Child Winter Heating Assistance: evaluation report

Methodology

This chapter provides an overview of the evaluation approach for CWHA. It describes short-term, medium-term, and long-term goals, outlines research questions driving the evaluation activities, and provides a summary of the different data sources used.

Overview of evaluation design

This evaluation will utilise relevant policy goals to estimate the impact of CWHA. The short-term goals are intended to be immediate and easy to measure as a direct impact of CWHA. Whereas the medium and especially the longer term goals are more distal and are also influenced by other factors.

Relevant short-term goals:

• Household income is increased across the winter months
• Reductions in financial pressure regarding increased heating costs
• Increased ability to heat the home during the winter months
• Mitigation of additional heating costs over the winter

Relevant medium-term goals:

• Improved health and wellbeing
• Mitigating against financial difficulties/insecurities over time
• Supporting people with payment costs at a point in time when it is most needed

Relevant long-term goals:

• Children and young people have the best start in life
• Children and young people have access to a full range of opportunities

It should be noted that the medium and long-term goals relate not only to CWHA but to the wider government goals for children, young people and their families, and are influenced by all social security benefits, as well as other interventions designed to support families with children across the Scottish Government. As such, CWHA will play an important, but not exclusive, role in contributing to these anticipated goals.

Summary of Data Sources

In accordance with the evaluation strategy[8], the evidence used in this report was drawn from multiple data sources, described below:

Bespoke commissioned research

Axiom was commissioned by the Scottish Government to conduct qualitative research with participants who had received CWHA on their experience of this, the impact that receiving the payment had on them, and suggested improvements to the payment process. The evaluation aimed to map findings onto the short-term as well as the medium/ longer-term policy aims of CWHA. It also aimed to gain an insight into how to improve the client experience, including receiving the payment so that it better addresses the CWHA policy aims.

In-depth interviews, lasting around 45 minutes, were conducted with 19 CWHA recipients across Scotland. Note that in this research, by using the term “recipients” we refer to parents, carers or guardians who receive the CWHA payment on behalf of the child and/or young person who is entitled to it. All interviews took place over the telephone due to preference, and fieldwork was conducted in February 2022. The findings from the commissioned research will be included under the relevant sections in this evaluation report in order to highlight how CWHA is addressing its policy aims. The full bespoke research project from Axiom is available in Annex A.

Official statistics

Social Security Scotland publishes Official Statistics which include information on applications, payments, and clients. The following publications are used as sources of evidence in this report:

• Child Winter Heating Assistance statistics: Winter 2021/2022
• Child Winter Heating Assistance: Winter 2020/2021

Please note that the latest publication (Child Winter Heating Assistance statistics: Winter 2021/2022) has updated information for winter 2020/21 within it.

Please also note the following technical points about how Official Statistics are presented throughout this report:

• Figures are rounded for disclosure control and may not sum due to rounding
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics.
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-‘.

Social Security Scotland Client Survey

This report draws on data conducted by the Client Survey team in Social Security Scotland. The Social Security Client Survey was open to everyone who at that time had received either (a) a Social Security Scotland benefit, or (b) a successful decision on a benefit application from Social Security Scotland’s inception. The Survey ran August/September 2020 (round 1) and May/June 2021 (round 2), however, only round 2 is significant to this report as CWHA was not launched until after the first round of the survey.

The Client Survey collected equalities and socio-economic information from respondents. It also asked about their experience of Social Security Scotland and receiving benefits. In total, the second round of the survey received 7,322 responses (around 7.4% of the total number of invites sent), of whom:

1. 836 had received CWHA and the qualifying benefit only, or were also in receipt of other benefits from Social Security Scotland.

2. 206 had received CWHA and the qualifying benefit only.

This report will discuss the Client Survey data concerning those who received CWHA and the qualifying benefit only. Although this is a relatively small subsample of the client survey (n = 206), it is necessary because experiences of other types of benefit may vary. Therefore, reporting on the experiences of those who received CWHA only will give the most robust insight into CWHA experiences specifically.

Please also note the following technical points about how Client Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly, within the ranges shown (e.g., 197-206 recipients of CWHA).
• Most results to the closed questions are rounded to whole numbers. As such, results (e.g. those presented in tables) may not sum to 100% due to rounding.
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-‘.

Limitations

This section discusses what can and cannot be determined from the data sources that are available for this evaluation. It will also highlight how this influences the extent to which robust conclusions can be drawn regarding the impacts CWHA has on its policy goals.

Commissioned qualitative research: This evaluation is largely dependent on findings from externally commissioned work. The work aimed to capture a variety of experiences based on rurality and socioeconomic status in order to provide context and narrative to the quantitative findings from the other data sources on CWHA. However, the work did have the following limitations:

• Though not the goal, the findings are not representative of all CWHA recipients because the overall sample was small, and participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.
• The sample approached for recruitment also involved those who were members of the Client Panels, not the wider client base of CWHA recipients. This meant that the sample was skewed towards those who are more generally motivated to participate in research. These people may have some other shared characteristics that in turn may skew the results.
• Findings are based on participants’ perceptions of impact, rather than objective measures of impact.

These are the main limitations from the commissioned qualitative work, but please see Annex A the full commissioned report for a more detailed discussion of the relevant limitations.

Social Security Scotland research: While the commissioned research was explicitly conducted for the current evaluation, Social Security Scotland research (the Client Survey and Official Statistics) was conducted to provide information relevant to operations and policy and does not therefore serve as a direct measurement of the policy goals concerning this evaluation.

On a similar note, the Client Survey was designed for all Social Security clients and not just those who were CWHA recipients. This means that the questions were worded in a general way to make them applicable to all benefits. Therefore, while it is a useful source of supplementary evidence, it is not explicitly designed to gather the views and experiences of CWHA recipients alone.

Furthermore, the Client Survey statistics discussed in this report were based on a small sample in the range of 197-206 recipients of CWHA, the number of which depended on the number of valid responses to the question being asked. Overall, these numbers represent a small proportion of the overall number of children or young people entitled to CWHA in winter 2020/21 (N = 18,315) and winter 2021/22 (N = 19,865). This sample was also self-selecting and no weighting had been conducted to control for potential response bias.[9] It cannot therefore be assumed that the results represent the CWHA client base as a whole.

Medium and long term impacts: Understanding the true impact of CWHA would require a rounded evaluation of progress towards its medium-term goals, and also its longer-term contribution to the wider Scottish Government impact of children and young people having access to a full range of opportunities, and children and young people having the best start in life. Doing so requires suitable time to have passed, and for the latter in particular, it would involve isolating the influence of CWHA from other contributing factors, such as wider social security benefits. A step in this direction would be to gain access to data with appropriate outcome variables e.g., from population surveys, or further bespoke research. However, at the time of writing there are no concrete proposals on future evaluations of CWHA. More details on options for extended policy evaluation are provided in the evaluation strategy report.[10]