Best Start Foods: evaluation

Methodology

This chapter provides an overview of the evaluation approach for Best Start Foods. It introduces the logic model and research questions driving the evaluation activities, and gives a summary of the different data sources used.

Overview of evaluation design and logic model

The Best Start Foods policy evaluation uses a "theory of change" model. The model shows the mechanisms whereby interventions (such as Best Start Foods) have a chain of immediate, short-term, and medium-term outcomes. If these outcomes are achieved, they can contribute to wider Scottish Government policy impacts in the long-term.

The Scottish Government's long-term policy impacts (e.g. reduced health inequalities) will take time to determine. They will also be affected by a range of factors in addition to Best Start Foods, making it difficult to measure and attribute changes specifically to that payment. However, the achievement of short-term policy outcomes (e.g. mothers and children eat more healthy foods) could reasonably be expected to contribute to these wider outcomes.

The logic model for Best Start Foods is below at Figure 1:

Immediate outcomes (Process Evaluation)

• Best Start Foods is well promoted
• Best Start Foods and its eligibility criteria are well understood
• Best Start Foods is taken up
• Making an application is clear and easy
• Applications and payments are processed in a timely manner
• Recipients treated with dignity, fairness and respect
• Card reduces stigma and is easy to use
• Card provides access to a wide range of retailers
• Card provides access to a wide range of healthy foods

Short-term outcomes (Policy Evaluation)

• Healthy foods are more affordable
• Supports healthier shopping habits and meal planning
• Mothers and children eat more healthy foods
• Reduced pressure on household finances

Medium-term outcomes (Policy Evaluation)

• Reduced incidence of food insecurity
• Increased healthy eating behaviours
• Improved health and wellbeing for children and mothers

Long-term outcomes and impacts

The long-term impacts in the logic model (reduced health inequalities and reduced child poverty) relate not only to Best Start Foods but to the wider government outcomes for children and their families, and are influenced by all social security interventions, as well as other interventions designed to support families with children across the Scottish Government. As such, Best Start Foods will play an important, but not exclusive, role in contributing to these.

Evaluation questions

Below are the key questions that informed the evaluation design:

1. To what extent did Best Start Foods achieve its immediate, short-term, and medium-term policy outcomes?

2. Is there any evidence of Best Start Foods contributing to long-term government policy impacts of (a) reduced health inequalities and (b) reduced child poverty?

3. What are the implications of the evaluation findings for future policy development?

Summary of data sources

In accordance with the evaluation strategy^[6], the evidence used in this report was drawn from multiple data sources, described below:

Bespoke commissioned research

ScotCen was commissioned by the Scottish Government to conduct qualitative research with Best Start Foods recipients, healthcare professionals who work with recipients, and food retailers. The research mainly focused on peoples' experience of receiving the benefit, and how this mapped on to Best Start Foods policy objectives (i.e. the immediate, short-term and medium-term outcomes outlined above). However, it also explored additional aims, such as potential barriers to applying for the benefit.

In-depth qualitative interviews were conducted with 33 Best Start Foods recipients across Scotland. Additionally, 5 interviews were held with healthcare professionals and 9 interviews were held with retailers. Fieldwork took place via telephone or video call between December 2021 and February 2022.^[8] The main findings from the commissioned research are presented in this report to provide a fuller understanding of the implementation and impact of Best Start Foods at this early stage of delivery. The full report from ScotCen, including more details on research methods and sample demographics, is available at Annex B.

Official Statistics

Social Security Scotland collects information on applications, payments, and recipients in the process of delivering the benefits. Some of this information is published online as Official Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Best Start Grant and Best Start Foods: high level statistics to 28 February 2022
• Social Security Scotland client diversity and equalities analysis to May 2021
• Social Security Scotland feedback: statistics to 30 September 2021

Of these three publications, the high level statistics data is used most frequently, and is hereafter referred to as "Official Statistics". The other publications are named in full when they are cited.

Supplementary analysis was carried out for the purposes of this evaluation to provide additional information on approved applications for Best Start Foods. This included a more detailed breakdown of relationship status, application channels, processing time categories, kinship status, and payment type. This analysis was produced using the same data as the Best Start Grant and Best Start Foods: high level statistics to 28 February 2022 publication.

Supplementary analysis was also carried out using the data from the Social Security Scotland client diversity and equalities analysis to May 2021 publication. This included breakdowns for approved Best Start Foods applications on a number of applicant characterstics including gender, age, ethnicity, health conidition, gender identity, sexual orientation, and geography.

The full supplementary analysis tables used in this report are provided at Annex A.

Please also note the following technical points about how Official Statistics are presented throughout this report:

• Figures are rounded for disclosure control and may not sum due to rounding
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics
• Most results are presented to zero decimal places. '0%' should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by '-'.

More detailed figures and information about the Official Statistics used in this report are provided at Annex A.

Social Security Scotland Client Survey

The Social Security Scotland Client Survey ran August/September 2020 (round 1) and May/June 2021 (round 2). It was open to everyone who at that time had received either (a) a Social Security Scotland benefit, or (b) a successful decision on a benefit application from Social Security Scotland's inception in September 2018 to March 2021.

The Client Survey collected equalities and socio-economic information from respondents. It also asked about their experience of Social Security Scotland and receiving benefits. In total, the survey received 10,575 responses (around 4% of the total number of invites sent), of whom:

1. 971 had only applied for Best Start Foods/Best Start Grant (on the joint application form), as opposed to having applied for Best Start Foods/Best Start Grant and other Social Security Scotland benefits. Data collected from these respondents is used to assess peoples' experiences and views of applying for Best Start Foods/Best Start Grant on the joint application form. This is because their views and experiences specifically relate to applying for only these benefits,.

2. 78 had only received Best Start Foods i.e. as opposed to having received Best Start Foods and other Social Security Scotland benefits. Data collected from these respondents is used to assess peoples' experiences and views of receiving Best Start Foods payments. This is because their views and experiences specifically relate to receiving Best Start Foods, as opposed to receiving potentially numerous benefits.

Please also note the following technical points about how Client Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly, within the ranges shown.
• Most results to the closed questions are rounded to whole numbers. As such, results (e.g. those presented in tables) may not sum to 100% due to rounding.
• Most results are presented to zero decimal places. '0%' should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by '-'.

Limitations

This section explains what can and cannot be determined from the available data sources, and how this influences the extent to which conclusions can be drawn about the early impact of Best Start Foods.

Role of qualitative research: The evaluation is largely dependent on findings from qualitative research commissioned by the Scottish Government. This provides a rich and detailed insight into the impact of Best Start Foods on recipients with a range of personal and demographic characteristics, and also the views of health professionals and retailers. However, the research also has the following limitations:

1. The findings are not representative of all Best Start Foods recipients, because (a) while diverse, the overall sample of participants was small, and (b) participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.

2. Findings are based on the participants' perceptions of impact, rather than objective measures of impact.

These are standard limitations of qualitative work. More detail on limitations has been provided at Annex B.

Role of Social Security Scotland research: The Client Survey findings contained in this report are based on up to 971 responses from people who applied for Best Start Foods/Best Start Grant on the joint application form, or who received Best Start Foods (and the number of respondents providing a valid answer to each individual question/statement varies within the ranges shown throughout the report).

Although the survey results provide insight into the views and experiences of a substantial number of Social Security Scotland recipients, it should be cautioned that this represents a small fraction of the 185,725 Best Start Foods applicants to 28 February 2022, and that views are drawn from a self-selecting sample of applicants. No weighting had been applied to counteract potential response bias. Therefore, it cannot be assumed that the results represent the views of Best Start Foods recipients as a whole.

It should also be noted that the Client Survey was designed for all Social Security Scotland recipients, not just Best Start Foods applicants. This means that questions are worded in a general way to make them applicable to all benefits. Therefore, while it is a useful source of supplementary evidence, it is not specifically designed to gather the views and experiences of Best Start Foods recipients about the benefit.

Medium and long-term impacts will take time and additional data to determine: Understanding the true impact of Best Start Foods would involve measuring (a) progress towards medium-term policy outcomes, and (b) its lasting contribution to wider Scottish Government outcomes. Doing so requires suitable time to have passed, and for the latter in particular, it would involve isolating the influence of Best Start Foods from other contributing factors, such as wider social security benefits and other government interventions designed to support families with children. A step in this direction would be to gain access to data with appropriate outcome variables e.g. from population surveys, or further bespoke research. However, at the time of writing there are no concrete proposals on future evaluations of Best Start Foods.

More detail on options for extended policy evaluation is provided in the evaluation strategy report.^[6]