Job Start Payment: evaluation

Methodology

This chapter provides an overview of the evaluation approach for Job Start Payment. It introduces the logic model and research questions driving the evaluation activities, and gives a summary of the different data sources used.

Overview of evaluation design and logic model

The Job Start Payment policy evaluation is based on a "theory of change" logic model. The model shows the mechanisms whereby interventions (such as Job Start Payment) have a chain of immediate, short-term, and medium-term outcomes. If these are met, they can generate longer-term outcomes and contribute to wider Scottish Government policy impacts.

Job Start Payment's medium-term outcomes, and the Scottish Government's long-term policy impacts (e.g. increased youth employment), will take time to determine, and may require access to robust quantitative data that is not currently available. They will also be influenced by other social security policies, as well as other Scottish Government interventions designed to support people into employment - making it difficult to measure and attribute changes specifically to that payment. However, the achievement of shorter-term policy outcomes (e.g. supports young people to take-up employment) could reasonably be expected to contribute to these wider outcomes.

The logic model for Job Start Payment is below at Figure 1.

The immediate outcomes of Job Start Payment relate to the delivery process of the benefit – i.e. promoting and administering the benefit. The short-term and medium-term outcomes relate to policy outcomes, which are linked with the intended aims of Job Start Payment.

Immediate outcomes (Process Evaluation)

• Job Start Payment is well promoted
• Job Start Payment and its eligibility criteria are well understood
• Making an application is clear and easy
• Applications are well administered
• Job Start Payment is taken up

Short-term outcomes (Policy Evaluation)

• Supports young people to take-up employment
• Helps to meet the costs of starting a new job
• Increased confidence when starting a job
• Increased confidence about employment prospects

Medium-term outcomes (Policy Evaluation)

• Helps young people to sustain employment
• Helps young people to build social networks
• Improved lifestyle and participation in society
• Improved health and wellbeing for recipients' families

Long-term impacts

• Increased youth unemployment
• Reduced youth unemployment
• Reduced labour market inequalities

Evaluation questions

Below are the key questions that informed the evaluation design:

1. To what extent did Job Start Payment achieve its immediate, short-term, and medium-term policy outcomes?

2. Is there any evidence of Job Start Payment contributing to the long-term government policy impacts of increased youth employment, reduced youth unemployment, and reduced inequalities in the labour market?

3. What are the implications of the evaluation findings for future policy development?

Summary of data sources

In line with the evaluation strategy^[6], the evidence used in this report was drawn from multiple data sources, described below:

Bespoke commissioned research

Diffley Partnership was commissioned by the Scottish Government to conduct primarily qualitative research on Job Start Payment. Data was gathered via:

1. One-to-one interviews with 26 Job Start Payment recipients, 6 unsuccessful applicants, and 7 non-applicants.

2. One-to-one and double interviews with 2 third sector representatives and 6 employabilty and skills professionals.

3. Discussion groups with 7 Social Security Scotland staff.

The qualitative data was supplemented by a short survey of 159 young people. The survey was mostly used as a recruitment tool for the qualitative research. However, it also gives some insight into awarness of Job Start Payment amongst 16-24 year olds living in Scotland.

The comissioned research had the following aims:

• To explore people's experience of applying for and receiving the benefit, and how this mapped on to Job Start Payment policy objectives (i.e. the immediate, short-term and medium-term outcomes outlined above).
• To investigate (a) factors which could impact take-up of Job Start Payment amongst eligible young people, and (b) factors associated with application denial rates.

Fieldwork took place via telephone or online video chat between November 2021 and March 2022.^[8] The main findings from the commissioned research are presented in this report to provide a fuller understanding of the implementation and impact of Job Start Payment at this early stage of delivery. The full report from Diffley Partnership, including more information on research methods and sample demographics, is available at Annex B.

Official Statistics

Social Security Scotland collects information on applications, payments, and recipients in the process of delivering the benefits. Some of this information is published online as Official Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Job Start Payment – high level statistics to 31 March 2022
• Social Security Scotland client diversity and equalities analysis to May 2021

Of these publications, the high level statistics data is used most frequently, and is hereafter referred to as "Official Statistics". The client diversity and equalities analysis publication is named in full when cited.

Supplementary analysis was carried out for the purposes of this evaluation to provide additional information on approved applications for Job Start Payment. This included a more detailed breakdown of the parental status of applicants. This analysis was produced using the same data as the Job Start Payment – high level statistics to 31 March 2022 publication.

Supplementary analysis was also carried out using the data from the Social Security Scotland client diversity and equalities analysis to May 2021 publication. This included breakdowns for approved Job Start Payment applications on a number of applicant characterstics including gender, age, ethnicity, health conidition, gender identity, sexual orientation, and geography.

The full supplementary analysis tables used in this report are provided at Annex A.

Please also note the following technical points about how Official Statistics are presented throughout this report:

• Figures are rounded for disclosure control and may not sum due to rounding
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics
• Most results are presented to zero decimal places. '0%' should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by '-'.

Limitations

This section explains what can and cannot be determined from the available data sources, and how this influences the extent to which conclusions can be drawn about the early impact of Job Start Payment.

Limited quantitative data: The only quantitative data used are Official Statistics on applications and payments.The Social Security Scotland Client Survey was also examined to determine if it could be used to assess Job Start Payment against its its intended outcomes. The survey ran in August/September 2020 (round 1) and May/June 2021 (round 2). It was open to everyone who at that time had received either (a) a Social Security Scotland benefit, or (b) a successful decision on a benefit application from Social Security Scotland's inception in September 2018 to March 2021. However, the survey only received 31 responses from people who had applied for Job Start Payment, and more than half of these respondents had applied for Job Start Payment and at least one other benefit, meaning that their views and experiences would potentially relate to numerous benefits (as opposed to having only experienced applying for or receiving Job Start Payment). Therefore, the Client Survey has not been used as a source of information in this evaluation.

Role of qualitative research: The evaluation is largely dependent on findings from qualitative research commissioned by the Scottish Government. This provides a rich and detailed insight into the impact of Job Start Payment on recipients with a range of personal and demographic characteristics, and the views of different stakeholder groups. However, the research also has the following limitations:

1. The findings are not representative of all Job Start Payment recipients, because (a) while diverse, the overall sample of participants was small, and (b) participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.

2. Findings are based on the participants' perceptions of impact, rather than objective measures of impact.

These are standard limitations of qualitative work. More detail on limitations has been provided at Annex B.

Medium and long-term impacts will take time and additional data to determine: Understanding the true impact of Job Start Payment would involve measuring (a) progress towards medium-term policy outcomes, and (b) its lasting contribution to wider Scottish Government outcomes. Doing so would require suitable time to have passed, and would involve isolating the influence of Job Start Payment from other contributing factors, such as wider social security benefits and other government interventions designed to support young people entering employment. A step in this direction would be to gain access to data with appropriate outcome variables e.g. from population surveys, or further bespoke research. However, at the time of writing there are no concrete proposals on future evaluations of Job Start Payment.

More detail on options for extended policy evaluation is provided in the evaluation strategy report.^{Error! Bookmark not defined.}