Rural Scotland - skills action plan: process evaluation 2019 to 2021

Chapter 2: Methodology

The evaluation was conducted by IFF Research, with support provided by a Research Advisory Group which included representatives from Skills Development Scotland (SDS) and Scottish Government. The evaluation consisted of a literature review and 30 qualitative depth interviews with key stakeholders.

Introduction

This chapter sets out the three research questions for the evaluation. It then details the full methodology and sampling approach used in the evaluation as well as outlining the approach to analysis.

Evaluation aims and objectives

In 2021, the Scottish Government and Skills Development Scotland commissioned IFF Research to conduct a process evaluation of the Skills Action Plan for Rural Scotland. IFF was supported through this process by a Research Advisory Group, which included representatives from SDS and Scottish Government.

The evaluation focused on three research questions:

1. How effective has the Implementation Steering Group (ISG) been at driving actions?

Specific topic areas included: how the partnership has evolved in terms of its role and remit; what value the partnership adds; and what evidence there is of the ways in which the group is working in partnership to deliver a more coordinated response to skills issues.

2. To what extent has the SAPRS delivered on the five broad priority areas agreed?

Specific topic areas explored included: recognition of the changing landscape; a reflection of the activities undertaken to implement the SAPRS; and what has been achieved (for example, new activities/COVID-19 specific activities; enhanced partnerships and collaborative working; improved access to skills provision; sharing of best practice).

3. Has the creation of SAPRS contributed to additional activity/ investment?

Specific topic areas included: how the partnership/plan is bringing about additionality or more effective activity; as well as examples of this in practice.

The process evaluation examined the reach of the SAPRS in terms of the sectors, groups and places that it was looking to support, the extent to which partners and stakeholders were satisfied with SAPRS activity, and any effects that taking this approach has had on the rural economy.

The findings of the research will be of particular value for informing the future direction of both the SAPRS and the ISG. These considerations, including the value of the approach, ways of working and responses to lessons learned, are set out in the Recommendations chapter.

Method

The evaluation methodology involved two stages: a review of existing literature, and 30 stakeholder interviews.

Literature review

The literature review sought to gain a broader understanding of the progress of the SAPRS, and to inform the topic guide for the qualitative interviews. This enabled the researchers carrying out the evaluation to contextualise stakeholder comments, situate the SAPRS within its broader context, and explore to what extent the SAPRS delivered on the five broad priority areas.

Key documents reviewed at this stage included two SAPRS Progress Update Reports^[18], compiled in Spring 2021 and Spring 2022 by SDS for members of the ISG. These were designed to be shared with partners and to provide learning for future activities. The two progress reports, alongside documentation relating to individual activities linked to the SAPRS, provided detail on the first research objective: "To what extent has the SAPRS delivered on the five broad priority areas agreed?".

In addition to these reports, the literature review included ISG meeting minutes^[19], and the results of an ISG members survey^[20] conducted in August-September 2021, which helped further understanding of the second objective: "How effective has the Implementation Steering Group (ISG) been at driving actions?"

Finally, the literature review explored research and insight papers covering broader themes such as the impact of coronavirus (COVID-19), and the labour market in Scotland. These provided important context for the Background chapter of this report.

Stakeholder interviews

Between December 2021 and June 2022 a total of 30 interviews were conducted with relevant stakeholders, 23 of which were with members of the Implementation Steering Group (ISG). ISG members represent organisations with a key stake in both the identification of challenges affecting skills in rural Scotland, and the means of addressing them. A group of wider stakeholders – those outside the ISG but representing organisations involved in activities contained within the SAPRS – were also invited to take part in interviews. This resulted in an additional 7 interviews. Organisations represented across the ISG and more widely included local councils, funding bodies, education and training bodies, representatives from sectors across the rural economy, Scottish Government and SDS.

Members of the Research Advisory Group initially identified stakeholders suitable for interview. Participants were selected to ensure that a good range of stakeholders, with different views and perspectives, and from a range of industries, were included in the interviews. Of the suggested participants, only a small number were unable or unwilling to participate. These were not generally skewed towards any particular industry. Although neither of the representatives from Highlands and Islands Enterprise were able to take part, this region was represented by other participants.

Interviews were carried out remotely, using Microsoft Teams, Zoom or telephone, and lasted for approximately 45 minutes. Interviewers used a semi-structured topic guide (see Annex A) designed by IFF, and agreed with the Research Advisory Group. This focused on participants' views on the following themes:

• Priority areas for the SAPRS
• Emerging issues and challenges affecting SAPRS content and delivery
• Reflections on ISG collaboration and outputs
• Where the SAPRS has added value
• Key lessons learned

All interviews were recorded (with participant permission) and notes written up based on the recordings into an analysis framework. The framework was structured around headings/themes relating to the objectives (for example, "Progress in priority areas", "Partnership working within the ISG" and so on), and included relevant additional information (such as interviewees' backgrounds).

Members of the IFF team then carried out detailed individual analysis of a specific topic area and wrote up analysis notes. Subsequently, the research team met for a full analysis session, where each team member led on their topic area, followed by full discussion and additional observations from the team. This session was then written up into an analysis template, which formed the basis of this report.