Scottish Rural Development Programme 2014-2020: ex-post evaluation - main report

2. Evaluation methodology

Introduction

This chapter provides details of the mixed-methods approach undertaken for the evaluation.

Secondary research

Most of the ex-post evaluation requirement was to be fulfilled through secondary analysis of existing data, evaluation reports, and other reports. The evaluation largely relied on insights and findings captured during the course of the SRDP 2014-2020 delivery period.

A range of existing information, documentation, and data was reviewed and synthesised, including:

• SRDP operational programme documents.
• SRDP scheme level evaluation reports.
• Annual Implementation Reports (AIRs), including the final 2023 AIR – these reports were based on a calendar year and provided a regular overview on the SRDP’s status and progress against targets.

The SRDP 2014-2020 Programme Document (page 805) sets out the process put in place by SG to review and monitor the quality of the implementation of the SRDP. This function was undertaken by the Rural Development Operational Committee (RDOC). Its role was to monitor progress towards the overall strategic direction set out in the Partnership Agreement (PA), and to advise Scottish Ministers and the MAs on any adjustments required to programmes, schemes or strategic interventions in order to ensure and maintain performance towards the targets and objectives of the PA. Among other things, the RDOC was required to consider and approve the AIRs. Following this, the AIR was then submitted to the EC for approval.

The purpose of the secondary research stage was to: take stock of existing information, data, and evaluation evidence; understand the CEQs and any existing reporting against these questions; and understand how SRDP schemes contribute to FAs. This process helped to build the evidence base for the evaluation and identified areas which required further exploration during the stakeholder interview stage, to fill gaps in intelligence to address the CEQs (where possible).

Stakeholder interviews

A targeted programme of stakeholder interviews was undertaken during September and October 2024.

Remote (on-line) interviews were undertaken with 18 people, in the main internal SG contacts.

A topic guide was used to inform semi-structured conversations with stakeholders, and interviews lasted on average 30 minutes. The conversations covered:

• contribution of SRDP schemes to relevant FAs.
• general views on how the scheme performed – what worked well and less well (and why), main barriers/challenges, key enablers.
• lessons learned.
• what has happened to the scheme since European funding came to an end.
• whether any wider information and data was held on the scheme that could inform the ex-post evaluation.

Qualitative analysis of interview notes was undertaken to identify key messages.

Research challenges and limitations

Limitations encountered in the design and conduct of the ex-post evaluation are described below.

Staff turnover within SG and across the various SRDP 2014-2020 schemes meant that some individuals involved in the design, delivery, and/or management of the schemes were no longer in the same post at the time of the ex-post evaluation. Some interviews were undertaken with individuals who had been involved from partway through, towards the end of the programme delivery period, or from when some schemes subsequently became part of the SRDP domestic programme.

SG was required to answer the 30 CEQs in enhanced AIRs (2016 and 2018), and these provide useful information which has been updated (where possible) as part of the ex-post evaluation. A mid-term evaluation of the SRDP was not undertaken given the EC requirement to submit enhanced AIRs. While data to inform the CEQs were not routinely collated, most CEQs were comprehensively answered in the enhanced AIR 2018^[4] and this identified which schemes were programmed under each FA and related activities. Limited impact data was presented given the relatively early stage the SRDP was at. Some of these data limitations extended to the ability to fully report performance against EC level CAP impact indicators, which has resulted in some reporting gaps in the ex-post evaluation.

During the 2014-2020 Programme period, some of the realised expenditure was for on-going commitments made under the 2007-2013 Programme, when there were different reporting requirements. It has not therefore been possible to populate some of the monitoring tables with the required data.

Almost all SRDP 2014-2020 schemes have been subject to an internal or external evaluation or review, and many reports are published and in the public domain. There are a couple of exceptions.^[5] Further, at the time of ex-post programme evaluation reporting, a couple of scheme evaluation reports have yet to be finalised and published.^[6]

Points to note about the existing evaluation evidence base include that:

• it was highly varied in nature, quality, and scope.
• most evaluations were undertaken when schemes were live and supported projects and activities were not all complete – there is often a time lag or delay between action and its resulting outcome.
• evaluations were not necessarily designed with relevant CEQs in mind or reported against relevant FAs - that is, this was not necessarily part of the scope of all evaluations.
• some evaluations were process evaluations and assessed how the scheme was set up, managed, and delivered, rather than also having a focus on impact, while others were simply a series of qualitative case studies – these evaluations did not all provide the data needed to answer the relevant CEQs comprehensively.
• in the main the evaluation reports did not provide further insights to help complete EC required data tables, albeit there were some exceptions.