Scottish Rural Development Programme 2014-2020: Evaluation of Capital Grant Schemes – Main Report

2. Study method

Introduction

The evaluation comprised a mixed-methods approach as described in more detail below.

Secondary research

A wide range of existing information, documentation, and data was reviewed and synthesised, including:

• grant scheme application forms and guidance.
• SRDP 2014-2020 programme document.
• Annual Implementation Reports (AIRs).
• data provided by SG on applications received (and outcome), and information on grant recipients and supported projects.

Primary research

The primary research stage comprised the following elements.

Stakeholder interviews

Remote (on-line) interviews were undertaken with 29 partners and stakeholders. A topic guide was used to inform semi-structured conversations with stakeholders, and interviews lasted on average 40 minutes. Prior to interviews, an information sheet was shared with stakeholders that provided information on:

• the purpose of the evaluation.
• the topic areas of interest.
• consent.
• voluntary participation.
• how the information provided would be used and protected.
• publishing of the final report and summary.
• who to contact if more information was required.

Qualitative analysis of interview notes was undertaken to identify and report on common themes and messages.

Grant recipient survey

SG issued an introductory email to grant recipients on 13 May 2024 to make farmers and crofters aware of the evaluation, its purpose, and that EKOS would be in touch by email with an on-line survey to ask for their views and experiences of applying for, and using, the grant funding.

An on-line survey aimed at grant scheme recipients was then issued by EKOS to the same grant recipients on 15 May 2024, with a reminder email sent on 29 May 2024. A total of 1,688 emails were sent and we received 393 bounce-backs – the survey reached 1,295 grant recipients.

The EKOS emails included information on:

• the purpose of the survey.
• an estimation of how long the survey may take to complete.
• the closing date for responses.
• consent.
• how the information provided would be used and protected.
• a weblink to the privacy notice.

The survey included a question on whether the grant recipient would like to be notified by email when the final report and summary are published by SG and providing them with a weblink to the documents.

The on-line survey was promoted in a range of ways including:

• through the area offices, Farm Advisory Service, and Scottish Association of Young Farmers Clubs.
• on SG social media channels and the Rural Payments & Services webpage.
• in the Scottish Rural Network newsletter and website.
• EKOS asked external stakeholders consulted as part of the evaluation to promote the survey via social media channels, direct email and/or organisation newsletters – and many did.

The survey, which was hosted on SNAP software, closed on 12 June 2024. When the survey closed EKOS downloaded responses into Microsoft Excel for data review and cleaning prior to undertaking quantitative and qualitative data analysis.

A total of 225 responses were received from grant recipients. The number of validated responses is 199 - this represents a good response rate of 15%.

The number of validated responses is lower than the total number of survey responses received because:

• 18 farmers and crofters did not consent to taking part in the survey.
• eight responses were removed as the farmer or crofter only answered one or two questions.

Case studies

The RAG were asked to identify potential project case study interviews, and area offices were also asked for suggestions. A total of circa 50 potential case studies, predominantly CAGS, were identified. This was then short-listed in discussion with SG to six plus a few back-up case study contacts to ensure a spread by:

• capital grant scheme.
• geography.
• applicant type.
• project type.

Relevant SG contacts checked whether the farmer or crofter was happy to take part in a short case study interview and brokered an email introduction with the EKOS team. A member of the EKOS team then followed up directly with the farmer or crofter to arrange a convenient appointment.

Prior to the case study interview, an information sheet was shared with the farmer or crofter to explain:

• the purpose of the evaluation.
• the topic areas of interest.
• consent.
• voluntary participation.
• how the information provided would be used and protected.
• publishing of the final report and summary.
• who to contact if more information was required.

Six case study interviews were undertaken, and these took on average 30 minutes to complete. Interview notes were reviewed to help draft the case studies. A draft case study was shared with the interviewee for review and comment prior to including in this report.

Presentations

A presentation was held with the RAG on 27 June 2024 prior to submission of the draft report. Another presentation to a wider group of internal colleagues will be scheduled later in 2024.

Research limitations

The main limitations encountered in the design and conduct of the evaluation are that:

• it was not possible to issue the on-line survey to all grant recipients supported to date by the three capital grant schemes. SG provided contact details of grant recipients for 2018 only (NECGS) and from 2019 onwards (SFGS and CAGS). Data on the current SG APEX IT system are incomplete, as prior to 2019 information and data on the grant schemes was recorded in spreadsheets at an area office level (there are 16 area offices). Work is ongoing to combine the two data sets.
• the partial data set meant that we relied on high-level information provided by SG regarding the total number of applications approved, total grant amount awarded to projects, and financial claims/expenditure.
• in the grant recipient survey a drop-down list of local authority areas was provided for respondents to select from in response to a question about location of farm or croft. This was considered the simplest way to frame the question. Responses were then re-coded where possible by EKOS into the four regions following survey closure. Helensburgh and Lomond sits under South Western Scotland while the rest of Argyll and Bute sits under Highlands and Islands (H&I) – as such, respondents based in this local authority area do not fit neatly under one SRDP region. A ‘best fit’ approach was undertaken, and all Argyll and Bute respondents were categorised as H&I.
• given the self-selecting nature of on-line surveys it was not possible to ensure a representative sample of the wider population of grant recipients by grant scheme – however, the vast majority of responses are from CAGS grant recipients, and this is by far, the biggest scheme.
• it was not feasible to undertake an unsuccessful applicant on-line survey (that is, non-grant scheme beneficiaries) to establish the counterfactual position. As no contract was entered into between SG and unsuccessful applicants this limited the ability of SG to share contact details with third party organisations, such as EKOS, for the purpose of research and evaluation.