Regional Land Use Partnerships: phase 1 process evaluation - final report

3. Overview of evaluation methodology

a. Logic model and Theory of Change model

To understand the full context of the RLUP development process, we structured our evaluation method around a logic model (LM). The logic model (see Figure 3a) is a schematic which outlines how RLUPs as an intervention collate a series of inputs (resources) to enable outputs (activities), which then forward the delivery of a specific set of outcomes. We have refined an overarching LM for the RLUP pilot work with a primary focus on Phase 1, which has been used to structure our evaluation.

While policy interventions are designed to lead to certain outcomes, realising these on the ground can be uncertain due to influences from a range of factors. A Theory of Change model (ToC) is a way of predicting outcomes of policy actions by capturing baseline conditions and identifying the factors which will influence results.^[1] In this evaluation, we revised the LM into a ToC diagram (see Appendix B: Detail on evaluation methodology) to provide a concise and interlinked ‘theory of change’ pathway from the inception of RLUPs towards long-term outcomes. These two linked schematics are cross-referenced within this report and provide a useful, visual guide to the written outputs.

b. Data collection

i. Desk-based research

Initial data collection involved reviewing documentation pertaining to RLUP development to date. This included:

• Relevant Scottish Government policies
• The body of work surrounding the previous 2013-2015 Land Use Strategy pilots
• Advice from the Scottish Land Commission
• Academic literature
• The original project specifications for these pilots
• Meeting agendas and minutes
• Other reports and updates.

This information review enabled us to establish an understanding of local circumstances (baselines) before meeting with representatives from the pilot regions. This also ensured the evaluation captured the separate starting points of each region. This initial review enabled efficient use of interview time such that these sessions could focus on exploring the qualitative aspects and lessons learned, following confirmation of the initial baseline data for each region.

ii. Interviews with pilots

Interview questions were developed by the evaluation team and refined in consultation with the SG policy team. For more detail on the questions and wider methodology, see Appendix B: Detail on evaluation methodology and Appendix D: Full list of interview questions. The pilots were contacted via emails provided by the SG RLUPs policy team. Interviews were scheduled with pilot representatives and all participants were provided with a project information sheet and each signed a data consent form. Interviews ran between 12 Jan – 2 Feb 2022 and lasted between 50 and 94 minutes. There were five interviews in total (one session with each pilot region); two interviews were one-on-one and three were in groups, with 2-4 pilot representatives attending. Eleven pilot representatives were interviewed in total. All interviews were conducted via MS Teams by the Principal Investigator from the SAC Consulting evaluation team.

c. Data analysis

Transcripts of the interviews and the background material were collated and analysed using a thematic analysis process. This involved reviewing the transcripts to develop a set of overarching codes to identify key strengths, weaknesses, opportunities and threats (SWOT) relating to the four main themes of the evaluation (governance, stakeholder engagement, natural capital, and data) developed from Ministerial expectations, outlined in the Bute House Agreement. In addition, results were considered under a ‘pilot process’ theme to capture the practical lessons around taking a pilot approach to this new RLUP policy mechanism. The thematic analysis identified emerging commonalities between pilots regarding strengths and weaknesses, and highlighted individual innovations and lessons learned. These are reported in Results and discussion, below. For more on the development of the methods used in this evaluation, see Appendix B: Detail on evaluation methodology.

d. Inception and Closing Workshops

Our evaluation was bookended by workshops led by the evaluation team and attended by the SG RLUP policy team and the entire cohort of pilots. The inception workshop of 8^th December 2021 included a presentation by the evaluation team on the process, the inputs required from pilots, and the desired outcomes. It also included a discussion around the enabling factors and existing activities taking place within each pilot. Since this evaluation will signpost future opportunities and threats for RLUPs, a key value added was the creation of space to discuss participants’ reflections on the process. Pilots came forward in round-table fashion with updates regarding ongoing activities. This was an opportunity to introduce the themes of the evaluation and get pilots talking and thinking about the ideas the interviews would interrogate.

Our closing workshop on 14 March 2022 provided an opportunity for pilot regions to revisit the priority outcomes for the evaluation. The evaluation team presented the headline findings from the data collection and thematic analysis process. The following plenary discussion with the pilots ensured validation of the results. These workshops were useful in that they made the evaluation process transparent to pilots and allowed them to raise any concerns, which led to increased trust and buy-in to the process.