Evaluation of the Regional Land Use Framework Pilots

2. Evaluation Methodology

2.1 Overview of Approach

The evaluation process began in September 2013, with an inception meeting involving CAG Consultants, the project leads from the pilot areas and representatives from the Scottish Government. Following this meeting, a detailed evaluation plan was prepared and agreed with the Scottish Government in January 2014. The evaluation plan set out a primarily qualitative approach, with a focus on process evaluation. Over the course of the evaluation, the following methods were used to gather data and information:

• Desk-based research and analysis of project documentation.
• Semi-structured interviews with pilot project managers and boards.
• Semi-structured interviews with stakeholders.
• Facilitated workshops.

The research was comprised of 4 evaluation stages.

Table 1: Project evaluation stages

Evaluation stage	Activity
Evaluation Stage 1 (completed May 2014)	Evaluation plan design Policy mapping and baseline recording Board meeting attendance Stakeholder telephone interviews Stage 1 Interim report
Evaluation Stage 2 (completed October 2014)	Review of project documentation Stakeholder mapping Board meeting attendance Stage 2 Interim report
Evaluation Stage 3 (completed March 2015)	Workshops with project management teams Workshops with key stakeholders Supplementary telephone interviews with stakeholders Stage 3 Workshop reports
Evaluation Stage 4 (completed May 2015)	Desk top review Clarification telephone interviews with pilot project staff Final report

2.2 Evaluation Methods

2.2.1 Desk-based research

Desk-based research was undertaken throughout the evaluation. As part of stage 1 of the evaluation a member of the research team worked with the pilots to develop a database of stakeholders in each area. The database comprised the local authority staff involved, land managers (private, public and third sector), policy makers, user groups (recreational and other special interest groups) and local communities.

This database was subsequently revised (in stages 2 and 3) and updated to produce a database of all of the stakeholders that were involved during the development of the pilots. The database included:

• details of the individuals and organisations involved in the pilot and the nature of their engagement;
• the development stage at which individual stakeholders became involved with the pilot;
• identification of the primary interests of stakeholders; and
• an attempt to classify stakeholders on the basis of how their primary interest relates to the four forms of high-level ecosystem services.

During stage 2, a desk exercise was initiated to record all of the main outputs of the pilots (e.g. reports, stakeholder events) from inception through to the conclusion of the pilot framework development process in March 2015. This product record was prepared as an Excel database and used to identify products and content particularly relevant to the research questions. This record is presented in table form in Appendix 2 of this report.

A final desk research exercise was undertaken in evaluation stage 4 with this being focused on a review of the final reports and GIS tools produced by the pilots, together with other relevant documentation as required to address the research questions.

2.2.2 Semi-structured Interviews with Stakeholders

In stage 1, CAG research team members undertook a series of telephone interviews with stakeholders using a questionnaire designed to explore attitudes, expectations, perceived risks and concerns at an early stage of the pilots' work. The questionnaire was agreed with the Scottish Government and the two pilot projects.

The interviews included a set of questions and statements with multiple choice response options (e.g. strongly agree to strongly disagree) enabling a categorisation of responses (see Appendix 3). The survey also gave the opportunity for respondents to explain and elaborate on their multiple choice responses in their own words, providing further information on attitudes, expectations and experiences of the pilots by respondents.

Each telephone interview lasted between 30 minutes and 1 hour, with the interviewer taking notes during the conversation.

The original intention was to interview between 25 and 30 stakeholders in each pilot area. However, in practice, it was only possible to identify 21 potential interviewees in Aberdeenshire and 28 in the Borders. All of the identified stakeholders were invited to participate in the evaluation of the pilots; 20 from Aberdeenshire and 21 from the Borders agreed to be interviewed. Interviews took place between 12th February and 14th March 2014.

2.2.3 Facilitated Workshops

Over the course of the project CAG staff attended a number of meetings involving project management staff and other key stakeholders (including representatives from wildlife, forestry, agriculture, environmental bodies). It had originally been intended that the evaluation would involve a series of stand-alone workshops, but in practice it was decided that this would prove impractical owing to the additional time demand it would impose on project staff and in particular stakeholders. Consequently it was agreed that researchers would look to 'piggyback' their sessions either before, after or as part of the regular programme of meetings being run by the two pilots.

During stage 2 of the evaluation, team members attended meetings with the two pilot steering boards in Aberdeenshire on the 25th September in 2014, and in the Scottish Borders on 6th October 2014.

In addition to observing the meetings, the researcher was provided with an opportunity to probe key points and issues arising and to pose a series of questions to board members. These questions were pre-agreed with the Scottish Government, and the pilots advised of the questions in advance (see Appendix 4 for further details).

Following the board meetings, the project leads were able to make themselves available for more in-depth discussion. Project leads subsequently also provided additional commentary and clarification via email and over the telephone.

In stage 3 of the evaluation two workshop sessions were held in each of the pilot areas. These took place on the 5th March 2015 (Scottish Borders) and 20th March 2015 (Aberdeenshire). The first workshop was restricted to members of the project management team (those involved in day to day operational activity or project oversight), the second was attended either by members of the project board (in the case of Aberdeenshire - this includes local authority and external stakeholders responsible for strategic oversight) or members of the pilot's key stakeholder group (in the Scottish Borders - representatives of various land management interests who acted as a key reference group)^[3].

Each workshop session focused on three headline topics and related sets of sub-questions (see Appendix 5). The questions were developed by CAG and informed by the Scottish Government; subsequently they were circulated by the pilots to enable participants to prepare for the sessions. The workshop questions were devised to generate insight in relation to the evaluation research questions and included a series of sub-questions relating to issues that emerged during the first and second rounds of the evaluation. Details of the questions can be found in Appendix 5.

The workshops were facilitated by CAG researchers and were run on an informal basis to encourage an open debate and exchange of views. Key discussion points were recorded by the facilitation team during the workshops and written up following the event. Owing to low attendance rates, particularly from amongst the private sector, researchers offered to hold semi-structured telephone interviews with representatives from the private sector, following the stakeholder workshop. The four participants in the telephone interviews were selected from lists of relevant stakeholders identified by the pilot project management teams.

2.3 Evaluation Risks and Challenges

The Evaluation Plan identified a number of risks for the evaluation, these are outlined below.

Stakeholder consultation fatigue

At the evaluation inception meeting, the pilots underlined the challenge they faced in managing and maintaining stakeholder engagement. In response, it was agreed that contact with stakeholders for the purposes of evaluation should be managed to ensure that the time burden was minimised. This was deemed to be critical, given the importance of stakeholder engagement to the pilot development process.

In practice, it was necessary to amend the Evaluation Plan several times to take account of the restricted availability of stakeholders.

• A proposal for first year evaluation workshops in the pilot areas was dropped. Instead, a telephone survey was conducted during stage 1 and members of the research team observed one pilot steering group meeting in each area.
• Workshops in stage 3 were supplemented by follow-up phone calls owing to the unavailability of some key stakeholders. This was necessary to strengthen input from the private sector.
• The original evaluation time plan was revised to better fit with pilot project plans. The timings of activities were adjusted so that meetings with stakeholders for evaluation purposes would coincide with established board and stakeholder meeting dates. Contact for evaluation purposes was planned to avoid times when consultations on the frameworks were taking place so that the processes were not confused.

Short Timeframes

All parties recognised that the limited lifespan of the pilots meant that impact would be difficult to deliver and record. This risk was mitigated by an evaluation design that considered the potential for impact. However, as potential is inherently difficult to measure, the assessment of cost and benefit has largely been restricted to a descriptive exercise.

Potential for suppression of innovation

It was necessary that the pilots felt enabled to test and develop solutions to constraints and challenges, and were allowed the freedom to tackle challenging objectives for the partnership. For this reason, the evaluation was designed to avoid the perception of policing and the consequent risk of suppressing innovation.