Community Empowerment (Scotland) Act 2015: asset transfer requests - evaluation

3. Approach to this evaluation

3.1. Evaluation stages

The evaluation was carried out in four consecutive stages, presented in Table 1. Myers et al.’s (2017) Theory of Change (Figure 1) informed the design, implementation and thematic framework adopted for the research.

3.2. Methods

The evaluation team collected primary (individual/group interviews and participant observations), and secondary (annual reports and supporting documents from relevant authorities) data to answer the research questions. These are discussed below together with the sampling strategy, our approach to data analysis and ethical considerations.

3.2.1. Interviews with relevant authorities

During Stage 1 (Table 1), the research team undertook in-depth interviews with 13 participants from 11 relevant authorities named in the Act, and one key stakeholder from a national asset transfer support organisation. The topic guides for these interviews were developed to generate data related to the interpretation and implementation of the Act. The interviews with individuals responsible for asset transfer request processes and/or supporting communities to submit asset transfer requests helped to gain an understanding of how the Act has been implemented by relevant authorities. Further questions regarding specific asset transfer requests were used to build case studies that gave insight into the potential for asset transfer requests to contribute to intermediate and longer-term outcomes in communities.

3.2.2. Interviews with community stakeholders

During Stage 2 (Table 1), the research team conducted one-to-one interviews with 12 community stakeholders involved in asset transfer request submissions. In addition to individual interviews, the research included one group interview with a further three participants from one community transfer body. In total, 15 community stakeholders were interviewed. The topic guides for these participants were developed to generate data related to the experience of the asset transfer request application process, the support offered by relevant authorities, actual and anticipated outcomes and perspectives on the meaning and possible measurement of community empowerment.

Table 1 Stages of the research, aims and activities

Stage 1 (Apr. 2018 – Sept. 2018)

Aim: identify activities and outputs related to asset transfer requests (ATR)

• Understanding implementation processes
• Identifying early patterns in ATR submissions across Scotland
• Highlighting how ATRs were being interpreted by relevant authorities (RAs)
• Exploring RA perspectives on potential outcomes

Data collection (primary)

• Interviews with RA representatives (n=13)
• Interview with one key stakeholder from a national ATR support organisation

Data collection and analysis (secondary; 2017-18)

• ATR reports and informal submissions (n=58) supplemented with available documents
• Identification of community transfer body (CTB) cases (Section 3.3.2)

Stage 2 (Oct. 2018 – May 2019)

Aim: to explore the experiences of community organisations leading ATRs

Data collection (primary):

• Interviews with community representatives (n=15)
  • participants (n=12) from eight CTBs
  • 1 group interview with participants (n=3) from one CTB (CTB 2)
• Participant observations (n=3)

Data collection and analysis (secondary; 2018-19)^[6]

• Formal ATR reports and informal submissions (n=42) supplemented with available documents

Stage 3 (Jun. 2019 – Sept. 2019)

Aim: Analysis of primary and secondary data.

• Comparative data analysis using 2017-18 and 2018-19 annual reports
• Themes identified across the qualitative data

Stage 4 (Oct. 2019 – Jan. 2020)

Aim: Final analysis

• Development of a revised Theory of Change
• Two stakeholder workshops held with asset transfer request stakeholders (30); emerging findings discussed contributing to strengthening a revised Theory of Change
• Producing final reports

3.2.3. Participant observation

During Stage 2 (Table 1), members of the research team attended events held by three community transfer bodies included in the sample. These were attended in order to observe support from communities and understand services being delivered by the community transfer bodies. Further, attending these events allowed the research team to have informal conversations with members of the wider community. One of the community organisations that had successfully completed an asset transfer request held regular meetings to manage the development of the asset. One of these meetings was attended, which offered the opportunity to ask questions and clarify plans and community transfer body decision-making. This gave an insight into the processes that follow the successful transfer of an asset and the ways that communities manage the various responsibilities, opportunities and challenges of an asset transfer (see Table 5 for details on participant observations).

3.2.4. Secondary data: Relevant authority annual reports

Secondary data collection spanned Stages 1–3. Two sets of relevant authority annual reports were collected and analysed for the evaluation. The first return of annual reports encompassed the period 1st April 2017 to 31st March 2018 while the second return covered 1st April 2018 to 31st March 2019.

Despite all relevant authorities having a statutory duty to publish annual reports which outline all asset transfer request activities, low publishing rates in both periods meant that additional steps were taken to collect data about asset transfer requests from relevant authorities. During Stage 1 of the evaluation, these steps included email communication from the Scottish Government, introducing relevant authorities to the evaluation team (April 2018) and reminding them of their statutory duty to publish (July 2018); and evaluation team email communication, requesting information on the intended online location of the published reports^[7] (June 2018) and requesting that all relevant authorities submit (July 2018). During Stages 2 and 3, the Scottish Government contacted relevant authorities, reminding them of their statutory duty to publish (May, July and August 2019).

The number of reports submitted by relevant authorities is presented in Table 6 and Table 7. Once available reports were collated, both sets of annual reports were analysed using the same systematic approach. Findings from this analysis of the annual reports are available on the Scottish Government website^[8].

In addition to the collection of annual reports, the research team gathered documents relating to asset transfer requests on relevant authority websites. Relevant documents included application forms, business plans and decision notices issued to inform community transfer bodies of the outcome of requests. Including this information has allowed for analysis of the types of organisations that are submitting requests, the purpose of the request and the type of asset. Where possible, the postcode of the requested asset was ascertained in order to map the asset transfer requests onto the Scottish Index of Multiple Deprivation (SIMD) – this to increase understanding about which communities are submitting asset transfer requests. Data were not available for all submitted asset transfer requests; where data are missing, this is noted in the tables and figures throughout the report.

3.3. Sampling

This evaluation adopted purposive sampling of relevant authorities, community transfer bodies and other stakeholders, with the aim of ensuring that the sample would have broad and diverse experience and knowledge of asset transfer requests.

3.3.1. Sampling of relevant authorities

Relevant authority interview participants were identified following the collation and analysis of relevant authority annual reports for the period 1^st April 2017 to 31^st March 2018, during Stage 1. Using information from the annual reports, we mapped out asset transfer request activities and identified key people responsible for asset transfer requests within relevant authorities. The selected participants occupy roles that are core to the development and delivery of asset transfer request processes within relevant authorities (Table 2).

Table 2 Interviewees per relevant authority (RA)

RA code	RA type	Number of interviewees
RA 1	Local authority	1
RA 2	Local authority	1
RA 3	Local authority	2
RA 4	Local authority	1
RA 5	Scottish Ministers/Other	1
RA 6	Education	1
RA 7	Transport	2
RA 8	Other	1
RA 9	Scottish Ministers/Other	1
RA 10	Scottish Ministers	1
RA 11	Health	1
Total		13

Although it was clear from the annual reports that the majority of asset transfer requests were received by relevant authorities (further information provided in Section 4.2), the sample was designed to include a range of relevant authority types (including local authorities, further education (colleges), regional transport partnerships, Scottish Ministers and ‘other’). The sample was also chosen to reflect the range of overall levels of asset transfer request activities. At the time of interviewing, two relevant authorities had not received any asset transfer requests. Another relevant authority had not received any formal applications at the time of the interview but noted, through their annual report, that they were involved in active discussions surrounding asset transfer requests and had received a number of enquiries. Given the significant scale of work undertaken prior to the submission of an asset transfer request, these relevant authorities were included in order to gain a fuller picture of the level of asset transfer request activities across Scotland.

3.3.2. Sampling of community transfer bodies

Overall, five community transfer bodies were included in the sample. To identify community transfer bodies for inclusion in the research, we followed a process in order to generate a list of possible case studies (see below). This process was conducted during Stage 1 of the evaluation as part of the secondary data analysis (Table 1):

• To identify community transfer bodies that would form the basis of the case studies, sampling began with an overview of the level of asset transfer activities across the different types of relevant authorities.
• Each relevant authority with experience of asset transfer requests was tabulated in order to identify the numbers of applications within each relevant authority. This generated a list of possible community transfer bodies to include in the sample.
• To identify potential community transfer bodies for case studies, the sampling parameters were narrowed to exclude asset transfer requests where decisions were ongoing and to focus on asset transfer requests where decisions had been confirmed.
• Each asset transfer request was categorised according to whether they were accepted or rejected; the type of asset requested; whether the request was for lease, ownership or management; and the SIMD decile associated with the postcode of the asset.

The final list, presented in Table 3, was selected to include representation across SIMD, type of asset, and rights requests (i.e. lease or ownership). This ensured that community transfer bodies represented a variety of experience and context.

Table 3 Community transfer bodies ( CTBs) – sample

CTB code	Whether ATR was accepted or refused	Asset type	Lease/ ownership	Asset SIMD
CTB 2	Accepted	Land	Ownership	6
CTB 7	Accepted	Building	Lease	10
CTB 9	Refused	Land	Ownership	3
CTB 10	Accepted	Building and land	Lease	8
CTB 11	Accepted	Building	Ownership	3

Three community transfer bodies listed in Table 3 were unable to participate in the study:

• CTB 9 did not want to take part as they had previously participated in research surrounding asset transfer requests. As a small, mainly volunteer-run organisation, they had limited capacity to be involved.
• The asset transfer of CTB 10 was concluded outside of the Act and beyond the scope of this evaluation.
• Following three attempts via email/ phone, there was no response from CTB 11.

After discussions with the Scottish Government, community transfer bodies 9, 10 and 11 were replaced with other community transfer bodies (Table 4). A revised sampling strategy was adopted to ensure that the sample included an asset transfer request received by a non-local authority relevant authority. The three alternative community transfer bodies were identified during Stage 2 of the evaluation as part of the secondary data analysis (Table 1). As such, the revised sample included community transfer bodies whose asset transfer requests were captured in 2018-19 annual reports.

Identifying potential interviewees from the wider community to explore the impacts of asset transfer requests was problematic because many asset transfers were still in progress, which means that it was not yet possible to assess the impacts of the transfer. Although all the selected case studies had received a final decision, not all of the assets had been officially transferred. Only two transferred assets were in use and running services during the evaluation. To address this, we identified a larger sample of community transfer bodies to participate in the research (Table 4).

Table 4 Community transfer bodies ( CTBs) – amended sample

CTB code	Relevant authority	Accepted/refused	Asset type	Lease/ ownership	Asset SIMD
CTB 1	Other	Refused	Building	Lease	5
CTB 2	Local Authority	Accepted (Also submitted a PR)	Land	Ownership	6
CTB 3	Local Authority	Accepted	Building	Ownership	2
CTB 4	Local Authority	Accepted	Building	Ownership	2
CTB 5	Other	Accepted	Land	Ownership	6
CTB 6	Local Authority	Accepted	Building	Ownership	2
CTB 7	Local Authority	Accepted	Land	Lease	10
CTB 8	Other	Accepted	Land	Ownership	5

From this sample of community transfer bodies, interviews were carried out with 15 community members directly involved in asset transfer requests. In addition, three participant observations (events/ meetings) were conducted (Table 5).

Table 5 Community transfer body interviewees and meetings/events attended

CTB name	Number of interviewees	Participant observations (meetings/ events attended[9])
CTB 1	1	0
CTB 2	5	1
CTB 3	2	0
CTB 4	2	0
CTB 5	1	0
CTB 6	2	1
CTB 7	1	0
CTB 8	1	1
Total	16	3

3.4. Analysis

3.4.1. Analysis – primary data

The interviews were audio-recorded and transcribed verbatim. All interview participants, and their respective relevant authorities/ community transfer bodies, were anonymised. Framework analysis was used to analyse both the interview transcripts and fieldnotes. This method is frequently used in applied policy research and aims to assess the impact of a policy as well as understand the ways in which a policy is implemented^[10]. Using framework analysis helped the research team to identify connections within the data and answer the research questions.

The thematic framework was informed by the Theory of Change developed by Myers et al.’s Evaluability Report (2017). As described at Table 1, the final stage of the evaluation focused on modifying and adding further details to create a revised Theory of Change based on primary and secondary data. As encouraged within a framework analysis approach, the framework was continually revised and refined to ensure it reflected the themes within the data.

Qualitative data were analysed in NVivo, a qualitative data analysis computer software package, to capture descriptive information surrounding the actions undertaken to support and promote the use of asset transfer requests. Given the non-standardised form of the reports, NVivo was particularly useful in capturing additional information included in some relevant authority reports (e.g. descriptions of asset transfer request procedures).

3.4.2. Analysis – secondary data

Quantitative data were extracted from the reports and analysed in SPSS, a software package that supports statistical data analysis. The analysis synthesised the number of asset transfer requests that were received and the various asset transfer request outcomes. Relevant authorities were categorised by type (for the full list of relevant authorities and the categorisation see Appendix 1). The data collated and analysed from annual reports submitted in 2017-18 and 2018-19 were also used to identify patterns and trends across the reporting periods.

Further, the Scottish Government Guidance on asset transfer requests (2017) published for relevant authorities was used to assess the content of the report (e.g. in terms of what may be missing in the reports). This enabled consideration of the extent to which this part of the Act is being implemented as intended.

Application forms, business plans and decision notices were analysed to supplement data within the annual reports. Details of the organisation submitting the asset transfer request, the location of the asset being requested and the purpose of the asset were coded in order to build a database of all asset transfer requests submitted across the two reporting periods. Given that the annual reports are publically available documents, these data were not anonymised.

3.5. Ethical considerations

A member of the research team provided all participants with an information sheet detailing the aims and purposes of the research. These sheets made clear that participation was voluntary and participants could withdraw from the study at any stage. Participants were given an opportunity to ask questions about the study before deciding whether to take part. In the case of telephone interviewees, verbal agreement to take part was audio recorded at the beginning of the interview. Face-to-face interviewees signed consent forms. All data has been anonymised with no personal details divulged in research outputs. We approached one community transfer body during the evaluation, seeking permission to present their asset transfer request as a case study. We explained that, outwith the specific case study section of the evaluation, their responses would remain anonymous. The community transfer body provided written consent to present their asset transfer request as a case study.