Community Empowerment (Scotland) Act 2015 - participation requests: evaluation

3. Approach to this evaluation

3.1. Evaluation stages

The evaluation was carried out in four consecutive stages with areas of overlap and continual analysis throughout the data generation (Table 1). Myers et al.'s (2017) Theory of Change informed the design, implementation and thematic framework adopted for the research.

3.2. Methods

To address the research questions, the study adopted a range of methods to collect primary data including in-depth interviews, participant observations and a focus group. Secondary data was sourced through the collation of participation request key documents including annual reports.

3.2.1. Data collection with public sector authorities

Across Stages 1 and 2, we undertook in-depth interviews (n=15) with stakeholders from ten public service authorities named in the Act and with one key stakeholder from a national organisation involved in promoting and supporting participation requests. Interviews with those responsible for participation request processes and supporting communities to submit participation requests helped to generate an understanding of how the Act has been implemented by public service authorities. The interviews were also used to give insights into intermediate and potential longer-term outcomes of participation requests.

During Stage 2, we held a focus group with four stakeholders from a public service authority that, according to their annual reports, sought to favour other pre-existing participatory processes over participation requests (Section 3.3.1 for further detail). Topic guides for these participants were developed to generate data related to how the Act has been interpreted and implemented, explore the alternative participation and engagement mechanisms offered by the public service authority and understand how participation requests are perceived and understood, in comparison to other processes.

3.2.2. Data collection with community members

During Stage 2, the research team conducted in-depth interviews with community members (n=12) from five community participation bodies with participation request submissions. Two of these community participation bodies had submitted two participation requests each. In total, seven participation requests were considered within the evaluation. The interviews were conducted in person or by telephone. The topic guides for these participants were developed to generate data related to the experience of the participation request application process; community participation bodies' motivations for submitting participation requests; the support offered by public service authorities; actual and anticipated outcomes from the participation requests; and perspectives on the meaning and possible measurement of community empowerment.

Table 1 Stages of the research, aims and activities

Stage 1

(Apr. 2018 – Sept. 2018)

Aim: to identify activities and outputs related to participation requests

• Understanding implementation processes
• Identifying early patterns in participation request submissions across Scotland
• Highlighting how participation requests were being interpreted by public service authorities
• Exploring public service authority perspectives on potential outcomes

Data collection (primary)

• Interviews with public service authority representatives (n=13)
• Interviews with one key stakeholder from a national participation request support organisation

Data collection and analysis (secondary; 2017-2018)

• Formal participation request reports (n=38) supplemented with available documents

Collection of informal data from public service authorities (detail provided in Section 4.1) (n=12)

Stage 2

(Sept. 2018 – May 2019)

Aim: to explore the experiences of community organisations involved with participation requests.

Data collection (primary)

• Interviews with community representatives (n=14)
  • 12 from five community participation bodies
  • 2 from two community groups
• Interviews with public service authority stakeholders directly involved in an outcome improvement process (n=2)
• Focus group with participants (n=4) from a public service authority
• Participant observations (n=6)

Data collection and analysis (secondary; 2018-2019)^[7]

• Formal participation request reports (n=29) supplemented with available documents
• Collection of informal data from public service authorities (detail provided in Section 4.1) (n=3)

Stage 3

(June 2019 – Sept. 2019)

Aim: to analyse primary and secondary data

• Comparative data analysis using 2017-2018 and 2018-2019 annual reports
• Analysis of the qualitative data

Stage 4

(Sept. 2019 – Jan. 2020)

Aim: Final analysis

• Development of a revised Theory of Change
• Two stakeholder workshops were held with participation request stakeholders (30); emerging findings were discussed and comments helped to further develop the revised Theory of Change
• Producing final reports

In addition to the interviews conducted with community participation bodies with participation request submissions, we also undertook interviews (n=2) with representatives from two community groups operating in an area where a public service authority had adopted a different approach to participation requests by actively minimising participation requests in favour of alternative participatory processes. The intention of the community group interviews was to explore how participation requests were perceived and understood, in comparison to other processes.

3.2.3. Participant observation

During the evaluation, a member of the research team attended meetings (n=6) of community organisations included in the sample, on occasions when participation requests were a key part of the meeting agenda. Attending these meetings allowed the research team to observe how the community organisations and the wider community were engaging with participation request processes. Further, these observations offered the opportunity to understand how participation requests fit within the broader work of the organisations, as well as the other mechanisms they engaged with to achieve their aims. During two of these meetings, the evaluation team were given an opportunity to formally solicit views on participation requests. During the course of the evaluation, two of the organisations took part in meetings related to outcome improvement processes. In one case, this consisted of a formal meeting between the community organisations and a public service authority. In another, an informal community engagement event was attended by the wider community. The research team attended both meetings, allowing us to observe interactions between community participation bodies and public service authorities, and understand the different ways in which an outcome improvement process may take place.

3.2.4. Secondary data: annual reports on public service authorities

Secondary data collection spanned Stages 1–3. Two sets of annual reports were collected and analysed for the evaluation: 2017-2018 (capturing the period 1st April 2017 to 31st March 2018) and 2018-2019 (capturing the period 1st April 2018 to 31st March 2019). Despite public service authorities having a statutory duty to publish annual reports which outline participation request activities, low publishing rates in both periods meant that additional steps were taken to collect data from public service authorities. During Stage 1 of the evaluation, these steps included Scottish Government email communication informing public service authorities of the evaluation (April 2018) and reminding public service authorities of their statutory duty to publish their reports (July 2018); and evaluation team email communication requesting information from public service authorities on the intended online location of the published reports^[8] (June 2018) and requesting that all public service authorities submit their reports (July 2018). During Stages 2 and 3 of the evaluation, emails from the Scottish Government were sent (May, June and August 2019) to public service authorities to remind them of the statutory duty to publish participation request annual reports.

Section 4.1 details the number of reports submitted by public service authorities. Findings presented for the period 2017-2018 include all reports and data made available by 31^st July 2018. Findings presented for the period 2018-2019 include all reports and data submitted by 23rd August 2019. Once collected, both sets of reports were analysed using the same systematic approach. Findings from reviews of annual reports for both periods are available online^[9]. In addition to the collection of annual reports, the research team gathered documents relating to participation requests from public service authority websites. Relevant documents included application forms and, in some cases, the outcome improvement process reports that outline any subsequent changes occurring as a result of participation requests. Including these data sources has allowed for analysis of the types of organisations that are submitting requests, the purpose of the request and the nature of any changes that may have occurred. Data were not available for all participation requests and some reports were incomplete with missing data; this is noted in the tables and figures throughout the report.

3.3. Sampling

This evaluation adopted purposive sampling of public service authorities, community participation bodies and other stakeholders, with the aim of ensuring that the sample would have broad and diverse experience and knowledge of participation requests.

3.3.1. Sampling of public service authorities

Public service authority interview participants were identified following the collation and analysis of annual reports for the 2017-2018 period. Information from the annual reports enabled a mapping of participation request activities and the identification of key people responsible for participation requests within public service authorities. For this evaluation report, interviewed public service authorities are labelled as 'PSA'.

The sample was chosen to reflect overall levels of participation request activities in different public service authorities. Given that local authorities received the majority of submitted participation requests (see Section 4.2 for further details), the majority (68%) of interviewees were stakeholders from local authorities. However, given that local authorities are only one type of public service authority named in the Act, it was important to represent the range of organisations involved. Table 2 shows the breakdown of the sample by public service authority type. PSAs 6-10 are different types of public service authority, as defined in Schedule 2 of the Act (Appendix 1 Public service authorities). In addition:

• PSAs 6 and 8 had not received any participation requests
• PSA 9 had been named as a secondary organisation on a participation request application^[10]
• PSA 2 were taking a different approach to participation requests (because they were trying to minimise participation request submissions by promoting other forms of engagement and participation)

Table 2 Sample by public service authority type

Code	Public service authority Type	Stage one participants	Stage two participants
PSA 1	Local Authority	2	2
PSA 2	Local Authority	2	4[11]
PSA 3	Local Authority	1	0
PSA 4	Local Authority	1	0
PSA 5	Local Authority	1	0
PSA 6	Transport	2	0
PSA 7	Health	1	0
PSA 8	Education	1	0
PSA 9	Other	1	0
PSA 10	Other	1	0
Total participants		13	6

3.3.2. Sampling – Communities

Five community participation bodies were included in the final sample. To identify community organisations for the sample, the evaluation considered the spread and level of participation request activities reported by public service authorities in 2017-2018 annual reports (see Table 1, Stage 1). Based on details from the reports, the evaluation sought to include community participation bodies that would reflect a variety of experiences and contexts: for example, whether their participation request was accepted or rejected, and whether the participation request resulted in a change to any service.

For this evaluation report, interviewed community participation bodies are labelled as 'CPB'.

The original sample included four community participation bodies. However, attempts to contact one community participation body in the sample were unsuccessful and additional community participation bodies were contacted. Prompt responses were received from relevant community participation bodies and they were included within the revised sample. During the course of the evaluation, one community participation body, CPB 5, submitted a second participation request, therefore the total number of participation requests covered within the evaluation was seven (Table 3).

Table 3 Number of participation requests included in the sample

Participation requests		Community participation body code	Public service authority code	Community participation body Interviewees	Observations
1	Accepted	CPB 1	PSA 4	2	1
2	Accepted			1	0
3	Accepted	CPB 2	PSA 5	1	1
4	Refused	CPB 3		2	1
5	Accepted	CPB 4		2	1
6	Accepted	CPB 5	PSA 1	2	1
7	Accepted			2	1
		Total		12	6

In addition, two interviews were conducted with representatives from two community groups that operate in an area where a public service authority (PSA 2) was actively and explicitly seeking to minimise participation request submissions. These community groups have been coded as CG 7 and CG 8 (Community Group). In total, the sample included 14 interviewees (community participation bodies=12, CG=2) and six observations were carried out (Table 1, Stage 2).

3.4. Analysis

3.4.1. Analysis – Interviews, focus group and observations

The interviews and focus group were audio-recorded and transcribed verbatim. All interview participants, and their respective public service authorities /community participation bodies, were anonymised. Framework analysis was used to analyse the interview transcripts and field notes. This method is frequently used in policy research which aims to assess the impact of a policy as well as understand the ways in which a policy is implemented^[12].Using framework analysis helped us to identify connections within the data and answer the research questions. The thematic framework was informed by the Theory of Change developed in Myers et al.'s (2017) Evaluability Report (Figure 1). The framework was continually refined to ensure it reflected the themes within the data, and led to the development of a revised Theory of Change (presented in Figure 6).

Qualitative data were analysed in NVivo, a qualitative data analysis computer software package, to capture descriptive information about the actions undertaken by public service authorities to support and promote the use of participation requests.

3.4.2. Analysis – Secondary data

Quantitative data were extracted from the reports and analysed in SPSS, a software package designed to conduct statistical data analysis. The analysis synthesised the number of participation requests that were received, agreed and refused across public service authorities, and any changes to services that were identified by the public service authorities during the reporting period. Public service authorities were categorised by type (for a full list and the categorisation see Appendix 1) in order to explore differences across the different types of public service authorities. Further, the Scottish Government Guidance on participation requests (2017)^[13] published for public service authorities was used to assess the content of the report (in terms of what may be missing in the reports). This approach enabled consideration of the extent to which this part of the Act is being implemented as intended.

Participation request application forms were analysed to supplement data within the annual reports. Details of organisations submitting participation requests, their purpose and the outcomes were coded in order to build an understanding, over time, of all participation requests submitted across the two reporting periods. Given that the annual reports are publicly available documents; these data were not anonymised.

3.5. Ethical considerations

A member of the research team provided all participants with an information sheet detailing the aims and purposes of the research. These sheets made clear that participation was voluntary and they could withdraw from the study at any stage. Participants were given an opportunity to ask questions about the study before deciding whether to take part. In the case of telephone interviewees, verbal agreement to take part was audio recorded at the beginning of the interview. Face-to-face interviewees and focus group participants signed consent forms. All data has been anonymised with no personal details divulged in research outputs. We approached one community participation body during the evaluation, seeking permission to present their participation request as a case study. We explained that, outwith the specific case study section of the evaluation, their responses would remain anonymous. The community participation body provided written consent to present their participation request as a case study.