Citizens' Assembly of Scotland: research report

Methodology

In this chapter we outline the programme of research undertaken collaboratively by the team of Scottish Government Social Researchers and three independent academic researchers. The research aims were threefold: to provide learning about the citizens' assembly process to support the delivery of the Assembly; to assess the Assembly's efficacy and impact; and to evaluate its success as a model of public engagement in Scotland. The first aim was met by the development of data briefings^[3] sharing insights from the research activities with the convener, organisers and stewarding group following each weekend. This report presents the findings in relation to the latter two aims of the research.

In order to answer questions about both the operation of the Assembly (internal dimension) and the relationship between the Assembly and wider Scottish society (external dimension), a mixed method design was developed. This approach is best suited to investigate the multidimensional nature of deliberative processes, making use of a combination of qualitative and quantitative strands of research to answer different aspects of the research questions in Table 1 (Escobar & Thompson, 2019).

Table 1: Research questions

Internal dimension

1. How do participants learn and form views through the Citizens' Assembly process?

2. How do participants experience the Citizens' Assembly process?

3. What was the quality of deliberation and facilitation at the Citizens' Assembly?

External dimension

4. How is the Citizens' Assembly perceived by the Scottish public?

5. How do understandings of elite representation, wider perceptions, and attitudes to engagement evolve throughout the duration of the Citizens' Assembly?

Overall learning

6. What are the lessons for informing other democratic processes and institutions in Scotland and internationally?

Data collection spanned 24 months, with surveys of members and expert speakers at the Assembly, non-participant observations, and recordings of small group deliberations taking place while the Assembly process was underway. A survey of the Scottish population and interviews with the Assembly's organisers, facilitators, stewarding group, politicians, civil servants and journalists were conducted once the Assembly had concluded. Throughout the duration of the Assembly process, the media coverage it received was monitored and logged for analysis. Taking a mixed methods approach, the development of certain research strands were guided by the findings of others. For instance, the sequencing of the research activities allowed the findings from non-participant observations to inform the development of the survey and interview guides.

The result is a comprehensive dataset, combining qualitative and quantitative data that gives a voice to the many participants involved in the Assembly process and allowing the findings to be grounded in their experiences. The research questions were addressed by integrating these different strands of research, allowing meta-inferences to be drawn across the datasets (Escobar & Thompson, 2019).

The Assembly's move online required adjustments to be made to the research to ensure consistency in the type and quality of the evidence collected across the face-to-face and the online weekends, which will be detailed below. The change in the Assembly's format provided an opportunity for the research to make an additional and unique contribution to the evidence base in comparing members' face-to-face and online engagement.

As a key output of the research, an anonymised dataset will be made accessible for use by researchers, practitioners and leaders in 2022, thus contributing to the global evidence base on democratic innovation.

Research programme

Non-participant observation

The research team was afforded an unprecedented level of access to the Assembly's proceedings compared to similar research projects through non-participant observation of the various formal and informal spaces at each of the eight Assembly weekends. At least three researchers attended each Assembly session, covering plenaries, deliberative sessions, facilitator briefings and debriefings, and the moments when members were on a break or having dinner. In addition, one or two researchers were present at other types of meetings, such as preparatory meetings by organisers and post-Assembly meetings between members and parliamentarians.

During the in-person weekends, researchers observed the proceedings from the side of the main room and were not present at the tables where members had their small group discussions. Once the Assembly moved online, the researchers joined the virtual breakout rooms via video conferencing software and had more direct access to the small group deliberations thanks to members' consent. To minimise the possible disruption to the Assembly, researchers' microphones and videos were kept off.

Fieldnotes were written following a flexible observation framework developed around the research questions to provide some guidance to researchers while allowing them to approach data generation with an open mind (see Appendix A). The resulting 23 sets (160,000 words) of fieldnotes were analysed thematically following inductive-abductive coding in Nvivo (see Appendix B).

Member survey

To address the first three research questions, a survey was developed to gather data directly from members about their experiences of taking part in the Assembly and their views about the Assembly process across a total of 12 time points (see Appendix C). All members who attended the weekends and consented to take part in the research were asked to complete a survey at the end of the first weekend and then at the start and end of weekends 2 to 4. Members were asked questions designed to track changes in knowledge and attitudes about the specific topic or activity at each weekend; their satisfaction with different aspects of the Assembly process; their experiences of deliberation and facilitation; and their views about democracy in Scotland and their role in it.

Prior to the Assembly moving online, members were invited to complete a survey in order to provide a new baseline of their knowledge and attitudes and explore their expectations ahead of reconvening online. The survey was expanded to consider the effect of the Assembly's mode of delivery on member' experiences. It also switched to being delivered online, with longer timeframes and additional modes of completion available to maximise the response rate and quality of data. To minimise the burden on members and mitigate the risk of research fatigue, only short post-weekend surveys took place in the week following weekends 5 to 7.

Finally, a post-Assembly survey was carried out in the week following the final Assembly weekend. This explored members' experiences, attitudes and understanding at the point of the Assembly concluding.

The sample size varied across each weekend according to the number of Assembly members in attendance who had agreed to participate in the research. The questionnaires were completed by between 85 and 113 members across the weekends and obtained a response rate of between 82% and 100%.

Responses were anonymised and a research identification code was generated for each members to use when completing the survey so their answers could be linked across the questionnaires. To best assess change over time, the decision was made to use responses from members who consistently answered every survey, thus reducing the sample size to 64. A list of variables used in the analysis of the member survey is included in Appendix C. Due to coding issues for the weekend 3 survey data, the decision was made to remove it from the analysis.

Population survey

A survey of the Scottish population was designed to respond to the external dimension of the research questions, assessing the awareness, understanding and perceptions of the Assembly across Scottish society (see Appendix D).

It gathered data from a demographically representative sample of the adult Scottish population. To appropriately compare responses to members of the Assembly, participants were also selected to be representative with respect to their political attitudes. A screening question was included in the survey to ensure that Assembly members were excluded.

Respondents were asked about their level of interest and participation in civic and political life; their perceptions and understanding of the Assembly; their attitudes and knowledge of the topics discussed during the Assembly; and their attitudes towards the Assembly's outcomes. This allowed the attitudes and knowledge of the population to be compared with that of individuals who were involved in this deliberative process.

The survey was administered online in two waves, the first taking place between the 11th and 22nd of March 2021 and receiving 1539 responses. A fresh sample was approached for the second wave, which took place between the 21st and 28th of September 2021 and received 1507 responses. A list of variables used in the analysis of this data is included in Appendix E.

Expert speaker survey

Fourteen expert speakers were invited to present formal evidence over the first four Assembly weekends. In order to gather their views and experiences of their preparation and participation, a short online survey was developed consisting of 15 closed and open-ended questions regarding their expectations of presenting at the Assembly, the preparation for their involvement, and their views on the Assembly process and members' engagement (see Appendix F).

All speakers were contacted by email and invited to participate via the Questback platform. Fieldwork took place between the 18th of June and 22nd of July 2020 while the Assembly was paused due to the COVID-19 pandemic and received ten responses, obtaining a 71% response rate.

Interviews

Following the conclusion of the Assembly, a maximal sampling strategy was taken to identify individuals from groups that could address either the internal dimension (i.e. how the Assembly operates) or the external dimension (i.e. the impact and relationship with Government and the wider Scottish society) of the research questions. The rationale and final sample sizes are outlined in Table 2.

A sample of 23 individuals from the Assembly's organisers (convener, secretariat and design team), facilitators and stewarding group were asked about their views and experiences relating to the planning, delivery, outcome and impact of the Assembly as well as their views on members' engagement and learning (see Appendix G).

In addition, a sample of 14 Scottish Government officials, politicians and journalists were interviewed and asked about their awareness and understanding of the Assembly (and citizens' assemblies in general); and their opinions relating to its outcome and impact (see Appendix H).

Table 2: Sampling strategy for the post-Assembly interviews

Sample group	Sampling rationale
Internal interviews
Facilitators N = 8	A gender balanced sample was selected from across the organisations involved (Diffley Partnership, Involve, the Democratic Society and the Scottish Community Development Centre), and from facilitators who had attended all or most of the Assembly weekends (in person and online) for continuity of experience.
Organisers (convener, secretariat and design team) N = 7	Leads in all roles: Convener x 1 Secretary x 1 Assistant Secretaries x 2 Lead facilitators x 2 - 4
Stewarding group N = 8	Sample selected from the members most engaged with the Assembly, ensuring a mix of academic and practitioner members.
External interviews
Politicians N = 7	Sample selected from across all main parties.
Civil servants N = 2	Sample selected from civil servants involved in the Government response to the Assembly.
Journalists N = 4	Sample selected from journalists identified as having provided coverage of the Assembly across national (Scottish) and local media; and across print, radio/TV and online media platforms.

In line with restrictions on face-to-face research as a result of the COVID-19 pandemic, interviews were conducted online using video conferencing software between January and May 2021. Following transcription of the audio-recordings, a thematic analysis was conducted taking a combined inductive and deductive approach to coding (see Appendices I and J). This allowed the resulting themes to reflect the data as accurately as possible.

Small Group Deliberations and Discourse Quality Index Analysis

In order to assess the deliberative quality throughout the Assembly, a sample of the small group discussions were recorded. As not all discussions in a citizens' assembly are designed to be deliberative, we identified the sessions from each weekend that we thought had the greatest chance of containing deliberative norms. Weekend 8 was excluded as it did not contain deliberative sessions. We then used an online random number generator to randomly select two tables/online breakout groups from between two and four sessions of each weekend. We ensured that only Assembly members who had consented to this part of the research were included and that the sample contained a range of different facilitators.

Following transcription of the audio recordings, unclear transcripts were removed and a subsample was selected for analysis based on the following criteria:

1. Deliberation: recordings that were less deliberative than initially anticipated were eliminated.

2. Type of communication: recordings from communicatively divergent sessions were eliminated. This refers to sessions designed to be explorative as opposed to communicatively convergent sessions in which members were encouraged to make decisions and seek agreement. While divergent communication^[4] serves an important purpose in deliberation as it encourages members to engage with the evidence and explore different intuitions, positions and prioritisations, instances of divergent communication were excluded to achieve a greater fit between the sample and the analytical tool that was employed.

3. Spread across the Assembly: in order to track any change or progression in deliberative quality across the weekends, at least one session per weekend was included. These were selected from recordings that met the other criteria in the sampling strategy so the difference in deliberative quality between groups and facilitators could be accounted for. Finally, an even spread of discussions was selected across the in-person and online sessions.

This resulted in a sample of 936 contributions (known as "speech acts") by Assembly members and 611 by facilitators on which to conduct a discourse quality index (DQI) analysis. This is a theoretically grounded instrument that involves quantitatively coding the extent to which discussions meet deliberative norms such as inclusion, reason-giving, focus on the common good or respect.

A coding framework (see Appendix K) was created based on the original codes developed by Steenbergen et al. (2003). In order to contextualise the framework to a citizens' assembly we adopted developments made by other researchers (Bobbio, 2013; Davidson, et al., 2017; Elstub & Pomatto, 2018; Marien, et al., 2020; Elstub, et al., 2021a; Elstub, et al., 2021b). In order to help us address some of our other research questions, we also coded for the extent the members expressed a need for more information, referred to information provided by the speakers, or discussed the Assembly process. Finally, to enable analysis of how deliberation evolved throughout the Assembly process, we coded the weekend, day, session number and table at which the discussion occurred.

Given the importance of facilitation to deliberative quality in mini-publics (Escobar, 2019), facilitators' contributions to the discussions were also coded following Ravazzi's (2013) coding framework. This includes aspects such as whether facilitators invite Assembly members to speak or justify their demands, how they deal with conflict between members or whether they offer their own opinions on issues being discussed (see Appendix K).

As is common practice in DQI analysis, each code was treated separately, rather than being aggregated into an overall score of deliberative quality (see Appendices M and N for the deliberation and facilitation scores). This is because not all deliberative norms carry equal credence (Davidson, et al., 2017). In reporting the findings, we have removed the first weekend from much of the analysis. As an introductory weekend it contained little deliberation and would have otherwise offset the aggregate results across the whole of the Assembly process.

Media monitoring

To assess how media reporting of the Assembly evolved throughout its duration and to track changing attitudes and perceptions about the Assembly, the media coverage of the Assembly was monitored from its announcement in August 2019 to its conclusion and the release of the Assembly's report in March 2021. Broad inclusion criteria were included to capture as much relevant data as possible. An initial media monitoring was commissioned by the Assembly's organisers between August 2019 and March 2020. This was supplemented with searches conducted by the research team using a number of databases: NewsBank, the National Library of Scotland Archives, Google, Google News, Newslookup and Bing.

The resulting dataset of 206 articles contains all mentions of the Assembly across local, national and international traditional online and print media between the period of August 2019 and March 2021. The media analysis followed the mixed-methods approach recommended by the bulk of the literature on the subject. A thematic analysis of the data was conducted in Nvivo, following three stages of iterative inductive coding (see Appendices O and P). In addition, the analysis produced a quantitative outline of the amount of coverage different themes received (see Appendix P).

Ethics

This research was conducted in accordance with Government Social Research (GSR) Ethics Principles and followed Scottish Government standard ethics procedures. Informed written consent was obtained from everyone that participated in the research. In addition, verbal consent was obtained from members prior to recording the small group discussions.