Citizens' Assembly of Scotland: research report

Chapter 1: Members' experiences of the Assembly

Research questions:

a) What is the level of satisfaction by Assembly members with all aspects of the process?

b) To what extent do Assembly members feel included/empowered throughout the process?

c) Group dynamics – Were there demographic/power dynamics at play? What features of the Assembly design had an impact on group dynamics and member satisfaction?

Data sources:

• Fieldnotes
• Member survey
• Internal interviews (organisers, facilitators, stewarding group)
• Sample of transcripts from the small group deliberations
• Expert speaker survey

This chapter explores how satisfied members were with different aspects of the Assembly, as well as their varied experiences as participants in the process. We focus in particular on questions of inclusion and empowerment, investigating group dynamics and some features of process design that had a significant impact on the quality of participation and interaction, and thus on members' satisfaction.

Members' satisfaction with the process

Exploring the question of members' satisfaction across the full research dataset strongly indicates that members were generally very positive and engaged throughout the process. In this section, we unpack this general finding, analyse key factors, and examine areas where satisfaction fluctuated over the course of the Assembly.

In the member survey, large majorities of participants reported high levels of satisfaction with every aspect of the process, from general organisation of logistics and proceedings to online delivery, quality of evidence sessions, quality of deliberation, and agreement with the final recommendations. This resonates with findings from fieldnotes, which also show that members expressed satisfaction with other aspects of the Assembly, including: the collegiality and friendships developed; the diversity of perspectives; the challenge of doing something unique and 'historical'; the level of respect and consensus-building; and the supportive work of organisers and facilitators.

The member survey asked about different elements within the Assembly process (see Appendix C). A first set of questions asked about the general organisation of the weekends and ongoing communication with the organisers. On a scale from 0 (dissatisfaction) to 3 (satisfaction), the average response was between 2.5 and 3 for every Assembly weekend, indicating high levels of satisfaction. However, the spread of the responses indicated that a small minority of people were very dissatisfied with the organisation in weekends 4 and 5, which coincides with the Assembly's transition from in-person to online due to the COVID-19 pandemic.

As a result of COVID-19 restrictions, the Assembly moved online on weekend 5 and participants were asked about their satisfaction with this new space – for example, whether there were distractions or technical challenges. Initially the mean response was 2.47/3 and by weekend 8 satisfaction rose to 2.71/3, once again indicating high levels of satisfaction with the (now digital) process. Finally, when asked to evaluate the diversity of viewpoints in the information provided, speakers' presentations, and other Assembly members, participants were overwhelmingly satisfied and became more so as the weeks went on. By weekend 8, the average satisfaction score on this measure was 2.73/3.

We combined various satisfaction variables into a Process Satisfaction Index ranging from 0 (dissatisfied) to 1 (satisfied). The chart below shows the aggregate points of change in satisfaction score (see Figure 3). It is remarkable that there was very little aggregate change over the course of the Assembly, with the mean satisfaction scores in weekends 1 and 7 only differing by 0.0004. However, this aggregate level hides the finding that at the individual level there were slight changes in scores taking place over time. By the end of the Assembly, only two respondents consistently had the same satisfaction score and only 16 had the same level of satisfaction in weekends 1 and 7. Nevertheless, high satisfaction levels throughout the Assembly were consistently supported by open text responses highlighting enjoyment as a key reason for continuing engagement with the process:

'I enjoyed taking part in this Assembly, meeting people from all over Scotland and hearing their views and stories.' (Assembly member, member survey)

Source: Member survey (due to coding errors in the data for weekend 3, this has been omitted from the analysis)

We also carried out tests to assess which participants were likely to be more satisfied with the process. Most demographic factors and opinion variables did not have a significant effect on a member's level of satisfaction. However, being a woman rather than a man decreased a person's satisfaction score by almost 10%. This resonates with recent research that suggests that women are not always equally included in different aspects of a citizens' assembly process (Gerber, et al., 2019; Harris, 2019; Harris, et al., 2021). We will return to this issue when discussing group dynamics.

A key factor in the consistently high levels of satisfaction relates to the work of the organisers and facilitators. The fieldnotes document sustained efforts by all organisers to support accessibility and inclusivity, both during the in-person and online phases of the process. The high retention rate when the Assembly was moved online, despite the extended gap between weekends 4 and 5, is illustrative of members' satisfaction and desire to be involved:

'I think the real yardstick of the success is that everyone came back.' (Facilitator, internal interview)

'[I]n the very dark days of the pandemic I didn't think people would want to come back and meet and discuss and debate and deliberate together, but they really did. And then hearing the stories of the Assembly members and hearing their absolute enthusiasm for each other and for their country and for how we can make things better was a revelation.' (Organiser, internal interview)

The online phase elicited a new set of challenges that affected the experience of some participants, albeit the organisers remained responsive to emerging support needs. The internal interviews and the fieldnotes emphasise factors such as: problems with technology, for instance regarding access and the flow of conversation; challenges related to the increased attention and concentration required of members; and above all else the loss of the informal spaces for connecting and strengthening relationships. The data also indicates problems stemming from the variable levels of accessibility of the evidence presented by the speakers, and how that caused some level of frustration and challenges for members. Some presentations were very difficult to follow, and members commented on being overwhelmed by the amount of evidence and some of the presentation styles. The dataset also includes instances when members' confidence in the process wavered, albeit this tended to be temporary and for a small number of participants.

Despite variation in the quality of deliberation and facilitation throughout the process (analysed in Chapter 3), the Assembly produced a clear output in the form of the members' vision and 60 majority-supported recommendations (58 of which received above 75% support) relating to future assemblies, incomes and poverty, tax and economy, support for young people, health and wellbeing, sustainability, and further powers for the Scottish Parliament.

Members expressed a high level of satisfaction with these outputs. The fieldnotes illustrate a sense of collective ownership of the recommendations, which is clearly reflected in the strong support expressed through the anonymous votes. When asked in the member survey, not a single participant reported supporting none of the recommendations. A majority of 57% supported some of the recommendations and a further 38% supported all of them. Furthermore, only 8% of respondents to the members' survey did not feel that they had exerted any influence on the recommendations. The vast majority agreed (78% strongly agreed; 14% tended to agree; 3% tended to disagree; 2% strongly disagreed) that the process through which the Assembly's recommendations were agreed was fair, implying strong consent.

The final Assembly sessions were rich in instances of members expressing delight with the process and the outputs, being proud of having taken part, noting how much they got out of the experience and connecting to other people, and being passionate about demanding that their recommendations should be taken forward and acted upon. We cannot assess in this report whether these high levels of satisfaction continued in the aftermath of the Assembly, albeit new research is taking place to examine that question.

Inclusion and empowerment

Our analysis of the dataset suggests that, overall, members felt included and empowered by the Assembly process – albeit there were important shortcomings as explored below. Positive indicators can be seen in the members' pride in and ownership of the recommendations, along with their capacity to represent the work of the Assembly in the media following the publication of the Assembly's report. Their excitement about being included in the process was noticeable from the outset:

'[P]eople were delighted to be asked … what they thought and delighted to be given the time to think about these things.' (Stewarding group member, internal interviews)

'[S]omebody mentioned it [was] like the golden ticket from Willy Wonka … I think that really surprised me … I just didn't think they would be so enthused about that … [I]t made me realise that citizens don't feel that they're listened to very often' (Organiser, internal interviews)

The organisers put in place a support structure for members' welfare and accompanied participants throughout the process. This started prior to the official sessions, with phone calls to members to introduce them to the process, answer questions and determine logistical and financial needs.

Digital exclusion was a risk during the move online, and members were provided with technical support, induction sessions and hardware where needed. Efforts were also made to prepare and enthuse members for their return, for example, providing summaries of work and accomplishments to date. The high retention rate of members can be seen as a measure of the success of these efforts:

'We did a survey with members to find out … their tech experience and tech needs … We then did an induction session with every member to make sure they were comfortable using Zoom, Jamboard, all the tools, and also just to get them back in the swing of things and to give them the opportunity to talk to each other. We then had to do quite a lot of work with individuals who either weren't comfortable or didn't have the right type of technology so getting them a [computer] and stuff like that.' (Organiser, internal interviews)

Members often expressed appreciation for the efforts made by organisers and facilitators to support accessibility and inclusion. The evidence sessions, however, did pose problems. They required a great deal of concentration from members and were sometimes pitched at too technical a level to be accessible to all – a shortcoming that can create a sense of disempowerment. Overall, the facilitators reported that this part of the process was both challenging and enjoyable for members. The dataset suggests that this dimension improved over the course of the Assembly, but there were some persistent issues that we will introduce below and analyse in more detail in Chapters 2 and 3.

Both the fieldnotes and the members' survey offer examples of procedural shortcomings that were also felt as exclusionary and disempowering. For instance, arbitrary voting thresholds, unclear procedures for proposition and review of vision statements, and questions about agenda-setting power (e.g. how key topics for evidence sessions were decided) elicited a sense of suspicion and disempowerment amongst some members on weekends 2 and 6. These issues stemmed from a lack of engagement with participants, which was subsequently corrected:

'The processes around weekend 6, and the vision, were probably the weakest part of the Assembly … I think we got [them] absolutely right in weekends 7 and 8, because we were … determined we were going to … have absolute transparency over decision making … and you need to have walked that through with members … I think that was a bit edgy in weekend 2, and … even more edgy in weekend 6. And it's getting that balance of direction, consultation, and clarity, for people.' (Organiser, internal interviews)

The Assembly design combined plenary and small group sessions, with the latter being where most of the personal interaction and group deliberation took place. Over the full course of the Assembly, the member survey indicates that between 85% and 95% of respondents consistently reported feeling they had ample opportunities to express their opinions in small group sessions. When asked if they felt able to speak, members consistently reported feeling included and respected by facilitators, with no more than 5% reporting that they did not feel so in any one weekend. Their open text responses are illustrative:

'[The facilitators] were both excellent, and helpful moving the conversation forward through some shy/awkward silences.' (Assembly member, member survey)

'Great facilitators, really made sure everyone got to speak.' (Assembly member, member survey)

'There was one lady we didn't hear a lot from … [the] facilitator brought her into the discussion and was very kind. Did include her but didn't push her too much.' (Assembly member, member survey)

A proxy to assess levels of empowerment is the extent to which members report being more able to participate in politics by the end of the Assembly. In open text responses in the member survey, participants consistently reported feeling empowered throughout the Assembly weekends:

'As the Assembly has progressed, I felt proud to be part of it. In the start I was probably a bit nervous and unsure, however I now feel part of something. I also feel as a collective we are making a positive contribution to some complex issues.' (Assembly member, member survey)

'Before the Citizens' Assembly I never really got involved in politics … I am now very passionate and committed to being part of changing Scotland … I feel empowered.' (Assembly member, member survey)

'I feel our input is going to make some radical changes to our country and its people.' (Assembly member, member survey)

The topic of public participation became a focal point in small group deliberations, as illustrated in these extracts from session transcripts where members discussed issues of inclusion, empowerment and the proposals for future assemblies:

'I think there's a consensus across the Assembly … that people are very passionate and committed to the process of the mission statement and … taking that forward. And I think the consensus that I feel is that people do want everyday members of the public represented, in some way, in politics.' (Assembly member, small group deliberation)

'And if we are gonna be introducing a type of Citizens' Assembly, that gives them a voice … you know, younger, whatever age group … that may not have been able to be heard before, may have an opportunity to be heard through this type of forum.' (Assembly member, small group deliberation)

Members often reflected on the sense of responsibility about the work they were doing for the country, which is arguably another proxy for empowerment. In the member survey, 84% felt that they had influenced the recommendations while 8% didn't. Across the entire dataset we found repeated comments about members' enjoyment and sense of making a difference.

Many participants in citizens' assemblies are not particularly politically active before they are invited to take part. The participatory experience of the assembly can therefore stimulate a change in civic attitudes and behaviour. Those who are already active can still find the distinct assembly experience transformative for their political behaviour. Participation can also change how people view the political system more generally. Consequently, we also explored how the attitudes to civic and political participation and democracy in Scotland evolved during the course of the Assembly.

In the member survey, participants were able to express their views on how ordinary citizens participate in politics, how they themselves participate, and the extent to which they think the Government listens to citizens. Beginning with self-reflection, Figure 4 shows that there was a significant change in how confident people felt in participating in politics from weekend 1 to weekend 8. Initially about the same proportion 'neither agreed nor disagreed' as 'agreed' that they knew enough to participate in politics, but by the end of the assembly the majority of members 'agreed' that they were knowledgeable enough to participate.

Source: Member survey (Question: I know enough to participate in politics)

In contrast, when asked about their opinions about other members of the public participating in politics, respondents showed increasingly negative perceptions as the Assembly went on (see Figure 5). By weekend 8, fewer respondents agreed that ordinary people are able to participate in politics and that they trusted ordinary citizens to make good political decisions than in weekend 1 (see Figure 6). One interpretation of this finding is that it may reflect increasing awareness of the wide-ranging nature of the issues considered and the importance of having opportunities to learn and deliberate about them. Another interpretation is that the context of the COVID-19 pandemic may have generated greater trust in scientific and technocratic expertise. However, these are tentative readings that warrant further research and it should be noted that respondents agreeing with these statements (i.e. considering ordinary citizens as able to participate in politics and trusting them to make good decisions) were still by far the largest group.

Source: Member survey (Question: I consider ordinary citizens able to participate in politics)

Source: Member survey (Question: I trust ordinary citizens to make good political decisions)

Finally, turning to measures of the extent to which Assembly members felt listened to by the Government we observe a marked change reflecting a positive effect. As shown in Figure 7, 44% of respondents in weekend 1 did not feel they had a say in what the Scottish Government does. By weekend 8 this had decreased to 33%, and 49% of respondents actively disagreed with the statement. Of all respondents, only ten did not change their minds about whether they influenced what the Scottish Government does. All ten either tended to agree or strongly agreed that they did not have a say. Over 70% of members changed their answer from the first survey to the second. Even between weekends 4 and 8, just under half the sample still changed their answer to this question. This suggests that opinions on how much people influence the Scottish Government were quite malleable throughout the process. Once again, the context of the COVID-19 pandemic may be a factor in the malleability of these responses (e.g. some participants may have answered the question in a general sense while others may have answered in reference to their immediate situation during the pandemic).

Source: Member survey (Question: People like me don't have a say in what the Scottish Government does)

Developments in self-confidence and internal efficacy (i.e. a sense that one can make a difference) were also noted by various observers of the Assembly sessions, including the organisers and facilitators:

'I've seen people genuinely change and feel listened to and feel valued, [it] has been actually quite overwhelming at times. But also the fact that they now feel that they have a voice and they want that voice to be heard.' (Organiser, internal interviews)

'I could see people become engaged more than as members of this specific Assembly. There were… more than a couple along the way… who said that before they were not interested in politics, they couldn't care less, they were totally switched off, and that now they felt like it's really important to be engaged, it's really important to have a say.' (Facilitator, internal interviews)

Many observations in the fieldnotes convey a sense of members' evolving attitudes on civic and political life; for example, developing self-confidence and a sense of agency, expressing growing faith in other members, or believing that it is possible to find some level of common ground across differences. There were many instances indicating that members felt that, as the title of the Assembly's report says, 'doing politics differently' was possible. Perhaps one of the most telling aspects about this was the amount of time members spent discussing institutional reform towards embedding democratic innovations in Scotland. This was reflected in their final vision and recommendations. For example, pertaining to the introduction and normalisation of citizens' assemblies. The data includes some instances where members expressed wishes that other people in Scotland could get a chance to participate in processes like this. However, a distrust in traditional institutions and political arenas also remained consistent amongst some members throughout the Assembly.

Group dynamics

In this section, we unpack further some findings about group dynamics at the Assembly. As a collective, members developed a strong sense of identity as well as interpersonal bonds. The observed strength in relationships between members was noted in fieldnotes and internal interviews, which sometimes elicited surprise, given the diversity of backgrounds and experiences represented in the Assembly.

'I could see where alliances … and friendships were being made. And I think the joy of that was some of them were between groups of people that you would expect to coalesce, and then other ones were between people who you would never in a million years think that they were going to … meet and discuss.' (Organiser, internal interviews)

The graphs below offer an overview of key group dynamics (see Figures 8-10). When members were asked about how they felt around their fellow participants, most respondents reported feeling comfortable throughout the process. When asked if they felt respected by other members, over 80% expressed agreement, with over 90% agreeing on weekends 1 and 8. Over 77% of respondents consistently disagreed with the statement 'I didn't always feel free to express my views for fear of others' reactions', with a maximum value of 89% of respondents disagreeing by weekend 8. Our analysis of a sample of deliberation transcripts also indicates that there was little disrespect shown amongst members (around 1% of all instances coded in the transcripts). All counts of 'no respect' occurred in the face-to-face sessions and none occurred online. In this sense, the move from in-person to digital seemed to foster even more respectful and considerate dynamics in small group deliberation.

Source: Member survey (Question: My fellow participants respected what I had to say, even when they didn't agree with me)

Source: Member survey (Question: I didn't always feel free to express my views for fear of others' reactions)

Source: Member survey (Question: I have had ample opportunity in small group discussions to express my views)

In the internal interviews, facilitators frequently reflected on the need to manage the balance of voices to ensure that all members had a chance to contribute; and they were particularly concerned about this during the move online. They did not make any reference to specific demographic characteristics influencing dynamics beyond noting that younger members were on occasion less engaged.

'I had a young guy that was in my group throughout, for about three weekends, and the secretariat kept putting him back in because they felt I had a relationship with him. But I think we totally failed to engage him. And I think the process wasn't designed to engage him. And I think the problem is, it was open enough to get him there, and he came to Clydebank and had a good time … But his lack of engagement was annoying the other citizens … I think, if we were going to do it again, there might need to be some different arrangements made for some folk who just don't relate to that sort of process.' (Facilitator, internal interviews)

According to our fieldnotes, gender inequalities seemed to shape dynamics in some of the small groups and plenaries – for example, men tended to speak more often when reporting back in the plenary sessions. Actions by facilitators and organisers sought to address this and representation improved towards the final sessions. In the member survey, similar proportions of men and women agreed that they felt respected in group discussions. However, when it came to whether they felt comfortable raising their opinions in the group, only one man disagreed whereas 18% of women disagreed. Women were also less likely than men to strongly agree that they had ample opportunity to express their views in group discussions: 86% of men strongly agreed while 7% tended to agree, for women the proportions were 64% and 24% respectively.

The analysis of the member survey data did not suggest that other demographic characteristics had a direct influence on group dynamics (e.g. disability; ethnicity). However, the fieldnotes (which included facilitator debriefs) provide some evidence that education level may have also influenced dynamics in some groups – for example when it came to having confidence crafting text, articulating propositions, or offering challenge and scrutiny.

Design choices in this type of deliberative process are often caught in the tension between structure and flexibility – that is, trying to plan as much as possible while leaving room to deal with contingencies such as the development of unequal group dynamics (Escobar, 2019; Faulkner & Bynner, 2020). In this Assembly, the design process was ongoing as the nature of the assembly's remit was broad and the route map unpredictable:

'I know that in between [weekends] they were desperately designing away and I'm sure that at times they felt that was almost from scratch each time, depending on how well the previous weekend had worked, and I'm sure that is the case – but my sense is that there was a purposefulness to where it was going. And the reason I say the design was good is I think a good design hinges on having the core question and the outcomes in mind all the time.' (Facilitator, internal interviews)

In contrast to the quote above, other facilitator interviews suggest that there was a fair amount of 'trusting in the process' going on (more or less happily) despite a lack of clarity about the route map for the Assembly. This could be related to the facilitators' limited involvement in the design process, with several noting that they focussed (again, more or less happily) on their individual roles and responsibilities weekend by weekend.

What is consistently clear across the internal interviews dataset is that the breadth of the remit, coupled with a pre-determined timeframe, had a knock-on effect on key aspects of the process design that shaped group dynamics within the Assembly. It made it challenging to design and implement the evidence sessions because designers did not know in advance what issues would be covered in subsequent stages. It also proved a challenge to the design of deliberative sessions due to the breadth of topics that needed to be addressed. A significant portion of the preparations were spent dealing with the uncertainty about how to proceed. Organisers frequently stressed that this was particularly challenging within the tight timelines. Regardless of their views on the breadth of the remit, interviewees agreed that it created challenges to design and delivery, resulting in uncertainty about the direction, endpoint, and output of the Assembly.

Other design shortcomings related to voting systems, revision processes (e.g. for proposals), agenda-setting, and changing procedures – which generated confusion and even suspicion amongst some members and facilitators. Perhaps the most visible manifestation of this was the design and implementation of weekend 6, where a 90% threshold was required to agree vision statements. There was lack of clarity of the rationale for this design choice and no involvement of members (or of the stewarding group) in setting this up. The fieldnotes for that weekend are full of things that went wrong and caused distress for facilitators – one put it bluntly: 'it was the most compromising experience of my professional career'. Some members also voiced discontent and confusion, and the open text responses in the member survey show suspicion amongst some participants. Parts of the procedure seemed to be made up on the go, creating substantial difficulties at the stage of developing, revising and voting vision statements. Some members noted that they may have voted differently if they knew that the statements could be revised and redrafted.

'I am never doing a significant vote live in a session ever again … Weekend 6 was the most stressful day of my working life … [A]ll it took was for two members to have a technical issue for that process to essentially not go well, not go right.' (Organiser, internal interviews)

'There was one weekend that I remember coming out the end of thinking we kind of lost control of things a bit there … although what we heard afterwards … from the members is the members didn't feel it in the same way as the facilitators had in terms of that didn't actually go that well.' (Facilitator, internal interviews)

Across the entire Assembly process, there seemed to be limited oversight of design and delivery by the stewarding group, which raises questions about overall governance of the assembly (see Chapter 6). Procedures seemed more carefully thought-out and worked better in the last two weekends. All in all, the process design did help produce clear outputs backed by substantial votes. However, the revision and refinement of the final recommendations was limited, which resulted in overlaps between some recommendations or in proposals for initiatives that may already exist (see Chapter 5 for more perspectives on these issues).

Conclusion

This chapter has shown that levels of satisfaction with the process were, and remained, very high amongst members for the entire duration of the Assembly. Likewise, our assessment also indicates high levels of inclusion and empowerment, despite a range of challenges related to process design and group dynamics. Inclusion and empowerment go hand in hand with opportunities for participant development, and some design features and facilitation dynamics may have undermined this (for further analysis see Chapter 3). The online setting may have also accentuated some of the challenges by reducing the relational aspects that contribute to developing confidence and internal efficacy – not just in formal sessions but across the many informal spaces in the process. Shortcomings related to agenda-setting, voting and revision processes prompted learning and adaptation which improved the dynamics of the Assembly over time. This shows the importance of reflective practice and collaboration between organisers and facilitators. Finally, members' sense of internal efficacy for civic and political life was positively affected by their experience at the Assembly.