Citizens' Assembly of Scotland: research report

Chapter 3: Facilitation and Deliberation at the Assembly

Research questions:

a) What is the quality of deliberation and facilitation in each weekend?

b) What participatory formats and facilitation techniques worked well? Which ones didn't?

Data sources:

• DQI analysis
• Fieldnotes
• Internal interviews (organisers, facilitators, stewarding group)
• Member survey

Deliberation is a communicative approach to democratic decision-making that aims to be inclusive, reasoned and respectful. In citizens' assemblies, the discussions are facilitated with the aim of ensuring the presence of these deliberative norms. One of the more frequently used methods to assess deliberative quality is the Discourse Quality Index (DQI). This is a theoretically grounded instrument that enables researchers to quantitatively code the extent to which discussions meet deliberative criteria such as inclusion, reason-giving, focus on the common good and respect (see Appendix K). Its application to a sample of recorded small group discussions enables an assessment of the extent to which deliberation occurred and which deliberative norms were more prevalent at different time points (see Appendices M and N). By interpreting the results in combination with interviews with the Assembly's organisers, fieldnotes made by the researchers following non-participant observations of the Assembly weekends, and surveys completed by members of the Assembly and expert speakers, it is possible to assess the quality of the deliberation and facilitation at the Assembly and discuss their contributing factors.

What was the quality of deliberation at the Assembly?

While the quality of deliberation varied between weekends, the DQI analysis found that some forms of deliberation remained more consistent throughout the Assembly (see Appendix M). The contributions members made to the discussions – known as 'speech acts' – consistently contained high levels of demands (83%), where the speaker refers to what should or should not be done within the issue under debate, and were almost entirely relevant to the discussion (99.5%). Conversations were also generally devoid of conflict and members were respectful towards each other, with no more than 5% reporting that they did not feel included or respected in any one week.

The DQI analysis found that most speech acts contained neutral respect towards demands (60%), one third contained no respect and 6% contained explicit respect. Members became increasingly respectful over the course of the Assembly, with over 40% of speech acts containing no respect towards demands during the in-person weekends compared to just over a quarter once the Assembly moved online. While the difference is statistically significant, this is likely to be due to the members getting to know each other better and honing deliberation skills with practice as the Assembly progressed, rather than the online format being more conducive to respectful discussion.

As is common in deliberative processes (Roberts & Escobar, 2015; Elstub, et al., 2021a), there is evidence that some forms of deliberation improved over the course of the Assembly as members got to know each other, became more familiar with the process and became more confident communicators.

'[S]eeing people that are maybe more quiet, you know, younger people, yeah, just people that maybe have trouble with their confidence and things, just seeing them come out their shells… I'm like, oh well, you're now able to speak in front of a room filled with people. Like, that is amazing to see.' (Facilitator, internal interviews)

For instance, members were increasingly likely to be assertive in their contributions to the discussions the closer they got to finalising the recommendations. The percentage of speech acts containing a demand increased from 73% to 94% between weekends 5 and 7.

Justifications and use of evidence

Members also became more likely to provide reasons (or justifications) for their demands as they started to focus on making proposals and developing recommendations. The overall level of reason-giving increased during the online portion of the Assembly, with almost a third less demands being made without justification (26% online compared to 41% in person). However the DQI analysis shows that the increase was largely in the percentage of inferior justifications, where the reason provided is not concretely linked to the demand, rather than qualified ones. Overall only 6% of demands were supported with qualified justifications.

It appears that the increase may be a result of more proactive facilitation in the later weekends – possibly following feedback from the research team to this effect – rather than being related to the online format of the Assembly. The DQI analysis found that facilitators only asked members to justify their demands 12% of the time over the course of the Assembly (see Appendix N). However, facilitators were more than twice as likely to do so during the online weekends (19% online compared to 7% in-person) and the results demonstrate that members provided more justifications when facilitators asked for them. The increases in requests for and provision of justifications may also partly be related to the nature of the tasks in the later weekends, when members were asked to make recommendations and an emphasis was placed on providing reasons for these.

Furthermore, a key aspect of citizens' assemblies is the use of expert evidence to increase participants' knowledge of an issue and inform their decision-making. However the DQI analysis found that little explicit reference was made to expert opinion throughout the Assembly, with 97% of contributions not referring to any expert opinion (Table 3). There is however a statistically significant difference in the use of expert opinion across the weekends. While weekends 2, 3 and 7 contained no explicit references to expert opinion, 40% of the contributions made in the first weekend did.

Table 3 - Content analysis results for sample of small group discussions

Weekends compared	Weekend
	1	2	3	4	5	6	7
Request for information
No request	62%	78%	91%	79%	65%	87%	92%
Request for process information	13%	13%	3%	3%	13%	8%	1%
Request for topic information	25%	9%	7%	18%	23%	6%	7%
Expert opinion
No opinions referred to	60%	100%	100%	96%	99%	100%	100%
Expert opinion supported	28%	0%	0%	4%	1%	0%	0%
Expert opinion contrasted	12%	0%	0%	0%	0%	0%	0%

Additionally, the DQI analysis tallied the number of times that members explicitly agreed or disagreed with expert opinions and found a statistically significant difference between the in person and online weekends. All seven instances when members disagreed with expert opinion and 24 out of the 26 instances when they agreed occurred during the in person portion of the Assembly. Interestingly, it is in the weekends that members most referred to expert opinion (1, 4 and 7) that they most frequently asked for more information during their discussions. This suggests that members' requests for additional information was influenced by the particular topics discussed during those weekends rather than the stage the Assembly was at.

The period of time the Assembly was paused in response to the COVID-19 pandemic may have had a negative effect on the mobilisation of expert evidence within deliberations, due to members having to recall evidence presented months earlier. Researchers observed that some members did refer to some of the evidence presented months earlier or available in their evidence hub in the final sessions, but this didn't seem very common despite the efforts made by the secretariat to provide members with summaries of the evidence and work done to date prior to reconvening.

These findings illustrate a discrepancy between the role of formal evidence within the Assembly and members' use of evidence within deliberations. We explored members' engagement with the evidence sessions in Chapter 2. However, while members expressed satisfaction with the evidence sessions and over 70% reported that they understood and trusted the expert speakers, they were observed to primarily draw on their own experiences and knowledge to provide reasons for their recommendations, rather than necessarily connecting this to evidence presented formally at the Assembly.

Level of scrutiny and challenge

The fieldnotes and internal interviews show that organisers and facilitators strongly emphasised finding "common ground" from the Assembly's outset. This arguably overvalued cohesion and consensus to the detriment of appreciating the importance of critique and disagreement in deliberative processes. As a result, the level of scrutiny and challenge during interactions with speakers and amongst members was one of the weakest or most uneven aspects across the Assembly. Most of the facilitators and some of the organisers interviewed felt (to varying degrees) that there was a lack of sufficiently robust deliberation or debate due to members not being sufficiently probed or challenged on their opinions or presented with different ones. Furthermore, while there was little conflict to resolve, facilitators generally did not intervene when it occurred. As we will discuss in Chapter 5, this had implications for the way in which the recommendations were received.

'[T]here wasn't really a point where you could sort of stress test that, like you couldn't argue it out with other people or you couldn't pin people down and go, well why do you think that?' (Facilitator, internal interviews)

'That real opportunity to engage with others that had different opinions from yourself and to be challenged on that view it feels like that really wasn't part of this process at all.' (Facilitator, internal interviews)

Comments made by a few of the facilitators during their interviews indicate that they did not feel able (or enabled) to support challenge and scrutiny. It is not possible to conclusively establish the source of this feeling or how widespread it was among facilitators, but some suggest that the organisers had a specific understanding of member welfare that may have been at odds with promoting facilitation techniques designed to support constructive challenge, explore differences and address disagreements. Some organisers offered a counterargument to this perspective, noting that there are differing perspectives on what counts as productive deliberation:

'I think it depends on your bar … My starting point is a quality conversation between people who are largely strangers on emotive and actually quite complex issues. So, in that whole range of human emotions and experience, I thought there were incredibly good conversations, incredibly positive ones where people respected each other, listened, learned, enjoyed themselves … Now, as I say, as a deliberative process, in a kind of rigorous, forensic engagement with evidence, through to conclusion, maybe not so strong. But in terms of quality human conversations, really powerful.' (Organiser, internal interviews)

In addition, some facilitators felt that the deliberation sessions were too output-driven, taking attention and time away from deliberation. Once again the remit and timing of the tasks presented challenges, with organisers and facilitators giving examples of deeper deliberation being interrupted by the need to move on to another task or topic.

'I think we spent a lot of time on stuff that wasn't deliberation, you know, we spent a lot of time voting, a lot of time creating words for things, but that's not deliberation.' (Facilitator, internal interviews)

'I think having such an open-ended set of remits … with the three questions, was almost kind of boxing the design into a corner before you started. And meaning that … if we don't do all these things, we're not answering the remit, but in order to be able to answer the remit, you can't go as deep into some of these things as you might like to.' (Facilitator, internal interviews)

However the DQI analysis found a substantial increase in the extent to which members engaged with each other's counter-arguments over the course of the Assembly. More than 22% of counter-arguments made by other members were ignored during the first five weekends, with between a third and half of counter-arguments being acknowledged but downgraded. Conversely, members engaged with counter-arguments more while drafting, voting upon and agreeing on a collective vision and recommendations. Over 55% of counter-arguments in weekend 6 were valued (highest of all weekends) and half of those made in weekend 7 received a neutral response (highest of all weekends).

Inclusion

A number of sources in our dataset (fieldnotes, DQI analysis and open text responses in the members' survey) suggest that levels of inclusion in the functioning of small groups was mixed across sessions. The issue of "dominant voices" kept recurring in some of the groups and some facilitators struggled to respond to this. Certain members went through entire sessions without making a single verbal contribution and some discussions were often dominated by a few participants. As demonstrated by the graph below (Figure 18), members who spoke most frequently also spoke longer on average (as determined by the number of words in a speech act).

Furthermore, there was a significant positive correlation between the length of a member's speech act and the level of justification used. This means that those members who were particularly vocal also had the most opportunity to persuade other participants.

However discussions seem to have become more inclusive as the Assembly progressed. Facilitators became more likely to encourage members to contribute to discussions, quieter participants seemed to gain in confidence and (some) dominant participants gained in self-restraint. As discussed in Chapter 1, members reported high levels of satisfaction with how they were treated by fellow participants and felt able to express their views and contribute to discussions. Furthermore, it is important to note that engagement and participation can take different forms depending on the individual. For example, listening rather than taking an active role in conversations, or contributing in written form.

'I noticed that happen quite a lot around the tables, that two or three people would be having a really interesting discussion about one of the topics and there'll be a couple of people listening in. And you get the sense that they are there kind of shaping things in their minds whilst not being active in the discussion, but they're still kind of … cogitating through.' (Organiser, internal interviews)

'[T]here's a mixed bag of actual engagement and by that I mean … taking part in discussions, putting ideas forward. You always have a cluster of some members that are more confident doing that. You'll have a cluster of members that are quiet at the beginning and build up confidence. You'll have some that look disengaged but actually they're listening. And earlier I mentioned one member that would draw stuff and make speech bubbles but wouldn't really say much in table discussions but would contribute in written form.' (Facilitator, internal interviews)

In terms of participatory formats, the combination and sequencing of plenaries and small group sessions worked well. The icebreakers seemed straightforward and well received, often consisting of basic introductions of members' backgrounds and interests. However the range of participatory formats and techniques were heavily skewed towards verbal contribution and oral skills, although some exercises allowed writing or using visual aids. The fieldnotes document the use of various other artefacts to structure interactions throughout the sessions (particularly the first four weekends of in-person gatherings), including flipcharts, cards, sticky walls, voting pads, templates, etc. These all seemed to be consistently accepted by members and there were few instances of members pushing back against a particular format or tool, although levels of engagement were variable.

Some facilitator interviews suggest that there was too strong an emphasis placed on verbal communication, with challenges to maintaining the balance of voices and potentially creating barriers to participation. These were often connected to the fact that facilitators were usually both facilitating and scribing. Conflating these two important functions seems to have created numerous challenges, particularly when the Assembly moved online. Using tools such as a digital whiteboard during the online phase presented an alternative, but in the context of the logistical challenges it presented they ended up being used by the facilitators rather than members themselves, thus reducing the already limited options for kinaesthetic engagement by members. This said, members who commented on these matters seemed to prefer that the facilitator managed the tools.

What was the quality of facilitation at the Assembly?

The quality of facilitation at the Assembly was highly variable, ranging from excellent to poor. There was a clear ethos of care amongst the facilitators, sustained through debriefs and reflective practice, and their commitment to the process was illustrated by many observations of enthusiastic, thoughtful and skilful facilitation. However there were also instances of facilitators overstepping their role, being unprepared or unsure about the task at hand, being confused by emerging situations, and having limited awareness of facilitation techniques and deliberative standards.

Facilitation techniques

There were varying levels of experience and skills, with some facilitators seemingly unfamiliar with basic facilitation toolkits (i.e. techniques to address dominant voices; techniques to deal productively with difference and avoid avoidance; approaches to drawing in all members; techniques for inviting scrutiny and constructive challenge, etc.). The fieldnotes suggest that members in some groups struggled with the way some questions and tasks were framed by facilitators, who sometimes seemed unsure about the purpose of the sessions and their contribution to the overall process.

In addition, the DQI analysis found fairly high levels of facilitators attempting to steer members' conversations with their own ideas. Facilitators took a position 15% of the time and introduced a new idea over 31% of the time over the course of the Assembly. Limited attempts were made to synthesise member's points (between 2% and 13%) despite members largely responding positively to their points being combined and reflected back at them.

However, facilitation generally did seem to improve, if not linearly throughout the Assembly, at least in its latter stages. As we've discussed above, facilitators increasingly prompted members to provide reasons for the demands. They were also more likely to encourage members to participate in the discussions. The percentage of facilitators' speech acts inviting contributions from members increased from just over 50% during the first weekend to 82% in weekend 7. It is possible that the improvement is, in part, due to feedback provided by the research team via their data briefings.

One factor behind this mixed picture is perhaps that the facilitators came not just from very different organisations and contexts, but from very different facilitation traditions (e.g. participatory, deliberative, dialogic, communitarian, organisational, market research) that in some cases are less steeped in deliberative approaches and standards. The interviews with facilitators strongly suggest an inconsistent understanding of deliberation and different levels of awareness about the facilitation techniques required. Some of the facilitators had clear prior experience of facilitating deliberation (including in other citizens' assemblies), while others had experience of public and community engagement more broadly – which may not always include deliberative techniques. This would explain some of the variation in both the techniques used and the overall quality of facilitation.

The majority of the facilitators interviewed reported feeling as prepared as they could be for the tasks, while specifying that plans and instructions were not shared far enough in advance to allow for enough preparation. Where these were shared ahead of time, facilitators commented that they were subject to changes at short notice. Some facilitators mentioned the additional challenges created by the complexity of the tasks and evidence:

'I did feel very stressed sometimes with the methods, and in particular I was incredibly stressed with … the design canvas and the game, it was the tax game … And I was really, really uncomfortable with that … because we just got it … literally before we went in.' (Facilitator, internal interviews)

Some facilitators deemed the plans to be overly structured and lacking sufficient flexibility to respond to group dynamics. As a result, some felt that their skillset and experience were underutilised: 'I just felt as though there was only part of me being used, and the rest was kind of wasted'. The interviews indicate that some facilitators were confident in the transferability of their skills and previous experience to the Assembly process, and were relying more on these than any training organised by the design team:

'I just went into it trusting that once I was actually asked to facilitate something, I would be able to do that, because I know I can do that.' (Facilitator, internal interviews)

'I think the people who were facilitators were all experienced enough to be able to bring in the techniques that they would normally deploy.' (Facilitator, internal interviews)

In person compared to online

Finally, with regards to any difference in the quality of deliberation and facilitation between the in person and online portions of the Assembly, the DQI analysis indicates that members generally seemed more respectful, engaged and considerate online. However, as discussed above, this improvement is likely to be due to the online sessions coinciding with the latter part of the process, by which time members had improved their deliberation skills but were also approaching finalising their decisions, which could have led to a keener focus.

The move online made it easier for some members to engage and harder for others. The member survey indicates that some members noted exclusions either through accessibility issues or group dynamics affected by dominant voices. Most facilitators noted the increased challenge of sustaining the inclusion of a range of voices and ensuring that all members could participate online. Part of this stemmed from the technological challenges, with connectivity issues and uncertainty about online conversational etiquette resulting in deliberation sessions that were more challenging to facilitate.

'I think oddly it kind of … makes some people feel more comfortable and confident in contributing, but then others less so. So, I think you … there is a kind of shift. It works better for some people, not so much for others.' (Organiser, internal interviews)

'People get kicked offline, some people don't like to show their face, so they turn their camera off and that makes it harder. Yeah, just any sort of internet problems, or sort of technical difficulties, and you know, unmuting and muting. So, you're sort of like, oh do you want to go next, and then they're sort of trying to unmute themselves, and then someone else speaks, you know, it can be messy.' (Facilitator, internal interviews)

'[T]he other thing is, obviously, louder voices over quieter voices, you definitely noticed that in the online. It was really hard to facilitate that online versus in person.' (Facilitator, internal interviews)

Facilitators often expressed relief that the groups were smaller online, stating that any larger than seven or eight people would have been very difficult to manage. There is a consensus among facilitators that the online format created increased challenges for facilitation and maintaining the balance of voices. Amongst the key factors, they noted: having to split attention across multiple screens; not being able to rely on body language as members could either not be seen properly or did not use video (due to connectivity issues or personal preference); support being less readily available compared to in-person (when facilitators could raise a hand to request support).

Once the Assembly moved online, the space for relational dynamics, interpersonal connections and informal interaction was very limited – there wasn't any visible attempt to nurture the social dimension of the digital Assembly. Members often expressed missing the social side of the experience, which can be an important foundation for deliberative work. This highlights the importance of informal and relational spaces in citizens' assemblies. The in-person phase of the Assembly was credited by many in the internal interviews as setting the relational groundwork for the online phase by providing informal spaces for members to get to know each other and develop trust and rapport.

'It's never going to be the same as an in person, you know, around a table, having banter in-between meetings, building relationships. It's harder online. But yeah, I think the fact that they knew each other, and they'd built those bonds, really helped.' (Facilitator, internal interviews)

'It certainly took away from the social aspect of the Assembly almost completely, it totally destroyed that in effect, and that is an important thing. It was fortunate that people had had a few weekends together so they'd made friends and stuff like that. You could tell that people were missing that.' (Facilitator, internal interviews)

Conclusion

In this chapter we discussed the quality of deliberation and facilitation at the Assembly. While members' deliberations were consistently respectful and contained a high level of relevance and demands, we found little explicit reference made to formal evidence presented at the Assembly and very low levels of qualified justifications overall. The quality of facilitation was highly variable and while facilitators demonstrated a clear ethos of care and there were examples of excellent facilitation, we found a wide range in levels of experience and skills and an inconsistent understanding of both deliberation and the techniques required to facilitate this type of process. As is common in deliberative processes, certain forms of deliberation improved over the course of the Assembly (e.g. assertiveness, reason-giving) partly due to more proactive facilitation. While these improvements seem to occur following the move online, this is unlikely to be the result of the format but rather that the move occurred towards the end of the process. The move online appears to have led to additional challenges to facilitation, although the impact on members is less clear with some members finding it easier or harder to engage.

The combination and sequencing of participatory formats worked well. However, they relied heavily on verbal contributions and oral communication skills. The levels of inclusions were inconsistent across groups and weekends, with facilitators repeatedly commenting on the challenge of dealing with 'dominant voices'. Certain aspects of design and facilitation may have presented challenges to inclusion and achieving more robust deliberation: lack of systematic use of go-rounds that give everyone a chance to contribute at least once on a given question; limited reciprocal deliberation, that is, direct exchanges between members as opposed to exchanges with or via the facilitator; limited opportunities to contribute to plenaries; and limited framing of mutual challenge and scrutiny as a positive and helpful dynamic for the Assembly.