Coronavirus (COVID-19): framework for decision making - overview of public engagement

3. Methodology

Over the course of the platform 'challenge', 4122^[2] ideas were published and 17,966 comments were posted. These came from 11,692 registered users, of whom 3,274 submitted ideas. Members of the public also engaged with the Scottish Government through email - 136 emails received during the challenge were also included in the analysis.

Ideas had 'topics' added to them by members of the moderation team and members of the Scottish Government's Social Research Group while the challenge was open. Topics were taken from a thematic coding framework that was developed by the Social Research Group to correspond with key themes of the Framework document and the Test, Trace, Isolate, Support strategy.^[3] The thematic coding framework was updated as new topics emerged during the challenge. The added topics are visible on the website, and provided a rapid 'coding' of ideas to the topics they covered for further analysis. An optimisation exercise was conducted once the challenge was closed in order to make sure similar topics, or different spellings, e.g. 'reopening' and 're-opening' were grouped together.

With a high volume of information on the platform, during the challenge social researchers identified emerging themes through analysis of the most engaged-with threads (by number of ratings, and number of comments) as well as searches by theme. Researchers also analysed ideas and comments as they were posted in real time. Therefore themes also emerged from threads with lower overall engagement. Throughout the challenge, social researchers produced daily analytical notes. Once the website was closed for submissions, researchers continued to carry out further analysis by framework themes and the most engaged with ideas.

Given that many ideas were rated or commented-on a small number of times, researchers also identified the 'most engaged with ideas' (idea threads with the highest number of ratings, or the highest number of comments) (See Annex B). These included 'highly rated' ideas - ideas with high engagement (among the top 20 threads by number of ratings) and high rating (above 4.5 stars). These were judged by researchers to be ideas with a wide degree of popularity. Researchers also identified ideas with lower ratings within the top 20 most commented-on group which attracted more polarised opinions. We have attempted to provide an overview of commentary on key ideas, but have not, in the time available, coded each comment as being in support or opposition to the original idea.

As a number of ideas covered similar issues, we have grouped these, where obvious, under the 'topics' in the thematic framework and reported the total number of comments and ratings as a measure of engagement in Annex C. However, in many cases, ideas may differ in their advocated approach to the extent that judging the support for them, relative to other ideas on the subject, is not possible. Alternatively, the number and characteristics of users rating ideas may not be comparable across ideas, so further caution should be exercised in comparing average ratings.

Over the course of the 'challenge', it was clear to researchers working on emerging themes that there was a broad degree of consistency from day to day. After the 'challenge', researchers, working within each of the main themes of the Framework and Test, Trace, Isolate and Support Strategy organised the key ideas and reported the main solutions and attitudes expressed. The two approaches were found to yield similar findings.

Respondents are self-selecting and do not represent a random sample of the population of Scotland. We did not require evidence of residence in Scotland, or ask people to report their demographic characteristics, so do not have independent evidence of representativeness. It is likely that the group who engaged with the platform were the digitally included and thus this is a reason for caution in interpreting the findings. However, it must be understood the platform was designed to solicit ideas from the public and give them the opportunity to comment on the Scottish Government's approach, not to measure their attitudes. All quotes used in this report are verbatim, although some have been shortened.

Finally, this analysis does not set out to be a detailed examination of all the ideas and their relative effectiveness or relevance to the issues. This is an overview of what those who engaged with the framework said to us. Given the rapid nature of this analysis, it has not been possible to be comprehensive, or definitively quantify the balance of opinion on the platform. It is one part of the information to be considered by decision makers.