Coronavirus (COVID-19): Our shared role in containing the virus: Analysis of responses submitted as part of the Scottish Government's second public engagement exercise

2) Methodology

Over the course of the platform “challenge”, 564 ideas were published on the site, with 1,633 comments and 1,242 users registered. Comments were also posted on social media in response to the promotions, including 229 Twitter replies and 1,900 Facebook comments. A small number of emails were also received via the site administration. We have included an assessment of these contributions throughout the report.

Once posted, ideas had “topics” added to them by members of the moderation team and members of the Scottish Government’s Social Research Group. Topics were taken from a thematic “coding” framework that was developed by the Social Research Group to correspond with the subjects in the seeded themes and anticipated commentary. The thematic coding framework was updated as new topics emerged during the challenge. The added topics are visible on the website, and ensured that contributions were rapidly categorised for further analysis. An ‘optimisation' exercise was conducted once the challenge was closed in order rationalise the topics to ensure consistency, and to make sure similar topics, or different spellings, e.g. “reopening” and “re-opening” were grouped together^[4].

Although responses were about a tenth of the volume seen in May, the level of engagement is nonetheless high relative to many Scottish Government consultations in a notably shorter timescale. During the challenge, social researchers identified emerging themes through analysis of the most engaged-with threads (by number of ratings, and number of comments) as well as searches by theme. Researchers also analysed ideas and comments as they were posted in real time. Therefore themes also emerged from threads with lower overall engagement. Throughout the challenge, social researchers produced daily analytical notes.

Once the website was closed for submissions, researchers continued to carry out further analysis by themes. Given that many ideas were rated or commented on a small number of times, researchers identified the “most engaged-with ideas” (idea threads with the highest number of ratings, or the highest number of comments) (See Annex A). These included “highly rated” ideas – ideas with high engagement (among the top 20 threads by number of ratings) and high average rating (above 4.5 stars) – but it also included ideas with high engagement by number of comments/ratings but with lower average ratings. The high engagement with these ideas was taken by the researchers to suggest a degree of traction among contributors. Further analysis took place to group ideas together by topic, and ensure that topics which attracted a large number of less engaged-with ideas were nonetheless identified if they had not been among the most engaged-with ideas.

As a number of ideas covered similar issues, we have grouped these, where obvious, under the ‘topics’ in the thematic framework and reported the total number of comments and ratings as a measure of engagement in Annex B. We have attempted to provide an overview of commentary on key ideas, but have not, in the time available, coded each comment as being in support or opposition to the original idea. Furthermore, in many cases, ideas may differ in their advocated approach and context, to the extent that definitively judging the level of support for them, relative to other ideas on the subject, is not possible. Alternatively, the number and characteristics of users rating ideas may not be comparable across ideas, nor the search strategies employed by users to find relevant ideas to them, so additional caution should be exercised in comparing average ratings.

Over the course of the “challenge”, it was clear to researchers working on emerging themes that there was a broad degree of consistency from day to day. While new restrictions were announced by the First Minister on 9 October, during the course of the challenge, this largely caused commentary to be directed at the specific measures, rather than potential possibilities, and not seen to change the overall sentiment. After the challenge, researchers, working within the seeded themes, organised the key ideas direct from the final dataset and reported the main solutions and attitudes expressed. The two approaches were found to yield similar findings.

Respondents were self-selecting and do not represent a random sample of the population of Scotland. We did not require evidence of residence in Scotland, or ask people to report their demographic characteristics, so do not have independent evidence of representativeness. It is likely that the group who engaged with the platform were the digitally included, and the peaks in activity on Day 1 and Day 7, suggest that many users were also likely to be active on social media. This is a reason for caution in interpreting the findings. However, it must be understood the platform was designed to solicit ideas from the public and give them the opportunity to comment on the Scottish Government’s approach, not to measure their attitudes. All quotes used in this report are verbatim, although some have been shortened. Full quotes are indented in the text, and single quotation marks used where the report’s discussion includes terms or phrases taken directly from from individual contributions.

Finally, this analysis does not set out to be a detailed examination of all the ideas and their relative effectiveness, accuracy or relevance to the issues. This is an overview of what those who engaged with the platform said to us. Given the rapid nature of this analysis, it has not been possible to be comprehensive, or definitively quantify the balance of opinion on the platform. It is one part of the information to be considered by decision makers.