Statistics.gov.scot improvement project: discovery user research report

Research overview

Background

Since the site launched around 8 years ago, the only targeted user research has been a short ‘true intent’ survey, which appeared to users of the site over a 3-month period in Summer 2021. This found that most of the 362 respondents had come to the site for the first time, to find out about the number of COVID-19 cases in specific areas. Almost three quarters of respondents did not fully complete their task.

The SG Open Data team, responsible for maintaining statistics.gov.scot, wish to make improvements to the service to improve the experience of users, and to ultimately improve data publishing and maximise the use and re-use of data. However, of the evidence that exists, there is uncertainty around who uses the site, and users appear to struggle to use the statistics.gov.scot service to accomplish their goals.

To ensure the design and implementation of improvements are fit for purpose, the team agreed to run an Agile Discovery to understand more about who the users are, and what they want and need. This report summaries the user research activities and findings conducted as part of the Discovery.

The team used Atlassian Jira and Confluence to respectively manage and document the project. While this report should be taken as the summative and final output of the user research workstream, the repository of information and outputs on Confluence and Jira offer further details of activities, draft outputs, and ways of working.

Methodology

To attend to these questions, the programme of user research took a qualitative, multi-phased approach, comprised of workshops, focus groups, interviews and co-design sessions, plus basic content analyses.

Prior to recruitment and any data generation, the project team completed a Data Protection Impact Assessment (DPIA), Risk Assessment Register, and User Research Ethics Plan. The DPIA was reviewed and approved by the IA&DP team, the DPO, and the IAO, with feedback incorporated where given. The Ethics Plan was reviewed and approved by an SG UR Lead, with feedback incorporated where given. Approved versions are stored in eRDM.

The final programme of research consisted of:

• 4 workshops with 18 SG participants.
• 4 focus groups with 9 external participants.
• 11 one-to-one interviews with experts and stakeholders, each lasting between 30 mins to 1 hour.
• 2 co-design sessions with 18 SG participants (including 10 repeat participants).

Data collection

Primary qualitative data was generated in the form of typed and handwritten notes, as well as participant sketches, both digitally (via online whiteboards) and on paper. Secondary qualitative data was gathered in the form of a high-level overview of emails to the shared mailbox statistics.opendata@gov.scot, and desk research covering various related documents, e.g. user guidance, reviews of other open data platforms. Data generated through the project is typically anonymous by nature, but where any names or other identifying information are given by participants, this is immediately redacted.

Data analysis

During each phase of the research, data was subject to basic qualitative content analyses. Analyses of written data followed a relatively swift and pragmatic version of template analysis, whereby initial coding of a subset of data formed a template to guide the analysis of the remaining data, with the template adjusted as required to incorporate new codes and sub-themes. Analyses of visual data were much simpler, where discussion and description of the images amongst the participants, and later the project team, led to understandings of the underlying principles.

All analyses were reviewed by the project team and further refined through presentations and report writing. This led to preliminary themes after the workshops, interim themes after the focus groups, complementary themes from interviews and co-design sessions, and a final summary of practical issues and overall findings. This was an iterative process, with themes becoming refined and stabilised through each phase of data collection and analysis. An illustrative example of these relationships is shown in Appendix F.

User groups

Largely for recruitment purposes, the research began by assuming that most users of the site were likely to be usefully categorised according to the following personas, adapted from existing research with the Office for National Statistics (ONS) and summarised here:

• Passive citizen: ‘I have a passing interest in the figures behind the headlines that affect me.’
• Inquiring citizen: ‘I want unbiased facts about topical issues.’
• Information forager: ‘I want local data to help me make practical decisions for my job or life.’
• Policy influencer: ‘I work in government or policy and use data for benchmarking or comparison.’
• Expert analyst: ‘I am an expert statistical user who creates my own analysis from data, using bespoke statistical models.’
• Technical user: ‘I create my own datasets from merged data.’
• Third-party organisation: ‘I use data or summary statistics for certain topics to further the aims of my organisation.’

These personas were initially helpful in recruiting users and identifying gaps in recruitment, but were subject to revisions as the research progressed and understanding about actual users was refined. For example, we immediately added data publishers as a separate group, being those who provide data to statistics.gov.scot (technically they are submitting draft datasets to be published on the site, hence sometimes being referred to as data providers).

As we worked through iterations of user groups and the user needs catalogue (see Appendix E for the final version), the project team determined that the following broad user groups were reasonably representative of the site’s actual (and potential) user base:

• general citizens (inc. those who self-identify with low digital literacy)
• inquiring citizens (e.g. charity/third sector users)
• commercial users (i.e. private sector, e.g. energy company, bank)
• technical/expert users (e.g. academia, data journalist, developers)
• public sector/policy influencers (e.g. policy advisers, statisticians, councils, health boards)
• data publishers (both regular and infrequent)

As a further development of the user groups and user needs catalogue, Appendix E also contains a summary of problem statements and technical challenges, related directly to user needs. These are also prioritised, with the intention being to help facilitate any improvements to the service.

Recruitment and participation

Participants were recruited using convenience snowball sampling, where known contacts were invited to participate through established networks, and asked to pass on the recruitment information to any other interested parties. The project team distributed recruitment information and screeners through personal invitations to participate, and the following online channels:

• Internal Microsoft Teams channels posts and messaging e.g. Community of Practice for Data Standards and Open Data.
• SG Yammer:
• Scottish Official Statistics – Community.
• Digital Data and Technology Profession (DDAT) – Community.
• Office of the Chief Statistician (OCS) Newsletters on 22/04/2024 and 29/04/2024.
• Open Data Scotland Slack channel.
• Data Lab Community forum posts (various groups).
• ScotlandIS
• LinkedIn

These posts reached 3000+ potential participants, of which 46 participated.

Specific details of participant roles and responsibilities are not included here to prevent identification. Participants can be roughly aligned with our user groups as follows:

• general citizens (0)
• inquiring citizens (1)
• commercial users (4)
• technical/expert users (14)
• public sector/policy influencers (13)
• data publishers (both regular and infrequent) (14)

There are some overlaps in role, but each user is only counted once, based on their principal use of statistics.gov.scot e.g. those in the public sector who are data publishers are counted as data publishers.

Although no general citizen and only one inquiring citizen users expressed interest in participating, these citizen user groups were retained to represent the needs of groups known to the team, both through the aforementioned ‘true intent’ survey and reviews of mailbox feedback. While opportunities for citizen participation were made publicly available, these were all online, and so it is perhaps unsurprising that no general citizens with low digital literacy expressed interest.

These gaps in participation were deemed acceptable for this Discovery, based on:

• the richness and utility of the findings generated with the remaining user groups and
• the reasonable assumption that if experienced users with high digital confidence were struggling with basic tasks, then relatively inexperienced users with lower digital confidence would also struggle.

Still, it is important that any future development of the site and service involves recruiting and testing with members of the general public (including low digital literacy). Ideally, regular cycles of user testing and evidence-based refinements would be incorporated into Alpha and Beta phases, within each sprint, for example. The aims of each round of testing should be clear and logical, e.g. it may be that testing with users of assistive tech would only make sense after any obvious accessibility issues (such as contrast ratios or basic keyboard navigation) have been resolved.

The research session schedule and participation were very well-organised by the Open Data Team, and participants were highly engaged, participative, and talkative. This is worth emphasising: it was a pleasure to run the sessions with SG.

It was also very helpful to have various members of the Open Data Team and SG more widely observing and/or participating in sessions as appropriate. Such first-hand experiences of real-time user feedback are always valuable for people in and out of the project team, towards appreciating both the reality of how the site is used, and the relative importance of making improvements.

The remainder of the document offers overviews of each stage of the research before summarising the main practical issues and overall findings. Outlines for each research session can be found in the Appendices.