Statistics.gov.scot improvement project: discovery user research report

Focus groups

Four focus groups were held from 25/06/24 to 02/07/24, with 9 people taking part. Participants were all external to SG, and all sessions were held online using Microsoft Teams. The focus groups were facilitated by Tom Farrington from Storm ID, with one member of the SG Open Data Team joining to help facilitate and note-take, and two members of the wider SG data community observing one session each.

The overall aim of these focus groups was to understand the perspectives of a range of external users, their expectations, needs, and how the website could be more user-friendly for them. The emphasis here was on group discussion, with some smaller group work involving fairly simple user scenarios and tasks.

Unfortunately, due to late cancellations, two of these sessions ended up with only one participant each. While this was not ideal, we simply spent longer on the usability section of the session, which allowed us to explore participants’ expectations and experiences in more depth.

The outline of the focus groups and user journey mapping scenarios were agreed in consultation with the Open Data Team. The outline is included in Appendix B, while the scenarios were selected from the same list as in Appendix A.

Anonymous qualitative data was generated in the form of notes typed by project team members into a secure Confluence Whiteboard, and digital sketches and typed notes on a Jamboard whiteboard by participants. This data was subject to a basic qualitative content analysis by the facilitator, being a version of template analysis as described in the Methodology section.

Interim analysis

The following analysis is interim in the sense of being produced immediately after the focus groups, before the remaining programme of research. Given the relatively similar aims of the workshops and focus groups, this stage of the analysis incorporates and builds upon the analysis of the workshop data to create broader interim themes. Each interim theme is accompanied by a set of sub-themes, each with illustrative quotations. These findings were presented to the wider Open Data Team on 11/07/24.

Interim theme 1: The site does not offer an intuitive user experience

From the current homepage as a starting point, participants found the website difficult to navigate and unintuitive to use. Specific issues are the volume of text, unexpected functions of familiar terms (e.g. ATLAS, data cart), unfamiliar terms (e.g. APIs, programmatically), two places to find two search bars, and the list of organisations presented without supporting information. Many participants struggled to decide where to start, and so had difficulty locating datasets, leading to frustration. During focus groups, one breakout group refused to accept the idea that any general citizen would start their search for data on statistics.gov.scot rather than Google. This was exacerbated by the confusing help journey, which leads the user away from the main site and is not up-to-date. The ‘ATLAS’ page, with the map of Scotland and some key statistics giving context, was seen by some participants as a more intuitive entrance point.

Data providers/publishers also found the process for uploading and updating datasets challenging, with some of these participants keeping their own step-by-step guidance to follow each time.

Sub-themes and illustrative quotations:

The site is jargon-heavy:

• “N-Triples doesn’t mean anything to me.”
• “they describe the data to someone who knows the data…not to a member of the public”
• “ATLAS is actually Scotland! Interesting, didn’t expect that.”
• “What is an API? What’s an R package? This isn’t oriented towards the citizen.”
• “…written by experts, for experts.”

Finding out how to use the site is a challenge:

• “I don’t know what there is or where to start.”
• “I feel I always have to click things I don’t want to click. I don’t feel in control.”
• “It’s hard to know what’s on stats.gov.scot. It might be on there, but I don’t know.”
• “I'm done, I’m off this website now.”
• “I’ve never figured out how to use this site.”

Publishing/providing or updating data is a convoluted experience:

• “the process is counter-intuitive…people forget what to do when uploading annual publications.”
• “It’s an ordeal, rather than just another job to tick off.”
• Offline support is excellent but online help/guidance is difficult to find and poorly presented.
• “[It’s] not easy to find help…it’s not really explaining anything.”
• “Interesting, help goes off to a different place/style. Am I still in SG land?”
• “‘Getting Started' is the most unhelpful guide ever!”
• “whenever I’ve had problems with the site I’ve had really good and helpful responses.”

Interim theme 2: The service is often slow and unresponsive…but not always.

During all research sessions, participants experienced some combination of slow loading times, timeouts, or errors. Data providers/publishers also found uploading slow. In all but one instance, these technical issues significantly marred participants' ability to effectively use the site, prompting some to leave the site and head to Google. The inconsistency is worth noting, as the one instance in which dimension-locking was relatively quick, and a dataset loaded relatively quickly for participants, seemed extraordinarily positive for all.

The dimension-locking feature was a particular point of frustration and confusion. More experienced participants actively avoided this, while less experienced participants soon gave up.

Sub-themes and illustrative quotations:

The site is slow to load:

• "Laggy, very slow. Many pages that fail to load, crash, or take forever to load."
• “When you click on something you expect it immediately.”
• “it takes days to upload some large datasets.”

Dimension locking is confusing for all participants and actively avoided by more experienced users:

• “Lock the value - what does that mean?”
• “why has that one got a little padlock next to it?”
• “…the horror: lock the value of all but two dimensions.”
• “Don’t know what a dimension or value is.”

Errors and timeouts are frequent:

• “Clicked on the search tab and got an error page that I’ve never seen before.”
• “[A] big red warning message that has made me feel sad!”

Interim theme 3: Simplicity first, complexity second.

While participants often simply wanted to quickly find and download known datasets, they were also keen to explore datasets by starting broadly and progressively disclosing complexity. The current processes for intentionally finding data and for discovering data are overly complex, and this was off-putting for participants. Participants also noted the tendency of the site, in content and function, to incorrectly assume a certain level of knowledge, sometimes statistical and sometimes site-specific. This left even very experienced participants feeling hapless and disempowered; further exacerbated by the aforementioned issues with jargon, online help and site performance.

Sub-themes and illustrative quotations:

Off-putting complexity of navigation options:

• “If I arrived here on a bus I’d want to get the first bus out of town.”
• “Browse the data. Don’t know what that means, but I’m not sure if that’s the data or the site.”
• “feels like I’m getting somewhere but nothing is quite covering what I’m looking for.”

Preference for starting broad and progressively revealing complexity:

• “I’d like to go in and it be very place-based…you could drill down to certain local authorities…you could profile what data there is for each geography.”
• “the Data Zone data is very detailed. It would be great to be able to group these by higher geographies…or…to have an A-Z and I could click on S.”
• “[I] want to break down by location but can’t.”

Participants feel stupid:

• “[It’s] assuming more knowledge than I have.”
• “this is so hard to interpret”
• “I feel like I’ve been failing as a statistician not understanding this site.”
• “…written by experts, for experts.”

Interim theme 4: Participants expect familiar and advanced search functionality

Participants were unsure which of the two search options would return the most useful datasets. Either way, searches often returned unexpected or irrelevant results, with no suggestions, causing frustration. SG participants suggested this could even be contributing to increasing FOI requests; a perhaps less radical corresponding position being that efficient search might at least reduce FOI requests. This chimed with the experience of an external researcher, who was keen to find data online to avoid FOIs wherever possible. Participants asked for a more advanced and user-friendly search capability to efficiently find the information they require, with suggestions of related datasets to aid further discovery.

Sub-themes and illustrative quotations:

Ineffective search functionality:

• “the search functionality doesn’t get the results you want.”
• “[Why] two search boxes? A bit more curation on the front page would be nice.”
• “When you search for something, such as hospitality and 0 results are given, perhaps it could return results for similar terms.”
• “I want things to be returned within 3 seconds or I’m leaving.”

Make data hard to find; expect FOIs:

• “This is why we get FOI'd so much - I can’t find my own data on here.”
• “[I’m] trying to find ways of getting information without having to use FOIs, as no one wants to go down [that] route.”

No advanced search:

• “Maybe a more advanced search might be useful…”
• [On a similar site] “advanced search is great and would cut out [refine] a lot.”

Interim theme 5: Comprehensive, up-to-date content, data, and information builds participant confidence

Participants immediately noticed indications that the site may be out of date and therefore not a reliable source of recent information, specifically the COVID-19 banner and the old name for NatureScot, both appearing prominently on the homepage. Participants were also disappointed to find some datasets without recent updates, and felt this undermined trust in the site. There was a sense from participants that the site was not fully delivering on its promise of providing ‘Open access to Scotland’s official statistics', as there are many datasets out there that are either not published here or easier to access elsewhere.

There was a genuine appetite for a feedback loop communicating who is accessing the site, what they are downloading, and how they are using this. For data providers, not knowing this left them unsure about how useful their datasets really are. Other participants found indications of relative popularity useful when assessing the suitability of datasets, and liked the idea of case studies showing how datasets had been used.

Sub-themes and illustrative quotations:

First impressions of the site being out of date:

• “Covid-19 up at the top - why is that still there?”
• “what is that big black COVID-19 thing? [Presumably] these are grim death stats on this site and this hasn’t been updated for a while.”
• “Scottish Natural Heritage changed its name years ago.”
• “Out-of-date data results in loss of trust.”

Expectations not met and confidence undermined:

• “It gives the impression it is all the data in one place, but it’s not.”
• “Loads of data isn’t on there…it’s quite misleading.”
• “why are you missing organisations on the right?”
• “who is it for? Do I want to the data to play with or do I want the answer?”
• “If we had a platform that worked well…you’ve got that engagement and you’d have people helping themselves.”
• “Not just a refresh but a rethink.”

Appetite for up-to-date information on dataset access and usage:

• “I don’t know who uses it…”
• “it felt like we were punting the data out into the void.”
• “[Number of] views indicates you are looking at things that people are using, and not niche.”
• “…interesting to see how many people have used it.”
• “There are stories about how the data can be used, commercially, socially.”