Scottish Household Survey User Engagement Report 2020

5. Results

5.1 Pre-development

The online survey sent out prior to developing a new data dissemination tool collected users’ view on the current data dissemination format and what they required from the new data tool. Asking users what they liked the most about the current MS Excel files, most users thought it was good to find all information in one place, as well as having a clear and simple presentation.

“I like the quantity of data available and relative ease of use of the tables”

“User friendly layout, easy to understand and interpret.”

“All data for a number of years is accessible in one place”

Asking users what they liked the least about the MS Excel files, there was frustration with local authority data not being published at the same time as national data as well as the large size of the Excel files themselves.

“Sometimes a little out of date by the time they are released.”

“The document is so large and the number of tables I am interested in is only a small proportion that it can be time consuming locating what I want.”

“I dislike the layout of the tables; it is irritating to scroll down given page (say housing) only to find data is supressed.”

When asked which functionalities that were required in the new local authority tables replacing the MS Excel files, over eight in ten said they needed interactive tables and charts. Over half of the respondents also required downloadable reports, graphs and tables. There was also a considerable proportion that reported the need for micro data.

Note: 28 respondents. Source: Pre-development survey 26-06-2019 to 15-10-2020

5.2 During development

With the required user functionalities and positive aspects of the current MS Excel files in mind, the Scottish Household Survey team developed an interactive dashboard called the Data Explorer. To involve users in the development process, a qualitative engagement approach was taken.

We invited users to test a beta version of the Data Explorer. Presentation and consultation was carried out by one of our team members either one-to-one or as focus groups between February and July 2020. In total, 47 users tested and provided feedback through an online questionnaire, as well as open-ended written and verbal comments on the beta version. There was a diverse group of users, including statisticians, policy analysts, local authority researchers, children, teenagers, students and academics.

We created an online tutorial and quiz as a way to engage users in the beta testing and to gauge how easy the Data Explorer was for individuals to navigate around and use. Most users could answer the questions in the online tutorial correctly, suggesting that the Data Explorer was easy to use and accessible.

The majority of the written and verbal feedback given during the beta testing phase was positive. Here are some examples:

“I am impressed with how user friendly the explorer is and how easily you can find information and manipulate it”

“In general, I found the layout easy to read and navigate. I had a couple of niggly issues with the charts.”

“I like how easy it is to navigate to each topic and find the information you are looking for and then be able to download it. I also liked how easy it was to update/change charts.”

When asked what they liked the least, users mentioned the format of reporting.

“Number of different options of tables in dropdowns is slightly overwhelming”

“I think you would need to use the 'get your own reports/data' a bit more to make sure you are getting the information you need.”

“Overall navigation was straightforward and layout clear. However, due to amount of different information presented, would potentially be quite difficult to locate one particular table if you were looking for a specific piece of info - splitting up tables in drop-downs into smaller groups might be helpful.”

The main issue mentioned was the difficulty to navigate around the many questions. As a response to this feedback, the team created a search bar and streamlined the question titles.

5.3 Post publication

The Scottish Household Survey data explorer and reports for 2019 data were published on the 15^th of September 2020. An online survey was sent out after public release of the various new products to gather user feedback. Response rate to the online survey was low (19 respondents), so any quantitative findings should be taken with caution. However, the open-ended questions in the questionnaire suggest that users in general are happy with the Data Explorer, the new format of the annual report and additional reports.

Data Explorer

The majority of users that had used the 2019 Data Explorer reported positive views of it. Almost eight in ten found the tool very useful, and two in ten thought it was quite useful to them. When asked through an-open ended question what they liked the most and the least about the new data tool, users mentioned the positive aspects of its flexibility, speed and accessibility. However, there was one report of a one-time crash of the dashboard.

Note: 13 respondents. Source: Post-publication survey 15-09-2020 to 15-10-2020

Comments about the data explorer:

“I really like the ability to choose what data to compare. The freedom of choosing specific data I am interested in really speeds up my analysis.”

“It is fantastic to be able to have the flexibility to query the data for the area of interest, or to download a topic report. A very user friendly tool.”

“Good, easy to navigate but seemed to crash once I tried to customise a chart.”

New annual report format

The majority of users were happy with the new annual report format and 95% found the various products useful (Figure 3). Furthermore, there was no evidence to suggest that users missed the previous, substantively longer annual report, with over seven in ten being content with the new headline report replacing it and only five percent not being content.

Note: 19 respondents. Source: Post-publication survey 15-09-2020 to 15-10-2020

Note: 19 respondents. Source: Post-publication survey 15-09-2020 to 15-10-2020

When asked which publications the users had used/read, figure 5 shows that several respondents accessed more than one product. This suggests that the various publications fill different purposes and potentially complement each other. Furthermore, it shows that different users require Scottish Household Survey information and findings in different formats. It is therefore more beneficial to produce several shorter publications than focusing on producing one, longer annual report, as was done previously.

Note: 19 respondents. Source: Post-publication survey 15-09-2020 to 15-10-2020

Asking users about the specific products released in September 2020^[1], the majority of those that read the headline report found it useful. To gain insight into what can be improved, an open-ended question was asked if there was something users particularly liked or disliked about the report. The comments suggest that users liked that it was informative and clear, and one user thought more detailed analysis would make it even more useful.

Note: 15 respondents. Source: Post-publication survey 15-09-2020 to 15-10-2020

Comments about the 2019 Headline Report:

“Informative precise and easy to read.”

“Well laid out, informative.”

“More equality analysis would be useful - the headlines might mask trends in sub-groups.”

Feedback on the 2019 Key Findings infographic suggest that the publication was useful to most of the users. Less than one in ten found it not very useful.

Note: 12 respondents. Source: Post-publication survey 15-09-2020 to 15-10-2020

For future improvement, users were asked if there was something in particular they liked or disliked about the key findings infographic. The comments suggest that users particularly liked the look and clarity of the report. There were no comments made about what users particularly disliked about the product.

Comments about the 2019 Key Findings Infographic:

“Consistent layout and appearance draws out key messages well.”

“Overall fascinating!! Makes me realise the importance of education for generally improving people's lives!!”

“Nice layout, easy to read important points.”

“Good reader friendly format.”

Asked through an open-ended question if there was anything users felt was missing from the 2019 SHS publication format, the majority of respondents expressed satisfaction with the current format. This again reinforces the notion that the new SHS products are useful, suitable, and overall endorsed by our users.

“Is there anything you feel was missing in the 2019 SHS publication format?”

“No. Generally I was impressed by the range of outputs that were made available.”

“No, very informative.”

“No, very detailed.”

“Could be slightly longer, difficult to quantify how to increase length but slightly more detail in some section.”

Lastly, the post-publication survey invited users to give any further comments about the communication of SHS 2019 data in order to receive feedback on any issues that were not covered by the questions asked in the survey. The answers received were all positive and give further evidence that users are satisfied with the new products and communication of such.

Further comments on SHS 2019 reporting:

“It is fantastic to be able to have local authority data available on the same day as the publication of the national data. The work that the team has put into making the outputs of SHS widely accessible to a range of audiences. A really valuable resource for Scotland. The team should take the credit for an excellent approach to dissemination of the results. I particularly liked the video accompanying the data explorer as it gave a great overview of what you can do with the data explorer tool. Fabulous work by all!”

“Well done on transforming the reporting of this valuable data!”

“I am happy with whole study and method of comms.”

The new 2019 data reporting format also received some comments via the SHS mailbox around publication time.

“Congratulations on an excellent set of outputs relating to the 2019 Scottish Household Survey!!”

“The new data explorer is a fabulous replacement for the local authority tables, and it is fantastic having access to local data on the same date as the national publication.”

“thanks for the report today, it’s great, really helpful, and congratulations on a really effective presentation of the data too – it really worked for me.”

“Love the interactive flip page pdf noise probably more than I should!”

“animation … I absolutely love that approach to presentation … congrats again on pulling all the info together.”

“The flip book noise is indeed ridiculously pleasing! Well done to Emma and your team on an impressive set out outputs.”