Scottish Crime and Justice Survey: user workshops - summary

5. Workshop 1: Survey Design Options

5.1 Introduction

This workshop centred on the SCJS design and a variety of ways in which the survey might be carried out in its new iteration. Discussion focused on the merits and limitations of a range of survey design options. This workshop represented a key point in the SCJS journey to select a preferred methodology for the survey under the new contract. Please see Annex A for the workshop paper, which was circulated to attendees prior to the workshop and contains details on the survey design options that were under discussion. 17 people attended the workshop.

5.2 Discussion

Discussion focussed on thoughts and input on the different survey design options under consideration. Comments made by attendees and responses by the SCJS team are arranged thematically.

Design Option 1 – 'Continuation of 2021/22'

This design option is a continuation of the SCJS methodology used for the 2021/22 year data collection. The 2021/22 survey is a multi-modal design developed in response to the COVID-19 pandemic. For more information, see the SCJS 'return to interviewing' page on the Scottish Government website. The key points of discussion at the workshop surrounding this approach were:

• Some attendees expressed support for the continuation of the 2021/22 survey design methodology.
• The SCJS team highlighted that over the pandemic changes were made to the survey design through necessity. Although it may appear that continuing the 2021/22 design is not a radical change in design, it is very different from previous SCJS years which adopted an exclusively face-to-face, in home approach. Considering this, it was stated that it is important that the 2021/22 survey year is not a 'standalone' year, meaning it cannot be compared to surrounding years' data. Whilst a key aim is for 2021/22 to form part of the SCJS time series, there is a risk this will not be possible. Therefore adopting the same design in the new contract mitigates this risk as the 2021/22 year would no longer be standalone in design.

Design Option 2 – 'Online First'

This design option is an 'online first' method, which aims to first engage survey participants via an internet based version of the survey. Fieldwork would commence by first contacting households via postal letter to request they select a random adult from the household and complete the survey online. If the household does not complete the survey via the internet, they would be re-contacted and offered other ways of completing the survey such as telephone, video call, or (circumstances permitting) face to face interviewing. The points of discussion arising at the workshop surrounding this approach are summarised below:

• Some attendees expressed support for this option as there is potential to reduce cost and improve efficiency.
• However, attendees also highlighted the risk in this approach as this method had not been trialled, and may result in the SCJS falling out of step with the other Scottish household surveys^[1].
• An attendee stated that, based on their own experience and research, there can be differences in response patterns when participants respond via face to face, or using telephone or online methods.
• The SCJS team noted that there has been support for the online first option amongst other stakeholders and colleagues, with a common reason for this being cost saving.
• However, it was highlighted by the SCJS team that there are potential issues on some aspects of data reliability, such as asking an adult in the house to randomly self-select a survey respondent. The SCJS team noted that they were aware through a recent international review that victimisation surveys that utilise online methods tend to have access to a population register, making the selection of a random adult easier.

Ethical Concerns on the use of Remote Methods

• Ethical concerns around the use of remote methods for the collection of sensitive data were raised. Specifically, concerns were raised about victims of partner abuse being able to respond to SCJS questions safely.
• The SCJS team confirmed they understand and always evaluate risk and ethical considerations of the survey, including remote contact methods. In the current survey, safeguarding measures are in place such as providing respondents with information on how to delete browsing history, and the provision of a 'quick exit' button in the online version of the questionnaire^[2].
• The SCJS team stated they will keep these considerations under review.

Time Series Impacts

• The potential impact on the time series was raised as one of the key things to consider when determining which survey design to select.
• An attendee questioned when the impact of a multi-modal survey design on the time series will be known.
• The SCJS team stated a key motivation is to maintain the time series, and decisions relating to survey design are taken with this in mind. It was stated that the impact on the time series is unknown until we are able to analyse the data from the 2021/22 survey year.

Rotational Topics

The workshop paper (Annex C) highlighted the potential for the survey to include 'rotational topics'. This proposal outlined a fixed or 'central' survey, and potentially introducing different modules to be asked on a rotational basis.

• On the topic of the potential inclusion of rotational modules, attendees asked for clarification that questionnaire modules that were rotated out would be rotated back in at a later point to permit trend analysis.
• The SCJS team stated that the potential to rotate modules in and out of the survey is still at an early stage of consideration. Any future decision on this matter will be made being mindful of user need.

Sample Size and Scope

• Attendees offered significant support for increasing the SCJS sample size.
• It was proposed that an increase in sample size may assist in matters of interest to those in attendance, such as capacity for demographic breakdowns and providing data by local authority.
• Attendees also offered support for increasing the scope of the SCJS sample.
• Suggested additions to the sample scope were those that have no fixed address and the inclusion of children.