Long term survey strategy: mixed mode research report

Appendix B – Topic guide for stakeholder interviews

Introductions (5 mins)

• Check time still OK and how long they have.
• Introduce self and role in research team – Ipsos working with Professor Peter Lynn, Lead on the ESRC’s Survey Futures project.
• Explain the research:
  • Commissioned to inform the SG LTSS
  • Focused on understanding pros, cons and possible mitigations associated with different mixed mode options for the three main SG surveys – the Scottish Health Survey, Scottish Household Survey and Scottish Crime and Justice Survey
    • (for non-research audience) – by mixed mode, we mean approaches that combine different ways of collecting data, such as inviting people to complete a survey online initially but following up non-responders or specific target groups via telephone or face-to-face. To date, with exception of Covid period and experiment with telephone data collection for the Crime Survey, all three surveys have been conducted F2F, so moving to mixed mode would be a significant departure that requires careful consideration.
  • Aim is to ultimately to produce a framework that can help guide SG discussions and decisions re. this, tailored to reflect the specific needs of each survey.
  • This will be informed by:
    • These initial interviews/workshops with key stakeholders, which are focused on ensuring the research is informed by a clear understanding of your priorities for the surveys.
    • A desk-based review of: a) literature on mixed mode surveys and b) a review of relevant surveys in the rest of the UK and elsewhere that cover similar topics to the SG three core surveys, to map what modes are used / learning from any transitions to new modes.
    • Further ‘expert interviews’ with those involved in mixed mode survey research and transitions outwith Scotland
    • A workshop with the SG to refine the framework.
• Interview today will focus on your priorities for the survey(s) you are involved in and your views on possible trade-offs between competing priorities.
• Pleased you’ve agreed, but taking part is completely voluntary – if you don’t want to answer particular questions, or don’t want your answers attributed, just let us know. We can share a copy of our notes on your interview afterwards and you can flag anything you’d rather wasn’t included (either at all, or in an attributable manner)
• Would like to record interviews, with your permission, so we can listen back and make sure we’ve captured your views accurately
  • Will be securely stored by Ipsos, not shared outwith the research team, and destroyed at the end of the project
• With your consent, we may also include quotes in the outputs.
  • We wouldn’t attribute these to named individuals – we’d use, e.g., ‘Scottish Government key stakeholder 2’.
  • However, we realise that given small number of people involved in this field in Scotland you may still feel you’re identifiable from this – so if you’d prefer not to be quoted, that’s fine.
  • We would like to include a list of organisations interviewed for the research.
  • We’ll come back to you at the end, and you can let us know whether you’re happy to be quoted and whether you’re happy for your organisation to be listed.
  • We’re also happy to share a copy of our notes on the interview before you decide.
• Any questions before we start?
• Ask for permission to record – for ourselves to listen back to, won’t share. Will hold recordings securely and destroy at end of project.
• OBS on – record consent to take part and be recorded.

Their role (5 mins)

• Ask them to describe their role, and how this relates to SG surveys
  • How much of your role is involved with surveys / data from surveys?
  • Which of the three main SG surveys are you involved in / are relevant to your role?
  • What other surveys do you use / are you involved with?

Uses (and future uses) of survey data (10 mins)

• What are the main ways you use the SHS/SCJS/SHeS (as applicable) in your role?
  • Are there things you particularly rely on the SHS/SCJS/SHeS for? Examples?
• What about (other) people within the SG? How do they use it?
• And external (non-SG) stakeholders?
  • Who else uses the data, as far as you’re aware?
  • For what purposes?
    • Outside the SG, who are the surveys most important to? And why?
• [See Annex for summary of stated uses and go through anything not already mentioned] Is this something the survey is used for at the moment? Who uses it in this way?
• Are there any tensions between what different stakeholders need from the surveys? E.g. different internal SG stakeholders (Ministers, policy, analysts), or internal vs. external stakeholders.
• Thinking about the next 2-5 years, do you think the Scottish Government will use the SHS/SHeS/SCJS in the same way?
  • What else might the Scottish Government use them for?
  • Anything they think the SG won’t use them for anymore? Why?
  • Anything you’d like to be able to use them for (but can’t currently)? (e.g. bigger subgroup samples, additional biomeasures)
  • What about further into the future – 6-10 years down the line?
• If time, for SG survey folk: How will future priorities be decided for the three surveys?
  • Will you look at them collectively or individually?
  • Who leads / is involved in deciding this?
  • What comes first in decision-making – deciding on budget, or deciding on priorities for users, or somewhere in between these?

Priorities and views on quality (unprompted) (10-15 mins)

• From your perspective, what are the main measures / (sets of) questions that are most important to you in the SHS/SCJS/SHeS?
• And what are the main features you are looking for from the SHS/SCJS/SHeS?
  • What are the main current features you value and would want to keep?
  • What are the main things you think would want to improve?
  • Probe fully on both.
• What does ‘high quality survey data’ mean to you?
  • What elements of quality are most important to you?
• What about ‘value for money’ – what does this mean in the context of SHS/SHeS/SCJS?
• What do you think ‘high quality survey data’ means to others, including:
  • Ministers?
  • Other Scottish Government colleagues. Probe RE:
    • (other) research colleagues
    • (other) analysts
    • (other) policy colleagues)?
  • External stakeholders? Probe RE:
    • Local authorities
    • Third sector organisations
    • Survey contractors
    • Academics
    • Others?
• And what about ‘Value for money’? What does this mean to others who might have an interest in the survey(s)?
• What contexts do you need to talk about the quality or VfM of the survey(s)? E.g. Submissions to Ministers, UKSA correspondence, grant proposals
  • (How) does how you need to frame these concepts vary depending on who you’re speaking to about them?
• When you’re thinking about cost vs. quality, how do you approach thinking through what trade-offs of quality are or are not acceptable?
  • Do different stakeholders weigh costs and quality differently? In what way? (i.e. do some think in terms of lower cost the better, and others not consider cost much/at all)

Priorities (prompted) (15-20 mins)

For anything not mentioned spontaneously, probe on how important a consideration each of the features listed below is:

General probes:

• How important is this to you personally, in terms of your needs and what you want to see from the survey(s) in the future?
• How much of a consideration/concern you think it is for (other) people in the Scottish Government when they’re thinking about what they need from the survey(s) – e.g. Ministers? Policy colleagues? Other analysts? What about external stakeholders?
• If you had to prioritise, what are the main things you would want to see in SHeS/SHS/SCJS in future – either things you would want to keep or things you feel need to change?
• Are there any of the things we’ve discussed that you see as a lower priority (for you / others)?

• Maintaining a good response rate?
  • Overall, and among particular sub-groups (e.g. by age)?
• The representativeness of the achieved sample – are the people who took part reflective of the Scottish population?
  • What elements of representativeness are particularly important to them / others? (e.g. geographical, age, etc.)
• Maintaining or increasing the sample size – overall and for particular sub-groups – e.g. age groups, local authorities?
  • Which sub-groups are particularly critical to them / others?
• Maximising the quality of respondents’ experience of taking part
  • What elements of this are most important to them / others?
• Reducing (or not increasing) the total cost of the survey?
• Increasing or maintaining the timeliness / speed of results from the survey?
• Avoiding any break in the time series that might reduce ability to compare with years prior to that break?
• Generally maintaining the ability to track change over time?
• Maintaining the credibility of the survey with external experts?
• Ensuring all respondents understand what they’re being asked in the same way?
• Being able to ask the same questions, in as similar way as possible to now? (e.g. questions with long sets of answer options, questions that require physical measures)
• Encouraging honest answers to sensitive questions?
• Being able to enable different (complex) routes through the questionnaire?
• Ensuring the survey is inclusive of people with different characteristics/needs?
• Maintaining a face-to-face fieldforce in Scotland?
• Reducing the impact of the survey(s) on the environment?

• SHeS – continuing to collect biological and/or physical data?
• SHS – continuing to collect objective data on the physical condition of people’s homes?

Attitudes to mixed mode (10-15 mins)

• Before this interview, how much have you been involved in thinking about or discussing the potential for using different modes of data collection on key Scottish Government surveys – i.e. moving away from a purely face-to-face design?
  • In general?
  • Re. the SHS/SHeS/SCJS?
• If have been involved in discussions:
  • Based on what you know/have discussed so far, how do you feel in general about the potential for moving SHS/SHeS/SCJS away from a purely face-to-face design?
    • From your point of view, what are the potential benefits?
    • And the potential risks?
  • And how do you feel in general about the potential for moving SHS/SHeS/SCJS to a design that combines different modes? (Want to get general sense of favourability/nervousness about this)
    • From your point of view, what are the potential benefits?
    • And the potential risks?
  • Are there any particular combinations of modes you think are more promising/more interesting for SHeS/SHS/SCJS (refer to annex A if needed)?
    • Why? What could be the benefits of this design?
    • What, if any, questions/concerns do you still have about moving to this design?
  • Are there any mixed mode approaches you are not considering seriously at this stage, or are not sure could work for SHeS/SHS/SCJS?
    • Why not? What are the main issues/challenges with it?
• If not been involved in discussions much:
  • At this stage, how do you feel about the potential for moving away from a face-to-face design for SHS/SHeS/SCJS to a design that makes greater use of other modes – online, telephone, or paper? (If necessary, talk them through different possible options)
    • What do you think could be the benefits (probe if possible around each mode/combination)?
    • What would be your main concerns?

Ending and next steps (5 mins)

• Do they have any questions for us about the research?
• Check they are happy to:
  • Be anonymously quoted?
  • Have their organisation listed in list of participants?
  • And whether they would like a copy of their notes (record this in recruitment sheet and record when you have sent these, via Itransfer and password protected).
• Discuss and agree next steps, e.g.
  • Anything they’ve agreed to send us
  • Or more general consent to get back to them if we have any further questions.
• Thank and close.