Scottish household survey 2017: consultation responses analysis

4. LOOKING AHEAD

Views on further reductions and how they should be achieved

Question 11 asked respondents their opinions on how further cost saving measures should be implemented in the future should they be deemed necessary. The question was split into two parts. Question 11(i) was a quantitative question where respondents were asked to select which of the following four options they preferred if future cost savings were needed:

i. a reduction in SHS sample size;
ii. a reduction in frequency of SHS data collection;
iii. a reduction in SHS topic coverage; or
iv. other - please explain.

Question 11(ii) asked respondents to explain why they had selected the option they had in question 11(i).

Sixty nine out of 99 (70 per cent) respondents provided a response to question 11(i). Sixty nine out of 99 (70 per cent) respondents also provided a response to question 11(ii), although respondents that answered 11(i) did not necessarily question 11(ii) (and vice versa).

Close to a third (30 per cent) of respondents did not select an option (see Figure 4-1). Around 20 per cent of respondents felt that the best option if further reductions were required was a reduction in SHS topic coverage. The least popular choice was a reduction in sample size (10 per cent of responses). A similar number of respondents selected a 'reduction in data collection frequency' and 'other' (18 per cent and 19 per cent respectively). A combination of the other options was the main reason given for selecting 'other' e.g. combination of sample size reduction and reduction in topic coverage.

Figure 4-1: Preferences for options if further reductions to the SHS are required

Figure 4-2 shows preferences for options if further reductions to the SHS are required by sector. With the exception of the third sector, all sectors said a reduction in sample size was the least preferred method of reducing the SHS further, excluding no responses which was the most common response from the local government sector. Central government and local government responses preferred a 'reduction in topic coverage' or 'other'. A reduction in data collection frequency was the clear favourite of other public sector responses, whereas there was no clear consensus amongst third sector respondents.

Figure 4-2: Preferences for options if further reductions to the SHS are required by sector

A third of respondents who preferred option A (biennial topics) said a reduction in data collection frequency was their preferred method if further reductions were needed (see Figure 4-3), followed by 'other' at 24 per cent and reduction in topic coverage at 18 per cent. Nearly three in ten respondents who had selected option B said a reduction in topic coverage was their preferred method if further reductions were required, followed by a reduction in sample size (21 per cent). Reduction in data collection frequency was not popular amongst those who preferred option B, with only 8 per cent selecting this option.

Figure 4-3: Preferences for options if further reductions to the SHS are required by preference for option A or B

Reasons given for selecting a reduction in topic coverage mainly revolved around concerns with the other options. A reduction in sample size was seen as adversely affecting data, particularly at smaller geography levels or data for smaller populations/sub-groups. A drop in the frequency of reporting was also viewed with concern as it would limit the usability of the SHS.

Just over half of those who did not select an option for question 11(i), i.e. the' no responses' did provide comments to question 11(ii). A number of these comments raised concerns that further reductions would erode user confidence in the survey.

Comments from those who selected the option to reduce the SHS sample size, indicated that they believed a reduction in sample size would not affect the way they use the survey; however, a reduction in the breadth of the survey would. One response suggested moving to a model similar to that currently used by the Scottish House Conditions Survey ( SHCS) i.e. collect data over a three year period from a third of the total sample each year.

Those who selected a reduction in the frequency of SHS data collection saw this option as being the least different from the current situation in terms of sample size and the breadth of the survey.

Several responses to this question, the majority from local authorities, called for a further consultation if further reductions were necessary.

Impact of further reductions

Question 12 asked 'What would be the impact on the work of your organisation if there were to be a further reduction?' Respondents were asked to outline foreseeable impacts on their work if the SHS sample, frequency of results or topic coverage were reduced.

Eighty out of 99 respondents (80 per cent) provided a response to this question. Half of the non-responders were local government organisations. Although the consultation asked for views on how different types of reduction would affect respondent's work, many respondents answered in a generic manner rather than singling out which type of reduction would affect them and how.

The biggest concern raised by respondents to this question was a loss in precision in the SHS (a quarter of responses) if the SHS sample size was reduced. This in turn would reduce the use, value and confidence in the SHS which could (and in some cases would) force users to stop using the SHS altogether and seek out alternative sources. For example, one response stated:

'[The SHS] would cease to be a viable source of data. We would ultimately either have to look for alternative methodologies (for developing a local income model) or the Scottish Government would have to cease the commission and not pursue this strand of work.' - Anon

Respondents that currently use the SHS as a monitoring tool expressed concern that they may no longer be able to use the survey in this way. Several respondents expressed concerns on how changes would impact on smaller sub-group analysis that is currently carried out. A number of responses highlighted how reductions would affect work that is done on behalf Ministers. For example, an individual from the Carnegie Trust UK said:

'We would be concerned if the changes impacted on the Scottish Ministers requirement to publish an annual report on Scotland's progress. Reductions in frequency of collection and topic coverage could result in this outcome. This would have a direct impact on our ability to provide information for Parliament and civil society to hold the Scottish Government to account.'

Several respondents highlighted issues they would have in getting future funding:

'It may make comparisons with previous years more difficult and perhaps even impossible. This could impact on the service funding we get from our local authority and/or NHS partners as we may not be able to demonstrate adequately improved uptake of our services.' - West Lothian Leisure

Central Scotland Green Network said that they may need to 'ask Government to fund surveys to give us our results'. This could potentially negate any cost saving as a result of this consultation.

Three responses from central government respondents noted concerns about the impact of reducing sample size, frequency and topic coverage on the national indicators. For example:

'As there are numerous National Indicators that are updated using data from the SHS reducing the frequency of collection would mean that several indicators would no longer be able to be updated on an annual basis, which would be problematic given the emphasis of Scotland Performs on short term change.'

A number of respondents noted that they were unable to answer the question without knowing exactly what the proposals for reductions would be.

Other comments about the SHS or the consultation

Question thirteen asked 'Do you have any other comments about the SHS or this consultation?'

Thirty two out of 99 respondents to the consultation (including six of the free text responses) provided their views or comments on the consultation or the SHS.

The vast majority of the comments received came from local government (56 per cent) and third sector respondents (34 per cent). Whilst in general the third sector called for a larger and more robust survey, local government mainly wanted to prevent any further reductions to the sample size of the survey.

A number of organisations, including the LGBF Board, Accounts Commission, COSLA, SOLACE and some individual local authorities largely recognised that increased budget pressures required a review of the SHS, but thought that the consultation was limited in scope in terms of its focus on two specific options to reduce costs. Many of these respondents called for a pause in order to carry out a comprehensive or fundamental review of the SHS in the context of the wider SG survey landscape, i.e. a review and redesign that would bring into scope other national survey products too, and that involved relevant stakeholders, including policy and technical colleagues. South Lanarkshire Council called for a re-designed survey which is 'clearer about the key uses to which it is put and its function as a survey mechanism for the public sector'.

The LGBF board cited four factors that supported a larger scale review, including:

• the current limitations in using SHS data at local level;
• the need to ensure the survey better reflects policy priorities such as prevention, outcomes and partnership approaches, locality and place, community empowerment/engagement and reducing inequalities;
• the need to examine opportunities to strengthen links and harmonise existing survey approaches across the public sector, as noted by the Outcomes Evidence and Performance Board;
• Safeguarding the future of the survey as there is a danger that we continue to cut away at the sample size until it diminishes beyond use.

Some of these points were echoed by the Accounts Commission who also added their concern about public confidence in the satisfaction indicators under both SHS options.

COSLA felt that the consultation did not prioritise the link between evidence and policy, that the proposals were driven by cost cutting rather than genuine consultation on what was the right thing to do, and that there was no thought to what would happen if further budget cuts came. They viewed the options as "…short sighted solutions which do nothing to safeguard the SHS". It was noted that the models adopted for 2017 and 2018 were linked in order to provide a useable sample and that any decision made now would dictate the process to be followed in 2018. It was also noted that a review should consider how any new powers for Scotland post the Smith Commission could or should be reflected in the survey.

Several local government respondents referred to a need to rethink data collection methods to either all or part of the survey, particularly if there are further pressures on costs to reduce sample sizes. This was frequently linked to comments about the limitations of current local authority sample sizes.

In order to increase the robustness and usability of the survey, the following suggestions were provided in responses to question 13:

• Around one in four respondents would avoid reducing the sample size any further, as this will leave the SHS useful only at Scotland level but not at local level.
• Around one in eight would like to increase the sample size as currently the SHS is of limited use to local authorities and/or could be of more use, including one respondent that wanted more regular national surveys that provided local authority data.

• Several respondents would like to expand the survey to allow for more questions to be added both to the SHS survey (e.g. informal volunteering, participation in sport and sport clubs) and/or SSCQ and for these questions to be stable over time.

• North Lanarkshire Council would welcome an opportunity to fund a boost to local survey samples to avoid potential costs of resourcing/procuring their own research, while two local authorities mentioned that they already undertake their own local survey.
• More topics disaggregated by ethnic group

Among other suggestions were to employ data-mining of private and public data sources for the fact-based questions on households to allow the SHS to focus on perceptions rather than facts.

A few respondents used question 13 to comment on the consultation itself, with mixed views on the consultation document in terms of the ease of understanding and the use of Citizen Space and/or the Word template that was provided as an alternative. Another respondent noted how helpful the SHS team had been to them in supporting their work.

Finally, the Child Poverty Action Group, which had concerns about both options, requested that the Scottish Government perform a Children's Rights and Wellbeing Impact Assessment in relation to the consultation. (This point was made in response to question 2 but has been reported in this section).