Scottish household survey 2017: methodology and fieldwork outcomes

6 Survey Response

Summary

• The final number of social survey interviews achieved was 10,683, matching the target, a response rate of 64 per cent.
• The SHS response rate has fallen in recent years, having fallen by 4% since 1999. The reduction in fieldwork performance resulted in a response rate of 64% in 2017. This was the same as in the previous year and 3 percentage points lower than the 2014 survey response rate of 67%.
• The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.
• There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland.

6.1 Introduction

This section presents the fieldwork outcomes for the sampled addresses.

The final number of social survey interviews achieved was 10,683 a response rate of 64 per cent. This was the first year since 2012 design came in that the target number of interviews had been met. The impact of this drop in response rate is discussed in section 6.3. The performance of the physical survey is described in section 6.5.

The SHS response rate has fallen in recent years, having fallen by 4% since 1999. The reduction in fieldwork performance resulted in a response rate of 64% in 2017. This was the same as in the previous year and 3 percentage points lower than the 2014 survey response rate of 67%. The maintenance of the response rate in a climate of declining response rates on other population surveys is commendable.

Survey response is an important indicator of survey quality as non-response can introduce bias into survey estimates. Standardised outcome codes (based on an updated version of those published in Lynn et al (2001)^[16]) for survey fieldwork were applied across the SHS, SHeS and SCJS. The outcome codes paper includes guidance on the appropriate categorisation of interview outcomes. This will allow consistent reporting of fieldwork performance and effective comparison between the performance of the surveys.

6.2 Scotland-Level Summary

Table 6.1 shows a detailed breakdown of the SHS response for all sampled addresses for Scotland. The addresses of unknown eligibility have been allocated as eligible and ineligible proportional to the levels of eligibility for the remainder of the sample. This approach provides a conservative estimate of the response rate as it estimates a high proportion of eligible cases amongst the unknown eligibility addresses.

The table shows that the overall household response rate was 63.9 per cent. This is below the long-term (1999/2000 to 2011) average response rate for the SHS of 67.9 per cent and below the 2012 to 2014 rates. The effects of the drop in response rate are discussed in section 6.3. It should be noted that from 2012 the calculation was changed slightly and a portion of the addresses of unknown eligibility are now considered to be eligible, whereas previously they would all have been classed as ineligible.

Table 6.1: Fieldwork outcomes, Scotland

Fieldwork Outcome	Sample	% of issued	% of eligible
Responding households	10,683	59.3%	63.9%
Random adult interview complete	9,812	54.4%	58.7%
Refused
Office refusal	445	2.5%	2.7%
Refusal at introduction/before interview	3,592	19.9%	21.5%
Broken appointment - no re-contact	202	1.1%	1.2%
Total refused	4,239	23.5%	25.3%
Non-contact
No contact with anyone at the address	688	3.8%	4.1%
Contact made at address, but not with target respondent	416	2.3%	2.5%
Total non-contact	1,104	6.1%	6.6%
Other non-response
Ill at home during field period	48	0.3%	0.3%
Away or in hospital throughout field period	108	0.6%	0.6%
Physically or mentally unable/incompetent	153	0.8%	0.9%
Language barrier	53	0.3%	0.3%
Lost interview	4	0.0%	0.0%
Other non-response (not covered by categories above)	28	0.2%	0.2%
	394	2.2%	2.4%
Unknown eligibility
Inaccessible	5	0.0%
Unable to locate address	26	0.1%
Unknown if occupied, due to non-contact	289	1.6%
Other unknown eligibility	0	0.0%
Total unknown eligibility	320	1.8%
Estimated eligible addresses in set of unknown eligibility addresses	304	1.7%	1.8%
Total eligible addresses	16,724	92.8%	100.0%
Not eligible
Not yet built / under construction	8	0.0%
Demolished/derelict	51	0.3%
Vacant/empty	754	4.2%
Non-residential	222	1.2%
Address occupied but not resident household	214	1.2%
Communal establishment / institution	22	0.1%
Other ineligible	11	0.1%
Estimated ineligible addresses in set of unknown eligibility addresses	16	0.1%
Total not eligible	1,298	7.2%
All issued addresses	18,022	100.0%

6.3 Drop in response rate

The reduction in fieldwork performance resulted in a response rate of 64% for 2017. This was the same as in previous year and 3 percentage points lower than the 2014 survey response rate of 67%. Response rates are shown in the Table 6.2: Response rates over time below.

Table 6.2: Response rates over time

	2001/02	2003/04	2005/06	2007/08	2009/10	2011	2012	2013	2014	2015	2016	2017
Response rate	67%	68%	69%	66%	69%	69%	67%	67%	67%	64%	64%	64%
Achieved sample	30,639	30,822	31,013	27,238	28,404	14,358	10,644	10,652	10,633	10,325	10,470	10,683

The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.

Due to a drop in the response rate in 2015, an analysis was conducted to investigate the likely impact of this drop in response rates. This was undertaken by modeling the 2014 sample results to examine what the impact of a 3% drop in response rates would have been had a lower number of interviews been achieved, and comparing this against the 67% full sample results.

Analysis showed that the demographic composition of the sample was largely unchanged (age, gender), with only the most deprived SIMD quintile and other urban areas showing a 1 percentage point drop in their respective shares of the total sample.

Analysis of mean differences in the population estimates from the two samples for a basket of full sample questions from the SHS, and full one third sample questions from the SHCS module, were also undertaken.

This showed that the absolute mean differences for the total population estimates across the different basket of questions within the household and random adults part of the survey, including the Scotland Performs National Indicators, were very small, at around 0.10 percentage points.

A few sub-group categories had one or two 'maximum' differences in estimates of around 1 percentage points, including age and social and private-rented sector sub-groups. However, these differences are unlikely to be statistically significant due to small subgroup sample sizes.

Sub-national analysis was not considered. It is expected that there would be a greater impact of this lower response rate for Local Authorities and other sub-national geographies.

6.4 Local authority performance

Table 6.3: Response rate and eligibility by local authority shows levels of ineligible addresses, response rate and random adult conversion.

Table 6.3: Response rate and eligibility by local authority

As in previous years, the highest levels of ineligible addresses were recorded in Argyll and Bute, Dumfries and Galloway, Na h-Eileanan Siar, Highland and Orkney Islands^[17]. For both Na h-Eileanan Siar and Argyll and Bute, high levels of ineligible addresses were expected as both areas contain a high number of holiday and second homes which are not eligible for the survey. Expected levels of ineligible addresses for Dumfries and Galloway, Highland and Orkney were all above 10%.

The two lowest household response rates in 2017 were in Glasgow City and East Renfrewshire, with Glasgow City having the lowest household response rate. In addition, 5 other local authorities had a response rate under 60 per cent in 2016.

The conversion from household interview to random adult completion was 92 per cent in 2017(this was 92% in 2016, 91% in 2015, 92% in 2014 and 93% both in 2012 and 2013).

6.5 Physical survey

As described in section 2.2, over one third of the SHS sample was assigned to the physical survey module. For completion of the physical survey, respondents had to agree to make an appointment for a surveyor to make a follow-up visit and to complete the appointment. Table 6.4 shows the number of households assigned to the physical module which responded to the main SHS and the conversion to completion of the physical survey.

There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland. The result of continuing lower than estimated levels of conversion from household survey to physical survey, meant that there were fewer than 80 physical survey responses for 9 of the local authorities in 2017^[18]. However, the decision to 'open up' an additional stream to the physical survey gave a lower shortfall on the minimum target of 80 per local authority than in previous years (16 local authorities in 2016, 17 in 2015, 24 in 2014, 21 in 2013 and 20 in 2012). The 3,006 surveys achieved for Scotland in 2017 met the target, the first time since prior to 2012.

Table 6.4: Conversion to full physical survey