Scottish Household Survey: response rates, reissuing and survey quality

4 Results

4.1 We briefly summarise the difference in the profile of people who respond at first issue to those who respond at reissue at the start of this section before examining the impact of excluding reissues on survey estimates.

How do those that respond at first issue differ from those who respond at reissue?

4.2 Reissues accounted for 16% of random adult interviews in 2014 and 18% in 2016.

Table 4.1: Profile of first issue random adult respondents compared with reissue respondents. SHS 2014 and 2016 unweighted

	SHS
	2014				2016
	First issue	Reissue	Final	N	First issue	Reissue	Final	N
Male	45%	45%	45%	4,442	45%	48%	46%	4,401
Female	55%	55%	55%	5,356	55%	52%	54%	5,241
Total	100%	100%	100%	9,798	100%	100%	100%	9,642
16 - 24	7%	11%	8%	787	8%	8%	8%	727
25 - 44	28%	33%	28%	2,787	27%	36%	29%	2,752
45 - 59	26%	26%	26%	2,532	25%	25%	25%	2,379
60+	39%	29%	38%	3,692	41%	32%	39%	3,784
Total	100%	100%	100%	9,798	100%	100%	100%	9,642
Urban	78%	88%	79%	7,752	77%	85%	78%	7,528
Rural	22%	12%	21%	2,046	23%	15%	22%	2,114
Total	100%	100%	100%	9,798	100%	100%	100%	9,642
15% most	14%	20%	15%	1,476	13%	17%	14%	1,349
Rest	86%	80%	85%	8,322	87%	83%	86%	8,293
Total	100%	100%	100%	9,798	100%	100%	100%	9,642
Single adult	18%	26%	19%	1,850	17%	26%	19%	1,807
Small adult	15%	16%	15%	1,494	15%	15%	15%	1,446
Single parent	5%	8%	5%	526	6%	8%	6%	570
Small family	12%	11%	12%	1,171	12%	11%	12%	1,153
Large family	6%	4%	6%	549	5%	4%	5%	433
Large adult	8%	7%	8%	792	8%	6%	7%	707
Older small	18%	12%	17%	1,659	19%	14%	18%	1,725
Single pensioner	18%	15%	18%	1,757	19%	16%	19%	1,801
Total	100%	100%	100%	9,798	100%	100%	100%	9,642
N	8,273	1,525	9,798		7,923	1,719	9,642

4.3 The profile of people who respond at reissue is broadly in line with the profile of those who respond at first issue in both years and the differences are not stark. Table 4.1 show the unweighted distributions across key sub-groups by when interviewed.

4.4 In both waves, men and those who were in the younger age bands comprised a higher proportion of reissue interviews than first issue interviews. This echoes findings from previous research reported in Chapter 2.

4.5 In relation to rurality and deprivation, reissue interviews were more likely to occur in urban areas, and in the 15% most deprived areas than first issue interviews were.

4.6 With regard to household type, Single Adult and Single Parent households comprised a higher proportion of reissue interviews than first issues, while the opposite was the case for Older Smaller households and Single Pensioner households.

What is the impact of reissuing on national estimates?

4.7 Table 4.2 shows the impact of reissuing on twelve estimates at the Scotland-wide level for the 2016 wave of the SHS.

4.8 The difference between the final sample estimates and the issue 1 survey estimate was small in absolute terms.

4.9 The average absolute difference between the final sample and the first issue estimates was 0.33 percentage points, and the maximum difference was 1.13 percentage points.

4.10 Table 4.2 also shows the standardised differences, the ratio of the absolute difference between estimates to the standard error of the main estimate. Overall, the average value of this ratio was 0.58 across all 12 estimates. In other words, the average impact of not including reissues equates to just over half of one standard error of the published estimates. The maximum value of this ratio among the twelve measures was 2.07, for the measure of providing unpaid help to organisations or groups within last 12 months.

Table 4.2: Impact of reissues on twelve key national estimates. SHS 2016

	Final estimate (64% RR)	N[6]	SE	CIs (+/-)	Issue 1 estimate (54% RR)	Difference	Diff/SE
Percentage satisfied (very or fairly) with local public services	56.1%	9,594	0.6%	1.2%	55.9%	0.23%	0.38
Percentage agreeing (strongly or slightly) they 'can influence decisions affecting my local area'	23.1%	9,642	0.5%	1.0%	23.3%	0.15%	0.30
Percentage using the internet for personal use	83.4%	4,707	0.7%	1.3%	84.1%	0.74%	1.13
Percentage rating neighbourhood as a very good place to live	56.7%	9,642	0.6%	1.2%	56.6%	0.05%	0.08
Percentage participating in a cultural activity or attended a cultural place or event in the last 12 months	92.0%	5,008	0.5%	0.9%	92.0%	0.01%	0.03
Percentage that make 1+ visits to the outdoors per week	48.5%	9,642	0.6%	1.2%	48.1%	0.35%	0.57
Percentage living within 5 min walk of greenspace	65.4%	9,642	0.6%	1.1%	66.4%	1.07%	1.84
Percentage providing unpaid help to organisations or groups within last 12 months	27.3%	9,642	0.5%	1.1%	28.4%	1.13%	2.07
Percentage participating in physical activity or sport in last four weeks	78.8%	9,642	0.5%	1.0%	78.8%	0.01%	0.02
Percentage rating general health as bad or very bad	7.7%	9,642	0.3%	0.6%	7.8%	0.07%	0.21
Percentage experiencing either discrimination or harassment	9.8%	9,642	0.4%	0.7%	9.9%	0.07%	0.20
Percentage households not managing well financially	8.2%	10,470	0.3%	0.6%	8.1%	0.04%	0.13
Average across the twelve measures.						0.33%	0.58

4.11 The results of the same analysis conducted on the 2012-2013 wave of the data presents a similar picture (Table 4.3). In terms of the absolute differences, the average among the 12 measures was 0.44 percentage points and the maximum difference was 0.93 percentage points.

4.12 With regard to the standardised measure, the average value of the ratio of the difference to the standard error of the main estimate was 0.86 across the 12 estimates, with a maximum of 1.66 for the measure, 'Percentage living within 5 min walk of greenspace'. Again, this means that the impact of not including reissues in the survey (and reducing the effective response rate by around 10%) would equate to less than one standard error for most measures.

Table 4.3: Impact of reissues on twelve key national estimates. SHS 2014

	Final estimate (67% RR)	N[7]	SE	CIs (+/-)	Issue 1 estimate (56% RR)	Difference	Diff/SE
Percentage satisfied (very or fairly) with local public services	61.9%	9,746	0.6%	1.2%	61.2%	0.74%	1.26
Percentage agreeing (strongly or slightly) they 'can influence decisions affecting my local area'	23.0%	9,798	0.5%	1.0%	23.1%	0.16%	0.31
Percentage using the internet for personal use	82.0%	4,787	0.7%	1.3%	82.5%	0.55%	0.83
Percentage rating neighbourhood as a very good place to live	55.8%	9,798	0.6%	1.2%	55.5%	0.38%	0.63
Percentage participating in cultural activity/ attending a cultural place/event in the last 12 months	90.8%	5,140	0.5%	0.9%	91.0%	0.15%	0.32
Percentage that make 1+ visits to the outdoors per week	48.4%	9,798	0.6%	1.2%	47.9%	0.48%	0.79
Percentage living within 5 min walk of greenspace	68.6%	9,798	0.6%	1.1%	69.5%	0.93%	1.66
Percentage providing unpaid help to organisations or groups within last 12 months	27.0%	9,798	0.5%	1.1%	27.5%	0.52%	0.96
Percentage participating in physical activity or sport in last four weeks	77.8%	9,798	0.5%	1.0%	78.2%	0.44%	0.88
Percentage rating general health as bad or very bad	6.6%	9,798	0.3%	0.6%	6.4%	0.15%	0.51
Percentage experiencing either discrimination or harassment	8.9%	9,798	0.3%	0.7%	9.2%	0.31%	0.91
Percentage households not managing well financially	11.2%	10,632	0.4%	0.7%	10.7%	0.47%	1.29
Average across the twelve measures.						0.44%	0.86

4.13 The scale of the impact of not including reissues, our proxy for examining the effect of setting a lower response rate target, is easier to visualise as estimates plotted with confidence intervals. Figures 4.1 to 4.4 show estimates of four of the measures plotted with confidence intervals.^[8]

• Satisfaction with local public services
• Providing unpaid help to organisations or groups within last 12 months
• Percentage participating in physical activity or sport in last four weeks
• Percentage households not managing well financially.

4.14 Table 4.4 presents the summary of the absolute impact of reissuing on the 12 national estimates. The average impact was 0.38 percentage points. This was similar between 2014 (0.44 percentage points) and 2016 (0.33 percentage points). Most of the estimates, 21 of 24, changed by less than 1 percentage point.

Table 4.4: Summary of absolute impact on the national estimates. SHS 2014 and SHS 2016

	Mean	Max	Count	Final minus Issue 1 estimate 0-1% points	Final minus Issue 1 estimate >1% points	Final minus Issue 1 estimate >3% points
2014	0.44%	0.93%	12	12	0	0
2016	0.33%	1.13%	12	9	3	0
Combined	0.38%	1.13%	24	22	3	0

Sample sizes: See Table 3.1.

4.15 Table 4.5 shows a summary of the impact on these estimates after standardisation. Overall, the average standardised impact on estimates was 0.72. This means that the impact of reducing the response rate by around 10% to 11% is of a similar magnitude to around three-quarters of the standard error associated with the estimates.

Table 4.5: Summary of average standardised impact of reissuing on national estimates. SHS 2014 and SHS 2016

	Mean	Max	Count	Diff/SE 0 to 0.5	Diff/SE >0.5	Diff/SE >1	Diff/SE >1.5
2014	0.86	1.66	12	2	10	3	1
2016	0.58	2.07	12	8	4	3	2
Combined	0.72	2.07	24	10	14	6	3

Sample sizes: See Table 3.1.

Analysis of impact of reissuing on estimates among key subgroups.

4.16 While the impact of reissuing on estimates at the national level was small, the impact on estimates among sub-groups could potentially be more considerable. Estimates for the 11 measures from the random adult section of the questionnaire were analysed. The impact on these estimates was calculated on a number of key sub-groups. These were gender, age, rurality, deprivation, tenure, area and household typology.

4.17 Overall, this meant that the impact was calculated for 704 estimates, 352 in 2014 and 352 estimates in 2016.^[10] Table 4.6 summarises the impact on the absolute difference of estimates among key subgroups.

Table 4.6: Summary of absolute impact on estimates among key subgroups. SHS 2014 and 2016

	Mean	Max	Count	Final minus Issue 1 estimate 0-1%	Final minus Issue 1 estimate >1%	Final minus Issue 1 estimate >3%	Final minus Issue 1 estimate >5%
2014	0.70%	5.5%	352	277	75	4	1
2016	0.82%	12.7%	352	249	103	8	1
Combined	0.76%	12.7%	704	526	178	12	2

4.18 Overall, reissuing had a larger impact at the sub-group level than the national level. This is primarily driven by the sample sizes. However most of the differences were still relatively modest. Overall, the average impact on estimates was 0.76%. The average impact was similar in 2014 (0.70%) and 2016 (0.82%).

4.19 Most estimates, 526 of 704, changed by less than 1%. Overall only 12 of the 704 estimates changed by more than 3%, and two changed by more than 5%. Estimates for Central Region^[11] accounted for 6 of the 12 estimates that changed by more than 3 percentage points. Similarly, 6 of the 12 estimates that changed by more than 3 percentage points related to the indicator for making one or more visits to the outdoors per week.

4.20 By way of illustration, Figure 4.5 shows the estimates for making one or more visits to the outdoors per week for Central Region. In 2016 this estimate changed by 4% for Central Region, from 38.4% to 42.4%. The overall sample size for the main estimate was based on 754 cases. This means that, as shown in Figure 4.5, the confidence intervals around this estimate equalled +/- 4.2%, and therefore that the standardised change equated to 1.88 of the standard error of the main estimate.

4.21 Finally, Table 4.5 shows a summary of the impact on these 704 estimates after standardisation.

Table 4.5: Summary of average standardised impact of reissuing on estimates among key subgroups. SHS 2014 and 2016

	Mean	Max	Count	Diff/SE 0 to 0.5	Diff/SE >0.5	Diff/SE >1	Diff/SE >1.5
2014	0.48	2.76	352	218	134	34	3
2016	0.53	6.21	352	215	137	49	17
Combined	0.51	6.21	704	433	271	83	20

4.22 Overall, the average standardised impact on estimates was 0.51. This means that the impact of reducing the response rate by around 10% is of a similar magnitude to one half of the standard error associated with the estimates. There is little difference with regard to the size of the impact by wave.

4.23 The impact was less than 0.5 for the majority of estimates (433 of 704).

4.24 The impact was greater than 1.5 for less than 3% of estimates (20 out of 704 estimates. The maximum value was 6.21 for all sub-group estimates included in this analysis and the second largest value was 2.76.

4.25 There was some evidence to suggest that the impact of reissuing on some sub-groups was greater than on others. However, this appears to be driven, at least in part, by the proportion of reissue interviews within particular subgroups. The maximum of the average standardised impact across the different measures but within sub-groups was 0.94 in the 2014 wave and 1.82 for 2016. These both related to estimates for the Central Region of Scotland. Note that 21% of interviews in Central Region were reissue interviews in 2014 while 27% were reissues in 2016^[12] compared to the overall average of 16% in 2014 and 18% in 2016.

4.26 The second largest average standardised impact across both years was for single adult households (0.74 in 2014 and 1.03 in 2016). Reissue interviews also accounted for a higher proportion of interviews among this sub-group than most other sub-groups (22% in 2014 and 25% in 2016).

4.27 As noted previously, the scale of the impact can be difficult to visualise. Figures 4.6 to 4.10 show estimates for five sets of sub-group estimates plotted with confidence intervals

• Satisfaction with public services among those in the most deprived 15% of areas
• Rate neighbourhood as a good place to live in Fife
• Rating general health as bad or very bad among those aged 60 and over
• Participated in a physical activity or a sport among those aged 16-24
• Providing unpaid help to organisations or groups within last 12 months among men