Scottish Household Survey: response rates, reissuing and survey quality

Executive Summary

Reissuing non-responses is a widespread practise in surveys, to maintain high response rates and reduce the risk of non-response bias. However, reissuing is costly. This paper assesses the impact of reissuing on survey estimates using data from two sweeps of the Scottish Household Survey (SHS), 2014 and 2016.

Method

The analysis explores how the published results of the survey would differ if reissuing had not been used. As reissues have increased the response rate by around 10-11 percentage points in the SHS, the analysis examines the potential impact on estimates of reducing the response rate target by this amount.

The impact of reducing response rates on a range of key measures is estimated by comparing the published estimates from the weighted full survey sample (after reissuing) with estimates from first issue interviews only, weighted as if they were the final achieved sample. The analysis examines the impact both at the national level and among key sub-groups. The scale of the impact on each estimate is reported in terms of the impact in absolute terms and also standardised to control for the effect of different base sizes and prevalence levels.

Differences between first issue sample and those who respond at reissue (before weighting)

Reissues are more common in urban areas than rural areas and in the most deprived 15% of areas than the rest of Scotland. Compared to the first issue sample, the reissue sample was younger and contained more men. Single adult households were also more prevalent in the reissue sample than the first issue sample.

The potential impact of these differences on the final weighted estimates is reduced by the fact that a number of these variables are included in the weighting strategy.

Twelve measures, covering a range of areas were analysed across both waves. The impact of reissuing on eleven key estimates at sub-group-level was also examined, looking specifically at gender, age, rurality, deprivation, tenure, area and household type. Across two sweeps this gave 704 estimates.

Findings

1. Overall, the impact of reissuing to increase the response rate on national estimates was small.

Most estimates saw a change that was less than one standard error of the published estimates. The average impact was equivalent to 0.72 standard errors and the maximum found was equivalent to 2.1 standard errors.

2. The absolute impact of reissuing on sub-group estimates was larger than for national estimates. However, this was because these estimates themselves are less precise as they are based on smaller sample sizes.

In terms of the standardised difference, the average change was equivalent to half of the standard error associated with the main estimates. For most estimates, the impact was less than one standard error. For only 20 out of 704 sub-group estimates the impact was greater than 1.5 standard error.

There did appear to be some particular sub-groups - particularly single adult households and households in the Central Region - that were more affected by reissuing than others. However, this was driven, at least in part, by these groups containing a higher proportion of reissue interviews than other sub-groups.

Conclusions

General reissuing is not having have a meaningful impact on the survey estimates. This is in line with previous research in finding that increasing response rates through reissuing has only a small impact on the survey estimates.