Health and care experience survey 2017 to 2018: technical report

4. Sample Design

Sampling Frame

People eligible to be sampled for the survey were those who were registered to a Scottish GP practice on 19 October 2017 and aged 17 or over at that date. Eligible people were identified by ISD Scotland, using an October 2017 extract from the Community Health Index (CHI) database. All data was accessed, managed and stored in accordance with the data confidentiality protocols described in the privacy notice for the survey^[3].

A small number of special practices, run by NHS Boards to provide primary care services to particular small groups of people (e.g. practices for homeless people) were excluded from the survey. Unfortunately, an error which occurred during the sampling process resulted in the Islands View Surgery in NHS Orkney (practice code: 38084) also being excluded from the 2017/18 survey. This affects not only the individual practice concerned, but also the organisational levels it is associated with: cluster; Health and Social Care Partnership; and NHS Board. This omission has been highlighted within the dashboard of local level results.

The addresses from CHI were cross checked against the Postcode Address File to ensure that they were complete. Any records where the address wasn’t recognised were removed from the survey sample frame, as were a small number of people who had requested not to be included in this or other surveys.

Sampling Design and Sample Size Calculation

Sampling was done within GP practice lists, to aim for sufficient responses to achieve a reasonably reliable result for each practice. The reliability of the result depends on the number of questionnaires returned, and also the variability of the responses.

The sample size that was calculated for each practice was based on the minimum number of responses that would be required to achieve an estimate of a percentage that has a 95 per cent confidence interval with width +/- eight percentage points, sampled from a finite population.

The formula for the minimum number of responses required (M) is

M = B / (1+(B-1) / N)

Where:

• N is the number of people registered with a practice on the sampling frame (i.e. the number of people aged 17 and over);
• B = z²p(1-p) / c² = 150 using the following definitions:
  • p is the proportion answering in a certain way, assume 0.5 to give maximum variability;
  • z is 1.96 for a 95 per cent confidence interval (using the standard normal distribution);
  • c gives maximum acceptable size of confidence interval, in this case 0.08 (eight percentage points).

Table 4 shows the minimum number of responses required (M) based on the assumptions above for some example practice population sizes.

Table 4: Minimum number of responses required for difference practice list sizes

In practice, if the underlying proportion is actually higher, or lower, than 0.5, then these numbers of responses would give narrower confidence intervals (or fewer responses would be required for the same accuracy).

The minimum number of responses required is adjusted upwards to allow for assumed non-response to the survey. Estimated response rates to the 2017/18 survey were calculated using a model that took into account different factors, including age and deprivation, which had an effect on the likelihood of a person responding to the survey. The model considered response rates from the 2015/16 Health and Care Experience Survey, but also took into account that a reminder letter would be sent out (which was not the case in the 2015/16 survey). Finally, sample sizes for practices were capped to a maximum number per practice for practices which historically have had extremely low response rates.

A total of 611,638 people were sampled for inclusion in the Health and Care Experience Survey 2017/18.

Sample Selection

For the majority of practices in Scotland, a random sample of the required number of people from each practice was taken from the sampling frame by ISD Scotland. For some practices with very small numbers of eligible people, all were included in the survey in order to meet the minimum sample size requirements identified from the calculation above. The sample was selected within SPSS software version 21.0.

Further references for this methodology are: Becker, R. A., Chambers, J. M. and Wilks, A. R. (1988) The New S Language. Wadsworth & Brooks/Cole.