Patient Experience survey of GP and local nhs services 2011/12 Volume 1: Technical Report

4 Sample Design

Sampling frame

4.1 Patients eligible to be sampled for the survey were those who were registered to a Scottish GP practice at 17 October 2011 and aged 17 or over at that date. Eligible patients were identified by ISD Scotland, using an October 2011 extract from the Community Health Index (CHI) database.

4.2 A small number of special practices, run by NHS Boards to provide primary care services to particular small groups of patients (e.g. practices for homeless people) were excluded from the survey. A small number of patients who had requested not to be included in this or other surveys were also removed from the survey sampling frame.

Sample size calculation

4.3 Sampling was done within GP practice lists, to aim for sufficient responses to achieve a reasonably reliable result for each practice. The reliability of the result depends on the number of questionnaires returned, and also the variability of the responses.

4.4 The sample size that was calculated for each practice was based on the minimum number of responses that would be required to achieve an estimate of a percentage that has a 95% confidence interval with width +/- 8 percentage points, sampled from a finite population. This level of acceptable uncertainty was agreed with stakeholders. For the 2009/10 survey the sample sizes were based on a 95% confidence interval with width +/-7 percentage points. The small reduction in the required accuracy of the results reduced respondent burden and costs. Unlike the 2009/10 survey, the results will not be used to allocate any funding to GP practices so they do not have to be quite as accurate.

4.5 The formula for the minimum sample required (M) is

M = B/(1+(B-1)/N).

Where:

• N is the number of patients in a practice on the sampling frame (i.e. the number of patients aged 17 and over)
• B = z²p(1-p)/c²
• p = the proportion answering in a certain way (assume 0.5 as this gives maximum variability)
• z is 1.96 for a 95% confidence interval (from standard normal distribution)
• c gives maximum acceptable size of confidence interval, in this case 0.08 (8 percentage points).

Based on the above, B=150

4.6 Table 3 below shows examples of this minimum number of responses required (M) based on the assumptions above for some example practice population sizes.

Table 3 Minimum sample size required for different practice list sizes

4.7 In practice, if the underlying proportion is actually higher, or lower, than 0.5, then these numbers of responses would give narrower confidence intervals (or fewer responses would be required for the same accuracy).

4.8 The minimum sample size required (Table 3) is adjusted upwards to allow for assumed non-response to the survey. Estimated response rates to the 2011/12 survey were calculated using a model that took into account different factors, including age and deprivation, which had an effect on the likelihood of a person responding to the survey. The model considered response rates from the 2009/10 GP Patient Experience Survey and 2010/11 GP Access Survey. This provided the most efficient way of determining the sample size.

4.9 The model also showed that response rates were lower the closer to Christmas that the survey was sent to people. Possible explanations for this may be surveys getting caught up in the Christmas post or people not being able to return the survey due to bad weather. This finding influenced the timetable for the 2011/12 survey. We sent surveys earlier than previously (all surveys were sent by 17 November 2011, whereas in 2009 the last mailing date was 4 December ) to ensure that the response rate was not affected by sending surveys too close to Christmas. The estimated response rate was 23 per cent and the achieved response rate was 24 per cent.

4.10 For the previous survey two reminders were sent to boost the response rate, but this year's survey was designed to achieve the required number of responses for each practice without reminders. It proved to be cost effective to send more surveys initially than to send fewer surveys and reminders.

4.11 The formula used to calculate the total sample size (S) for each practice can be simplified to

Where R is the assumed response rate for each practice.

4.12 A total of 605,896 patients were sampled for inclusion in the Patient Experience Survey of GP and Local NHS Services 2011/12.

Patient selection

4.13 For the majority of practices in Scotland, a random sample of the required number of patients (S) from each practice was taken from the sampling frame by ISD Scotland, using the "sample" function within the R software package. For some practices with very small numbers of eligible patients, all patients were included in the survey in order to meet the minimum sample size requirements identified from the calculation above.

4.14 The R software is free software and we acknowledge the following source: R Development Core Team (2007). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org.

4.15 Further references for this methodology are: Becker, R. A., Chambers, J. M. and Wilks, A. R. (1988) The New S Language. Wadsworth & Brooks/Cole.