Scotland's People Annual report: Results from 2009 Scottish Household Survey

Annex 4 Confidence intervals and statistical significance

The Representativeness of the Scottish Household Survey

Although the Scottish Household Survey ( SHS) sample is chosen at random, the people who take part in the survey will not necessarily be a representative cross-section of the population. Like all sample surveys the results of the SHS are estimates of the corresponding figures for the whole population and these results might vary from the true values in the population for three main reasons:

1. The sample source does not completely cover the population because accommodation in hospitals, prisons, military bases, larger student halls etc. are excluded from the sampling frame. The SHS provides a sample of private households rather than all households. The effect of this on the representativeness of the data is not known.

2. Some people refuse to take part in the survey and some cannot be contacted by interviewers. If these people are systematically different from the people who are interviewed, this represents a potential source of bias in the data. Comparison of the SHS data with other sources suggests that for the survey as a whole, any bias due to non-response is not significant. ⁸⁹

3. Samples always have some natural variability because of the random selection of households and people within households. In some areas where the sample is clustered, the selection of sampling points adds to this variability.

Each of these sources of variability becomes much more important when small sub-samples of the population are examined. For example, a sub-sample with only 100 households might have had very different results if the sampling had, by chance, selected four or five more households with children, or households including one or two adults of pensionable age and no younger adults.

Confidence Intervals

The likely extent of sampling variability can be quantified by calculating the 'standard error' associated with an estimate produced from a random sample. Statistical sampling theory states that, on average:

• only about one sample in three (33%) would produce an estimate that differed from the (unknown) true value by more than one standard error;
• only about one sample in twenty (5%) would produce an estimate that differed from the true value by more than two standard errors;
• only about one sample in 400 (0.25%) would produce an estimate that differed from the true value by more than three standard errors.

By convention, the '95% confidence interval' is defined as the estimate plus or minus about twice the standard error because there is only a 5% chance (on average) that a sample would produce an estimate that differs from the true value of that quantity by more than this amount.

There is no simple 'rule of thumb' for the size of standard errors: the standard error of the estimate of a percentage depends upon several things:

• the value of the percentage itself;
• the size of the sample (or sub-sample) from which it was calculated ( i.e. the number of sample cases corresponding to 100%);
• the sampling fraction ( i.e. the fraction of the relevant population that is included in the sample);
• the 'design effect' associated with the way in which the sample was selected (for example, a clustered random sample would be expected to have larger standard errors than a simple random sample of the same size).

Table A 4 at the end of this Annex shows the 95% confidence limits for a range of estimates calculated for a range of sample sizes, incorporating a design factor of 1.3 ⁹⁰ to account for the fact that some of the sample is clustered rather than a simple random sample. To estimate the potential variability for an estimate for the survey you should read along the row with the value closest to the estimate until you reach the column for the value closest to the sub-sample. This gives a value which, when added and subtracted from the estimate, gives the range (the 95% confidence interval) within which the true value is likely to lie. Where the exact value is not given in the table, we recommend using the closest value in the table.

For example, if the survey estimates that 16% (rounded to the nearest whole number) are Single Adult households and this has a confidence interval of ±0.8%, it means that, if the estimate were 16.0% we could be 95% confident that the true value for the population lies between 15.2% and 16.8%.

Smaller sample sizes have wider confidence intervals so looking at household type might show that in, say, Edinburgh, 24% of households are Single Adult households. However, if fewer than 2,500 households in Edinburgh are interviewed this estimate has a 95% confidence interval of approximately ±2.5%. Assuming that the estimate is 24.0%, this suggests that the true value lies between 21.5% and 26.5%. Clearly, the estimate for any single area is less reliable that the estimate for Scotland as a whole.

Statistical significance

Because the survey's estimates may be affected by sampling errors, apparent differences of a few percentage points between sub-samples may not reflect real differences in the population. It might be that the true values in the population are similar but the random selection of households for the survey has, by chance, produced a sample which gives a high estimate for one sub-sample and a low estimate for the other.

A difference between two areas is significant if it is so large that a difference of that size (or greater) is unlikely to have occurred purely by chance. Conventionally, significance is tested at the 5% level, which means that a difference is considered significant if it would only have occurred once in 20 different samples. Testing significance involves comparing the difference between the two samples with the 95% confidence limits for each of the two estimates.

For example, suppose the survey estimates that there are 14% Single Adult households in Stirling (±4.1%), 10% in Aberdeenshire (±1.7%), 15% in Fife (±2.0%), and 24% in Edinburgh (±2.5%). Assuming that the estimates' values are 'exact' ( i.e. that the figure underlying 10% is 10.0%), we can say the following:

• the difference between Stirling and Fife is not significant because the difference between the two (1%) is smaller than either of the confidence limits (at least ±2.0%). In general, if the difference is smaller than the larger of the two limits, it could have occurred by chance and is not significant;
• the difference between Stirling and Edinburgh is significant because the difference (10%) is greater than the sum of the limits (4.1 + 2.5 = 6.6%). In general, a difference that is greater than the sum of the limits is significant.

If the difference is greater than the larger of the two confidence limits but less than the sum of the two limits, the difference might be significant, although the test is more complex.

Statistical sampling theory suggests that the difference is significant if it is greater than the square root of the sum of the squares of the limits for the two estimates.

The difference of 5% between Aberdeenshire and Fife is greater than the largest confidence limit (±4.1%) but it is less than the sum of the two limits (4.1% + 2.0% = 6.1%) so it might be significant. In this case 4.1 ² = 16.81 and 2.0 ² = 4 giving a total of 20.81. The square root of this is 4.56, which means that the difference of 5% is significant (although only just). Similar calculations will indicate whether or not other pairs of estimates differ significantly.

It should be noted that the estimates published in this report have been rounded, generally to the nearest whole number, and this can affect the apparent significance of some of the results. For example:

• if the estimate for Aberdeenshire was 10.49% (rounded to 10%) and the estimate for the Fife was 14.51% (rounded to 15%) the difference would be calculated as 4.02% rather than 5%. This is below the calculated 'significance threshold' value of 4.56%;
• if, however, the estimate for the Lothians was 10.51% (rounded to 11%) and the estimate for Fife was 15.49% (rounded to 15%) the difference would be calculated as 4.98% rather than 5%. This is higher than 4.56%.

For this reason, caution should be exercised where differences are on the margins of significance. In general, we would suggest that differences should only be considered significant where the difference is clearly beyond the threshold of significance.

Statistical significance and representativeness

Calculations of confidence limits and statistical significance only take account of sampling variability. The survey's results could also be affected by non-contact/non-response bias. If the characteristics of the people who should have been in the survey but who could not be contacted, or who refused to take part, differ markedly from those of the people who were interviewed, there might be bias in the estimates. If that is the case, the SHS's results will not be representative of the whole population.

Without knowing the true values (for the population as a whole) of some quantities, we cannot be sure about the extent of any such biases in the SHS. However, comparison of SHS results with information from other sources suggests that they are broadly representative of the overall Scottish population, and therefore that any non-contact or non-response biases are not large overall. However, such biases could, of course, be more significant for some sub-groups of the population or in certain Council areas, particularly those that have the highest non-response rates.

In addition, because it is a survey of private households, the SHS does not cover some sections of the population - for example, it does not collect information about students in halls of residence. Please refer to the companion technical reports ⁹¹ for a comparison of SHS results with information from other sources.

Table A 4: Estimated sampling error associated with different proportions for different sample sizes

	100	500	1,000	2,000	3,000	4,000	5,000	6,000	7,000	8,000	9,000	10,000	11,000	12,000	13,000	14,000	15,000
5%	5.6%	2.5%	1.8%	1.2%	1.0%	0.9%	0.8%	0.7%	0.7%	0.6%	0.6%	0.6%	0.5%	0.5%	0.5%	0.5%	0.5%
10%	7.6%	3.4%	2.4%	1.7%	1.4%	1.2%	1.1%	1.0%	0.9%	0.9%	0.8%	0.8%	0.7%	0.7%	0.7%	0.6%	0.6%
15%	9.1%	4.1%	2.9%	2.0%	1.7%	1.4%	1.3%	1.2%	1.1%	1.0%	1.0%	0.9%	0.9%	0.8%	0.8%	0.8%	0.7%
20%	10.2%	4.6%	3.2%	2.3%	1.9%	1.6%	1.4%	1.3%	1.2%	1.1%	1.1%	1.0%	1.0%	0.9%	0.9%	0.9%	0.8%
25%	11.0%	4.9%	3.5%	2.5%	2.0%	1.7%	1.6%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%	1.0%	0.9%	0.9%
30%	11.7%	5.2%	3.7%	2.6%	2.1%	1.8%	1.7%	1.5%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%	1.0%	1.0%
35%	12.2%	5.4%	3.8%	2.7%	2.2%	1.9%	1.7%	1.6%	1.5%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%	1.0%
40%	12.5%	5.6%	3.9%	2.8%	2.3%	2.0%	1.8%	1.6%	1.5%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.1%	1.0%
45%	12.7%	5.7%	4.0%	2.8%	2.3%	2.0%	1.8%	1.6%	1.5%	1.4%	1.3%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%
50%	12.7%	5.7%	4.0%	2.8%	2.3%	2.0%	1.8%	1.6%	1.5%	1.4%	1.3%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%
55%	12.7%	5.7%	4.0%	2.8%	2.3%	2.0%	1.8%	1.6%	1.5%	1.4%	1.3%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%
60%	12.5%	5.6%	3.9%	2.8%	2.3%	2.0%	1.8%	1.6%	1.5%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.1%	1.0%
65%	12.2%	5.4%	3.8%	2.7%	2.2%	1.9%	1.7%	1.6%	1.5%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%	1.0%
70%	11.7%	5.2%	3.7%	2.6%	2.1%	1.8%	1.7%	1.5%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%	1.0%	1.0%
75%	11.0%	4.9%	3.5%	2.5%	2.0%	1.7%	1.6%	1.4%	1.3%	1.2%	1.2%	1.1%	1.1%	1.0%	1.0%	0.9%	0.9%
80%	10.2%	4.6%	3.2%	2.3%	1.9%	1.6%	1.4%	1.3%	1.2%	1.1%	1.1%	1.0%	1.0%	0.9%	0.9%	0.9%	0.8%
85%	9.1%	4.1%	2.9%	2.0%	1.7%	1.4%	1.3%	1.2%	1.1%	1.0%	1.0%	0.9%	0.9%	0.8%	0.8%	0.8%	0.7%
90%	7.6%	3.4%	2.4%	1.7%	1.4%	1.2%	1.1%	1.0%	0.9%	0.9%	0.8%	0.8%	0.7%	0.7%	0.7%	0.6%	0.6%
95%	5.6%	2.5%	1.8%	1.2%	1.0%	0.9%	0.8%	0.7%	0.7%	0.6%	0.6%	0.6%	0.5%	0.5%	0.5%	0.5%	0.5%