Scottish Cancer Patient Experience Survey 2024: technical report

Annex E – Confidence Intervals and Statistical Significance

Confidence intervals provide a way of quantifying sampling uncertainty. A 95% confidence interval means that, over many repeats of a survey under the same conditions, one would expect that the confidence interval would contain the true population value 95 times out of 100.

For example, given a results of 80% and a confidence interval of +/-3%, there is 95% confidence that the true result is between 77% and 83%.

Standard errors and confidence intervals have been calculated using the statistical software R and the R survey package.

Statistical Significance

A result can be described as statistically significant if it is unlikely to have occurred by random variation. Testing for statistical significance allows us to assess whether there have been significant changes in performance between two time periods or areas (e.g. between a specific Health Board and Scotland as a whole).

For each question where a comparison is possible – for example, between two different survey periods or between two areas – a test statistic is calculated to test the statistical significance of the difference between the two results.

The test statistic is calculated as:

where p₁ = the first point estimate or proportion

p₂ = the comparable second point estimate or proportion

where n₁ = the number of valid responses for point estimate 1

n₂ = the number of valid responses for point estimate 2

deff₁ = the design effect for point estimate 1

deff₂ = the design effect for point estimate 2

For point estimate 1 not to be statistically significantly different from point estimate 2 with 95% confidence, then the test statistic must be between ±1.96.