Scottish Social Attitudes 2015: Technical Report

6. Effect of 2015 sample size

6.1 Table 3 below shows the achieved sample size for the full SSA sample (all respondents) for all years the survey has taken place.

Table 3: Scottish Social Attitudes survey sample size by year

Survey year	Achieved sample size
1999	1482
2000	1663
2001	1605
2002	1665
2003	1508
2004	1637
2005	1549
2006	1594
2007	1508
2009	1482
2010	1495
2011	1197
2012	1229
2013	1497
2014	1501
2015	1288

6.2 The achieved sample size for SSA 2015 was smaller than in most previous years. As such, we have considered whether this reduction in the achieved sample size in 2015 (as a result of lower response rates) had an effect on:

• representativeness of the sample,
• accuracy of the estimates,
• ability to detect differences in estimates from different sub-groups,
• ability to detect differences in estimates across years (trend analysis).

6.3 In order to assess whether the sample's representativeness was affected, we compared the age and sex distribution of the population (ONS 2014 mid-year population estimates) with the achieved sample in 2015 and 2014 when the response rate was higher (achieved sample size=1501). The survey estimates have been weighted by pre-calibration weights so that we maximize the chance of observing a pure effect of potential differential non-response of males and females in different age groups.^[3] Unfortunately age and sex are the only characteristics available from ONS for use to assess the representativeness of the sample (therefore they are used for the final step of weighting - calibration).

6.4 The last two columns of Table 4 (below) present the difference in the distribution of 2015 and 2014 sample profile to the population estimates. The mean of the absolute values of the differences, used here to measure the relative difference of the sample to the population distribution, has increased only by 0.1 percentage point as compared with 2014. Therefore, we can claim that the reduction in the achieved sample size did not lead to a loss in representativeness of the sample with regards to age and sex.

Table 4: SSA 2014 and 2015 sample distribution compared with population totals

Age/sex	2014 MID-YEAR POPULATION ESTIMATES	Survey distribution*		Difference to population totals
		2015	2014	2015	2014
Male 18-24	5.8	5.0	4.3	-0.8	-1.5
Male 25-34	8.0	5.1	7.1	-2.9	-0.9
Male 35-44	7.6	7.2	6.7	-0.4	-0.9
Male 45-54	9.0	9.5	9.7	0.5	0.7
Male 55-64	7.6	9.3	7.2	1.7	-0.4
Male 65+	9.9	10.8	10.7	0.9	0.8
Female 18-24	5.8	5.0	5.8	-0.7	0.1
Female 25-34	8.2	7.9	7.1	-0.3	-1.1
Female 35-44	8.0	8.8	10.3	0.7	2.3
Female 45-54	9.5	10.4	9.6	0.8	0.1
Female 55-64	8.0	9.0	9.6	1.0	1.7
Female 65+	12.6	12.0	11.6	-0.6	-0.9
Mean absolute difference to population totals				1.0	0.9
Max value of a difference to population totals				1.7	2.3
Min value of a difference to population totals				-2.9	-1.5

*Survey estimates have been weighted by pre-calibration weight - a step before adjusting them to the distribution of mid-year population estimates.

6.5 Table 5 (below) presents the distribution of age and sex in the unweighted data. With respect to this characteristic, the mean absolute difference to population distribution is exactly the same for 2014 and 2015 raw data (1.8 percentage points).

Table 5: SSA 2014 and 2015 unweighted sample distribution compared to population totals

Age/sex	2014 MID-YEAR POPULATION ESTIMATES	Survey distribution (unweighted)		Difference to population totals
		2015	2014	2015	2014
Male 18-24	5.8	3.1	3.0	-2.7	-2.8
Male 25-34	8.0	4.8	5.7	-3.2	-2.2
Male 35-44	7.6	6.2	6.5	-1.4	-1.1
Male 45-54	9.0	8.8	8.7	-0.3	-0.4
Male 55-64	7.6	9.4	7.4	1.8	-0.2
Male 65+	9.9	12.9	12.3	3.0	2.4
Female 18-24	5.8	3.1	3.7	-2.6	-2.1
Female 25-34	8.2	7.8	7.2	-0.5	-1.0
Female 35-44	8.0	8.3	10.2	0.3	2.2
Female 45-54	9.5	10.0	8.6	0.5	-0.9
Female 55-64	8.0	9.8	10.2	1.8	2.3
Female 65+	12.6	15.7	16.4	3.2	3.9
Mean absolute difference to population totals				1.8	1.8
Max value of a difference to population totals				3.2	3.9
Min value of a difference to population totals				-3.2	-2.8

6.6 It is difficult to assess a general impact of a reduced sample size on the accuracy of estimates. The design effect (deff)^[4] used to calculate the effective sample size for analysis differs across estimates and depends on various aspects of complex survey design, weighting adjustments, so indirectly also on the sample size. We know that weights were a little bit more variable in 2015 than in 2014 (the design effect due to weighting in 2015 was 1.51 compared with 1.45 in 2014), which may be indirectly due to a smaller sample size. However, in order to assess a general design effect it is recommended to average deffs across several estimates. This was done for political party identification, strength of party identification, level of interest in politics and constitutional preference (altogether 19 deffs). The effect of the sample design and weighting together for these estimates is comparable to the one observed in 2014. In effect, in both 2014 and 2015 the effective sample size for the analysis constitute around 57% of the achieved sample size. This means that the loss in accuracy of survey estimates, presented in Table 6 (below) for estimates of prevalence rates of 50%, 30/70% and 10/90%, is mainly due to the reduced sample size. The magnitude of the loss is very small - 0.2 percentage points for the estimates on the total sample.

Table 6: Accuracy of estimates in 2014 and 2015

					50%	30/70%	10/90%
Total sample	Achieved sample size	average DEFF*	Effective sample size	Effectiveness of the design	Margin of rror	Margin of Error	Margin of Error
					(+/-)	(+/-)	(+/-)
2014	1501	1.77	850	56.6%	3.4%	3.1%	2.0%
2015	1288	1.75	735	57.0%	3.6%	3.3%	2.2%

*The value presented in the table is an average of design effects estimated using complex sample design for the same questions and their categories in 2014 and 2015: Party identification (party3), Strength of party identification (idstrng2), Level of interest in politics (politic2), constitutional perference (scotpar2).

6.7 Table 7 (below) presents the same but for subgroups: by sex and age. The effect of sample design and weighting (deff) is more even across both sex and age categories in 2015 as compared with 2014. For an estimate of prevalence rate of 50% (most conservative scenario), the change in precision as measured by margin of error, compared with 2014, estimates ranges from a loss of 1.2 percentage points to a gain of 0.2 percentage points. Despite a decrease in sample size for the age group 18-29 years old from 183 to 142 the precision is improved as a result of a decrease in the design effect for this group.

Table 7: Accuracy of estimates in subgroups in 2014 and 2015

	Year	N	Average deff*	Effective sample size	effectiveness	50%	30/70%	10/90%
						Margin of Error	Margin of Error	Margin of Error
						(+/-)	(+/-)	(+/-)
Sex
Males	2014	656	1.83	359	55%	5.2%	4.7%	3.1%
	2015	582	1.76	330	57%	5.4%	4.9%	3.2%
Females	2014	845	1.54	550	65%	4.2%	3.8%	2.5%
	2015	706	1.60	442	63%	4.7%	4.3%	2.8%
Age2
18-29	2014	183	2.70	68	37%	11.9%	10.9%	7.1%
	2015	143	2.05	70	49%	11.7%	10.8%	7.0%
30-39	2014	220	1.46	151	69%	8.0%	7.3%	4.8%
	2015	193	1.69	114	59%	9.2%	8.4%	5.5%
40-64	2014	665	1.34	495	74%	4.4%	4.0%	2.6%
	2015	582	1.36	429	74%	4.7%	4.3%	2.8%
65+	2014	430	1.11	388	90%	5.0%	4.6%	3.0%
	2015	368	1.28	287	78%	5.8%	5.3%	3.5%

*The value presented in the table is an average of design effects estimated using complex sample design for the same questions and their categories in 2014 and 2015: Party identification (party3), Strength of party identification (idstrng2), Level of interest in politics (politic2), constitutional preference (scotpar2).

6.8 We conducted power analysis to explore the ability to detect significant differences between the sub-groups at an acceptable level of precision. The statistical power is the probability of detecting an effect (e.g. difference in proportions) when it really exists in a population. An acceptable level is 80%, which tells us that the study has an 80% chance of detecting a statistically significant difference in a test for difference of proportions if there really was one in a population. Power is directly related to effective sample size, which is why we should expect a decrease in the ability of detecting differences in 2015, especially for sub-groups, as compared with 2014.

6.9 Table 8 (below) presents results of power analysis for two examples of estimates from 2015. The difference in the proportion of males and females trusting Scottish Government to work in Scotland's best long-term interest was estimated at 4.7 percentage points. However, it has not been found statistically significantly, because the effective sample sizes of the two groups allow only for detecting difference of at least 6.2 percentage points. However, if the effective sample sizes were increased to the levels from SSA 2014 the difference would still be too small to be detected: males and females' trust in the Scottish Government would need to differ by at least 5.8 percentage points. However, it is clear that smaller sample size results in a reduced ability to detect smaller differences between groups.

Table 8: Minimal detectable difference in proportions between males and females for two estimates from SSA 2015 (power=0.8, alfa=0.05)

Question: answer	Effective sample size (survey estimate)		Difference in estimates between the groups	Minimal detectable difference	Minimal detectable difference (if N in 2015=1500)
	Male	Female
How much do you trust Scottish Govt to work in Scotland's best long-term interest?: A great deal / Quite a lot	320 (75.0%)	456 (70.3%)	4.7	6.2	5.8
Trust in Scottish Government to make fair decisions? : A great deal / Quite a lot	345	621	5.9	6.6	6.1
	(52.4%)	(46.5%)

6.10 A very similar approach has been taken to evaluate the impact of a reduced sample size on the ability to detect differences in trends over time. Table 9 (below) presents an example of two views on how people want Scotland to be governed, and an estimates percentage of respondents supporting them in the 2011 and 2015 SSA surveys. A small difference was observed for the first opinion (Scotland should become independent) and it could not have been detected by statistical tests because the effective sample sizes allowed for detecting a change of 3.5 percentage points. If we were to increase the 2015 sample's size to 1500, the minimal detectable difference would decrease slightly but not enough to enable detecting the difference of 1.9 percentage points. From this example we can say that increasing the sample size from 1288 to 1500 only results in a small increase in the power to detect the difference. It should be noted that in 2011 the sample size was 1197.

6.11 If we compare the years when the achieved sample size was 1500, the minimal detectable difference would decrease by another 0.2 percentage points. It seems that the benefit of increasing the sample size by 200 in the ability to detect the difference across years is marginal. This example also shows how important it is to treat significance tests only as a guide rather than a strict rule to follow when deciding on the conclusions from a survey. The decrease of 5.7 percentage points in the support for Scotland remaining in the UK with its own elected parliament could be detected with statistical tests with the samples achieved in both 2011 and 2015.

Table 9: Minimal detectable difference in proportions between years for two estimates from SSA 2015 and 2011 (power=0.8, alfa=0.05)

Question: answer	Effective sample size (survey estimate)		Difference in estimates between the groups	Minimal detectable difference	Minimal detectable difference (if N in 2015=1500)
	2011	2015
Scotland should become independent, separate from the UK and EU	685	825	1.9	3.5	3.3
	11.6%	13.5%
Scotland should remain part of the UK, with its own elected parliament and some tax powers	641	768	5.7	5.2	5.0
	48.9%	43.2%