Scottish Survey of Literacy and Numeracy 2015 (Numeracy)

Chapter 6: Background notes

6.1 Sampling frame

The sampling frame for the pupil sample is all P4, P7 and S2 pupils attending all mainstream schools in Scotland that have registered to participate in the SSLN. The SSLN includes Gaelic medium and independent schools but excludes special schools. Pupils with Additional Support Needs attending mainstream schools are included in the sample and should be given the same level of support they would normally have for assessments in class. The P7 pupil cohort in the 2013 survey is the same as the S2 cohort in 2015. The pupils sampled to participate in the survey will not necessarily have been the same in each year.

The sampling frame for the teacher questionnaire is all P4, P7 and secondary teachers in all participating schools.

6.2 Sample design

The pupil sample design is a two stage stratified random sample, i.e. pupils are selected at random within schools and by gender. The sample consists of two P4 and two P7 pupils from every participating primary school and up to twelve S2 pupils from every participating secondary school. This produces a target sample size of around 4,000 pupils per stage. Pupil results are weighted to account for different school sizes, the small number of non-participating schools and gender and deprivation differences between the sample and the population.

The teacher questionnaire is allocated to all P4 teachers at half of participating primary schools and all P7 teachers at the remaining half of participating primary schools. Within secondary schools, the teacher questionnaire is allocated to ten teachers covering an equal distribution of four broad curriculum areas and Mathematics. This produces a total target sample size of around 5,400 teachers. Teacher results are weighted to account for non-response and differences in school size.

6.3 Response rate

The response rate at school level was 97 per cent in publicly funded schools and 33 per cent of schools in the independent sector.

6.4 Interpretation of SSLN results

As in all sample surveys, as the SSLN is based on a sample of pupils rather than on the whole population, the results shown are estimates. Therefore there is an element of uncertainty within the results because the pupils sampled may not reflect the population exactly.

Uncertainty around the results is estimated using standard errors. Standard errors are a measure of the variation in the data i.e. how each observation differs from the mean. As the SSLN sample design is not a simple random sample - in the SSLN pupils at small schools have a higher probability of being selected than pupils at large schools - this means that standard formulae used to calculate the standard error from a simple random sample would not be appropriate. Standard errors are therefore calculated empirically using the jackknife procedure.

Standard errors are in turn used to produce confidence intervals around the estimates. Confidence intervals show the range of values within which one can be reasonably confident that the actual value would lie if all pupils were assessed.

Ninety-five per cent confidence intervals for the main national estimates were calculated and were around ± two percentage points. This means that the true value of each estimate is likely to lie within two percentage points either side of the given estimate.

Where appropriate, confidence intervals are represented on charts by error bars to help demonstrate this level of uncertainty. Where the estimates are different but the error bars overlap we cannot be sure that the true values of each estimate are statistically significantly different from each other. Significance tests (t-tests) are used to assess the statistical significance of comparisons made.

Standard error data for the results, used to calculate these confidence intervals, are provided in the supplementary tables available at www.gov.scot/ssln.

6.5 Sources

Attainment data are derived from the results of assessments completed by participating pupils. For the numeracy survey the assessment consists of two written assessment booklets and one practical assessment. Written booklets consist of short answer (atomistic) tasks and multi-item (extended) tasks, which was based on a source datasheet with multiple associated questions. The practical assessment consists of a one-to-one verbal assessment between the pupil and the classroom teacher or another member of teaching staff.

All participating pupils complete an online questionnaire on factors that are likely to affect learning and attainment, such as pupil attitudes and experiences in class.

Sampled teachers are asked to complete an online questionnaire on their experiences and views on teaching numeracy.

All SSLN data was collected during the fieldwork period of 5 ^th May - 5 ^th June 2015.

6.6 Use made of SSLN data

The results of the 2015 SSLN will be used in line with the survey's three main objectives. These are:

• to monitor and report nationally on achievement in numeracy at P4, P7 and S2, in 2015 and over time
• to identify areas of numeracy strengths and weaknesses among pupils in Scotland to help inform policy initiatives and learning and teaching practices
• to gather information and report nationally on pupils' and teachers' experience of learning and teaching numeracy, along with their views about this experience.

In line with the aim to improve learning and teaching practice, Education Scotland has developed PLRs based on an in-depth analysis of 2011 and 2013 SSLN data. These resources are used by teachers, schools and authorities to support and inform learning and teaching practice in the classroom. These resources are available on the Education Scotland website.

The ways in which these materials can be used are set out below.

In the classroom, as a practitioner:

• as a resource for Professional Learning ( CLPL) through use of the reflective questions provided for self-evaluation
• to focus lesson planning linking to known areas for improvement
• as a resource with links to further reading to help develop new concepts and ideas in teaching numeracy
• to enhance children and young people's numeracy skills, through use of the example questions and links to additional support materials
• to share views on numeracy across learning through use of the activities for teachers to stimulate dialogue and debate on teaching practice
• to support children and young people's numeracy learning across the curriculum.

In school, as a leader or manager:

• "to inform development plans to improve standards in numeracy" as per the CfE Implementation Plan
• to inform school improvement plans - the resources include high level findings with reflective questions for whole school self-evaluation to focus discussions around school improvement planning in relation to numeracy
• to lead CLPL sessions - the resources include a range of materials which can be used to lead specific sessions focusing on particular areas of numeracy e.g. pedagogy across the school, development of numeracy skills and strategies for learning and teaching to support these skills
• to provide a focus for classroom observation - learning communities in schools can use the resources to identify areas for improvement in their own context. The PLR appendices contain exemplar sheets for focused observation at school level.

At local authority level, as a development officer or Quality Improvement Officer:

• to provide a focus when supporting individual schools or clusters, to identify clear targets for improvement

• to inform and expand the range of professional development opportunities available for teachers
• to clarify the aspirations contained in the numeracy Es and Os
• to identify clear targets for improvement
• to inform transition projects by promoting collegiality with staff from primary and secondary schools.

6.7 Supplementary tables

The survey contains a huge amount of data which cannot be summarised in this publication. This report seeks to highlight the key messages and give a flavour of the range of analysis possible. Detailed tables of the performance data and pupil and teacher questionnaire results are published as supporting tables alongside this publication, and provide a fuller picture of the findings.

The following list of tables are available at www.gov.scot/ssln

Attainment

1.1: Distribution of scores by stage
1.2: Summary of performance by stage
1.3: Summary of performance by stage and gender
1.4: Distribution of scores by stage and deprivation category
1.5: Summary of performance by stage and deprivation category
1.6: Summary of performance by stage and organiser
1.7: Summary of performance by stage and task type type
1.8: Summary of performance by stage and numeracy operator

Attainment over time

2.1: Summary of performance by stage in 2011, 2013 and 2015
2.2: Summary of performance by stage and gender in 2011, 2013 and 2015
2.3: Summary of performance by stage and deprivation category in 2011, 2013 and 2015
2.4: Summary of performance by stage and organiser in 2011, 2013 and 2015

Pupil Questionnaire

3.1: Pupils' learning in school - Class activities
3.2: Pupils' learning in school - Class teacher engagement
3.3: Pupils' learning in school - School engagement
3.4: What pupils think about their learning - Engagement
3.5: What pupils think about their learning - Usefulness
3.6: Pupils' confidence in learning
3.7: What pupils think about numeracy - General
3.8: What pupils think about numeracy - Organisers
3.9: School and home
3.10: School Environment: Behaviour
3.11: School Environment: Understanding
3.12: School Environment: Decision making
3.13: School Environment: Activities
3.14: ICT: Use of computers, tablets, etc.
3.15: ICT: Confidence in use of computers, tablets, etc.
3.16: ICT: Attitudes

Teacher Questionnaire

4.1: Pupils' classroom activities in numeracy
4.2: Schools' wider links
4.3: Teachers' resources in numeracy
4.4: ICT supported activities
4.5: Integrating numeracy skills into teaching various curriculum areas - Primary
4.6: Integrating numeracy skills into teaching various curriculum areas - Secondary non-Mathematics
4.7: Confidence in delivering the numeracy experiences and outcomes
4.8: Confidence in delivering the numeracy experiences and outcomes - Secondary non-Mathematics breakdown
4.9: Gathering evidence of pupils' achievements in numeracy
4.10: Evaluating and recording the evidence of pupils' achievements in numeracy
4.11: Career-Long Professional Learning ( CLPL) in numeracy
4.12: Career-Long Professional Learning ( CLPL) in numeracy - Impact

Survey Data

5.1: Participation figures

Standard Errors

6.1: Assessment data
6.2: Pupil questionnaire data
6.3: Teacher questionnaire data

6.8 Cost of compliance

One of the recommendations resulting from the UK Statistics Authority assessment of the SSLN was to publish an estimate of the cost to data suppliers for participation. The Government Statistical Service has devised a method for estimating the cost that avoids imposing an extra burden on data providers. The method for calculating cost to organisations, including schools, is:

Cost = (number of responses x median time taken to respond in hours x hourly rate of typical respondent) + any additional costs experienced by data providers.

This methodology has been applied to the SSLN administration model and the estimated cost of compliance for the SSLN 2015 (numeracy) survey was £460,000.

6.9 Further information

Further information on the SSLN, including the supplementary tables and Survey Design Document, is available from www.gov.scot/ssln.

There is a range of other reliable information on the performance of Scotland's school pupils.

Scotland participates in the OECD's triennial Programme for International Student Assessment ( PISA) survey. This assessment is carried out by 15 year-olds in over sixty countries, including all OECD countries, and as such is a key international benchmark of performance. The results of previous PISA surveys are available at www.gov.scot/pisa

The Scottish Government also publishes analysis of SQA exam results and leaver destinations. The latest post-appeal data are available at http://www.gov.scot/Publications/2015/06/2579

Media enquiries about the information in this Statistics Publication Notice should be addressed to: Russell Fallis, Tel: +44(0) 131 244 3558