Scottish Survey of Literacy and Numeracy 2016: literacy

Chapter 10: Background notes

10.1 Sampling frame

The sampling frame for the pupil sample is all P4, P7 and S2 pupils attending all mainstream schools in Scotland that have registered to participate in the SSLN. The SSLN includes Gaelic medium and independent schools but excludes special schools. Pupils with Additional Support Needs attending mainstream schools are included in the sample and should be given the same level of support they would normally have for assessments in class. The P7 pupil cohort in the 2014 survey is the same as the S2 cohort in 2016. The pupils sampled to participate in the survey will not necessarily have been the same in each year.

The sampling frame for the teacher questionnaire is all P4, P7 and secondary teachers in all participating schools.

10.2 Sample design

The pupil sample design is a two stage stratified random sample, i.e. pupils are selected at random within schools and by gender. The sample consists of two P4 and two P7 pupils from every participating primary school and up to twelve S2 pupils from every participating secondary school. This produces a target sample size of around 4,000 pupils per stage. Pupil results are weighted to adjust for possible non-response bias to ensure the sample reflects the profile of the general population.

The teacher questionnaire is allocated to all P4 teachers at half of participating primary schools and all P7 teachers at the remaining half of participating primary schools. Within secondary schools, the teacher questionnaire is allocated to ten teachers covering an equal distribution of four broad curriculum areas and English. This produces a total target sample size of around 5,400 teachers. Teacher results are weighted to account for non-response and differences in school size.

10.3 Response rate

The response rate at school level was 97 per cent in publicly funded schools and 33 per cent of schools in the independent sector.

10.4 Interpretation of SSLN results

As in all sample surveys, as the SSLN is based on a sample of pupils rather than on the whole population, the results shown are estimates. Therefore there is an element of uncertainty within the results because the pupils sampled may not reflect the population exactly.

Uncertainty around the results is estimated using standard errors. Standard errors are a measure of the variation in the data i.e. how each observation differs from the mean. As the SSLN sample design is not a simple random sample - pupils at small schools have a higher probability of being selected than pupils at large schools - this means that standard formulae used to calculate the standard error from a simple random sample would not be appropriate. Standard errors are therefore calculated empirically using the jackknife procedure.

Standard errors are in turn used to produce confidence intervals around the estimates. Confidence intervals show the range of values within which one can be reasonably confident that the actual value would lie if all pupils were assessed.

Ninety-five per cent confidence intervals for the main national estimates were calculated and were around ± two percentage points. This means that the true value of each estimate is likely to lie within two percentage points either side of the given estimate.

Where appropriate, confidence intervals are represented on charts by error bars to help demonstrate this level of uncertainty. Where the estimates are different but the error bars overlap we cannot be sure that the true values of each estimate are statistically significantly different from each other. Significance tests (t-tests) are used to assess the statistical significance of comparisons made.

Standard error data for the results, used to calculate these confidence intervals, are provided in the supplementary tables available at www.gov.scot/ssln.

10.5 Sources

Attainment data are derived from the results of assessments completed by participating pupils. For reading, the assessment consists of a paper reading booklet and an online reading assessment. The writing component consists of an assessment of two writing scripts per participating pupil. The listening and talking assessment consists of a group discussion between participating pupils based on a task designed to generate discussion.

All participating pupils complete an online questionnaire on factors that are likely to affect learning and attainment, such as pupil attitudes and experiences in class.

Sampled teachers are asked to complete an online questionnaire on their experiences and views on teaching numeracy.

All SSLN data was collected during the fieldwork period of 3rd May - 3rd June 2016.

10.6 Use made of SSLN data

The results of SSLN 2016 will be used in line with the survey's three main objectives. These are:

• to monitor and report nationally on achievement in literacy at P4, P7 and S2, in 2016 and over time
• to identify areas of literacy strengths and weaknesses among pupils in Scotland to help inform policy initiatives and learning and teaching practices
• to gather information and report nationally on pupils' and teachers' experience of learning and teaching literacy, along with their views about this experience.

In line with the aim to improve learning and teaching practice, Education Scotland has developed PLRs based on an in-depth analysis of previous SSLN data. These resources are used by teachers, schools and authorities to support and inform learning and teaching practice in the classroom. These resources are available on the National Improvement Hub.

The ways in which these materials can be used are set out below.

In the classroom, as a practitioner:

• as a resource for Career-Long Professional Learning ( CLPL) through use of the reflective questions provided for self-evaluation
• to focus lesson planning linking to known areas for improvement
• as a resource with links to further reading to help develop thinking around the learning and teaching of literacy skills
• to enhance children and young people's literacy skills, through reflective discussion of exemplar materials
• to share views on literacy across learning through use of the activities for teachers to stimulate dialogue and debate on teaching practice
• to support children and young people's literacy learning across the curriculum.

In school, as a leader or manager:

• "to inform development plans to improve standards in literacy" as per the CfE Implementation Plan
• to inform school improvement plans - the resource includes high level findings with reflective questions for whole school self-evaluation to focus discussions around school improvement planning in relation to literacy
• to lead CLPL sessions - the resources include a range of materials which can be used to lead specific sessions focusing on particular areas of literacy to provide a focus for classroom observation - learning communities in schools can use the resources to identify areas for improvement in their own context.

At local authority level, as a development officer or Quality Improvement Officer:

• to provide a focus when supporting individual schools or clusters, to identify clear targets for improvement
• to inform and expand the range of professional development opportunities available for teachers
• to clarify the aspirations contained in the literacy experiences and outcomes
• to identify clear targets for improvement
• to inform transition projects by promoting collegiality with staff from primary and secondary schools.

10.7 Supplementary tables

The survey contains a huge amount of data which cannot be summarised in this publication. This report seeks to highlight the key messages and give a flavour of the range of analysis possible. Detailed tables of the performance data and pupil and teacher questionnaire results are published as supporting tables alongside this publication, and provide a fuller picture of the findings.

The following list of tables is available at www.gov.scot/ssln.

Reading attainment

1.1: Distribution of scores by stage

1.2: Summary of performance by stage

1.3: Summary of performance by stage and gender

1.4: Summary of performance by stage and deprivation category

Reading attainment over time

2.1: Summary of performance by stage in 2012, 2014 and 2016

2.2: Summary of performance by stage and gender in 2012, 2014 and 2016

2.3: Summary of performance by stage and deprivation category in 2012, 2014 and 2016

Writing attainment

3.1: Summary of performance by stage

3.2: Summary of performance by stage and gender

3.3: Summary of performance by stage and deprivation category

Writing attainment over time

4.1: Summary of performance by stage in 2012, 2014 and 2016

4.2: Summary of performance by stage and gender in 2012, 2014 and 2016

4.3: Summary of performance by stage and deprivation category in 2012, 2014 and 2016

Listening and talking attainment

5.1: Summary of performance by stage

5.2: Summary of performance by stage and gender

5.3: Summary of performance by stage and deprivation category

Listening and talking attainment over time

6.1: Summary of performance by stage in 2012, 2014 and 2016

6.2: Summary of performance by stage and gender in 2012, 2014 and 2016

6.3: Summary of performance by stage and deprivation category in 2012, 2014 and 2016

Pupil questionnaire

7.1: Pupils' learning in school - Class activities

7.2: Pupils' learning in school - Class teacher engagements

7.3: Pupils' learning in school - School engagement

7.4: What pupils think about their learning - Engagement

7.5: What pupils think about their learning - Usefulness

7.6: Pupils' confidence in learning

7.7: What pupils think about literacy - General

7.8: What pupils think about literacy - Organisers

7.9: School and home

7.10: Pupils' activities outside of school

7.11: ICT: Use of computers, tablets etc.

7.12: ICT: Confidence in use of computers, tablets etc

7.13: ICT: Attitudes

Teacher questionnaire

8.1: Pupils classroom activities in literacy

8.2: Schools' wider links

8.3: Teachers' resources in literacy

8.4: ICT supported activities

8.5: ICT usage

8.6: Integrating literacy skills into teaching the various curriculum areas - Primary

8.7: Integrating literacy skills into teaching the various curriculum areas - Secondary non-English

8.8: Confidence in delivering the literacy experiences and outcomes

8.9: Gathering evidence of pupils' achievement in literacy

8.10: Evaluating and recording the evidence of pupils' achievements in literacy

8.11: Career Long Professional Learning ( CLPL) in literacy

8.12: Career Long Professional Learning ( CLPL) in literacy - impact

Survey data

9.1: Participation figures

Standard errors

10.1: Assessment Data

10.2 Pupil questionnaire data

10.3 Teacher questionnaire data

10.8 Cost of compliance

One of the recommendations resulting from the UK Statistics Authority assessment of the SSLN was to publish an estimate of the cost to data suppliers for participation. The Government Statistical Service has devised a method for estimating the cost that avoids imposing an extra burden on data providers. The method for calculating cost to organisations, including schools, is:

Cost = (number of responses x median time taken to respond in hours x hourly rate of typical respondent) + any additional costs experienced by data providers.

This methodology has been applied to the SSLN administration model and the estimated cost of compliance for the SSLN 2016 (literacy) survey was £394,000.

10.9 Further information

Further information on the SSLN, including the supplementary tables and Survey Design Document, is available from www.gov.scot/ssln.

There is a range of other reliable information on the performance of Scotland's school pupils.

The Achievement of Curriculum for Excellence (CfE) Levels 2015/16 Return is a census based data collection and gathers data on whether or not pupils have achieved the expected CfE Level for their stage based on the class teachers' professional judgement. The collection covers numeracy and the three elements of literacy (reading, writing, listening and talking) and four stages within Broad General Education: P1, P4, P7 and S3. These results are not directly comparable to the SSLN results due to the differing methodologies and reporting styles. Results for end 2015-16 were published in December 2016 and are available here: http://www.gov.scot/Topics/Statistics/Browse/School-Education/ACEL

Scotland participates in the OECD's triennial Programme for International Student Assessment ( PISA) survey. This assessment is carried out by 15 year-olds in over sixty countries, including all OECD countries, and as such is a key international benchmark of performance. The results of previous PISA surveys are available at www.gov.scot/pisa

The Scottish Government also publishes analysis of SQA exam results and leaver destinations. The latest post-appeal data are available at

http://www.gov.scot/Publications/2016/06/4523

Media enquiries about the information in this Statistics Publication Notice should be addressed to: Catherine Brown, Tel: +44(0) 131 244 2560.