Scottish Study of Early Learning and Childcare: phase 4 report

Appendix C – Details of data collection instruments

Ages and Stages Questionnaire (ASQ)

The Ages and Stages Questionnaire was administered as part of the keyworker observations.

Both the ASQ and SDQ questionnaires are widely used by Health Visitors across Scotland as part of their health reviews of pre-school children – the Child Health Programme.^[42] These questionnaires were also selected for inclusion in the Child Health Programme following an extensive review by academics and practitioners.^[43]

The ASQ is a structured assessment of a range of developmental domains to identify children at increased risk of developmental difficulties.^[44] The instrument includes 30 items split into five different domains: communication, gross motor, fine motor, problem solving and personal-social. The respondent indicates whether or not the child can complete the action or provide the response required by answering "yes", "sometimes" or "no". Each domain produces a summary score which can be used to indicate whether the child's development is on schedule, needs monitoring or requires further assessment. Whilst it is designed to be completed by parents, it was deemed suitable for completion by the child's keyworker at their ELC setting because it is informed by observation of the child.

Strengths and Difficulties Questionnaire (SDQ)

The Strengths and Difficulties Questionnaire was administered as part of the keyworker observations.

The SDQ is a commonly used behavioural screening questionnaire designed for use with children aged between four and 16. An adapted version is used for children aged two to four.^[45] The questionnaire includes 25 questions about a child's behaviour to which the respondent can answer "not true", "somewhat true" or "certainly true". Responses can be combined to form five different measures of the child's development, namely emotional symptoms (e.g. excessive worrying), conduct problems (e.g. often fighting with other children), hyperactivity / inattention (for example, constantly fidgeting), peer relationship problems (e.g. not having close friends), and prosocial behaviour (e.g. being kind to others). In addition, the first four measures can be combined into a "total difficulties" scale. Higher scores imply greater evidence of difficulties on each of the scales, with the exception of the prosocial behaviour scale where the reverse is true. In this report, recommended banded versions of the scales have been used to create the following categories: "close to average", "slightly raised", "high" and "very high", with "very high" indicating multiple difficulties identified.^[46]

Shortform Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS)

The short form of the Warwick-Edinburgh Mental Wellbeing Scale is a commonly used series of questions intended to measure wellbeing. Respondents are presented with seven statements about positive feelings and thoughts, and asked how frequently they have experienced them over the previous two weeks.

The scale is created by summing the frequency of responses for all seven items, with a score of 1 for "none of the time" and a score of 5 for "all of the time". This provides a score of between 7 and 35. In order to make the data comparable with other surveys, the scores have to be "Rasch-transformed".

Cut offs to determine "high" and "low" wellbeing can be identified from population norms. High wellbeing is defined as the top 15% of scores, in the range from 27.5 to 35.0, and low wellbeing as the bottom 15%, in the range from 7.0 to 19.5.

Details of the scoring are available from the WEMWBS website.

Home learning environment scale

Four items were used to create the home learning environment scale. These each related to the number of days on which certain activities between the child and parent were conducted. The scale was created by summing the number of days on which each of the four activities had occurred in the last week.

Cut offs to determine quartiles were created using data from Phase 1 of the study. This allows for comparison between phases. The bottom quartile included scores in the range 0 to 15, the second in the range 16 to 20, the third in the range 21 to 24, and the top in the range 25 to 28.

Parent-child warmth scale

The parent-child warmth scale used in the SSELC comprises the seven items that form the warmth dimension of the short form of the Mothers' Object Relations Scale (MORS-SF). Details of the scale are available from the MORS website.

Each item was scored from 1 to 5, with 1 being "None of the time" and 5 "All of the time". Summing the seven items created a scale from 7 to 35. Three groups were created of roughly equal size, based on Phase 1 responses. Because of the skewed nature of the data, the bottom tertile (a score of 7 to 31) should not be interpreted as "low warmth", as many of the respondents in this group would have answered "often" or "all of the time" to each of the statements. The middle tertile was for a score of 32 to 34 and the top tertile a score of 35.

Confusion, Hubbub and Order Scale

To form the scale, the first two items shown in Figure 4.10 were scored 1 to 5, with 1 being "strongly disagree" and 5 being "strongly agree"; the other two items were scored 1 to 5 in the reverse order, with 1 being strongly agree and 5 strongly disagree. The sum of the four items created a scale from 4 to 20, with higher values indicating more chaotic households. Tertiles were based on Phase 2 of the study, where the sample was representative of households with a four- or five-year-old child receiving funded ELC.

Infant / Toddler Environment Rating Scale (ITERS-3)

To gather information on the characteristics of ELC settings, inspectors from the Care Inspectorate (acting as observers independent of their regulatory roles) conducted observations of 149 settings using the most recent version of the Infant / Toddler Environment Rating Scale (ITERS-3). The ITERS-3 was developed in the United States by the Environment Rating Scale Institute along with the Early Childhood Environment Rating Scale (ECERS). Both scales are widely used in English speaking countries. In the United Kingdom, ECERS has been used in both the Effective Provision of Pre-School Education (EPPE) study and in the more recent Study of Early Education and Development (SEED) in England^[47]. In Scotland, ECERS was used as long ago as 1994^[48] and has seen many applications since that time.

Both environment rating scales have a positive international reputation as a way of assessing the quality of provision in a snap-shot observation and as a tool which gives researchers access to the everyday experiences of children in their ELC settings. The scales have high reliability at indicator and item level when used by trained observers^[49]. Validity is also high in terms of their relationship to other ways of assessing quality and to measures of children's outcomes. Further, in conjunction with academics and the Care Inspectorate, some minor amendments were made to ensure that the ITERS-3 was reflective of the aspects of quality that are expected in Scotland (e.g. that rainfall should not prevent outdoor play).

ITERS-3 was used for a number of reasons: it centres on the experience of the child in the setting; it allows for the effect of setting quality on child outcomes to be controlled for; and it is relatively easy to administer given that only one three-hour observation is required. The tool can also be used to determine if particular setting characteristics contribute to differential outcomes in children. Furthermore, ITERS-3 is designed for use in settings where most children are under 36 months and as such, it was deemed suitable for use with the eligible two-year-olds involved in Phase 4 of the SSELC. Although many settings observed did not have a specific two-year-old room, using ITERS-3 allowed for age-appropriate criteria to be observed.

It is important to note that these tools are not the only method of assessing setting quality in Scotland. Indeed, the Care Inspectorate ratings provide a broader measure of the quality of practice and policy within settings that have also been found to be related to children's outcomes in Scotland.^[50]

As with the Care Inspectorate inspection methodology, the setting observations focused on outcomes. However, the methodology differed in that the ITERS-3 tool was used to observe for three hours, with no consultation with setting staff and no professional dialogue or explicit feedback provided. This was because the observations were intended to be a snapshot to inform the study and control for variation in child outcome data, rather than serving as an assessment of an individual setting's quality. During the ITERS-3 observations, observers looked at the six domains specifically for two-year-olds. In contrast, during a formal inspection, Care Inspectorate inspectors consider a range of areas that impact on experiences for all children attending the setting, not just those in specific age groups. The key areas covered during a formal inspection are likely to include some or all of the domain areas but can also cover other aspects of the provision to evaluate the overall quality of the setting.

The ITERS-3 scale comprises 33 items across six different subscales: space and furnishings; personal care routines; language and books; activities; interaction, and programme structure.

• Space and furnishings include observation of indoor space; furnishings for care, play, and learning; room arrangement; and display for children.
• Personal care routines include observation of meals and snacks; toileting; health practices; and safety practices.
• Language and books include observation of talking with children; encouraging vocabulary development; responding to children's communication; encouraging children to communicate; staff use of books with children; and encouraging children's use of books.
• Activities includes observation of fine motor; art; music and movement; blocks; dramatic play; nature and science; maths and number; appropriate use of technology; promoting acceptance of diversity; and gross motor.
• Interaction includes observation of supervision of gross motor play; supervision of non-gross motor play and learning; peer interaction; staff-child interaction; providing physical warmth and touch; and guiding children's behaviour.
• Programme structure includes observation of schedule and transitions; free play; and group play activities.

In line with ITERS-3 guidance, each subscale is scored from 1 to 7. These scores are calculated by averaging the score for each item within the subscale. Each of the 33 items are also scored from 1 to 7. These scores are calculated using the indicators contained within each individual item. Indicators are grouped under scores of 1 (inadequate), 3 (minimal), 5 (good), and 7 (excellent), with each indicator providing an example of what should be observed relevant to each score. Indicators themselves are scored as yes or no depending on whether the indicator has been observed. In some cases, observers are able to record indicators or items as not applicable; these are then excluded when calculating item or subscale scores. A score of 1 is given if any indicator grouped under 1 is scored yes. For an item to score a 7, each indicator grouped under 7 must be scored yes.

How to access background or source data

The data collected for this social research publication:

☐ are available in more detail through Scottish Neighbourhood Statistics

☐ are available via an alternative route

☒ may be made available on request, subject to consideration of legal and ethical factors. Please contact socialresearch@gov.scot for further information

☐ cannot be made available by Scottish Government for further analysis as Scottish Government is not the data controller.