Health and social care - staff experience: report 2021

Appendix 6: Statistical Notes

Significance Testing

Significance testing has been carried out on the iMatter data, to assess whether the movements in results from 2019 to 2021 are likely to be 'true', rather than 'chance'. Specifically, a series of t-tests has been used to examine the size of change needed to give us a very high level of confidence that a 'true' change has happened.

The key element here is the number of responses – the larger the number of responses, the smaller the minimum change that can be deemed statistically significant (meaning that the change is highly likely to be 'true').

Overall Health and Social Care Level Data

• A change of 0.3, or even 0.2 at times, is significant across Health and Social Care as a whole. This generally means any change reported from 2019 to 2021 is likely to be a 'true' change. The table following summarises the level of significance in Health and Social Care scores, with annotation used:
  • Fairly significant significance ≤ 0.05 but ≤0.01
  • Significant significance <0.01 but ≥0.001
  • Very significant significance < 0.001
• Because iMatter reporting is based on whole numbers and therefore some significant movements may not be visible in the report. It is suggested that future reporting is to one decimal place, in order that all significant movements can be reported.
• The questions highlighted following are the ones where the reported score (reported as a whole number) is unchanged from 2019, but where there is still a significant movement in the data. It is noted that the following two statements have significantly higher scores in 2021 than 2019:
  • My direct line manager is sufficiently approachable (moving from 86.6 in 2019 to 86.8 in 2021)
  • I have confidence and trust in my direct line manager (moving from 83.7 in 2019 to 84.0 in 2021)

Board Level Data

As noted previously, the number of respondents (the achieved sample size) is key to the level of movement year on year that is significant. Therefore, for individual Boards, significant movements are:

• Boards with less than 800 responses per year: movements of 3 points are significant
• Boards with between 800 and 2,800 responses per year: movements of 2 points are significant
• Boards with over 2,800 responses per year: movements of 1 point are significant

Movements of 3pp are significant

• Healthcare Improvement Scotland
• NHS Orkney
• The State Hospital
• NHS Western Isles
• NHS Education for Scotland

Movements of 2pp are significant

• NHS Shetland
• NHS 24
• Golden Jubilee Foundation
• NHS Borders
• Public Health Scotland

Movements of 1pp are significant

• NHS National Services Scotland
• Scottish Ambulance Service
• NHS Dumfries & Galloway
• NHS Forth Valley
• NHS Highland
• NHS Fife
• NHS Ayrshire & Arran
• NHS Lanarkshire
• NHS Tayside
• NHS Grampian
• NHS Lothian
• NHS Greater Glasgow & Clyde

As seen with the National data there are incidences where significant movements are not evident in data reported at whole integer level only. Occasionally where there appears to be a movement in scores, but this is simply a result of rounding to whole integers and the difference is in fact, not significant.

The table following shows the level of significance in EEI score movements for Board receiving reports in 2019 and therefore having an EEI score that can be compared with their 2021 score.

Significance testing has also been carried out on individual iMatter questions, the results of which follow.

• Again, there are occasions, in larger Boards, where a reported 'no change' in score has a significant movement.
• Also, there are occasions where scores are reported as different from 2019, but that difference is not significant. This typically occurs in smaller Boards.