iMatter Health & Social Care Staff Experience Survey 2024

Appendix 6: Statistical Notes

Significance Testing

Significance testing has been carried out on the iMatter data, to assess whether year to year movements in results are likely to be ‘true’, rather than ‘chance’. Specifically, a series of t-tests have been used to examine the size of change needed to give us a very high level of confidence that a ‘true’ change has happened. The key element here is the number of responses – the larger the number of responses, the smaller the minimum change that can be deemed statistically significant (meaning that the change is highly likely to be ‘true’).

Overall Health and Social Care Level Data

• A change of 0.3, or even 0.2 at times, is significant across Health and Social Care as a whole. This generally means where reported scores have changed in 2024 from 2023, it is likely to be a ‘true’ change

Board Level Data

The number of respondents (the achieved sample size) is key to the level of movement year on year that is significant. Therefore, for individual Boards, significant movements are:

• Boards with less than 800 responses per year: movements of 3 points are significant
• Boards with between 800 and 2,800 responses per year: movements of 2 points are significant
• Boards with over 2,800 responses per year: movements of 1 point are significant

Movements of 3 points are significant

• Healthcare Improvement Scotland
• NHS Western Isles
• The State Hospital
• NHS Orkney

Movements of 2 points are significant

• NHS Shetland
• NHS 24
• NHS Golden Jubilee
• NHS Borders
• Public Health Scotland
• NHS Education for Scotland
• NHS Dumfries & Galloway

Movements of 1 point are significant

• NHS National Services Scotland
• NHS Tayside
• NHS Grampian
• NHS Lothian
• NHS Greater Glasgow and Clyde
• Scottish Ambulance Service
• NHS Forth Valley
• NHS Highland
• NHS Fife
• NHS Ayrshire & Arran
• NHS Lanarkshire

Full details of the significance testing can be found in the iMatter 2024 Data File^[16][17]. As can be seen with the National data there are incidences where significant movements are not evident in data reported at whole integer level only. Occasionally there appears to be a movement in scores, but this is simply a result of rounding to whole integers and the difference is in fact, not significant.

Note: where response rate is particularly high the extent of movement that is significant will be lower than those shown here.