Health & Social Care Staff Experience Survey 2023

Appendix 6: Statistical Notes

Significance Testing

Significance testing has been carried out on the iMatter data, to assess whether year to year movements in results are likely to be ‘true’, rather than ‘chance’. Specifically, a series of t-tests have been used to examine the size of change needed to give us a very high level of confidence that a ‘true’ change has happened. The key element here is the number of responses – the larger the number of responses, the smaller the minimum change that can be deemed statistically significant (meaning that the change is highly likely to be ‘true’).

Overall Health and Social Care Level Data

• A change of 0.3, or even 0.2 at times, is significant across Health and Social Care as a whole. This generally means any change reported from 2021 to 2022 is likely to be a ‘true’ change

Board Level Data

The number of respondents (the achieved sample size) is key to the level of movement year on year that is significant. Therefore, for individual Boards, significant movements are:

• Boards with less than 800 responses per year: movements of 3 points are significant
• Boards with between 800 and 2,800 responses per year: movements of 2 points are significant
• Boards with over 2,800 responses per year: movements of 1 point are significant

As can be seen with the National data there are incidences where significant movements are not evident in data reported at whole integer level only. Occasionally there appears to be a movement in scores, but this is simply a result of rounding to whole integers and the difference is in fact, not significant. The following tables illustrate where this is the case.

Movements of 3pp are significant

• Healthcare Improvement Scotland^[19]
• NHS Western Isles
• The State Hospital

Movements of 2pp are significant

• NHS Shetland
• NHS 24
• NHS Golden Jubilee
• NHS Borders
• Public Health Scotland
• NHS Orkney
• NHS Education for Scotland

Movements of 1pp are significant

• NHS National Services Scotland
• NHS Tayside
• NHS Grampian
• NHS Lothian
• NHS Greater Glasgow and Clyde
• Scottish Ambulance Service
• NHS Forth Valley
• NHS Highland
• NHS Fife
• NHS Ayrshire & Arran
• NHS Lanarkshire
• NHS Dumfries & Galloway

The data in the iMatter 2023 Data file^[20] shows that all movements in component scores are significant, even those that appear to show no movement when reported as integers.

Correlation Analysis

Correlation coefficients are used to measure how strong a relationship is between two variables. The Pearson correlation is also known as the “product moment correlation coefficient” (PMCC) or simply “correlation”. The Pearson’s correlation helps in measuring the strength and the existence of a linear relationship between the two variables and if the outcome is significant we conclude that the correlation exists.

Correlation analysis has been carried out on all 28 iMatter variables and the additional ‘raising concerns’ variables.

Generally, correlation is high within My Experience as an Individual, My Team/My Direct Line Manager and My Organisation themes, but is less strong across them. Elements within My Team have relatively low correlation with elements within My Organisation.

Several clusters exist with high levels of correlations across them, meaning that individuals typically score each statement in a similar or identical way:

Involvement

• I am confident my ideas and suggestions are listened to
• I am confident my ideas and suggestion are acted upon
• I feel involved in decisions relating to my job
• I feel involved in decisions relating to my team

Treatment

• I am treated with dignity and respect as an individual
• I am treated fairly and consistently

Line Manager

• My direct line manager is sufficiently approachable
• I have confidence and trust in my direct line manager
• I feel my direct line manager cares about my health and wellbeing

Team

• My team works well together
• I would recommend my team as a good one to be part of

Organisation

• I feel that board members who are responsible for my organisation are sufficiently visible
• I have confidence and trust in Board members who are responsible for my organisation
• I feel involved in decisions relating to my organisation
• I am confident performance is managed well within my organisation

Further details can be found in the iMatter 2023 Data file