Health and social care staff experience: report 2019

Appendix 3: Statistical Notes

Significance Testing

Significance testing has been carried out on the iMatter data, to assess whether the movements in results from 2018 to 2019 are likely to be ‘true’, rather than ‘chance’. Specifically, a series of t-tests has been used to examine the size of change needed to give us a very high level of confidence that a ‘true’ change has happened.

The key element here is the number of responses – the larger the number of responses, the smaller the minimum change that can be deemed statistically significant (meaning that the change is highly likely to be ‘true’). 

Overall Health and Social Care Level Data

• A change of 0.3, or even 0.2 at times, is significant across Health and Social Care as a whole. This generally means any change reported from 2018 to 2019 is likely to be a ‘true’ change. 
• However, the above finding highlights a slight challenge, since iMatter reporting is based on whole integers, and therefore some significant movements may not be visible in the report. We therefore recommend that future reporting is to one decimal place, in order that all significant movements can be reported.

Board Level Data

As noted above, the number of respondents (the achieved sample size) is key to the level of movement year on year that is significant. Therefore, for individual Boards, significant movements are:

• Boards with less than 800 responses per year: movements of 3 points are significant
• Boards with between 800 and 2,800 responses per year: movements of 2 points are significant
• Boards with over 2,800 responses per year: movements of 1 point are significant  

Regression Analysis

Regression analysis has been carried out on the 2019 iMatter data to examine the extent to which the overall thermometer question is related to the individual component questions. The scores for the thermometer question (‘Overall, working within my organisation is a ..very poor experience (0) . . . very good experience (10)’) were compared to each of the component scores (6 for Strongly Agree . . . 1 for Strongly Disagree) for each respondent.

The analysis looks at the degree of ‘similarity’ between the overall score and the component scores - that is, the extent to which respondents giving a high (or low) overall score also gave a high (or low) component score. In practical terms this provides insight into which components have the greatest influence on staff’s overall experience of working within their organisation.

Unsurprisingly, responses to ‘I would recommend my organisation as a good place to work’ were most closely related to the overall score, reflecting their broadly similar nature. The other components are shown below in order. For example, ‘I feel my organisation cares about my health and wellbeing’ has the next strongest influence on the overall score. 

1. I feel my organisation cares about my health and wellbeing
2. I would be happy for a friend or relative to access services within my organisation
3. My work gives me a sense of achievement
4. I am confident performance is managed well within my organisation
5. I have confidence and trust in senior managers responsible for the wider organisation
6. I have sufficient support to do my job well
7. I would recommend my team as a good one to be a part of
8. I feel appreciated for the work I do
9. I feel involved in decisions relating to my organisation
10. I am treated fairly and consistently
11. I am given the time and resources to support my learning growth
12. I have confidence and trust in my direct line manager
13. I am treated with dignity and respect as an individual
14. I get the information I need to do my job well
15. I get enough helpful feedback on how well I do my work
16. I am clear about my duties and responsibilities
17. I feel my direct line manager cares about my health and well-being
18. I am confident performance is managed well within my team
19. I am confident my ideas and suggestions are listened to
20. My team works well together
21. My direct line manager is sufficiently approachable
22. I feel involved in decisions relating to my job
23. I am confident my ideas and suggestion are acted upon
24. I feel involved in decisions relating to my team
25. I get the help and support I need from other teams and services within the organisation to do my job
26. I feel senior managers responsible or the wider organisation are sufficiently visible
27. I understand how my role contributes to the goals of my organisation