Health and social care staff experience: report 2018

Part 1: Response Rate Analysis

All questions on the iMatter questionnaire are mandatory to eliminate partial loss of data. Only those questionnaires that have every question answered can be included within the analysis. The Response Rate shows the number of staff issued with the questionnaire (Recipients) and the number of staff who responded (Respondents) as an overall percentage. A response rate of 60% is required for teams of 5 or more and 100% for teams of 4 or less to generate a report. This is to ensure anonymity and the higher the response rate, the more realistic the feedback of how staff feel about working in their team. In total 177,619 questionnaires were issued and 104,254 usable responses were received. This equates to an overall response rate of 59%.

Comparing 2017 to 2018 Response Rates

Response rates at individual Board level vary considerably from 51% in NHS Highland and NHS Western Isles to 91% in NHS Health Scotland. All Boards with a response rate of 60% or over have received a report.

Organisation	Response rate 2017	Response Rate 2018	Response Rate Movement (pp)
Health and Social Care	63%	59%	-4
Golden Jubilee Foundation	68%	63%	-5
Healthcare Improvement Scotland	80%	86%	+6
NHS 24	67%	70%	+3
NHS Ayrshire & Arran	63%	59%	-4
NHS Borders	61%	53%	-8
NHS Dumfries & Galloway	63%	59%	-4
NHS Education for Scotland	81%	84%	+3
NHS Fife	62%	53%	-9
NHS Forth Valley	65%	62%	-3
NHS Grampian	64%	60%	-4
NHS Greater Glasgow & Clyde	58%	54%	-4
NHS Health Scotland	85%	91%	+6
NHS Highland	58%	51%	-7
NHS Lanarkshire	65%	62%	-3
NHS Lothian	65%	63%	-2
NHS National Services Scotland	76%	77%	+1
NHS Orkney	73%	83%	+10
NHS Shetland	61%	56%	-5
NHS Tayside	65%	58%	-7
NHS Western Isles	52%	52%	0
Scottish Ambulance Service	64%	64%	0
The State Hospital	78%	77%	-1

Whilst the overall response rate dropped 4 percentage points (pp), there is considerable variation at individual Board level. NHS Orkney achieved a 10 percentage point increase in response rate from an already strong 73% to one of the highest at 83%.

NHS Fife saw the biggest drop in response rate from 62% to 53%. As the 2018 response rate was less than 60% NHS Fife did not receive a report for 2018. Other Boards with large drops in response rates were:

• NHS Borders -8 pp
• NHS Highland -7 pp
• NHS Tayside -7 pp
• NHS Shetland -5 pp

There are 3 Boards that did not receive a report in either 2017 or 2018. They are:

• NHS Greater Glasgow & Clyde
• NHS Highland
• NHS Western Isles

As would be expected from the overall response rate data, it is generally the Boards with a higher share of paper questionnaires that are less likely to get a report.

Three Boards did not achieve the 60% response rate threshold for online or for paper questionnaires:

• NHS Shetland
• NHS Highland
• NHS Western Isles

The difference between online and paper response rates

Overall 84% of the questionnaires were issued via email and 16% were on paper. The response rates differ greatly between the two methods, with online generating a far higher response rate than paper questionnaires.

Health and Social Care 2018	Volume issued	Usable Response Volume	% Response Rate
Email	149,557	95,693	64%
Paper	28,062	8,561	31%
Total	177,619	104,254	59%

This difference in response rate by method resulted in 92% of the total usable responses coming from online completions, and only 8% from paper questionnaires.

The variability in response rates across Boards is evident in both online and paper methods: The online response rate ranges from 91% for Health Scotland to 55% for NHS Shetland. The paper response rate ranges from 54% for Golden Jubilee Foundation to 6% for NHS Borders.

Health and Social Care 2018	Volume issued	% of Volume issued	Volume returned	% of Volume issued
Email	149,557	84%	95,693	92%
Paper	28,062	16%	8,561	8%
Total	177,254		104,254

Response Rates by Board

Note: 4 Boards (Health Improvement Scotland. NHS 24, NHS Education for Scotland, NHS Health Scotland) distributed all questionnaires electronically

	Response rates
	Email	Paper	Total
Health and Social Care	64%	31%	59%
Golden Jubilee Foundation	64%	54%	63%
Healthcare Improvement Scotland	86%	N/A	86%
NHS 24	70%	N/A	70%
NHS Ayrshire & Arran	64%	40%	59%
NHS Borders	62%	6%	53%
NHS Dumfries & Galloway	62%	32%	59%
NHS Education for Scotland	84%	N/A	84%
NHS Fife	60%	18%	53%
NHS Forth Valley	65%	24%	62%
NHS Grampian	66%	38%	60%
NHS Greater Glasgow & Clyde	62%	22%	54%
NHS Health Scotland	91%	N/A	91%
NHS Highland	57%	27%	51%
NHS Lanarkshire	66%	41%	62%
NHS Lothian	67%	43%	63%
NHS National Services Scotland	77%	15%	77%
NHS Orkney	83%	14%	83%
NHS Shetland	56%	33%	56%
NHS Tayside	62%	25%	58%
NHS Western Isles	59%	16%	52%
Scottish Ambulance Service	64%	31%	64%
The State Hospital	79%	45%	77%

Shaded boards did not get 2018 iMatter EEI report

The table above shows the response rate for questionnaires returned by email and by post for each Board. It illustrates that the postal response rate is always lower than the email response rate. However, it also shows that there is no clear relationship between the response rates for the two methods within each Board. Nor did we see any relationship between size of Board and response rates.

Geographic and National Boards

However, differences can be seen between the National and Geographic Boards (see below for lists of each type of Board) Response rates are higher among the National Boards, both for email and paper. Only 1% of paper questionnaires distributed were to staff in National Boards. Given higher response rates for online questionnaires generally, this increases still further the overall response rates for the National compared with Geographic Boards.

Response Rates and Volumes	Email	Paper	Total
Geographic Boards
Issued questionnaires	136,376	27,651	164,027
Responses	86,238	8,372	94,610
Response Rate	63%	30%	58%
National Boards
Issued questionnaires	13,181	411	13,592
Responses	9,455	189	9,644
Response Rate	72%	46%	71%
Share of questionnaires issued
Geographic Boards	91%	99%	92%
National Boards	9%	1%	8%
Share of responses
Geographic Boards	90%	98%	91%
National Boards	10%	2%	9%

Geographic Boards	National Boards
NHS Ayrshire & Arran	Golden Jubilee Foundation
NHS Borders	Healthcare Improvement Scotland
NHS Dumfries & Galloway	NHS 24
NHS Fife	NHS Education for Scotland
NHS Forth Valley	NHS Health Scotland
NHS Grampian	NHS National Services Scotland
NHS Greater Glasgow & Clyde	Scottish Ambulance Service
NHS Highland	The State Hospital
NHS Lanarkshire
NHS Lothian
NHS Orkney
NHS Shetland
NHS Tayside
NHS Western Isles

Response rate comparison 2017 to 2018

The overall response rate has dropped by 4 percentage points (pp) from 63% in 2017 to 59% in 2018. Within this the online response rate has dropped 4 pp, but the paper response rate has dropped by 7 pp.

Health and Social Care 2017	2017 Volume issued	Usable Response Volume 2017	% Response Rate 2017	% Response Rate 2018	Change in Response Rate (pp) 2017-18
Email	141,990	96,631	68%	64%	-4
Paper	30,597	11,599	38%	31%	-7
Total	172,587	108,230	63%	59%	-4

Compared to last year, the share of online questionnaires issued has increased marginally from 82% (to 84%) and the proportion of questionnaires completed online in 2017 was also lower at 89% compared to 92% in 2018.

Unusable responses

There were 8,625 recipients of the online questionnaire who clicked on the link but failed to finish the questionnaire.

A prerequisite of iMatter is that every question on the questionnaire is answered. If there are any errors on the questionnaire then it is not processed. For the online questionnaire this is monitored within the script and so non-completion is not an issue.

On the paper questionnaire it is not possible to monitor in this way and so it is only when returned questionnaires are processed that incomplete or incorrectly completed responses are identified.

The table below shows details of the reasons for paper responses being rejected and the incidence of each type.

Paper Questionnaires	Number	Percentage of Questionnaires Issued	Percentage of Questionnaires Returned
Questionnaires Issued	28,062
No Response	16,856	60%
Valid Response	8,561	31%	76%
Rejected Questionnaire	2,645	9%	24%
Partial Response	1,278	5%	11%
Completion Errors	670	2%	6%
Duplicate	245	1%	2%
Past Deadline	452	2%	4%

In order to achieve the required 60% response rate, only 1429 additional completed questionnaires would have been needed. Given 2645 paper responses were rejected, the 60% response rate target could conceivably have been met.

Partial Response Analysis

As shown above 1,278 returned paper questionnaires were only partial responses, and so have not been included in the results. This comprises 5% of all paper questionnaires issued, and 11% of those questionnaires returned. Further analysis shows that the level of missing questions is very low.

29% do not answer the thermometer question (overall experience). Of the Staff Governance Standards the My Team section is the most likely to be left completely blank.

The individual questions most likely to be left unanswered are as shown below. Many of these questions are ones that the respondent may feel unable to answer.

For example, some staff may feel that particular elements do not apply or are not relevant to them (e.g. relating to career progression, training etc.). Others are statements that the respondent may feel unable to answer because they do not have enough knowledge or perhaps do not fully understand the question.

Statement

I am given the time and resources to support my learning and growth
I am confident my ideas and suggestions are listened to
I am confident performance is well managed in my team
I get the information I need to do my job well
I feel senior managers responsible for the wider organisation are sufficiently visible
I am treated with dignity and respect as an individual
I have confidence and trust in senior managers responsible for the wider organisation
I feel involved in decisions relating to my job
I am confident performance is managed well within my organisation
I get the help and support I need from other teams and services within the organisation to do my job
I have sufficient support to do my job well
I am confident my ideas and suggestions are acted upon

Summary

Overall the response rate fell just short of the required 60% for a National Report to be issued, a drop of 4 percentage points from last year. Nine Boards also failed to reach the threshold, an increase from the three without reports last year.

Response rates vary considerably across the Boards, with the National Boards typically achieving higher response rates than Geographic Boards.

There is considerable difference in the response rates between the two methodologies. Online, which accounts for 84% of the volume of questionnaires issued, achieves a 64% response rate. Paper questionnaires, whist only a small proportion of the questionnaires, only achieved a 31% response rate this year, down 7 percentage points from last year. The requirement for 100% completion of the questionnaire leads to a notable volume of returned questionnaires being rejected. This report recommends further investigation into the paper questionnaire, both in terms of how it is distributed and how it is processed, in order to improve the response rate.

Whilst this report explores where the response rate has moved it cannot assess the reasons for that movement. However, given the variations at Board level both in absolute response rates and movement from last year, further work with staff is recommended, in order to understand why staff have chosen not to participate, or otherwise feel unable to do so. In particular, this might explore further why a small proportion of staff do not feel able to fully complete the questionnaire. It should be recalled that a fully-independent academic evaluation of the model has been commissioned, this will report in 2019 and may present findings in respect of these issues.