Health and Care Experience Survey 2023/24 Technical Report

Analysis and Reporting

The survey data collected and coded by IQVIA were securely transferred to Public Health Scotland and then from PHS to the Scottish Government. The information was analysed by Public Health Scotland and the Scottish Government using the statistical software package R.

Reporting the Sex of Respondents

Analysis of survey response rates by sex was undertaken using the sex of people in the sample according to their CHI record at the time of data extraction (25 September 2023). Sex and gender questions were not asked in the survey to reduce the respondent burden.

Reporting the Age of Respondents

Respondent date of birth was taken from their CHI record at the time of data extraction (25 September 2023). The age of respondents reported in the survey is as at 25 September 2023. i.e. the date when the sampling procedure commenced. This meant that an age question was not asked in the survey to reduce the respondent burden.

Number of Responses Analysed

The number of responses that have been analysed for each question is often lower than the total number of survey responses received. This is because not all of the questionnaires that were returned could be included in the calculation of results for every individual question. Respondents are instructed to skip any questions they do not wish to answer. Respondents may skip a question or their response may be excluded from analysis because:

• The specific question did not apply to the respondent and so they did not answer it. For example if they did not use Out of Hours services in the previous 12 months and therefore did not answer questions about their experience of it.
• The respondent did not answer the question for another reason (e.g. refused). People were advised that if they did not want to answer a specific question they should leave it blank.
• The respondent answered that they did not know or could not remember the answer to a particular question (for participants who answered questions over the phone/language line).
• Responses may be removed following validation checks, for example if a respondent selected an invalid combination of responses. Validation rules are automatically applied at the point of data collection through the online questionnaire (for example through enforcing “tick one box only” instructions) and validation checks are applied to ensure consistency between online and paper responses.

Weighting

When conducting a survey, it is important to have a representative sample of the population one is interested in. Applying weighting methods reduces potential bias by making the results more representative of the population.

Survey weights are numbers associated with the responses that specify the influence the various observations should have in the analysis. The final survey weight associated with a particular response can be thought of as a measure of the number of population units represented by that response.

A review of the weighting methodology was undertaken in 2017, leading to some changes in the weights applied. Details of the methodology used to calculate the weights is set out in Annex B.

Results at all levels of reporting are weighted. Only the “about you” questions are unweighted. This is because the “about you" questions are intended to provide an indication of the demographic profile of the survey’s respondents and survey was not designed to estimate the proportion of the population that has a particular demographic characteristic.

Analysis Software

Both the Scottish Government and Public Health Scotland use the software package R to analyse the data. This makes it easier for the Scottish Government and PHS to collaborate on the project, share code and reduces duplication of effort. The code used to analyse the data is available to view on a GitHub repository online.

Percentage Positive and Negative

Per cent or percentage positive is frequently used in reporting results from this survey. This means the percentage of people who answered that they had a positive experience. For example, when people were asked to rate the care provided by their general practice, if they answered “Excellent” or “Good”, their answers were counted as positive answers. Similarly, if people answered “Poor” or “Very poor”, their answers were counted as negative answers. Annex A details which answers have been classed as positive and negative for each question.

We report results in terms of per cent or percentage positive because they are easier to interpret, compared with reporting results on the five point scale that people used to answer the questions. There is also a belief that differences between answers on a five-point scale may be subjective. For example there may be little or no difference between a person who “strongly agrees” and one who “agrees” with a statement. In fact, some people may never strongly agree or strongly disagree with any statements.

Sampling error

This survey is of a sample of people registered with a General Practice and are therefore affected by sampling error. The effect of this sampling error is relatively small for the national estimates. However, when comparisons have been made in the analysis of the survey results, the effects of sampling error have been taken into account by the use of confidence intervals and tests for statistical significance. Only differences that are statistically significant are reported as differences within the analysis and all significance testing is carried out at the 5% level.

More information on confidence intervals, significance testing and how they’re calculated can be found in Annex C.

Quality Assurance of the National Report

A small group of Scottish Government analysts and policy leads were sent a draft version of the national report for quality assurance. Feedback included suggestions on ways in which to report data as well as comments about the context for the survey. These were taken into account in finalising the National Report. In addition, staff at IQVIA and Public Health Scotland carried out quality checks of figures used in the report.