Scottish Cancer Patient Experience Survey 2024: technical report

Analysis and Reporting

The survey data collected and coded by the survey contractor were securely transferred to Public Health Scotland (PHS) and then from PHS to the Scottish Government. The information was analysed by PHS and the Scottish Government using the statistical software package R.

Reporting the Sex of Respondents

Analysis of survey response rates by sex was undertaken using the sex of people in the sample according to their SMR01 record at the time the survey cohort was established (18^th January 2024). Sex and gender questions were not asked in the survey to reduce the respondent burden.

Reporting the Age Group of Respondents

Respondents age group was calculated from their date of birth in their CHI record. The age group of respondents reported in the survey is as at 18^th January 2024. i.e. the date when the survey cohort was established. Survey recipients were not asked for their age group in the survey to reduce the respondent burden.

Reporting the Cancer Group

The cancer group used in analysis is taken from the Scottish Cancer Registry (SMR06) record. Cancer types have been grouped to align to the definition of Less Survivable Cancers in the 2023 to 2033 Cancer Strategy for Scotland, while retaining as much comparability with previous surveys as possible and ensuring that no cancer group contains fewer than 50 responses so that any analysis by Cancer Group is robust and reduce the risk of someone being identifiable in the data set.

The survey results can be analysed by the following Cancer Groups:

• Breast
• Colorectal / lower gastrointestinal
• Gynaecological
• Haematological
• Head and neck
• Less Survivable Cancers
• Lung
• Oesophageal
• Other
• Prostate
• Skin
• Urological

More detail on the ICD codes used to determine Cancer Group can be found in the Glossary of this report.

Number of Responses Analysed

The number of responses that have been analysed for each question is often lower than the total number of survey responses received. This is because not all of the questionnaires that were returned could be included in the calculation of results for every individual question. Respondents are instructed to skip any questions they do not wish to answer. Respondents may skip a question or their response may be excluded from analysis because:

• The respondent did not feel that a specific question was relevant to their experience and so they did not answer it.
• The respondent did not answer the question because they did not wish to share that information (e.g. refused).
• The respondent answered that they did not know or could not remember the answer to a particular question.
• Responses may be removed following validation checks, for example if a respondent selected an invalid combination of responses. Validation rules are automatically applied at the point of data collection through the online questionnaire (for example through enforcing “tick one box only” instructions) and validation checks are applied to ensure consistency between online and paper responses.

Weighting

When conducting a survey, it is important to have a representative sample of the population we are interested in.

Weighting was introduced for the 2018 survey to take account of any non-response bias in the survey and ensure responses were representative of the sample population. Applying weighting methods reduces potential bias by making the results more representative of the population.

Survey weights are numbers associated with the responses that specify the influence the various observations should have in the analysis. The final survey weight associated with a particular response can be thought of as a measure of the number of population units represented by that response.

Details of the methodology used to calculate the weights is set out in Annex D.

Results at national, Regional Cancer Network and NHS Board levels have been weighted. Results at Cancer Centre level are presented unweighted.

The “about you” questions are unweighted. This is because the “about you" questions are intended to provide an indication of the demographic profile of the survey’s respondents, and hence how representative the survey sample is, rather than provide results on the demographic profile of cancer patients.

Analysis Software

Both the Scottish Government and Public Health Scotland use the software package R to analyse the data.

Percentage Positive and Percentage Negative

The term ‘percentage positive’ is frequently used in reporting results from this survey. This means the percentage of people who answered that they had a positive experience. For example, when people were asked to rate the administration of their care, if they answered “Very good” or “Good”, their answers were counted as positive answers. Similarly, if people answered “Bad” or “Very bad”, their answers were counted as negative answers.

We report results in terms of per cent or percentage positive because they are easier to interpret, compared with reporting results on the five point scale that people used to answer the questions. There is also a belief that differences between answers on a five-point scale may be subjective. For example there may be little or no difference between a person who “strongly agrees” and one who “agrees” with a statement. In fact, some people may never strongly agree or strongly disagree with any statements.

Respondents overall rating of care, on a scale from 0 to 10, is presented as an average score and also presented as a percentage positive score. Presenting it as an average score aligns with the reporting of this measure by other countries within the UK, while presenting it as a percentage positive statistic allows comparisons over time.

Sampling Error

All sample surveys are affected by sampling error. When comparisons have been made in the analysis of the survey results, the effects of sampling error have been taken into account by the use of confidence intervals and tests for statistical significance. Only differences that are statistically significant are reported as differences within the analysis and all significance testing is carried out at the 5% level.

More information on confidence intervals, significance testing and how they’re calculated can be found in Annex E.

Analysis of the Free-Text Comments

A thematic analysis of the free-text comments has been included in the National Report for the first time. This is in response to user demand for insights from the free-text comments to complement, support and provide context to the quantitative analysis of results.

The thematic analysis of the free-text comments is not a statistical analysis, but we have applied the Code of Practice for Statistics pillars of trustworthiness, quality and value to it. Below are details of how we have applied each of the pillars:

Trustworthiness: We have followed all the principles under the trustworthiness pillar of the Code.

Quality: We have used appropriate data, methods, and assured the quality of the analysis. However, the nature of the analysis means that we have not used statistical methods.

Value: The analysis is relevant to users as it has been produced in response to user feedback. It has been made equally available to all, is presented clearly to provide insights, and takes an innovative approach. We have made use of existing data, although we are unable to publish any of the underlying data for re-use because we need to ensure confidentiality.

Microsoft Excel was used to manage and analyse the data. Thematic analysis with an inductive coding approach was the method chosen to maintain the organic and rich nature of the data. Braun and Clarke’s (2006) 6-step method for thematic analysis was followed throughout to ensure a consistent approach to each survey question. Comments could be assigned multiple codes if the comment covered several aspects of the cancer care experience. These assigned codes were then collated into sub-themes and organised into key themes.

Quality Assurance of the National Report

A small group of Scottish Government analysts, Macmillan analysts and Scottish Government clinical advisors were sent a draft version of the National Report and the Technical Report for quality assurance.

Scottish Government analysts and Public Health Scotland analysts carried out quality checks of figures used in the report.