Scottish Cancer Patient Experience Survey 2015-16: technical report

Analysis and reporting

Analysis

The survey data collected and coded by the contractor were securely transferred to ISD via secure FTP and analysed using the statistical software package SPSS.

The analysis produced by ISD was transferred to the Scottish Government and Macmillan Cancer Support for inclusion in the national report.

Weighting the data

No weighting was applied to the survey data in any of the reporting formats. Unlike other Care Experience Surveys, which use stratified random sampling, this survey was a census of eligible patients.

There are ways that the data could be weighted using the differences in the characteristics of those sampled. However, historically patient experience survey results have only been weighted to allow for the different sizes of the strata alone.

Equivalent Cancer Patient Experience Surveys in other parts of the UK do not apply weighting.

Significance tests

Questions which produced 'percent positive' results were subject to significance testing to investigate differences between groups of respondents.

Differences between local level results and Scotland

For the purpose of the local level reports, significance testing was conducted to see if the results for each NHS Board, Cancer Network and Cancer Centre, were significantly different from the comparible figure for Scotland as a whole.

The method used to do this was the established approach used across the Care Experience Surveys:

Care Experience Surveys significance testing

This approach uses the normal approximation to the binomial theorem and is equivalent to constructing a 95% confidence interval for the difference between two results. If this confidence interval does not contain 0 then the result is statistically significant at the 5% level.

One point to note is that, given the census approach taken in survey, there is no design effect to take account of in the calculations.

Differences between most and least deprived areas

For the purposes of the national report, differences were tested between respondents from the most and least deprived SIMD quintile. This also followed the Care Experience Surveys standard approach (replacing Scotland and the local area in the calculation with the most and least deprived SIMD quintiles).

Differences between tumour groups

For the purposes of the national report, differences were tested between patients belonging to different tumour groups. A Pearson chi-square test was used to detect any statistically significant differences in the distribution of positive responses across all tumour groups (α = 0.05). This excluded patients where the tumour group was unknown, or their tumour type was included in the 'other' category.

Where statistically significant differences were found across the tumour groups, significance testing was conducted to see if each tumour group was significantly different from the figure for all tumour groups (this is simply the Scotland figure). This was done using the standard approach referred above.

Reporting thresholds

Reports for NHS Boards, Regional Cancer Networks and Cancer Centres were only produced if there were 50 or more respondents. If a particular question had less than 20 responses, the results for that question were suppressed.

Allocation of survey results to the correct NHS Board, Regional Cancer Network or Cancer Centre

As set out in the chapter on sampling, each patient in the survey was selected to take part on the basis of a specific hospital visit. The name of the relevant hospital was printed on the covering letter that the patient received with the survey.

The starting basis for the allocation of patient responses was that patients were allocated to the i) hospital, ii) NHS board, and iii) regional cancer network results all based on this specific hospital.

For example, someone that was selected for the survey on the basis of an inpatient stay at Ninewells hospital in Dundee would appear in the results for: Ninewells hospital; NHS Tayside; and the North of Scotland Cancer Network.

There were two exceptions to this approach: i) for questions that did not relate to hospital care and ii) for questions where the respondent indicated that a particular aspect of their care did not take place at the hospital on the covering letter.

Questions not relating to hospital care

Cancer patients do not necessarily live in the same NHS Board that they receive hospital care. This is relevant as some of questions in the survey relate to care that an individual would have likely received at home or the area where they live.

In order to account for this:

• Questions 1, 2 and 3 were assigned to the NHS Board / Cancer Network that the patient lived in at the time of diagnosis (these questions related to patients approaching their GP at the point that cancer was suspected).
• Questions 55, 56 and 58 were assigned to the NHS Board / Cancer Network that the patient lived in at the time of their hospital treatment (these questions related to community-based health and social services and GP care during cancer treatment)

Care that took place at a different hospital

For each aspect of hospital care covered in the survey, the questionnaire offered the respondents the chance to confirm whether their care took place in the hospital on the covering letter.

In instances where respondents indicated that this was indeed the case, the results were assigned to hospitals, NHS Boards and Regional Cancer Networks on the basis of the hospital on the covering letter.

Where respondents indicated that they did not receive that aspect of treatment at the hospital on the covering letter, the results were not assigned to any hospital, NHS Board or Cancer Networks. It was not clear where the respondent had received this care, so responses for that aspect of hospital care were included only in the national level results.

Sources of bias

Non-response bias

The greatest source of bias in the survey estimates is due to non-response. Non-response bias will affect the estimates if the experiences of respondents differ from those of non-respondents.

Other sources of bias

There are potential differences in the expectations and perceptions of patients with different characteristics. Patients with higher expectations will likely give less positive responses. Similarly patients will perceive things in different ways which may make them more or less likely to respond positively. When making comparisons between geographical areas within Scotland it should be remembered that these may be affected by differences in patient characteristics across the geography's cancer population.

In interpreting the results, consideration should also be given to differences in the nature of NHS Boards and Regional Cancer Networks in Scotland. Across NHS Boards, for example, there is a large variation in geographic coverage, population sizes and hospital sites as well as hospital type which should be borne in mind when reviewing survey findings.