Scottish Patient Experience Survey of GP and Local NHS Services 2011/12 Volume 3: Variation in the Experiences of Primary Care Patients

17 Annex A: Methodology

17.1 Detailed information about the survey can be found in the technical report that was published in 2012: http://www.scotland.gov.uk/Publications/2012/05/1477.

Analysis

17.2 Questions in the survey are typically on a five point scale. In order to simplify the analysis and results, answers have been combined into either positive or 'non-positive'. For example for the answer scale ranging from very poor to excellent, good and excellent have been classed as positive responses while very poor, poor and fair have been classed as non-positive responses. This is similar to the concept of percentage positive that was used in the local and national reporting of the survey results.

17.3 A multivariate multilevel model has been constructed for all questions relating to a patient's experience of GP and out-of-hours services. Multilevel modelling allows the analysis to take into account the hierarchal structure of the survey data. For questions about GP practices, the analysis considers that people are nested in GP practices that are nested within CHPs. If no CHP effect was found then CHP was removed from the model. For questions about out-of-hours services the models consider that patients are nested within NHS Boards. The decisions on how to model the questions were based on analysis and also generally reflect the way that services are delivered.

17.4 The SAS⁶ procedure proc glimmix which fits Generalised Linear Mixed Models (GLMMs) was used for the analysis.

17.5 The statistical modelling took into account the effects of different GP practices (including the effects of practice size and the percentage of patients living in a deprived area), Community Health Partnerships (CHPs) and NHS Boards. For experiences of out-of-hours services we also considered the effects of different services. The following personal factors were considered:

• age
• gender
• sexual orientation
• ethnicity
• religion
• deprivation (SIMD quintile)
• urban/rural classification
• work status
• being a carer
• health status
• health problem or disability that limits day-to-day activities
• a variety of disabilities (deafness, blindness, physical disability, learning disability, mental health condition, chronic pain lasting at least 3 months, and another long term condition)
• translation, interpreting and communication support needs
• how often people contacted their practice in the last year

17.6 We initially considered the effect of living in the most deprived 15% of datazones, but this didn't have an effect when SIMD quintile was included in the model and so it was removed from the analysis.

17.7 Models were fitted for all questions and tests used to determine if a particular factor had a statistically significant effect at the 5% level on the likelihood of a patient reporting a positive experience. Factors that did not have a statistically significant effect were removed from the model. An effect is statistically significant if it is so large that an effect of that size (or greater) is unlikely to have occurred purely by chance. Conventionally, significance is tested at the 5% level, which means that an effect is considered significant if it would only have occurred once in 20 different models by chance.

17.8 We performed various checks to ensure that the model assumptions were met. These checks included:

• Plotting predicted versus actual values for random effects to check that there was no strong evidence of outliers or lack of symmetry;
• Checking that the residuals for the random effects were approximately normally distributed.

Reporting of results

17.9 The odds ratio is used to show whether a particular group of patients is more or less likely to give a positive response when compared with a reference group. If an odds ratio (and its lower confidence limit) is above 1, then the group has reported better experience than the reference group, whilst if the odds ratio (and its upper confidence limit) are below 1, the reported experience of the group is worse. The reference group has usually been selected to be the largest group.

17.10 Even if there were no systematic difference between groups, we would not expect the responses to be exactly the same in our survey data. Random variations, or responses from a small number of respondents, are likely to make the value vary slightly from 1. The confidence interval allows us to judge when the difference from 1 is large enough to be interpreted as a difference attributable to the group, rather than random variation. If the confidence interval does not include 1 we say that the difference is statistically significant.

17.11 It should be noted that the odds ratios are not a direct measure of how positive the responses were from a particular group, but a measure of comparison showing how likely people within a given group were to give a positive response compared with those in the reference group for that factor. For example, for gender the results for males are compared to the results for females. This means that a result of 1.1 for males means that males are slightly more likely than females to report a positive experience.

Limitations of the analysis

17.12 The statistical techniques that have been used in the analysis presented here do not imply causality. We cannot tell from the associations identified whether the increased likelihood of reporting a positive experience are the direct result of the factors included in the model or whether some other factor not included in the modelling process is having a significant impact.

17.13 The modelling assumes that the relationships between age and experience, and the percentage of the practice list living in deprived areas and experience, are linear to simplify the reporting of results. These appear to be fairly reasonable assumptions.

17.14 Responses with missing values for one or all of the demographic questions (except for sexual orientation) are excluded from the analysis. This is a limitation of the technique and a possible source of bias, especially when several variables with a reasonably high number of non-responses are included in the model. Missing values for sexual orientation were included because the level of non-response was particularly high. Over 6 per cent of respondents did not answer this question which is more than double the total number that answered gay or lesbian, other or bisexual.

17.15 The model did not consider interaction effects to avoid over-complicating both the analysis and the interpretation of results. We did investigate some areas to see if there were any interaction effects, but did not detect any.

17.16 Patients report from their own perspective and judge against their own expectations which means that there are factors other than their own experience that influence what they report. Efforts are made to minimise the effect of subjectivity by asking questions that focus on specific events rather than overall satisfaction, and by testing questions with people from a range of backgrounds. The variations in experience reported here could reflect real inter-group differences in the quality of services received, or inter-group differences in subjective factors such as expectations, perceptions or the way questions are answered, or some combination of these factors.

17.17 The analysis covers a large number of questions and, when looking at so many results, it can be expected that some differences between groups will be found even if there are none. Therefore it is sensible to look at overall themes of results rather than focusing on particular questions which might lead to spurious conclusions. It is also better to focus on significant differences where the odds ratios are further from 1 as they will be of more practical significance. This is the approach that the report generally aims for.