Scottish Crime and Justice Survey 2014/15: Sexual Victimisation & Stalking

Annex 2. Data strengths and limitations

Summary: data strengths and limitations

One of main strengths of crime and victimisation surveys such as the SCJS is that they provide a complementary measure of crime to police recorded crime statistics. Police recorded crime is a measure of crime that police come into contact with. However, it is well established that people may be unwilling to report crimes for a range of reasons, including a perceived lack of benefit, fear of reprisal, vulnerability, an inability to identify assailants or unwillingness to bring the victims conduct to the attention of the police. People are also less likely to report some types of crime than others (for example, people are less likely to report sexual than property crimes). Police practices can also influence recorded crime, for example, officers may not record all crimes reported by the public.

These factors are unlikely to affect SCJS data. For example, people's attitudes toward the police are unlikely to affect SCJS data, nor are the data affected by police recording practices. SCJS data also provide a measure of prevalence, that is, the risk of experiencing different types of crime in a given time period. By contrast, police recorded crime can only measure incidence or the number of crimes. In addition, the SCJS collects demographic information, providing richer insights into who is experiencing crime. Furthermore, follow-up questions about incidents allow the SCJS to capture respondent's attitudes to the criminal justice system, including reasons for not reporting crimes to the police.

Sampling and crime type limitations

The main limitations of the SCJS result from sampling, and the types of crimes surveyed. In terms of sampling, the survey is of adults aged 16 and over, living in private residential households only. As such, the survey excludes persons under the age of sixteen, the homeless, and populations living in residences such as care homes, halls of residences, hospitals, prisons or other communal accommodation.

Survey Error

As discussed in the report introduction, the SCJS gathers information from a sample rather than from the whole population and survey results are always estimates, not precise figures. This means that they are subject to a level of uncertainty. To estimate the extent of this uncertainty, 95% confidence intervals for the statistics are calculated to define bands within which the 'true' value of survey estimates are likely to lie ( i.e. that value which would be obtained if a census of the entire population was undertaken). These confidence intervals are particularly important when making comparisons of SCJS estimates over short timescales.

Non-quantifiable errors: recall and accuracy

SCJS estimates are also subject to a margin of non-quantifiable error. For example: there may be errors in the recall of participants as to when certain incidents took place; respondents may have claimed to have reported a crime to police when they had not, feeling that this was the socially acceptable response; some incidents could also be inaccurately recorded by interviewers, or miscoded by the wider survey team. Although a number of steps in the design and implementation of the survey are taken to reduce such errors, they can never be fully eliminated.

There may be errors in the recall of participants as to when certain incidents took place, or the number of incidents that took place. This is particularly relevant to collecting data on more sensitive topics, which may be cumulative and ongoing. For example, it may be difficult for respondents to recall the exact number of incidents. Also, respondents may not want to either remember or report some experiences.

For further details on the respective advantages and limitations of SCJS data see the introduction to the 2014/15 Main Findings Report and the Technical Report.

Survey design changes

The collection of survey data on crime and victimisation in Scotland has undergone several major changes in methodology. Changing crime survey methodology in Scotland has implications for making comparisons across survey designs. As previous surveys had smaller sample sizes, estimates from earlier surveys are subject to a higher degree of uncertainty, and this report therefore focuses on the period from 2008-09 onwards.

Self-report data advantages

One of main strengths of crime and victimisation surveys is that they provide a complementary measure of crime to police recorded crime statistics. The limitations of police recorded crime are well established. For example, people may be unwilling to report crimes for a range of reasons, including a perceived lack of benefit, fear of reprisal, vulnerability, an inability to identify assailants, or an unwillingness to bring the victims conduct to the attention of the police. Some types of crime are also less likely to be reported than others. For example, sexual crimes are less likely to be reported to the police than property crimes. Police recorded crime may also be influenced by police practices, for example, officers may not record all crimes that are reported to the police.

The SCJS self-complete section is unlikely to be influenced by attitudes towards the police and are unaffected by police recording procedures. It is designed to allow respondents to answer questions on more sensitive and personal topics privately. Self-report data can capture crimes and experiences of a sensitive nature that respondents may be unwilling to report to the police, or to disclose in a face-to-face survey situation. For example, evidence from the Crime Survey of England and Wales shows that respondents are more likely to report sensitive issues in the self-completion module than in face-to-face interviews (2014; 3). Section 7.7 of the Technical Report discusses the self-complete section of the questionnaire in more detail.

Unlike police recorded crime (which measures incidence or the number of crimes), the SCJS can also provide a measure of prevalence, that is, the risk of experiencing different types of crime within a certain time period. The survey also provides information on respondent's reasons for not reporting experiences/crimes to the police, and on their attitudes towards the criminal justice system more broadly.

Self-report data limitations

A number of factors may affect the self-report data (as well as other types of survey data). These include the wording of questions and the presence or skills of the interviewer. The presence of other people in the house may also influence results. Although the self-completion module allows respondents to answer in relative privacy, respondents may be unwilling to disclose personal or distressing details.

Another important limitation to consider is where self-completion interviews on sensitive topics are administered by the interviewer, at the request of respondents, who, for example, do not wish to use the laptop/tablet to complete the interview themselves. In 2014/15 87% of respondents completed the self-completion section: 71% entered their answers directly in to the tablet PC themselves and 16% asked the interviewer to administer the questionnaire for them. Of those where the interviewer administered the self-completion, in five per cent of cases, the respondent completed the section themselves after the first few questions being administered by the interviewer. Steps are taken by the SCJS trained interviewers to ensure that the number of self-complete interviews that are interviewer-administered are minimised and this is monitored closely by the SCJS team and our survey contractors.

Under-reporting and under-representation is also a concern of this survey. For example, it is likely that there will be an under-representation of some groups, e.g. those who take drugs. In part, this will be due to the fact that some people who use drugs may live in accommodation not covered by a survey of private households (such as the SCJS) including, for example, hostels, prisons and student halls of residence. The survey is likely to under-represent those with the most problematic or chaotic drug use, some of whom may live in accommodation previously described and some of whom may live in private households covered by the survey, yet they may be rarely be at home or be unable to take part in an interview due to the chaotic nature of their lives.

Despite using Computer Assisted Self-completion Interviewing ( CASI) for this module, it is likely there will be some underreporting of illicit drug use, partner abuse and sexual victimisation and stalking among survey respondents. For example, each of these are sensitive topics, and illicit drug use is an illegal activity and as such some individuals may have felt uncomfortable reporting that they have taken illicit drugs, despite reassurances about confidentiality and anonymity.

Questions cover past use over varying periods (ever, in the last year and in the last month) and it is possible that some respondents may simply forget experiences, particularly if they last took it a long time ago.