Connecting Scotland phase 3 - digital support for employability: evaluation

Evaluation Approach

For each of the first the phases of Connecting Scotland, the same methods were used to evaluate the impact of the programme for users. These methods will be briefly outlined here. Greater detail on the methodology can be found in either of the Phase 1 or Phase 2 evaluation reports. The methods applied were designed to find out about users' experiences of the job market and of using digital technology, including, in both instances, barriers or difficulties encountered. The central data collection methods of the research have been carried out at various points throughout users' engagement with the programme (over the first year, or so) so that changes and impacts can be measured over time.

Data from Applications

Organisations submitting applications described the situations and needs of people with whom they worked as well as outlining the expected benefits that involvement in Connecting Scotland would have. Applications are not a neutral data source as organisations are representing the interests of those on whose behalf they are applying. Nevertheless, when taken together, information from applications provides a broad understanding of the issues faced by people in need of employability support and who are digitally excluded, as well as highlighting some of the groups most affected.

There were 525 applications; a sample of 226 was analysed.

Surveys

Connecting Scotland administers 2 main online surveys for users to complete. The first – the 'welcome survey' - is available to complete shortly after users have received their devices and seeks to understand people's experience of going online and using digital devices, as well as what people hope to achieve through involvement in the programme. For phase 3, we included specific questions on people's job-seeking status and barriers to employment with which they identified.

Users are invited to complete a second survey – the 'experience and impact survey' - when they have had their Connecting Scotland devices for between 9 and 12 months. The main purpose of this survey is to capture impacts and/or progress towards postive outcomes that have resulted from involvement with Connecting Scotland. Again, for phase 3, a number of questions were included that asked about people's employment prospects, and the extent to which equipment and support from Connecting Scotland had been helpful.

Each survey drew its sample from a cohort of Connecting Scotland users in phase 3. It was not a panel study, meaning that the same respondents will not necessarily have responded to both surveys. It is probable that several users did respond to both surveys, but participation is voluntary and respondents are anonymous.

The profile of respondents to both the welcome survey and the impact survey was similar. Respondents to the surveys were generally younger, with 60% being aged 16-44. The impact survey shows a 60%-40% gender split in favour of women. Around a third of respondents, to both surveys, were living alone. The biggest demographic difference between the surveys is that there were fewer households with children represented in the impact survey. The impact survey included a question relating to personal circumstances that was not included in the welcome survey. This showed that a quarter of respondents were single parents and a quarter were either disabled, or lived with someone else who was disabled. Categories were not mutually exclusive.

Qualitative Interviews

In between the delivery of the welcome and impact surveys, a sample of users took part in semi-structured qualitative interviews to provide more in-depth accounts of their experiences of job-seeking and Connecting Scotland support. Interviewees were recruited through contact with applicant organisations to build a sample of different users to include people of varying age, sex and job-seeking experience. The sample included disabled people, older (60+) job-seekers, people with experience of domestic abuse and people with dependent children.

The purpose of the interviews was to gain deeper insights into the aspects of support that people found more, or less, helpful as well as to understand more about how people had been using their devices, Interviews also present the opportunity to surface findings which researchers may not have anticipated.

Interviews were arranged at a time convenient to the participant and conducted over the phone, with a third person on the call to take detailed notes of the conversation.

Number of interviews = 18

Limitations of Approach

While all research participants are people who have received Connecting Scotland support, each stage of data collection is based upon a discrete sample of voluntary respondents. It is possible for someone to have completed both surveys and participated in an interview, however, the overall approach to research is not engineered to monitor change for individual users. Furthermore, because survey respondents remain anonymous, there is no way of identifying individuals from one data collection point to the next.

This is not a disadvantage, per se; it simply means that we are assessing change and impact at an aggregate level and using the findings to understand commonalities in experiences and outcomes across the cohort.

Throughout the research for each of the first three phases, we have consistently observed relatively low response rates to surveys, despite pursuing a number of strategies to boost participation. This does not invalidate the survey data that we have, though we could have greater confidence in the wider validity of the results with a more robust sample size.

The sample of qualitative interviewees is, by necessity, small and bound by time and resources. We have succeeded in achieving a sample that includes a range of demographics and characteristics. However, due to the approach to participant recruitment, interviewees were ultimately reached via only a few organisations. This might mean that there is a greater degree of similarity in people's experience of support than there might be in the cohort more widely.