New Scots Refugee Integration Strategy 2018 to 2022: evaluation

2. Methodology

2.1. Research aims and objectives

The overall aim of the evaluation was to gain a fuller understanding of what works in relation to refugee integration in Scotland, focusing on the implementation of the New Scots Refugee Integration Strategy as the focal point for coordinated action. The objectives of the evaluation were to assess:

• The processes through which the New Scots Refugee Integration Strategy 2018-2022 has been implemented
• The progress made towards achieving its intended outcomes
• What learning could be used to inform future policy and practice.

Linked to these aims were a number of research questions which are outlined in Table 2.1 below.

Table 2.1: Evaluation research questions

Process

Research questions:

1. What is working well/not well in how the New Scots Refugee Integration Strategy is being implemented? Does this vary between groups/areas?

2. What contextual factors influence implementation?

3. What are the factors that block successful implementation, and what factors enhance it?

4. Has the intended reach been achieved?

Outcomes

Research questions:

5. What can we learn about the progress that has been made towards achieving the four New Scots Refugee Integration Strategy outcomes? How does this vary across different groups/areas?

6. To what extent do the observed results and the perspectives of service users, staff and stakeholders suggest the New Scots Refugee Integration Strategy contributed to outcomes (whether improved, worsened or unchanged)?

7. How sustainable are these changes?

8. What unintended changes (positive or negative) has the New Scots Refugee Integration Strategy made?

Learning

Research questions:

9. What lessons learned and recommendations for future practice can be drawn from the experiences of the New Scots Refugee Integration Strategy to date?

10. What elements of the New Scots Refugee Integration Strategy are working well, and what elements are working less well?

11. What aspects of the New Scots Refugee Integration Strategy could be adapted or developed based on the insights from this work?

12. To what extent has the absence of a structured funding arrangement impacted the New Scots Refugee Integration Strategy and implementation, and how would the continuation of this policy impact future practice and implementation?

13. How have the outcomes of the New Scots Refugee Integration Strategy benefited from the partnership approach to policy and implementation?

2.2. Research design

To address the research aims and objectives, ScotCen and Matter of Focus conducted a collaborative mixed method study. The adoption of a mixed method design enabled data triangulation to strengthen the overall evaluation findings and provided both breadth and depth of scope.

The study consisted of five key strands which are outlined below, along with the lead organisation for each component:

1. Two Theory of Change workshops with partners involved in developing and implementing the New Scots Refugee Integration Strategy 2018-2022 (Matter of Focus)

2. Interviews with 30 stakeholders involved in developing and implementing the New Scots Refugee Integration Strategy 2018-2022 (ScotCen)

3. An online survey of stakeholder organisations (ScotCen)

4. Qualitative interviews with 21 refugees and people seeking asylum (ScotCen)

5. Self-evaluation workshops with 12 AMIF-funded projects (Matter of Focus)

ScotCen were responsible for conducting the survey; the interviews with individuals working in the field of integration; and the interviews with refugees and people seeking asylum. Matter of Focus led the Theory of Change workshops with New Scots partners and the self-evaluation workshops with funded projects. These utilised specialist evaluation software (OutNav) developed by Matter of Focus to help organisations evaluate their progress against specific outcomes.

Each of the elements of the overall research design is explored in more detail in the subsequent sections.

2.2.1. Theory of Change workshops

At the beginning of the project, two online workshops were held with New Scots partners involved in developing and implementing the New Scots Refugee Integration Strategy 2018-2022 . These workshops were led by Matter of Focus and chaired by the UNESCO Chair for Refugee Integration, Professor Alison Phipps of the University of Glasgow. The workshops aimed to work in a participatory way with members from the Scottish Government and the New Scots Refugee Integration Strategy and core groups to refresh the Theory of Change for the Strategy. This included an assessment by the group of the context for delivery using a tool developed by the Scottish Government, the Individual, Social, Material model (ISM) ^[25]. This model takes an approach to mapping context designed for policy makers and practitioners whose work ultimately aims at engaging people and influencing their behaviours in order to deliver improved outcomes.

The first workshop was held on 19 January 2022. It included participants from the Scottish Government, COSLA, Scottish Refugee Council, and other key organisations involved in the development and implementation of the Strategy. The second workshop was held on 7 February 2022.

These workshops helped develop a new Theory of Change for the Strategy (see Appendices C and D). This helped to explicitly set out how this group saw the actions which had been taken to implement the Strategy have contributed to its outcomes of the Strategy as a whole. The Theory of Change work was requested by the New Scots Core Group to help shape initial thinking about the next phase of the New Scots Refugee Integration Strategy. The resulting Theory of Change was shared with the funded groups so that they could explicitly link their work to the outcomes as expressed in this Theory of Change. The data gathered during the workshops also informed the development of the stakeholder interview topic guide in terms of both contextual factors and the Theory of Change itself.

2.2.2. Stakeholder interviews

In order to explore the development, implementation and impact of the New Scots Refugee Integration Strategy, ScotCen conducted interviews with a range of stakeholders working in the field of integration (n=30). The term stakeholder is used throughout the report to refer to the individuals from academia, government, local authorities, third sector and community groups who responded to the survey or participated in an interview.

Recruitment and sampling

The Scottish Government drafted a list of organisations and individuals involved in the development and implementation of the current Strategy and its previous iteration. Using contact details supplied by the Scottish Government, ScotCen sent email invitations to a selection of contacts inviting them to participate in an interview. The Scottish Government was not informed who was invited to, or subsequently participated in, an interview.

Having initially focused on organisations and individuals involved in Strategy development, stakeholder recruitment was expanded in June 2022 to include a selection of representatives from other organisations working directly with refugees and people seeking asylum, including some projects that had received AMIF funding. This allowed the research team to explore their views on the Strategy and their experiences as organisations working directly with refugees and people seeking asylum in Scotland.

Conducting the research

The stakeholder interview topic guide was developed in February/March 2022 in consultation with the Scottish Government and members of the New Scots Strategy Core Group. Two topic guides were developed for the in-depth interviews with key stakeholder organisations: one for core and thematic group members and one for non-group members. The content of the topic guides consisted of a wide range of questions relating to the New Scots Refugee Integration Strategy (2018-2022), including:

• Understanding and awareness of the New Scots Refugee Integration Strategy (2018-2022)
• Involvement in the development of the latest version of the Strategy
  • Experiences of collaboration
  • Aspects that have worked well/been more challenging
• Involvement and experiences within the core and/or theme group
  • Key aims and activities
  • Experiences of collaboration
  • Aspects that have worked well/been more challenging
  • Contribution to the group
• Own organisation’s work and influence on refugee integration in Scotland
  • Possible impact of the Strategy on organisation’s work
  • Alignment of the Strategy’s values with organisation
• Knowledge and awareness of the Strategy amongst organisations and individuals working in the field
• Implementation of the Strategy
  • Key factors enabling and hindering the Strategy’s implementation
  • Suggested improvements for the next New Scots Refugee Integration Strategy
• Outcomes of the Strategy
  • Progress achieved towards these outcomes
  • Key barriers and challenges to these outcomes
  • Potential impact of the Strategy on refugee integration in Scotland
  • Impact of reserved policy making on implementation of the Strategy
• Informing the next New Scots Refugee Integration Strategy
  • Lessons learned
  • Potential improvements to the Strategy
  • Gaps in the current Strategy.

All interviews were conducted by a member of the ScotCen research team, either by telephone or video call using Microsoft Teams. Interviews took place between 1^st April and 1^st August 2022 and took between 48 minutes and an hour and 52 minutes to complete.

Participant demographics

In total, 27 individual and paired interviews were conducted with 30 stakeholders. Representatives of core and theme groups, including New Scots partner agency representatives, were interviewed. In addition, interviews were conducted with representatives from both AMIF and non-AMIF-funded organisations that work directly with refugees and people seeking asylum.

Interviewees were based throughout Scotland, although a larger proportion were from Glasgow, reflecting the fact that many projects and organisations working with refugees and people seeking asylum are based in Glasgow.

The types of organisations represented included: Scottish Government; UK Government and Scottish Government bodies; statutory services; local authorities; education providers; members of Health and Social Care Partnerships; faith organisations; housing providers, and a variety of third sector organisations and arts organisations.

Staff interviewed included a mixture of those in senior management positions (including those in partner organisations) responsible for high level Strategy and those who delivered support to refugees and people seeking asylum on the ground.While participants from a wide range of organisations participated in an interview, despite several invitations issued, only one person from a local authority agreed to be interviewed. It is important to note that the research period coincided with the outbreak of the Ukraine conflict and the subsequent implementation of the Ukraine resettlement scheme, which placed significant unforeseen new requirements on local authorities, and which may have reduced the ability of local authority representatives to engage with this research. Nevertheless, a separate research project with local authorities was commissioned by the partnership and undertaken in parallel with this research, and this will provide additional insights into the experiences of local authorities.

Analysis

All transcripts were imported into and coded using NVivo 12, a software package which aids qualitative data analysis. Analysis involved a number of stages. Firstly, the key topics and issues which emerged from the research objectives and the data were identified through familiarisation with transcripts by at least two members of the ScotCen research team. A draft analytical framework was developed and agreed by the research team, set up in NVivo and piloted by two researchers on a few transcripts. The analytical framework was then refined after discussion and agreement within the wider project team. Once the analytical framework was finalised, each transcript was coded so that all the data relating to a particular theme could be viewed together.

Through reviewing the coded data, the full range of views and attitudes described by respondents were systematically mapped, and the accounts of different respondents, or groups of respondents, compared and contrasted.

ScotCen’s approach to qualitative analysis is designed to minimise bias in the interpretation of data. Multiple researchers across the team were involved in the analytical process (from reviewing transcripts, to framework development and coding), ensuring inter-rater reliability and validity to the conclusions. Regular analytical meetings were held by the team in order that any suggested modifications to the analytical framework were agreed by all of the researchers. All analysis and interpretation was checked by senior members of the team and a designated Quality Director.

2.2.3. Stakeholder organisations survey

Interviews with stakeholders were complemented by an online survey of a wider group of representatives of stakeholder organisations. The survey explored the views of those involved in activities which support the integration of refugees and people seeking asylum.

Recruitment and sampling

The Scottish Government, COSLA and SRC sent email invitiations, including a link to the survey, to a list of key contacts on behalf of ScotCen. The aim was to distribute the survey to as wide a range of organisations working with refugees and people seeking asylum in Scotland as possible, drawing on SRC’s database of organisations working in the sector, and ensuring distribution to key contacts in every Scottish local authority. Initially 718 invitations were sent to contacts of SRC (n=177), COSLA (n=250), GRAMNET and UNESCO RILA (n=80) and Scottish Government (n=211). All those invited to participate were asked to share with relevant contacts both within and outwith their organisations. Two reminders were sent by partner organisations to the same list of contacts. As the invitation email may have been forwarded on by organisations and individuals to other colleagues working in the area, it is not possible to know how many people or organisations received the survey invitation, or what the response rate might have been.

Conducting the research

A draft of the survey questions was agreed with the Scottish Government and New Scots partners (Appendix B). The online survey was developed using BUILD software. The online survey was live between 6^th June 2022 and 15^th August 2022. Responses to the online survey were only accessible by ScotCen researchers.

Analysis

The surveys were downloaded and transferred to SPSS, a quantitative data analysis package. The data were analysed using descriptive analysis, with frequencies and cross-tabulations conducted. As it was not a representative sample, it was not possible to conduct testing of statistical significance. Responses to the open-ended questions were analysed thematically. A coding framework outlining the key themes within the responses was designed through immersion in the data. Each response was then coded with the key themes written up under a series of headings.

The analysis presented in this report is based on 250 participants: 176 respondents who completed all questions and 74 respondents who partially responded (this included all respondents who answered beyond Question 8 which asked them about their awareness of the Strategy). An additional 79 respondents partially responded but did not answer beyond Question 8 so were excluded from the analysis. A further 232 respondents clicked on the survey but did not answer any questions.

Participant demographics

Table 2.2 shows the range of organisations that were represented in the survey. About one-third of respondents said they were from third sector organisations, with a similar proportion saying that they represented a local authority (see Table 2.2). Sixteen per cent chose community organisation, while just under one in ten said their organisation was an education provider.

Table 2.2: Survey respondents by organisational type

	N	%
Third sector organisation	86	34.4
Local authority	83	33.2
Community organisation	40	16.0
Education provider (school/college/university)	24	9.6
Faith and belief organisation/group	11	4.4
Scottish Government/agency	6	2.4
Answering in individual capacity/not associated with an organisation	6	2.4
NHS/Healthcare provider	3	1.2
Private sector organisation	2	0.8
UK Government/agency	1	0.4
Other (please write in)	16	6.4

NB: Base N= 250, percentages do not add up to 100 as more than one option could be chosen.

Written responses from the 16 respondents who selected ‘other’ included: charities, community interest companies, Health and Social Care Partnerships, museums, third sector partnerships and voluntary organisations.

The number of paid staff in respondents’ organisations varied (see Table 2.3). This was used as a proxy for organisational size to assess the diversity of the sample. The sample was predominantly made up of organisations with paid members of staff (90%). Half the sample reported that their organisation had more than 50 paid staff (42%), reflecting the number of responses from large organisations such as local authorities, or between 21 and 50 staff (8%). Around 10% of the individuals who responded were based in organisations run entirely by volunteers.

Table 2.3: Number of paid staff in respondents’ organisations

	N	%
0 – my organisation is run entirely by volunteers	26	10.4
1-5 paid staff	30	12.0
6-10 paid staff	25	10.0
11-20 paid staff	33	13.2
21-50 paid staff	19	7.6
More than 50 paid staff	106	42.4
N/A	11*	4.4
Total	250	100.0

*includes local authorities; education providers, community/third sector organisations, and respondents answering in an individual capacity

More than a quarter (26%) of respondents said their organisation worked across the whole of Scotland. There were survey responses from organisations working in each of Scotland’s 32 local authorities, though a larger proportion worked in Glasgow City than in other local authorities (23%), reflecting the history of refugee and asylum seeker resettlement and dispersal in Scotland. Nineteen responses (8%) represented organisations covering the whole of the UK, and 12 (5%) covered international areas.

Table 2.4: Areas covered by survey respondent organisations

	Frequency	%
Whole of Scotland	65	26
Glasgow City	58	23.2
UK wide	19	7.6
Perth and Kinross	18	7.2
City of Edinburgh	17	6.8
International/Outside UK	12	4.8
Inverclyde	13	5.2
Dundee City	12	4.8
Renfrewshire	12	4.8
Aberdeen City	11	4.4
North Lanarkshire	11	4.4
Stirling	10	4
Clackmannanshire	10	4
East Renfrewshire	10	4
Falkirk	10	4
North Ayrshire	9	3.6
South Ayrshire	9	3.6
West Dunbartonshire	10	4
Aberdeenshire	9	3.6
Angus	8	3.2
East Dunbartonshire	9	3.6
Fife	9	3.6
Midlothian	8	3.2
Argyll and Bute	7	2.8
East Lothian	7	2.8
South Lanarkshire	7	2.8
Dumfries and Galloway	6	2.4
East Ayrshire	5	2
Highland	6	2.4
Scottish Borders	5	2
Moray	5	2
Comhairle nan Eilean Siar	<5	<2
West Lothian	<5	<2
Orkney Islands	<5	<2
Shetland Islands	<5	<2
N/A	2	0.8

NB: Base N= 250, percentages do not add up to 100 as more than one option could be chosen.

Almost half of respondents (48%) said their organisation worked mainly in urban areas, while 41% said they undertook significant work in both rural and urban areas. Only 16 respondents said that they worked mainly in rural areas.

Table 2.5 shows that the majority of respondents said they worked with refugees (83%) and people seeking asylum (71%). Between 40% and 50% of respondents said their organisation worked with refugee-receiving communities; migrants other than refugees and people seeking asylum; the general public; unaccompanied asylum-seeking children; and others from disadvantaged backgrounds.

Table 2.5: Groups that survey respondent organisations work with

	N	%
Refugees	208	83.2
Asylum seekers	177	70.8
Refugee-receiving communities	124	49.6
Migrants other than refugees and asylum seekers	124	49.6
The general public	123	49.2
Unaccompanied asylum-seeking children	112	44.8
Others from disadvantaged backgrounds	101	40.4
Others based on their protected characteristics	66	26.4
People from a particular ethnic group	48	19.2
People from a particular faith group	37	14.8
N/A	4*	1.6

NB: Base N= 250, percentages do not add up to 100 as more than one option could be chosen.

*Mainly those who responded in an individual capacity/not associated with an organisation.

The activities that respondents’ organisations were involved in relating to the integration of refugees and people seeking asylum varied (see Table 2.6). The most-frequently cited activities were community integration and social connections (79%), health and wellbeing (72%) and language support (68%). The least-frequently selected activities still encompassed over one-third of the organisations’ work, including housing provision and support (43%), welfare benefits and rights (42%) and legal rights and citizenship (34%). The majority of organisations worked across a range of thematic areas with 87% reporting they were involved in more than one type of activity related to integration. More than a third (37%) selected between two and five thematic areas; 31% selected between six and nine; and 18% selected all ten options.

Table 2.6: Activities related to integration that survey respondent organisations are involved in

	N	%
Community integration and social connections	184	79.3
Health and wellbeing	167	72.0
Language support	159	68.5
Education	144	62.1
Employability/ entrepreneurship	126	54.3
Digital inclusion	124	53.4
Arts, culture and sport	111	47.8
Housing provision and support	99	42.7
Welfare benefits and rights	98	42.2
Legal rights and citizenship	79	34.1
N/A	3	1.3

NB: Base N= 250, percentages do not add up to 100 as more than one option could be chosen.

While individuals from a wide range of organisations participated in the survey, it is important to note that those who responded to the survey were self-selecting and may not represent the views of everyone working in the sector.

2.2.4. Refugees and asylum seeker interviews

To ensure that the evaluation would reflect the views and lived experiences of refugees and people seeking asylum living in Scotland, interviews were conducted with 21 individuals from a range of different backgrounds now living in Scotland. We considered a range of methodological approaches, including focus groups, for this research with refugees and people seeking asylum. However, given the potentially sensitive nature of some of the discussions, it was felt the most appropriate method would be individual interviews to allow interviewees to feel more comfortable in discussing their experiences and to explore these in some depth.

Recruitment and sampling

To ensure that a breadth of experience was represented in the research, ScotCen sought to recruit a diverse range of refugees and people seeking asylum in terms of age, gender, nationality, geographic location, programme (whether they had sought asylum or had arrived through a resettlement scheme) and length of residence in Scotland.

Recruitment was facilitated through support organisations for refugees and people seeking asylum across Scotland. This approach was chosen to facilitate engagement from refugees and people seeking asylum living in different geographic areas and to ensure participants were receiving the invitation via a trusted organisation. Organisations which shared information about the research were reassured that the focus of the research was the individuals’ experiences since arriving in Scotland and any support received, not specifically about the support received from their organisation. Organisations were encouraged to share information about the research widely amongst their membership and networks.

Information regarding the research (information sheet, privacy notice and invitation email) was produced by ScotCen and shared by organisations with refugees and people seeking asylum in contact with these organisations. These materials were translated into Ukrainian, Arabic, Pashto, and Dari, to allow for greater inclusivity during recruitment. Organisations requested materials to be translated into languages that were relevant to the individuals they were in contact with.

The invitation to participate in the research included contact details for the ScotCen research team. This enabled those wishing to participate in the research to express their interest with the research team directly. Alternatively, individuals could also express their interest to the support organisations, who would then organise an interview on their behalf. Once an individual expressed interest, a member of the research team made contact via email or telephone to complete a small number of screening questions. Screening questions asked about the individual’s age, gender, nationality, and length of residence in Scotland. Participants were also asked about whether a translator would be needed to fully participate in an interview. Participants were selected for interviews based on the sampling criteria agreed at project inception and invited to choose a date and time for their interview. Participants were given the option of a face-to-face, telephone or video interview.

Conducting the research

The interview topic guides were developed in August 2022 in consultation with the Scottish Government and members of the Strategy Core Group. The interviews explored individuals’:

• Experiences of living in Scotland, from arrival to present day, exploring any changes that have occurred and the impact of these changes;
• Experiences of accessing support, from when they had first arrived in Scotland to the present day, including what the support was, who offered it, and the impact of the support;
• Views on improvements to support for refugees and people seeking asylum in Scotland;
• View on improvements to make New Scots feel more welcome in Scotland.

All interviews were conducted either in person, by telephone or video call using MS Teams or Zoom at times and dates which met with the preferences of participants. Interviews took place bewtween 13^th September and 13^th October 2022 and took between 25 minutes and an hour and 20 minutes to complete. Eight participants were provided with an interpreter to aid participation in the interview.

With the consent of participants, all interviews were audio recorded and transcribed for ease of analysis. Verbal consent was recorded at the start of each interview. After completing an interview, participants received a £30 Love2Shop e-voucher as a one-off thank you gift for their time and for sharing their experiences.

Participant demographics

In total, 21 refugees and people seeking asylum were interviewed. Sixteen refugees and people seeking asylum self-described as female and five as male. Participants were aged between 22 and 72 years. Participants had travelled to the UK from seven different countries, and included participants with Syrian, Ukrainian, Algerian, South African, Ugandan, Nigerian, and Pakistan nationality. Over one half (n=12) of participants were Syrian.

Twelve participants were refugees and nine were people seeking asylum. Their length of residence in Scotland ranged from 2 months to 49 years. Participants lived in both rural (seven participants) and urban areas (14 participants) across six local authorities in Scotland. Four of the 12 refugees were in paid employment with the remaining either unemployed, volunteering, retired or in full-time or part-time education. Six of the nine people seeking asylum were volunteering at the time of interview.

While the research engaged with a diverse range of participants, it is not representative of all refugees and people seeking asylum. All participants self-selected to participate in the research. Furthermore, as participants were informed about the research via third party organisations, the research did not reach refugees and people seeking asylum who were not in receipt of any engagement or support from such organisations. The purpose of qualitative research is to present a broad range of views and perspectives rather than to be fully representative of a particular group. It was not possible within the scope of this research to speak to refugees and people seeking asylum in all parts of Scotland, or from all refugee communities.

Analysis

Analysis was undertaken in the same way as for the stakeholder interviews, as outlined in Section 2.2.2.

2.2.5. AMIF-funded projects

Over a series of four workshops, Matter of Focus worked with 12 of the 56 projects which had received AMIF funding. The purpose of the workshops was to enable the projects to develop and set out a Theory of Change in an outcome map which would demonstrate how their projects were contributing to achieving the New Scots Refugee Integration Strategy outcomes. This work was informed by an approach based on contribution analysis^[26] which aims to understand the difference that any initiative, policy, programme or project makes. It explores how the actions of an initiative contribute to making a difference, alongside other factors. In this approach, theories of change, at Strategy, programme or project level are mapped against the headings outlined in Figure 2.1.

Recruitment and sampling

A pool of AMIF-funded projects were selected by the New Scots project partnership using criteria listed below to ensure as wide and equally weighted a cohort of projects as possible. Of the 20 projects invited by the New Scots project partners to take part in a purposive sample, 12 chose to do so, in two cohorts.

The purposive sampling of the projects was based on the following criteria:

• Region (including a Glasgow cluster as well as others more widely spread across Scotland)
• Project topic
• Whether the project was designed to support innovation or widen good practice in refugee integration work
• Size of project (based on grant size)
• Project target group (who the project was working with), ensuring ethnic and gender diversity would be reflected across the invited projects.

ScotCen provided an initial list of 28 projects which represented diversity in terms project topic, target group and geographic spread. This was used as a starting point for the New Scots project partners to develop a list that would balance the criteria outlined above. Project Officers from the New Scots partner organisations were asked to review the proposed list, flag projects where they were aware that capacity to participate was low, and suggest replacements where necessary. A pragmatic approach was taken to inviting projects based on the selection criteria, by replacing them with others if they did not respond or were unable to take part.

Conducting the research

Matter of Focus held six sessions with the projects, taking them through a process to understand how, when and under what circumstances their work contributed to the outcomes they were funded for. This approach is informed by an understanding of complex systems theory: that on their own the projects don’t cause change, they contribute to better outcomes for people through complex and sometimes unpredictable processes, influenced by the relevant context^[27]. The sessions were:

1. Context and success story workshop

Considering the importance of contextual factors and sharing success stories of working with New Scots to help understand what is important when delivering work with refugees and people seeking asylum.

2. Outcome mapping workshop

Participatory outcome mapping to understand and link activities to project outcomes, and refining of each project’s Theory of Change.

3. OutNav orientation workshop

Training session in using OutNav cloud-based software used by Matter of Focus to underpin their approach to outcome monitoring and evaluation. The software provides a platform to manage outcome evaluation – a single place to hold theories of change and for collecting, collating and analysing data against them.

4. Data audit workshop

Supporting projects to develop a Strategy for data gathering and analysis and adding content to OutNav.

5. Collective Analysis (3 rounds)

Supported projects to analyse and illustrate their progress towards outcomes.

6. Learning event

Exploring projects’ learning about contributing to outcomes and about the New Scots Refugee Integration Strategy.

The agreement for working together and the questions that were explored during the learning event can be found in Appendices E and F respectively.

Participant demographics

Twelve AMIF-funded projects participated in the workshops. In total 20 projects were invited to take part in this aspect of the evaluation. Eight projects that had initially agreed subsequently decided not to take part. While the research team did not explore why this was the case, it was clear that during the time period of the reseach and the workshops, organisations working in the sector were experiencing very high workload and demand due to the developing Ukraine situation, and this probably had an impact on their capacity to participate in this process.

Table 2.7 below gives a typology of the 12 projects that participated in this work by sector and location. Support with outcome mapping and evaluation was taken up by more organisations from the third sector. All organisations were invited to send two people to attend the workshops. Four organisations attended with two people, while eight attended with one person. All of those who participated in the workshops were paid staff, including those with responsibility for managing projects. No-one receiving support from the funded projects attended the workshops.

Table 2.7: AMIF-funded projects that participated in the workshops

Organisation and/or Project Name	Organisation Sector	Location
Argyll and Bute Council - Sawa	Local Authority	Argyll and Bute
City of Edinburgh Council	Local Authority	Edinburgh
Glasgow Life – Adult Learners	Local Authority	Glasgow
Govan Community Project	Third Sector	Glasgow
Grampian Regional Equality Council - New Scots Active Citizens	Third Sector	Grampian
Ignite Theatre	Third Sector	Glasgow
Licketyspit – Porridge and Play Hubs	Third Sector	Glasgow
Media Education	Third Sector	Edinburgh
Midlothian Council New Scots Project	Local Authority	Midlothian
Outside the Box	Third Sector	Glasgow
Pollokshields Development Agency	Third Sector	Glasgow
Refuweegee – Digital Inclusion	Third Sector	Glasgow
West of Scotland Regional Equality Council – Community Integration Project	Third Sector	West of Scotland (multi area)

Analysis

Analysis was conducted across the AMIF-funded project outcome maps and data (Appendix G). Reports were pulled from the OutNav system for each of the 12 projects.These were then analysed using a thematic approach to understand:

1. What strategic outcomes in the New Scots Refugee Integration Strategy are projects contributing to?

2. What types of activity are projects delivering in relation to the New Scots Refugee Integration Strategy outcomes?

3. Are the projects reaching the beneficiaries they expected to?

4. What difference is this work making to beneficiaries?

5. What are projects learning about working with New Scots? What are the barriers and enablers in delivering this work?

The findings from this review were presented back to the projects in the learning session. Additional data from projects about their perceived impact were gathered at this event. Data gathered during the workshops and the learning event have been incorporated within this report.

2.3. Research ethics

NatCen’s Research Ethics Committee (REC) provided ethical scrutiny of this project. The REC processes are consistent with the requirements of the Social Research Association and the Government Social Research Unit Professional Guidance. The REC consists of researchers and Senior NatCen Staff who critically review all projects to ensure they meet our ethical standards. An application was made for a Stage 2 full review (involving a REC meeting) in January 2021 and ethical approval was granted in February 2021. The ethics application covered conducting all aspects of the evaluation (including the work undertaken by Matter of Focus), from recruitment and fieldwork, through to analysis and reporting. It covered how the research team would handle data, ensure confidentiality and anonymity of participants, and process for any disclosures of harm.

Recruitment

To recruit the organisational stakeholders for interview, a list of the contact details of individuals and organisations involved in the development and implementation of the Strategy was provided by the Scottish Government via a secure File Transfer Protocol (FTP) link. This was then stored in a secure folder on ScotCen’s servers which could only be accessed by the project team. ScotCen emailed invitations to a selection of these contacts inviting them to participate in an interview. Individuals were selected to represent a range of different organisations working in different parts of Scotland. Stakeholder recruitment was expanded in June 2022 to include a selection of representatives from other organisations working directly with refugees and people seeking asylum, including some projects which had received AMIF funding. These representatives were identified through interviews with other stakeholders. The names of individuals and organisations who participated were not shared with the Scottish Government or anyone outwith the project team in order to maintain confidentiality.

For the survey of stakeholder individuals and organisations, invitations were emailed by the Scottish Government, COSLA, SRC and University of Glasgow and included a link to the survey. The information collected as part of the survey related only to organisational demographics. No personal data were collected relating to individuals. Although survey respondents were invited to indicate the name of their organisation in their responses, this information was not used as part of the analysis and no information that could identify respondents was shared with the Scottish Government.

Refugees and people seeking asylum were recruited via AMIF-funded projects and other organisations who were asked to share information about the research (an information sheet, privacy notice and invitation email) with the people they worked with. This information included contact details for the research team so those interested in participating in an interview could contacted the research team directly. All materials were translated on request.

Refugees and people seeking asylum either contacted ScotCen directly to express an interest in taking part, or were invited to do so through a trusted organisation (for example, in cases where interpreters and translated materials were required for participation). Each volunteer was asked to complete a screening questionnaire which asked about their name, age, gender, address, country of origin, interpreter requirements and any other needs they might have to be able to participate. Screening questions could also be translated on request so individuals could respond in their first language. This information was collected to ensure a wide spread of participants took part. No personal information was shared directly with the Scottish Government and no information was used in the report which could identifiy indivduals or might link specific funded projects with participants. All refugees and people seeking asylum who took part were offered a £30 Love2Shop voucher as a thank you for offering their time.

Informed consent

Everyone invited to take part in an interview received an information sheet outlining the purpose of the research, details of the organisations conducting and funding the research, and what participation would entail. Participants were informed of the voluntary nature of the research, and of their rights to withdraw at any point without consequence. It was explained that all data collected were confidential and that no individuals or organisations would be named in the final report in order to preserve anonymity. A privacy notice was provided to all participants detailing the legal basis for data collection and processing, what data were collected, how they were handled, stored and deleted. Researchers provided a verbal summary of this at the start of the interview, before participants were asked if they were happy to proceed and happy to be recorded. Participants then provided verbal consent at the start of the recording. The same processes were followed when interviewing with the assistance of interpreters.

Conducting the research

All interviews were conducted by trained researchers with experience of conducting interviews with vulnerable groups and who were enrolled in Disclosure Scotland’s PVG scheme. Before fieldwork began, all interviewers participated in a briefing, which discussed issues that might be raised during interviews. Researchers were briefed to be alert to any signs of distress or reticence on the part of participants, and to take appropriate action (e.g., divert conversation to a new topic, pause the interview or stop it completely depending on the needs of the participant).

Refugees and people seeking asylum were asked if they had any requirements to help them participate in an interview. For example, they were offered to have someone accompany them to the interview, either to assist with interpretation or as a form of support. They could also choose the time of interview (including evenings and weekends) to suit them, and to help with childcare requirements. Participants were offered the choice of whether to participate in an in-person interview, online interview or over the telephone, with the intention of reducing barriers to participation for those without internet access.

Minimising harm

Discussing integration, particularly with refugees and people seeking asylum, is sensitive. Care was taken to try and minimise distress and manage any difficult dynamics to minimise the harm of the research on participants. For example, the researchers working with refugees and people seeking asylum emphasised with the interviewees that the focus of the research was on their experiences of living in Scotland. Participants were not asked about their past histories and potentially traumatising experiences.

The research materials for all evaluation activities were developed in consultation with and agreed by the Scottish Government and partners. Care was taken to ensure that the questions minimised distress and were non-judgmental. All interviews were conducted by a member of the core research team who are well trained and highly experienced in conducting qualitative research involving sensitive topics. Interviewers were trained in ScotCen’s organisational safeguarding procedures and knew how to apply these in the event of any safeguarding concern arising during the research.

Participants were offered the choice to take part in a telephone, video (via Teams or Zoom) or in an in person interview at a time which suited them. While it can be hard to pick up non-verbal communication over the telephone, researchers used the introduction to build rapport and put participants at ease before starting the interview. If participants become upset during an interview, they were offered the chance to pause the interview or stop. It was explained to participants before the start of an interview that they could choose to end their involvement at any time and that they could choose not to respond to any questions if they felt uncomfortable.

Transfer of data and storage of data

All data transferred between the Scottish Government and ScotCen was shared using a secure FTP link. All data, including transcripts of interviews, were stored in secure folders accessible only to named members of the research team. All transcripts are saved with a numerical file name and no data which might identify a participant is contained within the file.

Matter of Focus work with the funded projects

The ethical approach taken by Matter of Focus in their work with AMIF-funded projects is outlined in Appendix E. All projects were asked to agree to a confidentiality statement for any information and insights shared in the group sessions.

2.4. Strengths and limitations of the evaluation

There were a number of strengths and limitations associated with the research.

2.4.1. Limitations of the evaluation

• The second New Scots Refugee Integration Strategy is ambitious and overarching in its nature and scope. As a consequence, even those with the most knowledge of the Strategy were not always able to equate perceived impacts or developments related to the experiences of refugees and people seeking asylum with the Strategy itself. With such a wide-ranging Strategy, and with regular on-going work in the field of refugee integration, this is to be expected. The evaluation has sought to identify, where possible, impacts that can be linked to the Strategy.
• The lack of long-term data collection by the partners involved to monitor the implementation of the Strategy or progress towards outcomes means there was little large scale or objective data for the evaluators to draw on. As a result, the ability of this evaluation to evaluate the level of progress towards the Strategy outcomes is limited, although the data have given helpful insights into areas where progress appears to have been made and areas where it has been more difficult.
• As a result of the mode of recruitment for the survey, it was not possible to ensure or ascertain whether the responding organisations constitute a representative sample of refugee stakeholder organisations. A large number of recipients appeared to access but did not complete the survey. There is also a possibility of duplicate responses. However, 250 respondents completed the majority of the survey, with 176 fully complete responses. It is possible, though, that there was a bias in response towards those who had more knowledge of or were more engaged with the Strategy.
• Due to budget, there was a limit on the number of interviews which could be conducted with refugees and people seeking asylum in terms of how they have experienced integration. Therefore, we cannot draw conclusions or comparisons about how well integration is working in different geographical areas or with particular groups of refugees or people seeking asylum.
• The mode of qualitative recruitment may have introduced some bias into the sample of refugees and people seeking asylum, since only interviewees who were in contact with or receiving services from support organisations were contacted via the recruitment process. As a result, we did not reach any refugees and people seeking asylum who had not received support from organisations. However, the focus of the interviews was to explore the experiences of refugees and people seeking asylum since arriving in Scotland up until the time of the interview. This enabled the research to explore individuals’ experiences at times when they had not receive support. The interviews also explored any support received from any source, not just the support they received from the organisation who informed them about the research.

The work with the AMIF-funded projects also encountered a number of limitations:

• All AMIF-funded projects were required to develop monitoring and evaluation reports under the terms of their funding and as required by the Scottish Government and the European Union funder. Although the intention was for the work with Matter of Focus to complement and support the eventual project reporting it was challenging to engage the projects in an additional process of self-evaluation because their capacity was limited. This was made more challenging by the evaluation being delayed which resulted in the work with Matter of Focus starting once projects had already begun reporting to the Scottish Government, rather than at the start of their funding as previously planned. Both issues impacted on the overall number of projects who opted to participate in the self-evaluation support offered by Matter of Focus.

2.4.2. Strengths of the evaluation

• In order to evaluate the New Scots Refugee Integration Strategy effectively, the evaluation team sought to understand the views and experiences of as wide as possible a range of involved stakeholders, including organisations working directly with refugees and people seeking asylum in Scotland, AMIF-funded projects, and crucially people seeking asylum and refugees themselves. As such the evaluation included the insights of all the key informant groups.
• The data gathered as part of this research were extremely rich and provide greater insight into the experiences of refugees, people seeking asylum and those working to support and improve the integration of refugees and people seeking asylum. As with any qualitative data, the purpose was to provide a breadth of experience rather than quantify the experiences. Therefore, the in-depth interview data are unable to demonstrate how prevalent these views are across Scotland, and cannot be generalised to all refugees and people seeking asylum in Scotland or those working in this sector.
• Refugees and people seeking asylum were recruited with the support of organisations who provide services to these groups. This approach allowed us to work with a diverse group of refugees and people seeking asylum, and to explore key local issues for those in different areas. It also allowed us to draw upon the expertise of those working in the projects which helped to minimise any harm caused by overburdening participants with research requests and setting up interviews.