New Scots Refugee Integration Delivery Project: guidance

Background to the fund

The New Scots Refugee Integration Delivery Project (NSRIDP) is a new programme supported by the European Union Asylum, Migration and Integration Fund (AMIF). As part of this programme, up to £2.8 million will fund new projects to spread documented good practices and to support innovation in Scotland under the outcomes and objectives, and in relation to the beneficiaries of the New Scots Refugee Integration Strategy.

Recruitment of panel members

32 panel members were recruited from across Scotland to score applications, including members with lived-experience, members from refugee-assisting organisations (including third sector organisations and local authorities) and members from refugee-led organisations. The panel members broadly represented key stakeholders involved in refugee integration across Scotland and the project team are confident that the diverse expertise and experience greatly aided the scoring and evaluation process.

Training for panel members

The induction and training was delivered in two phases:

Phase 1

The aim of the first phase of training was to ensure panel members were equipped with the appropriate knowledge and skills to ensure that they were robust, fair and consistent in their approach to scoring applications.

Phase 1 involved comprehensive training sessions with small groups of panel members delivered by project officers from Scottish Government, Scottish Refugee Council and COSLA. Phase 1 covered the aims and aspirations of the NSRIDP, information on the roles and responsibilities of panel members, an overview of the scoring process, an introduction of the evaluation criteria and scoring matrix, and also included a mock scoring exercise.

Phase 2 

The aim of the second phase was to familiarise panel members with the online scoring portal and its functionality to ensure that they felt confident to use the portal independently when reviewing applications.

Phase 2 was delivered by Scottish Council of Voluntary Organisations (SCVO) staff with the support of a NSRIDP project officer and provided a comprehensive overview of these elements with 10-11 panel members attending each session.

Panel evaluation process

The 32 panel members were split up across 4 separate panels of 8. Each of the 4 panels scored applications from a range of topics. To avoid any conflicts of interest, panel members who submitted applications under a specific topic were ineligible to sit on a panel assigned to that topic. On this basis the Project Team assigned applications to individual panel members.

The Project Team paired up two panel members: ‘Person A’ and ‘Person B’ who both scored the same set of applications, independent of each other.

Scoring and assessing applications

The role of independent panellists was crucial to the openness and transparency of the scoring process. The diversity of panel members ensured panellists appreciated the importance of understanding the aims and aspirations of the NSRIDP.

Following their comprehensive training, panel members were assigned a set number of applications to mark. The scoring matrix and evaluation criteria provided panel members with a set of criteriato score applications against. Panellists considered the application forms and guidance, and sought to apply their training with consistency and fairness as independent scorers. 'Person A’ and ‘Person B’ then scored all the applications allocated to them and completed a short evaluation providing written feedback for each application.

Once panel members finished their scoring, each 'Person A' and 'Person B' pair met with a member of the NSRIDP Project Team for a moderation or ‘buddy’ meeting. This was an important opportunity for panellists to discuss their scores, and crucially, this acted as a mitigation process to address significant inconsistencies - ‘trigger’ points for review - that were occasioned by their separate scoring.

This process was established prior to the beginning of marking, and the ‘buddy’ meeting enabled any triggers to be addressed. Where one or more trigger unresolved after the ‘buddy’ meeting, the application was reassigned to two other panel members to score.

Once final scores from panellists were confirmed, the scores were subject to the funding model (see below),which identified the projects that should be awarded funding.

Reasons for delaying the announcement of successful applicants

It was crucial that our scoring process was rigorous and fair to all applicants. For this reason, when some panellists experienced a technical issue with accessing the full application form, we took additional measures to ensure that each application had been fully assessed. Our assessment panel reflects the diversity of those involved in implementing New Scots and many of our panel members have other commitments. As such, we extended the time allowed to panel members to complete their scoring. This process was also reviewed and approved by Scottish Government’s audit team, with the overall process taking an additional six weeks to ensure this review was comprehensive.

Overseeing the scoring process

NSRIDP Project Officers were involved through delivering training to panellists and facilitating buddy meetings between scoring pairs to discuss applications and address any trigger points that had arisen from the scoring process.

Administering the scoring process

The NSRIDP procured the services of the Scottish Council of Voluntary Organisations (SCVO) to develop a web-based scoring platform based on the Salesforce system. Panellists were able to view their applications online, submit their scores against the scoring matrix, and provide a short written evaluation of each application. Full training for the use of the platform was provided by the SCVO prior to the scoring process beginning.