Wellbeing Fund - open application process: evaluation

2. Evaluation aims, method and limitations

2.1 Aims

The first part of this evaluation aims to examine where and how the Wellbeing Fund Open Application funding was spent, and what the outcomes of the funding were from the perspective of the funded organisations. As it was not possible from the data available to draw direct conclusions about the overall impact of the fund, this report is not able to measure or comment on the fund's overall effectiveness in reducing the effects of the pandemic on target communities. Instead it focuses on what the funding was used for and where it was distributed. The main research aims were to:

• Examine the distribution of funding across the country by geographical area, deprivation levels, target groups, and type of project;
• To understand what the funding was used for and who was supported by the funded projects.
• Examine whether organisations used the funding as they had initially envisaged (and reasons why it was not used as originally envisaged);

The second part of the evaluation consists of a process evaluation which examines how well the design, management and implementation of the fund worked, and what lessons could be learned from the experience of implementing this fund to help in the development of future similar initiatives. The main research aims were to:

• Examine how well the design, management and implementation of the evaluation worked;
• Understand any challenges to the implementation of the fund;
• Make recommendations on how future similar emergency and non-emergency funding initiatives could be improved

2.2 Method

This evaluation report, conducted by Scottish Government analysts, brings together analysis from three main data sources: applications and awards data for all Wellbeing Fund Open Application Process applicants; monitoring returns data submitted by funding recipients; and interviews with key stakeholders.

Applications and awards data from all 1,563 applications to the fund was used in the analysis. This data included organisational and location information, as well as details of the proposed projects, target groups, and intended outcomes. This data was collected by the funding organisations and supplied to the Scottish Government. This data also formed the basis of the previous publication "Wellbeing Fund: Analysis of awards made through the Open Application Process and the Small Grants Fund" (Scottish Government, November 2020)^[2] and a published funding mapping tool.^[3]

Monitoring returns data was collected by the funding partners, with all successful applicant organisations asked to complete an end-of-project monitoring form. 916 of the 955 grant awardees (95.9%) completed the monitoring form in time to be included in this analysis. The monitoring forms included both closed- and open-ended questions requesting information on:

• what the funding was used for;
• how much was spent;
• whether any changes were made to the proposed project;
• what difference the project made;
• what challenges were faced;
• organisations' experiences of using the fund.

This information was linked with the application data for analysis, and is used to inform both the outcomes and process elements of the evaluation. Where open-ended responses have been analysed, a separate random sample of responses was generated for the responses to each question analysed.

Interviews were conducted with 16 key stakeholders, as well as one who provided information by email. Interviewees included two Scottish Government officials, two SCVO staff members, three staff members from funding organisations, and six TSI representatives. Three funding assessors also took part in an interview, and one provided information by email. These were semi-structured interviews, some of which were conducted individually, others in groups. They provided qualitative data used to gain an understanding of how the fund was designed, set up and implemented, how well the funding process worked, and what could be learned from this experience to improve future similar initiatives.

2.3 Limitations

As discussed in the aims, due to the nature and limits of the available data, it is beyond the scope of this project to make any definitive claims regarding the overall impact of the Wellbeing Fund Open Application Process. In particular, it was not possible to conduct direct evaluation with the individuals, groups or communities supported by funded organisations. It was also not possible to compare funded areas/organisations with those that did not receive this funding, or make any assessment of whether the funding been used more effectively in different way. Therefore, rather than seeking to make claims about the impact of the fund, the parts of this report focusing on outcomes instead present a picture of what the funding was used for, the extent to which it was used as intended, and who was supported by the fund.

By relying on organisations' own reporting about how they used the funds and how effective the projects were, the report cannot provide an entirely impartial, objective picture of the use of the funding, but it nevertheless provides important and useful insight into the successes and challenges of the funded projects.

It is also important to note that the Wellbeing Fund Open Application Process existed within the wider context of numerous Scottish Government and other emergency funds to support third sector organisations. This report is therefore limited by its focus on a single fund which represents a single aspect of a much wider funding landscape.

Analysis of the geographical data relating to applications and awards was limited by the fact that many applicant organisations were working across several areas and more than one local authority. This made it challenging undertake full analysis of how funding was spent in different local authorities or in areas of higher deprivation. Accordingly, the quantitative analysis presented in relation to geographical data relies on various assumptions, which are set in the relevant text.