Replacement for the European Structural and Investment Funds (ESIF) Post-EU Exit in Scotland: consultation report

6. Evaluation and Monitoring Progress

In order to ensure that any new fund is achieving its aims and objectives, it is important that an evaluation approach is developed in parallel.

Section 6 covers three questions from the online consultation:

Question 7: How could we best evaluate the success of this new fund?

Question 8: What relevant parts of the National Performance Framework should this funding be targeted towards?

Question 9: Which specific aspects of the monitoring and evaluation framework from European Cohesion Policy do you consider would be beneficial to retain for any new fund?

Summary of Main Points

It was considered essential that the approach to monitoring and evaluation was developed in parallel to the design of the new fund, and provided evidence of success, effectiveness and impact. A collaborative/co-design approach to its development was encouraged.

Strong support for the use of logic models, and for the use of a broader range of quantitative and qualitative measures to evaluate the impact of the new fund. Economic measures were considered important, but not sufficient, and there were calls for a wider range of social and environmental measures relating to areas such as wellbeing, carbon reduction, income inequality, community capacity building and health/mental health.

Measures of success should align to the National Performance Framework. All outcomes could be relevant, but where priorities were identified, the main outcomes were: Economy, Fair Work & Business, Communities, Environment, Poverty, and Education. Monitoring and evaluation should be simplified and proportionate. Processes and systems should avoid complexity and bureaucracy (compared to the current ESIF programme), and should make use of existing systems of data collection and reporting, where possible.

In terms of evaluation more generally, there was support for ongoing monitoring and evaluation to identify lessons learned and to improve project delivery during the life of the Fund (i.e. process evaluation), for the meaningful involvement of communities and beneficiaries in the evaluation process to better understand impact and the difference made (i.e. outcome evaluation), and for evaluations to include both quantitative and qualitative approaches to assess additionality/added value.

Question 7: How could we best evaluate the success of this new fund?

While not always phrased in these terms, most respondents pointed to the need for the monitoring and evaluation framework to adopt a "logic model" approach to assess the success of the new fund.

More specifically, there was wide-spread acknowledgment of the importance of capturing and tracking the inputs, activities, outputs, outcomes and impacts of the investments (including the establishment of a baseline position from which change could be measured). And that success should be measured against a set of "clearly defined targets which were relevant to, and aligned with, planned delivery".

While not mentioned to any great extent, there were a few comments that highlighted the inherent challenges in clearly defining how to evaluate the success of any new fund before its aims, objectives and priorities had been agreed and/or based on the limited information currently known about the UKSPF. When there is greater clarity on both fronts, it was suggested that evaluating the success of the new fund could be based on an assessment of the contribution that the investment has made towards the fund's aims and priorities, SMART^[19] objectives, outcomes, and an informed view taken on the additionality of the investment.

There was some reference to the planned UKSPF focus on productivity, and the main view was that this could be "too narrow" or "too limiting" a focus for evaluating the success of any new fund in Scotland. Rather, there was widespread support for an approach that took a "broader definition of success". This was often articulated as sustainable inclusive economic development/growth, alongside the health and wellbeing of people and communities, and the move to a low carbon economy.

In order to measure success a blend of economic, environmental, health and wellbeing and social measures were commonly proposed.

"Economic indicators, whilst a statistically reliable measure, only capture material wellbeing. These indicators do not account for social and environmental costs or reflect social inequalities and regional disparities in full. They also fail to capture other key aspects of wellbeing and quality of life (such as work-life balance, health and education)".
SCVO

The importance of designing an evaluation approach that would provide evidence on the extent to which fund aims and objectives have been met was emphasised.

There were many calls for a "comprehensive and consistent monitoring and evaluation framework to cover the lifetime of the programme" and/or for a "menu of indicators accompanied by clear definitions" to be provided at the outset.

The importance of developing an evaluation approach "in parallel to the design of the fund" rather than "bolted on at the end of the process" was also considered essential.

The main message was that there should be clarity on monitoring and reporting requirements, and audit and compliance arrangements, from the very beginning. There was also a request by some for requirements not to be subsequently changed during the delivery phase to avoid a need for "retrospective data collection".

The afore-mentioned points were considered important in order to:

• Provide potential applicants with clear and comprehensive guidance at an early stage. This would help ensure a shared understanding of what would be expected by way of monitoring, evaluation and reporting (i.e. outcomes and impacts to be reported, a consistent understanding of the data and performance information to be collected - when, how and by whom?).
• Ensure the evaluation approach reflected the potential broad scope and remit of the fund, and the diverse range of interventions it might support.
• Ensure a consistent approach to monitoring and evaluation was undertaken by successful applicants to the fund.
• Minimise the potential for non-compliance and/or clawback in funding.

A wider point raised, but to a very limited extent, was around whether managing authorities across the UK (if funding was devolved) would ensure that monitoring and evaluation approaches are comparable or aligned. It was suggested that this could allow for evaluations to be compared and contrasted, and for lessons to be shared more widely across the different UK nations.

There was wide-spread reference to the monitoring and evaluation process or framework "building on what already exists, rather than duplicating or creating something completing different". Common feedback focussed on the following themes.

Firstly, making use of existing mechanisms, approaches and performance indicators rather than reinventing the wheel was considered a sensible approach.

Many respondents suggested the Scottish Government could link and align the evaluation approach for the new fund with the National Performance Framework, and that this could have the added benefit of:

• Reducing the need for a different set of indicators or an additional monitoring and evaluation framework to be put in place.
• Minimising duplication of effort.
• Ensuring that a strong emphasis for the fund was directed at closing gaps in performance within Scotland.

Secondly, some respondents pointed to the wealth of existing knowledge and experience regarding the monitoring and evaluation of publicly funded programmes. It was considered vital that the Scottish Government worked with others to develop an appropriate approach or system for monitoring and evaluation - "rather than a top-down approach".

It was proposed that a collaborative or co-design approach could be a useful starting point. Learning the lessons from past experience and drawing on the skills, knowledge and expertise of delivery organisations (and others) with experience of delivering and/or monitoring and evaluating European-funding projects and programmes was considered crucial. It was suggested that this could also help ensure Key Performance Indicators (KPIs) and outputs were realistic, meaningful, and measurable. A wider suggestion, but not reported by many, included drawing on the findings within relevant reports published by organisations such as Evaluation Support Scotland (ESS) and the Institute for Voluntary Social Research (IVAR).

Evaluating the success of the fund against short, medium and longer-term results or outcomes, defined at a national level (but regionally specific/ responsive), was also considered essential. A greater focus on outcomes, rather than outputs, was generally considered to be a core component of evaluating the success of the fund. Evaluating progress over time was also emphasised given that impact might not be immediate and to accommodate the time-lag for some interventions to achieve desired outcomes (e.g. infrastructure projects, activities to tackle long-term structural barriers to inclusive economic growth).

"We must move away from time bound outcomes, which assume that every client will progress directly into employment following a fixed length intervention. Every young person is different and must be allowed to take an individualised journey to a successful outcome. Evaluation should recognise progress over time, rather than fixating on time bound milestones and outcomes".
The Young People's Consortium (The Prince's Trust, Barnardo's and Action for Children)

Some respondents, primarily those located in mainly rural areas, commented on issues considered pertinent to rural and remote communities, as reflected in the following quote.

"At the outset, there needs to be a recognition that impacts and outcomes in remote, rural and island areas will be significantly different to those of highly populated, urban areas. They may even take longer to achieve. Therefore, this will require regionally specific and meaningful indicators and outputs to be defined which will then make it easier to evaluate outcomes and report on funding achievements. It follows that if the fund is, in its design and purpose regionally responsive, then the measures identified and evaluated will likewise need to be appropriate to the region. For example, the relative value and benefit a job will bring to a remote, rural Highland & Islands community is proportionately greater than a job in more urban Scotland".
Highlands & Islands European Partnership

"Real-time" evaluation was mentioned by some respondents as a valuable approach to identify lessons learned (e.g. what has worked well and less well), and to improve project or programme delivery to achieve better outcomes, although this may be better understood as monitoring and or ongoing evaluation (rather than just ex-post evaluation).

Many respondents also noted the importance of building in the meaningful engagement and involvement of communities and end beneficiaries of projects and programmes in the evaluation process from the outset. "Listening" to the views of communities and "users", including adopting "participative approaches" were considered essential. And that it should not be given lip-service or seen as simply a "tick-box" exercise. It was suggested that this approach could ensure evaluations provided evidence of the wider range of benefits and impacts for different communities, regions and target groups.

There was some support for process as well as impact evaluations, and reference to the value of programme-wide and strategic level evaluations. While there was recognition that successful applicants have a key role to play in the ongoing monitoring and evaluation of projects and programmes, including capturing some outcome data, others highlighted the role of independent external evaluators in supporting this process.

A related point mentioned in a few cases was that the cost of undertaking evaluations could be prohibitive (e.g. for smaller groups and organisations). A suggestion was that this could be costed for within the total project cost.

Many respondents identified one or more overarching principles for the design and implementation of a monitoring and evaluation framework to assess the success of any new fund. These can best be summarised as follows:

• The approach to evaluation should be robust, rigorous and proportionate. There was recognition of the need for accountability and transparency on the one hand, and making reporting requirements more straightforward on the other.
• There were many calls for the approach to be less "bureaucratic", "complex" and "onerous" than arrangements under current European funded programmes. The points raised included that: evidence and reporting requirements should not act as a deterrent to potential applicants; arrangements should be less burdensome for successful recipients; and there should be an increased focus on improving the quality of data reported on to provide useful insights for grant recipients and for funders (e.g. to help refine and inform service delivery).
• There was wide-ranging support for an evaluation approach that placed sufficient emphasis on identifying lessons learned, identifying what worked well or less well (and why), celebrated and showcased examples of good practice, successes, and achievements, and captured any unexpected outcomes or consequences (both positive and negative).
• There was broad support for an equal emphasis to be placed on quantitative and qualitative approaches to evaluation to provide a holistic view of the success and value of activities funded. On the qualitative side, there was strong support for the inclusion of "softer" measures of success, understanding the client journey and distance travelled. Rather than a narrow focus on the "numbers".
• There was strong support for the provision of opportunities for those in receipt of funding to learn from each other, to enhance knowledge exchange/sharing, for reflection, and to embed continuous improvement within organisations.

Finally, there were some calls for any IT system to be user-friendly and "road-tested" prior to roll-out, and completed in advance of the launch of any new fund.

Question 8: What relevant parts of the National Performance Framework should this funding be targeted towards?

Overview

There was very strong support that future funding should align with the National Performance Framework and Sustainable Development Goals. The vast majority of respondents considered the National Performance Framework sufficiently broad to reflect a funding programme that covered economic, environmental, social and community aims and priorities.

"The National Performance Framework is a holistic tool which sets the clear, overarching priorities for all of us in Scotland to work towards".
Museums Galleries Scotland

"Overall, the National Performance Framework provides an excellent context within which any replacement for ESIF should operate - all working towards the same vision".
University of the Highlands and Islands

Some respondents indicated that "all" national outcomes could arguably have relevance for future funding.

Where respondents mentioned specific National Outcomes and/or Sustainable Development Goals of relevance to the funding, the vast majority went on to identify multiple elements of the Framework. The top six National Outcomes considered most important for future funding were:

1. Economy.
2. Fair Work & Business.
3. Communities.
4. Environment.
5. Poverty.
6. Education

The afore-mentioned outcomes were generally considered to be those which could have the most impact "on the economy and social cohesion of regions".

This was followed by support, but to a much lesser extent, for the remaining five National Performance Framework outcomes:

7. Health.
8. Culture.
9. Human Rights.
10. Children.
11. International.

Many respondents' commented that the focus for future funding should be on narrowing the gaps in performance within Scotland, and highlighted the interconnectedness of economic, social and environmental issues and National Performance Framework outcomes.

As highlighted earlier, there was strong support that decisions on how funding is targeted to deliver against the National Performance Framework Outcomes should, however, be decided at the regional/local level.

There were many comments that supported using the National Performance Framework, but with a caveat that it should not be too prescriptive - for example, it was reported that each outcome was likely to vary in level of relevance, importance and applicability at a regional/local level. The importance of "place" was again emphasised, and this encompassed the level at which priorities and outcomes should be defined and agreed.

"It will be for the community to decide which aspects of the National Performance Framework are of most relevance and then agree outcomes and delivery programmes with the relevant bodies".
Clyde Gateway

"The funding could be used to contribute positively towards many of the national outcomes, with local flexibility to determine which outcomes to focus on taking into account local circumstances".
South Aberdeenshire Local Action Group

"The structure of the National Performance Framework doesn't allow for a clear articulation on the value of place as it takes a whole-Scotland approach".
Universities Scotland

"Furthermore, specific metrics will have to be agreed in line with delivery within a place-based model".
University of the Highlands and Islands

Aligned to this point, is that there was some acknowledgement that funding to replace the ESIF would likely be limited, and would therefore need to be targeted (e.g. a focus on a smaller number of outcomes). Another respondent highlighted the climate emergency as an urgent challenge and that this alone could exhaust the funding. There was therefore support for a more "selective" or "targeted" approach based on critical need and where investment could make the biggest difference.

Many respondents felt that economic measures on their own would not be sufficient. There was some reference to "macro-level" economic indicators within the National Performance Framework (e.g. economic growth, research and development spend), and that individual projects would find it challenging to report against these measures. There was strong demand for a combination of economic measures and those that reflect Scotland's wider policy priorities (e.g. income inequality, greenhouse gas emissions, wellbeing).

And that there would need to be recognition within the framework that there might be a significant time lag between activity and impact. Aligned to this, was support for long-term multi-annual funding to provide stability, and for meaningful change and impact to be achieved.

Wider Points

Wider points raised (but not by many respondents) are captured below:

• Where possible existing National Performance Framework reporting channels should be used.
• The framework should take account of protected characteristics, vulnerabilities and inequality. For example, it was reported that the National Performance Framework lacked a clear measure or indicator of gender and racial inequalities.
• It was proposed that Environment, Equality, and Human Rights could be horizontal themes within the new funding programme.

Question 9: Which specific aspects of the monitoring and evaluation framework from European Cohesion Policy do you consider would be beneficial to retain for any new fund?

Around one-third of respondents left Question 9 blank, provided no comment, had insufficient knowledge on the monitoring and evaluation framework from European Cohesion Policy, or felt that it was difficult to respond in the absence of more detail on the UKSPF.

The most common feedback from respondents that provided feedback can be summarised as follows.

Firstly, there was strong support for monitoring and evaluation to be built into the process to provide evidence of success, effectiveness and impact. However, there was a request that should aspects of the monitoring and evaluation framework from the European Cohesion Policy be retained, then important considerations would be how to reduce bureaucracy to make the process less resource intensive and to reduce the reporting burden (i.e. time and cost implications). There was a strong plea for a simplified and agile approach (e.g. "proportionate to the scale and size of the project and risk of fraud", "sensitive to differing needs across Scotland"), and engagement with stakeholders and experts in monitoring and evaluation at the programme design stage.

"Bureaucracy should be reduced, auditing should be proportionate, outcomes should be more suited to the Highlands & Islands and evidence documentation needs to be manageable and consistent with the scale and size of the project. This would mean that there is less likelihood of mistakes being made and more emphasis can be placed on monitoring and evaluating outcomes rather than inputs. In order to assist this process, it is important that local authorities and partners have a greater role in design, governance and delivery at regional and sub-regional levels".
Highlands & Islands European Partnership

"A key issue was repeated change requirements for monitoring data and evidence of compliance - these over-arching requirements must be fixed at the outset of the programme and not subject to further change….There is a current over focus on evidence and, in particular, compliance which takes staff time away from working with their clients and creates significant barriers to engagement. Instead there should be flexibility to work with people in need".
Fife Voluntary Action^[20]

Secondly, there was strong support for aligning the monitoring and evaluation framework for the successor programme to existing mechanisms and systems (e.g. using existing indicators, systems and frameworks to avoid duplication). Similar points were raised around making the process for audit, compliance, monitoring and reporting less "restrictive", "onerous", "complex", and "risk-averse". A key message was that requirements should be clear from the outset and consistently applied over the funding period.

"Currently the burden of regulation and reporting is slanted towards the prevention of fraud and avoidance of risk rather than based on achieving outcomes. As such it tends to create rigid and highly proscriptive delivery models which are inevitably inefficient and cumbersome".
Apex Scotland

"There should be a greater degree of trust in local authority financial management and audit procedures than there has been for previous programmes".
North Ayrshire Council

"Verification and eligibility checks of supported participants should be simplified and the future fund interface should be designed with the user in mind rather than from an audit perspective. Audit requirements should be commensurate to the size of investment and the risk of fraud".
Clackmannanshire Council

"The fund has always focused upon audit of spend rather than audit of achievements and in many cases the level of evidence required for audit purposes is grossly disproportionate to the actual cost incurred. It would be more sensible to focus on achievement of the stated outcomes, and with payment of grant being more closely linked to the achievement of those outcomes. This would also encourage focus on performance".
Aberdeen City Council

Thirdly, there were calls for a greater focus on outcomes. Here, there were requests for a less "onerous" and "intrusive" approach with regards to the volume of monitoring data required under the current programme. It was felt that a different approach could allow more focus to be placed on results and outcomes, and less on inputs.

Linked to this point was support for continued adoption of a logic model approach (i.e. theory of change) to drive long term structural change, and to assess efficiency and impact of investment.

Aligned to this were a variety of points raised around the importance of:

• A common approach to data recording and storage. This could include programme-specific targets, baselines and milestones, as well as a clearly defined list of common indicators. The longer term focus of the successor programme should be on the additionality of economic, social and environmental outcomes.
• Meaningful indicators, results, and outcomes to be defined/agreed (at regional and local level) by all relevant stakeholders at the outset of any new funding scheme - and consistently applied over the funding period. This could include quantitative and qualitative measures of success.
• Recognition that impacts and outcomes in remote, rural and island areas were likely to be significantly different to those of highly populated, urban areas. And that some impacts might not be measurable or achieved over the short-term.
• Acknowledging that it can be challenging to disaggregate the impact of ESIF from other activities and funding streams, and that outcomes are influenced by a wider range of factors.
• A few respondents highlighted (again) the importance of independent evaluations.

Fourthly, there was strong support for a process of regular, ongoing review and evaluation:

• There was strong support for ongoing monitoring, reflecting a desire for learning outputs to be timely enough to influence and shape future delivery.
• There was also strong support for a focus on continuous learning to improve delivery and performance - rather than "target-driven behaviour" and "where improvement rather than compliance is the desired goal".
• Some respondents expressed support for the process of strategic planning, including: partnership involvement in programme design and delivery, and ex-ante,^[21] mid-term and/or ex-post evaluations. For ex-post evaluations it was considered important that these were timely enough to inform future programmes.
• A few respondents expressed support for the introduction of "peer to peer" monitoring and evaluation to overcome the perceived "top-down approach that currently exists. A related point was that this type of activity could have resource requirements for organisations.

A few wider points were raised, but not by many respondents.

Some respondents commented on challenges experienced with the various Management Information Systems (MIS) that are used for EU programmes,^[22] with the feedback that some are more user-friendly or intuitive than others. Differing systems were said to cause confusion, and the EUMIS system for ESF and ERDF was not viewed in a positive light. The respondent quotes below reflect the tone of the comments made.

"….an enhanced harmonised IT system should be designed, thoroughly road-tested and in place by the conclusion of the ESIF and for the start date of the replacement funding framework. The European Commission has acknowledged the need to achieve results "in a simple, fast, flexible and cost-effective manner" and so for its next budget period 2021-2027, it will seek to adopt a single rule book and harmonised reporting systems for its various funding programmes. Assuming that UK organisations will still be able to participate in certain transnational funding programmes, we suggest that the Scottish Government maintain close links with the European Commission as it designs its future processes for reporting, to ensure that the domestic and transnational systems do not greatly diverge, cause conflict or add complexity".
Dundee City Council

"A single system would be beneficial for the replacement fund, but it is also important to make use of other existing systems for capturing outcomes such as the National Performance Framework".
Fife Council

Where mentioned, the concept of Horizontal Themes was considered to have been useful (e.g. in terms of mainstreaming associated policies in Scotland). It was suggested that this could be enhanced to take account of climate change and the Just Transition to low carbon, and inclusion).

Linked to this, was reference to a Key Policies Group, which invited experts to develop Scotland's approach to the Horizontal Themes, and was considered to have worked well. Another point made was that more could be done to quantify and qualify contribution to any Horizontal Themes that are established.

Finally, a few respondents felt that a more sensible or appropriate approach would be for the Scottish Government to design its own monitoring and evaluation framework, including reference to alignment with the National Performance Framework.