European Social Fund and European Regional Development Fund: monitoring and evaluation strategy

Evaluation proposed

When

Questions to be answered

Proposed method

Resources and reasoning

Ex-Ante

Completed

Is programming and financial planning consistent with analysis of needs; and likely to achieve stated objectives and targets?

Internal baseline review conducted independently of MA External review of methodology, intervention logic and proposed objectives, activities and results.

SG Analytical Services carried out extensive baseline. External contractors acted as a check and balance on MA and SG policy decisions informed by that baseline and ensured logic and coherence within programmes.

Youth Employment Initiative 1

Completed (January 2016)

It has been difficult to establish successful actions under YEI in Scotland – what are the reasons for this?

• policy overlap
• client groups
• macro-economic changes

SG issued a survey as part of the evaluation to key stakeholders and Lead Partners in November 2015 to gather views on a range of issues connected with the YEI, including whether the long term outcomes (sustainable employment, apprenticeships, traineeships, continued education, etc.) are realistic. Views around the ineligibility of the unit cost model for YEI have been sought and a number of relevant     questions associated with match funding and the current policy focus have been raised. Analysis of the results will be detailed in the YEI Evaluation report due by end of December 2015. This will be incorporated into a wider lessons learned exercise capturing the views of policy and operational colleagues within Scottish Government alongside the views of stakeholders.

The Territorial Committee monitors impact of YEI and recommends improvements, specifically in relation to significant changes to the underlying labour market. SG draws on these resources to extract relevant information on policy developments and implementation issues.

Youth Employment Initiative 2

Completed (December 2018)

What types of skills training are being delivered to 16-24 year olds (ESF)?

How many 16-24 years olds have achieved sustainable employment, education or training (ESF)?

Has youth unemployment among 16-24 year olds reduced as a result of Structural Funds intervention, or other factors?

Desktop Data review.

Participant survey.

Comparison of Counterfactual Impact Evaluation (CIE) evidence on impact of different projects/schemes.

The MEG will be instrumental in the planning and development of the 2nd YEI evaluation.

The MEG/working groups will also help decide whether CIE is the best methodology to adopt.

Mid-Term

Reported to PMC in May 2017

How programmes are performing relative to their stated aims and overarching objectives.

Whether resources are being used in the most efficient and effective manner (i.e. could funding be used more successfully in different allocation?).

What the scale of European Structural fund investment is relative to other contributing factors.

What the most effective types of projects and schemes are, e.g. are some projects better at helping people into work than others?

Whether across projects and schemes there is sufficient coverage of the objectives of the Programmes and is the balance of interventions appropriate?

Whether the Programmes are on track to meet the performance reserve targets.

Whether the implementation processes put in place by the MA’s are effective for managing the Programmes?

Qualitative research with project/scheme and MA staff and programme stakeholders (e.g. PMC).

Review of programme and project/scheme documents

Analysis of macro- economic/external issues affecting successful implementation of the Programmes.

Questions asked in ‘omnibus’ surveys of the general public to test awareness of the Programmes.

Qualitative research with project / scheme and  MA staff and programme stakeholders (e.g. PMC).

Qualitative assessment of the outcomes achieved, e.g. the types of jobs, levels and types of skills, types of businesses targeted; and whether these continue to match SG and EU policy priorities.

Extensive update of ex-ante baseline will be carried out by SG Analytical Services.

External contractors will need to cover two aspects: whether the policy direction continues to be correct to contribute towards EU 2020; and whether the clustering of operations within Strategic Interventions is showing enough early signs of successful impacts and improved controls to continue, or whether stopping such interventions would be disruptive to long-term policy aims (e.g. where  something new is being tried).

Delivery Structures and  Partnership Working within ESF and ERDF Programmes

Completed (February 2022)

Delivery of programmes in 2014-20 is dependent on organisations working closely together to achieve  ‘more than the sum of their parts’. What factors make a successful partnership in this context, and can be replicated across the programme and into the future?

Has the Lead Partner model adopted in the 2014-20 programme been successful in achieving outcomes?

Whether the Lead Partner model would be one to continue into future funding programmes.

Has the Lead Partner model helped maintain a consistent level of delivery?

Has the Lead Partner model been flexible enough to allow adaptations mid-programme?

Is the Lead Partner model sufficiently transparent?

Lead Partner, delivery agents and stakeholder interviews.

A contractor will be procured to deliver the work, Lead Partner groups will have an input role in designing the scope.

The effect of funding in H&I Region

Completed (March 2022)

What impact funding has in the  H&I region where economic and remote geographic conditions are unique, as recognised by its status as a Transition region.

Whether there are types of projects where funding is  particularly effective in these type of regions.

Whether there are types of projects where funding is notably less effective in these types of regions.

Quantitative and qualitative assessment of impact in H&I communities.

Assessment of projects in those areas – effectiveness in terms of results; effectiveness in term of prevention of negative impacts.

A contractor will be procured to deliver the work, HITC and Lead Partners in the Highlands and Islands will have a significant input role in designing the scope.

Smart Growth and/or Inclusive Growth evaluation

Completed October 2022

Scope to be considered and developed during 2020. Potential topics may include:

• What impact has ESIF funding had on SMEs?
• More benefits/impacts on improving long-term unemployment than through funding available by other means.

Beneficiary survey

Lead partner input

A contractor will be procured to deliver the work, Lead Partner groups will have an input role in designing the scope.

Identify other areas of funding to improve long-term unemployment and review any evaluations carried out.

Low Carbon thematic evaluation

Low carbon has seen a significant expansion of activity under the 2014 programmes. Is it delivering benefits?

More benefits/impacts than in areas which are ‘tried and tested’ – Why?

Has the 2014-2020 programme contributed towards Scotland’s approach to Low Carbon.

How has the level of investment through ESIF to benefits delivered compared to specific Low Carbon projects.

Has there been sufficient flexibility to allow focus of funding to switch towards Low Carbon projects when needed?

Participant and beneficiary survey.

Detailed interviews with lead partners.

Desk based performance assessment from EUMIS and macro-economic updates.

A contractor will be procured to deliver the work, Lead Partner groups will have an input role in designing the scope.

Participant and beneficiary survey to be undertaken, including capturing the additional amount of other investment levered in.

Establish and review any evaluations already carried out within SG’s specific low carbon policy areas.

Ex-Post

Whether there are any over-arching achievements of the Programmes that can be summarised?

Questions asked in ‘omnibus’ surveys of the general public to test awareness of the Programmes.

Qualitative research with project / scheme and MA staff and programme stakeholders (e.g. PMC)

Questions asked in participant and business surveys to test awareness of how their support was funded.

TBC