Child Poverty Practice Accelerator Fund (CPAF) round 1: reflections and lessons

Annex 3: CPAF Evaluation Framework

CPAF Evaluation Framework

Section 1 / Introduction

This is the evaluation framework and guidance for the Child Poverty Practice Accelerator Fund (CPAF).

What is CPAF?

CPAF supports local authorities and health boards across Scotland to test approaches for accelerating their actions on child poverty. Run by the Scottish Government, it aims to:

Address the three key drivers of child poverty.

• Target the six priority families at the greatest risk.
• Engage those with lived experience of poverty in the design and implementation of interventions.
• Enhance local partnership working.
• Scale or maintain projects, if successful.
• Generate evidence and build the national evidence base.
• Innovate to accelerate practice to tackle child poverty

Who is this guide for?

This guide is for any recipients of the Child Practice Poverty Accelerator Fund.

Ideally, project leads and those involved in monitoring and evaluating the project should read this document.

The guide may also be of use to other non-CPAF funded projects interested in improving their approach to monitoring and evaluating their progress towards tackling child poverty.

How to use this document

1. Introduction

What is CPAF and how it is being evaluated

2. The CPAF evaluation framework

The overall framework and detail about the four key themes

3. How to develop an evaluation approach

A three-step guide to developing a project-specific evaluation approach. Information on data collection methods is provided

How is CPAF being evaluated?

When implementing or developing new approaches, it is important to understand quickly what works and what does not. Therefore, to support project implementation, CPAF recipients will collect data about the processes they are using and will share this with the Scottish Government.

Because CPAF is a new approach to tackling child poverty, the Scottish Government are mainly interested in understanding if, and how, the approach is working. The aim is to ensure that they can learn from people’s experiences and that any decisions are based on evidence.

Additional data projects could collect

Ongoing review and evaluation can improve project delivery. However, it is important evaluation is set up according to project needs.

Projects are encouraged to establish monitoring and evaluation frameworks to inform ongoing improvements. Projects have the flexibility to choose what data to capture and how to capture it, to help inform evidence gathering. Projects may choose to collect data about the process, impacts or economic value of the fund.

A process evaluation seeks to understand what has caused an intervention to meet (or not meet) its aims and objectives

An impact evaluation seeks to understand how well an intervention meets its intended objectives and outcomes.

An economic evaluation seeks to understand the value for money of an intervention

It is important to think about what evidence is useful:

• Evidence that can lead to additional funding opportunities (such as value for money/ return on investment)
• Evidence that can improve or change things elsewhere in the system
• Evidence that does not currently exist or is outdated.

Section 2 of this guide will help you to determine what is important to evaluate. Section 3 provides guidance on how to collect data.

Learning Together

Ongoing communication and collaboration are important to ensure that learnings are shared and widely understood. Data and insights collected by projects from their own evaluation approaches should be shared at quarterly meetings with Scottish Government.

The Scottish Government will use the Insights Report (Appendix I) to structure quarterly meetings. Projects may want to adapt the Insights Report to gather learnings from wider partners and stakeholders.

Projects may also choose to use the Monitoring Report template (Appendix II) for a structured approach to gathering ongoing information that can be used to inform the final report.

Using these two tools will ensure projects collect the minimum levels of data needed by the Scottish Government, as well as any additional data that is relevant for the project.

Tool

Insights report

Description

Insight reports provide real-time, actionable insights on what is causing a project to meet (or not meet) its objectives.

How often to collect data

Quarterly.

Meetings will be structured by the insights report (see Appendix I).

Projects can choose to collect data more frequently.

Tool

Monitoring report

Description

Monitoring forms collect regular data about project inputs and activities.

How often to collect data

Projects can choose how often to collect this data.

It is recommended that data is collected regularly through the duration of the project and shared with the Scottish Government in quarterly meetings. However, this is not a requirement. Data should be included in final reports.

Section 2 / The CPAF evaluation framework

The CPAF framework has four overarching themes.

The themes include prompts that can support projects to think about what information they need to collect to understand how and why the project is or is not working.

Table 1 provides an overview of the CPAF Evaluation Framework. The four overarching themes are listed along the top of the table with the evaluation questions, relating to each theme, listed underneath.

In the following sections further information is provided about each theme and potential methods. Section 3 provides additional detail about methods.

However, there are important points to consider:

• Projects should adapt the approach for their own needs. They do not need to answer all questions and may add others.
• Some data can be collected on a quarterly basis to support conversations with the Scottish Government at quarterly meetings.
• Other data may be more appropriate to collect at project mid-point and close, to include in project final reporting.

Table 1: CPAF Evaluation Framework

How is the design and implementation going?	How effectively is the project meeting its objectives?	How well is partnership working going?	What has the project team learned?
How well does the project engage those with lived or professional experience?	What is going well (strengths), not so well (barriers), and why?	How have existing and new partnerships been used to develop the project?	What has been learned and shared that adds to the knowledge base on child poverty?
What skills, capacity and resources are needed to implement?	What has had to change, how and why?	Which partnerships have been important for achieving the project objectives?	What has been learned and shared about the process for tackling child poverty?
How can the work be built on when funding ends?	How successfully is the project innovating?	What are the wider impacts of partnership working?	Should the project be scaled / replicated? If so, how?

1a/ How is the design and implementation going?

As CPAF is funding innovative approaches, it is useful to capture data that shows how the project design and implementation is going.

Collecting this data regularly – ideally on a quarterly basis – means that any time a change to the design or implementation is planned or made, it is possible to look back on this data to ensure important lessons are adopted.

Prompts

How well does the project engage those with lived or professional experience?

• What has been the process so far of building relationships and trust with people?
• What are the risks, challenges and opportunities?
• How is the project using / acting on this information? For example, is it impacting implementation plans, the way the project works with partners or wider policy and strategy development discussions?
• Is the format of engagement appropriate? Is it accessible and inclusive?
• What has the quality of the engagement been so far? Is it meaningful? Timely?

Potential methods

Co-design and participatory approaches – to observe the process and reflect on participants’ experiences of the process.

Interviews and focus groups – to ask people about their views.

Prompts

What skills, capacity and resources are needed to implement the project?

• Is the project being delivered within budget and other resource constraints? If not, why (also useful to reflect on underspend)?
• What are the perceived costs compared to business as usual?
• What are the perceived benefits for service delivery and service participants?
• Are the new skills/capacity able to be used elsewhere?
• Have the skills, capacity and resource needs changed overtime?

Potential methods

Monitoring report – to gather data on spend and resources.

Insight report – to gather reflections on progress.

Prompts

How can the work be built on when funding ends?

• Should work continue beyond the end of funding and how?
• How will the learnings be captured into wider strategic planning?
• How can elements of the service be continued as business as usual?
• How will learnings be embedded into service delivery?
• How will the project shape other ongoing or future work or funding proposals?

Potential methods

Insight report – to reflect on progress.

Process mapping – to look back on changes to internal workflows, processes, systems, roles and responsibilities.

1b/ How effectively is the project meeting its objectives?

Capturing data on how effectively the projects are meeting their objectives and creating impacts is useful to understand if CPAF is working as intended. It’s also important to understand how this may change as the project develops.

Reflecting on this data will help projects understand if they are on track. Ideally, data should be collected quarterly and at project close, but projects are free to choose a different timescale that suits their needs and should discuss this with the Scottish Government.

Prompts

What is going well (strengths), not so well (barriers), and why?

• Consider the objectives set out in the project plan, to what extent are they being met?
• Where are objectives not being met, what are the barriers to achieving the objectives?
• How is it meeting the wider CPAF objectives (such as tackling one of the three drivers, supporting any of the priority groups, or contributing to a reduction in inequalities)?
• Are there any other local or national objectives the project is contributing towards?
• How well is the project moving towards achieving its objectives for its target group?

Potential methods

Monitoring reports – to monitor evidence of impacts as the project progresses.

Insight reports – to understand what is going well/badly.

Prompts

Have there been any changes to the project objectives?

• What, why, and when?
• Have any unintended consequences happened so far, whether positive or negative? Should this change the project objectives?

Potential methods

Insight reports – to record changes and the reasoning behind them.

Prompts

How well is the project innovating?

• How is it changing how services are being designed or delivered?
• What are the barriers to this and how are they / can they be overcome?
• What new knowledge is it creating? How can the impact of new knowledge be captured?

Potential methods

Interviews, focus groups and/or surveys (with people not directly involved in delivery) – to measure innovation.

Prompts

If the project is reaching those with lived experience:

• Is it reaching intended low-income families with children or priority family group(s)?
• Why/ why not? What barriers are there? Is it accessible?

Potential methods

Monitoring reports – to count the types of families engaged.

Interviews – to get perceptions.

1c/ How well is partnership working going?

Partnership working is where two or more organisations work together for a common purpose. Capturing what makes a successful partnership is important. It is also useful to capture the wider impacts of partnership working as this will aid understanding of the secondary benefits from CPAF. Reflecting on this data will help projects to improve their approach to partnership working.

Ideally, data should be collected quarterly and at project close, but projects are free to choose a different timescale to suit their needs.

Prompts

How are existing and new partnerships being used to develop the project?

• Nature of relationship – how are the project and its partners working together? Are there shared values and mission? Are there established processes e.g., communication pathways?
• State of relationship – is the project strengthening the partnership?
• What new types of partnerships have formed?
• Strengths – what is working well in these partnerships?
• Challenges – what is not working so well?
• Areas for improvement – how could the partnerships be improved?
• How are partnerships changing over time? Are they sustainable?

Potential methods

Monitoring reports – to count the types of partners engaged.

Insight reports (from partners) – to understand what is going well/not so well.

Workshops – to get multiple people thinking about what works.

Prompts

Which partnerships have been important for achieving the project objectives so far?

• Why have some been important? Why have others been less important?
• Are there any gaps? Who should be contributing but is not involved?

Potential methods

Stakeholder map – to highlight key gaps.

Prompts

What are the wider impacts of partnership working?

• Is the project helping to support or improve processes, services, ways of working etc. for other partners?
• What is the downstream impact of the project? How will the project impact demand for other service in the system? How does this compare to the costs of delivering the project?
• Do these services have the capacity to handle changes in demand?

Potential methods

Ripple effect mapping – to track wider impacts.

Systems maps – to highlight key leverage points.

1d/ What has the project team learnt and how is this being shared?

Capturing how the CPAF projects contribute to knowledge about child poverty and understanding of how to tackle it helps ensure learnings influence future approaches. Sharing lessons ensures future approaches build on existing successes and avoid challenges.

Projects should collect lessons regularly in order to be able to share lessons in their quarterly meetings with the Scottish Government.

Prompts

What are you learning and sharing that adds to the knowledge base on child poverty?

• What have you learnt? How does this learning compare with that from other projects or from existing learning? What gaps in knowledge remain?
• How are you sharing learning to ensure that it informs future work/ approaches in your local area and elsewhere (other areas and/ or nationally)?
• How can learnings reach and support: those with lived experience; you/your team; your organisation; other organisations in your local area (partners and stakeholders).

Potential methods

Insights report – to record learning.

Lessons log – to create a complete record of lessons learned and shared.

Workshops – to present your learnings to others.

Prompts

What are you learning and sharing about the strategic process for tackling child poverty?

• What have you learnt? How does this learning compare with that from other projects or from existing learning? What gaps in knowledge remain?
• How are you sharing learnings to ensure that it informs future work/ approaches in your local area and elsewhere (other areas and/ or nationally)?
• Consider how learnings can reach and support: you/your team; your organisation; other organisations in your local area (partners and stakeholders); other organisations working on child poverty (especially other CPAF recipients); the Scottish Government.

Potential methods

Insights report – to gather reflections on your learning.

Lessons log – to create a complete record of lessons learned.

Workshops – to present your learnings to others.

Prompts

If the project should be scaled / replicated, how can this happen?

• What was essential to the successes or impacts of the project? Why?
• What challenges were encountered? Did they change over time? How were they solved?
• What considerations are there if implementing at a wider scale or in a different location?
• How can these lessons be effectively shared with other organisations?

Potential methods

Insights report – to reflect on successes and challenges.

Lessons log – to consider which learnings should inform scaling/ replication.

Section 3 / How to develop an evaluation approach

There are three important questions to consider when developing an evaluation approach:

• What needs to be measured?
• What types of data need to be collected
• How can data be collected?

These questions are useful to help you to develop your evaluation approach, but are not a funding requirement.

What needs to be measured?

Understanding what should be measured is essential in developing an evaluation approach. Developing a theory of change is a useful way to do this.

A theory of change is logic model. It visualises how a change, as a result of an initiative or approach, is expected to happen.

• Outcomes and impacts tell you the key things that should be measured in your evaluation.
• Inputs, activities and outputs can be monitored to keep a record of how the project has progressed and to help explain any impacts.

Below is a theory of change template showing what should be included in each section.

Appendix III is the theory of change for the CPAF programme. Please use this as a template for your own project’s theory of change.

Inputs

Resources needed to implement the project.

They include:

• Funding
• Partners
• Staff Time
• Technology

Activities

Actions that must be done to achieve the proposed outputs

Outputs

What is done / offered

Outcomes

Results that are directly meeting programme objectives.

Usually split into:

• Immediate / short-term
• Medium-term
• Longer-term

Impacts

Societal-level changes.

Often categorized by:

• Economic
• Social/Cultural
• Health
• Environmental
• Political
• May be indirect

What types of data need to be collected?

After deciding what should be measured, it is important to think about which types of data must be collected to provide the evidence needed.

For each measure/ indicator (i.e., each outcome or impact), think about what type of data is needed to show whether or not and to what extent the objective has been met.

Questions to ask when thinking about the types of data are:

• Will quantitative (numbers), qualitative (words) or both be best?
• Does the data already exist (secondary data) or is there a need to collect new data (primary data)?
• Who or what needs to be engaged to collect the data?

An example of what to consider for each indicator. Adapt this for your project.

Indicator	Type of data (qual/quant)	Does secondary data exist?	What/ who to collect data from?	How (method)?	When/ how often?	Comments (risks, practicalities, etc.)
Number of priority families who access a service	Quantitative	Yes	The service's enrolment data	Download data from dashboard	Monthly	Already have data sharing agreement
Experience of priority families who access service	Mixed	No	Families that have accessed service	Interview	3 months after they access service	Need to gain consent to recontact

How can data be collected?

The final stage is deciding how data can be collected – the methods. Usually, multiple data collection tools will be needed to gather the necessary data.

There are many potential data collection methods. The table below outlines methods that may be useful for CPAF projects. It explains how and when they can be used, practicalities to consider, and links to other resources.

CPAF projects are free to choose other methods where relevant.

When designing data collection tools, it is important to keep the needs and preferences of the user in mind. This means ensuring that data collection does not take too much time, is sensitive to people’s needs and preferences, and is suitable for the purposes of the evaluation.

Where data is being collected from those with lived experience or the general public, it is essential that the Government’s Social Research Guidelines are followed. This means ensuring:

• There is a clear and defined user need and public benefit.
• The approach is based on sound research methods and protect against bias in the interpretation of findings.
• Data protection regulations are followed at all times.
• Participation is based on specific and informed consent.
• The research enables the participation of the groups it is intending to represent.
• The research is conducted in a way that minimises personal and social harm.

Method

Monitoring forms

Description

Monitoring forms collect regular data about project inputs and activities. Appendix I includes the suggested monitoring form that CPAF projects can adapt.

Practicalities

You can use monitoring forms at different stages of a project to document or highlight any changes in participation, engagement or experiences.

Method

Insight reports

Description

Insight or lightning reports communicate a snapshot of how a project is progressing at different stages of a project. This is the minimum level of data required for CPAF quarterly meetings.

Practicalities

Keep your insights short and simple and use a format that is accessible to different people.

Method

Interviews

Description

Interviews can gather in-depth information about a subject or a person’s experience and perceptions. They can be one-to-one or in a group with staff, people with lived and living experience or other project stakeholders.

Practicalities

Ensure the people you are interviewing are reflective of those who are impacted by your project. You should ensure to follow relevant ethical research standards. You may want to consider a trauma informed approach.

Method

Focus groups

Description

Focus groups are used to facilitate discussion, understand needs and perspectives, generate ideas and recommendations. They can also be used to test ideas.

Practicalities

Focus groups rely on having a skilled facilitator to support the group to navigate different discussions.

Method

Surveys

Description

A survey is a list of questions that can be used to gather qualitative and quantitative information from many people.

Practicalities

Ensure that your survey is accessible to different people. Consider the content, the format and the channel / platform you are using to deliver it.

Method

Lessons log

Description

A record of lessons that have emerged from the project.

Practicalities

Make a simple table and add lessons as the project develops, ideally following each project meeting or milestone.

Method

Creative or arts-based methods

Description

Creative or arts-based engagement methods support people to share their insights and experiences using creative outlets such as photography, drawing, or role play.

Practicalities

Can be an alternative to ‘traditional’ research methods or used alongside them.

More accessible for people who may experience barriers to communication.

Method

Storyboarding

Description

Storyboarding helps you to describe the different sections or steps in someone’s experience or in a service journey.

Practicalities

You can use storyboarding to inform your journey, experience and process mapping activities.

Method

Method

Description

Description

Practicalities

Practicalities

Method

Journey, experience and process mapping

Description

Journey maps provide a detailed view of service design and delivery, including the steps someone takes and the people, systems and channels they engage.

Experience maps describe a person’s experience over a long time period, across different stages or life events. Process mapping describes the ‘backstage’ service delivery activities.

Practicalities

Journey and experience maps should be updated and refined at different stages of a project to reflect any changes or improvements that have been made and their impact on different people’s experiences.

Some organisations may also have access to software that allows them to map organisational and service processes.

Method

Systems thinking tools

Description

Stakeholder maps visualise everyone who plays a role in and is impacted by a project. Ecosystem maps show relationships and interactions between each stakeholder.

Ripple effects maps visualise impacts of a service or programme beyond its intended objectives. Systems maps provide a snapshot of the existing system prior to, during or after an intervention.

Practicalities

Engaging stakeholders and those with lived experience is strongly recommended for a useful and meaningful output for systems thinking tools.

Systems thinking tools should be returned to and updated at different points in the project to understand and document any changes.

Section 4 / Appendices

Appendix I: Insight reports

Insight reporting will be used to collect insights data quarterly throughout the duration of the project. The Scottish Government will share this in advance. Quarterly meetings will be structured by the four key questions. Projects should reflect on the questions and be prepared to talk through the main points.

The four key questions are outlined below.

• Working like clockwork
• What has worked well and why?
• What are your main successes?
• Hitting a bump in the road
• What has not worked well and why?
• What challenges have you encountered?
• Getting back on track
• What did you do to overcome these challenges?
• What have you done differently than planned?
• Sharing the knowledge
• Do you have any key insights, thoughts, or learnings?
• Is there anything else you need to move forwards?

Insight reports:

• provide real-time, actionable insights on what is causing a project to meet (or not meet) its objectives.
• should be collected regularly to show patterns over time.

Data should be collected from all involved in the planning and delivery of the project. You can then look across to understand critical success factors and barriers.

You can choose the best way to collect data. This could be an online form, email, focus group discussions. You can also build the questions into regular project meetings, in this case, ensure you take formal and structured notes.

Appendix II: Monitoring Report

A template monitoring report is also available as an Excel Spreadsheet. As noted, the monitoring report is optional to complete.

Projects can adapt the attached template to collect quantitative/ numerical data about things that are important to the project.

It is expected that most projects will collect the following information:

• The number of staff/ resources used.
• The number of partners involved.
• The amount of funding used.

Projects can also record progress against project-specific objectives. This can include things like number of:

• Target participants engaged
• Target participants who benefit
• Secondary beneficiaries
• New partnerships formed
• Number of staff trained
• Amount (£) of benefits allocated

It is suggested that data is collected on a quarterly basis and discussed in update meetings with the Scottish Government. However, projects can also adapt the frequency of data collection as needed. There is no requirement to collect quarterly, but projects will be expected to report on relevant information in the final report.

Appendix III: CPAF theory of change

This is the theory of change for the whole of CPAF. Please feel free to edit this for a project by adding or removing points as needed.

Inputs

CPAF funding

CPAF staff

Fund recipients’ and partners’ staff

People with lived experience of poverty

People with professional experience of tackling poverty

Local systems

Evaluation partners

[ add project-specific inputs, such as technology]

Activities

Funding allocated to project partners

Evaluation framework developed

Capacity building workshops hosted

Project partners finalise approaches and initiate projects

Project partners collect data to support evaluation

Projects meet quarterly to update and share lessons with Scottish Government

Engagement with people with lived experience

[ add project specific activities]

Outputs

Community of Practice formed

Insights from evaluation shared

Local partnerships formed

[ add project specific outputs]

Outcomes

Knowledge about child poverty developed

Knowledge about how to tackle child poverty developed

Local response to child poverty strengthened

Local partnership working strengthened

Partners capability for evaluation improved

[add project specific outputs]

Impacts

Reduced inequalities

Improved community physical and mental wellbeing

Reduced unemployment and increased workforce productivity

Improved educational outcomes for children in communities

Improved housing stability

Policy changes

Additional political and financial support for child poverty

[add project specific outputs]

Appendix IV: Additional resources

Child poverty resources

Resource

Executive summary for Best Start, Bright Futures

What does it help with?

Sets out Scotland's national mission to tackle child poverty. It is not solely a plan for the Scottish Government, it is a plan for all of Scotland, recognising the contribution all parts of society must make to deliver the change needed for children and families.

Resource

Child poverty system map developed by the Scottish Government

What does it help with?

The system map shows the range of factors that influence child poverty, and the relationships between them.

Resource

Year 4 review of Local Child Poverty Action Reports

What does it help with?

Summarises key trends and actions undertaken by local areas to reduce child poverty over the period of April 2021 to March 20222 to that key learnings can be shared and lessons learned.

Resource

An overview of poverty amongst priority family types

What does it help with?

An overview of evidence on the six priority family types identified as being at higher risk of child poverty.

Resource

An approach to reporting evidence using the priority family types

What does it help with?

This annex sets out the position of priority families in regard to available evidence, and data on the causes of poverty and effective measures to tackle it.

Resource

Guidance on developing a Local Child Poverty Action Report

What does it help with?

Guidance for local authorities on developing a local child poverty action report.

Resource

A report assessing system change in the child poverty system

What does it help with?

This report represents the first assessment of system change in the child poverty system. This report aims to identify and understand the range of approaches taken to system change and person-centred support in local child poverty systems.