Learning Connections Adult Literacies Phase-Two Pathfinders: Evaluation Framework

A ppendix I

ADULT LITERACIES PATHFINDERS
EVALUATION FRAMEWORK

May 2004

1. Introduction

The aim of this document is to provide a framework which can be used to assist the eight Second Stage Adult Literacies Pathfinders to evaluate the effectiveness and impact of their projects. Along with this has been produced an "evaluation toolkit". This provides the Pathfinders with a number of practical guides to assist them undertake different aspects of project evaluation.

First the framework sets out what it is each Pathfinder is being evaluated on. It then describes the different methods used in evaluation and then provides a list of the contents of the Evaluation Toolkit.

The evaluation framework has also been developed to be consistent with the Learning Evaluation and Planning ( LEAP) scheme. LEAP¹ was devised for the Scottish Executive to be used by partners in community learning.

2. What is being evaluated?

Following consultation with representatives of each of the Pathfinders, the main objectives of each for the evaluation were identified. While each Pathfinder had its own specific objectives these could be categorised under a number of headings. These eight criteria have been incorporated within the Evaluation Toolkit documents and are:

• Impact on the Target/Client Group
• Demonstration of Innovation
• Resources Development
• Development of Inter-Agency Partnership Working
• Raising Awareness of Literacies Issues
• Recording and Dissemination
• Project Management
• Best Value

2.1 Impact on the Target/Client Group

Clearly one of the main objectives is to make a difference to the individuals (and the organisations) with whom the Pathfinders are working. In terms of individuals, a key objective is the development of literacies and associated skills. The evaluation of many of the Pathfinders will require them to be able to demonstrate individual learner progress. The use of tools for self-evaluation could be important in this. Many of the projects already use Individual Learning Plans which could demonstrate individual learner progress.

This objective also focuses on both the ability of the Pathfinder to engage with the client group, involve significant numbers as well as providing the quality of education, support and resources which will lead to improvement.

User involvement is a priority within this objective. Users should be involved in the planning, delivery and evaluation of projects with involvement being demonstrated by:

• Learners being involved in designing materials and evaluating the methods used by the project.
• Feedback that learners have become more involved via new partnerships e.g. voluntary groups.
• Learners' voices having a place so they can help change the culture within organisations

Underlying the Pathfinder programme is the importance of targeting a number of "priority" groups. This aspect of the evaluation focuses on the ability and effectiveness of the individual Pathfinders to do this.

As well as having the objective of the development of literacies, a number of the Pathfinders have a number of specific objectives which are related to this, but can be described as "secondary". This does not mean that they are less important - only that the development of literacies in the individual should bring about other benefits. For example one Pathfinder is concerned with helping young people to sustain their tenancies and move towards independent living, another is concerned with bringing about improvements in the health of the group it works with and another focuses on helping parents develop the educational abilities of their children.

LEAP describes two levels at which these impacts can be analysed. "The first level identifies the major dimensions of each activity. We identify four for personal development

• Identify and engage the participants
• Create empowering learning
• Support sustained involvement in learning
• Support the transfer of learning

We identify five for community capacity

• Work with communities to plan and address their needs
• Support the development of skills and confidence of activists and organisations
• Promote broad based participation in community affairs
• Assist communities to exercise power and influence
• Assist communities to provide or manage services

The second level identifies what is involved in each dimension. We call this level the elements. For example in the dimension: Identify and engage the participants, we have elements like: establish contact with groups and organisations; gather information on perceived learning needs, motivation and aspirations." ²

2.2 Demonstration of Innovation

For all of the Pathfinders, the scope to demonstrate that their projects were innovative in their approaches is also seen as a key objective. Innovation encompasses a wide range of aspects including new approaches to engaging and sustaining the involvement of the target group, new materials, teaching methods and the use of technology.

2.3 Resources Development

For a number of Pathfinders the production of resources which help develop the literacies with their target group is seen as an important objective. These may be in the form of text based, audio visual media and interactive IT based. This objective is of particular priority to Learning Connections, enabling some of the important benefits of the pilot programme to be disseminated to other projects and services both within Scotland and further afield. The process of developing resources should also be part of the lessons learned from the project. The importance of developing quality resources which address barriers of accessibility and readability are crucial.

2.4 Development of inter-agency partnership working

An important theme for all of the Pathfinders was the ability to develop inter-agency partnership working. This is seen as important at a number of levels. First to maximise the resources that can be used in the delivery of literacies support to the client group, secondly to provide opportunities for "progression" for the learner and thirdly to engage with other agencies which are in contact with people who would benefit from literacies support and teaching.

The LEAP model is also concerned with ensuring that partners develop a shared perspective and understanding of the outcomes of the project and that this must be built in at the planning stages of the project in order that this can then be evaluated.(See for example LEAP handbook p.12)

While it is often recognised that effective inter-agency partnership working can be difficult to achieve, it is also recognised the importance of providing "joined-up" services especially to particularly disadvantaged or vulnerable client groups.

Partnership working is also an important aspect of ensuring the longer term sustainability of the Pathfinders, where other partners are a source of future support and resources. However, there is a need to have a shared understanding of how the level and effectiveness of Partnership working is measured.

It will be important to measure the effectiveness of these progression opportunities.

2.5 Raising awareness of literacies issues

Related to partnership working, is the objective of raising awareness of literacies issues among other agencies, groups and professionals. Projects such as the Great Northern Partnership have as a main objective, raising awareness of literacies issues and provision among health workers within the Social Inclusion Partnership area. Similarly projects such as Links to Literacies aims to raise awareness of the potential of voluntary and community organisations to be involved in the developments of literacies support. It is important to discern the impact of raised awareness on the activities of these organisations.

2.6 Recording and Dissemination

There is little point in a pilot programme such as Pathfinders if the experience of the individual projects is not well recorded and ultimately disseminated to others involved in literacies. While this objective is generally shared among the Pathfinders, it is of particular importance to Learning Connections, one of whose main tasks is to ensure that the lessons, both positive and negative from each of the Pathfinders can be used to improve the quality of literacies provision throughout the country. There is also value in dissemination between projects, which will be facilitated by the development of a website for Pathfinder projects.

2.7 Project Management

A theme that several Pathfinders stressed was the importance of effective project management. This covers several aspects of the project - for example its effectiveness at recruiting skilled staff, good reporting systems and financial control. It is also concerned with issues such as ensuring good access to the project's services for the client group. This can range from ensuring good physical access to premises through to ensuring that the project is promoted using accessible information.

2.8 Best Value

The Pathfinder programme represents a significant investment of public resources. This objective is concerned with being able to demonstrate that this investment is well used. Best value, like "value for money" is not however concerned with "cheapness". It is about demonstrating that the outputs and lessons from the projects would have been difficult, if not impossible to achieve with fewer resources and "inputs" than by other means.

Inputs (using the LEAP model), include:

• Community inputs - including skills, commitment (including voluntary effort), and motivation of local people)
• Partner inputs - from the main provider and its partner agencies including staff time and skills, buildings, equipment, services
• External inputs - in the form of government grants, support from external agencies

This aspect of the evaluation is also concerned with "the additionality" and support that each organisation sponsoring a Pathfinder can bring to the project.

3. How is Evaluation carried out?

In this section, the main methods of project evaluation are described. Different methods are used to evaluate different objectives. Two key concepts in the approaches to evaluation are formative and summative.

Formative evaluation is essentially concerned with assessing the processes and actions undertaken during the course of a project. The main focus is on monitoring progress and where using the results of this to adapt and develop the project during its life. Given the experimental nature of the Pathfinders, this aspect of the evaluation recognises that some aspects of the project will work, others may not. Success based on the ability to critically reflect on the work of the project, assess what works and build on this, what doesn't work and change this.

Summative evaluation is more concerned with "looking back" at a project following completion and assessing the overall impact of the project and the lessons which can be learned for the future, both for the project as well as others.

The Pathfinders will be subject to an independent external evaluation which will focus on the summative. The main focus of the work that individual Pathfinders will concentrate on is on the formative, although the data and information gathered during this stage of the project's life will also inform the final evaluation.

3.1 Baseline Data

In order to evaluate the impact of each Pathfinder it is important to have an understanding of the position within the community or target group the project is working with, prior to the project commencing. In some cases this information is relatively available and accessible, in others less so. The baseline data required for each Pathfinder will obviously vary but generally can be covered by the following headings:

• Area/community covered by Pathfinder
• Population/nos. of client group e.g. nos. of people with learning disabilities in area, no of asylum seekers
• Why literacies is an issue within the target group
• Evidence of the impacts of low literacies levels within the client group e.g. tenancy turnover, poorer health, higher levels of unemployment, poorer school performance of children

3.2 Statement of Project Objectives and Action Plan

The next step in the evaluation process is for each project to have a clear statement of its objectives and a plan to translate these into specific action and outcomes. This will also include an indication of the target timescales to achieve these. This should be produced by the end of September and sent to the Learning Connections Co-ordinator.

3.3 Progress Reporting

Pathfinders are required to provide periodic progress reports. These will record the action undertaken over a fixed period and the progress towards the Pathfinder's objectives. It is envisaged that periodic progress reports will be required quarterly, with these consolidated in an annual report. The first quarter report should cover the period October - December 2004) and submitted to the Learning Connections Co-ordinator by mid January 2005. Following this quarter, there will be a workshop in late January to review project progress, the experience of evaluating the first quarter of the project and to refine the evaluation framework in the light of this experience.

3.4 Final Reporting

Each Pathfinder should produce, close to its completion (dates for this will be agreed with each Pathfinder and their Learning Connections Co-ordinator), a final report. This will be a more detailed account of the project and an assessment of its performance, along with evidence of its work and achievements, the difficulties experienced and lessons learned. The final report needs to, as far as possible, provide quantitative data about the project. It can also provide a qualitative assessment of the project through the use of case studies which can provide evidence of the progress made both by individual learners (and clients) as well as the projects.

3.5 Learner Involvement

Feedback from learners involved in the Pathfinder will be an important element in any evaluation. Learner involvement can come in a number of ways. First, there is a need to be able to record individual learner progress. Second, the learner can be involved in providing feedback about the project, its benefits and how it might be improved. Thirdly, the Pathfinder may be able to develop ways in which learners are involved in the future development and management of the Pathfinder.

3.6 Consultation with Stakeholders

In assessing the impact of the Pathfinder overall, as well as the effectiveness of developing inter-agency partnerships, consultation with other stakeholders and partners is a central aspect of the evaluation. This can provide information on the success of increasing awareness, levering in additional resources and assessing the long term sustainability of the service.

4. The Evaluation "toolkit"

Accompanying this framework document are a number of "evaluation" tools to assist Pathfinders build up a comprehensive evaluation of their projects. It should be emphasised that not all "tools" will be applicable to every Pathfinder. Instead each Pathfinder can use those which best help it, or can adapt these to suit their individual circumstances.

The "tool kit" comprises:

4.1 Baseline Data Proforma

This is a short and simple proforma in which the baseline position prior to the establishment of the Pathfinder is recorded.

4.2 Statement of Project Evaluation and Action Plan

This is a summary in tabular form of the Pathfinder's objectives. This is based on the LEAP model, providing a framework for planning and evaluation of the main "elements" of the Pathfinder. While the LEAP model focusses on two underpinning aims, Promoting Personal Development and Building Community Capacity it is likely that most Pathfinders are predominantly concerned with the former, although some will also have community capacity building as their primary aim e.g. Learning Links

4.3 Quarterly and Annual Progress Report Proformas

These provide a simple monitoring and reporting system which takes the objectives of the Pathfinders and enables them to provide both their stakeholders and funders with an easy to assess account of their work over a particular period.

4.4 Final Report Proforma

This is a more detailed proforma which covers the development of the Pathfinder, its work and an assessment of the extent to which it has met its objectives. It also gathers together the views of project users, partners and other stakeholders and sets out any lessons learned for the future. The final report may also provide a detailed case for future support for the project.

4.5 Case Study Guidance

The toolkit provides guidance on the preparation of case studies for incorporation in periodic and final reports.

4.6 Client Satisfaction Interview Schedule

This interview schedule is designed to complement other forms of learner progress recording. The main aim of this schedule is to obtain learners' view on the project overall and ways in which it might be improved in the future.

4.7 Partner Consultation Interview Schedule

This schedule will enable Pathfinders to gather information from its partners and other stakeholders in a consistent way. It will also enable the Pathfinder to identify future potential of support and any action they may have to take to obtain this. It will also enable them to assess the extent to which the Pathfinder has raised awareness of literacies issues among other partner agencies.

5. Support with Evaluation

An overall aim of this evaluation framework is to provide support to each Pathfinder to undertake its own evaluation. The "toolkit" is being designed to minimise the additional work that each Pathfinder has to undertake, supplementing the development work being undertaken by each project. There is additional consultancy support for each Pathfinder, to assist with the evaluation process.

Support can be provided to adapt this framework to the needs of individual Pathfinders, fine tune any of the parts of the toolkits, and where necessary develop new tools specific to an individual project.

To obtain any assistance in relation to project evaluation, contact Anne Murray or Tony Graham at ODS Ltd - Tel 0141 424 3765 or e-mail admin@ods-glasgow.com

There will be a review workshop which will assess the experience of all Pathfinders on the evaluation process at the end of January 2005. Details of this will be sent out in early January.

TG/ab
ODS
June 2004