Learning Connections Adult Literacies Phase-Two Pathfinders: Evaluation Framework

4. Review of Project Activities

In this section we take each of the elements of the Evaluation Project and review the work carried out under each.

4.1. The Evaluation Framework

The underlying principle of the Evaluation Project as a whole was to engage with the Pathfinders and develop the project in consultation with them, rather than externally imposing it. This principle informed the development of the Evaluation Framework. A consultative workshop was held in 2004. This brought the eight pathfinders together with the Learning Connections Project Managers, support staff and the consultants working on the evaluation project.

The first priority for this workshop was to determine the criteria which would be used to evaluate the Pathfinders. A challenge for this part of the exercise was that each of the Pathfinders was quite different in terms of aims, target groups areas of operation and the approaches being developed. One aim of the evaluation project was to develop a framework which could be used by all eight pathfinders. It was recognised however that not all of the evaluation criteria developed would be of equal relevance to each of the Pathfinders. However it was acknowledged and accepted that overall, the pathfinders would aim to use all of the criteria, to a greater or lesser extent in their assessments of their projects.

Following consultation, it was agreed that eight evaluation criteria would be used

4.1.1. Impact on target group

Projects would evaluate themselves in relation to the impact they made on the target group. By this was meant their ability to engage with the particular group that was the focus of the Pathfinder, their ability to involve members of the target group and the overall impact they made in the development of literacies skills and awareness within this group. It was recognised and emphasised that by their very nature, the Pathfinders were more concerned with experimentation and developing new approaches and not purely with attracting large numbers of learners. At the same time, it was agreed that a realistic expectation of the Pathfinders was that this would involve sufficient numbers to be able to develop their approaches and demonstrate that key lessons had been learned which could be either replicated elsewhere or used as the basis of developing new projects in other areas and agencies.

This criteria highlighted the need for each of the projects to establish some very basic baseline date, which could be used to help them assess their success in relation to impact on the target group.

4.1.2. Demonstration of innovation

A key objective of the Pathfinders project as a whole was the use of innovative approaches to the development of literacies skills and of promoting awareness of literacies issues. Hence, it was considered vital that the individual projects should be able to demonstrate how the project was in some way innovative. Innovation was defined in this context, in its broadest sense i.e. it might apply to teaching and learning methods, identifying and attracting a particular target group, involving members of the target group in the development of their own learning or in the development of the project as whole. Essentially, the Pathfinders agreed to ask themselves 'what is new and different about our approach?'

In some respects, this was one of the simplest yet most challenging criteria for the Pathfinders. In some respects, innovation was self evident - it was one of the criteria which was used in the initial selection by Learning Connections for funding. However it was considered important that innovation was not only a feature of project design but that it was also sustained throughout the implementation and lifetime of the project

For some of the projects, it was relatively easy to see what was new and different. For some, there was a feeling that 'innovative' meant 'revolutionary'. It was important therefore that these projects were given support to clarify their own thinking in relation to innovation and that this could apply to a relatively simple aspect of the project - for example was the approach that was being used to identify and engage with a particular target group one that had been used before.

4.1.3. Resources development

Each of the Pathfinders had among their objectives, the development of resources for Literacies. For most, the focus was on the development of teaching/learning resources. This was not however the case for all of the projects. In those projects which were more concerned with promoting awareness of the need for Literacies provision or improved partnership working or co-ordination of agencies, resources development could apply to promotional material or awareness raising resources. It was also recognised that the production of 'physical resources' was potentially one of the most valuable aspects and legacies of the Pathfinder programme as a whole, with the potential for wider spread dissemination and application.

This criteria again was interpreted very flexibly and again it was agreed that it need not reflect 're-inventing the wheel' but could also be applied to the processes of adapting existing teaching or promotional materials and applying these in a new or different way.

4.1.4. Development of inter-agency and partnership working

Each of the Pathfinders recognised that their impact would be heightened by their ability to work with and engage a number of different partner agencies in their projects. This was considered important for a variety of reasons. First, increased partnership and inter-agency working would ensure that the resources that were being put into the individual Pathfinders could 'lever in' additional support and resources from these agencies. It was also recognised that in terms of Literacies development, the projects were by and large dealing with particular client groups at a particular stage of their literacies development. The projects were part of a spectrum of learning provision and it was critical that other agencies were aware of their work and could develop complementary services e.g. follow-on provision, additional support to learners or as in some cases, building support for the long term sustainability of the project.

In those projects where the main focus was developing networks of service provision or raising awareness of Literacies issues, the formation of partnership work and improving inter-agency working were particularly important criteria to be used for assessing success.

4.1.5. Raising awareness of literacy issues

Another of the key objectives for the Pathfinders as a whole was to raise awareness among a range of target audiences of the nature and extent of literacies and the difficulties many adults experienced with these. With the exception of two of the projects (Links to Literacies and Learning and Health) the remaining projects did not have this as a central objective. It was agreed however that this should at least be a 'spin-off' of each of the projects.

4.1.6. Recording and dissemination

As a pilot project, it was considered particularly important that each of the Pathfinders should establish and use of systematic approaches to recording their work and reporting on this. This was to enable the projects themselves to reflect on their work as well as provide useful information to both Communities Scotland and the wider literacies and community learning agencies and partnerships to benefit from the lesson learned from each. The dissemination strategy was two fold. First that the information recorded and the experience would be part of an overall dissemination strategy. At the same time, each of the Pathfinders agreed that it would be important to disseminate the lessons learned within their own area of operation and among the other local agencies and partners who had had an involvement in the project.

4.1.7. Project Management

This criteria was based on a recognition that there would be little value in the Pathfinders having innovative ideas on how to tackle literacies challenges within a wide range of target and client groups if they were unable to effectively deliver the projects 'on the ground'. This criteria therefore focused on the practical aspects of running the project. For example it would evaluate itself in the overall administration of the project - how effectively for example it recruited staff, deployed resources and generally managed the project.

4.1.8. Best Value

The final evaluation criteria was, one of the most difficult to assess. However it was agreed among all the stakeholders in the Pathfinders that there should be an attempt to assess the value the projects provided not only in educational terms but also in terms of the investment of resources that had gone into each of the projects. To what extent did that investment provide best value and value for money? With an overall investment of £1 million in the project it was important to form a view on whether this was being spent well or whether it may have made a greater impact used differently.

The evaluation framework document in full is included as Appendix 1.

4.2. The Evaluation Toolkit

The second main objective of the Evaluation Project was the development of an 'evaluation toolkit' for use by each of the Pathfinders. Again this was a task that would be carried out in consultation with the stakeholders including the eight pathfinders projects. The underlying principles were that the 'toolkit' should be uncomplicated and relatively easy to use. It should enable each of the Pathfinders to evaluate their progress at different stages of development and implementation of the project and it should embrace a range of methods, both quantitative and qualitative to inform their evaluations.

At the outset there was some debate about the need for a new set of evaluation tools. Some of the Pathfinders were already aware of, and intended to use the Learning Evaluation and Action Planning ( LEAP) framework which had already been developed for the Scottish Executive.

This presented a challenge to the consultants who then had to develop the toolkit in a way that complemented this existing approach and avoided either duplicating or contradicting it. In the end, we feel that we were successful in this.

The evaluation toolkit comprises a number of elements, which Pathfinders could use at different stages to assess and report on their progress. The toolkit was reviewed over the first few months of the project and a number of amendments were made in light of the Pathfinders actually using it and feedback from the pathfinders project Advisory Group.

The toolkit is essentially a set of pro formas or guidance notes which covered:

Baseline Study and data: This would enable a consistent approach to the collection of the baseline data which was considered crucial as the benchmark against which each of the Pathfinders would measure their progress.

Project Evaluation Action Planning Framework: This provided the basis of planning the project in terms of objectives, outputs and demonstrating that these had been achieved. Within the toolkit, there was provided a "blank" proforma as well as a completed "example"

Quarterly and Annual Reporting Proformas: These are relatively simple proformas which were to be used to report on progress of each of the projects in terms of the evaluation criteria.

Final Report Proforma: This was the template that was to be used by each of the projects for the preparation of their final reports.

Case Study Guidance: Case studies were suggested as part of the qualitative evaluation of each of the projects. This part of the toolkit provides guidance on what was expected of a case study write up.

Client Satisfaction Interview Schedule: Leaner or client feed back was considered an important element in project evaluation. This "tool" was designed to provide projects with a relatively simple guide to the areas to be covered in a client / learner interview.

Partner Interview Schedule: Particularly for those projects which were concerned with developing network and partnership working, this tool was developed to assist the projects assess their partner attitudes and raised awareness of literacies issues and provision.

The toolkit is included as Appendix 2.

4.3. Development Workshops

The overall approach being used in the Evaluation Project was one which placed importance on the involvement of the Pathfinders in its development. At an early stage in the project, the projects agreed that there would be value in holding over the course of the project lifetime, periodic workshops which would have two objectives. First they would enable the individual projects to come together and share their experience of developing and implementing their projects. Secondly, they would enable the projects to consider in a "learning environment", how they might develop and improve their practice.

Over the course of the project, in addition to the initial workshops used to develop the evaluation framework, two further workshops were held. On a practical note it is worth recording that there were a number of difficulties associated particularly with the organisation of the latter workshops. Originally two workshops were scheduled for June and August 2005. However, for a variety of reasons it provide impossible to get the level of commitment and involvement for these. Eventually the themes for these two workshops were combined in a single workshop session, held in November 2005.

Workshop February 2005: this workshop provided each of the Pathfinders with an opportunity to provide updates on their progress and identify any particular obstacles to this. This workshop was generally considered very useful by the individual projects. However, the main difficulty we experienced was that with each of the eight Pathfinders giving, in some cases very detailed presentations about their work to date, it was difficult to allocate the amount of time to discussion between projects which would have been desirable.

Workshop November 2005: a number of "development themes" that had been identified with the Pathfinders during their development period. This workshop focussed on two of these themes. First, how could they provide evidence that there was learner progression i.e. that individual learners involved in each of the project were making individual progress. This first theme was introduced by Kathy MacLachlan from the Department of Adult and Continuing Education at the University of Glasgow (and a member of the Pathfinders Evaluation Project Advisory Group).

Secondly, there was also a recognition that the projects were generally the first contact that many adult learners would make in "returning" to learning. It was recognised also that there would be a need for progression or "moving -on" strategies.

It had also been envisaged that this workshop would also focus on the theme of "marketing strategies" i.e. how projects could raise awareness among other partners and agencies of their work, however it was eventually felt that to incorporate these them would be to "over-load" the workshop and reduce the time available for participation by the individual project representatives.

4.4. Consultancy Support

To complement the develop work we carried out on the Evaluation Framework and Toolkit, each of the Pathfinders were offered, if they wished or needed, consultancy support with their implementation. There was a considerable variation in the use the projects made of this, with some projects drawing quite heavily on this, some occasional use, and one or two, minimal if any use.

The main areas where projects asked for support included:

• Advice of collecting baseline data
• Advice on defining project outputs and outcomes
• Advice on dissemination strategies

Support was provided through a range of approaches, ranging from on site visits to projects, telephone support and in some cases, by e-mail correspondence.

We were keen to promote the availability of this support, and at an early stage in the project visited almost all of the projects. Those one or two projects not visited were those who had not yet been able to put in the staffing infrastructure or declined any support - we concluded that they, as they were perfectly entitled to do, just wanted to get on with it on their own.

4.5. Web Site

At the outset of the project, it was agreed that a web site for the project should be set up.

Initially it was proposed that this would effectively be included within Communities Scotland own web site, however for a variety of reasons it was agreed to establish the site on a stand alone basis. For example, this would make it much easier for the consultants to manage the content of the site and update this, without having to rely on the Communities Scotland IT staff. At the time, Communities Scotland's web site was also going through a major redesign and this was likely to delay considerably our ability to get the Pathfinders web site up and running.

The web site was launched early in 2005 and enable users to:

• Download monitoring forms
• Read FAQs about using the evaluation framework
• Read information and news on the Pathfinder projects
• Share experience and views through a forum
• Browse Reference materials and link to other useful websites
• Access project quarterly and annual reports
• Obtain information on Project events

See Appendix 3.

While the site did not contain any sensitive or confidential information, Communities Scotland's IT section insisted it was both a "stand alone" site and was password protected, for access only by Pathfinders or Communities Scotland staff or others approved to be given a password. In order to develop the site simply and quickly, we did not build in any provision for recording of "site visitors". The password protection also meant that the potential audience or number of users was very limited. Ultimately, we feel the potential for the web site has not been, to date, as fully realised as it might have been.

In order to simplify the site, the Evaluation Toolkit could only be accessed by downloading. Projects could then for example complete the various proformas and return these by e-mail to Communities Scotland. Were the project to be developed further and perhaps with the benefit of hindsight, the site should be developed so that project progress reports could be completed and returned on-line. Communities Scotland could potentially insist on, or at least encourage this - it has developed a similar and more sophisticated approach for use with Housing Associations who are encouraged to complete and submit their Annual Performance and Statistical Return on-line.

4.6. Dissemination Conference

At the workshop in November 2005, there was discussion with the eight projects on the overall dissemination strategy for the Pathfinders. It was agreed that while each of the projects would be working on local dissemination activities, a national conference should be held at which the work of the eight projects could be "show cased".

A conference planning group was established with representation from the projects, Learning Connections and serviced by the evaluation project consultants.

It was agreed that the conference should be targeted at Community Learning Partnerships and Literacies Partnerships, to whom invitations were issued.

In order to ensure that as many of the Pathfinder staff and representatives would be able to participate, the conference was held at the end of March, just prior to the formal end of the project.

It was agreed that the main focus of the conference should be a series of participatory workshops which would enable participants to gain a detailed appreciation of as many of the projects as possible. These would be complemented with a "projects marketplace" so that delegates who were only able to attend workshops on two projects could meet with representatives of the remaining six projects.

Participants were asked to complete and return an evaluation questionnaire. Disappointingly we only received seven returns. Of these all agreed that the conference day was "interesting and informative". All but one agreed that the event "made them think about their own approaches" and "is likely to influence future work".

Respondents indicated that the most worthwhile elements were:

• workshops
• making contact with practitioners
• learning first hand what methods worked and didn't work

There were a couple of negative comments (from the same person): 'too much choice' (!) and 'not enough time to view all the projects'

When asked if there was anything else they would like to find out about the Pathfinders - two people said that they would like to know where / how to access the resources which had been developed - another suggested making them available online.

One of the more useful by-products of the conference was the booklet containing descriptions of the eight projects. This is included as Appendix 4.