Learning Connections Adult Literacies Phase-Two Pathfinders: Evaluation Framework

6. Conclusions

In this section we set out the main conclusions we have drawn following completion of the project. First, we take the project as a whole, and then we take each of the different elements of the project and set out separate conclusions for these.

6.1 General

Prior to being appointed to this project, we often commented that a fundamental problem with many project evaluations was that they came too late. By this we meant that the lessons learned often came after a project was finished. While this provided valuable information for others, it meant that those involved in the original project rarely gained the benefits of the process for their own projects. We were therefore encouraged by what we perceived as an enlightened approach that was being adopted by Learning Connections with the Second Phase Pathfinders.

In general, we conclude that this premise has been borne out with this particular project. Our overall impression is that the Evaluation Project has contributed to a process whereby each of the Pathfinders has critically reviewed its progress and that the process of self evaluation has introduced a rigour to this which would not otherwise be there.

At the same time, and we develop our thinking on this in more detail in the sections below, we would conclude that the approach we have developed has been possibly over mechanistic and over-relied on a 'reporting' process, rather than a more reflective one. While the evaluation framework and toolkit were designed to encourage self assessment, we generally conclude that they have been seen and used more as a processes of 'justification' and 'accountability' than as 'developmental' ones. In the next section, Recommendations, we set out ways in which this lesson might be applied in the future, with other projects. In general however, we feel that more emphasis (on our part), could have been placed on developing and supporting an internal process within each of the Pathfinders which could have been more participative and reflective. However, had this approach been adopted, we would have been less likely to have the legacy of evaluation resources which have been developed during the project.

Another significant factor worth commenting on has been the general process of establishing and implementing the Pathfinders. While we (Communities Scotland and the consultants) were keen to ensure that the evaluation process was embedded in the projects from the outset, we perhaps underestimated the difficulties and timescales that the projects would have in getting themselves 'up and running'. For example, recruitment of suitable staff proved difficult for some of the projects. Others such as Farm Plus had difficulties setting up IT links with learners. Consequently, and understandably, the focus of the project staff and the sponsoring agencies was on the practical establishment of the individual projects and less so, on the evaluation processes.

While not really a subject of the Evaluation Project or this report, it is worth commenting, in passing, that one of the underlying difficulties that the whole Pathfinder project experienced was a general shortage of suitably qualified and experienced staff to resource the Pathfinders. Across the country as a whole, the Pathfinders were creating a significant demand for approximately 12-15 relatively experienced staff in the field of literacies. Our perception was that this in itself had "over-heated" the market and that some projects had difficulties in attracting and recruiting suitably qualified staff. In the end, however, it is worth commenting that the projects were ultimately able to recruit and generally retain good quality and experienced staff.

6.2 Evaluation Framework

The evaluation framework and the criteria developed to be used in the assessment of the individual projects has, we conclude, provided a useful basis for this. We can see however from the final project reports, that again understandably, the projects have applied more importance to some criteria than others. For example, criteria such as 'Project Management' and 'Best Value' have probably been the least applied or have had a detailed assessment made against them.

On reflection, we would conclude that the evaluation framework could be improved by a number of measures. First, it could be simplified and the criteria used by each project reduced. Alternatively, we should have used a matrix approach where by each of the projects developed their own evaluation framework, using the suggested criteria but prioritising them thus:

Project	A	B	C	D	E	F	G	H
Evaluation Criteria
Impact	1		1				1	3
Innovation	3	4	2	1			3	4
Resources	2		3	2	1		2	2
Partnership		2			2	1
Awareness	4	1		3		2		1
Recording				4	3
Management			4			4
Value		3			4	3	4

(Top Priorities ranked 1-4)

This would enable the projects to 'tailor' the framework more closely to the objectives of their projects.

This would also enable the framework to embrace more criteria which could be used (optionally) to assess projects against, without making the process more elaborate or complex. For example one criteria we referred to in our proposal, but ultimately was not included in the framework was that of 'sustainability'. Paradoxically, this was an issue which became a pre-occupation with many of the projects and, with the benefit of hindsight, is probably more important than some of the criteria developed.

6.3 Evaluation Toolkit

The toolkit, we conclude provides a useful set of resources which could be made more widely available. The range of 'tools' is comprehensive however, it would be possible to develop this further to include one or two more which would assist further with assessing projects against all of the evaluation criteria. Originally for example the first version of the toolkit included a project financial monitoring 'tool'. This was withdrawn and replaced with the existing pro-forma developed by Communities Scotland. Similarly, we can now see that it would be relatively easy to develop a 'tool' which would assist projects assess their approach to, and competence in 'project management'.

Indeed, building on the conclusions on the restructuring of the Evaluation Framework, the toolkit could be redesigned so that there are individual tools relating to each of the evaluation criteria and projects using this could then select the appropriate tools. The existing guidance with the toolkit does indicate that projects should use it selectively, but with a few design changes, this could be clearer and easier to use.

The only significant difficulty that projects reported with the toolkit, was the issue which pre-occupied the Advisory Group in the early stages of the project - the differentiation between Project Outcomes and Project Outputs. While efforts were made to clarify this and illustrate the differences with examples, this is an issue which could be clarified or simplified further.

It was also gratifying from the dissemination event that some of the projects had gone on to use more multi-media approaches to record the work of their projects and learner experiences. The toolkit relies very much on written reporting and recording and it would be useful, in any refined version of this, to incorporate tools which give advice on the use of multi-media.

Below, we develop further, in Section 6.6, the potential for the toolkit to be used "on-line". Our conclusion is that at this stage, the toolkit is relatively unsophisticated i.e. a set of proformas and guidance that is downloaded, completed and then submitted to Communities Scotland. Further development could include more inter-active tools - for example the development of an interactive approach to assessing learners' progress and attitudes which could potentially be very valuable e.g. online survey questionnaires for completion by learners.

6.4 Consultancy Support

As we comment above, projects have varied considerably in their uptake of the offer of consultancy support. Our overall conclusion is that this has been a valuable aspect of the project. Inevitably it was up to each project to assess what use they would want to make of this and was not imposed on them.

Again, with the benefit of hindsight, it might have been possible to build into each of the projects, an obligatory quarterly review workshop, facilitated by the consultants which would help the projects to focus on reviewing project progress and supporting them to overcome any problems or issues they were experiencing.

6.5 Development Workshops

The overall conclusion based on both observation and feedback from the projects is that these have provided a valuable opportunity for the projects to interact and share experience of the issues and problems they have each faced. We are aware that the earlier workshops, because of time constraints, limited this and this has been addressed in the latter workshops held.

A suggestion made at the outset was that the workshops could be hosted by the individual projects, in order that all of the projects could develop a better understanding of the contexts and environments in which each of them operate. However for practical reasons, i.e. the geographical spread of the projects, this has proved impractical. In retrospect, it might have been worth making more effort to overcome these practical obstacles, with a format of quarterly meetings, hosted by the individual projects. Saying that, it has been difficult to get the commitment from all the projects to participate in the workshops we have organised - not because these were not seen of value, but just for practical and other workload reasons.

6.6 Web Site

The Pathfinders web site is we conclude a very valuable resource, with potential to be developed further. As we say above, the limitations on its use have been largely as result of the external constraints that have been imposed.

In relation to the potential to develop the web site, there is still scope for it to contain, for downloading, the resources that have been developed by the individual pathfinders. In one case, (Farm Plus) these is a link to the project's web site where the project's resources can be accessed. Where projects have their own web sites, then the Pathfinders web site could signpost these. In other cases, it should be possible to download resources directly from the Pathfinders' web site.

We would also conclude that there would be value in integrating the Pathfinders web site within the Communities Scotland web site and transferring responsibility for updating it.

At the same time, with the conclusion of the second stage Pathfinders, it is unlikely that there will be much more content added to the site. There is still scope for posting the Project's final reports and possibly the independent evaluation. It would therefore make sense to consider ways of incorporating the information, lessons learned and the resources developed with Communities Scotland's own web site.

6.7 Dissemination

Within the context of the Evaluation Project, the main dissemination event was the final conference, held in March 2006. As we say above this proved to be a relatively successful event and was well received by participants. However, we were disappointed by the significant level of 'no shows' i.e. delegates who had booked on the conference but failed to turn up. Below we make recommendations on how this problem might be addressed in the future.