Evaluation of the Food Education Programme (2012-2015)

Background

Policy Context

16. Scotland's National Food and Drink Policy ^[1] , recognises that some children do not understand how food is produced and cooked, or how it affects their long term health and the wider environment.

17. In order to deliver the food education component of Scotland's National Food and Drink Policy, an extensive Food Education Programme (throughout this document referred to as FEP) was implemented in 2010.

18. By developing a greater appreciation of where food comes from, it is expected that the FEP will allow young people to make informed choices and contribute towards a healthier, wealthier and environmentally sustainable Scotland.

19. Learning about our food from plough to plate is also seen as a great interdisciplinary topic to deliver the experiences and outcomes of Scotland's Curriculum for Excellence (CfE).

Programme Overview

Scottish Government Funding of the Programme

20. The FEP was piloted between financial years 2010 and 2012. During that time the Scottish Government's Food and Drink and Rural Communities ( FDARC) division provided approximately £560,000 of funding supporting six projects (Futures in Food, Seafood in Schools, Eco Schools, From Farm to Plate, Dumfries House Food Learning Centre and The Children's Orchard) and an Education Scotland Food, Health and Wellbeing Development Officer from Education Scotland.

21. Following the successful implementation of the pilot, the FDARC division provided a further £2.7 million of funding for the financial period 2012-2015. This funding was given to a wider range of projects (Chefs@School, Crofting Connections, Dumfries House Food learning Centre, Eco Schools, Futures in Food, Food for Thought, From Farm to Plate and Seafood in Schools) as well as the Education Scotland Development Officer, which sort to increase young people's understanding of the food they eat, building on the work funded during the pilot years.

22. On completion of the 2012-2015 programme, funding was extended for a further two financial years (2015-16 and 2016-17) to the same projects, with additional funding support of £1.67 million.

23. Consequently, since 2010, just under £5 million has been allocated to the Food Education Programme by the Scottish Government.

Programme Operational Years

24. This document reports on progress that has been made towards achieving the programmes' outcomes over the period 2012-13 to 2014-15. Only interim data is available for 2015-16 of the programme, and as such evaluation of this year is not included in this report. A separate report provides a detailed assessment of the pilot years 2010-12.

25. Throughout this report, following references to the programmes operational years apply:

• 2010-12 = Pilot
• 2012-13 = Year 1
• 2013-14 = Year 2
• 2014-15 = Year 3

Programme Objectives and Outcomes

26. The overall aim of the Food Education Programme is:

'to increase young people's knowledge and understanding regarding the social, cultural, economic, health and environmental aspects of the food we eat.'

27. The following programme outcomes were set at the outset :

• Opportunities to learn about food are implemented
• Food education is embedded in the Curriculum for Excellence (CfE) and there are prospects for continued learning beyond the life of the programme
• Industry investment in food education is demonstrated
• Knowledge and awareness regarding food has increased, whether social, cultural, economic, health, environmental or potential for a career in the food industry
• Positive change in attitudes/intentions/behaviour can be demonstrated regarding food issues, food choices and career options

28. The target population of the programme are primary and secondary pupils as well as other young people, including teachers and the wider community.

29. Projects were selected following an application process from external stakeholders with interests and expertise in relevant food and drink topics. Projects received funding where aims and objectives were aligned with Scottish Government priorities for Food Education. During 2012-15, the following projects were funded [note links to relevant websites provided were applicable]:

• Chefs at Schools [Federation of Chefs]
• Crofting Connections [Soil Association]
• Dumfries House Food Learning Centre
• Eco Schools [Keep Scotland Beautiful]
• Futures in Food [Scottish Food and Drink Federation, SFDF]
• Food for Thought Fund [Education Scotland]
• Food and Health Development Officer [Education Scotland]
• Royal Highland Education Trust, RHET
• Seafood in Schools [Seafood Scotland]

30. A summary overview and aim(s) of each individual project is provided in table 1.

Table 1: Food Education Programme Projects included in evaluation - overview and aims

Project [including total funding for period 2012-15]	Overview	Aims
Chefs at Schools - Federation of Chefs [£100,000]	Supported by the Federation of Chefs Scotland (not for profit organisation). Envisaged that chefs and cooks can have significant influence as leaders on food culture.	Aims to encourage culinary and food industry professionals to work with teachers and children across Scotland to bring food education to life.
Crofting Connections - Soil Association [£90,000]	Run jointly by the Soil Association Scotland, Scottish Crofting Federation and HIE. Enables young people living in crofting communities to learn about crofting (past, present and future).	Aims to increase children and young peoples' understanding of the connections between crofting, food, health and the environment; supports school and community local food-growing initiatives
Dumfries House Food Learning Centre [£225,000]	Dumfries House Trust is a charitable organisation located in East Ayrshire. It built a new Food Learning Centre to deliver food education opportunities for children and teachers.	Facilitates the teaching of pupils about how their food is produced, where it comes from, and how it affects their health and the environment
Eco Schools - Keep Scotland Beautiful [£225,000]	Inclusion of food as a theme in Eco Schools programme	To implement the new Food and Environment theme as part of the overall Eco Schools programme.
Futures in Food - Scottish Food and Drink Federation ( SFDF) [£270,000]	Futures in Food is a national programme coordinated by SFDF which brings together food industries and schools in partnership using food as a topic within CfE to allow pupils to learn about food and careers in the food industry.	Increase young people's awareness of careers in the Food and Drink Industry and inspire them to make the connection between food on their plates, learning in school and employment in the F&D industry.
Food for Thought Fund - Education Scotland [£1M]	Supports programmes that allow schools to work in partnership with industry/businesses which have a link to major cultural or sporting events e.g. the 2014 Commonwealth Games.	To support practitioners to build on/enhance existing practice in food education
Food and Health Development Officer - Education Scotland [£200,000]	The Education Scotland Food and Health Development Officer has a wide remit to support development of food education across schools in Scotland. Main tasks include: organisation of events, development, distribution and promotion of resources, offer and provide general support to teachers.	To provide support/resources to teachers and others in the private/public sector to use food as a topic for interdisciplinary learning.
Royal Highland Education Trust [£162,576]	Supports schools in teaching about food, farming and the countryside, through food and farming days and school farmer's markets and the education programme at the RHS.	Educate children & teachers about food and farming and create strong partnerships between RHET, farmers, schools and other food education partners.
Seafood in Schools - Seafood Scotland [£225,000]	Creation of collaborative clusters between the seafood sector and schools to promote learning on seafood.	To use seafood as a topic for interdisciplinary learning, delivering the experiences and outcomes of CfE and enhancing children's knowledge and understanding of the seafood supply chain

Programme Evaluation

31. Overall, a total of £2.7 of funding was allocated from 2012-15 to increase young people's knowledge and understanding regarding the social, cultural, economic, health and environmental aspects of the food we eat. In order to evaluate the impact this funding has had on the desired outcomes, an evaluation framework was designed and implemented.

32. Different types of evaluation methodologies were considered. These are summarised briefly in the table below.

Table 2: Types of Evaluation

	Process Evaluation	Impact Evaluation	Economic Evaluation
Question to answer	'How was the intervention implemented and delivered?'	'Did the intervention actually work (i.e. generate the expected impact)?'	'Was the impact worth the financial investment (i.e. value for money)'?
Aim of the evaluation	This type of evaluation aims to assess whether the intervention was implemented in the way it was supposed to be. This helps researchers to assess how and why any impact(s) was achieved, but not whether the impacts were achieved.	This type of evaluation focuses on what happened as a result of the intervention. It could be done via an attribution analysis or by a contribution analysis.	This type of evaluation places a monetary value on the impact ('the benefit') and compares this with the monetary value of the resources used to generate the impact ('the cost').

33. When undertaking evaluations, it is normally preferred to undertake an impact evaluation which is underpinned by a process evaluation. The aim of taking this approach would be to show clearly whether an intervention has caused the desired outcomes. This evaluation process is known as 'attribution'. This type of evaluation requires an assessment of:

• what would have happened anyway in the absence of the intervention (the counterfactual)
• what happened (the outcome) as a result of the intervention, and
• would any of the outcomes have occurred anyway (deadweight) perhaps as a result of other factors or influences.

34. While this evaluation would be ideal, measurement of attribution is often difficult, particularly when there are a wide range of factors influencing the outcome. This type of evaluation is often resource intensive.

35. Conducting an impact evaluation of the FEP's success was deemed to have had several limitations aside from cost that made any assessment of attribution quite difficult. These limitations included the following:

• The evaluation design of the programme was only initiated after the start of the 2012-15 funding round, consequently projects were not necessarily set up to provide relevant or comparable data to assess progress against some of the outcomes.
• There was an absence of data demonstrating the baseline situation and a lack of controls for many of the outputs and outcomes, therefore it was not possible to show if any changes are as a result of the FEP or other factors.
• A number of other food education related projects were taking place in schools that were not funded by the FEP. This increased the difficulty in attributing impacts on knowledge, attitudes, behaviours etc. solely to the FEP.
• There was a reliance on projects conducting self-evaluation with potential for under-reporting of challenges and negative unexpected consequences and a bias towards positive data.

36. Due to the challenges described, an assessment of attribution was not considered feasible. Instead the decision was taken to conduct a 'contribution analysis' in combination with an overview of how each project was implemented (process evaluation).

37. Contribution analysis offered an alternative to overcoming some of the difficulties or impracticalities around carrying out attribution analysis. It does not attempt to prove that one factor (the intervention being evaluated) 'caused' the outcome. Instead, it explores the contribution of the policy or intervention to observed outcomes.

38. It was assumed that the interventions provided by the projects within the FEP supported the programme's outcomes.

39. An evaluation framework was devised, identifying output and outcome indicators for the FEP and describing the source of indicator data to measure contribution of the FEP to achieving the programme's outcomes and objective. Some further information on the evaluation framework is provided in the sections that follow.

Data collection

40. Data was collected from various sources and comprised the following:

• Project progress reports which included: data from monitoring of activities, summaries of data from feedback forms and financial information and industry/external involvement in delivering food education activities.
• Interviews with project coordinators to supplement data from reports and provide more information on project delivery. The face to face interviews took place following completion of Year 1 and Year 2. The project Chefs@School only started in Year 2 of the FEP (2013-14); hence only one interview was conducted with that project coordinator. At the end of Year 3 a workshop attended by most project coordinators was organised to discuss progress thus far.

41. Additionally, some projects conducted their own evaluations for their individual projects. Education Scotland's Food, Health and Wellbeing Officer undertook a survey of primary and secondary schools which assessed the range and extent of food learning opportunities as well as resource usage. Crofting Connections commissioned an external organisation (University of Glasgow) to conduct an evaluation of their project. Other projects also conducted their own surveys which interviewed teachers, pupils and/or industry partners.

42. Following completion of Year 1 of the FEP, interviews were also conducted with Education Scotland's Health and Nutrition Inspectors with the aim of providing further information on opportunities developed, embeddedness of the programme in CfE and impact of the FEP on culture, learning and behaviour change.

Evaluation of project delivery

43. In order to assess project delivery, an assessment was undertaken which drew upon information gathered from the various data sources described above. The information gathered included:

• Individual projects' progress against targets set by the Scottish Government
• Individual projects' achievement of project aims
• Challenges faced
• Feedback from the target populations on successful delivery and satisfaction
• Identification of characteristics/features of successful projects

44. The assessment did not comment on the relative merits or difficulties of the projects in comparison to each other as evaluation activities and monitoring (across the various projects) were not necessarily comparable. Instead, the assessment highlighted general issues that had relevance to the implementation of food education.

Evaluation of programme outcomes

45. For the evaluation of programme outcomes, a set of indicators were selected to assess outputs and outcomes for the FEP. Table 3 overleaf sets out the outcomes of the programme alongside the associated indicators and how each has been assessed.

Table 3: Outcomes and associated indicators for evaluation

Outcomes	Assessment/Indicators
Opportunities to learn about food are provided to young people	This has been assessed using data provided by individual projects on: The number of pupils who have had an opportunity to learn about food through the FEP The number of schools supported by FEP projects to provide food education An increase in the number of pupils/schools engaged year on year is taken as an indication of success
Food education activities are embedded in the curriculum and teachers appreciate food as a learning resource and are confident to deliver food related learning	Embeddedness refers to: How confident and enthused teachers and schools are with using food as a learning resource How useful they find using food as a tool to implement the learning principles of the new CfE The extent to which food education activities are supported by a range of resources and which should contribute to the sustainability of providing food education opportunities through schools. The following indicators were identified as appropriate: Number of food education CPD events. Number of teachers attending CPD re food education. Feedback from CPD (or other events) indicating positive impacts on learning, enthusiasm and confidence. Resources - range, availability and feedback on use. Feedback on food as a learning resource
Industry investment in food education is demonstrated and has increased/continues to increase, with commitment of industry to continue with engagement/partnerships	Data collected from individual projects on the number of businesses and other external contributors that they have engaged with and supported FEP projects. Individual projects provide estimates of the financial contribution from various investments (time, resources, venues, prizes, etc). Year on year increase in investment is treated as an indicator of success. Projects provide a description of the range of businesses and external organisation/individuals they have engaged with to support the delivery of food education, including the nature of support given. Information on the nature of these partnerships was also gathered during interviews with project coordinators.
Knowledge and awareness regarding food has increased, whether social, cultural, economic, health, environmental or potential for a career in the food industry	The majority of individual projects collect feedback from pupils and teachers on learning primarily using post-activity feedback forms. Guidance was provided on generic types of questions to include in the feedback forms so as to collect consistent data across the projects. However, flexibilities were also applied to allow individual projects to tailor some of the questions to individual needs.
Positive change in attitudes/intentions/behaviour regarding food issues, food choices and career options	Due the long term nature of this outcome, it is not directly addressed within the planned review design. However, inferences to progress being made are reported in early and mid-term outcomes. A qualitative assessment based on attitude/intention/behaviour change feedback from teachers and pupil via feedback forms has also been carried out.

46. It should be noted that the vast majority of the data analysed was provided by individual projects directly. As such, this evaluation relies on the quality and accuracy of the outputs provided.

47. Furthermore, self-reporting by individual projects comes with some challenges:

• Consistency (or lack of): Each project was asked to provide information twice a year in September (as an Interim report) and in March (as an End of Year Report). In order to minimise disruptions in the evaluation a set of templates and forms [available at Annex 11] were distributed among food education partners. In most cases these were applied, but some projects used their own evaluation protocols and formats. As a consequence, care was taken through the evaluation to maximise comparability over time as well as between projects where applicable.
• Objectivity: Self-reported data collection relies on the honesty and objectivity of participants. The degree to which this is a problem will undoubtedly vary with the topic. Furthermore, it should be noted that people are in most cases unable to self-assess completely. Consequently, any self-reported information may not be as robust despite best efforts by individuals to be honest and accurate.
• Accuracy: The main issue in this respect relates to data overlap. There are some schools that were visited by more than one project, and engagement with these schools is reported individually by each project. Furthermore, some students may have taken part in FEP activities in subsequent years but each engagement is recorded as if it were a new engagement. Therefore, assessing the degree of overlap has proven very difficult.

48. During the evaluation of the programme care was taken to minimise these challenges. This took the form, among others, of templates, data checks, primary data collection via face to face interviews with project partners and HNIs (Health and Nutritious Inspectors) or data triangulation. Nevertheless, these challenges should be borne in mind when assessing the results presented in this report.

Structure of report

49. This report discusses firstly the FEP as a whole and evaluates how all projects have contributed to the programme's overall outcomes. The first section focuses on the delivery of the programme, and the second section evaluates progress that has been made in achieving the programme's outcomes and overall objective.

50. The report also includes a number of annexes which discusses progress made by individual projects in terms of delivery and achievement of programme outcomes and individual targets.