Farm Advisory Service - One to Many: evaluation
This report provides an evaluation of the One to Many component of the Farm Advisory Service, delivered by SAC Consulting and provided as part of the Scottish Rural Development Programme 2014-2020. It analyses what has been delivered and provides recommendations for the future.
3 What Has Been Delivered?
3.1 Overall Findings
- Primary interventions, such as events and use of the advice line, have been consistently delivered across the period while use of these services has increased.
- The FAS delivery has developed and improved over the years, with the addition of a newsletter, video content, podcasts and online tools which have enhanced the service.
- There has been a substantial growth in engagement with the FAS offering over this time, as measured in advice queries, event attendance, video views and social media presence.
- FAS feedback forms indicate high satisfaction among recipients for the service.
3.2 Achievement of KPIs
FAS key performance indicators were introduced in 2018. These are detailed in the annual reports for 2018-19, 2019-20, and on a monthly basis during 2020-21 (as the year has not yet been completed). As found in the annual reports, there are twelve KPIs attached to FAS One to Many contract delivery. These are:
1. Events
2. Event attendance
3. Event feedback – overall quality
4. Event feedback – relevance
5. Event feedback – intention to change
6. Bounce rate for the website
7. Website availability
8. 95 percent of call-backs are done within six working hours
9. 95 percent of routine queries are responded to within one working day
10. 90 per cent of detailed queries are responded to within two working days
11. 100 percent of complex queries are responded to within five working days
12. 100 percent of e-mail enquiries are responded to within one working day
As we can see, these relate primarily to event attendance, effective website maintenance and ensuring that queries are responded to. Based on the data available in FAS OtM annual and monthly reports, KPI 1 (number of events) was fulfilled in 2018-19 two months ahead of schedule. In 2019-20, the event KPI was fulfilled at the level of the number of events, but the total and average attendance at events was lower than the targets (4,917 attendants relative to a target of 5,400).
Looking at the KPIs overall in the period available, these have consistently been met, with the exception of frequent shortcomings with regard to KPI 6 – the bounce rate of the website – and KPI 11, which relates to responses to complex enquiries. Here, SAC Consulting have emphasised that while the responses are consistently made on time, the relevant paperwork is not always completed fast enough to confirm this. This appears to primarily be an issue of training and monitoring. A full review of the KPIs and SAC Consulting's performance against them can be found in the supplementary documents accompanying this report.
While the KPIs are indicative of the outputs of FAS, they do however raise important questions about how success in a future advice context should be measured. While website attendance, feedback forms and event participation are all useful metrics, it is worth considering how a future iteration of the FAS could also introduce KPIs specifically concerned with impacts. However, this is a challenging process, given the enhanced monitoring it would entail. This informs Recommendation 1.
Recommendation 1: Review Monitoring Framework. Review the KPI arrangements for a future service, and consider the possibility of developing outcome based KPIs. Review the options for establishing effective monitoring of farm improvements, ranging from centrally set and monitored goals to goals established and monitored by individuals or groups of participants.
3.3 FAS Events
A key part of FAS delivery has been the provision of events. With the exception of 2016/17, FAS have provided over 200 events per year, rising to 249 during 2019/20 (see Figure 3.1). The lower numbers in 2016/17 reflect the fact that, in that financial year, the FAS contract was only operational for nine months. During the 2020/21 financial year, to date, events have been altered to take into account restrictions on face to face meetings owing to the Covid-19 pandemic. As of August 2020, there had been 48 events, with a total of 1, 456 participants.
The events cover a wide range of topics, and have included 83 specifically aimed at New Entrants to farming, and 78 specifically focused on Women in Agriculture.
Event attendance has also increased over this period (see Figure 3.2). While it is impossible to state whether the attendees are unique, as individuals may have attended more than one event, the upwards trajectory is none the less positive. As we can see, when we compare 2017/18 to 2019/20, there is an increase of approximately 55%.
In Figure 3.3., the average number of attendees per event per year is provided, based on the available data. As this shows, while data is incomplete, the addition of online events has increased average event attendance in the 2020/21 period.
In response to Covid-19, online events have replaced in-person events. As shown in Figure 3.3, this has increased average attendance, although it is also possible that reducing barriers to attendance has also increased the number of participants who are not from core demographics, i.e. non-farmers and academics.
The events have taken place in a wide range of locations, consistent with the FAS' ambition to be a truly 'national' advice service (see Table 3-1).
| Region | No. of Events |
|---|---|
| Highlands and Islands | 211 |
| North East Scotland | 135 |
| South East Scotland | 44 |
| South West Scotland | 119 |
| Strathclyde and Central Scotland | 73 |
The FAS data also indicates that direct mail remains the most common means by which participants learned about events, although social media and local SAC Consulting offices remain important sources of information.
3.4 Events Feedback
FAS routinely collects feedback data during the events. On overall delivery, in the period May 2017 to March 2020, 68% of respondents rated the event as 'excellent', and 29% regarded the event as satisfactory. During the period April 2020 to August 2020 – during which the metrics of the feedback have been altered and therefore cannot be merged for analytical purposes - 58% of the respondents said the events were excellent and 39% regarded the events as good. As we can see, participant feedback has been consistently high throughout the duration of the programme. Similarly, other categories for feedback received positive results, as shown in Table 3-2 and Table 3-3.
| Less than Satisfactory | Satisfactory | Very Satisfactory | Excellent | |
|---|---|---|---|---|
| Was the information easy to understand? | <1% | 2.4% | 33% | 60% |
| Were there opportunities to ask questions? | <1% | <1% | 25% | 68% |
| How useful was the event? | <1% | 5% | 32% | 60% |
| How relevant was the content of the event? | <1% | 5% | 30% | 63% |
| How suitable was the event? | <1% | 2% | 32% | 63% |
| Time Keeping? | <1% | 3% | 33% | 58% |
| Poor | Fair | Good | Excellent | |
|---|---|---|---|---|
| Was the information easy to understand? | <1% | 3% | 39% | 58% |
| Were there opportunities to ask questions? | <1% | 5% | 39% | 54% |
| How useful was the event? | <1% | 5% | 42% | 53% |
| How relevant was the content of the event? | <1% | 4% | 35% | 60% |
| How suitable was the event? | <1% | 3% | 39% | 58% |
| Time Keeping? | 0% | 4% | 27% | 69% |
This event feedback is encouraging and informs the view that the intervention has been delivered effectively.
3.5 Impacts of Events
FAS also collected, via event feedback forms, data on whether participants intend to make changes on their farm as a result of the event. Of all participants, 85% reported that they intended to make changes to their farm as a result of the information they received at a FAS event. They were also asked what these changes were likely to be, with the answers displayed in Figure 3.5.
In the first instance, it is encouraging that such a high number of respondents indicated they would make changes. However, the benefits noted vary considerably. While over 40% of participants noted 'better decision making' as a change they would make, only 5% reported improved knowledge of climate change and 8% reported improved knowledge about waste. From this perspective, there may be scope to improve the extent to which participants are encouraged to make changes on these key issues. At the same time, it has been observed by SAC that, rather than delivering events specifically concerned with climate change, climate change discussion is 'mainstreamed' within other subjects, e.g. advice on fertilizer use will incorporate concerns about reducing emissions. Therefore, it may be the case that the 5% is an underestimate. However, given the uncertainty here, it may be the case that, going forward, monitoring focused on the on-farm impact of advice and impact related KPIs, as in Recommendation 1,[4] could supplement our knowledge of the impacts of advice.
3.6 Use of Advice Line
FAS OtM provides farm businesses with up to 30 minutes of free advice. Calls to the advice line per year have increased from 486 in 2016/17 to 1,639 in 2019/20, an increase of 237%. As noted in the 2019/20 annual report, the service received an average of 135 enquiries a month. Adjusted for office hours, this equates to approximately one call every 78 minutes during this period. The lower numbers in 2020/21 are likely to reflect the shorter period under discussion, although the impact of the Covid-19 pandemic on use of the advice line isn't entirely clear.
In terms of how calls progress, initial call handling is subcontracted to Ricardo, with SAC Consulting providing specialist and technical advice when a question exceeds a certain threshold. Non-technical calls – tier 1 – may be dealt with by the advice line staff, while more complicated enquiries – tier 2 – are dealt with by SAC Consulting directly. As we can see in Figure 3.6, both types of call have increased over the period in question.
This is, again, indicative of effective delivery of the contract. A key challenge, however, is clarifying the extent to which the increased demand represents increased use from a subset of users, an increase in the base, or both. This is because, while data is available on the use of FAS, we do not have representative surveys of the farming population that specifically ask about the use of FAS, and cannot therefore analyse the extent of coverage in this group. To address this going forward, see Recommendation 2.
Recommendation 2: Review Data Strategy. Consider potential mechanisms for regular, representation farm data collection to determine the extent to which FAS is used within the farming population as a whole and how the service is viewed. Consider opportunities for integrating monitoring of farm environments within a broader environmental monitoring strategy. Consider the possibility of a specific farm data strategy for monitoring and benchmarking environmental impacts.
3.7 Web Metrics
Measuring use of the FAS website over the period is unfortunately not possible. FAS noted that, during 2019/20, they became aware that the website metrics were not accurately capturing traffic. Following a rectification of this, website traffic increased substantially at this point, but this analysis cannot be retrospectively applied. However, Figure 3.7 reports the use of the website across 2020.
This indicates between 25, 000 and 43, 000 website sessions per month, which suggests that the website provides a valuable service. Further to above, implementing Recommendation 2 could help address questions of how widespread use of the website is within the farming population and where coverage should be expanded.
3.8 Social Media and Video Content
The One to Many service has also increased its social media presence over time. As of August 2020, they have 4, 215 Facebook followers. The development of this over the five years can be seen in Figure 3.8.
On Twitter, presence has similarly increased, from 212 followers in 2016/17 to 1, 579 in 2019/20, and 1, 886 by August 2020. During 2020, FAS tweets have been retweeted an average of 122 times a month.
FAS began producing video content in 2017/18. The graph below measures this in terms of the number of sessions on YouTube and the number of minutes watched. This has also increased substantially over the reporting period, with 57,120 minutes watched in 2019/20. FAS videos were also viewed 70,497 times in 2018/19 and 50, 508 times in 2019/20.
In 2019, the top five videos were: 'New Entrants Case Study: Stephen Withers and Neil Sandilands' (6,880 views), 'New Entrant Case Study: John Warnock and Iain Baillie' (4,794 views), 'Calf scour prevention' (2,010 views), Resilience workshops (1,172 views) and 'Intra Peritoneal injections to treat hypothermia in lambs' (1,157 views).
The growth of the FAS social media profile indicates that, over time, demand and awareness are slowly being achieved. It is also encouraging to see ongoing engagement with diverse formats.
3.9 Publications and Downloads
The FAS are responsible for publishing a wide range of guidance to inform farmers. The data available indicates that 292 publications were published online in 2019/20 and 258 in 2018/19. As noted in section 3.7, however, web metrics from prior to November 2019 are likely to be inaccurate and an underestimate of how many times these have been downloaded. However, the cumulative numbers from January to August 2020 indicate that, in this period, documents were downloaded 108, 674 times. The monthly downloads in 2020 can be seen in Figure 3.11:
With over 10, 000 downloads of technical information per month, these figures indicate that this is a valuable component of the service (although establishing the impacts of this information remains challenging). Among these downloads, the Farm Management Handbooks continue to rank highly within the top 10 monthly downloads (although, as a proportion of the whole, they do not appear to exceed 10%). This can be seen in Figure 3.12.
Given this consistent use of FAS resources, we can infer that, at least for some farmers, the presence of high quality downloads on the FAS website has been beneficial.
3.10 Podcasts
Over the course of the contract, FAS have started producing podcasts. In 2018/19, 22 podcasts were produced, and 34 were produced in 2019/20. In 2018/19, these podcasts were listened to 2,305 times and in 2019/20 they were listened to a total of 6,411 times. The most popular podcasts in 2018/19 and 2019/20 are reported below.
Top five podcasts in 2018/19:
- Finishing Lambs - dealing with the challenges (244 listens)
- Fodder beet: alternative feeding for sheep (241 listens)
- Practical grassland: making the most of your forage (211 listens)
- Moray Soil & Nutrient Network - Using organic manures to save on artificial fertilisers (194 listens)
- Growing crofting and smallholding (185 listens)
Top five podcasts in 2019/20:
- Women in Agriculture: Baddinsgill Farm, West Linton (337 listens)
- Women in Agriculture: Lynbreck Croft, Grantown-on-Spey (324 listens)
- Grant funding for woodland creation (276 listens)
- Women in Agriculture: Primrose Beaton of Lawrie & Symington (256 listens)
- Woodland Creation & Management - Frequently Asked Questions (240 listens)
While the audience for these podcasts appears to remain small, it is nonetheless encouraging that FAS have not only expanded their engagement into an additional medium, but that the audience appears to have grown over the period they have been producing them.
3.11 Croft and Small FarmAdvisory Service
FAS also provides a small farm and crofter advisory service. This is a subscription based service, which entitles crofters and small farmers to access two hours of remotely delivered advice from SAC Consulting, alongside a magazine and a subsidised consultancy service. In 2019/20 FAS reported that there were 2,188 crofter subscribers, compared to a target of 1,900, and 287 small farm subscribers, from a target of 235.
A survey of subscribers was undertaken by SAC Consulting in August 2019. It received 247 responses – an 11% response rate. The survey found that 12% of the sample contact their staff as often as 10-12 times a year, 51% make contact more than four times per year, and 39% are in contact two or three times, with 12% not contacting them at all. 87% of respondents rated the subscription as 'very good' or 'good', with only 2% rating it as 'poor' or 'very poor'.
3.12 Newsletter
In May 2018, FAS started providing a monthly newsletter to subscribers. As this has only recently become available, it is not helpful to look at longer term trends. At commencement, this had just under 3,000 subscribers and, as of August 2020, is at 5,299, which is a substantial increase over the period. The 'open rate' of the newsletter at this time was 49%.
3.13 Online Tools
In 2018, FAS started producing online tools, and have produced ten to date. These provide assistance in addressing issues such as soil management and nutrients, treating hypothermia in lambs, business management, resilience and women in agriculture. In 2019, FAS launched their first app, focused on soil nutrient advice.
3.14 Conclusion
Based on the reported activities of FAS, relative to the contract specification and with regard to the KPIs, it appears that SAC Consulting have fulfilled the contract and delivered the components of the Farm Advisory Service as required. There is also considerable evidence of effective delivery. The delivery of events has been well received and the consistent increases in FAS activity on social media, at events, online and the addition of new deliverables such as podcasts are indicative of a well delivered advice service. At the same time, as we can see from the accompanying recommendations, future delivery may seek to develop a more impact oriented monitoring framework, as well as considering the use of more robust data to determine the extent of FAS use among farmers.
Contact
Email: joseph.ritchie@gov.scot
There is a problem
Thanks for your feedback