Farm Advisory Service: enhanced monitoring and evaluation

2. Methodology

2.1 Overview of approach

The requirement was delivered by a telephone survey with a sample of 116 beneficiaries of one-to-one support.

The survey focused upon:

The extent to which actions recommended as part of the one-to-one advice have been taken

• The outcomes of the support
• Satisfaction with the support
• Any gaps in the support or suggested improvements.

The full questionnaire is provided in Appendix 1^[1].

2.2 Sample

Four databases of contacts provided by Ricardo of businesses who have received support through the four components of the FAS were reviewed. These covered the period from September 2016 (the inception of the support) up to 31 March 2019. Those who had received specialist advice only were excluded from this research as these are farmers in crisis and/or who are exiting their business.

Analysis of the databases indicated a total population of 295 for the telephone survey (once duplicates and opt outs had been removed). 116 interviews were achieved, a response rate of 39%^[2].

This response rate was achieved through:

• Arranging for Ricardo to send an initial email to beneficiaries informing them of the survey and encouraging participation
• Contacting beneficiaries multiple times
• A three week fieldwork period (14^th to 31^st May 2019)
• Flexibility in arranging telephone appointments.

Some minimum quotas for the number of achieved interviews by support type were initially proposed to ensure adequate minimum coverage of each. A larger sample (proportionally) of mentoring applicants was initially proposed but this was not achieved from the relatively small population size. Appendix 2 provides tables showing the profile of the achieved sample against the profile of beneficiaries. This shows a broadly similar profile in respect of the key characteristics: type of support received, farm size and farm sector. There is accordingly no need to weight the data.

The beneficiary database did not include complete data for age or whether the business was a croft or not so these characteristics have not been compared but are included in Appendix 2 for interest.

Not all respondents answered all questions. This is mainly explained by the routing of the questionnaire, for example, respondents were only asked for additional sales figures if they indicated that they had achieved additional sales. In addition, a small number of respondents were unable to remember the actions they had taken but were able to recall having support and their overall satisfaction with the service. The number of respondents answering each question is given throughout the report.

2.3 Reference to reports

The FAS support provided and the advice and recommendations given are quite specific and tailored to individual businesses. The preferred approach would have therefore been to have access to individual reports to inform each interview. However, this was not possible due to the FAS confidentiality requirements. Respondents were instead asked if they could find and refer to their report during the interview. This request was also included in the email that Ricardo sent out to respondents prior to contact as part of fieldwork. When respondents were called to arrange and conduct the interview they were again asked if they could have their FAS report to hand.

Just under a third, 30 per cent, of respondents referred to their reports during the interview. The majority of respondents answered from memory. In a small number of cases, respondents could recall some of the support they had received (and answer about this) but not recall all of the support they received.

2.4 Analysis

The sample includes one respondent who went through the questionnaire twice as he had received a Carbon Audit for two separate businesses. This means the analysis should be understood as analysis by farm rather than by individual.

The 'actions'^[3] section of the questionnaire (reported in section 3.2 and 3.3 of this report) was asked individually for each report a respondent received. For example, if a respondent had received an Integrated Land Management Plan (ILMP) only, they would have been asked these questions once in respect of the ILMP. If a respondent had received an ILMP report and two specialist advice reports, they would have been asked these questions three times in respect of each individual report. The actions section of findings that follows therefore presents percentages according to the proportion of reports. These findings can also therefore be sub-divided into individual support type: ILMP, specialist advice or carbon audit.

Respondents answered the other sections of the questionnaire in relation to all the support they received. For example, they indicated which benefits had arisen for all the support received together. Findings are therefore reported as the proportion of respondents. Analysing these findings by support type is consequently done by the type of overall package of support they received: mentoring (with other support); carbon audit only; ILMP with or without specialist advice; ILMP, carbon audit with or without specialist advice.

Benefits achieved and satisfaction have been analysed by support type received, farm sector (cattle and sheep or not cattle and sheep) and farm size (under and over 100 hectares). There is some relationship between support received and farm size which is likely to explain some similar patterns in findings by these sub-groups. 88% of those who received mentoring were from farms with under 100 hectares. 90% of those who received a carbon audit were farms with over 100 hectares.

Significance testing of differences between sub-groups, for example, by support type has not been used due to the relatively small size of the sample. Differences identified between sub-samples (for example, by support type, farm size etcetera) should be understood as indicative rather than as definitive.