Scottish Rural Network: evaluation

Literature Review

Key Findings:

• There is limited academic literature available on the subject NRNs. Available literature argues that national rural networks are challenging to evaluate owing to their loose structure and lack of clear intervention logic.
• An evaluation of the previous SRN (2007-2013) highlighted a lack of visibility, a lack of focus and the need to become a ‘network of networks’.
• Annual Implementation Reports have noted that the SRN has contributed to its objectives, although expressed concern that it may have overemphasised LEADER in its contributions to the RDP.

Academic Analysis

There has been limited work on rural networks in academic and policy literature, and limited comparison of networks across different national contexts. This section will, first, consider the limited existing academic analysis of NRNs. It will then consider existing Scottish Government research on the subject, including an evaluation of the previous Scottish rural network and the information available about previous SRN activities in Annual Implementation Reports (AIR) produced by Scottish Government for the EU.

Marquart^[11] conducted a survey of NRNs in Europe in 2010, which involved administering questionnaires to different rural networks. The research asked about the impact of the networks and found broad agreement amongst respondents that the networks facilitate the implementation of rural development measures and enhance the quality of rural development projects. However, despite this, networks were generally unable to provide evidence to underpin these views. Most networks, in this work, focused on outcomes like publications and website visitors. By contrast, concrete measurements that could detect improvements resulting from the network's actions were generally lacking. It should be emphasised that this does not show that the networks have no positive effect, but simply that such an effect is difficult to demonstrate. Overall, Marquart's main claim is that, without a clearly defined set of targets, it is difficult to evaluate the impact of the networks:

"…there still exists a clear need for systems to evaluate the instrumental effects of NRNs. The survey results reveal that although the network units are convinced of the positive effects of NRNs, they are unable to provide evidence to underpin this. This applies particularly to the impact of networking, which is often described as added value. Consequently, it will be hard to corroborate formally the assumed value of network activities."

Marquart frames this as a structural issue. The author argues that, at the EU level, many dimensions of the NRNs have been neglected, primarily intervention logic, definitions of good practice and governance, and an absence of clearly defined targets. As above, this closely parallels the situation here, where numerous participants in the evaluation viewed the network as positive, but could not necessarily identify clear evidence for this being the case.

Previous Evaluations and Reviews of the SRN

The 2007-2013 SRN Programme Evaluation

In 2013, an independent evaluation of the 2007-2013 SRN and SRDP communication plan was published.^[12] The evaluation notes that the budget of the SRN was modest compared to many other Member States, although the precise funding levels and comparisons to other NRNs is not discussed. The analysis involved an online survey with 346 responses, circulated via the SRN website and to other stakeholders, and a workshop with 18 rural stakeholders (along with additional interviews with internal Scottish Government personnel).

The primary findings of the evaluation were that, in large part, the SRN was 'not on the radar' of many stakeholders and that there were many competing sources of information and some overlap in provision. Of the survey respondents, 76% were aware of the SRN, with LEADER as the most common entry point. In general, stakeholders perceived there to be weak engagement with the SRN, although there was a strong recognition that the SRN could have a useful function as being the centre of a network of networks. The website was perceived relatively positively, and post-visit evaluations of project visits showed these were highly valued by participants. The evaluation concludes with a number of recommendations for how the SRN can overcome these challenges.

The evaluation also argued that, while the SRN website was perceived as helpful, the SRN had "not succeeded in engaging a wider constituency of SRDP beneficiaries and rural actors and remains very much community-focused."^[13] The evaluation advocated that the SRN could potentially act as a network of networks, to create links between rural communities. Reporting on a stakeholder workshop, they note:

"The workshop stressed the importance of an improved, better structured and clearer and more outcomes-focused approach to the NRN based on an understanding of the existing regional and sectoral networks coupled with an analysis of user group needs. In effect this should lead to a network of networks which adds value by improving links, synergies and complementarity strengthening networking and reducing duplication of effort. Clarity of purpose and the communication of this is a priority particularly if engagement with and networking of rural development actors such as LAGs and regional animaters is to be strengthened." (2013: 63)

This finding parallels a recurring theme within this evaluation: the need for more clearly defined outcomes on which to focus and a clear strategic vision for the organisation. It seems unlikely that a network of this kind would ever fully dispense with providing ad hoc support when appropriate. However, the absence of a clearly defined intervention logic and outcomes that the network intends to achieve make it difficult to assess the overall benefits that the network has produced.

SRDP Annual Implementation Reports

Since the launch of the current version of the SRN in 2015, it has been discussed in four Annual Implementation Reports (AIR). AIRs are produced for the purpose of monitoring the SRDP as a whole. Two of these reports were Enhanced Annual Implementation Reports (EAIR), which offer an extended description of the SRDP.

The AIR 2015 discussed the formation of the new Network Support Unit (NSU), which was established in the 2014-2020 programme as an in-house Scottish Government responsibility. Activities in 2015 were focused on developing a website, establishing links with the European Network for Rural Development and European Rural Networks, supporting the establishment of a new rural parliament, recruitment and raising awareness of the SRN with stakeholders and delivery partners.

Generally speaking, the assessments and evidence provided by the AIRs and EAIRs have been positive. In the 2016 AIR, the evaluation concluded that the SRN had substantially contributed to its objectives, with a similar conclusion in the 2018 AIR. However, the 2018 EAIR also noted that:

"It is clear the SRN has worked well with LEADER, but there is less evidence that the SRN has worked as effectively as it might with other schemes such as KTIF."^[14]

This finding echoes the views of several participants in the current evaluation.

SRN Self-Administered Surveys

In addition, the SRN has conducted several surveys which have been distributed through their mailing list and website. These surveys are generally small and self-selecting, with limited scope for general inference. However, given the consistencies of some of the findings, they are worth noting here for illustrative purposes.

2016 User Survey: This survey, with 199 respondents, found that while 75% and 74% of respondents were aware of the SRN's weekly newsletter and website respectively, only 32% were aware that the SRN provided funding information and 38% were aware of networking events. 'Word of mouth' was the most common means of hearing about the SRN, at 30%, although 23% of respondents had found out about the SRN through an online search engine. Among stakeholders and colleagues, 68% and 63% respectively, noted that, of their colleagues, 'some are aware' of the SRN, compared to 3% and 15% who said 'most'. Around 69% of respondents reported that they were not in any way involved with the SRN in 2015. Among those who had been involved, visiting the website or registering for the newsletter were the most common forms of involvement, although 27% mentioned meeting with one or more of the staff.

2017 Stakeholder Survey: In 2017 the SRN undertook a small survey of stakeholders, with 106 responses. This found, among other things, that 'word of mouth' was the most common means by which individuals became aware of the SRN and that, among the respondents, 54% reported that 'some' of their stakeholders were aware of the SRN (as opposed to 12% who reported 'most'). Similarly, 41% said that 'some' of their colleagues were aware of the SRN, compared to around 16% who said 'most'. Around 65% of the respondents had not attended an event run by the SRN. When asked about the extent of knowledge of SRN services, only 36% of the respondents were aware of funding information, and 35% were aware of networking events (compared to 74% who were aware of the website and 64% who were aware of the weekly newsletter). These proportions are comparable to those found in 2016.

2019 Communications Survey: In 2019, the SRN administered a small online survey (respondents = 82) to get feedback on their communications strategy. The survey found that around 41% of respondents had learned about the SRN through word of mouth (the next most popular option was social media, at approximately 14%). While this is insufficient data to generalise from, the clear prevalence of awareness facilitated by 'word of mouth', as distinct from social media or events, indicates that informal channels may be an important source of awareness about the network.

Conclusion

Previous research into rural networks clarifies a number of key features of the network. In short:

• While the goals of NRNs may be specified, they are relatively vague (i.e. do not specify clear, numerical targets) and emphasise process rather than outcomes. This makes them inherently challenging to evaluate. This is despite the fact that those involved in them frequently report that they perceive them to be useful.
• In the last funding period, the SRN was judged to be relatively unknown among stakeholders and a lack of focus was emphasised.
• The AIRs have noted that, while the SRN has effectively performed its roles, there was a perception that it had tended to focus on engagement with LEADER rather than with other schemes.
• Internal communications surveys conducted by the SRN are difficult to interpret, owing to their mixed findings and small samples. However, they appear to indicate that, for those who use the SRN's resources, use is relatively narrow (i.e. funding information or news about events). In terms of awareness more broadly, the surveys indicate that, for most respondents, the SRN is not known to the majority of their colleagues. However, it is hard to generalise much from this, as it not clear that awareness about the SRN in general, is a goal, as opposed to within particular communities. Moreover, general awareness about the SRN is more appropriately thought of as an outcome of effective engagement, than an end to be pursued in itself, so the implications of increasing awareness are not clear.

Following on from these findings, a key focus for the recommendations of this evaluation is identifying ways in which to make any future network more amenable to ongoing monitoring and evaluation.