One million acres by 2020: strategy report and recommendations

5. How will we measure success? Methodology

Workstream 2 of the 1 Million Acre Target Short Life Working Group has as one of its deliverable outputs: "A methodology for measuring progress towards the target."

Initially it was intended that the methodology would solely focus on counting the acreage coming into community landownership until 2020, in order to measure progress towards the target. However, based on discussions both within the Short Life Working Group and with external stakeholders, it has become evident that a more extensive analysis, also incorporating outcomes and impacts, would be deemed to be highly valuable.

Recommendation

16. A more extensive analysis of the impacts and outcomes of community land ownership would help relay the message that the target is not primarily concerned with the acreage in community ownership per se, but with the outcomes communities can achieve once they have taken ownership of this land.

Theme: Monitoring and Evaluating Progress

Therefore, 'measuring progress' is proposed to consist of two components:
1. Measuring acreage in order to evaluate progress towards the 1 million acre target; and
2. Measuring and analysing the outcomes resulting from delivering this Strategy's vision.

5.1 Measuring progress towards 1 million acres

From discussions with stakeholders it emerged that most of them would find it acceptable to be asked to provide a progress update on an annual basis. However, it was noted that public bodies and Local Authorities will already be required to publish an annual asset transfer report under the recently passed Community Empowerment Act. ^[68] As the guidance for the CEA is still to be completed, it is recommended that the measurement of progress towards the target is integrated with the reporting requirement under the CEA.

It was also suggested to make (some of the information) on community-owned assets public, for example through an online searchable database and/or visually, through map-based data. For example, see Scene Consulting's 'Energy Archipelago' ^[69] website for an example of a searchable online map-based database. Having information online, accessible to the public, would help not only to raise awareness, but would also allow community groups to include or update their own project.

5.2 Measuring progress - other outcomes

A key issue around the evaluation of projects and policies is the, at times, limited strength and quality of evidence of their effects. In part, this is due to problems of measurement and evaluation criteria not being established from the outset. ^[70] It is therefore recommended that evaluation is integrated from the outset and undertaken at several stages (during, on completion and post-completion). ^[71] However, due to the ever changing policy landscape and integration of policies and agendas, it is recognised it is not always feasible to set out evaluation criteria from the outset.

There is no universally applicable set of indicators that will be appropriate for a particular intervention. ^[72] Standardised performance indicators are often desirable, but may not always be useful measures of progress. It is therefore important that the indicators are established consensually, and that there is scope for those participating in, or benefiting from the policy or programme, to define the criteria against which 'success' is measured. Ideally, stakeholders would be involved in the selection of indicators in order to ensure that the indicators incorporate their practices, experiences and priorities.

Data is already being collected on an on-going basis by funders and support agencies (e.g. HIE, BIG). Collaborating with such agencies can help expand the evidence base without increasing the reporting burden on communities. Involving these agencies from the outset can also help to ensure that indicators and outcomes cover all relevant areas whilst also being appropriate and context-specific.

Measuring the benefits resulting from community asset ownership is not easy. A lack of accurate baseline data in the past has been raised as a concern. ^[73] Second, it can be difficult to identify reliable statistical impact measures for many of the benefits associated with community ownership. Communities do not operate in a vacuum, but are part of a much wider network of service provision and support, making it difficult to attribute benefits to one organisation or asset. ^[74]

Finally, outcomes are often not immediate or can fluctuate over time. Some organisations might find that the first few years after acquiring an asset is their 'honeymoon period', whereas other organisations might only begin to yield a surplus after acquiring several assets. Other socio-economic as well as environmental impacts are also likely to only be evident after many years. At the other end of the scale there are organisations like the Stornoway Trust who have owned an asset, in this case land, for so long that "even second-hand memories of [the transfer] have faded". ^[75]

Recognising these concerns, we have been considering how best to measure progress towards the 1 million acre target. Whilst some methodological issues (i.e. to separate the effects of the asset from other local or national activities) may be difficult to resolve, we believe others can be addressed within the scope of this target.

Recommendation

17. The SLWG recognise the need for robust baseline data, and would suggest this is improved with continued baseline data collection and analysis and greater collaboration with other funders and agencies to streamline reporting requirements.

Theme: Monitoring and Evaluating Progress

Early engagement with the community regarding their current situation, their aims and how to get there can also help make these evaluations more tangible for the groups involved. Second, we understand the difficulty in choosing reliable indicators that capture the many, diverse, benefits emerging from community ownership. Here, a suite of indicators can help to grasp some of these complexities and help to ensure that relevant outcomes are captured for each type of project. Finally, we appreciate that outcomes are often not immediate or can fluctuate over time. We therefore not only propose improved base-lining, but also more consistent recording of outcomes across time in order to obtain higher quality longitudinal data. However, we are sensitive to the fact that communities may feel overburdened with reporting requirements. We would therefore suggest that greater collaboration with other funders and agencies in other to streamline reporting requirements would be advantageous.