CONSULTATION ON INDEPENDENT ADVOCACY – GUIDE FOR COMMISSIONERS

12. Monitoring and Evaluating Advocacy

12.1 Both commissioners and advocacy groups have an investment in knowing that advocacy is effective. Public agencies have a duty to ensure that public money is being used well. Advocacy groups which seek public funds to help them do their work recognise their accountability to the public for how they use this money.

12.2 Advocacy groups know better than most that good intentions do not always lead to good outcomes. They know that this applies to their own work as well as to services, and welcome regular scrutiny.

12.3 Commissioners and advocacy organisations should be clear from the outset that as part of the Service Level Agreement there will be an agreed process for monitoring and evaluation. The SLA should also cite timescales for regular review meetings between the advocacy organisation and commissioners. However, the methods used for monitoring and evaluating the work of independent advocacy organisation must be credible both to the advocacy organisations themselves and to funders and should not impose a disproportionate burden on a small organisation.

12.4 Defining Quality

12.4.1 Different approaches to advocacy are needed for different people at different times and in different contexts, there is no one best model. Similarly, the evaluation method and criteria must be matched to the specific approach.

12.4.2 While advocacy organisations may differ in their approaches and beliefs, advocacy organisations have more similarities than differences and the core principles set out in Appendix 1 are therefore the same for all.

12.5 Monitoring

12.5.1 Monitoring is the process of checking continuously how things are going. From the outset, advocacy organisations should set up systems for gathering the routine information they need so they know how they are doing. For example, in the first few months the organisation and the funders might agree to monitor 'setting up' tasks such as getting the advocate recruitment procedures in place or providing training for the Management Committee or Board of Directors.

12.5.2 Commissioners can advise advocacy organisations on what information to collect and this information will be useful in discussions between the organisation and commissioners. However, advocacy organisations should see monitoring as primarily something which benefits them, not as a chore to please the commissioners. This means limiting the information collected to the useful minimum. In order to protect the confidentiality and anonymity of the advocacy partners, the information passed on to commissioners should not contain any identifying detail.

12.5.3 As well as routine monitoring, many advocacy organisations undertake periodic reviews of their work. These reviews may be purely internal, that is, undertaken by some combination of staff, management committee, advocates and people who need advocacy.

12.6 Evaluation

12.6.1 Evaluation involves a planned process of gathering information, reaching conclusions and making recommendations. An evaluation of an advocacy organisation will seek to take into account the perspectives of all those with a stake in the work: people who need advocacy, paid and unpaid advocates, staff and Management Committee/Board of Directors, members, funders, referrers and so on.

12.6.2 Evaluation means making a judgement of how good something is, not just whether or not it has complied with a funding specification. It means looking at outcomes as well as activities, at relevance as well as numbers, at what could have been done as well as what was done.

12.6.3 Evaluating advocacy is complicated. It means thinking carefully about the purpose of the organisation, and different stakeholders often have varying accounts of this. It means listening carefully to what people say about the difference it has made to people's lives. Sometimes the people whose lives have been affected most are not able to articulate this. It means balancing the visible stories of success with the invisible work of preventing worse from happening. It means putting a value on relationships as well as results. It means assessing how much impact advocacy has had on policies and practice in the service system, both in relation to individuals and more generally.

12.6.4 Commissioners and advocacy organisations should invest in regular independent external evaluation. The reports from these independent evaluations should be made available to funders, advocates and others.

12.6.5 Evaluation should be a constructive but challenging process - not an ordeal, but equally not simply a mechanism for encouragement and renewal. For this reason, it is important for advocacy organisations to undertake other renewal activities.

12.6.6 The SIAA has published 'Independent Advocacy - An Evaluation Framework'. The framework offers ideas for processes and tools that can be adapted by individual advocacy organisations to help them to evaluate and monitor their work.

12.6.7 In order for the evaluation to be effective, the independent advocacy organisation needs to demonstrate, through practical examples, how it meets all of the Principles and Standards set out in Appendix 1^[3].

12.6.8 There are three sections to the Framework. The first considers gathering and analysing numerical data and 'soft' outcomes, the second is a tool which can be used by organisations to measure their work against the Principles and Standards for Independent Advocacy in Appendix 1 and the SIAA Code of Practice. The third section is a tool which can be used by an external independent consultant when undertaking an evaluation.

12.7 Commissioning an independent evaluation

12.7.1 It is useful to draw up a clear specification for any evaluation, setting out the scope of the work, who is doing it, how it will be done, why it is being done, who wants it done, who is paying for it to be done, who will get the report and what sort of actions might be taken as a result of the evaluation.

12.7.2 For example, in planning one evaluation the following scope and focus was agreed by advocacy organisations and commissioners:

• Developmental - using a partnership approach, to highlight what is working well and where improvement and development is needed in future
• Service user-focused - exploring the relationship between partners and advocates, the experience of partners, the issues which have been important to partners and advocates and the roles advocates have taken up
• Organisation-focused - the work of the office and co-ordinator, the training and support given to volunteers
• Management-focused - management arrangements and the role of the Management Committee or Board of Directors
• External support - what support has been provided from commissioners and others, how this helps or hinders, how it could be improved
• Relationship with providers - how the project is perceived, what impact it has had on providers.

12.7.3 As well as considering current outcomes, an external evaluation should pay attention to the accountability, robustness and sustainability of the organisation. This might include, for example, issues such as:

• the composition and renewal of the Management Committee/Board of Directors. How well does the membership of the Committee/Board reflect the organisation's constituency? Are new people being recruited to the Committee/Board?
• the extent to which the organisation is addressing the needs of the most marginalised people within its constituency - for example, people from ethnic minorities, people who do not use words to communicate, etc.
• the reputation of the organisation within its community
• the match between the advocacy needs of the people the organisation serves and the skills and resources held within the organisation.

12.7.4 The SIAA Evaluation Framework includes details of areas for assessment in an external evaluation. The framework has been designed to allow it to be used in evaluating different models of advocacy. There are also recognised tools designed for evaluating citizen advocacy.

Question 3: Would you support a programme of evaluations based on the pilot model of evaluation set out in paragraph 5 of Part 2 of the consultation paper?

Yes No

If not, why not?