Primary care: national monitoring and evaluation strategy

Governance, Reporting and Resources

Decision-making

Ownership of this 10-year National Monitoring and Evaluation Strategy lies with the Scottish Government. The following governance and reporting arrangements will apply: 

• responsibility for delivering and reporting on the Strategy lies with the Scottish Government Health and Social Care Analysis Division (SG HSCA), supported by Health Scotland/Public Health Scotland
• the Senior Responsible Owner (SRO) for the Strategy will be the Head of Primary Care Division in the Scottish Government
• a Primary Care Monitoring and Evaluation Steering Group will provide analytical and operational oversight and direction for the Workplan
• SG HSCA will produce a 12-month Workplan which will be developed in collaboration with the Steering Group, and through appropriate consultation with the Scottish Government’s Primary Care Programme Board, other SG policy areas, the Primary Care Evidence Collaborative and other stakeholders
• the SRO will sign off the Workplan and resourcing decisions for monitoring and evaluation activity
• SG HSCA will provide updates to the Primary Care Programme Board and to the Steering Group
• lead responsibility for promoting and populating the Outcomes Framework with evidence and for revising it when required, at the national level, lies with Health Scotland and SG HSCA 

The Scottish Government will develop full terms of reference for the Primary Care Monitoring and Evaluation Steering Group. The Group’s core functions are likely to be to: 

• act on directions from the SRO and the Primary Care Programme Board 
• agree and propose to the SRO and Primary Care Programme Board an annual Workplan, defining required operational resourcing decisions
• be responsible for the ethical and analytical governance of the Strategy
• delegate activities to members of the Group and their organisations
• co-ordinate the best use of resources for Strategy delivery across represented organisations
• foster and maintain links with other policy areas and relevant programmes of monitoring and evaluation across government and the public sector
• quality assure outputs from research and data it commissions (in-house and externally)
• work closely with the Primary Care Evidence Collaborative

The Group is likely to include, as a minimum: SG HSCA Division; SG Primary Care Policy (including clinical advisers); the SG Person Centred and Quality Unit; Health Scotland, HIS, ISD and NHS Education for Scotland; representation from Integration Authorities; the SG Chief Scientist’s Office; the ALLIANCE; and the SG’s Chief Medical Officer Directorate. It will consult more widely, where appropriate, with stakeholders, including Integration Authorities, NHS Boards and Health and Social Care Partnerships, the third sector, members of the Collaborative, the Scottish Health Council (part of HIS) and other routes for engaging with lived experience, the wider academic and policy analysis community, and clinical interests.

Principles for Government Research and Statistics

Any data-collection or evidence analysis activities initiated under the Strategy will be subject to established research and data governance and legislation, and best practice in healthcare and public policy research. Depending on the nature of the data and analysis, the Scottish Government’s Protocol for the Publication of Research, National Statistics Codes or other publication requirements may apply. External studies will be procured through fair and open competition, in line with public sector procurement law and best practice and to ensure best use of public resources, unless there is sound justification for an alternative commissioning approach. 

Evaluation, whether undertaken internally or externally, on behalf of the Scottish Government, will reflect the Government Social Research (GSR) principles: 

• Principle 1: Sound application and conduct of social research methods and appropriate dissemination and utilisation of the findings
• Principle 2: Participation based on valid informed consent 
• Principle 3: Enabling participation 
• Principle 4: Avoidance of personal harm 
• Principle 5: Non-disclosure of identity and personal information 

All reports and other outputs should conform to the four principles for GSR products: rigorous and impartial; relevant; accessible; and legal and ethical. 

Roles and reporting 

The Scottish Government Health and Social Care Analysis Division has been tasked with delivering the Strategy and will monitor progress and report to the Steering Group. It will jointly produce and own, with Health Scotland, a short annual overview report, which maps progress against each section of the Outcomes Framework and details the work planned for the following year. It is likely to cover:

• a summary of research activity and findings;
• an update on quantitative indicator trends;
• evidence from other research sources (including specific evaluation projects, qualitative case studies) which demonstrate a contribution to whether and how primary care reform is being realised;
• a narrative overarching assessment of progress to date;
• relevant research and policy internationally which could inform ongoing primary care reform in Scotland.

More comprehensive reports, in 2021, 2024 and 2028, will synthesise the progress and learning, describe trends in key indicators, take-stock of the evidence-base, and identify gaps we need to address. Health Scotland will have a key role in synthesising evidence as it emerges. SG HSCA will be responsible for reporting on indicator framework data and changes over time.  

Resources

It will be challenging to deliver a comprehensive programme of monitoring and evaluation over the next decade in a context of competing priorities for public sector resources and a complex and evolving policy and delivery landscape. The need to be realistic and proportionate, only undertaking research that has genuine value, is keener than ever. At the same time sufficient investment of resources in research, evaluation and data is fundamental to ensuring good quality, cost-effective, evidence-informed policies and initiatives. The Scottish Government will use evaluation resources and research budgets strategically and effectively on the basis of annual workplan priorities and in consultation, particularly, with its national partners in evidence and analysis - ISD, HIS and Health Scotland. 

As already noted in this document, evidence for monitoring and evaluating primary care will not just come from activities undertaken or funded by the Scottish Government or the national boards. Sources for evaluation and research activity could come from the following:

• Organisations (e.g. SG, national health boards) undertaking research or analysis in-house
• Organisations commissioning others to undertake research (e.g. as Scottish Government Social Research projects)
• Funding for research and evaluation, including self-evaluation, being built into project grants by the funder
• Integration Authorities or Boards undertaking or commissioning local evaluation and self-evaluation which generate findings relevant to the national level
• Other funders - e.g. the SG Chief Scientists Office, National Institute for Health Research, Medical Research Council, Economic and Social Research Council, Health Foundation, and Wellcome Trust
• Building in-house capacity for (self-)evaluation and data analysis in organisations delivering change
• Collaborations or partnerships with national funders and think tanks.

Stakeholders, including public agencies and academic institutions, are encouraged to be strategic in their approaches to maximising the use of existing evidence; to exploiting existing funding sources; and to encouraging investment by significant national research funders to further the evidence base for primary care.

Anticipated challenges and risks to effective evaluation

It is our intention to be strategic in planning primary care evaluation, data collection and research over the next 10 years. We recognise that there are considerable challenges and risks for the success of this undertaking:

• Many outcomes will only be fully achieved over the longer term and system changes will take time.
• The availability, sufficiency and quality of primary care data are currently limited, and the supporting data infrastructure requires development.
• Complexity – primary care is part of a wider system undergoing significant change and establishing a “baseline” from which to document change is challenging. It will be challenging to attribute changes in a complex system to specific policies or set of policies. Established Contribution Analysis methods help with this.
• There is always the danger that we focus on what we can count or measure so that scarce evaluation resources are not available for telling the story, that we focus on the wrong things, or miss other valuable but “difficult to measure” things. 
• The results of evaluation need to be shared in a timely and effective fashion with those who are responsible for reforming primary care or their usefulness risks being diminished. 
• Local learning and success may not be generalisable or scalable and short-term pilots may not lead to sustainable, cost-effective changes. Many service redesign projects and tests of change are locally chosen and their potential might not be well understood when planning monitoring and evaluation, although lessons from the process of how they were introduced may be helpful.
• Availability of funding for the delivery of new models of care and for research and evaluation.
• The limited evaluation capacity and expertise of local and national organisations.