NHS Job Evaluation Handbook - Second Edition

NHS Job Evaluation Handbook

11. Consistency Checking

1 Achieving consistency in local matching and evaluations

1.1 The aim is to achieve consistency of local matching and evaluations:

• Internally, against other local matching and evaluations, in order to avoid local grading anomalies and consequent review requests.
• Externally, against national benchmark evaluations, in order to avoid locally matched or evaluated jobs getting 'out of line' with similar jobs elsewhere.

1.2 The first measure to ensure consistency of matching and evaluation is to follow the agreed procedures and to take such additional steps as will help to ensure that the panel gets it right first time. This includes ensuring that:

• All panel members have been fully trained in using the NHS Job Evaluation Scheme, in matching or local evaluation, as appropriate, and in the avoidance of bias.
• The panel is joint and representative in composition (differing occupational backgrounds, gender, ethnicity).
• Obvious sources of bias and inconsistency have been eliminated (e.g. exclusion by agreement with panel members known to have strong views for or against jobs to be evaluated; exclusion of those from the job group being matched or evaluated).
• Where possible, there is a mix and match of panel members at successive evaluation sessions, but preferably including at least one core panel member.

1.3 The most common source of inconsistency in local matching and evaluation is inadequate or inaccurate job information, whether in the form of a job description and any additional input for matching, or a completed and analysed JAQ for local evaluation. Possible steps to minimise inconsistencies arising from this source include:

• Joint pre-checking or vetting (by job evaluation leads or their nominees) of the written job information to identify obvious omissions or inaccuracies.
• Provision for matching or evaluation panel members to seek additional information from jobholders and/or line managers, where they are agreed that this is necessary.

1.4 Consistency prior to and during evaluation is improved by:

(1) Matching or evaluating jobs in family or equivalent groups (e.g. all finance jobs, all unique specialist jobs from an occupational group) as this allows for ongoing comparisons and provides some immediate internal consistency checks.

(2) Prior to matching or evaluation, reading the most relevant national profiles (e.g. finance profiles for finance jobs, specialist and highly specialist healthcare professional jobs for unique specialist healthcare jobs), noting features which are similar to those of jobs to be matched or evaluated locally and how they were evaluated.

(3) Avoiding being influenced by current or anticipated pay levels - best achieved by not having access to salary information (and reminding panel members that, if the outcome is out of line with current or anticipated salary, this will be dealt with later).

(4) Check individual factor matching and evaluations against national profile jobs with similar features during the process (not necessarily similar jobs e.g. the Physical Skills demands of an IT job requiring keyboard skills could be checked against clerical and secretarial jobs on this factor).

(5) Check factor matching and evaluations against jobs already matched or evaluated locally (this is standard job evaluation practice).

1.5 After the evaluations: either the evaluation panel itself, or a core consistency panel (e.g. job evaluation leads, sub-group of job evaluation project group), should undertake:

(1) Internal consistency checks: after every batch (five to ten) of local matching or evaluations, the checking panel should:

a. Check evaluations on a factor by factor basis by ranking the batch (and any previous batches) from top to bottom for each factor in turn, identifying and reviewing any apparent oddities.

b. Check total weighted score rank order, identifying and reviewing any apparent oddities (often called 'sore thumbs' in Job Evaluation jargon)

(2) External consistency checks: after each block of local matching or evaluations (e.g. each job family or occupational group), the checking panel should:

a. Compare outcomes with all relevant national profiles e.g. all those which are in the same job group, pay band.

b. Compare with other consistency-checked local matching or evaluation outcomes.

1.6 Post-matching or evaluation consistency checking is largely a matter of taking an overview of a batch of results and applying commonsense, but there are some useful questions to ask, for example:

(1) Do manager and supervisor jobs match or evaluate higher than the jobs they manage or supervise on those factors where this is to be expected e.g. Responsibility for Policy and Service Development, Responsibility for Human Resources, Freedom to Act? If not, is there a good reason for this?

(2) Do Specialist jobs match or evaluate higher than the relevant practitioner jobs on those factors where this is to be expected e.g. Knowledge, Analytical and Judgemental Skills, Responsibility for Human Resources (if teaching others in the specialism is relevant)? If not, is there a good reason for this?

(3) Do practical manual jobs match or evaluate higher than managerial or other jobs where hands-on activity is limited on those factors where this is to be expected e.g. Physical Skills, Physical Effort, Working Conditions? If not, is there a good reason for this?

Such checks are inevitably made in the first instance on the basis of job titles. If these checks throw up apparent anomalies, then the next level of checking is on the matching or evaluation documentation. If the inconsistency is not explained by the second level checks, then it may be necessary to raise questions with jobholders, line managers or trade union representatives.

All of the above consistency checks can be undertaken and are facilitated by a computerised system.

1.7 When a sufficiently large number of local matching or evaluation outcomes are available, it is possible to undertake some statistical consistency checks, for example:

(1) Check that all factor levels have been used.

(2) Check the distributions of level assessments for each factor. For some factors, for example, Knowledge, Freedom to Act, Analytical and Judgemental Skills, a broadly normal distribution is to be anticipated. For other factors, the distribution is likely to be skewed towards the lower levels, e.g. Planning and Organising Skills, Responsibility for Policy and Service Development, Responsibility for Human Resources. Where these patterns do not occur, further investigation may be required.

1.8 Any outstanding concerns about local consistency should:

(1) First be checked with trained matching or evaluation panel members from own, or a neighbouring trust if required, who were not involved in the original evaluations.

(2) Checked with Strategic Health Authority Job Evaluation Leads.

(3) If concerns cannot be resolved locally or regionally, they can be referred to national level for advice (note that reference upwards is a request for advice not an appeal system