Commissioner for Fair Access discussion paper: league tables

Discussion Paper: League Tables

This is one of a number of discussion papers published on the Commissioner for Fair Access’ website on key issues relating to fair access. The aim is to bridge the gap between detailed research (where it exists), which is often only accessible to experts, and the wider public conversation, especially in political circles and the media. The hope is that these discussion papers will contribute to, and stimulate, that conversation by presenting data and evidence as accessibly and objectively as possible. Each discussion paper will also include a commentary section by the Commissioner.

This discussion paper considers:

• The nature of league tables and university rankings;
• The criteria used to measure quality for different league tables and the challenges associated with those measurements;
• Concerns that have been raised within the sector about league tables, and their relationship to fair access;
• The Commissioner’s view on league tables and fair access.

Background

At its simplest, a league table or ranking system places universities in a list, from best to worst, according to a specific set of measures. A league table compiler may use 5 measures of quality (A—E) and decide on the importance of each by assigning a percentage weighting (A = 10%, B = 30%...). Suitable data to assess each measure is gathered for every institution being considered. Final positions within the ranking are determined by assigning a value to each measure, developing some kind of formula to account for the relative weighting of that measure, totalling the scores and placing every institution in a list from highest to lowest score.

There are many different types of university league tables or rankings. Some are global, attempting to determine the world’s best institutions, others provide a national picture or aim to measure one particular aspect of a university experience, such as their sports provision or sustainability. Some offer a whole institution summary others provide rankings for each subject area.

The Commission on Widening Access’ ( CoWA) recommendation 13 stipulated that the Commissioner for Fair Access should engage with those compiling rankings so that they give greater priority to socio-economic diversity and to ensure institutions are not penalised for taking actions necessary to ensure fair access. CoWA recognised, for example, that the entry grades of students play a significant role in league table rankings. This could incentivise institutions to increase their entry grades and work against those providing contextual offers or developing non-traditional entry routes. This would be to the detriment of efforts towards fair access.

Table 1: Examples of league tables

Examples of global rankings	Examples of UK national rankings	Examples of single measure rankings
Times Higher Education World University Rankings	Times/Sunday Times Good University Guide	QS Graduate Employability Rankings
QS World University Rankings	Guardian University Guide	British Universities and Colleges Sport
Academic Ranking of World Universities ( ARWU)	The Complete University Guide	People and Planet University League ( P&P)
U-Multirank
U-Map

Regardless of their limitations or concerns about their methodologies, league tables enjoy a high level of acceptance amongst stakeholders and the wider public because of their perceived simplicity and the consumer-type information they convey (European Commission, 2010). Taken at face value, they appear to show how universities compare to each other in a fair and objective manner.

They seem to offer transparency and could, therefore, assist applicants in making informed decisions and allow institutions to benchmark their performance against others.

The briefest of glances at a university website or prospectus reveals the importance of league tables and rankings. They are used by institutions in their advertising and recruitment as a measure of prestige (’Ranked 1st for campus environment’, ‘University named best buy’, ‘One of the world’s top universities’). Press releases are issued when an institution is ranked highly or improves upon their position and results from different outputs are emphasised to highlight positive outcomes whenever possible.

There is evidence that league tables, and the desire to climb higher through the rankings, can influence national policy and shape institutional decision-making and behaviour (Halzelkorn, 2009). Such impact seems likely to increase rather than diminish in the future. Bekhradnia (2016), writing recently for the Higher Education Policy Institute, argued that university rankings were dangerous but that it would appear ‘Canute-like to try to halt their rise’ or correct ‘their most serious flaws’.

How rankings are determined: some examples

Whilst rankings may seem objective, it is important to note that, in each case, the compiler has made subjective decisions on what constitutes quality, the ‘good’, ‘excellent’ or ‘best’, what is to be measured and what weighting is to be ascribed to each measure. Slight alterations to measures or weightings can produce very different results. Measures that value widening access rarely, if ever, feature.

The Guardian University Guide 2017 methodology

The Guardian uses eight statistical measures to approximate each university’s performance in teaching in each subject and to produce a score for ranking. Each institution is also given an overall score by averaging those subject scores to produce an institution-level table (more details can be found here). The rankings are based on official data collected by HESA (Higher Education Statistics Agency) including the Destination of Leavers from Higher Education Survey and in the National Student Survey published by Hefce.

Measure	Details	Weighting	Weighting for Medicine, Dentistry and Veterinary Sciences
National Student Survey ( NSS) score for teaching	In this annual survey of final year, first degree students, they are asked the extent to which they agree with four positive statements about their experience of teaching in their department.	10%	14%
National Student Survey ( NSS) score for assessment and feedback	The NSS asks for responses to five statements about how students were assessed and the usefulness of feedback.	10%	14%
National Student Survey ( NSS) score for overall satisfaction	The NSS asks students to provide an overall score regarding their satisfaction with the quality of their course.	5%	7%
Value Added	The Guardian describes this as a ‘sophisticated indexing methodology’. Students are given a probability of achieving a 1st or 2:1, based on their entry qualifications. If a good degree is obtained, points are awarded to reflect the difficulty of that achievement.	16.25%	5%
Student-staff ratio	The Guardian acknowledges that research intensive universities often have staff who spend significant amounts of time on research rather than teaching who will, nevertheless, be counted within this measure (research only staff are excluded).	16.25%	23%
Expenditure per student	Amount spent providing a subject (excluding costs of academic staff) divided by the number of students, plus institutional spend on services divided by total student population.	10%	14%
Entry scores	Average tariff scores of first year, first degree, full-time entrants aged under 21. Excludes departments dominated by mature entrants. Certain qualifications and access routes are excluded.	16.25%	23%
Career prospects	Proportion in further HE or professional study or graduate-level employment within 6 months of graduation.	16.25%	0%

THE World University Rankings methodology

The Times Higher Education world ranking claims to judge research-intensive institutions across all their core missions along 13 performance indicators to provide the most comprehensive and balanced comparisons.

Times Higher Education (2017)

U-Map Methodology

Rather than ranking institutions, the U-Map project aims to create a non-hierarchical classification of European institutions. The dimensions and indicators used are grouped into: teaching and learning profile; student profile; research involvement; regional engagement; involvement in knowledge exchange and international orientation. Institutional profiles can be viewed against one another and by focusing on a particular measure of interest, such as the proportion of mature students. Although still in development, the project aims to increase transparency whilst demonstrating the diversity of higher education institutions in Europe (more details can be found here). Such an approach could mitigate some of the issues around access, allowing for the recognition of excellence in this area.

Issues concerning the underlying data and measures used by rankings and league tables

A number of commentators have identified weaknesses in the measures and data used to compile rankings. Bekhradnia (2016), for example, has argued there should be no confidence in the quality of the data underpinning many global rankings. In many instances, universities supply their own data without any external audit. Where this data is not provided, some rankings simply include what publicly available data they can find, inevitably creating issues around quality control.

In an analysis of league tables and their impact on English institutions, HEFCE (2008) noted, ‘the measures used by the compilers are largely determined by the data available rather than a clear and coherent concept of excellence’.

The methodology employed by compilers can be difficult to understand, despite claims that league tables promote transparency. Many league tables provide some information on the measures, weightings and formulae employed, but it is often impossible for others to replicate the approach and thus understand its detailed working (Rauhvargers, 2011).

Survey questions, answers and details of respondents are often not publicly available for example. Concerns have been raised about the value of some of the indicators or measures employed, particularly by the global rankings. ARWU, for example, uses the number of Nobel prize winners as a proxy for the quality of teaching (Rauhvargers, 2011), a link which is indirect at best. Many, such as QS, employ ‘reputation surveys’ as measures of quality. Universities can nominate academics to complete the surveys, raising questions of peer review bias (Bekhradnia, 2016). Respondents select reputable institutions from a pre-prepared list and, in some years, just 5% of those contacted completed the survey (Rauhvargers, 2011). The value of such an approach in measuring the teaching or research quality of an institution can be questioned.

Concerns about league tables and rankings and their relationship to fair access

A number of concerns have been raised about the nature of league tables and their impact on institutions and policy makers. There are often negative implications for fair access, which is of particular relevance to this discussion.

1. All institutions are judged on the same definition of quality or excellence

The majority of league tables prioritise research excellence. They use data which they claim allows them to compare research performance, impact or reputation amongst institutions. It is possible to debate the extent to which they accurately measure this standard but that assumes it is worthwhile to do so. Not all universities prioritise, or would wish to be judged on, research alone. Yet in many league tables, this is the most important determinant. Those who prioritise regional engagement or some other mission, such as widening access, will not be fairly measured in these circumstances. League tables do not report on whether an institution is good at what it does; they aim to measure whether it has demonstrated quality by their definition.

2. Institutions can alter their missions

As a consequence of the point above, institutions can alter their mission or policy priorities to align with those measures valued within rankings. They may undertake activity specifically to improve their position within a league table and away from a previous mission. This is particularly relevant when efforts to improve rankings have a negative impact upon another agenda, such as fair access. In many rankings, the entry rates of incoming students are an important measure. The greater the previous attainment of students, the higher institutions score on this measure. If an institution introduces contextual admissions thus encouraging entry to those from disadvantaged backgrounds with lower attainment, this will have an adverse impact on their score. They are either penalised for undertaking a vital social mission or prioritise their position on a league table at the expense of access.

3. Social mobility is not currently reputational for institutions

Whilst there have been some efforts to include access within league tables, such as the Guardian’s ‘value added’ score, many measures run counter to it. Institutions that perform exceptionally well in relation to widening participation will not find this reflected in their ranking. In 2016, in its annual State of the Nation report, the Social Mobility Commission argued that a social mobility league table should be published annually to rank different institution’s performance in relation to widening access and to celebrate those who were undertaking effective activity in this area.

4. Hierarchical rankings judge quality only in relation to others

Where league tables publish results in an ordinal list, they have made judgements of institutions’ performance or quality in relation to each other. In these circumstances, quality becomes relational rather than substantive; it is not possible for all institutions to be excellent. One must be the best with others declining in quality as indicated by their position in the ranking. Two universities may have differences in their spend per student or ratio of doctoral to undergraduate students but it does not follow that one is necessarily better than the other. As Brink (2010) has argued, ‘ranking reflects the purpose of the ranker at least as much as any inherent properties of the entities being ranked… there is no reason to assume that quality inherently presents itself in a linear order’.

5. Evidence of ‘gaming’ the system

Rauhvargers (2011) pointed to concrete examples of institutions manipulating data or taking action to artificially inflate their scores, such as encouraging applicants with no real chance of success in order to improve on selectivity measures. Hazelkorn (2015) has argued that ‘allegations and admissions of gaming are most prevalent in the US but there is no reason to believe they don’t occur elsewhere’.

6. Reinforcing existing reputations

With many league tables relying on reputation surveys completed by academic staff or employers, they can reinforce existing notions of prestige. An employer may not have an accurate insight into a range of institutions and could instead merely note those universities they have heard most about. HEFCE’s (2008) comparison of two global rankings found that whilst they had only one indicator in common, they both came to very similar conclusions in terms of their top rated institutions. In the latest rankings for both THE and ARWU, for example, Edinburgh was ranked within five places in both top 200 lists and both placed Stanford and Cambridge within the top three. Newer institutions, and those prioritising access, are likely to perform poorly when judged by general reputation alone.

Commissioner’s commentary

League tables and other rankings now have a big impact on the behaviour of universities - too big perhaps but that particular horse has bolted and there is no possibility of closing the stable door now. Indeed there has been an exponential proliferation of league tables producing an almost Baroque profusion of variants.

The Commission on Widening Access recognised the potentially chilling effects of league tables on fair access. One of its recommendations charged me as Commissioner for Fair Access with initiating a dialogue with the compilers of league tables. This discussion document is designed as a contribution to that dialogue, but also to stimulate wider public debate.

There is no better illustration of the impact of league tables than the fact that several Scottish universities participated in the (English) Teaching Excellence Framework. There was no compulsion to do so (probably the opposite). There are also growing doubts about the TEF’s validity and methodology. Finally there are better, more collegial, ways to promote good teaching, as set out in QAA Scotland’s new enhancement priorities. But the prospect of achieving ‘gold’ standard was just too alluring, and the need to have the same kite mark as English universities too great. Above all the TEF will be a key ingredient in future league tables.

So how league tables are constructed matters. With one honourable exception they are based on a basket of metrics, hard data derived from official statistics (on research performance, student satisfaction, entry grades, facilities expenditure and so on) and soft data utilising ‘reputation’ surveys. These are then weighted to produce a single score. Then these scores are ranked in order. The honourable exception is U-map which produces a more nuanced multi-factor profile. Perhaps for that reason it has not caught on - there need to be winners (and losers) that shift around from year-to-year (although not beyond the bounds of credulity - ‘top’ universities need to be at or near the ‘top’).

There are two problems with league tables from the perspective of fair access:

First is that one-size-fits-all single score. Academically selective (and socially exclusive) universities clearly face different, and greater, challenges from institutions committed to widening participation. Also fair access depends on the efforts not only of single institutions but also on collaboration and building pathways between institutions in cities and regions;

Second, efforts to devise a proxy metric for fair access have proved unsuccessful so far. Measuring value added or learning gain are technically difficult, although not more so than measuring genuine teaching quality. I also suspect that, even if such proxies became part of the league-table metric mix, they would still get swamped by measures of performance.

I draw three conclusions from all of this:

The first is that there is no escape from league tables. They are not just the result of the commercial ambitions of the media and other compilers, who of course want to sell other services on the back of them. They are also part of a wider audit and performance culture - and so a key input into how universities are managed. League tables also impact just as heavily on institutional brands and staff morale as on student choice (perhaps more heavily). Finally they resonate around the world, maybe more loudly the further away from home;

The second is that, despite the difficulties, efforts to devise proxy measures of fair access such as value added and learning gain should be redoubled. Pressure needs to be brought to bear on the compilers of league tables to include them, and give them more weight. More difficult to achieve but still worth arguing for is greater encouragement, and use, of multi-factor institutional profiles such as U-map;

The third is a wilder idea. Perhaps those working to achieve fairer access, nationally and in institutions, should devise a separate ‘Fair Access’ league table. After all there are league tables of ‘green’ universities that have gained significant traction. No university wants to be at the bottom of any league table, and a ‘Fair Access’ ranking would boost some institutions discriminated against in existing league tables.

Peter Scott
Commissioner for Fair Access