Research Excellence Framework (REF)

By Bruce Philp

The REF is a process which is used to allocate central government research funding in UK universities.  It is the latest incarnation of the Research Assessment Exercise (RAE), which was initiated by Margaret Thatcher’s Government, and conducted in 1986, 1989, 1992, 1996, 2001 and 2008. One heterodox interpretation on the RAE has been offered by Lee (2009): ‘The RAE is essentially driven by the pro-market ideology adopted by the Thatcher, Major and Blair administrations since 1980 that universities are or should be like business enterprises, wealth creators as well as supporters of enterprise culture, and responsive to the needs of industry’ (p.184). And, this emphasis on internal markets and competitive processes resonates with other public sector innovations in the UK in this period.

The RAE and REF has come to dominate research activity in UK universities. Most significantly for economics research, it is the setting which drives academic appointments in UK university departments. The stated purpose of the REF, according to the Higher Education Funding Council for England (HEFCE), is as follows:

  • The funding bodies intend to use the assessment outcomes to inform the selective allocation of their research funding to HEIs, with effect from 2015-16.
  • The assessment provides accountability for public investment in research and produces evidence of the benefits of this investment.
  • The assessment outcomes provide benchmarking information and establish reputational yardsticks. (HEFCE 2011).

Heterodox economists would generally claim that the notions of “benchmarking” and “reputation yardsticks” are extremely constraining in an environment, such as economics, which is dominated by a single “mainstream” (or neoclassical) school of thought. For example the uncritical acceptance of the tools and ideology of this approach has prevented meaningful evaluation of contemporary market structures and governance. Meanwhile, it is these institutional forces — with recent emphasis on “impact” — which also provide the lifeblood for mainstream economics. It would be a Faustian pact, but for the fact that mainstream economics is severely limited in terms of the knowledge it creates.

Research Assessment Exercise (RAE) panels were categorised as Panel 34 for Economics and Econometrics, and Panel 36 for Business and Management. Many economists are entered into Panel 36, rather than Panel 34, though the process of cross-referral of submissions implies that ultimately the Economics and Econometrics Panel determine the landscape of economics research. In part the distrust of the process has been heightened in recent years by the fact there is asymmetric information. In previous RAE panels there have been identifiable heterodox economists. In the 2008 RAE there were no identifiable heterodox economists on the panel. And, in the 2014 Research Excellence Framework (REF), again, there are no panel members who are identifiable as heterodox, even peripherally (see Table 1).  For a discipline in schism — comprising heterodox and mainstream subdisciplines — the fact that one school is in complete control of funding creates such a bias it is stifling debate, systematically destroying the capability of the discipline to evolve and transform to provide a meaningful analysis of the contemporary socio-economy.

Sub-panel 18: Economics and Econometrics
Chair
Professor Peter Neary University of Oxford
Members
Professor Oriana Bandiera London School of Economics and Political Science
Professor Timothy Besley London School of Economics and Political Science
Professor David Blackaby Swansea University
Professor Stephen Bond University of Oxford
Professor Sarah Brown University of Sheffield
Professor Simon Gaechter University of Nottingham
Professor Rachel Griffith University of Manchester
Professor Andrew Harvey University of Cambridge
Professor Campbell Leith University of Glasgow
Professor Ben Lockwood University of Warwick
Professor David Miles Monetary Policy Committee
Dr Jorge Padilla Law and Economics Consulting Group
Professor Stephen Pudney University of Essex
Professor Alan Sutherland University of St Andrews
Professor Jonathan Thomas University of Edinburgh
Professor Ian Walker Lancaster University
Panel Secretary
Mr Simon Kerridge University of Sunderland

Table 1: Economics and Econometrics REF Panel

As a consequence of these pressures one of the problems which heterodox economists encounter is the apparent reluctance of leading research-orientated University departments to appoint them. This has been highlighted in the press by leading heterodox economists (for example Goodacre 2010; Dow 2010). The position is summarised succinctly thus:

Economics would … be stronger if it supported a range of approaches from which to draw.

But this is not the mainstream view of the discipline. There are disagreements within this view, but these are limited by the presumption both of independence from value judgments and of scope (in principle) for demonstrably best answers, implying that discussion about approach to economics is of little importance. This view has been strengthened in the UK by being institutionalised in the assessment of academic research, which has directly influenced government funding, and thus hiring incentives. (Dow 2010)

Does this, then, imply that the academic job market excludes heterodox economists? Plainly it is discriminatory, but it is especially so at the “top-end”, in institutions where the greatest weight (and time allowances) are given for research.

One of the problems of the RAE and the REF is that there are no official lists ranking journals and publications. And, the “mainstream” panel grade research behind closed doors, with only institutional results published. Given the nature of such deliberations, and the asymmetric information implied, university departments use various journal ranking lists — which are not officially endorsed by HEFCE — to inform their REF strategy and hiring policy. The HEFCE categories of attainment for individual publications are given in Table 2.

Grade Description
4* Quality that is world-leading in terms of originality, significance and rigour
3* Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence
2* Quality that is recognised internationally in terms of originality, significance and rigour
1* Quality that is recognised nationally in terms of originality, significance and rigour
U Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

Table 2:RAE/REF Grades

In economics, in the UK, there are two lists which are particularly influential:

  1. The Association of Business Schools (ABS 2010) List;
  2. The Keele List (Keele University, 2006).

If we consult these lists it is noteworthy that no economics journals which generally accept heterodox submissions are rated at world leading (4*) on either list. The inference from those who have complied these lists is clear: there is no world leading economics research in the Austrian, feminist, green, institutionalist, Marxian, post-Keynesian, radical or Sraffian economic traditions (or, indeed, any other heterodox approach).

The ABS (2010) list includes journals in non-economics business disciplines which may be open to heterodox approaches (for example Work, Employment and Society), but very little listed under economics accepts heterodox work. The Keele List actually grades the Economic Journal at 3*, along with the Cambridge Journal of Economics and the American Journal of Economics and Sociology. But, on both lists, mainstream American journals predominate in the 4* category, including the American Economic Review, Econometrica, Quarterly Journal of Economics and the Journal of Political Economy. It is also the case that the Journal of Economic Perspectives and the Journal of Economic Literature are included, even though the latter is a journal which does not have an open submission policy.

It also goes without saying that books don’t figure on the list, and economics departments and business schools are loath to enter such work into submissions. The irony of this, given the history of the discipline, is clear. One can imagine Adam Smith’s head of department at Glasgow informing him that An Inquiry into the Nature and causes of the Wealth of Nations is all very well, but books don’t count. Keynes, likewise, may have been told that Cambridge were including his journal articles in the submission, but The General Theory of Employment, Interest and Money would not be entered because it wasn’t bite-size enough.

Of course, there are other ways in which heterodox economics is marginalised. Almost all appointments in economics departments in leading UK-research institutions, explicitly or implicitly, seek mainstream economists. In policy terms the economists who are consulted are mainstream economists who use econometric methods underpinned by rational choice methodologies, assuming agents are self-serving and egoistic. In short, mainstream economics occupies a position of dominance across the practitioner and academic profession which organisations such as the AHE have been challenging, largely unsuccessfully, for a number of years.

The hierarchies of journals, and journal lists, are significant. However, there are limited opportunities for non-mainstream economists in intermediate-ranked journals. If we begin by considering journals at 2* they include Journal of Macroeconomics, the European Journal of Political Economy and Applied Economics (all mainstream). However, in addition, they include Review of Social Economy and International Review of Applied Economics, both of which are leading heterodox journals. The Keele list ranks the American Journal of Economics and Sociology, and the Cambridge Journal of Economics, at 3*. In social science based economics departments, which perhaps see the Keele list as a better indication of journal quality, this bears comparison with journals such as the Economic Journal and Oxford Economic Papers. But, these examples are isolated. World leading heterodox journals such as Journal of Economic issues and Review of Political Economy are graded at 1*, which is plainly bizarre (for a list of top heterodox journals see Lee and Cronin 2010).

There is the separate issue of particular sub-disciplines and the broader issue of heterodoxy. Goodacre, for example, points to the failure to recognise research on the history of economic thought as a particular problem of some of the leading lists.  We might also point to the discipline of economic methodology, where the Journal of Economic Methodology scores at 1* on both the ABS and the Keele lists.

The problems are significant for heterodox economics. Going forward, the AHE will continue to make representations to HEFCE and other organisations highlighting the discriminatory nature of research assessment in UK economics departments. This is why it is so important to support the organisation, through membership, the AHE Conference and postgraduate workshop. However, for individual researchers, the problems remain profound. The following should be borne in mind:

  1. The process of cross-referral and triangulation between panels implies it may well not be possible to avoid the power of the mainstream Economics and Econometrics REF sub-panel. This, then, suggests that the only effective way to improve the situation is to work to change it through organisations like the AHE, and other heterodox groups.
  2. Nevertheless, there is some scope for heterodox research to attain intermediate grades through identifying those journals, on the main lists (e.g. ABS and Keele), which are open to heterodox work.

In the long run it is only through sustained effort that we, as a heterodox community, can improve the situation. However, pragmatic responses need to be adopted by individual researchers in the short-run, to ensure the ground of economics remains contested.

ABS (2010). ABS Academic Journal Quality Guide Version 4. Available at: http://www.associationofbusinessschools.org/node/1000257 [accessed 13/12/11]

Dow, S. (2010). Nature and scope of economics is subject for wider debate.Financial Times, Letters, 18th August.

Goodacre, H. (2010). Some economists were never gurus or on a throne.Financial Times, Letters, 8th September.

HEFCE (2011). Research Excellence Framework. Available at: http://www.hefce.ac.uk/research/ref/ [accessed 12/12/11]

Keele University (2006). Keele Four-Four-Two List: Rankings of Economics Journals http://www.keele.ac.uk/cer/documents/KeeleEconJournalRanks442.pdf [accessed 13/12/11]

Lee, F. (2009).A History of Heterodox Economics: Challenging the mainstream in the twentieth century. Routledge: London.

Lee, F., and Cronin, B. (2010). Research Quality Rankings of Heterodox Economics Journals in a Contested Discipline. American Journal of Economics and Sociology 69(5), pp.1109-1452. AJES-Nov-2010-RankingJournals.pdf