REF Results and Research in the UK
In December 2014 results of the UK “Research Excellence Framework” (REF) were released, and now the funding councils for England & Wales and for Scotland are making decisions in light of these results. The REF is the latest version of the periodic ordeal suffered by UK scholars and universities in which departments submit up to four publications (from the preceding period since the previous such review) of each member of academic staff who is “submitted” for the review. These publications are rated by committees appointed for the particular subject areas/disciplines (or combinations of smaller fields of study) on a scale of 0 to 4 (4 = a publication of international significance that contributes in some noteworthy way to knowledge of a given subject, i.e., the sort of publication that everyone working on the topic should read and that will likely be cited for years/decades to come).
The REF exercise (formerly called the “Research Assessment Exercise” or RAE) produces ratings of departments, not named individuals. UK universities are funded by public funds in two streams or “envelopes”: A certain amount per student (the teaching “envelope”) and a certain amount in the research “envelope.” The latter amount is calculated by the funding councils using REF results. Indeed, the major point of the whole exercise is to give the funding councils a way of judging how much of the latter “envelope” to award to each university.
There are also some curious features of this whole exercise that show that it really has little to commend it as an academic assessment of publications and departments.
For example, these committees are expected to judge publications that in a good many cases have been published only short months (or even weeks) before the submission deadline. How a committee can judge in advance that a given publication is to become seen, for example, as discipline-changing so early in the life of the publication is a mystery to me. But, it’s what the government says must be done, and so . . . it’s done.
There is also some game-playing by university departments. As I’ve noted, the results don’t name individuals, but give a department-based assessment. Departments want to obtain a high rating, of course, both because of the prestige factor (the results typically used in advertising and recruitment) and because there are the financial consequences mentioned already. Departments, however, are free to submit as many or as few of their academic staff as they wish. So, if a given department has only some, or a few, people whose publications are likely to get high ratings, they can submit only these people, hoping to get a collective high rating for the department.
But the announced results don’t indicate what percentage of members of a department have been submitted. In Divinity here in Edinburgh, we submitted 90% of academic staff this time. But some places submitted much smaller percentages. This could give a misleading impression, the REF results perhaps reflecting the research of only a small portion of a given department. (Prospective postgraduate students take note.)
So far as biblical studies is concerned, this REF got off initially to a curious start. When the committee was initially announced, there was no one recognizable as a contributing biblical scholar on it. After strong representations to the body responsible for the REF (English Higher Education Funding Council) and the chair of the relevant committee, eventually two respected biblical scholars were added to the committee. This was a curious situation because I am reliably told that in this REF the publications in biblical studies comprised some 19% of the total of published works submitted in Theology and Religious Studies. Given that this committee received submissions in all areas of, and approaches to, religion studies and in all religions, that 19% has got to be one of the largest single subject areas, if not (perhaps) the largest. Anyway, it’s done, and now the funding councils will apportion funds according to their chosen policies. As Emeritus Professor, I’m out of the whole ordeal now, and thoroughly sympathetic with colleagues who aren’t.
Comments are closed.
Larry Hurtado's Blog 
The two things that really amaze me about the current REF are:
1) that the proportion of top-rated research overall has gone up so much (the Grauniad said it was up to 22% from 14%; this seems reasonable, but I haven’t checked further); and:
2) that with so many universities playing a clever selective submission game, we didn’t end up with one of the red bricks inadvertently beating Oxbridge and London.
Thanks yeah that sounds crazy they want to get their act together.
Perhaps a benefit of allowing departments to decide on which members of staff to submit, is that when departments make hiring decisions, they will be more inclined to hire promising junior researchers without an established publishing record. Otherwise, the departments will be more inclined to chase a small set of elite researchers. I think the funding formula that takes the product of quality and quantity takes care of the gaming problem to some extent.
In the last few RAE/REF exercises, in fact various universities did chase “elite” researchers, hiring them away from one university to contribute to the submission of another, often just months before the deadline for submissions. Yes, the funding process does take account of rating and number of people submitted, but I still think that a more even-handed process would involve a required 100% submission from all depts.
Hello Prof. Hurtado,
What recommendations would you make to improve the REF exercise? Given UK universities rely on funding from the government, and the government should apportion funds according to quality and quantity of research, something like REF is indispensible.
There is inevitably some gaming by departments to maximise the quality rating by restricting the % of staff submitted for assessment. But amount of funds is calculated on roughly the product of quality and quantity, so departments would lose out in funding by being too restrictive.
I am surprised to hear the original Theology & Religious Studies committee were not biblical scholars. Does REF publish names and affiliation of members of the assessment panels? This seems important for transparency. Perhaps the lack of biblical scholars reflects an increasing attitude that Religious Studies should become more broadly focused on the major world religions, and on socio-scientific approach to religion (though the latter category probably could be better assessed in the social sciences categories).
It seems artificial to cap the number of publications per individual to 4 as the typical volume of publications vary by disciplines.
Focusing on your discipline, how does the community of biblical scholars informally assess the quality of individual researchers and of departments? Is there an informal peer-review ranking for biblical scholarship?
Are you familiar with the funding system for state research universities in the US? Do you think UK should move towards the US model?
First, I’d require that all full-time members of a department must be submitted, regardless of what they’ve produced. That would give department ratings more readily comparable across the sector.
Given the expense of the REF (many millions of pounds), some have suggested that the research envelope decisions be made some other way, e.g., block grants to major places where the bulk of research in a given field is done. That’s highly contentious, however.
The names of the various committee members are published when the committees are set up and are available here: http://www.ref.ac.uk/media/ref/content/expanel/member/Main%20Panel%20D%20-%20Final%20List%20(Jan%202015).pdf.
In Biblical Studies (as in many Humanities fields), scholarship is heavily/particularly in book-form, and these books receive both pre-publication review and, more influentially, post-publication review. Thereafter, as one’s publications acquire positive assessment from other scholars in the field, one acquires a standing, and people in the field could readily identify top/influential scholars, although it would be very arbitrary and difficult to rank them. You could produce probably groups, but not a ranking of individuals that would be very much more than impressionistic.
But the assessment done by the REF isn’t, in my view, much better.
In the USA universities are either “public” funded (e.g., UCal Berkeley, UMichigan), which means by the individual 50 states (and in Canada by the several Provinces), or “private” (e.g., Harvard, Yale, et al.). So, because there is no central funding procedure (unlike the UK), there is no warrant for a national exercise such as the REF.
Thanks. Do you think there should be a longer interval between REF exercises, to minimise the problem of significant number of publications released too close to the cutoff date?
Back in the decades before the 1990s (the start of the era of rapid expansion of higher education in the UK), how was research funding allocated and do you think the older system was more conducive for scholars to focus on high-quality research?
Do you think Theology should be split from Religious Studies, as the former is confessional and the latter is more secular and requires no religious commitment? As I understand it, due to separation of church and state, Theology is excluded from Religious Studies curriculum in US state universities.
I can’t speak to UK higher education earlier than the late 1990s when I moved to the UK.
But I can correct one thing in your comment: Neither “Theology” nor “Religious Studies” is defined by, or requires, a confessional commitment, at least in UK universities generally. In the USA “Theology” (by which I presume that you mean broadly the study of Christian texts, ideas, etc.) is a part of major universities such as Harvard, Yale, Duke, Chicago, Vanderbilt, Emory, et alia. These are all “private” universities. It is the “state/public” universities (i.e., ones financed by the various US States) that did not have depts. of Religion at all until sometime in the 1960s. Then, to get around what was seen as the required “separation of church and state,” clever scholars proposed that “Religion/Religious Studies” was something different from “Theology”, which allowed depts. of Religion/Religious Studies” to be formed. There was some substance to this, but it was also a bit of a conjuring trick!
It’s difficult to come up with any kind of ranking based on the numbers (e.g., how do B’ham and Oxford compare?) per the REF site for Religious & Theo. Studies Any guidance would be appreciated.
There are several factors you might want to consider/explore. The REF results site lists submitting universities in Theology & Religious Studies alphabetically (http://results.ref.ac.uk/Results/ByUoa/33). It also lists the number of staff submitted (far right hand column). One question in each case: What percentage of total academic staff was submitted? You’d have to get that, I suppose, by going to the department web site and see if they list all their staff, and then compare the number submitted.
You can also compare the percentage-results, especially in the top two categories (three & four stars, the only levels that now generate funding for departments from the funding councils).
Also, if you click on the “Overall” beside each university, you’ll get a breakdown of “outputs” (i.e., the publications submitted), “impact” (i.e., the take-up or influence of research of that dept in the wider public sphere), and “environment” (which = various factors that make up a “research environment”, including, e.g., the number of PhD students, number of external awards, etc.). The last thing, “environment”, can be important, particularly for postgraduate students, reflecting how strong and well provided for may be the department as a setting for them.
I can’t believe that the REF doesn’t require every full-time member of the department be assessed. St Andrews could just, say, submit Tom Wright! Absurdity piled on absurdity. Thanks, LWH, for this characteristically informative and judicious warning…