On the Craziness that is REF
With apologies to anyone not obliged to take part in the looming UK Research Excellence Framework, and those who customarily (and rightly) expect to find on this blog site something about the New Testament and Christian Origins, the following rant about what my colleagues in UK universities are facing in the forthcoming national assessment of the research activities of academic staff.
In the Thatcher administration British universities were first saddled with a national assessment of research output and quality, the first few rounds (each about every five years or so) entitled the “Research Assessment Exercise” (RAE). This involved university departments preparing and submitting a list of academic staff, each of whom was allowed to submit up to four publications issued in the preceding time-frame for assessment by a national panel (basically, a panel for each major discipline). The public report on the assessment panels’ verdicts, however, was a collective one, each department given a composite rating.
The whole point of this exercise was so that the UK Treasury (via the university funding councils for England/Wales/N.Ireland and Scotland) could have some putatively scholarly judgment on which to base a differentiated award of research-related funding. The rating of each department had funding consequences for how much research-related income that department generated for the university.
As does any assessment scheme, this one had strong effects on the activities being assessed. It almost certainly led to a higher level of publishing activity in UK universities and across a wider swathe of academic staff. It also, however, led to “short-termism”, in which staff had to produce items within the approximately 5-year time-frame.
On an institutional level, it led to universities seeking to woo away “star” researchers, rather the way that UK football teams try to lure star players. One colleague lured from a Scottish university to a university in England just before the previous RAE deadline told me frankly, “They made me an offer I couldn’t refuse.” In another case, a colleague appointed at one university precisely to beef up their submission suddenly decided to bolt to another university just a few months before the deadline.
Another of the games played by universities and departments was to submit only a selection of their academic staff. This was intended to obtain a high collective rating for the department, but one based actually on an assessment of a selection of staff. I know of departments in which senior members of academic staff were not included in the submission, essentially because it was feared that their research output wouldn’t help the overall rating of the department. The result was a rating that was in fact not really representative of the research activity/excellence of those departments. But they were able to report (and make much of) the RAE rating as if it did really represent their academic staff. In the preparations for the most recent RAE, it was proposed (and seemed to be promised) that every participating department would have to make a 100% submission of academic staff, but that requirement went, and with it an element of integrity of the whole scheme.
Now, UK universities are gearing up for yet another turn of the Treasury-inspired screw, known as the “Research Excellence Framework” (REF), which replaces the RAE. (For a further explanation of the REF, see the Wikipedia entry here.) Once again, departments will be allowed to submit only a selection of academic staff. Indeed, a new wrinkle in the scheme will practically require it. It appears that the English and the Scottish funding councils (which use the assessment results to make funding decisions) have decided that the only research outputs that will attract their funding are those judged to be 3-star or 4-star (on a 4-star = maximum scale).
“Four-star” = research that is “world-leading in terms of originality, significance and rigour.” Three-star = “internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence” (i.e., just a shade below 4-star).
To get some perspective, in the REF scheme, a two-star rating of a publication = “recognised internationally in terms of originality, significance and rigour”, which sounds pretty damn good to me. But that’s not good enough to attract any funding in the forthcoming REF exercise.
If you take the rubrics seriously, a 4-star item would mean a publication that is a game-changer in a given field/subject, something that re-orients scholarly opinion. And a 3-star item would seem to mean something almost as profound in significance and impact. I’d imagine that most of us scholars would be delighted if, out of a career of scholarly publishing, even one of our books/articles was deemed to have had the sort of quality and effect that might appear to be represented in a 4-star rating. Indeed, when the 4-star scheme was introduced (in the previous RAE), assessment panels were strictly instructed that no more than 10% of publications should receive 4 stars.
So, we can predict what is going to happen, can’t we? Right now, departments up and down the UK are frantically trying to obtain a pre-submission assessment of the publications of their academic staff, aiming thereby to avoid submitting anyone who doesn’t have the promise of submitting a slate of 3-star and 4-star items. So, even more than in the previous assessments, we’ll have a skewed selection of academic staff submitted, which will give an even more unrepresentative picture of departments.
We’ll also likely see a certain amount of “grade-inflation”. There’s simply no way that, judged strictly, in any field, any more than a small percentage of publications deserve a 4-star or even a 3-star rating. But that won’t do. So, panels, unconsciously or consciously, will simply find that an impressive number of publications deserve such high ratings.
But among the daftness of the way the exercise is set up, here’s one final thing. Publications can be submitted so long as they’ve been published by the stated deadline, which precedes the assessment by only a few months. But how can we know that a given publication is a game-changer that changes the way scholars see a given problem or subject until sufficient time has elapsed for the publication to be reviewed, its finding assessed, and scholars then to acknowledge this effect? So, panels will be expected to make judgments about publications, in some cases very shortly after their appearance, and way before their actual significance has been established in the process of scholarly analysis.
So, if you’re not involved, have a moment of sympathy for academics in UK universities as they head into the final stage toward this periodic ordeal, this time with its own touches. ”It’s academia, Jim, but not as we know it.”