Modern science is all about evaluation these days – finding ways to work out whether researchers are giving true value for money. It's no easy task to measure success in the scientific sphere, and attempts to quantify something so complex and subjective using numbers always seem divisive and controversial.
I recently voiced my suspicion that some UK universities might be assessing their staff for the upcoming Research Excellence Framework (REF) exercise in a superficial manner expressly forbidden by its overseers. To learn a bit more, I asked readers in academic research in the UK to participate in an informal, unscientific poll to see how widespread the practice of using journal rankings or impact factor to evaluate the worth of individuals' research actually was. (The way the poll was designed, each question is independent, so the percentages refer to respondents to that question. All questions except number 6, which referred to a special case, had more than 200 respondents.)
The results are now in, and the trends largely support my initial concerns. The majority (64%) had had their papers assessed primarily using the forbidden "bibliometrics", or knew a colleague who had experienced this (72%). The reciprocal questions yielded largely comparable results. This isn't surprising considering that over half (57%) of respondents reported that their university representatives had admitted outright that this outlawed methodology would nevertheless be used.
Meanwhile, two-thirds (66%) said they didn't know exactly what methods had been used to judge their papers but were unhappy with the result. Most interesting of all, I think, is what happened to the 175 people who weren't happy with their initial assessments and complained: only 15% were able to secure a re-assessment using alternative methods.
So bibliometrics – despite being proven to be flawed for the purposes of assessing individuals – are still rearing their ugly head in the upcoming REF – and presumably for other important exercises like who to hire, promote, and shower with grant money.
This practice will never end until researchers themselves decide they've had enough and put a stop to it.
Jenny Rohn runs a cell biology lab at UCL, and tweets as @JennyRohn.