Naughty Twins and the Impact of Journals

Authors often assume that publishing their results in high-impact journals greatly increases the number of citations their articles will receive.

This assumption has been difficult to demonstrate empirically.  Highly-cited articles may appear in prestigious journals because of careful vetting and selection.  Alternatively, prestigious journals may have a lot to do with the citations a paper receives.

Separating cause and effect is difficult.

Measuring the effect of the journal on the articles published within requires controlling for the intrinsic “quality” of the article, which is not easy to do since each journal contains a set of unique articles.  Any analysis always comes down to comparing apples and oranges.

Publishing identical copies of articles in separate journals and observing their performance would solve the problem in methodology and allow apples to be compared with apples.  But multiple publication is largely considered an unethical practice.  Most academic journals stipulate that submitted manuscripts must be original and not previously published (also known as the Ingelfinger Rule).

And yet, republication happens, and the outcome of these duplicate articles creates a natural experiment upon which to answer the question of how much citation impact a journal exerts on its articles.

In their manuscript, “The impact factor’s Matthew effect: a natural experiment in bibliometrics,” released August 21st on the arXiv, Vincent Larivière and Yves Gingras, both at the University of Quebec in Montreal analyzed 4,532 pairs of identical papers published in two different journals.  By comparing identical papers, the authors were able to control for article quality and focus on the citation effect of the journal. Their approach is no different than measuring the effect of environment on human development when identical twins are raised in separate households.

The 4,532 pairs of papers were identified in ISI’s Web of Science using automated software designed to match titles, first authors, and number of references.  In the spectrum of big-picture at one end and granularity at the other, this was definitely a big-picture study.

Larivière and Gingras discovered that articles published in the higher-impact journal received, on average, twice as many citations than those articles published in the lower-impact journal (11.9 vs 6.3).  The percentage of uncited papers was also lower for the high-impact journals.

What is the cause of the different citation patterns?  The authors comment:

We know that many scientists look up the impact factors of journals in order to choose where to publish; they also tend to read journals with high impact factors in their own field and thus the papers in these journals become more visible.

In their manuscript, the authors imply that unethical authors are the cause of the natural experiment, submitting their work to multiple journals simultaneously.  This was not the case of Emerald Publishing, who engaged in systematic republication of hundreds of articles in their journals over several decades.

We also do not know if any of these republished article-pairs included planned statements of republication (such as “originally published in XXX”) with the result that the citer would give credit to the original publication.  (In the case of Emerald’s republications, there were no such indications.)

Their manuscript will be published in a forthcoming issue of the Journal of the American Society for Information Science & Technology (JASIST).  The authors have no intention of republishing it elsewhere.