New bot flags scientific studies that cite retracted papers

Most researchers don’t intend to cite retracted papers, but it can have serious consequences for science.

2 February 2021

llus_man/Shutterstock.com

An automated tool is targeting an issue that has plagued science for decades: retracted studies cited in newly published papers without acknowledgement that they have been pulled from the literature.

Launched in November, Scite Reference Check scans new article PDFs for references to retracted papers, and posts both the citing and retracted papers on Twitter. It also flags when new studies cite older ones that have issued corrections, errata, withdrawals, or expressions of concern.

An extension of scite.ai, a platform that monitors whether research findings are supported or contradicted by subsequent studies, Scite Reference Check keeps track of retracted papers by trawling through Crossref, PubMed, and the Retraction Watch Database.

Citing retracted papers can be legitimate, if it’s clearly acknowledged in the text that the cited paper has been retracted. But this is usually not the case.

An October 2020 study in Scientometrics is the latest of several studies that have suggested that as few as 4% of citations to retracted papers clearly recognize the retraction.

According to a recent report by Science, COVID-19 papers are also affected. The analysis found that 52.5% of recent articles citing two high-profile COVID-19 papers that were retracted by The Lancet and The New England Journal of Medicine did not mention their status.

It's not yet clear why more papers referenced the retracted studies appropriately in this example than in the Scientometrics analysis, but media attention could have helped to highlight the status of the retracted papers to the citing authors.

A tool such as Scite Reference Check could be a “major boon” in the prevention of citation of retracted publications, says John Budd, emeritus professor of information science at the University of Missouri in Columbia.

Budd co-authored a 2011 study that found that fewer than 4% of citations to 1,112 retracted papers published between 1997 and 2009 acknowledged that the cited paper had been pulled.

“It’s rather amazing that it’s taken this long for formal tools to be developed to deal with the issue," says Budd.

Easy mistake with serious consequences

There are many reasons why citations fail to acknowledge a paper’s retraction. Most often it’s due to ignorance, says Josh Nicholson, co-founder of scite.ai. “I don’t think authors that are citing these things are intentionally doing it.”

Authors may not be aware that the study they are referencing has been retracted, because the journal hasn’t made it clear on its website. Or they may cite a paper without reading it properly, because they’ve seen it referenced in a related study.

Jodi Schneider, an information scientist at the University of Illinois Urbana-Champaign in Illinois, who co-authored the Scientometrics analysis, says it’s an easy enough mistake for authors to make. “Even though I care an awful lot about not unintentionally citing retracted stuff, I don’t go and manually look at the references of manuscripts to check if they’ve been retracted.”

The ramifications of citing retracted papers can carry through to secondary literature, says Schneider. For instance, if a meta-analysis references a retracted study without acknowledging that it has been pulled from the literature, the meta-analysis itself becomes flawed.

In December 2020, Retraction Watch reported that a meta-analysis about heart disease was retracted after the authors realized that one of their reference studies had been pulled.

Part of growing efforts to automate checks

Scite Reference Check costs users around US$1 per manuscript. Some journals are already using the tool, such as Acta Orthopaedica. Nicholson says he’s planning to roll out the tool to a preprint server that will give authors posting papers on the platform access to it.

One limitation of Scite Reference Check is that it cannot distinguish appropriate citations (that acknowledge that the paper has been retracted) from inappropriate ones that don’t. Schneider doesn’t think that’s a big problem, because her research suggests that only about 5% of post-retraction citations fall into the former category.

There are similar tools already on the market, such as Zotero, a free open-source reference management system that alerts authors when they cite a paper that has been flagged by the Retraction Watch Database.

Dutch publisher Elsevier has a tool that automatically extracts citations from articles at the submission stage and validates them with PubMed, Crossref, and Scopus. An additional tool, which generates a specific alert for retracted citations, is under development.

Another team led by Ben Goldacre, a clinical epidemiologist at the University of Oxford in the UK, is developing a similar tool called RetractoBot. And the Minsk-based firm RedacTek is creating a retraction analysis tool that assigns a ‘retraction association value’, which is calculated by measuring the retraction status of primary, secondary and tertiary citations.

For those who just want to check a small number of references, the free-to-use database Open Retractions allows users to manually check if a specific article has been retracted, revised, updated, or had a note of concern published about it.

Editor’s note: The freelance author of this article was a reporter for Retraction Watch in 2016 and still does some work for its parent organization, the Center for Scientific Integrity. He was not involved with the development of the retraction database.