The number of times academic articles are cited by subsequent publications is among the time-honored measures used to assess scholarly standing and evaluate academic productivity. But not all of these citations are positive ones, and a paper published this week in the journal Proceedings of the National Academy of Sciences finds that as many as one in 50 citations in a top immunology journal were critical in nature.

Alexander Oettl, an assistant professor in Georgia Tech’s Scheller College of Business, is among the authors of a new study on negative citations in academic publishing.Photo Credit: Gary Meek, Georgia TechThese negative citations may point out limitations, inconsistences or flaws in previous work. The study found that these negative citations were more likely to criticize highly-read papers, and that the criticisms focused on specific sections of the papers. Negative citations tended to originate from scholars who were close to the authors of the original articles in academic discipline and social distance – but at least 150 miles away geographically.

The research, by authors at the Georgia Institute of Technology, University of Toronto and Massachusetts Institute of Technology, may be the first to systematically quantify and examine these negative citations. The authors hope to expand their study to other disciplines, and believe their work could ultimately lead to a re-examination of how citations are used in academia.

“Given that we rely so heavily on these citation metrics as measures of quality, it’s important to note that the intent of these citations isn’t homogeneous,” said Alexander Oettl, an assistant professor in Georgia Tech’s Scheller College of Business and one of the study’s three co-authors. “Criticism is really the cornerstone of the scientific enterprise, yet it’s difficult to get accurate measures of the extent to which criticism of academic papers truly occurs. The negative citations really are different from objective citations.”

Related article: Study Shows Researchers' Status Helps Some Scientific Papers Gain Popularity

The researchers, who also include Nicola Lacetera from the University of Toronto and Christian Catalini from MIT, began with 15,731 full-text articles that had been published in the Journal of Immunology, the top academic journal in that field. From those articles, they extracted 762,355 citations, which referred to 146,891 unique published papers.

Beginning with a manual process, they created a set of citations that was used to train a natural language processing program which classified the remainder of the citations as either objective or negative. The process identified 18,304 negative citations, about 2.4 percent of the total citations studied.

“The majority of these negative citations appear to take issue with the ‘results’ and ‘discussion’ sections of the papers, so they are really not attempting to overturn theory,” said Oettl. “They can be attempting to constrain results, note inconsistencies with other research, point out statistical flaws or correct other issues.”

In their study, the authors point out the relative rarity of these negative citations, and speculate that they could either demonstrate a “limited, uninfluential role” for criticism in science – or show a hesitancy to criticize.

Related article: Results of Research Ethics Investigation Reveal Editors Push Authors to Add Extra Citations

“Making these negative citations isn’t without cost,” Oettl noted. “There could be reputational harm from making these negative citations, and if your criticism turns out to be false, this could heavily impact your reputation within the field. Without this cost, perhaps we might see more overt criticism appearing in the journals.”

The researchers found that while negative citations tended to come from scientists close to the narrow academic topic, the criticism was more likely to involve geographic distance.

“We see that the probability of making a negative citation is much, much lower if you are co-located with the scientist whose work is being critiqued,” Oettl said. “That potentially speaks to the social component, the social cost of criticism – you don’t want to criticize someone you may run into on campus. Another possible interpretation is that these issues may be aired face-to-face among scientists who are located near each other geographically.”

In the study, frequently-cited papers were more likely than less-read papers to get negative citations, which stands to reason, Oettl said. “These are typically more influential papers, so this may have more to do with more people reading them and more of an incentive for scholars to take issue with important papers,” he explained. “Furthermore, less important papers may receive less scrutiny, as pointing out limitations and shortcomings of trivial work will not drastically shift the scientific frontier.”

Replication of previous work helps ensure the accuracy of research, and criticism of published studies can lead to correction that makes science more robust. But does the current scientific publishing system encourage enough discussion?

“The pessimistic view is that only one in 50 citations is negative in nature, so possibly there isn’t as much debate as we would want for a healthy discussion,” Oettl said. “But on the other hand, by the time a major manuscript makes it into print, it has undergone a tremendous amount of criticism. We now have some evidence of the extent to which criticism, in the form of negative citations, occurs within scientific manuscripts.”

Related article: Study Reveals Declining Influence of High Impact Factor Journals

The researchers chose immunology because they had access to the papers, which were consistent in format. They found similar numbers of negative citations in a small set of mathematics-related papers, but hope to expand their work to determine if their findings apply to other disciplines. Ultimately, they hope the work could change the way citations are used in science.

“We wanted to put forward a methodology to help future scholars think about how we can possibly expand this classification infrastructure and attract additional metadata to the citations that we make,” Oettl said. “Our paper is very descriptive, but it identifies something that nobody has really demonstrated before: that these negative citations behave very differently from others.”

Though the current study focused on a single discipline, immunology, “we would really like to see this work and methodology expanded to other disciplines,” Oettl said. “Ultimately, though, we care about what role these negative citations play in aiding scientific debate and in the advancement of the scientific frontier.”

The paper was published October 26 in the journal's Early Edition.