Scientific papers depend on readers trusting their knowledge. That’s why it’s disturbing that a new study by researchers affiliated with Cornell and UCLA found 146,900 false AI-generated citations in scientific papers hosted in all four major research databases.
The main limitation of large language models such as Gemini and ChatGPT is their tendency to produce information that sounds good but is not accurate, something known as hallucination. If a researcher relies on a chatbot to write citations without self-verification, the model may generate entirely fabricated references.
Although scientific papers are often hidden from the public eye, the research they report has a profound impact on our lives. Everything from the Internet to lithium-ion batteries started as a research paper.
But when scientists submit papers citing AI inventions, it can erode faith in the quality of the research.
Simple science
The research team analyzed 111 million references from 2.5 million scientific papers. Look for citations with titles that the group cannot match with any published material. Although some of these cases were spelling mistakes, the team also found some missing comments.
Unreliable researchers had fake citations long before the advent of chatbots, so the team also examined the numbers of unverifiable citations in studies published before 2023, when chatbots were not yet ubiquitous.
“We find a significant increase in missing indicators following LLM admission,” the authors wrote in the paper.
The team also found that bad citations were spread across many papers rather than concentrated in a few. That suggests the problem is widespread, as many researchers rely on AI-generated references without fully verifying them.
Warning signs
Usha Haley, a professor of management at Wichita State University, told CNET via email that she sees the rise of fake citations as a serious warning.
“False or AI-generated citations undermine the credibility of the scholarly record that provides the foundation upon which peer review and collated information are derived,” Haley said. “Disturbingly, this skepticism is now coming from within academia itself and early career scholars.”
The four databases where researchers find false citations are arXiv, bioRxiv, SSRN and PubMed Central. These organizations, known as scientific repositories, play a major role in the world of research.
Before a paper is published in a scientific journal, authors often upload it to a scientific repository, increasing its visibility and allowing the global scientific community to quickly access it. A new paper on AI citations is currently hosted on arXiv.
Recently, arXiv has taken steps to stem the flow of false citations. The organization announced on Tuesday that it will ban authors who post work with citations or any sign of AI content that has not been properly vetted.
“The scientific corpus is shrinking. A lot of AI stuff is either wrong or meaningless. It’s just noise,” arXiv science director Steinn Sigurdsson told CNET’s Katelyn Chedraoui in February. “It makes it hard to figure out what’s really going on, and it can misdirect people.”
