Should scientists rely on artificial intelligence to assist with research or help draft scientific papers? Most journals forbid listing A.I. as an author, and many conferences prohibit presenters from using A.I. tools outright. But on October 22, a bold—and controversial—experiment challenged those norms: an entire scientific conference in which every submitted and reviewed paper involved substantial A.I. participation. Even if humans contributed, the A.I. models were required to perform enough work to qualify as something akin to the study’s lead author.
“We’re witnessing a real shift in thinking,” James Zou, a computer scientist at Stanford University and co-organizer of the Agents4Science conference, told Kathryn Hulick at Science News. “People are beginning to treat A.I. as a potential co-scientist.”
The conference itself functioned as a research experiment. All submissions and their accompanying reviews—both by A.I. models and humans—were made publicly accessible. “Those of us who work with A.I. need to better understand where these systems excel and where they fall short,” said Margaret Mitchell, an A.I. ethics specialist at Hugging Face, speaking to Elizabeth Gibney at Nature. She noted that A.I. is prone to generating false positive findings. “Knowing how to evaluate A.I. agents at all remains an open research question.”
The virtual event drew roughly 1,800 registered participants, according to Jeffrey Brainard at Science. A total of 315 A.I.-generated submissions were reviewed by a panel of A.I. reviewers. Eighty papers advanced past this first stage before human reviewers helped narrow them down to 48 final selections. The overarching aim was to investigate whether A.I. can independently produce meaningful scientific ideas, hypotheses, and methodologies—and whether an A.I.-driven peer-review system can assess quality effectively.
“There’s still a stigma surrounding the use of A.I. in research, and many people feel pressured to conceal or downplay it,” Zou told Science. The organizers hoped that by conducting the experiment openly, they could begin gathering reliable data to address these questions.
Most accepted papers explored topics in artificial intelligence and human learning, per the conference website. Zou told Nature that the majority were computational in nature rather than based on laboratory experiments.
Three studies received the event’s “outstanding paper” award: one analyzing the behavior of A.I. agents in economic markets; another evaluating how reduced towing fees in San Francisco affected low-income residents; and a third probing whether an A.I. system could deceive A.I. reviewers by submitting flawed research.
Despite the enthusiasm surrounding the experiment, critics remain doubtful that A.I. can meaningfully conceptualize or author scientific studies. “If the authors and reviewers are A.I., then perhaps the attendees should be A.I. as well, because no human should mistake this for real scholarship,” said Raffaele Ciriello, a digital innovation researcher at the University of Sydney, in a statement issued before the conference.
“Science is not a factory that transforms data into conclusions,” he added. “It is a shared human effort grounded in judgment, interpretation, and critique. Turning research into an automated pipeline where hypotheses, experiments, and papers are generated and evaluated by machines reduces science to hyper-empiricism.”
Even researchers who used A.I. in their own submissions advised caution. Min Min Fong, an economist at the University of California, Berkeley, who co-authored the towing-fee study alongside A.I., told Science News that although A.I. greatly accelerated computational tasks, it also made fundamental errors. For instance, the model repeatedly misidentified the date when San Francisco’s new towing fees began. “The core scientific work is still fundamentally human,” she said.
Earlier studies have shown that A.I. reviewers often struggle to judge a paper’s originality and broader significance as effectively as human reviewers can, Matthew Gombolay of Georgia Tech told Nature. Another challenge: many large language models are overly agreeable. They fail to provide the kind of rigorous critique and diverse viewpoints that groundbreaking research typically demands, noted James Evans, a computational social scientist at the University of Chicago, speaking to Science.
A stronger experimental design, Gombolay suggested, would involve an existing academic conference randomly assigning manuscripts to either A.I. or human reviewers and then measuring which group ultimately produced more impactful or innovative outcomes.
