Same idea, different AI verdict

Transcript

The original narration, preserved as published. Relative dates refer to the episode’s coverage window.

Your idea did not change. The AI verdict did. Ai2's October twenty-seventh report tests whether language models can reliably judge a research idea's originality. The same ideas can receive very different verdicts.

Researchers compared six judges using human research ideas and ideas from a simple generator. They varied the instructions and other evaluation settings, rather than improving the ideas themselves.

In one GPT five point four comparison, removing a cue about reviewers' novelty judgments dropped reported accuracy from ninety-three point five percent to forty point nine. These were the same human-and-generated pairs, but the prompts supplied different information.

There is an important limit. The generated ideas were assumed less novel, not individually verified across the whole set. The generator had no literature search or feedback. This does not show that every AI idea is unoriginal.

Code and data are public. We have not independently reproduced the results. Before letting a judge rank your research, test whether reasonable instructions change its verdict. Watch whether an apparent breakthrough survives a change of evaluator.