AI Signal
Did the agent actually finish?
OpenAI’s Ironclad evaluation shows why a higher score still needs a closer look at the finished work.
· 1:04
Transcript
The original narration, preserved as published. Relative dates refer to the episode’s coverage window.
An agent can click every right-looking button and still leave the workflow wrong. OpenAI’s new Ironclad research makes that gap visible: the test is whether the finished process follows the business rules.
On October sixth, OpenAI described eleven research tasks for configuring contracts, approvals, and reusable legal terms. Each task had detailed criteria. The models practiced in hosted software environments, using synthetic training tasks.
GPT-six Astra reached a mean rubric score of fifty-five percent, versus forty-one point six for GPT-five point six Sol. Those are scores against task criteria, not percentages of fully completed customer jobs.
The reasoning settings differed. The reported time estimates were simulated, too, so you cannot read them as measured customer savings. Even with the improvement, there was still a lot left to get right.
If you are building an agent, define success in the actual application: approvals, exceptions, and resulting records. Check the finished state instead of accepting the agent’s summary. That is what a useful evaluation catches.