What to know
- Ironclad helped define 11 legal, commercial and procurement tasks.
- Astra’s mean rubric score was 55.0% in OpenAI’s evaluation.
- The reported times are research estimates rather than measured customer savings.
A domain-specific research evaluation
OpenAI announced an Ironclad research collaboration on October 6 to improve agents that operate specialized business software. Ironclad helped define 11 tasks covering legal, commercial and procurement workflows, including agreement setup and approval processes. Each task was assessed against between eight and 50 requirements.
In OpenAI’s evaluation, GPT-6 Astra scored 55.0% on average, compared with 41.6% for GPT-5.6 Sol. Estimated time per attempt was 19.2 minutes for Astra and 37.0 minutes for Sol. The models used different reasoning settings selected for their highest scores. OpenAI says simulated training tasks used filtered public SEC contracts, rather than nonpublic Ironclad customer contracts. These are company-reported research results, not an independent study of deployed legal operations.
Analysis: Completing the workflow is the relevant unit
Contracting software turns business requirements into interacting forms, templates and approval paths. A system can complete each visible screen while leaving the overall process incorrect. For example, it may collect a purchase amount but fail to route requests above the approval threshold to the finance team. The interface can look finished even though an important rule never takes effect.
Evaluating that work requires testing the resulting configuration with representative cases. Requests immediately above and below a threshold are useful because they reveal whether a boundary was implemented correctly. Exceptions need their own cases. An agreement that selects a different jurisdiction may require a different clause, so the test should inspect the generated document as well as the choices shown in the form.
Rubric scores help locate partial progress. They should not be interpreted as the share of contracts that a customer can safely delegate. A missed requirement may be more consequential than several correctly completed formatting tasks. Readers need to understand the distribution of errors and the importance of each criterion before translating an average into an operating decision.
Research progress leaves a deployment question
The estimated attempt times likewise require care. A faster attempt can still leave more work for a reviewer if it contains a difficult-to-detect configuration error. A customer trial would need to measure the time required to inspect the outcome, correct mistakes and rerun the relevant cases. That complete cost is more useful than the agent’s own runtime alone.
The collaboration offers a concrete way to evaluate agents inside a real software domain. Its next practical test is whether the resulting improvements hold across customers with different approval rules, templates and exceptions. Research tasks cannot capture every combination, and a business should retain a clear owner for the process an agent configures.
The published results establish progress on a defined evaluation. They also show why successful actions are an incomplete measure of competence. An agent handling professional work needs to leave behind a process that satisfies the original requirements, with enough evidence for another person to verify the result.
Sources & further reading
Factual statements are grounded in the linked material. Interpretation and illustrative examples are Byte Watchr analysis. Vendor claims are identified as claims, rather than independent testing.
The event date records the source announcement or documented operation. The coverage edition groups recent developments and is separate from the publication date. Actual publication is recorded above.
Corrections policy · About this byline
