Nvidia took the same AI model from 30% to 100% by changing what's around it. That's the real question before you buy an AI agent for your back office
Nvidia's own benchmark data shows the software wrapped around an AI model, not the model itself, decides whether an agent can reliably finish a long, multi-step task. That's the real diligence question for any GC vetting a submittal, RFI, or scheduling AI agent.
Nvidia published a result this week that has nothing to do with construction and everything to do with how you should shop for an AI agent: the same AI model went from a 30% score to a 100% score on a hard reasoning benchmark, and the model itself didn't change. What changed was the software wrapped around it, what the industry calls the "harness." If you're evaluating a vendor's AI agent for submittals, RFIs, or schedule risk, that's the diligence question worth asking before you ask which foundation model they use.
What did Nvidia actually show?
Nvidia built an agent architecture called AVO earlier this year to autonomously optimize CUDA GPU kernels — left to run unsupervised, it explored hundreds of code variations over multiple days and beat hand-tuned baselines like FlashAttention-4 by up to 10.5%. The architecture has two parts: an executor agent that proposes and tests changes, and a separate supervisor agent that audits the trajectory and pushes a new strategy when progress stalls, plus persistent memory that carries forward prior results instead of starting from scratch each round.
Nvidia then pointed that same, unmodified architecture at something completely different: ARC-AGI-3, a benchmark from the ARC Prize Foundation that drops an agent into unfamiliar game-like environments with no instructions and no stated goal. Running Claude Opus 5 inside the AVO harness, the system completed every level in every public environment — 100%. Claude Opus 5 without that harness, using only its own native tool use, was the best-scoring model tested at 30%.
Nvidia and outside reviewers flag that this isn't a clean side-by-side ablation — the two runs differed in more than just the harness. But the qualitative point lines up with Nvidia's separate published guidance on agent design: the scaffolding, not the model, is usually what decides whether an agent can reliably carry a task across many steps.
Why should a contractor care about a coding benchmark?
Because that's exactly the shape of the AI agents now being pitched into construction back offices. A submittal-processing agent, an RFI-drafting agent, or a schedule-risk agent isn't answering one question — it's supposed to carry state across an entire submittal log, an entire spec book, or an entire project schedule, over weeks. That's a long-horizon task in the same sense Nvidia's benchmark is, and Nvidia's data says the model name on the vendor's slide deck is a weaker predictor of reliability than what's built around it.
What should you actually ask a vendor?
Most sales conversations lead with "we're built on GPT-5" or "we run on Claude." That's the wrong first question. Ask these instead:
| Question | What it tells you |
|---|---|
| Does a separate process review or check the agent's output before it reaches a person? | Whether there's a supervisor step, or just one model producing an answer unchecked |
| Does it retain memory across the full submittal log or RFI history, or does each query start blank? | Whether it can actually track state over a multi-week job, not just answer one-off prompts |
| What happens when it's wrong or gets stuck? | Whether there's a defined recovery or escalation path, or a bad step just silently carries forward |
| Has it been tested on a full package or cycle, not a single demo question? | Whether the vendor has evidence for long-horizon reliability at all |
| Which foundation model does it run on? | Useful context — but Nvidia's own numbers say this predicts less than the answers above |
Where does this not apply?
Nvidia's benchmark has a clean win condition — the agent either clears the level or it doesn't. A submittal review or an RFI response doesn't work that way; it's a judgment call against a spec section, and getting it wrong on fire rating, ADA clearances, or lead-free requirements has real consequences. A strong harness is a signal that a vendor has thought about reliability, not a substitute for a person checking compliance-critical output before it goes out under your stamp.
The practical move: next time a vendor demo leads with the model name, ask them to walk through what happens on turn 40 of a task, not turn one. That's where the harness — or the lack of one — shows up.
For more on what's underneath the current wave of construction AI agents, see our look at the same architecture risk showing up in Copilot.
Forward this to the person on your team who's still arguing AI is overhyped.
Construction AI Brief publishes three times a week. Subscribe at constructionaibrief.com.
- What is an AI agent 'harness'?
- The harness is the software wrapped around a language model that turns it into an agent: how it stores memory across steps, whether a separate process reviews its work, how it recovers when a step fails, and how it uses tools. Two vendors can plug in the same underlying model and get very different reliability depending on the harness around it.
- Did Nvidia's test prove the harness alone caused the jump from 30% to 100%?
- Not as a clean controlled experiment. Nvidia and outside reviewers both note the harnessed and unharnessed runs differed in agent backend, memory handling, and evaluation setup, so the exact size of the gap isn't isolated to one variable. The direction of the finding, that scaffolding drives large swings in long-task reliability, is consistent with Nvidia's separate published guidance on harness design.
- What is ARC-AGI-3?
- ARC-AGI-3 is a benchmark launched in March 2026 by the ARC Prize Foundation that tests AI agents in novel game-like environments with no language, numbers, or familiar symbols to lean on, forcing them to explore, build a model of unfamiliar rules, and plan across many steps. At launch, every frontier model scored below 1% without added scaffolding.
- What should I ask an AI agent vendor before buying a submittal or RFI tool?
- Ask what happens after step one: does a separate process review or check the agent's output, does it retain project memory across the full submittal log or RFI history, and what happens when it's wrong or stuck. Which foundation model it runs on is a smaller factor than how it's built around that model.
- Does a good harness mean an AI agent doesn't need human review?
- No. Nvidia's own result was on a game benchmark with a clear win condition, not a spec-compliance judgment call. For submittals or RFIs touching fire rating, ADA, or lead-free requirements, a reviewer step inside the agent is a diligence signal, not a substitute for a human sign-off.