NVIDIA NeMo Blueprint Optimizes AI Agent Harnesses for Reliability

NVIDIA has introduced a comprehensive blueprint for optimizing AI agent harnesses using its NeMo platform. The workflow, detailed in a November 3, 2026 tutorial, focuses on improving agent reliability by addressing failures in prompts, tools, routing, and other harness components without altering model weights.

Identifying Failures with NeMo Compass

The optimization process begins with NeMo Compass, which analyzes stored traces to detect recurring issues and prioritize improvements. In a fictional example, the Hermes enterprise assistant incorrectly called the "send_message" tool without user confirmation after being asked to draft a project-update message. NeMo Compass flagged this failure by comparing the agent's actions against predefined behavior rules in an ETHOS.md file.

Compass findings are structured into a backlog of issues, supported by trace IDs linking errors to specific runs. NVIDIA reports that Compass identified anomalies in thousands of traces during internal benchmarks, outperforming coding agents tasked with similar evaluations.

Creating Tests with NeMo Eval Author

After identifying issues, NeMo Eval Author helps create executable tests and verifiers to reproduce and evaluate failures. In the Hermes example, Eval Author generated four Harbor evaluations to test whether the agent improperly sent messages without user consent. These evaluations were based on Compass findings, raw traces, and the agent's ETHOS.md file.

Eval Author also supports creating custom evaluations before production data is available, allowing developers to expand test suites as real-world failures emerge.

Updating and Measuring Harness Improvements

To address the identified failure, NVIDIA updated the Hermes agent's harness instructions, specifying that requests to draft messages should authorize preparation only, without automatic execution. Following this change, evaluations showed significant improvements:

• Development tasks: Pass rate increased from 3/6 to 6/6, with tool calls reduced from 10 to 6.
• Held-out tasks: Pass rate increased from 4/6 to 6/6, with tool calls reduced from 28 to 15.

Held-out tasks, concealed from the coding agents proposing changes, were used to validate generalization beyond development scenarios.

Broader Implications

The NeMo blueprint provides a reproducible regression-testing loop that evaluates real failure modes and generalization before deployment. NVIDIA emphasizes that this approach ensures reliability by isolating harness changes from confounding factors such as model identity or execution conditions.

While the reported results are based on narrow examples, the methodology offers a structured framework for improving AI agent performance in enterprise applications.