Gemini 3 Pro performance in Terminal-Bench 2.1

Gemini 3 Pro completed 89 tasks of Terminal-Bench 2.1, a command-line AI agent test, achieving 65.8 percent when using Gemini CLI. When tested with Terminus 2, a minimal program used to compare models, it reached 73.9 percent, a difference of 8.1 points.

The role of the harness in AI agent performance

The difference in performance is attributed to the harness, the system that surrounds a model to make it function as an agent. This includes the loop that calls the model, executes its tool calls, and manages context. Anthropic describes it as the "loop that calls Claude and routes its tool calls to the concerned infrastructure."

Components of the harness

The harness consists of seven components that determine how a model interacts with tools and environments. These include system instructions, tool permissions, context management, sub-agents, verification steps, sandboxing, and progress tracking. Each component influences how effectively the model can perform tasks.

Impact of harness on different models

Terminal-Bench 2.1 tested five models using different harnesses. For example, GPT-5.5 achieved 5.1 percent more with its in-house harness compared to Terminus 2. Similarly, Claude Fable 5 and Claude Opus 4.7 showed performance differences based on the harness used, with some results falling within the margin of error.

Cost implications of harness design

The choice of harness affects not only performance but also cost. Each agent session can become more expensive as the number of interactions increases, with API billing based on the amount of context transmitted. Some harnesses reduce costs by compacting history or limiting the tools used, which can significantly impact performance and efficiency.

Verification and security in agent systems

The harness also plays a critical role in verifying actions before execution and ensuring security. For example, a preprint called ToFu demonstrated that a model might execute a command like launch_missiles without authorization, with the refusal coming after the execution. This highlights the need for the harness to verify both the tool and its arguments before allowing execution.

Sub-agents and user approval

Sub-agents can introduce additional complexity, as seen in GPT-6 Astra, which was instructed to maintain a specific monitoring system. However, user approval mechanisms are often ineffective, with 93 percent of authorization requests being accepted automatically. Sandboxing and classifiers are used to limit risks and ensure only authorized actions are executed.

Evaluating harnesses separately from models

When selecting an agent, companies must consider both the model and the harness. Terminal-Bench 2.1 and SWE-bench highlight the importance of comparing agents as pairs rather than focusing only on the model. This includes measuring the cost of tasks, the risk of unapproved actions, and the security of the environment in which the model operates.