Benchmark Compares Claude Code, OpenCode, and Pi Agent Harnesses
A benchmark comparison of AI coding agent harnesses (Claude Code, OpenCode, and Pi) using DeepSeek V4 Flash revealed that while they produce identical code diffs, their execution times and token usage vary drastically. Specifically, Claude Code took nearly four times longer to complete the same tasks compared to the fastest harness. This highlights that the scaffolding or harness surrounding a Large Language Model (LLM) plays a critical role in operational efficiency, affecting both cost and latency even when using the same underlying model. It shifts the focus of AI engineering from just model quality to the optimization of agentic architectures. The benchmark was run using DeepSeek V4 Flash on vLLM at approximately 180 tokens per second, isolating the scaffolding as the sole variable. The differences stem from how each harness structures system prompts, handles tool calls, and explores codebases, with Claude Code showing a tendency to over-explore.
## BACKGROUND
In AI engineering, a "harness" or "scaffolding" is the wrapper framework (comprising system prompts, tool schemas, and execution logic) that turns a raw LLM into an active agent capable of interacting with a codebase. vLLM is an open-source, high-throughput engine designed for efficient LLM inference and serving.