Independent Run Replicates DeepSeek V4 Flash's 82.7% Score on Terminal-Bench 2.1
An independent evaluation using the Ante harness (version 0.preview.71) has successfully replicated DeepSeek V4 Flash's reported 82.7% accuracy on the Terminal-Bench 2.1 benchmark. This validation was conducted using the deepseek-v4-flash-0731 model accessed via OpenRouter. Independent replication of LLM benchmark scores is crucial for transparency, especially since DeepSeek's original claim relied on an unreleased internal evaluation harness. This successful run builds trust in the model's capabilities and demonstrates the reliability of public evaluation tools like Ante and Harbor. The evaluation consisted of 445 trials across 89 Terminal-Bench tasks, resulting in 368 successful trials with a standard error of ±1.79%. The test was configured with maximum reasoning effort and no skills enabled, showing that the model is sensitive to the specific harness setup.
## BACKGROUND
Terminal-Bench is a benchmark designed to measure how effectively AI agents can navigate and solve tasks within command-line interfaces. Harbor is an open-source framework used for evaluating and optimizing these agents. Evaluation harnesses like Ante provide standardized environments to run these tests, ensuring that benchmark results can be verified and compared fairly across different models.