The Need for Multi-Step Debugging and Agentic Coding Benchmarks for LLMs
A developer community discussion highlights growing dissatisfaction with traditional "one-shot" LLM coding benchmarks, calling for tests that evaluate multi-step debugging, code modification, and output analysis. Users are seeking benchmarks that require models to fix broken-by-design code and interactively analyze program outputs, including visual data. While one-shot generation benchmarks like HumanEval are common, they do not reflect real-world software engineering, which heavily involves debugging and iterative refinement. Shifting focus to agentic, multi-step benchmarks like SWE-bench or A-CODE-LLM Bench helps evaluate how LLMs perform as practical coding assistants in complex codebases. Existing agentic benchmarks like SWE-bench evaluate LLMs on resolving actual GitHub issues, while newer frameworks like A-CODE-LLM Bench test models across multiple automated validation steps using CLI tools. However, running these complex, multi-step agentic benchmarks locally remains a challenge for individual developers due to high resource requirements.
## BACKGROUND
Traditional LLM coding benchmarks often measure "one-shot" generation, where a model generates a complete block of code from a single prompt. In contrast, "agentic" coding benchmarks evaluate LLMs acting as autonomous agents that can run terminal commands, read error logs, edit multiple files, and iteratively test their solutions until they work.