Identifying Local LLMs and Strategies for Asking Clarifying Questions on Ambiguous Tasks
A discussion post on r/LocalLLaMA raised the question of which open-source LLMs, system prompts, or agent harnesses excel at recognizing ambiguous instructions and stopping to ask clarifying questions instead of proceeding with wrong assumptions. This highlights a major blind spot in standard LLM evaluation benchmarks, which measure coding speed and tool usage but fail to test how models handle ambiguity. Models that stop to clarify save user time and compute power by preventing long, autonomous chains of incorrect actions. The query asks whether stopping to clarify depends more on base model training, effective system prompting, or the architectural design of the agent harness controlling the execution loop. Current agentic workflows often suffer from over-confidence, executing multi-step tool calls based on vague prompts rather than seeking human feedback.
## BACKGROUND
An agent harness is the surrounding software infrastructure that manages an LLM's state, execution loops, tool integrations, and system context. While standard benchmarks measure capabilities like context window length, coding accuracy, and reasoning, real-world deployment frequently encounters incomplete user instructions where clarifying questions are necessary.