LLMs Fail to Detect Lookahead Bias Bug in Quantitative Trading Feature

In an experiment using the AQuA research framework, a code-generating LLM and a reviewer LLM both failed to detect a lookahead bias bug in a quantitative trading feature. The bug, which normalized intraday volume using the day's final total volume, was only caught during a manual audit after the feature's performance vanished on a clean data re-split. This case study highlights the limitations of using LLMs for code generation and review in highly complex domains like quantitative finance. It demonstrates that multi-agent LLM systems can align on plausible but incorrect explanations, prompting a shift toward structural constraints rather than relying solely on smarter reviewers. To address this vulnerability, the researchers updated the framework to AQuA v2, which seals data splits outside the agent's reach and replaces arbitrary code generation with a restricted registry of pre-defined, causal operators. While this prevents lookahead bias, it also limits the agent's search space and creative potential.