BrokenML: A New Benchmark for Evaluating AI Agents on ML Troubleshooting
A developer has introduced BrokenML, a novel benchmarking tool designed to evaluate how effectively AI agents can fix broken machine learning workloads across varying levels of cluster access. The benchmark tests agents on scenarios like PyTorch over Gloo or llama.cpp server setups, utilizing a separate verifier to re-run jobs on fresh clusters. This benchmark addresses a practical and critical problem in MLOps and agentic AI by testing whether autonomous agents can handle complex real-world infrastructure failures. It highlights the current capability gap between open and closed models, particularly regarding how well they perform debugging tasks using configuration files and code alone. The benchmark includes common faults such as shape mismatches, hanging collectives, straggler nodes, OOM errors, and silent loss divergence evaluated across three cluster access levels. Current limitations include using CPU over Gloo instead of GPUs and restricting testing to a single run per task per model.
## BACKGROUND
Slurm is a widely used open-source workload manager and job scheduler for Linux clusters, while Gloo serves as a collective communications library enabling distributed training processes to communicate. Meanwhile, llama.cpp is an open-source software library optimized for performing efficient local inference on large language models.