Specific Labs Releases Real-SWE Benchmark for Evaluating AI on Private Codebases
Specific Labs has introduced Real-SWE, a new benchmark designed to evaluate frontier AI models on real-world software engineering tasks using private, production codebases licensed from real companies. The initial release evaluates models across ten tasks and eight harness configurations, covering 640 scored rollouts. Existing coding benchmarks primarily rely on public open-source repositories, which risk data contamination and may not accurately reflect enterprise software development. Real-SWE provides a more realistic standard for assessing how effectively AI agents resolve software issues within complex, proprietary corporate environments. By leveraging licensed production codebases rather than public GitHub issues, Real-SWE mitigates the risk of models memorizing solutions during training. The benchmark measures model performance across multiple rollouts to evaluate both capability and solution reliability.
## BACKGROUND
LLM capabilities in code generation have traditionally been tested using benchmarks like SWE-bench, which tasks models with resolving real GitHub issues. However, because public repositories are often included in LLM training datasets, researchers need private benchmarks to accurately test model generalization.