Simon Willison Introduces `smevals`, a Lightweight LLM Evaluation Suite
Simon Willison, in collaboration with Prime Radiant, has released `smevals`, a lightweight, open-source evaluation suite designed to test LLM models, prompts, and harnesses. The tool allows users to define evaluations using YAML files, run them across multiple models, grade the results separately, and generate interactive HTML reports. As LLM development increasingly relies on automated agents and prompt engineering, there is a growing need for simple, agent-friendly evaluation tools. `smevals` addresses this by offering a lightweight alternative to complex benchmarking frameworks, making it easier for developers to quickly test and iterate on model behaviors. The tool is designed to be easily executed via `uvx` without manual installation, and it separates the execution of runs from the grading process. It also includes a built-in local server and a build command to compile evaluation results into static HTML dashboards.
## BACKGROUND
LLM evaluation (or "evals") is the process of testing language models to measure their accuracy, safety, and alignment on specific tasks. Traditionally, this requires heavy frameworks like EleutherAI's `lm-evaluation-harness`. Additionally, `uvx` is a tool runner from the fast Python package manager `uv` that executes CLI tools in temporary, isolated virtual environments.