lm-eval-ledger: An Open-Source Tool for Visually Inspecting LLM Benchmark Results

Developer jayminbhan has released lm-eval-ledger, an open-source benchmark harness and web application for evaluating Large Language Models (LLMs). It automatically logs generation data to a local SQLite database and provides a Flask web interface to inspect and compare question-by-question model responses. Standard evaluation frameworks often provide headline metrics or raw JSONL/Parquet dumps, requiring developers to write custom scripts to analyze output errors. lm-eval-ledger simplifies model debugging by offering built-in pairwise comparison, prompt inspection, extracted answers, and performance statistics in a web interface. The tool uses YAML configuration files to run multi-model and multi-task benchmarks via a single CLI command. It supports vLLM, SGLang, Hugging Face transformers, and llama.cpp backends across Linux and Windows, capturing metrics such as token generation speed (tok/s), character count, and stop reasons.