A collection of small domain-specific benchmarks for local LLMs

A developer has launched a website hosting a growing collection of over 30 small, domain-specific benchmarks designed to evaluate and compare the performance of local language models. The platform allows users to create benchmarks, set up model pipelines, and compare evaluation results. Standard academic benchmarks often fail to capture how LLMs perform on niche, real-world tasks. This project provides the local AI community with practical, domain-specific evaluations created by experts in fields like food safety, laser physics, and wireless networking. The platform integrates with llama-server using configuration files containing Hugging Face model IDs to fetch responses. It currently supports simple chat pipelines, with plans to expand into multi-turn conversations, RAG, and structured data extraction.