Standardizing Reproducibility for LLM Refusal Benchmarks in Model Cards
A community discussion highlights the lack of reproducibility in LLM refusal tables, specifically citing OrcaRouter's Qwen3.8-27B FP8 model card where refusal rates vary drastically between the base and abliterated versions without shared replication artifacts. The post asks what minimum artifacts—such as raw generations, decoding settings, or scoring code—are required to make these benchmarks independently verifiable. As open-source LLMs are increasingly modified via techniques like abliteration to bypass safety guardrails, standardized replication packets are crucial for verifying safety and performance claims. Without transparent benchmarks, developers cannot reliably compare models or trust uploader-reported refusal rates. The OrcaRouter Qwen3.8-27B FP8 card reports a 0–6% refusal rate for the abliterated checkpoint with thinking disabled, compared to 63.6–99% for the base model. However, the evaluation relies on an uploader-run, opening-phrase classifier without providing raw generations, decoding settings, or per-prompt labels.
## BACKGROUND
Abliteration is a technique used to uncensor LLMs by identifying and neutralizing the specific activation directions associated with refusal behavior, effectively bypassing safety alignment without full retraining. A model card is a short document accompanying a machine learning model that details its performance, limitations, and evaluation metrics to promote transparency.