Skip to content

docs/leaderboard/ — file guide

This is the only leaderboard/-named path tracked in git (see the repo .gitignore's carve-out comment) — everything under glossobench/leaderboard/ is local/generated scratch (freeze markers, per-run Elo boards from your own glossobench run invocations) and is intentionally not committed.

Two different kinds of file live here — do not confuse them:

File What it is Who writes it
<lang>/v<N>/leaderboard.json + index.html The CI-generated nightly board — a small, fixed, CPU-feasible model roster (ci_leaderboard recipe: Qwen2.5-0.5B/1.5B-Instruct, SmolLM2-1.7B-Instruct), judge-free, tiny row caps. Real cross-model Elo (genuine per-item pairwise matches via glossobench elo), but explicitly labeled DIRECTIONAL/demo, not a GlossoBench-7 claim — see the page footer. Regenerated by .forgejo/workflows/leaderboard.yml / .github/workflows/leaderboard.yml on relevant pushes, not on a timer. CI workflow, automatically
<lang>/v<N>/results_v1.jsonl Hand-compiled real results — the actual --intensity max ("fairest") + fast-intensity board for that language, one JSON object per model/run, each entry carrying source_commit/source_note for traceability to the exact build it was measured on. The Malay (ms) board is the one currently cited in the top-level README.md "Results" section; other languages get their own file here as results are compiled for them. Compiled by hand/script from real glossobench run output; not auto-written by any CLI command today.
<lang>/v<N>/supplementary_runs_v1.jsonl Real measurements that used different flags/methodology than the main results file (backend-parity checks, small-n judged-axis runs) — kept separate so they're never averaged into the main board by mistake. Same as above.

If you ran the benchmark yourself, your own run's full per-item summary lands at glossobench/output/<lang>/v<N>/<model>_ms.json (not under docs/ at all) — that's the file glossobench elo/compare/the CI renderer actually consume, and the file you'd attach to a leaderboard-submission PR.