docs/leaderboard/ — file guide¶
This is the only leaderboard/-named path tracked in git (see the repo
.gitignore's carve-out comment) — everything under glossobench/leaderboard/
is local/generated scratch (freeze markers, per-run Elo boards from your own
glossobench run invocations) and is intentionally not committed.
Two different kinds of file live here — do not confuse them:
| File | What it is | Who writes it |
|---|---|---|
<lang>/v<N>/leaderboard.json + index.html |
The CI-generated nightly board — a small, fixed, CPU-feasible model roster (ci_leaderboard recipe: Qwen2.5-0.5B/1.5B-Instruct, SmolLM2-1.7B-Instruct), judge-free, tiny row caps. Real cross-model Elo (genuine per-item pairwise matches via glossobench elo), but explicitly labeled DIRECTIONAL/demo, not a GlossoBench-7 claim — see the page footer. Regenerated by .forgejo/workflows/leaderboard.yml / .github/workflows/leaderboard.yml on relevant pushes, not on a timer. |
CI workflow, automatically |
<lang>/v<N>/results_v1.jsonl |
Hand-compiled real results — the actual --intensity max ("fairest") + fast-intensity board for that language, one JSON object per model/run, each entry carrying source_commit/source_note for traceability to the exact build it was measured on. The Malay (ms) board is the one currently cited in the top-level README.md "Results" section; other languages get their own file here as results are compiled for them. |
Compiled by hand/script from real glossobench run output; not auto-written by any CLI command today. |
<lang>/v<N>/supplementary_runs_v1.jsonl |
Real measurements that used different flags/methodology than the main results file (backend-parity checks, small-n judged-axis runs) — kept separate so they're never averaged into the main board by mistake. | Same as above. |
If you ran the benchmark yourself, your own run's full per-item summary lands
at glossobench/output/<lang>/v<N>/<model>_ms.json (not under docs/ at
all) — that's the file glossobench elo/compare/the CI renderer actually
consume, and the file you'd attach to a leaderboard-submission PR.