GlossoBench leaderboard — ms¶
Status: Live · Audience: All · bench_version: 1.0
Real --intensity max ("fairest") and fast-intensity runs on the ms language config. Judge-free (no paid judge required). Use the filter bar to narrow by board / language / intensity / run kind, or search by model name. Click any column header to sort — the MS column sorts descending by default; ranks re-number after sort + filter.
- Fairest board: 6 models, full 5-axis breakdown.
- Fast board: 10 runs, composite MS only (cheap intensity).
- Source:
docs/leaderboard/ms/v1/results_v1.jsonl(one JSON object per run, each carriessource_commit).
How to add a result: append a row to the JSONL, then re-run python website/scripts/render_leaderboard.py (or mkdocs build). See docs/leaderboard/FILEGUIDE.md for the file format.
Fairest board (--intensity max)
Full 5-axis (Knowledge / NLU / IF / NLG / Cultural). MS = register-composite mean score across the judge-free axes. Safety* proxy not shown here (separate axis; see README axis table).
| # | Model | MS | Knowledge | NLU | IF | NLG | Cultural | Judge | Wall (s) | Measured | Commit |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | SEA-LION v4.5 27B-IT v45 | 81.7 | 79.5 | 93.7 | 76.9 | 65.1 | 93.1 | judge-free | 1198.9 | 2026-07-25 | 3101af6 |
| 2 | Qwen3.5-27B (v4.5 base lineage) qwen35-27b | 80.3 | 80.9 | 94.0 | 73.1 | 64.0 | 89.7 | judge-free | 1171.3 | 2026-07-25 | 3101af6 |
| 3 | Qwen3.6-27B qwen36-27b | 77.5 | 74.9 | 91.2 | 73.1 | 65.5 | 82.8 | judge-free | 1290.6 | 2026-07-25 | 3101af6 |
| 4 | Gemma-4-31B-IT gemma | 76.5 | 83.1 | 93.6 | 73.1 | 39.8 | 93.1 | judge-free | 1796.7 | 2026-07-25 | 3101af6 |
| 5 | SEA-LION-v3-8B sealion8b | 69.7 | 58.7 | 80.1 | 61.5 | 62.1 | 86.2 | judge-free | 416.1 | 2026-07-25 | 3101af6 |
| 6 | SEA-LION-E2B-IT-2B e2b | 55.4 | 51.0 | 73.2 | 53.8 | 30.2 | 69.0 | judge-free | 1049.9 | 2026-07-25 | 3101af6 |
Fast board (--intensity fast)
Cheap-intensity composite MS + wall time. fast_seeds3 = 3-seed stability re-run of the same fast flags.
| Model | MS | Wall (s) | Run kind | Measured | Commit |
|---|---|---|---|---|---|
| SEA-LION v4.5 27B-IT v45 | 80.5 | 172.5 | fast | 2026-07-25 | 3101af6 |
| SEA-LION v4.5 27B-IT v45 | 80.5 | 157.5 | fast_seeds3 | 2026-07-25 | 3101af6 |
| Qwen3.5-27B (v4.5 base lineage) qwen35-27b | 78.5 | 148.8 | fast | 2026-07-25 | 3101af6 |
| Gemma-4-31B-IT gemma | 71.9 | 277.6 | fast_seeds3 | 2026-07-25 | 3101af6 |
| Gemma-4-31B-IT gemma | 71.9 | 299.2 | fast | 2026-07-25 | 3101af6 |
| Qwen3.6-27B qwen36-27b | 68.4 | 165.5 | fast | 2026-07-25 | 3101af6 |
| SEA-LION-v3-8B sealion8b | 67.8 | 99.9 | fast | 2026-07-25 | 3101af6 |
| SEA-LION-v3-8B sealion8b | 67.8 | 87.9 | fast_seeds3 | 2026-07-25 | 3101af6 |
| SEA-LION-E2B-IT-2B e2b | 53.8 | 205.0 | fast | 2026-07-25 | 3101af6 |
| SEA-LION-E2B-IT-2B e2b | 53.8 | 193.5 | fast_seeds3 | 2026-07-25 | 3101af6 |
Backend / methodology parity checks¶
These used different flags than the main board — kept separate so they are never averaged into a ranking. Not directly comparable to the boards above.
backend_parity_check — SEA-LION v4.5 27B-IT¶
- Flags: --thinking on (matched flags across backends)
- Backends: hf: MS 25.2 · vllm: MS 26.0
- Verdict: within CI -> hf and vllm backends equivalent after the v1.1.0 vLLM fixes (prompt_token_ids tokenization, skip_special_tokens, think-strip parity verified identical)
- Note: NOT comparable to the fairest board above -- --thinking on burns the response budget very differently than the fairest board's flags, so this is a much lower absolute MS by construction. A backend-equivalence check, not a ranking entry.
- Commit:
3101af6· Measured: 2026-07-25
Provenance: every row carries a source_commit and measured_at date. Scores are immutable per bench_version; old rows stay valid forever.
A separate CI-generated nightly demo board (small fixed model roster, directional Elo, explicitly NOT a GlossoBench-7 claim) is at leaderboard/ms/v1/index.html.
Related: README (GitHub) · METHODOLOGY.md · leaderboard file guide