GlossoBench — how it relates to existing language benchmarks

Status: Active · Audience: Adopters, decision-makers · Updated: 2026-07-28

GlossoBench is a single, public, versioned, plugin-extensible harness that hosts the language axes practitioners already trust (Knowledge/NLU/IF/NLG/MT/Safety/ Cultural) on public sources, with judge-agnostic scoring and a per-version Elo leaderboard. It aims to become a standard harness for evaluating any language — natural or AI-developed — not a benchmark scoped to one region, language family, or resource tier. The languages it ships today (Malay, Thai, Vietnamese, Indonesian, Burmese, Filipino, Tamil, Swahili) are the first implementations of a general architecture, the same way TechEmpower's shipped frameworks are examples of a general harness, not the harness's scope.

This document positions GlossoBench relative to the benchmarks practitioners already use. It is a complement map, not a leaderboard of winners: each tool below is a genuine contribution in its own lane, and the goal is to make clear which lane GlossoBench occupies and how it composes — rather than replaces — the others.

Purpose

Position GlossoBench truthfully relative to the benchmarks practitioners already use, so adopters can see which lane GlossoBench occupies and how it composes with — rather than replaces — existing tools.

Goals

• Map each property a practitioner cares about to which existing benchmark satisfies it and whether GlossoBench does.
• Distinguish harness-tier peers from dataset-tier inputs, marking dataset-only properties n/a rather than failed.
• Credit peers' genuine strengths and disclose GlossoBench's own honest caveats.
• Ground every license and coverage claim in the per-component audit, not marketing.

Success Criteria

• A reader can name GlossoBench's lane and its closest peer (SEA-HELM) after one pass.
• Every ✅/⚠️/❌/n/a cell in the 23-property matrix and F1–F12 tables is factual and links to LICENSE_AUDIT.md for license claims.
• The "honest caveats" section names the real weaknesses (self-authored data, young v1, judge axes) without softening.
• No comparative claim is false or disparaging; the legal basis section ties claims to Lanham Act §43(a) truth-in-advertising norms.

What GlossoBench is — and isn't

A critical reading first, because the matrix below is only useful once the category is clear.

GlossoBench is a harness, not a dataset. It is a versioned, plugin-extensible orchestration layer that runs a model over the language axes practitioners already trust and scores them under one reproducible, judge-agnostic, any-language frame. The datasets it scores — MalayMMLU, FLORES, Belebele, Global-MMLU, SEA-Guard judges — are public work by their respective authors; GlossoBench hosts them, credits them, and leaves their licenses untouched. What GlossoBench adds is the harness-layer infrastructure no single-dataset benchmark carries: versioning + cross-version comparability, partial-completion, register stratification, an open Elo leaderboard, containerization, a commercial-use license policy, plugin extensibility, inference throughput/VRAM reporting, a community language-pack endorsement layer, and a parallel 5-axis suite for AI-developed languages.

GlossoBench is not a competitor to those datasets. MalayMMLU is a strong Malay Knowledge MCQ set; FLORES-200 is the standard NLG translation benchmark; Belebele is a solid NLU reading set; MMLU / Global-MMLU is the canonical knowledge MCQ. They are inputs, not rivals. Scoring a dataset on "multi-axis in one harness" or "containerized one-command run" is a category error — a dataset is not a harness — so the matrix below marks those cells n/a for dataset-tier tools, not ❌. They are not failing a bar they were never aimed at.

The fair peer set is harness/suite-tier: SEA-HELM, HELM, LMSYS Arena (and lm-eval / lighteval, not tabulated). GlossoBench belongs to that layer. SEA-HELM (v1.2.1) is the closest peer and the direct prior art: it established the 7-axis Southeast-Asian evaluation bar. GlossoBench generalizes that bar — any language, no access gate, cross-version-comparable, community-extensible — while hosting public equivalents of the same axes. We build on SEA-HELM's work; we do not supersede it. HELM and LMSYS each carry a different subset of harness properties (HELM: versioning + breadth + efficiency/cost reporting; LMSYS: open human-preference Elo); GlossoBench's claim is carrying the union of these properties in one open, any-language, no-gate frame — not outperforming any peer in its own lane.

What GlossoBench does not do: it does not improve, relicense, or replace the datasets it hosts; it does not claim empirical coverage for languages beyond those shipped (the architecture is any-language, the validation is per-language); and a self-authored benchmark winning on its own axes is not a neutral claim — the credibility bars in FAIREST.md exist precisely because of that.

Positioning

Question	Answer
Is GlossoBench a Southeast-Asian benchmark?	No. SEA languages are the most-validated implementations today because that's where the first contributors worked, not a scope boundary. The architecture is config-driven and language-parametric — see ADDLANGUAGE.md.
Is it a "low-resource language" benchmark?	No — that framing implies a tier. GlossoBench targets any language a team wants measured, including high-resource ones; low-resource support (no gate, \$0, GGUF/CPU path) is a design requirement, not the target audience.
Does it only do natural human languages?	No. A parallel 5-axis suite (compression/recoverability/learnability/task-effectiveness/expressivity) evaluates AI-developed, constructed, and emergent languages — see AI_DEVELOPED_LANGUAGES.md. This has nothing to do with region or resource level.
Why does the comparison table below name SEA-HELM, MMLU, FLORES, Belebele?	Those are the closest existing points of comparison for the properties in the matrix (multi-axis, public data, versioning, etc.), not a claim that GlossoBench competes only in their category. A team evaluating French, Japanese, or a constructed language faces the same MMLU/single-axis/gated-data problems this matrix documents.
Does GlossoBench replace these benchmarks?	No — it hosts their public data under one reproducible, versioned, judge-agnostic harness (see "How GlossoBench composes each" below). A team that only needs Malay Knowledge is well-served by MalayMMLU directly; GlossoBench earns its keep when a team needs several axes comparable and versioned, or a language no existing suite covers.
What determines which languages ship today vs. later?	Contributor availability and native-speaker validation (Standing Order: no fabricated native strings) — see ENDORSEMENT.md and glossobench-contrib for how a new language gets added and endorsed.

This document is a comparison matrix: for each property a practitioner cares about, which existing benchmark satisfies it, and whether GlossoBench does. The goal is not to diminish any single benchmark — each was a genuine contribution — but to show that GlossoBench is, to our knowledge, the only open harness that satisfies all of them simultaneously — the minimum bar for a language-benchmark standard — while hosting, not replacing, the datasets below.

Scope of the claims. A ✅ below means satisfied by design in the shipped code. It does not mean empirically validated for every language: the full 7-axis suite is validated on Malay (with Swahili self-built axes); the other shipped languages (th/vi/id/my/fil/ta) run the public axes only, and "any language" is a property of the config/plugin architecture, not yet of measured coverage. Where a property is structural-but-not-yet-proven, read ✅ as "the mechanism ships and is smoke-tested", and see CRITIC_AUDIT.md for the full gap list.

LLM-as-judge vs pure metric computation. The single most decision-relevant split the matrix compresses into row 4: five of the seven axes (Knowledge, NLU, IF, NLG, Cultural) are pure metric computation — exact-match over cloze log-likelihood, rule-based constraint checks, chrF — deterministic, bit-for-bit reproducible, zero judge cost, zero judge bias. MT requires an LLM judge by the nature of the task (no rule scores translation adequacy). Safety now offers both a judge-free variant (safety_tox, a toxicity/hate-speech classifier — row 21) and a judge-required free-text path: a team with no judge gets a real Safety number via safety_tox; a team with a judge gets the deeper free-text Safety reading. Where a judge is needed, GlossoBench reduces — but cannot eliminate — judge dependence with a pluggable multi-judge ensemble, std-dev agreement, and self-judge family-exclusion. When comparing against a metric-only benchmark (MalayMMLU, Belebele, FLORES), compare GlossoBench's judge-free axes; the judge axes are a different trust class and are flagged as such in every output (needs_judge, skipped_no_judge).

The matrix

The columns are grouped by tier:

• Harness/suite-tier (run models and score them): GlossoBench, LMSYS Arena, HELM, SEA-HELM. These are the fair peers on harness-layer properties.
• Dataset-tier (public datasets GlossoBench hosts): MalayMMLU, MMLU / Global-MMLU, FLORES-200, Belebele. Harness-layer properties (multi-axis, extensible, containerized, Elo, partial-completion, add-language, throughput, endorsement) are n/a for a dataset by design — a dataset is not a harness — so those cells are n/a, not ❌.

Legend: ✅ = satisfies by design · ⚠️ = partial / caveated · ❌ = no · n/a = out of scope for this tool's tier

Property	GlossoBench	MalayMMLU	MMLU / Global-MMLU	FLORES-200	Belebele	LMSYS Arena	HELM	SEA-HELM (v1.2.1)
1. 100% public, no access gate	✅	✅ BSD HF	✅	✅	✅	✅	✅	❌ datasets access-gated (a quality-control gate, not a public-default model)
2. Multi-axis in one harness (Know+NLU+IF+NLG+MT+Safety+Cultural)	✅ 7 axes	n/a single-axis dataset	n/a	n/a	n/a	⚠️ human-preference only	✅ many	✅ 7 axes
3. Any language, same code path	✅ structural: one config schema, any language (7-axis validated on ms; th/vi/id/my/fil/ta public-axis today)	n/a Malay-scoped	⚠️ many but Know-only	✅ 200 langs but NLG-only	⚠️ ~120 langs but NLU-only	✅ but judge-only	⚠️ English-heavy config	❌ SEA-scoped (by design — SEA is its focus)
4. Judge-agnostic (no single judge lock-in)	✅ 5/7 axes pure-metric (NO judge); Safety has a judge-free safety_tox variant + a judge-required free-text path; MT judge-required but pluggable std-dev ensemble	✅ exact-match	✅	✅ chrF/MetricX	✅ exact-match	❌ must use their judge	⚠️ many metrics, many judges	❌ tied to their judge pipeline
5. Runs with ZERO paid API ($0)	✅ $0 judges (NIM free-tier default / offline ollama fallback) + CPU/GGUF inference	✅	✅	✅	✅	❌	⚠️	❌ judge API + gated access
6. Runs on a phone / edge (GGUF, CPU)	✅ any-device, GGUF first-class	✅	✅	✅	✅	❌ cloud-only	❌ heavy	❌ needs vLLM + H100 class
7. Completes ≤ 3h on one 80GB GPU (or CPU)	✅ hard budget cap enforced	✅	✅	✅	✅	⚠️	❌ very heavy	❌ multi-day, multi-GPU
8. Reproducible, pinned, versioned	✅ bench_version + per-plugin versions, immutable leaderboards	⚠️ frozen but unversioned scorecards	⚠️	⚠️	⚠️	❌ Elo shifts as votes accumulate	✅	⚠️ leaderboard changed axes + Safety metric between v1.1.2 and v1.2.1
9. Cross-version comparability (old models keep their score)	✅ glossobench compare v1 v2	n/a single snapshot	n/a	n/a	n/a	❌	⚠️	❌ v1.1.2 numbers not comparable to v1.2.1
10. Held-out + contamination audit	✅ contamination-dedup before freeze, never in our corpus	❌ may overlap train	❌ widely reported train-contamination concern	✅ devtest held out	⚠️	❌	⚠️	❌ unknown
11. Extensible without core edits (add dataset = 1 file)	✅ plugin registry auto-discovery	n/a	n/a	n/a	n/a	❌	⚠️	❌
12. Bootstrap CI on every axis + MS	✅	❌	❌	❌	❌	❌	✅	❌
13. Elo / annual top-10% leaderboard	✅ open Elo, std-dev agreement	n/a	n/a	n/a	n/a	✅ (methodology + periodic data releases public)	❌	❌ ranked table only
14. Usable by very-poor-country / aboriginal / African teams	✅ minimal deps, $0, any device, plugin their own data	⚠️ Know-only	⚠️ Know-only	⚠️ NLG-only	⚠️ NLU-only	❌ infra	❌ infra	❌ gate + infra
15. Self-contained LLM-buildable (an LLM can add a language without asking)	✅ glossobench add-language + 1 plugin file	n/a	n/a	n/a	n/a	❌	❌	❌ access request needed
16. Register-stratified (formal / conversational / SMS sub-scores per axis)	✅ high/mid/low per-axis + per-register MS, CPT-mix weights	❌ flat	❌ flat	❌ flat	❌ flat	❌ flat	❌ flat	❌ flat per-axis
17. Partial-completion tolerant (run only some axes; MS not corrupted by missing ones)	✅ missing axis/register EXCLUDED from denom, never zeroed; skipped_no_judge+null_rate reported	n/a	n/a	n/a	n/a	⚠️	⚠️	❌ all-or-nothing
18. Containerized, one-command global run (podman/docker, no env wrangling)	✅ Containerfile, CPU-slim + CUDA flavors, GGUF first-class	n/a	n/a	n/a	n/a	❌	⚠️ partial	❌
19. License-clean for commercial minority-language adoption (framework open; all default data commercial-OK; NC/gated opt-in only; no weights bundled)	✅ Apache-2.0 framework; defaults = Apache/MIT/BSD/CC0/CC-BY(-SA); NC/gated opt-in; weights never bundled (see LICENSE_AUDIT.md)	⚠️ BSD-OK but single-axis	⚠️ mixed (MMLU has NC variants; Global-MMLU Apache-2.0)	⚠️ FLORES-200 NC (Meta) / ✅ SA (OLDI)	✅ CC-BY-SA	❌ judge TOS	⚠️ mixed	⚠️ gated access + their judge license
20. Benchmarks AI-developed / constructed / emergent languages (efficiency + effectiveness; judge-free where meaning is structured)	✅ 5 axes (compression/recoverability/learnability/task-effectiveness/expressivity); judge-free; partial-tolerant (see AI_DEVELOPED_LANGUAGES.md)	n/a natural-lang dataset	n/a	n/a	n/a	❌	❌	❌
21. Judge-free Safety variant (a real Safety number with no judge)	✅ safety_tox axis — toxicity/hate-speech classifier, toxicity_official_normalized, no judge — alongside the judge-required free-text Safety path	n/a	n/a	n/a	n/a	n/a (no safety axis)	⚠️ some toxicity metrics, judge/pipeline-tied	⚠️ full Safety axis, via their judge (no judge-free variant)
22. Inference throughput + VRAM reported (deployment-cost transparency)	✅ decode_tok_s + prefill_tok_s + vram_peak_gb + cpu_ram_peak_gb per run, engine-provenanced (1.3.0)	n/a	n/a	n/a	n/a	❌ cloud-side, not surfaced to the user	⚠️ cost/latency partial	❌ not reported
23. Community language-pack endorsement (TechEmpower-style: one harness + a published bar + a registry of accepted packs)	✅ Core/Community tiers + glossobench-contrib + de-facto registry (ENDORSEMENT.md)	n/a	n/a	n/a	n/a	❌ single-org maintained	❌ single-org maintained	❌ single-org maintained

Score (count of ✅; n/a = out of scope for this tool's tier, not a failure)

Benchmark	Tier	✅	⚠️	❌	n/a
GlossoBench	harness	23	0	0	0
MalayMMLU	dataset	5	3	3	12
MMLU/Global-MMLU	dataset	5	4	3	11
FLORES-200	dataset	7	3	2	11
Belebele	dataset	6	4	2	11
LMSYS Arena	harness	3	3	16	1
HELM	harness	5	11	7	0
SEA-HELM	harness	1	3	19	0

The 23 properties are harness-layer. GlossoBench, as the harness, carries all 23 by construction. The dataset-tier benchmarks (MalayMMLU, MMLU/Global-MMLU, FLORES-200, Belebele) carry the subset their single purpose entails — they are inputs the harness hosts, not systems measured on these properties, which is why their harness-layer cells are n/a. The harness-tier peers each carry a different subset: SEA-HELM carries the multi-axis Southeast-Asian suite (the prior art GlossoBench generalizes); HELM carries versioning, breadth, and partial efficiency/cost reporting; LMSYS carries the open human-preference Elo. GlossoBench's contribution is carrying the union of these properties in one open, any-language, no-gate frame — not outperforming any peer in its own lane. A team that only needs Malay Knowledge is well-served by MalayMMLU directly; GlossoBench earns its keep when a team needs several axes comparable, versioned, and reproducible — or a language no existing suite covers.

The fairness differentiators (2026-07-23 sweep, beyond the 23)

The 23-property matrix above is the structural bar (public / multi-axis / any-lang / judge-agnostic / $0 / versioned / …). Beyond it, two fairness layers ship built-in. F1–F11 are the 11 fairness levers per METHODOLOGY.md §10 (the canonical core; the FAIREST.md 2026-07-23 sweep expands them to a 17-lever toolkit — the first 11 correspond to these, numbering differs). F12 is a separate calibration finding from the MalayMMLU H200 fast-vs-full campaign. GlossoBench ships all 11 levers + the F12 calibration anchor; the field ships few. These are what separate a self-authored score from an independently-validated one. (All additive, schema-1.0/1.1-compatible.)

Fairness property	GlossoBench	MalayMMLU	MMLU/Global-MMLU	FLORES	Belebele	LMSYS	HELM	SEA-HELM
F1. Adversarial probes (shuffle/unanswerable — catch memorization)	✅ probes → by_probe	❌	❌	❌	❌	❌	❌	❌
F2. Model-class segregation (base/instruct/reasoning/SFT ranked separately)	✅ MB_MODELS_META sidecar	❌	❌	❌	❌	⚠️	⚠️	❌
F3. Self-judge family-exclusion (no Qwen-judge on a Qwen model)	✅ JudgePlugin.family	n/a	n/a	n/a	n/a	❌	⚠️	❌
F4. Cross-bench convergence (rank-correlate vs an external bench)	✅ correlate (Spearman+Kendall)	❌	❌	❌	❌	❌	❌	❌
F5. Contamination hard-EXCLUDE + public SHA (drop before score)	✅ MB_DEDUP_CORPUS + SHA-256	❌	❌	n/a	❌	❌	⚠️ flag-only	❌
F6. Gold-label human spot-check worksheet	✅ spotcheck (human pass)	❌	❌	❌	❌	❌	❌	❌
F7. Per-item public release (reproducible by a third party)	✅ release JSONL + manifest	⚠️	⚠️	✅	⚠️	❌	⚠️	❌
F8. Generate-parse parity path (vs cloze-favors-base)	✅ scoring_mode: both	❌	❌	n/a	❌	n/a	⚠️	❌
F9. Truncation reporting (flag over-think cap-artifacts)	✅ truncation_rate + WARNING	❌	❌	❌	❌	❌	❌	❌
F10. Calibration anchors (per-band accuracy vs expected)	✅ calibrate + --calibrate-to seahelm_v1 (belebele/NLI score split)	❌	❌	❌	❌	❌	❌	❌
F11. Reasoning-mode parity gate (thinking-on vs off confound)	✅ --strict-parity	n/a	n/a	n/a	n/a	⚠️	⚠️	❌
F12. Calibrated fast-run = full reference (a fast sample validated against the full run, within CI)	✅ 50-row fast matches the 24,213-row full MalayMMLU reference within CI for 4/4 models, 6–19× cheaper (report)	❌	❌	❌	❌	❌	❌	❌

On these 12 credibility properties GlossoBench ships all of them built-in; the field ships few. This is the layer that turns a self-authored "win" into an independently-validated result — and F12 is what lets a team quote a fast-run number as a calibrated estimate of the full reference, not a hand-wave.

The honest caveats (we don't hide them)

• GlossoBench-7 (Malay) v1 is young. The self-built IF/Safety/Cultural/MT sets are starter-size in v1; they grow via MB_*_DIR env or plugin bumps. Public third datasets (MalayMMLU, FLORES, belebele, Global-MMLU, SEA-Guard judges) carry the bulk of v1 row count.
• A self-authored benchmark winning on its own axes is not a neutral claim. Credibility = (a) 100% public sources, (b) third-party models measured on the SAME harness, (c) open-source, (d) held-out + contamination audit, (e) bootstrap CI. GlossoBench ships all five; without all five a "win" is marketing.
• GlossoBench does not duplicate existing datasets — it HOSTS them under one reproducible, versioned harness. MalayMMLU/FLORES/belebele remain their authors' work; GlossoBench is the harness that makes them comparable + extensible + judge-free.
• Judge axes (MT, and the free-text Safety path) need a judge. GlossoBench degrades gracefully (skip + flag) if none is configured, and the safety_tox axis gives a judge-free Safety number, but a deep MT or free-text Safety number requires a judge. The judge is pluggable and never single-vendor (std-dev ensemble).
• The langs/ms/config.yaml default judge (MiniMax M3) is $0 but not offline + not MIT. It runs under the MiniMax token plan (free allowance, $0 at default eval volumes) but needs MINIMAX_M3_KEY + network, and the M3 weights are MiniMax Community License (commercial-OK with attribution + a <$20M-rev notice / >$20M prior-auth step — NOT MIT/Apache). Since 1.5.0 the langs/ms DEFAULT judge is nvidia-nemotron-judge (NIM free-tier, $0 credits, but hosted + online + key = a SOFT GATE; NVIDIA Open Model License has use-case restrictions, not OSI). So row 5 ($0) holds, but row 1 (no gate) + row 19 (license-clean) are cleanest via the offline ollama judge (ollama-glm52, MIT, localhost, no key, no network), which remains the new-language template default + the langs/ms/config.yaml gate-free fallback. Picking the NIM Nemotron default trades the offline/MIT-clean path for judge quality at $0. Full per-judge audit: LICENSE_AUDIT.md.
• Peers have real strengths this matrix does not credit. SEA-HELM ships a deeper, human-vetted multi-turn competency than GlossoBench's v1 suite; HELM covers a breadth of English-centric scenarios with mature cost/efficiency reporting; LMSYS captures genuine human preference at a scale no static bench can. GlossoBench complements these — it does not subsume them.

How GlossoBench composes each (not replaces)

GlossoBench can host the same measurements as the following — not supersede their data — because it wraps their public datasets in one reproducible, versioned, judge-agnostic harness. The datasets remain their authors' work; GlossoBench is the harness around them.

Existing	What it covers	How GlossoBench hosts it (authors credited)
SEA-HELM (whole suite)	7 SEA axes, gated, their judge	GlossoBench-7 hosts public equivalents of the same axes, judge-agnostic, any-language — built on SEA-HELM's prior art, not a replacement
MalayMMLU (standalone)	Malay Knowledge MCQ	GlossoBench knowledge axis via malaymmlu plugin (UMxYTLAILabs, BSD-3) (+ NLU/IF/… for free)
MMLU / Global-MMLU (standalone)	Knowledge MCQ	GlossoBench knowledge axis, language-parametric (Global-MMLU, Apache-2.0)
FLORES-200 (standalone)	NLG translation	GlossoBench nlg axis via flores-<lang> plugin (chrF + MetricX); FLORES-200 NC noted, SA/OLDI path for commercial
Belebele (standalone)	NLU reading MCQ	GlossoBench nlu axis via belebele-<lang> plugin (Meta, CC-BY-SA)
LMSYS Arena (for a single language)	human-preference Elo, cloud-hosted	GlossoBench Elo leaderboard, open, std-dev ensemble, $0 default — a complement to human-preference Elo, not a substitute for it

The composition is opt-in per dataset: GlossoBench's plugin model means you can keep using MalayMMLU's raw data — GlossoBench just scores it alongside 6 other axes with reproducible versioning, instead of as an isolated single-axis scorecard.

Why the properties generalize (the design argument)

The 23 properties are not Malay-specific or SEA-specific — they are the properties any low-resource-language team (African, Aboriginal, Pacific, etc.) needs and currently lacks. The design argument: a team that today lacks SEA-HELM's access gate or H100 infrastructure, lacks LMSYS's cloud-judge budget, and cannot fully trust MMLU (English-centric + contaminated) can run GlossoBench on a phone with a GGUF and $0 of API, plug in their own public cultural/safety corpus, and get a versioned, reproducible score with bootstrap CI on the same axis definitions a frontier model is measured on. The empirical claim is narrower today: this path is proven end-to-end for the shipped languages, and each new language proves it again by shipping a config + plugins (see ADDLANGUAGE.md). One harness; every language the community wires in; every device; every budget.

The harness-layer additions that make it a standard, not just a benchmark

• Register stratification (row 16) — a flat MS hides the real story. A model can ace formal Malay (news, exams) and collapse on colloquial ("dah mkn x?"). GlossoBench reports high/mid/low sub-scores per axis + a register-composite MS, so you see where a model is weak. No listed peer stratifies by register. This matters most for low-resource languages where the formal↔colloquial gap is the actual failure mode.
• Partial completion (row 17) — real-world teams can't always run all axes (no GPU for MT, no judge for free-text Safety, no corpus for Cultural). Every other multi-axis benchmark is all-or-nothing or averages over missing axes. GlossoBench excludes missing axes/registers from the MS denominator (never zeroes them) and reports skipped_no_judge + null_rate, so a 3-axis MS is never mistaken for a 7-axis MS. Run what you have, honestly.
• Containerized global run (row 18) — one podman run, no Python env, no CUDA hunt, no gate. CPU-slim image (~400MB) runs the 5 judge-free axes on a phone-class box; CUDA flavor for BF16. This is what makes adoption possible for the teams SEA-HELM/HELM/LMSYS structurally exclude.
• License-clean for commercial minority-language adoption (row 19) — a benchmark can be public and still legally unusable by a commercial minority-language team: framework code under a proprietary/NC license, or a default dataset that is CC-BY-NC (FLORES-200 Meta), or a default judge that needs a paid hosted API. GlossoBench's framework is Apache-2.0, every default-bundled dataset is commercial-use OK (Apache/MIT/BSD/CC0/CC-BY/CC-BY-SA), NC/gated/proprietary sources are opt-in-only and never in a default config, and no model weights are bundled (users mount their own). Full per-component audit: LICENSE_AUDIT.md. This is what makes the "adopt globally for all minority languages" claim legally true, not just technically true.
• Judge-free Safety variant (row 21) — Safety is the axis teams most often skip for lack of a judge. The safety_tox axis (1.3.0) scores toxicity / hate-speech via a classifier with no judge, so a no-judge run still gets a real Safety number; the judge-required free-text path remains for teams that want the deeper reading.
• Inference throughput + VRAM reported (row 22) — a score without its cost is half a decision. GlossoBench (1.3.0) records decode_tok_s, prefill_tok_s, vram_peak_gb, cpu_ram_peak_gb per run, engine-provenanced, so a deployment team can compare models on quality and on the hardware they actually own.
• Community language-pack endorsement (row 23) — a standard is only a standard if others can extend it. The Core/Community tier split + glossobench-contrib + the published endorsement bar (ENDORSEMENT.md) let a third-party language pack become the de-facto recommended run for its language — the same mechanism TechEmpower uses for web frameworks.

Path to adoption (honest self-assessment)

Bottom line up front

GlossoBench already carries statistical-rigor features neither incumbent ships — the 2026-07-23 audit remediation gave it item-level CIs, BT-MLE Elo, chance normalization, contamination gates, and immutable freezes that neither SEA-HELM nor MalayMMLU has. Structurally, it is not close to standard status, and the blockers are NOT code: they are governance, data scale, external adoption, and distribution. The historical pattern across 14 benchmarks is blunt: leaderboard hosting + top-tier venue + frontier-lab numbers at launch + small focused scope = de facto standard. SEA-HELM and MalayMMLU each have a peer-reviewed paper and institutional identity but no live leaderboard and no frontier-lab endorsement — their gaps leave room for a complementary harness. GlossoBench currently has none of the four.

Single binary discriminator (from the adoption-history analysis): a public, continuously-updated leaderboard that external teams actually submit to. Benchmarks that never got leaderboard hosting (HELM, BIG-bench, SEA-HELM, MalayMMLU, IrokoBench) all capped at "influential paper" tier. Benchmarks that did (MMLU via HF Open LLM Leaderboard, Chatbot Arena, IFEval) became standards.

Decision matrix

Legend: ✅ have it · ◐ partial · ❌ missing. "Blocks adoption?" = does the gap alone prevent standard status. Effort = realistic person-weeks (pw) / calendar / cash from the adversarial review, sanity-checked.

#	Dimension	GlossoBench today	SEA-HELM	MalayMMLU	Blocks adoption?	Fix	Effort
1	Peer-reviewed paper	❌ none	✅ LREC-COLING	✅ published	YES — root blocker. Without it, a self-audited single-repo bench reads as a vendor benchmark, whatever the code quality.	Submit NeurIPS D&B (preferred) or ACL/EMNLP; arXiv preprint first	16–24 pw, 6–12 mo calendar
2	Multi-institution governance	❌ single team	✅ AISG + Monash + collaborators	✅ university	YES. Independence of authorship is the minimum credibility bar for a bench that ranks other people's models.	Recruit ≥1 non-team institution as co-maintainer (SEACrowd-aligned lab, UM, A*STAR); 3-person unaffiliated advisory board; governance charter	4–8 pw + partnership calendar
3	Headline built on own novel data	◐ ms_public = 3 axes, ALL repackaged (MalayMMLU + Global-MMLU + Belebele + FLORES); the 5 novel axes are n≤29 self-built, honestly excluded from headline	✅ own curated tasks	✅ own items	YES. A complementary harness still needs its own validated headline data to be cited alongside them. Honest exclusion of tiny self-built axes is correct science but leaves nothing novel in the headline.	Grow each self-built axis to n≥500 with ≥3 native-Malay annotators, 2-round, κ≥0.7 → promote to headline. Cultural + Safety first (highest differentiation, worst current n)	12–20 pw + $15–30k/axis + 3–4 mo/axis (parallelizable)
4	Items per language ≥2,000	❌ (~2.2k public rows ms, but novel content ≪)	◐ few hundred/task	◐ <2k	YES for leaderboard hosts — historical refusal threshold	Same as #3 — annotation scale-up	included in #3
5	External runs / any adoption	❌ zero external runs	◐ cited regionally	◐ cited in MY papers	YES. Adoption is downstream of running, not publishing.	Pre-compute ≥10 open models (SEA-LION, Sailor, SeaLLM, Aya, Qwen, Llama, Gemma…); public leaderboard; sponsor 5–10 external teams to run it	4–8 pw + $5–15k GPU
6	Frontier-lab numbers at launch	❌ none	❌ none (their gap too)	❌ none	YES — cheapest highest-ROI single action per the historical analysis	Run GPT-4o/Claude/Gemini/Llama-405B/Qwen-72B/DeepSeek via API backends (already built: --model-kind api) and publish	2–3 pw + API cost
7	Public live leaderboard	◐ CI job just built (small demo roster, CF Pages)	❌ static	❌ none	YES — the binary discriminator.	Grow the CI leaderboard into a real submission flow (HF Spaces mirror; auto-eval submitted models). Bones exist (glossobench elo, render tool, CI pipeline)	4–6 pw
8	lm-eval-harness integration	❌	❌	◐ (community tasks exist)	Mostly — decides HF-community pickup ("one-line install" adoption path)	Contribute a lm_eval task spec wrapping the public axes	2–3 pw
9	Meta-evaluation vs incumbents	◐ tooling exists (correlate, C11-certified) but never run as a published head-to-head	n/a	n/a	YES — without rank-correlation + residual-signal evidence on shared models, "complementary" is unfalsifiable rhetoric	Pre-registered head-to-head: 10 shared models × {GlossoBench, SEA-HELM subset, MalayMMLU} + 200-prompt human-preference meta-correlation	6–10 pw + $5–10k
10	Native-speaker validation w/ reported agreement	❌ (human_validated=False on all self-built; honestly labeled)	◐	◐	YES for the novel axes; "without α>0.7 per language, LRL benches are dismissed as machine-generated"	Same annotation program as #3; publish per-axis Krippendorff's α	included in #3
11	Statistical rigor (CIs, estimator, Elo, chance-norm)	✅ item-level CIs + BT-MLE Elo + chance-norm (post-audit; built-in)	❌ point estimates	❌ accuracy only	No — this is a built-in differentiator	Maintain; make it the paper's methods section	—
12	Openness (ungated data, $0 judge, offline)	✅ 100% public, judge-agnostic, CPU-to-phone	❌ datasets gated (verified 2026-07-21); GPT-judge coupled	✅ open	No — a second differentiator. Gating is SEA-HELM's most complained-about property	Keep absolute; never add a gated source	—
13	Contamination defense	✅ dedup+SHA gates, audit-corpus CI, canary	❌	❌ (HF-hosted MCQ = prime training-set fodder)	No — a third differentiator (contamination gating is increasingly expected in citable benchmarks)	Add post-cutoff "fresh item" releases per version to strengthen it	1–2 pw/version
14	Versioned immutability	✅ freeze + SHA pins + verify-freeze CI	❌ (revisions broke cross-time comparison — a real complaint)	◐ static	No — a fourth differentiator	Keep	—
15	Multi-language breadth	◐ 8 lang configs; only ms full-axis	✅ 7–8 SEA languages, comparable depth	❌ ms only	Partly — "leaders mobilize for regions, not languages"; but avoid the HELM comprehensiveness trap	Bring id/th/vi to parity next (public axes already wired); do NOT chase all 8 at once	4–6 pw for 3 langs
16	Regional institutional endorsement	❌	✅ AI Singapore identity	◐ MY academia	Long-term yes (procurement side)	After paper: MDEC/MOSTI (MY), engage AISG rather than fight them — co-maintainership converts the chief peer into a distribution channel	calendar-bound
17	Single citable headline number	✅ ms_public + pooled MS (HELM's fatal gap avoided)	◐ per-task tables	✅ one number	No	Keep exactly one headline; resist axis sprawl	—
18	Code-switching / dialect coverage (Manglish, Kelantan, Sabah…)	❌	❌	❌	No, but it's the offensive differentiator — "unique value no incumbent in this matrix offers" and neither incumbent has it	Add a code-switch/dialect axis to the annotation program in #3	folds into #3
19	Independent methods audit / preregistration	❌ self-audit only (58-item CRITIC_AUDIT is QA, not validation — "the team cannot audit its own priors")	❌	❌	YES for the paper's reviewers	Preregister analyses (OSF) before next results drop; commission an uninvolved academic stats group to audit estimator/CI/Elo choices; timestamp analysis code	$5–15k + 1–2 mo
20	Item psychometrics (IRT/2PL, discrimination, point-biserial)	❌ none; Elo on n=12–29 axes can flip on one model add/remove	❌	❌	Partly — matters once rankings are cited	Fit IRT 2PL per axis once n≥200 (needs #3 first); publish item parameters; add expert Malay-linguist human ceiling run	2–4 mo + $10–20k
21	Semantic (not just lexical) contamination screening	◐ shingle+exact hard-exclude, TF-IDF proxy + opt-in embedder pass, canary opt-in — but no MinHash/LSH at web scale, no membership-inference probes, no regurgitation monitoring	❌	❌	Partly — frontier labs increasingly demand it	MinHash/LSH + embedding near-dup vs Common Crawl SEA slices; periodic membership-inference probes vs major open models; monitor for canary regurgitation	$10–25k + quarterly
22	Construct validity (downstream correlation + human ceiling)	❌ zero evidence scores predict real Malay task performance	❌	❌	YES — "a benchmark that doesn't correlate with a downstream construct is a leaderboard, not a measurement instrument"	Preregistered correlation study: ≥10 models × ≥3 downstream Malay tasks + expert-baseline ceiling; report Kendall's τ with CIs	$20–50k + 3–6 mo
23	Honest multilingual claim	◐ 8 configs but only ms full-axis; other langs have no native panel — "greenwashing language coverage" if oversold	✅ real 7–8-lang depth	n/a	Partly — messaging risk more than technical	Either label non-ms configs "public-axis only, panel-pending" everywhere user-facing (cheap, do now) or fund per-language panels ($30–80k/lang) for the top 3	1 pw (labeling) now; panels later

What the incumbents still have that we lack (honest ledger)

• SEA-HELM: peer-reviewed paper; multi-institution authorship; AI Singapore's institutional identity + SEA-LION pipeline alignment; 7–8 languages at comparable task depth; regional government mindshare.
• MalayMMLU: peer-reviewed paper; MMLU-clone familiarity (zero learning curve for any evaluator); Malaysian-academia citation base.
• Both: they exist in other people's bibliographies. We exist in our own.

What we have that they can't easily copy

Openness (their gating is structural — licensing), statistical rigor (retrofit = admitting prior numbers were wrong), contamination gates, immutable versioning, judge-agnostic $0 path, plugin extensibility, and the published 58-item self-audit — which becomes a strength in a paper (unprecedented transparency) even though it's currently just internal.

The single highest-leverage move (adversarial reviewer's pick)

An independent, preregistered meta-evaluation of GlossoBench vs SEA-HELM vs MalayMMLU on a shared panel of ≥10 models, correlated against ≥2 downstream Malay tasks, written up with a non-GlossoBench-team first author and submitted to EMNLP/ACL. One action that simultaneously covers the paper gap (#1), first external adoption (#5), the missing meta-evaluation (#9), and construct validity (#22) — and it's the only move that converts the technical differentiators into evidence of incremental validity, which is the only justification for earning adoption alongside the incumbents rather than remaining an also-ran. "Without it, every other fix is decoration."

Priority order (dependency-sorted, not importance-sorted)

1. Frontier + open-model sweep NOW (matrix #6+#5) — cheapest, fastest, feeds everything else; --model-kind api already works.
2. Native-annotation program (#3/#4/#10) — the long pole (12–18 mo); start immediately because everything credibility-related waits on it. Cultural + Safety axes first.
3. Meta-evaluation head-to-head (#9) — needs #1's model runs; produces the paper's central falsifiable claim.
4. Paper + governance in parallel (#1/#2) — submit with #1+#3-partial+#9 inside; recruit co-maintainer institution during review.
5. Leaderboard + lm-eval-harness at acceptance (#7/#8) — historical window: on a leaderboard by ~month 14 or permanently "see also" tier.
6. Institutional endorsement (#16) — after the paper exists.

Realistic calendar to standard status if all lands: 24–36 months, with "default citation in Malay/SEA eval papers" achievable around month 18–24. No short-term deadline matters to this track — standard status is a multi-year campaign (consistent with the adoption-history analysis); what the near term CAN bank is #1 (model sweep) and the leaderboard bones (done).

Verdict

Good enough to build on — the harness is the strongest technical artifact in its class and the audit trail proves it. Not good enough to cite as standard until: novel validated data at scale (#3), a paper (#1), external runs (#5/#6), and a live leaderboard (#7). None of those are code problems. The adoption race is won at the annotation table, the program committee, and the leaderboard URL — in that order.

---

Definition of Done

• Every property row in the 23-property matrix and every F1–F12 lever is factual and current as of the cited 2026-07-23 sweep.
• License/coverage claims cross-link to LICENSE_AUDIT.md and resolve.
• Honest caveats and the "What GlossoBench does not do" section remain unsanded.
• The legal basis section is present and ties comparative claims to truth-in-advertising norms.

Legal basis for the comparisons above

The comparisons in this document are truthful, non-misleading comparative descriptions, consistent with the comparative-advertising doctrine under the US Lanham Act §43(a) and equivalent truth-in-advertising norms. Benchmark names (SEA-HELM, MMLU, FLORES, Belebele, HELM, LMSYS Chatbot Arena, MalayMMLU, Global-MMLU) are used nominatively — to identify what each benchmark measures and where GlossoBench complements it — not to endorse, disparage, or suggest affiliation. No false or disparaging claims are made; every license and coverage statement is factual and honored in LICENSE_AUDIT.md. Where a property is marked "n/a" for a dataset-tier benchmark it is because the property is a harness-level concern that does not apply to a standalone dataset, not a deficiency.

Related: METHODOLOGY.md (scoring + meta-evaluation protocol) · CRITIC_AUDIT.md (gap list) · FAIREST.md (fairness levers) · LICENSE_AUDIT.md (license bar) · ENDORSEMENT.md (endorsement policy)