Skip to content

GlossoBench — License Audit & Compatibility Policy

Status: Active · Audience: Commercial adopters, legal review · Updated: 2026-07-24

This is a per-component audit of every license GlossoBench touches, a verdict for each (commercial-use OK / opt-in / block), the fixes applied, and the standing policy that keeps future additions safe for any minority-language adopter — including commercial ones.

Purpose

Per-component license audit of every license GlossoBench touches, with a verdict (commercial-use OK / opt-in / block), the fixes applied, and the standing policy keeping future additions safe for any minority-language adopter.

Goals

  • Give every default-bundled component a commercial-use verdict backed by a verified upstream license.
  • Keep NC / gated / proprietary sources opt-in only and never in a default config.
  • Document the standing rule a new dataset/judge plugin must follow to stay license-clean.
  • Cite OSI, SPDX, and Creative Commons as the standards authorities backing each license verdict.

Success Criteria

  • Every row in the TL;DR table has a verified license field and a verdict traceable to an upstream source.
  • No default config silently bundles an NC or gated component.
  • The "checks before any future dataset/judge is added" checklist enforces the four required plugin fields (license, commercial_use, attribution, human_validated).
  • The standards-alignment section maps every license GlossoBench uses to OSI/SPDX/CC-official recognition.

TL;DR

# Component License Commercial use? Bundled as default? Status
1 GlossoBench framework code Apache-2.0 (now) ✅ n/a (it IS the framework) FIXED (was PROPRIETARY)
2 Self-built IF / Cultural / Safety / MT CC0-1.0 ✅ ✅ OK
3 MalayMMLU BSD-3-Clause ✅ ✅ OK
4 Global-MMLU (ms) Apache-2.0 ✅ ✅ FIXED (was CC-BY-4.0)
5 Belebele (zsm_Latn) CC-BY-SA-4.0 ✅ (share-alike) ✅ FIXED (was CC-BY-4.0)
6 FLORES (OLDI openlanguagedata/flores) CC-BY-SA-4.0 ✅ (share-alike) ✅ FIXED (was tagged CC-BY-NC)
7 MetricX-24 (NLG metric) Apache-2.0 ✅ ✅ (opt-in, lazy) OK
8 IFEval (constraint checkers, faithful port) Apache-2.0 ✅ ✅ OK
9 SEA-Guard judge (Qwen-4B/8B-2602) Apache-2.0 ✅ ✅ (opt-in judge) OK
10 SEA-Guard judge (Llama-8B variant) Llama 3.1 Community License ⚠️ 700M-MAU clause ❌ opt-in only flagged
11 Default judge (langs/ms/config.yaml): MiniMax M3 via platform API MiniMax Community License (weights) + platform TOS ⚠️ conditional (attribution + <$20M-rev notice / >$20M prior auth; military-use prohibited) ✅ (langs/ms/config.yaml primary) $0 token plan; needs key+net
12 Default judge (template / offline alt): local ollama (GLM-5.2) MIT (weights) ✅ ✅ (_template.yaml/_schema.yaml default) FIXED (was hosted URL)
13 fasttext lid.176.bin (lang-id, optional) CC-BY-SA-3.0 ✅ (share-alike, viral) ❌ opt-in flagged; alt GlotLID-V3 Apache
14 OpenRouter judge (gpt-oss-120b) service TOS n/a (paid) ❌ never default opt-in
15 SEA-NLI (opt-in Malay cultural NLI, sea-nli-ms) MIT (data) ✅ (MIT) but access-gated on HF ❌ opt-in only (langs/ms_seanli/config.yaml, bench_version 1.0-seanli) $0; needs HF_TOKEN or MB_SEANLI_DIR mirror
16 Six-language public sweep (th/vi/id/my/fil/ta): Belebele NLU + Global-MMLU Knowledge + FLORES NLG Apache-2.0 (Global-MMLU) / CC-BY-SA-4.0 (Belebele, FLORES) ✅ (Belebele + FLORES = share-alike) ✅ (active in langs/{th,vi,id,my,fil,ta}/config.yaml) FLORES gated on HF → MB_FLORES_DIR mirror; Belebele + Global-MMLU public
17 BFCL (Berkeley Function-Calling-Leaderboard, function_calling axis) Apache-2.0 ✅ ✅ (default in langs/en/config.yaml, additive — outside bench_version-1.0 MS) OK

Every default-bundled component is commercial-use OK. The two non-commercial or use-restricted items (original Meta FLORES-200 CC-BY-NC; Llama-Community judges) are either swapped out or marked opt-in-only and never appear in a default config. One caveat (2026-07-23): the langs/ms/config.yaml default judge is MiniMax M3 (MiniMax Community License — commercial use OK with attribution + a <$20M-revenue notice / >$20M prior-authorization step, and a military-use prohibition; NOT MIT/Apache). It is $0 under the MiniMax token plan but needs MINIMAX_M3_KEY + network. The fully-permissive offline path — local ollama GLM-5.2 (MIT), no key, no network — remains the new-language template default + a bundled alt in langs/ms/config.yaml. A commercial minority-language adopter that needs the cleanest license path uses the ollama judge; one that wants judge quality at $0 accepts the MiniMax Community terms.

Every license GlossoBench uses is drawn from one of three recognized standards:

License OSI-approved SPDX-listed Creative-Commons-official
Apache-2.0 (framework code, Global-MMLU, MetricX-24, IFEval, SEA-Guard Qwen) ✅ ✅ (Apache-2.0) n/a
BSD-3-Clause (MalayMMLU) ✅ ✅ (BSD-3-Clause) n/a
MIT (GLM ollama judge, SEA-NLI) ✅ ✅ (MIT) n/a
CC-BY-SA-4.0 (Belebele, FLORES/OLDI) n/a ✅ (CC-BY-SA-4.0) ✅
CC0-1.0 (self-built datasets) n/a ✅ (CC0-1.0) ✅

Standards authorities cited: the Open Source Initiative (opensource.org/licenses) for OSI approval of software licenses; the SPDX spec (Linux Foundation, spdx.org/licenses) for standardized license identifiers; and Creative Commons (creativecommons.org) for the CC licenses. GlossoBench's framework is Apache-2.0 (OSI-approved + SPDX-listed); its hosted datasets are Apache-2.0 / BSD-3-Clause / MIT (OSI+SPDX) or CC-BY-SA-4.0 / CC0-1.0 (CC-official + SPDX).

Honest disclaimer: no single world body "approves" benchmark methodology or documentation. GlossoBench's legal posture rests on (a) every component carrying an OSI-approved / SPDX-listed / CC-official license and (b) truthful, non-misleading comparative claims (see COMPARISON.md). We cite OSI, SPDX, and Creative Commons as the relevant standards authorities; we do not claim endorsement by them or by any other body.

The three real problems found + fixed

1. Repo framework was PROPRIETARY — blocked ALL adoption

The parent repo LICENSE is "PROPRIETARY AND CONFIDENTIAL" and pyproject.toml declared LicenseRef-Proprietary. A benchmark whose code is proprietary cannot be adopted globally, full stop — it fails decision-matrix property

1 ("100% public, no access gate") at the source.

Fix: glossobench/ now ships its own LICENSE = Apache-2.0 and a NOTICE scoping it as a standalone permissively-licensed package carved out of the proprietary parent. The framework code (runner, plugins, axes, judges, scoring, config, CLI, self-built data, Containerfile) is Apache-2.0. The parent repo outside glossobench/ stays proprietary; GlossoBench has no dependency on it. This is what makes fork/redistribute/adopt by any team legal.

2. FLORES was tagged CC-BY-NC-4.0 (non-commercial) — blocked commercial adopters

The original Meta facebookresearch/flores release (FLORES-200) is CC-BY-NC-4.0 → non-commercial only. Our plugin was tagged that license but was actually loading openlanguagedata/flores (the OLDI release), which is CC-BY-SA-4.0 — commercial-use OK with a share-alike obligation on derivatives. The stale tag misrepresented the data.

Fix: FloresMalay.license corrected to CC-BY-SA-4.0, commercial_use = True, attribution names OLDI + the share-alike obligation. Docstring explains the distinction and what to set if a team chooses the Meta NC release instead (opt-in, commercial_use = False). We use OLDI so default adopters face no NC restriction.

3. Default judge pointed at a hosted service (soft gate)

OllamaJudge defaulted base_url to https://ollama.com/v1 — a hosted endpoint requiring OLLAMA_GL_KEY. For an offline-first / poor-country / aboriginal team this is a soft gate (needs network + a key), contradicting decision-matrix property #5 ("Runs with ZERO paid API ($0)").

Fix: default is now http://localhost:11434/v1 (real local ollama, $0, no key, no network). A hosted URL is only used if OLLAMA_BASE_URL is explicitly set. API key optional. The config schema comment now states OpenRouter/paid judges are never a default.

Full per-component audit

Framework code

  • License: Apache-2.0 (glossobench/LICENSE).
  • Scope: everything under glossobench/ except third-party dataset rows (which keep their own licenses — GlossoBench hosts/scores, never relicenses).
  • Verdict: ✅ commercial-OK, forkable, redistributable.

Self-built datasets (langs/ms/datasets/selfbuilt.py)

  • IF (Malay IFEval, 21 constraint types), Cultural (MCQ/NLI), Safety, MT.
  • License: CC0-1.0 (public domain) — the most permissive possible; no attribution even required, no restriction of any kind.
  • Verdict: ✅ maximum adoptability. A minority-language team can copy, modify, commercialize without any obligation.

MalayMMLU — UMxYTLAILabs/MalayMMLU

  • License: BSD-3-Clause (verified on the source GitHub repo UMxYTL-AI-Labs/MalayMMLU LICENSE.txt, Copyright Chee Seng Chan 2024 — the 3-clause BSD with a no-endorsement clause; not declared on the HF dataset card, so cite the repo, not the card).
  • Verdict: ✅ commercial-OK (OSI-approved, permissive).

Global-MMLU (ms) — CohereLabs/Global-MMLU (was CohereForAI/Global-MMLU)

  • License: Apache-2.0 (verified on the HF dataset card README, which states "this dataset can be used for any purpose, under the terms of the Apache 2.0 License"; the CohereForAI/ path still resolves via redirect).
  • Verdict: ✅ commercial-OK (attribution + explicit patent grant).

Belebele (zsm_Latn) — facebook/belebele

  • License: CC-BY-SA-4.0 (verified on the HF dataset card facebook/belebele; previously mislabeled CC-BY-4.0 — the share-alike obligation was missing).
  • Verdict: ✅ commercial-OK (attribution + share-alike on derivatives — a real but minor obligation, now disclosed in the plugin's attribution).

FLORES — openlanguagedata/flores (OLDI)

  • License: CC-BY-SA-4.0 (verified — OLDI/flores_plus release; commercial-OK, share-alike on derivatives).
  • Note: the archived Meta facebookresearch/flores (FLORES-200) is CC-BY-NC-4.0. We deliberately use the OLDI release. If a team swaps to Meta's, they must set commercial_use = False (opt-in, non-commercial only).
  • Verdict: ✅ commercial-OK (share-alike). Adopters shipping a derivative translation set must share-alike under CC-BY-SA-4.0 — a real but minor obligation, documented in the plugin's attribution.

MetricX-24 — google/metricx-24-hybrid-large-v2p6 (NLG metric)

  • License: Apache-2.0 (verified on HF model card + google-research/metricx).
  • Verdict: ✅ commercial-OK. Lazy-loaded; not bundled, user supplies.

IFEval — google/IFEval + google-research/instruction_following_eval

  • License: Apache-2.0 (verified). Our ifeval_checkers.py is a faithful port of the verifiable-instruction checkers; the upstream code is Apache-2.0.
  • Verdict: ✅ commercial-OK.

SEA-Guard judge — aisingapore/Qwen-SEA-Guard-{4B,8B}-2602

  • License: Apache-2.0 (verified) for the Qwen-based 4B/8B variants.
  • Verdict: ✅ commercial-OK. These are the recommended judges for the Safety axis (AUPRC). Not bundled; user supplies weights.

SEA-Guard judge — aisingapore/Llama-SEA-Guard-8B-2602

  • License: Llama 3.1 Community License (verified) — permits commercial use up to 700M monthly active users, plus attribution.
  • Verdict: ⚠️ opt-in only. Not a default; flagged in the plugin. Fine for small/minority-language deployments (well under 700M MAU); large deployers should use the Apache Qwen variants.

fasttext lid.176.bin (optional language-id)

  • License: CC-BY-SA-3.0 (verified) — viral share-alike; commercial-OK.
  • Verdict: ⚠️ opt-in only (not bundled). Alternative for teams that want permissive-only: GlotLID-V3 (Apache-2.0, superset of LID-176 languages) — recommended for the minority-language use case. Documented; no default dependency on the viral model.

Default judge (langs/ms/config.yaml): MiniMax M3 via platform API

  • License: MiniMax Community License (the M3 weights) + MiniMax platform API terms. Verified on the MiniMaxAI/MiniMax-M3 HF card (license: other / license_name: minimax-community) — NOT MIT/Apache despite some launch write-ups calling it MIT.
  • Commercial use: ⚠️ conditional. Allowed if (a) you prominently display "Built with MiniMax M3", and (b) for <$20M/yr revenue you send a one-time notice email, for >$20M/yr you obtain prior written authorization (api@minimax.io). Military use + the standard prohibited-use list are barred.
  • Cost/access: $0 under the MiniMax token plan (free allowance at default eval volumes); needs MINIMAX_M3_KEY + network (a soft gate vs the offline ollama path). GlossoBench ships no weights — it calls the platform API.
  • Why it's the langs/ms/config.yaml default: judge quality (M3 is a reasoning model). The trade-off vs ollama is the license conditionality + the key/network gate.
  • Verdict: ⚠️ commercial-OK-but-conditional. Fine for evaluation + for commercial adopters under $20M revenue with attribution; the MIT ollama path below is the cleaner commercial default.

Default judge (template / offline alt): local ollama (GLM-5.2)

  • License: MIT (GLM-4.5/5 weights, verified on zai-org HF cards).
  • Verdict: ✅ commercial-OK, fully permissive. User obtains the model under MIT; GlossoBench ships no weights. Default endpoint is localhost:11434 (offline, $0, no key, no network) — the gate-free path, and the _template.yaml / _schema.yaml default for new languages.

OpenRouter judge (gpt-oss-120b)

  • License: service terms (paid API).
  • Verdict: opt-in, never a default. A team that wants it sets OPENROUTER_API_KEY and adds the judge slug explicitly. No silent dependence.

SEA-NLI (opt-in Malay cultural NLI — sea-nli-ms)

  • Source: aisingapore/SEA-NLI — the 3-way NLI dataset behind SEA-HELM v1.2.1's Cultural axis. Malay subset (culture == "malaysian") = 195 rows (114 normal + 81 hard, the culturally-adversarial split).
  • License: MIT (commercial-use OK) — BUT the dataset is access-gated on HF (free aisingapore access token required to download).
  • Verdict: ⚠️ opt-in only. The gate conflicts with GlossoBench's "100% public, no gate, poor-country-runnable" default, so sea-nli-ms sets commercial_use = False, is NOT referenced by the gate-free langs/ms/config.yaml, and ships only via the separate langs/ms_seanli/config.yaml (bench_version 1.0-seanli — a separate leaderboard, not mixed with gate-free 1.0 runs). human_validated = True (native-speaker-curated by aisingapore). Gate-free offline fallback: MB_SEANLI_DIR → a local JSONL mirror (anyone who downloaded it once can re-run with zero network + zero token). Use it when you need the v1.2.1 apples-to-apples Cultural number; the gate-free malay-cultural-selfbuilt (CC0) remains the default Cultural data.

Six-language public sweep (th / vi / id / my / fil / ta)

  • What: the 2026-07-23 multi-language sweep adds configs + per-language dataset plugins for Thai, Vietnamese, Indonesian, Burmese, Filipino, Tamil — the public-axis path for languages with no self-built native content yet.
  • Datasets + licenses:
  • Belebele NLU (all 6) — facebook/belebele, CC-BY-SA-4.0, public (not gated; share-alike on derivatives). Configs: tha_Thai, vie_Latn, ind_Latn, mya_Mymr, tgl_Latn (Filipino = Tagalog — FLORES/Belebele have NO fil_Latn), tam_Taml.
  • Global-MMLU Knowledge (vi / id / fil only — Thai/Burmese/Tamil have no Global-MMLU config) — CohereLabs/Global-MMLU (was CohereForAI/), Apache-2.0, public. Configs: vi, id, fil (verify fil exact config name on first real load).
  • FLORES NLG (all 6) — OLDI openlanguagedata/flores, CC-BY-SA-4.0, gated on HF. Configs: same FLORES-200 codes as Belebele. No bundled mirror for these six (only Malay is bundled); use a MB_FLORES_DIR local mirror (pairs.jsonl or flores.eng + flores.<iso>) or HF auth. The runner skips NLG on load failure (partial-tolerant), so the offline default runs NLU (+ Knowledge where available) only.
  • Self-built axes (IF / Safety / Cultural / MT): intentionally NOT shipped for these six. Per Standing Order 4 we do not fabricate native strings or use LLM-generated data for gold; these axes grow when a native author writes CC0 seed rows (datasets/<lang>_selfbuilt.py), exactly as Malay + Swahili did. Each config lists them as commented growth slots.
  • Judge: nvidia-nemotron-judge (NVIDIA Nemotron-3-Super-120B-A12B via NIM free-tier, $0 credits, primary since 1.5.0; needs NVIDIA_NIM_API_KEY + network), consistent with langs/ms/config.yaml. minimax-m3 ($0 token plan)
  • ollama-glm52 (offline, no key) are the alt / gate-free fallbacks. Nemotron weights are NOT bundled (NIM-hosted or user-mounted; NVIDIA Open Model License has use-case restrictions — the judge plugin stays Apache-2.0, ours).
  • Verdict: ✅ every active dataset is commercial-use OK (Apache-2.0, BSD-3-Clause, or CC-BY-SA-4.0 share-alike). FLORES is the only gated one and it is mirror-preferred + skip-tolerant, so no adopter is forced through the gate.

BFCL — gorilla-llm/Berkeley-Function-Calling-Leaderboard (function_calling axis)

  • What: the 2026-07-29 v1.4.0 release adds a judge-FREE function_calling axis (tool-use / function-calling), scored by AST exact-match of the model's emitted function call(s) against a frozen ground-truth call list — no LLM judge (same verifiable principle as Knowledge cloze / Belebele MCQ).
  • Data: the BFCL single-turn trees (simple / multiple / parallel / parallel_multiple / irrelevance), loaded from gorilla-llm/Berkeley-Function-Calling-Leaderboard on HF (JSONL data files + possible_answer/ ground-truth files, joined on id). A GlossoBench-authored CC0-1.0 offline fixture is the zero-network smoke fallback.
  • License: Apache-2.0 (confirmed on the HF dataset card: license: apache-2.0; the Gorilla README states "All the models, and data used to train the models are released under Apache 2.0"). The offline fixture is CC0-1.0 (GlossoBench-authored).
  • Commercial use: ✅ Apache-2.0 is the most permissive common license; no attribution-virality, no revenue gate.
  • Default? ✅ default in langs/en/config.yaml (BFCL is English, so en is its native home). It is additive (additive: true): scored + reported per-axis but kept OUT of the bench_version-1.0 composite MS, so adding it does not change the frozen v1.0 headline's meaning. enabled: false opts out.
  • Scope (honest): v1 scores the single-turn trees only. The BFCL multi-turn trees need a tool-execution loop the single-turn runner does not run; a future function_calling_mt axis can add them. Stamped in every result's notes.
  • Verdict: ✅ Apache-2.0, commercial-use OK, public (not gated), tagged (license/commercial_use/attribution on the BFCLDataset plugin per the standing rule).

License-compatibility policy (standing rule)

To guarantee a minority-language team (including a commercial one) can run the default config with zero legal surprise:

  1. Default-bundled datasets MUST be commercial-use OK. Allowed: Apache-2.0, MIT, BSD-2/3-Clause, CC0-1.0, CC-BY-4.0, CC-BY-SA-4.0. Share-alike (CC-BY-SA) is allowed with the obligation noted in attribution.
  2. NonCommercial (CC-BY-NC-*), gated-access, or proprietary datasets are OPT-IN ONLY. The plugin sets commercial_use = False. The runner emits a warning when such a dataset is configured; the config schema flags it. They NEVER appear in a shipped default config.
  3. Every dataset plugin carries four fields: license (SPDX), commercial_use (bool), attribution (source credit), and human_validated (bool — whether a native speaker / the upstream authors reviewed the gold labels). Enforced by the base class; a missing license is a bug. human_validated is a provenance + credibility flag, not a license field: False (the honest default for self-built starters) means the gold is unvalidated until a glossobench spotcheck human pass; True only for upstream-human-curated sources (MalayMMLU, GlobalMMLU, Belebele, FLORES). It never gates loading — a False set still runs — but it is recorded in the per-item release manifest so a third party knows which gold has been human-checked. See CONTRIBUTING.md "The human_validated standing rule".
  4. No model weights are bundled. Judges run against user-supplied models (mounted or local ollama). GlossoBench ships no weights → no weight-license liability in the image/repo.
  5. Default judges are $0. Two shipped defaults: langs/ms/config.yaml uses MiniMax M3 via platform API ($0 token plan, needs MINIMAX_M3_KEY + network, MiniMax Community License — commercial-OK with attribution + the <$20M-notice / >$20M-auth step); the new-language template defaults to local ollama (localhost, MIT, offline, no key). A poor-country / offline team uses the ollama path with zero network or key dependency. Hosted/paid judges (OpenRouter) are opt-in and NEVER a default. The fully-permissive (MIT) default is ollama; the MiniMax default trades that for judge quality at $0 under a conditional license — pick per your commercial + offline constraints.
  6. GlossoBench never relicenses third-party data. It hosts + scores it; downstream users honor each source's own license. The NOTICE file lists all sources + their licenses.
  7. Framework code is Apache-2.0, carveable from the proprietary parent repo. Anyone may fork, modify, commercialize, redistribute GlossoBench itself under Apache-2.0.

This policy is what makes the decision-matrix claims (properties #1 public, #5 $0, #6 any-device, #14 very-poor-country-testable, #18 containerized) legally true, not just technically true.

Checks before any future dataset/judge is added

  • SPDX license set on the plugin?
  • commercial_use correct? (False for any NC/gated/proprietary source)
  • attribution names the source + author (CC-* requires it)?
  • human_validated set honestly? (False for self-built/unreviewed; True only after a real human review)
  • If commercial_use = False: is it opt-in only, NOT in a default config?
  • If a judge: weights not bundled? user supplies? license disclosed (MIT ollama vs MiniMax Community vs service TOS)? $0 default or paid opt-in?
  • Row added to the TL;DR table in this file?

Failing any of these is a licensing bug, not a style choice.

Sources

Definition of Done

  • Every component row in the TL;DR table carries a verified license and a commercial-use verdict traceable to an upstream source link.
  • The standing-rule checklist enforces license / commercial_use / attribution / human_validated on every new plugin.
  • No default config silently bundles an NC, gated, or proprietary component; opt-in-only items link back here.
  • The standards-alignment section maps each license to OSI / SPDX / CC-official recognition and links to COMPARISON.md.

Related: COMPARISON.md · ENDORSEMENT.md · SPEC_v1.0.md