Skip to content

GlossoBench — endorsement policy (Core vs Community)

Status: Active · Audience: Contributors, third-party teams · Updated: 2026-07-24

"GlossoBench-Endorsed" means a language pack / recipe / bench package meets the published bar below, is listed in RECIPES.md, and is the recommended run for that language or use case. Endorsement is how a community contribution becomes the de-facto standard run for its language — the same mechanism TechEmpower uses (one harness + a published bar + a registry of accepted implementations), split across two repos.

Purpose

Define what "GlossoBench-Endorsed" means and the bar a language pack / recipe / bench package must clear to be the recommended run for its language.

Goals

  • Keep endorsement a single, published, version-stable bar across Core and Community tiers.
  • Make the path from community contribution to de-facto standard run transparent and reproducible.
  • Ensure no endorsed pack depends on gated data, NC licenses, or fabricated native strings.

Success Criteria

  • Every endorsed row in RECIPES.md traces to a pack meeting all 7 endorsement criteria.
  • A Core pack ships in the wheel; a Community pack runs zero-install via --bench <git-url>.
  • De-listing (license change, contamination, breakage) moves a row to an archived section with reason — rows are never silently deleted.

The two tiers

Core — lives in THIS repo under glossobench/langs/<lang>/. Maintainer-curated, reference-quality, ships in the pip wheel and the container image. Endorsement = merged here. Today: ms (7-axis reference pack), sw (self-built axes), th/vi/id/my/fil/ta (public-axis).

Community — lives in glossobench-contrib as a bench package (packages/<name>/ — see SPEC_v1.0.md §15). Third-party language packs, self-built IF/MT/Safety/Cultural sets, model- or framework-specific configs. Endorsement = merged into contrib AND a row added to this repo's RECIPES.md Community table. Runnable by anyone with zero install: glossobench run --bench <git-url> --model /path --no-judge.

A Community pack that proves itself (adoption + validation + a maintainer willing to own it) can be promoted to Core — moved into glossobench/langs/<lang>/ so it ships in the wheel. Promotion is a PR to this repo and follows the same bar plus native-speaker validation.

Endorsement criteria (ALL required)

  1. 100% public data, no access gate on the default path. Gated sources opt-in only, never default.
  2. License-clean: every dataset plugin sets license (SPDX) + commercial_use + attribution + human_validated. Defaults must be commercial-use OK (Apache/MIT/BSD/CC0/CC-BY/CC-BY-SA); NC/gated → commercial_use=False, opt-in only (per LICENSE_AUDIT.md).
  3. Held-out: self-built rows never enter any training corpus; contamination dedup via MB_DEDUP_CORPUS with the published SHA-256 of the scored subset.
  4. Honest validation flags: human_validated=False until an independent native-speaker pass; small-n axes are directional, not headline.
  5. CI-green: the package loads via --bench (or the lang via --lang), plugins auto-discover, ruff + smoke tests pass.
  6. A judge-free quickstart runnable on CPU for $0 — the one-command path a stranger can copy-paste.
  7. No fabricated native strings: every native-language gold row is authored by a native speaker (standing rule). LLM-generated gold is not eligible.

For a cross-model "win" claim on an endorsed pack, the additional bar in FAIREST.md applies (third-party models on the same harness, bootstrap CI, contamination SHA, etc.).

How to submit

  1. PR your bench package to glossobench-contrib (packages/<name>/; copy packages/_template/).
  2. Maintainers check the 7 criteria (checklist in the contrib repo's CONTRIBUTING.md).
  3. On merge, a PR to THIS repo adds your row to RECIPES.md — that row is the endorsement.

De-listing: a pack that breaks the bar (license change upstream, contamination found, unmaintained breakage) gets its row moved to an "archived" section with the reason — rows are never silently deleted.

Definition of Done

  • The submitted pack meets all 7 endorsement criteria and is merged into Core or glossobench-contrib.
  • A row exists in RECIPES.md pointing at the pack with a runnable one-command path.
  • The pack loads via --bench (Community) or --lang (Core), plugins auto-discover, and ruff + smoke tests are green.

Related: RECIPES.md (endorsed registry) · ADDLANGUAGE.md (add a Core language) · LICENSE_AUDIT.md (license bar) · FAIREST.md (win-claim bar) · SPEC_v1.0.md (package format §15)