Skip to content

GlossoBench — TODO / roadmap

Status: Active · Audience: Maintainers, contributors · Updated: 2026-07-28

Everything left to do, grouped by what blocks it. None of it is broken-code work — the framework is complete and smoke-tested (full smoke suite green). This is the "make it a real benchmark with a real result" list. Fault IDs (A/B/C*) refer to CRITIC_AUDIT.md; the credibility sequencing lives in COMPARISON.md ("Path to adoption").

Purpose

Tracks all remaining work to turn GlossoBench from a smoke-green framework into a published, credible benchmark with a real result.

Goals

  • Land the headline third-party measurement + freeze a v1 leaderboard.
  • Validate the non-vLLM backends (SGLang, API, docker) against real engines.
  • Close the Tier-0 audit faults and grow self-built datasets to full statistical power.
  • Register a CI runner and exercise the release pipeline end-to-end.

Success Criteria

  • A frozen, immutable v1 leaderboard for ms with MS / MS_norm + sigtest + Elo CIs published in the README.
  • SGLang and API-as-model backends produce cloze/generate outputs matching an HF reference within tolerance.
  • Default glossobench run applies ensemble + dedup + freeze at -O3 and emits a full provenance block.
  • Nightly + release workflows fire on a registered runner and publish a container image to the registry.

Status legend: ☐ todo · ⚠️ blocked · ✅ done (for reference).

1. The headline measurement

The credibility claim is "third-party models on the same harness". A first real run has landed (regional 35B + SEA-LION v4.5-27B-IT across ms/vi/id/th — per-axis numbers in CHANGELOG.md); the full 7-axis freeze still needs:

  • ☐ Run a third frontier model (e.g. Gemma-family) for triangulation.
  • ☐ glossobench sigtest --lang ms --model-a <a> --model-b <b> → paired p-values.
  • ☐ glossobench elo --lang ms --models <a>,<b>,... → ranked table + CIs.
  • ☐ glossobench freeze --lang ms → immutable v1 leaderboard pin.
  • ☐ Publish MS / MS_norm + sigtest/elo tables (README "Status" → real numbers).
  • ✅ SEA-HELM v1.2.1 Cultural apples-to-apples path: langs/ms_seanli/config.yaml
  • sea-nli-ms (opt-in, HF-gated, MIT; gate-free via MB_SEANLI_DIR; bench_version 1.0-seanli, separate leaderboard). Use glossobench correlate --ours ours.json --external sea-helm.json for an independent Cultural convergence check.

Blocker: GPU time + weights download. No code work.

2. Backend runtime validation

SGLang and API-as-model backends ship with mock-wiring tests but have never run against a real engine/endpoint. Validate on first real use.

  • ⚠️ SGLang: pip install sglang on a GPU box, then --model-kind sglang. Confirm generate() + continuation-scoring cloze match an HF reference (reuse glossobench/tools/vllm_parity.py shape). Verify the prompt_token_logprobs container shape on a real engine; _sglang_lp handles float / [tok,lp] / dict.
  • ⚠️ API-as-model: point at a live OpenAI-compatible endpoint (MB_API_BASE, MB_API_KEY, MB_API_MODEL), --model-kind api. Some compatible servers lack /completions echo+logprobs — those should use a local backend for MCQ axes.
  • ⚠️ Docker in-container: podman build+run is green; verify docker (docker build -t glossobench -f Containerfile . && docker run --rm glossobench version).

3. CI — workflows written, runner registration pending

.github/workflows/{nightly,release}.yml target GitHub Actions + ghcr.io; .forgejo/workflows/{nightly,release}.yml are the Forgejo-native ports (secret-driven registry via FORGEJO_REGISTRY / FORGEJO_USER / FORGEJO_PUBLISH_TOKEN). Neither fires until a runner is registered (Forgejo) or the GitHub mirror is pushed (GitHub Actions fire as-is there).

  • ☐ Register a runner (or push the GitHub mirror) so nightly + release fire.
  • ☐ Set the publish-token secret (scoped package:write; env/secret only, never committed).
  • ☐ Cut the v1.0.0 tag → exercise the release pipeline end-to-end.

4. Dataset / content growth

  • ✅ NLG gate-free for Malay: FLORES devtest mirror bundled at langs/ms/bundled/flores/ (CC-BY-SA-4.0 + NOTICE), loader falls back to it before the gated HF path. See LICENSE_AUDIT.md §5 for the gate-bypass rationale + OLDI notification policy.
  • ☐ NLG for vi/id/th: bundle mirrors (same shape as ms) after the OLDI notification step, OR swap to ungated Tatoeba/OPUS parallel corpora. Until then NLG is gate-dropped for those languages (honest partial).
  • ☐ Canary GUIDs in bundled dataset rows — the --canary mechanism ships; embed a stable GUID per self-built dataset so a leak scan can grep raw data (see sop/growing-datasets.md).
  • ⚠️ Grow self-built seeds to full caps — the #1 statistical-power lever. Current: IF n=26, Cultural n=29, MT n=20 → MS CI ≈ ±0.15, no axis reaches significance. Full caps need native-authored content via MB_*_DIR (no fabricated native strings — standing rule), then glossobench.tools.dedup + freeze.
  • ⚠️ True embedding dedup — dedup --semantic-embedder PATH ships; point MB_DEDUP_EMBEDDER at a local Apache/MIT embedder (falls back to char n-gram TF-IDF proxy otherwise).
  • ⚠️ MetricX-24 — wired + offline-gated; never run with real weights (MB_METRICX_MODEL). NLG falls back to chrF++.
  • ⚠️ SEA-Guard judge — wired; weights not bundled (MB_SEAGUARD_MODEL). Safety axis skips cleanly without it.

5. Audit remediation (P0 validity floor)

The Tier-0 faults from CRITIC_AUDIT.md. Highest-leverage first:

  • ☐ A3 full-choice cloze (sum logprob over the whole choice, not first BPE token).
  • ☐ A2 dedup the public axes by default (MalayMMLU/Belebele/FLORES/ Global-MMLU); teach tools/dedup.py to read parquet/arrow (C18).
  • ☐ A7 make -O3 actually apply ensemble + dedup + freeze; ship ≥2 judges by default.
  • ☐ A4/A5/A6 item-level pooled MS + two-level bootstrap CI + ms=NaN on missing non-required axes.
  • ☐ A8/A12 real seeding + full provenance block in every summary.
  • ☐ A10/A11 MT metric rename/pairwise; Safety AUPRC direction fix.
  • ☐ C10 regenerate stale FREEZE.json SHAs from current plugin SHAs.

Reference: already done (do not redo)

  • 7-axis natural suite + 5-axis AI-developed suite, plugin registry, 3-axis versioning, register stratification, partial-completion, multi-judge ensemble, Elo (BT-MLE + bootstrap CIs), paired sigtest (permutation/McNemar/Holm/BH), Cohen's d + MDE + --strict-mde, Cohen's κ, CS/CA split, multi-seed, FormatSpread, semantic dedup (proxy + embedding), MetricX REF+QE, vLLM backend (GPU-validated), SGLang + API backends (mock-validated), canary mechanism, judge-bias audit, bench packages, intensity presets, thinking mode, container (podman-verified).
  • 11-lever fairness toolkit (probes, model-class segregation, self-judge family-exclusion, cross-bench correlate, contamination hard-EXCLUDE + SHA, spotcheck worksheet, per-item release, generate-parse parity, truncation reporting, calibration anchors, reasoning-mode parity gate).
  • Belebele NLU prompt-shape fix (labeled options inline + cloze on letters — pre-fix NLU numbers were at the chance floor and are invalid).
  • generate_batch NLG batching, bf16/sdpa defaults, chat-template policy (--chat-template always --no-think for fairest runs).
  • Full smoke suite green (100+ tests).

Definition of Done

  • Headline measurement published with sigtest + Elo tables, and v1 leaderboard frozen.
  • All Tier-0 (A*/C10) audit faults remediated; full provenance block present in every summary.
  • SGLang, API, and docker backends validated against real engines/endpoints.
  • CI runner registered, v1.0.0 tag cut, release pipeline publishes CPU + GPU images.

Related: CRITIC_AUDIT.md · COMPARISON.md · FAIREST.md · CHANGELOG.md