Submission¶
Submitting a model to the leaderboard takes two pull requests:
- one on the artifacts dataset on the Hugging Face Hub with the scores and discrete units of the model, so that anyone can check and reuse them,
- one on GitHub with the leaderboard scores, exported from the first.
Both use the same model key: lowercase letters, digits, and single hyphens (e.g. my-model).
1. Evaluate your model¶
There are two tracks, many_to_one (256 units) and one_to_one (as many units as phonemes, plus one), each with
two conditions:
- Zero-shot: the model as is.
- Finetuned on 10h: one model per language, finetuned on the 10h training set of that language and evaluated on that language only.
Choose the layer to submit using the dev languages only. You can choose a different layer for each track and condition, as long as all the finetuned models of a track use the same layer.
Extract the units (see Evaluate) and run the benchmark in a directory named after your model,
with one subdirectory per run, {condition}/{track}/{layer}/:
my-model/
├── info.json
├── zero-shot/
│ ├── many_to_one/6/
│ │ ├── units-{language}-{split}.jsonl
│ │ └── scores.jsonl
│ ├── one_to_one/6/
│ │ ├── units-{language}-{split}.jsonl
│ │ └── scores.jsonl
│ └── continuous/6/
│ └── scores.jsonl
└── ft-{language}-10h/
└── ... # same as zero-shot
The continuous/ directory holds the continuous ABX scores, computed on features rather than units, which are
reported in the many-to-one track. The features of a layer are stored in {language}/{split}/ subdirectories
(see Evaluate). The benchmark appends its results to the output file, so all the evaluations
of a run go to the same scores.jsonl. Delete it before running them again, otherwise the scores are duplicated:
❯ d=my-model/zero-shot/many_to_one/6
❯ python -m discophon.benchmark data $d $d/scores.jsonl --kind many-to-one
❯ python -m discophon.benchmark data $d $d/scores.jsonl --benchmark abx-discrete
❯ python -m discophon.benchmark data features/6 my-model/zero-shot/continuous/6/scores.jsonl --benchmark abx-continuous
❯ d=my-model/zero-shot/one_to_one/6
❯ python -m discophon.benchmark data $d $d/scores.jsonl --kind one-to-one
Pass --step-units if your units are not 20 ms apart.
Finally, write info.json with the step between units in ms (recorded for the users of your units), and the
submitted layers, by track and condition ("0" for zero-shot, "10h" for finetuned on 10h):
{
"step_units": 20,
"layers": {
"many_to_one": {"0": 6, "10h": 6},
"one_to_one": {"0": 6, "10h": 8}
}
}
You can include more layers or conditions than the submitted ones: only those listed in info.json are used
by the leaderboard.
2. Upload the artifacts¶
Open a pull request on the artifacts dataset with the hf CLI,
leaving out the lock files written by the benchmark:
❯ hf upload coml/discophon-artifacts my-model/ my-model --repo-type dataset --exclude "*.lock" --create-pr
3. Export the leaderboard scores¶
Clone the GitHub repository. discophon.leaderboard export reads the
submitted layers from your model directory, and writes the leaderboard scores to
leaderboard/scores/{track}/{key}.jsonl in your clone, for each track in info.json:
4. Register your model¶
Add an entry to leaderboard/models.toml:
[my-model]
label = "My Model"
category = "submission"
url = "https://huggingface.co/org/my-model" # checkpoint or paper
description = "One or two sentences on the architecture and training data."
From the root of your clone, check that the leaderboard is valid and renders as expected:
Then open a pull request on GitHub with leaderboard/models.toml and leaderboard/scores/*/my-model.jsonl,
and link it to the pull request on the artifacts dataset. We merge them together.
The tests check the scores: one layer per track and condition, every language and metric, no duplicates.