API reference¶

Benchmark phoneme discovery. Evaluate all languages and splits with available units.

The units should be saved in the directory path_units, in JSONL files named units-{code}-{split}.jsonl with keys file (str) and units (list[int]).

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset

• path_units (str | Path) –

  Path to the directory with the predicted units

• kind (Literal['many-to-one', 'one-to-one']) –

  Kind of assignment. If it is many-to-one, the number of units is set to the default (DEFAULT_N_UNITS). Otherwise, it is set to the number of phonemes plus one.

• step_units (int, default: STEP_UNITS ) –

  Step between consecutive units (in ms).

Returns:

• DataFrame –

  DataFrame with the results

Raises:

• NoPredictionsError –

  If there is nothing to evaluate.

ABX on all discrete units available.

The units should be saved in the directory path_units, in JSONL files named units-{code}-{split}.jsonl with keys file (str) and units (list[int]).

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset

• path_units (str | Path) –

  Path to the directory with the predicted units

• kind (Literal['triphone', 'phoneme'], default: 'triphone' ) –

  Kind of representations to use for ABX computation.

• step_units (int, default: STEP_UNITS ) –

  Step between consecutive units (in ms).

Returns:

• DataFrame –

  DataFrame with the results

Raises:

• NoPredictionsError –

  If there is nothing to evaluate.

ABX on all continuous features available.

The features should be saved in the directory path_features, in subfolders path_features/{code}/{split}.

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset

• path_features (str | Path) –

  Path to the directory with the extracted features

• kind (Literal['triphone', 'phoneme'], default: 'triphone' ) –

  Kind of representations to use for ABX computation.

• step_units (int, default: STEP_UNITS ) –

  Step between consecutive features (in ms). The feature frequency will be set to 1_000 / step_units

Returns:

• DataFrame –

  DataFrame with the results

Raises:

• NoPredictionsError –

  If there is nothing to evaluate.

Build the 2D cooccurrence matrix of shape (n_phonemes, n_units) as a DataArray.

Parameters:

• units (Units) –

  Predicted discrete units

• phones (Phones) –

  Gold phone annotations

• n_units (int) –

  Number of distinct discrete units in the evaluated system

• n_phonemes (int | None, default: None ) –

  Number of phonemes in the language under consideration. Either use this argument or language.

• step_units (int, default: STEP_UNITS ) –

  Step between consecutive units (in ms)

• step_phones (int, default: STEP_PHONES ) –

  Step between consecutive phones (in ms)

• language (str | Language | None, default: None ) –

  Evaluated language. Used to infer the number of phonemes if n_phonemes is not set. Do not set both at the same time.

Returns:

• DataArray –

  2D array for which the element (i, j) is the number of times the unit j has appeared where the underlying phoneme is i. The phonemes are sorted by frequency.

Compute the assigned sequences of phones from units, the cooccurrence matrix, and the kind of assignment.

Parameters:

• units (Units) –

  Predicted discrete units

• cooccurrence (DataArray) –

  Cooccurrence matrix between units and the underlying phones, computed with cooccurrence_matrix

• kind (Literal['many-to-one', 'one-to-one']) –

  Kind of assignment.

Returns:

• Phones –

  Assigned phones with this kind of mapping

Relabel clusters of units so that their labels follow the order of the phones they most likely represent.

Useful to plot units grouped in clusters (for instance, by a clustering of their embeddings) next to the phones: each cluster is matched to the phone with the highest mean probability over its units, and the clusters are then numbered in the order of the phones in proba, so that the plot is close to diagonal. The partition itself is unchanged: units that share a cluster still share one after relabeling.

Parameters:

• assignment (Iterable[int]) –

  Cluster of each unit, in the order of the unit coordinate of proba

• proba (DataArray) –

  Probability of each phone given each unit, with dimensions phone and unit, computed with probability_phone_given_unit

Returns:

• DataArray –

  New cluster of each unit, with dimension unit.

Full evaluation of phoneme discovery: PNMI, PER, F1 and R-value boundary detection.

Parameters:

• units (Units) –

  Predicted discrete units

• phones (Phones) –

  Gold phone annotations

• kind (Literal['many-to-one', 'one-to-one']) –

  Kind of assignment

• n_units (int) –

  Number of distinct discrete units in the evaluated system

• n_phonemes (int | None, default: None ) –

  Number of phonemes in the language under consideration. Either use this argument or language.

• step_units (int, default: STEP_UNITS ) –

  Step between consecutive units (in ms)

• step_phones (int, default: STEP_PHONES ) –

  Step between consecutive phones (in ms)

• language (str | Language | None, default: None ) –

  Evaluated language. Used to infer the number of phonemes if n_phonemes is not set. Do not set both at the same time.

Returns:

• PhonemeDiscoveryEvaluation –

  Phoneme discovery results in a dictionary with keys "pnmi", "per", "f1", and "r_val".

Compute PNMI.

The phone-normalized mutual information is the mutual information between phones and units divided by the phone entropy.

Both quantities are summed over their non-zero entries only, so the identity 0 * log(0) = 0 holds exactly and the result is guaranteed to lie in [0, 1]. Degenerate inputs have no information to normalize: if the matrix is empty or a single phone carries all the mass (the phone entropy is zero), PNMI is defined to be 0.0.

Parameters:

• cooccurrence (DataArray) –

  Cooccurrence matrix between units and the underlying phones, computed with cooccurrence_matrix

Returns:

• float –

  Phone-normalized mutual information (between 0 and 1)

Probability of each phone given each unit, ordered to plot the correspondence between phones and units.

The units that never occur are dropped. The units are grouped by their most probable phone, in the order of the phones in cooccurrence, and sorted by decreasing probability of that phone within each group, so that the plot is close to diagonal.

Parameters:

• cooccurrence (DataArray) –

  Cooccurrence matrix between units and phones, computed with cooccurrence_matrix

Returns:

• DataArray –

  P(phone|unit), with dimensions phone and unit. Each column sums to one.

Phone error rate.

Total edit distances divided by the total length of the target annotations.

Parameters:

• predicted_phones_from_units (Phones) –

  Predicted phones obtained with phone_assignments

• gold_phones (Phones) –

  Gold phone annotations

Returns:

• float –

  Phone error rate. Multiply it by 100 to get a percentage.

Raises:

• ValueError –

  If there are no files, or if some files have an empty predicted or gold sequence.

Phone segmentation evaluation.

Parameters:

• predicted_phones_from_units (Phones) –

  Predicted phones obtained with phone_assignments

• gold_phones (Phones) –

  Gold phone annotations

• margin_in_ms (int, default: 20 ) –

  Left and right margin around each gold boundaries (in ms). Predicted boundaries that fall in the resulting windows are considered correct. If two windows overlap, they are cut to the midpoint.

• step_units (int, default: STEP_UNITS ) –

  Step between consecutive units (in ms)

• step_phones (int, default: STEP_PHONES ) –

  Step between consecutive phones (in ms)

Returns:

• SegmentationEvaluation –

  Instance of a dataclass containing the segmentation results in attributes recall, precision, f1, os, and r_val. Use its describe method to get a summary of the segmentation evaluation.

ABX on discrete units.

Parameters:

• path_item (str | Path) –

  Path to the ABX item file

• path_units (str | Path) –

  Path to the predicted units: JSONL file with keys file (str) and units (list[int]).

• frequency (int | Decimal) –

  Feature frequency in Hz. It is the inverse of the step_units parameter used in other functions.

• kind (Literal['triphone', 'phoneme'], default: 'triphone' ) –

  Kind of representations to consider. If phoneme, we also compute the ABX in the "any" context condition, in addition to "within" context.

Returns:

• TriphoneABX | PhonemeABX –

  Dictionary of ABX discriminabilities with keys "within_speaker" and "across_speaker" if kind is "triphone", and with keys "within_speaker_within_context", "across_speaker_within_context", "within_speaker_any_context", and "across_speaker_any_context" if kind is "phoneme".

ABX on continuous representations.

Parameters:

• path_item (str | Path) –

  Path to the ABX item file

• path_features (str | Path) –

  Path to the extracted features: folder of .pt files with names corresponding to the file ids.

• frequency (int | Decimal) –

  Feature frequency in Hz. It is the inverse of the step_units parameter used in other functions.

• kind (Literal['triphone', 'phoneme'], default: 'triphone' ) –

  Kind of representations to consider. If phoneme, we also compute the ABX in the "any" context condition, in addition to "within" context.

Returns:

• TriphoneABX | PhonemeABX –

  Dictionary of ABX discriminabilities with keys "within_speaker" and "across_speaker" if kind is "triphone", and with keys "within_speaker_within_context", "across_speaker_within_context", "within_speaker_any_context", and "across_speaker_any_context" if kind is "phoneme".

Download and extract the DiscoPhon assets: manifests, alignments, items, and the audio we distribute.

Parameters:

• path_dataset (str | Path) –

  Target path to the DiscoPhon dataset.

Download the latest Common Voice release of language, and convert the clips needed by DiscoPhon to WAV.

The clips are resampled to 16 kHz directly from the archive, and written to path_dataset/audio/${code}/all. Requires the MDC_API_KEY environment variable, and to have accepted the release terms on Mozilla Data Collective.

This function can be resumed if interrupted: the download restarts where it stopped, and existing WAV files are skipped. The archive is kept in path_dataset/raw until all clips are processed.

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset. Must already contain the manifests.

• language (str) –

  Name, ISO 639-3 code, or Common Voice code of the language.

Raises:

• MissingClipsError –

  If some clips listed in the manifests are not in the release. The other clips are converted.

Check that the latest Common Voice release of language contains all the clips needed by DiscoPhon.

The release is streamed, and nothing is written to disk. Requires the MDC_API_KEY environment variable, and to have accepted the release terms on Mozilla Data Collective.

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset. Must already contain the manifests.

• language (str) –

  Name, ISO 639-3 code, or Common Voice code of the language.

Returns:

• DatasetDetails –

  The details of the checked release.

Raises:

• MissingClipsError –

  If some clips listed in the manifests are not in the release.

Filename for the predicted units of a (language, split) pair.

Filename for the gold phone alignment of a (language, split) pair.

Filename for the ABX item file of a (language, split, kind) triple.

Filename for the audio manifest of a (language, split) pair.

Read an RTTM file, with one column per field of the format.

Read a TextGrid file or directory of TextGrid files.

Write the given sequences of tokens as TextGrid files in the given output directory.

Write the intervals of a DataFrame as TextGrid files in outdir, one per file and in the tier tier_name.

If a TextGrid file already exists, the tier is added to it.

Write the speaker turns of an RTTM file as TextGrid files in outdir, in the tier tier_name.

Read the gold annotations and return a mapping between file names to the list of phonemes.

There will be one phone every step_in_ms ms.

Parameters:

• source (str | Path) –

  Path to the annotations file

• step_in_ms (int, default: STEP_PHONES ) –

  Step in ms between each phone.

Returns:

• Phones –

  Mapping between file ids and phones

Read the units from a JSONL file. Requires fields named file (str) and units (list[int]).

Other fields are ignored.

Parameters:

• source (str | Path) –

  Path to the units file

Returns:

• Units –

  Mapping between file ids and units

Raises:

• ValueError –

  If a file appears more than once, or if some units are missing.

Read the scores of the artifacts dataset.

Reads every {model}/{condition}/{folder}/{layer}/scores.jsonl in the dataset, or in one model directory if root holds an info.json. The condition is zero-shot or ft-{language}-{duration}, and the folder is many_to_one, one_to_one, continuous, or many_to_one-k{N}.

ABX scores within and across speakers are also averaged, under the name of the metric without the speaker condition. For example, triphone_abx_continuous averages triphone_abx_continuous_within_speaker and triphone_abx_continuous_across_speaker, and phoneme_abx_discrete_any_context averages phoneme_abx_discrete_within_speaker_any_context and phoneme_abx_discrete_across_speaker_any_context.

Parameters:

• root (str | Path) –

  Path to the artifacts dataset, or to one model directory in it

Returns:

• DataFrame –

  DataFrame with one row per score and columns model, folder, ft_lang (null for zero-shot), duration ("0" for zero-shot), layer, split, language, language_split (whether the language is a dev or test language), metric, and score, as stored (not in %).

Sonority class of each phoneme of the benchmark, for all languages.

Returns:

• dict[str, Sonority] –

  Mapping from each phoneme (in IPA) to its Sonority class.

LaTeX code of each phoneme of the benchmark with the tipa package, for all languages.

Returns:

• dict[str, str] –

  Mapping from each phoneme (in IPA) to its LaTeX code, such as \textipa{S} for ʃ.

Phoneme inventory of each language of the benchmark.

Returns:

• dict[str, list[str]] –

  Mapping from the ISO 639-3 code of each language to its sorted list of phonemes (in IPA).

Resolve a language identifier to its Language record.

The input is matched case-insensitively against any of the following identifiers:

• the English name (e.g. "German", "Mandarin Chinese");
• the ISO 639-3 code (e.g. "deu", "cmn");
• for languages available on Common Voice, the Common Voice locale code (e.g. "sw", "zh-CN").

A few languages also accept additional aliases: "mandarin" and "chinese" both resolve to Mandarin Chinese. Passing an existing Language instance returns it unchanged.

Parameters:

• n (str | Language) –

  Language identifier (name, ISO 639-3 code, Common Voice locale, alias) or an already-resolved Language instance.

Returns:

• Language –

  The matching Language record.

Raises:

• ValueError –

  If n does not match any known identifier.

Extract HuBERT continuous features for all utterances of a DiscoPhon split.

For each requested layer, the features are saved as PyTorch tensors at path_features / {layer} / {iso_639_3} / {split} / {fileid}.pt.

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset.

• path_features (str | Path) –

  Output directory under which per-layer feature tensors are written.

• language (str) –

  Language identifier resolved by get_language, either name or ISO 639-3 code.

• split (Literal['dev', 'test', 'train-10min', 'train-1h', 'train-10h']) –

  Dataset split to process.

• pretrained_model_name_or_path (str | Path) –

  HuBERT checkpoint or HuggingFace model identifier.

• layers (int | Iterable[int] | None, default: None ) –

  Layers to extract. If None, all encoder layers are used.

Files whose features already exist for all requested layers are skipped, so the extraction can be resumed.

Extract HuBERT discrete units for all utterances of a DiscoPhon split.

For each requested layer, the units are written to a JSONL file at path_units / {layer} / units-{iso_639_3}-{split}.jsonl, with one entry per utterance with keys file (str) and units (list[int]).

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset.

• path_units (str | Path) –

  Output directory under which the per-layer JSONL files are written.

• language (str) –

  Language identifier resolved by get_language, either name or ISO 639-3 code.

• split (Literal['dev', 'test', 'train-10min', 'train-1h', 'train-10h']) –

  Dataset split to process.

• pretrained_model_name_or_path (str | Path) –

  HuBERT checkpoint or HuggingFace model identifier.

• kmeans_by_layer (dict[int, MiniBatchKMeans]) –

  Mapping from 1-based layer index to the K-means model used to quantize that layer.

• layers (int | Iterable[int] | None, default: None ) –

  Layers to extract. If None, all the layers of kmeans_by_layer are used.

Raises:

• ValueError –

  If a layer is not in the model, or has no K-means in kmeans_by_layer.

Finetune HuBERT on DiscoPhon data with the default configuration.

Parameters:

• name (str) –

  Name of the run

• project (str) –

  Wandb project

• workdir (str | Path) –

  Path to workdir

• checkpoint (str | Path) –

  Path to pretrained checkpoint, or HuggingFace model identifier

• manifest (str) –

  Path to the manifest

• n_clusters (int) –

  Number of clusters

• target_layer (int) –

  Target layer

Compute the validation loss of every checkpoint of a HuBERT finetuning, and link the best one to best.pt.

The targets are the K-means units saved by finetune_hubert in checkpoints/kmeans.joblib, computed on the features of the pretrained model at target_layer. Every checkpoint is evaluated with the same masks and crops. The checkpoints already validated on the same manifest in output are skipped, and their losses are reused to choose the best one.

Parameters:

• output (str | Path) –

  JSONL file to which the losses of each checkpoint are appended

• checkpoints (str | Path) –

  Directory with the step_*.pt checkpoints and kmeans.joblib, where best.pt is created

• manifest (str | Path) –

  Manifest of the validation files

• pretrained (str | Path) –

  Path to the pretrained checkpoint the finetuning started from

• target_layer (int) –

  Layer of the pretrained model used to fit the K-means

• seed (int, default: 0 ) –

  Random seed of the masks and crops

Raises:

• ValueError –

  If there is no checkpoint in checkpoints.

Extract SpidR continuous features for all utterances of a DiscoPhon split.

For each requested layer, the features are saved as PyTorch tensors at path_features / {layer} / {iso_639_3} / {split} / {fileid}.pt.

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset.

• path_features (str | Path) –

  Output directory under which per-layer feature tensors are written.

• language (str) –

  Language identifier resolved by get_language, either name or ISO 639-3 code.

• split (Literal['dev', 'test', 'train-10min', 'train-1h', 'train-10h']) –

  Dataset split to process.

• checkpoint (str | Path) –

  Path to the SpidR checkpoint.

• layers (int | Iterable[int] | None, default: None ) –

  Layers to extract. If None, all student layers are used.

• batch_size (int, default: 1 ) –

  Number of utterances per forward pass. Padded batches give the same features up to numerical precision, and a batch size of 1 reproduces the published features exactly.

Files whose features already exist for all requested layers are skipped, so the extraction can be resumed.

Extract SpidR discrete units for all utterances of a DiscoPhon split.

The units are taken from the codebooks of the student network (one codebook per quantized layer). For each requested layer, the units are written to a JSONL file at path_units / {layer} / units-{iso_639_3}-{split}.jsonl, with one entry per utterance with keys file (str) and units (list[int]).

Parameters:

• path_dataset (str | Path) –

  Path to the DiscoPhon dataset.

• path_units (str | Path) –

  Output directory under which the per-layer JSONL files are written.

• language (str) –

  Language identifier resolved by get_language, either name or ISO 639-3 code.

• split (Literal['dev', 'test', 'train-10min', 'train-1h', 'train-10h']) –

  Dataset split to process.

• checkpoint (str | Path) –

  Path to the SpidR checkpoint.

• layers (int | Iterable[int] | None, default: None ) –

  Layers to extract. If None, all layers with a codebook are used.

• batch_size (int, default: 1 ) –

  Number of utterances per forward pass. Padded batches give the same units up to numerical precision, and a batch size of 1 reproduces the published units exactly.

Finetune SpidR on DiscoPhon data with the default configuration.

Parameters:

• name (str) –

  Run name

• project (str) –

  Run project

• workdir (str | Path) –

  Working directory for checkpoints and Wandb logs

• checkpoint (str | Path) –

  Path to the pretrained checkpoint

• manifest (str) –

  Path to the manifest

Compute the validation loss of every checkpoint of a SpidR finetuning, and link the best one to best.pt.

Every checkpoint is evaluated with the same masks and crops. The checkpoints already validated on the same manifest in output are skipped, and their losses are reused to choose the best one.

Parameters:

• output (str | Path) –

  JSONL file to which the losses of each checkpoint are appended

• checkpoints (str | Path) –

  Directory with the step_*.pt checkpoints, where best.pt is created

• manifest (str | Path) –

  Manifest of the validation files

• seed (int, default: 0 ) –

  Random seed of the masks and crops

Raises:

• ValueError –

  If there is no checkpoint in checkpoints.