opentau.configs.refs

Resolve $ref includes inside JSON config files.

Lets a JSON config split itself across multiple files. Anywhere a JSON value is expected, an object of the form {"$ref": "path/to/other.json"} loads that file’s content and inlines it. Sibling keys deep-merge over the loaded content (sibling values win on conflict), enabling small, targeted overrides of a referenced fragment.

Example:

// main.json
{
    "dataset_mixture": {
        "$ref": "mixtures/big_mixture.json",
        "weights": [0.5, 0.5]
    }
}

Rules:

  • $ref value must be a string. Relative paths resolve against the file that contains the $ref (not CWD).

  • Refs may appear at any depth, in dicts or list elements.

  • Refs may chain (a referenced file may itself contain $ref keys); cycles raise RefError.

  • Sibling keys are only allowed when the referenced content is a JSON object.

  • Currently only whole-file references are supported (no JSON-pointer fragments). HuggingFace Hub paths are not resolved here — only local files.

  • Path resolution follows symlinks (Path.resolve()). HuggingFace cache snapshots are symlinks into a content-addressed blob directory, so a relative $ref inside an HF-cached config will resolve against that flat blob dir — not the snapshot dir — and is unlikely to find the target. Use absolute paths or copy the config out of the HF cache before adding $ref includes.

Functions

resolve_refs(config_path)

Load a JSON file and recursively resolve all $ref includes.

resolve_refs_to_tempfile(config_path)

Resolve $ref includes and write the result to a temp JSON file.

Exceptions

RefError

Raised when $ref resolution fails.
exception opentau.configs.refs.RefError[source]

Bases: Exception

Raised when $ref resolution fails.

opentau.configs.refs.resolve_refs(config_path: str | Path) → Any[source]

Load a JSON file and recursively resolve all $ref includes.

Parameters:

config_path – Path to the root JSON file.

Returns:

The fully resolved JSON tree (typically a dict, but may be a list or scalar depending on what the file and its includes contain).

Raises:

RefError – If a referenced file is missing, contains invalid JSON, forms a cycle, has a non-string $ref value, or has sibling keys alongside a $ref whose target is not a JSON object.

opentau.configs.refs.resolve_refs_to_tempfile(config_path: str | Path) → Path[source]

Resolve $ref includes and write the result to a temp JSON file.

The caller is responsible for unlinking the returned path.