opentau.configs.default

Default configuration classes for datasets, evaluation, and logging.

This module provides default configuration classes for: - Dataset configuration and dataset mixtures - Weights & Biases (wandb) logging configuration - Evaluation settings and parameters

Classes

DatasetConfig(repo_id, vqa, root, episodes, ...)	Configuration for a dataset.
DatasetMixtureConfig(datasets, weights, ...)	Configuration for a mixture of multiple datasets.
EvalConfig([n_episodes, batch_size, ...])	Configuration for evaluation settings.
WandBConfig(enable, entity, project, run_id, ...)	Configuration for Weights & Biases (wandb) logging.

class opentau.configs.default.DatasetConfig(repo_id: str | None = None, vqa: str | None = None, root: str | None = None, episodes: list[int] | None = None, excluded_episodes: list[int] | None = None, image_transforms: ~opentau.datasets.transforms.ImageTransformsConfig = <factory>, revision: str | None = None, use_imagenet_stats: bool = True, video_backend: str = <factory>, stats: dict[str, dict[str, ~numpy.ndarray]] | None = None, data_features_name_mapping: dict[str, str] | None = None, prompt_substitutions: dict[str, list[str]] | None = None, robot_type: str | None = None, control_mode: str | None = None, tolerance_s: float | None = None, skip_timestamp_check: bool | None = None, val_split_ratio: float | None = None)

Bases: object

Configuration for a dataset.

You may provide a list of datasets here. train.py creates them all and concatenates them. Note: only data keys common between the datasets are kept. Each dataset gets an additional transform that inserts the “dataset_index” into the returned item. The index mapping is made according to the order in which the datasets are provided.

Parameters:

• repo_id – HuggingFace repository ID for the dataset. Exactly one of repo_id or vqa must be set.

• vqa – VQA dataset identifier. Exactly one of repo_id or vqa must be set.

• root – Root directory where the dataset will be stored (e.g. ‘dataset/path’). Defaults to None.

• episodes – List of episode indices to use from the dataset. If None, all episodes are used. Defaults to None.

• excluded_episodes – List of episode indices to drop from this dataset. Takes precedence over episodes: an index present in both is excluded. If None (default), no episodes are excluded. Note: on legacy v2.0 datasets (no per-episode stats) the listed episodes are dropped from training, but normalization stats stay the global (all-episode) aggregate — only v2.1+ can recompute them.

• image_transforms – Configuration for image transformations. Defaults to ImageTransformsConfig().

• revision – Git revision of the dataset repository to use. Defaults to None.

• use_imagenet_stats – Whether to use ImageNet statistics for normalization. Defaults to True.

• video_backend – Video codec backend to use. Defaults to a safe default codec.

• stats – Dictionary of statistics for normalization, keyed by feature name. Each value is a dictionary with ‘mean’ and ‘std’ arrays. Defaults to None.

• data_features_name_mapping – Optional mapping from standard feature names (camera0/camera1/…, state, actions, prompt, response, mistake, success) to this dataset’s own column names. The mistake and success roles feed the optional mistake metadata key: map a mistake-polarity column (True/1 = something went wrong) to mistake, or a success-polarity column (e.g. DROID’s is_episode_successful) to success — polarity is expressed by which role you map, so no inversion flag is needed. When both resolve, mistake wins (it is segment-grained). mistake must name a per-frame column; for an episode-level outcome map success instead, which resolves from a per-frame column or a per-episode key in the episodes metadata and also drives the value-function return bins. Two mixture entries may share a repo_id and control_mode while declaring different mappings (e.g. two camera views of one repo): each dataset instance resolves its own entry’s mapping. The global registry keeps only the last registration for annotation-script consumers — a warning fires when entries disagree. Defaults to None.

• robot_type – Optional override for the dataset’s robot_type metadata field. When provided (including the empty string), takes precedence over the value loaded from meta/info.json. None (default) leaves the loaded value untouched.

• control_mode – Optional override for the dataset’s control_mode metadata field. When provided (including the empty string), takes precedence over the value loaded from meta/info.json. None (default) leaves the loaded value untouched.

• tolerance_s – Optional per-dataset override for the timestamp-sync tolerance (in seconds) passed to LeRobotDataset’s load-time check_timestamps_sync call. None (default) inherits the mixture-wide DatasetMixtureConfig.tolerance_s value. Set this to a larger value (e.g. 1e-3) when a single dataset in the mixture has slightly off-fps timestamps but you don’t want to loosen the check for the others. Must be >= 0 when set. This also applies to frames decoded from video files: the same value is used as the per-frame match tolerance in query_video_frames_*, so loosening it widens the video-frame match window as well.

• skip_timestamp_check – Optional per-dataset override that bypasses the load-time check_timestamps_sync call entirely. None (default) inherits the mixture-wide DatasetMixtureConfig.skip_timestamp_check. True skips the check (a warning is logged); False forces the check on for this dataset even if the mixture default is True. Does not affect the record-time check inside add_episode.

• prompt_substitutions – Optional mapping from an on-disk task string (exact match against meta/tasks.*) to a non-empty list of non-empty substitute prompts. At fetch time a matching sample’s task is ALWAYS replaced by a uniform random draw from its list (include the original in the list if it should still appear); unmapped tasks pass through unchanged. Applies to the train split only unless DatasetMixtureConfig.val_enable_prompt_substitution is set. Keys that match no on-disk task string raise at dataset init. response/memory CE targets are NOT rewritten, so substitutes must be semantic paraphrases of the original task. Only settable for LeRobot datasets (repo_id), not VQA entries. Like every per-dataset field it has no CLI override path — set it in the JSON config; an external fragment can be inlined with {"$ref": "path/to/fragment.json"} (see opentau/configs/refs.py). Draws use the default torch RNG (see the DatasetMixtureConfig note). Defaults to None.

Raises:

ValueError – If both or neither of repo_id and vqa are set, if tolerance_s or val_split_ratio is out of range, or if prompt_substitutions is set on a VQA entry, maps a task literally named "$ref", has a non-string key, or contains an empty substitute list or empty/non-string entries.

__init__(repo_id: str | None = None, vqa: str | None = None, root: str | None = None, episodes: list[int] | None = None, excluded_episodes: list[int] | None = None, image_transforms: ~opentau.datasets.transforms.ImageTransformsConfig = <factory>, revision: str | None = None, use_imagenet_stats: bool = True, video_backend: str = <factory>, stats: dict[str, dict[str, ~numpy.ndarray]] | None = None, data_features_name_mapping: dict[str, str] | None = None, prompt_substitutions: dict[str, list[str]] | None = None, robot_type: str | None = None, control_mode: str | None = None, tolerance_s: float | None = None, skip_timestamp_check: bool | None = None, val_split_ratio: float | None = None) → None

control_mode: str | None = None

data_features_name_mapping: dict[str, str] | None = None

episodes: list[int] | None = None

excluded_episodes: list[int] | None = None

image_transforms: ImageTransformsConfig

prompt_substitutions: dict[str, list[str]] | None = None

repo_id: str | None = None

revision: str | None = None

robot_type: str | None = None

root: str | None = None

skip_timestamp_check: bool | None = None

stats: dict[str, dict[str, ndarray]] | None = None

tolerance_s: float | None = None

use_imagenet_stats: bool = True

val_split_ratio: float | None = None

video_backend: str

vqa: str | None = None

class opentau.configs.default.DatasetMixtureConfig(datasets: list[~opentau.configs.default.DatasetConfig] = <factory>, weights: list[float] | None = None, action_freq: float | None = None, image_resample_strategy: str = 'nearest', vector_resample_strategy: str = 'nearest', val_split_ratio: float = 0.05, n_obs_history: int | None = None, history_state_drop_prob: float = 0.3, subgoal_drop_prob: float = 0.75, subgoal_end_of_segment_prob: float = 0.25, response_drop_prob: float = 0.3, metadata_drop_all_prob: float = 0.15, metadata_drop_each_prob: float = 0.05, val_enable_optional_key_dropout: bool = False, val_enable_prompt_substitution: bool = False, require_non_empty_robot_type: bool = False, require_non_empty_control_mode: bool = False, emit_fps: bool = False, tolerance_s: float = 0.0001, skip_timestamp_check: bool = False)

Bases: object

Configuration for a mixture of multiple datasets.

This configuration allows combining multiple datasets with specified weights for training. The datasets are sampled according to their weights during training, and features are resampled to a common action frequency.

Parameters:

• datasets – List of dataset configs to be used in the mixture.

• weights – Optional list of weights for each dataset in the mixture. Must be the same length as datasets when provided. If None, weights are inferred from dataset lengths. Defaults to None.

• action_freq – Frequency at which actions from the dataset mixture are resampled, in Hz. None (default) disables resampling — each dataset is sampled at its native fps, so a single batch can mix samples from sources running at different rates (predicting chunk_size consecutive native frames per sample). Set a positive float to resample every dataset in the mixture to that common rate via nearest-neighbor frame selection. When using None, prefer also setting emit_fps=True so the policy can condition on the per-sample rate.

• image_resample_strategy – Resample strategy for image features. Must be one of ‘linear’ or ‘nearest’. Defaults to ‘nearest’.

• vector_resample_strategy – Resample strategy for non-image features, such as action or state. Must be one of ‘linear’ or ‘nearest’. Defaults to ‘nearest’.

• val_split_ratio – Mixture-wide default fraction of each dataset reserved for the validation split (only used when TrainPipelineConfig.val_freq > 0). A per-dataset DatasetConfig.val_split_ratio overrides this value for that dataset; None there inherits this mixture default. Must be in [0, 1]. Defaults to 0.05.

• n_obs_history – Number of historical observation steps to include. When set to T, each camera returns shape (T, C, H, W) and state returns shape (T, max_state_dim). When None, the default single-step behavior is preserved with rank-3 camera tensors (C, H, W) and rank-1 state tensors (max_state_dim,). Note that n_obs_history=1 produces rank-4 camera tensors (1, C, H, W) with a leading singleton dimension, while state collapses to rank-1 (max_state_dim,) (the length-1 delta-timestamps query is squeezed like the None case), so downstream consumers must rank-normalize state themselves. Defaults to None. The temporal stride between sampled observations is read from the policy config’s history_interval attribute (defaults to 1 when the policy doesn’t define one), so observations are sampled at timesteps \(t - (T-1)k,\; t - (T-2)k,\; \ldots,\; t\).

• history_state_drop_prob – Probability of dropping the observation history during a single __getitem__ call. When it fires, the historical steps are masked via obs_history_is_pad (set all True) and the historical camera frames are zeroed; the current step — current observation.state and current camera frame — is kept. state is deliberately NOT zeroed here: it is MEAN_STD-normalized downstream, so the dropped history is zeroed after normalization inside the policy (zeroing a raw state pre-normalization would map 0 to -mean/std, an out-of-distribution extreme). Must be in [0, 1]. Defaults to 0.3.

• subgoal_drop_prob – Probability of dropping all subgoal images during a single __getitem__ call. Must be in [0, 1]. Defaults to 0.75.

• subgoal_end_of_segment_prob – Probability of sampling the subgoal frame at the end of the current segment (vs. uniformly in the next 4s of wall-clock time). Must be in [0, 1]. Defaults to 0.25.

• response_drop_prob – Probability of dropping the response (subtask text) during a single __getitem__ call. Only rolled when subgoals are not dropped. Must be in [0, 1]. Defaults to 0.3.

• metadata_drop_all_prob – Probability of dropping speed, mistake, quality, robot_type, and control_mode together during a single __getitem__ call. Must be in [0, 1]. Defaults to 0.15.

• metadata_drop_each_prob – Per-field independent drop probability for speed, mistake, quality, robot_type, and control_mode. Only rolled when metadata_drop_all_prob did not fire. Must be in [0, 1]. Defaults to 0.05.

• val_enable_optional_key_dropout – Whether to apply the five *_drop_prob rolls above to the validation split. Defaults to False — validation evaluates on un-masked samples so metrics aren’t polluted by training-time augmentation. Subgoal frame sampling (end-of-segment vs. uniform in the next 4s) stays active either way; only the masking logic is gated.

• val_enable_prompt_substitution – Whether the per-dataset DatasetConfig.prompt_substitutions swaps also fire on the validation split. Defaults to False — validation evaluates on the on-disk prompts. Flip to True when val loss should match the training prompt distribution (with always-replace semantics the original prompt never appears in training unless listed among its own substitutes).

• require_non_empty_robot_type – If True, every dataset in the mixture must have a non-empty robot_type after the optional DatasetConfig.robot_type override has been applied. Defaults to False (empty / missing values are allowed).

• require_non_empty_control_mode – If True, every dataset in the mixture must have a non-empty control_mode after the optional DatasetConfig.control_mode override has been applied. Defaults to False (empty / missing values are allowed).

• emit_fps – Whether __getitem__ returns the effective per-sample frame rate (action_freq if set, else the dataset’s native meta.fps) as the fps metadata key (torch.long scalar, paired with fps_is_pad=False). Default False — fps conditioning is an opt-in feature so pre-PR checkpoints resume without the policy’s metadata prefix gaining an unfamiliar FPS: segment. Flip to True for new training runs that want per-sample frame-rate conditioning (especially heterogeneous-frequency mixtures where action_freq=None lets each dataset run at its native rate). Unlike the other metadata fields, fps is not rolled by metadata_drop_*_prob — it’s an intrinsic property of the chunk, not a noisy label, so it is always present (non-pad) for LeRobot samples when emit_fps=True. VQA samples (no temporal axis) emit fps=0, fps_is_pad=True regardless so heterogeneous VLA + VQA batches stay schema-aligned.

• tolerance_s – Mixture-wide default tolerance (in seconds) for the load-time check_timestamps_sync call inside LeRobotDataset.__init__. Each dataset’s frame-to-frame timestamp spacing must lie within 1/fps +/- tolerance_s or the check raises. Defaults to 1e-4. A per-dataset DatasetConfig.tolerance_s overrides this value when set. Must be >= 0. This also applies to frames decoded from video files: the same value is used as the per-frame match tolerance in query_video_frames_*, so loosening it widens the video-frame match window as well.

• skip_timestamp_check – If True, bypass the load-time check_timestamps_sync call for every dataset in the mixture (a warning is logged per dataset). Useful as a debug knob when the timing data is known-bad but you still want the mixture to load. Defaults to False. A per-dataset DatasetConfig.skip_timestamp_check overrides this value when set. Does not affect the record-time check inside add_episode.

Note

Dropout rolls use the default torch RNG. PyTorch DataLoader workers auto-seed each process’s torch RNG from the base seed + worker id, so workers sample independently. For reproducibility the caller should seed via torch.manual_seed(...) in the main process before constructing the DataLoader.

Raises:

ValueError – If weights is provided and its length doesn’t match datasets, if action_freq is not None and not positive, if resample strategies are invalid, or if any drop probability is outside [0, 1].

__init__(datasets: list[~opentau.configs.default.DatasetConfig] = <factory>, weights: list[float] | None = None, action_freq: float | None = None, image_resample_strategy: str = 'nearest', vector_resample_strategy: str = 'nearest', val_split_ratio: float = 0.05, n_obs_history: int | None = None, history_state_drop_prob: float = 0.3, subgoal_drop_prob: float = 0.75, subgoal_end_of_segment_prob: float = 0.25, response_drop_prob: float = 0.3, metadata_drop_all_prob: float = 0.15, metadata_drop_each_prob: float = 0.05, val_enable_optional_key_dropout: bool = False, val_enable_prompt_substitution: bool = False, require_non_empty_robot_type: bool = False, require_non_empty_control_mode: bool = False, emit_fps: bool = False, tolerance_s: float = 0.0001, skip_timestamp_check: bool = False) → None

action_freq: float | None = None

datasets: list[DatasetConfig]

emit_fps: bool = False

history_state_drop_prob: float = 0.3

image_resample_strategy: str = 'nearest'

metadata_drop_all_prob: float = 0.15

metadata_drop_each_prob: float = 0.05

n_obs_history: int | None = None

require_non_empty_control_mode: bool = False

require_non_empty_robot_type: bool = False

response_drop_prob: float = 0.3

skip_timestamp_check: bool = False

subgoal_drop_prob: float = 0.75

subgoal_end_of_segment_prob: float = 0.25

tolerance_s: float = 0.0001

val_enable_optional_key_dropout: bool = False

val_enable_prompt_substitution: bool = False

val_split_ratio: float = 0.05

vector_resample_strategy: str = 'nearest'

weights: list[float] | None = None

class opentau.configs.default.EvalConfig(n_episodes: int = 16, batch_size: int = 16, use_async_envs: bool = True, max_episodes_rendered: int = 16, grid_size: tuple[int, int] | None = None, video_crf: int = 30, video_preset: str = 'veryfast', video_frame_stride: int = 2, keep_per_episode_videos: bool = False, recording_root: str | None = None, seed: int | None = None, decorrelate_rank_seeds: bool = False, goal_frames_dir: str | None = None, seed_list: str | None = None, dataset_repo_id: str | None = None, robot_type: str | None = None, control_mode: str | None = None)

Bases: object

Configuration for evaluation settings.

Parameters:

• n_episodes – Number of episodes to run during evaluation. Defaults to 16.

• batch_size – Number of environments to use in a gym.vector.VectorEnv. Only used for environments that are not already vectorized. Defaults to 16.

• use_async_envs – Whether to use asynchronous environments (multiprocessing). Defaults to True. RoboCasa eval requires the async backend for any multi-env (batch_size > 1) build: its per-env EGL/GL offscreen render contexts cross-contaminate in a single process, so SyncVectorEnv feeds the policy the wrong env’s pixels (issue #449). A multi-env RoboCasa Sync build is therefore auto-promoted to async; LIBERO is unaffected.

• max_episodes_rendered – Maximum number of episodes to render as videos. Defaults to 16.

• grid_size – Grid dimensions for video summary (rows, cols). If None, will be auto-calculated as a square grid. Defaults to None.

• video_crf – H.264 constant-rate-factor for the uploaded grid-summary video (higher = smaller file / lower quality, 0-51). Defaults to 30.

• video_preset – x264 encode preset (ultrafast..veryslow); an encode-speed vs compression-ratio knob that does not change the quality target. Defaults to “veryfast”.

• video_frame_stride – Keep only every k-th frame of the grid-summary video (k>1 shrinks the upload ~linearly and speeds playback up k x). Defaults to 2.

• keep_per_episode_videos – If False, delete the per-episode eval_episode_*.mp4 clips after the grid summary is built (they are never uploaded to wandb). Defaults to False.

• recording_root – Root directory for saving evaluation recordings. Defaults to None.

• seed – Master seed for the eval simulations (env scene generation). When set, takes precedence over the top-level cfg.seed for seeding the eval environments; when None (default), falls back to cfg.seed. Does not affect the global set_seed. Defaults to None.

• decorrelate_rank_seeds – If True, each accelerator rank evaluates a distinct, orthogonal slice of scenes (for tasks deliberately replicated across ranks to gain coverage). If False (default), all ranks seed identically, so the eval is reproducible across world sizes. Defaults to False.

Raises:

ValueError – If batch_size is greater than n_episodes, or if any of video_crf, video_preset, video_frame_stride is out of range.

__init__(n_episodes: int = 16, batch_size: int = 16, use_async_envs: bool = True, max_episodes_rendered: int = 16, grid_size: tuple[int, int] | None = None, video_crf: int = 30, video_preset: str = 'veryfast', video_frame_stride: int = 2, keep_per_episode_videos: bool = False, recording_root: str | None = None, seed: int | None = None, decorrelate_rank_seeds: bool = False, goal_frames_dir: str | None = None, seed_list: str | None = None, dataset_repo_id: str | None = None, robot_type: str | None = None, control_mode: str | None = None) → None

batch_size: int = 16

control_mode: str | None = None

dataset_repo_id: str | None = None

decorrelate_rank_seeds: bool = False

goal_frames_dir: str | None = None

grid_size: tuple[int, int] | None = None

keep_per_episode_videos: bool = False

max_episodes_rendered: int = 16

n_episodes: int = 16

recording_root: str | None = None

robot_type: str | None = None

seed: int | None = None

seed_list: str | None = None

use_async_envs: bool = True

video_crf: int = 30

video_frame_stride: int = 2

video_preset: str = 'veryfast'

class opentau.configs.default.WandBConfig(enable: bool = False, entity: str | None = None, project: str = 'opentau', run_id: str | None = None, name: str | None = None, notes: str | None = None, tags: list[str] = <factory>, group: str | None = None, job_type: str | None = None, mode: str | None = None, on_resume: ~typing.Literal['fork', 'continue'] = 'fork', allow_resume: bool | None = None, disable_artifact: bool = False, disable_video: bool = False)

Bases: object

Configuration for Weights & Biases (wandb) logging.

Parameters:

• enable – Enable Weights & Biases logging. Defaults to False.

• entity – The entity name in Weights & Biases, e.g. your username or your team name. Defaults to None.

• project – The project name in Weights & Biases, e.g. “pi0”. Defaults to “opentau”.

• run_id – If provided, the run will be forked from this run ID. Defaults to None.

• name – Name of the run, shown in the UI. Defaults to None.

• notes – Description of the run, shown in the UI. If None and enable is True, will prompt the user for input. Defaults to None.

• tags – Tags to be added to the run in the UI, e.g. [“robot”, “v1.0”]. Defaults to empty list.

• group – Used to group runs in the UI, e.g. “experiment_1”, “experiment_2”. Defaults to None.

• job_type – Used to group runs in the UI, e.g. “train”, “eval”, “test”. Defaults to None.

• mode – Allowed values: ‘online’, ‘offline’, ‘disabled’. Defaults to None (which uses ‘online’).

• on_resume – What to do when resuming a run that has a run_id and a training step (i.e. resuming from a checkpoint). Allowed values: ‘fork’ (default) creates a NEW run that branches from the parent at the resume step via wandb’s fork_from; ‘continue’ resumes the same run in place (resume=’allow’). Forking keeps the parent run immutable and records server-side lineage; the cost is that a job preempted/requeued N times produces a chain of N linked runs.

• allow_resume – DEPRECATED, use on_resume instead. If set, it is mapped to on_resume for backward compatibility (True -> ‘continue’, False -> ‘fork’) and a FutureWarning is emitted. Defaults to None.

• disable_artifact – Set to True to disable saving an artifact despite training.save_checkpoint=True. Defaults to False.

• disable_video – Set to True to skip logging eval grid-summary videos to wandb. Defaults to False (videos are logged).

__init__(enable: bool = False, entity: str | None = None, project: str = 'opentau', run_id: str | None = None, name: str | None = None, notes: str | None = None, tags: list[str] = <factory>, group: str | None = None, job_type: str | None = None, mode: str | None = None, on_resume: ~typing.Literal['fork', 'continue'] = 'fork', allow_resume: bool | None = None, disable_artifact: bool = False, disable_video: bool = False) → None

allow_resume: bool | None = None

disable_artifact: bool = False

disable_video: bool = False

enable: bool = False

entity: str | None = None

group: str | None = None

job_type: str | None = None

mode: str | None = None

name: str | None = None

notes: str | None = None

on_resume: Literal['fork', 'continue'] = 'fork'

project: str = 'opentau'

run_id: str | None = None

tags: list[str]

to_wandb_kwargs(step=None)

Convert configuration to keyword arguments for wandb.init().

Parameters:

step – Optional training step number. If provided along with run_id, used for resuming or forking runs. Defaults to None.

Returns:

Dictionary of keyword arguments suitable for passing to wandb.init().