opentau.policies.pi07.high_level_planner.configuration_pi07_high_level

Configuration module for the PI07 high level planner Policy.

This module defines the PI07HighLevelPlannerConfig class, which handles the configuration parameters for the PI07 high level planner. It includes settings for the model architecture, optimization, scheduling, and data processing.

Classes

PI07HighLevelPlannerConfig(n_obs_steps, ...)

Configuration for the π07 high-level planner policy.

class opentau.policies.pi07.high_level_planner.configuration_pi07_high_level.PI07HighLevelPlannerConfig(n_obs_steps: int = 1, normalization_mapping: dict[str, ~opentau.configs.types.NormalizationMode] = <factory>, input_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, output_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, device: str | None = None, use_amp: bool = False, use_torch_compile: bool = False, torch_compile_mode: str = 'default', pretrained_path: str | None = None, skip_normalization_weights: bool = False, skip_input_resolution_check: bool = False, save_normalization_stats: bool = True, dataset_names: list[str] | None = None, dataset_to_norm_index: dict[str, int] | None = None, cloud_vlm_latency_mean: float = 0.0, cloud_vlm_latency_std: float = 0.0, cloud_vlm_latency_lower: float = 0.0, cloud_vlm_latency_upper: float = 0.0, action_decoder_latency_mean: float = 0.0, action_decoder_latency_std: float = 0.0, action_decoder_latency_lower: float = 0.0, action_decoder_latency_upper: float = 0.0, max_state_dim: int = 32, resize_imgs_with_padding: tuple[int, int] = (448, 448), empty_cameras: int = 0, prompt_max_length: int = 256, memory_max_length: int = 52, response_max_length: int = 52, metadata_max_length: int = 52, subtask_indicator_max_length: int = 4, memory_indicator_max_length: int = 4, discrete_action_tokenizer_path: str = 'physical-intelligence/fast', dropout: float = 0.1, attention_implementation: str = 'eager', freeze_vision_encoder: bool = True, gradient_checkpointing: bool = False, vlm_config: ~opentau.policies.pi07.gemma3_with_expert.Gemma3WithExpertConfig = <factory>, optimizer_lr: float = 2.5e-05, optimizer_betas: tuple[float, float] = (0.9, 0.95), optimizer_eps: float = 1e-08, optimizer_weight_decay: float = 1e-10, scheduler_warmup_steps: int = 1000, scheduler_decay_steps: int = 30000, scheduler_decay_lr: float = 2.5e-06)

Bases: PreTrainedConfig

Configuration for the π07 high-level planner policy.

The high-level planner takes images, a language instruction, robot state, and past memory, then autoregressively predicts updated memory and the next subtask string. This config controls model architecture, tokenizer limits, initialization, and optimizer/scheduler presets.

Parameters:

• n_obs_steps – Number of observation steps to use. Only 1 is currently supported. Defaults to 1.

• normalization_mapping – Mapping from feature type names to normalization modes. Defaults to identity for visual features and mean-std for state.

• max_state_dim – Maximum dimension for state vectors. Shorter vectors are zero-padded. Defaults to 32.

• resize_imgs_with_padding – Target (height, width) for image resizing with aspect-ratio-preserving padding. Must match the Gemma 3 vision tower’s image_size (the projector hardcodes patches_per_image = image_size // patch_size). Defaults to (448, 448) to match vlm_config.gemma3_config.vision_config.image_size.

• empty_cameras – Number of empty (zero-filled) camera inputs to add. Defaults to 0.

• prompt_max_length – Maximum token length for the composite language prompt (task + past memory + state). Defaults to 256.

• memory_max_length – Maximum token length for the updated memory sequence. Defaults to 52.

• response_max_length – Maximum token length for the subtask response sequence. Defaults to 52.

• metadata_max_length – Maximum token length for episode metadata strings. Defaults to 52.

• subtask_indicator_max_length – Number of tokenizer pieces for the fixed "Subtask: " span (encode(..., add_special_tokens=False)). Used to align CE slices with the prefix layout. MUST equal len(tokenizer.encode("Subtask: ", add_special_tokens=False)) for whatever tokenizer the model uses; otherwise the memory CE slice is misaligned. Defaults to 4.

• memory_indicator_max_length – Number of tokenizer pieces for the fixed "Updated Memory: " span. Used for documentation and layout checks. MUST equal len(tokenizer.encode("Updated Memory: ", add_special_tokens=False)) for whatever tokenizer the model uses. Defaults to 4.

• dropout – Dropout rate applied in the transformer expert. Defaults to 0.1.

• attention_implementation – Attention backend — "eager", "sdpa", or "fa2". Defaults to "eager". "sdpa" dispatches to torch.nn.functional.scaled_dot_product_attention (typically ~2x faster on A100/H100 + bf16). "fa2" is accepted for backward compatibility but logs a warning and falls back to eager. The value is propagated to vlm_config in __post_init__ so a single --policy.attention_implementation override reaches the engine.

• freeze_vision_encoder – Whether to freeze the SigLIP vision encoder during fine-tuning. Defaults to True.

• gradient_checkpointing – Wrap each interleaved Gemma 3 + expert decoder layer body in torch.utils.checkpoint.checkpoint during training. Trades roughly one extra forward pass per step (~25-33% compute) for a large slice of activation memory per rank, enabling larger per-rank batch sizes. Only safe under plain DDP (MULTI_GPU), single-process (NO), or DeepSpeed ZeRO-1/2 — src/opentau/scripts/train.py raises if the accelerator’s distributed_type is anything else (ZeRO-3, FSDP). Propagated to vlm_config.gradient_checkpointing in __post_init__. Defaults to False.

• optimizer_lr – Peak learning rate for AdamW. Defaults to 2.5e-5.

• optimizer_betas – Beta parameters for AdamW. Defaults to (0.9, 0.95).

• optimizer_eps – Epsilon for AdamW. Defaults to 1e-8.

• optimizer_weight_decay – Weight decay for AdamW. Defaults to 1e-10.

• scheduler_warmup_steps – Linear warmup steps. Defaults to 1_000.

• scheduler_decay_steps – Cosine decay steps. Defaults to 30_000.

• scheduler_decay_lr – Final learning rate after decay. Defaults to 2.5e-6.

__init__(n_obs_steps: int = 1, normalization_mapping: dict[str, ~opentau.configs.types.NormalizationMode] = <factory>, input_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, output_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, device: str | None = None, use_amp: bool = False, use_torch_compile: bool = False, torch_compile_mode: str = 'default', pretrained_path: str | None = None, skip_normalization_weights: bool = False, skip_input_resolution_check: bool = False, save_normalization_stats: bool = True, dataset_names: list[str] | None = None, dataset_to_norm_index: dict[str, int] | None = None, cloud_vlm_latency_mean: float = 0.0, cloud_vlm_latency_std: float = 0.0, cloud_vlm_latency_lower: float = 0.0, cloud_vlm_latency_upper: float = 0.0, action_decoder_latency_mean: float = 0.0, action_decoder_latency_std: float = 0.0, action_decoder_latency_lower: float = 0.0, action_decoder_latency_upper: float = 0.0, max_state_dim: int = 32, resize_imgs_with_padding: tuple[int, int] = (448, 448), empty_cameras: int = 0, prompt_max_length: int = 256, memory_max_length: int = 52, response_max_length: int = 52, metadata_max_length: int = 52, subtask_indicator_max_length: int = 4, memory_indicator_max_length: int = 4, discrete_action_tokenizer_path: str = 'physical-intelligence/fast', dropout: float = 0.1, attention_implementation: str = 'eager', freeze_vision_encoder: bool = True, gradient_checkpointing: bool = False, vlm_config: ~opentau.policies.pi07.gemma3_with_expert.Gemma3WithExpertConfig = <factory>, optimizer_lr: float = 2.5e-05, optimizer_betas: tuple[float, float] = (0.9, 0.95), optimizer_eps: float = 1e-08, optimizer_weight_decay: float = 1e-10, scheduler_warmup_steps: int = 1000, scheduler_decay_steps: int = 30000, scheduler_decay_lr: float = 2.5e-06) → None

property action_delta_indices: None

Returns None; action deltas are not used by this planner.

attention_implementation: str = 'eager'

discrete_action_tokenizer_path: str = 'physical-intelligence/fast'

dropout: float = 0.1

empty_cameras: int = 0

freeze_vision_encoder: bool = True

get_optimizer_preset() → AdamWConfig

Returns the default AdamW optimizer configuration.

Returns:

An AdamWConfig populated from this config’s optimizer_* fields.

get_scheduler_preset() → LRSchedulerConfig

Returns the default cosine-decay-with-warmup scheduler configuration.

Returns:

A CosineDecayWithWarmupSchedulerConfig populated from this config’s scheduler_* and optimizer_lr fields.

gradient_checkpointing: bool = False

max_state_dim: int = 32

memory_indicator_max_length: int = 4

memory_max_length: int = 52

metadata_max_length: int = 52

n_obs_steps: int = 1

normalization_mapping: dict[str, NormalizationMode]

property observation_delta_indices: None

Returns None; observation deltas are not used by this planner.

optimizer_betas: tuple[float, float] = (0.9, 0.95)

optimizer_eps: float = 1e-08

optimizer_lr: float = 2.5e-05

optimizer_weight_decay: float = 1e-10

prompt_max_length: int = 256

resize_imgs_with_padding: tuple[int, int] = (448, 448)

response_max_length: int = 52

property reward_delta_indices: None

Returns None; reward deltas are not used by this planner.

scheduler_decay_lr: float = 2.5e-06

scheduler_decay_steps: int = 30000

scheduler_warmup_steps: int = 1000

subtask_indicator_max_length: int = 4

validate_features() → None

Adds placeholder camera features for empty camera slots.

Dynamically inserts zero-filled camera entries into self.input_features for each configured empty camera, so the model receives a fixed number of image inputs regardless of which cameras are physically present.

vlm_config: Gemma3WithExpertConfig