opentau.policies.pi07.low_level.configuration_pi07_low_level

Configuration module for the π07 Low-Level Component.

This module defines the PI07LowLevelConfig class, which handles configuration parameters for the π07 low-level component. This component uses SpaceTimeSiglip (the Gemma 3 SigLIP vision tower wrapped with space-time separable attention) as a video encoder, processes temporal state sequences (one continuous token per timestep), and supports optional subtask response, subgoal image, and metadata conditioning.

Classes

PI07LowLevelConfig(n_obs_steps, ...)

Configuration for the π07 low-level component.
class opentau.policies.pi07.low_level.configuration_pi07_low_level.PI07LowLevelConfig(n_obs_steps: int = 6, normalization_mapping: dict[str, ~opentau.configs.types.NormalizationMode] = <factory>, input_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, output_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, device: str | None = None, use_amp: bool = False, use_torch_compile: bool = True, torch_compile_mode: str = 'default', pretrained_path: str | None = None, skip_normalization_weights: bool = False, skip_input_resolution_check: bool = False, save_normalization_stats: bool = True, dataset_names: list[str] | None = None, dataset_to_norm_index: dict[str, int] | None = None, cloud_vlm_latency_mean: float = 0.0, cloud_vlm_latency_std: float = 0.0, cloud_vlm_latency_lower: float = 0.0, cloud_vlm_latency_upper: float = 0.0, action_decoder_latency_mean: float = 0.0, action_decoder_latency_std: float = 0.0, action_decoder_latency_lower: float = 0.0, action_decoder_latency_upper: float = 0.0, chunk_size: int = 50, n_action_steps: int = 50, history_interval: int = 1, max_state_dim: int = 32, max_action_dim: int = 32, resize_imgs_with_padding: tuple[int, int] = (448, 448), empty_cameras: int = 0, prompt_max_length: int = 256, discrete_action_max_length: int = 32, discrete_action_tokenizer_path: str = 'physical-intelligence/fast', metadata_max_length: int = 52, response_max_length: int = 52, proj_width: int = 1280, per_group_projection: bool = False, dropout: float = 0.1, num_steps: int = 5, max_delay: int = 0, attention_implementation: str = 'eager', freeze_vision_encoder: bool = True, train_expert_only: bool = False, knowledge_insulation: bool = True, vlm_config: ~opentau.policies.pi07.gemma3_with_expert.Gemma3WithExpertConfig = <factory>, spacetime_layer_stride: int = 4, gradient_checkpointing: bool = False, optimizer_lr: float = 2.5e-05, optimizer_betas: tuple[float, float] = (0.9, 0.95), optimizer_eps: float = 1e-08, optimizer_weight_decay: float = 1e-10, scheduler_warmup_steps: int = 1000, scheduler_decay_steps: int = 30000, scheduler_decay_lr: float = 2.5e-06, warn_outlier_threshold: float = 32.0)[source]

Bases: PreTrainedConfig

Configuration for the π07 low-level component.

The low-level component generates continuous action chunks via flow matching and discrete FAST action tokens through the VLM backbone. It uses SpaceTimeSiglipVideoEncoder (the Gemma 3 SigLIP vision tower wrapped with space-time separable attention) as a video encoder and projects temporal state sequences into per-timestep continuous tokens.

Parameters:

  • n_obs_steps – Number of temporal video frames passed to the SpaceTimeSiglip encoder per forward call. Also the count of evenly-spaced historical frames the inference buffer keeps (the encoder sees the same number of frames at training and inference time). Must equal dataset_mixture.n_obs_history.

  • history_interval – Temporal stride between the n_obs_steps stacked frames in the inference buffer, in environment steps. Defaults to 1.

  • chunk_size – Trained action-chunk length, i.e. the prediction horizon the model always decodes at inference. Upper bound for n_action_steps. Defaults to 50.

  • n_action_steps – Inference execution horizon – how many actions from each predicted chunk are executed before the policy re-queries with fresh observations. Must be <= chunk_size. Defaults to 50.

  • normalization_mapping – Mapping of feature names to normalization modes.

  • max_state_dim – Maximum dimension for state vectors. Defaults to 32.

  • max_action_dim – Maximum dimension for action vectors. Defaults to 32.

  • resize_imgs_with_padding – Target (H, W) for video frame resizing with padding. Must match the Gemma 3 vision tower’s image_size (the projector hardcodes patches_per_image = image_size // patch_size). Defaults to (448, 448) to match vlm_config.gemma3_config.vision_config.image_size.

  • empty_cameras – Number of empty camera inputs to add. Defaults to 0.

  • prompt_max_length – Maximum token length for language prompts. Defaults to 256.

  • discrete_action_max_length – Maximum length for FAST action tokens. Defaults to 32.

  • discrete_action_tokenizer_path – HuggingFace repo id or local path of the FAST action tokenizer loaded into self.discrete_action_processor. Accepts anything AutoProcessor.from_pretrained(..., trust_remote_code=True) can resolve. Override this to use a tokenizer specialized to a different dataset mixture (see opentau.scripts.fit_fast_tokenizer); the value flows through to the policy’s auxiliary cross-entropy target at training time but is unused at inference (the flow-matching head produces continuous actions). Defaults to "physical-intelligence/fast".

  • metadata_max_length – Maximum token length for metadata strings. Defaults to 52.

  • response_max_length – Maximum token length for high-level planner subtask responses. Defaults to 52.

  • proj_width – Width of the action projection layer. Must match the Gemma-v1 action expert’s hidden_size so that suffix embeddings are compatible with the expert’s transformer layers. Defaults to 1280 (the expert hidden size in Gemma3WithExpertConfig).

  • per_group_projection – When True, use an independent state/action projection per (robot_type, control_mode) group (shares the normalization-head grouping). Defaults to False.

  • dropout – Dropout rate. Defaults to 0.1.

  • num_steps – Number of flow-matching denoising steps. Defaults to 5.

  • max_delay – Maximum number of prefix action steps for real-time inference. Defaults to 0.

  • attention_implementation – Attention backend — "eager", "sdpa", or "fa2". Defaults to "eager". "sdpa" dispatches to torch.nn.functional.scaled_dot_product_attention (typically ~2x faster on A100/H100 + bf16). "fa2" is accepted for backward compatibility but logs a warning and falls back to eager. Propagated to vlm_config in __post_init__ so a single --policy.attention_implementation override reaches the engine.

  • freeze_vision_encoder – Whether to freeze the SigLIP vision tower. Defaults to True.

  • train_expert_only – Whether to train only the action expert. Defaults to False.

  • vlm_config – Bundled Gemma3WithExpertConfig for the Gemma 3 VLM backbone + Gemma-v1 action expert.

  • spacetime_layer_stride – Wrap every Nth SigLIP encoder layer with space-time separable attention. Defaults to 4 (every 4th of the 27 SigLIP layers, indices [3, 7, 11, 15, 19, 23]), matching the MEM paper / pi05_mem (#171).

  • gradient_checkpointing – If True, wrap each SpaceTimeSiglip video encoder layer and each interleaved Gemma 3 + expert decoder layer body in torch.utils.checkpoint.checkpoint during training. Trades roughly one extra forward pass per step (~25-33% compute) for a large slice of activation memory per rank, enabling larger per-rank batch sizes. Only safe under plain DDP (MULTI_GPU), single-process (NO), or DeepSpeed ZeRO-1/2 — src/opentau/scripts/train.py raises if the accelerator’s distributed_type is anything else (ZeRO-3, FSDP). The engine half is plumbed via vlm_config.gradient_checkpointing in __post_init__; the video-encoder half is plumbed at model construction time. Defaults to False.

__init__(n_obs_steps: int = 6, normalization_mapping: dict[str, ~opentau.configs.types.NormalizationMode] = <factory>, input_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, output_features: dict[str, ~opentau.configs.types.PolicyFeature] = <factory>, device: str | None = None, use_amp: bool = False, use_torch_compile: bool = True, torch_compile_mode: str = 'default', pretrained_path: str | None = None, skip_normalization_weights: bool = False, skip_input_resolution_check: bool = False, save_normalization_stats: bool = True, dataset_names: list[str] | None = None, dataset_to_norm_index: dict[str, int] | None = None, cloud_vlm_latency_mean: float = 0.0, cloud_vlm_latency_std: float = 0.0, cloud_vlm_latency_lower: float = 0.0, cloud_vlm_latency_upper: float = 0.0, action_decoder_latency_mean: float = 0.0, action_decoder_latency_std: float = 0.0, action_decoder_latency_lower: float = 0.0, action_decoder_latency_upper: float = 0.0, chunk_size: int = 50, n_action_steps: int = 50, history_interval: int = 1, max_state_dim: int = 32, max_action_dim: int = 32, resize_imgs_with_padding: tuple[int, int] = (448, 448), empty_cameras: int = 0, prompt_max_length: int = 256, discrete_action_max_length: int = 32, discrete_action_tokenizer_path: str = 'physical-intelligence/fast', metadata_max_length: int = 52, response_max_length: int = 52, proj_width: int = 1280, per_group_projection: bool = False, dropout: float = 0.1, num_steps: int = 5, max_delay: int = 0, attention_implementation: str = 'eager', freeze_vision_encoder: bool = True, train_expert_only: bool = False, knowledge_insulation: bool = True, vlm_config: ~opentau.policies.pi07.gemma3_with_expert.Gemma3WithExpertConfig = <factory>, spacetime_layer_stride: int = 4, gradient_checkpointing: bool = False, optimizer_lr: float = 2.5e-05, optimizer_betas: tuple[float, float] = (0.9, 0.95), optimizer_eps: float = 1e-08, optimizer_weight_decay: float = 1e-10, scheduler_warmup_steps: int = 1000, scheduler_decay_steps: int = 30000, scheduler_decay_lr: float = 2.5e-06, warn_outlier_threshold: float = 32.0) → None
property action_delta_indices: list[int]

Get indices for action delta features.

Returns:

List of indices indicating which action features should be treated as deltas, or None if no delta features are used.

attention_implementation: str = 'eager'
chunk_size: int = 50
discrete_action_max_length: int = 32
discrete_action_tokenizer_path: str = 'physical-intelligence/fast'
dropout: float = 0.1
empty_cameras: int = 0
freeze_vision_encoder: bool = True
get_optimizer_preset() → AdamWConfig[source]

Returns the default optimizer configuration.

get_scheduler_preset() → LRSchedulerConfig[source]

Returns the default scheduler configuration.

gradient_checkpointing: bool = False
history_interval: int = 1
knowledge_insulation: bool = True
max_action_dim: int = 32
max_delay: int = 0
max_state_dim: int = 32
metadata_max_length: int = 52
n_action_steps: int = 50
n_obs_steps: int = 6
normalization_mapping: dict[str, NormalizationMode]
num_steps: int = 5
property obs_buffer_size: int

Total raw frames the observation buffer must keep.

With n_obs_steps=T and history_interval=k, the buffer stores the most recent (T-1)*k + 1 frames so that T evenly-spaced frames can be selected.

property observation_delta_indices: None

Get indices for observation delta features.

Returns:

List of indices indicating which observation features should be treated as deltas, or None if no delta features are used.

optimizer_betas: tuple[float, float] = (0.9, 0.95)
optimizer_eps: float = 1e-08
optimizer_lr: float = 2.5e-05
optimizer_weight_decay: float = 1e-10
per_group_projection: bool = False
proj_width: int = 1280
prompt_max_length: int = 256
resize_imgs_with_padding: tuple[int, int] = (448, 448)
response_max_length: int = 52
property reward_delta_indices: None

Get indices for reward delta features.

Returns:

List of indices indicating which reward features should be treated as deltas, or None if no delta features are used.

scheduler_decay_lr: float = 2.5e-06
scheduler_decay_steps: int = 30000
scheduler_warmup_steps: int = 1000
spacetime_layer_stride: int = 4
train_expert_only: bool = False
use_torch_compile: bool = True
validate_features() → None[source]

Validates the features and adds empty cameras if configured.

vlm_config: Gemma3WithExpertConfig
warn_outlier_threshold: float = 32.0