opentau.datasets.vqa.base

Base class for vision-language vqa datasets.

This module provides the base class for all vqa datasets, which are used for training vision-language-action models on image-text tasks without robot actions. VQA datasets provide images, prompts, and responses for tasks like visual question answering, spatial reasoning, and object vqa.

The base class handles common functionality including:

• Metadata creation with ImageNet statistics for images

• Zero-padding of state and action features for compatibility

• Standard data format conversion

• Integration with the dataset mixture system

Classes:

VQADataset: Abstract base class that all vqa datasets inherit

from. Provides common functionality for metadata creation, data format conversion, and zero-padding of missing features.

Example

Create a custom vqa dataset:

>>> from opentau import register_vqa_dataset
>>> @register_vqa_dataset("my_dataset")
>>> class MyVQADataset(VQADataset):
...     def __getitem_helper__(self, item):
...         return {"image": ..., "task": ..., "postfix": ...}

Classes

VQADataset(cfg[, num_frames, num_episodes])

Base class for vision-language vqa datasets.

class opentau.datasets.vqa.base.VQADataset(cfg: TrainPipelineConfig, num_frames: int = 1, num_episodes: int = 1)

Bases: BaseDataset

Base class for vision-language vqa datasets.

VQA datasets are used for training vision-language-action models on image-text tasks without robot actions. They provide images, prompts, and responses for vqa tasks.

num_frames

Number of frames in the dataset.

num_episodes

Number of episodes (always 1 for vqa datasets).

meta

Dataset metadata containing features and statistics.

__init__(cfg: TrainPipelineConfig, num_frames: int = 1, num_episodes: int = 1)

create_meta() → VQADatasetMetadata

Create metadata for the vqa dataset.

Initializes metadata with ImageNet statistics for images and zero statistics for state and actions (since vqa datasets don’t have them).

Returns:

VQADatasetMetadata object with initialized info and stats.