opentau.datasets.vqa

Vision-language vqa datasets for multimodal learning.

This module provides datasets for training vision-language-action models on image-text vqa tasks without requiring robot actions. VQA datasets are designed to help models learn visual understanding, spatial reasoning, and language vqa capabilities that can be transferred to robotic tasks.

VQA datasets differ from standard robot learning datasets in that they:

• Provide images, prompts, and responses but no robot actions or states

• Use zero-padding for state and action features to maintain compatibility

• Focus on visual question answering, spatial reasoning, and object vqa

• Enable training on large-scale vision-language data without robot hardware

The module uses a registration system where datasets are registered via the @register_vqa_dataset decorator, making them available through the available_vqa_datasets registry.

Available Datasets:

• CLEVR: Compositional Language and Elementary Visual Reasoning dataset

  for visual question answering with synthetic scenes.

• COCO-QA: Visual question answering dataset based on COCO images,

  filtered for spatial reasoning tasks.

• VSR: Visual Spatial Reasoning dataset for true/false statement

  vqa about spatial relationships in images.

• dummy: Synthetic test dataset with simple black, white, and gray

  images for testing infrastructure.

Classes:

VQADataset: Base class for all vqa datasets, providing

common functionality for metadata creation, data format conversion, and zero-padding of state/action features.

Modules:

base: Base class and common functionality for vqa datasets. clevr: CLEVR dataset implementation. cocoqa: COCO-QA dataset implementation. dummy: Dummy test dataset implementation. vsr: VSR dataset implementation.

Example

Use a vqa dataset in training configuration:

>>> from opentau.configs.default import DatasetConfig
>>> cfg = DatasetConfig(vqa="cocoqa")
>>> dataset = make_dataset(cfg, train_cfg)

Access available vqa datasets:

>>> from opentau import available_vqa_datasets
>>> print(list(available_vqa_datasets.keys()))
['clevr', 'cocoqa', 'dummy', 'vsr']

Modules

base	Base class for vision-language vqa datasets.
clevr	CLEVR dataset for visual reasoning and vqa tasks.
cocoqa	COCO-QA dataset for visual question answering and vqa tasks.
dummy	Dummy vqa dataset for testing and development.
vsr	VSR (Visual Spatial Reasoning) dataset for true/false statement vqa.