opentau.datasets.vqa.cocoqa

COCO-QA dataset for visual question answering and vqa tasks.

This module provides the COCO-QA dataset implementation for training vision-language models on visual question answering tasks. The dataset is filtered to only include ‘where’ questions, focusing on spatial reasoning tasks that are relevant for robotic manipulation.

The dataset is loaded from HuggingFace (ThucPD/coco-qa-vi) and automatically filtered to retain only spatial reasoning questions.

Classes:

COCODataset: Dataset class that loads, filters, and formats COCO-QA data

for vqa tasks.

Functions:

_img_to_normalized_tensor: Convert PIL Image to normalized torch tensor

with channel-first format and [0, 1] normalization.

_filter_dataset: Filter dataset samples to only include ‘where’ questions

for spatial reasoning tasks.

Example

Use COCO-QA dataset in training:

>>> from opentau.configs.default import DatasetConfig
>>> cfg = DatasetConfig(vqa="cocoqa")
>>> dataset = make_dataset(cfg, train_cfg)

Classes

COCODataset(cfg)

COCO-QA dataset for visual question answering and vqa tasks.

class opentau.datasets.vqa.cocoqa.COCODataset(cfg: TrainPipelineConfig)

Bases: VQADataset

COCO-QA dataset for visual question answering and vqa tasks.

Loads the ThucPD/coco-qa-vi dataset from HuggingFace and filters it to only include ‘where’ questions for spatial reasoning tasks.

__init__(cfg: TrainPipelineConfig)