opentau.datasets.grounding.loc_codec

Codec between pixel coordinates and PaliGemma <locNNNN> strings.

PaliGemma’s 1024-bin grounding format quantizes a coordinate axis into 10 bits and emits each bin as a single <locNNNN> token (zero-padded to four digits — <loc23> does not match the tokenizer). A bounding box is encoded as four loc tokens in (y_min, x_min, y_max, x_max) order (y-then-x; do not swap), then a space, then the label, and `; ` separates multiple boxes:

“<loc0234><loc0567><loc0890><loc1023> dog ; <loc0010><loc0050><loc0200><loc0500> cat”

A point is two loc tokens in (y, x) order:

“<loc0234><loc0567> spout”

The 1024 grid is abstract — it is not the input image resolution. Coordinates are normalized using the original image dimensions, then quantized as int(round(coord_norm * 1023)) and clamped to [0, 1023]. Pass the original image’s (width, height) from the dataset (e.g. Image.open(...).size), NOT the post-resize tensor shape that the policy actually consumes.

TODO: eval-side decoding will use loc_tokens_to_xyxy / loc_tokens_to_points against decoded response strings to recover bounding boxes for IoU/mAP. Tracked as a follow-up to the configurable response-formatter work.

Functions

loc_tokens_to_points(s, img_w, img_h)	Parse a string of loc tokens into (x, y) pixel points.
loc_tokens_to_xyxy(s, img_w, img_h)	Parse a string of loc tokens into (x_min, y_min, x_max, y_max) pixel boxes.
point_to_loc_tokens(x, y, img_w, img_h)	Encode an (x, y) point as two loc tokens in y-then-x order.
xywh_to_loc_tokens(box_xywh, img_w, img_h)	Same as xyxy_to_loc_tokens but accepts COCO-style (x, y, w, h).
xyxy_to_loc_tokens(box_xyxy, img_w, img_h)	Encode an (x_min, y_min, x_max, y_max) box as four loc tokens.

opentau.datasets.grounding.loc_codec.loc_tokens_to_points(s: str, img_w: int, img_h: int) → list[tuple[float, float]]

Parse a string of loc tokens into (x, y) pixel points.

Tolerant and segment-aware in the same sense as loc_tokens_to_xyxy: the input is split on ;, and each segment must contribute exactly two loc tokens (in (y, x) order per the PaliGemma convention) to yield a point. Segments with any other count are dropped — a malformed segment cannot shift later ones.

opentau.datasets.grounding.loc_codec.loc_tokens_to_xyxy(s: str, img_w: int, img_h: int) → list[tuple[float, float, float, float]]

Parse a string of loc tokens into (x_min, y_min, x_max, y_max) pixel boxes.

Tolerant and segment-aware: the input is split on the encoder’s segment separator (;), and each segment must contribute exactly four loc tokens to yield a box. A segment with any other count (0, 1, 2, 3, 5, …) is dropped silently — its tokens do NOT spill into the next segment, so a single malformed box cannot misalign every subsequent one. Garbage strings or partial decodes return [].

Parameters:

• s – A string that may contain <locNNNN> tokens, e.g. a decoded response. Non-loc text within a segment is ignored.

• img_w – Original image width in pixels.

• img_h – Original image height in pixels.

Returns:

A list of (x_min, y_min, x_max, y_max) tuples in pixel coordinates.

opentau.datasets.grounding.loc_codec.point_to_loc_tokens(x: float, y: float, img_w: int, img_h: int) → str

Encode an (x, y) point as two loc tokens in y-then-x order.

opentau.datasets.grounding.loc_codec.xywh_to_loc_tokens(box_xywh: tuple[float, float, float, float], img_w: int, img_h: int) → str

Same as xyxy_to_loc_tokens but accepts COCO-style (x, y, w, h).

opentau.datasets.grounding.loc_codec.xyxy_to_loc_tokens(box_xyxy: tuple[float, float, float, float], img_w: int, img_h: int) → str

Encode an (x_min, y_min, x_max, y_max) box as four loc tokens.

The output order is <loc Y_min><loc X_min><loc Y_max><loc X_max> (y-then-x), matching PaliGemma’s convention.

Parameters:

• box_xyxy – (x_min, y_min, x_max, y_max) in pixel coordinates of the original image.

• img_w – Original image width in pixels.

• img_h – Original image height in pixels.

Returns:

A four-token string with no separators.