The missing step
Standard vision-language models map visual features directly to text. They may answer correctly, but the answer does not expose what objects were localized or which geometry was used.
arXiv:2603.18795v1 | multimodal spatial reasoning
Perceptio: spatial tokens before words.
Yuchen Li1,2,*, Amanmeet Garg1, Shalini Chaudhuri1, Rui Zhao1, Garin Kessler1
1 Amazon Prime Video Team2 MBZUAI
* Yuchen Li is a PhD student at MBZUAI. This work was completed during an internship at Amazon.
Perceptio turns perception into an explicit autoregressive trace: first segmentation, then depth, then language. The result is a multimodal model whose answers are conditioned on generated 2D and 3D structure instead of hidden visual intuition alone.
1
Image
pixels
2
[seg]
object mask tokens
3
[depth]
binned 3D geometry
4
Language
spatial answer
Generated sequence
<image>[seg]mask_018mask_041[/seg][depth]z_03z_18z_32z_57[/depth]Themugsitsinfront
Generation trace
Watch spatial tokens become the model's scratchpad.
Generated [seg] tokens
It first emits mask tokens that isolate the objects needed for the response.
<image>[seg] mask_12 mask_48 [/seg][depth] z_03 z_18 z_32 z_57 [/depth]The mug is in front of the cabinet.
Motivation
VLMs can speak fluently while being spatially vague.
The Perceptio step
The paper inserts spatial tokens into the target sequence. Generated masks and depth become the visible bridge between image evidence and the final natural-language response, trained through the same next-token objective.
Method
Perception is written into the sequence before language gets a turn.
Why it matters
Spatial reasoning errors often happen because language tokens must infer geometry implicitly. Perceptio makes geometry an explicit intermediate product.
Tokenization
SAM2 masks become segmentation tokens. DepthAnything-V2 depth maps are discretized with a VQ-VAE codebook into depth token sequences.
Training signal
Beyond language loss, Perceptio adds marker-token, count-token, and Hamming-distance losses so the spatial trace is learned directly.
Inference behavior
The model first externalizes where objects are and how far they are, then uses those tokens as context for the response.
Paper visuals
Original figures from the arXiv HTML version.
Figure 1: Perception-enhanced generation
The paper's overview figure shows the central story: Perceptio emits segmentation and depth tokens before the final language response.
Figure 2: Architecture and training flow
The model connects image features, mask/depth token targets, and language generation in one autoregressive sequence.
HardBLINK depth example
HardBLINK depth example
RefCOCOG segmentation example
Results
Explicit spatial traces make the answer easier to trust.
RefCOCO cIoU
+0.8 to +1.4 on referring-expression segmentation benchmarks.
HardBLINK
+10.3 on a spatial-relation benchmark designed to stress visual layout.
MMBench
+1.0 on a general multimodal benchmark, despite the spatial focus.
Traceability
The generated answer is preceded by inspectable segmentation and depth traces.
Takeaway
A model that can show its spatial work.
The contribution is not another captioning wrapper. It is a sequence design: perception tokens are treated as first-class outputs. That makes segmentation and depth both supervision targets and inference-time context, giving the final language a compact visual scratchpad.