Perceptio: letting a vision-language model sketch space before it answers

Tue, 06 Oct 2026 00:00:00 +0000

Paper: arXiv 2603.18795 · Hugging Face · Long-form write-up on Zhihu (in Chinese)

The problem

Large vision-language models are very good at what is in an image and surprisingly weak at where. Ask which of two objects is closer to the camera, or to segment “the second mug from the left”, and the model has to infer the geometry implicitly. It never writes down a spatial interpretation that it, or we, could check.

The idea: perceive first, then answer

Perceptio puts perception into the token stream. Before it answers, the model generates two kinds of spatial tokens inside the same autoregressive sequence:

Segmentation tokens based on SAM2, which ground the object being talked about;
Depth tokens from a VQ-VAE codebook that we distil from a strong monocular depth teacher, so a dense depth map becomes a short, discrete sequence.

The answer is then conditioned on the model’s own spatial “sketch”. You can think of it as a chain of thought made of masks and depth codes instead of words, and, because the tokens decode back into a mask and a depth map, it is one you can look at.

Making depth tokens trainable

Generating long runs of depth codes is unstable if you only use the usual next-token loss. Two things helped:

Composite depth-token objectives: a marker, a token and a count loss, used together;
Soft merging, which makes depth reconstruction from the predicted tokens differentiable, so the model is also supervised on the depth map it implies.

The model is built on InternVL and co-trained on a mix of tasks, so the same perception tokens serve segmentation, spatial question answering and general VQA.

Results

Compared with the strongest prior baselines reported in the paper:

Benchmark	Gain
Referring segmentation, RefCOCO / RefCOCO+ / RefCOCOg (cIoU)	+0.8 / +1.4 / +1.1
HardBLINK spatial understanding (accuracy)	+10.3%
MMBench (accuracy)	+1.0%

The HardBLINK jump is the result I care most about: it is the benchmark where “just look harder” does not work and explicit geometry does.

What’s next

I think explicit, inspectable intermediate perception is a good direction for spatial reasoning in multimodal models, and for world models more broadly. That is the line I want to keep pushing after my PhD.