Discrete Latent Reasoning

01 / The idea

Reasoning doesn’t have to
spell everything out.

Chain-of-thought reasoning works by putting intermediate steps into words. But those words come at a cost: longer sequences and more autoregressive generation. Latent reasoning asks a natural question: can a model do useful reasoning in a more compact space?

Continuous latent states are difficult to supervise and inspect. Discrete Latent Reasoning (DLR) gives that space explicit anchors: a vocabulary of learned latent tokens that can be generated, trained, and decoded.

Compress the representation.
Keep the reasoning accessible.

We render reasoning traces into images, learn a discrete codebook from their visual features, and extend a language model’s vocabulary with those codes. The result is a single autoregressive model that predicts both text and latent tokens.

~20×shorter reasoning trajectories

5reasoning benchmarks

3language-model training stages

Compression example: 108.4 CoT tokens vs. 5.4 DLR latent tokens, averaged across four grade-school benchmarks on Qwen3-VL-4B (Table 1). Token compression is not a measured wall-clock speedup.

02 / From text to latents

A vocabulary for thought.

Rendered text provides a bridge between language and compact visual features. DLR uses a DeepSeek-OCR2-based encoder to turn images of reasoning traces into causally ordered features, then maps them to a learned codebook.

Inside the compressorSelect a step to explore

Conceptual illustration of codebook construction; example symbols are not model outputs.

The codebook is initialized through clustering and trained with stochastic vector quantization. An auxiliary OCR decoder reconstructs the original text, encouraging the learned codes to preserve recoverable semantics.

See the complete architecture Figure 2 from the paper ↗

DLR architecture: rendered chain-of-thought features are quantized into a latent codebook, then integrated into a language model through alignment, SFT, and reinforcement learning.

03 / Learning to reason

Discrete tokens.
A familiar training recipe.

Lightweight input and output projectors connect the codebook to the language model. With a joint text + latent vocabulary, the model can learn through standard next-token prediction and reinforcement learning.

I

Align the two spaces

Freeze the language model and train the projectors. A latent prefix teaches the model to recover the corresponding reasoning text.

Latent-text alignment

II

Learn the latent trajectory

Supervised fine-tuning teaches the model to generate latent tokens and then a final answer. Explicit boundaries keep the sequence structured.

Supervised fine-tuning

III

Refine with rewards

GRPO optimizes freely generated trajectories. The paper combines answer correctness, response format, and a process-alignment reward based on decoded reasoning.

Reinforcement learning

question→<latent>z₁z₂…zₙ</latent>→answer

A schematic sequence; the paper also wraps the final answer in <answer> boundaries. At inference time, the model generates latents directly—rendering is an offline training step.

04 / Accuracy & compression

Small trajectories.
Substantial reasoning.

DLR improves on prior latent-reasoning baselines across the paper’s benchmarks while using compact trajectories. Explicit CoT remains more accurate in many settings; the practical question is the tradeoff between accuracy and reasoning length.

Explore the resultsPaper, Table 1 ↗

BackboneBenchmark

MethodAccuracy (%)Reasoning tokens

Qwen3-VL-4B four-task average: DLR (Full), 63.2% accuracy / 5.4 reasoning tokens; CoT-SFT, 79.3% / 108.4 tokens. Enable JavaScript to explore all benchmark comparisons.

View numerical results & uncertainty


Method	Accuracy (%)	Reasoning tokens

Reported means from Table 1; per-benchmark uncertainty is shown in the numerical table. #L counts reasoning tokens, excluding the final answer. The four-task average is the paper’s reported average.

Scaling to harder mathematics

On MATH-500, DLR (Full) reaches 44.6% accuracy with 28.4 latent tokens on Qwen3-VL-4B, compared with RoT’s 33.2% at 64 tokens. CoT-SFT reaches 55.8% with 291.5 tokens.

MATH-500 / DLR (Full)Table 2 ↗

Qwen3-VL · 2B34.0%28.9 latent tokens

Qwen3-VL · 4B44.6%28.4 latent tokens

Qwen3-VL · 8B54.0%29.5 latent tokens

05 / Reading the latents

A window into
compressed reasoning.

A short latent sequence does not have to be opaque. The auxiliary decoder maps generated latent trajectories back into readable text, providing a way to inspect the computations they carry.

Two examples from Figure 4 comparing input questions, original chain-of-thought text, and text reconstructed from DLR latent trajectories. — Figure 4 / From the paperClick to inspect ↗

The case studies show recovered mathematical structure, alongside reconstruction artifacts. Names and units may change; the appendix also reports numerical reconstruction noise. Decoded text is a useful probe, not a guarantee of an exact or faithful transcript.

What remains open?

Rendering and codebook construction add offline preparation cost. Codebook size and quantization settings may need domain-specific tuning. Interpretability depends on the auxiliary decoder’s reconstruction quality, and the evaluation focuses on mathematical reasoning.

06 / Open training code

Build on the recipe.

The repository contains codebook training and latent-token export, a local TRL fork, latent alignment, SFT, GRPO, and evaluation tools, with example launch scripts for eight GPUs.

01 Learn the codebookTraining, token export & decoding↗02 Train the latent modelAlignment, SFT, GRPO & evaluation↗03 Explore the launch scriptsConfigurable eight-GPU examples↗

Release scope: model weights and generated datasets are not included. The provided GRPO script implements final-answer rewards; the paper’s process-alignment reward additionally requires the auxiliary decoder and an evaluator LLM.

View repository ↗

07 / Citation

If you find this work useful.

The full method, experiments, and limitations are available in the paper.

BibTeX

@article{chang2026discrete,
  title   = {Why Struggle with Continuous Latents?
             Interpretable Discrete Latent Reasoning
             via Rendered Compression},
  author  = {Chang, Shuochen and Liu, Qingyang and
             Wang, Shaobo and Gao, Bingjie and
             Ma, Qianli and Zhao, Haonan and
             Miao, Yibo and Sun, Yulin and
             Peng, Zelin and Li, Jiangtong and Niu, Li},
  journal = {arXiv preprint arXiv:2606.29712},
  year    = {2026},
  url     = {https://arxiv.org/abs/2606.29712}
}