On the Limits of Token Reduction for Efficient Unified Vision Language Training

Abstract

What we study

Unified vision-language models (VLMs) jointly learn visual understanding (e.g., text prediction conditioned on images) and visual generation (autoregressively predicting image tokens) with a single next-token objective. However, joint training is expensive, and token-reduction methods that work for inference-time or single-task settings do not directly translate to training-time efficiency. We investigate whether token-reduction acceleration can be extended to unified training and what its fundamental limits are.

Contributions

We make four main contributions:

• Unified redundancy analysis: characterize an asymmetry in token usage across depth for understanding vs generation.
• Task-specific accelerators: propose training-time token reduction methods for understanding and generation separately.
• Synergy loss discovery: show that combining task-specific token reduction under unified training can collapse joint performance.
• Lessons for unified acceleration: effective strategies must preserve shared cross-task structures rather than composing isolated optimizations.

Core idea

Through attention-allocation analysis, we uncover a depth-dependent asymmetry: late-layer visual tokens are substantially redundant for visual understanding, while visual generation maintains persistent dependence on image tokens across depth. This observation motivates task-aware accelerators that reduce image-token computation in a way that matches each objective's token-utilization pattern.

Task-specific accelerators

HiMix (Understanding)

For understanding, we reduce image-token computation by removing image tokens from the query stream while retaining them in key/value projections. This eliminates expensive image-to-image quadratic attention, while keeping the necessary text-to-image interactions.

HMGen (Generation)

For generation, we must preserve autoregressive structure: predicted image tokens need to remain valid queries for subsequent image-token prediction. HMGen therefore skips image-token attention/FFN updates only in a set of designated "shallow" middle layers, forwarding image-token hidden states without participation in updates. Additionally, HMGen introduces separate projection parameters for image vs text tokens in full layers to stabilize hierarchical conditioning.

Table 1. HiMix for visual understanding

Table 2. HMGen for visual generation

Why unified token reduction fails

When task-specific token reduction is applied under unified training, we observe a consistent synergy loss: unified training's mutual performance gains disappear, and both understanding and generation can collapse compared to the unified baseline. The central issue is that token dropping changes which tokens participate in attention, which parameters receive gradients, and thus fragments the shared cross-task optimization dynamics.

Table 3. Unified training: synergy loss

Synergy-aware mitigation

To mitigate synergy breakage, we introduce a separate image projection strategy: instead of fully sharing image-related projection parameters, we partially decouple them by decomposing image projection matrices into shared and image-specific components. This partially restores shared pathways while still allowing token participation to differ across tasks.

BibTeX

@inproceedings{chen2026token_reduction_unified_vlm,
  title={On the Limits of Token Reduction for Efficient Unified Vision Language Training},
  author={Chen, Siyi and Zhuang, Weiming and Li, Jingtao and Lv, Lingjuan},
  booktitle={CVPRW},
  year={2026}
}