Does a Language Model Have to Do All Its Work in Tokens?

Imagine asking a model to write a project handover from a pile of notes. It needs to decide what information belongs together, carry that information forward, and turn it into precise sentences.

Does all of that work need to happen at the same granularity as the words we eventually read?

Cola2 explores a different division of labour: compress text into a shorter sequence of learned representations, generate in that space, and use another component to produce the tokens.

The timing needs a distinction. The paper was submitted on October 22, 2026. The new development is the November 6 release of inference code and a downloadable checkpoint, visible in the source history and model history.

That makes the idea more concrete. We can inspect how it is implemented, rather than assume what generating in a latent space means.

Don't language models already use vectors?#

Yes. Ordinary language models already turn tokens into numerical representations and process continuous hidden states. The change is where the compressed sequence enters the generation process.

The difference is that it makes a compressed sequence of latents a target for generation. A latent is a learned numerical representation; it isn't a readable outline or a named concept we can inspect like a JSON field.

The released architecture groups contiguous tokens into spans of one to eight tokens, targeting about two tokens per latent on average. Each latent has 64 dimensions.

So the mental picture should stay modest. This isn't one vector containing an entire paragraph's meaning. It is a shorter representation with a learned boundary between what to preserve and what the text generator can supply.

How does it turn those representations into language?#

There are three main jobs in the implementation.

The compression encoder groups the existing text into spans. A diffusion Transformer, or DiT, proposes new latent blocks by refining noise. The token decoder turns those latents into text, using the text context too.

A useful analogy is an editor working from a compact set of notes. The notes constrain what should be said; the writer still supplies wording and connective detail. But unlike our editor's notes, the model's latents are learned vectors, not sentences or verified facts.

There is also an important difference between the research framework and this release. The paper studies masked-diffusion token decoders, while the public checkpoint's text realization path is span-causal: it commits spans from left to right. Generating latent blocks doesn't mean the complete answer's words appear in one parallel operation.

The cover diagram is the authors' original method figure. Their three-minute overview offers another view. At 00, a schematic maps representations to words; at 01, it illustrates latent denoising. I inspected these frames. This is an authored explanation, not a recording of measured inference speed; its MDM scenes also illustrate the broader research framework.

Who decides what the compressed version should keep?#

This is the part I find most interesting.

Compressing something is easy if we don't care what gets lost. A handover that preserves the general topic but drops a deadline is shorter and less useful.

The training description lets token-prediction training update the encoder through its latents. The representation learns from what the decoder needs. The decoder also sees clean, noisy and absent latent conditions, balancing reconstruction with its own ability to generate language.

Meanwhile, the latent generator's training signal is detached from the encoder. The authors separate learning to predict the representations from shaping what those representations contain.

My reading is that the compression scheme and the writer need to learn a shared contract. Otherwise, we might build representations that are convenient to predict but poor at carrying the details needed downstream.

Why encode the generated text again?#

Because a proposed representation and the words actually selected aren't automatically identical.

The inference documentation describes latent feedback: after committing text spans, the system re-encodes them under their fixed span layout before continuing. Later generation follows the text that was emitted.

As an illustrative example, imagine an internal proposal steering toward “dispatch today,” while the selected sentence says “dispatch tomorrow.” Continuing from the proposal alone could carry the wrong history forward. This is a toy explanation, not a captured Cola2 output.

Re-encoding addresses consistency between two internal views. It doesn't establish that the dispatch date is true. That still needs evidence from the application.

What do the results actually show?#

The released results are from the authors:

Two rows favour Cola2; the overall average doesn't. The model card describes the latent DiT as not yet converged and the checkpoint as a base model, not an instruction-tuned assistant.

The training-budget comparison needs even more care. The authors report 346.03 billion source tokens, but their training account says the decoder and encoder embedding were initialized from Qwen3-4B-Base. That brings inherited training into the picture. Dividing Llama's reported token budget by Cola2's new training tokens doesn't establish a from-scratch efficiency gain.

Different data, initialization and evaluation details matter alongside the generation method.

Does compression help when the comparison is controlled?#

The authors also provide a separate controlled scaling study, matching model size, data and evaluation settings. Here the uncompressed Cola-MDM has the strongest aggregate scaling. Compressed Cola2-MDM improves more slowly, then narrows its gap to LLaDA as training continues.

That is a useful result without turning it into a winner label. Compression creates another tradeoff to investigate; it doesn't automatically improve every task.

Has anyone outside the project tried it?#

I found a November 6 report by Efrain Garay describing CPU offload on an RTX 4070 Ti SUPER and difficulty running the official harness. It is an early account, not an independently reproduced result here.

There are reasons for caution: it calls the model 4B, while official metadata lists roughly 8.77B total parameters. Its storage descriptions also conflict with that metadata, and its small custom test isn't the paper's protocol. I couldn't verify the underlying run artifacts, so I wouldn't use its speed or accuracy figures to choose hardware. The setup and unresolved discrepancies are recorded in the evidence notes.

I haven't run Cola2 myself, and I couldn't verify a controlled independent test video of this checkpoint.

Does a shorter latent sequence mean a faster application?#

Not by itself. There is still latent denoising, token decoding and feedback work. The project's limitations explicitly distinguish theoretical savings in the latent prior from end-to-end speed.

For the handover example, I'd test whether compression preserves names, dates, negations and dependencies, alongside total latency and memory. Repeated runs matter too: the reproduction guide says latent noise remains stochastic even with greedy token selection, and the historical paper seeds aren't published.

What interests me is making the division of labour learnable: what should the compressed representation retain, and what can the decoder reconstruct or supply?

For an application, the test is whether that division saves work while preserving the details we cannot afford to lose.