Squeezing a prompt into a few tokens
The compression half of Project Salience. Learned "soft tokens" (vectors fed to a frozen Llama in place of real words) were fitted to stand in for a whole prompt or conversation, down to 2 tokens, and then masked so only a handful of numbers per token could change.
Question
How much of a prompt can be packed into a few vectors and still make a frozen model answer the same way? Can a 100-word user message become 32 soft tokens, or 2? And how many of the numbers inside each soft token actually need to carry information?
This grew out of the memory injection work in the same project. Injection added memory on top of the prompt. Compression tries to replace the prompt.
Approach
The model under test was always a frozen Llama 3.2 1B Instruct (2,048 numbers per token). A soft token is a vector of that size fed in where a word embedding would normally go. Nothing inside Llama was trained. Training data was about 9,200 chat conversations (UltraChat-style prompts with the replies regenerated by Llama 1B itself, so the target is the model’s own behaviour). Work ran on A100 and B200 GPUs.
Fidelity was measured against the same model reading the full prompt (the “teacher”):
- KL divergence: how far the model’s next-token probabilities with soft tokens drift from the teacher’s, averaged over the reply. 0 means identical.
- Generation match: let both versions write 32 tokens greedily and count how many agree.
| Experiment | What was compressed | Into what | How it was fitted |
|---|---|---|---|
| Canvas | A whole multi-turn conversation | 10 to 256 memory slots, read as soft tokens | A writer network (~75M params) updates the slots token by token |
| Capacity test | N real tokens (3 to 180) | 1 or 2 free vectors | Direct optimization of the vectors, KL on 1 to 5 following tokens |
| Stacking test | Two 100-token windows | 2 vectors each, then stacked to 4 | Each pair fitted alone, then tested together |
| Per-conversation prefix (“gold”) | One user message | 32 soft tokens (or 2 to 50) | Direct optimization per conversation with Muon, KL on the reply |
| Masked prefix | One user message | 32 soft tokens, only the first N of 2,048 numbers free, the rest fixed at 1 | Same as above |
| Gold writer | Any user message | 32 soft tokens | A network trained to output the per-conversation prefixes directly |
| 25-to-1 compressor | Any token sequence | 1 vector per 25 tokens | 6-layer transformer (512 wide) trained on KL to the teacher |
Results
Canvas (Mar 23 – Apr 21, 2026). Error on the first assistant tokens, as cross-entropy (lower is better).
| Setup | Loss |
|---|---|
| No useful context (random guess over the vocabulary) | 11.76 |
| Raw token embeddings through a separately trained key/value projector | 9.0 |
| Same embeddings fed as soft tokens through Llama’s own forward pass | 6.95 |
| Trained canvas, 100 slots (several runs) | 5.2–5.9 |
| Trained canvas, 256 slots | 5.36 |
| Llama reading the full conversation | ~0.5 |
Masked prefix, one conversation, 32 soft tokens (KL after about 4,150 steps, last real user token still fed alongside the prefix):
| Free numbers per soft token | Free numbers in total | KL |
|---|---|---|
| 3 | 96 | 2.48 |
| 4 | 128 | 1.70 |
| 6 | 192 | 0.86 |
| 8 | 256 | 0.51 |
| 10 | 320 | 0.41 |
| 16 | 512 | 0.22 |
| 32 | 1,024 | 0.066 |
| 64 | 2,048 | 0.024 |
Which numbers were left free (the first N, every 32nd, or four separate blocks) changed KL by less than 5%. Weighting the first reply token in the loss brought a run to KL 0.030 overall and 0.034 on that first token.
Per-conversation prefixes and transfer
| Test | Result |
|---|---|
| One prefix optimized hard on a single conversation (32 tokens) | KL 0.0008 over 512 reply positions |
| 20 conversations pulled toward that prefix as a shared anchor | KL 0.0015, cosine similarity to the anchor 0.97, to each other 0.94 |
| Starting conversation B from conversation A’s prefix instead of random | KL 1.40 at step 0, against about 9 from random |
| 10 random starts, 2 soft tokens, same conversation | only 29–31% of the variation was shared; the rest depended on the start |
Gold writer. Validation KL across 10 held-out conversations.
| Writer | KL |
|---|---|
| Fixed anchor prefix, ignores the input | 0.171 |
| Trained to copy the per-conversation prefixes (MSE only), 1 to 12 layers, 2k to 6k wide | 0.21 |
| MSE plus a slowly ramped KL loss | 0.17 |
| Residual writer (anchor plus a learned correction) | 0.169 |
Bigger writers (up to 8,192 wide), deeper ones (4 layers), and more data (237 to 9,000 conversations) all stopped at the same level.
Checking generated text. The KL 0.030 prefix above was run without the last real user token. Its 32-token reply matched the teacher’s on 1 token (3.1%) and drifted off topic from the first word.
Findings
- Soft tokens have to go through the model’s own forward pass. Hand-built keys and values were about 20 times too small and the model could not read them (loss 9.0 against 6.95).
- A trained canvas never got close to the full conversation. It stalled near 5.4 against 0.5, so most of the conversation was lost.
- Very few numbers per soft token are needed to move the output. At first KL roughly halved with every two extra free numbers, then gains slowed. 64 of 2,048 per token was close to a perfect match.
- Those masked results leaned on one real token. The last word of the user’s message was still fed in next to the prefix, and without it the same prefix produced unrelated text. The dimension scan has to be redone with the prefix alone.
- Prefixes for different conversations share a lot of structure (warm starts were 6.4 times closer than random), yet every writer collapsed to one generic output that was no better than the fixed anchor.
- Averaged KL with the true previous token fed in (teacher forcing) is a poor proxy. Small KL hid a wrong first token, and generated text was the only honest check.
Next
- Redo the masked-prefix scan with the prefix alone and score it by generated text, to find how many free numbers the real task needs.
- Train a small writer that maps Llama’s own final hidden state for the prompt into the free numbers of the prefix (64 per token), trained on the teacher’s generated replies instead of KL.
- Test the 25-to-1 compressor and the two-window stacking on the same generation check.