Giving a frozen LLM a memory
The injection half of Project Salience, a large project that also covered prompt compression. Memory vectors added straight into the hidden states of a frozen Llama, with only small projector networks trained, scaled from 3B to 8B and tested on whether the model answers from memory it was never shown as text.
Question
Can a language model remember facts about a person without those facts being pasted into the prompt? The idea: encode each memory as a vector, add it directly into the model’s hidden states (its internal working signal, layer by layer), and leave the model’s own weights untouched.
Project Salience was one large project, and injection was its first half. The later work on packing a whole prompt into a few learned tokens is in Squeezing a prompt into a few tokens.
Approach
A frozen Llama with small trained “projectors” that map a memory vector to the model’s hidden size and add it at chosen layers. Only the projectors learn. Memories were encoded with a sentence-embedding model (MiniLM, 384 numbers per memory, then BGE-large, 1024). Training data was synthetic: memory notes, a question, and a gold answer, generated with a larger model.
| Setting | 3B phase | 8B phase |
|---|---|---|
| Base model | Llama 3.2 3B Instruct | Llama 3.1 8B Instruct |
| Hidden size | 3,072 | 4,096 |
| Injected layers | 4 to all 28 | 3 to all 32 |
| Trainable params | projectors only | ~168M at 32 layers, ~15.7M at 3 |
| Data | 4K to 18K examples | 18K, 110K, then 175K (14 question types) |
| Hardware | A10G at first (Hugging Face Space), then A100 | A100 80GB |
Three numbers track progress. Val loss is the model’s error on held-out answers (lower is better). PPL, perplexity, is roughly how many tokens the model is torn between at each step (lower is better). Lift is token-level F1 (word overlap with the gold answer) with injection on, minus F1 with injection off. Lift is the control: it shows how much of the answer came from the memory and not from the base model guessing.
Results
Runs 1 to 13 (Feb 24 to Mar 1, 2026)
| Run | Model | Layers | Data | Val loss | PPL | F1 on | F1 off | Lift | Change |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 3B | 4 | ~4,000 | 1.60 | 4.95 | 0.320 | 0.183 | +0.137 | Proof of concept |
| 2 | 3B | 8 | 5,310 | 1.4159 | 4.13 | 0.412 | 0.187 | +0.225 | Spread layers, 1024-dim, “no memory” examples |
| 3 | 3B | 15 | 5,310 | 1.4124 | 4.11 | – | – | – | More layers, same data: no gain |
| 4 | 3B | 8 | 18,008 | 1.2578 | 3.52 | 0.462 | 0.248 | +0.215 | 3.6x data |
| 5 | 3B | 15 | 18,008 | 1.1661 | 3.21 | – | – | – | 15 layers |
| 6 | 3B | 28 | 18,008 | 1.1025 | 3.01 | – | – | – | All layers |
| 7 | 3B | 28 | 18,008 | 1.0932 | 2.98 | – | – | – | Batch 16 instead of 32 |
| 8 | 3B | 28 | 18,008 | 1.1110 | 3.04 | 0.528 | 0.262 | +0.265 | Long warmup |
| 9 | 8B | 32 | 18,008 | 0.9581 | 2.61 | – | – | – | Bigger base model, 4 epochs |
| 10 | 8B | 32 | 110,971 | 0.8443 | 2.33 | – | – | – | 6x data, 7 question types, 2 epochs |
| 11 | 8B | 32 | 174,741 | 0.9003 | 2.46 | – | – | – | All 14 question types, 2 epochs |
| 12 | 8B | 3 | 174,741 | – | – | – | – | – | Three layers only |
| 13 | 70B (4-bit) | 48 of 80 | 174,741 | – | – | – | – | – | Largest base model, 4-bit weights |
F1 for Runs 4 and 8 was measured on 200 test questions.
8B evaluation on the 14-type test set (147 questions, Feb 28, 2026)
| Metric | Injection on | Injection off |
|---|---|---|
| F1 | 0.4305 | 0.0755 |
| Exact match | 5.4% | 0% |
| Facts correct (judged by a second model) | 40.3% | 17.5% |
| Correctly says “no record” when nothing is stored | 95.1% | 15.0% |
| Lift | +0.355 |
An earlier pass with the same setup on Feb 27 gave lift +0.320. Irrelevant memories raised perplexity by only 1.3–2.0%, so a wrong memory did little harm. Weak spots with injection on: every numeric-precision, temporal-math, comparison and multi-hop question in the set failed the judge.
Stage 5: memory card (Mar 2–4, 2026)
One fixed-size, multi-channel “card” per person (identity, relationships, preferences and so on), updated every turn and injected into the 8B model through a learned encoder, trained on B200 GPUs on synthetic multi-turn chats about made-up personas. In a small conversation test, asked about a user who works at Stripe, the model with the card still answered “Google”. After this the project turned from adding memory to compressing the prompt itself.
Findings
- Additive injection works. Even the first 4-layer run lifted F1 by +0.137, though it acted as a topic hint: the subject was right, the specific names and numbers were not.
- Data was the main lever on the 3B model. Going from 5K to 18K examples beat every layer or batch-size change, and 15 layers only helped once there was enough data.
- A bigger base model broke the 3B floor. On the same 18K examples, 8B reached val loss 0.96 against about 1.09–1.11 for 3B.
- Training on “no memory” questions taught the model to decline instead of making things up: 95% correct refusals with injection on.
- Exact recall of numbers, dates and codes never worked. Opaque tokens such as PINs and IDs made injection actively worse in an early review.
Next
- Exact recall of numbers, dates and IDs, which additive injection never managed. A memory bank the model attends to, one vector per memory instead of one blended vector, was the planned fix.
- The rest of Project Salience continued as prompt compression.