Research – Foundations

Giving a frozen LLM a memory

The injection half of Project Salience, a large project that also covered prompt compression. Memory vectors added straight into the hidden states of a frozen Llama, with only small projector networks trained, scaled from 3B to 8B and tested on whether the model answers from memory it was never shown as text.

StartedFeb 24, 2026
Last updateMar 4, 2026
F1 lift from injection on an 8B model, 147 held-out questions (0.43 with memory, 0.08 without)+0.355

Question

Can a language model remember facts about a person without those facts being pasted into the prompt? The idea: encode each memory as a vector, add it directly into the model’s hidden states (its internal working signal, layer by layer), and leave the model’s own weights untouched.

Project Salience was one large project, and injection was its first half. The later work on packing a whole prompt into a few learned tokens is in Squeezing a prompt into a few tokens.

Approach

A frozen Llama with small trained “projectors” that map a memory vector to the model’s hidden size and add it at chosen layers. Only the projectors learn. Memories were encoded with a sentence-embedding model (MiniLM, 384 numbers per memory, then BGE-large, 1024). Training data was synthetic: memory notes, a question, and a gold answer, generated with a larger model.

Setting 3B phase 8B phase
Base model Llama 3.2 3B Instruct Llama 3.1 8B Instruct
Hidden size 3,072 4,096
Injected layers 4 to all 28 3 to all 32
Trainable params projectors only ~168M at 32 layers, ~15.7M at 3
Data 4K to 18K examples 18K, 110K, then 175K (14 question types)
Hardware A10G at first (Hugging Face Space), then A100 A100 80GB

Three numbers track progress. Val loss is the model’s error on held-out answers (lower is better). PPL, perplexity, is roughly how many tokens the model is torn between at each step (lower is better). Lift is token-level F1 (word overlap with the gold answer) with injection on, minus F1 with injection off. Lift is the control: it shows how much of the answer came from the memory and not from the base model guessing.

Results

Runs 1 to 13 (Feb 24 to Mar 1, 2026)

Run Model Layers Data Val loss PPL F1 on F1 off Lift Change
1 3B 4 ~4,000 1.60 4.95 0.320 0.183 +0.137 Proof of concept
2 3B 8 5,310 1.4159 4.13 0.412 0.187 +0.225 Spread layers, 1024-dim, “no memory” examples
3 3B 15 5,310 1.4124 4.11 – – – More layers, same data: no gain
4 3B 8 18,008 1.2578 3.52 0.462 0.248 +0.215 3.6x data
5 3B 15 18,008 1.1661 3.21 – – – 15 layers
6 3B 28 18,008 1.1025 3.01 – – – All layers
7 3B 28 18,008 1.0932 2.98 – – – Batch 16 instead of 32
8 3B 28 18,008 1.1110 3.04 0.528 0.262 +0.265 Long warmup
9 8B 32 18,008 0.9581 2.61 – – – Bigger base model, 4 epochs
10 8B 32 110,971 0.8443 2.33 – – – 6x data, 7 question types, 2 epochs
11 8B 32 174,741 0.9003 2.46 – – – All 14 question types, 2 epochs
12 8B 3 174,741 – – – – – Three layers only
13 70B (4-bit) 48 of 80 174,741 – – – – – Largest base model, 4-bit weights

F1 for Runs 4 and 8 was measured on 200 test questions.

8B evaluation on the 14-type test set (147 questions, Feb 28, 2026)

Metric Injection on Injection off
F1 0.4305 0.0755
Exact match 5.4% 0%
Facts correct (judged by a second model) 40.3% 17.5%
Correctly says “no record” when nothing is stored 95.1% 15.0%
Lift +0.355

An earlier pass with the same setup on Feb 27 gave lift +0.320. Irrelevant memories raised perplexity by only 1.3–2.0%, so a wrong memory did little harm. Weak spots with injection on: every numeric-precision, temporal-math, comparison and multi-hop question in the set failed the judge.

Stage 5: memory card (Mar 2–4, 2026)

One fixed-size, multi-channel “card” per person (identity, relationships, preferences and so on), updated every turn and injected into the 8B model through a learned encoder, trained on B200 GPUs on synthetic multi-turn chats about made-up personas. In a small conversation test, asked about a user who works at Stripe, the model with the card still answered “Google”. After this the project turned from adding memory to compressing the prompt itself.

Findings

  • Additive injection works. Even the first 4-layer run lifted F1 by +0.137, though it acted as a topic hint: the subject was right, the specific names and numbers were not.
  • Data was the main lever on the 3B model. Going from 5K to 18K examples beat every layer or batch-size change, and 15 layers only helped once there was enough data.
  • A bigger base model broke the 3B floor. On the same 18K examples, 8B reached val loss 0.96 against about 1.09–1.11 for 3B.
  • Training on “no memory” questions taught the model to decline instead of making things up: 95% correct refusals with injection on.
  • Exact recall of numbers, dates and codes never worked. Opaque tokens such as PINs and IDs made injection actively worse in an early review.

Next

  • Exact recall of numbers, dates and IDs, which additive injection never managed. A memory bank the model attends to, one vector per memory instead of one blended vector, was the planned fix.
  • The rest of Project Salience continued as prompt compression.