Your AI agent's memory is a privacy risk: new ICML research

TL;DR: I read the ICML paper on agent memory privacy expecting academic noise. What I found changed how I build memory systems. Key-fact summarization cuts extraction risk by 76%. But deleting the source leaves recoverable copies 20% of the time. Full-pipeline purge is the only real fix.

If you’re building AI agents that remember users across conversations, there’s a new paper from ICML 2026 you need to read.

The paper, Deployment-Time Memorization in Foundation-Model Agents, introduces a measurement framework for what the authors call “deployment-time memorization”: the information an agent accumulates during use through its memory system, which is fundamentally different from the parametric memorization people usually worry about in LLMs.

Key takeaways:

• Deployment-time memorization is different from parametric memorization: it’s what your agent stores in its memory system during use, not what the model absorbed during training
• Key-fact summarization reduces extraction risk by 76% on Gemma 3 12B and 64% on GPT-4o-mini while preserving personalization recall
• Raw-only deletion leaves derived summary copies recoverable in ~20% of instances: deleting the source isn’t enough
• Full-pipeline purge (source + all derived representations) or tombstone redaction are the only approaches that push residue to zero
• The Forgetting Residue Score (FRS) is a new metric for measuring whether deleted information is truly gone

The core problem

Foundation-model agents are increasingly long-lived systems. They remember users across sessions, conversations, and tasks. That’s the whole point of agent memory: without it, every interaction starts from scratch and the agent has no context about who you are or what you’ve been working on.

But this creates a new privacy surface. When an agent stores information about a user in its memory system, that information can potentially be:

• Extracted by an attacker who gains access to the memory store
• Recovered after deletion from derived tiers (summaries, compressed representations)
• Leaked through the agent’s outputs without the user’s knowledge

The paper formalizes this as a privacy-utility tradeoff measured by two metrics: Personalization Recall (PR): how well the agent remembers what it should: and Adversarial Extraction Rate (AER): how easily an attacker can extract stored information.

The three design knobs

The authors sweep three memory-design choices:

1. Summarization aggressiveness: do you store raw conversation logs, key-fact summaries, or something in between?
2. Retrieval breadth (k): how many memory items do you retrieve per interaction?
3. Deletion mode: raw deletion (remove the source) vs full-pipeline purge (remove source and all derived copies) vs tombstone redaction (mark as deleted, never surface)

Key findings

Summarization works. Switching from raw conversation storage to key-fact summarization reduces canary extraction by 76% on Gemma 3 12B and 64% on GPT-4o-mini, while preserving nearly all personalization recall. Once content is compressed into summaries, increasing retrieval breadth (k) no longer restores the leakage: the information simply isn’t there to extract.

But deletion is broken. This is the finding that should concern every agent builder. Raw-only deletion (removing the source conversation) leaves derived summary copies recoverable in approximately 20% of instances. You can delete the original, but if your memory pipeline created a summary from it and stored that separately, the summary survives: and the attacker can still extract information from it.

Only full-pipeline purge: deleting the source and all downstream derived copies: or tombstone redaction: marking deleted items so they’re never surfaced: drives the worst-tier residue to zero.

What this means for agent builders

Three practical takeaways:

1. Design memory tier hygiene from day one. If your agent creates summaries, embeddings, or compressed representations of user data, those derived tiers need their own deletion lifecycle. Deleting the source is not enough. Every memory pipeline should trace through all derived representations and treat deletion as a cascading operation.

2. Measure your Forgetting Residue. The paper introduces FRS (Forgetting Residue Score) to quantify whether deleted information remains recoverable from derived memory tiers. Run this against your own agent’s memory system: if you’re not measuring deletion fidelity, you don’t know whether your delete button works.

3. Prefer summarization over raw storage. Beyond the privacy benefits, summarization forces your memory system to be intentional about what it keeps. Raw storage is lazy and expensive: summarization is a design constraint that improves both privacy and performance.

The full paper is available on arXiv: 2606.10062. If you’re building agents with persistent memory, this is 4 pages that will change how you think about deletion.

I’ve written about agent memory management and why agents forget conversations: this paper adds a security and privacy dimension that most memory system designs overlook. Worth reading alongside those posts.

FAQ

What is deployment-time memorization in AI agents? Unlike parametric memorization (where the model remembers training data), deployment-time memorization is when an AI agent stores user information in its memory system during use : and that information can be extracted by an attacker or persists even after deletion.

What is the Forgetting Residue Score (FRS)? FRS is a new metric introduced in the paper that measures whether deleted information remains recoverable from derived memory tiers : like summaries or compressed representations that were created from the original data.

How can agent builders prevent memory leakage? Use key-fact summarization instead of raw storage, implement full-pipeline purge (not just raw deletion), and test deletion fidelity across all memory tiers rather than assuming one deletion call is enough.

Related Posts

• AI agent error handling patterns. Retry strategies, fallback behaviors, and graceful degradation for production agents
• Preventing AI agent hallucinations. 7 techniques to reduce hallucination rates and improve agent reliability
• Your AI Agent Just Scaffolded a Project from 2020. How npm silently downgrades packages and why AI agents don’t catch it

---

This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at hello@agenticup.dev.