MEMORYLENS: BENCHMARKING LLM MEMORY DECAY ACROSS 100 CONVERSATION TURNS
I built MemoryLens after noticing a glaring gap: every RAG tutorial claims superior recall, but nobody had actually measured how memory architectures degrade as conversations grow longer. So I quantified it.
Recall/token improvement
Chunked RAG recall @ T=100
Lower cost vs naive
Conversation turns tested
THE PROBLEM WITH LLM MEMORY
LLMs forget. When a conversation exceeds the context window, tokens get dropped, and the model loses facts it was told 30 messages ago. Naive truncation (just dropping old tokens) is the default behaviour in most production chatbots. RAG-based memory retrieves relevant chunks instead of dropping them blindly. But how much better is it, really, at turn 50? Turn 100?
I modelled memory decay using Ebbinghaus forgetting curves alongside exponential and linear decay models, then ran statistical multi-seed benchmarks across five memory backends.
FIVE MEMORY ARCHITECTURES TESTED
EVALUATION METRICS
All five metrics are computed without requiring an API key. Content-based evaluation only:
BENCHMARK RESULTS
At turn 100, Chunked RAG achieves 85% recall using 38 tokens/query. Cascading Temporal delivers 5.45× better recall-per-token efficiency than naive truncation. Naive truncation bottoms out at 31% recall.
SummaryMemory showed high recall but suffered from hallucination drift: the LLM-generated summaries introduced factual contamination at scale, making it unreliable for high-stakes applications.
WHY THIS MATTERS FOR PRODUCTION RAG
If you're building a customer support chatbot, a coding assistant, or any long-horizon conversational AI and using naive truncation, you're silently losing context that directly impacts response quality. MemoryLens gives you a reproducible framework to test your memory strategy before it fails in production.
OPEN SOURCE
MemoryLens is fully open-source. It ships with 24 integration tests, CI via GitHub Actions (Python 3.10–3.11), a Streamlit dashboard, and supports Groq, OpenAI, Anthropic, OpenRouter, and Ollama as LLM backends, no API key required for the content-based metrics path.
Written by Neal Daftary, AI & ML Engineer, Nirma University, Ahmedabad