Agent Memory Leaderboard

Agent Memory Leaderboard Official.
First open championship for agent memory systems.
Neutral · Reproducible · Fair.
mailto: [email protected]

[Agent Memory Leaderboard @AgentMemoryL]

Agent Memory Challenge 2026 Cycle 2 opens September 20.

A shared evaluation for long-term Agent Memory across Textual, Coding, and Multimodal tracks—testing not just what agents store, but whether they retrieve current, useful evidence when it matters. https://t.co/hyjO72lMNk

[Agent Memory Leaderboard @AgentMemoryL]

Thank you for raising this. The current rule is intended to control evaluation variables, not to claim that GPT-4o-mini is technically superior. To clarify, open-source entries are not required to call an LLM during Add or Search. Systems that use only local embeddings, rerankers, keyword search, vector search, or other non-generative components are eligible. If a system does use a generative model in Add or Search, the current Cycle 2 rule requires that model to be GPT-4o-mini. We chose a fixed model for this cycle because downstream model choice can materially affect the result through summarization, extraction, query rewriting, memory consolidation, and retrieval planning. Fixing that variable makes scores more comparable and makes independent reproduction easier. The platform’s own Answer and Eval models are separate from the participant’s internal Add/Search implementation. We agree that the policy has limitations. A frozen open-weight model could improve transparency, but it also introduces hardware, quantization, runtime, and implementation differences. A compute budget alone would not fully control those variables. Separate model tiers are also a reasonable direction, but they would require enough submissions per tier, tier-specific reporting, and a carefully validated normalization protocol. For Cycle 2, we are keeping the rule fixed so that all results are evaluated under one stable contract. For a future cycle, we are open to adding a model-agnostic track, a frozen open-weight reference model, or explicitly separated model tiers. Any such change would be announced before the evaluation window and would not be applied retroactively to existing results.

[Agent Memory Leaderboard @AgentMemoryL]

Exactly. Freshness is not just a ranking signal.

A meaningful test needs a real conflict: plausible older evidence, a later correction, and a task where the agent’s next action depends on resolving the difference correctly.

That makes it possible to evaluate three things separately: whether the system retrieves the relevant provenance, recognizes supersession, and returns only the evidence needed for the current task rather than amplifying stale context.

[Agent Memory Leaderboard @AgentMemoryL]

Exactly. A larger window can preserve more history, but it can also make stale assumptions harder to notice.

Once an agent has to act again—after a decision, correction, or environmental change—the question becomes whether it can retrieve the evidence that should govern the next action, rather than simply carry forward everything it has seen.

That is where retrieval, temporal signals, and provenance start to matter together.

[Agent Memory Leaderboard @AgentMemoryL]

Agent Memory Challenge 2026 Cycle 2 is now open.

Long-term memory is not just about storing more history. It is about retrieving the right evidence, recognizing what has changed, and avoiding stale context when an agent needs to act.

Three tracks: Textual · Coding · Multimodal
Open-source Methods · Commercial Products

Over USD 22,000 prize pool for eligible open-source teams.

A shared Add/Search interface. Standardized Answer/Eval. Public, comparable results.

Join: https://t.co/t2QajOb7Fe

[Agent Memory Leaderboard @AgentMemoryL]

For the full technical overview of Cycle 2—including the shared Add/Search evaluation boundary, Textual, Coding, and Multimodal tracks, and the principles behind reproducible Agent Memory evaluation: https://t.co/ro6e6G36ON

[Agent Memory Leaderboard @AgentMemoryL]

Agent Memory Challenge 2026 Cycle 2 opens September 20.

A shared evaluation for long-term Agent Memory across Textual, Coding, and Multimodal tracks—testing not just what agents store, but whether they retrieve current, useful evidence when it matters. https://t.co/hyjO72lMNk

[Agent Memory Leaderboard @AgentMemoryL]

Read the participation guide: https://t.co/DieiWkPhUC
API integration guide: https://t.co/bvyn2jjZKU
Open-source evaluation framework and updates: https://t.co/P946npqzo5
Cycle 2 runs through October 31.
Official results are planned for mid-November.

[Agent Memory Leaderboard @AgentMemoryL]

For the full technical overview of Cycle 2—including the shared Add/Search evaluation boundary, Textual, Coding, and Multimodal tracks, and the principles behind reproducible Agent Memory evaluation:
https://t.co/ro6e6G36ON

[Agent Memory Leaderboard @AgentMemoryL]

Agent Memory Challenge 2026 Cycle 2 opens September 20.

A shared evaluation for long-term Agent Memory across Textual, Coding, and Multimodal tracks—testing not just what agents store, but whether they retrieve current, useful evidence when it matters. https://t.co/hyjO72lMNk

[Agent Memory Leaderboard @AgentMemoryL]

That is exactly the direction we think evaluation needs to move toward.

In practice, one common failure case is when an old, highly similar statement keeps outranking a later update that is phrased differently or contains only a partial correction. The system retrieves both, but does not reliably identify which one should govern the next action.

Conflicting-memory and stale-memory cases make that failure visible.

[Agent Memory Leaderboard @AgentMemoryL]

Agent Memory Challenge 2026 Cycle 2 opens September 20.

A shared evaluation for long-term Agent Memory across Textual, Coding, and Multimodal tracks—testing not just what agents store, but whether they retrieve current, useful evidence when it matters. https://t.co/hyjO72lMNk

[Agent Memory Leaderboard @AgentMemoryL]

Well put. A memory system can retrieve highly relevant text and still fail the task if that text is no longer valid.

The meaningful test is whether a newer signal changes what the agent does next, while preserving enough provenance to explain why an earlier belief was no longer used.

[Agent Memory Leaderboard @AgentMemoryL]

Exactly. “Current” is not the same as “most similar.”
A useful evaluation needs to distinguish at least three cases: a later correction or supersession, historical evidence that should remain traceable, and content that must be removed for privacy or safety.
The hard part is testing whether the agent changes its next action—not merely whether the newer text appears somewhere in Top-K.

[Agent Memory Leaderboard @AgentMemoryL]

Hi tetsuo — I’m part of the organizing team behind the Agent Memory Leaderboard (AML).

Your point that coding agents can degrade across sessions—and that runtime traces can reveal recurring failure patterns—is very close to a question we are trying to make measurable.

AML Cycle 2 is now open, including a Coding Memory track. We evaluate whether an agent can retrieve useful historical engineering evidence under a shared Add/Search and downstream Answer/Eval protocol—not simply retain a longer transcript.

We would value your perspective on what an evaluation should measure when context drifts, old decisions become stale, or runtime evidence should change future agent behavior.
https://t.co/bWD4Pf61w7

[Agent Memory Leaderboard @AgentMemoryL]

The filtering signal is especially important here.

If ~36% of rollouts are removed, headline capability trends need to be read alongside per-category acceptance rates and post-filter performance. Otherwise, an aggregate gain can partly reflect a changing training distribution rather than uniform improvement on the hard tail.

Would be interesting to see whether the agentic and code subsets diverge most under that filter.

[Agent Memory Leaderboard @AgentMemoryL]

This is exactly the kind of correction that makes a leaderboard credible.

An impressive result can still be valuable, but only if its evaluation conditions are visible and comparable. Access to source papers changes the task—not just the score.
Clear provenance for runs, resources, and evaluation conditions should be treated as part of the result itself.

[Agent Memory Leaderboard @AgentMemoryL]

Hi Mia — I’m part of the team behind the Agent Memory Leaderboard (AML).

AML Cycle 2 opens on September 20, with Textual, Coding, and Multimodal Memory tracks. We evaluate participant-hosted Add/Search systems under one standardized Answer/Eval flow.

Your work around local LLMs, long-context workloads, agent setups, and reproducible performance experiments feels highly relevant to the developer audience we hope to reach.

We’re inviting a few technical AI creators to collaborate on the launch. The deliverable would be one Quote Post of our official launch post, based on the visual and technical brief we provide, with your own take.

This is a paid collaboration. Would you be interested? If yes, please share your rate for one Quote Post.

https://t.co/bWD4Pf61w7

[Agent Memory Leaderboard @AgentMemoryL]

Great suggestion. Temporal/versioned identifiers make it possible to evaluate not only whether relevant evidence was retrieved, but whether it is still valid in the current state.
Precision@k alone can hide stale context. A stale-evidence rate under explicit updates would make that failure mode measurable.

[Agent Memory Leaderboard @AgentMemoryL]

What should Agent Memory be judged on?

Not whether a system can write a convincing final answer—but whether it can reliably retrieve the right evidence from long-running experience.

In 2 days, Cycle 2 of the Agent Memory Challenge opens.

One shared Add/Search interface.
One standardized Answer/Eval pipeline.
Textual, Coding, and Multimodal Memory.

Measure memory. Compare what matters.
September 20, 00:00 UTC+8
https://t.co/bWD4Pf61w7

[Agent Memory Leaderboard @AgentMemoryL]

Exactly. Recall is only useful when it preserves the state of knowledge.
A memory system should distinguish what was said, what later changed, and what remains current—without erasing the evidence trail.
That is why temporal reasoning and memory governance matter alongside retrieval in AML.

[Agent Memory Leaderboard @AgentMemoryL]

For the evaluation framework, API contract, and benchmark updates:
https://t.co/bHyMLiNw8e

We welcome technical feedback and Benchmark Contributions from the community.

[Agent Memory Leaderboard @AgentMemoryL]

Hi Diogo — I’m part of the organizing team for the Agent Memory Leaderboard (AML).

Your work and writing on reliable no-human-in-the-loop automation resonates with a problem we are actively working on: as agents operate over longer horizons, memory cannot just be stored context—it needs to be isolated, traceable, and evaluated by its downstream value.

AML’s first cycle had 136 registrations and 67 systems complete official evaluation. The second Agent Memory Challenge opens on September 20, with Textual, Coding, and Multimodal tracks. Systems expose a common Add/Search interface, while AML standardizes the downstream Answer and Eval pipeline.

We’d value your critical perspective on what a meaningful memory benchmark should measure for reliable automation.

• Challenge & evaluation: https://t.co/t2QajOb7Fe
• Rules: https://t.co/DieiWkPPKa
• API guide: https://t.co/bvyn2jkxAs
• GitHub: https://t.co/P946npr7dD