Research prototype · October 2026
Agent Memory Lab
Can a coding agent tell which project decision is still in force after a migration or rollback?
Give decisions a history
Imagine a service starts on cron, moves to a queue, then rolls back to cron. All three notes can look relevant in search. To answer what the service uses now, you need to know when each change took effect and which decision it superseded.
I built a local TypeScript CLI with persistent SQLite storage for dated facts and their source revisions. It can update its index when a file changes, resolve current and historical questions, and return the supporting sources. If two active facts disagree, it reports the conflict.
Measure retrieval and answers separately
I compared keyword search, a hybrid search baseline, and temporal lookup over six runs with 3,600 synthetic questions and 18,000 retrieval evaluations. The facts were explicitly annotated. The table shows how often each method retrieved the expected evidence within five results, averaged across the runs at each corpus size.
| Method | 20 entities | 100 entities |
|---|---|---|
| BM25 keyword search | 66.7% | 70.1% |
| Hybrid search | 41.8% | 35.2% |
| SQLite temporal lookup | 100% | 100% |
The perfect score reflects a benchmark where dates, fields and source links were already available. Extracting those facts from ordinary project notes and measuring the cost of annotation remain open questions.
What the model still got wrong
In a separate set of local Qwen tests, the model sometimes answered “unknown” with the correct fact in its context. In another case it gave a confident answer despite conflicting sources. A small Clef Flash routing pilot scored 8 of 9 evaluable questions, while the rule-based baseline scored 9 of 9.
I kept the prototype’s rule-based answer path for explicitly recorded facts, with citations and a way to abstain. The most useful result was seeing exactly where the system failed after retrieval had already succeeded.