Retrieval Scoped by Subject
A factorial measurement of conversational memory on LoCoMo, and the negative controls that locate the effect
Hanzo AI Research
Abstract
We measure a graph-structured conversational memory against a flat nearest-neighbour index on LoCoMo — 10 conversations, 272 sessions, 5,882 turns, 1,986 questions — using the benchmark's own scorer, ported to Go and verified against the original Python on all 1,986 questions and every system reported. Three results follow, and the negative controls are the substance of the paper. First, the aggregate over all five question categories is degenerate: a system that replies “No information available” to every question scores 0.228, above every system here that has to find its own evidence, and only an oracle handed the dataset's own annotated evidence scores higher, at 0.252. A quarter of LoCoMo's questions are adversarial and the official metric awards a point for declining them, so any mean over the full set measures the abstention rate before it measures the memory; the answerable and adversarial halves have to be reported apart. Second, the effect that survives is subject scope. Restricting retrieval to the person a question names scores 0.502 against 0.200 on the 446 adversarial questions (95% CI over conversations [+0.247, +0.347]) at no measurable cost on the 1,540 answerable ones (0.142 against 0.144, CI [-0.010, +0.006]). The mechanism is reader-independent: with the result limit lifted, the graph memory reaches 76% of the annotated evidence for answerable questions and 27% for adversarial ones, where the flat index reaches 99% and 96%. It is selectively blind, and the blindness falls where the trap turns are. Third, the graph itself is inert — empty it entirely and every number is byte-identical.