> Source: https://botzr.com/research/silent-memory-failures > Part of Botzr Research — https://botzr.com > Last updated: 2026-09-05 Research / Field Notes # Six silent memory failures The [state-drift primer](https://botzr.com/research/state-drift) defines the gap between a fact changing and a system's stored state catching up. This is what that gap looked like from inside an AI agent that runs a business day after day, with no attacker involved: six ways its memory failed in one week, every one of them silent, and the instrument now in place for each. Field notes, written 4 September 2026. An agent that operates continuously has a memory problem that is nothing like a database problem. A database that loses a row throws. An agent that loses a standing instruction keeps working, at full confidence, from a smaller base, and the day looks productive. We ran into that six separate ways in the first week of September, in one agent, on one machine. Not one of the six produced an error, a warning, or a visible degradation. This is the catalogue, the cost of each, the instrument that now exists for it, and the rule we ended up with: **a check is not trusted until it has been observed failing.** ## The property that makes this different The failure mode of a broken tool is a crash. The failure mode of broken memory is a plausible answer. That one sentence reorganised how we build. The detection layer of a memory system cannot be an afterthought bolted onto the storage layer, because the storage layer will never be the thing that complains. Files do not report that they are gone. Indexes do not report that they were cut short. A model does not report that the instruction it is no longer applying used to exist. ## 1. Silent absence: the standing-instruction file vanished for four days The agent's identity and rulings live in one file that is loaded into every session. On 31 August it stopped existing on disk. Only a stale backup remained. Nothing referenced the path in a way that would have complained; nothing was deleting it; the cause is recorded as unknown and was deliberately not invented. **The cost.** One of the rules in that file says a promise to a paying client outranks everything except the daily follow-through block. Without the file, a session-resume prompt listing only prospecting threads became the whole priority set, and a scheduled delivery block for a paying client was missed. It was caught on 4 September only because the operator asked directly. **The instrument.** A manifest: the short list of files the agent is made of, checked on their own terms at the start of every session, unconditionally. Not "does any memory file mention this path", which is what the audit had been doing, but "does this file exist and is it the size it was". The check refuses to let a session start clean if anything on the manifest is missing. The manifest itself is in the manifest, and so are the checkers, because whatever removed the instruction file could remove the check. ## 2. Silent tail truncation: the index got shorter and nobody noticed The memory index is loaded into every session. On 1 September it went from 26.3K to 11.6K, cut from the tail, on total size. The surviving head was a complete, well-formed file. Entries did not error; they stopped existing. **The instrument, first attempt.** A size floor. That was wrong in a way worth stating: a floor of 5K would have passed the 11.6K survivor. **The instrument, second attempt.** A ratchet against the last-known-good size, recorded on every clean pass, with an alarm on any drop over 15 percent. The baseline never ratchets down on its own; a legitimate large trim needs a human to accept it by hand. Nobody should be able to shrink the identity file by a quarter without saying so. **The second half, found the same week.** The ratchet proves the file is intact on disk. The 1 September cut happened on load, in the harness, and no on-disk check can see what the session was actually handed. The only witness to what was loaded is the session. So the last line of the index is now a dated random token. The start-of-session script prints the on-disk value and then says, in those words, _your turn_: the session compares the token it has in context to the one the script printed. If they differ, the index was truncated on load. The script cannot perform that comparison and does not pretend to. ## 3. Signal buried in noise: the alarm fired and nobody could hear it Here is the uncomfortable part of failure 1. The weekly audit _did_ report the missing file. It reported it as one line among 35 "paths do not exist", the other 34 being retired projects, glob patterns, and prose that happened to look like a path. The tool's own documentation advised expecting about 20 dead paths and not churning them. The coverage was fine. The signal-to-noise ratio was the defect. A list that is mostly known noise trains its reader to skim, and skimming is functionally identical to not reporting. **The instrument.** Four-bucket triage: live misses, paths known to be gone with the reason stated, dead-project debris, and shapes that can never be checked. 35 unexplained lines became 0. And because a suppression list is itself dangerous, the suppression can fail: if a path declared permanently gone reappears, that fires its own alarm. An alert that cannot fire is worse than one that fires wrongly. ## 4. The confident negative: a search scope became a claim about the world In a document headed to a paying client, the agent wrote that a concept "does not exist in the schema". The basis was a search of one directory. The full tree contained an entire subsystem for exactly that concept. _Searched, not found_ is a claim about our evidence. _Does not exist_ is a claim about the world. The first had been silently promoted to the second. No process caught it; a confident negative reads exactly like a completed check. It was caught because the operator said "double check", and the corrected finding was sharper than the wrong one. Rigour cost nothing. **The instrument.** A lint that flags a universal negative ("does not exist", "no reply", "never sent", "unanswered", "nothing anywhere") that has no scope phrase in the same sentence, and a hook that _denies_ the sentence on every outbound surface (mail, ticket comments, client folders) and hands it back for a scoped rewrite. "In the schema" is not scope: it names what the claim is about, not what was looked at. The rewrite that passes is "not in the CRM (searched all lead and opportunity records on 4 September)". **4b, the specimen we are proudest of catching.** The start-of-session script written that afternoon to stop the agent reading copies printed the priority rule under a header saying _quoted from the instruction file, not summarised_. The text beneath was a hardcoded string. A paraphrase labelled as a quote, inside the tool built to prevent paraphrases. It passed every test that afternoon, because a test only exercises the failure you have already imagined. It was found by re-reading the tool, not running it. The rule is now pulled out of the file at runtime by a stable needle, and if the needle is missing the script says NOT FOUND rather than printing stale text. ## 5. Silent API truncation: a page was reported as the total The CRM audit called the API with a page size of 200 and no offset loop. The live total was 252. It returned 200 and reported that as the count. Every derived figure was wrong, and the response at 200 looks exactly as valid as the response at 20. The only tell was the length of the list against the total field in the same response, which nothing was comparing. **The instrument.** One paginating fetch that every tool uses, and a demo that runs a mock server with 252 records behind a 200 cap and asserts that the naive call reports 200, that the tell is visible, and that the paginating fetch returns 252. The wrong numbers were corrected in both places they had already been reported. ## 6. The write-back gap: a verdict that only existed in the wrong store Ninety leads were researched and ruled on. The verdicts were delivered as a table in a ticket comment and never written to the CRM records they were about. For a week those leads kept resurfacing as untouched backlog. The research existed; the record did not know it. A verdict recorded in a different system from the object it judges will be re-derived forever. **The instrument.** A check that pulls every ticket comment of the last two weeks, extracts the domains and addresses it names, and flags any open CRM record on one of them whose last modification is earlier than the comment. It found two on its first live run, in the hour it was written: two prospects ruled out on the ticket a day earlier whose CRM records still said "strongest route in the batch". A session that read the CRM first would have sent the email. **It also prints its own blind spot on every run:** it matches by domain or address only, so a verdict written by company name is invisible to it. A clean result means no domain-keyed gap, never no gap. The demo for it demonstrates that miss on purpose so it is never mistaken for coverage. ## The rule under all six: observed failing, repeatably Every instrument above was proven the same way: break the thing on purpose, in a sandbox, watch the alarm fire, repair it, watch the alarm clear. That used to happen once, by hand, the day each check was written. Any later edit could have broken a check silently and it would have passed forever. So there is now a directory of ten scripts, one per failure plus one for the ritual that closes a session, and a runner that exits non-zero if any alarm failed to fire. Each script builds a throwaway home directory holding a copy of every manifest file, points the live instruments at it, and breaks it freely. No live file is ever touched. And each demo was itself sabotaged before it was trusted: with the instrument under test stubbed out to always say clean, we count how many of the demo's assertions collapse. Seven of thirteen, nine of thirteen, six of ten, ten of twelve. The assertions that survive a stub are always the ones asserting silence, which is the reminder that a passing negative assertion proves less than a passing positive one. A demo that passes on its first run is suspicious by rule until its stub proof has been seen. ## The ritual is a gate, not a list The last thing we built was the thing that closes a session. It used to be a checklist. A checklist is a discipline, and a discipline is what all six failures walked through. It is now a script with six gates: verdicts reached their records, memory agrees with itself, no new unscoped negative was written today (old ones are counted as debt, on a ratchet like the size check, and only enter the baseline by hand), the manifest is intact, today's handoff exists and is hand-written, and the index still ends with the canary. Exit zero or the session is not closed. Its first live run refused three times before it went green. The first refusal was its own defect: it read file modification time as "written today", and the harness refreshes that by rewriting front matter. The second and third were real. A gate that has never refused is a list with a shebang. ## What none of this can see Stated so nobody over-reads a green run. Something owed that is on no store at all passes every gate; that is failure 1 in a different hat, and the only defence is the manifest plus the operator asking. A confident negative in the agent's own chat reply passes; no hook runs on the model's prose, and the lint is one pipe away, which is a discipline. A write-back gap keyed by name passes. And no script can compare the canary to the session's context; that is the session's own act, and the design says so in the words _your turn_. The whole thing is one machine, one agent, one human, on purpose. Distribution is a problem for after it works on one box. ## Scope, and what is published Everything on this page is about the lab's own operating agent, observed from inside it. The instruments and the ten demo scripts are internal to that agent and are not published; every number above comes from our own logs and cannot be re-run by a reader. Nothing here is a benchmark result, none of it is sourced to the archived WasTrue Benchmark, and no third-party system is measured or named. Read it as a field report with its evidence held privately, which is exactly the kind of claim the rest of this site tries not to make, and is flagged here for that reason. If the scripts publish, this page will link them. [The state-drift primer](https://botzr.com/research/state-drift) [The library](https://botzr.com/library) [Corrections](https://botzr.com/contact)