Your AI QA agent
has amnesia. We fixed it.

AI QA agents have amnesia. Every session they re-crawl, re-discover selectors that already passed, and re-attempt repairs that already succeeded. We gave our agents memory—but deliberately not the fluffy “let the LLM remember stuff” kind. Every recalled fact is evidence-backed: it cites a verified execution, a passing run, or a real repair. No evidence, no memory.

That constraint is the whole point. QA is not a general knowledge problem. A selector is useful because it held on the application. A repair is useful because it was followed by a pass. Without that provenance, a memory system can turn yesterday’s plausible guess into today’s confident failure.

What generic agent memory gets wrong

Most agent-memory designs optimize for recall: save a transcript, summarize it, embed it, retrieve it later. That is reasonable for a personal assistant. It is insufficient for test automation.

A test agent operates against a changing target. UI structure shifts. Scripts are edited. A one-off workaround becomes a liability. And a prior agent’s self-summary can be both fluent and ungrounded. The failure mode is not simply forgetting; it is remembering the wrong thing with no way to explain why it was trusted.

So Project Memory starts with an evidence-first contract. We deterministically extract facts from execution records and repairs. An LLM is not asked to invent the factual substrate.

• stable_locator: a Playwright locator that held across at least two passing runs;
• assertion_pattern: a verified expect(...) shape;
• failure_pattern: a classified locator, timeout, assertion, navigation, or execution failure;
• healing_outcome: a repair that was followed by a passing run;
• route_transition: observed navigation; and
• agent_episode: a distilled session goal, tool sequence, and outcome.

The difference is subtle but important. A proposed fix is not a healing outcome. A locator that looks sensible is not stable. An agent self-summary is not evidence on its own.

Hybrid retrieval for the way QA questions are asked

Some QA questions are lexical: “What locator passed for the Save button?” Others are semantic: “How did we recover from a settings-page failure after the UI changed?” A single retrieval technique is a poor fit for both.

We use PostgreSQL full-text search alongside pgvector cosine similarity in a dedicated 384-dimensional all-MiniLM-L6-v2 space. The result lists are fused with reciprocal rank fusion, using k = 60:

RRF(document) = Σ 1 / (60 + rank_in_result_list)

That gives us candidates from both exact language and related historical situations. But retrieval rank is not the final word. We re-rank by evidence support, recency, contradiction count, and outcome. A recent passing repair should outrank a weakly supported note that merely sounds similar.

Vectors are helpful infrastructure, not a reason to fail closed. If vector retrieval is unavailable, the system degrades safely to lexical and structured retrieval. It never silently returns an empty corpus and suggests there is no prior knowledge.

Stale does not mean deleted

Memory is not forever, but evidence is. A versioned recall score decays with staleness: when a script is edited after its last pass, the relevant learning ranks lower. Repeated contradictions suppress recall further.

We do not destroy the source evidence. We change its rank. That makes the system reversible and explainable: an engineer can see that a locator was once reliable, then see why it stopped being promoted.

Consolidation follows the same principle. Near-duplicate learnings merge deterministically, scoped by project and memory type; a semantic fact is never merged with an episode just because their text is similar. A contradicted memory cannot gain score by being a merge survivor. Dry-run mode shows proposed merges before they are applied.

Recall is useful only if it is safe

Reintroducing historical text into an agent prompt creates a trust boundary. We treat recalled content as untrusted input: it is bounded, sanitized, length-capped, and marked with an advisory="untrusted" delimiter. Prompt-control tokens are stripped, and agents are told to use the item as evidence context—never to execute instructions found inside it.

We also never embed raw transcripts, prompts, traces, or screenshots. And the project/team authorization boundary is enforced inside the database query before ranking, not as a filter applied after a global search. Recall is project-isolated by construction.

Why Claude or Codex alone cannot produce high-quality tests sustainably

Give Claude or Codex an app and either can produce a credible Playwright test. That is useful. It is not sustainable QA. Sustainable quality means knowing whether a test works on the real application, retaining the evidence when it does, noticing when that evidence goes stale, and preventing one project’s history from influencing another. Those are systems problems, not prompt-writing problems.

Claude and Codex are excellent collaborators inside that system. They can propose a test, explain a failure, compare a retrieved repair with the current page, and decide which follow-up check is worth running. But there are parts of the loop they cannot do at all—because those parts require external, deterministic proof rather than a model’s interpretation.

• They cannot prove that a locator works. A model can suggest getByRole(); only a real browser execution can show that it found the intended element and that the test passed.
• They cannot prove that a repair succeeded. A patch or self-summary is a hypothesis until a later run verifies it. That is why a healing_outcome requires a passing execution, not an agent’s confidence.
• They cannot make stale evidence current. A model may recognize a familiar pattern, but it cannot know whether an edited script or changed UI invalidated prior evidence without the versioned execution history and a fresh check.
• They cannot enforce the authorization boundary. Project isolation has to happen in the database query before retrieval and ranking. It cannot depend on a prompt asking a model to ignore information it should never have received.
• They cannot turn untrusted text into trustworthy instructions. Sanitization, length limits, prompt-control stripping, and the advisory="untrusted" boundary are system controls. They are not properties an LLM can guarantee about text it is asked to read.

This is not a limitation to work around with a bigger prompt. It is a division of responsibility: models reason over evidence; the system collects, scopes, validates, and ranks it. Keeping those jobs separate is what makes recalled knowledge inspectable instead of merely persuasive.

That is where QualityMax is already ahead. We are not asking an agent to remember more. We are giving it a verified execution loop and a memory layer with provenance, decay, contradiction handling, retrieval fallbacks, and prompt-safety controls. Claude or Codex can make the next decision better; QualityMax makes the decision repeatable, measurable, and accountable across the next hundred runs.

Measure before adding memory to the prompt

Memory can look impressive in a demo while quietly exposing stale or weakly grounded context in production. We do not skip the measurement step.

Writes and recall are guarded independently:

PROJECT_MEMORY_LEARNINGS_WRITE_ENABLED
PROJECT_MEMORY_LEARNINGS_RECALL_ENABLED

Shadow retrieval can run recall and log metrics without injecting anything into the agent prompt. That lets us measure embedding, database, and total latency before the retrieved content can influence an action.

We also evaluate against an offline replay set of labeled historical queries, comparing structured, lexical, vector, and hybrid retrieval. The harness measures Recall@5, MRR, grounding, and stale exposure against that curated replay set—not invented headline metrics.

From rediscovery to verified reuse

The payoff is not an agent that blindly repeats the past. It is an agent that can start from a verified prior result, inspect the live application, and decide whether the evidence still applies.

When a prior repair replaced a brittle CSS selector with a role locator and the next run passed, that outcome is worth remembering. When the next agent sees a related failure, it has a grounded direction instead of a cold start. If the UI or script has changed, staleness and contradiction controls keep that direction from being over-promoted.

That is the kind of memory QA agents need: not a general knowledge base, and not an unexamined transcript archive. A practical record of what worked, why we believe it worked, and when to stop trusting it.