Meta-engineering AI systems: from traces to better memory

Viacheslav Dubrov

2026-09-12 · Updated 2026-09-12 · 22 min · AI Agents

This article opens Meta-Engineering AI Systems, a six-part guide to closed-loop AI engineering: using agents to investigate failures, test changes, and use the results to improve AI systems.

An AI application’s behavior depends on more than its model. Prompts, retrieved information, stored memory, tools, and application code all affect what happens. When an answer is wrong, several changes may look plausible. The engineering problem is deciding which change helps, what it costs, and what else it breaks.

I use meta-engineering here for designing the improvement process itself: what evidence the agent sees, what it may change, how we test its proposals, and who decides whether to adopt them. Closed-loop describes how that process learns from its experiments. Each result informs what we try or use next, including when a proposed change fails.

We can automate that work in stages. A person may specify the exact changes to test, define a set of permitted alternatives for a search algorithm, or delegate the next proposal to an LLM agent. The runner tests each candidate and preserves evidence for the next decision. This series focuses on bringing agents into that process while keeping execution, evaluation, and adoption under explicit control.

The question running through all six parts is how much of that work we can delegate to an agent while keeping the evidence and the decision to adopt a change trustworthy. We will start by mapping the whole system, then build a small working version around agent memory.

What closes the loop

There are two different loops to keep in view. Inside an agent application, the runtime loop chooses an action, calls a tool, observes the result, and continues the user’s current task. The improvement loop works across versions of that application. It investigates behavior from completed runs and tests changes intended to make later runs better.

The improvement loop starts with evidence: an incorrect answer, an unwanted state change, excessive cost, or another observable failure. A trace records the steps behind that outcome. The proposer chooses a candidate, a specific version of a change. A runner executes the candidate on defined tasks, and an evaluator checks the resulting behavior against the requirements. In this article’s live experiment, an LLM agent is the proposer:

Evaluation informs a decision; it does not authorize itself. Someone must decide whether the measured benefit justifies adopting the change, including its effects on cost, permissions, and other required behavior. In this series we begin with a human reviewer. The proposer can suggest a change, but it cannot rewrite the checks, increase its own budget, or grant itself approval.

There are two return paths. The experiment history feeds later proposals, including when a candidate is rejected. If a candidate is approved and adopted, its behavior supplies new evidence from use. Keeping a change reversible matters when that evidence contradicts the original experiment. Recording a score is only one step; the loop closes when the evidence changes what we try or use next.

The thing being improved is the target. It might be a memory policy, a retrieval pipeline, tool-selection code, or the program that assembles a model’s context. Model training is another possible intervention. The loop can work with fixed model weights: the agent may improve the surrounding program without training a model. This first demo already includes the outer LLM agent. Later parts strengthen the evaluator and test whether agentic search is worth its cost compared with simpler methods.

Choose how much of the search to delegate

Suppose you already know what to try: compare two models in an extraction step, or test memory thresholds of 0.6, 0.7, and 0.8. You can supply those choices yourself and automate execution, scoring, and reporting. After reading the results, you choose the next experiment. The feedback loop works even though the system did not invent the candidates.

I use four working modes to make that division visible. The figure shows one way to delegate progressively more experiment design: start with exact model choices, let the system search settings and approved components, or ask an agent to investigate failures and choose the next experiment. The human specifies less of each attempt while still setting the rules for the search.

The mode names describe practical arrangements for this series, rather than an industry-standard autonomy scale. What may change and who chooses the change are separate decisions. Optuna’s search-space examples combine model choice with parameter ranges. Scikit-learn’s pipeline search compares alternative components through ordinary grid search. Replacing a component therefore does not, by itself, make the process more agentic. The figure’s widths illustrate the division of work in these examples; they are not measured shares of control or effort.

Agent-led investigation describes how the next attempt is chosen. The agent reads failures and earlier results, proposes a change, and adapts after the test. This follows the distinction in Anthropic’s workflow and agent patterns: a predefined process can automate execution, while an agent directs its next steps from feedback. Hybrid methods are possible too: DSPy’s MIPROv2 uses a model to propose instructions and Bayesian optimization to search their combinations. A fixed model comparison needs no LLM proposer; its results enter the feedback loop when they guide the next experiment or system version.

Proposal, execution, and adoption permissions remain separate. An agent may draft a change for approval before each run, or test permitted changes unattended while a person reviews adoption. An approved component list still needs compatible implementations; permission to select an adapter does not include permission to write one. In every mode, the agent stays within the allowed changes, fixed checks, and budget, and cannot grant itself deployment approval. Delegating more work also leaves room for supervision: Anthropic’s study of deployed agents describes users shifting from approving individual actions to monitoring and intervening.

All modes can use the same experiment ledger: a record of every attempted change. Keep the candidate and its parent, the exact change and who supplied it, the test and environment versions, the result, cost, and any failure or rejection. That history lets a human hand the next proposal to an agent—or take it back—without losing the evidence. Scores from different test versions still need to be distinguished.

This demo combines configuration search with agent-led investigation. We define five settings, the tests, the budget, and the selection rule. The agent chooses values and the next hypothesis from feedback; Python runs the tests automatically. Human release review remains separate. The repository also accepts a candidate JSON supplied by a person through lab evaluate. Model sweeps, component adapters, and agent-written code are broader design options for later parts, not implemented modes of this memory demo.

Why examine this now

The feedback principle is familiar engineering. The recent work that interests me puts coding agents inside the investigation and experiment process. It gives us concrete implementations to examine, rather than asking us to assume that autonomous improvement will work.

Karpathy’s autoresearch specifies a compact experiment: modify a training program, run it under a fixed training-time budget, inspect the validation result, and record whether the attempt was kept, discarded, or crashed. The instructions separate the editable training file from the fixed evaluation code. That separation makes the proposed work and its success criterion inspectable.

Meta-Harness, a March 2026 preprint, studies a different target: the code that controls what information an LLM application stores, retrieves, and presents to its model. Its proposer can inspect earlier candidates’ source, scores, and execution traces through a filesystem. Experiment history becomes working material for the next investigation.

These projects motivate the series’ engineering question: once an agent can investigate and propose changes, what must the surrounding system do to make those experiments useful? More attempts alone do not establish better decisions. A weak evaluator can reward a harmful change, and a search process can repeatedly exploit that weakness. We will test the value of agentic proposals against simpler alternatives, rather than assume an advantage.

How the six parts build one system

The articles will develop one cumulative companion project. Each part takes a question left open by the previous experiment and adds the mechanism needed to investigate it.

Privacy checks and human review belong in the first version. Part 5 strengthens that protection as the proposer gains capabilities. Likewise, Part 6 investigates a possible use of the accumulated evidence; each earlier part must remain useful without training a new model.

Give the outer agent a small, inspectable target

Imagine an assistant that remembers Ada’s account details. On January 1, it learns that she lives in Berlin. On January 3, it learns about a move to Paris effective January 10. Asked for her city on January 5, it answers Paris.

The memory contains both facts. The retrieval rule prefers the more recently received one without checking when it becomes valid. That is a concrete failure an improvement agent can investigate.

The target is deliberately just a memory tool, not a complete assistant. It receives prepared facts, stores or rejects them, and retrieves a value for a question. Its writer, date handling, retrieval, and answer selection are ordinary Python. We leave conversation extraction and natural-language answer generation outside this experiment so we can identify what a rule change actually did.

The LLM is in the outer loop of the Python experiment. The agent receives test results, decides what settings to change, and gets the next result back. The recorded campaigns use an actual model to choose those changes.

This division lets us test the engineering loop without introducing a second source of model behavior inside the memory tool. In a larger application, both loops could use models. Here, one LLM is enough to make the improvement process agentic. Another LLM can fill the same proposer role; its proposals must follow the same schema and pass the same checks.

Inspect the experiment

The companion repository on GitHub contains the outer agent, the memory tool, tests, and compact reports from three real LLM campaigns. Complete requests, responses, and traces are available in a checksummed evidence archive. The experiment report follows the proposed changes, their measured results, and each decision to stop. Later in the article we will use the repository to run a fresh campaign.

Understand one test before reading the score

A scenario is one complete test story. An event is one action sent to the memory tool: write, query, or delete. The future-move scenario has four events:

1. Save city = Berlin, effective January 1.
2. Save city = Paris, received January 3 but effective January 10.
3. Ask for the city on January 5. Expect Berlin.
4. Ask for the city on January 12. Expect Paris.

The inputs are already structured. For example, the Paris fact identifies the owner (north workspace, user ada), subject (account), property (city), value (Paris), and effective date. The strings Berlin and Paris come from the test data, not a model generating memories from conversation.

Confidence is a supplied score used to decide whether to store a proposed fact. The writer compares it with min_confidence. A delivery = courier proposal scored 0.4 is rejected at the baseline threshold of 0.7, but stored at a threshold of 0.2. The test author supplies the score; it is not a measured probability that the proposal is correct. We evaluate how the storage rule handles those scores, not whether a model can estimate them reliably.

Each scenario starts with an empty Python list of memory records. Accepted updates close the previous value’s validity period and append a new record. After the two city writes, the records describe:

The answer function returns the first retrieved record matching the requested property. The evaluator compares that value with the expected answer. A complete scenario passes only if every answer is correct and every applicable data-rule check passes. Deletion, owner isolation, and prohibited writes have explicit checks alongside answer quality.

The tests supply owner IDs directly; this demo has no authentication system. A real service must get those IDs from the authenticated request. Rejecting an input explicitly labelled as an instruction also does not demonstrate detection of instructions hidden in ordinary text.

Give memory a schema, too

The facts need rules of their own. I use Schema-Guided Agent Memory (SGAM) for the pattern in which schemas govern stored state and its lifecycle. Here we demonstrate a small part of it: typed facts, ownership, source references, validity intervals, and deletion. The later SGR schema will govern what the outer agent can propose; this memory schema governs what the tool can store.

The Paris record after the second write contains:

{
  "tenant": "north",
  "user": "ada",
  "entity": "account",
  "key": "city",
  "value": "Paris",
  "confidence": 0.95,
  "source": "user",
  "valid_from": "2026-01-10",
  "schema_version": 1,
  "memory_type": "fact",
  "id": "m002",
  "source_event_id": "future-move:event-2",
  "observed_at": "2026-01-03",
  "valid_to": null,
  "supersedes_memory_id": "m001"
}

source_event_id points to the event that supplied Paris. supersedes_memory_id links Paris to the Berlin record, m001. schema_version: 1 identifies the record format; it does not mean the program can migrate old data automatically.

MemoryRecord and Memory.write() enforce that format. Missing source references, invalid dates, and malformed fields are rejected before stored history changes. A closed interval must end after it starts. Berlin can end on January 10 while Paris starts that day; neither record has an empty interval.

Suppose the next write says Rome, also effective January 10. The writer rejects that conflict and leaves Paris unchanged. A second Paris fact for the same date simply reuses the record. An update with an earlier effective date is also rejected: this small writer does not reconstruct late-arriving history. These are fixed policies the outer agent cannot change. Its deduplicate setting controls repeated confirmations with a later effective date.

The baseline still deliberately ignores subject and date filters when reading. It stores valid records but can choose the wrong one. That is the fault we give the agent to repair. Owner isolation applies to every configuration, and deletion removes all versions of the requested property within that owner’s subject.

This is an in-memory demonstration of those SGAM rules. It has no durable database, retention service, or migration system. Separate writer regression tests check rejected writes and interval boundaries; they are not extra scenarios in the 20-story campaign score.

Let the agent choose a change

A campaign is one attempt to improve the original configuration, starting with fresh agent history. An iteration is one call in which the agent proposes a change or chooses to stop. The next iteration receives feedback from the earlier ones.

The campaign begins by running only the baseline. It passes 13 of 20 scenarios. We then give the agent:

• the current settings and their meanings;
• the baseline’s measurements;
• traces from failing scenarios, rejected writes, and duplicate confirmations;
• the changes it is allowed to propose;
• earlier proposals and their results, when there are any.

The first request contains no ready-made improved configurations. The repository also has four hand-prepared configurations for explaining memory mechanics, but the live agent does not start with those answers.

The agent can change five settings in the existing tool:

A proposal can change at most two settings. Numeric values have bounds: confidence from 0 to 1, and top_k from 1 to 8. The contract records the tool’s allowed configuration changes; the agent schema and validator add the proposal rules.

The agent has no shell or filesystem tools in this process. It cannot edit Python, expected answers, owner isolation, deletion behavior, the memory schema, conflict handling, scoring, budgets, or release authority. Its output is data that Python may accept or reject. This is a small configuration experiment, not a sandbox for arbitrary code written by an agent.

Make every proposal an SGR decision record

I use Schema-Guided Reasoning to make the decision inspectable. Every model response has the same fields:

The OpenAI Responses request uses a strict JSON Schema generated from Pydantic models. Objects reject extra fields, and each patch field is required but may be null, meaning “leave this setting unchanged.” Structured Outputs constrains the response shape. Python still checks that references exist, values are allowed, and the proposed configuration has not already been tested.

The schema does not prove the hypothesis or expose the model’s internal reasoning. These are concise decision records that we can check against evidence.

In the first recorded iteration, the agent identified cross-subject retrieval and facts being returned outside their validity interval. Its actual response proposed this patch:

{
    "min_confidence": null,
    "filter_entity": true,
    "time_aware": true,
    "deduplicate": null,
    "top_k": null
}

Those two settings were chosen by the model. The runner supplied the parent ID and assigned the candidate ID; the model could not redirect the change to an arbitrary parent or file.

Follow the change through Python

Both the baseline and the new configuration store the same Berlin and Paris records. The proposed change affects which records retrieval may consider.

In Memory.query(), these switches activate two filters:

if self.config.filter_entity:
    eligible = [r for r in eligible if r["entity"] == event["entity"]]
if self.config.time_aware:
    eligible = [r for r in eligible if valid_at(r, event["as_of"])]

The validity check includes the start date and excludes the end date:

def valid_at(item: dict, at: str) -> bool:
    return item["valid_from"] <= at and (
        item["valid_to"] is None or at < item["valid_to"]
    )

The dates use YYYY-MM-DD, so their string order matches their calendar order. For the January 5 question, Paris is excluded because it becomes valid on January 10. Berlin remains available. For the January 12 question, Paris is valid and Berlin is historical.

Python runs the proposed configuration on all 20 stories. This first change improves success from 13/20 to 19/20, with all implemented hard checks passing. It changes two filters together, so the suite-wide gain measures their combined effect. The city trace identifies what the date filter did in this particular case.

The runner selects this configuration as the parent for the next experiment. That selection does not deploy it. The model gets the measured result, the selected settings, and the remaining evidence back in the next request.

The next decision must use the result

Here is the complete first campaign. Each row is a real model response, not a prewritten step in a demonstration script:

The second response cited the duplicate-confirmation scenario. That story writes language = German three times. Deduplication keeps one record instead of three while preserving the answer. Across the suite, mean stored records fell from 1.70 to 1.60 with success still at 19/20.

The third proposal addressed a different failure. A useful language preference had a supplied confidence of 0.6, below the current 0.7 threshold. Lowering the threshold to 0.6 admitted it while still rejecting the uncertain courier proposal scored 0.4. Success reached 20/20; mean stored records became 1.65, because memory now kept the additional useful fact.

On the fourth call, the agent chose to stop: it had no further supported change to propose from the supplied evidence. That is a model decision, not proof that the configuration is globally optimal.

The next request contains the current results and previous decisions. After seeing that its retrieval fix helped, the agent moved on to storage and write admission. The runner did not supply those next patches or a prewritten sequence of steps.

The selection rule is fixed: require all hard checks to pass, prefer better complete-scenario success, and at equal success prefer fewer stored records. A worse result leaves the previous parent in place. The tests also exercise that rejection path with a scripted regressing proposal; it was not a measured quality regression in these three live campaigns.

A separate development run contains a real validation rejection: the agent cited agent-01, a candidate ID, as though it were a scenario. The response matched the JSON schema, but its reference did not identify a supplied event, so Python rejected it before evaluation. The controller returns the rejected response and specific validation error as feedback; offline tests check that behavior. The three campaigns reported below had no validation failures.

Repeat the search campaign, not the deterministic test

The same memory configuration and inputs produce the same answers. Repeating that evaluation adds no answer-quality evidence. The runner checks a new configuration once and reuses verified parent results for comparisons.

The outer agent can choose different proposals, so we ran three independent campaigns. Each began at the same 13/20 baseline, received fresh history, and had at most four model decisions. No campaign received the previous campaign’s discoveries.

All three campaigns chose the same sequence: enable both retrieval filters, deduplicate confirmations, lower the confidence threshold to 0.6, then stop.

There were 12 model calls: nine proposals that Python evaluated, followed by three stop decisions. Each campaign executed the baseline plus three new configurations on 20 stories: 80 memory-tool executions per campaign. The 20 stories remained the same throughout.

All three selected the same final settings. This is a small observation about one model, prompt, target, and public test suite. It does not estimate how reliably the optimizer will improve unfamiliar systems. A stronger comparison belongs later in the series: repeated campaigns under matched budgets, competing proposal methods, and tests the proposer cannot inspect.

Read quality and cost at the right level

The experiment has two different costs. Running the memory tool uses local CPU time and makes no model calls. Running the outer proposer uses input and output tokens. A zero provider-cost field in a memory evaluation report describes only the inner tool; it is not the cost of the campaign.

For the recorded study on September 11, 2026, we used GPT-5.6 Luna (gpt-5.6-luna), reasoning effort low, the saved SGR schema and instructions, and a maximum of 4,096 output tokens per call. The program disabled automatic SDK retries and set a 60-second request timeout. Exact requests, returned model IDs, usage, and call durations are retained.

The token-based cost estimate for the 12 model calls is USD 0.039177, against a configured USD 0.50 budget. The calculation uses the published model rates, includes the cache-write surcharge reported in usage, and ignores cache-read discounts. It is a conservative estimate from recorded usage, not an invoice. Hardware and development time are outside that figure.

Before each call, the runner reserves an upper estimate based on bounded input size and maximum output tokens. If the remaining budget cannot cover that reservation, it stops. Failed or interrupted calls remain in the record; when usage is unavailable, their reserved amount is retained rather than treated as zero.

For answer quality, 20/20 means every answer and applicable hard check passed on these 20 prepared stories. It does not mean production-ready memory. All stories are public development cases; selected traces and suite feedback are available to the proposer. They cover updates, historical questions, uncertainty, duplicates, distractors, owner separation, deletion, and disallowed writes. The search, evaluation, and adversarial labels organize the original suite; they do not make any of these cases a hidden test set.

This small suite draws ideas from LongMemEval, MemoryAgentBench, VehicleMemBench, and GateMem. The inputs and scoring here are our own; the results do not reproduce those benchmarks.

You can also compare four hand-prepared configurations in the repository. They illustrate why accepting more facts can hurt answers and how storage can improve without higher answer quality. They also show the human-specified side of the same experiment machinery: the author supplies the candidates and Python evaluates them. They are useful teaching comparisons, not evidence that agentic search beats a competent human or another search method.

Run a fresh agent campaign

To let the agent choose new proposals, clone the companion and download the recorded evidence:

git clone --branch v0.2.2 --depth 1 https://github.com/slavadubrov/meta-engineering-ai-lab.git
cd meta-engineering-ai-lab
uv sync --frozen
uv run --frozen python scripts/fetch_evidence.py
uv run --frozen python -m lab verify artifacts/agent-study-03 --source

The download checks SHA-256 and restores the full recordings under artifacts/. It makes no model calls. If you already downloaded the evidence, skip that step and run the verification command.

Save an OPENAI_API_KEY in a local .env.local file. The repository ignores that file. Then run:

uv run --frozen --env-file .env.local python -m lab campaign \
  --live --campaigns 3 --iterations 4 --budget-usd 0.50 \
  --output artifacts/my-agent-study

--live explicitly enables provider calls. Having a key in the environment does not turn the offline memory commands or browser into a live agent. --campaigns 3 creates three independent agent histories; --iterations 4 limits decisions within each one. Use a new output directory for each study: existing evidence is never overwritten.

Open artifacts/my-agent-study/report.md for the campaign outcomes and links to each request, response, and Python evaluation. Compare the proposed changes with the recorded campaign above: the agent may choose differently even though the memory tests are deterministic.

The README maps the code and commands. The frozen campaign report and archive checksum and example index let a reader inspect this article’s evidence directly.

Keep experiment history separate from user memory

The memory tool stores Ada’s city and language. The experiment history stores what the agent saw, the proposed patch, any rejection, the resulting measurements, and which configuration became the next parent. These stores serve different purposes.

Each iteration keeps the exact request with instructions and schema, the provider response, usage and timing, validation outcome, and any completed evaluation. Failed proposals stay visible. File hashes bind the saved evidence to the implementation and inputs used for the experiment.

These are synthetic facts, so the public record can retain full states. Real traces need a separate retention policy: deleting a fact from the memory tool does not erase earlier copies from experiment logs or backups. The demo’s deletion test checks the tool’s records and later reads, not erasure from every possible store.

What the next article needs to challenge

We now have an agent inside the improvement loop. It proposes a change, receives a measured result, chooses the next change, and can decide to stop. Invalid proposals follow a separate rejection path and remain in the record. Its authority is limited to configuration; testing and release review remain separate responsibilities.

The next risk is the evaluator itself. If success rewards keeping every fact, the agent may learn to keep guesses. If a suite asks only about the current city, it may miss a change that destroys useful history. A more capable proposer can exploit weak measurements more efficiently.

Part 2, Make It Scorable Before You Make It Autonomous, asks how to test those measurements before giving the proposer more freedom. The first campaign’s 20/20 is the starting point for that investigation.