EvergreenYada is an open-source agent harness for TypeScript backends. You plug in a model and register your APIs and MCP servers as tools. Yada runs the tool loop, asks before anything writes, and survives restarts.
We built a real product on Yada 0.3.0 and kept a list of everything that hurt. It came to twelve items. Fixing them became Yada 0.4.0. The cleanup afterwards turned up three flaky tests, a race condition that had shipped in every release since 0.1.0, and a CI check that blocked every pull request that didn't touch the docs.
This post walks through all of it, code included.
One rule shaped every fix. Yada is general-purpose, so nothing specific to the app was allowed in: no tool names, field names, URLs or business rules. Each fix had to be a capability any app could use. It needed a neutral name, and it stayed off by default unless the old behaviour was a bug.
Round 1: twelve issues from one app
The list fell into three groups: things we saw at runtime, capabilities we had to build around, and suspicions from reading the code that we didn't believe until a test failed.
A message sent as the run finishes gets lost
The chat widget lets users keep typing while the agent works. A message sent mid-run steers the run through POST .../messages. But if the run finished just before the message arrived, the server answered 409 and the widget showed "Could not send your message".
The server was right. The run had ended, so there was nothing left to steer. The widget just didn't know what to do next.
So the fix lives in the widget. On a 409, it fetches the run and replays any events it missed. If the run completed, it starts a new run with the same text:
const job = await options.client
.getRun(options.conversationId, input.id, stale.signal)
.catch(() => undefined);
if (!job || !this.isCurrent(stale) || !isFinished(job.status)) return false;
stale.abort();
const controller = new AbortController();
this.controller = controller;
for (const event of job.events)
if (event.sequence > this.cursor) {
this.cursor = event.sequence;
await this.showEvent(event, controller.signal);
}
if (job.status === "completed") {
this.appendMessage(text, "user");
this.prompt.value = "";
this.submit(text);
}
There's a second, subtler case. The server can accept a steering message that the model never reads, because the run ends before the model's next turn. The widget now remembers the IDs of the messages it steered, and when a run ends it checks which ones were consumed. Anything unread goes into a new run, or back into the input box if the run failed:
const text = job.steering
.filter((item) => !item.consumed && this.steered.has(item.id))
.map((item) => contentText(item.input))
.join("\n\n");
this.steered.clear();
if (!text) return;
if (job.status === "completed") this.submit(text);
else if (!this.prompt.value) this.prompt.value = text;
Retries reuse the same steering ID, so a flaky network can't send a message twice.
`unknown format "uuid"`, over and over
Every MCP tool schema with "format": "uuid" logged a warning when it was compiled, and the format wasn't actually checked. The noise came from Yada, not the MCP SDK. We compiled schemas with Ajv and never registered the standard formats. Now schemas are compiled in one place, with formats on and Ajv's logger off:
import { Ajv2020 } from "ajv/dist/2020.js";
import { Ajv } from "ajv";
import addFormats from "ajv-formats";
const withFormats = addFormats as unknown as <T>(ajv: T) => T;
export function compileSchema(schema: Record<string, unknown>) {
const options = { strict: false, logger: false } as const;
return withFormats(
schema["$schema"] === "https://json-schema.org/draft/2020-12/schema"
? new Ajv2020(options)
: new Ajv(options),
).compile(schema);
}
This changes behaviour, so it went into the upgrade notes. If a server's output fails a format check after a write has already run, the action is treated as possibly applied and goes to reconciliation. It isn't silently accepted.
Per-actor MCP connections
Worker profiles had a fixed tool list, and each MCP connection served a single actor. An app with many signed-in users, each with their own OAuth account on the MCP server, had to build its own routing layer.
createMcpActorPool makes that first-class. You get one tool catalog, a pooled OAuth connection per actor, idle eviction, and a shared connection for tools the app marks as public.
const pool = await createMcpActorPool({
config: {
name: "crm",
url,
readTools: ["search", "get_account"],
confirmTools: ["update_account"],
},
redirectUrl: "https://your-app.example/oauth/mcp/callback",
store: storage.oauth,
authorize: async (identity, tool) => canUse(identity, tool),
publicTools: ["search"],
idleMs: 300_000,
maxConnections: 100,
});
const profile = { /* ... */ tools: pool.tools };
// A signed-in route: send the browser to the returned URL.
const url = await pool.beginConnect(identity);
// The public callback route needs no session: the actor comes from the state.
const actor = await pool.completeConnect(new URL(request.url).searchParams);
That last line fills a real gap. OAuth callbacks often arrive without the app's session cookie, so the pool recovers the actor from the one-time state instead of making the app do it.
If an actor without a connection calls a tool that needs one, they get a clear not_connected error the model can explain to the user, not a generic failure.
Arguments the model shouldn't fill in
Some MCP write tools require an idempotency key. The model shouldn't make one up, because it would pick a new value on every retry, and that defeats the point.
hiddenArguments removes named arguments from the schema the model sees and fills them in when the tool runs:
await connectMcp(
{
name: "orders",
url,
confirmTools: ["place_order"],
hiddenArguments: { place_order: ["request_id"] },
},
access,
signal,
);
The value comes from the operation ID, so every retry of the same operation sends the same key. It's shaped like a UUID because that's what most servers expect:
export function injectArguments(
names: readonly string[] | undefined,
args: unknown,
id: string,
) {
const values = (names ?? []).map((name) => {
const hash = createHash("sha256").update(`${id}:${name}`).digest("hex");
return [
name,
`${hash.slice(0, 8)}-${hash.slice(8, 12)}-5${hash.slice(13, 16)}-8${hash.slice(17, 20)}-${hash.slice(20, 32)}`,
];
});
return { ...(args as object), ...Object.fromEntries(values) } as unknown;
}
Nothing here names a specific argument. It works for any tool and any field.
Remembered approvals you couldn't turn off
When users approve a write, they can tick "remember", and identical writes in the same run or conversation go through without asking again. Some apps need every write approved, and they had no way to say so. The widget always showed the checkbox and the server always accepted remember.
Now the handler and the worker both take a list of allowed scopes:
createHandler({ /* ... */ rememberApprovals: [] });
createWorker({ /* ... */ rememberApprovals: [] });
The worker option matters more than it looks. It decides which existing grants still apply, so turning remembering off also cancels grants made before you changed the setting. The widget hides the checkbox when remembering is off, so apps don't need CSS hacks or a proxy that rewrites requests.
Knowing when a run ends
The widget fired yada-run when a run started, but nothing when it finished. Now it also fires yada-run-end with the final status (completed, failed or cancelled), including for runs resumed after a page reload:
widget.addEventListener("yada-run-end", (event) => {
const { runId, status } = event.detail;
refreshSidebar(runId, status);
});
Errors you can tell apart
Tool call errors all collapsed into one kind of failure. A caller couldn't tell "the user's OAuth grant was revoked" from "the server is down" without poking at OAuth state.
ToolCallError now carries two things: what went wrong, and whether the request could have reached the server.
export type ToolCallErrorCode =
| "access_denied"
| "unauthorized"
| "not_connected"
| "oauth_failed"
| "session_expired"
| "request_failed"
| "invalid_arguments"
| "tool_failed"
| "invalid_output";
export type ToolCallDelivery = "not_sent" | "maybe_sent" | "received";
export class ToolCallError extends Error {
constructor(
message: string,
readonly code: ToolCallErrorCode,
readonly delivery: ToolCallDelivery,
) {
super(message);
}
}
For writes, delivery is what counts. "Never sent" is safe to retry. "May have arrived" needs reconciliation. A write that's known to be unsent now ends with a new undelivered action status, instead of looking like a possible double charge.
Erasing a user
eraseConversation throws if the conversation has an active run, so every app wrote its own loop to erase a user: page through conversations, skip the busy ones, remember to come back later. Now it's one call:
const { remaining } = await storage.retention.eraseActor(identity);
// remaining: conversation IDs with unfinished runs. Resolve them, then call again.
It erases the actor's conversations, long-term memory and stored MCP OAuth records. It carries on past conversations it can't erase yet, and it's safe to run again.
The four we had to prove first
The last four came from reading code, not from seeing failures, so each one started with a failing test.
Expired MCP sessions. reconnectingMcp only noticed a dead connection when the client closed or the tool list changed. A server that expires an HTTP session answers 404, and Yada kept using the dead session anyway. A test server that expires sessions confirmed it. A 404 now maps to a typed error, marked not_sent because the server rejected the session before running anything:
if (status === 404)
return new ToolCallError(message, "session_expired", "not_sent");
reconnectingMcp catches exactly that error, re-initializes, and retries once:
async function withSession<T>(
callSignal: AbortSignal,
operation: () => Promise<T>,
): Promise<T> {
await ready(callSignal);
const used = current;
try {
return await tracked(operation);
} catch (error) {
if (!(error instanceof ToolCallError) || error.code !== "session_expired")
throw error;
if (current === used) expired = true;
}
await ready(callSignal);
return tracked(operation);
}
The retry is only safe because the error says the request never ran. A call that may have run is never repeated, so a write can't happen twice.
No token revocation. disconnect() cleared local tokens but left them valid at the server. Now it also revokes them (RFC 7009) when the server's OAuth metadata advertises a revocation endpoint. That part is best effort. Local tokens get cleared either way.
OAuth store growth. consumePending wrote a row even when the state didn't match, so anyone could add rows by hitting the public callback with made-up state values. Now it reads first and returns without writing on a mismatch, on every storage driver:
async consumePending(id, pendingState) {
const record = await records.read(key(id));
if (token !== undefined) owned(record?.value, token);
if (state(record?.value).pending?.state !== pendingState)
return undefined;
return records.mutate(key(id), (value) => { /* ... */ });
}
storage.oauth.prune() and oauth.forget() clean up records that are no longer needed.
Quota overshoot. Cost quotas were checked when a run was submitted, but nothing was reserved. Submit ten runs at once and they all saw the same unused budget and all went through. Now each run reserves its maximum cost at submission and settles to the real cost when it ends. There's also an optional global budget across all actors:
const quota = {
costMicros: { limit: 5_000_000, windowMs: 86_400_000 },
scope: "actor",
globalCostMicros: { limit: 50_000_000, windowMs: 86_400_000 },
reserveCostMicros: (profileId) => profiles[profileId].limits.maxCostMicros,
reservationTtlMs: 3_600_000,
};
Reservations live with the quota record, so the guarantee holds across any number of workers.
Round 2: three flaky tests
After 0.4.0 shipped, two storage tests started failing in CI every so often, and a third showed up later. None of them was a product bug, though one led us to a real one.
The OAuth lease that was too short
This test starts several processes that compete for one OAuth record. One of them deliberately stalls while holding the lock, to prove the others wait instead of corrupting the record. Every contender used a 300 ms lease. On a busy machine, a healthy contender could take longer than 300 ms between taking the lock and writing. Its lease expired and it got fenced out. Under load, the test failed in 4 of 6 runs.
Only the stalled process needs the short lease, so everyone else now gets 30 seconds:
const store = createOAuthStore(
recordStore(db, config.encryptionKey),
stall ? 300 : 30_000,
);
After that it passed 8 runs out of 8 on Postgres and MySQL, with the machine's load average around 340.
The 1-second wait
The other two had the same cause: vi.waitFor gives up after one second by default. That's plenty on a quiet laptop and not enough on a busy CI runner. Locally, the worker resume test failed in 4 of 10 runs. Raising the timeout to 30 seconds doesn't slow down the passing case at all, because waitFor returns as soon as the check passes:
await vi.waitFor(
() => {
expect(provider.requests).toHaveLength(1);
},
{ timeout: 30_000 },
);
The job queue test also raised its worker lease from 100 ms to 1 second, for the same reason as the OAuth test. After that, the worker resume test passed 15 runs out of 15.
Making the database tests fast
The database tests were slow enough that a full run felt like a chore, so we looked into it. Each test creates a fresh database, and Postgres and MySQL spend most of that time syncing files to disk. Test data doesn't need to survive a crash, so the containers now keep their data directories in memory:
services:
postgres:
image: postgres:17
options: --tmpfs /var/lib/postgresql/data --health-cmd pg_isready
mysql:
image: mysql:8.4
options: --tmpfs /var/lib/mysql --health-cmd "mysqladmin ping -pyada-test"
Setup per test dropped from 1.9 s to 0.15 s on Postgres, and from 11 s to 0.3 s on MySQL. Locally it's the same flag:
docker run -d -e POSTGRES_PASSWORD=yada -p 5432:5432 \
--tmpfs /var/lib/postgresql/data postgres:17-alpine
Round 3: the flake that wasn't
While we were merging the flaky-test fixes, a different test failed once in CI:
FAIL tests/quota.test.ts > tenant quotas on postgres
> does not exceed the limit under concurrent submissions
AssertionError: expected [ 202, 202 ] to have a length of 3 but got 2
Eight submissions race against a limit of three. Three should get 202 and five should get 429. This time only two got through. A rerun passed, and it would have been easy to call it another flake and move on.
Instead we ran it 25 times in a loop. It failed 3 times, always the same way: two accepted, six refused, no server errors. The limit wasn't being exceeded, which is the failure you'd normally worry about. A slot was being used up by nobody.
Following the slot
Every quota check is a read-modify-write on one record, retried on conflict:
for (let attempt = 0; attempt < 20; attempt++) {
const record = await read(key);
const next = change(record?.value);
try {
await write(key, next.value, record?.revision ?? 0, next.active);
return next.result;
} catch (error) {
if (!isRetryable(error)) throw error;
}
// back off and try again
}
Updates are guarded by a revision number. The first write to a new record works differently. There's no row yet, so it inserts, ignores a duplicate-key conflict, and reads the row back to see whose insert won:
await db("yada_records")
.insert({ id, value: encrypted, revision: 1 /* ... */ })
.onConflict("id")
.ignore();
const raw = await db("yada_records").where({ id }).first();
if (rowSchema.safeParse(raw).data?.value !== encrypted)
throw new StorageConflict();
Values are encrypted with a random IV, so no two writers ever produce the same ciphertext. If the stored value isn't ours, we lost. That logic is sound, except for the gap between the insert and the read-back:
- Request A inserts the record with
used: 1. Its insert wins. - Request B reads the record A just inserted, increments it to
used: 2, and updates it. - A reads the row back, sees B's value, and concludes its insert lost.
- A retries and increments the counter again, to
used: 3.
A counted itself twice. The limit was hit after two real requests, so the third was refused. Any mutate that creates a record under concurrency could double-apply the same way, and this code had been there since 0.1.0.
The fix
Run the insert and the read-back in one transaction:
await db.transaction(async (tx) => {
await tx("yada_records")
.insert({ id, value: encrypted, revision: 1 /* ... */ })
.onConflict("id")
.ignore();
const raw = await tx("yada_records").where({ id }).first();
if (rowSchema.safeParse(raw).data?.value !== encrypted)
throw new StorageConflict();
});
If A's insert wins, nobody else can see the row, let alone update it, until A commits. So A's read-back always sees its own value. A writer whose insert was ignored waits for the winner to commit, reads the winner's value, and correctly retries.
A test that fails for the right reason
The first version of the regression test fired eight concurrent increments at a fresh key and checked the total. It passed with and without the fix, which made it useless. All eight writers read "no record" at the same moment and raced on the insert, so none of them could land in the gap after it.
Staggering the starts by a millisecond each puts later writers in the gap:
for (let round = 0; round < 40; round++) {
const key = JSON.stringify(["counter", String(round)]);
await Promise.all(
Array.from({ length: 8 }, async (_, index) => {
await delay(index);
await records.mutate(key, (value) => ({
value: ((value as number | undefined) ?? 0) + 1,
result: undefined,
}));
}),
);
expect((await records.read(key))?.value).toBe(8);
}
Without the fix, it failed in 5 of 5 runs on Postgres and 3 of 5 on MySQL, counting 9 instead of 8. With the fix, it passed 5 of 5 on both, and the original quota test passed 25 out of 25. The fix shipped in 0.5.0.
Round 4: the CI check that blocked everything
Required, but never run
The pull request with the flaky-test fixes passed every check and still couldn't merge. Branch protection required a check called pages, which builds the docs site. That workflow only runs when the site, the docs or the package files change:
on:
pull_request:
paths:
- "site/**"
- "docs/**"
- "package.json"
- "package-lock.json"
# and the scripts that build the site
A pull request that only touched tests never got a pages result, so GitHub waited for it forever. Running the workflow by hand didn't help either, because a manually triggered run doesn't count toward a pull request's checks.
It's an easy trap: put a path filter on a required check and some pull requests can never merge. We removed pages from the required list. It still runs on every pull request that touches the site, and a red result is still visible.
Four minutes for a docs build
While we were looking at pages, we noticed it was also slow: about four minutes for a static site. The step timings showed where the time went:
| Step | Time |
|---|---|
npm ci |
49 s |
npx playwright install --with-deps chromium |
130 s |
| Page checks | 38 s |
More than half the job was spent installing a browser for the accessibility and layout checks. Our first attempt switched to the smaller headless-only Chromium build. One run took 15 s for the install and looked like a win. The next run on main took 106 s.
The flag alone couldn't explain numbers like that. --with-deps installs Chromium's system packages through apt, and that's where both the time and the variance came from. GitHub's Ubuntu runners already have what headless Chromium needs, so we dropped it. The checks don't need npm install scripts either:
- run: npm ci --ignore-scripts
- run: npx playwright install --only-shell chromium
Three runs took 75 s, 90 s and 94 s, with the browser install down to 3 or 4 seconds. The first run on main took 104 s, down from 200 s.
What the first attempt taught us is that one fast run proves nothing. We only trusted the result after timing three.
What we took away
- Run the harness under a real app. Every runtime issue here came from a product, not a unit test. Scripted providers and quiet laptops don't hit the steering race or the expired session.
- Prove a suspicion before you fix it. Four issues came from reading code. Each got a failing test first, so we knew the fix actually changed something.
- Don't retry a flaky test away. The quota failure looked like noise. Running it 25 times turned it into a reproducible bug that had shipped in every release.
- Check that a regression test fails without the fix. Our first test for the race passed either way.
- Time things more than once. One fast CI run nearly convinced us the wrong change was the right one.
Yada is MIT licensed and on GitHub. To try it, run npx @edgaralochi/yada init my-agent.
