Goldie Blocks

Why the interview agent moved off gpt-5-nano, why reasoning effort was dropped to "none" rather than tuned, and what the ping-instrumented eval A/B actually showed

Decision

Amended by owner ruling, 2026-09-25: the model is now gpt-6-luna, still at reasoningEffort: "none". A one-off probe on the dev deployment ran both models through Goldie's settings on one tool call: both called the tool and answered correctly; gpt-6-luna took about 2.0 s against 4.6 s (one sample, indicative only). OpenAI documents Chat Completions function calling for GPT-6 Luna only at reasoning_effort: "none", which is this setting. The record below is the original decision and stands for its reasoning.

The agent's model is gpt-5.6-luna at reasoningEffort: "none". This replaces gpt-5-nano, which had been run at "low" and briefly at "minimal". The interview modules this was first set on were deleted with the interview surface; the ruling carried over and lives on the one agent definition, convex/goldie/agent.ts.

No dependency upgrade was required or performed. gpt-5.6-luna is callable on the already-installed @ai-sdk/openai@3.0.53 / ai@6.0.168 pair — the package's typed model-id union does not list it, but the union's trailing (string & {}) member accepts any string, so an unlisted model id type-checks and calls through to OpenAI unchanged.

Context

Interview turns felt slow, with no way to tell whether the cause was prompt size, model reasoning, or an uncached prompt. A ping action was added: a context-free "reply with exactly: OK" round trip against the same agent, reporting total/model latency, token counts, cache-read tokens, and reasoning tokens — isolating model/network latency from agenda assembly, writeModule, and audit-insert overhead.

The ping data ruled out the two suspects that looked likeliest going in:

Prompt size / caching was not the driver. OpenAI's automatic prefix cache was already covering ~96% of the ~19KB system-prompt-plus-tool-schema prefix (4352 of 4552 input tokens cached on a typical ping). Cutting prompt size further would not have moved the needle much.
Reasoning tokens were the driver. At reasoningEffort: "low", nano spent 250–400+ hidden reasoning tokens on a call whose only visible output was the word "OK" — output that should cost single digits of tokens. That reasoning spend, not prompt size, tracked directly with both the latency and its turn-to-turn variance (pings ranged 1.1–10s at "low").

The nano reasoning-effort A/B

Dropping nano's reasoningEffort from "low" to "minimal" was tested against a frozen eval script, 3 runs per setting:

Setting	Pass rate	Median latency	Max latency	Invented writes
nano / `"low"`	1/3	3918–5484ms	6193–8122ms	2 of 3 runs
nano / `"minimal"`	1/3	2225–2576ms	2511–3007ms	2 of 3 runs

Latency roughly halved with no reasoning tokens spent (confirmed via ping: reasoningTokens: 0 at "minimal"), but the same invented-write failure recurred at the same rate with the same fields (has_grandchildren, partner_marriage_will_continues_yes_no) regardless of reasoning effort. That ruled out reasoning effort as the cause of the invention bug — it was never protecting against it, so lowering it cost nothing on that axis.

Why the invention bug was model capability, not prompt wording

The system prompt already stated the never-invent rule about as explicitly as prompt engineering gets: "Never invent, default, or assume a value... A field you have not asked about yet is null. Always" and "Guessed values are worse than missing ones." nano violated this instruction anyway, specifically on plausible adjacent fields sitting in the same turn's schema as fields it was legitimately asked about (a grandchildren field after a children question; a marriage-continuation field after a marriage question) — a known weak spot for smaller/cheaper models under structured-output schema pressure, regardless of how unambiguous the instruction is.

If the fix were prompt clarity, a different model running the same prompt would show the same failure. It did not.

The luna result

Switching to gpt-5.6-luna at reasoningEffort: "none" (Luna's documented latency baseline) and re-running the same eval, same frozen script, 6 total runs:

Run	Result	Median latency	Max latency
1	PASS	2090ms	2178ms
2	PASS	2261ms	3191ms
3	PASS	1837ms	2058ms
4	PASS	1835ms	2238ms
5	PASS	2095ms	2555ms
6	PASS	1824ms	1996ms

6/6 PASS, zero invented writes in any run, median latency tightly clustered 1824–2261ms — faster than nano at either reasoning effort, and the invention bug that recurred in 4 of 6 nano runs did not appear once. Same prompt, same schema, same never-invent instruction; only the model changed.

Verification method: `pingModel`, isolated from the real agent

Before switching the real agent's model, the ping gained a second action, pingModel, which takes an owner-supplied model id and calls the plain ai SDK's generateText directly — no Agent, no thread, no agent wiring at all. This let gpt-5.6-luna be confirmed callable and functioning (valid "OK" reply, real usage data) with zero risk to the live interview flow, before touching INTERVIEW_MODEL_ID. It remains in the codebase as a lightweight, agent-independent model-swap diagnostic.

Why no SDK upgrade

npm view showed newer majors available (@ai-sdk/openai@4.0.26, ai@7.0.45), which might be expected to matter for a GPT-5.6-era model. They are not currently installable together with the rest of the stack:

@convex-dev/agent@0.6.4 (the latest published version) peer-depends on ai@^6.0.35 — it does not declare support for ai@7.x.
ai@6.x's own dependency on @ai-sdk/provider@3.0.8 is incompatible with @ai-sdk/openai@4.x's required @ai-sdk/provider@4.0.4 — the provider interface versions do not line up across that boundary either.

So the ai / @ai-sdk/openai / @convex-dev/agent stack is locked together at its current majors until @convex-dev/agent ships an ai@7-compatible release. This is not a blocker for the model switch itself — gpt-5.6-luna calls correctly on the installed versions — but it does mean GPT-5.6-era features that need the newer SDK surface (explicit cache breakpoints via prompt_cache_options, reasoning.context persistence, pro mode) are not available through this stack yet.

Revisit when

@convex-dev/agent ships a release with an ai@7-compatible peer dependency — re-evaluate the SDK upgrade at that point, ideally as its own isolated change with its own gate run, not bundled with a model or prompt change.
The eval script's known-flaky residual patterns (occasional last-turn guesses, a raw-key leak reply, and a missed write) reappear on luna; re-run the A/B before assuming they are the same nano-era issues.
Luna's pricing or availability changes materially — the per-turn cost comparison against nano was not re-verified in this record beyond the latency numbers above.
A workload on this agent needs a capability Luna does not have (the two-model routing pattern — Luna for extraction, a stronger model for genuinely interpretive work — was raised as a design option but not implemented; nothing here escalates to a second model yet).

Interview Turns Run gpt-5.6-luna at reasoningEffort "none"