Minimal-testing skill: what the Codex runs actually show
Internal experiment recap · Draft · Not a product claim
Keep the skill and keep using it. On the cleaner Codex pair it cut session tokens from 878,495 to 396,241 and replaced a multi-path browser tour with a shorter flow, while both agents still shipped a working inventory page. It does not prove the agent will ever do only one check, and the first pair is too contaminated to count.
The question
Does minimal-testing stop Codex from burning tokens on smoke tests, retesting, and website tours, without shipping a worse page?
What was inspected
The governing text is the repo skill. It requires one targeted functional check by default, a second check only for a second behavior the user named, and a ban on suites, extra viewports, and unrelated tours. File existence is not a pass.
| Field | Value |
|---|---|
| Path | minimal-testing/SKILL.md |
| Lines | 182 |
| SHA-256 | d10996d89f04d4d042f49cb6382b3a6938cfa295369ac9e1564e5eb2e853f148 |
Meter. Codex threads.tokens_used and each thread’s token_count events. Totals sum every model call’s prompt, including cached repeats. They are not “new unique tokens.”
Setup. Both pairs used gpt-5.6-sol at medium effort.
| Control | With $minimal-testing | |
|---|---|---|
| Thread | 01a04700-e0cc-…b432 | 01a04700-eeae-…b2ec |
| Name | test-no-skill | test-yes-skill |
| tokens_used | 1,097,996 | 475,733 |
| Model calls | 20 | 11 |
| Also loaded | impeccable and minimal-testing | minimal-testing (named) |
Both jobs shared /Users/anon5376/Documents/ChatGPT/test. The control hit “port 8765 already in use.”
Agents cannot see the meter. Asked for session tokens, they answered ≈50,000 and ~52,000. Last-call context was ~48k and ~68k. Cumulative totals were already 427,354 and 1,029,748.
| Control (skills off) | With $minimal-testing | |
|---|---|---|
| Thread | 01a04bb0-78e5-…c796 | 01a04bb3-b1d5-…f1d3 |
| Folder | …/ChatGPT/control | …/ChatGPT/test 2 |
| tokens_used | 878,495 | 396,241 |
| Model calls | 19 | 10 |
| Browser/MCP calls | 12 | 4 |
| index.html lines | 360 | 215 |
| Skills loaded | Sites + in-app browser | browser + named skill |
| Final check claimed | Search, detail, sell-to-zero, Spare C, HTTP 200, console, retries, reload reset | Search Gadget B → details → Mark sold to 0 |
| Spec coverage | All named behaviors present | All named behaviors present |
Both pair-2 pages were read as bytes. They hardcode the same three products, live name search, a same-page detail panel, Mark sold, and “Out of stock” at zero. Neither added tests, extra routes, or a framework.
What the evidence concludes
FACT: on the cleaner pair, naming $minimal-testing coincided with fewer model calls, fewer browser calls, and less than half the stored session tokens.
FACT: the extra control tokens bought a longer QA tour and more CSS, not extra product behavior.
FACT: the skill text was not followed to the letter. Treatment packed search, select, sell, and out-of-stock into “one flow.” Control added HTTP, console, retries, and a reset on top of the planned tour.
INFERENCE: the skill works as a scope brake against other skills and against extra verification theater. Most of the token gap is extra rounds re-sending ~50k of cached context, not 2× more unique writing.
What does not follow: a guaranteed 2.2× cut on the next UI task. That the skill forces a single click. That pair 1 is a valid A/B.
What remains unresolved
OPEN: whether implicit invocation (no $) is enough. Pair 2 treatment used an explicit mention. Pair 1 control still self-loaded the skill, so implicit behavior is mixed, not measured.
OPEN: whether Cursor agents obey the same text. This report is Codex-session evidence plus the repo file.
UNKNOWN: cost on non-UI tasks, APIs-only, or larger apps where one check is genuinely insufficient.
FAIL relative to the skill text: packing several named UI behaviors into “one flow.”
What decision followed
Keep minimal-testing installed and invoke it on UI work. Do not treat pair 1 as proof. Do not rewrite the skill around the 2.2× headline. The next edit, if any, should forbid packing several named UI behaviors into one flow unless the user asked for that path as a single story.
What happens next
- Owner: you. Optional skill edit: one check equals one behavior unless the prompt names a single path.
- Acceptance evidence: a third Codex pair, new folders, other skills off, sequential. Compare
tokens_used, call count, and whether the final answer lists more than one behavior. - Stop or redirect: if a clean pair shows no gap, stop claiming usage savings. If the treatment page misses a required behavior, the skill is over-tight; restore an explicit second-check rule.
Verification and limits
Skill bytes hashed in this workspace. Pair-2 HTML read in full. Token figures are Codex thread records inspected in the working session that produced this report, not a fresh database dump at publish time.