What does a production audit actually find?

What did the audit find, in total?

Sixty-three findings: 7 critical, 29 high, 20 medium, 7 lower-severity. All thirteen layers returned something. Five specific areas were checked and came back clean — SQL injection, webhook signature verification, payment idempotency, cross-user cache leakage, and cross-tenant data access.

The spread is the point. A security review alone would have found one of the seven critical findings. A performance pass would have found two others and missed the rest. The reason the check has thirteen layers is that the things which actually take an app down are distributed across all of them, and each one is invisible from inside the others.

Twenty-one of them are still open — see below.

What was the single worst thing you found?

The database caps a query at 100 bound parameters, and six operations built their queries by adding one parameter per row. Any store with more than a hundred products could not be deleted, imported to, synced, or listed. It needed no traffic, no attacker and no unusual conditions — a normal catalog was enough.

Deleting an account failed in the same way, and because the record of the deletion was written in the same all-or-nothing batch, nothing happened at all: not a partial delete, not an error anyone could act on. A mid-size apparel brand carries 200 to 2,000 products, so this wasn't an edge case — it was the first real customer import.

The part that stays with you is that the correct way to write it was already in the codebase, two files away, carrying a comment explaining exactly why it existed. It just wasn't carried across when the deletion logic was consolidated into one place. The consolidation meant to make deletion safer is what broke it.

The fix took a day. The part that matters took an hour: a check wired into the build that now fails the build if anyone writes it the old way again. A fix removes one instance; a guard removes the class.

How can something be broken and invisible at the same time?

Because the code that noticed was written to swallow what it noticed. The shopper analytics wrote every open session for a store into a single stored value, and that store caps a value at 128 KiB — so past a few hundred concurrent shoppers the write threw. The throw landed inside a fire-and-forget handler that logged to a console nobody reads.

So there was no error page, no alert, no log entry and no failed request. The merchant's dashboard simply stopped climbing. And it would have stopped first on the day the most people arrived at once — the campaign, the launch, the sale — which is the day that data is worth the most.

This is the failure shape that makes audits worth buying, and it is the one you cannot find by using your own app. Everything looks correct at your desk. It looks correct in staging. It looks correct right up to the moment it matters, and then it fails without telling anyone.

Did the AI-built code get anything right?

A great deal, and the list is longer than most people expect. A dedicated security review of the authentication, session, ownership and payment paths found one real vulnerability — no authentication bypass, no cross-tenant data access, no injection, no webhook forgery.

Specifically already correct before anyone looked: webhook signatures verified properly, with a replay window and a constant-time comparison; an append-only credit ledger with a uniqueness constraint, where refunds deliberately use a different key from the charge so a refund can't be swallowed by it; one-time codes stored hashed, single-use, with an atomic attempt counter — because the obvious read-then-write version lets two simultaneous attempts both slip past the cap; password resets whose link invalidates itself the moment a new password is set, by construction rather than by bookkeeping.

There was also one genuinely excellent architectural decision — the whole asset path is built so that bandwidth, the scariest-looking cost in a 3D product, is free — and the audit's own notes name it as such and tell any later pass not to touch it.

This is why the distinction that matters isn't AI versus human. It's built versus reviewed. The code wasn't bad. It had never been read by someone asking a different question than the one it was written to answer.

How long did the work take?

The audit ran on 28 July. The first wave of fixes — including every one of the things that were broken with no traffic at all — was live in production on 1 August. Three days. The full programme ran about four weeks, in waves, and it is still going.

Both halves of that are worth saying plainly. The audit itself is fast, and the highest-severity fixes ship inside the first week. The tail is real work: eight or nine working sessions, several findings that were discovered by the fixes rather than by the audit, and a remainder that can't be closed from a keyboard at all.

So when we say days rather than weeks, we mean the first fixes ship in days. Anyone promising you the whole thing is finished in days is describing a smaller job than this one.

What's still open, and why?

Twenty-one findings. They fall into three kinds: deliberate trade-offs that were considered and rejected, work that can't be done from a keyboard, and work that is scheduled and not yet done. The open list is published as a count, not as an inventory — this is a live app, and naming which defences are absent would be handing out a map.

The deliberate ones are the interesting category. A supplier's account-level concurrency cap has no engineering fix, and the note says explicitly not to open extra accounts to get around it — that would be a terms-of-service violation, revocable without warning. A timeout that discards slow work was lengthened rather than shortened, because the supplier won't say whether a job is queued or stuck, and when you can't tell them apart the only safe way to be wrong is late. Each of those is written down with its reasoning, so nobody re-proposes it in six months thinking it was an oversight.

The ones that need something other than typing: a load test nobody has run, a device nobody has in the room, a lawyer. We'd rather name those as open than quietly count them as done.

Do these numbers say anything about my app?

No — and be suspicious of anyone who says they do. This is one app, examined completely. One app is not a distribution, so nothing here supports a claim like “most AI-built apps have N of these”. We don't have that data, and we won't quote it until we do.

Two more limits worth being straight about. This app was built by a developer using AI assistants, not by a non-technical founder using a no-code builder — so the tool isn't the parallel. The failure classes are: things that break at a size you haven't reached yet, things that fail silently, protections that are present but not actually working, and one input somewhere that gets treated as code. Those transfer to anything built quickly, whatever built it.

And the capacity numbers in the audit are arithmetic against documented platform limits and a reading of the code — no load test has ever been run against this app. That's stated in the audit's own documents, and it's the single biggest caveat on everything in it. A finding that says “this breaks at roughly this size” is a prediction. A finding that says “this is broken at a hundred and one products” was measured.

Who ran this audit — a person or a tool?

Both, in the order that matters: AI for coverage, a developer for judgement, and the developer is accountable for every finding. The same method we describe everywhere else on this site — the long version is on the 13-layer production check.

What that looked like here, concretely. The cross-site-scripting finding was reproduced end to end in a real browser — a store seeded with the payload as its name, the page inspected, the injected code observed running — and then the same test was re-run against the patched version. The policy that now backs the fix was rolled out in report-only mode first and driven in a real browser, which caught something no amount of reasoning would have: part of the 3D stack evaluates a string at runtime, so the stricter policy everyone would have written by default would have broken the product.

Same pattern on the performance work: a belief about how many parallel downloads were optimal was tested on live production against a real GPU, and the measurement said the opposite of the reasoning — more parallel fetches were slower. Three of the sharpest findings in this project weren't bugs at all. They were confident answers nobody had measured.

What actually changed, measured

Every row here was measured, before and after, with the method and the date recorded. What is not here matters as much: page load time, error rates and uptime have no “before”, because nothing was measuring them. You cannot compare against a number nobody took.