Mythos 5. The determinism problem.

In May we published what AI security scanning actually costs. Three scanners on one codebase agreed on 5% of findings. One scanner run three times reproduced 17% of its own results. A $315 scan generated $128,000 in triage labor. The most common response we got was patience. The models were early. The next generation would fix it. To see those early results, here’s part one of the series.

Over the summer, the next generation shipped. We tested it under the same protocol. It got worse.

We ran the same scanner three times against the same commit of the same application on the same day. No code changed between runs. Nothing changed at all except the clock.

01 What we ran

We used Claude Security in Standard mode, now running on Mythos 5, against a modern 80,000-line application, 55,000 lines of which are Java. Three runs against the same commit on the same branch, each in an isolated project with no shared context, all in one day.

Run One returned 29 findings for $250 in 37 minutes. Run Two returned 15 findings for $225 in 30 minutes. Run Three returned 21 findings for $255 in 50 minutes. Total spend was $730 across 117 minutes of scanning.

One detail matters more than the model version. Part one tested a codebase that had already absorbed years of static analysis, dynamic analysis, instrumentation, bug bounty and annual pentests. We said at the time that this weeded out the easy findings and gave the models a real workout. Vendors took that framing and ran with it.

This codebase does not qualify. It is new. It is modern. It is AI-heavy service code nobody has picked over. This is the easy case, and the scanner still cannot agree with itself.

02 Reproducibility went down

Twelve percent of the 42 candidate findings were reported by all three runs.

Part one measured the same thing the same way. The Sonnet-based scan reproduced 17% of its own findings across three runs. The Opus scan managed 25%. Same protocol, same style of codebase, same three-run intersection. The newest and most capable model in the series scored 12%.

Fifty-seven percent of the findings were reported by exactly one run and never seen again.

Twenty-four of 42 findings are ghosts. They appeared once, in one report, and no subsequent scan of identical code ever mentioned them again. Another 13 appeared in two runs out of three. Whichever scan you happened to keep determines which two thirds of this list you work on.

03 The finding set is not converging

This is the number that changed my read of the whole problem.

After run One we had 29 unique findings. Run Two added four the first run never mentioned, bringing the total to 33. Run Three added nine more, bringing it to 42.

Run Three added more new findings than run Two did. On identical code. After $475 had already been spent looking at it.

The curve is not flattening. It is steepening. Forty-two is not the number of candidate issues in this application, it is the number we happened to reach after three passes, and there is no evidence in this data that a fourth pass would not add another nine. We do not know where this converges. Neither does the vendor, because nobody has published a convergence curve for any of these tools.

That single chart invalidates the entire premise of a scan. A scan is supposed to be an assessment of an application. This is a sample from a distribution, and we cannot see the distribution.

04 Why this is structural

Each run starts with nothing. No memory of the previous run, no record of which files it opened, no note of which call paths it already traced. It picks a place to start, and every choice after that narrows what it will ever see. The first file it opens shapes which call it follows, which shapes what loads into context, which shapes how much is left to examine before the pass ends.

A traditional static analyzer walks the same abstract syntax tree in the same order every time.The traversal is fixed, so the conclusions are fixed. Change nothing and you get the same answer back, which is the entire reason anyone was ever able to build a baseline.

An agentic scanner decides where to go next.That decision comes out of a model, and 80,000 lines do not fit in a single reasoning pass, so the scanner has to choose which parts of your application it will actually look at. Choosing under that constraint is sampling. Three runs took three different walks through the same code and came back describing three overlapping but different applications.

That is what the convergence chart is showing. Run Three did not surface nine new issues because it tried harder. It surfaced them because it went somewhere the first two runs never went.

Setting temperature to zero does not fix this, and it is worth being precise about why. Tool call ordering and context assembly still vary in an agentic system, so the walk still moves. But grant the vendors a perfectly deterministic scanner tomorrow. You would get the same 29 findings every single time and never see the other 13. Determinism without coverage is a consistent blind spot, which is easier to build a program on and no safer to trust.

This is where context-window constraints hit real-world application architecture. Complex vulnerabilities rarely live in a single method; they require tracing untrusted data flows across multiple files and layers—from an API controller, through middleware, down to a database query. Because LLM-based scanners must drop and assemble context to stay within reasoning windows, they inherently fragment cross-file data-flow tracking. In contrast, runtime instrumentation continuously observes data execution in memory as code runs, requiring no context-window sampling or path guesswork to trace an exploit.

05 Spend does not track output

Run Three cost the most, ran 13 minutes longer than run one, and returned eight fewer findings.

You cannot buy your way out of this with more compute or more patience. There is no relationship between what you spend on a pass and what the pass gives you back. The most expensive, longest-running scan of the three was not the most thorough.

06 Which report you opened decides how bad your application is

Run one reported 11 HIGH severity findings. Runs Two and Three reported four each.

If you scanned this application once and took the output into a risk review, the answer to “how exposed are we” is either eleven serious problems or four, depending on which pass you were handed. Run Three also pushed six findings into LOW, three times as many as run one rated that way.

Sixteen findings were rated HIGH by at least one of the three runs. Seven of those sixteen were reported by a single run and never reproduced. Genericized, they include an authentication bypass by spoofing, a missing authentication control on a credit tracking controller, a second missing authentication control on a balance management controller, an integer overflow, a resource exhaustion condition, an improper cleanup on a thrown exception and a business logic error. CWE-290, CWE-306 twice, CWE-190, CWE-770, CWE-460 and CWE-841.

Five of those seven came from run One. Two came from run Three. Run Two contributed none. If you ran this scan once, you had a two in three chance of never hearing about any given one of them.

07 What actually survives

Eighteen findings appeared in more than one run. Of those, six held the same severity, nine held the same CWE and six kept the same category name. Twelve of eighteen were rated differently depending on which report you read.

Narrow it to the five findings all three runs agreed existed, the reproducible core of this application, and the labels still move. Two of the five held a stable severity. One went HIGH, MEDIUM, MEDIUM. Another did the same. A missing authorization finding went LOW, then MEDIUM, then back to LOW. It bounced.

The CWE moved too. A client-side enforcement issue was CWE-602 in run one and CWE-807 in ru

This state churn directly pollutes your Vulnerability Management systems. When findings bounce between CWE-602 and CWE-807, or bounce from HIGH to LOW, your ASPM (Application Security Posture Management) deduplication logic fails. Instead of updating an existing record, the system creates duplicate Jira tickets for the exact same underlying issue. This doesn't just create noise; it corrupts your Mean Time to Remediate (MTTR) metrics and generates cascading ticket bloat for the triage team.

Two findings out of 42 held both their existence and their severity across all three passes. That is the stable core of this assessment. Two.

This state churn directly pollutes your Vulnerability Management systems. When findings bounce between CWE-602 and CWE-807, or bounce from HIGH to LOW, your ASPM (Application Security Posture Management) deduplication logic fails. Instead of updating an existing record, the system creates duplicate Jira tickets for the exact same underlying issue. This doesn't just create noise; it corrupts your Mean Time to Remediate (MTTR) metrics and generates cascading ticket bloat for the triage team.

The same instability shows up at the file level, where no finding-matching rule is involved at all.

Twenty-eight source files were flagged across the three scans. Eight were flagged every time. Fourteen were flagged once.

If you are rolling any of this up to the OWASP Top 10, to ASVS, or into a PCI or SSDF control narrative, you are rolling up noise. The category your compliance report depends on is itself a sample.

08 Now try to close a ticket

You cannot verify a fix. That is what changed my mind between part one and part two, and it worries me more than any number above.

Part one argued that unstable findings break your trend lines. That was too polite. The problem is more basic than reporting.

You have no way to know. Twenty-four findings in this dataset showed up in one scan and never again, and nobody touched a line of code. When coverage is based on random sampling, a 'clean' rescan is functionally meaningless. It is impossible to distinguish a remediated vulnerability from an un-traced execution path. Your verification step cannot distinguish a fixed vulnerability from a finding this pass never went looking for.

Now run it backward. Run Three surfaced nine findings no previous scan had reported, in code that had not changed since the last two scans. In any normal workflow those read as regressions. Your team opens nine tickets to investigate code nobody has edited.

Every security automation pipeline assumes state continuity between scan N and scan N+1. Without deterministic execution paths, you cannot establish a verifiable baseline or programmatically confirm remediation. No new criticals in the merge gate. Risk acceptance with an expiry date. SLA clocks that start when a finding appears and stop when it clears. Every one of those controls requires that a finding staying, appearing or vanishing carries information.

At 12% reproducibility, none of them carry information.

09 The economics improved. Nothing else did.

Give credit where it is due. This generation is cheaper and faster per line of code than the last one. A single pass averaged $243 for 80,000 lines, roughly $3.04 per thousand lines against the $4.72 we measured in part one, in about half the wall clock time per line.

That improvement makes the reliability problem worse, because it removes the last excuse for running one scan and calling an application reviewed. If a pass is cheap, the honest thing to do is run several. Run several and you are staring at the convergence chart.

Getting to five reproducible findings on this application took $730 and 117 minutes. That is $146 per reproducible finding, none of which has been validated as real yet. For the two findings that held both existence and severity, it is $365 each.

10 What we are not claiming

Three runs, one codebase, one scanner. This is an existence proof, not a benchmark. One reproducible failure is enough to invalidate a claim of determinism, and that is the only claim we are making.

None of the 42 findings has been confirmed as a true or false positive. That work is underway and it will be its own post. Everything above measures run-to-run consistency, not correctness. A finding all three runs reported is reproducible, not right, and a finding reported once may still be real. That is exactly the problem.

Our clustering rule was deliberately generous. Findings counted as the same issue if they shared a file path and either landed within five lines of each other or carried the same CWE, with at most one finding per run allowed into any cluster. A stricter rule produces worse agreement, not better.

The reproducibility comparison to part one is a like-for-like one. Twelve percent, 17% and 25% are all three-run intersections on a single codebase, measured the same way. We ran a third pass here specifically so that number would be comparable rather than convenient.

11 What this is actually good at

It finds things. That is the entire value proposition, and it is a real one.

These models reason about intent in a way that pattern-matching static analysis cannot. Broken access control, insecure direct object references and missing ownership checks across multi-tenant logic are genuinely hard for traditional tooling. The strongest confirmed true positives in part one came from AI scanners, and the one finding that survived all three runs here at a stable HIGH severity is an insecure direct object reference. That capability is real and it is worth having.

For AppSec leaders, the path forward is positioning the tool correctly in the architecture: treat an AI scanner like an automated penetration tester, not a continuous build gate. Pen-testers excel at discovering novel, context-heavy logic flaws, but no architect would wire a human pen-tester into a merge gate to verify a bug fix on every pull request.

Use AI scanning for ad-hoc, periodic discovery. But for continuous verification, merge-gate enforcement, and establishing a stable risk baseline, you must rely on deterministic runtime execution that yields the same answer every time.

Do not make it a system of record. Wiring a non-deterministic scanner to a merge gate introduces probabilistic failure modes into your deployment pipeline. The gate logic is driven by sampling noise, not actual security posture. Do not track burndown from it, because the denominator moves on its own. Do not verify a remediation with it, because a clean rescan proves nothing. Do not put its severity ratings in front of a board as a measure of application risk, because we just watched those ratings bounce in both directions on code that never changed.

The gap this exposes is the one we keep landing on. Static analysis of any kind, deterministic or probabilistic, describes what your code might do. It cannot tell you what your application is doing in production right now, what is reachable by real traffic, or what is being probed. Runtime tells you that, and it tells you the same thing every time you ask.

Consistency is the property that makes every other property of a security tool useful. A scanner that answers differently each time you ask is a discovery aid with an invoice.