The question almost never arrives as a question. It arrives as a budget line — "we already do an annual pen test, why would we also do this?" — and it deserves a better answer than the one it usually gets, which is a slogan about shifting left. Penetration testing is genuinely valuable, the teams asking have usually been buying it for years, and telling them it is the wrong thing is both untrue and a good way to lose the argument.
The honest answer is that the two practices answer different questions, and neither question is optional. A penetration test asks: given what we actually deployed, what can an attacker do to it right now? A threat model asks: given how we intend this to work, what could go wrong, and which of those things should change the design? One is empirical and one is analytical. One operates on a running system and one operates on an architecture. The distinction is not seniority or rigour — it is what each method is capable of seeing at all.
The short version: A pen test finds implementation defects in what exists. A threat model finds design defects, including missing controls that were never built and therefore cannot be tested. A pen test gives you proof; a threat model gives you coverage. Each is blind exactly where the other sees, and the useful arrangement is not choosing between them but wiring them together — the model scopes the test, and the test's findings are evidence about the model's quality.
The distinction that actually matters
Most comparison tables set these two side by side on axes like cost and frequency, which is accurate and not very useful. The axis that changes decisions is what kind of defect each method is structurally capable of finding.
| Threat modeling | Penetration testing | |
|---|---|---|
| Question answered | What could go wrong with this design? | What can be exploited in this deployment? |
| Operates on | An architecture — diagrams, data flows, trust boundaries, intended behaviour | A running instance — the code, the config, the infrastructure as it exists today |
| Finds best | Missing controls, wrong trust assumptions, flawed authorisation design, unsafe data flows across boundaries | Implementation bugs, injection, broken access control in reachable endpoints, misconfiguration, exploit chains |
| Structurally cannot find | Whether the control was implemented correctly, or at all | A control that was never designed, so leaves no artefact to test |
| Output | Design decisions and a ranked risk register | Verified findings with reproduction steps and evidence |
| Certainty | Hypotheses. Some will not be real. | Proof. What is reported was demonstrated. |
| Coverage | As broad as the diagram — including parts not yet built | Bounded by scope, time-box, and tester skill |
| Timing | Design time, and again when the design changes | After a system exists, typically periodically or pre-release |
| Depends on | The knowledge of the people who built it | The skill of the tester and the honesty of the scope |
Read the "structurally cannot find" row twice. It is the whole argument. Everything else in the table is a difference of degree; that row is a difference of kind, and it is why the two practices do not substitute for one another no matter how much you spend on either.
What a penetration test cannot find
Not because testers are insufficiently skilled — because of what a test is. Four categories are reliably invisible to it:
- The control that was never built. A test probes what exists. If nobody ever designed tenant isolation into the export job, there is no isolation logic to bypass; the tester would have to independently notice its absence, from outside, in a time-box, without the design context. Sometimes they do. It is not something to rely on.
- Trust assumptions that hold today. "This queue is only written to by our own services" may be true right now and is a design assumption, not a security property. A pen test against the current environment confirms the assumption rather than questioning it. The threat model is where you ask what happens when it stops being true — a new consumer, a partner integration, a compromised sibling service.
- Anything out of scope, and scope is negotiated. Third-party integrations, the CI/CD pipeline, the admin tooling, the staging environment with production data. A model covers attack surface the engagement's rules of engagement explicitly excluded — and the exclusions are usually chosen for practical reasons that have nothing to do with where the risk is.
- Threats with no exploitable instance yet. Everything about a system that is still being designed, plus everything about the system you are about to build next quarter. This is the category where a test's score is definitionally zero and the model's value is highest.
There is a fifth, subtler one: a pen test tells you what an attacker could do, not what would matter if they did. Business impact is not discoverable from outside the organisation. The tester who compromises a service does not necessarily know whether that service holds the data that would end the company or the data nobody would notice missing — which is why a report ranked purely by CVSS so often misranks the risks relative to your actual business.
What threat modeling cannot find
The symmetry is important, and skipping it is how threat modeling advocates lose credibility with engineers who have shipped real systems.
- Whether the control works. A model can record that the API enforces per-tenant authorisation. It cannot tell you that one endpoint reads the tenant ID from a request header instead of the session. Only touching the running system finds that.
- Drift. The design was sound, the implementation was correct, and eighteen months later a bucket policy is public and a debug endpoint is deployed. Models describe intent; intent decays.
- Real exploitability and chaining. A model produces plausible attack paths. A tester produces the one that worked, often by combining three findings each individually rated low — which is precisely the kind of composition that is hard to see on a diagram and obvious with a shell.
- Unknown unknowns in your dependencies. The vulnerability in the library you did not know you had, in the transitive dependency you never chose.
And a general caveat that applies to the whole practice: a threat model is only as good as the diagram it is built on, and diagrams are drawn by people who believe they know how the system works. Where that belief is wrong, the model is confidently wrong in the same place. A pen test is one of the few things that reliably catches that class of error, which is a good reason for a modelling team to want tests done rather than to treat them as competition.
The cost argument, stated honestly
The usual version of this argument cites a specific multiplier — fixing a flaw in design costs some exact fraction of fixing it in production. Those figures come from studies whose methodology and applicability to modern delivery are genuinely disputed, and quoting them to an engineering audience invites a rebuttal you will lose.
The defensible version needs no numbers. Changing an authorisation model on a whiteboard means editing a diagram. Changing it after launch means a migration, a backfill, a compatibility window for existing clients, a coordinated release, and a conversation with whoever owns the roadmap you are now displacing. The direction of that inequality is not controversial to anyone who has done both. What is unknowable is the multiplier, so do not claim one.
The corollary matters more than the cost point itself: it explains why a finding's timing changes its nature. The same flaw found at design time is a decision, found in a pen test is a defect, and found by an attacker is an incident. Each stage adds constraints that had nothing to do with the security issue. That is the whole reason to do analysis before there is something to test — not because testing is inferior, but because by the time you can test, your options have narrowed.
How they feed each other
Treated as alternatives, both get worse. Treated as a loop, each makes the other sharper:
- The model scopes the test. Handing a tester a data flow diagram with trust boundaries marked, plus your ranked register, converts the first day or two of reconnaissance into actual testing. You are also telling them where you think the risk is — which is exactly the hypothesis you want an adversarial expert to attack.
- The test validates the model. Every finding should be locatable on the diagram. Trace it back: which component, which flow, which trust boundary. If it lands somewhere the model said was fine, that is not a bad day — it is the most informative output of the engagement.
- Misses are diagnostic. A finding your model never anticipated tells you something specific about how you model, not just about the bug. Was the component missing from the diagram? Was it there but a trust assumption went unquestioned? Was the threat identified and then dismissed as low likelihood? Three different fixes, and the third is the interesting one.
- The model turns findings into fixes that hold. A report says the endpoint leaks other tenants' records. The model is where you ask whether the same pattern exists on the eleven endpoints the tester did not have time to reach — and whether the right remediation is patching this one or moving the check somewhere it cannot be forgotten. That is the difference between remediating a finding and removing a class of finding.
One practical habit makes this concrete: after every engagement, spend an hour mapping each finding onto the existing model before anyone starts fixing anything. It routinely reveals that three "separate" findings are one design decision, and it is the cheapest quality feedback a security programme gets.
Where the other assessment types fit
The debate is usually framed as a pair, which flatters both. In practice a programme has more than two instruments and they overlap unevenly.
| Activity | Primary question | Best at | Blind to |
|---|---|---|---|
| Threat modeling | What could go wrong by design? | Missing controls, trust assumptions, breadth including the unbuilt | Implementation correctness, drift |
| Penetration testing | What is exploitable now? | Proof, chaining, creative misuse of what exists | Absent controls, out-of-scope surface, business impact |
| Red teaming | Would we detect and respond? | Testing the blue team, not just the system — see TTPs | Comprehensive coverage; it is a narrow, deep campaign |
| SAST / DAST / SCA | Do known bad patterns appear? | Continuous, cheap, catches regressions and vulnerable dependencies | Anything requiring intent — logic and authorisation flaws |
| Secure code review | Is this implemented as designed? | The gap between model and code, in the code that was reviewed | Runtime and configuration reality |
| Bug bounty | What did everyone else miss? | Continuous coverage, diverse attacker perspectives | Prioritised or systematic coverage; you get what researchers find interesting |
Threat modeling is the only row that operates before the artefact exists, and the only one whose output is a decision rather than a defect. That is its distinctive contribution, and it is also why it never shows up in the same report as the others — a design flaw prevented leaves no trace, which is an unfair disadvantage in any budget conversation and worth naming out loud when making the case.
What compliance actually requires
A practical note, because this is often what settles the argument in regulated environments: several frameworks require both, in different words.
PCI DSS v4.0 mandates periodic penetration testing outright under its Requirement 11 testing obligations, and separately expects secure design and threat identification during development. ISO/IEC 27001:2022 carries both sides in Annex A — technical vulnerability management alongside secure system architecture and engineering principles, the second of which is a design-time expectation that a test cannot evidence. SOC 2's Common Criteria similarly look for risk identification and analysis as a process, not merely a periodic test result.
The pattern is consistent: auditors want evidence that you find problems and evidence that you designed to avoid them. A pen test report satisfies the first and says nothing about the second, which is why a maintained model with dated decisions is useful audit evidence in a way that a stack of test reports is not. Check the current text of whichever standard applies to you rather than taking a blog's word for the clause — including this one.
So which do you do first?
It depends on one thing: whether the system exists yet.
- Being designed now. Model. There is nothing to test, and the window where findings are free closes when the code is written.
- Live, never tested, unknown state. Test first. You need to know what is exploitable today, and a model built on an unverified understanding of a system nobody has probed will inherit that team's blind spots. Then model, using the findings as calibration.
- Live, tested regularly, findings keep recurring. Model — and this is the strongest signal there is. Recurring findings of the same class across engagements are not a testing problem; they are a design problem that testing has been reporting to you for years without being able to name.
- Genuinely can only afford one, ever. Then buy the test, and do the modelling yourselves. Testing requires expertise and tooling you probably do not have in-house; modelling requires a whiteboard, ninety minutes, and the people who built the system — who are the only people who can do it well anyway.
Where the comparison goes wrong
- "We're covered, we pen test annually." Coverage is the specific thing an annual scoped test does not provide. It provides depth on a slice, once a year, on what happened to be in scope.
- "The model said it was fine, so we can skip the test." Inverted, and worse. A model is a set of hypotheses about a system nobody has attacked.
- Comparing on cost per finding. A metric that rewards whichever activity reports more issues, and punishes the one whose successes are invisible.
- Letting the pen test define the risk register. The register should hold your risks. A test contributes findings to it; it does not get to determine what is in it, or the register silently inherits the engagement's scope.
- Treating the test as the deadline. Once a pen test date becomes the moment security is considered, everything shifts right — including the modelling that was supposed to happen months earlier.
- Arguing the comparison at all with an engineering team. Nobody has to lose. The framing that works is that the test tells you what you built wrong and the model tells you what not to build — and the second one is cheaper because it is earlier, not because it is cleverer.
The two practices come from different traditions and attract different temperaments, which is probably why they get pitched against each other more than the underlying logic warrants. A tester is trying to prove a specific thing is possible. A modeller is trying to enumerate what might be. Those are complementary dispositions, and the programmes that work well tend to have both in the room — often literally, because the person who has spent a decade breaking systems is unusually good at spotting the optimistic assumption on somebody's attack tree.
Your regulator or customer may require both — see the industry threat modeling guides for what applies in SaaS, fintech, healthcare, medical devices, automotive, AI/ML, and critical infrastructure. To start the design-time half, the free templates include a pre-labelled DFD and a risk register you can hand to your next tester.