Chapter 12 of 12
What to Look for in an API Security Testing Tool
Eight criteria that decide whether a scanner is still being read in month three, written as questions to put to any tool including the one you already run.
Tools in this category demo identically. They all produce a list with red items on it, and the differences only show up in the weeks after the trial, when somebody has to read run number twelve.
These are the questions that predict that. They apply to any tool including the one you already run.
The criteria
1. Does it start from your specification
Ask how the tool learns what your API contains.
A crawler follows links, and an API has none, so a crawling tool sees whatever you manually pointed it at. A tool that reads your specification knows every path, method, parameter and body schema before it sends anything, which means coverage becomes a property of your contract rather than of a discovery heuristic. It also means probes fit the endpoint, and a probe that fits produces far fewer false positives than one that does not.
2. Can it hold more than one identity
This is the question that decides whether the tool can test the top of the OWASP list at all.
Three of the ten categories need two identities compared against each other, and a tool with one credential slot cannot express the comparison. Ask how identities are stored, which authentication schemes they support, whether secrets are encrypted at rest and never returned, and what the tool does when an authorization check is configured without enough identities. Skipping silently is the wrong answer, because a check that quietly does nothing is worse than one that is absent.
3. What does it do to the target
A scan is traffic, so ask what the traffic looks like before you point it anywhere.
Which probes write data, which generate load, what the concurrency and delay defaults are, and whether there is a wall-clock limit. Then ask which of the aggressive checks are off by default. A tool with fuzzing and rate-limit bursts on out of the box will eventually be blamed for an outage. A tool that cannot be paced at all cannot be pointed anywhere shared.
4. Is confidence separate from severity
A single ranking collapses two different questions, as the triage chapter covers.
Ask whether the tool tells you how sure it is, separately from how bad it would be. Then ask the sharper version, which is whether the tool ever reports a weaker finding at a lower grade rather than promoting it. A tool that grades every detection as high is telling you about its marketing rather than its evidence.
5. Can you reproduce a finding without the tool
Every finding you act on has to be verified. Ask what the tool hands you for that.
The good answer is a request you can run yourself, plus the raw evidence behind the decision. The poor answer is a category name and a description. Whoever has to fix it then reconstructs the probe from prose. This single property does more for adoption than anything else on this list, because it is the difference between a finding and an accusation.
6. What happens to a finding you closed
Ask whether a resolving status needs a reason, and whether a later run reopens an item it sees again.
A tool that lets you close things without a note produces a list nobody can audit six weeks later. A tool that keeps something closed after the fix was reverted produces a list that is wrong. Both failure modes end the same way, which is a report people stop reading.
7. Does it leave the tool
Ask for the export format and check that your code host reads it natively.
SARIF is the answer worth wanting. The details matter more than the checkbox. Ask whether severity is carried in the property your host buckets on, whether results land on the endpoint rather than on one line, and whether each result has a stable fingerprint.
8. Is it honest about its limits
Ask which categories the tool does not cover and see whether you get a straight answer.
Every scanner has a blind spot, since business logic is not automatable by anyone. A vendor who names the gap is telling you what the rest of your programme has to cover. A vendor who implies complete coverage is selling you a false negative. The limits chapter is what a straight answer looks like.
A scorecard you can use
| Criterion | The question | Poor answer |
|---|---|---|
| Surface discovery | Does it read the specification | It crawls, or you list endpoints by hand |
| Identities | Can it compare two callers | One credential slot |
| Impact on the target | What does a run cost the system | No pacing, aggressive probes on by default |
| Confidence | Is it separate from severity | Everything is high |
| Reproduction | What do you get per finding | A description |
| Triage | Are closures recorded and reopened | Close with no note, never reopened |
| Export | Does your code host read it | A PDF |
| Limits | Which categories are not covered | Complete coverage implied |
How to actually evaluate
Do not evaluate on a deliberately vulnerable sample application, because every tool looks competent against one.
Point the tool at a staging deployment of your largest real API and read the first fifty findings rather than the summary. Count three things. How many you could reproduce from what the tool gave you, how many were genuinely wrong, and how long a run takes. Those three numbers are the tool's real cost, since a scanner that produces forty items nobody can verify is a net loss whatever it detects.
Then check the second run. A tool that reports the same fifty findings again, with no memory of what you decided, will not survive contact with a team.
In Routebase
Routebase answers these criteria in a particular way, and the previous chapters describe each answer in full.
The scanner reads the specification you already maintain, so coverage follows your contract rather than a crawl. Identities are personas with encrypted credentials that are never returned to the browser, and a scanner that needs two of them cannot be saved with fewer. Pacing is explicit, with concurrency, delay and a time budget on every profile, and the two load-generating features are off until you turn them on. Severity and confidence are separate fields, and the weaker authorization detector reports at a lower grade rather than being promoted to match the stronger one.
Every finding carries a reproduction curl command and the raw evidence. Closing one requires a note, reopening happens automatically on the next scan that sees it, and the export is SARIF 2.1.0 with the numeric severity property, endpoint-level locations and per-result fingerprints. The category with no scanner is named as not covered in the documentation rather than implied away.
See Security Overview for the workflow and the Scanner Reference for what each scanner checks, including the Not covered section.
Frequently asked questions
What should you look for in an API security testing tool?
Ask whether it reads your specification, whether it can hold more than one identity, what it does to the target while it runs, and whether every finding comes with a way to reproduce it. Those four separate a scanner from a report generator. The feature lists tend to converge, and the answers to those questions do not.
What is the difference between DAST and API security testing?
DAST means dynamic application security testing, which is the general category of probing a running system rather than reading its source. A classic DAST tool was built for web applications and discovers its targets by crawling links, which finds very little of an API. A tool built for APIs starts from the specification instead, so coverage is a property of your contract rather than of the crawler.
Do you need a scanner if you already have a penetration test?
They answer different questions and the schedule is the reason. A penetration test is a deep look at one moment, and a scanner is a shallow look at every moment. Without the scanner, the twelve months between tests are unobserved, and without the test, everything that needs a model of your business stays unexamined.
How should you evaluate an API security scanner?
Point it at a staging deployment of your largest real API and read the first fifty findings rather than the summary. Count how many you can reproduce from what the tool gave you, how many are genuinely wrong, and how long the run took. That afternoon tells you more than any feature comparison, because it measures the work the tool creates as well as the work it saves.
Last reviewed by The Routebase Team.