How do you measure API quality?

By combining a few things a machine can count and then treating the result as a pointer rather than a verdict. Rule compliance, documentation coverage, security posture and internal consistency are all countable, so a weighted score over them is a reasonable way to find the specification that needs attention. Whether the API is any good is a different question that no count answers.

What are good API quality metrics?

The countable ones are rule compliance and the share of endpoints carrying a summary, a description and a declared response. Add to those the share declaring a security requirement, and how uniform naming and error shapes are across the document. The ones that matter more are harder to get, such as how long a new consumer takes to make a successful first call and how many support questions an API produces.

Should teams be measured by an API quality score?

No, and the reason is mechanical rather than cultural. As soon as the number is a target it stops describing quality and starts describing effort spent on the number, so descriptions get padded and inconvenient rules get switched off. Use it to decide where to look, and use the things it points at as the actual conversation.

What is API documentation coverage?

The share of endpoints that carry the things a reader needs, which is normally a summary, a description, a declared response with a schema and an example. It is worth measuring because a missing description is a certain defect. It is a floor rather than a target, since it says nothing about whether the descriptions are any good.

Measuring API Quality

A quality score is useful for one job and harmful for another, and the two are close enough that organisations regularly do the second by accident.

What a weighted score is made of

Every score of this kind is a handful of countable things, weighted and added up.

Five dimensions with their default weights, and the question each one actually answers. Four of the five are counts over the description itself, which is what makes the number cheap to produce and narrow in what it can mean.

Rule compliance is the share of applicable style guide rules the specification passes. It is the most direct of the five, and it inherits whatever your guide happens to contain.

Documentation coverage counts what a reader needs on each endpoint, which is normally a summary, a description, a declared response with a schema and an example.

Security posture counts declared security requirements, whether schemas carry validation constraints, and whether anything looks like a field that should not be in an API at all.

Consistency compares the document against itself. Whether path segments follow one convention, whether error responses share a shape, and whether property naming is uniform.

Architecture health sits outside the document and describes how the service relates to its neighbours, so it is the one dimension that needs analysis beyond the specification.

Coverage is not quality

The largest gap between what a score says and what people hear sits in the word coverage.

Coverage says a description exists. It cannot say the description explains anything, and the two are only weakly related. A team pushed on coverage will produce a summary for GET /orders/{id} that reads "Gets an order by id", which moves the number and helps nobody.

That does not make it worthless. A missing description is a certain defect, so a coverage number is a genuine floor. Read it as the absence of one known problem rather than the presence of quality.

What the number cannot see

Four things sit outside any score computed from a description, and they happen to be the four that decide whether an API is good.

Whether the resource model fits the domain. A document can be immaculate and still model the business wrongly, and the score has no opinion.

Whether consumers succeed. The time from reading your documentation to a working first call is the number that actually reflects quality, and it is measured against your API rather than against your specification.

Whether the API should exist. Two teams shipping overlapping APIs both score well.

Whether it is stable. A specification that changes shape every sprint can hold a perfect score throughout, while the consumers absorb every change.

What happens when you manage by it

This is the failure the rest of the chapter exists to prevent.

A measure that becomes a target stops being a measure, because effort moves to the number rather than to the thing it was counting. The mechanism is not abstract, and it shows up in the same three ways every time.

Descriptions get padded, since the check is presence and presence is cheap. Rules get switched off, because a rule nobody passes is easier to disable than to fix and the score improves either way. Work gets reordered so that the countable parts go first, which means the design questions nobody can count go last.

The honest arrangement is to use the score as a filter rather than a grade. It tells you which specification to open, and the conversation after you open it is about what you found there. Compare a specification against its own history rather than against another team's number, because two APIs at different ages and different levels of ambition are not comparable by construction.

The measurements worth adding

If you want numbers that resist the problem above, take them from outside the document.

Time to first successful call is the strongest one, and it moves when the getting started page genuinely improves. Support questions per API tell you where the documentation failed, and each repeat question is a specific defect. Failed searches in your documentation portal are a ranked list of things readers expected and did not find. Measuring API documentation covers those in detail.

None of them are as convenient as a single percentage, which is exactly why they are harder to game.

In Routebase

Each specification gets a quality score aggregated from five weighted dimensions, and the weights are yours to set.

Style guide compliance carries a quarter by default, documentation, security and consistency carry a fifth each, and architecture health carries the rest. The settings page requires the five to total one hundred and refuses to save otherwise, which keeps the weighting an explicit decision rather than a drift. Architecture health scores full marks until an architecture analysis has run for the project, so it never costs points by default.

Two thresholds turn the score into an alert rather than a dashboard. A minimum quality score and a minimum documentation coverage each come with a live preview, counting how many specifications would fall below the value you are typing. That preview is the fastest way to see whether a threshold is ambitious or unreachable. Alerts on breaking changes and on drafts left untouched beyond a configurable window round it out.

The score is deliberately a pointer. It reports which specification is furthest from the standard you set, and the Style Guide guide covers the dimensions, the weights and the thresholds behind it.