# Automated API Testing in CI/CD — Routebase

> Where the gate belongs, why exit codes are a contract, how to make a failure readable in the pipeline, and how to fix flakiness before people learn to rerun.

Canonical page: https://routebase.dev/guides/api-testing/automated-api-testing-in-ci-cd/
Chapter 9 of 12 · API Testing · Last reviewed 2026-09-12 · The Routebase Team

A suite that only runs when somebody remembers to click is a suite that runs on the days it is least needed. Automated API testing is not the sophisticated part of the discipline, and it is the part that decides whether any of the rest of it has an effect.

## The gate belongs before the merge

Put the fast tests where a failure costs the least. That is the pull request, against a preview or staging environment, before anyone has built on top of the change.

_Figure: Each stage runs what it can afford, and only the pull request holds a gate, because that is where a regression still costs a rebase._

| When | What runs | Why there |
|---|---|---|
| Pull request | Functional and contract tests, plus a light security pass | Finishes inside a review wait, and a fix is still a rebase |
| Merge to main | The same suite against staging | Catches what only appears after integration |
| After deploy | Smoke suite plus a contract check against the deployed environment | Confirms the release actually serves what it promised |
| Nightly | Full suite, load profile, full security profile | Has time to be thorough, blocks nobody |

The rule behind the table is that a gate must be fast enough that nobody resents it. Ten minutes is where people start looking for the flag that skips it, and a skipped gate protects nothing.

## Exit codes are a contract

A test runner in a pipeline communicates almost entirely through its exit code, so the meaning of each code deserves the same care as an API response.

The distinction that matters is between "the tests ran and failed" and "the run never happened". Both are non-zero, and they mean completely different things to whoever is looking at the red build.

| Code | Meaning | What the pipeline should do |
|---|---|---|
| 0 | Everything passed | Continue |
| 1 | Tests ran, something failed | Fail the build and show the report |
| Higher | Configuration, network, authentication or a missing resource | Fail the build, and say the run did not happen |

A pipeline that branches only on zero against non-zero will report an expired credential as a test failure. Somebody then spends an hour reading assertions before noticing the run never reached the API.

Once published, those codes are part of your CI contract. Changing what a code means breaks every pipeline that branched on it, which makes it a breaking change even though no API was touched.

## Make the failure readable where people look

The default output of most runners is a log, and a log is the worst place to read a test result. Every CI platform can render a structured report, and the format they all understand is JUnit XML.

The difference is concrete. Without it, a reviewer sees a red step and opens 400 lines of output. With it, they see "3 of 47 tests failed" with the three names, the assertion that broke and the expected against actual values, in the pull request itself.

Security results have their own format, namely SARIF. A code-scanning tool that ingests SARIF puts findings on the lines and routes they belong to, and it recognises the same finding across runs instead of reporting it as new each time.

Upload both even when the gate fails. A failed step that also published its report explains itself, while a failed step that published nothing sends somebody digging through logs.

## Credentials for a pipeline

The pipeline needs an identity, and it should not be a person's.

Use a machine credential with only the permissions the step requires, which for a test run is the permission to execute tests in one project. A key that can also delete specifications is a key that will eventually be used by a script nobody reviewed carefully.

Three practices repay the small effort.

**Scope it to one project.** A credential restricted to a single project cannot touch the rest of the organisation, whatever the pipeline is asked to do.

**Store it as a pipeline secret, never in the repository.** This is obvious and it still happens, usually inside a snippet somebody pasted from a working example.

**Give it an expiry and rotate it.** A key with no expiry outlives the integration it was created for and stays valid long after the pipeline is gone.

## Flakiness is a defect, not weather

The real cost of a flaky test is not the failed build. It is that people learn to press rerun, and after a few weeks that reflex applies to real failures too.

Three causes account for most of it.

**Shared mutable state.** Two pipelines running against the same environment create the same record, or one deletes what the other is reading. The fix is data the run owns, which [test data](/guides/api-testing/api-test-data/) covers.

**Timing.** A wait that is long enough on a quiet afternoon is not long enough when the build agents are busy. Poll for the state you need with a deadline instead of sleeping for a duration.

**Assertions on values the API never promised.** Array order without an explicit sort, generated identifiers and exact timestamps are not contractual, so asserting on them is asserting on luck.

When one appears, quarantine it. Move it out of the gate so it stops blocking, keep it running so it stays visible, and fix the cause on a normal ticket. What must not happen is a permanent skip, because a permanently muted test is a deleted test with extra steps.

## Scheduled runs are not a substitute

A cron schedule and a pipeline gate answer different questions, and teams sometimes buy one and expect the other.

The gate answers whether this change is safe to merge. The schedule answers whether the deployed thing still works, including after changes nobody put through the pipeline, such as a configuration edit, a dependency update or an upstream provider changing behaviour.

Run both. The schedule wants a notification path that reaches a human, because a nightly run that fails into an empty inbox is a cron job and not a test.

## In Routebase

Every suite carries a CI/CD section that generates a ready-made snippet for GitHub Actions, Azure Pipelines, GitLab CI, Jenkins and Bitbucket Pipelines, each with its target filename. The snippet posts to the run endpoint with an API key, parses the response, and exits non-zero unless every test passed. Nothing has to be installed on the runner.

_Screenshot: The generated snippet reads the key from your CI secret store, so no credential is written into the workflow file._

The `Routebase.Cli` test runner is the shorter path where installing a tool is acceptable. One command runs a suite against a named environment and writes JUnit XML that your platform renders as a native test report, so failed assertions appear as failed tests. Its exit codes are a published contract, with zero for success, one for a failed run, and separate codes for configuration, network, authentication and missing resources. A key that authenticates but lacks a permission exits with the authentication code in every command, and that is the most common pipeline misconfiguration there is.

Security scans run from the same pipeline, either as three REST calls or as one CLI command. That command queues the scan, polls it, writes SARIF and sets the exit code from a severity threshold you choose. There is deliberately no server-side pass or fail verdict, because the threshold belongs to your team and not to the tool.

Where a schedule fits better than a pipeline, a suite runs on a cron expression in a timezone and against an environment you pick. Project-wide webhooks then push the outcome to your own systems, with an HMAC signature on every delivery. Scheduled runs land in the same history as manual ones, with the same export and sharing options.

Details are in [Scheduling](https://docs.routebase.dev/scheduling/), [CLI Overview](https://docs.routebase.dev/cli-overview/) and [CLI in CI/CD](https://docs.routebase.dev/cli-in-cicd/).

## Frequently asked questions

### How do you run API tests in a CI/CD pipeline?

Give the pipeline a scoped credential, point it at a deployed environment, and run the suite as one step that exits non-zero when a test fails. Publish the results in a format your platform renders as a test report, so a reviewer sees which assertion broke instead of a wall of log output. The whole thing is usually three lines of pipeline configuration once the suite exists.

### What should an API test runner's exit codes mean?

Exit zero for a run where everything passed and one for a run where the tests failed. Reserve the higher codes for the cases where the run never happened, such as a configuration problem, a network failure or a credential without the right permission. A pipeline that treats every non-zero code the same reports infrastructure problems as test failures, which sends people to debug the wrong thing.

### Should a failing API test block a merge?

Yes for the fast, deterministic tests, because that is the entire reason to run them before merge. Contract and functional tests against a preview environment finish inside a normal review wait and catch the regressions that are cheapest to fix. Load tests, long soak runs and the full security profile belong on a schedule, since a gate that takes ten minutes gets disabled.

### How do you deal with flaky API tests in CI?

Treat a flaky test as a defect in the test, not background noise, because the cost is that people learn to press rerun and stop investigating. Most flakiness comes from shared mutable state, from waits that are long enough today, or from assertions on values the API never promised. Quarantine the test so it stops blocking, fix the cause, and put it back, since a permanently muted test is a deleted test with extra steps.

---

[Routebase](https://routebase.dev/) — [Sign up](https://app.routebase.dev/): Every account starts with a 14-day Pro trial — no credit card required.