Skip to main content
Tobias Lekman13 min read

Quality You Can Prove

Regulated teams can show what they tested and why. The practices behind that, from GAMP 5, work for any product, sized to the risk.

Listen to this article · 14:18

“We Tested It” Is Not an Answer

Picture two incident reviews. In the first, a payment export sent the wrong amounts to a bank for a day. Someone asks whether the export was tested, and the answer is yes. It has unit tests, coverage is high, and the pipeline was green. Nobody can say which test checked amounts in a currency without decimals, because none did, and nobody had written down that it mattered.

In the second, an auditor on a pharmaceutical project asks the same question about a different system. The team opens the trace matrix and finds the requirement, the risk it was given, the test that covers it, and the approved result from the release now in production. The conversation takes five minutes.

The second team is not better at testing. They work to a discipline that makes them say up front what matters, test that, and keep the proof. In life sciences that discipline is GAMP 5, and almost all of it works for products that will never meet a regulator.

This post is for engineering leads, product owners and CTOs who build software outside regulated industries and want quality they can show, not only claim. If you already work in GxP, the ideas will be familiar. The useful part may be how far down you can scale them.

What GAMP 5 Is, in Plain Terms

GAMP 5 is a guide from ISPE, the International Society for Pharmaceutical Engineering, on how to show that a computerised system is fit for its intended use. Pharmaceutical companies use it because regulators expect proof that software affecting patient safety, product quality or data integrity works as intended.

It has a reputation for paperwork, and some of that is earned. The core of it is short:

  1. Know what the system is for.
  2. Find out where it can do harm.
  3. Spend your testing effort where the harm is.
  4. Keep evidence that you did.
  5. Keep all of that true when the system changes.

The second edition, published in 2022, leans further this way. It asks for critical thinking over documents for their own sake, accepts exploratory testing where the risk is low, and lets you rely on a supplier’s own testing instead of repeating it. Read like that, it describes good engineering judgement and asks you to write it down.

Where Teams Usually Start

Outside regulated work, most teams measure quality by what is easy to count: the number of tests, coverage and a green pipeline. That is a natural place to start, and it catches a lot. It limits you when the product grows, because coverage tells you which lines ran, and nothing about which promises were checked. A suite with 90 per cent coverage can miss the one rule that would have stopped the incident, because nobody wrote the rule down.

Coverage tells you which lines ran. It does not tell you which promises were checked.

Regulated teams often fail the other way. They write a document for every screen and take a screenshot of every click, so the effort goes into paperwork instead of the risky parts. Both problems have the same fix.

Six Practices That Travel

1. Write Down the Intended Use

Before you test anything, write what the feature is for, who uses it, and what it must never do. GAMP calls this the intended use, and everything else hangs off it. “Exports the day’s payments in the bank’s file format” is a start. “Never changes an amount, a currency or a payee” is the line that becomes your most important test.

2. Assess the Risk of Each Function

GAMP asks what happens to patient safety, product quality and data integrity if a function fails. Swap in your own three: harm to a customer, lost money, and data you cannot get back. For each function, ask how bad a failure would be, how likely it is, and whether you would notice before a customer does. A function where failure is severe, plausible and silent is high risk. A typo on a settings page is not.

You do not need a scoring spreadsheet to start. A label of low, medium or high on each backlog item, agreed with the product owner, is enough to change where the effort goes.

3. Size the Effort to the Risk

This practice makes the rest affordable. High risk functions get tests written from the requirement, with expected results, and someone reviews the outcome. Low risk functions get exploratory testing, or nothing beyond what the pipeline already runs.

GAMP also sizes the effort by how much of the software is yours. It separates infrastructure, software you use as it comes, software you configure, and software you write yourself, and expects most of the testing where you wrote the code. The same holds for any product. You do not need to retest how your cloud provider’s queue works, but you do need to test the configuration you gave it and the code that reads from it.

Assurance sized to the risk

The same practices at every level. Only the depth changes.

  • Intended use

    Low riskOne line in the backlog item.

    Medium riskA short statement per feature, with what it must never do.

    High riskWritten and agreed with the owner before anything is built.

  • Testing

    Low riskExploratory testing, or the supplier's own evidence.

    Medium riskAutomated tests in the pipeline.

    High riskAutomated tests written from the requirement, plus a review of the results.

  • Traceability

    Low riskNone needed.

    Medium riskThe requirement ID in the test name.

    High riskRequirement, risk, test and result linked, and checked on every release.

  • Evidence

    Low riskThe pipeline log.

    Medium riskTest reports kept with each release.

    High riskReports kept with each release, with who approved it and when.

  • On change

    Low riskRun what the pipeline runs.

    Medium riskRerun the affected tests.

    High riskAssess the risk again, then rerun the affected tests.

4. Make the Trace Cheap

Traceability is the link from a requirement, through its risk, to a test and its result. Laid out as a matrix, it gives one answer per requirement: proven, failing or open. A requirement with no test shows up as a row, before a customer finds the gap.

A trace matrix. Five steps link a requirement to its risk, its test, the test’s result and the evidence. In the example matrix, two requirements are proven by passing tests, one high risk requirement has a failing test, one has a test that was not run, and one has no test. A reviewer reads two proven, one failing and two open, and the release waits on the high risk rows that are not proven.

In regulated work the matrix often lives in a spreadsheet that drifts out of date. You can keep it in the code instead. Give each test an ID, say which requirements it covers and how risky they are, and record every check it makes:

export.qualify.test.ts
import { oq } from "@lekman/qualify/vitest";
oq(
"OQ-EXPORT-003",
"The bank file never changes an amount",
{
given: "a day of payments in three currencies",
when: "the export writes the bank file",
then: "every amount in the file equals the amount in the ledger",
requirements: ["EXPORT-003"],
risk: "high",
scope: "read",
},
(ctx) => {
const file = exportBankFile(ledger);
ctx
.check("bank file", "EUR keeps two decimals")
.that(file[0])
.equals("Acme Ltd;EUR;100.50");
}
);
test_export.py
import qualify
@qualify.oq(
"OQ-EXPORT-003",
"The bank file never changes an amount",
given="a day of payments in three currencies",
when="the export writes the bank file",
then="every amount in the file equals the amount in the ledger",
requirements=["EXPORT-003"],
risk="high",
scope="read",
)
def test_bank_file_keeps_amounts(ctx: qualify.Context, ledger) -> None:
file = export_bank_file(ledger)
(
ctx.check("bank file", "EUR keeps two decimals")
.that(file[0])
.equals("Acme Ltd;EUR;100.50")
)
BankFileTest.kt
import com.lekman.qualify.OQ
import com.lekman.qualify.QualifyContext
import com.lekman.qualify.Risk
import com.lekman.qualify.Scope
class BankFileTest {
@OQ(
id = "OQ-EXPORT-003",
title = "The bank file never changes an amount",
given = "a day of payments in three currencies",
`when` = "the export writes the bank file",
then = "every amount in the file equals the amount in the ledger",
requirements = ["EXPORT-003"],
risk = Risk.HIGH,
scope = Scope.READ,
)
fun bankFileKeepsAmounts(ctx: QualifyContext) {
val file = exportBankFile(ledger)
ctx.check("bank file", "EUR keeps two decimals")
.that(file[0])
.isEqualTo("Acme Ltd;EUR;100.50")
}
}
BankFileTests.cs
using Lekman.Qualify;
public class BankFileTests
{
[OQ("OQ-EXPORT-003", "The bank file never changes an amount",
Given = "a day of payments in three currencies",
When = "the export writes the bank file",
Then = "every amount in the file equals the amount in the ledger",
Requirements = ["EXPORT-003"], Risk = Risk.High, Scope = Scope.Read)]
public void BankFileKeepsAmounts()
{
var ctx = QualifyContext.Current;
var file = BankFile.Export(ledger);
ctx.Check("bank file", "EUR keeps two decimals")
.That(file[0])
.IsEqualTo("Acme Ltd;EUR;100.50");
}
}

The test still runs in the test runner you already use, and the same runner selects it by ID when that requirement changes: vitest -t OQ-EXPORT-003, pytest -k OQ-EXPORT-003, a JUnit tag in Gradle, or dotnet test --filter-display-name "[OQ-EXPORT-003]*". As it runs, each test writes a record of what it claimed and what it checked, and a trace matrix is built from those records.

These examples come from qualify, part of my qa-toolkit on GitHub. It has the same test harness for TypeScript, Python, Java and Kotlin, and C#, one evidence format they all write, and the report shown in the next section. It is free under the MIT licence: use it as it is, or take the shape and change it to fit your own tests. The specification in the repository describes the evidence format, so a version you build yourself can still produce records the same tools can read.

On a regulated product I work on, the operational qualification tests, which check that the deployed system does its job in its real environment, are named this way and run in the pipeline after every deployment. The trace matrix is a report generated from the test results, so it cannot drift from what actually ran. Nothing about that setup needs a regulator.

5. Keep Evidence Someone Else Can Read

Regulated teams keep records to a set of principles called ALCOA+: attributable, legible, contemporaneous, original and accurate, plus complete, consistent, enduring and available. For a product team, that means test results tied to a version, kept with the release, with who approved it and when, for as long as someone might ask a question about it. Most CI systems delete test reports after a few weeks, so that is the first setting to check.

In practice, each test can write its own record as it runs: what it claimed, what it checked, what it found, and when. This is the record for the high risk test from the example above, trimmed to one check:

OQ-EXPORT-003.json (trimmed)
{
"id": "OQ-EXPORT-003",
"level": "OQ",
"title": "The bank file never changes an amount",
"given": "a day of payments in three currencies",
"when": "the export writes the bank file",
"then": "every amount in the file equals the amount in the ledger",
"requirements": ["EXPORT-003"],
"risk": "high",
"outcome": "pass",
"startedAt": "2026-11-20T08:17:37.788Z",
"checks": [
{
"context": "bank file",
"description": "EUR keeps two decimals",
"matcher": "equals",
"expected": "Acme Ltd;EUR;100.50",
"actual": "Acme Ltd;EUR;100.50",
"pass": true
}
]
}

Next to it, the run records the commit, the environment and who started it. JSON is for machines and for the next tool in the chain. A reviewer wants to read it, so the same records render as a report: a cover page, the run details, the trace matrix, one section per test, and a signature block. Select a page to open it:

Payment export: operational qualification5 pagesDownload the PDF (65 KB)

The report says what the JSON says, in a form someone can sign and file. Notice the last row of the trace matrix on page 2: the high risk requirement whose test never ran is listed on its own, where a reviewer cannot miss it.

The test for this practice is simple. Could someone outside the team find out, six months from now, what was tested in the release that is in production today?

6. Keep It True After Change

A validated system stays validated only while you control how it changes. GAMP expects every change to be assessed for what it affects, the right tests to run again, and a periodic review of whether the system still does what it is for. Your pipeline already runs the tests. The habit to add is one question on each pull request: does this change touch a high risk function? If it does, the tests for that requirement run again and a second person reviews it.

What to Leave Behind

Not everything travels. Ink signatures, a screenshot for every test step and a validation document for every screen answer regulatory questions you do not have. If a practice produces paper nobody reads, drop it. The point is evidence someone would actually use.

The same goes for what your tools produce. Test frameworks, coverage reports and now AI assistants can generate pages of documentation for every change, and most of it is never opened. Each page still has to be stored, kept current and explained when someone asks about it, so it adds work for the team without adding confidence. Be deliberate about it. State in your organisation’s policies or engineering practices exactly what a release needs: which tests, which records and which approvals, for each level of risk. Anything outside that list is optional. Do not let your colleagues drown in documentation and process to prove they did their job. The goal is adequate quality and risk reduced to a level you can accept, and the paperwork exists only to show you reached it.

Scaling Up or Down

A two person startup and a bank should not do the same amount of this, and GAMP agrees. Start with labels and IDs. As the product handles more money, more customers or more sensitive data, move functions to the right in the table above. More of them get tests written from the requirement, evidence is kept longer, and a second person reviews the result. You can scale the other way too, and move a function back to low risk when it stops mattering.

This pairs with how you measure value. In Measure Value From Day One I argued for writing test cases for whether the work was worth doing. The practices here are the other half: test cases for whether the product does what it is for, safely.

Where to Start

  1. Pick the three functions in your product that would hurt most if they failed silently.
  2. Write one line of intended use for each, including what it must never do.
  3. Give each a requirement ID, put the ID in the names of the tests that cover it, and write the tests that are missing.
  4. Check how long your CI keeps test reports, and keep the reports for each production release for at least a year.
  5. Add one question to your pull request template: does this touch a high risk function?

None of this needs a regulator, a quality department or a new tool. It needs you to decide what matters before you test, and to keep the proof afterwards.

Tobias Lekman
Tobias Lekman
Cloud & Security Architect · MD

25 years building secure digital solutions for regulated and modern teams.

Work with us →