Tests in the Pipeline
By the end of this lesson
Run tests on every change and keep them reliable.
A test suite that runs when somebody remembers to run it protects nobody. The value arrives when every change is checked automatically, before it reaches a branch other people build on.
Two requirements make that real, and both are about behaviour rather than configuration. The suite has to run on every push and every pull request, so nothing merges unverified. And a failure has to stop the change, because a red build that people merge anyway is a slower way of having no tests.
Everything else in this lesson follows from those two, including the one that teams find hardest: a test that fails intermittently has to be treated as a bug, not as something to re-run.
name: Tests
on:
push:
branches: [main]
pull_request:
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: 9.0.x
- run: dotnet restore
- run: dotnet build --no-restore --configuration Release
# Fails the job on the first failing test, which fails the workflow
- run: >
dotnet test Ordering.Domain.Tests
--no-build --configuration Release
--logger "trx;LogFileName=unit.trx"
--results-directory ./test-results
- uses: actions/upload-artifact@v4
if: always()
with:
name: unit-test-results
path: ./test-results
integration-tests:
runs-on: ubuntu-latest
needs: unit-tests
services:
sqlserver:
image: mcr.microsoft.com/mssql/server:2022-latest
ports: ["1433:1433"]
env:
ACCEPT_EULA: "Y"
# Container lives for one job only. Anything with real access
# belongs in the pipeline's secret store, never in this file.
MSSQL_SA_PASSWORD: LocalCiOnly_ReplaceMe1
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: 9.0.x
- run: dotnet test Ordering.IntegrationTests --configuration Release
env:
ConnectionStrings__Ordering: "Server=localhost,1433;Database=ordering_ci;User Id=sa;Password=LocalCiOnly_ReplaceMe1;TrustServerCertificate=True"- The two triggers are the important part. push covers the main branch, pull_request covers every proposed change, and together they mean nothing reaches main without the suite running.
- dotnet test returns a non-zero exit code when any test fails. The step fails, the job fails, the workflow fails, and the pull request shows as blocked. Nothing extra is needed to make a failure stop the change — the mistake is adding something that prevents it.
- --no-build in the first job reuses the compiled output from the build step. On a large solution this saves a minute or two per run, and it also proves the Release configuration compiles before any test runs.
- The trx logger writes structured results, and uploading them with if: always() means the artefact exists precisely when you need it — after a failure. Without if: always() the upload is skipped on the run you wanted it for.
- Integration tests get their own job with a database as a service container, so the fast suite reports in under a minute while the slower one runs separately. needs: unit-tests makes it wait, because there is no point paying for a database when the unit tests have already found a problem.
- The password is a placeholder for a container that exists for the length of one job. It is written in the file only because both the service and the test need the same value; any credential that reaches a real system belongs in the pipeline's secret store and is referenced, never pasted.
What a pipeline has to provide before anyone will trust it:
- A result on every pull request, visible in the request itself, so a reviewer is not deciding whether to check.
- A failure that blocks the merge. Configure the branch to require the check; otherwise blocking is a matter of etiquette.
- A named failing test in the output. "Job failed" sends someone digging through logs; a test name and an expected-versus-actual message is a diagnosis.
- Fast feedback on the common path. If the unit suite takes fifteen minutes, people stop waiting for it and start merging on hope.
- Retained results as artefacts, so a failure can be investigated after the fact rather than reproduced from memory.
- The same commands that work locally. A pipeline that passes only with special arguments is a pipeline nobody can debug.
A flaky test is a bug. It passes and fails against the same code, and it is the single most damaging thing that can happen to a suite — where the damage is not the failed build but what the team learns from it.
The first time an unexplained red turns green on a re-run, somebody notices that re-running works. After that, re-running is what happens — and it happens for real failures too, because from the outside they look identical. The suite still runs, still reports, and no longer stops anything. That is worse than having no pipeline, because the badge says otherwise.
So treat an intermittent failure the way you would treat an intermittent bug in production, because that is what it usually is. A test that fails one run in twenty is often reporting a genuine race, an ordering assumption or a time-dependent behaviour in your code. Sometimes the test is at fault instead. Both are defects; neither is fixed by running it again.
A response to a flaky test that does not end in a retry:
Record it before it disappears
Open an issue with the test name, the failure message and a link to the run. Flakes are forgotten between occurrences, so the third one looks like the first and nobody knows it is the third.
Reproduce it deliberately
Run the test in a loop, run it with the rest of its class, and run it with parallelisation disabled. Which of those changes the outcome tells you whether you are looking at shared state, concurrency or something time-dependent.
Check the usual causes first
Shared mutable state between tests, the system clock, random data, an ordering assumption, an unawaited task, or two tests using the same database rows.
Decide whether the code or the test is wrong
A race the test exposed is a production bug and the more important find. An assertion on something the code never promised — an ordering the query does not guarantee — is a test bug. Say which one you concluded, in the issue.
If you cannot fix it now, quarantine it with a deadline
Exclude it from the blocking job by trait, with the issue reference and a date in the comment. It keeps running where someone still sees it, and the deadline is what stops a quarantine becoming permanent.
Never make a retry the resolution
An automatic retry policy converts a visible intermittent failure into an invisible one. The behaviour is still there, the evidence is gone, and it will next appear in production.
Summary
- Run the suite on every push and every pull request, and require the check so a failure genuinely blocks the merge
- dotnet test fails the job by exit code; keep test results as artefacts with if: always() so failures can be investigated
- Give integration tests their own job with a service container, so the fast tier stays fast
- Treat a flaky test as a bug: reproduce it, decide whether the code or the test is wrong, and quarantine with a deadline if you must
- An automatic retry converts a visible intermittent failure into an invisible one and teaches the team to ignore the suite
Practice
Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.
Think about it
The test that fails twice a week
One integration test fails roughly twice a week, always on the pipeline and never locally. A re-run fixes it every time. Someone proposes adding an automatic retry for that test so the team stops losing time.
What would you argue, and what would you do instead?
Show solution
Start with what the flake is evidence of. Failing only under the pipeline's conditions — slower machine, containers starting concurrently, tests running in parallel — is the signature of a race or a shared-state problem. Those are production defects more often than test defects, and "it only happens under load" describes the incident you have not had yet.
The retry hides exactly the signal you need. After it, the behaviour still exists and nothing reports it, so the next occurrence is in production with a customer attached.
What to do instead: reproduce it under pipeline-like conditions. Run the test in a loop, run it in parallel with its siblings, run it against a cold container. If the test asserts on something the code never promised — a result order the query does not define — the test is wrong and should be tightened. If two tests share rows, isolate the data. If your code has a race, that is the find of the week.
If it cannot be fixed this week, quarantine it out of the blocking job with the issue number and a review date, and say so in the standup. That is honest — the team knows one area is unverified — whereas a retry claims coverage that does not exist.
Worth conceding: the time being lost is real, and the proposal comes from a reasonable place. The disagreement is about which cost is larger, and an invisible intermittent defect is the larger one.
Try it yourself
Split a slow pipeline
Your single test job now takes 22 minutes: 400 unit tests (90 seconds), 120 integration tests (7 minutes), and 40 browser tests (13 minutes). People have started merging without waiting.
Propose a job layout and triggers. State what each job blocks, and what you are accepting as a consequence.
Show solution
A workable layout: job one runs the unit tests on every push and pull request and blocks the merge — 90 seconds is a wait people will tolerate. Job two runs the integration tests on the same triggers and also blocks; seven minutes is acceptable for a check that catches wiring and migration problems, and it can run in parallel with job one rather than after it.
Job three keeps five browser tests as a smoke set and runs them after deployment to staging, blocking promotion rather than the merge. The remaining 35 run on a nightly schedule against staging.
What that accepts, stated plainly: a defect that only the full browser suite catches is found the following morning. In exchange, every merge is checked by something people wait for, which is the property the current 22-minute job has already lost.
Two things to do alongside the split. Ask why 120 integration tests are needed — some of them are probably covering logic that belongs in the unit suite, and moving them down improves both jobs. And configure the repository to require the two blocking checks, because otherwise "blocks the merge" is a convention rather than a rule.
# Fast tier: blocks every merge
on:
push:
branches: [main]
pull_request:
jobs:
unit-tests: # ~90s, required check
runs-on: ubuntu-latest
steps: [...]
integration-tests: # ~7m, required check, runs alongside unit-tests
runs-on: ubuntu-latest
services:
sqlserver: { image: "mcr.microsoft.com/mssql/server:2022-latest" }
steps: [...]
---
# Slow tier: separate workflow, not on the merge path
on:
deployment_status: # 5 smoke journeys after a staging deploy
schedule:
- cron: "0 2 * * *" # the remaining 35, nightlyKnowledge check
Nothing is recorded and there is no score. The explanation appears either way.
Saved in this browser only.
End of the published lessons
That is everything written so far in Testing
More lessons in this course are on the way. In the meantime, the course page shows the full roadmap, and the projects are the best way to consolidate what you have covered.