Skip to main content
ANVISoftware Solutions
Lesson 16 of 16Advanced20 min

Production Considerations

By the end of this lesson

Add health checks, resource limits and sensible logging.

An image that starts on your machine is not yet an image something else can operate. Whatever runs it in production — an orchestrator, a managed container service, or Compose on a single host — needs four things from the container that development never asked for.

It needs to know whether the application inside is working, not only whether the process is alive. It needs boundaries on memory and CPU so one container cannot starve its neighbours. It needs the logs somewhere it can collect them. And it needs the container to shut down in an orderly way, because containers are replaced routinely rather than exceptionally.

The logging point deserves stating up front, because it changes how you write the application rather than how you run it. Write log lines to standard output and standard error and nothing else. Every container platform captures those two streams and forwards them to wherever logs are kept, so a line written to stdout ends up searchable alongside every other service. A line written to a file inside the container is in a filesystem that disappears with the container, on a host nobody is watching, competing for disk with everything else on the machine.

Dockerfile — the last few lines, with a health check
Dockerfile
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS final
WORKDIR /app
COPY --from=build --chown=appuser:appuser /app/publish .

ENV ASPNETCORE_HTTP_PORTS=8080
EXPOSE 8080
USER appuser

# Ask the application whether it is working. Not whether the process exists.
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD ["dotnet", "/app/Anvi.Employees.Api.dll", "--health-check"]

ENTRYPOINT ["dotnet", "Anvi.Employees.Api.dll"]
  • HEALTHCHECK runs a command inside the container on a schedule. Exit code 0 means healthy and 1 means unhealthy; docker ps then shows the status next to the container.
  • --interval is how often the check runs, and --timeout is how long a single check may take before it counts as a failure. Keep the timeout well below the interval so checks cannot overlap.
  • --start-period is a grace window after start-up during which failures do not count against the container. An application that spends 15 seconds warming up would otherwise be marked unhealthy before it ever had a chance.
  • --retries is how many consecutive failures are needed before the status changes. More than one absorbs a single slow response without declaring an outage.
  • The command here calls back into the application rather than using curl, because a minimal runtime image has no curl and adding one puts a network tool into production for the sake of a health check. Implementing a --health-check switch that exercises your readiness logic and returns an exit code keeps the dependency inside your own code.
  • A health check should test something meaningful and stay cheap. Confirming the application can serve a request and reach its database is useful. Running a report on every interval is a self-inflicted load, and a check that calls three other services turns their problems into your unhealthy status.

Docker's HEALTHCHECK and an orchestrator's probes answer the same question in different places, and when both exist only one of them has any authority.

 HEALTHCHECK in the imageOrchestrator probe (for example Kubernetes)
Defined byThe Dockerfile, so it travels with the imageThe deployment manifest, separate from the image
Who acts on the resultDocker sets a status. Compose can wait on it with condition: service_healthyThe orchestrator restarts the container, or removes it from load balancing
Does an unhealthy container get restarted?Not by Docker on its own — the status is reported, not enforcedYes, a failing liveness probe restarts it
Is traffic withheld while unhealthy?No. A published port keeps accepting connectionsYes, a failing readiness probe takes it out of rotation
Where it is authoritativeLocal development, Compose, single-host deploymentsAnywhere an orchestrator is running the container
When both existTypically ignored. Kubernetes does not read itThis is the one that decides

Memory and CPU limits both cap consumption, and they fail in completely different ways. Confusing the two leads to a long search for the wrong problem.

 Memory limitCPU limit
What happens at the limitThe kernel kills the process that asked for moreThe process is throttled — it waits for its next slice of time
Is it survivable?No. The container dies immediately and restartsYes. Everything keeps working, more slowly
How it looksAn abrupt exit, often with code 137, and no error in the application logsRising response times, timeouts downstream, no crash
Where to lookdocker inspect shows OOMKilled true on the container's statedocker stats shows CPU pinned at the ceiling you set
How to set it--memory 512m, or a memory limit in the platform's manifest--cpus 0.5, meaning half of one core's time
Getting the number wrongToo low restarts a healthy application under normal loadToo low makes a healthy application slow in a way that is easy to misdiagnose
compose.yaml — the operational settings on the API service
YAML
services:
  api:
    image: anvi/employees-api:1.5.0
    environment:
      ASPNETCORE_ENVIRONMENT: Production
      ASPNETCORE_HTTP_PORTS: "8080"
      Logging__LogLevel__Default: Information
    ports:
      - "8080:8080"

    # Restart on failure, but stop trying if it fails immediately and forever
    restart: on-failure:5

    # How long to wait after SIGTERM before the process is killed
    stop_grace_period: 30s

    # Collected from stdout and stderr, with a cap so a loop cannot fill the disk
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "0.75"
        reservations:
          memory: 256M
  • restart: on-failure:5 restarts the container when it exits non-zero, up to five times. Unlimited restarts hide a crash loop behind a container that looks like it is running; a cap makes the failure visible.
  • stop_grace_period is the window between the stop signal and the kill. The default is 10 seconds, which is short for an application finishing a database transaction. Set it slightly longer than your slowest reasonable request, and no longer — the platform waits this long for every replacement.
  • The logging block configures Docker's json-file driver, which captures stdout and stderr. max-size and max-file rotate the files and cap total disk use. Without them, one badly behaved loop can fill a host's disk, which takes down every container on that host rather than only yours.
  • On a real platform the driver is usually not json-file — logs go to the cloud provider's logging service or a collector. The application does not change either way, which is the point of writing to stdout: the destination is an operational decision made outside your code.
  • deploy.resources.limits sets the ceilings from the comparison above. Compose on a single host applies these; a swarm or orchestrator reads the same block. The memory number is a hard stop, so leave headroom above the application's observed peak.
  • reservations is a floor rather than a ceiling — the amount the platform sets aside for this service when deciding what fits on a host. It does not restrict the container.
  • Deliberately absent: no build section and no secret values. This is a deployment description referencing an image that was built and scanned elsewhere, with configuration and credentials supplied by the environment.

Summary

  • A health check should test whether the application works, not whether the process exists — and stay cheap enough to run every interval
  • HEALTHCHECK in the image is authoritative under Compose; an orchestrator's probes take over and act on the result
  • A memory limit kills the container (exit 137, OOMKilled) while a CPU limit only throttles it, so the symptoms differ completely
  • Write logs to stdout and stderr so the platform collects them, and cap log size so one loop cannot fill a host's disk
  • Handle SIGTERM and set a grace period longer than your slowest request, or every deployment drops work in flight

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Try it yourself

Trigger the two failure modes on purpose

Run the employees API with --memory 64m, which is below what it needs. Watch it exit, then inspect the container's state and find the OOMKilled flag and the exit code.

Run it again with a sensible memory limit and --cpus 0.1. Send it a burst of requests and compare response times against an unrestricted container while watching docker stats.

Show solution

The memory case ends the container abruptly. The application logs show nothing useful, because the process was killed rather than given a chance to report anything, and the exit code is 137. That combination — no error, code 137, OOMKilled true — is the signature to recognise, and recognising it saves you from reading the application's code looking for a bug that is not there.

The CPU case never fails. It gets slow, and the slowness appears as timeouts in whatever calls the API. Nothing in the container's own logs says the CPU was capped, which is why docker stats and the limit you set are the first two things to check when a container is inexplicably sluggish.

The reason to cause both deliberately is that you will meet them under pressure otherwise. Killed and throttled look nothing alike once you have seen each one, and they look identically mysterious if you have not.

Shell
docker run -d --name api-starved -p 8080:8080 --memory 64m anvi/employees-api:1.5.0
sleep 20
docker ps -a --filter name=api-starved
docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' api-starved
docker logs api-starved
docker rm -f api-starved

docker run -d --name api-throttled -p 8080:8080 --memory 512m --cpus 0.1 anvi/employees-api:1.5.0
docker stats --no-stream api-throttled
docker rm -f api-throttled

Think about it

Think about it

The employees API has an endpoint that generates a payroll export and takes up to 45 seconds. The service is deployed with the default 10-second grace period, and deployments happen twice a week during working hours.

Describe what a user running an export sees during a deployment, and set out two different ways to fix it.

Show solution

The container is told to stop, stops accepting new requests, and waits. Ten seconds later it is killed with the export half finished. The user sees the connection drop with no error message from the application, and a retry may hit a container that is also about to be replaced. Reported as a bug it looks intermittent and unreproducible, because it only happens during a deployment.

The direct fix is to raise the grace period above the longest request — 60 seconds, say. It is one line and it works. The cost is that every deployment is slower by up to that amount for every container, and you have made deployment speed depend on your slowest endpoint. A 45-second request also holds a connection and a thread for 45 seconds, which is its own scaling problem.

The structural fix is to stop doing long work inside a request. The endpoint accepts the job, returns an identifier immediately, and a background worker produces the export; the client polls or is notified. Now no request lasts more than a moment, the grace period can stay short, and a container replacement interrupts a job that can be retried rather than a user's connection. The cost is real: another moving part, somewhere to store job state, and a more involved client.

Both are defensible. The first is right if this is the only slow endpoint and deployments are infrequent. The second is right if long operations are a normal part of the product, and it is the one that keeps working as the application grows.

Saved in this browser only.

End of the published lessons

That is everything written so far in Docker

More lessons in this course are on the way. In the meantime, the course page shows the full roadmap, and the projects are the best way to consolidate what you have covered.