Application Lifecycle

A process starts, does work, and stops. The interesting part is the stop, because it is the only phase that happens on someone else's schedule. Your application does not decide when a deployment replaces it, when an orchestrator moves it to another node, or when an operator restarts it.

Graceful shutdown is the behaviour of stopping well when asked: stop accepting new work, let work already in progress finish, release what you hold, and exit. The opposite is being killed mid-operation, which turns a routine deployment into a small pile of half-finished work.

The host implements the sequence. What you have to do is honour it — which in practice means forwarding cancellation tokens and keeping shutdown work short.

public sealed class EmployeeImportJob(
    IHostApplicationLifetime lifetime,
    IServiceScopeFactory scopeFactory,
    ILogger<EmployeeImportJob> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        lifetime.ApplicationStopping.Register(
            () => logger.LogInformation("Stop requested; finishing current batch"));

        try
        {
            using IServiceScope scope = scopeFactory.CreateScope();
            var service = scope.ServiceProvider.GetRequiredService<EmployeeService>();

            int imported = await service.ImportPendingAsync(stoppingToken);
            logger.LogInformation("Imported {EmployeeCount} employees", imported);
        }
        catch (OperationCanceledException)
        {
            logger.LogWarning("Import interrupted by shutdown; remaining records will be retried");
        }
        finally
        {
            // The work is finished, so ask the host to shut down
            lifetime.StopApplication();
        }
    }
}

• ApplicationStopping fires as soon as a stop is requested, before the drain period. Registering a callback there is how you log the transition or flip a health endpoint to unhealthy, and it must return quickly because the shutdown clock is already running.
• OperationCanceledException during shutdown is not a failure. Catching it separately and logging at Warning keeps it out of your error rate, which matters because every deployment would otherwise produce errors.
• StopApplication requests shutdown from inside the application. This is the shape of a job: do the work, then ask the host to stop, and the process exits with a success code. Without it, a job-style application finishes its work and then sits there indefinitely.
• The comment on the finally block is worth reading twice: StopApplication requests a stop, it does not perform one. The host still runs the full shutdown sequence, which is what makes this safe.

HostApplicationBuilder builder = Host.CreateApplicationBuilder(args);

builder.Services.Configure<HostOptions>(options =>
{
    // How long the host waits for in-flight work during shutdown.
    // The default is 30 seconds. Match it to your longest reasonable unit of work.
    options.ShutdownTimeout = TimeSpan.FromSeconds(25);
});

builder.Services.AddHostedService<LeaveRequestConsumer>();

// A consumer that finishes the message it is holding, then stops
public sealed class LeaveRequestConsumer(
    IMessageQueue queue,
    IServiceScopeFactory scopeFactory) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            LeaveRequestMessage? message = await queue.ReceiveAsync(stoppingToken);
            if (message is null)
            {
                continue;
            }

            using IServiceScope scope = scopeFactory.CreateScope();
            var service = scope.ServiceProvider.GetRequiredService<EmployeeService>();

            // Deliberately not passing stoppingToken: once started, this finishes
            await service.ApplyLeaveRequestAsync(message, CancellationToken.None);
            await queue.AcknowledgeAsync(message, CancellationToken.None);
        }
    }
}

• ShutdownTimeout is the host's patience. When it expires, the host stops waiting and the process exits with work still in progress. Longer is not automatically better: it delays every deployment and can exceed the grace period your platform allows.
• The token is passed to ReceiveAsync, so waiting for a new message is interrupted immediately on shutdown. That is the part you want cancelled promptly — waiting for work is not work.
• It is deliberately not passed to ApplyLeaveRequestAsync. Once a message is being processed, cancelling halfway through risks a partial change with no acknowledgement. Finishing takes a bounded amount of time and leaves the system consistent.
• That distinction is the whole craft of graceful shutdown: cancel the waiting, finish the working. Forwarding the token everywhere is not the goal, and neither is ignoring it.
• The cost of finishing is that your unit of work must be shorter than the shutdown timeout. If a message can take two minutes, either shorten the unit of work or accept that deployments will interrupt it.

This matters most during a rolling deployment, which is how most services are replaced. The platform starts a new instance, waits for it to report healthy, then asks an old instance to stop. That request is a SIGTERM signal to the container, and the host translates it into the sequence above.

The platform does not wait indefinitely. It allows a termination grace period — 30 seconds by default on Kubernetes — and then sends SIGKILL, which cannot be caught, handled or delayed. Whatever the process was doing stops mid-instruction.

The number that bites is the relationship between the two. If your ShutdownTimeout is longer than the platform's grace period, your careful drain is cut short by a kill you did not plan for, and the graceful path is never actually exercised. Keep the application's timeout comfortably below the platform's, and set the platform's from how long your work genuinely takes.

There is a second gap worth knowing about, because it produces errors that look like a shutdown bug and are not. A load balancer needs time to notice an instance is going away. If the process exits the instant it receives SIGTERM, traffic is still being routed to it for a few seconds and those requests fail. The usual answer is to fail readiness immediately on ApplicationStopping and then keep serving for a short, deliberate period before draining.