Lesson 18 of 18Advanced19 min

API Performance

By the end of this lesson

Measure latency and remove the common causes of slow endpoints.

Performance work starts with measurement, and the reason is not discipline for its own sake. Without numbers you will optimise the code you happen to be looking at, which is rarely the code that is slow. Every experienced developer has spent a day making something twice as fast that accounted for two percent of the response time.

The measurement to start from is latency per endpoint, recorded as a distribution rather than a single number. That distinction does more work than any individual optimisation, because an average is the most common reason a real problem stays invisible.

How to read a latency distribution:

p50, the median: Half of requests were faster than this. A reasonable description of a typical request, and it says nothing about the worst ones.
p95: One request in twenty was slower. This is usually where users start describing the application as unreliable rather than slow.
p99: One in a hundred was slower. It looks rare until you count calls: a screen making ten API calls hits a p99 response about ten percent of the time, so the slowest one percent shapes how the product feels.
The average, and why it misleads: Thousands of 20ms requests hide hundreds of 4-second ones inside an average of 120ms. Two systems with very different behaviour can report the same average, and a graph of averages can stay flat through a serious regression.
Throughput, separately: Requests per second is a different measurement from latency, and improving one can worsen the other. Track both, and know which one your problem is about.

The causes behind most slow endpoints, roughly in the order they turn up. None of them is exotic:

N+1 queries — one query for a list, then another for each row in it
Missing indexes, so a filter or a sort reads the whole table
Oversized payloads — returning every column and every related record when the caller uses four fields
No paging, so the response size grows with the table
No caching for data that changes rarely and is read constantly, such as a department list
Blocking on asynchronous work with .Result or .Wait(), which occupies a thread that could be serving another request
Chatty design that makes a client call five endpoints to draw one screen
Work done inside the request that could happen afterwards, such as sending an email or generating a report

The N+1 pattern, and the same result in one query

// Slow: one query for the departments, then one more per department
var departments = await db.Departments.ToListAsync(ct);
var slowSummaries = new List<DepartmentSummary>();

foreach (var department in departments)
{
    var activeCount = await db.Employees.CountAsync(
        e => e.DepartmentId == department.Id && e.Status == EmploymentStatus.Active, ct);

    slowSummaries.Add(new DepartmentSummary(department.Id, department.Name, activeCount));
}

// One query, returning exactly the three values the response needs
var summaries = await db.Departments
    .OrderBy(d => d.Name)
    .Select(d => new DepartmentSummary(
        d.Id,
        d.Name,
        d.Employees.Count(e => e.Status == EmploymentStatus.Active)))
    .ToListAsync(ct);

The first version runs 1 + N queries. With 40 departments that is 41 round trips to the database, and each one carries the network latency between your server and the database whether it returns one row or none.
Every individual query is fast, which is what makes this hard to spot from a slow-query log. The cost is the number of round trips, not the cost of any one of them.
It also scales with your data rather than with your traffic. It was fine with 5 departments in development and it is 41 round trips now, without any code changing.
The second version asks the database to do the counting and returns one row per department. One round trip, three columns, no entities materialised.
The diagnostic that finds this is counting queries per request rather than reading the C#. Log the query count for each request, or watch it in your profiler, and an endpoint issuing dozens becomes obvious.

A loop that produces improvements you can defend, rather than changes you hope help:

Record per-endpoint latency with percentiles
p50, p95 and p99 per route, plus request count. Without the count you cannot tell a slow endpoint that matters from one called twice a day.
Rank by total time, not by worst case
An endpoint at 200ms called 50,000 times a day costs more than one at 4 seconds called twice. Multiply latency by volume and start at the top.
Look at the query count and the payload size before the code
Those two numbers identify the majority of real problems. A request issuing 60 queries or returning 4MB has already told you what is wrong.
Change one thing
One fix, measured. Three at once leaves you unable to say which helped, and one of them may have made things worse while another hid it.
Measure the same numbers again, under realistic data
A development database with a thousand rows will not show an index problem. Verify against a data volume close to production, or the numbers are about the wrong system.
Keep the measurement in place
The value of instrumentation is noticing the next regression on the day it ships, rather than during the incident that follows it.

Trade-off

Every fix here costs something

Caching trades freshness for speed. A cached department list is fast and can be wrong for as long as its lifetime, so a cache needs a deliberate expiry and a story for invalidation. Caching to cover a query problem hides the problem and adds a second one.

Indexes speed up reads and slow down writes, because every insert and update maintains them, and they occupy storage. Adding one to fix a slow filter is usually right; adding six because they might help is not.

Smaller payloads mean more endpoints or more parameters, so a caller that needs the full record makes a second call. That is a better trade for a list endpoint than for a detail endpoint.

Moving work out of the request makes the response fast and the system harder to reason about, because the caller now finishes before the work does and has to find out separately whether it succeeded.

None of this is an argument against the techniques. It is an argument for applying them where a measurement says they are needed, which is the same reason the loop above starts with numbers.

Common mistake

Common performance mistakes

Optimising by intuition. The rewritten method was not the slow one, and the week spent on it is not recoverable.

Tracking averages. A flat average line through a serious p99 regression is the most common way a performance problem goes unreported.

Loading entities to compute a number. Fetching 5,000 employees to count the active ones does work the database would have done in one statement.

Blocking on async calls with .Result or .Wait(). Under load, threads accumulate waiting on I/O, and the symptom is an application that slows down everywhere at once rather than on the endpoint responsible.

Caching a value that must be current. Fast and wrong is worse than slow and right, particularly for permissions, prices and stock levels.

Adding indexes without checking write cost. An insert-heavy table with nine indexes has a new and less obvious problem.

Testing performance against a development database. A thousand rows hides every index problem you have, which is why the endpoint was fine until it was not.

Summary

Measure per endpoint as a distribution — p50, p95 and p99 — because an average hides the requests users notice
Rank work by latency multiplied by call volume, not by the worst single case
Query count and payload size identify most real problems before you read any code
N+1 queries, missing indexes, oversized payloads, absent caching and blocking I/O are the usual causes
Caching costs freshness and indexes cost write speed, so apply them where a measurement asks for them

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Try it yourself

Count the queries

Pick a collection endpoint that returns related data — employees with their department, or orders with their lines.

Log or observe the number of database queries a single request issues, and the size of the response body. Compare the query count with the number of rows returned.

Show solution

A query count that tracks the row count is the N+1 pattern, and the fix is a projection or an explicit include so the related data comes back in one statement.

The payload size is worth the same attention. A list endpoint returning every column of every related entity is often several times larger than the screen needs, and the cost is paid on serialisation, on the network and again on the client.

Both numbers come from the same request and neither requires a load test. Most first performance wins are found this way rather than with specialised tooling.

Think about it

Reading the numbers

An endpoint reports an average of 120ms, a p50 of 40ms, a p95 of 900ms and a p99 of 4.2 seconds. It is called 20,000 times a day.

What do these numbers suggest, and what would you look at first?

Show solution

The gap between the median and the tail is the finding. Most requests are quick, so the code path is not inherently slow; something specific is making a minority of requests very slow. An average of 120ms describes neither group.

At 20,000 calls a day, the p99 is roughly 200 requests taking over four seconds, every day. If a screen makes several calls, a noticeable share of page loads include one of them.

The usual causes of that shape are data-dependent work — a caller whose department has thousands of employees, a query whose plan changes with parameter values, a cache that misses for less common inputs, or contention such as lock waits during a busy period.

What to look at first is the slow requests themselves rather than the aggregate: capture the parameters, the query count and the row counts for requests over one second. The distribution told you a subgroup exists; only the individual traces say which subgroup.

Knowledge check

Nothing is recorded and there is no score. The explanation appears either way.

Why are percentiles more useful than an average when measuring endpoint latency?

They are cheaper to calculateAn average can hide a slow minority behind a fast majority, while p95 and p99 describe the requests users complain aboutAverages are only valid for fast endpointsPercentiles include network time and averages do not

An endpoint issues one query for a list of 40 departments and then one query per department. What is the problem and the fix?

The queries are individually slow, so they need indexesIt is 41 round trips whose count grows with the data; a single projected query returns the same result onceNothing, because each query is fastThe result should be cached indefinitely

Saved in this browser only.

End of the published lessons

That is everything written so far in Web API

More lessons in this course are on the way. In the meantime, the course page shows the full roadmap, and the projects are the best way to consolidate what you have covered.