Skip to main content
ANVISoftware Solutions
Lesson 14 of 19Professional22 min

Event-Driven Architecture

By the end of this lesson

Communicate through events, accepting eventual consistency.

Event-driven architecture takes the observer idea across a process boundary. Something happens, the component that owns it publishes a fact, and other components react in their own time, in their own processes. Orders announces that an order was approved. Invoicing raises an invoice. Reporting updates a figure. Orders was not told about either of them.

Two properties you had in process are now gone, and almost every difficulty in this lesson comes from losing them. The publisher no longer knows whether a reaction ran, or when, or whether it succeeded. And a message can arrive more than once, because the delivery guarantee every practical broker offers is at-least-once. Design for both from the first handler, not after the first incident.

Publishing a fact without losing it — state and outbox in one transaction
C#
// The contract. Past tense, versioned, carrying what a subscriber needs.
public sealed record OrderApproved(
    Guid EventId,
    int OrderId,
    string OrderNumber,
    decimal Total,
    string CurrencyCode,
    DateTime ApprovedUtc)
{
    public const string Type = "orders.order-approved.v1";
}

public async Task ApproveAsync(int orderId, string approvedBy, CancellationToken token)
{
    Order order = await db.Orders.SingleAsync(o => o.Id == orderId, token);
    order.Approve(approvedBy, clock.GetUtcNow().UtcDateTime);

    var fact = new OrderApproved(
        Guid.CreateVersion7(), order.Id, order.Number, order.Total, "GBP", order.ApprovedUtc);

    db.OutboxMessages.Add(new OutboxMessage
    {
        Id = fact.EventId,
        Type = OrderApproved.Type,
        Payload = JsonSerializer.Serialize(fact),
        OccurredUtc = fact.ApprovedUtc,
    });

    // One transaction covers the state change and the intent to publish.
    await db.SaveChangesAsync(token);
}
  • Saving the order and publishing to a broker are two separate systems, and no transaction spans them. Either can succeed alone. Putting the publish after the save does not fix it: the process can stop in between, and then the order is approved and nobody was told.
  • The outbox row is written in the same transaction as the state change, so both land or neither does. A background publisher then reads unsent rows, publishes them, and marks them sent.
  • That publisher is where duplicates come from. If it crashes after the broker accepted the message but before the row was marked sent, it will publish the same message again on restart. This is not a flaw to engineer away; it is the reason the receiving side must tolerate repeats.
  • Guid.CreateVersion7 produces a time-ordered identifier, which keeps index inserts roughly sequential and makes the event id a sensible deduplication key. On .NET 8 or earlier, generate the identifier however you normally would — what matters is that it is stable across republishes of the same fact.
  • The versioned type string is the contract between teams. Subscribers match on it, so publishing orders.order-approved.v2 alongside v1 lets you change the shape without waiting for another team's release.

Between that commit and the last handler finishing, the system is inconsistent in a way a user can see. The order is approved, the invoice does not exist yet, and the dashboard total is the one from before. Usually the window is milliseconds. When a consumer is restarting, or a queue has backed up behind a slow handler, it is minutes.

The expectation this breaks has a name worth knowing: read-your-own-writes. A person clicks approve, the page reloads, and the thing they did is not there. They conclude the click did not work, so they click again. No amount of care inside the handler prevents this, because the cause is the design, not a defect.

So the decision belongs with whoever owns the product, and it has to be made before the code is written. The realistic options are: show the outcome from the command's own response rather than re-reading, show an explicit pending state, or wait for the reaction before responding — which gives back the decoupling you paid for. Engineers picking silently is how a working system becomes a ticket saying the system is broken.

A handler written for a message that arrives twice
C#
internal sealed class RaiseInvoiceOnOrderApproved(
    AppDbContext db,
    TimeProvider clock,
    ILogger<RaiseInvoiceOnOrderApproved> log) : IConsume<OrderApproved>
{
    private const string Handler = nameof(RaiseInvoiceOnOrderApproved);

    public async Task ConsumeAsync(OrderApproved fact, CancellationToken token)
    {
        bool done = await db.HandledMessages
            .AnyAsync(h => h.EventId == fact.EventId && h.Handler == Handler, token);

        if (done)
        {
            log.LogInformation("Skipping {EventId}: {Handler} already ran.", fact.EventId, Handler);
            return;
        }

        db.Invoices.Add(Invoice.For(fact.OrderId, fact.OrderNumber, fact.Total, fact.CurrencyCode));
        db.HandledMessages.Add(new HandledMessage
        {
            EventId = fact.EventId,
            Handler = Handler,
            HandledUtc = clock.GetUtcNow().UtcDateTime,
        });

        try
        {
            await db.SaveChangesAsync(token);
        }
        catch (DbUpdateException ex) when (IsDuplicateKey(ex))
        {
            // A concurrent delivery got there first. The invoice exists; there is nothing to do.
            log.LogInformation("Duplicate delivery of {EventId} lost the race.", fact.EventId);
        }
    }
}
  • Idempotent means running twice leaves the same result as running once. It does not mean the message arrives once — no broker can promise that end to end, whatever its documentation calls the feature.
  • The check and both inserts commit together, so a repeat delivery cannot squeeze between deciding and acting.
  • The check on its own is not enough, and this is the part that gets missed. Two deliveries can be handled at the same instant on two instances: both queries find nothing, both insert. A unique constraint on EventId plus Handler is what actually makes it safe, and the catch block is the expected path rather than an error path.
  • The key includes the handler name because three handlers each need to process the same fact once. Keying on the event id alone means the first handler to finish blocks the other two.
  • IsDuplicateKey is a small helper you write, because the exception details are database-specific. Keep it in one place rather than matching error numbers at each call site.
  • When the side effect is outside your database — an email, a payment, a file upload — it cannot join the transaction. Then you need the remote system's own idempotency key, which is why the payment port in the adapter lesson carried a Reference.

Assume every one of these about any message your code receives, because each will happen in a system that runs long enough:

  • It arrives twice. A publisher retry, a consumer that crashed before acknowledging, or a broker redelivery after a visibility timeout.
  • It arrives out of order. OrderCancelled can reach a consumer before OrderApproved, so a handler that assumes sequence will act on a state that has already moved on.
  • It arrives late. Minutes after the fact, if the consumer was down or a queue was backed up. A handler that reads "the current state" will see something newer than the fact describes.
  • It arrives with fields you have never seen, because the publishing team added them. Deserialisation must tolerate that rather than throw.
  • It is replayed deliberately, weeks later, because a new subscriber needs history or an old one had a bug.
  • It describes an entity that has since changed again. The fact is still true about the moment it happened; it is not a description of now.

Summary

  • A publisher announces a fact; reactions happen in other processes on their own schedule
  • State changes and the intent to publish belong in one transaction, or facts get lost or invented
  • At-least-once delivery is the norm, so every handler must be idempotent and defended by a unique constraint
  • Eventual consistency reaches the screen, and what the user sees is a product decision
  • Use direct calls when the reaction is part of the operation or the caller needs the answer

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Think about it

The user cannot see their own change

Approving an order publishes a fact. Invoicing raises the invoice a second or two later. The approvals screen reloads from the read side immediately and shows no invoice, so users approve the same order two or three times.

List the options, say who should choose between them, and name what each one costs.

Show solution

Option one: stop re-reading. The approve response already knows what it did, so the screen shows the approved state from that response. Cheap, effective for this screen, and does nothing for the user who reloads the page.

Option two: show the pending state honestly. The row says "invoice being raised", and the screen refreshes when it appears. This is usually the right answer, and its cost is real interface work plus a decision about what to show if the invoice never arrives.

Option three: disable the button after the first click and rely on an idempotent command. This removes the duplicate approvals but leaves the user confused about whether anything happened.

Option four: wait for the invoice before responding. Now the two services are coupled again and an invoicing outage becomes an ordering outage. You have paid for a broker and bought back the coupling.

Whoever owns the product chooses, because every option changes what a person sees and how much the business can tolerate. Engineers can rule out option four on technical grounds and cost the rest, but picking silently is how a correct implementation becomes a complaint.

Whatever is chosen, the command still needs to be idempotent. Users will double-click regardless, and the interface is not a safety mechanism.

Try it yourself

Replay a day of events

Take a service that consumes events and replay yesterday's messages into it, either from a broker feature or from a stored copy.

Record what goes wrong. Then write down which of those problems would also occur from a single duplicate delivery in normal operation.

Show solution

The usual result is duplicated work: second invoices, repeated emails, audit rows twice. Every one of those is the same defect a routine redelivery would cause, which is the point of the exercise — replay makes a rare problem reproducible on demand.

The second finding is usually events that cannot be replayed because they are not self-contained. A handler that receives an order identifier and then reads the order's current state gets today's state for yesterday's fact, and produces something that is not wrong in any obvious way, which makes it worse.

The third is ordering. Replay often delivers in a different order from the original, and handlers that assumed sequence fail in ways that look like business errors rather than technical ones.

Replay is worth being able to do for its own sake: a new subscriber needs history, and a handler released with a bug needs its work redone. A system where replay is unsafe has no recovery path except manual database edits.

The test to keep: process the same message twice in an integration test and assert the state is identical to processing it once. That turns idempotency into something the build checks rather than something the team intends.

Saved in this browser only.