Retain audit trails
A full history is recorded natively. Compliance with audit requirments happens naturally, without developers needs to maintain a separate service.
01 · An Overview
Event-sourcing is an alternative approach for data and state management that is more flexibile and scalabile than a traditional CRUD setup.
Instead of recording the consequences of changes, you simply record the changes themselves. They are the source of truth, and you derive the resulting state when needed.
Especially when building distributed microservices, event-sourcing has become a necessity and the standard.
carts| id | items | total | status |
|---|---|---|---|
| 42 | — | £0 | open |
A cart is represented by a single row, and every change overwrites that row.
cart-42A cart is represented by an append only stream of immutable facts.
The relational table has no record the mug was ever considered.
The event log retains this interaction.
Traditional CRUD systems store the current state, overwriting it on every change. Event-sourcing stores the changes themselves, as an append-only sequence of facts. The current state is derived from them. This small change of perspecive has big consequences.
Throughout this page we’ll follow a single shopping cart, cart-42, as a customer adds coffee
beans, dithers over a mug, and eventually checks out.
02 · Key concepts
An event is a fact about something that has already happened. It is usually named in the past
tense, such as item-added, item-removed, or checked-out.
Events are immutable. Once something has happened, it has happened, and so a stream of
events is append only.
To derive the current state of affairs, you replay the start (or a cached, prior moment in
time). Functional programmers will reconise this as a fold or a reduce.
The fundamental reframe is that the event stream is the source of truth, and any state you need is derived from it.
state = empty_cart()
for event in stream:
state = apply(state, event)The current state is never stored on its own. It is derived by replaying the events in order.
A request like “add a mug to cart 42” is a command, an instruction from one service to
another. To execute the command, the system first replays the stream to check the command makes
sense. (You can’t remove an item than is no longer there, or add to a cart that’s already been
checked out.) If the command succeeds, the outcome is recorded as new event.
Each command is checked against the current state. If it's valid, it becomes an event.
cart-42 cart-42pastAs you scroll back in time, notice that the state is cheap throw away and rederive. The event stream is read-only, so any number of views of the past can coexist.
Replaying a stream works well for understanding one cart. But questions like “which carts are open right now?” or “how much coffee did we sell today?” span thousands of streams, and you don’t want to replay every one for every query.
Instead, you build projections. A projection listens to events, often across a whole
category of streams, and maintains read model: a table, document, or search index shaped for
a particular question.
You can have as many projections as you like, each optimised for its own purpose. Separating the
write side (streams) from the read side (projections) is known Command Query Responsibility
Segregation, or CQRS for short.
Also, projections are disposable. If your team identifies a new question to ask, you can write a new projection to answer it.
carts| cart | items | total |
|---|
The question “what do people remove from their carts?” was never anticipated. In a CRUD system that information would long ago have been overwritten. Here, a new read model can replay from the very first event, then continue to report as new events flow in.
What happens if two services try to append new events at the same time? Simultaneous events, while rare, could lead to trouble if two systems are unaware of the others’ changes.
To prevent this, event stores rely on optimistic concurrency control. During a write operation, each service must specify which version of the stream it has acted upon. If the current version no longer matches, the append fails, and the rejected service must reprocess its command.
cart-42 cart-42. It's at version 5.No service holds a lock on the data. Conflicts are rare and caught when they happen. The rejected writer reprocesses against a fresh set of facts instead of silently overwriting it’s compeitor.
03 · The benefits
A full history is recorded natively. Compliance with audit requirments happens naturally, without developers needs to maintain a separate service.
"What did this customer's cart look like when they phoned us?"
Travel back to any momemnt in time for shared, accurate views.
Not everything you'll want to know can be known up front. Developers don't have to decide which interactions are imporant enough to include in their schema.
Copy the events behind a bug report into a test and replay them. You've reproduced the exact scenario and can see exactly what happened.
Events also make excellent integration points. Other services can subscribe to the events of
yours that they care about, such as checked-out, without accessing to your database. They operate
from their own projections.
Event-sourcing tends to pay off most in domains where what happened matters as much as what is. Finance, logistics, ordering, bookings, compliance-heavy workflows: anywhere you’ve been asked “how did it get here?” and couldn’t answer.
04 · Alternatives
Event-sourcing isn’t the only technique for maintaining history or sharing changes, and is by no means a silver bullet. Here’s how it compares with other common practices:
Row in cartsOnly the latest state survives. What it was before, and why it changed, are gone.
Stream cart-42Every change is kept, in order, along with what it meant.
carts + carts_auditHistory is a second copy, written by extra code. Nothing guarantees it is complete, and you can’t rebuild state from it.
Stream cart-42The log is the data. If a change isn’t in the log, it didn’t happen.
CDC feed from cartsYou see which columns changed, but not why. Did the customer remove the mug, or did it go out of stock?
Stream cart-42Events record intent in the language of the business, not as diffs of a table.
Topic cart-updatesA broker moves events between services. The state they describe still lives in some other database, which is the real source of truth.
Stream cart-42With event sourcing the events are the source of truth. You load an entity from its own stream, and you can still publish those events to a broker.
| CRUD | Audit tables | Change data capture | Messaging / Kafka | Event-sourcing | |
|---|---|---|---|---|---|
| Source of truth | Current state | Current state | Current state | Usually a separate DB | The events |
| Keeps full history | No | If code remembers | While retained | While retained | Yes, by design |
| Captures business intent | No | Partly | No, row diffs | Yes | Yes |
| Can rebuild state from history | No | Rarely | Rarely | Rarely | Yes |
| New read models over past data | No | Hard | Hard | If retained forever | Yes |
| Familiar to most teams | Very | Very | Fairly | Fairly | Less so |
Also, these approaches aren’t mutually exclusive. Event-sourced systems often publish their events to a broker for other services to consume, and plenty of successful services have a mix of some event-driven and some CRUD, for where history matters and where it doesn’t.
05 · Considerations
Event-sourcing’s simplicity can belie
Events are forever
You can’t migrate old events. When the shape of an event changes, you version it and ensure your code can read both versions. Be thoughtful when naming and modeling events.
Read models lag behind
Projections are usually updated asynchronously and can be a moment behind the stream. Services need to tolerate this eventual consistency or read from the stream when they can’t.
A new frame of mind
Streams, projections, subscriptions, replays, and versioned events are new concepts for many teams. Debugging gets easier, but there is more to learn and more infrastructure to be aware of.
Personal data and erasure
An immutable log and the right to be forgotten are in tension. Keep personal data out of event payloads where you can, or encrypt it per person and delete the key (“crypto-shredding”) when asked.
Event-sourcing can be overkill for small or constrained projects with no need to retain history. However, most commercial applications eventually reach a level of complexity such the issues arrising call for evented solutions. Transitioning a legacy, CRUD system into an evented system can be painful, but not every domain must operate the same way. Begin where history and intent matter most.
06 · In practice
logicblocks.event.store gives you the building blocks described above for Python, using
asyncio.
Events are grouped into a log of categories of streams. State is derived by projectors, and open-source adapters let you record these stream in your database of choice.
Here’s cart-42 from this page:
from logicblocks.event.projection import Projectorfrom logicblocks.event.store import EventStore, adapters, conditionsfrom logicblocks.event.types import NewEvent, StreamIdentifier
class CartProjector(Projector[StreamIdentifier, dict, dict]): def initial_state_factory(self): return {"items": {}, "status": "open"}
def initial_metadata_factory(self): return {}
def id_factory(self, state, source): return source.stream
# One handler per event: "item-added" is handled by item_added, and so on. def item_added(self, state, event): sku = event.payload["sku"] state["items"][sku] = state["items"].get(sku, 0) + 1 return state
def item_removed(self, state, event): sku = event.payload["sku"] state["items"][sku] -= 1 if state["items"][sku] == 0: del state["items"][sku] return state
def checked_out(self, state, event): state["status"] = "checked-out" return state
async def main(): store = EventStore(adapters.InMemoryEventStorageAdapter()) cart = store.stream(category="carts", stream="cart-42")
# publish raises an `UnmetWriteConditionError` if another event # writes before this one can complete. await cart.publish( events=[ NewEvent(name="item-added", payload={"sku": "beans"}, metadata=None), NewEvent(name="item-added", payload={"sku": "mug"}, metadata=None), ] )
# Optimistic concurrency: only append if nobody else has written since we read. latest = await cart.latest() await cart.publish( events=[NewEvent(name="item-removed", payload={"sku": "mug"}, metadata=None)], condition=conditions.position_is(latest.position), )
projection = await CartProjector().project(source=cart) print(projection.state) # {'items': {'beans': 1}, 'status': 'open'}