Skip to content

An introduction to event-sourcing

An interactive walkthrough. The basic concepts, why it's so powerful, and how to get started.

01 · An Overview

Event-sourcing is an alternative approach for data and state management that is more flexibile and scalabile than a traditional CRUD setup.

Instead of recording the consequences of changes, you simply record the changes themselves. They are the source of truth, and you derive the resulting state when needed.

Especially when building distributed microservices, event-sourcing has become a necessity and the standard.


Customer opens the shop
Traditional table carts
iditemstotalstatus
42—£0open
 

A cart is represented by a single row, and every change overwrites that row.

Event stream cart-42

    A cart is represented by an append only stream of immutable facts.

    The relational table has no record the mug was ever considered.

    The event log retains this interaction.

    Traditional CRUD systems store the current state, overwriting it on every change. Event-sourcing stores the changes themselves, as an append-only sequence of facts. The current state is derived from them. This small change of perspecive has big consequences.

    Throughout this page we’ll follow a single shopping cart, cart-42, as a customer adds coffee beans, dithers over a mug, and eventually checks out.

    02 · Key concepts

    An event is a fact about something that has already happened. It is usually named in the past tense, such as item-added, item-removed, or checked-out.

    Events are immutable. Once something has happened, it has happened, and so a stream of events is append only.

    To derive the current state of affairs, you replay the start (or a cached, prior moment in time). Functional programmers will reconise this as a fold or a reduce.

    The fundamental reframe is that the event stream is the source of truth, and any state you need is derived from it.

    Event Stream
      Fold
      state = empty_cart()
      for event in stream:
          state = apply(state, event)
       
      Current state

      The current state is never stored on its own. It is derived by replaying the events in order.

      A request like “add a mug to cart 42” is a command, an instruction from one service to another. To execute the command, the system first replays the stream to check the command makes sense. (You can’t remove an item than is no longer there, or add to a cart that’s already been checked out.) If the command succeeds, the outcome is recorded as new event.

      Commands
      • Coffee beans£12
      • Mug£8
      • Grinder£45
      • Filters£4

      Each command is checked against the current state. If it's valid, it becomes an event.

      Stream cart-42
        State cart-42past

        As you scroll back in time, notice that the state is cheap throw away and rederive. The event stream is read-only, so any number of views of the past can coexist.

        Replaying a stream works well for understanding one cart. But questions like “which carts are open right now?” or “how much coffee did we sell today?” span thousands of streams, and you don’t want to replay every one for every query.

        Instead, you build projections. A projection listens to events, often across a whole category of streams, and maintains read model: a table, document, or search index shaped for a particular question.

        You can have as many projections as you like, each optimised for its own purpose. Separating the write side (streams) from the read side (projections) is known Command Query Responsibility Segregation, or CQRS for short.

        Also, projections are disposable. If your team identifies a new question to ask, you can write a new projection to answer it.

        All streams
        Category carts

          Read models
          Open carts
          cartitemstotal
          Revenue
          £00 orders
          Non-coffee revenue
          £0excl. beans
          New · Most removed products
          • Coffee beans0
          • Mug0
          • Grinder0
          • Filters0

          The question “what do people remove from their carts?” was never anticipated. In a CRUD system that information would long ago have been overwritten. Here, a new read model can replay from the very first event, then continue to report as new events flow in.

          What happens if two services try to append new events at the same time? Simultaneous events, while rare, could lead to trouble if two systems are unaware of the others’ changes.

          To prevent this, event stores rely on optimistic concurrency control. During a write operation, each service must specify which version of the stream it has acted upon. If the current version no longer matches, the append fails, and the rejected service must reprocess its command.

          Writer A · laptop
          Hasn't read yet
          Stream cart-42
            Writer B · phone
            Hasn't read yet
            1. Both services load cart-42. It's at version 5.
            2. Each decides on a change, based on the version it receives.
            3. A appends first, expecting version 5. It matches, so the event is accepted.
            4. B appends, also expecting version 5. The stream is now at 6, so the append is rejected.
            5. B reloads, reprocessing against the new state, and retries, this time expecting version 6.

            No service holds a lock on the data. Conflicts are rare and caught when they happen. The rejected writer reprocesses against a fresh set of facts instead of silently overwriting it’s compeitor.

            03 · The benefits

            09:41:02item-addedbeansweb
            09:41:49item-addedmugweb
            09:44:15item-removedmugsupport
            09:46:30checked-outweb

            Retain audit trails

            A full history is recorded natively. Compliance with audit requirments happens naturally, without developers needs to maintain a separate service.

            as of 09:44

            Travel back in time

            "What did this customer's cart look like when they phoned us?"

            Travel back to any momemnt in time for shared, accurate views.

            Ask new questions

            Not everything you'll want to know can be known up front. Developers don't have to decide which interactions are imporant enough to include in their schema.

            replaying production events locally…

            Debug by replaying

            Copy the events behind a bug report into a test and replay them. You've reproduced the exact scenario and can see exactly what happened.

            Events also make excellent integration points. Other services can subscribe to the events of yours that they care about, such as checked-out, without accessing to your database. They operate from their own projections.

            Event-sourcing tends to pay off most in domains where what happened matters as much as what is. Finance, logistics, ordering, bookings, compliance-heavy workflows: anywhere you’ve been asked “how did it get here?” and couldn’t answer.

            04 · Alternatives

            Event-sourcing isn’t the only technique for maintaining history or sharing changes, and is by no means a silver bullet. Here’s how it compares with other common practices:

            CRUD · Row in carts

              Only the latest state survives. What it was before, and why it changed, are gone.

              Event sourcing · Stream cart-42

                Every change is kept, in order, along with what it meant.

                CRUD Audit tables Change data capture Messaging / Kafka Event-sourcing
                Source of truth Current state Current state Current state Usually a separate DB The events
                Keeps full history No If code remembers While retained While retained Yes, by design
                Captures business intent No Partly No, row diffs Yes Yes
                Can rebuild state from history No Rarely Rarely Rarely Yes
                New read models over past data No Hard Hard If retained forever Yes
                Familiar to most teams Very Very Fairly Fairly Less so

                Also, these approaches aren’t mutually exclusive. Event-sourced systems often publish their events to a broker for other services to consume, and plenty of successful services have a mix of some event-driven and some CRUD, for where history matters and where it doesn’t.

                05 · Considerations

                Event-sourcing’s simplicity can belie

                Events are forever

                You can’t migrate old events. When the shape of an event changes, you version it and ensure your code can read both versions. Be thoughtful when naming and modeling events.

                Read models lag behind

                Projections are usually updated asynchronously and can be a moment behind the stream. Services need to tolerate this eventual consistency or read from the stream when they can’t.

                A new frame of mind

                Streams, projections, subscriptions, replays, and versioned events are new concepts for many teams. Debugging gets easier, but there is more to learn and more infrastructure to be aware of.

                Personal data and erasure

                An immutable log and the right to be forgotten are in tension. Keep personal data out of event payloads where you can, or encrypt it per person and delete the key (“crypto-shredding”) when asked.

                Event-sourcing can be overkill for small or constrained projects with no need to retain history. However, most commercial applications eventually reach a level of complexity such the issues arrising call for evented solutions. Transitioning a legacy, CRUD system into an evented system can be painful, but not every domain must operate the same way. Begin where history and intent matter most.

                06 · In practice

                Event-sourcing with logicblocks.event.store

                Section titled “Event-sourcing with logicblocks.event.store”

                logicblocks.event.store gives you the building blocks described above for Python, using asyncio.

                Events are grouped into a log of categories of streams. State is derived by projectors, and open-source adapters let you record these stream in your database of choice.

                Here’s cart-42 from this page:

                from logicblocks.event.projection import Projector
                from logicblocks.event.store import EventStore, adapters, conditions
                from logicblocks.event.types import NewEvent, StreamIdentifier
                class CartProjector(Projector[StreamIdentifier, dict, dict]):
                def initial_state_factory(self):
                return {"items": {}, "status": "open"}
                def initial_metadata_factory(self):
                return {}
                def id_factory(self, state, source):
                return source.stream
                # One handler per event: "item-added" is handled by item_added, and so on.
                def item_added(self, state, event):
                sku = event.payload["sku"]
                state["items"][sku] = state["items"].get(sku, 0) + 1
                return state
                def item_removed(self, state, event):
                sku = event.payload["sku"]
                state["items"][sku] -= 1
                if state["items"][sku] == 0:
                del state["items"][sku]
                return state
                def checked_out(self, state, event):
                state["status"] = "checked-out"
                return state
                async def main():
                store = EventStore(adapters.InMemoryEventStorageAdapter())
                cart = store.stream(category="carts", stream="cart-42")
                # publish raises an `UnmetWriteConditionError` if another event
                # writes before this one can complete.
                await cart.publish(
                events=[
                NewEvent(name="item-added", payload={"sku": "beans"}, metadata=None),
                NewEvent(name="item-added", payload={"sku": "mug"}, metadata=None),
                ]
                )
                # Optimistic concurrency: only append if nobody else has written since we read.
                latest = await cart.latest()
                await cart.publish(
                events=[NewEvent(name="item-removed", payload={"sku": "mug"}, metadata=None)],
                condition=conditions.position_is(latest.position),
                )
                projection = await CartProjector().project(source=cart)
                print(projection.state) # {'items': {'beans': 1}, 'status': 'open'}