3. Why Event Sourcing? — AI Native Code

Event sourcing is about persisting data in a way that preserves every single bit of information. It’s about representing objects as a sequence of events that took place through the time and led them to the current state.

For instance, if I were to persist information about my pocket money (i.e.: 100 EUR), I could simply save the latest state somewhere in a variable or database:

User	Amount
Rinat	100

Now, whenever there is a change, we would overwrite this value with the new value (discarding the previous one). Then at some point in time we will have something like this written in the database:

User	Amount
Rinat	67

Simple and elegant (and works perfectly in a large number of scenarios). However, we are performing a logical compression here (lossy one) and discarding some information. Let’s see what would happen if we were to preserve all the changes:

Transaction	Change	Amount
Got from ATM	100	100
Bought metro tickets	-12	88
Grabbed a lunch	-8	80
Found a coin	1	81
Took taxi	-14	67

Obviously, if we have such a sequence of events, we can always “reconstruct” the current balance, by doing a total:

Balance: 100 - 12 - 8 + 1 - 14 = 67 EUR

In essence, the final state (Balance) is a left-fold function of the previous states (equivalent of IEnumerable.Aggregate in .NET, std::accumulate in C++ or array.reduce in JavaScript).

What’s the point in storing all these intermediate steps, when you can just save the final balance?

Benefits

It all boils down to a few benefits of Event Sourcing:

Sometimes we need to store changes to prove or explain what happened in the past. This is what banks do, for example.
We want to scale our system out, improve its performance or availability by leveraging the transaction logs capabilities of an event stream. This is what databases do under the hood.
We want to stick with the ideas of Domain-Driven Design, leverage benefits of Event Storming and in general align the codebase with the language of the customers and other stakeholders.
We want to write non-fragile unit tests for our system that verify business behaviors but avoid changing internal implementation details or even changing stacks.

Agentic coding

That last reason is the most important of them all nowadays.

I used Event Sourcing over the decades, mostly reaching out to it in cases of really complex business products and large teams. These benefits made it worthwhile in such projects.

However today modern Agentic Coding tools shift the economics of software engineering a lot: manual coding becomes essentially free, provided humans take care of all the other aspects of software engineering, verification and product development.

This lowers the barrier to using event sourcing in the projects and brings into the forefront its most valuable benefit: ability to capture business requirements into non-fragile and additive full-stack specs. This solves the pain of code drift between vibe-coded text specifications and similarly vibe-coded implementation.

In the next module units, testability will be our primary focus.

Storage ignorance

It is easy to save data as a stream of events. All we need is to define POCO (Plain Old CLR Object) classes (one for each event) and then serialize them to one of many formats available: Google ProtoBuf, JSON, Binary, XML etc.

Define events:

GotMoneyFromAtm! (amount, transaction, time)
BoughtMetroTickets! (count, amount, machine, time)
GrabbedALunch! (amount, cost, time, menu, place)
FoundACoin! (amount, gps, time)
TookTaxi! (amount, rideDuration, taxiCompany, route, time)

That’s how the definitions could look for C# language, if you used T4 of Visual Studio.

Flexibility

Given the sequence of events, we can project them to any desired structural representation. For instance, we could write a projection that would summarize all our expenses and produce the latest balance.

However, we can do much more:

Generate list of cities, in which coins can be found most often.
Get a list of taxi companies that are the cheapest ones or fastest ones.
List TOP 5 favorite places for having a lunch on Monday.

We don’t need any complex queries in order to do that. Try doing that if you have just a single field of Balance, or even if you had a list of changes (credit/debit).

However, as long as you have a stream of events, you can project it to any form, even a conventional SQL database.

Messaging capabilities

Events represent serializable and immutable data structures that are appended to an append only stream. As such, they share all the capabilities of messages. So we can:

Multiplex them to multiple subscribers (i.e.: have a list of 5 last restaurants visited by Lady Gaga; while keeping that list up to date and continuously replicated around the globe to load balance the pressure of fans).
Implement redundancy and reliability (i.e.: write events to 3 different locations at once, where we have 1 master and 2 slaves).
Support load balancing (competing consumers) and parallel processing.
Support incremental sync that fits in a few lines of code.

Here are these lines of code from the production system (we are replicating from remote to cache):

var next = _cache.GetCurrentVersion();

while (true)
{
    var items = _remote.ReadRecords(next, BatchSize);
    if (items.Length == 0)
        break;

    next = items.Max(m => m.Version);

    _cache.AppendNonAtomic(items);
    logger(string.Format("Loaded {0} records", items.Length));
}

Of course, in a more conventional system (that does not employ event sourcing) you can leverage something like SQL Replication or Microsoft Sync Framework.

Performance

Performance and scalability are byproducts of inherent capabilities offered by event sourcing:

events can be published as soon as they happen;
it is dead easy to precompute read models from the events (do a projection to some desired state);
events can be processed in parallel;
events can be multiplexed to multiple subscribers, where each one could maintain its own copy of some read model.

What is a read model? “Balance” is one sample of a read model; “List of favorite Gaga’s restaurants” is another one. Essentially read model is some view (precomputed result of an SQL Query in SQL World).

Since we have more flexibility with projecting events and passing them around, we can easily do more interesting things, reaching up to the speeds of LMAX (which was described by Martin Fowler):

Keep read models (precomputed query results) directly in memory in some MemCached instances (if server goes down, we can always repopulate cache from the history of the events).
Run multiple processing units in parallel (if any one goes down, switch to the backup).
Benefit from the real time signal processing knowledge of robotics and critical systems (imagine, what could do an FPGA/ASIC optimized for parallel processing of events).

Developer’s life

There are a few more interesting aspects of event sourcing:

Simplifies deployment and maintenance (less SQL, upgrade scripts and versioning).
Reduces expenses on both the hardware and software (no need to have extremely powerful and redundant servers or commercial databases).
Integration between systems is more straightforward (all Enterprise Integration Patterns apply here directly).
Since no data is ever lost, we gain full audit (plus the ability to go back at any point in time) and excellent debugging capabilities.
Event Sourcing is a natural fit for software developers that want to capture the essence of business domains (especially the most complex ones) while staying decoupled from platform complexity or hard performance issues.
ES approaches help to provide clear answers to some of the new challenges brought to us by the market and technology: Cloud Computing, Big Data processing, Mobile and occasionally connected systems, real time business intelligence.

There also are some financial and political benefits for project stakeholders to be interested in. They all revolve around the ability to have better flexibility in project delivery, managing resources and risks.

Ability to keep things simple, defer important decisions and adapt business solutions can be a powerful enabler in large conservative organizations. Smaller companies (such as lean startups) can also gain more competitive advantage and reduce time to market with such approaches.

Here we are getting into the area of synergy effects with CQRS/DDD methodologies and their practical application to distributed environments.

Downsides

Event Sourcing is not a silver bullet. It is a different approach to think and represent changes and data. If you are a C# or C++ developer, then this feels like going back to assembler. If you are a project manager, it’s like consciously going back from Microsoft Project Server to task lists and custom budgeting software.

Defining these events is a complex art of its own, which requires skills in domain modeling (hint: if you have a lot of events with following words in their names, then you are doing something wrong: Create, Insert, Update, Delete, Set, Change, Add). Domain Driven Design (as both a book and a body of knowledge) is an entry point into this skill.

Concerns

There are a few additional concerns that might look like downsides of approach, but in fact are not that important.

Extra storage costs are usually negligible when compared to the business value that might be created.

Slower performance is not an issue, since we can optimize IO via snapshotting and persistent read models. And leveraging push based nature of events, we can get immediately invalidating caches. In short, there are multiple technical solutions that could be plugged later, if there is such need.

Fragility (losing an event in the past causes the entire stream to be corrupt) is not an issue, since you can determine yourself the levels of SLAs to go for (via replication and redundancy). Corruption in any single replica can be reliably detected using git’s approach: event includes SHA1 signature computed against its contents and signature of the previous event.

Versioning is sometimes perceived as a problem, since our systems tend to grow and event contracts (schemas) can gradually evolve to new formats that are incompatible with the old saved events. Yet, if approached consciously this can be solved (and solution is more elegant and simple than SQL migration scripts). I use a combination of 3 elements here:

Using serializer that is evolution friendly (i.e.: Google ProtocolBuffers is not only among the fastest serializers, it also allows you to rename class members without breaking anything).
Using in memory upgraders which are simple classes stacked on top of the event streams for replays and accessing history. They can split, merge, convert events or even fill in missing fields (if this data is available in the other events). Upgraders are just a lazy and quick replacement for rewriting the entire event stream history completely.
Using careful event modeling (which is usually based on the Ubiquitous Language coming from DDD) to define events in a way that they will rarely change a lot (they will be based on concepts that do not change, no matter how their relations evolve).

Stories

As said by Mike Nichols in DDD/CQRS mailing list:

My experience has been that ES and its promotion of business semantics over technological terms has a way of bending my mind toward modeling behaviors rather than essence.

It also lets me avoid the ceremony of modeling state that doesn’t contribute to behaviors.

I see this as a good thing and as a side effect I probably use language the business person understands more.

I just can’t find a reason to use ORM in my domain anymore. ES seems to let me more rapidly model … having a change log/audit trail is about the furthest thing from my mind when I reach for it.