Event sourcing is a pattern where application persists its state as a sequence of events that happened in the past.
State and events
For example, customer management application, if it were to use event sourcing, might store events like:
- customer was created
- customer was awarded VIP status
- customer was converted into a lead.
Using these events, we can transform them or project them into any state that is needed for the application.
Traditional CRUD application framework like Ruby on Rails or Django will have usually database and a set of tables in which it will store facts or state about the current business entities.
Event sourcing comes before that. It is a sequence of events that are recorded throughout time. If we have a point in time in which we’re interested (it could be the present or something in the past), we can use all the events that happened before that point in time. We can take information from them and create any needed structure of facts and state.
This process of deriving is called projection. Instead of storing state and information in some tables from the start, we store them as a history of events, and then use this history of events to create a state.
Transport Tycoon
We’ll use the domain of Transport Tycoon. It’s a game from the old ages, probably before the 2000s.
The domain of Transport Tycoon was used in my DDD katas. It’s a simplistic domain. In this domain, we have a factory, and there are two warehouses, warehouse A and warehouse B. Factory produces goods, and we have to deliver these goods to the warehouses. We also have a port. All these points are connected by the roads, and there is water between port and B.
We have a couple of trucks and a ferry that can travel and transport goods. Currently, the factory has four cargos named A, B, B, A, which have to be delivered to their respective warehouses. If we were to build an application in Django or Ruby on Rails or anything that deals with the database tables first, we’d probably create:
Cargo: This entity would be represented in the table as two columns, where one column iscargo IDand the other column iscargo destination. We have four cargos and we’ll have four rows.Location to Cargo: In this table, we’ll list all of our four locations, and in theCargocolumn, we’ll list the cargoes that are currently assigned to that location. At this point in time, where all cargo stays at factory, we have all four items added to the factory row. That’s not how relational table works, but let’s keep it at this.Transport: This transport table will list all the current available transports, the cargo that they are carrying, and the location.
We’ve missed an important domain concept. Since we haven’t started recording anything important in the tables, we can modify the structure. The concept was estimated time of arrival. It takes five hours to go for a truck from factory to A, one hour for a truck to go from factory to port, and four hours for a ship to travel between port and B.
We adjust our table by adding estimated time of arrival column. It estimates the time at which the transport, if it is en route, will arrive to its location. We have three transports. At the beginning of the game, they’re stationed at the factory (the trucks), and the ship is at the port.
For example, the first truck takes the first cargo and starts moving to location A, and the estimated time of arrival would be five hours. We mark that by removing one cargo from the factory, adding to the truck, and changing the location of the truck to A, marking it that it is still driving by adding ETA of 5. Same would be with the second truck, except that it takes the second cargo and starts driving to the port, and it should arrive earlier.
One hour later, the ship should take this cargo. The truck passes the cargo to the ship, and the ship starts going to the warehouse B with estimated time of arrival of 5. The second truck will start going back to the factory.
We have four cargoes at warehouses A and B, and all trucks and ships are back at their stations. What if we needed at some point to derive additional facts from these truck travels? For example, marketing department or the business guys might come up with the questions:
- What is the transport idle time?
- What is the mileage of transport? We need to do maintenance on trucks after 10,000 miles.
- What are our delivery bottlenecks, so that we can optimize them?
Based on the information in the current facts table alone, it is not possible to get these answers. Perhaps there was some information recorded in the cells, but we deleted that by overwriting with a new state. Or some information wasn’t even recorded. Event sourcing is a pattern that differs from the fact-based storage or state-based storage in that it records information as a sequence of events and then derives facts from them.
Recording events
The first event would probably be recorded at zero o’clock, and it will say that some cargo had to be delivered, that we have four pieces of cargo, and they are destined to go to warehouses A and B. The second event that we can record at time zero would be about the transport, that we have two trucks at the factory and one ship waiting at the port.
At hour zero, we can record an event that the first truck loaded cargo number one that has to go to the warehouse A from the factory, and it departed to warehouse A. Estimated time of arrival is five. Another event that we can record at zero is about the second truck, that it loaded cargo for warehouse B from factory and that it departed to port.
Estimated time of arrival is one. One more event would be recorded at time one, that the truck unloaded second cargo at port and that it started moving back to factory with estimated time of arrival at two. One more event that we can record is that ship number one loaded second cargo for the warehouse B at port and departed to the warehouse B. As trucks and ships move around, we will record events that signify important state transitions.
Rebuilding state
The process of deriving facts from events, it’s called projection. When we’re talking about projecting events to a state, the projected fact, they can be called views, but we’ll refer to them as facts. We have our existing table structure.
We can fill this structure using the events alone. The first event: cargo was planned. That information is probably enough to fill the cargo table. We also know that the cargo is planned to be delivered from the factory. At time zero, we know that there are four pieces of cargo that are stationed at the factory. We can add the factory row to the location table.
At time zero, we also knew, based on our event, that there are three pieces of transport stationed at the factory and at the port. We can fill some part of the transport table. At time zero, the transport loads the first cargo from factory and departs to A. That information is enough to update the factory row and the truck row in the corresponding tables.
We start updating tables in a similar way to how they were updated based on the domain model alone, but now we’re using only information from the events. The second event is similar. Second truck loaded cargo to warehouse B from factory and departed to port with estimated time of arrival of one. We move the second cargo from the factory to the truck, mark the target location as B, estimated time of arrival of one.
At hour one, truck unloads the second cargo B at port and departs. At some point in time, the second cargo is at port. When we were recording our domain previously directly into the facts, we never had this situation where the second cargo was at port. We had the situation where cargo was passed directly to the ship.
The cargo has to be unloaded from the truck somewhere before it can be loaded to the ship. At some point, ship loads the second cargo from the port and departs to B.
New projections
Events are suitable for more than that.
We can take the existing events and project them to more facts to produce more information. Let’s keep only the events that deal with the transfer of the second cargo. There are not all of the events, but that’s enough for us. Using these events, we know that the cargo was planned at time zero and that it was already loaded at time zero, that the first truck unloaded at the port at one o’clock, and then the ship loaded and departed to the warehouse B.
We can create cargo milestone history that says that the second cargo was going from factory to B and that it took two transports with the specific departure and arrival times at specific locations and at specific destinations. We were able to take past events and, based on the information contained in them, create a new table, cargo milestone history, and fill it with the data as if this table existed from the beginning of times.
Normally, applications store their state in databases. They have tables there. It could be relational tables, it could be a document database. These databases store facts. As the world around the application changes, it records most of that information by updating facts and rewriting them with new information. If at some point we need to create new facts that leverage some of things that happened in the past, it is not possible.
With event sourcing, we take a step back. Instead of creating facts tables directly, we invest an extra effort to record domain events about what happened in the domain model throughout time. Using these events, we can write projections that allow us to replay these events and build up facts. We can also, after some time, define new projections that will allow us to replay events from the start and create new collections of facts, as if these facts were being collected from the very beginning of time.
That is useful if you’re building a report that has to cover all transactions, or if you’re adding a feature and need to change the columns in the database and fill them as if this feature was available from the very beginning of time.