PageSourceSearch

https://www.ahri.net/2019/07/practical-event-driven-and-sourced-programs-in-haskell/

html ahri.net collected 2026-09-25 20:00:16 UTC 50,235 bytes, 220 lines download raw bytes

1<!DOCTYPE html>
2<html>
3
4<head>
5  <meta charset="utf-8">
6  <meta name="viewport" content="width=device-width, initial-scale=1.0">
7  <title>Practical event driven &amp; sourced programs in Haskell</title>
8  <link rel="stylesheet" href="https://stackedit.io/style.css" />
9</head>
10
11<body class="stackedit">
12  <div class="stackedit__left">
13    <div class="stackedit__toc">
14      
15<ul>
16<li><a href="#practical-event-driven--sourced-programs-in-haskell">Practical event driven & sourced programs in Haskell</a>
17<ul>
18<li><a href="#terminology--high-level-architectural-view">Terminology & high level architectural view</a></li>
19<li><a href="#event-driven-systems">Event-driven systems</a></li>
20<li><a href="#event-sourcing">Event sourcing</a></li>
21<li><a href="#wait-we-store-all-events">Wait, we store all events?</a></li>
22<li><a href="#querying-the-data">Querying the data</a></li>
23<li><a href="#inherent-costs">Inherent costs</a></li>
24<li><a href="#an-example-of-this-styles-use">An example of this style’s use</a></li>
25<li><a href="#storage-space-concerns">Storage space concerns</a></li>
26<li><a href="#a-worked-example-in-haskell">A worked example in Haskell</a></li>
27<li><a href="#closing">Closing</a></li>
28</ul>
29</li>
30</ul>
31
32    </div>
33  </div>
34  <div class="stackedit__right">
35    <div class="stackedit__html">
36      <p><a href="/">&lt;- ahri.net</a></p>
37<h1 id="practical-event-driven--sourced-programs-in-haskell">Practical event driven &amp; sourced programs in Haskell</h1>
38<p>I’ve been interested in event-driven systems for a long time now, along with the intriguing concept of storing every event to be sourced later to construct new “materialised” forms. I’m using these techniques in an upcoming project, and decided to open-source the event database I wrote as <a href="https://github.com/ahri/eventdb">EventDB</a>, a Haskell library providing a simple, safe and reasonably efficie
38nt way to store and access an event stream.</p>
39<p>I’m going to provide a quick overview of the technique of driving a system through events, and what sourcing those events means in this context, and then show some code. Feel free to <a href="#a-worked-example-in-haskell">skip ahead</a> to the code if you’re already familiar with this style!</p>
40<h2 id="terminology--high-level-architectural-view">Terminology &amp; high level architectural view</h2>
41<dl>
42<dt><em>Command</em></dt>
43<dd>A small chunk of data expressing a desired change to the <em>state</em>.</dd>
44<dt><em>State</em></dt>
45<dd>A digested form of the <em>events</em>, usually there will be one used to validate <em>commands</em> and others built for specific <em>querying</em> roles.</dd>
46<dt><em>Validation Error</em></dt>
47<dd>The meaningful result of a failed <em>command</em>.</dd>
48<dt><em>Event</em></dt>
49<dd>An isolated piece of information recording an immutable fact.</dd>
50<dt><em>Query</em></dt>
51<dd>A read-only request made to a <em>state</em>.</dd>
52<dt><em>Store</em></dt>
53<dd>Some durable place to keep all <em>events</em>.</dd>
54<dt><em>Subscriber</em></dt>
55<dd>Any downstream system that would like to act upon any/all <em>events</em>.</dd>
56</dl>
57<h3 id="normal-operation">Normal operation</h3>
58<div class="mermaid"><svg xmlns="http://www.w3.org/2000/svg" id="mermaid-svg-VbAH8sfXVVmdUlmV" width="100%" style="max-width: 461px;" viewBox="0 0 461 657.5782699584961"><g transform="translate(-12, -12)"><g class="output"><g class="clusters"></g><g class="edgePaths"><g class="edgePath" style="opacity: 1;"><path class="path" d="M281.5725789582073,32.51942889954611L105.5,78.35832977294922L105.5,116.71665954589844" marker-end="url(#arrowhead321)" style="stroke: #333; stroke-width: 3.5px;fill:none"></path><defs><marker id="arrowhead321" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M105.5,163.43331909179688L105.5,188.43331909179688L150.55953173587096,237.37378735592594" marker-end="url(#arrowhead322)" style="stroke: #333; stroke-width: 3.5px;fill:none"></path><defs><marker id="arrowhead322" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M154.9834510878517,292.76176377095715L118,349.6366424560547L118,391.3533020019531" marker-end="url(#arrowhead323)" style="stroke: #333; stroke-width: 3.5px;fill:none"></path><defs><marker id="arrowhead323" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M204.05451535297632,243.9878344447732L291.25,188.43331909179688L291.25,140.07498931884766L291.25,78.35832977294922L291.25,40" marker-end="url(#arrowhead324)" style="stroke: #333; fill:none;stroke-width:2px;stroke-dasharray:3;"></path><defs><marker id="arrowhead324" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M112.73824600124345,409.8570626322019L86,453.06996154785156L86,491.4282913208008" marker-end="url(#arrowhead325)" style="stroke: #333; stroke-width: 3.5px;fill:none"></path><defs><marker id="arrowhead325" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M126.60874323093748,406.4414784910097L205.5,453.06996154785156L239.99453223915071,491.4282913208008" marker-end="url(#arrowhead326)" style="fill:none"></path><defs><marker id="arrowhead326" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M86,538.1449508666992L86,576.5032806396484L171.9255690461727,614.8616104125977" marker-end="url(#arrowhead327)" style="stroke: #333; stroke-width: 3.5px;fill:none"></path><defs>
58<marker id="arrowhead327" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M284.5224457822568,614.8616104125977L383.5,576.5032806396484L383.5,514.78662109375L383.5,453.06996154785156L383.5,401.3533020019531L383.5,349.6366424560547L383.5,262.3558158874512L383.5,188.43331909179688L383.5,140.07498931884766L383.5,78.35832977294922L300.10685868810344,34.642849790696104" marker-end="url(#arrowhead328)" style="stroke: #333; fill:none;stroke-width:2px;stroke-dasharray:3;"></path><defs><marker id="arrowhead328" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M268.94801482842945,491.4282913208008L282,453.06996154785156L282,401.3533020019531L282,349.6366424560547L201.11241440421895,284.6658982788865" marker-end="url(#arrowhead329)" style="stroke: #333; fill:none;stroke-width:2px;stroke-dasharray:3;"></path><defs><marker id="arrowhead329" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g></g><g class="edgeLabels"><g class="edgeLabel" style="opacity: 1;" transform="translate(105.5,78.35832977294922)"><g transform="translate(-32.5,-13.358329772949219)" class="label"><foreignObject width="65" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>command</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform=""><g transform="translate(0,0)" class="label"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(118,349.6366424560547)"><g transform="translate(-37.5,-13.358329772949219)" class="label"><foreignObject width="75" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel">yes, <i>events</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(291.25,140.07498931884766)"><g transform="translate(-63,-13.358329772949219)" class="label"><foreignObject width="126" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel">no, <i>validation error</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(86,453.06996154785156)"><g transform="translate(-22,-13.358329772949219)" class="label"><foreignObject width="44" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>events</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(205.5,453.06996154785156)"><g transform="translate(-22,-13.358329772949219)" class="label"><foreignObject width="44" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>events</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(86,576.5032806396484)"><g transform="translate(-22,-13.358329772949219)" class="label"><foreignObject width="44" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>events</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(383.5,349.6366424560547)"><g transform="translate(-81.5,-13.358329772949219)" class="label"><foreignObject width="163" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel">process new <i>commands</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(282,401.3533020019531)"><g transform="translate(-17,-13.358329772949219)" class="label"><foreignObject width="34" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>state</i></span></div></foreignObject></g></g></g><g class="nodes"><g class="node" style="opacity: 1;" id="START" transform="translate(291.25,30)"><circle x="-10" y="-10" r="10"></circle><g class="label" transform="translate(0,0)"><g transform="translate(0,0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"></div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="EXEC" transform="translate(105.5,140.07498931884766)"><rect rx="0" ry="0" x="-38" y="-23.35832977294922" width="76" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-28,-13.358329772949219)"><foreignObject width="56" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Execute</div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="VALID" transform="translate(173.5,262.3558158874512)"><polygon points="48.9224967956543,0 97.8449935913086,-48.9224967956543 48.9224967956543,-97.8449935913086 0,-48.9224967956543" rx="5" ry="5" transform="translate(-48.9224967956543,48.9224967956543)"></polygon><g class="label" transform="translate(0,0)"><g transform="translate(-21,-13.358329772949219)"><foreignObject width="42" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Valid?</div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="MULTIPLEX" transform="translate(118,401.3533020019531)"><circle x="-10" y="-10" r="10"></circle><g class="label" transform="translate(0,0)"><g transform="translate(0,0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"></div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="STORE" transform="translate(86,514.78662109375)"><rect rx="0" ry="0" x="-66" y="-23.35832977294922" width="132" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-56,-13.358329772949219)"><foreignObject width="112" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Commit to <i>store</i></div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="APPLY" transform="translate(261,514.78662109375)"><rect rx="0" ry="0" x="-59" y="-23.35832977294922" width="118" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-49,-13.358329772949219)"><foreignObject width="98" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Apply to <i>state</i></div></foreignObject></g></g></g><g class="node" style="opacity: 1;
58" id="PUBLISH" transform="translate(224.25,638.2199401855469)"><rect rx="0" ry="0" x="-97.5" y="-23.35832977294922" width="195" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-87.5,-13.358329772949219)"><foreignObject width="175" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Publish to any <i>subscribers</i></div></foreignObject></g></g></g></g></g></g></svg></div>
59<h3 id="replaying-from-store-e.g.-to-bootstrap-program-start">Replaying from store (e.g. to bootstrap program start)</h3>
60<div class="mermaid"><svg xmlns="http://www.w3.org/2000/svg" id="mermaid-svg-uZAjqel7zJEGxRNI" width="100%" style="max-width: 630px;" viewBox="0 0 630 62.71665954589844"><g transform="translate(-12, -12)"><g class="output"><g class="clusters"></g><g class="edgePaths"><g class="edgePath" style="opacity: 1;"><path class="path" d="M40,43.35832977294922L65,43.35832977294922L90,43.35832977294922" marker-end="url(#arrowhead348)" style="fill:none"></path><defs><marker id="arrowhead348" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M219,43.35832977294922L266,43.35832977294922L313,43.35832977294922" marker-end="url(#arrowhead349)" style="fill:none"></path><defs><marker id="arrowhead349" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g><g class="edgePath" style="opacity: 1;"><path class="path" d="M431,43.35832977294922L473,43.35832977294922L515,43.35832977294922" marker-end="url(#arrowhead350)" style="fill:none"></path><defs><marker id="arrowhead350" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1px; stroke-dasharray: 1px, 0px;"></path></marker></defs></g></g><g class="edgeLabels"><g class="edgeLabel" style="opacity: 1;" transform=""><g transform="translate(0,0)" class="label"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(266,43.35832977294922)"><g transform="translate(-22,-13.358329772949219)" class="label"><foreignObject width="44" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>events</i></span></div></foreignObject></g></g><g class="edgeLabel" style="opacity: 1;" transform="translate(473,43.35832977294922)"><g transform="translate(-17,-13.358329772949219)" class="label"><foreignObject width="34" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"><span class="edgeLabel"><i>state</i></span></div></foreignObject></g></g></g><g class="nodes"><g class="node" style="opacity: 1;" id="START" transform="translate(30,43.35832977294922)"><circle x="-10" y="-10" r="10"></circle><g class="label" transform="translate(0,0)"><g transform="translate(0,0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;"></div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="READ" transform="translate(154.5,43.35832977294922)"><rect rx="0" ry="0" x="-64.5" y="-23.35832977294922" width="129" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-54.5,-13.358329772949219)"><foreignObject width="109" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Read from <i>store</i></div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="APPLY" transform="translate(372,43.35832977294922)"><rect rx="0" ry="0" x="-59" y="-23.35832977294922" width="118" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-49,-13.358329772949219)"><foreignObject width="98" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Apply to <i>state</i></div></foreignObject></g></g></g><g class="node" style="opacity: 1;" id="USE" transform="translate(574.5,43.35832977294922)"><rect rx="5" ry="5" x="-59.5" y="-23.35832977294922" width="119" height="46.71665954589844"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-49.5,-13.358329772949219)"><foreignObject width="99" height="26.716659545898438"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; white-space: nowrap;">Use new <i>state</i></div></foreignObject></g></g></g></g></g></g></svg></div>
61<h2 id="event-driven-systems">Event-driven systems</h2>
62<p>Any interactive system is driven at least in part by events, usually triggered by a user; these might be click, typing, closing a browser tab, etc. The difference in a system self-described as “event-driven” is the embodiment of those events as first-class objects recognised globally as being a set of <em>all</em> things that happen to the system.</p>
63<p>It’s probably worth mentioning that in order to remain sane the events tend to be a little less granular than “User clicked screen at point 1207x1682”, instead opting for atomic aggregates of raw events with some special meaning for the system, e.g. “User changed email address to <a href="/cdn-cgi/l/email-protection#37515858775556451954585a"><span class="__cf_email__" data-cfemail="65030a0a250704174b060a08">[email&#160;protected]</span></a>”.</p>
64<h2 id="event-sourcing">Event sourcing</h2>
65<p>Event sourcing is a fancy term for maintaining an audit log of all events occurring in a system, and <em>sourcing</em>
65 the overall state of that system by walking the log, digesting the entries into a state of some sort - in Haskell terms this is a “fold”. Everything else you might read is fluff around that central (and simple) concept.</p>
66<p>I first heard about event sourcing at a previous job when I was working with C# and we had <a href="https://twitter.com/gregyoung">Greg Young</a> come along to talk about his <a href="https://eventstore.org/">Event Store</a> software, and the simplicity of the idea (coupled with his <a href="https://www.youtube.com/watch?v=8JKjvY4etTY">energetic delivery</a>) left it bubbling around in my head for the past few years. With the recent rise in popularity of <a href="https://kafka.apache.org/">Apache Kafka</a> (an event queue/store/processor) event-driven systems are being more widely discussed. It’s also worth noting that by configuring Kafka to hold events forever, it can behave as an event store too.</p>
67<h2 id="wait-we-store-all-events">Wait, we store all events?</h2>
68<p>Yep, one interesting characteristic of an event-sourced system is that all events should be held, immutably, forever! Disk space is cheap these days, right?! All kidding aside, events don’t need to take up that much space, and there are a couple of approaches to reducing storage use that I’ll <a href="#storage-space-concerns">cover later</a>. That said, this requirement should highlight that not all systems are suited to being event-sourced in such a puritanical sense.</p>
69<h2 id="querying-the-data">Querying the data</h2>
70<p>One very important observation is that in a primitive form the data stored in an audit log isn’t particularly useful because we can’t easily query it; we have to digest those events into some useful form, which will normally be a program state but could equally be updating a relational database, or any other form of database for that matter. Once we have the data in a more useful form we can query it as usual.</p>
71<p>You may be wondering, then, what the point of maintaining the log is: it’s there as the central form of truth for our program, because it’s immutable it allows us to trust the information held there, and to derive new digested (“materialised”) forms of the data in the face of changing requirements. This allows us to skirt the issue of database design being an up-front task that could have ramifications years into the future - as our needs change we can simply fold over the events and produce a new state in whatever guise that takes - be it in-memory, held in a database, or perhaps a specially tuned form for an admin interface vs. the one used in a public web app. This flexibility is enticing!</p>
72<p>On the other hand, how many systems you’ve worked with ever swapped database, despite that awesome selling-point on whatever ORM (Object-Relational Model) you used that it supported 5 different databases? I wouldn’t invest in an event-sourced system for this nifty trick any more than I’d take the hit on magical complexity that comes with an ORM when I only had to talk to one database.</p>
73<h3 id="cqs-commandquery-separation"><a href="https://en.wikipedia.org/wiki/Command%E2%80%93query_separation">CQS</a> (Command/Query Separation)</h3>
74<p>CQS and <a href="https://en.wikipedia.org/wiki/Command%E2%80%93query_separation#Command_query_responsibility_segregation">CQRS</a> are object-oriented concepts talking about splitting responsibilities for dealing with data, where commands write data and queries read it. As a simplification we can consider queries to be dealt with already by existing technologies; e.g. using SQL to query a read-only database satisfies the requirement of a query in this context.</p>
75<p>This style is natural to Haskell programmers, as purity and immutability are so strongly built into the language that reads and writes of data tend to be strongly delineated.</p>
76<p>Commands are simple bits of data that, through execution against a state, result in zero or more events, which in turn are stored and applied back to the program state.</p>
77<h2 id="inherent-costs">Inherent costs</h2>
78<p>The benefits we get with any abstraction always come at a cost, there’s always a compromise we make in adopting them whether it’s in terms of performance, code 
78complexity or maintenance.</p>
79<p>Building a system around events can be overkill; it gives us this superpower to go back to the start and rebuild everything in a new form, but that means an extra level of abstraction from solving the problem at hand, and also incurs an overhead because the system is now “eventually consistent” as the data we’re querying may not yet be up to date.</p>
80<p>For more in-depth write-ups on the costs involved I suggest reading <a href="https://chriskiehl.com/article/event-sourcing-is-hard">Event Sourcing is Hard</a> and <a href="https://medium.com/@hugo.oliveira.rocha/what-they-dont-tell-you-about-event-sourcing-6afc23c69e9a">What they don’t tell you about event sourcing</a>.</p>
81<h2 id="an-example-of-this-styles-use">An example of this style’s use</h2>
82<p>I’m building an application with a fat-client model, where those clients may spend plenty of time offline, some notable characteristics of this application are:</p>
83<ul>
84<li>Clients can produce and validate commands locally,  and apply the resultant events to a local state for “pre”-validation and a nicer user experience</li>
85<li>Clients transmit commands to the server,  when they’re online</li>
86<li>The server re-validates the commands based on its (trusted) state</li>
87<li>Any valid commands produce events</li>
88<li>The server <em>stores them</em></li>
89<li>The server then re-transmits those events to any subscribers - thus keeping the system as a whole in-sync (eventually)</li>
90</ul>
91<p>This model allows me to avoid multiple patterns and paradigms, and other costly abstractions, so that I can share the logic on clients and the server.</p>
92<p>Choosing an event-driven architecture means that I can take a consistent approach to the rather inconvenient aspect that clients might be offline a lot, and it’s low-traffic enough that storing all the events generated by a user feels like a reasonable task.</p>
93<h2 id="storage-space-concerns">Storage space concerns</h2>
94<p>Storing every event could become costly depending on how many we’re dealing with. As such we can take a number of approaches to limit this cost (from least-to-most radical):</p>
95<ol>
96<li>Compress the data</li>
97<li>Rebuild the log from time to time, from the current state; this normalises the events, but may lose data in the process</li>
98<li>Build state snapshots and throw away all prior events - should only be considered if we wish to keep modelling our system as eventually-consistent and event-driven, but no longer need to rewind time, Kafka seems like a good solution for this case</li>
99</ol>
100<p>Approach (2) is a little more subtle, being a compromise between the two extremes, so an example is in order:</p>
101<p>If the state is a number and the current value is 5, perhaps this was arrived at through 3 events; “+3”, “+8” and “-6”, but as we understand the state, we know we can arrive at the current state with a single event: “+5”.</p>
102<p>We lose the information about how many amendments were needed, which might be useful for understanding our user experience, but we save space so it may be a reasonable trade-off.</p>
103<p><em>Note that this example shows events relative to a state, which is something that we should avoid in the general case (though it may be unavoidable) - guidance on events may be covered in another post.</em></p>
104<p>In more clear-cut cases we may have retired some event that is no longer contributing to the state at all, so we can just strip all occurrences to save the space. Of course this means that it may be difficult to put them back if we change our minds later!</p>
105<h2 id="a-worked-example-in-haskell">A worked example in Haskell</h2>
106<p>As a simple worked example of event sourcing we can consider a single bank account, let’s start with some data types!</p>
107<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token keyword">newtype</span> <span class="token constant">AcctHolder</span> <span class="token operator">=</span> <span class="token constant">AcctHolder</span> <span class="token constant">String</span>
108<span class="token keyword">newtype</span> <span class="token constant">Balance</span> <span class="token operator">=</span> <span class="token constant">Balance</span> <span class="token constant">Integer</span>
109
110<span class="token keyword">data</span> <span class="token constant">State</span> <span class="token operator">=</span> <span class="token constant">State</span>
111    <span class="token punctuation">{</span> <span class="token hvariable">holder</span>  <span class="token operator">::</span> <span class="token constant">TVar</span> <span class="token constant">AcctHolder</span>
112    <span class="token punctuation">,</span> <span class="token hvariable">balance</span> <span class="token operator">::</span> <span class="token constant">TVar</span> <span class="token constant">Balance</span>
113    <span class="token punctuation">}</span>
114</code></pre>
115<p>Straightforward so far, we’re representing the whole program state as a single account for simplicity, and we’re using <
115a href="https://hackage.haskell.org/package/stm">STM</a>’s <a href="https://hackage.haskell.org/package/stm-2.5.0.0/docs/Control-Concurrent-STM-TVar.html">TVar</a> abstraction to define granularity in the model.</p>
116<p>Now, the rest of the <a href="#cqs-commandquery-separation">CQS</a> and event machinery:</p>
117<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token keyword">data</span> <span class="token constant">Command</span>
118    <span class="token operator">=</span> <span class="token constant">Rename</span> <span class="token constant">String</span>
119    <span class="token operator">|</span> <span class="token constant">Deposit</span> <span class="token constant">Integer</span>
120    <span class="token operator">|</span> <span class="token constant">Withdraw</span> <span class="token constant">Integer</span>
121
122<span class="token keyword">data</span> <span class="token constant">Event</span>
123    <span class="token operator">=</span> <span class="token constant">Renamed</span> <span class="token constant">String</span>
124    <span class="token operator">|</span> <span class="token constant">Deposited</span> <span class="token constant">Integer</span>
125    <span class="token operator">|</span> <span class="token constant">Withdrew</span> <span class="token constant">Integer</span>
126    <span class="token keyword">deriving</span> <span class="token punctuation">(</span><span class="token constant">Show</span><span class="token punctuation">,</span> <span class="token constant">Read</span><span class="token punctuation">)</span>
127</code></pre>
128<p>We’ve defined some operations on the account, in imperative language, and specified the events we expect to see, in past-tense. In this case there’s a 1:1 matching, but that doesn’t need to be the case, indeed we could trivially have events with no corresponding command, e.g. we could model interest rate alterations with events that have no command. This level of complexity will do for now though!</p>
129<p>We’ve also had the compiler derive instances of <code>Show</code> and <code>Read</code> for simple serialisation purposes - of course more efficient representations may be worthwhile, but this is not a post about efficient binary representation so we’ll stick with simplicity.</p>
130<p>Now, for some functions to play with these data structures:</p>
131<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token hvariable">exec</span> <span class="token operator">::</span> <span class="token constant">State</span> <span class="token operator">-&gt;</span> <span class="token constant">Command</span> <span class="token operator">-&gt;</span> <span class="token constant">STM</span> <span class="token punctuation">(</span><span class="token constant">Either</span> <span class="token constant">String</span> <span class="token punctuation">[</span><span class="token constant">Event</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
132<span class="token hvariable">exec</span> <span class="token hvariable">state</span> <span class="token hvariable">cmd</span> <span class="token operator">=</span> <span class="token keyword">case</span> <span class="token hvariable">cmd</span> <span class="token keyword">of</span>
133    <span class="token constant">Rename</span> <span class="token hvariable">x</span>        <span class="token operator">-&gt;</span> <span class="token hvariable">pure</span> <span class="token operator">$</span> <span class="token constant">Right</span> <span class="token punctuation">[</span><span class="token constant">Renamed</span> <span class="token hvariable">x</span><span class="token punctuation">]</span>
134
135    <span class="token constant">Deposit</span> <span class="token hvariable">x</span>
136        <span class="token operator">|</span> <span class="token hvariable">x</span> <span class="token operator">&lt;</span> <span class="token number">1</span>     <span class="token operator">-&gt;</span> <span class="token hvariable">pure</span> <span class="token operator">$</span> <span class="token constant">Left</span> <span class="token string">"Deposits of less than 1 are not allowed"</span>
137        <span class="token operator">|</span> <span class="token builtin">otherwise</span> <span class="token operator">-&gt;</span> <span class="token hvariable">pure</span> <span class="token operator">$</span> <span class="token constant">Right</span> <span class="token punctuation">[</span><span class="token constant">Deposited</span> <span class="token hvariable">x</span><span class="token punctuation">]</span>
138
139    <span class="token constant">Withdraw</span> <span class="token hvariable">x</span> <span class="token operator">-&gt;</span> <span class="token keyword">do</span>
140        <span class="token punctuation">(</span><span class="token constant">Balance</span> <span class="token hvariable">bal</span><span class="token punctuation">)</span> <span class="token operator">&lt;-</span> <span class="token hvariable">readTVar</span> <span class="token operator">$</span> <span class="token hvariable">balance</span> <span class="token hvariable">state</span> <span class="token comment">-- here we express a fine-grained dependency</span>
141        <span class="token hvariable">pure</span> <span class="token operator">$</span> <span class="token keyword">if</span> <span class="token operator">|</span> <span class="token hvariable">x</span> <span class="token operator">&gt;</span> <span class="token hvariable">bal</span>   <span class="token operator">-&gt;</span> <span class="token constant">Left</span> <span class="token string">"Insufficient balance"</span>
142                  <span class="token operator">|</span> <span class="token hvariable">x</span> <span class="token operator">&lt;</span> <span class="token number">1</span>     <span class="token operator">-&gt;</span> <span class="token constant">Left</span> <span class="token string">"Withdrawls of less than 1 are not allowed"</span>
143                  <span class="token operator">|</span> <span class="token builtin">otherwise</span> <span class="token operator">-&gt;</span> <span class="token constant">Right</span> <span class="token punctuation">[</span><span class="token constant">Withdrew</span> <span class="token hvariable">x</span><span class="token punctuation">]</span>
144</code></pre>
145<p>This function allows us to execute commands against a state, doing validation on the command issued in the context of the current state, but at the same time usage of the <code>readTVar</code> means we are telling STM that we depend on the balance of the account, and any change to that balance will trigger an abort and re-run of the logic.</p>
146<p>Note especially that no writes occur to the state; we’re just producing events to write. Now we need to apply them to the state:</p>
147<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token hvariable">apply</span> <span class="token operator">::</span> <span class="token constant">State</span> <span class="token operator">-&gt;</span> <span class="token punctuation">[</span><span class="token constant">Event</span><span class="token punctuation">]</span> <span class="token operator">-&gt;</span> <span class="token constant">STM</span> <span class="token punctuation">(</span><span class="token punctuation">)</span>
148<span class="token hvariable">apply</span> <span class="token hvariable">state</span> <span class="token hvariable">evs</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token builtin">flip</span> <span class="token hvariable">traverse_</span><span class="token punctuation">)</span> <span class="token hvariable">evs</span> <span class="token operator">$</span> <span class="token operator">\</span><span class="token keyword">case</span>
149    <span class="token constant">Renamed</span> <span class="token hvariable">x</span>   <span class="token operator">-&gt;</span> <span class="token hvariable">writeTVar</span> <span class="token punctuation">(</span><span class="token hvariable">holder</span> <span class="token hvariable">state</span><span class="token punctuation">)</span> <span class="token operator">$</span> <span class="token constant">AcctHolder</span> <span class="token hvariable">x</span>
150
151    <span class="token constant">Deposited</span> <span class="token hvariable">x</span> <span class="token operator">-&gt;</span> <span class="token keyword">do</span>
152        <span class="token punctuation">(</span><span class="token constant">Balance</span> <span class="token hvariable">bal</span><span class="token punctuation">)</span> <span class="token operator">&lt;-</span> <span class="token hvariable">readTVar</span> <span class="token operator">$</span> <span class="token hvariable">balance</span> <span class="token hvariable">state</span>
153        <span class="token hvariable">writeTVar</span> <span class="token punctuation">(</span><span class="token hvariable">balance</span> <span class="token hvariable">state</span><span class="token punctuation">)</span> <span class="token operator">$</span> <span class="token constant">Balance</span> <span class="token operator">$</span> <span class="token hvariable">bal</span> <span class="token operator">+</span> <span class="token hvariable">x</span>
154
155    <span class="token constant">Withdrew</span>  <span class="token hvariable">x</span> <span class="token operator">-&gt;</span> <span class="token keyword">do</span>
156        <span class="token punctuation">(</span><span class="token constant">Balance</span> <span class="token hvariable">bal</span><span class="token punctuation">)</span> <span class="token operator">&lt;-</span> <span class="token hvariable">readTVar</span> <span class="token operator">$</span> <span class="token hvariable">balance</span> <span class="token hvariable">state</span>
157        <span class="token hvariable">writeTVar</span> <span class="token punctuation">(</span><span class="token hvariable">balance</span> <span class="token hvariable">state</span><span class="token punctuation">)</span> <span class="token operator">$</span> <span class="token constant">Balance</span> <span class="token operator">$</span> <span class="token hvariable">bal</span> <span class="token operator">-</span> <span class="token hvariable">x</span>
158</code></pre>
159<p>It’s worth noting here that no checks are made here; when applying events to a state we are dealing with facts that have happened in the past; we can’t reject them now.</p>
160<p>If it seems a bit disjoint to have both <code>exec</code> and <code>apply</code> that’s because it is: we split these up in order to re-use <code>apply</code> in another context; when we replay all events to construct a new state.</p>
161<p>Now we’ve got the events we can write them to the state, again specifying writes to only the related data, take the <code>Renamed</code> event as an example; changes to the balance won’t trigger the abort/retry behaviour, this general capability allows us to express the narrowest set of dependencies possible, thereby improving potential concurrency for our system. To put this another way: if we depend on every bit of our state then <em>any</em> alterations to the state invalidate our current operation.</p>
162<p>We wouldn’t have much of an event-sourced system if we didn’t write events to the audit log in order to source later. We can encode a transaction that calls <code>exec</code> and <code>apply</code> and writes the resultant events using STM’s <code>atomically</code> function:</p>
163<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token hvariable">transact</span> <span class="token operator">::</span> <span class="token constant">Connection</span> <span class="token operator">-&gt;</span> <span class="token constant">State</span> <span class="token constant">TVar</span> <span class="token operator">-&gt;</span> <span class="token constant">Command</span> <span class="token operator">-&gt;</span> <span class="token constant">IO</span> <span class="token punctuation">(</span><span class="token punctuation">)</span>
164<span class="token hvariable">transact</span> <span class="token hvariable">conn</span> <span class="token hvariable">state</span> <span class="token hvariable">cmd</span> <span class="token operator">=</span> <span class="token hvariable">atomically</span> <span class="token operator">$</span> <span class="token keyword">do</span>
165    <span class="token hvariable">result</span> <span class="token operator">&lt;-</span> <span class="token hvariable">exec</span> <span class="token hvariable">state</span> <span class="token hvariable">cmd</span>
166    <span class="token keyword">case</span> <span class="token hvariable">result</span> <span class="token keyword">of</span>
167        <span class="token constant">Left</span> <span class="token hvariable">_</span>    <span class="token operator">-&gt;</span> <span class="token hvariable">pure</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">-- in this case we're just ignoring errors</span>
168        <span class="token constant">Right</span> <span class="token hvariable">evs</span> <span class="token operator">-&gt;</span> <span class="token keyword">do</span>
169            <span class="token hvariable">apply</span> <span class="token hvariable">state</span> <span class="token hvariable">evs</span>
170            <span class="token hvariable">writeEventsAsync</span> <span class="token punctuation">(</span><span class="token builtin">fmap</span> <span class="token punctuation">(</span><span class="token constant">C</span><span class="token punctuation">.</span><span class="token builtin">pack</span><span class="token operator"> . </span><span class="token builtin">show</span><span class="token punctuation">)</span> <span class="token hvariable">evs</span><span class="token punctuation">)</span> <span class="token hvariable">conn</span>
171</code></pre>
172<p>All we’re doing here is throwing away invalid commands; in a real program we might do something useful in this case, but as an example this is already complex enough without that complication! We then use the <code>show</code> instance of <code>Event</code> to encode our events into simple strings and write them to the database in a single transaction.</p>
173<p>Now, to put it all together we can create a state:</p>
174<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token hvariable">state</span> <span class="token operator">&lt;-</span> <span class="token constant">State</span> <span class="token operator">&lt;$&gt;</span> <span class="token hvariable">newTVarIO</span> <span class="token punctuation">(</span><span class="token constant">AcctHolder</span> <span class="token string">"John Smith"</span><span class="token punctuation">)</span> <span class="token operator">&lt;*&gt;</span> <span class="token hvariable">newTVarIO</span> <span class="token punctuation">(</span><span class="token constant">Balance</span> <span class="token number">0</span><span class="token punctuation">)</span>
175</code></pre>
176<p>and execute a bunch of commands against that state, concurrently:</p>
177<pre class=" language-haskell"><code class="prism  language-haskell"><span class="token hvariable">conn</span> <span class="token operator">&lt;-</span> <span class="token hvariable">openConnection</span> <span class="token string">"/tmp/eventdb-bank-acct-demo"</span>
178<span class="token hvariable">mapConcurrently_</span>
179    <span class="token punctuation">(</span><span class="token operator">&gt;&gt;=</span> <span class="token hvariable">transact</span> <span class="token hvariable">conn</span> <span class="token hvariable">state</span><span class="token punctuation">)</span>
180    <span class="token operator">$</span> <span class="token punctuation">(</span><span class="token builtin">replicate</span> <span class="token number">10</span> <span class="token operator">$</span> <span class="token hvariable">randomCommand</span><span class="token punctuation">)</span> <span class="token operator">&lt;&gt;</span> <span class="token punctuation">[</span><span class="token hvariable">pure</span> <span class="token operator">$</span> <span class="token constant">Rename</span> <span class="token string">"Jemima Schmidt"</span><span class="token punctuation">]</span>
181</code></pre>
182<p>We’ll skip the implementation of <code>randomCommand</code> in the interests of brevity, but the <a href="https://github.com/ahri/eventdb/blob/d4472ec/app-bank-acct-demo/Main.hs">full source code</a> of a working program is available for inspection.</p>
183<p>The demo program linked above does a little bit more than we have here, specifically it replays the event log into a new state in order to compare against the original to verify that the resulting program state is indeed exactly the same.</p>
184<p>Now, to run our program (Windows is not currently supported, sorry):</p>
185<pre><code>$ git clone https://github.com/ahri/eventdb.git &amp;&amp; cd eventdb &amp;&amp; git checkout d4472ec &amp;&amp; stack build &amp;&amp; echo &amp;&amp; stack exec -- bank-acct-demo
186
187Initialising State 'John Smith' 0 and creating DB in "/tmp/eventdb-bank-acct-demo"
188Executing random commands concurrently...
189Resulting state: State 'Jemima Schmidt' 5
190
191Replaying events against State 'John Smith' 0
192Deposited 2
193Deposited 5
194Withdrew 5
195Withdrew 1
196Renamed "Jemima Schmidt"
197Deposited 4
198
199Comparing states...
200State 'Jemima Schmidt' 5 == State 'Jemima Schmidt' 5 ~ True
201</code></pre>
202<p>We can observe a few things from this output:</p>
203<ol>
204<li>the 10 commands are fired seemingly at random</li>
205<li>Not all of the commands succeed;
205 recall that we throw away invalid commands</li>
206<li>the audit log (or event store) is replayed</li>
207<li>the resulting replayed state is identical to the original</li>
208</ol>
209<p>We can keep running the program via repeated executions of <code>stack exec -- bank-acct-demo</code>, to observe that in every execution, regardless of the random commands selected, the bank account balance never goes below zero despite the clear race condition inherent in multiple <code>Withdraw</code>s being executed.</p>
210<p>Altering the program to increase the number of concurrent commands attempted may help to convince us of the soundness of the logic.</p>
211<h2 id="closing">Closing</h2>
212<p>I feel that event-driven systems are quite interesting, and that event-sourcing as a tool is quite compelling in its simplicity. Characterising a system in this manner allows us to embrace any disjoint system’s inherent eventual-consistency in an honest way, and, once we’re used to developing a system with this style we can feel some safety in being forced to deal with the unfortunate situations arising in complex, loosely connected systems.</p>
213<p>To me this paradigm feels similar to the benefit that I get from Haskell’s compiler forcing me to (if I were looking at it cynically) jump through hoops, that ultimately saves me from swathes of bugs I’d otherwise introduce and in the best case discover immediately at runtime, and in worse cases leave to fester for someone else to find. It’s abrasive in some ways to be forced to write a program in this style, but on balance I feel that the cost is worthwhile versus the benefits when applied to suitable contexts.</p>
214
215    </div>
216  </div>
vendor: 109 bytes, lines 216-217
216
217<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script>
217<script type="module" src="https://static.cloudflareinsights.com/beacon.min.js/v31edd6df95cf4e85bb4c19e7a9bdbcba1788362987495" integrity="sha512-iIg7k2xntmwu6/uSb5tpc/hySgZc4eoL31yB29W6tJFo2akwjPWcEqnCEdJvGexCL0KEQwVYv5BlowfhVz26hg==" data-cf-beacon='{"version":"2024.11.0","token":"f4b276638b624d88a186f8431f2c59ac","r":1,"spa":2}' crossorigin="anonymous"></script>
217
218</body>
219
220</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.