<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Artur Wojnar on Medium]]></title>
        <description><![CDATA[Stories by Artur Wojnar on Medium]]></description>
        <link>https://medium.com/@arturwojnar.dev?source=rss-84316362a96d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d4_FzdKwOaSrqhPB2hib_Q.jpeg</url>
            <title>Stories by Artur Wojnar on Medium</title>
            <link>https://medium.com/@arturwojnar.dev?source=rss-84316362a96d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:29:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arturwojnar.dev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Comparing Time Availability archetype implementations]]></title>
            <link>https://medium.com/@arturwojnar.dev/comparing-time-availability-archetype-implementations-628499efaa28?source=rss-84316362a96d------2</link>
            <guid isPermaLink="false">https://medium.com/p/628499efaa28</guid>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[archetypes]]></category>
            <dc:creator><![CDATA[Artur Wojnar]]></dc:creator>
            <pubDate>Mon, 24 Jun 2024 09:26:12 GMT</pubDate>
            <atom:updated>2024-08-04T20:06:02.655Z</atom:updated>
            <content:encoded><![CDATA[<h4>Finding a tailored performant solution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2eGTEOUoCTUoizuk" /></figure><h3>Before we start…</h3><p>This article is a direct follow-up to <a href="https://x.com/jakubpilimon?lang=en">Jakub Pilimon’s</a> workshop “<em>Domain-Driven Design with functional elements</em>”, where we had a chance to mull over one of the <em>Time Availability</em> archetype’s implementations.</p><p>🔎<em>This workshop wouldn’t have been possible</em> <em>without </em><a href="https://www.linkedin.com/in/%F0%9F%A7%99-mateusz-kubaszek-58306466/"><em>Mateusz Kubaszek</em></a><em> and his </em><a href="https://discord.gg/efwHbdJp"><em>Order Of Devs community</em></a><em>.</em></p><p>Along with <a href="https://bd90.pl/">Kamil Kiełbasa</a>, I discovered that this problem can also be solved by one <em>PostgreSQL’s</em> <em>GiST </em>constraint. Kamil also made a nice summary of it on <a href="https://bd90.pl/system-rezerwacji-w-5-minut/">his blog</a> (for Polish readers).</p><p>Lastly, I summarised Jakub’s workshop at my company, where one of my colleagues, <a href="https://www.linkedin.com/in/wojciech-ryczko/">Wojciech Ryczko</a>, pointed out something simple and obvious, yet maybe not as evident as you will see later if you keep reading. Wojciech said this problem can be handled simply with <em>SELECT </em>and <em>WHERE </em>clauses. I was like — “<em>Yeah, it can be… but it’s pointless!</em>”.</p><p><strong><em>We all came together to question how slow the other two solutions are compared to the “best” option </em></strong><em>explained by Jakub</em> (Jakub learned about that at one of the <em>DDD </em>conferences).</p><p><strong>And sometimes, you’re pleasantly surprised by how kind people can be.</strong> <a href="https://www.linkedin.com/in/lukasz-rynek/">Łukasz Rynek</a> is definitely one of those people! He graciously met with me and Mateusz Kubaszek to discuss his large-scale solution. 🎉🥰</p><p><strong>To all of you mentioned here, huge huggy thanks for the inspiration and late-night</strong> <strong>chit-chat.</strong></p><p>🔎<em>Everything described here, including test results, can be found in my </em><a href="https://github.com/arturwojnar/examples/blob/main/postgresql-availability"><em>GitHub repository</em></a><em>.</em></p><h3><strong>What is the Time Availability archetype?</strong></h3><p><em>An Archetype </em>is a defined<strong> solution to a business problem</strong>.<br><em>The Time Availability</em> answers the question: “<em>Can I schedule resource X between D1 and D2?</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*r8Ybx3YL2H5PRrPd5xucPw.png" /><figcaption>Image 1. Oops! Can&#39;t be in two places at the same time!</figcaption></figure><p>Certainly, you bumped into this problem class. <em>Can I book that cottage house for the upcoming weekend? Can I “book” the babysitter, Alice, for this evening? Can I schedule a Bali massage with Nu Luh next Wednesday at 16:00? Can I have this doctor’s call at 14:15-14:30?</em></p><p>Look at <em>Image 2</em>. <strong>It’s a generic object’s representation of the archetype</strong>.<br>A ResourceLock represents a single, continuous lock on a resource. The first solution relies on multiple granular 15-minute time slots that constitute locks. The other two solutions use <em>locks </em>with single time slots (their length can be as long as you want; business-wise, they are reservations). Generally, it&#39;s all about the consistency unit of our aggregate.</p><p>ResourceAvailability is our RootAggregate. It is responsible for creating <em>locks </em>and determining whether a resource is available. Its key quality is that it allows us to decide the scope of data we want to keep consistent (as it is for ResourceLock)—it depends on what resource availability means to us.</p><p>For instance, if we are interested in reservations of doctors’ visits, which depend on doctors’ calendars, then the <em>resource availability </em>will be a single doctor’s visit, whereas the consistency scope will also be on the level of single appointments. If we create something like <em>Airbnb</em>, many users can try to make bookings for the same apartment at different date ranges so that booking dates can conflict. Thus, the <em>consistency scope</em> will be all bookings (locks) for a given apartment (resource).</p><p>Here, I will focus on the latter case. Our primary technical challenge is <em>multiple simultaneous attempts to lock or unlock</em> the same resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*37KmGSHEP9khAKjl2n_w5w.png" /><figcaption>Image2. Object-oriented representation of the TimeAvailability archetype.</figcaption></figure><p>Let’s dive into three technical solutions for the <em>Root Aggregate</em> implementation of the <em>Time Availability</em> archetype.</p><h3>High-level considerations</h3><p>Invariants I’m defining for further consideration are:</p><ul><li><em>A resource can have 0 or multiple locks</em></li><li><em>Locking is making a reservation for a resource for a specific time period aligned to 15 minutes so that you can start the reservation at 16:45 but not at 16:40</em></li><li><em>A reservation can be made by one requester</em></li><li><em>Locking can’t be done for less than 15 minutes</em></li><li><em>Unlocking can be done by the same person who made the locking</em></li><li><em>When unlocked at specific dates, the resource can be locked again by another requester</em></li></ul><p>Quality attributes are:</p><ul><li><em>Lock attempts of the same resource can happen at the same time</em></li><li><em>At the same time, there can be over a million different locks</em></li><li><em>Per one minute, over 1000 lock or unlock requests can happen</em></li><li><em>The longest lock shouldn’t be greater than 30 hours</em></li></ul><p>Implementing the given requirements, such as guaranteeing 15-minute alignments, is straightforward. However, there are two major problems to solve. <strong>The first is how to detect <em>overlapping locks</em></strong>. <strong>The second is handling <em>concurrent access</em></strong>, ensuring that, for example, two identical requests won’t create <em>duplicate locks</em>. We need to <em>reject</em> conflicting locks. The preferable <em>concurrency control method, </em>in this case, is <em>pessimistic locking</em>.</p><p>The boundary of our <em>Root Aggregate</em> is all <em>locks </em>for a given resource; otherwise, we won’t be able to detect overlaps.</p><p>I tested how the solutions would work when detecting conflicts on the app side, but this required loading a significant amount of data within a single <em>Aggregate</em>, significantly impacting performance.</p><p><strong>After juggling various implementations, I realized that the optimal way to implement this <em>Aggregate </em>is entirely on the <em>RDBMS/Repository</em> side </strong>(just like <a href="https://www.linkedin.com/in/jakub-pilimon-449b7984/">Jakub Pilimon</a> claimed)<strong>.</strong> This approach aligns with the <em>Root Aggregate</em> definition as a <em>write model</em> that <em>enforces consistency</em> and protects <em>transactional boundaries</em>. Additionally, it allows us to meet <em>non-functional</em> requirements regarding the amount of data.</p><p>🔎 <em>I like Scott Millett and Nick Tune’s definition from the Patterns, Principles and Practices of Domain-Driven Design. ISBN 978–1–118–71470–6, page 320, chapter 14. It goes: “Domain-Driven Design has the Aggregate pattern to ensure consistency and to define transactional concurrency boundaries for object graphs”.</em></p><p>🔎<em>I have to explain myself. I feel I must! I use (now partially) Prisma in my examples because I just wanted to try it out. </em><strong><em>Generally, I’m strongly against any ORM</em></strong><em> except simple CRUD cases. But you know that engineer’s curiosity… Long story short — avoid Prisma as it lacks many features and sets you far away from the database; Also, its migration tool just… su*ks! It’s more reasonable to pick up </em><a href="https://github.com/drizzle-team/drizzle-orm"><em>DrizzleORM </em></a><em>if you need an ORM.<br>I’m much </em>keener<em> on an advanced wrapper like </em><a href="https://github.com/porsager/postgres"><em>porsager/postgres</em></a><em>. I migrated to it, and the world is again green, sunny and beautiful.</em></p><h3>Testing</h3><p>All test cases for all three solutions are analogous. Here, I’m presenting the tests for <em>time slots</em> for context and reference. If you are interested, you’ll find other test cases on my GitHub repository, linked in the corresponding sections later.</p><pre>import { describe, expect, test } from &#39;@jest/globals&#39;<br>import { PrismaClient } from &#39;@prisma/client&#39;<br>import { PostgreSqlContainer, StartedPostgreSqlContainer } from &#39;@testcontainers/postgresql&#39;<br>import util from &#39;util&#39;<br>import { TimeSlot, TimeSlotRepository } from &#39;./timeslots&#39;<br><br>describe(`time slots`, () =&gt; {<br>  jest.setTimeout(60000)<br><br>  const resourceId1 = 1000<br>  const resourceId2 = 2000<br>  const requester1 = `artur`<br>  const timeslots = [<br>    new TimeSlot(new Date(`2024-05-22 10:00:00`), new Date(`2024-05-22 10:15:00`), resourceId1, requester1),<br>    new TimeSlot(new Date(`2024-05-22 10:15:00`), new Date(`2024-05-22 10:30:00`), resourceId1, requester1),<br>    new TimeSlot(new Date(`2024-05-22 10:30:00`), new Date(`2024-05-22 10:35:00`), resourceId1, requester1),<br>    new TimeSlot(new Date(`2024-05-22 10:45:00`), new Date(`2024-05-22 11:00:00`), resourceId1, requester1),<br>  ]<br><br>  let prisma: PrismaClient<br>  let container: StartedPostgreSqlContainer<br><br>  beforeAll(async () =&gt; {<br>    container = await new PostgreSqlContainer().start()<br>    const datasourceUrl = `postgresql://${container.getUsername()}:${container.getPassword()}@${container.getHost()}:${container.getPort()}/${container.getDatabase()}?schema=public`<br>    prisma = new PrismaClient({ datasourceUrl })<br>    await prisma.$connect()<br>    await runMigrations(datasourceUrl)<br>  })<br><br>  afterAll(async () =&gt; {<br>    await prisma.$disconnect()<br>    await container.stop()<br>  })<br><br>  test(`time slots can be added`, async () =&gt; {<br>    const repo = new TimeSlotRepository(prisma)<br><br>    await repo.createMany(timeslots)<br>    const result = await repo.find(resourceId1)<br><br>    expect(result.length).toBe(4)<br>    expect(result).toEqual(<br>      timeslots.map((data) =&gt; ({<br>        ...data,<br>        id: expect.any(Number),<br>      })),<br>    )<br>  })<br><br>  test(`can&#39;t add an overlapping timeslot`, async () =&gt; {<br>    const repo = new TimeSlotRepository(prisma)<br><br>    await expect(() =&gt; repo.lock(timeslots)).rejects.toThrowError()<br>  })<br><br>  test(`the same time slots can be added but for different resource`, async () =&gt; {<br>    const repo = new TimeSlotRepository(prisma)<br><br>    await repo.lock([<br>      new TimeSlot(new Date(`2024-05-22 10:00:00`), new Date(`2024-05-22 10:15:00`), resourceId2, requester1),<br>      new TimeSlot(new Date(`2024-05-22 10:15:00`), new Date(`2024-05-22 10:30:00`), resourceId2, requester1),<br>      new TimeSlot(new Date(`2024-05-22 10:30:00`), new Date(`2024-05-22 10:35:00`), resourceId2, requester1),<br>      new TimeSlot(new Date(`2024-05-22 10:45:00`), new Date(`2024-05-22 11:00:00`), resourceId2, requester1),<br>    ])<br>    const result = await repo.find(resourceId2)<br><br>    expect(result.length).toBe(4)<br>    expect(result).toEqual(<br>      timeslots.map((data) =&gt; ({<br>        ...data,<br>        resourceId: resourceId2,<br>        id: expect.any(Number),<br>      })),<br>    )<br>  })<br><br>  test(`slots can be unlocked`, async () =&gt; {<br>    const repo = new TimeSlotRepository(prisma)<br><br>    await repo.unlock(resourceId1, requester1)<br><br>    const result = await repo.find(resourceId1)<br><br>    expect(result).toEqual(<br>      timeslots.map((data) =&gt; ({<br>        ...data,<br>        id: expect.any(Number),<br>        locked: false,<br>      })),<br>    )<br>  })<br><br>  test(`slots can be locked again`, async () =&gt; {<br>    const repo = new TimeSlotRepository(prisma)<br><br>    await repo.createMany(timeslots)<br>    const result = await repo.find(resourceId1)<br><br>    expect(result).toEqual(<br>      timeslots.map((data) =&gt; ({<br>        ...data,<br>        locked: true,<br>        id: expect.any(Number),<br>      })),<br>    )<br>  })<br>})</pre><h3>“Timeslots solution”</h3><p><strong>This solution is well-suited for reading and writing because it’s backed by a simple index</strong>, as seen in <em>Image 3</em>. Locks are split into 15-minute (the size is just an example) time slots, allowing us to build a unique<em> B-tree</em> (<em>non-clustered</em>) index on the resourceId and the startTime. This enables us to catch collisions since only 15-minute time slots can be saved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wM-cFw4pqphrWeCnETFUJA.png" /><figcaption>Image 3. Unique index handles “conflicts”.</figcaption></figure><p><strong>As you can surely notice, that handy solution comes at a price.</strong> What if we switch business-wise to one-minute or 20-minute time slots? What if things change, and at some point, we’d like to have one-hour or one-day time slots to cut down on data amount? Then, it turns out, the solution is not very flexible, and to fulfil domain challenges over time, we will be forced to bend our architecture to its extent. Presumably, with this specific approach, we won’t be able to be as flexible as with the other two ways.</p><p>The table and the related indexes have been created with the PSQL:</p><pre>CREATE TABLE &quot;TimeSlot&quot; (<br>    &quot;id&quot; SERIAL NOT NULL,<br>    &quot;requesterId&quot; TEXT NOT NULL,<br>    &quot;resourceId&quot; INTEGER NOT NULL,<br>    &quot;startTime&quot; TIMESTAMP(3) NOT NULL,<br>    &quot;endTime&quot; TIMESTAMP(3) NOT NULL,<br>    &quot;locked&quot; BOOLEAN NOT NULL DEFAULT true,<br><br>    CONSTRAINT &quot;TimeSlot_pkey&quot; PRIMARY KEY (&quot;id&quot;)<br>);<br><br>CREATE INDEX &quot;TimeSlot_resourceId_requesterId_idx&quot; ON &quot;TimeSlot&quot;(&quot;resourceId&quot;, &quot;requesterId&quot;);<br>CREATE INDEX &quot;TimeSlot_resourceId_idx&quot; ON &quot;TimeSlot&quot;(&quot;resourceId&quot;);<br>CREATE UNIQUE INDEX &quot;TimeSlot_resourceId_startTime_key&quot; ON &quot;TimeSlot&quot;(&quot;resourceId&quot;, &quot;startTime&quot;);<br>CREATE UNIQUE INDEX &quot;TimeSlot_resourceId_startTime_locked_key&quot; ON &quot;TimeSlot&quot;(&quot;resourceId&quot;, &quot;startTime&quot;, &quot;locked&quot;);</pre><p>Here’s the implementation of the <em>lock </em>method belonging to a <em>TimeSlotRepository</em>:</p><pre>async lock(<br>  slots: TimeSlot[]<br>) {<br>  const values: Array&lt;[string, number, Date, Date]&gt; = slots.map((slot) =&gt; [<br>    slot.requesterId,<br>    slot.resourceId,<br>    slot.startTime,<br>    slot.endTime,<br>  ])<br><br>  await this._sql.begin(async (sql) =&gt; {<br>    await sql.unsafe(`<br>      LOOK AT THE NEXT LISTING FOR FORMATTED SQL 😄<br>    `)<br>  })<br>}</pre><pre>DO $$<br>DECLARE<br>    slot JSONB;<br>    v_locked BOOLEAN;<br>BEGIN<br>    FOR slot IN SELECT * FROM jsonb_array_elements(&#39;${JSON.stringify(values)}&#39;::jsonb)<br>    LOOP<br>        INSERT INTO &quot;TimeSlot&quot; (&quot;requesterId&quot;, &quot;resourceId&quot;, &quot;startTime&quot;, &quot;endTime&quot;, &quot;locked&quot;)<br>        VALUES (<br>          slot-&gt;&gt;0,<br>          (slot-&gt;&gt;1)::INTEGER,<br>          (slot-&gt;&gt;2)::TIMESTAMPTZ,<br>          (slot-&gt;&gt;3)::TIMESTAMPTZ, <br>          True<br>        )<br>        ON CONFLICT (&quot;resourceId&quot;, &quot;startTime&quot;)<br>        DO UPDATE SET &quot;startTime&quot; = EXCLUDED.&quot;startTime&quot;,<br>            &quot;endTime&quot; = EXCLUDED.&quot;endTime&quot;,<br>            &quot;requesterId&quot; = EXCLUDED.&quot;requesterId&quot;,<br>            &quot;locked&quot;=True<br>        WHERE &quot;TimeSlot&quot;.&quot;locked&quot;=False<br>        RETURNING &quot;TimeSlot&quot;.&quot;locked&quot; INTO v_locked;<br><br>        IF NOT FOUND THEN<br>            RAISE EXCEPTION &#39;CONFLICT&#39;;<br>        END IF;<br>    END LOOP;<br>END $$;</pre><p>So, we either INSERT if a given time slot doesn&#39;t exist, or we UPDATE to lock the time slot again. The ON-CONFLICT pattern used in the above SQL is equivalent to the UPSERT operation. If this clause ends up with no result, meaning neither INSERT nor UPDATE was done, it indicates a conflict occurred.</p><p>Oh, of course, there has to be also some code to convert <em>a date range</em> into <em>time slots</em> like this:</p><pre>const TO_MINUTES = 60000<br><br>function getSlotsForDateRange(requesterId: string, from: Date, to: Date) {<br>  // Calculate slots required<br>  if ((from.getTime() / TO_MINUTES) % 15 !== 0) {<br>    throw new Error(`The date has to be a multiple of 15 minutes`)<br>  }<br>  if ((to.getTime() / TO_MINUTES) % 15 !== 0) {<br>    throw new Error(`The date has to be a multiple of 15 minutes`)<br>  }<br><br>  const durationMinutes = (to.getTime() - from.getTime()) / TO_MINUTES<br>  const slotsRequired = Math.ceil(durationMinutes / timeSlotSize)<br><br>  return Array(slotsRequired)<br>    .fill(0)<br>    .map((_, i) =&gt; {<br>      const startTime = new Date(from.getTime() + i * timeSlotSize * TO_MINUTES)<br>      const endTime = new Date(startTime.getTime() + timeSlotSize * TO_MINUTES)<br>      return new TimeSlot(startTime, endTime, this.resourceId, requesterId)<br>    })<br>}</pre><p>To <em>unlock, </em>we should rely on <em>startTime </em>on which the index is placed:</p><pre>async unlock(resourceId: number, requesterId: string, startTime?: Date, endTime?: Date) {<br>  const startDates = Array((endTime.getTime() - startTime.getTime()) / (15 * 60000))<br>    .fill(0)<br>    .map((_, i) =&gt;<br>      dayjs(startTime)<br>        .add(15 * i, &#39;minutes&#39;)<br>        .toDate()<br>        .toISOString(),<br>    )<br>  const result = await this._sql`<br>    UPDATE &quot;TimeSlot&quot;<br>    SET &quot;locked&quot; = false<br>    WHERE &quot;resourceId&quot; = ${resourceId}<br>    AND &quot;requesterId&quot; = ${requesterId}<br>    AND &quot;startTime&quot; IN (${this._sql(startDates)})<br>  `<br>  return result.count<br>}</pre><h3>“The SELECT solution”</h3><p>The obvious solution is a SELECT FOR UPDATE query with a WHERE clause checking for overlapping dates. <strong>However, we can&#39;t manipulate the </strong><strong>lockedcolumn this time because we don&#39;t have time slots, only entire reservations.</strong> It&#39;s unlikely that all users will be locking and unlocking the same date ranges every time. Therefore, when a <em>reservation </em>is unlocked, it is marked as <em>deleted </em>and, consequently, can&#39;t be reused later.</p><p>You can find the tests here, and the full implementation is <a href="https://github.com/arturwojnar/examples/blob/main/postgresql-availability/src/select/select-timeslots.ts">here</a>.<br>The table and the related indexes have been created with the PSQL:</p><pre>CREATE TABLE &quot;TimeSlot2&quot; (<br>    &quot;id&quot; SERIAL NOT NULL,<br>    &quot;requesterId&quot; TEXT NOT NULL,<br>    &quot;resourceId&quot; INTEGER NOT NULL,<br>    &quot;startTime&quot; TIMESTAMP(3) NOT NULL,<br>    &quot;endTime&quot; TIMESTAMP(3) NOT NULL,<br>    &quot;deleted&quot; BOOLEAN NOT NULL DEFAULT true,<br><br>    CONSTRAINT &quot;TimeSlot2_pkey&quot; PRIMARY KEY (&quot;id&quot;)<br>);<br><br>CREATE INDEX &quot;TimeSlot2_resourceId_requesterId_idx&quot; ON &quot;TimeSlot2&quot;(&quot;resourceId&quot;, &quot;requesterId&quot;);<br>CREATE INDEX &quot;TimeSlot2_resourceId_idx&quot; ON &quot;TimeSlot2&quot;(&quot;resourceId&quot;);<br>CREATE UNIQUE INDEX &quot;TimeSlot2_resourceId_dates_deleted_idx&quot; ON &quot;TimeSlot2&quot;(&quot;resourceId&quot;, &quot;startTime&quot;, &quot;endTime&quot;, &quot;deleted&quot;);</pre><p><em>Locking </em>implementation:</p><pre>DO $$<br>DECLARE<br>  v_count INTEGER;<br>BEGIN<br>  SELECT 1 AS &quot;count&quot;<br>  INTO v_count<br>  FROM &quot;TimeSlot2&quot; t<br>  WHERE &quot;t&quot;.&quot;resourceId&quot;=${resourceId}<br>    AND &quot;t&quot;.&quot;startTime&quot; &lt; &#39;${to.toISOString()}&#39;<br>    AND &quot;t&quot;.&quot;endTime&quot; &gt; &#39;${from.toISOString()}&#39;<br>    AND &quot;t&quot;.&quot;deleted&quot;=False<br>  LIMIT 1<br>  FOR UPDATE;<br><br>  IF FOUND THEN<br>    RAISE EXCEPTION &#39;CONFLICT&#39;;<br>  END IF;<br><br>  INSERT INTO &quot;TimeSlot2&quot; (&quot;requesterId&quot;, &quot;resourceId&quot;, &quot;startTime&quot;, &quot;endTime&quot;, &quot;deleted&quot;)<br>  VALUES (<br>    &#39;${requesterId}&#39;, <br>    ${resourceId},<br>    &#39;${from.toISOString()}&#39;,<br>    &#39;${to.toISOString()}&#39;,<br>    False<br>  );<br><br>END $$;</pre><p>The query analysis shows that the database engine utilized the <em>B-tree index</em> placed on the resourceId column and planned a <em>Bitmap Index Scan</em> for this query, which is not bad. The entire table is not scanned; the bitmap structure keeps pointers found during the index scan and in the end, because of the LIMIT, only one row is locked. As the <a href="https://pganalyze.com/docs/explain/scan-nodes/bitmap-index-scan">pganalyze docs</a> say:</p><blockquote>You can think of a bitmap index scan as a middle ground between a sequential scan and an index scan. Like an index scan, it scans an index to determine exactly what data it needs to fetch, but like a sequential scan, it takes advantage of data being easier to read in bulk.</blockquote><blockquote>The bitmap index scan actually operates in tandem with a Bitmap Heap Scan: it does not fetch the data itself. Instead of producing the rows directly, the bitmap index scan constructs a bitmap of potential row locations. It feeds this data to a parent Bitmap Heap Scan, which can decode the bitmap to fetch the underlying data, grabbing data page by page.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AhpQMrjwat7wW8OF" /><figcaption>Image 3. Bitmap Index Scan for the <em>SELECT FOR UPDATE</em> query.</figcaption></figure><p><em>Unlocking </em>looks this way:</p><pre>async unlock(resourceId: number, requesterId: string, startTime: Date, endTime: Date) {<br>  try {<br>    const result = await this._sql`<br>      UPDATE &quot;TimeSlot2&quot;<br>      SET &quot;deleted&quot; = true<br>      WHERE &quot;requesterId&quot; = ${requesterId}<br>      AND &quot;resourceId&quot; = ${resourceId}<br>      AND &quot;startTime&quot;=${startTime.toISOString()}<br>      AND &quot;endTime&quot;=${endTime.toISOString()}<br>    `<br>    return result.count<br>  } catch (error) {<br>    console.error(&#39;Error unlocking timeslot:&#39;, error)<br>    throw error<br>  }<br>}</pre><p>It is worth noting that the two covered implementations rely on unlocking, which is a simple UPDATE using dedicated indexes. Thus, their performance results will likely be similar.</p><h3>“The GiST solution”</h3><p>The <a href="https://www.postgresql.org/docs/8.1/gist.html"><em>GiST </em></a>(<em>Generalized Search Tree</em>) is one of PostgreSQL’s indexes that can be combined with a set of constraints. <strong>The interesting one for us is the exclusion constraint, an extension of uniqueness, but with that difference, the exclusion guarantees that no two rows overlap.</strong></p><p>In the listing below, you can see how a table with an exclusion constraint can be created. The configuration possibilities are quite impressive. I ensured that the resourceId was taken into account and compared by exact equality (<em>operator &quot;=&quot;</em>), and the date_range was compared by checking for overlap (<em>operator &quot;&amp;&amp;&quot;</em>). Additionally, only rows within the <em>GiST </em>index with falsy deleted values can be found as colliding.</p><pre>-- create the extension first<br>CREATE EXTENSION IF NOT EXISTS btree_gist;<br><br>CREATE TABLE &quot;timeslot3&quot; (<br>    &quot;id&quot; SERIAL NOT NULL,<br>    &quot;requesterid&quot; TEXT,<br>    &quot;resourceid&quot; INTEGER,<br>    &quot;date_range&quot; tsrange,<br>    &quot;deleted&quot; BOOLEAN,<br>    CONSTRAINT &quot;timeslot3_pkey&quot; PRIMARY KEY (&quot;id&quot;),<br>    CONSTRAINT &quot;timeslot3_excl&quot; EXCLUDE USING GIST <br>    (<br>        &quot;resourceid&quot; WITH =, <br>        &quot;date_range&quot; WITH &amp;&amp; <br>    ) WHERE (&quot;deleted&quot; IS FALSE)<br>);<br><br>CREATE UNIQUE INDEX &quot;TimeSlot3_resourceId_daterange_deleted&quot;<br> ON &quot;timeslot3&quot;(&quot;requesterid&quot;, &quot;resourceid&quot;, &quot;date_range&quot;, &quot;deleted&quot;);</pre><p>Please note that the timeslot3_resourceId_daterange index is applied to a set of most of the existing columns (requesterid, resourceid, date_range) to satisfy the access pattern for unlocking.</p><p><em>Locking </em>and <em>unlocking </em>is then easy:</p><pre>async lock({ requesterId, resourceId, date_range }: TimeSlot) {<br>  await this._prisma.$executeRawUnsafe(`<br>    INSERT INTO &quot;timeslot3&quot; (requesterId, resourceId, date_range, deleted)<br>      VALUES (<br>        &#39;${requesterId}&#39;,<br>        ${resourceId},<br>        daterange(&#39;${date_range[0].toISOString()}&#39;, &#39;${date_range[1].toISOString()}&#39;, &#39;[]&#39;), False)<br>  `)<br>}<br><br>async unlock(requesterid: string, resourceid: number, startTime?: Date, endTime?: Date) {<br>  return await this._prisma.$executeRawUnsafe(`<br>    UPDATE &quot;timeslot3&quot; SET &quot;deleted&quot;=True<br>    WHERE &quot;requesterid&quot;=&#39;${requesterid}&#39; AND<br>      &quot;resourceid&quot;=${resourceid} AND<br>      &quot;deleted&quot;=False AND<br>      &quot;date_range&quot;=tsrange(&#39;${startTime.toISOString()}&#39;, &#39;${endTime.toISOString()}&#39;, &#39;[)&#39;)<br>  `)<br>}</pre><p>The SQLs for this solution are the easiest ones because we rely only on the GiST index with the exclusion, no jumble and no puzzles.</p><h3>Comparing performance</h3><p>To examine which solution performs the best, I defined four metrics:</p><ol><li><em>Average time to successfully add a new lock (lock, aka reservation)</em></li><li><em>Average time to fail when adding a new lock (conflict)</em></li><li><em>Average time to unlock</em></li><li><em>Average time to lock the same resource again</em></li></ol><p>Initially, I generated 10,000 locks and obtained comparable results for all three methods. Therefore, I finally decided to conduct tests with the following set:</p><ul><li><em>1,000 resources</em></li><li><em>Each resource has 1,000 locks</em></li><li><em>Each reservation lasts 30 hours (120 time slots for the first solution)</em></li></ul><p>Initially, I generated 10,000 locks and obtained comparable results for all three methods. Therefore, I finally decided to conduct tests with the following set:</p><ul><li>1,000 resources</li><li>Each resource has 1,000 locks</li><li>Each reservation lasts 30 hours (120 time slots for the first solution)</li></ul><p>That gives 120 million rows for the first solution and a million for the other two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_N2DuZo2zCgPePlPIM3CQ.png" /><figcaption>Image 4. Comparison of the average time of operations of all three solutions for a million reservations.</figcaption></figure><p><strong>Basically, we can confidently say that all three solutions are equally efficient.</strong> A million rows is not a small amount, so it should have shown significant differences in performance. But it didn’t. What’s more perplexing is that, with my current knowledge, I can’t explain why the second solution (the “<em>SELECT</em>” one) appeared so effective.</p><p>I bumped up the number of rows for the <em>SELECT </em>and <em>GIST </em>solutions to 120 million to make them more comparable to the first solution regarding the absolute number of tuples. <em>But, again, the implementations don&#39;t diverge from each other.</em></p><p>Unfortunately, this means I have to do more tests. Nothing proves more about the quality of our solution than reliability while <em>concurrent access</em>.</p><h3>Comparing performance on concurrent access</h3><p>Let’s test how our solutions will cope under medium load. Imagine that 10-thousand simultaneous requests to lock a resource happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EszmgNoc0mOppIQuOMdfeQ.png" /><figcaption>Image 5. 10000 new reservations done concurrently.</figcaption></figure><p><strong>Without a shred of doubt, all three functions do well and are comparable</strong> (if your client is OK with a request taking a second and a half).</p><p><strong>And how do things look for a million simultaneous requests?</strong> <strong>The proportions remained consistent</strong> (Image 6), but the average time surged to the minute level, from one and a half to over three minutes. On this scale, the GiST solution has the most noticeable advantage. However, if your system is expected to handle this kind of load regularly, then your customer likely won’t appreciate these results.</p><p>In that situation, you can test your implementation on <em>a</em> <em>vertically scaled PostgreSQL</em> instance and see how more <em>physical memory</em> and <em>vCPU</em> impact the performance. If the results still won’t be satisfying, then maybe the <em>immediate consistency</em> solution is not for you, and your attention should be drawn toward a <a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction">compensation mechanism</a> for a <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/write-ahead-log.html">Write-Ahead Log pattern</a> implementation, that is, for bigger loads.</p><p>Additionally, consider tailoring your solution to your specific business case, which may have particular requirements to help narrow down some scenarios, such as predefined appointments. Lastly, you can explore a solution based on specialized tooling (see the next chapter).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PB2rTM1N3quy0HwuyMmAag.png" /><figcaption><em>Image 6. A million new reservations done concurrently.</em></figcaption></figure><p><strong>I also performed tests for concurrent access for unlocking — the results are good and acceptable even for a million simultaneous requests.</strong> But I knew the results upfront. I’m not a Nostradamus — all of the implementations are straightforward and rely on indexes, and they come down to a single UPDATE statement. The performance of <em>the time slot solution</em> depends on how many slots are updated at once.</p><h3>Solutions based on tooling</h3><p>So far, I’ve only been focusing on “traditional” solutions based on a relationship database, varying in a consistency unit (“small” timeslots consisting of a reservation or whole reservations). The closest solution to this subtopic is the GiST one.</p><p>These days, we read about more and more tools that resolve a particular problem. <strong>Despite not being the biggest fan of solutions driven based on a specific technology, I do see at least a few strong proses of “<em>Technology Driven Development</em>”</strong>. One of those is that someone has already done that. If we’re fine with the license, deploying, maintaining, potential outages, and difficult-to-oversee future changes in this dependency, and if you’re dedicated to expertise in this tool (often a superficial understanding of your 3rd party technology is not enough), then maybe it’s a way to go.</p><p>What technologies can we use for our case?</p><p><a href="https://www.linkedin.com/in/lukasz-rynek/">Łukasz Rynek</a>, a Tech Leader at <a href="https://docplanner.tech/">Docplanner </a>(on the Polish market known as “<em>Znany Lekarz</em>”), was kind to help me understand their approach. <em>The Docplanner</em> system helps you find a specialist and book a visit at a convenient time. Specialists are ranked depending on many factors to suit the best patients&#39; needs.</p><p>First, Łukasz mentioned that their solution is not as generic as I described. What we’ve been primarily focusing on in this article is a system in which users can try to reserve whatever time range they want to, and the system cares about the consistency of all of that. As I wrote in “<em>What is the Time Availability archetype?</em>” this is a case when users rely on someone else’s calendars that are predefined in the system. Hence, the users compete for those predefined slots to change them into their bookings.</p><p>🔎<em>If you’re interested in the Docplanner company and their work, please visit their </em><a href="https://medium.com/docplanner-tech"><em>Medium blog</em></a><em>.</em></p><p>Secondly, Łukasz said <strong>they work in a highly distributed system where possible reservations and available visits are managed by </strong><a href="https://www.elastic.co/elasticsearch"><strong>Elasticsearch</strong></a>. Although most operations leave the system in a consistent state, sometimes it’s not. It is an <em>eventually consistent</em> solution; some deviations and unwanted effects are mitigated with compensation.</p><p><strong>Mateusz Kubaszek, whom I mentioned initially, suggested that Apache Flink can handle the problem successfully</strong> (quoting the docs—“Stateful Computations over Data Streams”). We can use Flink&#39;s capabilities to process and analyse streaming data to achieve the desired functionality in Apache Flink, where no two timeslots can overlap. The idea is to maintain a state that keeps track of the current time slots and checks for overlaps whenever a new request occurs (by continuous processing streams).</p><p>🔎<em>At the time of writing, Microsoft SQL Server doesn&#39;t have a dedicated index/constraint to exclude conflicting date-time ranges. The same goes for MongoDB.</em></p><p><strong>What else can be done?</strong> We can implement a mechanism that if the load is too big, let’s say — it goes under a million simultaneous requests. Then, it switches from immediate consistency to something else. We can gather incrementing logs (<a href="https://martinfowler.com/articles/patterns-of-distributed-systems/write-ahead-log.html">Write-Ahead Log Pattern</a>) for that more intense time period and analyze them by a separate process, which aims to generate results that will eventually transform into target locks or be rejected.</p><p>You can always seek for further database optimizations.</p><p><strong>Do you have other ideas? Or have you already done it somehow? Please share!</strong></p><h3>Summary</h3><p>If you had pointed a gun to my head demanding a short, most likely safe option for your project that doesn&#39;t experience a million concurrent requests at the same time the whole time, I’d say try with PostgreSQL and a GiST index with an exclusion constraint.</p><p>But I don&#39;t have any gun at my head, and I have still some time 🙂</p><p>It all comes (as it mostly does) to your business case. Let’s remember our example with booking predefined slots. That’s a perfect limitation of all possible time slots. Maybe your domain allows you to create one-month upfront slots where users can reserve 15-minute time slots, but only between 9 A.M and 5 P.M., That can be <em>a car repair shop</em> type of business. Depending on the <em>service type</em>, whether a seasonal tyre change or a more complicated air conditioning cleaning, users can book a visit with duration and time varying depending on the service type.</p><p>I’m pretty convinced that many domains work that way.</p><p>If reservations are more generic without any boundaries dictated by the business and you cannot cheat on the immediate consistency, then all of the solutions I’ve brought up can be valid.</p><p>Time slots are clever because they perfectly fit database engine indexes. From that perspective, they feel like a well-sharpened knife, especially when you can narrow the number of slots. <strong>But maybe the knife doesn&#39;t have to be perfectly sharpened if the chef is a two-star Michelin winner?</strong></p><p>Unfortunately, it’s <strong>the least flexible solution</strong> because the business can come up with the idea of 3-minute time slots. Then, depending on the resource type or reservation type, the slots will be different lengths. It’s a lot of the play.</p><p>That’s why the third solution would be considerable if you use PostgreSQL. I haven’t found a weak spot; on the contrary, it’s the best option for bigger loads, as seen for a million simultaneous requests.</p><p>But why does the solution based on the <em>Bitmap Search</em> keep up with the others? I don&#39;t know. <strong>Several dozen years of relational database optimization have done their job</strong>. <strong>If you know, dear reader, please comment on this post. I will be more than happy.</strong></p><p>If your database engine doesn’t provide an equivalent of what we’ve done here, then the “<em>SELECT</em>” solution may work for you.</p><p>You might think that since time availability issues are so common, the solutions to handle them would be well-known and thoroughly documented. However, my article aims to highlight just how complex and challenging these problems can be. I was personally surprised by what I discovered.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=628499efaa28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The ultimate solution to everything]]></title>
            <link>https://medium.com/@arturwojnar.dev/the-ultimate-solution-to-everything-4e698edfc868?source=rss-84316362a96d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e698edfc868</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[legacy-code]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <dc:creator><![CDATA[Artur Wojnar]]></dc:creator>
            <pubDate>Fri, 23 Feb 2024 14:04:37 GMT</pubDate>
            <atom:updated>2024-08-26T13:41:30.287Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0zZHHI2tkfkO9wEW1Ciww.png" /></figure><p><em>“</em><strong><em>STOP </em></strong><em>using it”</em></p><p><em>“If you do </em><strong><em>THAT, THEN YOU DO IT WRONG!</em></strong><em>”</em></p><p><em>“A technique you </em><strong><em>must </em></strong><em>know”</em></p><p><em>“The X framework is </em><strong><em>DEAD</em></strong><em>”</em></p><p><em>“</em><strong><em>The rise</em></strong><em> of Y framework”</em></p><p>If you roll your eyes or, worse, you feel troubled whenever you read an eye-catching, heart-breaking, and click-bait title like one of those above, you’ve finally found a good place for you (I hope so). Let’s delve into a discourse on how we approach problem-solving.</p><p>Renowned astrophysicist Neil deGrasse Tyson once remarked:</p><blockquote>“The best thing we have going for us is our intelligence, especially pattern recognition, sharpened over aeons of evolution</blockquote><p>While pattern recognition is undoubtedly valuable, it can also be a double-edged sword.</p><p>When I learn a new thing, I see it everywhere. When I learned about <a href="https://en.wikipedia.org/wiki/SOLID">SOLID </a>someday, I became a <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> <em>neophyte</em>, saluting strict rules as a soldier bravely standing for the rights of good taste.<br>Whether adopting <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> design principles or fervently embracing the latest framework, we can become entrenched in rigid thinking.</p><p>Despite many years passing, I still catch myself, let’s say, easily falling in love with new things. Maybe it doesn’t last long months, but some days or weeks for sure, looming later over me like an embarrassing memory from school past (you have it too, right?).</p><p>It’s a great way to learn — seeing patterns and introducing them to our solutions- but let’s try to remember to cool down passion and turn on critical thinking. Also, it’s helpful to recall one of the many embarrassing moments!</p><p>People on the Internet often want to sell you something; they are in their “<em>in-love-and-no-question-asked</em>” moments. We have had tiny, truly innovative things in the last few years.</p><p>About becoming a soldier of a new standard: <em>There are soldiers with </em><a href="https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215"><em>blue uniforms</em></a>, right? In that particular case, a specific contradiction arises.</p><p>I read <a href="https://en.wikipedia.org/wiki/Domain-driven_design">DDD</a> as a bunch of experience gathered under some named ideas (like <em>Aggregate</em>), and I think we’ve gotten to this point by now that naming something explicitly in a codebase by its name may be narrowing, making others hold you accountable because this ”<em>pattern says that and that</em>”. So, we tend to lose something far more important from sight — pragmatism and business goals. That’s funny — <a href="https://en.wikipedia.org/wiki/Domain-driven_design">DDD</a> is about business and domains, but because of treating the <a href="https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215"><em>Blue Book</em></a> as the Bible, we may achieve the opposite — focusing too much on technical aspects and definitions of ideas that should only be a guide.</p><p>It’s a lesson I’ve learned: <em>every battle to follow the rules costs you nerves and your client real money</em>.</p><p><em>We tend to follow algorithms (patterns) to solve our problems</em>. We expect to implement the same successful solution of others to achieve the same success. Yet, it almost always requires an individual approach unless we’re solving a problem from a tutorial.</p><p>I’ve learned through experience that blindly following rules can be costly regarding time and resources. Instead, I advocate for a more pragmatic approach focused on managing risk effectively. <br><em>I gathered my most important rules under yet another acronym: 3R</em>.</p><p><strong>Risk management rule<br></strong>There’s no infinite time for planning and development, so everybody talks about trade-offs. Not because something is impossible but because we can’t do 100 architecture iterations or work on one story by one year.</p><p>I think about programming and designing architectures as <em>legacy code management</em>. Legacy code is always with us, but if contained in <em>predictable boxes</em> can be manageable, and the overall project’s costs are more satisfying. First things first: for me, legacy code is code that a developer calls “<em>we have to fix it someday</em>” — it is an implementation that is not done by the book and, at some point, can cause problems.</p><p>I always ask myself and my teams about a decision: “<em>What’s the worst thing that can happen?</em>”</p><p>A few examples:</p><ol><li>Supporting subdomains are places where less-experienced developers can learn and grow without stressful expectations of more complicated implementations of core subdomains (this is also a good place to outsource).<br>I agree with this area’s higher legacy code level (<em>supporting subdomain</em>). Thanks to that, the team doesn’t have to deliver the highest quality. In the worst scenario, I’m okay with that to rewrite that part when a subdomain drifts toward bigger complexity (because the business also shifts in a new direction or is discovering new possibilities).<br>Supporting subdomains are often <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUDs</a>, so they probably rely on <a href="https://martinfowler.com/eaaCatalog/transactionScript.html">Transaction Scripts</a> or <a href="https://www.martinfowler.com/eaaCatalog/activeRecord.html">Active Records </a>— that’s why I cover them with container tests; with that, I know at least that that part works as an end product.</li><li><em>Modules</em>. It’s a matter of cohesion and design independence from other modules. If the modules are not self-evident in the bounded context, then I use the <a href="https://mrpicky.dev/process-level-event-storming-lets-get-this-process-started/"><em>Event Storming Process Level</em></a>. The same is true for supporting subdomains. The modules give me, to some extent, a guarantee that legacy code in one of the modules won’t necessarily propagate to others. So, again, I try to mitigate the risk of spreading the legacy code to other modules like a virus.</li><li><em>Minimise </em><a href="https://en.wikipedia.org/wiki/Generalization"><em>generalisation</em></a>. If you don’t work in a <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall </a>and don’t know the full scope of the product, then it’s impossible to predict all change vectors and how your codebase will evolve.<br>I unzip the thinnest umbrella of commons over modules to minimise coupling between them. It is easy to bump into the Liskov problem and try to modify a “<em>generic class</em>” to support “<em>one thing more</em>”. With that, I mitigate the risk of tightening up things that should be loosely coupled and also of diluting commons, which eventually become a burden (legacy code)</li><li>It’s a recent problem I hope I fixed for good. I tended to make commons from, e.g. <a href="https://martinfowler.com/bliki/ValueObject.html"><em>value objects</em></a> that, similarly to the above point, will lead to the same problem but with the leaking domain. I mitigate the risk by sharing only an infrastructural commons</li><li>Suppose a project is still in the phase of a modular monolith with well-defined data ownership. In that case, I agree with simplifying and embracing database operations from different modules in one database transaction to avoid eventual consistency problems. At some point, when the project evolves to the point where it has to split into separate processes, this implementation will be something to refactor, but up to this point, we save time and money.</li><li><a href="https://www.jimmybogard.com/vertical-slice-architecture/">Slicing</a> the codebase. It’s something other than modules — the slices are more granular, referencing one feature. With that approach, I avoid expanding relationships and unnecessary dependencies between features (the opposite of the <a href="https://www.oreilly.com/library/view/software-architecture-patterns/9781491971437/ch01.html">Layered Architecture</a>).</li></ol><p>It is a fair summary of my thoughts to say that I strive to keep legacy code manageable because I know that it’s inevitable. <em>Legacy code is yet another tool to fit a project implementation into a multidimensional shape created by the intersection of different non-functional constraints</em> <em>like time, budget and team capabilities</em>. It’s like packing numerous pieces of luggage into the car trunk; some of them will get squeezed, and some of the boxes may get deformed a bit, but after all, it’s about protecting what is inside and not forgetting to pack an undeniably needed thing.</p><p>Risk management is broad, but I don’t want to contain everything here. Speaking of that, I encounter two bigger building blocks of my thinking (at least starting from R)</p><p><strong>Redundancy<br></strong>I’ve liked redundancy after years of trying to be <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>. It’s the harmful rule, a relic of the past, yet another example of how thinking is necessary before applying “<em>good practices</em>”.</p><p><em>WET &gt;&gt; DRY</em><br><em>WET </em>stands for <em>Write Everything Twice</em>, the opposite of <em>DRY</em>. But I don’t use it as a hammer. My rule of thumb is to apply <em>DRY </em>on the module level, but for everything else, think 5 times before making a generalisation (see point 3 in the previous section). Infrastructure-specific and typing utils are probably good candidates for a separate package or a common code in a mono repo.</p><p>Worth noticing is that a shared package (kernel) means one <a href="https://www.oreilly.com/library/view/building-evolutionary-architectures/9781491986356/ch04.html"><em>Architectural Quanta</em></a>. Because the DRY part is the single source of truth, it couples components that refer to it.</p><p>One perfect example of a generalisation problem is <em>orphaned classes</em> (or <em>orphaned files</em>). This perspective is extremely useful when we’re still in the extensive domain discovery phase and/or we work on modular monolith.</p><p>If you add another directory to the namespace or directory structure, you may generate some architectural problems despite the fact that it makes total sense from the developer’s point of view.</p><p>Let’s take a look at the following structure:</p><pre>/patient-calendar<br>  # info whether a user wants to receive notifications or in which language<br>  get-patient-info.code <br>  /schedule<br>    cron.code<br>    generate-event.code<br>  /notification<br>    generate-notification.code<br>    push-notification.code</pre><p>It seems to be OK, and maybe it is, but by asking, “How should I split that into services?” some problems will arise. Should <em>the schedule</em> be a separate service? If so, what’s with the <em>get-patient-info</em>? Should it also be a separate thing? Or maybe we should deploy a service called<em> patient-calendar</em>?<br>The orphaned classes’ perspective resolves that dilemma by defining a component as the last directory (<em>leaf </em>directory). With that, the schedule and the notification are components, and the patient calendar can be considered a subdomain. The get-patient-info file is the orphaned class, not belonging to any particular component but (likely) shared between them.</p><p>The rule distilled from these considerations can be summarised as follows: <em>Code can exist only in node leaves (components)</em>.</p><p>The refactoring process for orphaned classes is called the Flatten Components Pattern (introduced by Neal Ford). One way is redesigning with, e.g. Event Storming or merging an orphaned class like this:</p><pre>/patient-calendar<br>  /scheduler<br>    get-patient-info.code<br>    cron.code<br>    generate-event.code<br>  /notification<br>    get-patient-info.code<br>    generate-notification.code<br>    push-notification.code</pre><p>Neal Ford describes orphaned classes and the Flatten Components Pattern:</p><blockquote>“Regardless of the direction of flattening, make sure source files reside only in a leaf node namespace or directories so that source code can always be identified within a specific component</blockquote><p>I like making refactoring like this because it’s also part of the risk/legacy code management — potential split into separate deployable units will be much simpler, and orphaned files tend to hurt their dependencies by violating of Liskov rule.</p><p><strong>Restrained ocean<br></strong>If my kid wants to play and sees too many toys, she ends up trying to play with every toy for a few seconds, so eventually, she plays with nothing and gets unfocused. If you’re a parent, you know that for sure 🙂</p><p><em>A restrained ocean means limiting an ocean of possibilities</em>. Instead of trying to grasp the whole underwater space and all the creatures living there, it’s better to refrain from absorbing everything to not overload our senses. I purposefully limit the possibilities of plays for my daughter so that she can enjoy her time instead of falling into chaos.</p><p>I try not to fall into the so-called analysis paralysis, expecting to come up with an ultimate complex solution at once. Instead, I take small steps and morph my current solution, splitting the problem into smaller ones like the divide-and-conquer algorithm.</p><p>A few examples:</p><ol><li>When a new project kicks off, I do not usually pick up too many new things (to the team or the company), technology-wise — like frameworks, 3rd parties, and cloud services. Most of the stack should be known to me and the developers.</li><li>When starting an implementation, I start with requirements and risks I’m sure will impact the solution immediately, leaning over cohesion and coupling; I give the implementation a chance to evolve later or be changed completely or even removed without the need for long and bloody surgery involving other components.<br>I.a., if stories or requirements are prioritised (like with <em>MoSCoW</em>) and I see <em>COULDs</em>, then it is a signal those have a much bigger chance to change in the future. They are only guidance for me on the direction of evolution, but they should not significantly influence the implementation/architecture of <em>MUSTs</em>.</li></ol><p>Let’s come up with an example:</p><ul><li><em>MUST</em>: if a patient bought ten or more blood tests in a current calendar year, then the next 5 should have a 5% discount if made in the next 6 months after receiving the discount</li><li><em>COULD</em>: if a patient has invited a new patient to the system, then the inviting patient should get a one-time 2% discount for the next blood test, and the discount cumulates the others</li></ul><p>The <em>COULD </em>gives me a feeling that different types of discounts may be applied cumulatively. I know it’s an important aspect of the business’s competition, so I’m preparing a slice that treats the known discount type separately. <em>With that, I can process other discounts later</em>, <em>either in a pipeline or independently</em>, if they can be calculated without context.<br>The result is a blood test price with applied discounts described by dedicated events. Thanks to that design, different discounts are not coupled together. An example implementation can look like this:</p><pre>type DiscountCommand = OrderBloodTest | CancelBloodTestOrde<br>type DiscountEvent = BloodTestsNumberDiscountApplied<br>type DiscountCounter = {<br>  state: NotReached | Active | Expired<br>  year: NaturalNumber<br>  lastBoughtAt: Date<br>  boughtTestsNumber: NaturalNumber<br>  _version: NaturalNumber<br>}<br><br>calculateTestDiscount(command: DiscountCommand, state: DiscountCounter): DiscountEvent {<br>  // implementation<br>}</pre><p>In conclusion, I move away from the allure of one-size-fits-all solutions and embrace a more nuanced approach to problem-solving. By combining well-known patterns with critical thinking and risk management, Thanks to, I can navigate the ever-changing landscape of technology with greater confidence and clarity.</p><p>Anyway, catch you later!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e698edfc868" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to successfully deliver an MVP to the market]]></title>
            <link>https://medium.com/@arturwojnar.dev/how-to-successfully-deliver-an-mvp-to-the-market-56c14bbe35dd?source=rss-84316362a96d------2</link>
            <guid isPermaLink="false">https://medium.com/p/56c14bbe35dd</guid>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[monolithic-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Artur Wojnar]]></dc:creator>
            <pubDate>Wed, 09 Aug 2023 18:39:51 GMT</pubDate>
            <atom:updated>2024-08-26T13:35:13.023Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OWSctyfHybRU8MxyEm75FQ.png" /></figure><p><strong>I’ve only worked in software houses, where we usually kick off startup projects</strong>. So we begin tiny, without guaranteeing that the market will appreciate our effort and the client’s plan. This situation also indirectly implies the non-functional requirement — the budget is not a bottomless pit, so we have to thoroughly plan our work to not sink all the client’s money into a product <strong>yet to be released.</strong></p><p>Here’re some of my thoughts on <em>how </em>we can help ourselves to successfully deliver the MVP on the market. But please, <strong>don’t treat bits of advice and experiences described here as something that will work for you</strong>. Read the first sentence of the very first paragraph again and consider if there’s a chance you are in this place too.</p><p><strong>First of all, I try to organize the process with my client for delivering pieces of the unfinished and partially mocked MVP</strong>, following the well-known and hated rule “<em>time-to-market</em>”, to let the client have presentations for <em>investors</em>, <em>potential customers</em>, and <em>future users</em>. <strong>This speeds up something inevitable — that something will change in the requirements</strong>, and there will be some adjustments <em>even though</em> in the design phase, we went through tests with users, and even though during workshops we did our best to tease every detail out of the client. <br>Our client has to agree with this approach and has to<strong> </strong>have the possibility to work that way. We can hint at this and politely explain that it’ll probably be the best for their business but the client has the final call, as always.</p><p><strong>We want to get to know as much as we can as fast as we can</strong>, thus delivering the product step by step is one of the ideas to go to save time and some portion of the budget.<strong> </strong>What other actions can be taken to <em>mitigate</em> the risk of the failure specter?</p><p>Although we’ve been struggling to improve the process of delivering the MVP making it gradual as it was planned and needed for the next demos for investors, <em>there are still dark clouds looming over our concerned heads</em> making us think whether it’ll end up with a big storm.</p><p>Staying in the area of processes and the business itself <strong>- </strong>sometimes the business side is people with ideas about what to do, with grants or investors.<strong> </strong>And surprise, surprise — <strong>they don’t play heads or tails with our work and our processes, with software development in general</strong>. We can politely and calmly explain and elaborate on everything. You know. <em>EVERYTHING</em>. Tell how sprints work, what Scrum celebrations are, why they are important, what an estimation means, etc…</p><p><strong>Alternatively, you can spur the business side on hiring someone technical, a CTO, who will take over the laborious job</strong> <strong>and will strike a balance between your team and the non-tech stakeholders</strong>, pushing away from you preposterous, fanciful, and tiring questions.<br>Also, having a CTO on the client&#39;s side puts him in a “<em>last-word</em>” decision-making position, which is good because there are fewer duties on you. I’m not talking about keeping your hands clean, but <strong>the CTO may be a person who will be on the same longitude and latitude as the client so that they skip problems with remote communication</strong>. Again, that should be the person who has some technical knowledge but is also business-empathetic and business-involved. They participate in conferences and all sorts of business meetings. The CTO should represent the technical aspect of the client’s needs.</p><p><strong>Of course, this deal may sometimes be frustrating</strong>, <strong>you never know who you will be working with</strong>. What the background of this person is, maybe that person is short-fused, always bummed up, or does everything down to the wire, or is newly baked in the CTO role and will tear into it with the greatest passion trying to splurge. <em>Let’s not jinx it!</em> The possible downsides do not outweigh the advantages.</p><p><strong>So now, we’re getting to the most tricky advice I would’ve given myself a few years ago. It’s a character trait — being humble</strong>.</p><p>That person on the other end of the call that <em>pixel-ish face</em> on a video conference is also a part of the team, and with high confidence, this person also wants the best for the project. They might be <em>annoying sometimes</em>, and you will disagree with them, but that’s part of the process. Respect their opinion. You don’t know everything, you haven’t been everywhere and you haven’t seen everything.</p><p>And remember that they make a final call, even if you dislike that from your developer’s perspective. Take a breath and think — maybe that can eventually turn out better, indeed?</p><p>It’s difficult to find balance here, because <strong>a</strong> <strong>CTO may tend to micromanagement</strong>. Our aim is to not let that happen and nip it in the bud because it will make the whole team demotivated and <em>ultimately</em> the team’s pent-up feelings will surely blow up.</p><p>Our aim is to discuss ideas thoroughly with the CTO, gently push them toward a good solution if that’s really the best one and <strong>take responsibility for that</strong>. CTOs have to trust you and the team and you have to gain their trust by making good decisions and being professional.</p><p><strong>About the obvious — when you got to the development process it means that before you had passed through workshops with the client.</strong></p><p><strong>You know what the <em>core domain is</em></strong>. By “you” I understand the contractor (<em>development team</em>) and the principal (<em>client</em>). Even when the client is convinced that they planned and know everything, because they have tons of documents and wireframes, you must go through a few days’ workshops to visualize processes and embrace the scope of the MVP.</p><p><strong>Speaking of which, one of the goals except a common understanding of the product is defining what features the MVP includes</strong><em>. </em>Only then the development team will be able to provide reliable estimation. <strong>There are a bunch of techniques you can apply to make the workshops more structural</strong>: <strong><em>User Story Mapping</em>, <em>Example Mapping</em>, and <em>Event Storming</em></strong>. But more important than any of the techniques is to talk to each other and to analyze product functionalities.</p><p><strong>The important thing about every workshop is to define a single goal</strong> that is understood by all stakeholders and to work solely on it. Thanks to that you can avoid <em>context switching</em> and maximize chances to get more detailed information and spot more intricacies rather than having a superficial view on a few problems.</p><p><strong>One thing that is beside me and always results in something good — tests with users</strong> performed by the UX team show whether planned features make sense or how they can be improved. This can be done earlier by the client, but not necessarily, and not always it’s done thoroughly. So, <em>limited trust here.</em></p><p><strong>Prepare the first version of the architecture.</strong></p><p><em>How are you going to deploy it? What aggregates do you encounter? What modules do you see? What other non-functional requirements influence the architecture? Maybe part of the system will be certificated as a medical device? Maybe you want to store and process users’ data only in one module and in a separate database? Maybe the business is somewhere OK with eventual consistency? Maybe the business side expects the history of changes to be kept somewhere. Maybe you will be consuming telemetry? What database(s) will fit your case at a given moment?</em></p><p><strong>Before you answer any of the above questions, remember to focus on the functionalities and the domain instead of on technology and infrastructure.</strong></p><p><strong>Before you pull toys out of the sleeve</strong>, <strong>like queues, event sourcing, CQRS, NoSQL databases, and new frameworks</strong>, <strong>think about the members of the team and their skills.</strong> My <em>simple</em> rule is that I don’t introduce more than one new thing at a time to not bog down the development process and flatten the learning curve. For me a new thing is something that no one on the team knows, if there’s at least one person (but me) who knows this, then I’ll not treat this as something new.</p><p><strong>My current aim is to work on a chassis repository for bootstrapping new projects</strong>.</p><p>I believe that this kind of standard in a software house helps to organize knowledge and speeds up project bootstrapping. Aligning knowledge contained within that repository is also a matter of internal training.</p><p><strong>Make sure that your team understands architectural assumptions and the decisions behind them</strong>. Make sure they know the importance of the processes (e.g. API first) and the documentation. Confront them all with that, discuss it in detail, and in the end help them adapt to the changes.</p><p><strong>Set a cyclic meeting, e.g. once a week to take control of the evolution process of the architecture and to address doubts that have been piling up. </strong>Especially at the beginning (the first few months), when developers get to know each other, the new project, and potential new rules and guidelines. That’s why, as mentioned above, I believe that these rules and guidelines in a software house company shouldn’t deviate much between teams.</p><p>Work on documentation and track your decisions, you can e.g. follow one of many ADR (Architecture Decision Record) templates. I use a GIT repository for storing it to push those documents through a regular code review process. The decision records should be short, spikes may produce longer analysis documents.</p><p>You can accuse me of inconsistency here because developing cumbersome documentation is the opposite of cutting edges where possible. That’s right, <strong>but <em>without </em>documentation, we can’t really follow the project in the post-MVP phase</strong>. It’s not possible to write accurately the docs that should have been settled a few months ago. No one would remember the exact motivations behind decisions.</p><p><strong>Another way I find effective is prototyping using cloud-native solutions</strong>. <br>Did you know that to make an on-prem Kubernetes (as of the moment of writing — version 1.7.1) secure and production-ready you need to go through over<em> 300 pages</em> of <a href="https://learn.cisecurity.org/benchmarks">CIS report</a>? Did you know that to do the same but with a cloud-managed cluster you have to go through over <em>170 pages</em>?</p><p>My recent EKS implementation included the following controllers/daemon sets: <em>CoreDNS</em>, <em>Metrics Server</em>, <em>Helm</em>, <em>Flux</em>,<em> Sealed Secrets</em>, <em>AWS Distro for OpenTelemetry Collector</em>, <em>Fluent Bit</em><strong>.</strong> Everybody may feel dizzy with all of these integrations and possible bugs caused by <em>a version mismatch</em>. Additionally, having yet another plugin can mean having another container running.</p><p><strong>It’s a lot of maintenance, a lot of configurations, and a lot of spots and intersections where something may go wrong</strong>. So, if you don’t want to occupy your busy developers with DevOps or, <em>Kyrie eleison</em> (!) if you don’t have much of this skill, then<em> pack all the toys back into a bag and focus on what’s really important — functionalities, not the infrastructure</em>.</p><p><strong>Another issue regarding the K8s that I’d like to tackle is the vendor lock. <em>THE </em>mythical vendor lock</strong>. If we use a managed version of the orchestrator, then — whatever the efforts — we’ll bump into the vendor lock as well. Examples: logs harvesting to send them to the cloud, distributed cache on a cloud bucket, sending metrics to the cloud, mounting cloud storage with a <em>FUSE adapter</em>, <em>cloud load balancer</em>, and <em>cloud OIDC provider</em>.</p><p>As one of my teammates once said —<em> coupling to the cloud for a managed K8s cluster is way more subtle and implicit than using a cloud-native solution directly</em>.</p><p>Of course, in most of these scenarios, you can substitute cloud solutions with more generic <em>SaaS</em> and Kubernetes-hosted services (like, for example getting rid of the cloud load balancer and having <em>Nginx</em> or <em>Traefix</em> instead). But that means more tooling and integrations on your side. So, the question is — is it worth it?</p><p><strong>In my life, I migrated a managed Kubernetes cluster from AWS to Azure. And I have to admit there’s a lot of not obvious caveats and intricacies</strong>. <strong>Hence, my thumb rule and decision heuristic is to start with a cloud-native service</strong> like <em>AWS Elastic Container Services</em> or <em>Lambdas</em>.</p><p>It’s simple, observability is better, it’s much less responsibility on the team side and it’s also less error-prone. <em>I find it quicker and more cost-effective to migrate the cloud-native solution to another cloud provider or to a non-cloud-native approach</em> (like migrating from Lambdas or ECS to a K8s or EC2s). Moreover, the cloud providers give you a set of tools that are perfectly compatible with each other. Can you say the same about Kubernetes plug-ins?</p><p><strong>If you work on Docker images, then it seems that what and how runs them doesn’t make much difference</strong>. Making the K8s up and running, secure and usable is cumbersome and put a significant burden on your shoulders.</p><p>A real big-scale scenario was migrating one of Prime Video services from AWS Step Functions and Lambdas to ECS and EC2s. Engineers did it because they relied on virtualization, a good plan and that was just another step in their architecture evolution. I will come back to it later.</p><p><strong>It’s staggering for me that microservices are one of the most controversial topics. </strong>When this defiant buzzword is thrown in space you, hear mumbling, someone sounds off about what they think. Appetites for the microservices have been largely undimmed, they seem to be beckoning, but people often forget that this is just another screwdriver, another tool.</p><p>Let’s refer again to the Prime Video case. The noise around that proved to me once again that I shouldn’t trust most of the blog posts on the Internet even if the author is not an anonymous developer (like me, your Favorite Modest Author). A prominent example is <a href="https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-serverless-or-microservices-59625580">this post</a> (sic!).</p><p>Developers dig their heels in and refuse to read with understanding.</p><p><em>The Video Quality Analysis</em>, one along many Amazon’s teams, motivated its initial choice as follows:</p><blockquote><em>“We designed our initial solution as a distributed system using serverless components saw (for example, AWS Step Functions or AWS Lambda), which was a </em><strong><em>good choice for building the service quickly</em></strong><em>.</em></blockquote><p><em>The link to the original Amazon post you can find </em><a href="https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90"><em>here</em></a>.</p><p>So, it was handy and cost-effective for them to bootstrap their app with Step Functions, a few Lambdas, Media Conversion Service, and S3. They continue:</p><blockquote><strong><em>“We realized that distributed approach wasn’t bringing a lot of benefits</em></strong><em> in our specific use case, so we packed all of the components into a single process.</em></blockquote><p>They explicitly pointed out it’s about their specific use case.</p><p>And the last interesting thing in the context of my text:</p><blockquote><em>“Conceptually, the high-level architecture remained the same. </em><strong><em>We still have exactly the same components as we had in the initial design</em></strong><em>.</em></blockquote><p>In this case, they, among others, hit the AWS region’s limits which limited their scaling capabilities. Also, having distributed architecture forced them to use S3 which has been replaced by in-memory storage in one-process monolith service.</p><p>The takeaway here is that <em>in this particular case</em> due to arbitrary region limits and the racked-up costs (high demand), they found the single process on EC2s and ECS a much better fit because everything is close to each other so there’s no latency for infrastructure communication.</p><p><strong>It was the evolution process</strong>. They kicked off with Lambdas, Media Converter, and the distributed system because they were focused solely on functionalities and testing their product. <em>And the product worked out, it passed the market test.</em></p><p>The Prime Video’s case showed also, as I pointed out a few paragraphs earlier, that it wasn’t a barrier not to overcome to switch from Lambdas to a more conservative and traditional solution — the codebase within the vendor-locked solution has been mostly reused.</p><p>The same applies (and even more) to the ECS, which the Prime Video team thinks of as a more traditional tool because it doesn’t force you to use microservices as it happens naturally with Lambdas. The ECS is yet another orchestrator where you have to deliver a containerized application, so what’s the problem with migrating it to on-prem infrastructure later when the software is more mature and needs are different?</p><p><strong>Honestly, I’ve never started with microservices architecture and that’s my other thumb rule not to do so as I want to bypass overcomplicated architectures consuming precious time which is running out.</strong> The architecture has to serve the business goal.<br>But I have to highlight one thing that I believe is widely misunderstood; The tautology is <em>microservices =&gt; distributed system</em>. It means that if you have microservice architecture it implies you also have a distributed system but not the other way around. Having a distributed system doesn’t mean your architecture can be called microservices.</p><p><strong>The microservices are fine-grained</strong>. The perfect examples are AWS Lambdas and <em>AWS Step Functions </em>implementing the <em>Saga pattern</em>. That toolset forces fine-grained services with a single responsibility. Of course, you can still create a distributed big ball of mud, reference all functions to a single database, make all Lambdas hungry, and kill off yourself with overcomplicated communication.<strong> No tool or technology saves you from doing stupid things</strong>. You can grab a hammer and try to bump a bicycle tire, why not?</p><p><strong>Service-oriented architecture and monoliths are coarse-grained services.</strong> They are perfect to learn about your business boundaries, bounded contexts, rules, constraints, and aggregates. It’s easier to split a bigger service than merge an existing service into one when you discover that something went wrong. That thing is comprehensively described in <strong><em>Neal Ford’s Software Architecture: The Hard Parts</em>.</strong></p><p>If I learn that, for instance, I have to implement data streaming of health monitoring devices, then it’s likely I’ll dedicate a separate service to that to take the burden of real-time data processing. If I learn that the important non-functional requirement is the security of patients’ records, then maybe I will cut off this piece of the cake and put it in a separate closed box with dedicated security measures and layers that other parts of the system don’t need. If I learn that there’s also a development team on the client’s side, then maybe we’ll agree to work on separate monoliths (services). But it doesn’t mean that suddenly I work with microservices architecture. It means that I analyze the requirements and <strong>I try to do my best to deliver the value whilst having the option to evolve what is currently implemented into something more advanced and focused in the future</strong>.</p><p><strong>These days everything is a distributed system, even monoliths. </strong>You have your backend, you have a Keycloak instance (with its database), and you have of course your main database. So, already you have a few moving parts. You can’t span a change over both databases with a single transaction. Also, probably, you have more than that. You have a notification service, static storage, etc. <strong>Worth noticing is also that microservices are monoliths too</strong>. Right? But distributed. They all serve the common purpose of implementing business flows. If you shut down one of the services will the whole system still be delivering the full value?</p><p><strong>My point is that a modular monolith is not a constant struggle to have everything contained in a single process</strong>. If I have a modular monolith with that one split service for the data streaming I still call the whole thing a modular monolith because I’m still learning along with my client about the product’s domain and I don’t see a reason to overload me and the team with hundreds of problems that pop up with microservices.</p><p>The microservices are not an end in itself — the end in itself is to follow your domain and organization requirements and adapt the architecture towards them to achieve their goals to let your client successfully sell their application.</p><p><strong>Wrapping up the above paragraphs — I start my projects as modular monoliths because there’s still a lot to learn about the domain and that includes me, the team, and as well as the clients themselves.</strong></p><p>The software development and the architecture for me is risk management.<strong> This is the mantra I follow. This is also a paraphrase of the idiomatic “it depends”.</strong></p><p>I have to gather functional and non-functional requirements that have been determined for the time being and know who I will be working with. Mix it all up, assess a hazard, and propose some solutions. <strong>There’s no single effective recipe on how to successfully deliver an MVP to the world</strong>, only some heuristics I learned after a lot of smaller or bigger failures. Heuristics I’ve just shared with you, Dear Reader. And again, <em>don’t </em>treat them as tools that will help you, but rather think of them as inspiration and an incentive to confront your own views and your experience.</p><p><strong>There’s also an area, I imagine pretty big, of things that I haven’t tried and which can serve the purpose.</strong> One I can think of is feature toggle and Canary deployment. It’s something that so far I haven’t had a chance to work with and have an opinion about.</p><p>If I stand by my resolution, then I will thoroughly describe how I go through the design and decision-making process up to the implementation phase based on a real-world scenario.</p><p>Artur</p><p><strong>Bibliography</strong></p><ul><li><a href="https://www.amazon.com/Strategic-Monoliths-Microservices-Addison-Wesley-Signature/dp/0137355467">Strategic Monoliths and Microservices: Driving Innovation Using Purposeful Architecture</a></li><li><a href="https://www.amazon.com/Serverless-Architectures-AWS-Peter-Sbarski/dp/1617295426">Serverless Architectures on AWS, Second Edition</a></li><li><a href="https://www.amazon.com/Building-Evolutionary-Architectures-Automated-Governance/dp/1492097543">Building Evolutionary Architectures: Automated Software Governance</a></li><li><a href="https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90">https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56c14bbe35dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HL7 FHIR: Foundations in the architectural context]]></title>
            <link>https://medium.com/@arturwojnar.dev/hl7-fhir-foundations-in-the-architectural-context-a4a676c0aeab?source=rss-84316362a96d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4a676c0aeab</guid>
            <category><![CDATA[architecture-design]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[hl7]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[fhir]]></category>
            <dc:creator><![CDATA[Artur Wojnar]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 00:00:37 GMT</pubDate>
            <atom:updated>2024-08-26T14:10:28.869Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*duZOIwvrUq1K3AVJduhMWA.png" /></figure><p><strong>The Fast Healthcare Interoperability Resources (HL7 FHIR) standard describes how computer systems can exchange healthcare information. Seeing how already some 30% of the world’s data is generated by the healthcare industry — and a lot of it is highly sensitive — it’s no wonder this standard came into existence. Let me try to explain why it’s important and how it works from a software developer’s perspective.</strong></p><h3>The common language for healthcare data exchange</h3><p><strong>Communication is everything</strong>. Would you imagine our world without the possibility of communicating with each other? Definitely, we wouldn’t be in the same place we are now, and it’s enough to say that we, as humankind, are still working on improving this skill (we’ve even written a whole article about the importance of <a href="https://revolve.healthcare/blog/communication-bioinformaticians-engineers-biotech-software-development/">good communication between software engineers and bioinformaticians</a>).</p><p>We know how to use languages because they all are structured systems. Processors know how to execute a set of instructions that varies depending on architecture. Moving on to the world of high-level software development, there’s, for instance, the Auth0 protocol, which is about passing users’ identities (mostly) from a provider to other systems. We also have well-documented REST APIs that — for documentary purposes — may use OpenAPI specification, and thanks to that, the API consumers know how to authenticate to and communicate with it.</p><p>It makes sense that there’s also a standard in the healthcare world for information exchange. This standard is HL7 FHIR — pronounced “fire” — and it recently (2019) got a big update. It’s now way more mature and easier to implement than it used to be. And it’s a good thing it exists because we’re all much more concerned about everything that has to do with healthcare these days.</p><p><strong>HL7 FHIR was created by Health Level Seven International (HL7), a healthcare standards organisation. Its key point is interoperability, which facilitates the exchange of patient data</strong> (EPR—Electronic Patient Record).</p><p>The FHIR standard consists of a lot of resources grouped into five levels</p><ol><li><strong>Level 1 — Foundation.</strong> This level includes resources like the Questionnaire, its Responses, or a structure that embraces other resources (List).</li><li><strong>Level 2: supporting implementation and bindings to external specifications.</strong> While using FHIR resources to increase and improve interoperability, we can reference (with the help of CodeSystem resource)values to other specifications that will provide a meaningful description of a value, e.g.pointing out that we’re dealing with kilograms or meters. It’s not only about SI (<em>International System of Units</em>) symbols, though. We can describe our data as a disease, a mental state, a heart rate or a question about a person’s stress level. Those definitions can be found in terminology systems like SNOMED or LOINC. Moreover, you can define your public system. Also, this level represents the Exchange module that answers how others, especially healthcare applications, can communicate with your system.</li><li><strong>Level 3: linking to real-world concepts in the healthcare system.</strong> It provides resources such as <em>Patient </em>or <em>Organisation</em>.</li><li><strong>Levels 4 and 5 provide even more varied resources to facilitate record-keeping and data exchange</strong> for the healthcare processes.</li></ol><h3>Resource representation</h3><p>Everything in FHIR can be expressed as JSON, XML, or RDF.</p><p>You can find an example of a questionnaire <a href="https://www.hl7.org/fhir/questionnaire-example.json.html">here</a>. It’s worth noticing that the whole document is self-descriptive. For instance, item 1.1.1.1 defines an answerValueSet, referring to an <a href="http://hl7.org/fhir/ValueSet/yesnodontknow">external terminology system</a> that specifies three possible answers (yes / no /don’t know). Another question is described by one of the SNOMED database values pointing to an angina disorder. Some questions are conditional (property enableWhen), and some are grouped or nested.</p><p>It’s also possible to express an intention of how a set of extensions should display the question, e.g. we can define that we want to have radio buttons in a vertical orientation, like so:</p><pre>&quot;extension&quot;: [ { &quot;url&quot;: &quot;http://hl7.org/fhir/StructureDefinition/questionnaire-itemControl&quot; &quot;valueCodeableConcept&quot;: { &quot;coding&quot;: [ { &quot;system&quot;: &quot;http://hl7.org/fhir/questionnaire-item-control&quot; &quot;code&quot;: &quot;radio-button&quot; &quot;display&quot;: &quot;Radio Button&quot; } ], &quot;text&quot;: &quot;Radio Button&quot; } }, { &quot;url&quot;: &quot;http://hl7.org/fhir/StructureDefinition/questionnaire-choiceOrientation&quot; &quot;valueCode&quot;: &quot;vertical&quot; } ]</pre><p><strong>It’s demanding to describe properly and in detail how our system thinks about its own data or</strong> (as in the example above) <strong>what its expectations regarding data acquisition are.</strong> But it’s <strong>worth the effort</strong> because other parties can interpret our data more accurately and, potentially, link them with their own.</p><h3>Cloud implementation</h3><p>The FHIR specification documents the RESTful API and search engine that a compliant system should follow and implement to be understood by other healthcare systems (FHIR-based).</p><p>Luckily, <strong>all big cloud providers support the FHIR standard in one way or another</strong>. Let’s take a look at the AWS approach. AWS is the thing. We can trust their work. Let’s check the <a href="https://aws.amazon.com/solutions/implementations">AWS Solutions Implementations</a>, a fully-fledged solution for <a href="https://aws.amazon.com/solutions/implementations/fhir-works-on-aws/?did=sl_card&amp;trk=sl_card">FHIR-based</a> systems as a CloudFormation template with an exhaustive overview of it.</p><p>It’s worth noticing that the architecture of such a solution is not trivial. In the case of AWS, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*3DByfptMf-JnT0OF" /></figure><p>The solution stores all unstructured FHIR resources on AWS S3 and all structured ones in DynamoDB-from where they are redundantly streamed to the Elasticsearch engine.</p><p>Another thing is how the solution validates the correctness of FHIR documents that are sent to the RESTful API — a very important step in the system since it’s one of the things that ensure data integrity. AWS engineers have decided to use an open-source 3rd-party implementation of the FHIR specification created by the Java community — <a href="https://hapifhir.io/">HAPI FHIR</a>, which is the product of <a href="https://www.smilecdr.com/">Smile CDR</a>. In my opinion, it’s also the most reliable tool we can use to validate (FHIR) resources, and it is recommended even in the FHIR docs. Other tools (mostly NPM packages) that I tried aren’t trustworthy — their results may not be correct regarding the FHIR specification.</p><p>See <a href="https://github.com/awslabs/fhir-works-on-aws-deployment">the AWS GitHub repository</a> for this solution.</p><p>There’s a <a href="https://aws.amazon.com/about-aws/whats-new/2021/07/amazon-healthlake-generally-available/">new</a> fully managed HIPAA-compliant FHIR server on AWS: <a href="https://aws.amazon.com/healthlake/">Healthlake</a>. It is only one of the FHIR server implementations, but there are plenty of them. <a href="https://darrendevitt.com/13-fhir-server-providers/">Check </a><a href="https://www.linkedin.com/in/darrendevitt/">Darren Devitt&#39;s list,</a> which will give you a glimpse of how much is going on in the FHIR world.</p><h3>Partial support</h3><p><strong>If we want to fully comply with FHIR, we must use a solution like the one described above</strong>. FHIR resources are cumbersome and arbitrary, and—most likely—they don’t reflect your whole system’s domain—it’s a data projection. The data represents the system’s domain from the perspective of healthcare’s common language, but it doesn’t model your processes or business requirements or constitute a business’s competitive value.</p><p>Actually, <strong>the FHIR API is just a complicated CRUD system</strong>. We can say that making software able to exchange FHIR information isn’t the goal in itself; rather, it’s a matter of having a generic domain in terms of Eric Evans’ <a href="https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215/ref=asc_df_0321125215/?tag=plshogostdde-21&amp;linkCode=df0&amp;hvadid=504239934199&amp;hvpos=&amp;hvnetw=g&amp;hvrand=16957646779703600099&amp;hvpone=&amp;hvptwo=&amp;hvqmt=&amp;hvdev=c&amp;hvdvcmdl=&amp;hvlocint=&amp;hvlocphy=1011516&amp;hvtargid=pla-449269547899&amp;psc=1">Domain-Driven Design</a>. Thus, this part of the architecture will probably be separate and redundant from the main application, and the data that goes there will eventually be consistent (consistent between the FHIR system working alongside the main one and receiving data from it).</p><p>But is it the only way? <strong>Sometimes, the requirement is to have data that can be exported</strong> in a common predictive format like FHIR. That data can be patient records, observations (laboratory data or vital signs), or questionnaire responses (e.g. stress level monitoring).</p><p><strong>On the other hand, the requirement can be to import some data</strong> in a well-known format like FHIR. Imagine a situation where your system processes patients’ observations—temperature, pulse, weight, and blood pressure—to determine the risk of developing disease X. Your input comes from several clinics and hospitals, all of which are FHIR-compliant.</p><p><strong>In both cases </strong>— exporting and importing of data — <strong>your application has to adapt to the common language</strong> and use a pattern like <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer">Anti-corruption Layer</a>or <a href="http://ddd.fed.wiki.org/view/open-host-service">Open-Host Pattern</a> <strong>but may not necessarily need a full-fledged FHIR API.</strong></p><p>FHIR is a step in a good direction, trying to embrace healthcare language and its requirements while aiming for interoperability with the great support of cloud providers. But as always, there are two sides to the coin.</p><p>At the time of writing this article, the standard&#39;s current version is R4 (4.0.1), and version R5 is being developed (4.6.0). <strong>With every new version, some relying parties will lose compatibility with it</strong> and thus with the systems implementing it.</p><p>Besides, the FHIR specification is just a text block, and <strong>specific implementations can vary. What can help is relying on well-known external implementations</strong>, like HAPI (which I mentioned before) or SMART Health IT, the <a href="https://smarthealthit.org/about-smart-2/">goal of which</a> is that “an innovative app developer can write an app once, and expect that it will run anywhere in the healthcare system”.</p><p>Some differences that serve particular business purposes can also be a problem. <strong>Sometimes the interoperability isn’t necessary business-wise</strong>, as <a href="https://searchhealthit.techtarget.com/news/252453591/Why-the-FHIR-standard-may-not-get-us-to-interoperability">pointed out</a> by Joseph Kvedar, M.D., the vice president of connected health at Partners HealthCare in Boston:</p><blockquote>I work for a healthcare provider — it’s in our best interest to keep your data from going to a provider down the street. It’s in Epic’s best interest to keep your data unique so you don’t switch to Cerner. Until we see the quality of care suffer massively or see revenues drop because patients are not coming to us because of interoperability, it’s nice to have.</blockquote><p><em>Joseph Kvedar, M.D.</em></p><p><em>Originally published at </em><a href="https://revolve.healthcare/blog/hl7-fhir-foundations-in-the-architectural-context"><em>https://revolve.healthcare</em></a><em> on July 20, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4a676c0aeab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why make a decomposition of the monolith database?]]></title>
            <link>https://medium.com/@arturwojnar.dev/why-make-a-decomposition-of-the-monolith-database-f91aea41af6c?source=rss-84316362a96d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f91aea41af6c</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <dc:creator><![CDATA[Artur Wojnar]]></dc:creator>
            <pubDate>Sun, 12 Mar 2023 23:02:39 GMT</pubDate>
            <atom:updated>2024-08-26T13:13:58.902Z</atom:updated>
            <content:encoded><![CDATA[<h4>Summary and review of chapter 6 of <em>Software Architecture: The Hard Parts </em>by Neal Ford</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3yu28kDetoXrT_cn8aAsJA.png" /><figcaption>Figure 0. A typical emotion when you look at your database schema.</figcaption></figure><h3>Introduction 1</h3><p>Neal Ford is great. Erudite and intelligent speaker, receptive and perceptive. Highly experienced Software Architect, Solutions Architect, or whatever you’d like to call his range of responsibilities (these positions are overloaded in meaning, but that’s another topic!). With his books, he can hold your attention and make you feel that you are spending your precious time in a very satisfying way. And that’s a thing! Right? I bet, you can list many books that were OK, full of excellent insights but written in a way you had problems not falling asleep over the next page (yeah, I’m looking at the “biblical” <em>Blue Book</em>).<br>Ford’s books are a different piece of writing. Let’s dive into the pages of this chapter, taking what’s best there and extending contained knowledge with my humble insights and experience.</p><h3>Introduction 2</h3><p>Have you ever gotten stuck with your task because of a change introduced by another developer in a completely separate service? How did that happen? Oh, of course! It’s because of our integration pattern based on the <strong>database contract¹</strong>. We extracted part of our application responsible for communication with pharmacy stores and gathering data about scanned and bought medicines, separating it from the patient-facing application. Thanks to this we could meet the requirements of the <a href="https://en.wikipedia.org/wiki/Service-level_agreement"><em>SLA</em></a><em> </em>of our system again. By doing so, we migrated to the Service Oriented Architecture, meaning that we split up the monolith application into two services with a shared database. This was still the monolith application — Here I can reference Neal Ford’s book again and the idea of <strong>architectural quantum²</strong>. Well, this concept has been described in his two other books, including <em>The Hard Parts</em>. But going back to the topic — <em>architectural quantum</em> says that if two services e.g. share the same database, they create one component, one deployable unit with high coupling, and we cannot treat them independently. This concept is wider — Ford mentions synchronous connascence; If two services share a resource or communicate with each other in a synchronous way, then they become <em>one quantum</em>.</p><p>Chapter 6 is about that — we start from having an <em>SOA</em> application (coarse-grained services) and we want to try to break apart its database so we can get rid of their common static coupling (the shared database).</p><p>The original chapter title is <em>Pulling Aparat Operational Data.</em></p><h3>Why split data?</h3><p>Reasons why you should consider the effort of disintegrating data are drivers and non-functional requirements you can come up with after giving them some thought:</p><ul><li><em>Loose coupling</em>. Ford has split it into two separate disintegrators:<br>- The number of services impacted by a database table change<br>- Prevention of consolidating services into one architectural quantum</li><li><em>Scalability</em></li><li><em>The performance</em> described in the book as <em>connection management</em></li><li><em>Fault tolerance</em> &amp; <em>Security</em></li><li><em>Database type optimization</em></li></ul><h3>Loose coupling</h3><p><strong>We used to break </strong>apart application functionalities into components, create abstractions for them, and use design patterns to model them in a more understandable and unified way. It <strong>often follows</strong> with treating databases only as bags for data — we add entries, some indexes or not, thinking that the database will deal with everything (in many cases that will be the truth, relational databases forgive a lot). We may add some <em>foreign keys</em> between two columns because we can. But at some point, we may encounter popping up regression and increased size of estimations for changes planned for the application.</p><p>A team, trying to cut down on costs and working on the <em>MVP</em>, could agree, not to handle multiple databases, but it’s not the case anymore. It became a real problem, involving more and more of the client’s budget.<br>Introducing a breaking change to the database like</p><ul><li>changing the name of a column or property</li><li>removing a column or property</li><li>dropping a table or collection</li></ul><p>may entail the necessity of refactoring other parts of the system depending on these data. Later, the team has to deploy several services and go through test protocols for these services. That results in increased estimations. On the other hand, if we forgot about one of the dependent services then we would face a regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtB9n2_Cqk9Z0lKCH6L3NA.jpeg" /><figcaption>Figure 1. Services A, B, and C depend on one database.</figcaption></figure><p>A change in one of the services may impact or even force the change in other services.</p><p>Let’s imagine that <em>Service A</em> is responsible for handling nurses, <em>Service B</em> for patients, and <em>Service C</em> for patients’ therapeutical indications and notes provided by nurses. Let’s also assume that this division is the best at the very moment and it’s the result of many analyses. It turns out that, despite having separate tables for nurses and patients, there’s also a table <em>Users</em>, because the two entities have a lot in common: e.g. <em>forename</em>, <em>surname</em>, <em>phone number</em>, <em>email address</em>, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-23aHnggwYmMYGJmOeCPgw.jpeg" /><figcaption>Figure 2. Generalization results in one architectural quantum.</figcaption></figure><p>As a result,<em> team A</em> and <em>team B</em> or <em>C</em> can affect each other while having separate domain models and seemingly the corresponding tables. That’s a bad thing because the desire was to make the teams much more independent and the communication between them should only occur on the API contract level. So, answering the author’s question — “<em>how many services are impacted by a database table change?</em>” — the answer is 2 and we can’t think of those services as independent.</p><p>Unfortunately, the services still consolidate into one <em>architectural quantum</em> by sharing the same static coupling that is the shared database. That may be OK if the teams understand the consequences of it.<br>Going further, if at some point the team decided to separate databases e.g. to ramp up fault tolerance, then the <em>Users </em>table would have to be either merged with the <em>Nurses </em>and <em>Patients </em>tables or duplicated in the two separate databases.</p><h3>Scalability and Performance</h3><p>I decided to join the <em>Connection Management</em> (<em>Performance</em>) and the <em>Scalability </em>factors since one is a consequence of another, but the first one is less obvious and a way more precise.</p><p>For now, our project, as mentioned earlier, relies on an <em>SOA </em>architecture with a shared database, which means we have N services that create connection pools to the very same database. One open connection can handle <a href="https://www.postgresql.org/message-id/4A8B56E2.6050704@postnewspapers.com.au">one query at a time</a>, so depending on the configuration the connection pool can consist of 5, 10, or 100 connections, depending on the expected load. That may not be a lot jointly, but each service can be run multiple times so that instances are capable of carrying increased traffic by (surprise, surprise!) scaling out. Those abrupt picks in load can quickly saturate database connections which will result in some clients waiting and waiting, seconds by seconds, minutes (?!) by minutes! And they say that 10 seconds is enough <a href="https://www.nngroup.com/articles/response-times-3-important-limits/">to make our users feel impatient</a>. I can say only for me, but I personally get annoyed faster.</p><p>Taking a simple example. If we deployed 20 services, each having 15 connections in the connection pool, and by average we always run at least 2 instances of every service, then by simple math we end up with <em>20 * 15 * 2 = 600</em>. Still, during Black Friday it’s not 2 on average but 5, which gives 1500!</p><p>The result is that we scaled out the services but it didn’t help because queries can get stuck along with system users lining up in a very long queue. Let’s remember also, that we are limited by the <em>max_connections</em> setting which varies on an OS, but by default is 100, which means that any new connection after reaching the maximum will <a href="https://www.postgresql.org/docs/current/runtime-config-connection.html">result in a failure</a>.</p><h3>Fault tolerance &amp; Security</h3><p>We have N services smiling at us on the production cluster; Teams can work on separate service source codes and deploy them separately (unless you link them by your data — see <a href="#d599">Loose coupling</a>). It’s Sunday morning when, due to maintenance work scheduled by a vendor, your database crashes. It may be because of something — let’s say it’s a matter of deprecated configuration of one of the plugins you use, the plugin that is required only by exactly one of your services (<em>Service C</em>). Unfortunately, that scales onto the replica sets, too. As the result, your entire system is down. If we had data split up for <em>Service C</em>, kept separately in a dedicated database, the unlucky maintenance upgrade wouldn’t cause this big-scale damage, leading to the frustration of customers and your boss. Then, only one service would be acting up, not all, so only a subset of functionalities would be unreachable as depicted in <em>Figure 3 </em>(assuming that the responsibilities of the services are split wisely and the services don’t need each other all time to provide functionalities to end users).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVF-wnkfDw9zZNuoaTKg9w.jpeg" /><figcaption>Figure 3. An outage in database C impacts only Service C.</figcaption></figure><p>Similarly, what can happen in the case of a shared database breakdown can also apply to security concerns which is one of the aspects I missed in the book.</p><p>If our database gets compromised, then <strong>all data get compromised</strong> off hand, including sensitive and private data of our patients and their medical records. What a mess! If our concern is security and this is one of the important non-functional requirements, then knowing which data are the most valuable, we may decide to move them to a separate database, although it’s not necessary from the domain perspective.<br>Keeping those data separate also may not be sufficient, because our whole network could be hijacked. <strong>The best way to mitigate this possibility is to keep your security-sensitive database in a separate <em>Virtual Private Cloud⁴</em></strong>; Then you are guaranteed that traffic is enclosed within each VPC and there’s no room for misconfigured firewalls (security groups, when talking in the context of AWS). You can add an explicit VPC Peering connection between two networks to keep traffic private.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hCEBFHbPw3YVKl78efUt2A.jpeg" /><figcaption>Figure 4. Two networks. Each is independent with separate addresses pool.</figcaption></figure><h3>Database type optimization</h3><p>The most popular database type is a relational database. It’s also the most generic one giving you the opportunity to model almost everything you can think of — of course with an appropriate level of complexity. Relational databases have been developed for over 50 years, which is plenty of time to optimize a product in a very sophisticated way. It implies that those databases “forgive” developers’ many mistakes.</p><p>Because relational databases have been with us for so long, it’s obvious that they have been studied and taught in schools and academies for decades, and students (me included!) have been analyzing which <em>normal form</em> a database follows, what isolation level is desired, whether it’s possible to drive a database into a deadlock? What is <em>ACID</em>? And so on…</p><p>But the world and <em>IT </em>made lots of steps forward, our community got clouds, and various SaaS products solving well-known problems like authentication, or video streaming; Requirements and needs had gotten more and more advanced, so eventually, we got on-demand support for full-text search, IoT, data warehouses, data lakes, AI platforms. There is a rule in the economy saying that the more complex the requirements become, the more complex software becomes and the more complex the software becomes, the more complex the requirements get. Some applications which used to take long months or years to develop, today may take a few days or weeks.</p><p>Among others, this is why we noticed something we can call an upsurge in the database market resulting in the creation of many problem-specific databases. I’m saying that we solve more and more specific problems and that’s why the NoSQL databases exist — to help us in solving particular problems. That’s true that maybe some of them get increasingly all-purpose, but it’s tough to beat relational databases’ performance.</p><p>These days we can choose from a variety of database products that belong to one or more types:</p><ul><li><em>Columnar databases</em>, like Apache Cassandra, HBase, Google BigTable, Azure ComosDB</li><li><em>Wide-Column databases</em>, like ScyllaDB and most of the databases listed above</li><li><em>Document databases</em>, like Atlas MongoDB, CouchDB, AWS DynamoDB, Azure CosmosDB</li><li><em>Key-value databases</em>, like ScyllaDB, AWS DynamoDB, Azure ComosDB, Redis, Couchbase, Cassandra, RocksDB, TerracotaDB</li><li>Here we can also split this category into a subcategory of <em>in-memory key-value</em> databases like Redis</li><li><em>Time series databases</em>, like InfluxDB, TimescaleDB, MongoDB</li><li><em>Graph databases</em>, like Neo4J, TypeDB, OrientDB</li><li><em>State-transition databases</em>, like Event Store</li><li><em>Full-text search databases</em>, like Elasticsearch, Amazon ElastiCache</li></ul><p>You can read up on database internals in the great book <a href="https://www.oreilly.com/library/view/database-internals/9781492040330/"><em>Database Internals by Alex Petrov</em></a>³.</p><p>One caveat regarding my Ford’s book edition: It’s written that one of the document-based databases is <em>Amazon DocumentDB</em>. This database is objectively far behind its competitors, just look at the ratings on the “<a href="https://db-engines.com/en/system/Amazon+DocumentDB">DB-engines</a>”, see that only three clients are mentioned in<a href="https://aws.amazon.com/documentdb/"> AWS official documentation</a>, or read what <a href="https://www.mongodb.com/compare/documentdb-vs-mongodb">the official Atlas MongoDB opinion</a> is. If you think about using the document-based database on AWS while you think the <em>DynamoDB </em>is too much for you, then you can look at the official <a href="https://aws.amazon.com/solutions/partners/mongodb-atlas/">Atlas MongoDB integration on AWS</a>.</p><p>So, why exactly should we care about databases?</p><ul><li>One point has already been mentioned and this is a particular use case optimization, e.g.:<br><strong>- Patients’ therapeutical indications</strong> handled by Service C can be in fact a graph, where nodes are medicines, supplements, therapies, or specialized books, and edges (relationships) describe transitions and possible options. When a patient liked book A then we can easily traverse the graph by the book’s category and get the top 5 other similar positions; If one of the book’s categories is about building self-esteem then we can get therapy nodes that focus on this topic.<br>You can get inspiration from other use cases from <a href="https://neo4j.com/use-cases/">Neo4J docs.<br></a>Worth noticing is the fact that this type of database is not standalone but rather supporting and it should be a read model taken from another database.<br><strong>- You want to skip</strong> bulky ORMs and simply store and index whole documents, but…<br><strong>- The main point</strong> is that you can store whole Aggregates as a single document<br><strong>- Patients’ telemetry</strong> can be stored in a time series database and processed and analyzed later</li><li><em>Scalability</em>. <em>Reliability</em>. <em>Fault tolerance</em>.<br>Lots of <em>buzzwords</em>, but the younger the age of the NoSQL database is also their leverage. They often have been constructed from scratch, sometimes based e.g. on MySQL database like it is with <em>Amazon DynamoDB</em> or Meta’s database, but their architecture is also often distributed. You can scale out relational databases vertically, and you can have read replica sets, but if you want to have <strong>sharding </strong>(horizontal scaling), this is when the problems start; Fortunately, we can use totally distributed <strong>cloud-native serverless databases</strong> (with completely different philosophy) like AWS DynamoDB.<br>Very short and high-level description of how this database work is: your table gets divided into partitions and DynamoDB “knows” how to get to a partition and its data by the special hash function (see Figure 4). Partitions also get replicated over multiple Availability Zones; In front of partitions are run scaled-out <em>Request Routers </em>(APIs) that handle your requests and deal with “joins” in-flight. <em>DynamoDB </em>defines several access patterns to retrieve and store data and it means there are many constraints on what you can do and it’s easy to get messy if you don’t know what you do or you don’t have a proper design. You can have only a few <em>Global Secondary Indexes</em> which create a new <em>B-Tree</em> structure being a different read model of your data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EL_ubo1Ar1qTBb-kLo8GaQ.jpeg" /><figcaption>Figure 5. A high-level picture on scalability of DynamoDB.</figcaption></figure><ul><li>As you see, it is a different approach with the sharding by default in which you have to first have well-defined boundaries and responsibilities within your domain design resulting in data ownership letting to have a reliable and performant data structure.<br>Although <em>DynamoDB </em>supports transactions providing <em>ACID </em>guarantee, it’s not a natural thing for distributed systems and for sure it’s not a performant one; On the contrary to the <em>ACID</em>, the distributed systems define <em>BASE </em>(you know, like in chemistry) that focuses mainly on the eventual consistency.<br>If you’re interested you can dive into this topic on <a href="https://www.alexdebrie.com/posts/dynamodb-no-bad-queries/">this blog by Alex Debrie.</a> Also, you can try one of Reinvent YouTube’s videos <a href="https://www.youtube.com/watch?v=yvBR71D0nAQ">like this one</a>.<br>Remember, that cloud-native database will always be more scalable than on-premise databases.</li></ul><h3>Why keep data together?</h3><p>It’s tempting to divide our services more and more and their data with it to be as micro as the biggest microservices purist’s dream in the world, but be careful! There is a line you should not cross. The bad news is the line is rather vague and there’s no mathematical equation to calculate the borders of your modules — only heuristics and discovering Aggregates boundaries. So, why keep data together? Ford calls the reasons the data integrators.</p><ul><li>We need the atomicity of a business operation, one ACID transaction spanning crucial data. The saga pattern is not enough for us because of the <a href="https://www.scylladb.com/glossary/cap-theorem/">CAP theorem’s tradeoffs</a>.</li><li>Modules/Services are hungry, they need to talk to each other all the time to generate their own state and proceed with a change. It means that something went wrong on the design level, requirements could change and you need to redesign your system and modules/services responsibilities. Maybe the solution to problems would be merging some modules.</li></ul><p>Not everyone has to strive to create maximally independent services meeting all of the ”-ity” non-functional requirements like reliability, scalability, or availability. With the shared database, our architecture is more straightforward. It also entails the fact that it is more cost-effective when we don’t have to worry about losing the ACID and transactionality over our tables.</p><h3>Wrap-up</h3><p>That wasn’t a precise and exact summary of the sixth chapter of <em>Software Architecture: The Hard Parts by Neal Ford</em>⁵; I took parts that were the most valuable for me, I wrote what takeaways were contained on the book’s next pages, and whatever inspired me got a more detailed extension as I felt it needs more elaboration or small emendation (like with the AWS DocumentDB).</p><p>I haven’t written anything about techniques of data separation, or how to approach it. I believe that it’d require separate posts but based on a practical example — that is not the case in Neal’s book, unfortunately, it’s just a few theoretical paragraphs and your modest author also does not have experience in that.</p><p>If you haven’t heard about data ownership, you should probably get your head around it. It’s a change of customs, a change of mindset because it’s much simpler to link all the data together and access it with one transaction. If you can’t, and your only available option is data join in-memory, but you know it’s not something you should abuse, the day-to-day routine becomes more demanding. Because then the design-first matters, fine-grained planning matters, discussing tasks and stories with your teams matter, and <strong>constantly evaluating and challenging your architecture matters</strong> — architecture and design made once is not over, it’s just the beginning of evolution and discussion over them⁶.</p><p>[1]: Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley. 2003.</p><p>[2]: Building Evolutionary Architectures: Support Constant Change. Neal Ford, Rebecca Parsons, Patrick Kua. O’Reilly Media. 2017</p><p>[3]: Database Internals. Alex Petrov. O’Reilly Media. 2019</p><p>[4]: AWS Security. Dylan Shields. Manning Publications 2022 (ISBN 978161729733). Page 104–109.</p><p>[5]: Software Architecture: The Hard Parts. Neal Ford, Mark Richards, Pramod Sadalage, Zhamak Dehghani. O’Reilly Media. 2021 (ISBN: 9781492086895)</p><p>[6]: Continuous Architecture in Practice: Software Architecture in the Age of Agility and Devops. Murat Erder, Pierre Pureur. Addison-Wesley Professional. 2021</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f91aea41af6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>