<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Engineering at Birdie - Medium]]></title>
        <description><![CDATA[Birdie has been delivering software solutions to Elderly care for 6 years. We like to think we’ve learnt a thing or two in this time. We’re very excited to share it with the world. See our open roles here: https://www.birdie.care/join-us#jobs-live - Medium]]></description>
        <link>https://medium.com/engineering-at-birdie?source=rss----40f364345252---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Engineering at Birdie - Medium</title>
            <link>https://medium.com/engineering-at-birdie?source=rss----40f364345252---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:06:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/engineering-at-birdie" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Diving Deeper: Birdie’s Latest Contribution to Debezium Open Source project — Autoset Replica…]]></title>
            <link>https://medium.com/engineering-at-birdie/diving-deeper-birdies-latest-contribution-to-debezium-open-source-project-autoset-replica-3fbfb59e58dd?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/3fbfb59e58dd</guid>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[postgresql-replication]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Miguel A. Sotomayor]]></dc:creator>
            <pubDate>Thu, 19 Oct 2023 14:38:51 GMT</pubDate>
            <atom:updated>2023-10-19T14:38:51.166Z</atom:updated>
            <content:encoded><![CDATA[<h3>Diving Deeper: Birdie’s Latest Contribution to Debezium Open Source project — Autoset Replica Identity for PostgreSQL Connector</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVRunWmHYK26NaHbHOVVrQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jmsdono?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">jms</a> on <a href="https://unsplash.com/photos/0ZBRKEG_5no?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Birdie keeps growing and growing. It’s not just about the number of care recipients we’re helping through our partners, but also how our apps are becoming more advanced and packed with features. To keep up with the pace, we adopted a micro-services setup. This allows our teams to create and roll out new features independently, but the downside is that it makes it harder to centralise and share data.</p><p>To get around this, we’ve been using a technique called “change data capture” (CDC) to clone our service databases to Kafka and on to our data warehouse. But here’s the thing: we’re still growing, adding new services and data sources, and updating the existing ones all the time. For our pretty small data team, it was getting way too hectic to manage the increasing number of data flows, so we knew we needed a new game plan.</p><p>We enabled our delivery teams to manage their own CDC pipelines, but it turned out to be a tougher nut to crack than we thought. They kept running into issues — a particularly frequent and painful one was getting source tables ready for replication. In this piece, we’ll talk about how we upgraded our source connectors to take this problem away from the teams giving them the headspace to concentrate on more important things.</p><h3>CDC: A Brief Overview</h3><p>Change Data Capture, or CDC, is a technology pattern that identifies and tracks changes in a data source, and in response, generates events that capture and describe these changes. CDC is often used for replicating and distributing data across different databases, data warehouses, or other systems. In the context of Birdie, we use CDC to propagate data from our PostgreSQL databases to Kafka. In particular, we are using Debezium as CDC.</p><h3>The Pain Point</h3><p>One of the major challenges while setting up connectors in PostgreSQL is managing the dependencies. PostgreSQL requires three key dependencies for ingesting tables in Apache Kafka:</p><ul><li>The Kafka Connect role used to access to PostgreSQL must have access to the tables ingested in Kafka.</li><li><strong>REPLICA IDENTITY</strong> must be set for tables ingested in Kafka. In our case, we set FULL value to this property.</li><li>A <strong>publication</strong> must exist for tables ingested in Kafka.</li></ul><p>Mistakes in managing these dependencies often lead to the need for removing and redeploying connectors — a frustrating process indeed.</p><h3>Introducing Autoset Replica Identity</h3><p>Our team at Birdie sought to minimize the time squads spend building connectors and managing their dependencies, and that’s where our contribution to Debezium comes in. While Debezium already has a mechanism to <a href="https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-publication-autocreate-mode"><strong>automatically create publications</strong></a>, it did not offer a way to set <strong>REPLICA IDENTITY</strong> automatically.</p><p>After discussion with the Debezium team, we decided to extend Debezium’s capabilities by adding a feature to set <strong>REPLICA IDENTITY</strong> automatically.</p><p>The result? The “Autoset Replica Identity” feature.</p><h3>Using Autoset Replica Identity</h3><p>With this feature, users simply set up the option in the connector configuration, without having to manually adjust the <strong>REPLICA IDENTITY</strong> for each table. This new configuration option, <strong>replica.identity.autoset.values</strong>, accepts a comma-separated list of fully-qualified table and replica identity tuples. For example:</p><pre>{<br>  &quot;replica.identity.autoset.values&quot;: &quot;schema1.*:FULL,schema2.table2:NOTHING,schema2.table3:INDEX idx_name&quot;<br>}</pre><p>In the example above, all tables in <strong>schema1</strong> are set to <strong>FULL</strong>, <strong>schema2.table2</strong> is set to <strong>NOTHING</strong>, and <strong>schema2.table3</strong> is set to use an index <strong>idx_name</strong>.</p><h3>Benefits &amp; Drawbacks</h3><p><strong>Benefits:</strong></p><ul><li><strong>No user intervention</strong> is required, greatly simplifying the setup process.</li><li>Transparency is maintained as users can check if these properties are enabled or not.</li><li>Contribution to an open-source project with a large community and substantial support.</li><li>Engineers will have more confidence in deploying connectors.</li></ul><p><strong>Drawbacks:</strong></p><ul><li>The Kafka Connect role must share the ownership of the table with the current owner.</li></ul><h3>Conclusion</h3><p>Birdie’s commitment to revolutionizing the aging experience through technology extends to our dedication to improving the tools we use every day. Our latest contribution to Debezium is a testament to that commitment, simplifying and streamlining the setup process for PostgreSQL connectors.</p><p>We invite you to try out this feature in Debezium 2.3 and let us know your thoughts. We believe in open collaboration and look forward to your feedback. Together, we can continue to refine, improve, and expand our work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fbfb59e58dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/diving-deeper-birdies-latest-contribution-to-debezium-open-source-project-autoset-replica-3fbfb59e58dd">Diving Deeper: Birdie’s Latest Contribution to Debezium Open Source project — Autoset Replica…</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A guide to observability at Birdie]]></title>
            <link>https://medium.com/engineering-at-birdie/a-guide-to-observability-at-birdie-29b3459c9a78?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/29b3459c9a78</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Lorenzo Pieri - 404answernotfound]]></dc:creator>
            <pubDate>Thu, 07 Sep 2023 10:16:19 GMT</pubDate>
            <atom:updated>2023-09-07T10:16:19.562Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fABMfTV15p56VE5X" /><figcaption>Photo by <a href="https://unsplash.com/@etiennegirardet?utm_source=medium&amp;utm_medium=referral">Etienne Girardet</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s close our eyes for a minute and imagine a world where everything just makes sense and if it doesn’t, well, no worries, because we still have the chance to find meaning on any event, based on all the clues and cues that surround us.</p><p>From an Engineering perspective it might sound too good to be true but indeed, this is the power we rely on at Birdie whenever we want to make an informed choice or need to gather relevant informations about choices that we have already taken.</p><p>A decision, in simple terms, is something that might affect the way a business responds to events or how external factors could respond to these business decisions. Sometimes a decision can be dependent on a hunch or a feeling from someone inside a company but most of the times it really depends on data but where does this data come from?</p><p>Well, we could just say that most businesses are what we can call “data driven” and all the decisions are taken based on relevant outcomes from this “data driven approach”.</p><p>“Alright Lorenzo, but you are very well aware that Data, most of the times, is just a garbage collection thrown into very spacy baskets with no one looking into them”.</p><p>Well, <em>Troy</em>, you are right, most of the times that is exactly what happens but we should also be aware that that’s not the expected way to deal with this. Afterall, why would you bother collecting data at all, if you don’t want to make use of it. You should understand that this action of “collecting” any sort of information does bring a little overhead to any request carried out by our systems, so why collect anything at all?</p><p>“Well, everyone does logging!” — you might argue, <em>Troy</em>.</p><p>That’s not a reasonable answer, <em>Troy</em>.</p><p>Why would we collect any data at all? T<strong><em>o make informed decisions that align with the company values and objectives</em></strong>. That is good but how do we do that?</p><p>At <em>Birdie</em>, as Engineers, we rely upon flows of data that get collected, sampled and aggregated in our systems to be sent to other applications that make use of these to extrapolate meaning and show us how our system fares based on latest changes to the codebase, the number of requests that we receive and so many more valuable insights. This is what we define as <em>an observable system</em>.</p><p>We are also making heavy use of Event Sourcing, as such every event that is generated by the system can be used as <em>audit trail, data payload and as part of a snapshot to rebuild the system (and its aggregates) at any point in time</em>. This is not an article about Event Sourcing but I am sure that you can already sense the deeper understanding of the system and the data that we can have if we use Event Sourcing and Observability in conjunction.</p><p>As per the definition stolen by a dear colleague, <em>a system is observable if we are able to, without making any code changes, ask questions about the running system, understand how it behaves and identify issues</em>. Sounds delightful, but, it takes time, knowledge and a lot of pressure to build a diamond out of carbon, doesn’t it? And that’s exactly what happened at Birdie and, I’m sure, at many other companies out there, after all, you don’t want to start over-engineering by default, do you <em>Troy</em>?</p><p>At <em>Birdie</em>, not unlike many others, we started our journey into Observability with a simple yet very dependable logging system called <em>Bunyan</em>. Now, <em>Bunyan</em> is surely a recognised and famous tool used everywhere in the industry. It’s a fast and simple JSON logging module that you can inject anywhere in your Nodejs applications and it will just work. This approach works decently and it’s possibly one of the fastest.</p><p><em>Spoiler alert: there is an even faster one with time and space complexity O(0). Not collecting logs at all. Mind you, not the best choice.</em></p><p>The way pretty much everyone deals with this type of logging systems is to wildly hoard any logs and then cat logs.txt | grep &quot;any text in the logs&quot; which is completely fine but I’m sure, <em>Troy</em>, we can agree there must be a better system. One that is more reliable than the one we just described.</p><p>We can also agree on the fact that this kind of logging doesn’t come easy with correlation and causality between the different logs that get added to the mix. Just like a bowl of cereal, you just throw everything in and enjoy. This approach screams something like: “Send it all in, we’ll figure it out later!” and don’t get me wrong, it’s a fine approach for any starting points but if you feel your apps are beyond that, then you might be interested in relating this events to one another and be able to rebuild a story out of them, <em>a trail</em> of how log A generated log B up to log X, whatever that might be. That’s Observability, mostly.</p><p>With an observable system you, the <em>incredible engineer</em>, will be able to rebuild this trail of events based on all the hops that were generated between single interactions. This is powerful enough in Monolithic systems but just try to think of the insights that you might be able to gather in architectures like <em>Micro services.</em></p><p><em>Just like an onion</em>, each hop (in the system or in the network) will generate an additional layer of knowledge that will embrace the previous request and it will be up to you to scout for answers in what is a very precise, concise and knowledgable piece of information.</p><p>This piece of information is called <em>a trace</em> and a trace will contain all the information you need about how a process interacted with your systems and which events were generated based on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-9cs7DutvSHxPvtzPTcuiA.png" /></figure><p>A <em>Trace</em> is usually made up of 3 things:</p><ol><li>The Trace ID, which is shared between all <em>spans</em> that belong to that trace</li><li>The Span ID, which is the ID of the single spans</li><li>The Parent ID, which starts as undefined and changes for each subsequent span, based on the previous one. This is what we can refer to as the <em>causality</em> of the current span.</li></ol><p>Yes, it’s practically a <em>linked list of knowledge except that it’s not. It’s more like a tree. Which tree? A Baobab. You are welcome.</em></p><p>You might already be thinking: <em>Wow, this is incredible. I’ll get to it immediately.</em> Of course you will, <em>Troy</em>, but that’s not all there is to it, so read on!</p><p>With such a system in place we now have the granularity of the single logged <em>event</em> (which again, is a span) wherever it happens in our services. <em>Orwell would be so proud of us</em>. We can now see the trail of an event from start to finish, and that’s because every single <em>span</em> also describe itself as a log comprised on the span information, <em>the duration</em> of the span and <em>a timestamp</em> of when the span occurred.</p><p>“Alright, I’ll just delete all my console.logs and add Tracing to my systems!”. Yes, <em>Troy</em>, but also no. Very much no.</p><p><em>Tracing</em> and <em>Logging</em>, just as <em>Metrics</em> (which is another one of their buddies when talking about Observability) are really powerful when together so you don’t really need to choose one over the other, you just need to make sense of all of them and make good use of this new found knowledge at your disposal.</p><p>“Again, I get it, now stop being a bother and tell me <em>how</em> to implement this in my systems!”. Love the energy <em>Troy</em>.</p><p>As many answers by engineers that have reached <em>senior level and beyond</em>: it depends.</p><p>What I can tell you is that there are many ways to build an Observable system and at Birdie we used more than one approach and built up from there, little by little. There were many <em>Application Performance Monitoring</em> systems (APM from now on) to choose from and we also built our internal packages to deal with <em>tracing</em> using HTTP headers as well but now we are actively using <em>OpenTelemetry SDK</em> which is the <em>de facto standard</em> (which is the <em>de facto way of making something sound very important</em>) to collect traces in Distributed Systems like the ones we operate every day.</p><p>Traces (along with their spans) get collected and <em>refined</em> through an open source sampler proxy called <em>Refinery</em>. Sampling happens heuristically and relies on the amount of events that were sampled up to that point.</p><p>The basic gist of it is that we collect <em>all 500s</em> and a sampled amount of HTTP codes in the 200s, 300s and 400s range. After all, the most important information we can get is based on the errors that our systems might output.</p><p>But collecting is not enough and that’s where Monitoring tools and APMs come into play. We are just switching to a new monitoring system called <em>Honeycomb</em> which we found out to be very simple to get used to. With Honeycomb we can ask the right questions to the system at any point in time and we can configure and build these questions based on all the informations we have collected. Through this system, we are also able to visualise how our services are keeping up, the amount of requests they receive and if any issues are happening <em>in real time</em>.</p><p>Another important aspect of <em>Honeycomb</em> is that we can build personalised dashboards to check on our internal and external SLOs/SLIs/SLAs (Service Level IN (”objective”, “indicator”, “agreement”)) and <em>error budget</em> that we have as squads working in autonomous teams.</p><p>At Birdie, we are very much invested in bringing the best services and service levels to our end users and we care deeply for our systems. That is why we make ourselves accountable through the usage of SLOs and <em>error budget</em>, allowing us to be more aware of the scope of our applications and demystify all the processes that are part of our systems.</p><p>Thank you for taking the time to read through this with me and <em>Troy</em>! Happy Observability!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29b3459c9a78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/a-guide-to-observability-at-birdie-29b3459c9a78">A guide to observability at Birdie</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving mountains: How we succeeded in migrating huge datasets into Birdie]]></title>
            <link>https://medium.com/engineering-at-birdie/moving-mountains-how-we-succeeded-in-migrating-huge-datasets-into-birdie-f7303eebc949?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/f7303eebc949</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Eben Ennim]]></dc:creator>
            <pubDate>Tue, 25 Apr 2023 10:13:52 GMT</pubDate>
            <atom:updated>2023-04-25T10:13:52.321Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DgZPIzHkN92H6poW" /><figcaption>Photo by <a href="https://unsplash.com/@danielleone?utm_source=medium&amp;utm_medium=referral">Daniel Leone</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>This piece is the first of a series of pieces covering the technical challenges Birdie’s engineers faced when migrating enterprise clients into our system. Watch this space for more!</strong></blockquote><p>Picking the right tool for a task or job is essential. In fact, the difference between a job done right and a mess to clean up could rest solely on the tool used.</p><p>A key factor in making the right choice of tool is an understanding of the job at hand. For example, if you were cutting through metal, you would need a different tool as compared to if you were cutting through wood.</p><p>I’m sure you’re probably wondering if this is a DIY article or an engineering blog by now but stay with me for a moment.</p><p><a href="https://www.birdie.care/">Birdie</a> is an all-in-one home healthcare technology that has been growing tremendously to help improve the lives of older adults. We provide a software solution to care providers all over the UK. This allows them to assess, plan, roster and finance the care they deliver to older adults.</p><p>We often provide our software to large partners like <a href="https://www.birdie.care/blog/birdie-and-home-instead">Home Instead</a> who deliver care to tens of thousands of people. They have come from existing systems and therefore require that their data is effectively migrated over to <em>our</em> system.</p><p>The process of migrating over this existing data as you would appreciate presents a few challenges</p><ol><li>Migration implies care agencies start using birdie instead of their existing system and this is a significant operational change for their entire staff (think moving from Windows to MacOS)</li><li>We needed to have a migration process that allows us to incrementally migrate data, as opposed to a one-off file transfer to accommodate scenarios where larger agencies with multiple branches and different migration dates per branch could do so with little stress</li><li>There was every possibility that we would not have access to their existing system’s API. Instead, the customer will have to transform their own data to conform to a schema expected by birdie. Every care agency might have to have different flavours of mappings due to their existing implementation details.</li><li>Whatever system we generated would need to be very reliable. It requires a high level of automated testing, the necessary observability (logs &amp; artefacts’ archives) and appropriate monitoring in place.</li></ol><p>To solve the challenges outlined above, you guessed it — the right tool had to be chosen.</p><p>Another factor to consider in making the right choice of tool is the ergonomics of the tool. You would want to have a tool that fits comfortably and allows you to execute the task easily.</p><p>My team was tasked to solve this data migration challenge. We had to make the hard decision of picking the tool to use.</p><p>The team settled on using <a href="https://elixir-lang.org/">Elixir</a> for this purpose and there are a couple of observations that I made that confirm the choice of tool was the right one.</p><ul><li>Simplicity: the language offered uniformity not just for the sake of it but it promoted; simpler development, simpler testing, simpler deployment, simpler maintenance.</li></ul><p>All the things needed to address the challenges mentioned above were self-contained within Elixir’s ecosystem. We could literally define the system in one language. Background jobs, queues and data persistence (plus much more!) had adequate implementations in Elixir. We did not have to include external solutions which would come with some extra overhead.</p><ul><li>Improved collaboration: this simplicity allowed two-thirds of the engineers, who had never used the language before, to contribute equally. We mobbed (group coded) on parts of the system the day after the Principal engineer suggested that we were going to use the language. The only prerequisite was an open mind.</li></ul><p>This article would be painfully long if I included the detailed architectural decisions and technical considerations we made using Elixir to solve the data migration challenge but watch this space for articles on the topics enumerated under the simplicity observation.</p><p>To tie it up, clearly defining the task at hand to be solved goes a long way toward determining the tool needed. Once that box is checked, assess the tool that gets you to start delivering value with speed and ease, then go for it.</p><blockquote><strong>This piece is the first of a series of pieces covering the technical challenges Birdie’s engineers faced when migrating enterprise clients into our system. Watch this space for more!</strong></blockquote><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us#jobs-live</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7303eebc949" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/moving-mountains-how-we-succeeded-in-migrating-huge-datasets-into-birdie-f7303eebc949">Moving mountains: How we succeeded in migrating huge datasets into Birdie</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Hours to Minutes: Making older React Native Apps easy to set up]]></title>
            <link>https://medium.com/engineering-at-birdie/from-hours-to-minutes-making-older-react-native-apps-easy-to-set-up-ad9c1997fe6d?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/ad9c1997fe6d</guid>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[expo]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Marin Godechot]]></dc:creator>
            <pubDate>Mon, 27 Feb 2023 09:41:31 GMT</pubDate>
            <atom:updated>2023-02-27T09:42:02.827Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UbO4iZuLuzFbnvHp" /><figcaption>Photo by <a href="https://unsplash.com/de/@lucas_degenhardt?utm_source=medium&amp;utm_medium=referral">Lucas Degenhardt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As developers, we’ve all been there. We get excited to learn a new programming language or work on a new project, only to find ourselves stuck for hours on setting up the development environment and searching for obscure error messages online. This is especially true for React Native development, where getting your native environment set up can be a nightmare. But what if I told you that it doesn’t have to be this way?</p><p>Thanks to Expo and its tools like Expo Snack, Expo Go, and Expo Dev Client, setting up a new React Native project has become a breeze. But what if, like us at birdie, you have a massive React Native app that was created years ago that is still on older versions of React Native and Expo SDK, preventing you from playing with the new toys? Sure, you could invest the time to upgrade everything, but that would take ages, it might be better to explore other options to enhance the developer experience in the immediate term.</p><p>What makes setting up a React Native development environment so hard is that you need to have three parts configured properly: the JavaScript part, the Android native part, and the iOS native part. Each of these parts can already be a pain on its own, but the native parts are particularly challenging. For Android, you might need to have Java, Gradle, and Android Studio set up and configured, while for iOS, you need Ruby, Cocoapods, and XCode.</p><p>When we got new engineers to work on our mobile app, it was often the case that one of the pieces wasn’t installed or configured properly. The error messages we got weren’t that helpful either, so we ended up spending hours trying to figure out what was wrong. Things got even worse when engineers started using M1 MacBooks since not all required software was compatible with ARM yet.</p><p>It got to the point where making a trivial change in the app would be considered a high cost for a team if they hadn’t setup the app locally in a long time. With a quickly growing engineering team, we needed to fix this urgently.</p><p>Given that the hard part is the native side, but we very rarely make changes to the native code, why build it again and again when developing locally? Our solution was to remove the need for setting up the native environment entirely when working on the JavaScript code. By building the native parts of the dev app in advance, engineers simply need to download the pre-built dev app and set up the JavaScript side with a quick yarn install &amp;&amp; yarn start. It now takes less than 5 minutes for a newjoiner to be able to start working on our app.</p><p>We did this by using another Expo product, EAS, which allowed us to build the native parts in advance and bypass the need for developers to set up their local environment.</p><p>EAS Build is designed to work with any React Native project, which allowed us to set it up with no changes to our app code. Using Github Actions, we now have dev apps built automatically for iOS and Android whenever we create a new native version.</p><p>Here is how we have it setup:</p><p>After following <a href="https://docs.expo.dev/build/setup/">Expo’s instructions</a> on creating an account and setting up our app, we configured EAS through the eas.json file like so:</p><pre>{<br>  &quot;cli&quot;: {<br>    &quot;version&quot;: &quot;&gt;= 0.55.1&quot;<br>  },<br>  &quot;build&quot;: {<br>    &quot;dev&quot;: {<br>      &quot;distribution&quot;: &quot;internal&quot;,<br>      &quot;android&quot;: {<br>        &quot;gradleCommand&quot;: &quot;:app:assembleDebug&quot;,<br>        &quot;image&quot;: &quot;ubuntu-18.04-jdk-11-ndk-r19c&quot;<br>      },<br>      &quot;ios&quot;: {<br>        &quot;simulator&quot;: true,<br>        &quot;buildConfiguration&quot;: &quot;Debug&quot;<br>      },<br>      &quot;env&quot;: {<br>        &quot;ENVFILE&quot;: &quot;.env.dev&quot;<br>      }<br>    }<br>  }<br>}</pre><p>By specifying the exact gradle command to use for Android and build configuration to use for iOS, we made sure that it would work with our existing native dev build setup which doesn’t necessarily match Expo’s defaults.</p><p>By specifying which server image to use for Android, we make sure that the build tools on the server are compatible with our versions of Expo SDK and React Native. If you use the recommended version of React Native for your Expo SDK, you don’t have to do this as EAS will pick a compatible server for you.</p><p>To create a dev build, from our CI or from a laptop, we use eas-cli like this: eas build --profile dev --platform all --non-interactive.</p><p>Our mobile app is versioned in a <a href="https://semver.org/">SemVer</a> format, with the implication that a native change leads to at least a minor version bump. This allows us to only build dev apps when needed by having a step in our CI that looks for version changes and calls the command above when there is a major or minor version change.</p><p>Engineers can then download the latest build from EAS and not worry about the native side of the app until the next minor version change, at which point they need to download the new build.</p><p>This solution worked for us, but you could also build a similar system with fastlane and your preferred CI service. We chose EAS due to its ease of use and setup, plus our intention to migrate other build and release processes there in the future.</p><p>Going forward, our plans include upgrading our React Native and Expo SDK versions to have access to all the latest tools and features. We also plan on using Expo Dev Client, which should make our development process even smoother.</p><p>No more hours spent trying to configure the development environment — just a quick and easy setup process so we can get back to what we do best: building delightful features for our users.</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us#jobs-live</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad9c1997fe6d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/from-hours-to-minutes-making-older-react-native-apps-easy-to-set-up-ad9c1997fe6d">From Hours to Minutes: Making older React Native Apps easy to set up</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Disasters & Platform teams — How to deliver applications you can be confident in.]]></title>
            <link>https://medium.com/engineering-at-birdie/disasters-platform-teams-how-to-deliver-applications-you-can-be-confident-in-5cb375d6e3b4?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/5cb375d6e3b4</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[disaster-recovery]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Harry Morgan]]></dc:creator>
            <pubDate>Wed, 11 Jan 2023 13:48:46 GMT</pubDate>
            <atom:updated>2023-01-11T13:48:45.971Z</atom:updated>
            <content:encoded><![CDATA[<h3>Disasters &amp; Platform teams — How to deliver applications you can be confident in.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JbCWQXW43zDfVmCe" /><figcaption>Photo by <a href="https://unsplash.com/es/@nasa?utm_source=medium&amp;utm_medium=referral">NASA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>No one likes to think about the worst case scenario.</p><p>Bringing an umbrella in case it rains, clicking save on your document every 5 minutes or looking both ways before crossing the road. These behaviours become natural over time, and are learned from years of experience and come with knowledge of what might happen if we don’t do them.</p><p>Disaster Recovery is a practice that attempts to mitigate the impact of disasters on a system. At Birdie, that could be any kind of event that might disrupt a modern distributed system.</p><h3>What is a “disaster”?</h3><p>At Birdie, a disaster is something that could stop us providing our service to our customers — those are care providers, care managers and ultimately vulnerable people who rely on our system. This can be anything from a natural disaster damaging data centre infrastructure, a malicious attack by a hacker, or simply human error in development. These events are rare but one of the right size could bring a halt to most businesses in the line of fire.</p><p>We cannot tolerate the consequences of these events. It would mean loss of data, leaving carers at loose ends in arranging their visits or logging important information. Ultimately having a negative impact on the lives of carers and care managers, which is the very opposite of our mission.</p><h3>Recovering from disasters</h3><p>Disaster Recovery (DR) is the practice of having tools in place that we can rely on to ensure Birdie will still exist if the worst happens. It’s a goal that engineering organisations build towards.</p><p>What’s important to note here is that Disaster Recovery is very much a moving target, not one Platform team at any business can say they’ve “completed disaster recovery”. It’s a set of practices that move with your organisation and evolve as your engineering organisation evolves and progresses.</p><p>At Birdie these practices have an origin within The Owls (The Infrastructure &amp; Security team). We develop and advocate for tooling to achieve our best effort Disaster Recovery process. Considerations of these processes can take many forms, and span across a development lifecycle. A few technical areas to note:</p><p><em>Application code</em></p><ul><li>Can we rely on our application code to deploy reliably onto infrastructure that it doesn’t usually?</li><li>Are there any hardcoded parameters in our production environment that we’d miss, or configuration that was manual and not documented?</li></ul><p><em>CI/CD</em></p><ul><li>Will our deployment practices slow us down in the event of a disaster? Are there any secrets or variables being injected into code that can’t be replicated to other environments or regions?</li><li>Can we rely on our pipelines to be successful in the event of a disaster? How confident are we in builds succeeding?</li><li>Are our deployment practices reproducible locally? What if our CD runners aren’t available?</li></ul><p><em>Infrastructure</em></p><ul><li>Is our infrastructure maintained in code, and can it be replayed at whim?</li><li>Are all resources in our infrastructure available in regions we plan to deploy into?</li><li>Are there regular backups available and are we confident in being able to restore from these backups?</li></ul><p><em>End-User</em></p><ul><li>What will the impact on the end-user be if we have to recover from a disaster? Will their data persist? Will our DNS records change?</li></ul><p>Again, it’s important that all of these aspects are considered in developing processes for disaster recovery — but these responsibilities don’t just sit with the platform team in an organisation — it’s important that there is buy-in. <em>Another</em> thing for developers to consider — how can we ensure the burden of DR is shared?</p><h3>Sharing the love</h3><p>Everyone is very busy.</p><p>We’ve all got roadmaps filled to the brim until Q1 2050 and requirements flying in left, right and centre. How can an engineering team be expected to plan for heavy and slow investigations and possibly redevelopment? This is the kind of work where the benefit can border on the ambiguous. Building and tearing down a deployment, untying hard-constraints in pipelines, ensuring container images are backed up and updated regularly, environment variables aren’t hard-coded — this is work that drags — and once you’re done, you don’t have a feature in front of you or a carers’ need satisfied.</p><p>This perceived problem in this case, I believe, comes down to communication and interactions between platform and development team. If a platform team considers a development team <em>their customer,</em> the problem gets reframed slightly — and this reframing can be really important in delivering a robust disaster recovery setup.</p><p>On one end, a platform team can focus on delivering tooling and abstractions that can benefit development teams in their comprehension and understanding of their role in disaster recovery. In a developer-is-the-customer frame, these products are delivered to engineering teams and as with any client relationship, feedback can be provided and time can be spent pairing on implementations and communicating expectations.</p><p>This means engineering teams or platform teams aren’t expected to “solve” disasters recovery themselves, but work with a shared tooling and requirements with which they can plan steps to make services resilient. Following this, the onus of prioritisation is on the product managers and developers within a team (another hard thing), but the work is less abstract — it’s no longer “DR for 2 sprints” — it can be thought about more as “continuous improvement” work.</p><p>And that, I think, is an important pattern to adopt in engineering.</p><p>We’ve already seen the rapid rise of shift-left methodologies and practices like <em>DevOps</em> and <em>DevSecOps</em> move more of the operations side of engineering into the development lifecycle. Concepts like deployments, infrastructure and security are now commonplace within development teams — and their maintenance is considered when planning, because it is work owned and operated by that given team. I simply believe building disaster resilient code (application or infrastructure) is another concept that should be shifted left.</p><p>I feel it is important to note at this point that Birdie haven’t nailed this either. We’re currently in the throngs of jamming disaster recovery tooling into already-built systems and already-planned backlogs — but with a vision to come out the other side. These tangles can be avoided.</p><h3>Building for confidence</h3><p>Disasters aren’t edge-cases. Any incident can snowball into a disaster without the proper foresight. But if you’re lucky enough to never see your system fall away from you — what exactly is in it for you?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*iHajmOPBdru8AVs1kh8Jfw.jpeg" /></figure><p>Well, a lot. But the headlines are:</p><p>Confidence and portability.</p><p>That doesn’t just mean confidence internally. Being able to answer questions from customers regarding recovery time if your system is compromised with certainty is a reassurance we must afford people in a world ever governed by and dependent upon complex distributed systems.</p><p>While portability doesn’t quite strike the same vital chord — it’s a very underrated aspect of a modern system. Having the ability to prop up your application in new environments or regions gives a product <em>agility</em> — being able to navigate complex problems like data residency requirements or international expansions with ease, as the bulk of the technical problem has already been solved.</p><p>Ultimately, I believe instilling resilience as a core value in an engineering organisation is just as vital as considering security, fast-paced deployments and robust testing when beginning to build infrastructure or applications. It encourages a culture of confidence within an organisation — confidence in deploying consistently and in large volumes, confidence in building-out on-call practices, confidence in growing a user-base at pace.</p><p>So to bring it back to our clickbait-y headline, how do we deliver applications we can be confident in? As far as I believe it’s a team effort. It’s on everyone to consider resilience and recoverability in every stage of the development lifecycle, have platform teams deliver tooling and products to application engineers that demystify and untangle complexity, make deployments lightweight and portable, practice recovering from disasters, and understand that disaster recovery isn’t a box that is ticked, but a ladder you climb, which you’re constantly adding rungs to.</p><p>That’s a lot to consider and change. Organisations that aren’t already building for disasters will find it hard to prioritise and fit in, but ultimately, at the end of the article, if nothing else is taken from it; I hope engineers or managers might ask themselves; “am I confident that my system could recover from a disaster?”.</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us#jobs-live</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cb375d6e3b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/disasters-platform-teams-how-to-deliver-applications-you-can-be-confident-in-5cb375d6e3b4">Disasters &amp; Platform teams — How to deliver applications you can be confident in.</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Puzzling Postgres: a story of solving an unreproducible performance issue]]></title>
            <link>https://medium.com/engineering-at-birdie/puzzling-postgres-a-story-of-solving-an-unreproducible-performance-issue-778075ed7998?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/778075ed7998</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[William Duclot]]></dc:creator>
            <pubDate>Tue, 03 Jan 2023 15:43:24 GMT</pubDate>
            <atom:updated>2024-03-14T12:33:00.393Z</atom:updated>
            <content:encoded><![CDATA[<h3>Puzzling Postgres: solving an unreproducible performance issue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-yO8WWakKxdqQd3D" /><figcaption>Photo by <a href="https://unsplash.com/es/@rosssneddon?utm_source=medium&amp;utm_medium=referral">Ross Sneddon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>TL;DR</h3><p><em>Note: this is a spoiler of the investigation below.</em></p><p>First surprise: some Postgres drivers (eg pgx in Go) use prepared statements under the hood, without the user explicitely opting in.</p><p>Second surprise: in Postgres, queries executed via a prepared statement can have a different query plan than if executed directly. The query planner can choose to use a “generic” query plan that ignores the value of the prepared statement’s parameters, potentially resulting in a very inefficient query plan for the <em>actual</em> values you send.</p><p>Third surprise: Postgres’ query planner has a bug in its cost estimation.</p><p>Consequence: we had a query that was particularly slow in production, but it was very hard to reproduce as we didn’t know our ORM was using prepared statements under the hood and nothing was looking problematic with a simple EXPLAIN ANALYZE.</p><ul><li><a href="https://github.com/authzed/spicedb/issues/486">The original bug report in an application</a> using pgx</li><li><a href="https://www.postgresql.org/message-id/17540-7aa1855ad5ec18b4%40postgresql.org">The bug report to the Postgres community</a>. The conclusion is that there’s indeed an issue in the query planner.</li></ul><h3>Background</h3><p>Birdie provides technology for homecare providers to deliver outstanding care. In the backend, our services need to do some access control to ensure that users can only access what they are allowed to: for this we were testing the authorization system <a href="https://github.com/authzed/spicedb">SpiceDB</a>, an open-source implementation of <a href="https://authzed.com/blog/what-is-zanzibar/">Google’s Zanzibar</a> which we had identified as a good fit for our requirements.</p><p>SpiceDB’s job is to receive requests like “does user X have permission Y on the resource Z?” and answer with “yes” or “no”. We configured SpiceDB to use Postgres as datastore.</p><h3>The problem</h3><p>As I tried to send some shadow traffic to our deployment of SpiceDB, things went very wrong. Response times were expected to be in single-digit-milliseconds, but I was observing it to be in <em>minutes.</em> Timeouts and failures were all over the place. It didn’t look good.</p><p>I direct my eyeballs at the most obvious thing to spring to mind: the database. A quick run-through of the AWS RDS monitoring yields a clue: CPU was 100% maxed out. I’m already in discussion with the SpiceDB maintainers at this point (a lovely bunch of people), warning that there seems to be something very wrong with the Postgres adapter (which at the time was kind of beta, CockroachDB was the better-supported option).</p><p>I’ve done my share of drilling into Postgres shenanigans in the past, I decided to try and gather as much information as possible to zero in on the problem collaboratively. First stop: AWS Performance Insights.</p><h3>Performance insights</h3><p>AWS Performance Insight is an AWS service to monitor your database, in particular, it shows the top queries by “wait type” (which is basically different types of load): CPU, disk read, disk write, locks… It is an unintuitive tool, limited in functionality, terribly buggy, basically unmaintained, but yet one of the most useful tools we have to figure out what is wrong in a database.</p><p><em>(EDIT 2024: Perf Insights has been worked on since then, “buggy” and “unmaintained” isn’t a fair assessment anymore although it’s still limited).</em></p><p>At the time, this is what Performance Insight showed us:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NFQFVO9BSiZlRZPY3aSnug.png" /></figure><p>Lots of green. Green means CPU. We already knew that the DB CPU was maxed out so not much learning. The “average active session” unit is a bit tricky to interpret, but it’s good enough to point out that having it at &gt;200 is <em>insanely</em> high (our now-healthy database is below 2).</p><p>The useful thing about Performance Insight is that it tells you exactly <em>which</em> database queries are causing this load. I don’t have a screenshot of the queries breakdown at this time, but it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_iSsfQdYkKVE6P5IdgU_Bg.png" /></figure><p>In my case the top-most heavy query, that was single-handledly responsible for all this CPU usage, was:</p><pre>SELECT MIN(id), MAX(id) FROM relation_tuple_transaction<br>WHERE timestamp &gt;= $1;</pre><h3>A first shot at EXPLAIN ANALYZE</h3><p>Alright, I easily identified a query that’s using up a crazy amount of CPU, that’s a good start. It smells a lot like a missing index, so let’s EXPLAIN ANALYZE this. Digging in the SpiceDB code, I find that this $1 is always the same value (pseudocode below, actual code <a href="https://github.com/samkim/spicedb/blob/79622b7b5e9ba8c68efc5cfff06164495a176b0d/internal/datastore/postgres/postgres.go#L533">here</a>).</p><pre>pg.query(<br>	&quot;SELECT MIN(id), MAX(id) FROM relation_tuple_transaction WHERE timestamp &gt;= $1&quot;,<br>	[&quot;now() - interval &#39;5s&#39;&quot;]<br>)</pre><p>The “5s” is actually configurable, but I’m certain it’s the value used for my deployment of SpiceDB. Let’s run the EXPLAIN ANALYZE:</p><pre>EXPLAIN ANALYZE SELECT MIN(id), MAX(id) FROM relation_tuple_transaction<br>WHERE timestamp &gt;= now() - interval &#39;5 seconds&#39;;</pre><pre>Aggregate  (cost=8.38..8.39 rows=1 width=16) (actual time=0.009..0.009 rows=1 loops=1)<br>  -&gt;  Index Scan using ix_relation_tuple_transaction_by_timestamp on relation_tuple_transaction  (cost=0.42..8.37 rows=1 width=8) (actual time=0.005..0.006 rows=0 loops=1)<br>        Index Cond: (&quot;timestamp&quot; &gt;= &#39;2022-03-17 11:17:34&#39;::timestamp without time zone)<br>Planning Time: 0.183 ms<br>Execution Time: 0.035 ms</pre><p>Hm. That’s fast.</p><ul><li>The “cost”, ie the estimation of how expensive this query will be, is low: 8.39 is not a lot in “cost units”</li><li>The “actual time” is fast: 9 microseconds</li><li>The query plan seems perfectly reasonable</li></ul><p>I’m confident this is the problematic query, I’m also confident that we’re always querying with now() - interval &#39;5s&#39;. Then… why is my database being overloaded when this query seems so fast and cheap? Volume (queries per second) comes as an obvious hypothesis, but built-in metrics confirm that we’re not seeing a high volume of queries: it does look like we’re dealing with a reasonable volume of very CPU-intensive queries. Which is contradicted by my EXPLAIN ANALYZE. Which confuses me.</p><h3>Shot in the dark</h3><p>I have no idea what’s going on and I don’t have much to dig into. In other words, I’m in the dark: let’s shoot in the dark to hopefully hit <em>something</em> that we can dig into.</p><p>Among various things I tried, I ended up running a query that gives me an interesting result:</p><pre>-- Same query, with a different value for `timestamp`<br>EXPLAIN ANALYZE SELECT MIN(id), MAX(id) FROM relation_tuple_transaction<br>WHERE timestamp &gt;= &#39;2022-03-15 00:00:00&#39;;</pre><pre>Result  (cost=1.07..1.08 rows=1 width=16) (actual time=28103.240..28103.242 rows=1 loops=1)<br>  InitPlan 1 (returns $0)<br>    -&gt;  Limit  (cost=0.43..0.54 rows=1 width=8) (actual time=28103.219..28103.220 rows=1 loops=1)<br>          -&gt;  Index Scan using pk_rttx on relation_tuple_transaction  (cost=0.43..193195.54 rows=1823603 width=8) (actual time=28103.218..28103.218 rows=1 loops=1)<br>                Index Cond: (id IS NOT NULL)<br>                Filter: (&quot;timestamp&quot; &gt;= &#39;2022-03-15 00:00:00&#39;::timestamp without time zone)<br>                Rows Removed by Filter: 548517<br>  InitPlan 2 (returns $1)<br>    -&gt;  Limit  (cost=0.43..0.54 rows=1 width=8) (actual time=0.016..0.016 rows=1 loops=1)<br>          -&gt;  Index Scan Backward using pk_rttx on relation_tuple_transaction relation_tuple_transaction_1  (cost=0.43..193195.54 rows=1823603 width=8) (actual time=0.015..0.015 rows=1 loops=1)<br>                Index Cond: (id IS NOT NULL)<br>                Filter: (&quot;timestamp&quot; &gt;= &#39;2022-03-15 00:00:00&#39;::timestamp without time zone)<br>Planning time: 0.375 ms<br>Execution time: 28103.275 ms</pre><p>Setting the timestamp to a different value, in particular a date that’s further in the past (a couple of days back), gives me a very different result.</p><ul><li>The query took <em>very long</em> to execute, 28s here</li><li>The query plan is wildly different and much more complex</li><li><strong>Interestingly,</strong> the cost estimation is still very low. It’s even <em>lower</em> than the estimation I had using “5s ago” as a filter.</li></ul><p>That’s a super interesting result. I’m not sure yet how relevant it is to my problem, but it looks shifty enough that I’d like to see if I can draw some correlations with my problem. Here’s what goes through my head:</p><ul><li>This execution time is consistent with the response times I’m observing in reality. 28s <em>has</em> to mean a lot of CPU is being spent (very likely from this Rows Removed by Filter: 548517). This query plan is consistent with what the database monitoring shows me.</li><li>The “estimated cost” is very low but the query is actually very CPU-intensive: <strong>that smells like a query planner issue.</strong> That’s not good news: famously, Postgres’ query planner is a black box that users have very little control over, if there’s a problem there I might not be able to find a workaround.</li><li>The actual value of timestamp used in the WHERE seems to matter a lot for the query planner.</li></ul><p>That gives me a direction for investigation: is it possible <em>somehow</em> that the query plan I get when executing EXPLAIN ANALYZE with the “5s ago” filter isn’t the one that’s being used for SpiceDB? Even if I’m absolutely certain that SpiceDB also uses “5s ago”?</p><h3>Dig into the planner statistics</h3><p>When talking about variations in query plans, the usual suspect is Postgres’ use of statistics.</p><p>Postgres maintains some stats about the content of your tables (row count, values distribution…) to help create optimised query plans. These statistics change with writes (eg table growing in size) and the query planner might decide to start using a different query plan for a given query. There’s no way to anticipate that change accurately, and there’s no way to force another query plan: that’s why Postgres’ query planner is considered to be a black box.</p><p>So when confronted with the problem space of “changing query plans”, “statistics” tingles my spidey sense.</p><p>In this case, however, my hypothesis is that I am getting <em>different query plans</em> for the <em>same query</em> on the <em>same database</em> and on the <em>same dataset</em>. That doesn’t really sound like the common issue of statistics changing.</p><p>For peace of mind, I run a VACUUM ANALYZE FULL (⚠️ this will lock your table for a while) to definitely clean up statistics and dead-tuples-related noise. No change. Looks like it’s not a statistics problem.</p><p>I’m out of theories: looks like my Postgres-fu has a blind spot. it’s time to pay a visit to <a href="http://postgresql.org/">postgresql.org</a> and <a href="http://stackoverflow.com/">stackoverflow.com</a> to learn more about query planning.</p><h3>Pulling the thread</h3><p>Honestly, neither the documentation nor Stackoverflow really gives me an epiphany. Nothing seems to explain how a given query on a given dataset would yield different query plans.</p><p>But I do smell a faint trail. Although it doesn’t seem to apply to my case, there’s something that keeps appearing through my research: prepared statements (eg <a href="https://dba.stackexchange.com/questions/289130/postgresql-changes-same-query-plan-at-runtime">this life-saving StackExchange thread</a>).</p><p>All over the internet, the <em>only</em> thing that seems to be related to inconsistent query plans is prepared statements. The StackExchange thread explains it well:</p><blockquote><em>PostgreSQL has a built-in heuristics for [prepared] statements whose plans can be cached: it plans them with the actual parameter values for the first 5 times, and then it switches to a generic plan if that plan promises not to perform worse</em></blockquote><p>That <em>would</em> explain it. If prepared statements are somehow involved, PG will take into account the actual value of the timestamp parameter for the first 5 executions: that&#39;ll be fast. From the 6th execution onwards, it doesn&#39;t consider what value timestamp actually has and always applies a generic query plan: this generic query plan is probably the same as I’ve observed with a different value for timestamp: it is much much slower and CPU-intensive.</p><p>Excellent theory! Except…</p><ol><li>Nowhere in the code does SpiceDB use prepared statements. The SpiceDB maintainers confirm they don’t use query plans.</li><li>Postgres “switches to a generic plan if that plan promises not to perform worse”, but the “generic plan” seems to perform <em>much worse.</em> We saw 28s execution time, and the CPU is maxed out: there’s no way this query plan is better even for different values of the timestamp filter: why would Postgres decide to switch to it?</li></ol><h3>Hidden prepared statements</h3><p>Let’s focus on the first open question: “nowhere in the code does SpiceDB use prepared statements”. I’m confident about that, but still… this behaviour of prepared statement matches my observations too closely, and nothing else does: my inner Holmes screams “<em>when you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth”.</em> Even if I don’t understand <em>how</em> prepared statements could be the cause, <em>only</em> prepared statements seem to explain what I’m observing*.*</p><p>Around then, I’m also noticing something I should have noticed much earlier. Let’s go back to the very start of my investigation, I identified this query as being the culprit for CPU load:</p><pre>SELECT MIN(id), MAX(id) FROM relation_tuple_transaction<br>WHERE timestamp &gt;= $1;</pre><p>Notice this $1? I didn’t really think anything about it, but now... It’s not saying now() - interval &#39;5s&#39;, even though I <em>know</em> that’s the value that’s always used, it says $1. And this $ thing is a prepared statement thing: that’s a parameter placeholder.</p><p>I don’t understand <em>how</em> it’s possible that prepared statements are involved, but by now I’m pretty sure that they are.</p><p><a href="https://stackoverflow.com/questions/70244423/different-execution-plan-for-local-postgresql-client-for-same-query">This StackOverflow question</a> which explains a similar problem gives me another clue. It mentions pgx, which is the Go library that SpiceDB uses. New hypothesis: is it possible that pgx automatically creates prepared statements for queries?</p><p><a href="https://stackoverflow.com/questions/70244423/different-execution-plan-for-local-postgresql-client-for-same-query">Yes it is</a>. The README mentions as a feature “automatic statement preparation and caching”. So while I was expecting (as did the SpiceDB maintainers) that calling pgx with a statement and parameters would result to something similar to libpq&#39;s <a href="https://www.postgresql.org/docs/13/libpq-exec.html#LIBPQ-PQEXECPARAMS">PQExecParams</a>, it actually creates a prepared statement, more akin to <a href="https://www.postgresql.org/docs/13/libpq-exec.html#LIBPQ-PQEXECPREPARED">PQExecPrepared</a>.</p><p>Something something leaky abstractions.</p><p>At this point, we had enough information for Jake from Authzed (the maintainers of SpiceDB) to manage a consistent reproduction of the problem: <a href="https://dbfiddle.uk/5N7GmTuz">https://dbfiddle.uk/5N7GmTuz</a>.</p><h3>Bad generic query plan</h3><p>Now we know where this bad query plan comes from. There’s still another open question: why would Postgres decide to switch to a generic query plan that’s much worse-performing?</p><p>We’ve actually already answered that: the cost estimation of this query plan is completely off, making it a candidate over the custom query plan despite being horrendously slow in reality. In other words, it <em>promises</em> not to be worse, but doesn’t uphold the promise. In other words, it’s a bug in the query planner.</p><h3>Working around the issue</h3><p>Fine, I know where this bad query plan comes from: it’s a prepared statement’s generic plan. We have multiple options to fix this in order from “easy but workaroundy” to “complex but addressing the root cause”:</p><ol><li>I can fix it for my particular deployment of SpiceDB. Luckily, one of the few things that the query planner <em>does</em> give you some amount of control over are the use of generic query plans. By enabling the setting plan_cache_mode to force_custom_plan, I completely disable the use of generic query plans database-wide: prepared statements will always use a custom query plan. It worked: CPU immediately drops to nothing and my response time goes from minutes to milliseconds. Woop woop.</li><li>Other SpiceDB Postgres users will encounter the same issue of course. SpiceDB can work around this issue by making sure that <a href="https://github.com/authzed/spicedb/pull/496/files">a prepared statement isn’t used</a> for this specific query.</li><li>None of this would have been a problem if Postgres’ query planner didn’t yield a bad query plan. It’s a bug of the query planner, <a href="https://www.postgresql.org/message-id/17540-7aa1855ad5ec18b4%40postgresql.org">it is now in the Postgres community’s hands</a>.</li></ol><p>This bug report went quickly over my head, but there was an agreement that there is indeed a problem in the query planner. A patch came out of it but as mentioned in the email chain, “<em>it’s just a band-aid that happens to help in this exact scenario. It’s not doing much for the bad cost estimation that’s the root of the problem</em>”.</p><p>Whilst it looks like this might not be fixed soon I hope this article has shown how Postgres performance issues can be debugged!</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us#jobs-live</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=778075ed7998" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/puzzling-postgres-a-story-of-solving-an-unreproducible-performance-issue-778075ed7998">Puzzling Postgres: a story of solving an unreproducible performance issue</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing structure and support for career development at Birdie]]></title>
            <link>https://medium.com/engineering-at-birdie/introducing-structure-and-support-for-career-development-at-birdie-41727828d0c3?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/41727828d0c3</guid>
            <category><![CDATA[systems-change]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jason Murphy]]></dc:creator>
            <pubDate>Tue, 13 Dec 2022 14:36:00 GMT</pubDate>
            <atom:updated>2022-12-13T14:36:00.093Z</atom:updated>
            <content:encoded><![CDATA[<p>If, like me, you’re a fan of 80s synth pop bangers <strong>and</strong> professional growth, then I think you’ll agree with me and The Korgis when we say <a href="https://www.youtube.com/watch?v=Q4huTptveEs">“Everybody’s Got To Learn Sometime”</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8knIOyqSSnVbEKiC" /><figcaption>Photo by <a href="https://unsplash.com/@itfeelslikefilm?utm_source=medium&amp;utm_medium=referral">🇸🇮 Janko Ferlič</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here at birdie, we understand that as knowledge workers the one aspect that underpins all of our jobs is to be continuous learners.</p><p>Over the last few months, as an Engineering Management function, we’ve put our focus on providing our early career software engineers with the best onboarding experience and support to facilitate their learning and development. I want to tell you about some of the processes and policies we’ve implemented and our approach to introducing those changes.</p><h3>Introducing changes to our systems</h3><p>When making changes to complex adaptive systems, the Cynefin framework can guide our approach. To quote <a href="https://lizkeogh.com/cynefin-for-everyone/">Liz Keogh’s excellent blog post about Cynefin</a>, “Human systems, particularly, are <em>Complex Adaptive Systems</em> or CASs; systems in which the agents of the system (people) can change the system itself. When we want to create changes in human systems safely, we have to be careful not to make those changes too big or sweeping, because we don’t really know what will happen in any given context, even if we have some good ideas.”</p><figure><img alt="The Cynefin Framework by Dave Snowden. CC BY-SA 3.0" src="https://cdn-images-1.medium.com/max/426/1*F_rJhbo9DQTo2eIT96V8Aw.png" /><figcaption>The Cynefin Framework by Dave Snowden. CC BY-SA 3.0</figcaption></figure><p>We should take the approach of “probe, sense, and respond”.</p><p>So we needed to identify a probe for the problem under consideration.</p><p>We work in a high-paced industry and profession where a lot is expected of us and when we’re in the early stages of our careers that can sometimes be overwhelming. Having spoken with our Intern and Associate Engineers, one thing that became immediately apparent was a desire for structure and time.</p><p>Having identified what to probe, we identified some actions to take. We went about designing some processes and policies that would address those needs. We broke them down into three categories:</p><ul><li>Explicitness</li><li>Resources</li><li>Deliberate learning and practice</li></ul><h4>Explicitness</h4><p>We tried to create some policies that would help early career engineers, and new joiners specifically, with their start to life at birdie. Some of these suggestions may seem basic, but they were gaps that people were reporting they found themselves in. The main idea is to take as much responsibility off of the new birdie as possible, so we create tailored onboarding plans for each person so they know what to do over the first weeks of their onboarding and what is expected of them. We also want to put the responsibility on each member of the team to reach out to the new joiner to make sure that lots of pair programming or ensemble programming happens hopefully reducing the “awkwardness factor” a new person experiences.</p><h4>Resources</h4><p>We know that the amount of topics and technologies that full stack software engineers in cross functional teams need to know is vast.</p><ul><li>Backend and frontend tools and technologies</li><li>Ops, deployment, monitoring, and alerting tools</li><li>Software engineering concepts such as Test Driven Development, Event Sourcing, Domain Driven Design to name but a few</li><li>Company specific tooling and processes</li></ul><p>Where to find good quality information for all of these topics isn’t always obvious. We’ve attempted to tackle this issue by having a collection of all the relevant topics for an engineer at birdie, and within those topics having curated resources (online courses, videos, blog posts, books).</p><p>We also signpost people within the company who are experts in particular topics so that anyone can ask them questions or set up some focussed mentoring sessions.</p><p>As well as this curated library of topics and resources, we also have company subscriptions for some learning platforms where people can do short, focused courses on anything they are currently working on, or would like to expand into.</p><h4>Deliberate learning and practice</h4><p>Being deliberate about how you go about learning and practicing something is very important.</p><p>Imagine you’re learning how to play your favourite song on the piano and you know that the chord progression is C#m7, A#m7, A6, C#m7, G#7, F#m9, B9, B7 (we all know what your favourite song is of course). While playing, you always make a slight mistake when moving from C#m7 to G#7. The approach you’ve decided to take to learn the song is to always play the full progression from start to finish. And now you practice. And practice. And practice. You put hours and hours of practice in. Your dedication cannot be questioned. You make the same mistake on the transition from C#m7 to G#7 most of the time. And after all those hours, you’ve completely internalised the mistake you always make when moving from C#m7 to G#7.</p><p>If you do lots of practice on something but in a non optimal way then that’s the way that you will continue to do it, mistakes included.</p><p>A better way of approaching this is to focus on what you’re trying to learn and achieve. Since the most immediate issue you have is the transition between C#m7 to G#7, taking 5 or 10 minutes every day just moving from C#m7 to G#7, again and again, is what you should be practicing.</p><p>It can sometimes be boring and feel like you’re not playing a song (which of course you’re not). You make mistakes but you correct them in the moment and practice what you’ve learned. You sometimes feel like you’re getting nowhere.</p><p>After a week or so, you find that you don’t make that mistake anymore and have built up the skills and muscle memory to nail that transition and when you play the extended progression you don’t make that mistake anymore! You’ve got to where you wanted to be, and in actuality in a much shorter timescale than by just trying to play the whole song again and again.</p><p>This principle of deliberate practice extends to any field: identify what you want to learn, zoom in on it, practice that, include it in your wider context.</p><p>Finding ways to nurture this approach within birdie was the final piece of our initial initiative to support the learning and development of our early career engineers and we’ve approached it in two ways.</p><p><strong>We’ve started a weekly “code club”</strong></p><p>We get together and deliberately practice some aspect of software engineering.</p><p>At the moment we’re practicing Test Driven Development. We take a coding kata and we practice the TDD approach</p><ul><li>Red (add a failing test case)</li><li>Green (make it pass in the <strong>simplest way possible</strong> — this is like moving from C#m7 to G#7 again and again — it can sometimes feel weird and forced)</li><li>Refactor (look for opportunities to “make it better”)</li></ul><p>We’ll continue practicing this for however long is appropriate, and then we’ll pick something else to deliberately practice.</p><p><strong>We’ve instituted a policy of ring fenced learning time</strong></p><p>All of our Intern and Associate Software Engineers have 1 day a week available to them where they can focus completely on learning and not have to worry about “delivery”. It should be related to the context of their squad and birdie, but within those constraints they are free to use it and approach it however they wish.</p><h3>Continuing to improve our systems</h3><p>Much like building software, building systems within a company is an incremental and iterative process. Having introduced these changes to our systems we need to constantly inspect and respond to what we learn so that we can, as Woody Zuill says, “Turn up the good” on what works and, by implication, drop what doesn’t work.</p><p>In the context of this case study, the policies and processes we introduce, remove, tweak, and experiment with will be always up for discussion and evolution but will be guided by birdie’s commitment to “providing our early career software engineers with the best onboarding experience and support to facilitate their learning and development”.</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us#jobs-live</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41727828d0c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/introducing-structure-and-support-for-career-development-at-birdie-41727828d0c3">Introducing structure and support for career development at Birdie</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Want to know how we query an incredible amount of data?]]></title>
            <link>https://medium.com/engineering-at-birdie/working-hard-ea2696c65183?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea2696c65183</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Tim Robinson]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 08:04:47 GMT</pubDate>
            <atom:updated>2022-12-01T08:17:02.796Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building scalable systems for an ever-growing dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mNeB4FcCoaWpPhM5" /><figcaption>Photo by <a href="https://unsplash.com/@smigielski?utm_source=medium&amp;utm_medium=referral">Adam Śmigielski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I never liked getting new shoes when I was younger. Out-growing my favourite pair of shoes and being forced by necessity to seek bigger ones carried a sort of painful inevitability to it that didn’t sit well with me.</p><p>Of course, I <em>did</em> always get bigger shoes when I needed to. The only alternative was to keep my oversized feet in undersized shoes, pretending that everything was fine, when in reality the shoes had stopped working for me long ago.</p><p>Companies, like people themselves, experience these sorts of growing pains all the time.</p><p>Whether it’s outgrowing a phyiscal office because you’ve just hired your thirty-eighth employee, or growing too large for your current systems to keep up with your customer base, all successful companies will eventually hit scalability problems of some kind.</p><p>If you’re a successful startup that’s followed the trendy advice to <a href="http://paulgraham.com/ds.html">do things that don’t scale</a>, then you’re perhaps especially likely to hit this wall.</p><p>But having to scale things up doesn’t have to be a problem, as long as you come prepared.</p><blockquote><strong>“Every problem is a gift — without problems we would not grow.” </strong>— Anthony Robbins</blockquote><h4>Off the scales</h4><p>For our engineering team at <a href="https://www.birdie.care/">Birdie</a>, addressing scalability had become something we were eager to tackle, before those loosely-realised concerns graduated into hard-and-fast problems.</p><p>As our user count and data stores grew, it became clear that the current setup simply wasn’t going to cut it. We were beginning to reach the jagged edge of how much our system could support. Thankfully, we’d spotted the early warning signs before our users did.</p><p>I’ve previously touched on why <a href="https://medium.com/engineering-at-birdie/how-we-made-our-app-80-faster-3c028aae4dd">keeping things performant is important</a>, but scalability isn’t just about making sure things don’t slow down.</p><p>It’s also about making sure things don’t collapse around you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7NiuPVtNDv9HVKwT19uvw.png" /><figcaption>Our Head of Platform, Ryan, highlighting warning signs of our scalability problems</figcaption></figure><h4>One log to rule them all</h4><p>There were several different fronts for the engineering team to start working on, from optimising queries, to reworking endpoints, and generous amounts of load testing.</p><p>My squad owned a very critical piece that needed to be looked at — the care log.</p><p>In many ways, the care log represents the very core of Birdie’s mission. It consists of a timeline of visit data, logged by caregivers using the Birdie app. It’s a live, ever-growing feed of the visits that have been recorded through Birdie.</p><p>The log itself is <em>vital</em> for the care agency to know what visits they’ve delivered, for family members to receive constant updates on their loved one’s wellbeing, and even for healthcare workers to know in an emergency what medications have recently been administered.</p><p>And because the care-log plays such a pivotal role, it gets an <em>exceptionally</em> high amount of traffic, which was very gradually starting to dampen our system’s performance.</p><p>Our Postgres database holding the care log information was coming under a lot of strain. One of the best parts about the care log is how flexible it is — you can filter for different visits, or different dates, perhaps visits delivered by a specific caregiver, or perhaps every caregiver across the last year.</p><p>Even with indexes and query optimisations, the <em>sheer weight</em> of new visit data being pumped in to the table every second, combined with the complex and numerous queries used to fetch that data, proved to be a recipie for serious growing pains.</p><p>Making these queries scalable and performant was the challenge our squad was facing. But how do you take such flexible queries, such large amounts of data fetching, and make it performant?</p><p>Enter… the <strong>Unified Log.</strong></p><h4>Elastic fantastic</h4><p>If you’ve ever gone looking for a fast way to query a lot of data, you’ll have almost certainly come across <a href="https://www.elastic.co/elasticsearch/">Elasticsearch</a>. I won’t be able to say it better than their website does:</p><blockquote>Elasticsearch is fast. Really, really fast.</blockquote><p>Sounds like it might be fast. We needed a solution that could be lightning-fast and properly scalable, and Elasticsearch fit the bill nicely.</p><p>But we couldn’t exactly spend time <em>replacing</em> our Postgres database with a brand new Elasticsearch setup. After all, we needed a solution more quickly, and ideally one that didn’t involve isolating the care log data from everything else.</p><p>Thus, the service known as the Unified Log (UL), a new query layer on top of our existing API, was conceived. Birdie loves an acronym, and the UL was no exception.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*bfPkjVrZzgTK9LbuZOlpxQ.png" /></figure><h4>The care log librarian</h4><p>Imagine you’re in a library, and you’re looking for something very specific. You’re seeking a history book on ancient Rome, with a red cover, that starts with the letter “R” and it was written some time between 1989 and 2016.</p><p>You start wandering the innumerable shelves with your specific criteria in mind. You’re not <em>completely</em> lost because you know it must be in the history section somewhere, but <strong>it’s going to take you a good amount of time to narrow it down.</strong></p><p>Now imagine instead of going out and attempting to find the book yourself, you present your specific criteria to a librarian, and query them for help. They tell you that you’re looking for the book “Rubicon” and take you to exactly where it’s sitting on the shelf.</p><p>The library is our database, the books are our data, and the ever-so-helpful librarian is our Unified Log.</p><p>When loading the care log, a request is made to our API’s endpoint. Previously, this endpoint would take any query params provided in the URL and use them to query the Postgres database directly, returning the resultant data to the user.</p><p>Now, instead of going directly to the main Postgres database, we instead make a request to the UL, which incredibly quickly calculates the IDs of the care log items that we need to fetch. Those IDs are then used to query the Postgres database, which is a <em>very</em> fast query to run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lyxZGP_V2VgCgaLCKZGtyw.png" /><figcaption>The CPU strain on our read-replica DB dropped dramatically after rolling out the UL</figcaption></figure><h4>Awesome! But how does it work?</h4><p>There were two main problems we had to tackle with this: Keeping the UL up-to-date with our Postgres DB, and safely making the switch to querying via the UL.</p><p>Birdie has long made use of a tool called <a href="https://kafka.apache.org/">Kafka</a> and <a href="https://docs.confluent.io/platform/current/connect/index.html#kafka-connect">Kafka Connect</a>. Without spending too long explaining what it is (I recommend reading about it, it’s a big <em>topic</em> — wink — in itself), Kafka is an event-streaming platform that allows us to send data across to our distributed systems.</p><p>Whenever care log data in our Postgres DB was created or updated, it would send the relevant DB row to Kafka. The UL service itself would then receive these messages from Kafka and use it to update its own internal records in its Elasticsearch cluster.</p><p>Thus the UL was kept up to date with any changes, and because it was only used for querying to fetch IDs, it only needed to hold minimal information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqQYvPk4KtMnF1zJ4R3KIw.png" /><figcaption>How a typical document in the UL’s Elasticsearch cluster is structured — minimal information stored.</figcaption></figure><p>The second problem was how to make the transition safely. How can we make sure the UL is returning accurate IDs?</p><p>We decided to adopt a pattern we call “shadowing”, which has become a popular method here to cautiously roll something out.</p><p>In shadow mode, the API endpoint would query the DB directly and return the results to the user just as before, but it would also send the same query filters to the UL. The user would not have to wait for this to complete so it wouldn’t affect performance time, but the IDs returned by the UL would be compared with those fetched directly from the DB.</p><p>Every now and then, a query would come along and produce IDs that didn’t quite match, which would be logged and flagged to our squad. We’d investigate, and fix the problem.</p><p>Eventually, after several consecutive weeks of idential IDs coming back from both the UL and the main DB, the UL was deemed ready to be rolled out. <strong>We flipped the switch, and it was live.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pNiZycqRq_gVznMZiwlDNw.png" /><figcaption>A rough idea of how our shadowing logic behaved</figcaption></figure><h4>The cornerstone of data querying</h4><p>So where are we now? These days, the importance of the Unified Log has only increased.</p><p>What started as a way of speeding up the care log has become the de facto way of querying log data in the Birdie system.</p><p>Initially only responsible for care log data, the UL is now also responsible for querying…</p><ul><li><strong>Alerts </strong>— These are issues with care delivery we’ve flagged to an agency that they might need to look into, anything from a scheduled visit that started late to a medication that hasn’t been taken.</li><li><strong>Actions — </strong>These are actions that the agency wishes to perform and keep track of, from making sure someone is assigned to onboarding a new client, to reviewing a client’s care plan every three months.</li><li><strong>Notes — </strong>These are any notes recorded on the system by the agency.</li></ul><p>The raw power of Elasticsearch through the UL has given us the ability to provide the ultimate tools an agency needs to audit their data.</p><p>Through a feature in Birdie called the Inbox, users can now shape any kind of query they desire, to surface a range of different data in one unified timeline as they see fit.</p><p>We even leverage our UL setup to automatically generate available filters on the frontend for our users to experiment with. But that’s a topic for another time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2-uYN1aCj3DKJQ5I1scLVA.png" /><figcaption>Our response times on the care log endpoint dropped impressively after the UL rollout</figcaption></figure><h4>Final thoughts</h4><p>All in all, the Unified Log has given us the stability and scalability we needed to keep pushing forwards.</p><p>In building it we not only improved the speed of our endpoints, but we also took considerable strain off of our database.</p><p>It was, and remains, by no means an obvious solution, but it’s brought about a service layer we now couldn’t live without.</p><p>Never underestimate where a bit of lateral thinking can get you.</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us#jobs-live</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea2696c65183" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/working-hard-ea2696c65183">Want to know how we query an incredible amount of data?</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I learnt as an Intern at Birdie]]></title>
            <link>https://medium.com/engineering-at-birdie/what-i-learnt-as-an-intern-at-birdie-9f65193d2b7f?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/9f65193d2b7f</guid>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[women-in-stem]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Hyosun Lee]]></dc:creator>
            <pubDate>Tue, 22 Nov 2022 09:39:34 GMT</pubDate>
            <atom:updated>2022-11-22T09:54:33.359Z</atom:updated>
            <content:encoded><![CDATA[<h3>What I learnt as an intern at Birdie</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEQW67c_X9lLl0mx5TegVA.jpeg" /></figure><p>I joined Birdie as an intern software engineer in July and it has been amazing so far, so I wanted to share how my intern life was and what I have learned along the way.</p><p>If you ask me to describe my first few weeks at Birdie, I’d have to say I was a symbol of imposter syndrome.</p><p>I graduated from a 16 week full stack development bootcamp and landed my first job here at Birdie. During the bootcamp, I was trained in a very fast-paced curriculum while having to digest multiple concepts in one day. After learning so much, I had this voice in my head saying, ‘It’s time to work on lots of tickets, time to show off your amazing skills and surprise everyone!”. But as you would expect, there was slim chance an intern who just started this month would smash all blockers, save all of the struggling seniors and be the hero. I was obsessed with the idea that I needed to make huge achievements or show people that I am an extremely capable person at work.</p><p>Of course, my hero fantasy didn’t happen. I spent my initial days working on through my onboarding plan. It wasn’t exactly ‘cracking-things-and-being-super-cool’ Hyosun but I was learning! Naturally, though, I started worrying. What if I am just a useless intern? What if I don’t meet their expectations at all? Shouldn’t I already know about these things? I decided to be honest and talk about this to my line manager. Right there, he gave me a very clear answer:</p><p>He said that as an intern, my job <em>is</em> ‘Learning’. Birdie expects me to learn from the experiences here and to grow. He said that I was following the onboarding process very well, showing a great attitude by being eager to learn and trying to catch up with the basic knowledge and tools we use. Most importantly, he encouraged me to keep sharing thoughts like this and asking questions and feedback. He told me to keep going and suggested documenting my learning journey so that I can have a sense of achievement from those visual goals.</p><p>It has been 5 months since then. Of course, from time to time, I want to challenge if I am going the right way or not but I don’t worry too much anymore as it motivates me to be a better version of myself. I have very supportive, open-minded squad members telling me ‘we got your back!’ and having a line manager who is willing to spend his time with me is incredibly helpful to resolve any concerns I have.</p><p>In Birdie, you can have lots of great opportunities to use for your growth. I took advantage of some of these cases during my internship and learned many things from them including what attitude I need to have as I grow.</p><p>Since I joined Birdie, I was assigned a tech mentor. We have weekly sessions every Thursday for an hour. We cover some concepts that I don’t understand or useful stacks for work. We explore them on our codebase or practice together by making some games.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*JuR749whAlXDUcRsWFjyBg.gif" /><figcaption>The game I built with my mentor</figcaption></figure><p>Every session is very informative and has lots of things to learn but one of the best parts I like is at the end of every project, we talk through what went well and what can be better. This is not only about the projects but also about mentoring as well. Even though he is a senior and my mentor, he always asks for feedback and tries to find a way to be a better engineer and a better mentor.</p><p>As another example, I was able to work with another squad for a week in the summertime. It was an interesting week for me because how they work and what they are working on was slightly different from our squad. The part that I want to mention is their ‘book club’ session. It is a weekly session to discuss and share your thoughts about certain topics that have been prepared a few days before such as a youtube video, an article, a book or some documents. The session is quite casual and everyone can freely share their opinions. I had a chance to attend once while I was working with Bills and I really enjoyed it! It was a valuable time to learn something new and to see how senior developers bring newness to their work and try to have dynamic insights.</p><p>Lastly, there is a session called ‘Lunch &amp; Learn’. Similarly, with the book club in the previous squad I mentioned, it is quite a casual atmosphere to share something that you are interested in or something new that you learned. The topics can be anything. Sometimes it is about a new interesting library, sometimes it is about design practices and sometimes it is about cultural differences. Having this session with various topics during lunchtime can be a short learning opportunity and a good stress refresher from work!</p><p>We have so many unplugged communities in Birdie including off_nerds, off_music, off_en_lang, jr_devs etc. My first community channel was women_in_tech. I still remember the feeling from my first day. I was looking around the channel before joining and saw that they were already welcoming me! It made me feel like I was already in the nest.</p><p>At 4pm on Friday every week, we have an hour weekly retro meeting and it starts with introducing and welcoming new Birdies. New Birdies introduce themselves with some uncanny facts about themselves and sometimes it is really funny which makes it easier for other Birdies to remember who you are. I was also welcomed so much at my first weekly retro meeting. A hundred people welcomed me like a family member. I felt like a celebrity.</p><p>Birdie has such a lovely company culture and atmosphere. I’ve been met with such kindness during my internship journey. Everyone at Birdie has always been so keen to help me, give advice and help me to grow which I am so grateful for. Especially my squad members! They have given me so much support and helped me to feel empowered. Thanks to my squad, I was able to overcome my fears and stop being afraid to get involved and be proactive. It would have been much more difficult to settle down if they were not there.</p><p>This has been my intern life so far, I hope this shows what working at Birdie as an intern actually looks like.</p><p>I started my time here at Birdie with a lot of self-doubts but after 5 months, I think I’ve grown up so much already. I can’t wait to grow up more and find out what the next chapters have in store for me.</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f65193d2b7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/what-i-learnt-as-an-intern-at-birdie-9f65193d2b7f">What I learnt as an Intern at Birdie</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop writing bad documentation… Write RFCs instead.]]></title>
            <link>https://medium.com/engineering-at-birdie/stop-writing-bad-documentation-write-rfcs-instead-f904d79c1cdb?source=rss----40f364345252---4</link>
            <guid isPermaLink="false">https://medium.com/p/f904d79c1cdb</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Emma Mitchinson]]></dc:creator>
            <pubDate>Tue, 15 Nov 2022 15:14:35 GMT</pubDate>
            <atom:updated>2022-11-17T11:21:39.960Z</atom:updated>
            <content:encoded><![CDATA[<h4>They’re not just a request for comment, but a request for collaboration and communication.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e8SBp2jwEYn6dbIo" /><figcaption>Sassy cat judges your lack of documentation — Photo by <a href="https://unsplash.com/@chmarco?utm_source=medium&amp;utm_medium=referral">Marco Chilese</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you’re working in the tech world, there’s a fairly decent chance that you’ve been tripped up or slowed down by bad documentation.</p><p>There’s tons of valid reasons why there might be incomplete, unclear, or outdated documentation for the feature, package, API or even app you’re trying to work with. Yet having this empathy for overworked and understaffed developers unfortunately doesn’t remedy the frustration you have to face. There’s no easy way to make time for writing documentation if time is already a constraint, and so it tends to take a backseat. But if there was a way to have a decent foundation for this neglected task, I’d like to think more of us developers would take it up.</p><h4>Here’s where the RFC comes in.</h4><p>An RFC is a way for us to explore and justify your technical decision-making to your current and future colleagues at the same time. A way to provide reasoning and context for future developers without littering your codebase with comments. An all-encompassing document outlining what problem needs to be tackled, what options were considered and exactly how it is going to be solved.</p><blockquote>Sounds like you want me to rewrite war and peace every time I want to write some code…</blockquote><p>I’ll admit that an ‘all-encompassing document’ sounds great and awful at the same time — how is anyone meant to dedicate the time and energy to reading you waffling on about refactoring your monolith repository for 18 pages front and back? So, while it’s important to provide sufficient context and detail for your readers, it’s also important to preserve the concision and clarity of the document, by ensuring there’s no repetition and, most importantly, making it a group effort!</p><h4>Transform learning within engineering teams</h4><p>During my first weeks at Birdie, I was subject to so much new information, I didn’t know what to do with it; while it was fascinating to learn so much, I needed a way to help solidify what I thought I’d learnt.</p><p>One of the ways I remedied this for an especially large and involved bit of tech debt was by writing the first RFC within Birdie that painstakingly listed out all the tasks we needed to complete to get the task done. At times, I was even referring to specific files within the codebase, and pointing out very particular assumptions I was making. By having this reviewed by my peers, both them and I were certain we were on the same page, and anything I didn’t quite understand was painfully obvious.</p><p>On reflection, it became clear that writing RFCs could be super helpful for any new starter, whether new to software engineering, or new to a team. It’s fairly common knowledge that it’s hard to know if you really understand a concept until you try to explain it to others, and I think RFCs are a prime way to do this.</p><h4>Transform communication within engineering teams</h4><p>Even for seasoned developers, RFCs are still a really valuable tool. Onboarding to a new team, and having to learn the entire stack can be daunting, even moving teams within the same company can involve learning a whole new product and stack. Having an encyclopedia of documents explaining all parts of the codebase could be indispensable for saving time and mitigating against gaps in knowledge.</p><p>We had a time interval-based parameter we needed to add to one of our entities; originally we had settled on saving a 6 monthly interval in milliseconds. The problem we’d overlooked is the user expectation of a 6-monthly interval. An event recurring every 6 months from the 1st of January would be expected to next occur on the 1st of July, which due to varying month length, is non-trivial to do. This is something we probably wouldn’t have realised until it was too late, had we not had a conversation off the back of an RFC comment that challenged our decision.</p><h3><em>What’s the difference between an RFC and ADR?</em></h3><p>I’ve seen the use of these two acronyms used interchangeably quite frequently, despite the fact that…</p><ul><li><em>ADRs are </em><strong><em>short records of decisions</em></strong><em> that have already been made.</em></li><li><em>RFCs are </em><strong><em>longer</em></strong><em> </em><strong><em>exploratory</em></strong><em> </em><strong><em>documents</em></strong><em>, for getting feedback on any plans and for making decisions. Generally, they’re the precursor for an ADR.</em></li></ul><p>I’m an advocate for both; I believe they both have a place within software engineering.</p><p>Having said that, there seems to be a lot of confusion between the two and this can result in the incoherence of a set of documentation and uncertainty — should I be questioning the decision made here? Has the decision already been made? Are you asking for my feedback or just letting me know what you’ll be doing?</p><h3>Writing an effective RFC</h3><p>Get your colleagues involved! Don’t make your RFC your little baby- encourage challenges to your approach and any questions or alternative solutions. The size and reach of the project should govern who else needs to give their input — for a small piece of tech debt, there’s no need to get the whole team involved, potentially only those who have dealt with something similar, or who wrote the code originally. Try and set a feedback window, so you’re not getting any doubts right before you’re about to start actually building what you’ve outlined.</p><p>Having a peer review process also helps ensure that there’s no missing context — if a reviewer doesn’t understand what they’re reading, chances are another developer looking back after a few months trying to understand the problem space, won’t either.</p><p>A faux pas of RFC writing can be a lack of structure in the document. A wall of text with seemingly random titles and headings that once made sense but now merely add to the confusion. Sticking to a template can help mitigate against this.</p><p>There’s also the foe of a difficult-to-find RFC. At Birdie, we store our RFCs within a Notion database, and this means we always know where to find them. Having said that, Notion’s search functionality leaves a little bit to be desired, so finding a mention of a specific word or phrase can be super trying. It’d be a good idea to try and add tags or categories to the RFC, maybe outlining the context, the tech used, or the feature in question, in order to make it easier to find later.</p><h3>The structure of a Birdie RFC</h3><p>Using a document template can make writing RFCs easier for you and your team. Instead of a blank sheet, being presented with something you can fill in the gaps for, can reduce the time and effort needed. Always keep in mind who the audience is, remember the reader might not always have full context on the problem space, and without this, it’ll be hard for them to give any sort of meaningful feedback.</p><p><strong>Title<br></strong>The title of an RFC should be a precursor to your context. Try to avoid acronyms, unless they’re widely used within the industry and will be correctly interpreted by all engineers, veterans, and newbies. Try and think of what you’d search for when trying to find this RFC in 6 months time.</p><p><strong>Context<br></strong>Akin to the abstract on a research paper, this should be a short overview of what you’re going to address in this document. Outline what the initial problem is, whether it’s a feature, an optimisation, or some technical debt, and what you hope to achieve or resolve following this bit of work. It might also be good to outline what sort of feedback, if any, is needed.</p><p><strong>Options<br></strong>In order for your readers to understand why the final plan of action was chosen, it can be useful to add quick summaries of what other approaches could have been to tackle the problem, what the pros and cons were for those methods and why they inevitably weren’t the way to go.</p><p><strong>Plan of Action<br></strong>The chunky bit. This is a breakdown of the tasks required to complete this bit of work and solve the problem. Whether it’s an overarching plan of the architecture of an app or broken down step by step into tickets, it’s important that you convey the approach in a logical and easy-to-understand format.<br>By breaking your large piece of work down at this point, not only does writing tickets become easier, but it can also help uncover any potential technical implementation questions, and consequently it can double up as your technical refinement.</p><p><strong>Next steps<br></strong>Out of the scope of this bit of work, what, if anything, are you going to be working on next? By adding this detail, it can help give even better context</p><p><strong>Questions<br></strong>Give your readers the option to add questions they want other reviewers’ opinions on, or add outstanding sources of concerns that you’d like the input of others on before you can begin the work</p><p><strong>Appendix<br></strong>An ideal place to list any references, sources or links out to conversations and other planning resources.</p><h3>The ideal RFC…</h3><p><strong>….ensures I am super confident in the work I’m doing, because I know we’ve collaborated with the whole team to come up with the most efficient and effective solution to our problem.</strong></p><p><strong>…allows me to be certain that I or anyone else could explain or lead the work, because the whole team have had the opportunity to read, reflect and fully understand the work we’re about to undertake and how we’re going to do it.</strong></p><p><strong>…is available to refer back to it in a year’s time when I will be able to fully grasp the context of the problem, and be reminded why we took the actions we did.</strong></p><p>Once a clear, concise, yet thorough RFC is complete, the author has received enough feedback and the team should be aligned and comfortable with the approach that has been outlined, and confident that it’s the best solution possible given the time and resources available.</p><p><em>We’re hiring for a variety of roles within engineering. Come check them out here:</em></p><p><a href="https://www.birdie.care/join-us#jobs-live">https://www.birdie.care/join-us</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f904d79c1cdb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-birdie/stop-writing-bad-documentation-write-rfcs-instead-f904d79c1cdb">Stop writing bad documentation… Write RFCs instead.</a> was originally published in <a href="https://medium.com/engineering-at-birdie">Engineering at Birdie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>