<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dave Allie on Medium]]></title>
        <description><![CDATA[Stories by Dave Allie on Medium]]></description>
        <link>https://medium.com/@daveallie?source=rss-f2e2079965e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*P0xuoLSw4XEFZ0enA4N0tQ.jpeg</url>
            <title>Stories by Dave Allie on Medium</title>
            <link>https://medium.com/@daveallie?source=rss-f2e2079965e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:00:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@daveallie/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why startups are career accelerators]]></title>
            <link>https://medium.com/visibuild/why-startups-are-career-accelerators-1da90f7b11d?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1da90f7b11d</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 03:22:37 GMT</pubDate>
            <atom:updated>2022-07-26T03:22:37.248Z</atom:updated>
            <content:encoded><![CDATA[<p>Startups are fast-moving versions of their (much) older relatives — big corporates (or as our CPO Shaun likes to say, “hella corps”). Working at a startup and getting your hands dirty with new tools, directly impacting the success of the business, and exposing yourself to other areas of a business is a surefire way to accelerate your hard skills, your understanding of the big picture, and your career.</p><p>Thinking about potentially joining a startup? Read on. Absolutely convinced that big-corps are the way and not interested in startups? Read on and let me convince you otherwise.</p><h3>Gain exposure to new tech</h3><p>Startups are not anchored to tooling, language, or approach requirements, and encourage thinking outside the box where product differentiators are involved. Moving fast and freely is in their best interest, so focusing on a customer problem and the technical approach to solve it is often more important than making sure everyone is using the exact same outdated libraries.</p><p>Startups additionally allow you to build the critical context of when to use a tool or an approach, not just what that tool is, through debate, experimentation, or exposure. This experience and knowledge will allow you to take your learnings between industries and companies, creating clean system design, correct tooling choices, and knowing where to start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9XmM2JIpfbv7gKJ-YiKcRg.png" /><figcaption>Shaun presenting an idea for Keanu Reeves style Matrix view</figcaption></figure><p>Larger companies focus on homogeneity, forming an extremely well-trodden path upon which a set of well-known approved tools are used. They discourage (or even prevent) the ability to stray from that path. While it is possible to gain new exposure by hopping between teams periodically, this comes at a cost in the form of abandoning your team and starting over, or alienating others to the point of no mutual trust. Even within this homogeneity, big companies have a build-up of legacy baggage and complexity, drastically reducing their ability to move freely. Building a new small service within a massive network of outdated, incompatible, or incomplete micro-services isn’t fast or free. Don’t believe me? Watch this <a href="https://www.youtube.com/watch?v=y8OnoxKotPQ">too real skit</a>.</p><h3>Be heard, make your mark</h3><p>Startups encourage everyone to promote important and impactful problems for customers. At Visibuild, we use a combination of weekly WIP (work in progress) chats and Problem Docs (more on Problem Docs in a future blog post!) to help us communicate the root of the customer problem, why it’s important, and some potential ways to address it. These problems and solutions are synthesised from talking with customers, demoing prototypes, and internally iterating on ideas. From the point of a problem doc, building out and delivering the solution back to customers closes the loop from ideation to delivery. Being involved in helping define what’s important to the product and company both improves your critical thinking when it comes to product direction and is a whole lot more fun than just churning through tickets day after day.</p><p>Startups breed a culture of experimentation and ideation through the use of hack days, innovation sessions, or <a href="https://www.thesprintbook.com/the-design-sprint">design sprints</a>. These sessions focus on pushing boundaries by exploring new ways to approach and solve problems and are a core mechanism for product and personal growth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hVizirnYbrTLA2O9" /><figcaption>Visibuild team design sprint for a redesigned home page — <a href="https://medium.com/visibuild/what-makes-good-products-great-make-things-personal-72a8fb7ad8ac">see Renee’s thoughts on home page personalisation</a>.</figcaption></figure><p>Getting a hand into the planning process and getting your ideas executed at big tech companies is nigh impossible. 2-pagers, 6-pagers, PR/FAQs, impact estimation, effort planning, review meetings, and cross-team alignment, all to get to the point of your idea being deprioritised for the upcoming 12 months of planned work. Better luck next time! Profit margins, long-distance planning, and big egos all conspire to make it punishing to see your ideas out in the world. Experimentation at big corporates is also tough. Generally, managers are not incentivised to experiment. They commit their teams to large potions of work in advance and work through that backlog over the year, typically without built-in slack for much else. At these big companies where everything is decided by financial impact, experimentation (which is difficult to quantify) is routinely undervalued and deprioritised.</p><h3>Understand how the sausage gets made</h3><p>In a startup you work extremely close with customers, and the barriers that are usually in the way are removed. You don’t need to ask anyone’s permission or fill in any forms to talk to your users. There is a massive difference between talking to customers directly to understand their problems, and hearing that same information second or third-hand.</p><p>Drawing your own conclusions from customers, instead of blindly following one person’s opinion is what makes a diverse and complete product. Following that customer discussion through product ideation to completion and taking part in the entire journey is something uniquely startup. Peeling back the layers of the business, and understanding how each part functions will help you communicate and work more closely with those areas in your startups (or other businesses) in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dc2vvhNQBRDnH9_t2pc6cA.jpeg" /><figcaption>Getting out to site to talk to customers</figcaption></figure><p>Understanding customers in large companies is like playing a game of Telephone, by the time the feedback gets to you, it has been reshaped by a bunch of people, twisting the original intent. Direct access to customers is often impossible. Beyond that, you’re discouraged from straying too far from your prescribed path, you’re either hired as a specialist or shaped into one. Keeping a foot in multiple camps, or exploring the process within the rest of the business is not typically possible.</p><p>Feeling stagnant, or think that the issues with hella corps mentioned hit a little too close to home? Then maybe it’s time for a change of pace, maybe it’s time to join a startup.</p><p>Thinking about stepping on the gas and accelerating your career? <a href="https://www.linkedin.com/company/visibuild-pty-ltd/jobs">Visibuild is hiring</a>! Feel free to visit our LinkedIn jobs, or to reach out to me directly!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1da90f7b11d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visibuild/why-startups-are-career-accelerators-1da90f7b11d">Why startups are career accelerators</a> was originally published in <a href="https://medium.com/visibuild">Visibuild Product Team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ULIDs and Primary Keys]]></title>
            <link>https://medium.com/@daveallie/ulid-primary-keys-7620b0da1a78?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7620b0da1a78</guid>
            <category><![CDATA[ulid]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[uuid]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Tue, 15 Feb 2022 21:30:39 GMT</pubDate>
            <atom:updated>2022-02-16T01:31:50.187Z</atom:updated>
            <content:encoded><![CDATA[<p>When it comes to picking the type for your database’s primary keys, there are a few divided camps. When making this decision for <a href="https://www.linkedin.com/company/visibuild-pty-ltd/">Visibuild</a>, I was torn between the simplicity of sequential IDs and the longevity/future benefits of non-sequential IDs.</p><p>Ultimately, I decided to go with non-sequential IDs to make it easier to deal with sharding and regional databases down the line. Although, even after I decided to go with non-sequential IDs, that wasn’t the end of it. By its very nature, the bucket of “non-sequential ID” types is all possible ID types except sequential ones, and unsurprisingly there are a lot of different options out there.</p><p>This is a bit of a comparison of the few front-runners I looked at, including the one I landed on (ULID), and how I implemented it for our PostgreSQL database.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href">https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href</a></iframe><h3>UUID</h3><p>Let’s start where most people do, with UUIDs.</p><p>UUIDs (universally unique identifiers) are 128-bit labels typically represented as a hexadecimal string broken up into 5 sections containing 32 bits, 16 bits, 16 bits, 16 bits, and 48 bits respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w7s-hL448fnUCNi5.png" /></figure><p>Within the UUID standard (which you can read about in <a href="https://datatracker.ietf.org/doc/html/rfc4122">RFC 4122</a>), there are 5 different versions of that same 128-bit label. The differences between the versions are which inputs are required to generate the UUID and the structure of the bits that are output.</p><p>Version 1 requires knowledge of the computer’s MAC address to generate the label. Version 2 is only guaranteed to be unique if at most one is generated per computer about every 7 minutes. Versions 3 and 5, are deterministic based on the supplied input. This leaves only version 4 as a suitable choice for scalable, non-deterministic UUIDs.</p><h4>UUIDv4</h4><p>When people say UUID, they’re almost always referring to UUIDv4. It’s the most prevalent and widely supported UUID standard, even the example UUID given above is a UUIDv4. As UUIDv4 is based solely on randomness, it is extremely portable and can be used with very little prior knowledge about the state of the system.</p><p>UUIDv4 are composed of <strong>122</strong> bits of randomness, and <strong>6</strong> bits of version/variant identification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*esUXlhXz2g5ohXbl.png" /></figure><blockquote>This version isn’t interactive, if you’d like to play around with generating UUIDs check out this post on <a href="https://blog.daveallie.com/ulid-primary-keys">my blog</a>.</blockquote><p>The only non-random sections of a UUDv4 are the version and variant. The version is represented by 4 bits and is always set to 4 for UUIDv4, you can see it highlighted as the first orange nibble above. The variant is represented by 2 bits and is set to 2 for the UUIDv4 standard, you can see it highlighted as the second orange nibble above.</p><p>The variant is only the first 2 bits of the second highlighted nibble. While the remaining 2 are random, meaning the value of that nibble can be 8 ( 1000), 9 ( 1001), a ( 1010), or b ( 1011)</p><p>UUIDv7 and its siblings (UUIDv6 and UUIDv8) are the new hotness in the UUID space. They are currently still in a <a href="https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format">RFC Draft</a>, which if approved, would join version 1 through 5 as part of the UUID specification.</p><p>The three new versions are all based on a similar structure of a combination of time bits in the most significant positions, followed by random bits. The difference between the three is which kind of clock is used to generate the time bits. Version 7 from the draft is using Unix Epoch time, making it the likely candidate for adoption in the future.</p><p>Even within version 7, there are a few different ways to represent the time. Different levels of sub-second precision can be used, but the more precise in time you get, the fewer random bits you’ll be able to use. For the rest of this article, I’ll just be looking at the millisecond precision setup of UUIDv7.</p><p>With the millisecond precision setup, there are <strong>48</strong> bits of time, <strong>6</strong> bits of version/variant, <strong>12</strong> bits for a monotonic sequence, and <strong>62</strong> bits of randomness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HOfkl-eVGKMB0g_M.png" /></figure><blockquote>This version isn’t interactive, if you’d like to play around with generating UUIDs check out this post on <a href="https://blog.daveallie.com/ulid-primary-keys">my blog</a>.</blockquote><h4>UUIDv7</h4><p>UUIDv7 always dedicates the first 36 bits to the seconds of the timestamp, and then a variable amount of bits for the sub-second precision, in our case that’s 12 bits for the milliseconds within the second.</p><p>If you click “Regenerate” a few times, you should see the first 9 characters of the timestamp only change once a second, these are the bits representing the Unix Epoch time in seconds. The last three bits are the milliseconds within that second.</p><p>The version and variant follow the same setup as UUIDv4, however the version is set to 7.</p><p>The main value for the timestamp at the front of the UUID is that UUIDv7s (and the other new UUIDs in the draft) are monotonically increasing (within the same sub-second), meaning that a UUIDv7 generated now is always going to be larger than a UUIDv7 generated a millisecond ago.</p><p>You’ll also notice a new kind of section in UUIDv7, the “monotonic sequence” section. This section of 12 bits is designed to ensure that the UUIDs generated within the same sub-second are also monotonically increasing. In our case, for each subsequent UUIDv7 generated within the same millisecond, the monotonic sequence counter will increase by one. Note that this is only relevant for UUIDs generated on the same machine within the same millisecond, it will not ensure any ordering of UUIDs if they’re generated independently of each other in the same millisecond.</p><p>Here are some example UUIDs generated in the same millisecond:</p><ul><li>061ff8d8-e24b-7000-8092-ca1e5440d491</li><li>061ff8d8-e24b-7001-b653-1c41e471cd78</li><li>061ff8d8-e24b-7002-9bc2-82b5da559f1d</li></ul><p>When using UUIDv7 as a primary key, you get the same sortability as a sequential ID with the flexibility of client-side/distributed generation. If you’re looking to do a performant sort on creation time using UUIDv4, you’d need a separate indexed creation time column, but if using UUIDv7 you can use the already indexed primary key as the sort column.</p><h3>ULID</h3><p>In late 2017, almost 4 years before the UUIDv7 specification was drafted, the Universally Unique Lexicographically Sortable Identifier (or ULID) was proposed, not as an RFC Draft, but as a grassroots proposal over GitHub. You can take a read of the (quite concise) <a href="https://github.com/ulid/spec">specification for ULID</a> on GitHub.</p><p>A ULID is a 128-bit label, just like a UUID. It’s sortable, has millisecond precision, and is monotonically increasing, just like UUIDv7.</p><p>ULIDs are typically represented as a <a href="http://www.crockford.com/base32.html">Crockford’s Base 32</a> encoded string, instead of a hexadecimal string like UUIDs. As an example, instead of 017eb31e-1440-b69e-d82f-5f0937f823c8, the same value can be represented as 0GWWXY2G84DFMRVWQNJ1SRYCMC.</p><p>For the purpose of comparison with UUIDs, I’m going to represent all ULIDs as hexadecimal strings, in the same 8, 4, 4, 4, 12 segments which we have been using.</p><p>Within a ULID, there are <strong>48</strong> bits of time and <strong>80</strong> bits of randomness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sAb5mquDEQx7KE_g.png" /></figure><blockquote>This version isn’t interactive, if you’d like to play around with generating UUIDs check out this post on <a href="https://blog.daveallie.com/ulid-primary-keys">my blog</a>.</blockquote><p>While there are 48 bits of time information in both UUIDv7 (with millisecond precision) and ULID, the ULID standard encodes the time as Unix Epoch in milliseconds for the whole 48 bits whereas UUIDv7 splits the time into 36 seconds bits and 12 millisecond bits.</p><p>ULIDs also remove the bits pertaining to the version and variant, giving us 6 extra bits to play with but removing all indication that ULIDs are in fact ULIDs. Because UUIDs encode version and variant information inside the label, they can be reliably decoded into their composite parts (e.g. timestamp and randomness). If you wanted to extract the timestamp from a ULID, the client would need to know that the string being processed is a ULID, and not an invalid (or potentially valid looking) UUID.</p><p>ULIDs are monotonically increasing, but there is no defined monotonic counter like in UUIDv7. Instead, ULIDs generated in the same millisecond on the same machine will have sequential randomness sections.</p><p>Here are some example ULIDs generated in the same millisecond:</p><ul><li>017ece40-2a1e-63ac-a58d-e336f30c1d76</li><li>017ece40-2a1e-63ac-a58d-e336f30c1d77</li><li>017ece40-2a1e-63ac-a58d-e336f30c1d78</li></ul><p>The main differences between UUIDv7 and ULID are:</p><ul><li>ULID assumes client knowledge if decoding is required</li><li>UUIDv7 handles several timestamp types, and as a result uses those bits less effectively</li><li>UUIDv7 encodes the version and variant information inside the label</li><li>UUIDv7 will work until 4,147AD whereas ULID will work until 10,889AD</li><li>ULID monotonicity is built into the random bit instead of as standalone bits</li><li>ULID has 80 random bits vs 64 random bits for UUIDv7</li></ul><h3>Implementing UUIDs/ULIDs in PostgreSQL</h3><blockquote>This section is taken through the lens of PostgreSQL, if you’re using a different database engine then you’ll likely need to look elsewhere to understand how to implement these kinds of primary keys.</blockquote><p>While all of these UUID/ULID formats can be generated by the client before inserting them into the database, for the purpose of simplicity and consistency, having them be generated within the database engine is preferred.</p><p>Thankfully, PostgreSQL includes a <a href="https://www.postgresql.org/docs/14/datatype-uuid.html">uuid data type</a> which accepts a case insentive 128-bit hexadecimal string, parses it into binary data and stores it as 16 bytes of data. This is drastically better than storing the value as a string in the database, which would take up 37 bytes of space including the dashes, or 33 bytes of space with the dashes stripped out.</p><p>Here are some implementation details for populating those uuid columns and setting up tables with UUID/ULID primary keys:</p><h4>UUIDv4</h4><p>PostgreSQL has built in support for UUIDv4 through the pgcrypto or the uuid-ossp extensions.</p><pre>CREATE EXTENSION IF NOT EXISTS pgcrypto;<br><br>SELECT gen_random_uuid();<br>--&gt; f449a5bc-a221-4e9d-8819-e7f22b83d8ae</pre><p>This makes it extremely easy to setup a table with a UUIDv4 primary key:</p><pre>CREATE TABLE my_uuidv4_things(<br>  id UUID NOT NULL DEFAULT gen_random_uuid(),<br>  name TEXT NOT NULL,<br>  PRIMARY KEY (id)<br>);<br><br>INSERT INTO my_uuidv4_things(name) VALUES (&#39;foo&#39;);<br><br>SELECT * FROM my_uuidv4_things;<br>--                  id                  | name<br>----------------------------------------+------<br>-- 8501364b-b669-4c17-bd98-00bad8cd8f7d | foo</pre><h4>UUIDv7</h4><p>There don’t seem to be any existing built-in or extension functions that support generating UUIDv7s in PostgreSQL. To generate a UUIDv7, the function below could be tweaked in order to support the correct formatting. I haven’t gone ahead and done it here, because I was more invested in ULIDs by this point.</p><h4>ULID</h4><p>There are a few different implementations of ULIDs in <a href="https://github.com/iCyberon/pg_ulid">Go</a>, or in <a href="https://github.com/geckoboard/pgulid">plsql</a>. However, both of these implementations return the Crockford Base 32 text representation of ULID (taking up 26 bytes), instead of as a UUID datatype (taking up 16 bytes). Due to the natural multiplexing nature of PostgreSQL (and RDBMSes in general), having a consistent, shared monotonic counter is difficult, if not impossible. The <a href="https://github.com/oklog/ulid">Go library</a> used by the Go PostgreSQL implementation includes an implementation of the monotonic counter, however, it’s not used by the Go PostgreSQL extension. The plsql implementation doesn’t have a monotonic counter at all.</p><p>I wanted a simple ULID PostgreSQL implementation without the complexity of the monotonic counter that generated a UUID data typed value. So I wrote one.</p><p>Here it is:</p><pre>CREATE EXTENSION IF NOT EXISTS pgcrypto;<br><br>CREATE OR REPLACE FUNCTION generate_ulid() RETURNS uuid<br>    AS $$<br>        SELECT (lpad(to_hex(floor(extract(epoch FROM clock_timestamp()) * 1000)::bigint), 12, &#39;0&#39;) || encode(gen_random_bytes(10), &#39;hex&#39;))::uuid;<br>    $$ LANGUAGE SQL;<br><br>SELECT generate_ulid();<br>--&gt; 017eb31e-1440-b69e-d82f-5f0937f823c8</pre><p>Breaking down the function:</p><ul><li>The first half lpad(to_hex(floor(extract(epoch FROM clock_timestamp()) * 1000)::bigint), 12, &#39;0&#39;) gets the milliseconds since the Unix Epoch, and converts it to a hexadecimal string, of length 12 (it&#39;ll have a leading 0 for the next 500 years or so).</li><li>The second half encode(gen_random_bytes(10), &#39;hex&#39;) generates 10 random bytes and converts them to hexadecimal.</li></ul><p>Together these, make the 16-byte (or 128-bit) ULID label, which is finally cast to a UUID data type with ::uuid.</p><p>Our new function can now be used as a default value for a primary key:</p><pre>CREATE TABLE my_ulid_things(<br>  id UUID NOT NULL DEFAULT generate_ulid(),<br>  name TEXT NOT NULL,<br>  PRIMARY KEY (id)<br>);<br><br>INSERT INTO my_ulid_things(name) VALUES (&#39;foo&#39;);<br><br>SELECT * FROM my_ulid_things;<br>--                  id                  | name<br>----------------------------------------+------<br>-- 017eb31e-1440-b69e-d82f-5f0937f823c8 | foo</pre><h3>Benchmarks</h3><p>So how does it perform?</p><p>When compared to the native implementation of gen_random_uuid(), generate_ulid() performs <strong>73% slower</strong> when <strong>generating</strong> 10 million values, but only <strong>31% slower</strong> when <strong>generating and inserting</strong> 1 million values.</p><p>Meaning that while ULIDs are slower to generate (at least with this implementation), they are much faster to insert. My guess is that due to the sequential and highly clustered significant bytes of the ULIDs, they are much faster to create the index entries for. Meanwhile, the UUIDv4s are extremely sparse, so they are much slower to create the index entries for.</p><p>Here’s how I got these numbers:</p><p>createdb ulid-test</p><pre>EXPLAIN ANALYSE SELECT gen_random_uuid() FROM generate_series(1, 1000000);<br>-- 6766.681ms<br><br>EXPLAIN ANALYSE SELECT generate_ulid() FROM generate_series(1, 1000000);<br>-- 11750.966ms</pre><pre>CREATE TABLE uuid_keys(id UUID);<br>CREATE TABLE ulid_keys(id UUID);<br><br>EXPLAIN ANALYSE INSERT INTO uuid_keys(id)<br>SELECT gen_random_uuid() FROM generate_series(1, 1000000);<br>-- 1372.470ms<br><br>EXPLAIN ANALYSE INSERT INTO ulid_keys(id)<br>SELECT generate_ulid() FROM generate_series(1, 1000000);<br>-- 1803.472ms</pre><h3>Conclusion</h3><p>Using ULIDs as database primary keys comes with a performance cost. Yet, it’s one I’m more than willing to pay in order to get the benefits of a sortable, globally unique identifier.</p><p><em>Originally published at </em><a href="https://blog.daveallie.com/ulid-primary-keys"><em>https://blog.daveallie.com</em></a><em> on February 16, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7620b0da1a78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So long, Amazon]]></title>
            <link>https://medium.com/@daveallie/so-long-amazon-99d0d2ae409e?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/99d0d2ae409e</guid>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Fri, 30 Jul 2021 00:00:00 GMT</pubDate>
            <atom:updated>2022-02-16T01:22:09.075Z</atom:updated>
            <content:encoded><![CDATA[<p>This month was my last at Amazon. The last two years of building software at mind-boggling scales for Amazon customers around the world has been engaging, challenging and insightful. I’m sad to have hung up my badge and walked out the doors for the last time, but thankful for the knowledge I’ve picked up and the advice I’ve received along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JWBpyW9wtzpg7T2n.jpg" /></figure><p>I thought I’d take the time to reflect on some of the best and worst parts of life at Amazon as an engineer. While it doesn’t come at all close to a comprehensive list, and may differ greatly from other people’s ups and downs, these are the aspects that resonated with me or irked me during my time there.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href">https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href</a></iframe><h3>The Great</h3><p>Before starting at Amazon 2 years ago, I had some preconceived notions of how Amazon built, launched, and maintained software. I was picturing a slow, lumbering giant, filled with many small, and easily forgotten cogs. With the overall machine struggling to course-correct due to its own inbuilt inertia. However, throughout the interview process, and in my first few days, I was completely purged of my delusions.</p><h4>Decision Making and Ownership</h4><p>Amazon splits high-level business focuses into separate areas, each with its own clear and distinct owners. This split of responsibility is repeated again and again until you’re left with an atomic product owned by a product manager. For the majority of my time at Amazon, I was working in the grocery space, focusing on the technology and experience of substitutions for a customer’s items. This split of responsibility and clear ownership meant that our team’s output had a single point of approval, our PM. There was always a single person responsible for ensuring that what we were building was moving us towards our end goal and any deviations in our projects could be approved without escalating all way up to Jeff.</p><p>This is the true representation of decentralised decision-making in practice. Ownership is effectively delegated and owners are empowered to independently make decisions on behalf of their product space. As an engineer, it was extremely relieving that in a company as big as Amazon, I only needed to convince my team and one or two other people in order to get things done. I have previously witnessed attempts at decentralised decision-making where owners attempt to involve everyone in all decisions and delegate ‘decision owners’, but still want a strongly weighted voice in the decision. This always resulted in huge amounts of confusion around ownership, and a risk of accidental undermining.</p><h4>Amazon’s Experts</h4><p>The availability of extremely smart, and helpful people at Amazon was amazing. I had direct lines of communication to people with an insane amount of experience across the whole software development lifecycle, with expertise in just about every area. There’s bound to be at least a few experts at your company or within your development team, wherever you work. The Encyclopedia Britannicas of the tech, tooling, and processes you use on a daily basis. The people you go to as a last resort once every other well of information has come up dry, who always seem to have the answer. Now imagine expanding your team to hundreds of thousands of people, and scale the number of experts as well. That’s what it’s like working at Amazon. If you have a question, about literally anything, chances are you can find an expert at Amazon, and they’re just a DM or an email away.</p><h4>Leadership Principles</h4><p><a href="https://www.amazon.jobs/en/principles">Amazon’s Leadership Principles</a> are thoughtfully considered and unironically embodied by everyone within the company. They aren’t just plastered to the wall and paid lip service. They reflect both the values of the employees and the company. It could just be brainwashing, but I truly believe that the leadership principles are useful core tenants to lean on in all parts of Amazon. They are used during 360° reviews, hiring, promoting, and are considered during almost every critical decision point. Including system architecture designs, launch decisions, and feature prioritisation. Having a common framework for backing up thoughts and decisions which represent the best outcome for the company meant that disputes were infrequent, and when they did come up, you were assured that the other side of the argument was trying to fulfill the same criteria as you.</p><h3>The Not So Great</h3><p>While there was a lot of great things at Amazon, there were some inescapable not so great things about working there too.</p><h4>Golden Handcuffs</h4><p>Leaving money on the table is never a good feeling, especially when a large portion of that money comes in the form of <a href="https://www.google.com/finance/quote/AMZN:NASDAQ?window=5Y">ever-growing Amazon stock</a>. Amazon has perfected the process of fitting golden handcuffs to their employees, through a combination of a high base, and a generous amount of stock units. It’s pretty clear how the base component works, they just pay equal or more than most employers in the cities they operate ( <em>at least for corporate employees</em> 😓), the catch comes with the stock units. Amazon awards the stock on a rolling schedule years before they vest. So if you do decide to leave, you’re going to be walking away from a few years worth of promised and pre-awarded stock units. Personally, I found it a tough pill to swallow, but I wasn’t dependent on those future earnings, so I just stopped considering un-vested units “mine”, and it became clear that the only thing I was losing from quitting was future potential and nothing with value today.</p><h4>Internal Tooling Support</h4><p>Amazon has an unbelievable number of internal tools, services, and frameworks. They have their own internal version of StackOverflow, internal wikis, internal video hosting platform, custom-built Git repo hosting and code exploring, CI/CD platform, deployment services, handfuls of different design systems and component libraries, internal URL shortener, the list goes on. If you can picture a public SaaS product, Amazon most likely has an internally built version that does the same thing. Unfortunately, if the tool or framework you’re looking at isn’t on the critical path for a majority of engineers at Amazon, then you’re probably picking up a deprecated, scheduled to be sunset, or completely undocumented piece of tech. In my experience, Amazon’s external documentation isn’t fantastic, but it’s consistent and serviceable, the same can’t be said about all internal documentation.</p><p>You’re as likely to find a completely documented tool as you are to find a completely undocumented one. It’s infuriating typing the same queries again and again into the internal StackOverflow or wiki searches to discover not a single shred of information or replication of your issues. Often I would end up resorting to searching through other team’s code to find out how they’re using the tool and following up with DMs and emails to get my own use-case working. Once everything is working, there’s no incentive beyond goodwill to help backfill the documentation for other teams, so the issue never self-corrects. Even in my last week, I was receiving DMs from an engineer in another country asking for help troubleshooting a specific issue with implementing an internal framework ( <em>which I didn’t own</em>), just because I had set it up for one of my team’s services over a year prior.</p><p>Because there’s no external funding for these tools, if you’re not able to prove that fixing bugs, making security patches, or adding features to the tool is adding value to the company (often indirectly through productivity improvements of other teams), then it’s likely that you’ll have to put that tool into KTLO (keep the lights on) or sunset it completely. One day you’ll open a tool, and there’ll be a big red banner at the top of the screen with a date, threatening to turn it off unless some other team has the courage to take up the mantel of ownership. While this does result in almost a Darwinian system of tools where only the best and most-used survive, it also has the tendency to catch extremely useful, but underused tools in the crossfire.</p><p>To get around this, some teams have procedures in place to encourage other teams to commit fractions of their time to improving tools in exchange for a few benefits. The most extreme example of this I saw was an internal design system that had a “Standard” and “Premium” tier complete with a comparison table showing the differences between them. There was no difference between the actual design system or components between the tiers, the differentiators were things like access to a private Slack channel with the authoring team, influence on the roadmap, and workshops. While it is possible for teams to charge each other internally for use of their services, or tools, it is often reserved for situations where usage is directly tied to an external cost (for example sending SMS). Instead, if you want access to the Premium tier of this design system, you had to commit an engineer on your team to help build out part of it, reducing the load on the owners. I had a chuckle when I first found the tiers, but after thinking on it more, it troubled me that there needed to be such backroom deals to keep important tools off life support.</p><h3>Would I do it all again?</h3><p><strong>Absolutely!</strong> I had an amazing time at Amazon, and learnt a lot about what it takes to operate within one of the biggest companies in the world. My time there was invaluable and granted me a significantly different perspective from my previous roles, where I could remember the name of everyone in the company. I’m truly grateful for the opportunity I was given.</p><h3>What’s Next?</h3><p><strong>I’m promoting myself to customer</strong> and am off to tackle a new challenge! I’m joining a founding team to help solve big problems in the construction sector with new software. If that sounds interesting, and you’d like to chat more about it, feel free to reach out on <a href="https://www.linkedin.com/in/daveallie">LinkedIn </a>or drop me an <a href="/service/mailto:dave@daveallie.com">email</a>.</p><p>I want to take this opportunity to thank profusely the 3 managers I had, and 2 teams I was able to be a part of, during my time at Amazon. All of you are awesome, and hopefully we can work together in the future. 😍</p><p><em>Originally published at </em><a href="https://blog.daveallie.com/leaving-amazon"><em>https://blog.daveallie.com</em></a><em> on July 30, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99d0d2ae409e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preventing erroneous Rails migrations]]></title>
            <link>https://medium.com/tanda-developers/preventing-erroneous-rails-migrations-e5677346063?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5677346063</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Sat, 22 Jun 2019 03:21:15 GMT</pubDate>
            <atom:updated>2022-02-16T01:23:53.849Z</atom:updated>
            <content:encoded><![CDATA[<p>At Tanda, we’ve never really had a smooth ride with migrations, a medium-sized development team, and shared development resources (such as a common database for staging data and development).</p><p>Alex wrote about some of our woes with structure.sql which you can check out <a href="https://medium.com/tanda-developers/rails-teams-what-are-you-meant-to-do-with-structure-sql-9a12db79e88c">here</a>, and until recently, one of the bigger risks of our development workflow was the possibility for migration code that has never been run before to make it into production. I’m going to dive into how it can happen, and what we did to prevent it from happening.</p><p>After they’re created, database migrations go through a few different checkpoints at Tanda before making it to production. These are the three big ones:</p><ul><li>Getting run on the developer’s machine</li><li>Passing code review</li><li>Getting run on a staging site</li></ul><p>Between each of these steps, the migration might need to be adjusted by the developer, and depending on when the migration is changed, it’s possible for those changes to slip out into production with only the review step. Here’s how.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href">https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href</a></iframe><h3>Breaking a Migration</h3><p>I have my great new migration here:</p><pre>class MyMigration &lt; ActiveRecord::Migration[5.2]<br>  def change<br>    MyModel.where(my_field: 1).update_all(my_field: 2)<br>  end<br>end</pre><p>Fantastic, it looks like it’s going to work, it runs fine locally, my structure.sql or schema.rb is updated to include my new migration version. I commit the migration and the changes to the structure file, tests are run, everything passes, and it is deployed onto a staging server.</p><p>However, when it’s reviewed, my reviewer suggests I use MyModel.where(my_field: 1).batches.update_all(my_field: 2) because the table is big and the update should be broken into many smaller updates. Sound’s good, I update the migration file locally and commit again.</p><pre>class MyMigration &lt; ActiveRecord::Migration[5.2]<br>  def change<br>    MyModel.where(my_field: 1).batches.update_all(my_field: 2)<br>  end<br>end</pre><p>There was no change to my structure file, so nothing to commit there. Again, tests pass, it’s deployed, but this time, the migration isn’t run during my staging server deployment because it has already run. My reviewer comes back, approves my changes, and I merge into master where the production deploy kicks off.</p><p>If you haven’t seen the issue with the migration yet, I don’t blame you, however, Ruby doesn’t have much in the way of a forgiving nature. When the migration is run in production, it crashes, complaining that batches isn’t a method. In fact, it’s right, the correct method is in_batches , so what went wrong, how could this have happened?</p><p>Primarily, it was a process issue. The developer should have re-run the changed migration locally and the reviewer should have been more sure about the changes they were suggesting. If there’s one thing development tools are good for, it’s removing process problems.</p><h3>Fixing the Issue</h3><p>All the structure file does is let us know that the migration has run, <em>at one point</em>. What we needed was a way to signify that changes to a migration had been run, and the way we did that was with a concept called a ‘migration signature’.</p><p>We needed two things to make this work, a place to put the signature, and a way to generate it from the migration source. With those two, a rough workflow it all together looks like this:</p><ol><li>When the ‘up’ part of the migration is run locally, we analyse the migration file and generate a signature based on the contents</li><li>We save that migration signature somewhere</li><li>In CI, we analyse the migration files again and compare them to their signatures</li><li>If the expected and actual signatures don’t match, we know that a migration has been changed and has not been re-run</li></ol><p>Storage was the easiest part of the puzzle, we would store the migration signature in the file as a comment. In the case of our original migration above, after running it, this is what the file would look like:</p><pre># migration_signature: xxx</pre><pre>class MyMigration &lt; ActiveRecord::Migration[5.2]<br>  def change<br>    MyModel.where(my_field: 1).update_all(my_field: 2)<br>  end<br>end</pre><p>This way, migration signatures are right next to their migration, making checking and debugging a lot easier.</p><p>Generating the migration signature was a little bit less straight forward. The first idea was to use a hash of the migration file, however as we store the signature in the same file we’re signing, if we tried to sign the same file twice, we’d get two different signatures. So instead, the AST of the file was extracted and hashed to get the signature, this has the added benefit that any non-ruby changes will not cause a change in the signature. So signing a signed file, adding/removing comments and whitespace will not change the signature.</p><p>Going back to the original example, when the migration is changed and not run again, the signature no longer matches the file source.</p><pre># migration_signature: xxx</pre><pre>class MyMigration &lt; ActiveRecord::Migration[5.2]<br>  def change<br>    MyModel.where(my_field: 1).update.update_all(my_field: 2)<br>  end<br>end</pre><p>So in CI, when checking migrations for signatures, we can flag this migration and mark the build as failed with an error message:</p><pre>Missing or invalid migration signature in migration: 20190621040327_my_migration. <br>Please re-run your migration to receive an updated signature.</pre><p>That’s all there is to it. With a small utility, we have protected against a process problem, and encouraged good development practices at the same time.</p><p>We opened sourced this tool and made it into a gem for anyone to use, you can find it here:</p><p><a href="https://github.com/tandahq/migration_signature">TandaHQ/migration_signature</a></p><p>If you have come across this issue before in your development team and navigated around it in a different way, I’d love to hear about it, drop a comment down below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5677346063" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tanda-developers/preventing-erroneous-rails-migrations-e5677346063">Preventing erroneous Rails migrations</a> was originally published in <a href="https://medium.com/tanda-developers">Tanda Product Team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rustyboy — Part 1: Welcome to Emulation Station]]></title>
            <link>https://medium.com/@daveallie/rustyboy-part-1-welcome-to-emulation-station-34baeaacdf62?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/34baeaacdf62</guid>
            <category><![CDATA[gameboy]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[emulator]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Sat, 27 Apr 2019 00:12:43 GMT</pubDate>
            <atom:updated>2022-02-16T01:23:52.016Z</atom:updated>
            <content:encoded><![CDATA[<h3>Rustyboy — Part 1: Welcome to Emulation Station</h3><blockquote>This was originally posted back on <a href="https://blog.daveallie.com/rustyboy-part-1/">my personal blog site</a> in July 2018.</blockquote><p>I’ve seen a fair spread of emulators over the last 10 or so years and played around with a fair few of them as well (if you weren’t playing Pokemon on the VisualBoyAdvance while pretending to do school work then, I’m sorry to say, your childhood was lacking). It wasn’t until I stumbled across an old <a href="https://www.reddit.com/r/programming/comments/41bo7h/ive_started_livecoding_an_n64_emulator_in_rust/">reddit post</a> that I seriously considered building an emulator myself.</p><p>In the reddit post redditor yupferris was live streaming building an N64 emulator in Rust. Ferris walked through some of the tools he used, how he started the project, and most importantly, he showed off how much documentation existed for these older systems. There were docs on how sound worked, how the CPU ticked, how graphics were rendered and just about anything you could want to know about the internal working of the console. That was it I was hooked, time to build an emulator!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YW4DzhClV36ejLgq.png" /></figure><p>How hard can it be?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href">https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href</a></iframe><h3>Picking a console and a language</h3><p>The first step in building an emulator is deciding what to emulate. Some people suggest writing your first emulator for the <a href="https://en.wikipedia.org/wiki/CHIP-8">CHIP-8 system</a>, but on closer inspection, it looked almost a little too easy. Emulating the original <a href="https://en.wikipedia.org/wiki/Game_Boy">Game Boy</a> seemed complex enough to keep me scratching my head, but not so hard that I’d lose interested when I hit a roadblock. Plus, Game Boy had Pokemon. Who can say no to Pokemon?</p><p>Emulating a Game Boy definitely isn’t a new thing, just a quick Google will return hundreds of results of Game Boy emulators written in every language imaginable, but that didn’t bother me, it showed me that it was possible and there was absolutely no excuse as to why this emulator couldn’t be finished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tvBFOOw66i62vFDS.png" /><figcaption>Take a gander at those bezels. From a time when screen-to-body ratios weren’t as important.</figcaption></figure><p>The language choice was easy. I’d been <a href="https://github.com/daveallie?utf8=%E2%9C%93&amp;tab=repositories&amp;q=&amp;type=&amp;language=rust">messing around in Rust</a> on a few very small projects, but this seemed like the perfect opportunity to stretch my legs in Rust and start leveraging some of the more advanced features in a slightly larger codebase. And thus, my lil’ Rustyboy was born.</p><h3>Getting started</h3><h4>CPU</h4><p>Unsurprisingly, Tetris for the Game Boy is a very simple ROM that doesn’t utilise many of the advanced features of the Game Boy (aside from randomisation). Running Tetris gave me a tangible, and measurable goal. I started out by coding a simple CPU loop that ran completely unthrottled and crashed whenever it came across an instruction it didn’t recognise. And oh boy, it crashed a lot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/0*uTyMY1Iooeky84d5.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HlbuvFgJB33FNvLV.png" /></figure><p>Every time it crashed, I dug through the <a href="http://marc.rawer.de/Gameboy/Docs/GBCPUman.pdf">Game Boy CPU Manual</a> (hosted by Marc Rawer), found the relevant opcode, and wrote out the CPU instruction. There were some easy ones like NOP (literally, “no operation”) which were one-liners, but then there were also ridiculous ones, like DAA, which was implemented 100 different way depending on where you looked. It was slow but steady, each time I added an operation the emulator made it a few steps further.</p><p>One of the bigger mistakes I made here was attempting to emulate a Z80 CPU instead of the Game Boy CPU. I found a few articles saying that they were quite similar, and others that the Game Boy used the off-the-shelf Z80. So when I came across <a href="http://clrhome.org/table/">http://clrhome.org/table/</a> — which clearly laid out all the different operations and their opcodes, I threw Marc’s CPU manual out the window. Instead of scrolling through pages and pages of a PDF, it was all there in front of me. By the time I realised that the Game Boy did not, in fact, use a Z80, the damage was done. Half of the instructions I had written had correct timings and were referenced by the correct opcode, but the other half was a mess of incorrectly timed and referenced instructions that occasionally moved the program counter to wrong parts of the ROM. It took me a lot longer than I’d like to admit to find and fix those instructions.</p><h4>MMU</h4><p>Some instructions required reading from, or writing to, particular addresses in memory. Addressing the memory is done using a 16-bit value, meaning there only 65,536 addressable location in memory, and only half of that is for accessing data on the ROM. This creates a theoretical upper limit for ROM sizes of 32kB. However, looking at something like the Pokemon Red ROM, it’s exactly 1MB, so how the heck is all that data accessible? This <a href="http://gameboy.mongenel.com/dmg/asmmemmap.html">Game Boy memory map</a> might give it away, but I’ll be exploring it more in a future post.</p><p>In the beginning, like any good programmer, I chose to ignore the harder parts, and let future me deal with it. Instead of worry about 1MB ROMs, or writing out the interfaces for all the physical parts of the Game Boy, I wrote a very simple <a href="https://en.wikipedia.org/wiki/Memory_management_unit">MMU</a> that stubs almost every (non-ROM) read and hopes for the best. There was no way that Rustyboy could ever boot Pokemon in this state, but for a ROM as simple as Tetris, it didn’t matter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Rd7ZK4S8Ea5flzJ.png" /></figure><h3>Kind of working v1</h3><p>Commenting out all the panics, and attempting to run Rustyboy with the Tetris ROM, produced something like the following:</p><p>Each line contains the program counter (current address to lookup), and the operation found at that address to be executed.</p><p>There was no screen, no throttling and absolutely no pausing. Rustyboy would scream along at 100% CPU usage and try to process as many instructions as it could until it crashed or ended up in a loop waiting for graphics to render. It wasn’t much, but I was over the moon.</p><h3>Next Up</h3><p>This post is part of a series, in the next post I will talk about finding and fixing those broken instructions using a debugger that I wrote into Rustyboy.</p><p>You can also find the full repo at <a href="https://github.com/daveallie/rustyboy">https://github.com/daveallie/rustyboy</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34baeaacdf62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stealing Fuel Data]]></title>
            <link>https://medium.com/@daveallie/stealing-fuel-data-303f3327434c?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/303f3327434c</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[data-scraping]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Fri, 19 Apr 2019 08:12:47 GMT</pubDate>
            <atom:updated>2022-02-16T01:23:50.483Z</atom:updated>
            <content:encoded><![CDATA[<p>A little over a year ago, I discovered that the ACCC (Australian Competitor and Consumer Commission) <a href="https://www.accc.gov.au/consumers/petrol-diesel-lpg/petrol-price-cycles">reports on fuel costs</a> in the major cities of Australia.</p><p>I was pretty interested in what kind of insights could be pulled out of the historical data, however, I ran into a few speed bumps when attempting to get the raw cost data. The ACCC doesn’t present the raw costs, instead, they have an image of a graph containing the cost of fuel over the last 45 days for each city. Each day the image is updated, the oldest data point is removed, and a new data point appears.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*PCEupjZcabBMvFaB.png" /><figcaption>Brisbane’s fuel price from 1st Jan 2019</figcaption></figure><p>Frustratingly, the raw values aren’t included anywhere on the page, and after coming up empty-handed, I decided to try a different avenue. Below each image was <em>“Source: FUELtrac”</em>. A quick search took me to their website where they advertise a range of reports, including, fuel prices for Australia. As far as I could tell, the only way to get reports from FUELtrac was entering your details, starting a commercial process, and purchasing the report. Being a side project, I wasn’t really interested in spending money to get fuel data.</p><p>As this point it became a bit of a challenge to get the fuel prices for free, the man be damned.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href">https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href</a></iframe><p>There are a few sites around like <a href="https://petrolspy.com.au/map/graph/brisbane">Petrol Spy</a> which reports fuel prices, but does the exact same thing as the ACCC. It presents pre-rendered graphs and doesn’t expose the underlying data anywhere. Things started to look pretty grim, the people paying for these reports or manually collecting data weren’t just going to give it away.</p><p>I went back to the ACCC images and took a closer look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*Br9D6TZH7F9Jb-Fx.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*VXnP2TrvVHLvHdbI.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*IereMCtnPEXFEaiO.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*4ywi4BT1tUCAmzAo.png" /><figcaption>Samples of 4 different graphs for Brisbane’s fuel prices</figcaption></figure><p>It turns out, every single image has the exact same dimensions and the exact same amount of whitespace around the actual chart. The perfect kind of image for a scripted analyse and data extraction process. However, each image still only had 45 days of data, and to build a decent data set a lot more than 45 days of data was going to be required. I needed the ACCC images over a range of time. Again, the ACCC didn’t seem to want to grant easy access to their images, the historical images were impossible to find, and the main images shown for each city had generic URLs: <a href="https://www.accc.gov.au/sites/default/files/fuelwatch/brisbane-ulp.png">https://www.accc.gov.au/sites/default/files/fuelwatch/brisbane-ulp.png</a><br>With the underlying image being updated daily.</p><p>So I built a small lambda to download that image each day and save it into an S3 bucket (<a href="https://github.com/daveallie/data/blob/master/accc-fuel/lambda/index.js">source here</a>) and promptly forgot about the whole thing. That was over a year ago.</p><p>I picked up this project a week or two ago, coming back to over 500 days of data, determined to get the raw data from the image. I used OpenCV and Python, and have linked the source code at the bottom of the post.</p><p>To start, I wanted to determine the pixel position of each dot on the graph in relation to the top left of the image. Thanks to the visual difference of the dots on the line graph, using a bit of image manipulation, it is possible to get the individual positions of the dots.</p><h3>Extracting the Dot’s Positions</h3><p>Grayscaling the image and applying a threshold converts every single pixel to either black or white, removing all the extra noise of the background colour of the graph and the axis. In this case, I chose to threshold exactly halfway between black and white. So if the pixel was more black than white, it would be converted to completely black, and the rest were converted to completely white. In order to make future transformations easier, the image was also inverted, I’ll explain why in a bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*hgI9zU-tXuT10qu2.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*SAJwJu2viKx3f5qU.png" /><figcaption>Comparison of original and thresholded image</figcaption></figure><p>From here, the image was eroded. This can be imagined as shrinking the areas with white pixels and growing the areas with black pixels. The idea of this step is to end with an image containing just the dots from the graph and nothing else. If too much is eroded, the image will be completely black, if not enough is eroded, there will be white pixels that aren’t part of the dots. This is why it’s important to invert the image, if it hadn’t been inverted, the graph lines would have become larger when eroding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*PWh0-2_-IU1hS2X6.png" /><figcaption>Erosion value of 0</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*9xSJ9gcPseebSA3kbHpnLQ.png" /><figcaption>Erosion value of 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*AkIrxueAnwtHAV7k.png" /><figcaption>Erosion value of 4</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*fq2kOtzPPrRWjpiz.png" /><figcaption>Erosion value of 6</figcaption></figure><p>An erosion amount of 4 was found to be perfect, we’re left with a black image with a small cluster of white pixels where the dots were on the original image’s graph. For the next step, it is important that dots are not connected. So while an erosion amount of 2 did get rid of most of the axis, most of the dots are still connected by a thin white line.</p><p>Next, the image was analysed for contours. This process attempt to build and return paths around borders of shapes. Seeing as the only non-black pixels are our dots, we’re returned a list of borders that make up each dot. From there, it’s one line of code to get the locations of the centres of those shapes. Plotting the centres on both the eroded image, and the original, we can be sure that all the points were found, and no random clusters of pixels were incorrectly identified as a point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*qKUTaMoIfcMeiUzQ.png" /><figcaption>Points highlighted on eroded image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*vgLj5t1twTYEkJun.png" /><figcaption>Points highlighted on original image</figcaption></figure><p>Now that we can accurately identify points from the graph in the image and their position within the image, we need to turn those values into a fuel cost for each day. I cut some corners to get the date of each point, we’ll get to that in a bit, but first, the cost of fuel for each point.</p><p>Lucky for us, the images are all the same dimensions, with the graph in the exact same position within each image. Each graph starts at 10 pixels from the top of the image and is 330 pixels tall, and with a quick transformation, we can scale each point in the image to a percentage distance from the bottom of the graph. A point at the bottom of the graph would have a value of 0, and a point at the top, a value of 1. If we know the graph starts at 110 cents and goes to 140 cents, then the value of a point 30% of the way up the graph can be calculated as 119 cents (110 + (140–110) * 0.3 = 119).</p><p>Unfortunately for us, each image can have a different minimum and maximum values on the y-axis (the fuel cost axis), and because we need the min and max values to find the value of points, we need to somehow dynamically pull those values out of each image.</p><h3>Determining the y-values</h3><p>I don’t really know how to use many OCR libraries, or what their training data looks like. Also, given the extremely limited number of pixels used for each digit of the fuel’s cost, I didn’t feel confident trying to implement OCR for these values. Instead, I looked around in what OpenCV could do and stumbled across <a href="https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html">template matching</a>. Template matching is a way of taking a template image and an image potentially containing that template, to return the confidence and location of the template within the larger image.</p><p>So if we know what the number 1 looks like, we can confirm its existence and position in a picture of 120. Another way to use it would be to give it a collection of pixels and ask it how confident we are that it is what we have determined previously to be the number 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/0*FhlaccHi3YklqWfD.png" /><figcaption>120 from the y-axis up to 10x its original size</figcaption></figure><p>Thanks again to the graph being a consistent size, the minimum and maximum fuel costs on the y-axis can also be found in the same position on every graph. If we take those numbers and split them up into their digits for the purpose of template matching, then we have a significantly lower number of templates to match against. Instead of having to know what all numbers look like between 90 and 190, we can just know the numbers 1 through 9 and construct the value based on the digits.</p><p>We can take a very lazy approach to split the image into images of the digits. By inverting and thresholding the image, we can find rows where there are no white pixels, indicating gaps between the images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/0*UkVy4YV0OKbyQWar.png" /><figcaption>120 from the y-axis inverted, thresholded with empty columns highlighted in red</figcaption></figure><p>That gives us a splitting point for the image to get the digits, but for purpose of the upcoming template matching, the digit images were also padded to 5 pixels wide in order to standardise them. Below is the final extracted digits from the 120 image above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/0*XWYul0WHU6uXVEJf.png" /><figcaption>Each of the three digits extracted from the original image, background is just for visibility</figcaption></figure><p>To find the value of each digit, a small python script was built to take the original graph image, extract the digits through the method just explained, and then run through template matching. On its first run, the script has no idea what a 1, 2 or 0 looks like. So when it can’t match the image against a known digit with 95% confidence, the script will show the image and wait for a key input. Pressing 0–9 will identify the image as that digit. The image is then averaged with all other images for that digit to help to create a better picture of what that digit looks like. Because the digits were quite consistent and small, I only had to identify each digit once before the script knew what to look for.</p><p>From the trained templates, the images above can be identified as 1, 2 and 0, making up the number 120. The trained templates were saved to disk so we can use them in the main script.</p><h3>Putting it all together</h3><p>At this point, I started getting pretty excited, we are pretty much there. From an image, we can break it down into individual graph points, extract the minimum and maximum fuel costs for the graph and with those two pieces of information, determine the fuel cost represented by each point in the graph. The only thing we’re still missing is the date of the fuel costs.</p><p>We do have a bit of an advantage though, all the points in the graph are separated by exactly one day. So if we knew the date of the first measurement on the graph, then we could extract the dates for the rest of the points. Also, as each day there’s a new image with the oldest data point removed and a new data point added, we can continue to determine the date of the points, as the next image will just start one day after the last image started.</p><p>In order to remove jitter from the image processing, the values for each date were extracted from multiple images and averaged. After running the script a few times across different numbers of images I started to notice a lot of variance in the costs.</p><p>Unfortunately, the ACCC had to make my life complicated one final time. There were days in my data where the image wasn’t updated for a day or two, and then it jumped forward 2 days. I don’t know if it’s an issue with their system or a lazy intern, but it started to throw out the cost for fuel.</p><p>To fix this, we need to introduce a delta check for each new image. Once the values for each point are extracted from subsequent images, the delta of days between the images is assumed to be somewhere between 0 and 10. Where 0 would be an image with the same dates, and 10 would be an image where 10 days were skipped. A day delta of 1 is standard and was the assumption that was made about the images before discovering this problem.</p><p>To find the actual day delta, it is guessed to be each of the values between 0 and 10, and the difference between the guessed costs on the current image and the known costs for dates from previous images is summed. The day delta which produces the lowest variance in costs is taken. After running the script I found that about every 5th or so image had a 2–3 day skip in it.</p><blockquote><strong>Edit: </strong>As a couple of people have pointed out, a 2 day skip every 5 days is called a weekend 🤦‍♂️. I was so close to the data that I didn’t even realise it lined up with the working week.</blockquote><p>Finally, we have accurate fuel cost data, can be continuously retrieved and fed into a database. We did it, screw the ACCC. As developers, it’s our responsibility to open up the internet and the data throughout it. Don’t take no for an answer, go steal data off the internet!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KukJ9gNlDAh8T7JE.png" /><figcaption>Final graph of fuel costs</figcaption></figure><p>Thanks for reading! You can find the source code, and raw fuel costs <a href="https://github.com/daveallie/data/tree/master/accc-fuel">here</a>. I also threw the data in a Google Sheet to graph it, if you want to clone that and play around with it, you can find it <a href="https://docs.google.com/spreadsheets/d/1F05ATFtte4KU_qefRaklKzUMJQwdL8KTX7gTQCWTpNU">here</a>.</p><p>This is my first attempt at doing anything related to image processing, so if you spot an error, please let me know!</p><blockquote><strong>Edit:</strong> A few days after publishing this, I deployed an undocumented API for this data. If you consider yourself a power-user, have a play with it <a href="https://fuelapi.daveallie.com/v1/brisbane/ulp/2018-12-12">here</a>. If you have questions about this article, the data, or the API, please reach out.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=303f3327434c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grapple — An actual interruptable download accelerator]]></title>
            <link>https://medium.com/@daveallie/grapple-an-actual-interruptable-download-accelerator-17bee72df982?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/17bee72df982</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Wed, 12 Sep 2018 05:15:35 GMT</pubDate>
            <atom:updated>2022-02-16T01:23:49.362Z</atom:updated>
            <content:encoded><![CDATA[<p>A bit ago I needed to pull some large binary files down from a remote server and, at the time, had an extremely spotty internet connection. After about the 5th attempt to download the file and having my internet drop 15 minutes into the download, I gave up. Chrome’s download manager wasn’t cutting it, so I opened up GitHub, had a quick look at how other people were writing download tools, and stumbled across <a href="https://github.com/derniercri/snatch">Snatch</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Ff%2Fsqh0dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href">https://medium.com/media/2d1d494f40be802ac873c65cfa616b68/href</a></iframe><h3>Snatch</h3><p>Snatch’s description was “A simple, fast and interruptable download accelerator, written in Rust”. Exactly what I wanted, and written in a language I was interested in! Not only that, they were also a trending Rust repository at the time and had almost 600 stars. Taking a closer look at the repository’s readme, that excitement turned to disappointment pretty quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwHdCE5JSHBQ5K3SQl39mg.png" /></figure><blockquote>the <em>Interruptable</em> feature is not implemented yet.</blockquote><p>What… the… hell…</p><p>How can you have an <strong>interruptable</strong> download accelerator, without the interruptable part?!</p><h4>Proposing Interruptablity</h4><p>I tried to get comfortable with the codebase to see if writing in the interruptable feature would be difficult. I opened an issue, wrote a PR to fix it, and gave a bit more thought to the feature. I formalised my thoughts for how I believed the feature should be written and wrote up a draft proposal for it in <a href="https://github.com/derniercri/snatch/issues/32">the open issue for interruptablilty in the repo</a>.</p><p>The way Snatch worked was to first pre-allocate the space needed to download the file. Start n threads at even intervals of the file and have each of the threads download its section. If the download was interrupted, there was no way to determine how much of the file each thread had downloaded so all threads had to start from the beginning of their section.</p><p>The changes I proposed to get the feature working were as follows:</p><ul><li>The first 32 bits would be a 32-bit integer that represented the number of 64kB chunks in the file</li><li>Let’s call this number n</li><li>The next n bits were bit flags representing if the relevant chunk had been downloaded</li><li>e.g. For a 256kB file, 0101 would mean that the 2nd and 4th chunk had been completely downloaded</li><li>The rest of the file would follow</li></ul><pre> 1    2             3<br>|-|-------|---------------------|</pre><pre>1: 32-bit number holding the number of 64kb chunks<br>2: n bits holding bit flags for each chunk<br>3: File contents</pre><p>Restarting the download, each thread could look at the bit flags in section 2 and determine if part of the download could be skipped. When the download was complete, the first 32 bits could be read which would let us know how many bits in section 2, so we could remove both section 1 and section 2, without touching the file contents.</p><p>This approach definitely had a few limitations. As we could only hold state for 2³² 64kB chunks, it had a maximum file size limit of 256TB (2³² * 64 * 1024 / 1024⁴). Threads would need to download a range of bytes that both started and finished on a 64kB chunk so that no chunk was downloaded by two separate threads. There were also a couple of other small caveats that would mean slight modifications to the download and threading logic.</p><p>I was hoping after laying it all out to the author of the project, he would approve of it, and I could forge ahead and build it. Unfortunately, he didn’t seem too interested in actually having the interruptable feature written, and instead wanted to polish every little aspect of the download part before considering interruptability.</p><p>So naturally, being a developer, I just went off and wrote my own package.</p><h3>Grapple</h3><p>I present <a href="http://github.com/daveallie/grapple">Grapple</a>, an actual interruptable download accelerator. I took some pretty heavy inspiration from Snatch as Grapple was initially intended to be a proof of concept for the feature, but rapidly eclipsed Snatch’s feature-set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*UCTNuPpT-ap6w6z5vj392w.gif" /></figure><p>The actual implementation was a little different to what I initially proposed. It turns out it’s really bloody hard to efficiently remove the first n bytes of a file. Conversely, it’s really easy to truncate the end of the file, so one of the changes was to move all the metadata to the end of the file. Also, to drastically increase the file size limit, I used a 64-bit integer for the number of chunks and increased the chunk size from 64kb to 128kB. This brought the new file size limit up to 2,147,483,648PB. I’ll probably be dead before files get large.</p><p>New file layout:</p><pre>           3              2    1<br>|---------------------|-------|-|</pre><pre>1: 64-bit number holding the number of 128kb chunks<br>2: n bits holding bit flags for each chunk<br>3: File contents</pre><p>I use Grapple about once a week, and it hasn’t crashed on me yet (<em>touch wood</em>). I still go back to it now and again to make some enhancements, just recently I added bandwidth limiting per thread to prevent saturating my internet connection and upsetting my housemates. Grapple has definitely been one of the more useful tools I’ve written, and I intend to keep updating it over time as I need more and more from it.</p><p>Just for fun, as I was writing this, I went back to look at Snatch to see if the interruptable feature was implemented yet. Snatch is deprecated in favour of a fork, Zou, and this was in the readme:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9L1a8zN7IUZZkODfNWkAEA.png" /></figure><p>The interruptable feature <strong>still</strong> isn’t done. I think I’ll just stick to using Grapple.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17bee72df982" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tanda Student Clubs Promise]]></title>
            <link>https://medium.com/tanda-developers/tanda-student-clubs-promise-88dd900b921?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/88dd900b921</guid>
            <category><![CDATA[students]]></category>
            <category><![CDATA[university]]></category>
            <category><![CDATA[hackathons]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Wed, 25 Apr 2018 00:35:25 GMT</pubDate>
            <atom:updated>2018-05-01T23:31:20.775Z</atom:updated>
            <content:encoded><![CDATA[<p>At Tanda we love student clubs. Tanda was born out of a student club at QUT where our four founders met. As we’ve grown the business, we’ve always tried to support clubs as much as we can, through running our own events, sponsoring theirs, or just putting up bar tabs for no reason.</p><p>The thing that’s both exciting and challenging about working with student clubs is that in some ways you start fresh every year. Clubs are always trying to grow and improve constantly (just like we are), but their leadership changes every 12 months.</p><p>To make things a bit simpler for everyone involved, I thought I’d write down what we offer to all student clubs. You don’t need to do anything to be eligible for this offer, other than to be a student club at a university, and to take us up on it.</p><p><strong>Hackathons: $10/ticket</strong></p><p>Whenever we run a <a href="https://medium.com/tanda-developers/looking-back-on-4-years-of-hackathons-86387d3b7dc4">hackathon</a>, one of the questions we ask when buying a ticket is “which student club do you come from?”. For every ticket sold where this question is answered, we’ll give $10 to the relevant club.</p><p>Get in touch with us ahead of time so that we can ensure that your club’s name is on the list. You will need to send us an invoice for $10 x number of tickets after the event — we’ll tell you the exact amount after the event.</p><p><strong>Hires: $500-$1000/hire</strong></p><p>If a student club directly introduces us to a member of theirs who we end up hiring, we’ll give the club $1000 for a full time hire, or $500 for a part time hire.</p><p>Note: having your member attend one of our events doesn’t count for this. It needs to be a direct introduction — send us an email, cc the member, tell us a little bit about them, and we’ll take it from there. If you aren’t sure who to email, you can start with <a href="/service/mailto:developers@tanda.co">developers@tanda.co</a>, or come to one of our events and meet our team!</p><p>The club will need to invoice us once we’ve made the hire and the member has started working at Tanda.</p><p>btw, this is the same offer that we make internally to everyone at Tanda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ij8RmHcVxiIFnFkXmmUYOg.jpeg" /></figure><p>We are putting both of these offers online so there is no confusion about them, and in the hope that more clubs take us up on them. If there’s something else you think we can do to support clubs, please tell us about it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88dd900b921" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tanda-developers/tanda-student-clubs-promise-88dd900b921">Tanda Student Clubs Promise</a> was originally published in <a href="https://medium.com/tanda-developers">Tanda Product Team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brisbane Tanda Hackathon 2018]]></title>
            <link>https://medium.com/tanda-developers/brisbane-tanda-hackathon-2018-5e911c17d614?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e911c17d614</guid>
            <category><![CDATA[hackathons]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Tue, 24 Apr 2018 08:45:33 GMT</pubDate>
            <atom:updated>2018-04-24T08:46:44.749Z</atom:updated>
            <content:encoded><![CDATA[<p>Last weekend was the annual Tanda Brisbane Hackathon, our fifth and largest hackathon event in 4 years. With 160 tickets sold and the Brisbane office prepared for the masses we were ready, or so we thought. There were a few hiccups, including a production outage, but all in all another awesome weekend of hacking.</p><p>This year we decided to move away from a technical theme and went with the theme of Employee Experience; How can we make the lives of employees better, let them love their workplace, and remove all the hurdles of leaving one job and starting another. This is something that we’re spending more and more time thinking about at Tanda, especially with the release of our Employee App.</p><p>Unfortunately, I missed out of the initial brief and rundown of the hackathon on Friday because someone at Tanda decided that running a 160 person event wasn’t hard enough, so they pushed a sneaky bug into production a week before. As people started to arrive the bug reared up and took down our production site. Friday night outages are a bad time even on a good day, but throw a hackathon in the mix and you’ve ruined 160 other people’s weekend as well. Thankfully the operations team saved the day and brought the site up just in time for the pitches!</p><p>With 160 people and a room full of ideas, we had people pitching things from an anonymous cross-company chat service to an offboarding tool that directs you to the nearest pub when you get fired. There was a pretty amazing assortment of ideas, and it’s really cool to see both a mix of completely new and previously unthought of projects mixed in with recurring favourites (Tinder for Tanda 4 will be a winner, I’m calling it now).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0c-2RFPMveCUPmNv." /><figcaption>160 people don’t even fit in two photos</figcaption></figure><p>For the remainder of the evening, everyone split up into teams and got to work either planning, coding, or (somehow) both at once.</p><p>The Saturday was at River City Labs, an amazing co-working space just off Brunswick St, where the teams buckled down and got some serious work done. Some teams knew what they had to accomplish, wired themselves in for the day and barely said a word, while other teams and more whiteboard diagrams than I could count, mapping out every little part of their product. They all had one thing in common though, downing tools at the smell of food.</p><p>Following lunch, there were a lot fewer questions about the best approach for the problem and a lot more questions about how much could be hardcoded and if the judges would notice. The final couple of hours really make or break the teams, and leading up to 7pm, there were some teams that said it couldn’t be done, however almost all of those teams went on to pitch what they had put together over the weekend.</p><p>7pm finally arrived, and the pitches commenced. There were 20 pitches, and with a short break in the middle, both the attendees and the judges managed to get through all 20.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C_zY3990bsqbf0EQ." /><figcaption>Say hi to our 3 awesome judges: Bronwyn Voyce, Joanna Cantly, and Dr Daniel Thomas</figcaption></figure><p>Once it was all said and done the judges had 2 awards to announce: the Technical Award, and the overall Tanda Hackathon winner.</p><p>The technical award went to ETAssistant, the team behind an Android Auto integration that would notify your manager or team through any chat service that you were running late, and exactly how long it would take you to get to work given the current traffic conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ku7sGhS3wTz2ZZvL." /><figcaption>ETAssistant, the technical award winners</figcaption></figure><p>The overall award went to I’ll Get back2U. They created a messenger bot for employees to allow them to lookup shifts, how much they’re going to get paid and what their leave balance is, all with a fair few gifs sprinkled in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zmn6HDBMi1I0qknw." /><figcaption>I’ll Get back2U, the overall winners (+ Dan from Tanda getting his squat on)</figcaption></figure><p>A great weekend of hacking, and an enormous effort from everyone who came. I can’t wait to do it all again next year!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e911c17d614" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tanda-developers/brisbane-tanda-hackathon-2018-5e911c17d614">Brisbane Tanda Hackathon 2018</a> was originally published in <a href="https://medium.com/tanda-developers">Tanda Product Team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we sped up our asset precompile by removing it]]></title>
            <link>https://medium.com/tanda-developers/how-we-sped-up-our-asset-precompile-by-removing-it-641cca04289c?source=rss-f2e2079965e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/641cca04289c</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[assets]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Dave Allie]]></dc:creator>
            <pubDate>Mon, 12 Jun 2017 00:13:58 GMT</pubDate>
            <atom:updated>2017-06-12T09:20:53.312Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*UaMhvZoeKuw3DcYQM_3TEg.png" /></figure><p>It seems like every time I look back at our build pipeline our builds are taking longer and longer to go from commit to being out in the wild. A lot of that time is spent running our test suite and static code analysis tools, which can’t be reduced much further. Those pipelines aren’t getting much shorter as it’s not really in our best interest to tell our developers to stop writing new tests. So the savings have to come from somewhere else. Luckily, there’s still a very large portion of time that is spent getting our code ready and onto all the production boxes. So let’s have a bit of a dig to try and see if we can cut some time off our deployment process.</p><h3>Initial Benchmarks</h3><p>The most important part of benchmarking, and determining if you’ve made any impact at all, is having a good initial benchmark. So I trawled through our build history analysing time stamps to find a typical deployment. Below is an extract of the relevant log lines with timestamps.</p><pre>* 04:37:36 executing `deploy:update_code&#39;<br>* 04:38:06 executing `bundle:install&#39;<br>* 04:38:25 executing `deploy:assets:precompile&#39;<br>* 04:42:01 executing `deploy:assets:upload&#39;<br>* 04:45:01 executing `deploy:assets:upload_manifest&#39;<br>* 04:45:36 executing `deploy:assets:clean&#39;<br>* 04:45:47 executing `deploy:restart&#39;<br>* 04:46:20 finished</pre><p>Here is a more readable breakdown on the time taken to perform each task.</p><pre> Event            | Duration<br>------------------+------------<br>Update Code       |    30s<br>Install Deps      |    19s<br>Compile Assets    |   215s<br>Upload Assets     |   180s<br>Upload Manifest   |    35s<br>Clean Assets      |    11s<br>Restart Processes |    33s<br>------------------+------------<br>      Total       | 523s (8:43)</pre><p>Straight away, a couple of the steps stick out. Compiling the assets, and uploading them take about 75% of the total time. It is pretty ridiculous to consider that these steps always run on deploy, even when there are no asset changes between the currently live version and version in the process of deploying.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2FDimSimilarCapybara&amp;url=https%3A%2F%2Fgfycat.com%2Fgifs%2Fdetail%2FDimSimilarCapybara&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FDimSimilarCapybara-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="1280" height="546" frameborder="0" scrolling="no"><a href="https://medium.com/media/50f2832ca624d4385c806ed640909079/href">https://medium.com/media/50f2832ca624d4385c806ed640909079/href</a></iframe><h4>Why does the precompile take so long?</h4><p>If you didn’t know, Rails’ asset precompile is supposed to be almost instant when assets are unchanged. So then why is it taking so long for our builds regardless? It turns out that Sprockets leverages a cache directory holding partial asset artefacts which can be instantly compiled into production assets. Whenever the original assets change, these artefacts are invalidated and as a result the build process has to start from scratch.</p><p>Due to the nature of some our assets, and our slightly unreliable CI/CD cache, we don’t carry our Sprockets cache between builds. In the future we should come back and reevaluate this, potentially utilising S3 to carry zipped asset artefacts between builds of the same branch. But currently, each build has to compile the assets from scratch, which leads to the extensive build times seen above.</p><h3>Conditional Asset Precompile</h3><p>So, given that the source assets aren’t changing <em>every </em>deploy, what can we do in these deploys to prevent building identical production assets? Surely we can just reuse the assets that were compiled last build? To answer that, we’ll need to dig into how Rails determines which minified asset to serve and where we store our assets.</p><h4>Rails asset manifest</h4><p>When Rails completes the asset precompile, it appends a fingerprint to each compiled asset. For example, application.js might be compiled to application-7d25452ceb63594739af24cde73b6499.js where the fingerprint section (7d..99) is the hash of the content of the file. This just means that if you run the precompile twice without changing anything, the exact same file will be generated, with the exact same file name. Not only does that help with <a href="https://www.keycdn.com/support/what-is-cache-busting/">cache busting</a>, but it also means we can keep a historical set of unique assets in the event that we need to rollback a release.</p><p>Alongside these files, Rails also generates a manifest file. When application is running, this manifest file tells Rails that application.js can be found at application-7d25452ceb63594739af24cde73b6499.js. Effectively, it’s just a mapping between original file names, and file names with fingerprints. Similar to the fingerprinted file names, running the asset precompile twice with the same assets will generate the exact same manifest file.</p><h4>Tanda’s asset storage</h4><p>In the breakdown above, there was a step titled ‘Upload Assets’. This step is where we take the compiled, fingerprinted assets and upload them into an AWS S3 bucket. As the asset’s filenames include fingerprints, we know that if the file exists on S3, the contents will identical, so we don’t need to re-upload it. This saves us some time during the deploy, but more importantly, if the assets haven’t changed since the last deploy, then we know that our compiled assets are already on S3.</p><p>Tanda’s web instances don’t have a copy of the compiled assets on them, just the asset manifest containing the mapping between the original file name and fingerprinted file name. This, coupled with some Rails configuration, tells our Rails instances that application.js can actually be found at //s3-ap-southeast-2.amazonaws.com/asset-bucket/application-7d25452ceb63594739af24cde73b6499.js.</p><h4>Reusing the manifest</h4><p>Putting all of this together, we know that if the assets are unchanged and we can get a copy of the previous manifest, we don’t need to rerun the precompile or upload anything to S3. We can just drop the old manifest onto the boxes as part of the deploy, and voilà, we have access to the correct, compiled, production assets. All currently running Rails instances have the current manifest (or else they themselves wouldn’t be able to access assets), so we can get a copy of it from them.</p><p>The harder task is determining if the assets have changed without running the precompile. The fingerprint added by Sprockets is a hash of the contents of the compiled file, so we can’t compute this hash to unless we run the intensive task of compiling the assets. But we need the hash to determine if the asset is unchanged so we can decide to skip the precompile. Catch-22. Let’s look at another approach.</p><p>We know that when the original assets change, so too do some (or all) of the compiled assets. So what if we ‘fingerprint’ the uncompiled assets and compare it to the ‘fingerprint’ of the currently deployed assets. That way, if the fingerprints match, we can skip the precompile and the upload, using the currently deployed manifest instead.</p><p>So how the heck do we generate this ‘fingerprint’? A <a href="https://unix.stackexchange.com/questions/35832/how-do-i-get-the-md5-sum-of-a-directorys-contents-as-one-sum">quick Google search</a> revealed that creating a hash for a whole directory isn’t that hard, and thankfully Rails’ directory structure dictates that all raw assets live in app/assets. So the ‘fingerprint’ for our assets can be computed as the fingerprint for the whole app/assets directory.</p><pre><strong>find app/assets -type f | sort</strong><br><em>app/assets/javascripts/application.js<br>app/assets/javascripts/availability/app.js<br>app/assets/javascripts/availability/bookmark_bubble.js<br>app/assets/javascripts/availability/index.js<br>...</em></pre><pre><strong>find app/assets -type f | sort | xargs md5sum</strong><br><em>5c..ff  app/assets/javascripts/application.js<br>b7..ed  app/assets/javascripts/availability/app.js<br>9a..15  app/assets/javascripts/availability/bookmark_bubble.js<br>8b..7d  app/assets/javascripts/availability/index.js<br>...</em></pre><pre><strong>find app/assets -type f | sort | xargs md5sum | cut -c1-32<br></strong><em>5cf52293e4cecba984fb1967741af3ff<br>b781f4df4db63ca255bb48ac6cef34ed<br>9a0afefd9a8821d09d596ac3fedcae15<br>8b21d90afbbc3f6d5e7296e168e58a7d<br>...</em></pre><pre><strong>find app/assets -type f | sort | xargs md5sum | cut -c1-32 | md5sum | cut -c1-32<br></strong><em>87cfe7fb67a9268deebb6638e76a5c37</em></pre><p>First, I grabbed the full list of assets using find, then I piped each of them into md5sum, giving me a hash for each file. Finally, I piped all those hashes back into md5sum giving me a hash of all the hashes. If a file changes, its hash will change, which will also mean the hash of hashes will change. This final hash is exactly what we were looking for, a ‘fingerprint’ for the assets directory.</p><h3>Results</h3><p>I modified the build process, running the new ‘fingerprinting’ command on the new app/assets directory being deployed and on the currently deployed app/assets directory. In the case they matched, I downloaded the manifest file from a currently deployed box. A few empty commits and a bit of debugging later, I got this:</p><pre>* 05:59:43 executing `deploy:update_code&#39;<br>* 06:00:11 executing `bundle:install&#39;<br>* 06:00:52 executing `deploy:assets:check_hash&#39;</pre><pre>Local hash:<br>b4ea3b43fb2e76a20e4da6f49b78bbca</pre><pre>Server hashes:<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca<br>b4ea3b43fb2e76a20e4da6f49b78bbca</pre><pre>Assets unchanged, skipping precompile</pre><pre>* 06:01:02 executing `deploy:assets:upload_manifest&#39;<br>* 06:01:38 executing `deploy:assets:clean&#39;<br>* 06:01:48 executing `deploy:restart&#39;<br>* 06:02:29 finished</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2FSparseVigilantDotterel&amp;url=https%3A%2F%2Fgfycat.com%2Fgifs%2Fdetail%2FSparseVigilantDotterel&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FSparseVigilantDotterel-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="1280" height="544" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba82f6ed23dfd52747d2839a17cf244e/href">https://medium.com/media/ba82f6ed23dfd52747d2839a17cf244e/href</a></iframe><p>If you didn’t do some quick mental math from the timestamps above, the breakdown below should reveal it all. In short, the build process went from 8:43 to 2:46 when there were no new assets. Thats a speedup of 3.15x! I don’t know about you, but I’d call scraping 6 whole minutes off the build process is a good day’s work.</p><pre> Event            | Duration<br>------------------+------------<br>Update Code       |    28s<br>Install Deps      |    41s<br>Check Asset Hash  |    10s<br>Upload Manifest   |    36s<br>Clean Assets      |    10s<br>Restart Processes |    41s<br>------------------+------------<br>      Total       | 166s (2:46)</pre><p>There are a lot more improvements that could be made to this process in the future. As an aside, at Tanda we use Webpack for all of our new assets, and Sprockets for all existing assets. This process could be improved by creating a seperate ‘fingerprint’, like we did for all the original assets, for Sprockets and Webpack separately. The Sprockets precompile takes about 3 minutes, whereas the Webpack precompile takes about 7 seconds. If we could correctly split assets and compute these fingerprints, changing the Webpack assets wouldn’t trigger the Sprockets precompile which would be another big win.</p><p>All in all, the takeaway from this exercise is this: <strong>don’t be afraid to dig a little deeper into production critical processes.</strong> Definitely get your code heavily reviewed and tested before attempting to use it in production, but don’t hold back thinking that the process will fix or improve itself. If a part of your development cycle isn’t living up to your expectations, and it hasn’t been fixed already, then either no one else has noticed, or they’re too scared to touch it. So heed the wise words of Master Yoda and don’t fear digging deeper.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2FAbleRegalHookersealion&amp;url=https%3A%2F%2Fgfycat.com%2Fgifs%2Fdetail%2FAbleRegalHookersealion&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FAbleRegalHookersealion-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="1280" height="546" frameborder="0" scrolling="no"><a href="https://medium.com/media/588e63ecf8a50394f4a16b446ab1cb37/href">https://medium.com/media/588e63ecf8a50394f4a16b446ab1cb37/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=641cca04289c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tanda-developers/how-we-sped-up-our-asset-precompile-by-removing-it-641cca04289c">How we sped up our asset precompile by removing it</a> was originally published in <a href="https://medium.com/tanda-developers">Tanda Product Team</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>