<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Niall O&#39;Higgins on Medium]]></title>
        <description><![CDATA[Stories by Niall O&#39;Higgins on Medium]]></description>
        <link>https://medium.com/@niallo?source=rss-5ed1aec8c690------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8GQthhN1d2GS4Y33ivpDYw.jpeg</url>
            <title>Stories by Niall O&amp;#39;Higgins on Medium</title>
            <link>https://medium.com/@niallo?source=rss-5ed1aec8c690------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:43:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@niallo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NFT Is The New Rolex]]></title>
            <link>https://medium.com/@niallo/nft-is-the-new-rolex-78547c89a998?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/78547c89a998</guid>
            <category><![CDATA[nft]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[crypto]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 16:58:30 GMT</pubDate>
            <atom:updated>2022-01-18T16:58:30.373Z</atom:updated>
            <content:encoded><![CDATA[<p>Why JPEGs are selling for hundreds of thousands of dollars</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*9kNYLHh_tcPwpbX8bnxfqw.jpeg" /><figcaption>Custom Bored Ape Rolex Skydweller from <a href="https://twitter.com/machibigbrother/status/1474016912225218561">https://twitter.com/machibigbrother/status/1474016912225218561</a></figcaption></figure><h3>What is an NFT?</h3><p>In case you have been living under a rock, here are the most important things to understand about contemporary NFTs:</p><ul><li><strong>Unique</strong> (ie Non-Fungible) digital item (today: mostly images), part of a <strong>collection.</strong></li><li>Ownership is <strong>transparent</strong> &amp; <strong>provable</strong> on the blockchain. I can cryptographically prove that I own the <a href="https://opensea.io/collection/doodles-official">Doodle</a> NFT which is my profile pic.</li><li>Very <strong>low friction to trade </strong>24/7/365 on a blockchain like Ethereum or via a marketplace like <a href="https://opensea.io/collection/doodles-official">OpenSea</a>.</li><li><a href="https://www.kslaw.com/news-and-insights/not-your-standard-orange-grove-non-fungible-tokens-securities-laws">Probably not</a> a regulated security or commodity in the USA.</li></ul><h3>Why do insane people buy NFTs for hundreds of thousands of dollars? They’re just JPEGs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*wJl6_ExzVezUSJk3qR6F8g.png" /></figure><p>The eternal question posed to all bubbly crypto assets — from Bitcoin to Bored Apes.</p><blockquote>Why on earth are some magic numbers that don’t exist in the physical world so valuable? This must be a scam.</blockquote><p>The simple answer is because <strong>people are willing to pay that much for them.</strong></p><p>But why? Let’s dig into this a bit more.</p><h3>Rolex &amp; Luxury Mechanical Watches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4-zoVtQT2rJCXdvYW19eQ.png" /></figure><p>Luxury watch prices don’t make a lot of sense either. Nor does the fact that you <a href="https://www.nytimes.com/2021/09/08/fashion/watches-rolex-audemars-piguet-patek-philippe-mille.html">can’t currently buy a Rolex at retail</a> even if you wanted to. Why would anyone spend <a href="https://www.thewatchbox.com/watches/rolex/rolex-submariner/">well over $10,000 USD</a> for a basic Rolex Submariner? Interestingly they were sold for about $150 USD in the 1950s (~$1100 USD adjusted for inflation) and the technology hasn’t particularly improved since then.</p><p>A modern electronic or quartz watch is much cheaper, more reliable, lower maintenance and more technologically advanced. Arguably Rolex and other luxury mechanical watches are outdated and inferior time keeping devices. Who wants to wind their watch daily and send it away for periodic servicing?</p><p>The fact is, Rolex and luxury watches in general command prices far above what it costs just to perform the job of accurately telling the time. Applying <a href="https://hbr.org/2016/09/know-your-customers-jobs-to-be-done">Jobs-to-be-done theory</a> to Rolex will give us some interesting ideas:</p><ul><li>Rolex does the same job as <strong>very nice jewelry. </strong>They look nice on a wrist. (<strong>Aesthetic appeal</strong>)</li><li>Rolex has a <strong>limited supply. </strong>(<strong>Rarity</strong>)</li><li>Rolex is a visible <strong>marker of status/class/wealth </strong>because they are expensive, recognizable and well-known as such among the general public. (<strong>Status</strong>)</li><li>Many luxury watches have interesting histories — e.g the Submariner was the first commercially available timepiece to be able to function submerged to up to 100m. The Cartier Santos was the first wrist watch ever made and the Omega Speedmaster was the first watch worn on the moon.<strong> </strong>(<strong>Notable history</strong>).</li><li>Many Rolex watches are associated with famous celebrities or characters e.g. the <strong>Submariner was worn by Sean Connery and Roger Moore in numerous James Bond films</strong>. (<strong>Celebrity association</strong>).</li></ul><p>Gathering these together we find that beyond the function of <strong>telling the time</strong> a Rolex watch also performs the jobs of <strong>aesthetic appeal, rarity, status, notable history </strong>and<strong> celebrity association.</strong></p><p>However Rolex can only do these jobs for you in the physical world!</p><h3>V0 NFTs: On the Internet nobody knows you’re a dog (with a Rolex)</h3><p>On the Internet, nobody can see you let alone what you’re wearing on your wrist. Therefore a Rolex watch isn’t going to to be able to do any of the jobs listed above.</p><p>But as more and more of our lives have moved out of the physical world and into the digital world on the Internet — accelerated by Covid — these jobs formerly done by Rolex now need to be done by something else.</p><p>Enter NFTs. Let’s look at how they can do the jobs of Rolex or other luxury watches:</p><ul><li>NFTs <strong>look good as profile pics</strong> online. This is obviously subjective, but generally people think they look cool. <strong>(Aesthetic appeal)</strong></li><li>NFTs have <strong>limited supply</strong>. Typically no more than 10k for a PFP (ProFile Pic) collection. (<strong>Rarity)</strong></li><li>NFTs can be a marker of status/wealth due to their price and rarity. E.g. Crypto Punks are coveted among Crypto OGs. (<strong>Status)</strong></li><li>NFTs are much newer than watches but the valuable ones have interesting histories. E.g. Crypto Punks were one of the earliest NFTs and inspired the development of the ERC-721 standard. (<strong>Notable history).</strong></li><li>NFTs are quickly being adopted by celebrities, sports figures, etc. E.g. Bored Apes are owned by <a href="https://hypebeast.com/2021/8/stephen-curry-bored-ape-yacht-club-nft-55-ethereum-purchase">Steph Curry</a>, <a href="https://www.benzinga.com/markets/cryptocurrency/21/11/24052185/jimmy-fallon-buys-a-bored-ape-yacht-club-nft-here-are-the-details">Jimmy Fallon</a> and many other well-known celebrities. <strong>(Celebrity association)</strong></li></ul><p>So here we can clearly see that NFTs are doing many of the same jobs in the digital realm that Rolex and luxury watches do in the physical.</p><h3>V1 NFTs: From Luxury Watches To Luxury Communities</h3><p><strong><em>This section was written by my friend </em></strong><a href="https://twitter.com/htormey"><strong><em>Harry Tormey</em></strong></a><strong><em>, an Engineering Manager on a web3 team at Coinbase.</em></strong></p><p>Another purpose an NFT can serve is to signify membership or support for a club or group of people with shared interests.</p><p>The physical world has plenty of examples of exclusive members-only clubs; a good example of this can be seen with <a href="https://www.sohohouse.com/en-us/houses">Soho House</a>. Soho House operates private members clubs, hotels, and venues around the world. Its members, who have access to these venues, are drawn mainly from the media, arts, and fashion industries.</p><p>Membership is selective and its membership policies focus on creativity above net worth and job titles. Professions that are likely to be admitted to the club include Blogger, Fashion Assistant, Socialite, and those working in Advertising. Examples of professions likely to be excluded include Consulting, Finance, and Law.</p><p>In the U.S annual memberships cost $2,100 to join a single club and $3,200 to have access to all of the Soho Houses with discounts given to younger presumably less well off members. So what are the benefits of being part of an exclusive club-like Soho House?</p><ul><li><strong>Community</strong>: You want to be part of a club that puts you in contact with a certain group of vetted people with whom you aspire to socialize, share your professional and or social interests.</li><li><strong>Benefit</strong>: You want to get access to perks or benefits. In the case of Soho House that would be access to amazing venues all around the world.</li><li><strong>Cause</strong>: Perhaps not so relevant to Soho House, but you want to be part of a group of people working towards a common goal.</li></ul><p>On the internet members-only clubs that do this have been a thing for a long time. There are plenty of paid private websites, Discords, or Facebook groups so you might be wondering what’s different about NFT’s? NFT-based communities are different in the following ways:</p><ul><li><strong>Verifiability</strong>: In the same way you can’t verify if someone owns a Rolex online you can’t verify someone is actually a member of an exclusive social club. NFT’s bridge this gap and allow members to signal in a cryptographically verifiable way that they are part of the group.</li><li><strong>Portability</strong>: Anyone can start a Telegram, Discord, Website, game or provide any service and use something like <a href="https://collab.land/">Collab.land</a> to gate access to members that hold a particular type of NFT. The ability to determine membership in a group without having to ask permission from a central authority is a powerful thing as it allows the concept of the group to persist digitally across multiple services.</li><li><strong>Tradability</strong>: You have the ability to sell your NFT on marketplaces such as OpenSea.</li></ul><p>Some NFT based communities are also <a href="https://en.wikipedia.org/wiki/Decentralized_autonomous_organization">DAO</a>’s, they have all of the above attributes but in addition, they also allow:</p><ul><li><strong>Voting</strong>: DAO’s allow members to vote on decisions that impact the community and, for example, what funds raised by the community are used for. Examples of the types of proposals being voted on by DAO’s can be seen on the website <a href="https://snapshot.org/#/">Snapshot</a>.</li><li><strong>Shared Programmable Bank</strong>: Many DAO’s that use NFT’s have a treasury. Funds, usually denoted in Ethereum or <a href="https://en.bitcoinwiki.org/wiki/ERC20">ERC-20’s</a> get deposited in the address associated with the treasury and can be used by the community. These funds can be distributed based on pre-agreed rules codified by smart contracts or by voting on proposals.</li></ul><p>So what are some examples of NFT based communities that exist today and what benefits do they confer? Let’s compare and contrast three popular projects:</p><h3>Developer DAO</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8aFkgN-OD5449ZnO" /></figure><p><a href="https://www.developerdao.com/">Developer Dao</a> is a DAO that consists of thousands of people, mainly developers, who are interested in participating, learning, and contributing to the future of Web3. The community is organized into <strong>guilds </strong>that focus on specific aspects of this mission.</p><p>The NFT representing membership in the DAO is a series of words, some of which are related to software development. In other words, not a PFP (ProFile Pic).</p><p>Access to the Developer DAO discord currently requires that you hold one of the <a href="https://twitter.com/developer_dao/status/1459794282626396163">genesis NFT tokens</a>. This was a limited series of <a href="https://jonkuperman.com/how-to-join-developer-dao/">governance NFT</a> tokens minted when the community was founded by <a href="https://twitter.com/dabit3">Nader Dabit</a>. In future they plan on issuing more NFT’s and opening the server up again to new members. Membership benefits include:</p><ul><li>The ability to vote on proposals via the project’s <a href="https://snapshot.org/#/devdao.eth">snapshot</a>.</li><li>Access to a high quality network of developers interested in web3.</li><li>Access to events put on and sponsored by the DAO.</li><li>The project is planning on issuing a <a href="https://forum.developerdao.com/t/governance-token-proposal-request-for-feedback/835">governance token</a>, twenty five percent of which will be issued to holders of the project’s genesis NFT at the time of the snapshot.</li><li>The project has a treasury which for example includes tokens from its <a href="https://twitter.com/developer_dao/status/1471510755090513921">partnership</a> with <a href="https://twitter.com/gitcoin">gitcoin</a> and donations from other organizations. These proceeds can be used to fund future proposals voted on by token holders.</li><li>It’s a great place to find web3 projects to work on, attract other developers to your project, and get folks to review or signal boost your content or project on Twitter and other forums.</li></ul><h3>Board Ape Yacht Club</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bLdXc-ZfJSsiu5C3" /></figure><p><a href="https://boredapeyachtclub.com/#/"><strong>Bored Ape Yacht Club</strong></a> is a limited NFT collection where the token itself doubles as your membership to the club. Many people who hold one of these tokens set it to be their public profile picture on twitter. Membership benefits include:</p><ul><li>A distinct PFP (ProFile Pic) NFT that allows you to signal that you are a member of a club that’s expensive to join. Many well-known celebrities such as<a href="https://hypebeast.com/2021/8/stephen-curry-bored-ape-yacht-club-nft-55-ethereum-purchase"> Steph Curry</a> and <a href="https://www.benzinga.com/markets/cryptocurrency/21/11/24052185/jimmy-fallon-buys-a-bored-ape-yacht-club-nft-here-are-the-details">Jimmy Fallon</a> own one.</li><li>Additional NFT drops to Ape holders which have value, e.g. <a href="https://opensea.io/collection/bored-ape-kennel-club">Bored Ape Kennel Club </a>where you could adopt a pet dog, <a href="https://opensea.io/collection/mutant-ape-yacht-club">Mutant Apes</a>, etc. Each of these are worth a lot of money in their own right as individual NFTs.</li><li>Events such as concerts, a real-life yacht party as well as <a href="https://www.inputmag.com/culture/bored-ape-yacht-club-nft-nyc-ape-fest">carnival weekends</a> have all been benefits for Ape holders.</li><li>Full commercial ownership of the ape held — for example, you could create a character with a <a href="https://www.coindesk.com/layer2/2021/12/13/neil-strauss-pens-the-bored-ape-yacht-club/">full back story and publish a book</a>. Universal Music Group has <a href="https://nftnow.com/news/bored-ape-yacht-club-band-announced-by-umg/">started a band</a> made up of 3 bored apes and 1 mutant ape.</li></ul><h3>Crypto Coven</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RiM93RFY8oLHvB7O" /></figure><p><a href="https://www.cryptocoven.xyz/"><strong>​​</strong>Crypto Coven</a> is a limited NFT collection with a distinct well-executed look and aesthetic. Like Board Ape Yacht club many people set their social media profile picture to be their witch. Unlike Developer DAO, access to their Discord server does not require holding a token. Unlike BAYC owning a token doesn’t convey any explicit benefits, however it’s worth considering the reasons why someone might own one, which include:</p><ul><li><strong>Aesthetic appeal: </strong>This can be seen in the quality of the art, website and smart contract they developed for their launch. As a result lots of high profile people working in crypto have set their witch NFTs to be their public profile picture on twitter and other social media sites.</li><li><strong>Community</strong>: All of the founders of crypto coven are anonymous women who are talented developers, artists and writers. Unlike a lot of crypto communities this one has a high percentage of women. Many of the women involved with the community have impressive resumes with jobs at high profile tech companies and venture capital, however, in <a href="https://www.cryptocoven.xyz/rollout">rolling out their token</a>, crypto coven placed a big emphasis on <a href="https://twitter.com/natashajuliakim/status/1467900290045292545">inclusivity</a> and diversity so it’s not just all rich people.</li><li><strong>Benefit</strong>: According to the roadmap on their website you are also supporting the development of an immersive, interactive world building experiment which will leverage the witch token you own.</li><li><strong>Cause</strong>: By setting your profile picture to be a crypto coven witch you are essentially associating with those women and signaling that you support them. They provide a non gated discord that gives you access to this community of people, who are interested in participating, learning, and helping others contribute to the future of Web3</li></ul><h3>What NFT Should I buy?</h3><p>To answer this question I am going to talk about several NFT’s I own and why I own them. I am first going to give you some background information about me so you can better understand my motivations. The salient points about me for this conversation are:</p><ul><li>I’ve been a developer for more than ten years. For the last four years I’ve been working full time in crypto.</li><li>I’m an engineering manager who manages a team building a web3 product at Coinbase</li><li>The brand <a href="https://en.wikipedia.org/wiki/Supreme_(brand)">Supreme</a> is not my cup of tea. I’m a fan of horror, role-playing games, dark music, and, <a href="https://en.wikipedia.org/wiki/Alexander_McQueen">Alexander McQueen</a> is one of my favorite fashion designers.</li></ul><p>The NFT’s I decided to acquire are from Developer DAO and Crypto Coven. You can see them here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*SDGJqU9WQFXN389scXbWUg.jpeg" /><figcaption>Harry’s Developer DAO NFT</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*Gz_LGra2rK5gUQszDLhRLQ.jpeg" /><figcaption>Harry’s Crypto Coven NFT</figcaption></figure><p>So why did I acquire these NFT’s?</p><ul><li><strong>Aesthetic appeal: </strong>This is somewhat subjective but if I’m going to put it as my profile picture, I want it to look cool and ideally reflect my personal and professional interests (<strong>Crypto Coven</strong>)</li><li><strong>Community</strong>: I work in crypto and I want to be able to network with and learn from other people working in crypto. (<strong>Developer DAO, Crypto Coven</strong>)</li><li><strong>Cause</strong>: I want to help promote the industry and help women, in particular, find rewarding jobs working in crypto (<strong>Crypto Coven</strong>)</li><li><strong>Benefit</strong>: I’m a hiring manager. I want to hire other people either in or trying to break into the industry. Getting access to events and discords that are full of qualified leads is a big benefit for me(<strong>Crypto Coven, Developer DAO</strong>)</li><li><strong>Tradability</strong>: While I don’t own NFT’s with the expectation of profit, if I get bored, it’s nice to know that I might be able to get some of the money I paid for the NFT back or even make a profit. As a example the price I paid for my Crypto Coven witches was .05 ETH, today the floor price for the project is 0.74 ETH (<strong>Crypto Coven, Developer DAO</strong>)</li></ul><p>In picking a crypto community to be a part of or NFT to represent you, you likely will have different interests and needs than I have but I think this is a good framework for making this decision.</p><p>The use cases listed above are only scratching the surface of the jobs that NFTs can do. There are a number of exciting new possibilities that we will be talking about in future articles. In our next article which will be published on Harry’s blog we will dive into play-to-earn games.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78547c89a998" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ethereum VM Supremacy: 4 Lessons From UNIX Which Explain It]]></title>
            <link>https://medium.com/@niallo/ethereum-vm-supremacy-4-lessons-from-unix-which-explain-it-5a658f5cd7ef?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a658f5cd7ef</guid>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Tue, 28 Dec 2021 19:09:15 GMT</pubDate>
            <atom:updated>2021-12-28T20:10:19.364Z</atom:updated>
            <content:encoded><![CDATA[<h4>Ethereum: One of the earliest production blockchains with a Turing-complete virtual machine enabling <a href="https://en.wikipedia.org/wiki/Smart_contract">Smart Contracts</a>.</h4><h3>Why ETH is going to zero !!!!!</h3><p>Ethereum’s pain points are well-known:</p><ul><li><strong>Slow:</strong> 13 TPS vs VISA’s 1700 TPS</li><li><strong>Expensive: </strong><a href="https://etherscan.io/gastracker">High gas cost</a> — easily $10 USD for a simple transfer, or &gt;$100 USD for smart contract interaction such as an NFT purchase or DeFi deposit/withdrawal.</li><li><strong>Insecure: </strong>The <a href="https://medium.com/hackernoon/the-evm-is-fundamentally-unsafe-d69f2e3b908b">EVM is fundamentally unsafe</a> and smart contracts get <a href="https://www.cnn.com/2021/12/12/tech/crypto-exchange-hacks-explainer/index.html">hacked</a> every day.</li></ul><p>Clearly <a href="https://techcrunch.com/2018/09/02/the-collapse-of-eth-is-inevitable/">the collapse of ETH is inevitable</a> right?</p><h3>And yet Ethereum &amp; the EVM are thriving</h3><p>Not only has the USD price of ETH increased about 10x in 12 months, but the total value of assets locked in the Ethereum network has grown to over $100b USD today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ujtsqTvsNeqi9Z6zCEXFRg.png" /></figure><p>Digging a bit deeper, many of the so-called “Eth Killer” chains are compatible with the Ethereum VM, or are working on compatibility:</p><ul><li>Avalanche <a href="https://docs.avax.network/learn/platform-overview/README#contract-chain-c-chain">C-Chain</a></li><li>NEAR’s <a href="https://docs.near.org/docs/develop/eth/evm">project Aurora</a></li><li>Binance Smart Chain (<a href="https://docs.binance.org/smart-chain/guides/bsc-intro.html">BSC</a>)</li><li>Solana’s <a href="https://neon-labs.org/">NEON</a></li><li><a href="https://smartbch.org/">SmartBCH</a> sidechain for Bitcoin Cash</li></ul><p>Not to mention Ethereum side-chains/L2&#39;s:</p><ul><li>Optimism / <a href="https://www.optimism.io/faqs">Optimistic Ethereum</a></li><li>MATIC / <a href="https://docs.polygon.technology/docs/develop/getting-started/">Polygon</a></li><li><a href="https://developer.offchainlabs.com/docs/inside_arbitrum">Arbitrum</a></li></ul><p>Zoom into some TVL charts and you will see EVM-compatible chains are growing deposits like crazy too but I digress.</p><p>What is happening here? I thought Ethereum was crippled by scaling problems and doomed to collapse. How could it be that the core primitive of Ethereum — the EVM, the thing that makes Ethereum useful — keeps growing and gaining market share as a smart contract platform?</p><h3>4 Lessons From How UNIX and POSIX Took Over The World</h3><p>In 1969 a little operating system called UNIX started development at Bell Labs.</p><p>What are 4 reasons UNIX became popular and other OSes did not?</p><ul><li><strong>Portable</strong> to many hardware platforms<strong>,</strong> owing to largely being written in the C language rather than machine code.</li><li><strong>Composable</strong>: Combine small tools via pipes and shell scripts, program using nice APIs for signals, processes, everything-is-a-file philosophy, etc.</li><li><strong>Popular with developers/researchers</strong> because the source code was low-cost/free and easy to access and share.</li><li><strong>Scale</strong>. Powered large scale super computers and Internet infrastructure.</li></ul><p>In 1988 the POSIX or IEEE 1003 standard was released, based on UNIX, and specified the core OS interfaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ePtiOgXaGZ4yjNo1aQN40w.png" /></figure><p>Today, <a href="https://gs.statcounter.com/os-market-share#monthly-202009-202109">64% of worldwide operating system market share across all devices is UNIX-inspired and POSIX-compliant</a> (Linux/Android + iOS/macOS) and if you add in Windows’ arguable POSIX compliance, you can make that figure closer to 96%.</p><p>It is astounding that potentially <strong>96% of all devices today are running an OS compatible with a standard largely developed in the 1970s</strong>.</p><h4>SIDEBAR ON POSIX COMPLIANCE</h4><blockquote>In 1991, Linus Torvalds released the first version of the Linux kernel — which became a free OS inspired by UNIX and mostly compliant with the POSIX standard.</blockquote><blockquote>In 1993, Microsoft released the POSIX subsystem for Windows NT 3.5, which implemented IEEE 1003, and all NT-derived versions up till and including Windows 7 were <a href="https://qr.ae/pGq5LB">basically POSIX compliant</a>.</blockquote><blockquote>In 2007, Mac OS X 10.5 was certified POSIX compliant and iOS (based on the same Darwin base OS) was released.</blockquote><blockquote>In 2008, Android was released which was based on Linux and therefore mostly POSIX-compliant.</blockquote><h3>How The EVM is like UNIX</h3><p>What are some of the reasons that EVM usage is expanding among these competitor chains to Ethereum? They are reminiscent of the 4 lessons of UNIX:</p><ul><li><strong>Portable</strong>. There are<a href="https://eth.wiki/concepts/evm/implementations"> multiple existing EVM implementations</a> written in different languages and many <a href="https://docs.smartbch.org/smartbch/smartbch_whitepaper-en#moeingevm">interesting improvements</a> have been published. You can re-use existing smart contract code and deploy pretty much unmodified Dapps to any EVM compatible chain. EVM compatibility is not tightly coupled to the blockchain consensus mechanism.</li><li><strong>Composable</strong>. You can stand on the shoulders of giants - build your Dapp on top of other EVM DeFi primitives aka <strong>Money Legos </strong>such as AMMs, DEXes, insurers etc. All of this reduces friction to getting liquidity and usage on your chain/Dapp and serves as a competitive moat for the EVM.</li><li><strong>Developer adoption. </strong>There is a critical mass of documentation, existing source code, training programs, programmer communities, etc in EVM smart contract development. Developers can re-use their knowledge of Solidity, dev tools, concepts, documentation and source code very easily on any EVM-compatible chain.</li><li><strong>Scale. </strong>Billions of dollars of deposits secured, millions of real users, many thousands of Dapps. Powers the real world Dapp infrastructure.</li></ul><p>I argue that EVM is not likely to be disrupted in the Clay Christensen<strong> </strong>sense<strong> </strong>and replaced in the same way that Yahoo was by Google for search, or Myspace was by Facebook.</p><p>Instead, the EVM is more like UNIX or POSIX: A core standard, developed relatively early in the history of blockchain, that due to its fundamental attributes and adoption, will takeover and hold very significant market share for the foreseeable future.</p><p>My advice: If you’re going to build Dapps, build them on the EVM. If you’re building a smart contract blockchain, build it on the EVM.</p><h3>The Counter Case: Alternatives to an EVM future</h3><p>What else could happen? What if I’m wrong about the UNIX/POSIX analogy altogether? I thought it would be nice to talk to a some very knowledgeable folks in the Crypto space to get their thoughts and then share their perspectives.</p><p><a href="https://twitter.com/_patrickogrady"><strong>Patrick O’Grady</strong></a><strong>, Ava Labs (and Coinbase alumni)</strong></p><p><a href="https://twitter.com/_patrickogrady">Patrick O’Grady</a> — who recently <a href="https://twitter.com/_patrickogrady/status/1471223325045100545?s=20">released</a> an interesting feature for Avalanche called <a href="https://github.com/ava-labs/subnet-evm">Subnet EVM</a> which makes it easy for anyone to release their own <a href="https://docs.avax.network/build/tutorials/platform/subnet/create-evm-blockchain/">EVM-compatible blockchain on Avalanche</a> — gave me some of his thoughts on EVM and smart contract platforms.</p><p>In our conversation, he agreed that the EVM is the dominant smart contracting platform and mentioned that he hasn’t received any inbound requests from developers asking for another smart contract-compatible VM on Avalanche. He also pointed out that blockchains that have developed their own smart contract-compatible VMs are now going to great lengths to support Solidity using an “EVM” deployed as a contract (Aurora on NEAR and Neon on Solana).</p><p>Patrick thinks that the sticky piece of the EVM is really just Solidity-compatibility. The plethora of Solidity code samples under permissive OSS licenses makes it trivial for anyone to launch a powerful dApp overnight. In the future, there will be a proliferation of EVM-like VMs that support most of Solidity but make major changes behind the scenes to increase throughput (which may end up looking nothing like the <a href="https://ethereum.org/en/developers/docs/evm/">existing EVM spec/design</a>). He pointed out that we’ve already seen this idea play out in the L2 space to ensure anything that occurs in a rollup can be re-executed on ETH in the case of a challenge and in ZK-land as Solidity must be simplified to support execution in a circuit.</p><p>Lastly, he mentioned how excited he was for Subnet EVM to become a testbed for all sorts of EVM experimentation. It is now one of the easiest ways to spin up a live EVM for testing (all networking, consensus, low-level storage abstracted away by AvalancheGo) and hopes the innovations that come out of tinkering with it make it back to the rest of the ETH community. If you are curious about testing more complex modifications/getting funding to work on interesting optimizations, you can <a href="https://twitter.com/_patrickogrady">reach out to him on Twitter</a>.</p><p><a href="https://twitter.com/jai_prasad17"><strong>Jai Prasad</strong></a><strong> (Crypto Product Manager at Coinbase)</strong></p><p>I also ran some of these ideas about EVM supremacy by my colleague <a href="https://twitter.com/jai_prasad17">Jai Prasad</a> (one of my favorite PMs on the Crypto team at Coinbase) and he sees a multi-standard world. In his view, the EVM, for next 5 years, will be the dominant standard and garner a lot of attention, but it will not make sense for all use cases. Solana &amp; NEAR will be used for some specific high throughput use-cases like gaming or high frequency trading</p><p><a href="https://twitter.com/cgst"><strong>Cristian Strat</strong></a><strong>, Runloop (and Coinbase alumni)</strong></p><p>I chatted with <a href="https://twitter.com/cgst">Cristian Strat</a>, founder at DeFi simulation company <a href="https://runloop.xyz/">Runloop</a> about this topic. He thinks the analogy with POSIX is intriguing, but doesn’t think it helps the case for EVM supremacy that much. He had the following comments:</p><p>You could argue there’s a need for standardizing interfaces. We could perhaps use the Solidity language (though not any particular implementation) to describe those interfaces, just like POSIX defines interfaces with C. POSIX-like compliance for programmable blockchains would be pretty helpful and well-supported by your analogy. By comparison, the EVM spec (opcodes, interpretation, etc.) is too far down the stack to serve this purpose IMO.</p><p>The other thing to consider is that a lot of the exciting stuff our computers do (hardware-accelerated graphics, UI kits, drivers, etc.) is outside the scope of POSIX. To get the unique advantages of each platform and compete with the native apps, you’ll likely have to roll up your sleeves and learn the ins and outs of your platform. For example, the EVM is probably not the right abstraction for writing efficient provable programs. Even if you could do it with zkEVM/similar, you’re at a disadvantage compared to folks who use more efficient circuits. We saw this with dydx — they didn’t just port to StarkEx; they redesigned the system to leverage StarkEx.</p><h3>Conclusion</h3><p>In the end, only time will tell whether the EVM reigns supreme as a smart contract platform decades from now, in a similar way to the POSIX standards which emerged from UNIX in the 1970s.</p><p>Or perhaps the EVM standard is just a stepping-stone, like many earlier technologies —maybe more like MS-DOS than UNIX— something that had a huge amount of market share for a good while, but ultimately was replaced by better technologies.</p><p>It is inarguable that today, however, the EVM is having its moment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a658f5cd7ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reliability Engineering at Coinbase]]></title>
            <link>https://medium.com/the-coinbase-blog/reliability-engineering-at-coinbase-8b6956ba802f?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b6956ba802f</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[reliability]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Thu, 21 Mar 2019 17:35:24 GMT</pubDate>
            <atom:updated>2019-03-21T18:06:39.627Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Reliability Engineering?</h3><p>Why is Reliability Engineering relevant at a company like Coinbase? Why would we want to build a Reliability Engineering team?</p><blockquote>“Our goal is to make Coinbase the <strong>most trusted</strong> and easiest to use digital currency exchange.”</blockquote><p>-Brian Armstrong, Co-founder &amp; CEO</p><p>It all comes back to what our CEO Brian Armstrong said about Coinbase wanting to be the most trusted. Our goal in the cryptocurrency industry is to <a href="https://blog.coinbase.com/the-vision-mission-and-strategy-for-coinbase-944b79a64a7c">create an open financial system for the world</a> — and part of that requires us to build the most trusted digital currency exchange. In order to be the most trusted exchange, we need to be the most reliable. Being reliable is a competitive advantage in our industry, while being unreliable is a serious risk to our business.</p><p>Before you get too deep into this article, please note that we’re actively hiring great Reliability Engineers, so if any of this sounds interesting to you please head over to our Senior Reliability Engineer job posting <a href="https://www.coinbase.com/careers/positions/1525954">here</a>.</p><h3>What is Reliability Engineering?</h3><p>The mission of the Reliability Engineering team at Coinbase is:</p><blockquote>“Help engineers design &amp; keep their promises in production.”</blockquote><p>The word “promise” in our mission statement is a reference to <a href="https://www.amazon.com/Thinking-Promises-Designing-Systems-Cooperation-ebook/dp/B01092PYG8">Promise Theory</a> which was invented by Mark Burgess. While we use many of the principles from the <a href="https://landing.google.com/sre/books/">Google SRE books</a>, we found Promise Theory to be more human-friendly than the term “Service Level Objective” which is a bit jargon-y. Based on the investigations into safety and reliability by people like <a href="https://en.wikipedia.org/wiki/Sidney_Dekker">Sidney Dekker</a> and companies such as Toyota (see <a href="https://en.wikipedia.org/wiki/The_Toyota_Way">the Toyota Way</a>), we consider reliability to be ultimately a human challenge. For this reason we preferred to reference a concept which every human already understands — that of making and keeping promises.</p><p>Major differences between Reliability Engineering at Coinbase vs Site Reliability Engineering (SRE) at some other companies:</p><ul><li><strong>We are generalist software engineers first and foremost.</strong> We focus on solving challenges by writing better software rather than adding more and more humans to push buttons. Everyone on the team is a strong software engineer, working on multiple software systems in a variety of programming languages.</li><li><strong>We do not have front-line pager responsibility</strong>. We are on-call for the systems that we ourselves own (e.g. <a href="https://blog.coinbase.com/logs-metrics-and-the-evolution-of-observability-at-coinbase-13196b15edb7">the Coinbase observability stack</a>), but we are not the first line of incident response for other teams. Service and product teams have their own pager rotations.</li></ul><p>We like to apply the metaphor of ‘teaching a person to fish vs giving them a fish’ to how we operate — our mission is to “teach teams to fish” in terms of reliability. This is in contrast to “giving them a fish” by handling front-line pager duties on their behalf. Another way of putting it is that our goal is to up-level every engineering team at Coinbase to be self-sufficient in Reliability Engineering.</p><h3>How do Reliability Engineers work?</h3><p>One of the important things to realize about reliability engineering is that it is inherently cross-cutting throughout the organization. Reliability is not itself a functional silo — it is a value and a business output. Our customers are every single engineering team at Coinbase. Since we work with so many customers, we have defined different models of engagement to meet their needs:</p><ul><li><strong>Advisory</strong>. This is answering questions, or responding to ad-hoc requests without formal deliverables. For example responding to “Help me monitor/scale/improve my thing” questions in Slack, or jumping into production incidents to support responders.</li><li><strong>Consulting</strong>. We often run structured reliability workshops and pairing sessions with other teams. In these engagements, we have a shared goal (in our case, <a href="https://en.wikipedia.org/wiki/OKR">OKR</a>) with the team we’re consulting with — thus there is a measurable outcome. While consulting engagements are formal, they are typically part-time endeavours.</li><li><strong>Embedding.</strong> Sometimes teams will need full-time reliability support from our engineers, and they request that we physically sit and work with them, participating in their standups, sprint plannings, etc. This is where we use embedding. Similar to Consulting, this work has a shared goal and measurable outcome (OKR) — the difference is the reliability engineer is a temporary (typically, one calendar quarter) member of the customer team.</li></ul><p>Beyond the various ways we engage with customers, we follow a standard “agile” software engineering process. We have a weekly planning meeting to update our Kanban board, conduct monthly retrospectives and hold daily standups. Longer-term strategy and measurements are captured in quarterly OKRs which we derive from customer feedback and internal discussion.</p><h3>Introducing the Coinbase Reliability Engineering Team</h3><p>The Reliability Team was founded in 2018 with one engineer (Luke Demi) and myself (Niall O’Higgins) as manager. Since then, we’ve grown to 7 engineers and shipped a lot of improvements.</p><p>In the words of folks on the team, here are some accomplishments we can speak about publicly as well as impressions and experiences from working on reliability!</p><p><strong>Luke Demi</strong></p><p>After joining Coinbase in 2016, my initial efforts within the company focused on building self-service infrastructure for engineers. However in 2017 as interest in cryptocurrency surged, Coinbase began to experience outages across our systems. Solving these <a href="https://blog.coinbase.com/how-were-scaling-our-platform-for-spikes-in-customer-demand-4a047cb3139c">types of reliability problems</a> excited me, so I dove in head first to get to the bottom of these issues.</p><p>We were able to survive 2017, but it was clear that in order to withstand future surges and provide a reliable experience for our customers we would need to make reliability a core component of the engineering culture at Coinbase.</p><p>I find the Reliability Team exciting because we’re able to both advise teams on best practices for choosing reliability indicators (Service Level Indicators AKA SLIs) and promises (AKA SLOs) as well as build the tools that let engineers understand the performance of their systems in production.</p><p><strong>Maksym Naboka</strong></p><p>I joined Coinbase in July 2018. Being the 3rd engineer on the Reliability Team was an amazing experience. There are so many things I love about the company and I’d like to highlight few of them:</p><ul><li>An opportunity to work with / learn from smart and talented people.</li><li>Project ownership. An engineer on the Reliability Team owns a project all the way through from design to shipping.</li><li>Ability to contribute to Open Source.</li><li>Learn, learn and learn. Coinbase provides so many opportunities to learn new technology. It feels like we are utilizing every spare minute to learn new things! We have Lunch &amp; Learn sessions with guests from leading technology companies, every engineer has an annual educational budget to go to conferences or take online classes.</li><li>Delicious meals on site :)</li></ul><p><strong>Amy Li</strong></p><p>When I first joined the Reliability Team in November 2018, I was under the impression that I would be thrown into the deep end of blockchain — drowning in Bitcoin, Ethereum, and smart contracts. Colleagues also warned me of endless firefighting and nightmarish on-call rotations. Fortunately, this was not the case.</p><p>The Reliability Team doesn’t work with blockchains directly and aren’t the first ones being paged for every single incident. Each Coinbase team owns the daily operations of their specific products or services. This allows for distributed knowledge across the organization.</p><p>As a new college graduate I initially felt overwhelmed, but everyone on the team has been incredibly supportive and willing to share their knowledge. Within a month, Niall and I improved our incident management system by integrating it with JIRA. I wrote my first design document to further integrate PagerDuty with our incident management system and I am continually making incremental changes to our system.</p><p>One of the most important things I’ve learned is that working with amazing team members is priceless. The Reliability Team is a group of curious, empathetic, and intelligent individuals and there’s no other group I would rather be with for five days a week.</p><p><strong>Paul Henry</strong></p><p>The most interesting part of being on the Reliability Team for me is our high-level perspective across the organization. Since we are not tasked with handling day-to-day operations of any specific Coinbase product (Coinbase.com, Coinbase Pro, Coinbase Wallet, etc), we can focus on improving the ability for teams to observe and understand their systems. This means that teams can move faster, incidents are resolved quicker, and there’s a decentralization of knowledge across the organization.</p><p>Here’s some examples of improvements that I’ve contributed to over the past year:</p><ul><li>Writing lightweight stats, tracing, and logging libraries for the various languages in use across the organization.</li><li>Contributing to “paved roads” for various languages and ensuring that developers have a good starting point for new services, with sane defaults.</li><li>Introducing new vendors (such as Datadog) to bring more dimensions of observability, unlocking new ways of monitoring systems.</li><li>Bringing a perspective of reliability to technology choices made by teams and helping them ask the right questions.</li><li>Contributing to our deployment tooling to integrate high level monitoring by default on all services.</li><li>Enabling the use of gRPC across the organization through client generation in various languages and integration into our AWS architecture. See blog post “<a href="https://blog.coinbase.com/grpc-to-aws-lambda-is-it-possible-4b29a9171d7f">gRPC to AWS Lambda: Is it Possible?</a>”</li></ul><p>In addition to shared tooling, we engage with many teams across the organization by running workshops, review sessions, and office hours.</p><p>Workshops are hands-on sessions that focus on topics like observability tooling and promise construction, within the context of that team’s services or problem domain.</p><p>Review sessions happen both early in the design process for services and later when they are nearing production. These reviews do not act as a gate or “green check mark” for teams, but instead make sure that they are asking the right questions and highlighting ways that the reliability team can level up teams across the organization.</p><p>Office hours are open time every week for any engineer to bring problems or feedback to our team by pairing with an engineer. Topics usually include: how to build effective monitors and dashboards, integrating tracing or metrics libraries, what database should I use for this particular problem, and more.</p><p>At the end of the day, my favorite part about the Reliability Team is the diverse set of engineers we have. The breadth and depth of knowledge shared by everyone is a great support structure for tackling a problem of any scale.</p><p><strong>Jordan Sitkin</strong></p><p>I have an unusual background for an infrastructure engineer. I studied graphic design in school and worked for the first half of my career as a designer. Joining the Reliability Team was, for me, the latest step in a long, ongoing journey away from the front end. I’ve really enjoyed the new challenges I’ve faced on this team and have been pleasantly surprised at how often my experience as a designer ends up being relevant here.</p><p>My favorite part about being on the Reliability Team is being close to where the excitement is happening across the company. The greatest need for reliability expertise is often around new product launches or new-found success of some existing product. We’ve been pursuing a new model of embedding reliability engineers in other teams where their expertise is needed most. I’m personally currently embedded in the Consumer team, which is responsible for Coinbase.com and the Coinbase mobile apps. I’ve enjoyed feeling close to the front lines of product development while still focusing on infrastructure.</p><p>Another rewarding aspect of being on the Reliability Team has been turning our work into conference talks. Over the past year I had the chance to speak at <a href="https://www.youtube.com/watch?v=i6ws1JpvNs0">MongoDB World</a> and QCon about designing load testing strategies. I had never given a talk before, so this was a great learning opportunity for me and I ended up having a lot of fun doing it.</p><p>Working on the Reliability Team is one of the most fun positions at Coinbase because we get to be a part of so many different initiatives and projects across the company. We’ve got a great diversity of expertise on the team. I’ve never learned so much so quickly.</p><h3>Reliability Engineering and the Future</h3><p>In the past year, our team has helped all of Coinbase build a culture of reliability in the following ways:</p><ul><li>Moving the entire engineering team from a reactive stance on reliability (firefighting, etc.) to a proactive one (installing smoke detectors) with service level indicators and promises.</li><li>Providing a world-class observability stack comprised of three pillars — tracing, metrics and logs.</li><li>Designing and implementing high-performance infrastructure services.</li></ul><p>We look forward to doing much more over the next year such as:</p><ul><li>Building the serverless foundation to accelerate feature development.</li><li>Helping move to a service oriented architecture by building core infrastructure such as the service mesh.</li><li>Leveling up every single team in terms of performance engineering, quality and incident response.</li></ul><p>If any of this sounds interesting to you please head over to our Senior Reliability Engineer job posting <a href="https://www.coinbase.com/careers/positions/1525954">here</a>.</p><p><em>This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b6956ba802f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-coinbase-blog/reliability-engineering-at-coinbase-8b6956ba802f">Reliability Engineering at Coinbase</a> was originally published in <a href="https://medium.com/the-coinbase-blog">The Coinbase Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Hard Lessons from Scaling Continuous Deployment to a Monolith with 70+ Engineers]]></title>
            <link>https://engblog.nextdoor.com/3-hard-lessons-from-scaling-continuous-deployment-to-a-monolith-with-70-engineers-99fb6dfe3c38?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/99fb6dfe3c38</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 21:41:18 GMT</pubDate>
            <atom:updated>2017-12-08T22:25:56.483Z</atom:updated>
            <content:encoded><![CDATA[<p>Like many successful tech startups (Instagram, Slack, New Relic, and many more) a large portion of Nextdoor is a monolithic application. While we’ve also created a significant number of microservices — especially in places where high throughput is needed — a large amount of our product and feature work still takes place in our Django monolith.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ira6drmdCBpJ64zH6VNI4g.png" /><figcaption>Our developers work on unrelated components of the monolith at the same time</figcaption></figure><p>This means that the majority of our 70-member (and growing) engineering team is regularly working on a large monolithic Python codebase. We recently completed a very significant project to move from ~4 releases of the monolith per week to Continuous Deployment. See our related post <a href="https://engblog.nextdoor.com/how-nextdoor-made-a-10x-improvement-in-release-times-with-docker-and-amazon-ecs-35aab52b726f">How Nextdoor Made a 10x Improvement in Release Times with Docker and Amazon ECS</a>.</p><p>With Continuous Deployment at Nextdoor, changes are released to production as soon as they are ready and as quickly as our build and deployment infrastructure can manage. Engineers land directly to master and we avoid long-lived branches. Furthermore, the workflow around deployment and rollback of the monolith is fully automated. There isn’t even a button that an engineer needs to push! Typically, we have dozens of releases per day in a fully-automated system where any engineer can deploy or rollback production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1G1SIuocTUu747RqCuJQig.png" /><figcaption>High-level continuous deployment workflow</figcaption></figure><p>Making this operate smoothly has required lots of thought and polish on our workflow and the tooling around it. There isn’t much public information on how these details are managed at scale nor what kinds of tools you need to do this well. Since releasing code is at the core of our business, we invested in building a great tool to manage this complexity, named Conductor. We hope to open source Conductor in the near future. In this blog post we aim to share the biggest lessons we learned the hard way, and give a sneak peek into our tooling and workflow.</p><h3>1) Continuous Deployment is much harder with lots of engineers and code</h3><p>It’s one thing to practice Continuous Deployment (henceforth abbreviated to CD) with a microservice owned by a team consisting of a handful of engineers. A small team will introduce changes at a slower rate than a large team. When you have 70 engineers working on the same large application, the rate of change is much higher. Changes are being made in completely different areas of the application by different teams at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*sLjYMmLYR_H1bqHH5nGszQ.png" /><figcaption>Microservices typically have fewer changes per release vs monolith</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*Ha4DsPxRVdqFXZ6dKH5hAw.png" /></figure><p>Furthermore, in a microservice the surface area for defects is much smaller. A defect in a microservice will generally be limited in terms of its potential blast radius compared to a defect in a monolith. In contrast, one small bug in a monolith can take down all functionality of the site. This is one of the downsides to a monolithic application — slowness or errors in one part can adversely effect the entire system, rather than just their local area. There is little isolation. Therefore, managing the high rate of change and blast radius is a critical ingredient in successful CD workflows.</p><h3>2) Use trains to manage many changes</h3><p>With CD of a microservice, you might be able to have the release size consist of a single change — i.e. do one release for each individual change. However at the scale of the Nextdoor monolith, this isn’t usually possible. Given the test, build and deploy overhead of our monolith (about 25 minutes total), if we released a single change at a time, changes would get backed up and it could take hours between engineers landing their changes on master and them going out to production. We aim to get changes live as quickly as possible once they’re landed on master. It becomes difficult and inconvenient for engineers should there be too large a delta or too much uncertainty around when the change they landed will be live on Production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLS2A3aDNZydW0HG6z1Ekg.png" /><figcaption>An example of a train, including number of commits, deployment time, and a “rollback” button.</figcaption></figure><p>This is where we introduce the concept of a “train”. A train is basically the smallest unit (or batch) of releasable changes, and we use the following rules to negotiate what is on a train.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*B7FXI7_15FXbG2r4YsFrOg.png" /><figcaption>Train creation logic</figcaption></figure><ul><li>There can be only one train “running on the tracks” at a time.</li><li>When a change lands on master, if there is already an existing train, it is queued for the next train. To use a train-based metaphor, the changes are “waiting on the platform” until a new train arrives. Note that before a change can be landed on master, a large suite of automated tests must pass and it must be accepted by a human reviewer.</li><li>If there is no existing train, a new train will be created containing all changes up to HEAD of master. Thus, all of the queued changes or those “waiting on the platform” are put on this new train.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eyszTgZaSbtnMswo4Zl8hg.png" /><figcaption>Train phases (left to right)</figcaption></figure><ul><li>Trains go through three phases. First, Delivery. This phase includes build and deployment to our Staging environment.</li><li>Then, Verification. This includes both automated verifications such as unit test runs, smoke tests along with human verification. Human verification is tracked by polling the state of tickets — we use JIRA. Tickets are automatically created for each engineer with non-dark launched changes on the train once it is delivered to Staging.</li><li>Finally, Deploy. Once all of the verifications are complete — which means all the automated tests have passed and any manual verification tickets have been closed by the engineers— the train will auto-deploy to production.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OOEmtRfQz_Hydxp2ggjltA.png" /><figcaption>Train in verification phase. Human verification completed, waiting on automated tests.</figcaption></figure><ul><li>Once a train starts to deploy to production, a new train is implicitly created as soon as there are any queued changes. To maximize throughput, we don’t wait for deployment to complete before starting on the next train.</li><li>If there is a problem with a train, it can be manually extended via a button in the UI. Any engineer can press a button to include a fix or revert commit for something urgent. Train extension simply pulls in all queued commits — up to HEAD of master. This will slow down the current train, and therefore everyone else on it, so it should be used sparingly. Ideally, automated test coverage is good enough that you don’t find problems at this stage and there is no need to extend for a revert or fix.</li></ul><p>One of the advantages of the way we have implemented trains is that releases are typically quite small. The smaller your releases the better — since the likelihood of a problem increases with every additional change. Furthermore, when something does go wrong, it is much easier to find the offending change from within a small batch size than a large one.</p><p>Engineers are encouraged to put their changes behind a feature flag and we provide a framework called Feature Config which powers this. If a change is behind a feature config, we let engineers bypass the manual verification process. The rationale here is that the feature can be manually verified at any time, potentially with a small number of users initially, and if it has a bug, it can be switched off instantly.</p><h3>3) Kill release teams — democratize deployment workflows instead</h3><p>If you’re an engineer who has just finished a change, you have a strong incentive to see it released as soon as possible. Similarly, if you as an individual engineer introduce a defect you have a very strong desire to fix it quickly and get that fix live on production immediately. You’ll be much happier and more productive if you can drive this workflow yourself.</p><p>If the workflow is instead driven and controlled by a dedicated team (e.g. a “release team”) the incentives aren’t necessarily the same. For example, if a bad data migration is in a release, it’s best to give the people who had changes in that release visibility so that they can quickly debug it and fix it.</p><h4><strong>Align Incentives</strong></h4><p>One of the philosophies at Nextdoor is to align incentives at the right levels to boost productivity as much as possible. Toward this goal, we pick a random individual who has a change on the train to be the “train engineer”. The train engineer is responsible for frontline triage of issues. They act as a release engineer but for a very small release. For example, if an automated test fails during the verification phase of their train, they will be sent a Slack message telling them to triage the issue and resolve the problem. This is typically resolved by them reading test failure output, triangulating a likely candidate change on the train, and then co-ordinating with that change’s author to get a fix or a revert onto the train to get it moving again.</p><p>In practice, the train engineer system works well. If a particular train is delayed or has issues, there is clear individual responsibility for tracking down verifiers, performing a rollback or extending the train with a fix. Since the train engineer by definition has a change on the train themselves, they also have a strong incentive to resolve problems quickly. By selecting the train engineer at random, release process knowledge and load is gradually spread throughout the organization.</p><h4>Get Humans Out Of The Way</h4><p>Another observation was that human verification can be extremely slow. For example, perhaps an engineer has a change on the train and then they go into an interview for an hour or a series of meetings — meanwhile everyone else is held up. To tackle this problem, we have created a culture where it’s agreed that excessively slow verifications aren’t acceptable behavior. It’s considered as serious a breach of etiquette as breaking tests in master. Other people on the train will go and track down the person with some urgency, or possibly find someone else to verify it — or in extreme cases revert the offending commit entirely.</p><p>While this culture helps to a significant degree, humans will make mistakes. The best way to avoid having a human forget to do their verification is to make it unnecessary for them to verify in the first place. To aid in this we have introduced additional tools and workflows where you can bypass needing to verify your change on Staging entirely and therefore not risk holding everybody up:</p><ul><li>Pre-land verification environments, called Preview Environments. These are unique, per-code-review, fully-isolated Staging-like environments which engineers can use to verify their changes before they land them on master. If they use this capability, they bypass Staging and its manual verification requirements entirely. Thus there is no human verification requirement for their change during the release process.</li><li>Feature Config. If your change is behind a feature config, you don’t need to verify manually on Staging since your change has no impact on Production by default. It can be rolled out to a limited audience and quickly turned off if it has bugs or issues.</li></ul><h3>Conclusion</h3><p>At Nextdoor, we have been continuously deploying dozens of releases per day of our large monolith for about 10 months. We have 70+ engineers working on this codebase. The benefits of Continuous Deployment to the speed of our product development organization have been significant.</p><p>However, the unique challenges posed by both a monolithic application and a rapid pace of change have required us to invest heavily in workflow optimization and tooling to achieve this. It took a team of 3 engineers about 9 months to figure out all of the intricacies of the workflow, developer user experience, and to then build the associated tooling. At the core is the Conductor microservice, which glues together Jenkins, Slack, Github and JIRA to present a coherent user interface and enforce rules. Since every engineer at Nextdoor interacts with Conductor to ship their code we wanted to make it as pleasurable as possible to use.</p><p>We plan to open source Conductor — which was designed to be modular with pluggable support for third party services— in the near future.</p><p>I’d like to give a huge shout out to the engineers on the Dev Tools Team who have contributed to this project and this blog post: <a href="https://medium.com/u/4148785e95d2">Steve Mostovoy</a>, <a href="https://medium.com/u/f75d092779f4">Rob Mackenzie</a>, <a href="https://medium.com/u/21224cebe9ea">Mikhail Simin</a> and <a href="https://medium.com/u/f106a2736754">Alex Karweit</a>.</p><p>Find this sort of stuff cool? The Nextdoor engineering team is always looking for motivated and talented engineers.</p><p><a href="https://nextdoor.com/jobs/">Careers at Nextdoor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99fb6dfe3c38" width="1" height="1" alt=""><hr><p><a href="https://engblog.nextdoor.com/3-hard-lessons-from-scaling-continuous-deployment-to-a-monolith-with-70-engineers-99fb6dfe3c38">3 Hard Lessons from Scaling Continuous Deployment to a Monolith with 70+ Engineers</a> was originally published in <a href="https://engblog.nextdoor.com/">Nextdoor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Nextdoor made a 10x improvement in release times with Docker and Amazon ECS]]></title>
            <link>https://engblog.nextdoor.com/how-nextdoor-made-a-10x-improvement-in-release-times-with-docker-and-amazon-ecs-35aab52b726f?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/35aab52b726f</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Tue, 10 Jan 2017 18:11:05 GMT</pubDate>
            <atom:updated>2017-01-10T18:12:33.421Z</atom:updated>
            <content:encoded><![CDATA[<p>At the beginning of 2016, Nextdoor production releases took about an hour. That is to say, once all the code in the new release was tested and verified, it would still take an entire hour before our users saw those changes. Packaging the software took about 25 minutes and then actually deploying that package to production would take another 30 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*eAvw5tBmwRIsYp30awKGjw.png" /></figure><p>“That’s madness,” you say. “Why on earth did things take so long?”</p><p>Let’s dive deep into what made this so slow, and how we made a 10x improvement.</p><h4>Snail-paced staging</h4><p>Before we release code to production, our engineers perform acceptance testing for their changes on a production-like environment, which we call staging. Product managers, designers, and engineers can use staging to help verify new features or behaviors before the code is released to all of our users.</p><p>Staging has a similar dataset to production, and runs all the services in the same way — the only difference is how many EC2 instances are powering it. Our staging deploy pipeline used the same architecture as production; hence, staging was similarly slow to update. This meant that we were heavily limited in the number of changes we could test per day on staging. If you’re an engineer working on a complicated new feature and you only have 2–3 opportunities to test it in a full production-like environment per day, that severely reduces your speed of iteration. Slow staging and production turn-around was hurting our engineering team’s ability to move fast and get new features out quickly.</p><h4>Too much toil and trouble</h4><p>Not only was this process slow, it was also far more manual than we wished. Each release would require running several scripts and inputting values by hand, which introduced potential for operator error. None of the release process should require manual intervention — we knew there should just be a single “deploy button” to push.</p><h4>Roll backs are wrong</h4><p>Finally, a frightening implication of all this lag was that if we needed to push an emergency fix, we’d be looking at a minimum of a 1 hour turn-around — and potentially almost double that if we needed to test it on staging first. This meant that when things went wrong, we tended to roll back the entire release rather than roll forward with a patch. Rolling forward is preferable to rolling back, since important features and fixes aren’t reverted due to an unrelated bug.</p><p>It was obvious to us all that the slowness in our build and deployment pipeline was becoming a major drag on productivity. We needed to dig in and solve this.</p><h4>Enter the Dev Tools Team</h4><p>One of the non-technical reasons why things were so slow was that the various components of the build and deploy processes didn’t have clear team ownership, and had been neglected. The innards of the build &amp; release systems had grown organically over the years and consisted of spaghetti code with various fixes slapped on top. We’d never taken a step back and thought holistically about the entire system and its architecture.</p><p>We decided we needed to create a new team which could focus on repairing our build and release pipelines. Staffed with 3–4 engineers, the high-level goal of the Dev Tools Team has been to maintain an engineering platform that supports rapid development and release of high quality code. Our philosophy is that our engineers are customers, and our job is to make their experience developing, testing, and deploying their work pleasant and efficient.</p><p>Our plan was simple and consisted of 3 steps:</p><ul><li>Understand the existing build and deploy pipeline.</li><li>Automate it.</li><li>Optimize it.</li></ul><p>And our team for this project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*eo32foZSx_dPyXXpX4fUFQ.jpeg" /></figure><h3>Understanding unhurriedness</h3><h4>25 minutes “build” for Python app — WAT</h4><p>Our primary service is a large Django/PostgreSQL monolith which runs under nginx/uWSGI. Python is an interpreted language. It doesn’t require compilation. This might make you think that there’s nothing to build, but that turned out to be not entirely true.</p><p>We distributed our release to our servers as a self-contained binary Debian package. This Debian package contained all the application source code, the Python virtual env, uWSGI and various other binary requirements.</p><p>In order to build Debian packages in a clean, reproducible way, you need to use a tool such as sbuild or cowbuilder which effectively build them in an <a href="https://en.wikipedia.org/wiki/Chroot">empty chroot environment</a>. This is great — except for the fact that you need to populate the contents of the chroot with system libraries. Anyone who has used a Debianoid Linux is familiar with waiting for apt-get install to complete.</p><p>We also needed to compile frontend assets (<a href="https://engblog.nextdoor.com/nextdoor-loves-webpack-796fc3249c12#.o5sobm5s0">using Webpack and associated tooling</a>), and push those to the CDN before starting the deploy.</p><p>Finally, the Debian package itself needed to be uploaded to our Debian package host — which could take several minutes in practice for bizarre reasons which I’m not going to delve into here.</p><p>Since we had not architected any of this for end-to-end performance, we ended up with a breakdown like the following:</p><ul><li>apt-get install system package into Chroot (10 minutes)</li><li>pip install Python packages (5 minutes)</li><li>Compile frontend assets &amp; upload to CDN (5 minutes)</li><li>Upload resulting Debian package to hosting provider (5 minutes)</li></ul><p>This added up to around 25 minutes.</p><h4>30 minutes to deploy the Debian package — WAT</h4><p>So we have a binary Debian package. It should just take a few minutes to push that out to our servers, right? Not so fast. We had architected our deploy to use a<a href="https://engblog.nextdoor.com/how-we-rollback-code-releases-in-less-than-a-minute-936c4855c381#.ggc50u4je"> Red/Black process inspired by Netflix</a>. We would boot an entirely new set of servers to run the new release, install the Debian package onto those and then switch traffic at the load balancer to point to the new servers. In case of a rollback, we still had the old release running, so it would simply be a matter of pointing the load balancer back at the old servers.</p><p>This architecture has a number of nice properties — such as exercising our ability to replace machines frequently. However, the cost was that it was very slow:</p><ul><li>Bidding, booting, and configuring the new EC2 instances (25 minutes)</li><li>Wait for new version of services to become healthy and join load balancer (5 minutes)</li></ul><p>This added up to around 30 minutes. Staging used the same Red/Black architecture, and so had a similar performance profile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*G_WJh53CFt_ZY22V2aWdJw.png" /></figure><p>One final note on this architecture is that you end up booting tons of new instances. Each time you do this, there is a chance of a failure. EC2 itself can have outages, Apt repos can go down, etc. At a certain scale, we began having quite frequent failures in bringing up new instances — which very often required painful manual intervention to work-around. This kind of operational toil was a drag on our team.</p><h4>Automation and the lack of it</h4><p>There is an old rule-of-thumb in software engineering that, if you find yourself performing the same manual task three times, it is worth automating it. Our release engineers were doing a lot of repetitive manual tasks, and we had tons of opportunities to apply this adage.</p><p>Here are some of the most onerous examples of manual tasks in our staging pipeline:</p><ul><li>If an engineer wanted to get a commit into the existing release branch, they would manually add a comment with their SHA to a special JIRA ticket.</li><li>The release engineer would periodically run a “git cherrypick” script by hand to scan the JIRA ticket for new SHAs and apply those to the branch in git.</li><li>The release engineer would be the first to encounter any merge conflicts, even though they don’t have any context on the changes.</li><li>The release engineer would manually trigger a new build, wait, and then babysit the deployment to staging.</li></ul><p>We knew that none of this was necessary. With better tooling, engineers could be responsible for performing their own cherrypicks and resolving any merge conflicts. Builds could be automatically triggered whenever new commits landed on the branch. A new build could be automatically deployed to staging and a Slack notification sent to inform the developer that their change is ready for acceptance testing.</p><p>Our first major achievement was automation of the existing staging pipeline, which freed us from operational overhead enough to work on building the new architecture. As we gradually dug ourselves out of the hole of manual operational toil, we continued to invest in improving automation and reliability.</p><h3>Migrating from Debian/Amazon EC2 to Docker/Amazon ECS</h3><p>While we already had some organizational experience with both Docker and ECS — our dev boxes were built on Docker Compose, and we had numerous microservices in production which we deployed on ECS. This alone wasn’t the driving reason for our migration.</p><h4>Win 1: Huge build speed-ups with Docker Layer Caching</h4><p>As described above, we specifically wanted to increase the speed at which we could build and release new code. On the build side, we realized that we could gain huge speed-ups through caching infrequently-performed but expensive operations. Perfect examples are installation of system packages — which rarely change — and Python requirements which change more frequently, but still not that often.</p><p>While we could have built caching into the legacy Debian packaging system, we felt that it would be at least as much work as moving to Docker. Debian packaging tools are byzantine and nobody on our team had a good understanding of them. Furthermore, Docker has quickly emerged as a standard packaging and runtime format. If you package your application as a Docker image, you can build and run it in tons of different environments. Since we had adopted Docker already for our microservices and development environments, and therefore already had a fair amount of in-house familiarity with it, it was a natural fit.</p><p>In re-architecting our package build process under Docker, we were obsessive in optimizing our use of the provided layer caching. By fanatically avoiding doing more work during builds than we needed to do, we were able to get the build time from ~25 minutes to ~2–3 minutes on a machine with a warm cache. Since our cache hit rate is very high (system packages and Python dependencies change very rarely), this easily provided a 10x improvement in average build time.</p><h4>Win 2: Reusing machines with ECS Docker Host Clusters</h4><p>It was fantastic to decrease our build times. However, we still had an unacceptably slow deployment speed. This was mainly because we would bring up a whole new set of machines on each deploy. This is inherently time-consuming and error-prone. What if we instead maintained an elastic pool of servers which were always ready? That’s essentially what Amazon ECS offers. You run a cluster of Container Instances (which can be scaled easily), and then ECS takes care of deploying and scheduling your application containers on that cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*0_7rqj56U45InRv79hnkIA.png" /></figure><p>Since we had migrated our monolithic Django app to Docker, it was fairly straight-forward to see how this would work by migrating our staging system to ECS. We quickly saw huge drops in deployment times — to about and average of 2–3 minutes on staging. Production takes a little longer because there are so many more container instances, but is on the order of 7–8 minutes. This could be sped up further by having additional ECS host capacity.</p><h4>Production cutover — with zero downtime</h4><p>It’s one thing to migrate your staging environment to a new architecture, but it’s another to migrate the full production workloads and release process. Not only did we need to come up with a plan to perform the migration with zero downtime to users, we also needed to have some way to do this gradually and roll back in case we found issues.</p><p>Like many real-world monoliths, the Nextdoor Django app in fact runs as seven different “service flavors”. These are instances of the exact same code, running in a slightly different mode and serving different sets of traffic. For example, there is an “api” service flavor for mobile clients and a “taskworker” service flavor for asynchronous job execution. Each service flavor runs under a dedicated Amazon ELB. These service flavors provided a natural grouping upon which to perform the migration. We could migrate production one service flavor at a time, using the DNS on the ELB to roll back to the EC2-based system if necessary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*qcCkDohVHSPVweejZx8iIw.png" /></figure><p>We started with one of the low traffic service flavors, and worked our way gradually to migrating the highest traffic ones as we gained confidence. The entire migration took about 3 weeks, mainly due to two tricky issues which took us quite a lot of time to debug: processes dying mysteriously and occasionally hung asynchronous taskworkers.</p><h4>Issue 1: OOM kills under Amazon ECS</h4><p>There isn’t a direct way to translate from a system where the minimum unit of scaling is an entire EC2 instance to a system where the unit of scaling is a single container. We had previously thought only in terms of scaling coarsely — by booting or terminating EC2 instances. We hadn’t thought of more fine-grained scaling — i.e. by X units of CPU and Y memory units. We needed to think deeply about our application’s resource requirements and tune the number of worker processes given the amount of memory we were allocating. We couldn’t simply say “let’s just boot another X machines and let the application have as much memory as it wants”. ECS will require you to set clear limits, and if your containers exceed those limits they will be killed unceremoniously.</p><p>In an environment like <a href="http://deeplearning.net/software/theano/tutorial/python-memory-management.html">Python where the interpreter seemingly never returns memory once it requests it from the OS</a>, your processes can seem to grow in memory forever. This manifested as us initially setting memory limits too low and suffering OOM kills of important processes as a result. It took a week of careful watching and tuning to get things to a point we were happy with. We found ECS memory usage graphs were problematic as — given metrics are collected only one per minute — they could easily miss spikes.</p><h4>Issue 2: Mysterious hanging taskworkers</h4><p>One of our largest service flavors is our <a href="https://engblog.nextdoor.com/nextdoor-taskworker-simple-efficient-amp-scalable-ac4f7886957b#.yj5o0weda">asynchronous job processing system we name “taskworker”</a>. We run a big pool of taskworker processes which pulls tasks from Amazon SQS queues.</p><p>We migrated a portion of this service to Docker/ECS and started to notice some very strange behaviors. In certain cases, the Docker/ECS taskworkers were getting stuck and hanging forever. However the legacy Debian/EC2 taskworkers didn’t have this problem. Doing some deep system call-level debugging, we managed to track down the source of the hang. Only under Docker/ECS, our taskworkers would read from a PostgreSQL connection without a timeout — and for some reason the result would never arrive nor would any error occur on the socket — thus hanging the process. We came up with various crazy theories about why this was happening such as Docker networking issues, kernel bugs, and more. We also started to work on handling stuck taskworkers by doing operations in a child process, and handling a timeout in the parent process which would notice the child was stuck and restart it.</p><p>However the Infrastructure team found the underlying issue independently — in porting the Debian package build to Docker we had inadvertently upgraded the PostgreSQL C library “libpq” from 9.1 to 9.6. This later version of libpq seemed to have some subtle backwards-compatibility issues communicating with our 9.4-version database servers. It would issue a read on a socket without a timeout and no error would ever occur — hanging the process forever.</p><p>When we pinned the version of libpq back to 9.1 — where it had been in our Debian build — this problem disappeared.</p><h3>Conclusion</h3><p>It took our team around 5 months to re-architect the build and deployment system to run under Docker/ECS and completely overhaul all our automation. This investment has paid off enormously in terms of the following benefits:</p><ul><li>Build &amp; deployment down from about an hour to 5–6 minutes (ECS capacity willing).</li><li>Fully automated staging environment. Developers push commits onto a release branch and they are live on staging for verification within 5 minutes — no manual operation required.</li><li>Vastly simplified build and release process for production. Operator simply types “!release” in a Slack channel and a Slack button is displayed to confirm deployment of release. Click the button and the production deployment process is begun.</li><li>Massively improved release reliability. ECS and Docker have helped us simplify our systems so that they have many fewer moving parts and points of failure. Not having to worry about bringing up large numbers of new EC2 instances on every release drastically reduces our exposure to random failures.</li></ul><p>However, it’s not all rainbows and unicorns. We have run into numerous issues with both Docker and Amazon ECS, which we plan to go into in future blog posts. Key learnings:</p><ul><li>Docker layer caching works, but it is coarse-grained. Furthermore, <a href="https://github.com/docker/docker/issues/20316">due to some changes, the cache cannot be populated by a Docker pull</a>. It must be built locally. Expect to invest quite a lot of time on optimizing if you have a complex build and you want to make it very fast.</li><li>Amazon ECS works reasonably well, although it has rough edges. In particular, you cannot differentiate between a failing health check and startup grace period. If your application takes a while to become healthy on startup, you must set a long health check duration. You cannot say “wait for 10 minutes for my application to pass health check on startup, then ratchet down the threshold”.</li><li>To migrate to Docker, be prepared to think deeply about your resource utilization in terms of CPU and memory required per container.</li></ul><p>What’s your deployment process like? We’d love to hear about your experiences making builds and deployments fast!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35aab52b726f" width="1" height="1" alt=""><hr><p><a href="https://engblog.nextdoor.com/how-nextdoor-made-a-10x-improvement-in-release-times-with-docker-and-amazon-ecs-35aab52b726f">How Nextdoor made a 10x improvement in release times with Docker and Amazon ECS</a> was originally published in <a href="https://engblog.nextdoor.com/">Nextdoor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fury lifts the heavyweight belt and ushers in post-Klitschko era]]></title>
            <link>https://medium.com/@niallo/fury-lifts-the-heavyweight-belt-and-ushers-in-post-klitschko-era-a0304bd36a6?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0304bd36a6</guid>
            <category><![CDATA[boxing]]></category>
            <category><![CDATA[sports]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Sun, 29 Nov 2015 06:01:13 GMT</pubDate>
            <atom:updated>2015-11-29T06:07:07.593Z</atom:updated>
            <content:encoded><![CDATA[<p>Tyson Fury — perhaps the most oddball boxing personality of all time — defied the odds and beat a weary-looking 39-year-old Wladimir Klitschko to a fairly boring unanimous decision this evening. The awkward but massive 27-year-old cousin of WBO middleweight world champion <a href="https://en.wikipedia.org/wiki/Andy_Lee_%28boxer%29">Andy Lee</a> —who was coincidentally trained by the late great <a href="https://en.wikipedia.org/wiki/Emanuel_Steward">Emanuel Steward</a>, just as Wladimir was — succeeded where for the past 10 years many others have failed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*Wg_mfosuKZtOJl1HhuMtLQ.gif" /><figcaption>self-uppercuts happen to the best of us</figcaption></figure><p>As the 39-year-old Klitschko walked into the ring looking haggard and frankly a bit wrecked, it appeared to me that perhaps Fury’s bizarre pre-fight antics — including <a href="http://www.dailymail.co.uk/sport/boxing/article-3246215/Tyson-Fury-dresses-BATMAN-press-conference-Wladimir-Klitschko-entered-superhero-s-theme-tune.html">a staged Batman appearance at the press conference where he tussled with and subsequently apprehended a costumed villain planted in the audience</a> — had indeed thrown a spanner in the workings of Dr Steelhammer’s brain box.</p><p>As early as the second round there was a clear sense that something was wrong with Wladimir as the then-champion threw close to no punches whatsoever. The taller Fury managed to use his greater speed and reach to negate the Ukranian’s offense all the while “Mayweathering” him — continually beating Klitschko to the punch and racking up points by touching him with backhand jabs.</p><p>According to Compu-Box, in the first two rounds, Klitschko landed one power shot while Fury landed four.</p><p>Simply put, this pattern continued for the rest of the bout — producing what was ultimately a boring fight which won’t do much to change popular perceptions of the basically-dead spectacle of contemporary heavyweight boxing.</p><p>Despite increasingly desperate pleas from his corner, Wladimir was unable to adjust to an opponent who was taller, faster and more scientific than any recent foe. His typical routine of jabbing and head-hunting from a comfortable distance followed by holding and attempting to tie up the other man simply didn’t work against someone who was both larger and a skilled in-fighter. This resulted in Klitschko handing over points to Fury who took every opportunity to land punches in the clinch, and due to his size was never much slowed by the attempted leaning of the smaller Ukranian.</p><p>When Klitschko should have been deploying his greater power to the body of the challenger on the inside, he was wasting time, energy and opportunity by using a tactic that simply wasn’t working.</p><p>Ultimately, Fury had an effective plan which he managed to execute well while Wladimir showed a complete lack of ability to adjust and just didn’t seem to be mentally committed to the contest at all. I see this fight as a champion performing absolutely abysmally while a challenger performed well enough to pry out a win. The overall lack of activity make this fight to be a bit of a stinking anti-climax.</p><p>After over 10 years of Ukranian dominance it’s nice to see a changing of the heavyweight guard however <a href="http://www.dailymail.co.uk/sport/boxing/article-3308639/Is-Tyson-Fury-fit-fight-Wladimir-Klitschko-world-heavyweight-title-Read-vile-homophobic-slurs-bizarre-rants-devil-worshippers-Armageddon.html">Fury’s outspoken, repugnant and moronic bigotry</a> make it difficult for me to be much of a fan. With <a href="http://www.badlefthook.com/2015/11/28/9812628/klitschko-vs-fury-wladimir-klitschko-to-exercise-rematch-clause">Wlad saying he’ll exercise his rematch clause</a> next year we’ll get to see a replay before long. But given he’ll be 40 by then and his heart doesn’t seem to be in it, how likely is it he’ll be able to make the needed adjustments? For my own part, I’m looking forward to Fury getting in the ring with some more interesting opponents such as <a href="https://en.wikipedia.org/wiki/Deontay_Wilder">Deontay “started boxing late” Wilder</a> or the hyped up British heavyweight <a href="https://en.wikipedia.org/wiki/Anthony_Joshua">Anthony Joshua</a>.</p><p>In the mean time treat yourself to the <a href="https://www.youtube.com/watch?v=qmDwmpISZ3I">grim, beer-filled post-fight press conference</a> featuring both PhD-holding Klitschko brothers answering in German and then self-interpreting their responses into English.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0304bd36a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fast Polygon Self-Intersection Detection in JavaScript]]></title>
            <link>https://engblog.nextdoor.com/fast-polygon-self-intersection-detection-in-javascript-a405ba40dd50?source=rss-5ed1aec8c690------2</link>
            <guid isPermaLink="false">https://medium.com/p/a405ba40dd50</guid>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Niall O'Higgins]]></dc:creator>
            <pubDate>Wed, 21 May 2014 19:52:00 GMT</pubDate>
            <atom:updated>2016-02-22T20:33:04.424Z</atom:updated>
            <content:encoded><![CDATA[<p>We do a lot of work with polygons at Nextdoor. Given that we manage tens of thousands of private neighborhood communities, we have a huge number of geometric neighborhood boundaries which are stored as polygons.</p><p>The quality and accuracy of neighborhood boundaries is hugely important to our product, so we work to build good boundary editing tools.</p><h4><strong>What is a Polygon Self-Intersection</strong></h4><p>One of the common classes of inconsistencies in neighborhood boundaries is self-intersection. This is when a boundary intersects itself.</p><p>This can be difficult for a human editor to spot, especially when editing a boundary polygon with many points. What if our boundary editing tools could automatically detect such self-intersections after each edit, highlight the location and notify the editor?</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*gQojkA-eQ6LLt5qu.png" /></figure><h4><strong>Self-Intersection Algorithms</strong></h4><p>It turns out that implementing polygon self-intersection checks in the browser is not only possible, it’s straightforward and performant. The algorithms to find self-intersections are well researched and documented. For example the <a href="http://en.wikipedia.org/wiki/Bentley-Ottmann_algorithm">Bently-Ottman algorithm</a> is one efficient approach — O((n + k) log n) — with many existing implementations.</p><h4><strong>JavaScript Spatial Predicates With JSTS Topology Suite</strong></h4><p>An excellent library for doing geospatial processing in JavaScript is the <a href="https://github.com/bjornharrtell/jsts">JSTS Topology Suite</a>. This is in fact a port of the <a href="http://www.vividsolutions.com/jts/JTSHome.htm">Java JTS library</a> to JavaScript. JSTS has a similar API and feature set to the Java JTS library.</p><h4><strong>Self-Intersection Detection in JavaScript with JSTS</strong></h4><p>The first step is to ensure we have our polygon co-ordinates in a representation which JSTS can understand.</p><p>First of all ensure you’ve included the JSTS JavaScript library:</p><pre>&lt;script src=“/js/jsts.js” type=“text/javascript”&gt;&lt;/script&gt;</pre><p>Here is a function which converts <a href="http://geojson.org/">GeoJSON</a> co-ordinates to JSTS coordinates:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/edb47497ee53ac6a64f118aa06a3b029/href">https://medium.com/media/edb47497ee53ac6a64f118aa06a3b029/href</a></iframe><p>Now we can write a simple function to wrap JSTS to detect the self-intersections:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0df732e2840af6c00d9aa2e87505691b/href">https://medium.com/media/0df732e2840af6c00d9aa2e87505691b/href</a></iframe><h4><strong>Conclusion</strong></h4><p>Using this code, you can implement your own self-intersection detection in a browser! It’s easy to integrate this code with <a href="https://developers.google.com/maps/documentation/javascript/">Google Maps JavaScript API</a>. If you’re interested in working on these kinds of problems and more, <a href="https://nextdoor.com/jobs/">we’re hiring</a>.</p><p>Niall O’Higgins<br> Senior Software Engineer</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a405ba40dd50" width="1" height="1" alt=""><hr><p><a href="https://engblog.nextdoor.com/fast-polygon-self-intersection-detection-in-javascript-a405ba40dd50">Fast Polygon Self-Intersection Detection in JavaScript</a> was originally published in <a href="https://engblog.nextdoor.com/">Nextdoor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>