<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Davi de Castro Reis on Medium]]></title>
        <description><![CDATA[Stories by Davi de Castro Reis on Medium]]></description>
        <link>https://medium.com/@daviws?source=rss-13a36f31dadc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iaLQ6_ri0mtbB4sepAF6Ug.jpeg</url>
            <title>Stories by Davi de Castro Reis on Medium</title>
            <link>https://medium.com/@daviws?source=rss-13a36f31dadc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:03:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@daviws/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Engrave your SDLC]]></title>
            <link>https://daviws.medium.com/engrave-your-sdlc-04b428a40dcf?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/04b428a40dcf</guid>
            <category><![CDATA[bazel]]></category>
            <category><![CDATA[cuelang]]></category>
            <category><![CDATA[nushell]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 01:10:20 GMT</pubDate>
            <atom:updated>2026-11-15T11:50:37.340Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build (agentic) codebases that blossom, not rot</h4><p>Not many years ago, it was very hard for people to move around codebases. Even highly technical people with extensive domain knowledge would frown on grabbing random code and starting to work on it. They would rather have lunch with the codebase owner and discuss the problems and solutions. And that was very effective, because it takes a while to master a new tool, and new codebases came with several of them.</p><p>The industry was split into two schools on how to organize codebases. One was centralized, and most large companies took that route. Most codebases at Google use the same tooling, so you can shift teams around and become productive quickly. This enabled the never-ending defrags, short for defragmentation, that the company did to reallocate engineers across projects, a nuisance for employees but invaluable for an organization that needed to react to market changes. Facebook adopted the same approach, as did most large tech companies. Startups are averse to centralization and usually fall into the second school: instead of letting codebases grow, they build new ones. The trade-offs are real and yield different answers depending on organization size and maturity. Then agents arrived, and suddenly every codebase gets a new contributor every few minutes.</p><h3>Common verbs, engraved</h3><p>When I first built <a href="https://github.com/bonisoft3/sayt">Sayt</a> (pronounced /seɪ/, like the verb “say”), I tried to create codebases that feel familiar but use their own custom-fit tooling. The trick was to acknowledge that software has different points in its lifetime, and people consume code in different roles. I discuss this idea extensively in my book, <a href="https://www.amazon.com/dp/B0CXW4F1XN">Scaling your Artifacts with Your Team</a>, and you see the Sayt acronym born out of that title. In the early days of a codebase, you have one to a handful of people who are experts in that codebase. They use highly specialized, efficient flows, custom-built for each task. As a codebase becomes successful, more people contribute casually, and even those involved full-time can no longer know it all in depth; each change they make will require either a larger span across parts of it or (slow) coordination across experts.</p><figure><img alt="Sayt banner: “A dev cycle that works for anyone, anywhere, anytime.” above the command sayt build, with plants growing from a dashed timeline." src="https://cdn-images-1.medium.com/max/1024/1*bOM0PBUayapVSgI9l7xqDg@2x.png" /></figure><p>The idea in Sayt was to find the common verbs that most codebases share and encode a mapping from those early-day flows into the common target. For example, how does one build this code? How is it launched? A perfect mapping would defeat the purpose; the efficiency of the early-day workflows comes precisely from the deviation that enables specialization. Is npm run dev what we should see as the launch command? Is npx tsc a lint or a build command? Will it use the tool versions I expect? Does it even work outside the first developer&#39;s laptop? When you adopt Sayt in your codebase, you are engraving the answers to these questions into the configuration.</p><h3>Work with your tools, not around them</h3><p>Encoding bespoke practices in version control results in a codebase that is predictable and self-explaining. And Sayt makes those properties robust. It can bootstrap itself and your build tools anywhere, from your colleague&#39;s laptop to a scratch Docker image. Unlike systems that want to own your codebase, like Bazel or Dagger, Sayt lets you work with your tools. It only provides the machinery to describe how it is used and to support the operations around it.</p><p>You describe your flows in YAML, and if they get too complex, you can jump into CUE, a superset of JSON that offers strong composition capabilities but is still pure. If you need impure operations, you can reach out to Nushell, a shell language that is also, surprisingly, a superset of JSON.</p><p>To give a taste of each format’s expressiveness, here is the release verb configured three ways. The args are passed through to the underlying release tool.</p><p>.say.yaml, plain:</p><pre>say:<br>  release:<br>    args: &quot;--changelog --clean&quot;</pre><p>.say.cue, with a constraint, a reference and a string operation:</p><pre>package say<br><br>import &quot;strings&quot;<br><br>_target: *&quot;staging&quot; | &quot;production&quot;<br>_flags: [&quot;--changelog&quot;, &quot;--clean&quot;, &quot;--release-notes=notes/\(_target).md&quot;]<br><br>say: release: args: strings.Join(_flags, &quot; &quot;)</pre><p>.say.nu, reading the environment:</p><pre>let tagged = $env.GITHUB_REF? | default &quot;&quot; | str starts-with &quot;refs/tags/&quot;<br><br>{<br>  say: {<br>    release: {<br>      args: (if $tagged { &quot;--changelog --clean&quot; } else { &quot;--snapshot --clean&quot; })<br>    }<br>  }<br>}</pre><p>Sayt, CUE and Nushell all work across macOS, native Windows, or Linux (glibc or musl) and can be installed with Mise, a portable Rust binary you can bootstrap from virtually anywhere. None of this is about your code; Sayt is all about the surroundings of a codebase and how it fits into general software engineering practices. In any Sayt-enabled codebase, on any machine, you can just git clone and sayt build. Or, as the name suggests, just say build. You commit a tiny wrapper, saytw, in your repository, and Sayt self-bootstraps, saving contributors the trouble of installing it.</p><p>It is a simple concept, but not easy to execute. Take reproducibility versus flexibility: if you are coding in PHP, Sayt will induce you to pin your PHP version. But you can&#39;t use the same exact binary on all platforms. On your laptop, you may want the glibc-linked version, but in CI you may need the musl-linked version. Or you need to favor the version that lives in a fast, unthrottled artifact store. There are always details that need to be accommodated, or not, and Sayt makes these decisions explicit. Sayt steers you in pragmatic directions, while solving the &quot;works on my machine&quot; problem.</p><h3>Verbs, platforms and directories</h3><p>In Sayt, we have defined five verb pairs that mean roughly the same thing at different altitudes, from a naked machine where you just cloned your repo, to the cloud where your code is deployed. In each pair, one verb does something, and the other validates the result. Getting to the verb names themselves is a big challenge. People and tools attribute different semantics to those names, and we wanted to find a reasonable common ground. It is up to the user to map their interpretation of those names to actual operations in their codebase. The exercise invariably increases the quality of the codebase. Sometimes no reasonable mapping exists, and Sayt lets you change everything: the semantics of each verb, the tools associated with it, and even the list of verbs itself. Even if you change everything, you still have Sayt&#39;s powerful configuration mechanism to enforce your own conventions in code.</p><figure><img alt="The five Sayt verb pairs, each a verb that does and a verb that validates, by layer: toolchain (setup, doctor; @bare), static (generate, lint; @repo), app (build, test; @host), stack (launch, integrate; @compose) and public (release, verify; @preview)." src="https://cdn-images-1.medium.com/max/1024/1*qabwrJjfHqrL2y9DIpFTGg@2x.png" /></figure><p>Read the layers top to bottom, from fastest to slowest. In the toolchain layer, setup installs your tools through mise and doctor checks that each tier has what it needs. In the static layer, generate produces code and configuration and lint validates it. In the app layer, build and test map to the tasks your editor already runs. In the stack layer, launch brings the whole system up with Docker Compose and integrate exercises it end to end. In the public layer, release ships it with tools like GoReleaser or Skaffold, and verify confirms the result is healthy. Each layer also has a default platform, from @bare to @preview, and no verb blocks the other.</p><p>Each Sayt verb is part of a triple (verb, @platform, directory/). When you run sayt build, it is shorthand for sayt --verb build --platform host --directory $PWD. Each of these three dimensions can independently fully replace the command&#39;s behavior through configuration. This gives choices to different codebases; some will have sayt build@debug (a custom debug platform), others will choose sayt -d debug/ build (a custom debug directory). Both are legitimate, and different choices make sense as your codebase grows. This is the main insight behind Sayt: software is dynamic, and your tools should understand not only the single developer coding flow, but also the long horizons of software development life cycles, where people come and go, markets change, and products transform. But they shouldn&#39;t trade one loop for the other; your tools need to deliver excellence in both the inner loop that runs in minutes and the outer loop that runs over years.</p><h3>From humans to agents</h3><p>What was first built for humans now needs to prove valuable for LLMs or perish. A growing share of the code produced in our industry is no longer authored by humans. Developers no longer need to master any tools, just the concepts behind them. The LLM will figure out the details of how to operate a codebase. Every time. From scratch.</p><p>But like humans, LLMs have limited context. They can learn fast, but if they need to jump from place to place, they will spend more time learning than doing. Agent fleets already have specialized roles; they master subsets of the codebase or are given personas so they can focus on specific knowledge areas. Organizational complexity seems to be an emergent property, regardless of whether an organization&#39;s members are made of flesh or silicon. This may not hold true forever; the speed at which LLMs are evolving is astonishing. But at least for now, two things seem to hold true: monorepos are a superpower for agents, who can connect far-away dots much faster than going through indirections, and Sayt-enabled codebases let them quickly start operating that far-away code and validate hypotheses. Combine monorepos with Sayt, and you can unleash your agents and see your codebase flourish.</p><h3>The Sayt TDD loop</h3><p>Sayt also exposes a skill for LLMs: the Sayt TDD loop. Whenever something goes wrong (a build breaks, a bug is reported), you or your LLM can initiate the Sayt TDD loop. It starts by noting at which level the problem was reported, because that is where it needs to go back to verify correctness. But instead of iterating at the level of the report, which can be slow, the loop instructs you to keep narrowing the verbs and then the specialized actions the verb can call until you can reproduce the problem in a layer that iterates quickly. Once you fix it there, you start to bubble up, and if more problems arrive, you rinse and repeat until you can bubble all the way to the reported level and claim a fix. This avoids two traps that LLMs often fall for: first, wasting a lot of time iterating on a slow tier. Second, declaring success too early, only for the frustrated user to open the UI and see the problem again. With the Sayt TDD loop, the LLM itself will open the UI.</p><figure><img alt="The Sayt TDD loop: “Ping-pong until green. Then cascade.” A detected bug drops back through the layers to where it reproduces fastest; each layer ping-pongs until green, then the fix advances one layer at a time until it is green and merged." src="https://cdn-images-1.medium.com/max/1024/1*9n47s5E_hq6NRxpg0_GFrg@2x.png" /></figure><p>As mentioned, Sayt leads to configuration that lets you run your code everywhere. This greatly reduces production skew, as well as the skew between a developer’s machine and CI. When an LLM sees a Sayt-enabled CI failure, it can easily reproduce it locally for much faster iteration, and once it gets a local fix working, it follows the Sayt TDD loop discipline to drive CI green on the first try.</p><h3>Try it</h3><p>If you want to try Sayt, point your favorite code assistant to <a href="https://github.com/bonisoft3/sayt">bonisoft3/sayt</a> and ask it to engrave your SDLC, then enjoy a portable codebase that does not rot, where agents roam free at full speed.</p><h4>Bonus: just say build</h4><p>In my own repositories, I go one step further: a .justfile at the repo root aliases sayt to say for the whole tree, so the command I actually type is exactly just say build.</p><pre>[no-cd]<br>say *args:<br>  {{justfile_directory()}}/saytw {{args}}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04b428a40dcf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing a portable codebase]]></title>
            <link>https://daviws.medium.com/writing-a-portable-codebase-3c4cb681f763?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c4cb681f763</guid>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Thu, 22 Aug 2024 12:36:30 GMT</pubDate>
            <atom:updated>2024-08-22T17:06:52.208Z</atom:updated>
            <content:encoded><![CDATA[<p>Every new codebase is started by a single individual working on one computer. That does not last long, though. New environments where the code must run and be built pop up quickly. Sometimes, it is a new, shiny laptop used on the run. Often, it is a ci/cd pipeline where you want to mimic the local commands. And if the code is valuable, there will be new contributors with their operating systems and habits. Very quickly, things get out of control as each machine where the code runs has its quirks, and a far-from-trivial amount of work needs to be done to make the codebase portable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*700vFKjHqzA9Zyq-.jpeg" /></figure><p>There are two ways of approaching this problem. You can try to control the environment, for example, by making all machines look the same, or you can try to make the codebase portable so that it works on different machines. Large organizations take the first path by specifying hardware and operating systems and tooling for their employees and deployment environments. In contrast, loosely centralized or fully distributed organizations, such as open source, have no option and go through the latter.</p><p>Both are roads full of obstacles and pains, with significant trade-offs. There is a third path somewhere in between, making the controlled environment portable. Our industry has attempted this many times with limited success. For running software in production, we have certainly seen great advancements, from VMs to containers, but the results in the development environments have been paltry, at least. Tools like Vagrant, Devcontainers, and Tilt have tried to backport production isolation layers to developers&#39; screens for a while. Although they are not without merit, widespread adoption is yet to be seen. The problem is that too many workflows, idioms, and tools exist.</p><p>Reproducibility in a dynamic environment is also very hard to achieve. Production code is primarily read-only, and you can pack your release in a nice sha256 signed blob, like a docker image, and be confident you have the same bits running every time. However, by definition, development environments are constantly changing. The span and the frequency of variation across dependencies, operating systems, memory amount, disk speeds, and CPU architecture are way beyond what is seen in production, and on top of that, code is being live edited and experimented with. You get surprising, long-lasting side effects from things like compilation or integration tests, and suddenly, your code feels like a delicate flower that can only blossom in the fragile, poorly understood garden that is your local computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*enDRmPE2-Xh7m_-B8Zy9Ng.png" /><figcaption>Check out <a href="https://geek-and-poke.com/">https://geek-and-poke.com/</a> for more fun</figcaption></figure><p>The exercise we will do in this write-up is to build a pragmatic portable codebase. We will leverage the local environment and tools for the more developer-oriented stages and move towards stronger isolation layers as we dive into the operations world. This is a balancing act since the software development landscape is ever-changing. We will introduce concepts, which tend to last longer, back them up with tools, which inevitably rot, and seam them together with conventions that shall help bridge the lifetime gap among those.</p><p>We require that everything can be bootstrapped and successfully operated in Windows, Mac, and major Linux distributions. Also, it should support Apple and Intel silicon and be easily lifted to container-centric environments for things like CI/CD and cloud deployments. We will restrict ourselves to a multi-repo design for simplicity but hint at extending the design to larger monorepos with internal dependencies. The ideas are polyglot in essence, and can be applied to most common programming languages, and we will showcase a handful of them. Finally, the techniques are gradual, which means you can start a new codebase with them or incrementally move an existing codebase towards portability.</p><p>From the social perspective, the codebase should have a smooth onboarding experience. It should suit companies hiring engineers fast through a central process with locked-in standardized laptops or a hobby project that takes contributions from random individuals on the internet writing code on their personal computers. It should also be a comfortable place for LLMs, the programmers of the new world. For all these to work, we rely on existing, battle-proven, and future-safe tooling rather than trying to write an all-encompassing system as many have tried before.</p><h3>The developer toolbelt</h3><p>Three leading IDEs are widely used today. Visual Studio Code leads the pack as a highly extensible solution, IntelliJ and the JetBrains family offer the most integrated solution, and NeoVim found a niche. There are many other interesting options and a bunch of new exciting players popping up, like Zed or Helix. From our perspective, we will focus on Visual Studio Code, not for its merits, which are many, but because its popularity makes it widely supported by other tools.</p><p>Hence, we will start by leveraging Visual Studio Code configuration to describe how code should be built and how unit tests can be run. This can be expressed in the .vscode/tasks.json file, and plenty of recipes can be found around the internet on how to set this up for any language or framework. It also has cross-platform support, offering different invocations per operating system. Whether you use Visual Studio Code or not, this standard configuration can be leveraged to integrate with other tools. For example, the <a href="https://github.com/NathanVaughn/vscode-task-runner">vscode-task-runner</a> allows you to run vtr build on the shell and build your code the way Visual Studio Code would have. Or, if you are into NeoVim, you can use <a href="https://github.com/stevearc/overseer.nvim/blob/master/doc/guides.md#vs-code-tasks">Overseer</a> for a more tightly integrated experience. Ultimately, this is the most popular configuration format for building and testing code; you only need to commit it to your repository to let many tools benefit.</p><p>The next item in our toolbelt is a command runner. For this, we will use <a href="https://github.com/casey/just">casey/just</a>. Besides being an excellent portable tool, it has a few critical properties. First, it is composable through its import statement. This is critical for monorepos, and we favor tools that have a composition mechanism. Second, it has repository-scoped invocation semantics, which means that it can fall back to the git root for a default configuration. With this, one can run just build in any directory, and a root .justfile can delegate to vtr build which will run the instructions on that directory .vscode/tasks.json . This is an excellent first win; it does not matter if it is your operating system, the editor you use, or your programming language. With minimal intrusion, any code can now be built with a single uniform command in the terminal.</p><p>No matter how powerful the tools you have, every codebase relies on some ad-hoc scripts. Bash is a popular option but does not scale well regarding composition and readability. It is not that widely available either. Mac ships with an old version, Windows has Cygwin and git-bash distros, and the high overhead WSL one, and it is too easy to shoot yourself in the foot. Posix shell fares only slightly better. Powershell is portable but a vast and hard-to-install beast. We will go with nushell , which hits many high notes, is relatively small, easy to write scripts, and has excellent startup time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*RJcYdyrWQYknImQr.png" /><figcaption>Golang and Rust are the language of choice for the best modern terminal utilities</figcaption></figure><p>A uniform set of command line utilities is also desirable. Because we want those to be small, fast, and portable, not by coincidence, they tend to be all written in Rust or Golang. There are many <em>modern unix tools </em>lists, but here are some that you may find helpful: rg, fd, fzf, bat, zoxide, jq, yq, lsd, duf, hyperfine, and dive. They all run just fine in Windows and may suffice for your muscle memory to work across machines.</p><h3>Bootstrapping your environment</h3><p>We will need a package manager to install our command runner, scripting shell, and any other utility. Unfortunately, no good package manager exists that works across all three operating systems yet. For Windows, scoop is the best option. For everything else, we will go with pkgx.sh, the new shiny toy from the homebrew creator. Reproducibility is a crucial factor, and both these tools have reasonable support for version pinning. But Pkgx has a killer feature no other package manager offers: lazy installation support. This helps with isolation and works wonders with docker, enabling tiny images. In the world of fast and cheap downloads, expensive CPUs, and uploads, this is a match made in heaven for docker build cache mounts.</p><p>There is one last challenge. How do we install the package manager itself in a portable fashion? We need to rely on tools that are shipped by default. On Linux and Mac, a POSIX shell is always available. For Windows, we can rely on PowerShell. The trick here is that if you write two scripts, one without an extension and another with theps1 extension, you can use the same call syntax on all operating systems. For example, if we create the two scripts below on the root of our repository and name them boostrap and bootstrap.ps1 , we can run ./bootstrap from the command line, and Windows will magically run ./bootstrap.ps1 whereas in Linux or Mac, boostrap will be picked.</p><pre>#!/bin/sh<br><br>command -v pkgx &gt;/dev/null || eval &quot;$(curl -Ssf https://pkgx.sh)&quot;<br>command -v nu &gt;/dev/null || pkgx install nushell.sh@0.96.1<br>command -v just &gt;/dev/null || pkgx install just.systems@1.29.1</pre><pre># Check if scoop command is available<br>if (-not (Get-Command scoop -ErrorAction SilentlyContinue)) {<br>  Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser<br>  Invoke-RestMethod -Uri &quot;https://get.scoop.sh&quot; | Invoke-Expression<br>}<br><br># Check if nu command is available<br>if (-not (Get-Command nu -ErrorAction SilentlyContinue)) {<br>   scoop install nu@0.96.1<br>}<br><br># Check if just command is available<br>if (-not (Get-Command just -ErrorAction SilentlyContinue)) {<br>   scoop install just@1.34.0<br>}</pre><h3>Just say the word</h3><p>Let us wrap this up by showing the rest of the needed scripts and configuration files. In the end, we will end up with the following steps that we could run in GitHub actions as a CI/CD, in either Mac, Ubuntu, or Windows machine, for any framework. Moreover, the same commands would work inside a docker container, as the onboarding instructions for the first day of a new developer in a company, in the DEVELOPING.md of an open source project with new contributors joining with their own laptops, or even as the prebuild steps of a GitHub devcontainer. Each of these environments may need some surrounding machinery to enable the cache to work correctly, but we will skip that in this text.</p><pre>jobs:<br>    strategy:<br>      matrix:<br>        os: [ &quot;ubuntu-latest&quot;, &quot;macos-latest&quot;, &quot;windows-latest&quot; ]<br>    runs-on: ${{ matrix.os }}<br>    steps:<br>      - run: ./bootstrap<br>      - run: just setup<br>      - run: just build<br>      - run: just test</pre><p>We have seen the ./bootstrap script, which is, in fact, two simple scripts. To add the remaining commands, we will use a .justfile in the root of the repository to pick up the command and delegate it to a portable nushell script. That script will be responsible for locally installing all the dependencies required to work with the codebase. See below the .justfile. The Medium.com code box lacks native syntax highlight for it, but justis still popular enough that it has syntax plugins for most editors and is mature enough to not only have cross-platform support but also good composition support.</p><pre>set shell := [&quot;nu&quot;, &quot;-c&quot;]<br><br>@_default:<br>  just --list --unsorted<br><br>[no-cd]<br>build:<br>  just say build<br>[no-cd]<br>test:<br>  just say test<br><br>[private]<br>[no-cd]<br>say target *args:<br>  nu {{justfile_directory()}}/say.nu {{target}} {{args}}</pre><p>The say.nu is a straightforward script that will delegate to the package managers for the setup command and will delegate to vscode-task-runner the build and test commands. Let us start with the latter two.</p><pre>#!/usr/bin/env nu<br><br>def test [...args] { vtr test ...$args }<br>def build [...args] { vtr build ...$args }<br><br>def --wrapped vtr [...args: string] {<br> pipx vscode-task-runner ...$args<br>}<br><br>def pipx [pkg, ...args] {<br> if ((sys host | get name) == &#39;Windows&#39;) {<br>  vrun pipx run -q $pkg ...$args<br> } else {<br>  vrun pkgx +pypa.github.io/pipx pipx run -q $pkg ...$args<br> }<br>}<br><br>def --wrapped vrun [cmd, ...args] {<br>  print $&quot;($cmd) ($args | str join &#39; &#39;)&quot;<br>  ^$cmd ...$args<br>}</pre><p>As you can see, we try not to affect the local machine as much as possible. The usage of vscode-task-runner, in particular, is tricky since it is a Python-based dependency, which is probably worth rewriting in Golang to avoid the cost of installing the runtime. Nonetheless, on Mac and Linux the usage of Pkgx.sh alleviates this cost since we can install pipxlazily, which in turn offers a cacheable one-off mechanism for running vtr. On Windows, until native Pkgx.sh support is launched, we need to rely on a global pipxinstallation.</p><p>The setup command will install all the local dependencies for the development. Let us assume this is pnpm/nuxt project, but the same instructions are easily applicable to most languages/frameworks. Let us extend say.nu to implement it. Ah, if you are wondering why the namesay.nu , it is just a contrived acronym for <a href="https://www.amazon.com.br/Scaling-Your-Artifacts-Team-Microservices-ebook/dp/B0CXW4F1XN/ref=sr_1_1">Scaling Your Artifacts with Your Team</a> that happens to sound nice when coupled with just. Let us add a mainfunction as well to route between setup, build and test.</p><pre>def setup [...args] {<br>  if ((sys host | get name) != &#39;Windows&#39;) {<br>    open .pkgx.yaml | get -i dependencies | filter { is-not-empty } | split row &quot; &quot; | par-each { |it| vrun pkgx install $it }  <br>  } else {<br>    open .pkgx.yaml | get -i get env.SAY_SCOOP_INSTALL | filter { is-not-empty } | split row &quot; &quot; | par-each { |it| vrun pkgx install $it } <br>  }<br>  # fallback to .pkgx.nu for non-standard installation needs<br>  if (&#39;.pkgx.nu&#39; | path exists) { nu &#39;.pkgx.nu&#39; }<br>}<br><br>def --wrapped main [<br>   --directory (-d) = &quot;.&quot;,  # directory where to run the command<br>   subcommand?: string, ...args] {<br>  cd $directory<br>  match $subcommand {<br>    &quot;setup&quot; =&gt; { setup ...$args },<br>    &quot;build&quot; =&gt; { build ...$args },<br>    &quot;test&quot; =&gt; { test ...$args },<br>    _ =&gt; {<br>       $&quot;subcommand ($subcommand) not found&quot;<br>    }<br>  }<br>}</pre><p>The .pkgx.yml is a standard configuration from Pkgx.sh, and if you are on Mac or Linux, you can use its shell integration to have a venv style locally isolated local setup by just calling dev instead of calling just setup. For Windows, we abuse a bit its env variable support to encode the packages we need to feed to scoop.</p><pre>dependencies: nodejs.org@20.13.1 pnpm.io@9.1.2<br>env:<br>  SAY_SCOOP_INSTALL: nodejs-lts@20.13.1 pnpm@9.1.2</pre><p>The final step is to add tasks.json , which usually can be auto-generated from within Visual Studio Code, or you can ask an LLM to write it for you. Here is a working example. Notice we chose typecheck to represent the build action, but each codebase can find what would be more appropriate.</p><pre>{<br>    &quot;version&quot;: &quot;2.0.0&quot;,<br>    &quot;tasks&quot;: [<br>        {<br>            &quot;type&quot;: &quot;pnpm&quot;,<br>            &quot;script&quot;: &quot;nuxi typecheck&quot;,<br>            &quot;group&quot;: &quot;build&quot;<br>        },<br>        {<br>            &quot;type&quot;: &quot;pnpm&quot;,<br>            &quot;script&quot;: &quot;nuxi test&quot;,<br>            &quot;group&quot;: &quot;test&quot;<br>        }<br>    ]<br>}</pre><p>By setting up two standard configuration files, namely vscode/tasks.json and .pkgx.yaml, each one connected to powerful tools, we unlocked several workflows. Some are very popular, but others are niche ones, which can only be supported by popular tools that can afford them. Furthermore, by bringing less than a hundred lines of glue scripts, we exposed a uniform interface to interact with diverse codebases in distinct environments.</p><p>Arguably, this could be done with a proper programming language, like typescript. The modern nodejsdoppelgangers, namely bun and deno , are very impressive pieces of software with composition support, great tooling, extensive libraries, negligible cold start times, and an ok-ish runtime size. Using a single programming language across frontend, backend, and DevOps is a sweet proposal as it smooths the boundaries and context switch costs. However, the boundaries often exist for a valid underlying reason, not a technology limitation. Specialized tools are not only about increasing productivity on the happy paths; they also introduce friction as you inadvertently try to break the boundaries. Striking a balance is not easy, but it is worth pursuing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bj_h7c-kZw3de1Nn" /><figcaption>It takes time to understand, accept, and then embrace the fragmentation in the software world</figcaption></figure><p>In the real world, people will write a Makefile to cover all the developer experience journey and fail quickly as the codebase scales. Then, they will try to replace it with an all-encompassing tool, be it bazel or a homegrown solution, and when you do this, you gain a full-time job and lose the rest of the ecosystem. We avoided both traps by sticking to idiomatic solutions of well-maintained software solutions. Finding the glue is difficult because you need to have broad and deep knowledge of the software-building process, but once it ticks, things work. There is no need to be perfect either; local escape hatches are everywhere whenever things fall short, and the concepts will survive a handful of tooling iterations as the landscape evolves.</p><h3>Hire your first AI programmer</h3><p>As we see English taking over as the primary programming language and regular code becoming just a state storage, it is easy to wonder if setting up a new codebase makes sense. No one can say for sure what will happen in the future. Still, as of now, predictability may be the most lacking property of LLMs, and a solid codebase with strong conventions and testing mechanisms may be the perfect playground for our new creative peers. Let us jump fearlessly in the new world by adding one more command to our portable codebase: just chat.</p><p>Although many tools offer AI agents that can write and compile code, <a href="https://aider.chat/">Aider</a> seems to have a good set of trade-offs. You can use it with <a href="https://ollama.com/">ollama</a>, which is not great and not fast, or you can create a free <a href="https://groq.com/">Groq</a> account to see fast and ok-ish responses from Llama70B, or you can use a commercial Openai key for a decent experience, or you can go straight to Sonnet to get the best. The trade-offs change every month, so keep an eye on their <a href="https://aider.chat/docs/leaderboards/">leaderboard</a>. Let us extend say.nu to offer the chat command.</p><pre>def chat [...args] {<br> if ($env.GROQ_API_KEY | is-empty) and ($args | is-empty) {<br>  log warning &quot;GROQ_API_KEY is empty and no arguments provided&quot;<br>  exit 1<br> } else if ($args | is-empty) {<br>  aider --model groq/llama-3.1-70b-versatile<br> } else {<br>  aider ...$args<br> }<br>}<br><br>def --wrapped aider [...args: string] {<br> pipx aider-chat ...$args<br>}</pre><p>Now you can type just say chat to get Aider to work in your codebase with the groq/llama70b combo. Or if you have an Anthropic key, you can do just say chat --sonnet. Lastly, let us wrap this up by registering the command in the .justfile so we can drop the say and just chat!</p><pre>[no-cd]<br>chat:<br>  just say chat</pre><h3>Next chapters</h3><p>In the next installment in this series, we will dive into the DevOps world by introducing docker as an isolation layer and setting up several caching layers to build an efficient CI pipeline. From there, we will continue into the ops/sre discipline, taming Kubernetes and performing end-to-end tests to deploy your code in production safely. Finally, we will extend the whole thing into a humble monorepo, supporting several languages and dozens of services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*-gAxdcvkrbjtjHF0.png" /><figcaption>Happy chatting!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c4cb681f763" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using docker for integration tests]]></title>
            <link>https://daviws.medium.com/using-docker-for-integration-tests-dfc02b5ffc8c?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfc02b5ffc8c</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[testcontainer]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Sun, 28 Jul 2024 17:54:58 GMT</pubDate>
            <atom:updated>2024-08-03T12:15:22.205Z</atom:updated>
            <content:encoded><![CDATA[<p>A robust testing pipeline is surprisingly hard to achieve. One crucial property of tests is that they need to be reproducible. At odds with that property is another important one: tests have dependencies. In fact, most bugs are found in the interaction of the systems, which is why integration tests are paramount.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/rauchg/status/807626710350839808&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/88775516caa823b1f1c17b63695b4e45/href">https://medium.com/media/88775516caa823b1f1c17b63695b4e45/href</a></iframe><p>The problem with integration tests is that they are hard to set up. With unit tests, you mock everything but the small piece of code you are testing, and then you are in stateless nirvana, where start-up time is fast, and everything is reproducible. But with integration tests, you need to prepare the surrounding state, bring up a database, spin other services your test depends upon, write results to disk, and clean up everything after you are done. You have state, side effects, and a non-neglectable amount of non-determinism, as requests flow in parallel with no predictable order, networks fail, and the disk gets full. It is a launch-and-pray situation.</p><p>You want to increase the reproducibility of your integration tests. For that, you need isolation. Luckily, the industry has settled on containers as the de facto isolation layer for running computations. They have more shared state than virtual machines or distinct bare metal servers, but they are fast to boot and reasonably secure. Nowadays, if you want to launch a server in any environment, you write a Dockerfile and build a container image, which will run in its own filesystem with its own network adapters. And if containers are good for production, they should also be suitable for setting up the environment for your integration tests.</p><p>And lots of people write their integration tests that way. They write a compose.yaml file where they describe the dependencies for the test, bring it up with some scripts, run the integration test, and shut down the environment in the end. It works well for one test or even for many tests that can share the testing environment, but it stops scaling once the need for specific setups for different integration tests arises. Enter <a href="https://testcontainers.com/">testcontainers</a>, a nice project that offers an embeddable library in several programming languages that allow you to describe your dependencies inside your testing code, leveraging dependency injection frameworks, a tight integration with testing frameworks and IDEs, and many other goodies. The project also provides several tuned containers for common dependencies, like Redis or Postgres.</p><p>But this scenario still needs to be improved. You are running dependencies as containers, but your code is being built locally and runs locally. That means that you need to make a lot of effort to achieve hermetic builds and ensure any side effects from the integration tests do not linger once the tests are complete. Building hermeticity is a challenging problem, and our industry has split into two directions to solve this problem. Either you use out-of-tree build systems, like Bazel and its doppelgangers, or you rely on CI/CD pipelines which spin fresh VMs for your builds. They both have downsides: Bazel is an engineering marvel but has a very steep learning curve and integrates poorly with IDEs and open-source projects in general. On the other hand, systems like GitHub actions or Circleci are expensive and complicated to simulate locally. If you could build your code with the idiomatic tools of your language, like cargo or gradle, but inside a container, and run it inside a container as you do with the dependencies, you could solve both these problems. So, let us see how we can do it.</p><p>Building the code for your integration tests in Docker is easy. It is similar to how people compile the code for releases, although they are usually focused on image size. Here, we want to focus on the development cycle, such as in GitHub devcontainers. Some base image, slightly featureful, that you can use for interactive, incremental development. Running the tests, however, gets more complicated because you need Docker to spin up the dependencies, and Docker needs some help running inside Docker itself. There are some popular patterns to work around this, like docker-in-docker (aka DinD) or docker-outside-docker (aka DoD), or even more native but harder-to-set-up solutions like Sysbox. They all come with trade-offs regarding start-up time, setup pain, and capabilities.</p><p>What none of these solutions let you do is leverage the “docker build” system as a trail of record for your tests. You can only access Docker in the “docker run” steps. And then you lose all the incremental, remote caching magic from Docker, a capability that bazel also developed to the limit. The almost paranoid stance of Bazel and, to a lesser extent, Docker regarding hermeticity enables large-scale, remote caching, a contrast to regular build systems, which have trouble even supporting local incremental builds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*PEFrxPUwc8nSKRBh.png" /><figcaption>Incremental builds are important even in tiny codebases since they are part of the programmer&#39;s inner loop. In large codebases, they can kill productivity.</figcaption></figure><p>The docker-outside-docker trick is to mount the docker socket as a file inside the container. But that is not a regular file, it is a disguised HTTP server exposed as a unix socket. Because we cannot mount external volumes during the build step and only copy regular files, we cannot inject the socket into the environment where the build steps are executed. We can do docker-inside-docker, and for example, Earthly uses that technique with syntax sugar on top, but moving caching from the external Docker into the inside Docker becomes very expensive. And although Earthly is a project full of benefits, much of the open-source ecosystem only integrates well with Dockerfiles (I am looking at you Skaffold).</p><p>There is, however, something else that we can inject into a docker build environment besides files. We can inject a network. If we expose our unix socket as a port in the local system instead, we can access that in the Docker build steps. Here is a Dockerfile that uses this technique. We are using the Wolfi Linux base image, which is an excellent choice as it has the benefits of Alpine without its problems.</p><pre>FROM cgr.dev/chainguard/wolfi-base<br>ENV DOCKER_HOST=host.docker.internal:2375<br>RUN apk add docker-cli<br>RUN docker ps</pre><p>If you try to build this image, it will fail in the last command. When the command docker ps runs, it will try to reach an unsafe HTTP server on host.docker.internal:2375, and unless you have manually configured your docker desktop to serve that, the connection will be refused. Also, note the usage of the host.docker.internal domain, which is created by Docker for you. Let us use the network-swiss-army-knife socat to expose the unix socket over the network and get the right incantation for docker buildto guarantee the build step can communicate with the host on all platforms.</p><pre>socat -v TCP-LISTEN:2375,fork UNIX-CONNECT:/var/run/docker.sock&amp;<br>docker build --add-host=host.docker.internal:host-gateway .</pre><p>And now the build should succeed. However, suppose your build steps contain an actual invocation of an integration test that uses testcontainers. In that case, you must modify the Dockerfile contents to use socat in the inverse direction, exposing the network socket as a unix socket. Also, testcontainers uses ryuk to make sure the containers it spawns are collected after you are done with them, and you will need to inform testcontainers about where docker is running by setting the TESTCONTAINERS_HOST_OVERRIDE variable.</p><pre>FROM cgr.dev/chainguard/wolfi-base<br>ENV TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal<br>RUN apk add docker-cli<br>RUN socat -v UNIX-LISTEN:/var/run/docker.sock,fork TCP:host.docker.internal:2375 &amp;./gradlew integrationTest</pre><p>Now you have integration tests that pass as part of a Docker build. This is super useful as you can use it in your CI/CD and use Docker remote caching capabilities, like GHA, to give you something closer to the large-scale Bazel experience without abandoning the toolset you are familiar with.</p><p>This setup may look like the holy grail of Docker inception, but there is more. Although you are no longer leaking state into the host filesystem and nicely capturing your test reports in Docker’s overlayfs, you are still leaking port 2375 to the host, breaking some of the isolation. We can improve this by running the initial socat call inside Docker itself. This makes the solution more portable across Windows and Mac as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*xuBaLgHUWUH6FK1D.jpeg" /><figcaption>Running docker-inside-docker-inside-docker can get as confusing as a Nolan movie.</figcaption></figure><p>For that, we will create a container that is the devcontainer from where we will kick the build of the image which runs the integration tests. For the devcontainer, we will use pkgx.sh as the primary package manager, since it can generate small lazy images through its caching mechanisms. The integrate image will set up the build environment, and use socat to expose the docker socket to the RUN line where we call python testing library unittest, which will pick up our integration test (despite its name).</p><pre>FROM cgr.dev/chainguard/wolfi-base AS devcontainer<br>RUN apk add curl libgcc docker-cli docker-compose<br>RUN install -D /usr/bin/docker-compose /usr/local/libexec/docker/cli-plugins/docker-compose<br>RUN curl -Ssf https://pkgx.sh | sh <br>RUN pkgx install socat python<br>RUN mkdir -p /var/run/<br>RUN python -m pip install testcontainers[postgres] psycopg2-binary<br>WORKDIR /usr/src/app<br>COPY Dockerfile compose.yaml test.py ./<br><br>FROM devcontainer AS integrate<br>ENV DOCKER_HOST=host.docker.internal:2375<br>ENV TESTCONTAINERS_HOST_OVERRIDE=gateway.docker.internal<br>RUN socat UNIX-LISTEN:/var/run/docker.sock,fork TCP:host.docker.internal:2375 &amp; \<br>  python -m unittest discover</pre><p>And here is the toy Python integration test we will run in our proof of concept.</p><pre>import unittest<br>import psycopg2<br>from testcontainers.postgres import PostgresContainer<br><br>class MyIntegrationTest(unittest.TestCase):<br>    def test_postgres_version(self):<br>      pin = &quot;sha256:36ed71227ae36305d26382657c0b96cbaf298427b3f1eaeb10d77a6dea3eec41&quot;<br>      with PostgresContainer(&quot;postgres:16-alpine@&quot; + pin, driver=None) as postgres:<br>        cursor = psycopg2.connect(postgres.get_connection_url()).cursor()<br>        cursor.execute(&#39;SELECT version()&#39;)<br>        version = cursor.fetchone()<br>        self.assertEqual(version[0][:13], &quot;PostgreSQL 16&quot;)</pre><p>Finally, we will use compose to launch the devcontainer and invoke the proper build command to build the integrate image from within devcontainer. Notice this time we need to distinguish the host.docker.internal from gateway.docker.internal, since we are exposing the docker socket in a IP that is dynamically decided by docker when it brings up the devcontainer.</p><pre>services:<br>  integrate:<br>    build:<br>      target: devcontainer<br>    network_mode: host<br>    volumes:<br>      - //var/run/docker.sock:/var/run/docker.sock<br>    command: |<br>      sh -c &#39;\<br>        socat TCP-LISTEN:2375,fork UNIX-CONNECT:/var/run/docker.sock&amp; \<br>        DOCKER_HOST_IP=$(hostname -i) docker compose build do-integrate&#39;<br>  do-integrate:<br>    build:<br>      target: integrate<br>      network: host<br>      extra_hosts:<br>        - gateway.docker.internal:host-gateway<br>        - host.docker.internal:${DOCKER_HOST_IP:-host-gateway}</pre><p>And voilá. You now have made your integration tests into a hermetic, portable, incremental, and remote cacheable computation that you can trivially use in CI/CD setups, local debugging sessions, or whatever you prefer. By moving everything inside the docker build step, we bypassed a big conundrum of build time cache mounts and run time volume mounts in Docker. It is worth saying Podman has build time mounts, but they lack the GC capabilities of the build cache mounts. We can also use the build time cache mounts to save on image size and hence remote cache space, at the cost of extra downloads. This is a sweet deal for github actions, including the native GHA docker remote caching, or even Circleci with the upcoming docker cache s3 support. But if you want the best of both worlds, you can do remote builds in a platform that supports affinity like <a href="https://depot.dev/">depot</a> or <a href="https://docs.earthly.dev/earthly-cloud/satellites">earthly satellites</a>.</p><p>There is still a small gotcha: the devcontainer stage will be built twice, once as a standalone image and a second time as the base stage for integrate. Caching fails because the DOCKER_HOST_IP is a dynamic value injected in the /etc/hosts file, busting the cache. That can be fixed with yet another hack, by injecting the value through a build time secrets mount, but it will be left as an exercise for the reader (and if you are on mac, notice extended attributes will also break cache, and that <a href="https://apple.stackexchange.com/a/463956">xattr does not work in modern terminals</a>).</p><p>To wrap it up, here is one last iteration in the Dockerfile, pinning all external dependencies for improved reproducibility and using cache mounts. Distributing computations is always subject to trade-offs involving the cost of hardware, speed of networks, and multiple other factors. As the rule of thumb du jour, aim to minimize uploads, followed by CPU usage, and freely use cheap local storage followed by downloads. Using <a href="https://pkgx.sh/">pkgx.sh</a> as the package manager is quite clever as you end up with small images that save a lot of space on your docker cache and pay nothing on a cache hit, and you download dependencies on a cache miss.</p><pre>FROM cgr.dev/chainguard/wolfi-base:latest@sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92 AS devcontainer<br>RUN apk add curl libgcc docker-cli docker-compose<br>RUN install -D /usr/bin/docker-compose /usr/local/libexec/docker/cli-plugins/docker-compose<br>RUN mkdir -p /var/run/ &amp;&amp; mkdir -p /usr/local/bin/<br>RUN curl -L https://github.com/pkgxdev/pkgx/releases/download/v1.1.6/pkgx-1.1.6+$(uname)+$(uname -m).tar.xz  | tar xvJ -C /usr/local/bin/ -f-<br>RUN --mount=type=cache,target=/root/.cache,id=root-dot-cache --mount=type=cache,target=/root/.pkgx,id=root-dot-pkgx \<br>    pkgx install socat@1.8.0.0 python@3.11.9<br>RUN --mount=type=cache,target=/root/.cache,id=root-dot-cache --mount=type=cache,target=/root/.pkgx,id=root-dot-pkgx \<br>    python -m pip install testcontainers[postgres]==4.7.2 psycopg2-binary==2.9.9<br>WORKDIR /usr/src/app<br>COPY Dockerfile compose.yaml test.py ./<br><br>FROM devcontainer AS integrate<br>ENV DOCKER_HOST=host.docker.internal:2375<br>ENV TESTCONTAINERS_HOST_OVERRIDE=gateway.docker.internal<br>RUN --mount=type=cache,target=/root/.cache,id=root-dot-cache --mount=type=cache,target=/root/.pkgx,id=root-dot-pkgx \<br>    socat UNIX-LISTEN:/var/run/docker.sock,fork TCP:host.docker.internal:2375 &amp; \<br>  python -m unittest discover</pre><p>In retrospect, this was incredibly hard to get right due to all the tiny details one must pay attention to. And you can’t easily test cache hits other than inspecting logging output or timing runs. But it works in the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Rn3Qe8czozOYiBI_8IQLA.gif" /><figcaption>Hitting Postgres for your integration tests with half a second rerun is nice.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfc02b5ffc8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Your Artifacts With Your Team]]></title>
            <link>https://medium.com/better-programming/scaling-your-artifacts-with-your-team-de9b0c7775b3?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/de9b0c7775b3</guid>
            <category><![CDATA[conways-law]]></category>
            <category><![CDATA[monolith]]></category>
            <category><![CDATA[hypergrowth]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[crud]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Thu, 16 Feb 2023 18:56:19 GMT</pubDate>
            <atom:updated>2023-07-21T04:49:12.975Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l-PwMY8N0meG5ktu" /><figcaption>Photo by <a href="https://unsplash.com/@_maxxmiller?utm_source=medium&amp;utm_medium=referral">Maxx Miller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The term microservices has become ubiquitous in our industry, comprising many different meanings. From the engineer’s perspective, it promises the green-field dream, with framework freedom and a blind eye to existing code. This aspect is often emphasized by AWS, a vendor tailored for that. From a systems perspective, it promises stability through asynchronous communication and event-based systems.</p><p>Confluent, the makers of Kafka, have based their whole business strategy on this. On the tooling front, version control and integrated IDEs welcome the piecemeal approach, avoiding the scale needs of humungous codebases, as the success of scaffolding technology like create-react-app, championed by Facebook has shown.</p><p>Finally, the service mesh should eat the microservices chassis complexity, letting developers focus on business problems, a core tenant of Google Cloud Platform, operating on different layers of abstraction, from Kubernetes to Anthos.</p><p>For a time, it seemed a consensus was about to emerge that microservices were simply a superior architecture, despite the evidence that most successful large and small companies did not rely on it. Instagram is a large monolith Django application, Facebook sustained its growth in a simple PHP frontend, and Google, which pioneered containers, has many of its key services spanning hundreds of engineers, like GWS or Mustang. On the mid-size spectrum, <a href="https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity">Shopify</a> and <a href="https://hanselminutes.com/847/engineering-stack-overflow-with-roberta-arcoverde">Stack Overflow</a> are also monoliths.</p><p>Successful startups are invariably based on a simple CRUD server with some auxiliary flows, a highly efficient architecture for small groups of engineers. The challenge microservices initially came to solve was how to go from this small organization to a large one without facing the incredibly hard path of scaling in a centralized fashion taken by the early Big Five, namely Google, Amazon, Facebook, Apple, and Microsoft.</p><p>Instead, they proposed to scale as a group of small startups, an option AWS was famously created for, as it supported Amazon’s continued hypergrowth. Later, companies like Twitter, Uber, and Airbnb tried to mimic that with limited success. Netflix may be the most publicly successful case.</p><p>Recently, this consensus started to fade. Companies that went through the journey of adopting microservices started to talk about macroservices or well-sized services as a more suitable architecture. Event-oriented communication did not fully replace RPC technologies, and streams are still far from replacing databases as the source of truth.</p><p>The advances in scaling the tech stack of the verticalized trillion-dollar companies started to leak through their own SaaS or engineers creating their own super-scalable SaaS startups. But the industry shakeup brought important innovations which, when carefully combined with the modern incarnations of traditional technologies, can make the journey of a small to large company way smoother and help at any point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rUDh8eKg-vLS-aGe-MY5g.png" /><figcaption>Breaking the monolith. Image generated by DALL-E</figcaption></figure><p>In the last couple of years, I had the chance to experience those architectural decisions from many angles. In this write-up, I capture some of the thoughts I had and the techniques I developed as both an individual contributor and a leader in a large company, in my startup, and in a scale-up. With no further ado, here are my perspectives and guidance on how to scale your artifacts as your team grows.</p><h3>Software Engineering is Programming Over Time</h3><p><strong>TL;DR</strong>: You do not have a crystal ball. Invest in decisions that are resilient to change and try to make the rest reversible.</p><p>A few important perspectives are often missed when talking about software engineering. The book “<a href="https://www.amazon.com.br/Software-Engineering-Google-Titus-Winters/dp/1492082791">Software Engineering at Google</a>” defines software engineering as programming over time. The key insight is that the optimal way someone builds software artifacts is tightly connected to the expected lifetime of that software artifact. Since that quantity is often unknown, one also needs to account for uncertainty and decide not only the best strategy for writing software for the problem at hand but also the change management strategy if the expected lifetime of the software is found to be shorter or larger.</p><p>This uncertainty permeates the decision-making when writing software. Different engineers will have different guesses based on the specific problem at hand or on their past experience in different organizations, and they may arrive at distinct conclusions. Take the common task of doing something in parallel.</p><p>One might argue it is better to write a throw-away shell script using ampersands; another will argue it is better to write a small Golang CLI with proper channels and some types, and some will prefer to deploy a rust server with provably correct concurrency. The correct answer not only depends on much broader reasoning but also depends on information that is not yet available, and the discussion is a bit like sports cheering.</p><p>If you accept uncertainty as a premise, you stop looking at the problem as striking a balance since the problem is dynamic — with strong chances that whichever balance you pick today will turn out wrong tomorrow. Borrowing some machine learning lingo, we can say that instead of trying to tune precision and recall on every decision, you want to develop principles that improve the area under the curve of your model.</p><p>If you find principles that have little to no cost when you are right and that decrease the cost of the inevitable mistakes over time, you can iterate much faster and buy yourself the privilege of quickly adapting to the learnings and changes on the business side. That is the main pillar of this essay: what are the practices in your software architecture you should adopt, given that you cannot know how long your software will live?</p><h3>First Comes Data</h3><p>TL;DR: use semantic versioning for the private state, protocol buffers for everything else</p><p>There is little doubt that the most valuable asset in companies today is data. Once a startup is past the MVP phase, data will be its guide to finding the most promising markets, and if all goes well, it will become its treasure as it unlocks strong opportunities. And with a lot of luck and hard work, data can then become a moat, unveiling the powerful scale effects underpinning the most successful companies in our industry.</p><p>Data comes in many forms. Tables in a traditional database like Postgres are a common architecture to store durable data decoupled from code. Large amounts of nested data are often found in NoSQL databases, like MongoDB, which traded some of the ACID properties to scale geographically and in size. The NeoSQL databases like spanner tried to combine properties of those systems, and there are still myriads of variants, like those focused on graph operations, like Neo4J, or like Datatomic with its strong functional bias. They are usually what comes to people’s minds when we talk about frequently written data that should live long. It is also where most systems are born.</p><p>There is also long-lived, but read-most data. This used to be the realm of the business intelligence teams and is now populated with a variety of professionals, from data engineers to machine learning experts. Data-warehouse systems like Redshift are the norm in this world, with columnar storage being the key defining property of those architectures.</p><p>Search-heavy systems with inverted indices on their core, like Elasticsearch, have arisen recently, bringing temporal data and logs to the core of the data analysis, and powerful mixed-paradigm data lakes like Snowflake have made strides in our industry.</p><p>Non-durable data is also present in several parts of a system, with its own challenges. External caching saw a revolution ignited by Memcached and consolidated by Redis. Modern systems like DAX brings external caching access to the microsecond latency level, on par with in-memory JSON parsing. SQLite has become ubiquitous, often used creatively in durable and non-durable scenarios. Inverted lists for search systems are often mmap’ed to memory or even recreated on-demand on system load. In the frontend space, sophisticated data management solutions, the so-called state libraries, like redux, promise features like time machine debugging.</p><p>The distinction between data and code gets blurry when we talk about machine learning models. They can be a h5 blob deployed as an asset in a binary or an independently versioned artifact served by TensorFlow Serving. Even though our mental model is always tied to Von Neuman’s architecture, with code and data flowing as separate entities, code is part of the system state in a distributed system. And as we will see, the lifetime of code needs to be managed just like any other piece of data.</p><p>As we discussed, software is programming over time, so we need to talk about data over time as well. This is often discussed in the context of schema evolution. Ruby and Rails and Django are popular ORM systems that come with migration tooling, a common but rather limited mechanism for schema evolution. So-called migrations are SQL-based transformations deployed with code that can evolve the schema of a relational database together with the changes you have decided as your system has evolved. When carefully crafted, they can rollforward and rollback the database to match the code expectations.</p><p>They have one severe shortcoming though, which is that they are contained in the database. They have no effect on the data warehouse or other data copies and will generally result in breakages on analytical pipelines. It is beyond their reach to rewrite log lines from the past or invalidate caching layers like Redis, which may end up with a mix of data schemas across its data entries.</p><p>Another popular approach to dealing with data over time is schema versioning. The ecosystem around serialization format Avro champions that solution, and central schema databases, dubbed schema registry, are offered both in open source format and as SaaS services by some vendors. Whereas migrations fall short for being insufficient on where they can be applied, schema registries are a poor solution to schema evolution because they rely on cumbersome and mostly overlooked compatibility modes when specifying how schemas mixed in the axis of time. Having access to all historical schemas of a given table will hardly suffice to efficiently write analytical code, which has to reduce or join data represented in multiple schema versions over the years.</p><p>Both migrations and schema registries are shy, if not shortsighted solutions, to the problem of schema evolution, offering only localized solutions that ignore that data crosses several systems with different properties in a given instant and over time as well.</p><p>But not all is lost. Protocol buffers are a schema language born inside Google with a strong focus on the schema lifetime and a highly efficient associated serialization machinery with powerful backward and forward compatibility semantics. They are pervasive in the Golang ecosystem but available in virtually every language. The modification rules for protocol buffers can be mechanically verified by different tools like Buf Breaking Changes Detector and offer a powerful set of properties around data over time in exchange for some extra weight on the data designer.</p><p>The trade-off is well worth it, as it allows both the core database and all the derived ones, from data lakes to message queues, to be resilient to schema changes across many different operations. I would say this is one of the core secret sauces of Google’s infrastructure. You can look years behind and still make sense of the data, even in the presence of cumulative software changes as products and people come and go.</p><p>A schema language with automatically enforced backward/forward compatibility rules and associated serialization machinery like Protocol Buffers is a strong requirement for any distributed system, and it is no coincidence that most large successful organizations adopt it, from Google itself to Netflix’s appraised microservices architecture. Facebook’s Thrift drank from the same fountain, and Cap’nProto gives a glimpse of what can be accomplished if you dare take it into the future.</p><p>But there is an overhead associated with it, from a small hit to developer productivity perspective, since you have extra cognitive and typing overhead from the schema language and tooling to performance, given the cost of tag-encoding that enables the forward/backward compatibility properties over the wire.</p><p>It would be amazing to use protocol buffers for Scala case classes, C++ pod types, or the new Python Data Classes. But it is a trade-off that makes sense to pay for whenever your data gets serialized. If it goes to the network, if it goes to the database, or even if it goes to logs, you should use protocol buffers. Because now, your data is part of an ecosystem that does not evolve atomically and needs the bells and whistles from protocol buffers.</p><p>You have a terrific scape-hatch in JSON encoding for systems that do not natively support protocol buffers. By adopting the best practice of keeping your protos mostly flat, you can handle your data easily in most environments.</p><p>For data that is purely private, you can get away with something else. Think about in-memory vectors, caching layers, or even the code itself. These are data pieces that often cannot accept, or we cannot impose the overheads associated with protocol buffers. For those, the industry has successfully adopted the semantic versioning protocol. You could think of major version changes as imposed migrations to systems that consume that data. Semantic versioning only offers backward/forward compatibility across minor versions, but by being explicit, it often allows different major versions to live side by side.</p><p>We see that happening all the time as we push releases in systems like Kubernetes, where minor versions live together, and major versions require new endpoints, as commonly done in public REST APIs. We tend to think that only the binary artifacts require semantic versioning, but you can use the same strategy for table names in cache systems or for completely private databases. It is all about whether you have atomic updates or not. If you start to share your redux data stores across UI parts that get updated independently, soon you land in the same problem, hence the usage of hash-prefixed URLs, where you feed semver into the hash function. And once you start to expose the data, be it through logs or through your warehouse system, you really want to promote it to protocol buffers.</p><p>Remember, we are ultimately talking about distributed systems here. What differentiates a distributed system from a centralized one is whether they have the same notion of “now,” They can live in the same microchip or in a cluster. What matters is if they have the global Newtonian clock, or if they are in Einstein’s reality where time depends on the point of reference. Once the data gets published to a system with a different clock, you need to handle schema evolution for your organization to succeed.</p><p>But despite how hard it is to grasp all this, the solution is simple. Always do semantic versioning. Once you start serializing your data to the network or to a database, write a protocol buffer to describe it and enforce break-changing detection in your CI pipeline. This is the single most important property for your system, doesn’t matter if you are doing tiny microservices or a big monolith. Because a monolith also becomes a distributed system when you add a data lake or even when your clients store its responses.</p><h3>Mind the Cross-Cutting Concerns</h3><p>TL;DR: adopt tooling on the boundaries and kill private tooling</p><p>Cross-cutting concerns are the problems that arise in most software engineering projects but are not related to the business problem at hand. Some of the classic examples are logs and security. In modern applications, the set of cross-cutting concerns tends to be pretty high. You typically have authentication/authorization. You have several forms of observability, comprising from logs to tracing, several experiment classes, from a/b testing to canary releases, and deploy concerns, from the simple CI to a robust CD system. And on it goes.</p><p>Heroku famously published a collection of best practices they have collected over the years from those developing applications in their platform. Under the name “<a href="https://12factor.net/">12-factor app</a>”, the document collected extremely pragmatic and battle-tested solutions for problems every developer faced for some of the classic cross-cutting concerns of the time. The industry benefited tremendously from adopting the opinionated solutions provided in that document, and many of them are today not only considered a (somewhat aging) standard but available in refined form as out-of-the-shelf solutions you can download as open source or buy in SaaS form.</p><p>When an organization is in its infancy, the cost of cross-cutting concerns is low. Early successful companies will usually have a simple CRUD monolith; if they have drunk the microservices cool-aid, they will have a handful of services. At worst, you will have addressed a given cross-cutting concern, e.g., the CI pipeline, once or twice with deeper thoughts and maybe ignored it or hacked it a handful of times. The reality is that there isn’t much meat for the concern to cross. As companies grow, this starts to change. If you try to scale the monolith further, your cross-cutting concerns solutions need to grow in sophistication.</p><p>For example, your CI pipeline will start to have slow start times or use too much memory. If you started adopting microservices from the beginning, you would be swamped in setting up new, different CI setups that rotten independently across microservices and need to be kept in sync. And then, you will create a scaffolding tool that will become the cross-cutting concern and start to deviate resources from your business problem.</p><p>The reality is that there is no simple solution. Efficient and effective solutions for cross-cutting concerns are paramount for scaling a software organization but are irrelevant to the business. They are always a sunk cost that will be competing for resources and needs to be protected somehow. You will need error budgets, dedicated teams, and several processes. But it will always feel bad for the business, and it will always feel exciting for the engineering organization, even risking creating a miasma that can easily contaminate the company culture.</p><p>A robust strategy is to adopt or buy cross-cutting concerns solutions from companies or organizations dedicated to this problem (and hence the business goals are aligned) and kill the internal tooling. You will still dedicate significant resources, but not to build tools that fit your (secondary) use case, but to fit your use case to the existing tools. Resist the urge to write your own Kubernetes wrapper or a create-react-app clone for your backend. They start simple and grow into large projects that slow down your progress.</p><p>The tricky part is always when you start to consider what happens when they cross boundaries over time and over teams. Once an internal tool is no longer private to a team (or even to an individual), the organization should see that it is time to outsource someone who does that as their core business. In the long run, a company must be good at building what it sells, not what others sell.</p><p>Let us go through some of the common software-related cross-cutting concerns which pop up as your organization grows.</p><h4>The cloud</h4><p>TL;DR: never do anything on-premise</p><p>Unless you are competing with GCP, Azure, or AWS directly, there is no point in running your infrastructure today. If internal policies or practices bind you, invest your time in changing them. If you are bound by regulatory power, you must comply, but really verify as that is becoming less and less common. You cannot underestimate the high cost of running on-premise.</p><p>Also, you should aim to be in a single cloud. They have strengths and weaknesses, but combining them is a worse scenario than any individual cloud. Do not believe you can increase stability by being multi-cloud. This will only make sense if 5+ 9s stability is a crucial selling point for your product, and this will only be the case if you sell infrastructure SaaS. Unless you are dedicating the majority of your organization&#39;s resources to reach this very hard goal, the load balancer you need to put on top of the clouds is a cross-cutting concern that will drag efforts from your business and fail more often than the underlying services.</p><p>If you use multiple clouds, which tends to happen even if you try not to, strongly avoid overlapping services. Cross-cloud communication is expensive and slow, and the cognitive cost of distinct tooling is also high. On the other hand, intra-cloud interop is not only fast and cheap but tends to be smooth as well since the cloud vendor has done the hard work for integration. If their exact implementation is not a fit for you, do not patch on top of it; instead, try to modify your requirements to match their offer.</p><p>Remember, it does not matter if you are running a gigantic monolith or a thousand microservices. If your architecture prevents you from using the cloud, the cost is too high. And do not forget that the cloud is also evolving. If your monolith requires 32GB of RAM, it is probably not a big deal since this can easily be provisioned on modern clouds. A few years ago, at this size, you would probably be already forced to review your architecture.</p><h4>Service mesh</h4><p>TL;DR: Use a high-level service mesh. Do not use plain Kubernetes.</p><p>Services, no matter their size, need to be brought live and need to communicate. The modern service mesh provides tooling for several important aspects of the software’s lifetime in production. Talking about GCP alone, you can look at plain Kubernetes, Istio, and Anthos. The higher the abstraction you can plug, the more you get. The service mesh operates on the boundaries of your services, and the more services you have, the more critical it becomes. Organizations that try to run microservices at scale and build their service mesh simultaneously will divert an incredible amount of resources from their core business.</p><p>An attractive sweet spot now is running Cloud Run for Anthos. This is a very high-level abstraction layer without imposing many constraints on your organization. Every cloud has something similar, like AWS Fargate or Azure Containers. What I believe sets Cloud Run apart is that it is close enough to the serverless offer without the downsides. Serverless is a pretty exciting trade-off, which imposes on developers the requirement of fast startup times, and it gives you automatic balancing, pay-for-what-you-use semantics, and a bunch of other goodies. It also imposes some restrictions on how to handle state.</p><p>Overall the restrictions by serverless are good practices software should adopt anyways, for various reasons, but it is still a world that requires a very strong opt-in from your organization. Cloud Run gives the same benefits and is a bit less stringent in imposing good practices. You can still use highly-optimized in-process concurrency frameworks and standard protocols (gRPC, HTTP), but the hourly restart means you don’t need to track down that nasty leak that only manifests itself after a week. And you can get pretty much all of the benefits of the pay-for-what-you-use model.</p><p><em>I remember a similar journey on a smaller scale as we started to build web services over twenty years ago. Full-blown server applications were being replaced by small cgi-bin scripts in Perl, powered by the Apache chassis. Those felt a lot like the microservices of that time. On the limit of simplicity, inetd mode forked a service reading from stdin and writing to stdout, very much like serverless nowadays. The world goes round and round.</em></p><h4>Developer environments</h4><p>TL;DR: favor vanilla stacks and limit their number; offer five-second modify/test cycles, syntax highlighting, and breakpoints from the get-go.</p><p>The discussion around microservices often evolves around people having the freedom to use the programming languages and tools they like. When a company is born, the first person writing code will pick the stack they like the best and will make the initial investment to have a productive environment. For a greenfield project, this is usually very easy to do. Just clone some skeleton git repository or use a scaffolding tool. As the organization grows, the dynamics change. New engineers come with their own experiences and preferences and get frustrated if they are not productive from the first day.</p><p>The reality is that setting up a development environment is a complex cross-cutting concern. As your company scales up, people come and go, and projects live and die. The requirements are also not obvious either. Since most people in a company in hypergrowth have joined recently, they want the development environment to be tuned for the day one productivity use case. On the other hand, the company has money to hire a lot of people because their existing code generates a lot of value, and getting the core seasoned team still productive and happy after two years is probably more valuable, even though it affects fewer individuals. These two use cases can often clash.</p><p>Once you are past a few dozen engineers, you should start thinking about developer experience as an internal product. A simple answer that is often flawed is to have a dedicated team. It gets better if it has a PM with OKRs and everything, but the risk is that the team forgets this is a cross-cutting concern and starts to build tools instead of adopting/buying from organizations that are thinking about the problem much more holistically.</p><p><strong>You need a team, but it is a procurement and integration group.</strong></p><p>Docker was one of the first external offers that helped with the environment skew problem by letting people adopt a mostly hermetic toolset. More recent offers, like devcontainers and GitHub Codespaces bring the solution to a whole new layer, and hopefully soon, people shall be able to code with a simple laptop, leaving all the heavy lifting to cloud-based systems. Jetbrains’ gateway and many others are pointing in the same direction. Just remember that whatever you go with, your engineers can use breakpoints, syntax highlighting, and quick tests on day one and in year two as well.</p><p>Regardless of your solution, or if you deploy thousands of services or a handful of them, the more diverse your environment is, the more expensive it will be to maintain it. Because of that, you want to have a few general-purpose stacks only. For example, it could be Golang in the backend, Python for data science, and Flutter in the frontend. Or it could be Kotlin, Julia, and JavaScript. Or node for isomorphic js code and scala for data. As will discuss later, there is often a need for special-purpose projects, and for those, you probably cannot afford the cross-cutting concern of having a top-notch developer experience. But for the core projects, focus on a few stacks, and deliver the best developer experience.</p><p>The bridge between the developer cycle and production, with staging and testing environments, among others, is also a difficult challenge. Skaffold is a new kid in town offering what docker-compose in a sense failed to deliver. Not only do you get continuous build (a breeze with gradle+jib integration), but you also override can backends at will to run private environments, locally or in the cloud, or point to staging or production. Resist the urge to create per-team shared backends, though. SRE should manage the global staging and production environments, and developers who need to modify multiple services can simply expose them under their own namespace. Shared state is hard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IoMoOnftZTz93XDD.png" /><figcaption>It takes a bit to get skaffold running, but it works pretty well. See <a href="https://cloud.google.com/blog/topics/developers-practitioners/simplify-your-devops-using-skaffold">https://cloud.google.com/blog/topics/developers-practitioners/simplify-your-devops-using-skaffold</a> for details.</figcaption></figure><p>Finally, remember that your developer experience squad goal is not to build wrappers on top of vendors. Otherwise, these become internal products and start to drag resources from your business. Their goal is to delete the wrappers and make sure your requirements can be adjusted to match the vendor’s offer.</p><h4>Observability</h4><p>TL;DR: company-wide top-notch logs infrastructure, APM metrics, canaries and flag flips, out of the box</p><p>There is a blurry barrier between the developer experience and managing production. Code usually is graduated from the developer environment to different deployments, sometimes called staging, or beta, until finally reaching production. As this happens, the ability to observe and control the environment also changes for a myriad of reasons. Extensive tooling is required for this journey, and their quality is key to fostering collaboration between the SRE team (or the group ultimately responsible for 24/7 production responsiveness) and those doing day-hour shifts and producing new code.</p><p>In the past, the DevOps movement advocated for full ownership of production by engineers. In the microservices world, we still see residues of that school. As organizations grow, though, this hasn’t held well in practice. The requirements of a few critical systems, including the mesh itself, will always end up in the hands of some specialized coordinator group. You can and still should have developer-driven systems in their infancy. The key is to make the road towards managed production cheaper than to deviate from it.</p><p>Isolate change observation is a critical aspect that drives people away from shared deploys and towards microservices. Unfortunately, a small service does not suffice for proper changeset isolation. The extensive usage of feature flags tends to be cheap enough and with a high payoff. This way, you can require code pushes to always be free of behavior changes and let engineers flip individual flags with close inspection. And you even get finer-grained backward/forward compatibility as a bonus.</p><p>This can make a daily monolith push safer than microservices pushes that bundle code and behavior changes together. I can’t emphasize enough how much of a game-changer feature-flags are for a codebase. As always, you can check Martin Fowley’s excellent writing for an <a href="https://martinfowler.com/articles/feature-toggles.html">overview of feature-flags</a>, and then check out one of the several vendors that exist nowadays, from LaunchDarkly to Unleash, with many others in between. Don’t try to build it yourself, though. It is a cross-cutting concern; those who develop it as their first-class product will surely do a better job.</p><p>Pre-production and production observability tools are also typical cross-cutting-concerns because when needed, they are often across several services. You need the ability to join logs across services, and your idealized distributed tracing infrastructure won’t be properly configured if every engineer is spinning their own services with complete freedom. A proper microservices chassis requires opt-in effort from the organization that is virtually always beyond the skills or the effort budget of the teams working on the business’s problems. The solution is always the same.</p><p>Buy and adopt the best tools, and adapt yourself to them and not the other way around. ELK was a decisive innovator in this space, and today cloud vendors offer native alternatives with a similar feature set. Advanced tracing with correlation ids seems like a must, but, in practice, I have seldom seen teams that can keep that working, and even when it works, the extracted value seems to be not worth it. Give it a try, but don’t sweat too much.</p><h4>Security</h4><p>TL;DR: pay people to hack your system</p><p>No discipline of software development is more of a crossing-cutting concern than security. Security is not only very counter-intuitive, but it also requires a lot of cross-field knowledge and experience. From stack overflows to physical threats with guns, from XSS attacks to social engineering, it all needs to be taken care of. And even if you create the ultimate layered architecture to protect your assets, there will always be multiple bypasses, sometimes by mistake, often due to requirements. Never underestimate how deep an environment can be injected.</p><p>The security engineer belt is full of tools. Periodic pen tests, automated linters, SSO, and cloud policies. The one recent addition that makes a difference is the service provided by companies like HackerOne. As with all cross-cutting concerns, you should always look at outsourcing all the aspects you can. This is hard for security since it is highly intertwined with your products. This new crop of SaaS offers that brings a pay-as-use red team yields a great value simply because they emulate the threat model very closely. The services ain’t cheap, but there is enough competition in this space now that you may find a vendor that suits your needs. And once you sign the contract, you won’t be capable of ignoring security anymore, as you will find yourself flooded with work to do, and when the hose stops dropping, throw more money.</p><p>Also, don’t try to come up with your own security solutions. Do not write your encryption algorithm, do not invent your authentication solution. You cannot afford the resources a proper implementation of those systems requires, and you cannot afford an amateur in-house solution. And if you have not been hired as a security expert, you are an amateur in this field.</p><p>Luckily, the cloud vendors provide everything you need, and if you do a good job, you can get authorization into your storage layer. The trade-off with a poorly executed microservice strategy here shows up very quickly. You will need to configure SSO in different programming languages, have different secret stores, and so on and so forth. Move as much as you can to your service mesh chassis. Still, there is always some work that needs to be done inside the services, and if you cannot share libraries and best practices, you may end up with a very uneven level of security across your microservices. One failure may end up leaking too much.</p><h4>Concurrency models</h4><p>TL:DR: do not mix models, favor idiomatic, and be explicit about your choices</p><p>Significant complexity is introduced in a system with multiple, non-coordinates services due to the lack of a central clock. But even within a single service, there are already challenges in implementing concurrency to leverage the multiple cores and machines made available by modern architecture.</p><p>The simplest programming model is a purely blocking single-thread model. From the programmer’s perspective, there are very few traps, with zero implicit data sharing and all the coordination handled at the OS level. Concurrency is achieved through multiple processes, one of the recommendations of the 12-factor best practices. This trade-off suits simple CPU-bound loads well, but when your computation is mostly IO-bound, you start to need some form of multiplexing to achieve a reasonable throughput.</p><p>Unfortunately, no great general technical solution has yet been found to deliver highly efficient concurrency solutions without involving the programmer. Languages like C and C++ were developed in a single-core world and provided little support for concurrency. Java was one of the first languages to bring the problem front and center, advocating preemptive concurrency with a thread-per-request model, but the overhead was always significant. The reactive school of thought advocated cooperative concurrency, and several languages adopted syntax-sugar wrapping promise objects, often with the async/await keyword pair. Specialized libraries, like NumPy and spark, offered their own notions of concurrency optimized for their own use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JHmqWOqE3EGutHGj.jpg" /><figcaption><a href="https://www.amazon.com.br/Seven-Concurrency-Models-Weeks-Threads/dp/1937785653/ref=sr_1_1">I have not read this book, but the fact that it exists definitively tells something about the state of the industry regarding concurrency</a></figcaption></figure><p>The concurrency model for a service may be seen at first glance as a private matter, but it is really a cross-cutting concern. First, because code is shared among services (either through public libraries or private ones), which often have expectations on the concurrency model. Second, the sort of problem that arises from improperly mixed concurrency models can decrease application performance by a thousand times. Subtle issues like thread starvation, unbounded concurrency, or blocking reactive threads will often lead people to conclude that code cannot scale and try to fix that by splitting services, only to end up in the same situation when the system grows by another order of magnitude. Ultimately, this comes from the fact that resources are not infinite, and you need to have reasonable utilization regardless of your architecture. If you fail on that, your users will have the services denied, typically once you get concurrency in the hundreds.</p><p>Unfortunately, there is no vendor that solves this problem. Part of the solution is trying to be idiomatic. In some stacks, this is easier, JavaScript does not have threads anyway, and the async/await syntax is well established. Python is also single-threaded but highly parallel through extensions. It also has a lot of blocking stuff and only recently adopted async/await constructs. Java and friends have a mixed model, and Kotlin structured concurrency is a great relief when it comes to error handling.</p><p>Go with goroutines do not offer strong guarantees but is simple, and programmers have been successful in applying its model. C++ and even Rust are still far from settling on a global idiomatic way of leveraging non-blocking io and multiple cores. Frontend programmers are also exposed to very similar challenges due to the rendering thread model used in native code and, to a lesser extent, in web code.</p><p>Above all, try to avoid mixing models within the same service or even stack. For example, the typical scientific Python code is CPU-bound and should live in its own single-threaded service. Stream and batch processing, often done by libraries like Kafka Streams, Apache Spark, or Apache Beam, also should be in its own services, using a thread-per-request model and a bounded thread pool. The typical CRUD application should choose between thread-per-request or a reactive model following the API of its database.</p><p>This will usually mean the idiomatic form of the language, but for some languages, it can go either way. Make sure you are explicit about the choice and provide configuration at the mesh level to support the option. Enforce it in code reviews and load-test your service to detect accidental bottlenecks when the models get mixed. Finally, make sure you have reasonable timeout defaults in the mesh and deploy backpressure mechanisms like circuit breakers everywhere. There will be problems, and you should both be robust to them and have the ability to diagnose them once they arrive.</p><h4>Version controls and dependency management</h4><p>TL;DR: one repo per stack, protocol buffers in a submodule</p><p>The way code gets organized is also highly influenced by Conway’s law. If you look at the open source community, you will see code freely available and globally indexed, but the repositories are very independent, and releases are mostly uncoordinated, relying on best-effort semantic versioning to avoid conflicts. The private for-profit organizations are usually more centralized, as they minimally share a mission.</p><p>Most large successful companies like Google and Facebook use large monorepos and vendoring as the code-sharing mechanism. This has been a rather successful trade-off as the barrier of consuming internal code decreases by a lot, mostly at the expense of tooling, which has trouble dealing with the scale. There is also a trade-off for developers, as now every new state of code needs to be tested with all downstream consumers at every change.</p><p>The first trade-off around tooling is purely a technical one. IDEs like IntelliJ, version control systems like git, or build tools like Gradle, are all fine-tuned to the open source model or small company model, which is where most of their users are. As your codebase grows, you leave this sweet spot and start to feel you should go to the multirepo paradigm and retain your tooling. However, those inside the large organizations that scaled monorepos found that the cost of not having global changes and the ability of vendoring dependencies will increase as your organization gets larger and will often try to scale tooling instead.</p><p>There are no vendors today that can give you the same experience developers from the Big Five get, and this is a cross-cutting concern that is just too expensive to tackle nowadays. Hence, you need to settle somewhere. In practice, it seems viable to have one monorepo per stack, each with a different level of maturity. The JavaScript ecosystem has several mature tools that spawned from Lerna, including NX which is inspired by the internal Google model. The Java world, through Gradle’s multi-project and composite builds, has reasonable support as well. Python still lacks much of what is needed, but cross-chain tools like Bazel can be of help. It is a significant upfront cost, but paying it once and gradually per stack seems feasible.</p><p>The other trade-off is more social in nature and comes from the fundamental property one wants in its codebase. If you upgrade a given dependency, do you want everyone else to upgrade at the same time? In the manyrepo world, the answer by default is no, requiring less coordination. The effect in practice is that different people in different teams stumble upon the same problem multiple times, increasing development costs and sometimes creating strong diversions that remove the ability of internal code sharing.</p><p>On the other side, if you share dependency versions, you will often find roadblocks like the need of upgrading the JVM globally, a large undertaking for the project that finds itself blocked. In practice, these are few and apart, and by having a developer experience dedicated team they can be made much cheaper. Most of the time, upgrading dependencies together is free, and for small diversions, you can always have escape hatches by letting subprojects override explicit versions for their needs.</p><p>How to choose which dependencies in your codebase is a matter of its own, and I recommend going through one of my past articles for guidance.</p><p><a href="https://medium.com/swlh/dependency-management-a6cce61660d">Dependency Management</a></p><p>Even if we have one repo per stack, where do data models and other stack-neutral artifacts go? If you use protocol buffers for your data models, you are lucky. Protocol buffers are a very special kind of dependency since their backward/forward compatibility is not given by semantic versioning but instead by their schema evolution rules. Also, they are naturally cross-language artifacts. This means they can be vendored on the individual per-language repositories without needing atomic global updates.</p><p>A good solution for achieving this is exposing them as a git submodule on each language repo. Protocol buffer encoded data also have the same good properties as protocol buffer schema files, and similarly, submodules might be a good way of sharing cross-stack data, like mock table data and so on, as long as the data is encoded with protocol buffer binary format (or JSON, given that you follow the more limiting JSON schema evolution rules).</p><h4>Design systems</h4><p>TL;DR: pick modern, highly customized design systems</p><p>I will skip this one and recommend this excellent article:</p><p><a href="https://medium.com/havingfun/rebuilding-loggis-design-system-on-top-of-material-ui-9555fede0466">https://medium.com/havingfun/rebuilding-loggis-design-system-on-top-of-material-ui-9555fede0466</a></p><h4>Lakehouses</h4><p>TL; DR: leverage modern cloud infrastructure for effortless extract-load, DBT for transform, and protocol buffers for robust schema evolution</p><p>As we have said, data comes first. And the data analysis part of modern companies usually lives segregated from the serving systems. There are good reasons for that; the main one is that the data view wants to be aware of the whole history of the data, whereas serving systems are primarily concerned with the current state of the data. Another reason for becoming less relevant is that serving systems usually perform better with row-oriented storage, whereas data systems benefit from columnar storage.</p><p>Most data projects spend most of their time struggling with the second reason, moving data from databases to columnar storage. Modern data storage, like Google’s AlloyDb, takes care of this automagically. For more traditional data sources, one can resort to tools like GCP DataStream or Airbyte that keep data synchronized between your database and your data warehouse for you. There is an extensive range of vendors in this space, and you should resist the urge to build your Kafka pipeline that listens to data and transforms and then stores it in your warehouse or lake. Forget ETL; embrace the ELT rage and leverage simpler tools and mental models. Reach out to DBT or Bigquery within your data lake for large transforms and golden datasets, and enjoy on-the-fly-way transformation for your explorations.</p><p>The complicated part comes when you remember that data exists across time. Don’t underestimate how valuable a fresh copy of your database is, but you want more than that; you want code that is robust to database changes; you want past versions of it; you want a log of the communication among your services. Since all those use cases cross the time axis, you need a robust schema evolution strategy, which, unlike database migrations, works across systems. Again, protocol buffers come to the rescue. If you follow their strict forward/backward compatibility rules, most problems will disappear. Use protocol buffers to describe your models and your events, and if you use the Outbox Pattern, those are one and the same.</p><h3>A Growth Story</h3><p>So far, we have warned about the risks of a poorly executed migration to microservices and spent significant time discussing how data and cross-cutting concerns should fit into your plan. In fact, those are the critical aspects to have in mind and usually the ones least talked about. Once that learning settles, let us take a look at how things usually play out in the open.</p><p>Probably one of the things that make microservices migrations particularly challenging is that we start discussing social matters, like the two-pizza team and Conway’s law, but what is taking over people’s sleep are purely technical problems, like system downtimes and slowness. When we mix scaling employees and customers, we get into a perilous place. Let us see an example that is very typical when companies go from ~20 to 50+ people.</p><h4>In the beginning, there was ORM/REST</h4><p>Startups need to transform themselves fast to get to the product market fit or die. The zero-to-one stage is very different from the late stage of companies. Whereas in the latter deep understanding of a problem and specialized tools reign, in the former stage, you need very general tools that you can quickly bend towards whatever the unserved demand you identified that month. And there is no better general tool for bringing online simple business requirements than relation databases (maybe spreadsheets, but let us skip that part).</p><p>Because of that, virtually all unicorns start with something that looks like Django or Ruby on Rails. Those systems are incredibly adaptive and allow your engineering team to support sales and product, as they cover ground discovering or creating a new market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPZFVsePX3WXG467C6mPAg.png" /><figcaption>Living the simple life in the garden</figcaption></figure><p>Up to twenty employees, no one will call such a system a “monolith” or complain about legacy code. After all, it is a very robust design, which companies have used to serve billions of customers for at least a couple of decades. Latency is really small, and observability is great. But when the company hits the jackpot, it will usually grow to, let’s say, 50 employees, and things will start to change. Not because you have more customers. After all, this design has supported decades of large systems, but because you have more complex requirements.</p><h4>The horror</h4><p>If, and only if, a company is successful, it will get swamped by the need to address the more sophisticated needs of clients. They will ask for features that are inherently slow or require batching for economies of scale. They will demand faster response times or sophisticated analytical capabilities. This invariably needs what I call “The Horror,” a distributed system disguised as a simple CRUD with appendages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hf2OFvKNUqhQcEe0qpcJew.png" /><figcaption>In a distributed system, there is no global clock, and you now live in Einstein’s relativity world.</figcaption></figure><p>For offering analytical capabilities, developers will introduce a de-normalized version of the data, typically in the form of an Elasticsearch cluster of some warehouse technology, to improve response times, a Redis will be thrown on top of the database, and for those slow/batchy requests, there will be a cron job which periodically scans the database looking for non-consumed tasks and run them asynchronously, toggling the “completed” column in the end.</p><p><em>That last one is the infamous database-as-queue anti-pattern and a guaranteed way to have your system slowly degrade and suddenly halt as you generate periodic load peaks, eventually resulting in a snowball self-inflicted denial of service attack. Good luck taming that once it pops up.</em></p><p>At this point, the company is still mostly run by the early engineers who know the system inside out and keep patching problems. Newcomers, however, will start calling it “the monolith,” despite it being anything but that. Data is spread everywhere, code is pushed independently in the cron jobs, and there is concurrent data access from many sources. Goodbye, atomicity.</p><p>Finally, after the 50 engineers barrier gets broken and we start to get close to an organization of over 100 individuals, the old timers will feel ashamed for the system failures. As they get outnumbered, the company will decide to move to microservices.</p><h4>Rinse and repeat</h4><p>The decision to move to microservices is well-known to be expensive. To justify its cost, companies expect that after a fixed time frame, they will have stability and the ability to iterate fast. Naive groups estimate this in a couple of months of work, and more seasoned groups will define it as a multi-year undertaking delivering progressive improvements along the road.</p><p>However, the actual benefit that gets delivered on day one is that newcomers get to write code the way they want without the overhead of coordinating with the existing codebase. In a hypergrowth scenario, typically, startup folks will hire more startup folks, and they will build startups. With multiple groups undergoing the same journey with their tools of choice, you will end up with multiple CRUD stacks, which are highly idiomatic at day one but evolve to have their cron jobs, their own cross-cutting concerns solutions, and the same problems you had before.</p><p>You will evolve from “The Horror” to the “Never Ending Survival Mode,” where you get the same design mistakes replicated over and over again, often interacting through one (or more) higher-level Kafka-based event bus. As problems like idempotency are never properly addressed, data gets corrupted subtly in many ways. You are always giving up fixing the problems and rewriting your system in a new microservice. The cycle goes on and on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gCIUEWRVrJvDo6VW7twiDA.png" /></figure><p>This moment is often an existential risk for the company. Engineering resources can be fully diverted into this new architecture that is never delivered. When the new design starts to fail in different but familiar ways, you will often overhear that you have a “distributed monolith,” or the “microservices death-ball,” or the “big ball of mud architecture.” People talking about those also hint that you could do “microservices the right way,” and your problem would be gone (more likely, you would be back to square one).</p><p>What happens typically, though, is that the company will lose around one to two years of progress, and demographics will come into play. Enough newcomers will become old-timers and consider they already have their microservice and start resisting new newcomers who want to rewrite their systems. People will start to talk about macroservices, complain about abandoned microservices and rotten scaffolding tools, and start sharing domain best practices through libraries. Eventually, the engineer population growth decreases enough (overall or just in the core areas). In that case, the pressure to scale communication among employees decreases, and the work towards improving local quality and scaling towards numbers of clients gets traction, hopefully bringing stability and some of the efficiency gains desired.</p><p>This pyrrhic victory is a good scenario. In the worst-but-not-unrealistic scenario, competitors will outpace you, and the company will die. But even in a good scenario, it is not only that you lost time. The place you arrive still hardly feels like the promised nirvana. Simple things became simple, but hard things are still impossible. You never built the in-depth, domain-dependent platforms that the Big Five had the chance of building through their even more painful journeys and where they dug their moats deeper and deeper. Every new country you must launch still feels like building a whole new company from scratch. Every bug needs to be fixed a hundred times. Every team makes the same mistakes and writes the same workarounds with their mannerisms. You are slow because you are redoing the same work over and over again and failing the same way.</p><p>How can things be different?</p><h3>Surviving Growth</h3><p>If we want to do “microservices the right way,” we need to define it. Most of the time, the definition of a service is very open. At best, we can say a service should serve a bounded domain, itself a weak definition. The two-pizza team rule recommends that services do not get bigger than a dozen people. Let us try to rethink our growth story aiming for roughly the following targets: around one service for every dozen engineers, making the total number at most three times that, or at least a third of that, depending on how strongly connected your organization is.</p><p>This rules out the 100+ people working in a monolith and a company with more microservices than engineers. This is what we need to have in mind to scale the number of programmers in a company. But before we get there, we need to ensure our systems can scale to millions of clients. Once you detangle the problems, you can succeed.</p><h4><strong>First data</strong></h4><p>The first step when you start building a distributed system is to make sure you have a data language where you can specify schemas and ensure they can evolve without breaking any services. Because distributed systems have no global clock or state, you cannot assume things like atomic pushes or system-wide changes. We have discussed this already, but it is worth recapping the summary: ban migrations and adopt protocol buffers. The latter can be a bit hard due to tooling. But no worries, if you adopt the schema evolution rules enforced by protocol buffers, you will be good. I will name any table that follows those rules a “pbtable” (short for “protocol buffers table”), and we will see how interesting they are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXX6Xg1CC6ZfByzcnE6HbA.png" /><figcaption>Couple your data evolution contract with the data itself by adopting protocol buffers</figcaption></figure><p><em>One creative technique I came up with recently to create a “universal” protocol buffer-based ORM is to use a Postgres JSONB column together with generated columns and INSTEAD rules, so you can have both the JSON view when working with full protos and a more regular relational view when just accessing individual fields. Let us see how it plays out in the long run.</em></p><h4>Scale reads</h4><p>If your system runs on the internet, it uses HTTP, which co-evolved with a sophisticated caching infrastructure to support the public web. This means that one can easily split traffic resulting in writes in your database from those that may not. The latter is likely to use the GET verb and is often responsible for the majority of the load of a system in its early stage of life.</p><p>Hence, before we even start talking about bounded domains, independent pushes, or code ownership, all concerns related to the number of engineers working in a system, we will leverage this natural read/write split to scale the system in the number of requests, by explicitly declaring that thing by default are eventually consistent, and that calls which generate side-effects go through a different path than intrinsically idempotent reads. This can be done nowadays by creating one (or many) read-only replicas of your data storages, a functionality broadly available today, and by creating replicas of your service that can only access the database read replicas. Finally, use the routing infrastructure to send all GETs to these newly created services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_8K20fQpzgVP289jmc-q_A.png" /></figure><p>This can be done without writing a single line of code; it is a pure SRE-led effort that can be done in a few days and will let you scale reads to millions of users. Yes, you are losing the strong consistency guarantee you had before, and a few use cases may break. You can add those requests to an exception list to fix them at your pace. Still, in practice, those will be few and apart because, often, the strong consistency guarantee is already absent for a myriad of reasons, from application code mistakes to subtle things like retries, load balancing, pushes, and so forth. When your millions of users start to reload your site, you add more database replicas and more copies of your nanoservice. You can scale horizontally as much as you want now.</p><p>We will call the new deployment, which initially runs a copy of the code of the monolith (but without write access to the database) a nanoservice. It is completely stateless, and although it can be pushed together with the monolith, it can also be pushed independently, and the code can slowly fork into something completely different. Yes, it is reading data from a replica of the monolith’s database, which may look like a violation of microservices’ rule of non-shared databases, but we will get to it in a second.</p><p>Now, contrast this with the usual strategy of extracting microservices from your CRUD service. Instead of creating a new deployment, you need to bootstrap a new codebase. And instead of replicating the database, you need to create a new one. Instead of days of work, this will take months. But worse than that, you are not truly solving the scalability problem. If you get too many reads, you can’t just increase the database because writes are still hitting there, and they cannot scale horizontally. You still have a design that is trying to keep strong consistency guarantees everywhere, even though you don’t need them most of the time.</p><p>Let us look at this from another angle. In a microservice architecture, each microservice should only talk directly to its own database. But often, it needs to join data that is born in a different microservice. You could make a request to that other microservice, coupling them together and mostly defeating the point of separating them. Instead, it would be best if you favored events and the CQRS (command-query responsibility separation) pattern. When the data is born in the first microservice (the command side of CQRS), an event is published, and your microservice can process it and store it locally with its data. Then when the request lands, everything is available and can be served locally (the query side of CQRS). It is sort of a long path where data gets copied around to be used in local joins. And because there are copies, we are again in an eventually consistent world.</p><p>Funnily, from the technical perspective, we landed in the same place by doing a read-only database replica plus a deployment clone in a few days and extracting a microservice over a few months. This is clear when you can think about the process that is replicating your database as a CQRS pipeline, with your nanoservice being the query side of CQRS. Every write in the database is an event that gets picked by infrastructure and results in an identity transformation, aka a copy of the data to the new microservice. There is a small caveat, though. The database replica gives decoupled data, and the cloned deployment gives decoupled code. But you still don’t have a decoupled database schema because you don’t control the database replication code. That lives in the infrastructure. If the database schema changes, you will need to coordinate the deployment of your CRUD service and your new nanoservice.</p><p>And finally, we get back to protocol buffers. In a traditional events+CQRS microservice architecture, people tend to think of the data flowing through the events as an ETL pipeline. But if you want the infrastructure to move the data around, your transformations need to come last, making this more like ELT pipeline. Instead of the low-coupling coordination around schema happening in the event publication/consumption, it needs to happen earlier as there is no transformation opportunity before the data movement. And with protocol buffers, it happens at schema design time by following protocol buffers rules. This means a read-only replica of a “pbtable” is in itself very much like a microservice. A somewhat dumb one as it only runs SQL queries that are highly decoupled (data, code, and schema), and you can add arbitrary code on top.</p><p>A database replica, some promises around schema evolution, a deployment clone, and routing GETs to it. Making eventual consistency explicit, we can scale the number of clients. Now we have our foundation from where we can start doing “microservices the right way.”</p><h4>Scale writes</h4><p>Handling writes is one of the hardest problems in software engineering. It is no surprise many programmers love functional languages, where side effects are kept at bay, and bugs are easily caught since everything is easily reproducible. But the harsh reality is that most useful systems have a complex state that must be dealt with. And this gets even more difficult when you have a distributed system, and there is no longer a global clock where you can observe the full state at once.</p><p>Writes in a database are side-effects; they modify the state of the system. In a CRUD system are usually associated with the HTTP POST verb (but really all other than GET). Modern database technology cannot yet easily scale writes (for many fundamental reasons), and developers need to use domain knowledge to make writes cheaper and more independent to make a system scale. When one extracts a microservice from another service, she is implicitly stating the writes across two services no longer need to be atomic, making them more independent. And when a large transaction is broken into several smaller ones, it results in cheaper individual writes. Finally, write latency can also be hidden by making it asynchronous from the client’s perspective, at the cost of not being able to handle errors while the client is looking.</p><p>Let us take a closer look at these ideas. Let us pick whichever POST request SRE believes is the most expensive for the system. It goes like this: open a transaction, read some data, compute, write some data, repeat this a couple of times, and close the transaction. This is the infamous read-modify-pattern, aka RMW, applied thrice, and it is notoriously hard to scale since the database needs to hold a lock over several operations, which are often fast the moment the code is written but get slow over time. As code changes, hotspots appear, and load increases. How can we break it into smaller and more independent parts?</p><p>Let us start decreasing the size of the transaction. Instead of three RMW cycles inside a single transaction, let us modify the code to have three transactions, each with a single RMW. The large transaction is more of an incidental and unnecessary thing, as it tends to be the case as code grows organically. If it is not, there is no way around it. You need to understand and modify the code so that splitting the transaction is possible. If you don’t, microservices won’t save you either. After all, you can’t extract the code. But often, one can modify the transaction boundaries, so let us assume it is true in this example. The problem after you break the transactions is that not only you are consuming data from potentially different states, which we deemed ok, but you are also now publishing piecemeal state changes from each transaction, and this broken-down state may be inconsistent.</p><p>The piecemeal state publication can be fixed by modifying the code to only make the changes visible in the final write. An elegant way of doing that is moving from the side-effect world of the transactional database to the functional world of event-oriented systems. The outbox pattern is a technique that is getting increased adoption because it bridges those two worlds very simply. When you want to publish an event, instead of working with queue APIs, you write to the database, and the infrastructure (e.g., debezium) will make that write into an event for you.</p><p>This is not just an improvement in ergonomics; it lets you publish both side-effects and events atomically, which is impossible if you reach those two worlds independently. So, now you can have each one of your RMWs decide whether to make its final effect visible or not by writing to the destination table or to delay its visibility by writing to an events table so a different replica of the service can pick it up from there with all the data it needs. This may all seem a bit too complicated at first, but with the proper syntax sugar, it is no different than adding celery’s .delay in your function calls, and it will suffice to scale horizontal resources like the CPU and thread-pool of your service. And, of course, the smaller critical sections will alleviate much of the database load.</p><p>Now, let us break one of the RMWs into a read/modify phase and a write phase. For that, we will create another clone of the deployment, call it a microservice, and think of it as the “command” side of the CRQS pattern. This deployment will be connected to the read-only database replica and be capable of publishing events through the outbox pattern through a dedicated database, but it won’t be able to write to the original database. When the POST arrives in our monolith, we will forward it to this new deployment. There, the read and modify parts will run as usual, but we will intercept the write call and instead write its input as an event in the events table. This will get routed back to our monolith, which is allowed to write in the database and will now run the final, quick write part, making the side effects visible.</p><p>Technically, this is similar to the outbox pattern but takes care of buffering the input for the monolith and goes by the lesser well-known name of <em>inbox pattern</em>. The multiple writes that yield events until it leads to a consistent state as if it was an atomic operation are part of a distributed transaction implemented with the choreography flavor of the SAGA pattern. You have redesigned your three read-modify-write from RMW:RMW:RWM to RM-&gt;RM-&gt;RM-&gt;WWW. All this is harder than it sounds, and remember; it can be done request by request, focusing on the slower ones. Once you manage to decrease the database timeouts to &lt;10ms for all operations and the total request budget to &lt;100ms, you should be safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S7RehKflFSyjMOETSl54ww.png" /></figure><p>The usage of the inbox and outbox pattern for communication among the services has several interesting properties. First, as events always originate from a write in the database, there is the opportunity to create an idempotency key for them automatically, making it much simpler (and explicit) to handle at-least-once semantics in the application code. On the consuming side of events, instead of using the pull-based made popular by kafka, we can use a push-based model with HTTP or gRPC callbacks, which completely isolate application programmers from the complex queue APIs, and they are free to continue working with the simple semantics from the CRUD world. Again, this is more than ergonomics.</p><p>The push-based model has the advantage of allowing very high throughput by sharing the parallelism configuration across consumers and producers, but it makes the scaling a shared enterprise (e.g., you can increase service replicas, but if there are not enough Kafka partitions, you won’t get more throughput). In the push-based model, you lose some efficiency, but concurrency is much simpler, and even a satisfactory form of backpressure can be achieved by using circuit breakers at the infrastructure level. This is all possible working directly with queues, but in practice, teams never manage to get all the (many) details right.</p><p>Up to this point, we just improved the request by moving the read load to the read-replica of the database and bringing the new deployment to take care of the CPU load. From the client&#39;s perspective, nothing has changed it, and the monolith is still hanging on that POST as we will distribute that load and wait for the write call to wrap it up. We have improved throughput significantly, but the latency gains may still not be enough. In that case, we use the ultimate trick by making this an async request from the client&#39;s perspective. For that, we route the POST itself to the microservice, and once we get the request, we just write it as an event before any computation and return success.</p><p>The event will trigger the choreography we described. We will treat the communication with the client as we have treated the communication across the deployments. This is a powerful trick that is often abused in a microservices architecture. First, it requires a change of signature, requiring an idempotency key in the request. Otherwise, the client cannot safely do retries. Second, it requires machinery for the client to get the status of the command, whether it succeeded or more importantly, failed. In a straightforward CRUD architecture, failure happens with the client on the line, so you can communicate and react to failure.</p><p>In async architecture, you need some sort of callback which is hard to implement and often is forgotten with significant consequences. Minimally, you should have mesh-level configuration to send failures to dead letter queues and connect pagers to them. Don’t make clients’ requests async unless you must.</p><p>So, unlike scaling the reads, scaling the writes require understanding a few more concepts and some modifications to the application. But more of these pieces are now available in the cloud. For example, GCP supports outbox pattern and PubSub callbacks out of the box, and Microsoft’s Dapr can be used where those are not native. In fact, virtually all of this could be provided by a Kubernetes operator, and I eagerly await it to show up somewhere. Application code still needs to either be explicit or use some sort of syntax sugar for remote procedure calls, available in frameworks like Celery Tasks or Apache Spark. Moving all this to the mesh and libraries removes the traps people face when implementing ad-hoc retries (e.g., no jittering nor exponential backoffs) and error handling (swallowing errors, dead letter queues rotting everywhere).</p><p>Despite the availability of infrastructure, this is still a much simpler task than “extracting a microservice” since you will have to face all those problems there as well. In fact, what we are doing here is the first step of extracting a microservice, by solving the technical problems first so that we can scale the number of clients. Only then we move to solve the social problem of scaling the number of engineers.</p><h3>Wrap It Up</h3><p>Up to this point, we have discussed nothing about bounded domains, different programming languages, or application semantics. Instead, we created some clones of the monoliths and splitted the requests among them, according to whether they were GET or POST. The former went to the replica we called a “nanoservice,” and most of the latter to the one we called a “microservice.” In the monolith, we kept the requests that could not afford eventual consistency.</p><p>We scaled the database with read replicas and improved decouplement by changing the communication across the deployments to be event-based, which was done with minimal code changes due to the use of the outbox pattern on the producer side and callbacks on the consumer side. We used protocol buffers to prevent the need for deployment coordination when the data schema evolves. The blast radius has decreased a lot, and the usage of circuit breaker in the mesh should take care of the more egregious failure modes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSpjSKHTpKFYrZipaA7gxA.png" /></figure><p>And what we have in the end is, from the technical perspective, very much like a microservices architecture. We are still considering everything as one big bounded domain, and we still have a single, vertically scaling write database, but this is the starting point to scale the number of engineers. In the number of clients, this design can handle millions or even billions of clients. The rules are simple. If something is slow/expensive/fragile and you don’t have the time to improve it, make it eventually consistent. If it is a read, move it to your nanoservices and enjoy infinite linear scalability. If it is a write, move it to your microservice and make its response async, at the cost of giving up on error screens and monitoring dead letter queues instead. And that is all.</p><p>In fact, this is how programmers instinctively approach these problems, but things will break everywhere without making a clear bridge between the eventual consistent world and the transactional world.</p><p>Finally, your deployments can now start to diverge. For convenience, we cloned them, making them assets built together from a monorepo, but deployed with different database access permissions. But that can change. You can make them different build targets and only ship relevant code. You can split them into different repositories if scaling the tooling is getting too hard. There may be one request you really need to optimize, so go ahead and use a different programming language that is a better fit for that specific technical problem, even if all the services belong to the same bounded domain.</p><p>But you can do it at your time when you see a benefit in doing so, not because if you don’t, your system will overload. And you can create new “monoliths” or new “microservices” or new “nanoservices.” As long as you follow the database access and communication rules, you should be good.</p><p>And as time passes, you may see a few more specialized load types. Maybe you have some machine learning computation that you want to be in Python or that you need to run on a GPU-enabled machine. You can add a service with no access to any database and call it a “lambda service.” Or you may be missing that time-based aggregation that needs a peek at the event stream. For that, plug Kafka stream or something similar directly in the event bus, and enjoy the exactly-once-semantics you always struggled to achieve when your system did not separate the side-effect-inducing computations from the event world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*COC7LZVsB70tmqkMvUXa4A.png" /><figcaption>Add some “stream services” with exactly-once-semantics like Kafka streams for those rare but powerful window-based joins and some “lambda services” for specialized CPU (or GPU) bound loads.</figcaption></figure><p>We have created a microservices chassis without ever extracting a microservice from our monolith. And we did it in days for the read case and probably weeks for the write cases. Not in months, not in years. With things in peace, in the next section, we will see how to leverage that chassis to let the newcomers have fun. You survived the 30–50 people growth phase. Let’s see how we go from that to a thousand and beyond.</p><h3>Thriving on Extracted Value</h3><p>Finally, with a stable and scalable system, we can make sure new people joining the company can create new value without worrying that the existing systems that pay for their salaries are on the verge of a breakdown.</p><p>The idea that we will explore is the one of freedom with responsibility. As we have seen, complexity in distributed systems is higher the closer the code is to the data. State is a hard thing to tame, and even communicating around it can be difficult. Newcomers will move faster if they don’t need to coordinate with several other teams and learn the intricacies of each part of a multi-year codebase. But to produce value, they will need access to the system data. We will categorize this need for data in three dimensions, whether they need to read, modify or write data, and whether they need to do it now, i.e., they cannot tolerate eventual consistency.</p><p>With the answer to these three questions, we will define four “classes” of services: monoliths, microservices, nanoservices, and lambda services. Respectively, those will have the main role of doing CRUD, event processing, querying, and computing. Let us talk about them.</p><h4>Lambda services</h4><p>Lambda services only do computation. They do not talk to other services. They read from a database, and they do not write data. All they do is receive a request, manipulate the data in the request, and send back a response. And yeah, AWS Lambda or Knative Function would be very convenient ways of implementing them.</p><p>For example, if you need to solve a combinatorial problem, there is a good chance you will want to reuse or-tools. One reason is the parallelism properties of the library depend on the chosen algorithm, and their scheduling needs may conflict with the surrounding application.</p><p>A more mundane reason is that your main backend language might be typescript, and there are no bindings for or-tools in that language. In that case, you spin a lambda service which is a thin wrapper on top of or-tools exposing a simple gRPC API for solving your specific combinatorial problem. Other isolation techniques like process forking, swigging, or shared memory have a much bigger bug surface, are way more fragile, and will rarely work in practice. If it must be colocated, go with a sidecar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OSOgQUTGwbIpTB2Z4i0ucA.png" /><figcaption>The bug surface in a stateless service with no dependencies is surprisingly small</figcaption></figure><p>Although the limitation of not talking to other services or to the database may seem like a big limitation at first, if you adopt the event carried state protocol pattern, you will be able to express a lot of needs with lambda services. And they are a joy to test and debug as they need no mocking.</p><h4>Nanoservices</h4><p>For nanoservices, let us go with an example. The company is running a Superball campaign and needs a landing page with some dynamic elements, e.g., stock stickers, that can sustain a heavy load. For rendering that page, only reads are necessary, and if the data is slightly out of date (e.g., seconds), it is no big deal. In that scenario, what we have dubbed as a nanoservice suffices. The fancy name means that this service is not allowed to write or modify any data and that it reads its data from eventually consistent database replicas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_rGJmXZ4kQLUKpFYBLaGIg.png" /></figure><p>Despite those limitations, nanoservices are very powerful. If you are storing your data in “pbtables,” a nanoservice can access it, and in a well-designed system, this can be virtually all data. It suffices for the data owner to promise not to break forward/backward compatibility, and then you can spin a read-only replica. They are a great place to put business logic and slow joins or scans. Since we aim not to get flooded with services with no maintenance, we require a project to budget two engineers over a quarter to be deserving of a new nanoservice. If you think of your architecture using the CQRS pattern, nanoservices are where the query logic goes. Nanoservices already have some cross-cutting concerns, like accessing the database, so they are preferably developed in one of the approved stacks, but if someone thinks a given new stack is ready to be used, give it a try.</p><h4>Microservices</h4><p>We reserved the name microservices for services that work like the idealized microservices people often aim for. They receive events and publish events and hold no private state. Whereas, nanoservices are reading eventually consistent data, microservices are where writes that will yield an eventually consistent view go. In a CQRS architecture, you would see them as the command service. If you are a fintech, a typical task that would suit well a new microservice is computing an investment portfolio for a new user. It can be done using potentially stale data, the computation deserves more than a few milliseconds, so it is naturally asynchronous, and you want to write its result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PU_rJedZr_3nJWdKQ8mIWg.png" /></figure><p>If you are using the outbox pattern and a push model for your consumers, writing a microservice is no different than a regular HTTP service. You get requests, compute and write in the database. There is no need to write in a queue like Kafka or poll from it and worry about partitions and coordinate consumer and producers’ capacity when scaling up or down. These are all concerns from the push model, which was made popular by kafka, due to its extremely high throughput, but very few applications need to optimize that. Let outbox pattern infrastructure publish the event for you and configure infrastructure-level callbacks to send the next HTTP request. Virtually everyone is better served with this simple push model with requests and responses.</p><p>Also, microservices will usually stitch data from several sources, so to have decent throughput, it is important to pay attention to the concurrency model. Minimally, you need to avoid blocking on I/O. Whereas CRUD services are happy with thread-per-request, microservices usually favor a reactive model.</p><p>Finally, because the microservices responses will eventually be published and modify our state, we need to be more careful when developing those. So, for spinning a new one, we will require three engineers working over two quarters, and the code being developed in one of the approved stacks.</p><h4>CRUD services</h4><p>Although CRUD services are usually where companies start and where most engineers have experience, they are the most complex class of services to develop because they deal with strong data consistency in the presence of reads and writes. This is a great property, but as Uncle Ben would point out, it comes with a great responsibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C7xsfPY-uZbs2fw7L9dCZQ.png" /></figure><p>There are several sorts of hard to foreseen traps as a CRUD service or a monolith scales. They are less severe than people think, and, as we discussed, several companies have scaled them to billions of users. Most traps will be avoided if you keep (all) database timeouts in the &lt;100ms range. You will still have a somewhat hard time mocking state for testing or being in peace by settling on a simple thread-per-request concurrency model and letting the database do the heavy lifting. You must protect yourself from poorly behaved inbound traffic with circuit breakers. But as long as you don’t give up on the first bump and stop to rewrite it as a (broken) constellation of undisciplined microservices, it will work.</p><p>If you are introducing a new bounded domain with its data model for the next 100 billion dollar project, do not hesitate to spin off a new django/rails server even if your colleagues look you down. It works. But because getting to that 100 B will take some time, to create a new CRUD service, you should have a team that looks like an early startup, with around six engineers working over three quarters. And to ensure SRE will carry the page once it goes to prod, as should be the case for all data-bearing services, you must stay in a company-approved stack.</p><h4>Freedom with responsibility</h4><p>We have introduced four “classes” of services with the goal of giving people freedom with responsibility. The less dependent on state a service is, the more freedom people in the organization have in creating them. As state gets involved, more responsibility is required.</p><p>Notice that the definition of the classes doesn’t say anything about how those services are implemented. They are only an SRE concern, a set of configurations in the deployment that will grant the service access or not to data operations. In fact, the same binary may be deployed simultaneously as different service classes. For example, we could use this trick to route the command endpoints of a binary to its microservice deployment, whereas the query endpoints go to a nanoservice. Not only is this a good separation for resource allocation, but it allows us to establish a rule that services only talk to lower classes of services, avoiding the infamous Christmas tree light problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubWwxBdWceUGbqRhP-MBDQ.png" /></figure><p>More importantly, we have fully detached the problems of scaling your artifacts in the number of users and in the number of developers. For the former, split your service into deployments that have a different level of access to the database, and make timeouts tight when close to data. For the latter, let people create bounded domains that connect to the company business and not the technical constraints. Inside each bounded domain, repeat the scaling formula by starting with a crud service, and as things grow, create new deployments and re-route endpoints.</p><p>For needs that are born more independently, introduce new services within that bounded domain according to your needs. New bounded domains let you break the 50-engineer barrier, and now you can scale to hundreds of developers working together. Because client scalability is taken care of, you can focus on people, bow to Conway’s Law and work on the business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Y1sMtjih8wMF-_Kk-2Y1w.png" /></figure><p>A final interesting observation is that the amount of state does not increase linearly with the company size. In fact, with a strong product market fit, it is more important to optimize value extraction from existing datasets than introduce new ones. This means that the number of people working in crud-services will not increase as fast as those working in micro-services once a tipping point is passed. And as it keeps going, this will be repeated for nano-services and lambda services.</p><p>When you have thousands of engineers, optimization needs for value extraction will be so extreme that the company will build its libraries, and data will be so distant that domain-independent services will pop up everywhere. Get to the tens of thousands of engineers and stay on that path as long as Google, Microsoft, or Amazon to finally reach the moment you pivot to sell cloud services :-).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de9b0c7775b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/scaling-your-artifacts-with-your-team-de9b0c7775b3">Scaling Your Artifacts With Your Team</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge 4.0]]></title>
            <link>https://daviws.medium.com/knowledge-4-0-bbae005f5a3e?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbae005f5a3e</guid>
            <category><![CDATA[knowledge]]></category>
            <category><![CDATA[wittgenstein]]></category>
            <category><![CDATA[yuval-harari]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Tue, 15 Jun 2021 03:22:47 GMT</pubDate>
            <atom:updated>2021-07-05T10:42:00.172Z</atom:updated>
            <content:encoded><![CDATA[<p>A few years ago, like many others around the world, I read Yuval Harari’s Sapiens book. It is a fascinating look at human history, from the perspective of anthropology, sociology and economics. Also, as many others in the computer science field, I have been thinking and working with what Andrej Karpathy dubbed “Software 2.0”, a bold and pragmatic view of how machine learning is transforming our industry. Sipping from ideas of these writings and my own experience, I decided to write about Knowledge 4.0, an idea that has been in my mind for a while.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L0ZGJPzrNLwDbZu2.jpg" /></figure><p>Intelligence has always been somewhat ill defined. In the midst of the current deep learning explosion, many have once again tried to define what generalized artificial intelligence is, as Turing’s Test started to feel irrelevant. Critics say this is just part of the artificial intelligence hype cycle, whenever we reach a target, we redefine the problem.</p><p>As someone who has spent a lot of my adult lifetime thinking about the representation of information, my mind usually wanders through a more humble concept, that of knowledge. Whereas intelligence revolves around some process of creating or manipulating ideas, knowledge is usually seen in a much more static fashion. A snapshot of the world, or some simplification of it, but ultimately a useful representation of a state of things. The product of the application of (some sort of) intelligence.</p><p>As humans, our ability to create and access knowledge has changed meaningfully no more than a handful of times, but always with vast consequences to our societies.</p><p>In our early nomadic days, we lived as hunter-gatherers, and the amount of knowledge each group of humans possessed was rather limited. Each individual only knew what he had seen or felt through his other senses. At best, one acquired limited knowledge indirectly from observation of its physically colocated peers. In fact, in its book Harari claims that humans at that time had bigger brains and were more “intelligent”. It is no surprise. Beyond some innate traits from our genes, the knowledge our ancestors needed to survive had to be recreated constantly through sensorial processing in hostile and constantly changing environments. Whereas Sapiens looks at us through the lenses of work and societies, the ideas we develop here are all contained in the world of culture.</p><p>The first revolution came when a fantastic and rarely recognized technology was invented: speech. The ability to communicate is innate to us, and can be observed in many other species, including not only primates but birds and even more primitive animals. A gorilla can master many symbols through its hands. But modulating sound through our digestive apparels to convey a large set of phonemes and combining them to encode vast amounts of distinct symbols is a technological invention of our society. One that took some brilliant individuals to bootstrap certainly, as there is no evidence it has been developed more than once, and in fact several feral kids have been observed in that regard, and speech has never seen developing in fully isolated societies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*sZf8BZEVgv2l98nU" /><figcaption><a href="https://cargocollective.com/vickinerino">Illustration by Vicki Nerino</a></figcaption></figure><p>Speech was so utterly successful that it has been adopted by virtually a hundred percent of the population, an adoption rate probably higher than any other technology ever created in history. The creation of knowledge did not change much yet, one still had to rely on its sensorial experiences to capture the world, and use its intelligence to compress that input in a representation that fitted its brain, and now that could be expressed in words, so it could be passed along. But the transmission of knowledge now flew through this funny channel, waves in the air, captured by little bones on the side of our heads, and mapped to abstract symbols our brains agreed upon convention as we were taught our local languages.</p><p>With languages, we could create stories. Small, compressed representations of aspects of the world, that we access to know how to react to situations we had never experienced directly and master nature. As our pool of knowledge grew, we tamed agriculture and settled. We created rhymed poetry, a funny trick to decrease entropy in speech, making memorization easier and stories more noise-resistant. As we now knew more, and required less intelligence to survive, our brains shrank.</p><p>As knowledge storage and transmission media, the combination of speech and memory was still limited. To access knowledge one had to be physically colocated with another individual already possessing the knowledge, and that individual had, of course, to be alive. Time had to be spent both by the speaker and by the listener as well. And if someone or some group which exclusively possessed some knowledge died, then the knowledge was gone. It could only be recreated through sensorial experience and intelligence. It took hundreds of thousands of years for the next leap, written speech.</p><p>For the first time, knowledge had a general physical version, that we could touch with our hands, move around, share with friends. Knowledge, once encoded in a book, can be decoded by many people. With an enormous increase in the payoff of the encoding investment, the world was transformed. Shared stories had much more reach, and our culture could spread across much larger groups. Accidental tampering of data became much less of an issue. When the manuscripts of the dead sea were discovered, we could verify that over thousands of years, little of the Torah had changed, a property impossible to hold through oral history alone. The invention of the press in the 14th century super-charged the properties of written media, and ignited a series of revolutions that brought us to the modern world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*qACcR2J-P99IWakXMoRSoQ.jpeg" /><figcaption>Newspapers, TV, radio, made the world smaller from our perspective, but larger from the eyes of the memes that spread further and survived longer</figcaption></figure><p>Thanks to writing, we went from the village level to the nation-state level. Ancillary developments, like the TV brought us to a global culture. The TV with its cameras may at a first glance seem revolutionary by itself, after all it captures a much more raw form of sensorial input. However, one does not efficiently access knowledge though a raw camera feed, and what we watch on TV are scripted shows, which goes through the same process of one-time compression through intelligence into story-telling, so that knowledge can be transmitted through the air and decoded in parallel by its viewers. A richer media, but ultimately of the same nature of written speech. I used to joke that God gave us this miracle but restricted it to a handful of channels such that all of us would need to buy cable to pay for our sins.</p><p>Jokes aside, up to this point, we were pretty much ruled by Shannon’s law. What we call work essentially was using intelligence and muscles to put things into that knowledge pool and taking stuff from there for real world application. The one place we could bypass the choke point of our brains was the custom tools we built, which encoded knowledge in a repeatable, readily-available form. Once a car is built, you can just drive it. You no longer need to access the knowledge pool to figure out how to build a new car every time you want to travel around. But unlike the knowledge pool, where ideas were freely composed and accumulated in a large collective pool, machines were clumsy in that regard.</p><p>The third revolution is much more recent, and its impact has been equally transformative. Whereas the transformations in knowledge we have discussed so far have been on how to encode and transmit it, the third revolution happened on the decoding and application side. We invented software, or how we soon learned, thanks to Alan Turing, a universal machine. Something that could compute any computable result. Creation of knowledge is still the same complex process that requires intelligence. A computer scientist still needs to observe the world and compress it into useful abstractions that fit in her brain, and she will later encode that as computer instructions, instead of words. But software no longer requires decoding, and composes nicely, even better than the knowledge representations we had so far. Software is a configuration of a universal machine, and dispenses the limitations of the special purpose machines we had built so far. To use software, we do not need to understand it, we do not need to read it and we are no longer constrained by the role of our brains in Shannon’s law.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*VDbmRf44dWTfC_rN.jpg" /><figcaption>The ability of software to compose has enabled to build knowledge towers of unthinkable height</figcaption></figure><p>A young Richard Stallman ignited a revolution within the revolution when he understood the composition power of software, and created the GPL, a legalese powered ancient blockchain of sorts, creating the largest collective intellectual work ever built by humankind. It was readily available, and free even beyond the beer and freedom meanings often discussed. It was free from us, its creators. Sensorial information and the procedures we derived from that to act on the world in our interest were compressed into lego blocks that could be assembled to create more and more complex tools. Although Stallman’s vision became fractured as the internet emerged and siloed software, the power of software composition was far from halted.</p><p>Some of the most revered contemporary art museums installations are powerful exercises in semiotics. I remember going through the corridors of Shanghai’s MoCA, and viewing the spectacular Shan Shui Within exposition, where artists tried to capture the evolution of traditional Chinese culture as it faces our modern era. As I observed the exposition, my mind started to wander, dreaming about a large installation that goes through all the knowledge pieces that goes into creating a smartphone. Despite being ubiquitous, smartphones are an unparalleled feat of engineering, and sometimes we forget how much of a revolution they represent. The amount of man-power that goes into each of these machines is worth trillions of dollars. They are the embodiment of knowledge captured over many generations and geographies, and encoded as software, readily available for our use. There is no individual that could in a lifetime read a mere fraction of the books that contain the knowledge necessary to build a smartphone. Before the invention of software and the computers that execute it, we simply lacked the tools for building something like that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7wttV_qHqNj2ExSJ.jpg" /><figcaption>There would be no bigger upgrade to Duchamp’s seminal work than adding an individual playing with its phone as it goes on with its duties.</figcaption></figure><p>Since the third revolution of knowledge, anything new we discover about the world, any new procedure, any new observation, can be captured in software, and stay forever readily available. Once again, part of our brains can shrink, since our society no longer requires the difficult task of decoding information from the knowledge pool to apply it. It is readily available, and if the Idiocracy movie comes true, as long as they still have computers, they could still access the most modern algorithms to act upon the world, without having any idea of how they work.</p><p>The fourth revolution is in motion now, and we are just seeing it start. This time it is happening in the encoding side of our knowledge machine. Speech allowed us to transmit knowledge among humans, the written word enabled us to broadcast it across generations, and software removed the cost of accessing that knowledge, while turbocharging our ability of composing any piece of knowledge we created with the existing humanity-level pool. What we call now machine learning came to remove one of the few remaining costs in our quest of conquering the world: creating knowledge. It is not that workers will lose their jobs in the near future, this is the revolution that will make obsolete much of our intellectual activity for understanding the world. We will be able to craft planes without ever understanding <strong>why</strong> birds can fly.</p><p>Polanyi’s Paradox explores the idea that we are capable of much more than we are able to explain. Even though we are capable of consuming information now through written text, augmented sensory abilities, from microscopes to dark energy telescopes, for that knowledge to be captured, it still needs to go through our brains and leave it in compressed form, probably first as the excited words of a scientist, then latex equations and ultimately code flowing from the fingers of a programmer. This is an expensive, lossy process. For example, the periodic table is nothing but a lossy, very clever low dimensional representation of the complex reality of the atomic world. The process of synthesizing knowledge this way is what we call “intelligence” at action.</p><p>But what if you can bypass the paradox altogether by removing ourselves from that pipeline? What if we feed the microscope image directly to our computers and let them register the knowledge directly? Raw recording of data doesn’t sound much of a revolution, but what if we could register just the bits that matter for later leveraging that knowledge? Isn’t that what the scientist does when she looks for the most elegant equation which can describe hundreds of data points with a few symbols? Maybe our taste for symmetries and patterns is just the result of evolution favoring those who learn faster and remember cheaply.</p><p>The ability of discovering useful representations of knowledge automatically is the fourth revolution. The techniques for that had been brewing for a long time under the machine learning umbrella, but it was only with the advancements from the last decade that learning feature representations became a central part of the discipline, in many ways eclipsing the more intelligence-like parts of decision making. In the world of “deep learning”, compressing, learning, knowing, storing, teaching, decompressing, those all look the same.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fsheetsu.com%2Ftables%2F4dab6b6ebc&amp;dntp=1&amp;display_name=Sheetsu&amp;url=https%3A%2F%2Fsheetsu.com%2Ftables%2F4dab6b6ebc&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=sheetsu" width="700" height="294" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0586d640e5268cb558b6a5fbf91f528/href">https://medium.com/media/d0586d640e5268cb558b6a5fbf91f528/href</a></iframe><p>It is on top of this idea, of a noisy encoder-decoder, that we start to see breakthroughs in translation software. The impressive result of word2vec took the world by storm, the first large-scale knowledge embedding, the result of pointing our “telescope” to all of the written books and asking the computers to predict the next word in each phrase. To be up for that task, computers had no choice but to learn the underlying structure of language. They learned that queens were to kings as mothers were to fathers. Beyond structure, they amassed geography and culture (including its prejudices) indirectly. They learned that apples are more popular than jabuticabas among english speakers, and that we all seem to believe that all princes are blue-eyed europeans. GPT-3, the latest incarnation of this idea can write poems better than myself.</p><p>But who was teaching that to the computer? It was itself, and in machine learning lingo, we started to call it a self-supervised algorithm, a terminology consolidated by a recent popular article by Yann Le Cun. I would rather take a narrower view of what is going on here. The word2vec embedding was created as the computers tried to learn the only function for which there is abundant labeled information: the identity function. Simply put, the identity function generates copies of something. Given 𝑥, it tries to output 𝑥 as faithfully as it can.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*N-ngqKRKHhB9rLt7.jpg" /><figcaption>Cloning is the identity function at action for live beings</figcaption></figure><p>The deep neural networks we are all working with now are capable of memorizing anything we feed to them. Assessing the error of the identity function is still not trivial in all domains, but it certainly is in finite discrete spaces like NLP, and we are on the verge of seeing that capability expand. And through the combination of the identity function to provide an infinite stream of labels, and neural networks for encoding embeddings, together with a well designed choke point, we have a universal compressor generator, which may be able to capture everything in the world in bits and bytes.</p><p>As we master Software 2.0, we will become more and more capable of feeding to the common pool of models and embedding layers and leverage it to whichever tasks we want. Anything in the realms of language processing, speech recognition or translation has been operating in this mode for a few years now. Image algorithms are incredibly close, and we are now seeing areas that were dominated by combinatorial algorithms, like logistics, going in the same direction.</p><p>This is knowledge 4.0. It only needs sensors and computing power to be generated. We may think of it as incredibly sophisticated cameras. Take enough pictures of the sky across the world, and a thousand years from now someone may be able to say how many millimeters it likely rained in a rural area of California on any day of the twenty-first century. Inject enough people with them to monitor blood pressure and composition, cross the models and we can say what is the heart attack probability for any population that lives in sunny areas. These applications are not unlike the colorization of old pictures or the upscaling of hundred year old videos from Paris or New York which have baffled so many in Youtube. Given 𝑥, produces 𝑥 a century afterwards, no matter how crappy your sensors to capture 𝑥 were. Modern phones already do machine learning adjustments just to compensate for cheap lenses.</p><p>Today medicine is seen as a complex subject, with lots of fragmented information across a vast array of sub-fields, some relatively well understood, like those dealing with the body mechanics, others not so much, as our internal chemistry, and many still very primitive, as those dealing with our mind. It may be that the human body is so complex that we may never be able to reduce it to something we can understand. But with knowledge 4.0, we won’t need it, and from a pragmatic perspective, doctors won’t stand a chance. Learned embeddings targeting the identity function will be inscrutable, but when used to feed more targeted models will allow for vastly better diagnostics and treatments. The computer will become a master of any field, but there is a good chance in many fields it won’t be able to explain to us what it learned. As Wittgenstein put it, if a lion could speak, we could not understand it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTji2I4o1Ir8POCpNff7EQ.jpeg" /><figcaption>Practical freedom from Shannon’s law only exists beyond humanity’s garden</figcaption></figure><p>As computing power increases, and we deploy sensors everywhere, there will be nothing left to learn from primitive instruments like our eyes and ears. Even though these are supported by our incredibly powerful brains, ultimately knowledge accumulation is capped by our ability to talk, write papers or programs. Neural networks are inspired by how our brains work, but we know they are far apart, and in fact we are far from knowing how brains work. Regardless, we have not yet discovered telepathy, so brains can only exchange information among themselves very slowly and indirectly through genes or through the media we know, a rather saturated channel with all its memes. It may be that despite being dumber, computers will benefit from the faster model of knowledge composition through a hive-mind of embeddings and models. This jump is so big that for practical purposes we may say we have escaped Shannon’s law.</p><p>So far we have been talking about knowledge 4.0 restricted to the role of encoding data from sensors. For that, the identity function enables the learning of everything that has already happened, but that is just the start. With the techniques from reinforcement learning we can also reason about simulated worlds, and connect far apart events as part of our learning process. Our computers can dream and experience a million thousand lives to sense everything and master any subject.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*liSpNvcoqNpvRp3S.jpg" /><figcaption>There isn&#39;t a safer learning space than a simulation</figcaption></figure><p>Take the work of Deep Mind on Chess, Shogi, Go and Starcraft II with its AlphaZero learning algorithm. Given a generative engine able to contain the dreams to the rules of the game, the computer just played itself several times, generating its own labeled data. When it played a human, it easily outperformed it because it had experienced so many similar scenarios that it had taught itself how to behave to win. Had the games with humans been played with different rules, never seen by the machine before but well known to the humans, the machine would likely have been crushed. But within a controlled simulation, everything is self-supervised, and if the rules of the simulation transfer to reality, the machine will be unbeatable.</p><p>Also, in simple prediction as enabled by the identity function we discussed before, the path to input reconstruction is often something we humans can relate to. It is just filling in details from memory and common sense. But when AlphaZero was learning to play Chess at a superhuman level, it was never constrained by common sense — “It doesn’t play like a human, and it doesn’t play like a program,” said its creator Demmis Hassabis, completing: “it plays in a third, almost alien, way” — .</p><p>Reinforcement learning backpropagates knowledge across far apart events making not only knowledge representation inscrutable, but any intermediate actions derived from it equally cryptic. High level Go players often have trouble describing why a given move is good, but when AlphaZero is playing Polanyi’s paradox effect is in a different league altogether. For it to explain to us <strong>why</strong> it played a given move is not unlike the human player explaining its move to a cat. Yes, there is a whole field of research on explainable models, but ultimately there is a trade-off between being able to explain intellectual activities to humans and performing them at superhuman level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hHYlDtN43_iuSC2FOrwL5Q.jpeg" /><figcaption>The creation of Adam — humanity meets androids Credit: fishsnack on deviantart</figcaption></figure><p>The knowledge 4.0 revolution allows us to freeze a frame of the world simply by training for the identity function. That alone is extremely powerful, as the future tends to resemble the past, but it goes beyond that by allowing us to apply the same technique to every imaginary world we can describe and then mimic complex series of actions it discovers to be successful on those worlds. There is no need to drive millions of hours and crash thousands of cars in the real world to teach a computer to drive. With knowledge 4.0, we can add an incredible version of São Paulo to GTA, and let the computer learn how to drive there. Yes, the physics engine won’t be as perfect as reality, but it might be close enough, and it will get better every year. We are seeing this playing out right now as the large technology companies mix maps services with deployed sensors in existing vehicles to train models mixing observation, simulation and prediction as they rush to build self-driving cars.</p><p>I guess this is what our task is now. Throwing sensors everywhere, so machines become our eyes, and playing “what if” games with them just to see they master each of those make-believe setups. Human thinking won’t be abolished, after all the former is a rather challenging engineering task, and the latter a deep intellectual one. But the <strong>why </strong>— well, that word is gone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbae005f5a3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dependency Management]]></title>
            <link>https://medium.com/swlh/dependency-management-a6cce61660d?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6cce61660d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[dependency-management]]></category>
            <category><![CDATA[loggi]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Tue, 04 Aug 2020 01:58:05 GMT</pubDate>
            <atom:updated>2022-10-03T14:05:18.632Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/887/0*JS18N95K7iRIVxdJ" /><figcaption><em>“If I have seen further,” Isaac Newton wrote in a 1675 letter to fellow scientist Robert Hooke, “it is by standing on the shoulders of giants”, referring to the work of Descartes and Hooke which he built upon.</em></figcaption></figure><p><strong>Why Bother</strong></p><p>Writing software is a very expensive process, and most systems we interact with today, as technologists or even as regular consumers, are the product of thousands of years of work by different individuals all across the world.</p><p>For example, the Linux kernel alone is estimated to have cost USD 1.4B dollars. Developing a full Linux distro is said to cost <a href="https://www.linuxfoundation.org/press-release/2008/10/linux-foundation-publishes-study-estimating-the-value-of-linux/">60,000 Person-Years or 10.8B dollars</a>. This is more money than the combined venture funding received by Latin America tech companies ever. But regardless of the fact that none of these companies could afford building a full-fledged distro, all of them benefit from many different flavors of GNU/Linux.</p><p>Software might be the first kind of knowledge that can be composed. It is hard to develop, but it is often easy to use, and is not even that hard to combine. And the latter is what brings the real magic. It is the closest thing to “building on the shoulders of giants” in the realm of really building stuff. Thanks to the legal, cultural and intellectual foundation laid out by the Free Software movement and others, this often comes for free. Even when there is a dollar price, from the cloud providers or other commercial mechanisms, it is rarely comparable to the high cost of working out all the details alone.</p><p>In this document we try to give some guidance on how to benefit from this collective investment that humanity has done for us, and how to see the evidence of when you should be contributing to the pool, not taking from it. Both are beasts hard to tame, and even harder to master. Nonetheless, here is some wisdom to help.</p><p><strong>Applicability</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IiNrEqzspfALt6TS" /><figcaption>“Creativity is just connecting things. When you ask creative people how they did something, they feel a little guilty because they didn’t really do it. They just saw something. It seemed obvious to them after a while; that’s because they were able to connect experiences they’ve had and synthesize new things.”. Jobs</figcaption></figure><p>The first question is when you should consider adopting an existing library to solve a problem. The answer is quite straightforward: always. You may not necessarily end up using a library, but the potential benefits are so great that you should at least always spend a bit of time on seeingwhat are the choices you have.</p><p>Besides the reasons for the development cost described in the last section, there are many other positive aspects of using a well-chosen library. First, future development is paid for. Not only you get the benefits from the code that already exists, but you also get for free <strong>future development</strong>. For example, from the adoption of <a href="https://developers.google.com/optimization">or-tools</a> in our codebase to today (roughly 6 months), we have gained almost 2x performance improvements just by bumping versions.</p><p>Sometimes libraries can look overwhelming. They have APIs that seem to do more than you need. And you get the feeling that you would be better off by writing the small subset of functionality you need yourself than spending the time putting yourself inside the head of the library author and asking why things are not just what you expect. Be careful on that line; thereality is that you are very likely to bump into the same obstacles that made the library author choose a different path than yours. So, what at first is causing you discomfort, is quite often you getting the benefits of <strong>time-traveling magical hindsight</strong> from someone who went through the same journey you are about to onboard.</p><p><strong>The trade-offs</strong></p><p>Now that you are convinced that you should consider using a library in your codebase, you need to decide which one to pick, if any. Different organizations adopt different strategies, and using some proxy for popularity seems to be the most common, as you can see in this <a href="https://medium.com/mwfire/a-short-beginners-guide-to-choosing-third-party-libraries-44692cb74f4c">beginner’s guide for choosing a library</a>, this <a href="https://opensource.com/life/14/1/evaluate-sustainability-open-source-project">writeup on open source sustainability</a>, or this <a href="https://www.freecodecamp.org/news/how-to-choose-between-open-source-libraries-using-popularity-c71677785542/">short piece</a> presenting more popularity signals, such as stack overflows questions. The scientificliterature on some of these signals, like <a href="http://janvitek.org/events/NEU/6050/Ps/star.pdf">github stars</a>, shows there is good information there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/0*hdDHM05vskuHYtIo" /><figcaption>Latin for “Here be dragons” in the ancient Hunt-Lenox Globe, advising those heading to Southeast Asia.</figcaption></figure><p>Although it is better than nothing, I find it an insufficient framework, and in Loggi we have adopted instead a set of orthogonal metrics based on a model for how libraries interact with our codebase and engineers. The central insight is that <strong>third-party libraries bring (great) incremental benefits, but their costs are compositional</strong>. Here is a nice equation to describe it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*hefpA02VAoYk07--" /></figure><p>What we are capturing is that for each library <em>xi</em> we adopt, our productivity is incrementally increased by some amount <em>f(xi)</em>, which depends essentially on the library itself. However, we pay an overall penalty by having to integrate the library with our codebase, and that penalty, <em>g</em>, depends on the interaction of the library with all the others in the codebase, <strong>x</strong>. The overall effect is significant and exacerbated for each library we add, like composite interest, so while we see the summation of benefits, our productivity loss gets multiplied by the more distinct libraries we use.</p><p>Let us use a simple example to understand. Assume you have a frontend codebase using vanilla js (i.e., no libraries or frameworks). You decide, very correctly according to our model, to save a lot of work by adopting AngularJS. This is one of the most popular and well-maintained frameworks, so it hits home on any “choosing a library guide”. That makes the first component of our equation increase, and has little impact on the second, since you use no other libraries. Now, rinse and repeat, and add another great library to your codebase, ReactJS. Equally popular and well maintained. Now you have a more productive codebase right? Well, not. The left side of the equation continued to increase, but the right side increased faster since it is super hard to combine ReactJS and AngularJS, and now you have the worst nightmare for many developers.</p><p>This is not just a hypothetical example. It happens every dayin many companies, and when it is noted, it is too late. For large organizations, it is often deemed unavoidable, and the microservices movement (and its less famous cousin, microfrontends) focuses on dealing with that problem at the infrastructure level instead of preventing it. In the past, we had in our codebase cocoa, reactjs, react-native, kotlin, angular, and other frameworks I don’t even remember, and for every new component in our design system, we had to reimplement it in all these codebases. You can’t avoid the dragons, but you must work hard to keep them at bay.</p><p>Before we go deeper into how to estimate <em>f</em> and <em>g</em> in the equation, notice I chose to only model whether we use or not a library, not how much we use it. This yields a vital corollary: you can always use a library someone else is using, no questions asked. Without that, the decision-making process becomes just too convoluted, and people will fall into analysis paralysis. In practice, it is a reasonable approximation, and reverting past decisions are separate projects, not part of the day-to-day work.</p><p><strong>Computing your choices</strong></p><p>To estimate how much benefit and cost a given library can bring to our codebase, we will look at four independent dimensions: <strong>value, momentum, surface and dispersion</strong>. The first one is where the conversation started: how many engineer-years would take to develop that code in-house. The second is closely related with popularity, and captures how likely your decision is to withhold the test of time. The last two are related to the penalty side of our equation. Surface is about the cognitive cost that a library brings to your organization, and dispersion captures how many roles the library may end (intently or not) playing in your code. Let us try to create a visualization for some definition of these metrics to see how it feels.</p><p>Let us talk about <strong>value</strong> first, which is the simplest concept. We can decide the scale for that is indeed the number of the engineers-years invested in that library. The <a href="https://en.wikipedia.org/wiki/COCOMO">COCOMO</a> methodology is an established technique for deriving that, and we can find the resulting number for open source libraries in openhub.net. For example, React took <a href="https://www.openhub.net/p/facebook-react/estimated_cost">72 years to develop.</a> We will use bubble graph, and encode the time to develop as the bubble size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/135/0*bqYK8TpC333LkJSg" /></figure><p>For <strong>momentum</strong>, we want something like github stars, but <a href="https://blog.tidelift.com/dont-judge-a-project-by-its-github-stars-alone">without the downsides</a>. In particular, some notion of vitality of the project is important to give us confidence that the library will not rot, and hopefully will thrive bringing new value. This being the most common way people pick libraries, others have thought about this problem before, and we will just take a ride on libraries.io <a href="https://libraries.io/npm/libraries.io/sourcerank">SourceRank</a>. That metric goes from 0 to 30&#39;ish, and we will represent it as the color of our datapoint, going from red to yellow to green. Google spreadsheet plots are not that flexible, so we will discretize it to 5 colors. React is very healthy at 34 points, so it gets a nice green tint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/141/0*MxEFdQ5i7cQ6JQLv" /></figure><p>The concept of <strong>surface</strong> is less commonly talked about, and we couldn’t find any well established metrics for it. This JSConf talk on <a href="https://youtu.be/4anAwXYqLG8">Minimal API Surface Area</a> is very much on target, and even mentions lodash, my favorite example of a library we should not adopt. It saves a little bit of typing for sure (low value), but it is even funny how it automatically gets mentioned on virtually every code review. The way this article extends the <a href="https://medium.com/@tannerlinsley/reducing-my-open-source-lifestyles-api-surface-area-420f82b7ecd8">notion of surface cost</a> to the personal life of an open source programmer also connects to the concept we are defining here. Essentially, how many new contact points an extra library brings to your codebase and organization? The larger the surface, more it hurts your codebase, so it better brings a lot of value. We will try to capture that by defining the x-axis of our graphic as the log of the number of files in our codebase with imports from that library. We compute it with some bash-fu: rg “import.*’react” .|cut -d: -f1 | sort -u|wc -l. For react, this yields 3060, whereas lodash scores at 450, and we take the log. Let us see how they look in our graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/0*m7giXmXVz5dcXwMm" /></figure><p>To wrap up our graph, we will use <strong>dispersion</strong> as the scale for y-axis. This is a way to express how much your library resembles a swiss-knife, the ultimate tool, which can do everything, and does everything poorly. When you pick a well-focused library, it is clear which role it will play in your codebase, and it leaves room for libraries focused on different topics. But if you pick a large framework, which does a lot of things, there won’t be room for another one of those. For example, when we chose Micronaut as the JVM framework for our codebase, we knew we were giving up on Spring (<a href="https://www.godeltech.com/news-events/spring-framework-5-the-swiss-knife-for-microservices/">a swiss knife itself</a>). Although this is hard to define formally, we try to proxy it through the number of names a programmer needs to summon to use the library. Again, with some bash-fu we count unique symbols imported in our codebase: rg “import.*’react” .|cut -d: -f2-|tr ‘ ‘ ‘\n’| tr -d ‘{},’ | grep -v ‘from\|import\|\’.*\’;\|^’|sort -u|wc -l. Material-ui is at 164, close to react, and in the other extreme we have quagga2, exporting a single symbol. Let us see the full graph with some of the libraries in our codebase, again taking the log for the axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*zGo6rT3aFXFHoELE" /></figure><p>So, value and momentum contribute to the positive side of our equation, <em>f</em>, while surface and dispersion contribute to the negative side, <em>g.</em> Looking at the graph, you want to have most of your libraries close to the plane origin, and you want to be very careful with the right-upper side of the graph, where only a few large, green circles can live. This visualization hints that lodash, redux and relay are not generating the same kind of benefit as the other libraries around them, namely react, material-ui and moment. The fairly new xstate and formik are a bit risky, and react-testing-library is getting a free pass for being a testing-only dependency. Yup, dexie and guagga2 are very specialized and well contained, and hence an easy choice to have in the codebase.</p><p>As for the equation itself, it provides intuition, but we don’t actually compute its value for the whole codebase, after all the inputs are noisy and it is an oversimplification of a complex social process. Nonetheless, you want to feel that you can get a positive result. Here is how a localized exercise looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wgOA30dNKA77MmSO" /></figure><p>The plot and the whole computation are connected to a <a href="https://docs.google.com/spreadsheets/d/1few3FllntRi28yJcR0-tOcxNKj0gCkM4wB0GqIWzU48/edit#gid=0">google spreadsheet</a> where you just need to feed SourceRank, person-years of development and the output of the bash incantations mentioned here to give your own codebase a try. The final numbers may be too noisy, but I suspect you will learn a lot and be able to think better on your policy of adopting 3rd party libraries.</p><p>If you are looking for a new job opportunity, we are hiring new Loggers to work in Brazil and Portugal, check out our open positions <a href="https://bit.ly/dependency-management">here!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6cce61660d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/dependency-management-a6cce61660d">Dependency Management</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spraying our bike shed with some new colors]]></title>
            <link>https://medium.com/loggi-product/spraying-our-bike-shed-with-some-new-colors-c5f4fbc5847f?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5f4fbc5847f</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[micronaut]]></category>
            <category><![CDATA[logistics]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Fri, 12 Jun 2020 19:55:14 GMT</pubDate>
            <atom:updated>2020-09-16T01:03:17.353Z</atom:updated>
            <content:encoded><![CDATA[<p>About an year and a half ago, Loggi backend codebase mostly consisted of a large Django application, and several &quot;nanoservices&quot;, with a wide range of stacks that never got traction. Rust, Elixir, C++, Nodejs, you name it.</p><p>We felt that our Django stack served a large class of applications very well, but the company was starting to write stuff that did no fit that preferred stack paradigm well, and we needed a new, different and well maintained stack in our toolbelt.</p><p>So, as small, medium and large decisions start, we wrote a design doc, which I am now sharing in case it helps those facing similar dilemmas. It has been 18 months, and it feels the experiment that was spawned by this document worked. Our team grew from 40 to 200 engineers, and our django stack continues strong in its focus. But many teams with different needs adopted the new stack, unlocking new idioms, and still interoperating successfully with the rest of the company.</p><p>Without further ado, here is our initial analysis, and how we decided to settle with Kotlin as our second backend language, JVM as our runtime, Micronaut as the overall framework for it, NoSQL databases as the primary storage, and protocol buffers as our Esperanto.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfzlNzN2yP-7QdYMb5wMIw.jpeg" /><figcaption>A pre-covid day of work in the Loggi Tower</figcaption></figure><h3>Goals</h3><ul><li>Propose an additional programming stack for the Loggi ecosystem, focused on non-crud applications.</li><li>Describe several provisional sub-decisions that cover the relevant dimensions for the new stack and are deemed good enough.</li><li>Define the communication surfaces for the new stack.</li></ul><h3>Non-Goals</h3><ul><li>Replace the existing Python/Django/Pgsql stack that currently powers most of the company.</li><li>Impose the described sub-decisions related to the new stack.</li></ul><h3>Overview</h3><p>Currently Loggi mostly relies on a classical Python/Django/Pgsql CRUD architecture to power its operations. Since a lot of the needs of the company fit that paradigm well, attempts to change that status quo have gotten little traction.</p><p>That stack however suffers from a few downsides which have become more prominent as the classes of applications in the company become more diversified, a natural result of the company growth and the technical complexity of the solutions it aspires to implement.</p><p>We propose an additional programming stack which aims to serve better non-crud applications by improving on the following intrinsic limitations of the Python/Django stack: a) lack of a mature type system, b) poor multi-core/concurrency support.</p><p>It is important to note that neither of these downsides are simply bad things. They are part of trade-offs that enable some very desirable traits. For example, the weakly typed nature of python was a big enabler for the Django ORM layer, one of the main reasons of the framework success. Also, the existence of the maligned Global Interpreter Lock (GIL) in cpython means a whole class of bugs is less likely in our codebase.</p><p>Nonetheless, the other side of the equation is equally real, and codebases with much less dynamic data models, as our national deliveries integration code, benefit little from the flexibility provided by this stack, and impose a penalty on developer productivity due to the attrition level to expose more strict data schemas. Others, like our address parsing service, require high parallelism in both cpu bound or i/o bound load, and again the solutions in the Python/Django stack are far fetched.</p><p>In the detailed design below we discuss some of the dimensions where the proposed Kotlin/Micronaut/Nosql stack differs from the Python/Django/Pgsql one, and also how it fulfills the expectations we believe any new framework introduced should. We also discuss why pick grpc/protocol buffers as the basis for interoperability between stacks.</p><h3>Detailed Design</h3><h4>Multicore</h4><p>In the Python/Django/Pgsql stack we currently have live, usage of multi-core is accomplished through multiple processes. This approach has merit, but it results in complex settings involving memory usage, blocking primitives, inbound throughput/latency and backend capacity.</p><figure><img alt="Funny break dance  where a guy named CPU 0 does all the hard work while the others (CPU1, CPU2, CPU3) just observe and cheer." src="https://cdn-images-1.medium.com/max/320/0*Dr2IWnyULSEORjCt" /><figcaption>How our python services often look</figcaption></figure><p>It goes without saying that some form of tuning will exist no matter which technical choices are made, since ultimately they are simply a side effect of not having infinite resources. In fact, process based concurrency is the <a href="https://12factor.net/concurrency">12 factor recommendation</a>. But due to the difficulty of achieving reasonable concurrency in the python eco-systems, we should adopt a more natural framework when this property is necessary.</p><h4>Type system</h4><p>Although we frame this discussion around the presence of a strong type system, the best way to see this is as the amount of static analysis benefits we have in each ecosystem. Python is very lacking in this domain, and although tools like PEP8 and mypy improves things a bit, and other static analysis/code smells tools also bring some benefit, those are mostly afterthoughts for the language, and struggle to deliver a benefit similar to what you get from static JVM languages, and more in particular from Kotlin.</p><p>Besides the traditional type support, we highlight the first class support for strongly-typed protocol buffers messages, and the syntax sugar and type system support for preventing nullability problems in Kotlin.</p><p>Again, it is important to highlight that a strong type system is a not a benefit per se, and the programming community has yet to settle on the code defect/productivity benefits trade-offs in the whole spectrum of options that exist in this world. But for the scope of this discussion, it does seem that Kotlin position itself in a rather different camp for which there is perceived demand in the company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/0*_rsaV5wcljgvCN4B" /><figcaption>Brilliant trade-off illustration published in <a href="https://twitter.com/01k/status/1067788059989684224">https://twitter.com/01k/status/1067788059989684224</a></figcaption></figure><p>Notice this is one of the camps the Node.js ecosystem did not fare very well in our informal comparisons, although it does excel in performance and offers great performance through its async based model, despite not exposing multiple cores to users. But from the more strongly typed options in that ecosystem, only typescript has significant traction, and javascript interoperability is not closely as smooth as the Kotlin/Java gap.</p><h4>GRPC/Protocol Buffers</h4><p>There are multiple alternatives for communicating between services in the same stack, or between different stacks. REST is probably the most popular choice nowadays, and has ubiquitous support. In the frontend/backend edge GraphQL is growing. Both technologies are used at Loggi. On the async front, the company also uses Kafka and AMQP (for Celery).</p><p>Each of those offers <a href="https://speakerdeck.com/nbarbettini/api-throwdown-rpc-vs-rest-vs-graphql">different properties</a>. GRPC is most often chosen in the <a href="https://philsturgeon.uk/2018/05/21/picking-an-api-paradigm-implementation/">microservices backend</a> scenario and is highly optimized for performance, leveraging HTTP/2 natively, and highly efficient binary format through protocol buffers. REST focus on the notion of resources, and hence offers <a href="https://blog.goodapi.co/rest-vs-graphql-a-critical-review-5f77392658e7">rich caching capabilities</a> deployed across the internet. GraphQL has chosen the query as its central concept, and its niche is client/server communication where both endpoints are controlled by the same company. The async protocols, kafka and ampq, are often connected to a CQRS infrastructure.</p><p>In our decision, we start with the premise that GRPC is the most efficient choice for backend/backend communication. Furthermore, we emphasize the ubiquitous support of protocol buffers as a schema language, with good schema evolution semantics, which makes it a good choice for serialization in a diverse set of environments.</p><p>Given that premise, we claim that there is strong interoperability between protocol buffers/grpc and the other choices, and given its low overhead, we conclude that building the other protocols on top of it instead of the other way around is the right choice. That choice is supported by existing prior art, covering reasonably well all the use cases.</p><p>As supporting material, see <a href="https://www.infoq.com/presentations/api-pb-grpc">Lyft talk on using protocol buffers/grpc</a>. For REST interoperability, we can leverage simple annotations in protocol buffers, consumed by <a href="https://www.envoyproxy.io/">Envoy Proxy</a>, for GraphQL, we can leverage <a href="https://github.com/google/rejoiner">Rejoiner</a> and for Kafka, we could adopt <a href="https://github.com/mailgun/kafka-pixy">Kafka Pixy</a>. Furthermore, Grpc and Envoy are at the core of <a href="https://medium.com/pismolabs/istio-envoy-grpc-metrics-winning-with-service-mesh-in-practice-d67a08acd8f7">Istio</a> and the core tracing and observability toolset it provides.</p><p>Finally, protocol buffers 3.0 do provide a very good escape hatch through its <a href="https://developers.google.com/protocol-buffers/docs/proto3#json">json representation</a>, whenever we believe GRPC, its adapters or protocol buffers are inappropriate.</p><h4>Object-Relational Mapping</h4><p>The Micronaut framework has first-class support for ORM. It is arguable whether it is at the same level as Django, but since it is not a goal to compete in this front, we simply reiterate that as of now we do not allow code in the Kotlin/Micronaut/Nosql to use its ORM facilities, avoiding analysis paralysis and fragmentation in our codebase.</p><figure><img alt="A cartoon where one says &quot;are you still doing research, get out and smell the roses&quot;, and the other starts researching roses" src="https://cdn-images-1.medium.com/max/1024/0*yDdGCfhSr89toDwt" /><figcaption><a href="https://blog.tilt365.com/uncertainty-is-a-sign-of-great-wisdom-unless-you-dont-ever-get-into-action-93276d46af3f">Research is not useful if you never get to action</a></figcaption></figure><p>Besides avoiding the conflict with existing codebases, this decision has the goal of sparking more interest and developing more expertise in NoSQL development in the company. Traditional SQL applications with very high level ORM abstractions do have programmer productivity benefits, but often lead to applications that scale poorly and are hard to manage in production, since developers are shielded from too many details. Our main codebase, Loggi-Web is a classical example for this problem. In the so called NewSQL databases, scalability is less of a issue, but latency, poor data modeling and cost are still concerns.</p><h4>Reactive programming</h4><p>The <a href="https://www.reactivemanifesto.org/">reactive manifesto</a> has declared a series of (arguably too) bold characteristics that modern software should aim to have. Micronaut declares first-class support for many of those characteristics, notably non-blocking i/o. Without proper support across your library set, the goals declared in the manifest are rarely achieved.</p><p>Even with proper support, it is often hard for a codebase to stay faithful to the principles from the manifest due to the cognitive overload of the ideas, and the extra cost it imposes to programmers. Nonetheless, this stack puts us in a good position to try, and partial success can still be useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/0*vZNlpBph3b58G12J" /><figcaption>A truly reactive codebase is as common as a dog who can sit like this</figcaption></figure><h4>Tooling support</h4><p>Kotlin is made by JetBrains, probably the best authors of full blown IDEs after Microsoft, and that means that its compiler toolchain has first class support editor integration. Besides the native integration in Intellij, also from JetBrains, it has decent support in Vim, VS Code, and even Emacs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*Acd5TTmBQwV0JOH5" /><figcaption>XKCD in its inifinite wisdow</figcaption></figure><p>Micronaut is written by OCI, the authors of Grails, and a notoriously developer conscious company. For example, a nice touch in Micronaut is that it offers scaffolding through the mn command line tool, and it is easy to install on mac and linux.</p><h4>Developer excitement</h4><p>Since being adopted by Google for Android development, Kotlin has been rising steadily. It is the <a href="https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted">fastest growing language in github</a> and the <a href="https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted">second most loved programming language according to stackoverflow</a>. Its horizontal versatility, supporting backend programming, android and other experimental targets, like the browser and IOS, may not be practical but it is certainly exciting.</p><h4>Micronaut goodnesses</h4><p>As for micronaut, although quite knew, it packs a punch. Solving some long standing pains of the most popular JVM framework, Spring Boot, while keeping its (arguably) good idioms, has bought it a lot of positive emotions as can be <a href="https://twitter.com/search?q=%23micronautfw">seen in twitter</a>. The framework is pretty modern, and compared well and even favorably to many other choices. Within the jvm world, we notice it does not suffer from the slowness and hard-to-debug characteristics of Spring Boot due to runtime reflection (it still suffers from spooky-action-at-a-distance due the use of dependency injection idioms). It is not tied to a build system that is struggling, like Play Framework. It does not have a high memory footprint, like Dropwizard.</p><p>It is also relevant that the framework is deemed cloud native, and has some <a href="https://docs.micronaut.io/latest/guide/index.html#cloud">interesting features</a> on that front. In particular, we can highlight environment detection, jaeger tracing, service discovery and serverless functions. Notice since we plan to use Istio, some of the of the features offered are redundant.</p><p>Other features worth mentioning are the strong support for configuration with env vars overrides and reloadable components, fast startup and graceful shutdown support, out-of-the-box support for many popular storage layers (kafka, cassandra, mongo, redis, postgres), and command line support through picocli.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FZPKA8hg390ZP2%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FZPKA8hg390ZP2%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FZPKA8hg390ZP2%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="248" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a55592c29bc450b01dd62de013a2d03/href">https://medium.com/media/1a55592c29bc450b01dd62de013a2d03/href</a></iframe><h4>Library availability</h4><p>Kotlin has great compatibility with Java, in both directions. And Java has the largest set of mature libraries in the enterprise and open source world. For example, Grpc was deemed a key library for this stack, and Google publishes an official java library.</p><p>The Rust ecosystem, although vibrant, has not nearly the same level of support. The Golang community is a bit more mature on that front, but also has not reached the same level of consolidation.</p><p>This was perhaps the most compelling reason to land in a JVM stack as our first alternative stack.</p><h4>Testability</h4><p>The junit/mockito combo is a very mature and powerful testing strategy. The extra syntax sugar and support that kotlin demands is already available, and micronaut provides a convenience bridge library so tests can easily tap in the dependency injection graph.</p><h4>Performance</h4><p>The Kotlin/Micronaut/Nosql stack builds on top of the JVM, one of the fastest vm-based environment. It is still behind ahead-of-time compiled languages, like C/C++ or Rust, and JVM structures are notoriously memory hungry. But those limitations are well known and the ecosystem offers good alternatives when necessary, like the <a href="https://github.com/vigna/fastutil">fastutil</a> library for compact data structures or the many j<a href="https://github.com/bytedeco/javacpp-presets">avacpp-presets</a> for fast (potentially gpu) math operations.</p><p>Notice that arguably Node and Golang perform better on I/O heavy loads, or even in general loads. And the fact that we are using Kotlin brings some additional concerns with <a href="https://medium.com/@BladeCoder/exploring-kotlins-hidden-costs-part-1-fbb9935d9b62">hidden costs</a>. Nonetheless, this is a <a href="https://sschakraborty.github.io/benchmark/dont-jump-to-conclusions.html">trade-off</a> which seems <a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html">acceptable</a>.</p><h4>Early adoptions risks</h4><p>We are doing two moves of early adoption in the proposed stack. One is the Kotlin language, the other is the Micronaut framework. We believe the first is mitigated by the large body of production code in Kotlin being created since Google <a href="https://techcrunch.com/2017/05/17/google-makes-kotlin-a-first-class-language-for-writing-android-apps/">declared it an official language for Android development</a>. The fact that we are using it in the backend make things slightly different, but does not seem enough to justify a veto the adoption.</p><p>The second move, the adoption of the Micronaut framework is definitively more risky, since its first stable release was <a href="https://objectcomputing.com/news/2018/10/23/micronaut-10-ga-released">a little over a month</a> before this stack was proposed. Micronaut is backed by the same company as the well-established (but shrinking) Grails framework, and it should be the <a href="https://twitter.com/nulleric/status/1062347320963014657">default application context for Grails 4.0</a>, which is a second path for production adoption.</p><p>Nonetheless, there are real risks that we will be hit by bugs due to the lack of maturity of the framework, and the framework may even be discontinued. Despite those risks, the modern feature set offered by Micronaut is arguably unmatched, except maybe for Spring Boot. If the worst comes to it, migration to Spring Boot would be painful, but possible.</p><p>The other frameworks we looked closely were Spring Boot, Spark Java, Vert.x (all three with <a href="https://medium.com/@napperley/kotlin-server-side-adoption-d160ede1877c">Kotlin support</a>), Scala Play, Dropwizard and Express.js. We did not analyzed in depth solutions in the golang eco-system for the reasons discussed in the library availability section, but it is worth noting the ecosystem is very friendly to grpc in case we eventually need interoperability.</p><p>Spring Boot was the second contender, but it is large, complex and slow (to boot), due to its ancient J2EE origins. Its lifecycle management and annotation-centric style are arguably a great productivity boost for a team of diverse skill levels, and is well captured by Micronaut. Since Micronaut was built exactly to compete with Spring Boot on modern application architectures, it is no surprise it drops the bad parts, and leave the good ones.</p><p>Spark Java, Vert.x were interesting options, but the former does not cover much other than the http server api, and much of the value we get from Micronaut are the consistent backend apis, cloud integration and lifecycle management. Vert.x offers much of this, but ties us too closely to the internal event loop model. Dropwizard has a better surface, being a great library for monoliths, including health checks and configuration, but it is memory hungry, slow to start. On the other side of the spectrum, Scala Play and its microservices cousin Lagom cover too much surface, much of which we intend to leverage Envoy Proxy for, and are highly tied to akka, which is notoriously difficult to master, and SBT, which is in a rather poor shape.</p><p>In the end, Micronaut both offered most of what we needed, and brought less burden than others, and the risk seemed worth it. The fact that it is a modern framework that relies on existing mature libraries for almost everything outside its small core, decreases our chances of hitting hard to solve bugs.</p><h3>Wrapping it up</h3><p>One and a half year later, the design doc above seems to have aged well. Kotlin continues to be strong, Micronaut is solid and advancing faster every quarter, Istio has gotten more traction, and grpc/protocol buffers are everywhere. Developers got exposed to a whole new set of idioms, and multiple new, different, applications were successfully brought to production.</p><p>Each company or group of developers certainly can and should approach this problem in a different way, and will definitively arrive at different conclusions. But often the journey matters more for success than where you want to land, so hopefully the points we brought here can be useful for others in their own unique scenarios.</p><p>If you are looking for a new job opportunity, we are hiring new Loggers to work in Brazil and Portugal, check out our open positions <a href="https://bit.ly/3gQtTPX">here</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5f4fbc5847f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/loggi-product/spraying-our-bike-shed-with-some-new-colors-c5f4fbc5847f">Spraying our bike shed with some new colors</a> was originally published in <a href="https://medium.com/loggi-product">Loggi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Techniques and limitations for experiments]]></title>
            <link>https://daviws.medium.com/techniques-and-limitations-for-experiments-be0c318f369e?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/be0c318f369e</guid>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Sat, 09 May 2020 02:37:13 GMT</pubDate>
            <atom:updated>2020-05-14T23:10:48.618Z</atom:updated>
            <content:encoded><![CDATA[<p>Since I joined Loggi, we have been exploring some techniques for a more data-driven decision making process. In this document I describe some of those, from the perspective of what they bring to the table and their limitations.</p><p>The <a href="https://en.wikipedia.org/wiki/Hierarchy_of_evidence">Hierarchy of Evidence</a> is a well established classification of knowledge gathering mechanisms which bring different levels of assertiveness to the decision making process. As you go up in the pyramid, the cost of generating data increases, as does the confidence in the information. As you go down, the easier it is to generate input for your discussion, but the less likely the input will be a true predictor of the effects of a change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*4RAYyYewZieo1NSs.jpg" /><figcaption>Evidence Hierarchies (<a href="http://flinders.libguides.com/c.php?g=203799&amp;p=1719021">http://flinders.libguides.com/c.php?g=203799&amp;p=1719021</a>)</figcaption></figure><p>Only a few decades ago companies were operating mostly on the base levels of this pyramid, and discussions around quality of evidence were restricted to the areas of medical clinical trials and social sciences. These areas had limited ability to observe and control its subjects of study, and only the most expensive studies were capable of reaching the higher layers of the pyramid.</p><p>As the first big internet companies started to get traction, this all changed. As <a href="https://www.theatlantic.com/business/archive/2011/11/the-big-data-boom-is-the-innovation-story-of-our-time/248215/">The Atlantic</a> puts it: “Science has been dominated by the experimental approach for nearly 400 years. Running controlled experiments is the gold standard for sorting out cause and effect. But experimentation has been difficult for businesses throughout history because of cost, speed and convenience. It is only recently that businesses have learned to run real-time experiments on their customers. The key enabler was the Web — Consider two “born-digital” companies, Amazon and Google. A central part of Amazon’s research strategy is a program of “A-B” experiments where it develops two versions of its website and offers them to matched samples of customers. Using this method, Amazon might test a new recommendation engine for books, a new service feature, a different check-out process, or simply a different layout or design. Amazon sometimes gets sufficient data within just a few hours to see a statistically significant difference.”</p><p>In 2012, Wired claimed that <a href="https://www.wired.com/2012/04/ff-abtesting/">A/B testing was changing the rules of business</a>. Some parts of Loggi look “big data” enough now, and we want to jump on that bandwagon. On the rest of this document we discuss the shortcomings of our current strategy, based on <a href="https://en.wikipedia.org/wiki/Quasi-experiment">quasi-experiments</a> , and the techniques we can use to start to run randomised controlled trials (RCT) experiments at minimal extra cost, and benefit from the same unfair advantages other big data companies have been leveraging.</p><p><strong>Detailed Design</strong></p><p>In this section we discuss the classes of quasi-experiments currently being used in Loggi, their strengths and limitations, and we discuss the caveats and advantages of running RCT A/B experiments using our driver fleet, where we have a meaningful amount of events to enable this technique.</p><p><strong>Quasi-experiments and other studies</strong></p><p>There is a myriad of possible <a href="https://childhoodcancer.cochrane.org/non-randomised-controlled-study-nrs-designs">designs for non-rct experiments</a>. A very simple one in our field is to <strong>launch and see</strong> what happens to the metric you care after launching. This can be seen as an uncontrolled longitudinal study, or a case series. These experiments have very low internal validity because there is no isolation of external factors. For example, rain can affect Loggi metrics, and if you launch in a sunny week you will be happy with your results even if they are potentially negative. See the graph below from a <a href="https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7">similar discussion of the shortcomings of this class of experiments done by Airbnb</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*56KQe65jVubuMO28" /><figcaption>An Airbnb experiment (<a href="https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7">https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7</a>)</figcaption></figure><p>By simply looking at that curve you can mistakenly assume this was a good launch, since the metric goes up afterwards. However, external changes often have more impact than a product change, and through a proper A/B experiment Airbnb found this was neutral and rolled it back. Interestingly enough, later they rolled back the roll back because even though the experiment was not positive, they thought it would be worth the cost to accommodate the use case, a pure product decision, not backed by data, but made safe by data.</p><p>These external factors are called <a href="https://hackernoon.com/what-is-a-confounding-factor-6f30487de3e9">confounding factors</a> in experiments lingo, and are very hard to isolate without a randomized controlled trial. Probably the biggest challenge is that they are often hidden variables, in the sense that you don’t even know they exist. More often than not, when something is not going well, we trigger multiple efforts to fix it. In ops, sales and engineering. How do we know that the change we did in eng is really positive, or if it is a loss masked by a much better idea done by ops? Or maybe it just happened that another company in the ecosystem decided to run an experiment themselves, something we can’t even confirm?</p><p>One should always favor A/B experiments since they give you strong causality guarantees. Quasi-experiments are much more fragile, even though at a first glance they seem simpler. But sometimes A/B experiments are just not possible, as is the case when you don’t control the assignment of users, or when pure random assignment does not result in well isolated groups. Netflix has recently started to discuss how they still use quasi-experiments for a <a href="https://medium.com/netflix-techblog/quasi-experimentation-at-netflix-566b57d2e362">directional read on causality</a>, for say, the effectiveness of their TV ads (on third party networks). In a recent talk, one can see the level of complexity and care they take to <a href="https://channels.theinnovationenterprise.com/presentations/quasi-experimentation-at-netflix-beyond-a-b-testing">compare users from different geographical cities</a>. For example, they avoid comparing pre-test and post-test averages, and instead uses the full time series of each city to derive predictions, and check how much test and control derive from the prediction. Check some online decks for <a href="http://howardsei.org/uploads/QuasiExperimental.pdf">examples</a> and <a href="https://www.researchgate.net/profile/Dr_Senthilvel_Vasudevan2/post/How_effective_are_quasi_non-randomized_studies/attachment/59d63613c49f478072ea3c1a/AS%3A273601507332105%401442243090775/download/NonRandomizedTrial.pdf">explanations</a> of quasi-experiments in the human sciences field.</p><p>Among the most popular of the quasi-experimental designs are (prospective) cohort studies. These studies are sometimes called follow-up or longitudinal studies. The term cohort comes from the old Roman armies, where it was used to describe a large circumscribed group of similar or identical soldiers (eg, all foot soldiers, cavalry, or archers). They then moved together, as a group, through space and time. A variation of those are retrospective cohort experiments, which are observational studies and sit right below in the evidence hierarchy picture we used. Cohort experiments in general are a cheap and intuitive form of <a href="https://ux.stackexchange.com/a/35107">comparing two populations</a>, but have poor validity and are therefore a poor choice for comparing features and their effects. One way to see is that <a href="https://medium.com/earnest-product-management/the-who-and-the-how-using-cohort-analysis-and-a-b-testing-to-increase-conversion-44da0e674e1">cohort studies can tell you who is different, but you need a/b to find out the what makes them different</a>.</p><p><strong>Randomized Controlled Trial Experiments</strong></p><p>In the age of big data, randomized controlled trial experiments are often simply called A/B experiments. The definition, nonetheless, is still rather formal and strict, and much care has to be taken in order to produce a reasonable causality proof. Several companies, like Optimizely, VWO and Google offer products which take care of the several details that are needed to get A/B experiments right, while others as <a href="https://medium.com/netflix-techblog/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15">Netflix</a>, <a href="https://blog.twitter.com/engineering/en_us/a/2015/the-what-and-why-of-product-experimentation-at-twitter-0.html">Twitter</a> or <a href="https://patreonhq.com/a-framework-to-design-randomized-experiments-the-right-way-3640d4271d1f">Patreon</a> just give us a glimpse of their frameworks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/0*9seLU6BquhlNz1fD" /></figure><p>Although we are currently discussing adopting some of these tools, many important decisions related to A/B experiments are not abstracted away by them, and we need to discuss them, both now using our ad-hoc infrastructure, and later as we offload the heavy lift. Google has a dedicated Udacity course on the topic, with a <a href="https://towardsdatascience.com/a-summary-of-udacity-a-b-testing-course-9ecc32dedbb1">great short textual summary</a>, and for the more math inclined, you can <a href="https://rstudio-pubs-static.s3.amazonaws.com/201749_9fc280333a5c4f448687e1d99b9bdf76.html">dive into the equations on this tutorial</a>.</p><p>However, the most important skill is certainly learning how to frame your questions and problems as A/B experiments. For example, it is hard to use an A/B experiment to figure out if restaurants owners prefer more seasoned drivers or are fine with those who just joined. After all, there are not enough sales happening for us to meet A/B experiment criteria, and we cannot isolate that criteria when a sale is closed (or not). But we can use an A/B experiment to see the delivery success rate for a set of restaurants where we randomly assign seasoned or not drivers at each delivery. They are not the same questions, but the response for the latter may be enough for the decision we need to make.</p><p>It is a trade-off on which questions you can ask and the quality of the answers you get. You can still use a retrospective study to claim that restaurant owners prefer seasoned drivers, but you will never know what is responsible for the numbers you observed. Maybe you just got a star salesperson dedicated to sell proposals with seasoned drivers only.</p><p><strong>Some rules of thumb</strong></p><p>Let us assume we want to frame as many questions as we want as A/B experiments. Some of the details we need to pay attention to are how to do the <strong>diversion</strong> of the users, also known as sampling, how to decide the <strong>size of the population</strong>, and <strong>how long</strong> to run the experiment. The response for each of these is quite involving and we discuss them below, but here are Davi’s rules of thumb that can be applied: sample the users randomly without any zooming (e.g., do not filter by hour or city), have at least hundreds of events (converted users) in the experiment every day, and run the experiment for one or two business cycles (e.g., weeks).</p><p>Let us visit the first important decision, how to <strong>divert</strong> your users. This is about saying who goes in an experiment, in its control or neither. Remember, you must have a randomized control, otherwise you are in the quasi-experiment level. This can be implemented by <a href="http://blog.richardweiss.org/2016/12/25/hash-splits.html">taking a mod of a hashed id</a>. Details matter, so ask for help (<a href="https://www.optimizely.com/optimization-glossary/aa-testing/">a/a experiments</a> and <a href="https://blog.twitter.com/engineering/en_us/a/2015/detecting-and-avoiding-bucket-imbalance-in-ab-tests.html">statistical tests can help</a>), and in a full framework, one can use salted hash functions to create many experiment layers. But despite the technicalities, there is still a rather important decision to take: what is the id? For example, we recently ran an experiment using the itinerary Id as the identifier, but it turns out that this gets recreated on retries, and our routes were bouncing around experiment buckets. We fixed that by using the order id instead. The response is not always clear, but the guidance is: always go with the most stable id. Web based experiments sometimes use things like ip address+user agent as a pseudo id, but cookies are more stable and always superior.</p><p>The second important decision is the <strong>size of the population</strong>. To define that, one needs the notion of a conversion (or event) and a baseline conversion rate. For example, your population are offers, and your conversion is the acceptance of an order. Finally, you need to decide what is the minimum effect that interests you. If you want to see if your change increases acceptance rate by 1%, you need way more users than if it suffices to know whether the change is bigger than 10%. With those numbers, you can use <a href="https://www.optimizely.com/sample-size-calculator/">Optimizely Sample Size Calculator</a> or other <a href="http://www.evanmiller.org/ab-testing/sample-size.html">similar sites</a> to get your population size. Besides the nice FAQ and video on that page, you can check many other <a href="https://towardsdatascience.com/the-math-behind-a-b-testing-with-example-code-part-1-of-2-7be752e1d06f">in-depth</a> <a href="https://byrony.github.io/understanding-ab-testing-and-statistics-behind.html">explanations</a> of the <a href="https://towardsdatascience.com/the-art-of-a-b-testing-5a10c9bb70a4">math</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_uVMhdjL6MZQi4Zu" /></figure><p>The third question we need to pose ourselves is <strong>how long to run an experiment</strong>. The naive answer is to wait for convergence and stop, but the way p-value works, if you <a href="https://heap.io/blog/data-stories/dont-stop-your-ab-tests-part-way-through">mistakenly keep checking for convergence</a>, you will find it even when it is not there. So, you need to be mindful to not call your experiment before you planned. Also, you need to run it long enough so that all important internal variations are represented (e.g., our peak traffic on Mondays). Plan for <a href="https://www.crazyegg.com/blog/ab-test-business-cycle/">one or two business cycles</a>.</p><p>It takes sometime to learn how to analyse experiments results, but for now, take some time to familiarize yourself with the many caveats in A/B experimentation. There are a lot of tricks, so do not be afraid to ask for help. Check some hints on how to <a href="https://instapage.com/blog/validating-ab-tests">validate a/b tests</a>, some of the <a href="https://www.kdnuggets.com/2017/05/must-know-key-issues-problems-ab-testing.html">must known key issues</a>, <a href="https://www.widerfunnel.com/3-mistakes-invalidate-ab-test-results/">mistakes that can invalidate your results</a>, <a href="https://splitbase.com/ab-testing-threats/">common validity threats</a> and some <a href="https://www.kdnuggets.com/2017/05/must-know-key-issues-problems-ab-testing.html">additional guidelines</a>. A few highlights follow.</p><p>The first one is the problem of side effects leaking across the experiment buckets. This is something every experiment in a network-like environment needs to face, and in Loggi this can easily happen as the driver fleet individuals communicate a lot among themselves. There is no easy way around, and often we think about per-city experiments for that, but that breaks the randomized sampling and we no longer have proper RCT due to selection bias, <a href="https://www.kdnuggets.com/2017/05/must-know-key-issues-problems-ab-testing.html">the archenemy of A/B experiments</a>. See how <a href="https://tech.okcupid.com/the-pitfalls-of-a-b-testing-in-social-networks/">okcupid dealt with this problem</a>, which is probably what Uber and other large systems with similar problems do. Unfortunately, we are not big enough to get there yet, so for now we just try to formulate our hypothesis to not fall into that trap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FZUKU1JU0fYMilW9" /><figcaption>Okcupid regional clustering (<a href="https://tech.okcupid.com/the-pitfalls-of-a-b-testing-in-social-networks/">https://tech.okcupid.com/the-pitfalls-of-a-b-testing-in-social-networks/</a>)</figcaption></figure><p><strong>Experiments as a Software Engineering Practice</strong></p><p>Although the most recognizable format for an A/B experiment is checking the impact of a feature launch, there are some standard formulations with slightly less intuitive definitions that are still very useful as software engineering practices. Two of them are worth highlighting as we use them often at Loggi.</p><p>The first one are <a href="https://martinfowler.com/bliki/CanaryRelease.html">canary launches</a>. The name comes from the practice used in coal mines to detect hazardous gases by bringing a caged canary down the tunnels. If the bird became silent or died, this was a sign that something was seriously amiss and that the miners should leave. One can see canary launches as A/B experiments where the monitored metrics are the system stability numbers usually tracked by SREs. User assignments follow the usual RCT practice, making this a powerful technique for defining causation, albeit hard to narrow to a specific feature, since the change is the full set of new code in the canary. Usually statistical significance is not closely monitored either because the baseline for failures is so small that any digression is deemed problematic. All of our apps use canary launches nowadays, and our backends also rely on a limited form of it, doing per-machine diversion.</p><p>The second one are <a href="https://developers.redhat.com/blog/2018/04/17/istio-dark-launch-secret-services/">dark launches</a>. This is like an A/B experiment where you clone your requests for sampling, and you drop your responses to stay restricted to infrastructure side-effects. On dark launches sometimes small variations, like in hardware utilization or in algorithm output, matters, and statistical significance can play an important role. In Loggi, we usually dark launch Vehicle Routing Problems like algorithms to be confident they behave well before applying to the real world.</p><p><strong>Final Remarks</strong></p><p>Collecting and analysing data by itself is not enough for a successful data driven culture. As seen in the many links shown here, these techniques all have very real limitations and pitfalls. Given the wrong incentives, they can be used to make any point, right or wrong, intentionally or not (<a href="https://blog.twitter.com/engineering/en_us/a/2015/the-what-and-why-of-product-experimentation-at-twitter-0.html">see Twitter’s recommendation on HARKing</a> or <a href="https://www.youtube.com/watch?v=42QuXLucH3Q">this video on irreproducible research and its science validity</a>). Being faithful to hypothesis you are studying, and keeping a high level of skepticism is a must. The scientific community as a whole has <a href="https://simplifaster.com/articles/p-hacking-harking-scientific-replication/">noticed that risk already</a>, so do not forget <strong>your intuition and that of your colleagues are as important as any of these tools</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be0c318f369e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Porting code with deep learning]]></title>
            <link>https://daviws.medium.com/porting-code-with-deep-learning-e9833b08a9ed?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9833b08a9ed</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Wed, 29 Aug 2018 11:04:40 GMT</pubDate>
            <atom:updated>2018-08-30T17:36:00.472Z</atom:updated>
            <content:encoded><![CDATA[<p>If you studied machine learning in college or through some tutorials in the internet, you probably have this feeling that the problems solved by it are all similar in some sense. Detecting people in photos, classifying sentiment in a text. It always feels like some task that an unhappy employee is doing in a far away country.</p><p>But when the theory books start to discuss how <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem">some machine learning technique captures any function</a>, this often feels a bit disconnected from reality. Those tasks are not that close to what we computer scientists usually describe as functions. There are some nice articles out there that can help fixing that misunderstanding, but in this one I will just show that you can also use machine learning to reproduce some boring programming function using the same techniques that a machine learning expert uses to learn how to execute those mechanical turk like tasks. And no, I will not use xor as my example.</p><p>The function I will use is Python’s <a href="https://docs.python.org/3/library/urllib.parse.html">urllib.parse.urlparse</a>. And I will do it without ever looking at its source code. My equivalent learned function can easily be ported to java, or c++, or any platform with some support for machine learning models. And funnily enough, if the source code of urlparse gets lost, we will still have this learned black box with its functionality.</p><p>I will also take this a bit further, and after learning how to parse urls, we will choose a few arbitrary functions, like hex, and use machine learning to copy them using the same architecture from urlparse, modifying a single parameter in a function call to achieve this. We will even create a model for the inverse of a function, actually learning a function that no one ever wrote code for.</p><p>And to make things even more concrete, I will finally take those functions we translated from python to a machine learning model and deploy them in the browser, using tensorflowjs as the platform. It is as is tensorflow is the new bytecode. I can save my model and suddenly I can have a RPC service built on top of tensorflow-serving that knows how to parse urls, or I can use the model as a transformer in a spark-ml pipeline, or I could use its intermediate state as part of a search engine. With predictable latency and memory requirements.</p><p>It is knowledge captured in matrices. It is <a href="https://medium.com/@karpathy/software-2-0-a64152b37c35">Software 2.0</a>.</p><p>For all these tasks, I chose to use Google Colab with its free GPU offering and I detail each step in the sections below. First, learn how to install Apache Spark with anonymous s3 access to access training data. Then see how to prepare your data for both the development cycle and learning production models. See the powerful seq2seq+attention model we use to learn arbitrary functions, and apply it to copy existing functionality from python libraries. Finally, deploy your models in the browser.</p><h3>Setting up Apache Spark</h3><p>As our first step , we will install Apache Spark. That requires a working jdk and plus a manual hadoop installation, since we cannot use the bundled hadoop versions due to lack of proper anonymous s3 support. We use the shell access support to complete these installations and <a href="https://spark.apache.org/docs/2.3.1/hadoop-provided.html">glue them together using the SPARK_DIST_CLASSPATH env var</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ecaa49110c60c219e79f21a25530034/href">https://medium.com/media/7ecaa49110c60c219e79f21a25530034/href</a></iframe><p>Now we need to access our installation in python, and there is already a tutorial on how to setup <a href="https://mikestaszel.com/2018/03/07/apache-spark-on-google-colaboratory/">Apache Spark on Google Colaboratory</a>. But we need extra steps to access AWS s3 files. That is why we needed a 2.8.x hadoop version, otherwise <a href="https://issues.apache.org/jira/browse/SPARK-21077">we cannot access public s3 files</a>. See below a modified version of the tutorial that takes care of this. As a bonus, we setup the new Apache Arrow support to <a href="https://arrow.apache.org/blog/2017/07/26/spark-arrow/">get faster conversion to pandas</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b9953c1889474e3589367668e38d3f8/href">https://medium.com/media/3b9953c1889474e3589367668e38d3f8/href</a></iframe><p>Finally, we can read anonymous data in s3. There are many public datasets one can access in AWS, and they are now starting to offer data in parquet format, the most convenient columnar storage out there. The code snippet below reads some urls and other metadata from the multi-petabyte CommonCrawl repository, ready to be used in your machine learning investigations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fae245d17011d9ed0f94b0d7c5b702a1/href">https://medium.com/media/fae245d17011d9ed0f94b0d7c5b702a1/href</a></iframe><h3>Data wrangling</h3><p>The usual machine learning toy example assumes you can load all data in memory, and usually does that in a nice <a href="https://paulovasconcellos.com.br/28-useful-pandas-functions-you-might-not-know-de42c59db085">pandas dataframe</a>, from where one can derive all sorts of useful charts and statistics. However, python and pandas in general are quite memory hungry, and even loading a million urls in memory is more than enough to blow up the limits. In our case, we would like to potentially access all training data we have available, a meager 144M urls. Let us use spark to get a handle to all the data, but for now we will load only a small sample of that into pandas.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bf438ed6f6634275a60df72de99853c/href">https://medium.com/media/3bf438ed6f6634275a60df72de99853c/href</a></iframe><p>In this situation, usually people end up with two separate pieces of code, one more debugging oriented, and another more production oriented, and sometimes they diverge creating quite a pain for the machine learning practitioner. According to a friend, the term for this problem is “training serving skew” and there is no magical solution. After all the goals of extracting the last bits of performance and memory are at odds with the idea of keeping extra data around and doing extra computations for debugging.</p><p>Here we carefully organized our code to create a good compromise. First, let us work in the boundaries of the model, doing what is usually called vectorization. On the input side, we need code to take strings and create vectors, and in the output side, we need to be able to convert the one-hot encoded output representation into a string. We can test this part of the code by using to_categorical as if it was a learned identity function, and see if we can reproduce the input after it goes through it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a50da79d6531fda04b5db39240eb0b9c/href">https://medium.com/media/a50da79d6531fda04b5db39240eb0b9c/href</a></iframe><p>Now we write carefully thought out numpy code which we will use to process each batch, and on top of this we overlay pandas transformations for visualization and analysis. The key here is the insight that doing batch level transformation suffices to leverage numpy vectorization gains, and still allows for reasonable memory consumption when moving small collections of batches into the pandas world. In particular, using locals() is a great trick to automatically derive your pandas column names. The code below shows the efficient code to generate a dictionary with the features ready to be used in training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb192c45fe0e06a6321b0fa69008e2c7/href">https://medium.com/media/bb192c45fe0e06a6321b0fa69008e2c7/href</a></iframe><p>And the next block of code shows how we can chain a simple function to capture the same information in a memory hungry but nice to see pandas dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1629855c49308cd78144d609b500892b/href">https://medium.com/media/1629855c49308cd78144d609b500892b/href</a></iframe><p>We do one extra step in the data preparation to wrap our data in a generator. The reason for that is that a full pass in our data would take too long, and we would be looking at sad empty graphs in Tensorboard because its curves only get updated on each <a href="https://github.com/keras-team/keras/issues/6692">epoch</a>. Furthermore, when you consider that you have infinite training data, defining what is your training data is somewhat ill defined and arbitrarily defining what is an epoch makes sense.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2f4d1f685bb5a15f541b33741711cfb/href">https://medium.com/media/f2f4d1f685bb5a15f541b33741711cfb/href</a></iframe><h3>Learning</h3><p>In this step, we assemble and train the neural network which mimics urlparse. The architecture is an almost vanilla version of the sequence classification network. We have a pair of embedding layer to represent each byte of the input or output, a pair of LSTM layers to encode the network state as it traverses each url and its the so-far-parsed output, and an attention mechanism to make sure we can handle large sequences.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6284306319ac49ddfc01fa1810006bf8/href">https://medium.com/media/6284306319ac49ddfc01fa1810006bf8/href</a></iframe><p>One tricky part when using tensorflow under keras is the usage of global graphs and sessions, which is already confusing in tensorflow, but even more when you use keras because the abstractions are more hidden and harder to manipulate. In particular, carefully calling K.clear_session() is key to avoid leaking state as you execute your training cell multiple times.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c91fc5940bb5a42bf496a9e8e95e6d49/href">https://medium.com/media/c91fc5940bb5a42bf496a9e8e95e6d49/href</a></iframe><p>The other gotcha here that cannot be ignored is that Keras layers GRU and LSTM do not really use the GPU. You need to use the CuDNN version of these layers, <a href="https://github.com/tensorflow/tensorflow/releases/tag/v1.9.0">released with Tensorflow 1.9</a>, to see a meaningful improvement from using this hardware. With that in mind, let us start training a simple model to test our code. In this first step, we will get a simpler problem, defining the protocol given the first 32 characters of a url, and we will use the small dev data from a pandas dataframe, with no validation data and only the default instrumentation from keras. In this first moment, the important thing is to see if our code does not blow up and that the model starts to overfit, ideally in a few minutes. We will add a few more bells and whistles after that, and spend more time training for more complex problems.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37ba23c50bf2028c1139eeb0ee79a2a4/href">https://medium.com/media/37ba23c50bf2028c1139eeb0ee79a2a4/href</a></iframe><p>The seq2seq architecture is able to infer the next character given an input and the output generated so far. In training, the full output is available, but during inference, we need to do it character by character. This is called <a href="https://towardsdatascience.com/seq2seq-model-in-tensorflow-ec0c557e560f">decoding</a>, and you get several helpers functions in tensorflow for that. Here, we implement it in an very inefficient but clear numpy code. And we can finally see the results of our trained development model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6ec6d043dff1232330196b16db129c8/href">https://medium.com/media/d6ec6d043dff1232330196b16db129c8/href</a></iframe><h3>Learning for real</h3><p>Our toy model trained quickly and achieved perfect accuracy. However, it simply has not seen enough data to generalize, and is working in a very simple problem. Really learning to parse urls will require more data and more time. For that, let us add some extra instrumentation, using tensorboard to follow learning, and a custom callback so we actually see the parsing attempts at every epoch as the model converges. These callbacks are particularly useful when you have some problem preventing your model from learning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd1385be51ca67975ef85858331a9702/href">https://medium.com/media/bd1385be51ca67975ef85858331a9702/href</a></iframe><p>Because this time our goal is not to simply overfit the model, we need validation data. In the code below we just get the first batch of the training data, and we use it. Extra batches or a generator itself could be better, but Keras <a href="https://github.com/keras-team/keras/issues/2702">does not play nice</a> with these more sophisticated options.</p><p>Also, notice that the performance improvement of the CuDNN layers is over an order of magnitude, but it comes at the price of not supporting the mask_zeroflag. That means that for batches where there is a very long sequence, the learning process and the loss function will be mostly concerned about the padded zeros in all other inputs of the batch. Luckily, since we define the padding length per batch, not over all the collection, the problem is mostly contained. Still, if you really care about it, you could use the loss function below, which caps the contribution of the loss by the padding to half of the loss. You still lose the computing cycles for learning how to generate those zeros, but the CuDNN performance gains far out weight that slow down.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6130e8c483f443eb5cb6ea16c2e87d1b/href">https://medium.com/media/6130e8c483f443eb5cb6ea16c2e87d1b/href</a></iframe><p>With these tools in place, we can now train our copy of the url parser function. Because we organized our code having in mind both the development and production cycle, we can now train this complex model using almost the same code as the one that we used for the simple protocol extracting model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ad5e73772da58052719ae3cf798fa22/href">https://medium.com/media/6ad5e73772da58052719ae3cf798fa22/href</a></iframe><p>And with a bit less than an hour training, we now have a url parser with perfect accuracy. Let us see the results when we parse some of the urls in our dev data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/069dfa72d543dc10cb83702e09e6b0b9/href">https://medium.com/media/069dfa72d543dc10cb83702e09e6b0b9/href</a></iframe><h3>Reusing the architecture</h3><p>The architecture we developed knows how to receive bytes in the input, and how to generate bytes in the output. It has nothing specific about parsing URLs, and in fact we can use to copy virtually any python function that takes a string and whose output can be represented as json. Let us try it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/142852af94885ee84cbc8843d687fbfa/href">https://medium.com/media/142852af94885ee84cbc8843d687fbfa/href</a></iframe><p>More impressively, we not only can copy python code that exists, but we can use our architecture to learn the inverse of these functions, essentially learning about functions that no one ever wrote code for. Below you can see the code we used to define a my_date_format function, which display human friendly dates for humans who like hexadecimal, and how we created a table to define its inverse, and learned a model who can parse dates like that. For the learning to finish, it would have taken a few hours, but you can see that with a short amount of time we were able to get my birth datetime parsed correctly with a day off.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27f07338b5e466cdd5fa45b4b1ae66bd/href">https://medium.com/media/27f07338b5e466cdd5fa45b4b1ae66bd/href</a></iframe><p>In the next installment, we will see how to move the predictfunction out of python code and into our tensorflow graph. With that, our general sequence learning architecture can be deployed in any environment where tensorflow runs. And we will actually show how to do it in tensorflowjs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9833b08a9ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playing Kaggle with scala]]></title>
            <link>https://daviws.medium.com/playing-kaggle-with-scala-c8d5e155c0c4?source=rss-13a36f31dadc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8d5e155c0c4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <dc:creator><![CDATA[Davi de Castro Reis]]></dc:creator>
            <pubDate>Mon, 31 Jul 2017 23:00:15 GMT</pubDate>
            <atom:updated>2017-08-08T20:03:28.059Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Playing Kaggle with Scala</h3><p>I had some fun and challenges with the unusual choice of picking Spark ML as my primary building block in one of Kaggle competitions. Read on to learn more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*JyN3qTL62XeuzAJa6J7bZQ.jpeg" /></figure><p>The <a href="https://www.kaggle.com/c/quora-question-pairs">competition problem statement</a> was to detect duplicate Quora questions, which involves a lot of text analysis, an area we at <a href="https://www.worldsense.com/">WorldSense</a> like a lot. The solution I attempted leverages only algorithms readily available in Spark ML, and is a simple pipeline with data cleaning, TF-IDF, LDA, and a linear classifier.</p><p>The tools of the trade in these competitions are invariably from the python ecosystem. The completeness of the machine learning libraries, the interactive programming model with <a href="http://jupyter.org/">Jupyter</a>, and the robust visualization support are unmatched in other language. On the other hand, python can be quite slow at runtime when you need to do something that is not readily available as a computation supported by a library implemented in C, and the lack of a type system can be unnerving.</p><p>So, while <a href="https://www.kaggle.com/t/654733/worldsensers">we</a> developed a competitive model in the top 11% using <a href="https://keras.io/">Keras</a>, <a href="https://www.tensorflow.org/">Tensorflow</a>, <a href="http://scikit-learn.org/stable/">Scikit-Learn</a> and <a href="https://radimrehurek.com/gensim/">Gensim</a>, I also thought it would be interesting to see how hard it would be to write a complete solution for the problem using the Spark/Scala eco-system. I did manage to reach a 0.45 log loss score, which is somewhat bleh, but still meaningful. There were many possible improvements, but the amount of work I had to get there was just too much, and I did not feel like further improving it.</p><p>Keep reading if you want to learn the details of <a href="https://github.com/worldsense/kaggle/">the code</a> I wrote for each step in this machine learning task from Quora.</p><h4>Loading the data</h4><p>The very first step when doing machine learning is to load your data in the data structures you need. In my case, since I will be using Spark&#39;s machine learning libraries, I want to have the data in a spark dataframe (analogous to a pandas dataframe in python, which in turn are analogous to the R dataframes). The original data is stored in a csv which can be directly loaded in Python, the lingua franca of Kaggle competitors. In Spark, I needed a few custom settings to get it right.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98410f85bc28094caf0b89ed766cc41e/href">https://medium.com/media/98410f85bc28094caf0b89ed766cc41e/href</a></iframe><p>During data loading, it is also usual to do small transformations to match the environment expectations. For example, here I renamed a field with a underscore in the name, which is not polite Scala. After that, I could leverage Spark 2.0 strongly-typed dataframes sibilings, called <a href="https://databricks.com/blog/2016/01/04/introducing-apache-spark-datasets.html">datasets</a>. Those are a joy to work with, so as soon as I loaded the data I converted it to the Features case class, surfacing those types and making coding easier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e8c419323496c41cbd83556926a83f2/href">https://medium.com/media/2e8c419323496c41cbd83556926a83f2/href</a></iframe><h4>Data Cleaning</h4><p>I have loaded the data doing the minimal amount of transformations, but before feeding it to our machine learning algorithms, I needed to clean it. This is inevitable, every real world data comes with some noise that either breaks or harms the learning process. Even Kaggle competition data has its share of problems, ranging from the data itself, like missing values, to completely borked rows that need to be dropped.</p><p>For the integer values, one usually needs to look out for values outside the domain valid range, or dataset level constraints violations, like non-unique ids. In this specific dataset, I only had to handle missing values, which could be easily replaced. For free text values, like what is given in question1 and question2, there are a bunch of transformations which are usually required, regarding encoding, diacritical representation, how to deal with math symbols, and so on. Some of those decisions are not strictly in the domain of data cleaning, and are heuristics in nature, while others are more straightforward. Here is what I did:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62f40eeb273a60d6a76b01764dfcc4e1/href">https://medium.com/media/62f40eeb273a60d6a76b01764dfcc4e1/href</a></iframe><p>The implementation of UTF8 normalization and diacritical handling relies on <a href="https://www-01.ibm.com/software/globalization/icu/">IBM ICU</a>, a state of the art library for dealing with unicode. I highly recommend feeding all your text through it due to its robustness at dealing with crazy input. This will often save you from all sorts of failures when you feed your input to more fragile code further down in your pipeline.</p><h4>Vectorization</h4><p>Machine learning works in the domain of vectors and matrices, a very natural world for the image processing community, which developed most of the field over the last decades. Words and characters are somewhat strangers in this universe, and machine learning with text will invariably involve converting your words to a vector representation.</p><p>I started by first converting our phrases to sequences of words, a process named <a href="https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization">tokenization</a>. I also did small, lossy transformations (where we lose potentially useful information), to normalize the resulting representation. After this step, punctuation will be gone, as will be the so called call stopwords, which are the words, like &quot;the&quot;, that are defined as carrying very little information and believed to introduce noise in our learning. The exact set of stopwords is problem specific, and we started with <a href="https://github.com/6/stopwords-json/blob/master/dist/en.json">this list</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50a42a1e8f4494f3c9c472e8bc5a3769/href">https://medium.com/media/50a42a1e8f4494f3c9c472e8bc5a3769/href</a></iframe><p>After that, I transformed each word, which is a variable length series of characters, into a fixed size vector. In the world of deep learning, this is often done through a table of <a href="https://www.tensorflow.org/tutorials/word2vec">word vectors</a>, which was trained with an unsupervised learning algorithm over a large corpora. I used a more traditional technique, where we weight terms by their frequency times their <strong>inverse document frequency*</strong>, the so called TF-IDF vectorization. Its biggest downside is that it loses positional information of the words, but despite this significant shortcoming, and can still yield interesting results. On the upside, it is readily available in Spark ML and easily understood (*<strong>Learn more</strong>: <a href="http://spectrum.ieee.org/geek-life/profiles/computer-science-a-womans-work">inverse document frequency, or IDF, was conceived by Karen Sparck Jones, one of the many great women in computer science</a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c120806fd981565c09d76cade730606d/href">https://medium.com/media/c120806fd981565c09d76cade730606d/href</a></iframe><p>You will notice the class MultiColumnPipeline in both this snippets. This is an adapter I had to write to deal with the fact that the questions are given in two columns. The adapter takes a traditional spark ml estimator, which trains data in a column and applies its transformation to the same column, and wraps it in a modified pipeline which trains in the concatenated data of multiple columns and applies its transformation in each individual column.</p><h4>Latent Dirichlet Allocation</h4><p>We now have, representing each question in our training set, a variable length sequence of numeric vectors. But machine learning algorithms do not usually directly handle variable length sequences, since they don&#39;t know how to assemble those vectors in a matrix representation. This needs to be done somehow, either in a straightforward way, through padding and pruning, through an attention technique, or through and algorithm that is capable of doing this transformation. In our case, we use the <a href="http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/">Latent Dirichlet Allocation</a> algorithm, which is readily available in Spark ML, and has the nice property of being somewhat interpretable, an important aspect often overlooked by machine learning practitioners. A more modern architecture, using word vectors and attention is described in this <a href="https://explosion.ai/blog/deep-learning-formula-nlp">excellent writeup</a> from the authors of the python library <a href="https://spacy.io/">Spacy </a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8be18ed4df4f0ec0e29c4c280f6710e2/href">https://medium.com/media/8be18ed4df4f0ec0e29c4c280f6710e2/href</a></iframe><h4>Logistic Regression</h4><p>Now that we have fixed size vectors representing each of our sentences, in principle we all we need is to calculate the distance between these two vectors and feed the result to Kaggle. For example, we could use straightforward algebraic formulations, like the cosine between the two vectors, or take advantage of LDA probabilistic nature and compute the <a href="https://blakeboswell.github.io/2016/gensim-hellinger/">Hellinger distance</a> between the vectors. That choice would define the typical numerical values we get in the output, and we would need to adjust the client code consuming these numbers to properly interpret the distribution of values we generate.</p><p>We do have a specific goal for the distance though, which is maximizing log-loss in the Kaggle competition. In a real world scenario, other goals could be present, like how fast the distance computation is, or how easy it is to interpret. But since in our problem statement, all we need is to maximize our metric, an interesting option is to just use another component in our machine learning pipeline to do that. Here, we simply decided to feed each pair of vectors to a <a href="http://eli.thegreenplace.net/2016/logistic-regression/">Logistic Regression</a> learner, which will yield the probability of the two questions being repeated, as defined by our training data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5025de557e331cea1b6b77f1b10057fc/href">https://medium.com/media/5025de557e331cea1b6b77f1b10057fc/href</a></iframe><h4>Evaluation Metrics</h4><p>We have been given a specific metric to optimize, but when working in a problem in machine learning, it is often the case that you want to look at different metrics while you develop until you develop the intuition of what each one is capturing, and grow confident that what you are optimizing really matters. The Spark ML library is not very rich in terms of implemented metrics or visualization tools, but regardless I tried to look at what was convenient enough. First, the code that puts the stages and trains the model we want to inspect:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1c4dbe54df1a9852c2ab0b36e80d6cc/href">https://medium.com/media/e1c4dbe54df1a9852c2ab0b36e80d6cc/href</a></iframe><p>And now the helper functions that compute the metrics and print them in the screen in our development iterations. The lda topics, albeit not a metric per se, serve the same noble goal here: understanding what is happening as we play with the code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0481db8cb9bc7c8e00dd063bc3acffd1/href">https://medium.com/media/0481db8cb9bc7c8e00dd063bc3acffd1/href</a></iframe><p>It is important to not try to optimize the numbers you see here, otherwise you will be falling in the overfit trap, since you do not know how well your model will generalize to unseen data. In the following section we discuss how to alleviate that problem, but nonetheless looking at numbers at every step in your machine learning pipeline is key to develop intuition.</p><h4>Hyperparameters</h4><p>As the last piece of our puzzle, we will now wrap fully functional machine learning pipeline we have in a loop that will define the best parameters for the machine learning algorithms themselves. This is called hyper-parameter selection, and even though it could be seen as a fine tuning step, often the impact of hyper-parameters in the learning process can be huge, and searching that space is a must to achieve good results in a task.</p><p>This is the time we harvest the benefits of having expressed all of our computation as a Pipeline , the abstraction Spark ML borrowed from Scikit-Learn. This means that the Spark knows how run our learning algorithm multiple times and compare the results of each run.</p><p>There are many interesting hyper-parameters to tune, like the list of stopwords, or how one normalizes diacriticals and capitalization, which arguably is a change of the learning algorithm itself, but ultimately that is what all hyper-parameters are. However, each hyper-parameter you try to change increases exponentially the time to train. The reason for that is that our pipeline assumes that every parameter in the model is dependent on each other. In practice, it is often ok to assume independence among most parameters, and that can be expressed by tuning each parameter set at a time.</p><p>Below we see the code we used to decide on vocabulary size, minimum frequency, lists of stopwords, number of topics and max iterations for the LDA and logistic regression steps, with a total of over a hundred models being trained. Luckily, the cross validator has a hook to define the metric we want to optimize, and even though I had to implement a custom evaluator for log-loss, it was straightforward to plug it in.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c91a9405eaeb11ef991d6e8aca80510a/href">https://medium.com/media/c91a9405eaeb11ef991d6e8aca80510a/href</a></iframe><p>Another important aspect here is that I cross validated the model over 5 folds for each hyper-parameter set, which helps avoiding overfitting the hyper-parameters to the training data, as was the case before introducing the cross validation.</p><p>Usually this is not the step where one stops in a Kaggle competition. In fact, here is not a single best set of hyper-parameters, and in practice each set captures different angles of the problem. In particular, when one starts to think of models as hyper-parameters themselves, you will see that you want to build many different models and combine them somehow. And the secret to score high in those competitions is to mix and match a lot of techniques and discoveries and create an <a href="https://www.analyticsvidhya.com/blog/2015/08/introduction-ensemble-learning/">ensemble</a>, using something like XGboost. We will not go into this here, even though XGBoost do have <a href="http://dmlc.ml/2016/10/26/a-full-integration-of-xgboost-and-spark.html">nice bindings</a> for Spark/Scala.</p><h4>Submission</h4><p>Now that I have a model using the best set of hyper-parameters that had been found, I need to apply it to the test data provided by Kaggle and submit the results. First, I loaded the test data into the same format that the model expects as input.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4232b4e183b9ce1c3774429f763378f6/href">https://medium.com/media/4232b4e183b9ce1c3774429f763378f6/href</a></iframe><p>Then I applied the model and finally wrote the output file. I just needed some tricks to rename the output columns and again and I had to set the right csv parameters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13bd573e3999d4257601230084e88890/href">https://medium.com/media/13bd573e3999d4257601230084e88890/href</a></iframe><p>To wrap it up, see how simple and clear the main function can be when one adheres to the <a href="http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html">pipeline model</a>, be it either in the Scala or the in Scikit-Learn world.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60363980728a8cd5d02dd3174808ee73/href">https://medium.com/media/60363980728a8cd5d02dd3174808ee73/href</a></iframe><h4>Conclusions</h4><p>Even with the recent advancements in the area, machine learning is still a complex field and achieving above average results in practical problems depends on a lot of tools which unfortunately are still sub-par in the Scala/Spark ecosystem. Even though I have presented an end to end solution with all the typical bells and whistles, the lack of an interactive environment with visualization support like Jupyter prevents exploring efficiently the problem space, and I got with the initial solution I thought about. In the same fashion, the lack of a large, integrated, easy to use, collection of data cleaning algorithms, learners, transformers and evaluation metrics, place a big cost of each idea one wants to try, and since much in this field is still experimentation, development cost dominates. For example, not having a readily available LSTM implementation already puts you way too much behind the state of the art.</p><p>On the bright side, having the type-system on your side is very helpful. An integrated environment like Intellij with proper auto-completion, good testing frameworks, and the superior speed of data crunching with Scala brings its own benefits. If I had Keras and some subset of Scikit-Learn in Scala, I would probably never look back.</p><p>Footnotes</p><ol><li>There are a handful of projects which provide Scala or Spark support in Jupyter, or provide a similar experience in another environment. Databricks proprietary notebooks are a good option if it fits in your budget. I use <a href="https://github.com/alexarchambault/jupyter-scala">jupyter-scala</a> and <a href="https://github.com/alexarchambault/plotly-scala">plotly-scala</a>, but the productivity is just not the same yet.</li><li>Unfortunately the data provided by Quora for this competition was not properly sanitized and inane things like supposedly random identifiers were actually leaking information about the results, making the overall competition less interesting than it could have been. See <a href="https://www.kaggle.com/c/quora-question-pairs/discussion/34560">https://www.kaggle.com/c/quora-question-pairs/discussion/34560</a> for details.</li><li>Our competitive solution, which benefitted from the leaky data and was written with keras/scikit-learn using LSTM and other more modern techniques scored 0.15664, which was position 335 overall by the end of the competition.</li><li>Python can often outperform java/scala, but that is only the case when the majority of your code is actually running in C, and is properly using multiples cores and/or the GPU (cython being one of the possible <a href="https://explosion.ai/blog/multithreading-with-cython">workarounds</a> nowadays). In the JVM, your performance will usually be more uniform, and depending what you are doing, that may be a better trade-off. In many ways, it is sad that things just don&#39;t simply work.</li><li>There many other machine learning libraries in Java other than Spark ML. But I don&#39;t see any of them matching scikit-learn breadth of scope and maturity, and I chose Spark ML due to the deep integration with Spark itself.</li><li>It would be only natural that Spark ML cross validator would train the models in parallel in the cluster. Unfortunately, that is not the case: <a href="https://stackoverflow.com/questions/41479359/spark-achieve-parallel-cross-validation-for-scala-api">https://stackoverflow.com/questions/41479359/spark-achieve-parallel-cross-validation-for-scala-api</a></li><li>Writing MultiColumnPipeline was the main software engineering task to make this work. That class is an adapter that takes two columns, concatenates them, train the given learner on the full data, and then applies the learned model in each of the columns individually. Had this code not been written as a an adapter pipeline, we would have deviated from the Spark ML Pipeline model and would not have been able to use its cross validation functionality.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8d5e155c0c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>