<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dima on Medium]]></title>
        <description><![CDATA[Stories by Dima on Medium]]></description>
        <link>https://medium.com/@Dima?source=rss-335654e24a38------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*a7FAdaYGhE0WlvNaZVtYnw.jpeg</url>
            <title>Stories by Dima on Medium</title>
            <link>https://medium.com/@Dima?source=rss-335654e24a38------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:00:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Dima/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Miro & Styra: Battle-testing Enterprise OPA]]></title>
            <link>https://medium.com/miro-engineering/miro-styra-battle-testing-enterprise-opa-d4b8e005e25c?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4b8e005e25c</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[miro]]></category>
            <category><![CDATA[styra]]></category>
            <category><![CDATA[open-policy-agent]]></category>
            <category><![CDATA[authz]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Wed, 19 Jul 2023 15:09:13 GMT</pubDate>
            <atom:updated>2023-08-09T16:17:17.229Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Learn how Miro extensively benchmarked Styra’s Enterprise OPA and found it to be 10,000 times more performant than the open source container.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Ny9DxH10nBFYNir-uId8A.jpeg" /><figcaption>The logo of <a href="https://github.com/StyraInc/regal">Regal</a>, the linter for Rego, is shamelessly used here, because it fits the illustration best!</figcaption></figure><p>End-user authorization powers the business rules in the Miro application around what our users are allowed to see and which actions they are allowed to perform. The goal, as part of decomposing the monolith, is to <a href="https://medium.com/miro-engineering/how-miro-leverages-open-policy-agent-to-implement-authorization-as-a-service-763f08469e5">build a centralized authorization service</a> powered by <a href="https://www.styra.com/blog/how-to-write-your-first-rules-in-rego-the-policy-language-for-opa/">Rego policy</a>. Over time, nearly every incoming request will hit authorization, oftentimes more than once. Part of the challenge is that every board session requires real-time authorization to enable a great end-user experience, mandating a high-throughput decision engine that can handle large volumes of requests with low latencies.</p><p>A Miro-specific requirement is to enable pushing real-time AuthZ decision updates about the currently online users to downstream services. Thus, we’re looking at the ability of the policy decision engine to rapidly re-evaluate thousands of policies each time we are notified about a change in the data that may affect the result of these policies.</p><p>At the technical level, we are evaluating a standard HTTP API endpoint vs. streaming gRPC with a high concurrency of requests, using both the openly available <a href="https://hub.docker.com/r/openpolicyagent/opa/">Docker container of OPA</a>, the custom-built <a href="https://go.dev/">Golang</a> wrapper over the open source OPA, and <a href="https://www.styra.com/blog/introducing-styra-load-enterprise-opa-distribution-for-data-heavy-authorization/">Styra Enterprise OPA</a>. We’ll discuss the practical challenges of implementing each solution and the comparative advantages of a commercial vs. an open-source or hand-rolled approach.</p><h3><strong>Miro’s technical requirements</strong></h3><p>There are two fundamental measurements to Miro’s POC with Enterprise OPA: throughput and latency. We need high throughput in order to support many concurrent authorization decisions around board interactions. We need low latency for each decision in order to avoid visible lag or janky-ness in the user experience.</p><p>Requiring both high throughput and low latency may be a cornerstone of the use case for OPA that we have at Miro, and ensuring we are in good shape at both fronts makes the problem all the more interesting.</p><p>The set of systems under test (SUTs) and the means to interface with them is as follows:</p><ol><li>The open source OPA container, interfaced via the standard HTTP API.</li><li>Enterprise OPA, interfaced via the standard HTTP API.</li><li>Enterprise OPA, interfaced via the standard gRPC API.</li><li>An in-house solution that uses the OPA SDK as a black box, custom gRPC API.</li><li>An in-house solution that uses the OPA SDK as the “gray box”, custom gRPC API.</li></ol><p>When it comes to latency, the first major takeaway is that <strong>streaming gRPC hands down beats the HTTP interface</strong>, in mean and median latency, as well as in high percentiles.</p><p>The tribute has to be paid to the open source OPA implementation, as once we put our own thin gRPC wrapper around it, the very OPA engine performs steadily and predictably. Thus, the first conclusion is that if predictable latency is required, it is feasible to use the open source OPA engine, but <strong>do get away from the HTTP API as soon as possible</strong>. Depending on how creative you are willing to get, the open source OPA offers several ways to programmatically connect to it, and both <a href="https://www.openpolicyagent.org/docs/latest/integration/#integrating-with-the-go-sdk">OPA SDK</a> and <a href="https://www.openpolicyagent.org/docs/latest/integration/#integrating-with-the-go-api">OPA API</a><em> </em>are well documented and mature.</p><p>One important caveat that should be noted here is: as part of Miro’s problem statement, we need to frequently alternate data update requests with policy evaluation requests. This makes using the HTTP API an even less feasible option for us. Because even if the underlying Rego engine is massively parallel and lightning fast, once we begin pushing updated data into this engine, we will have to wait until it settles, and this slashes end-user throughput and dramatically increases end-user latency.</p><p>With the above said, we are in the realm of gRPC. Following the original set of five alternatives, (3) makes (2) obsolete, (4) makes (1) obsolete, and we are down to three fair gRPC alternatives: (3), (4), and (5).</p><p>The Rego engine that Styra offers, both in the open source OPA and as part of Enterprise OPA, is implemented in Golang, which makes it massively parallel thanks to the use of native goroutines. Both engines perform steadily, and thus latency is no longer a constraint, and we can focus solidly on the throughput aspect.</p><h3><strong>Architecture and implementation</strong></h3><p>When it comes to throughput, we need to compare three gRPC options:</p><ul><li>Enterprise OPA,</li><li>Using the open source OPA engine as the “black box”, and</li><li>Leveraging our knowledge of open source OPA’s internals to get the best out of it.</li></ul><p>The “black box” option, in a nutshell, is just about adding a thin layer on top of the open source OPA engine to make it gRPC-enabled.</p><p>And the final “greybox” approach is about applying nontrivial, implementation-dependent optimizations in an attempt to boost the performance of the baseline solution of interfacing with the open source OPA engine via gRPC.</p><h3><strong>Claims and results</strong></h3><p>First and foremost, data volume and data mutation performance.</p><p><em>[Claim A]</em>: When it comes to memory usage, <strong>both the open source OPA container and our “black box” solution use 25x .. 30x more RAM compared to optimized solutions</strong>. Nothing can be more clear. If you need your Rego policies to deal with ~gigabytes, or even high hundreds of megabytes of data, using the open source OPA container, and/or using the OPA SDK as the “black box” from your own Golang wrapper code are just non-starters. The “grey box” OPA API is the way to go.</p><p><em>[Claim B]</em>: In addition to memory bloat, <strong>the throughput of data updates is literally ~thousands of times slower when it comes to using the HTTP API to the open source OPA container compared to the Enterprise OPA solution</strong>, or compared to the sophisticated in-house “hack” that utilizes nontrivial low-level optimizations. Same as with memory usage, even if your total “active set” of data is just dozens of megabytes, having to update it at high frequency (say, due to push notifications that come from a certain Kafka topic) would slash the performance of end-to-end policy evaluations, as the OPA part of the system would use the vast majority of time applying those updates. In fact, it would simply come to a stall, not being able to handle any real load, starting from low hundreds of data mutations per second.</p><p>Being far better than the open source OPA when it comes to dealing with external data is the major selling point of Enterprise OPA. And indeed, Enterprise OPA delivers on this point! Some ~30x less RAM usage and literally 1,000x+ boost to the ability to work with the data dynamically is an impressive accomplishment. To put it bluntly: with any reasonable throughput of data updates, which is always the case with end-user authorization, Enterprise OPA is just what fits the bill, while the open source OPA solution chokes along both RAM and CPU dimensions.</p><p><em>[Claim C]</em>: Now, when it comes to frequent data updates, Enterprise OPA is again leaps and bounds ahead of our thin gRPC wrapper around the open source Golang OPA engine. This has to do with the means the open source OPA/Rego engine accesses the data that is fed into it. In a nutshell, this data has to be prepared to be “Rego-friendly,” to which there is a substantial price to pay. This price can be paid when the data updates are applied, and this is where the ~1,000x slowdown of applying data updates comes into play. Or this price can be deferred, effectively until the point in time when a particular bit of data needs to be accessed by the Rego policy itself. However, at the end of the day, <strong>this price of “preparing” the data to be “Rego-friendly” for the open source OPA engine must be paid. </strong>And in the environment where data updates are frequent, this does bring the open source OPA engine to a halt. By comparison, Enterprise OPA soars in such a scenario.</p><p>Now, for our own, Miro-internal POC reasons, after many experiments, even the route of injecting custom OPA/Rego functions did not prevent us from paying the full price. Eventually, we managed to arrive at a not-too-hacky way that makes it possible to only pay it once, though.</p><p>After several careful and diligent implementations (thanks to software engineer <a href="https://medium.com/@ionutzp">Ionut Popa</a>), what worked out at the end of the day is to pre-prepare the data to be “OPA-friendly” at data update time, but only use shallow copies while “overloading” the payloads for each individual policy evaluation request. Ultimately, performance-wise, despite the tremendous effort we have put into these experiments, <strong>Enterprise OPA still beats our best POC candidate by a factor of four.</strong></p><h3><strong>Conclusion</strong></h3><p>At the end of the day, we conclude that Enterprise OPA does ship with a number of significant improvements over the open source OPA solution. Fresh out of the box, Enterprise OPA beats the open source OPA ~30x in RAM usage and ~1,000x in performance of data updates.</p><p>Moreover, while we have put a lot of effort into trying to squeeze the best out of the open source OPA engine, its performance is still below what Enterprise OPA offers. In our experiments, we effectively ended up hacking into the OPA engine, cutting all the corners we can, and our best shot still loses to Enterprise OPA by a factor of four.</p><p>It is worth noting that we did not have access to the source code of Enterprise OPA while running the experiments covered in this post. Having access to the source code of Enterprise OPA would unlock the ability for Miro to embed it into our application and to reap these ~1,000x performance benefits that we saw in our black-box test runs.</p><p>To learn about Miro’s specific challenges with authorization and how OPA helps us address service-to-service and real-time, high-throughput authorization, check out <a href="https://medium.com/miro-engineering/how-miro-leverages-open-policy-agent-to-implement-authorization-as-a-service-763f08469e5">this recent blog post</a>.</p><p>Do you have ideas or a different approach to authorization? Let us know in the comments — we’re interested in what you have to say!</p><p>Are you interested in joining our team? Then check out our <a href="https://miro.com/careers/open-positions/">open positions</a>.</p><p>Finally, don’t forget to try <a href="https://www.miro.com/">Miro</a> or the <a href="https://developers.miro.com/">Miro Developer Platform</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4b8e005e25c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/miro-engineering/miro-styra-battle-testing-enterprise-opa-d4b8e005e25c">Miro &amp; Styra: Battle-testing Enterprise OPA</a> was originally published in <a href="https://medium.com/miro-engineering">Miro Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High-Performance OPA]]></title>
            <link>https://medium.com/@Dima/high-performance-opa-6400fc0570d1?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/6400fc0570d1</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cpp]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Tue, 19 Jul 2022 21:10:25 GMT</pubDate>
            <atom:updated>2022-07-19T21:10:25.348Z</atom:updated>
            <content:encoded><![CDATA[<h3>Transpiling Rego policies into C++.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9C4FKNVBa5p9jI-MC9DUyw.png" /></figure><p><a href="https://dimakorolev.substack.com/p/high-performance-opa">High-Performance OPA</a></p><p>#OpenPolicyAgent #OPA #Rego #Transpilation #Performance #100KQPS</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6400fc0570d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design Meetup]]></title>
            <link>https://medium.com/@Dima/system-design-meetup-8db8d2797cf6?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/8db8d2797cf6</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Sat, 26 Feb 2022 22:50:36 GMT</pubDate>
            <atom:updated>2022-02-26T22:50:36.052Z</atom:updated>
            <content:encoded><![CDATA[<p>Kafka and RabbitMQ, and the future of this Medium.</p><p>Hi folks,</p><p>For a bit over a year now I’m hosting the <a href="https://youtube.com/dimakorolev">System Design Meetup</a>. The last episode, about Kafka and RabbitMQ, was quite interesting, and I’ve cut a few shorts from it, <a href="https://dimakorolev.substack.com/p/kafka-and-rabbitmq">here</a>. Teasers:</p><ul><li><strong>An intro to Kafka as the product</strong>, in five minutes.</li><li><strong>An intro to RabbitMQ and Erlang</strong>, all under ten minutes.</li><li><strong>Why append-only queues are always faster than DB transactions</strong>, in two minutes.</li><li><strong>How Kafka consumers works, leveraging consumer groups in two ways</strong>, ten minutes.</li><li><strong>Using heartbeats to re-create total order from partial order</strong>, three minutes.</li></ul><p>If you want to join the meetup, <a href="https://tinyurl.com/sdm-slack-invite">this link</a> should take you to our Slack.</p><p>Not much else to say. Medium followed the path of Quora, which followed the footsteps of Facebook : paid beats organic.</p><p>I’ve moved to <a href="https://dimakorolev.substack.com/">Substack</a>, not necessarily because I believe it is better, and not because I plan to monetize my posts, but largely because I like Substack’s policy of allowing one to export the list of subscribers’ emails. Good incentives always beat good decisions in my book, and, with such a policy (for as long as it is in place, at least), I am quite confident the platform will be growing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zceZd65YUHTNveuVOHSieg.jpeg" /><figcaption>Including an image here because Medium doesn’t let me to it later, during publishing. Hehe.</figcaption></figure><p>Thanks,<br>Dima</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8db8d2797cf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Management as Code]]></title>
            <link>https://medium.com/dima-korolev/management-as-code-88919b00f6a8?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/88919b00f6a8</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[management-software]]></category>
            <category><![CDATA[configuration-as-code]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[jira]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 14:13:03 GMT</pubDate>
            <atom:updated>2020-07-14T14:13:03.259Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jf31a2ckH4yU7BDhoP2ERw.jpeg" /><figcaption>Stock photo found via Bing Image Search.</figcaption></figure><p>The tech scene has largely converged to the understanding that configuration as code is a good mantra.</p><p>I wonder when would we embrace the idea of management as code?</p><p>As an engineer, I think the largest win of configuration as code over the previously existing practices is that we have killed drag-and-drops and other UI-based operations in favor of configuration files.</p><p>And when configuration is stored in human-readable files, it allows us to follow best engineering practices, most importantly reproducibility, code reviews, and integration testing.</p><p>If some configuration, say, a new environment or a new deployment, has succeeded once, it can be projected to succeed the next time with, ideally, zero extra cost. If some configuration has succeeded at altering the state of three servers, it’s a great start on the path to correctly altering the state of 10, 100, and 1000 machines.</p><p>Here I am not saying scaling is a straightforward problem. It never is. But reproducible individual steps most certainly lay the foundation on which it is a lot easier to predictably build large-scale systems.</p><p>I think it’s time we, the engineers, do the same to Jira.</p><p>With configuration as code as the best practice with us for just a few years, we have collected more than enough data to know exactly what is easy and what is complicated. We know exactly where time is lost, and we know exactly who and how managed to get things done a lot faster than usual.</p><p>We no longer live in the world of “I’m a believer in approach X”. The world today is a lot closer to “we know for a fact that approach Y is superior to approach Z”.</p><p>And the world of today is a lot more meritocratic, as we don’t have to rate candidates, for example, by how confidently they speak, or by how many configurations they have built in the past several years. Instead, we can accurately judge whether their understanding of the process of building and maintaining production services is correct. Much like algorithmic problems and big-O notation enable us to ask questions with definitively right and definitively wrong answers, in the world of configuration we have dramatically shrink the space where one’s hand-waving and a “strong” resume can take them places; we’re looking at the true substance these days.</p><p>Knowledge is the reduction of room for uncertainty, in science, and, especially, in engineering. And with configuration as code we, as the industry, have made a big step forward.</p><p>Now, when it comes to project management, imagine if each “permission obtained”, “access granted”, “cross-team dependency resolved”, or some “project successfully derisked by eliminating an unnecessary costly dependency” is no longer a vaguely recalled poorly told story, perceived and judged differently by different people.</p><p>Imagine it’s a well-traceable trail of pull requests, each of which visible to all or most of the organization, and each of which can be used later. Either as a reference of how one can best get things done in an effective way, or an example of how lost productivity and wasted time look like.</p><p>Won’t this be a magical world?</p><p>Let’s make it happen.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88919b00f6a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dima-korolev/management-as-code-88919b00f6a8">Management as Code</a> was originally published in <a href="https://medium.com/dima-korolev">Live Long and Prosper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[During the past few weekends I was thinking of a problem that came up at work.]]></title>
            <link>https://medium.com/@Dima/during-the-past-few-weekends-i-was-thinking-of-a-problem-that-came-up-at-work-b5ad8eea2acc?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5ad8eea2acc</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Sun, 07 Jun 2020 13:54:24 GMT</pubDate>
            <atom:updated>2020-06-07T13:54:24.306Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Image taken from https://indianexpress.com/article/technology/science/ai-models-can-help-unravel-mystery-of-human-brain-study" src="https://cdn-images-1.medium.com/max/759/1*CkT2WEDedUbNbDBPZSFCHQ.jpeg" /><figcaption>Image taken from <a href="https://indianexpress.com/article/technology/science/ai-models-can-help-unravel-mystery-of-human-brain-study-5112260/">https://indianexpress.com/article/technology/science/ai-models-can-help-unravel-mystery-of-human-brain-study-5112260/</a></figcaption></figure><p>During the past few weekends I was thinking of a problem that came up at work.</p><p>Leaving the details out, a decent chunk of the problem has to do with transforming JSON objects dynamically. According to the rules that by themselves are dynamic. According to the dynamic set of rules that themselves have to be expressed in a JSON object.</p><p>Given this part of the problem is very much open-ended — and also is not, strictly speaking, 100% an engineering one — I resorted to a TDD approach of assuming certain product usecases would be covered, and designed the end-to-end architecture around this assumption. Ultimately, of course, it has come to figuring out how to best make the very inner engine work.</p><p>The first decision, that was already well crystallized in my head by the time to build the engine, is that the engine itself must be a functional, not an imperative one.</p><p>Building imperative state machines sucks. It sucks doubly when the ultimate end user is not an algorithms and data nerd to the degree I am one. Also, whether you like it or not, where there is a dynamic, flexible language used by non-engineers, those people would inevitably find hilarious ways to shoot themselves into the foot, which I could not even think of.</p><p>So, let’s keep it purely functional. No variables. No global state. No side effects. Pure transformation, from one object into another, perhaps with several steps in between, which by themselves are, of course, purely functional.</p><p>Originally I was planning to introduce strong typing to the objects and sub-objects to later be transformed, but have quickly let go of this idea, when the colleagues I have shared early results with were genuinely wondering if I am writing an obfuscation engine.</p><p>Very quickly, the above train of thought has come to a simple observation that I only need a very narrow subset of the engine to make the first step.</p><p>The nature of this project is that other teams will be working on implementing the rules application engine on their own platforms. So, while I would, of course, provide them with the full spec, several reference implementations, and quite a few test cases, the more compact and self-contained I make the very rule engine language, the more the other teams would thank me later on.</p><p>So, it quickly became apparent that some pre-processing step, a transpiler if you with, would be in order. Once I was past this pivotal moment, it was the very next step to realize that any and all “strong typing” I was hoping to introduce would be done before the transpilation phase.</p><p>By then it was, thus, exclusively about formulating the minimalistic self-contained version of the engine running the above transpiled code.</p><p>Obviously, the next question I asked myself was: should I transpile the input functional code into the output imperative one, or should I stick with the functional paradigm all the way?</p><p>In my head, this question has quickly evolved into a different one: Would a functional rule-applying engine be as simple to implement as an imperative one?</p><p>I gave it a shot, and it turned out pretty damn well to my taste. Would be curious if someone else could see major flaws which I may have missed. The main ideas of the language are outlined below.</p><p>The rule definition language consists exclusively of function definitions.</p><p>Zero-parameters functions are constants. Such as pi. Or such as a pre-determined list of JSON keys to be filtered out from the input object.</p><p>Other functions have one or more parameters. Such as object or string operations, or comparators, or conditional evaluation, a.k.a. f = g ? x ? y.</p><p>The most important piece is that each function consists of exactly one statement. Because if the function originally consisted of several statements, the transpiler can trivially decompose it into the trivial operations before sending it further to the low-level rules application engine.</p><p>Add functional operations, such as filter, map, and reduce, into the mix. Complement them with several ways to call the function — by its immediate name, or as returned from another function, and with explicitly provided list of arguments or on an array of arguments — and the language becomes Turing-complete, at least to the best of my understanding.</p><p>For the type system, I was lazy enough (and quite short on time, to be honest), and have simply adopted what JSON has to offer: numbers, strings, booleans, null-s, arrays, or objects, which are maps from strings to anything valid. Add type “function” into the list, and it becomes complete.</p><p>Last but not least: since there are is no global state and no mutable variables, given the same arguments the same function would evaluate to the same result.</p><p>So, the very low-level engine is a lazy evaluation of single function with a fixed input. Yes, this simple.</p><p>With the above being said, here are the real questions:</p><ol><li>Did I miss anything obvious?</li><li>Was this done well enough before, so that I don’t re-invent the wheel?</li><li>Why isn’t this idea more widespread?</li></ol><p>The latter point requires elaboration. Part of me is a systems engineer. Part of me knows how to build high-performance systems. The above design, to me, is screaming for a JIT implementation. And, unlike most “imperative” JITs (Java, JavaScript, Lua), this one is very easy to build in a way that is constantly trying to evaluate the same piece of code in multiple different ways, picking the best-performing one on the fly.</p><p>In other words, the above idea can be compared to WebAssembly, but for Haskell, not for JavaScript.</p><p>How to solve the above better would be an interesting conversation to have.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5ad8eea2acc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Ethics of Applying ML/AI]]></title>
            <link>https://medium.com/@Dima/on-ethics-of-applying-ml-ai-ea588432016a?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea588432016a</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Wed, 15 Nov 2017 19:46:18 GMT</pubDate>
            <atom:updated>2017-11-17T08:54:48.314Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E2a9B4b3CAK3988pOOQbZQ.jpeg" /><figcaption><a href="https://en.wikipedia.org/wiki/Ex_Machina_(film)">https://en.wikipedia.org/wiki/Ex_Machina_(film)</a></figcaption></figure><p>There generally are two good approaches to building data-driven solutions, with a bad one in between.</p><p><strong>Good approach #1</strong>: Formulate the problem along with measurable metrics, then employ machine learning, and be open-minded at interpreting and acting upon what that machine learning has discovered, however controversial or contradictory it is.</p><p><strong>Good approach #2</strong>: Decide up front that the problem is too socially-sensitive, and do not use machine learning to solve it. Collect data, involve humans, and have a human, or a team, responsible for each decision it makes. Use data science in the back office to make sure humans are not making obvious blunders, but don’t mix the two.</p><p><strong>Bad approach</strong>: Claim AI / ML is used, train and validate some models, and then, upon receiving the results that are different from what were expected, tweak the way those results are used.</p><p>Generally speaking, building ML- and AI-based software <strong>reduces</strong> the risk of the developer / product manager / founder to incorporate their own biases into the product.</p><p><strong>TL;DR</strong> how to do it right:</p><ol><li>If you decide to use AI/ML, be honest about it from day one, and do so openly and transparently.</li><li>If the AI/ML solution misbehaves, <strong>don’t</strong> try to tweak its outputs to have it “conform” to some “standards” you’d like to hold it to. Instead, admit that the original goal set for the ML/AI did not account for certain types of skewed solutions, adjust the objective function, re-train, iterate, repeat.</li></ol><p>A human’s knowledge of “outcome Y is better than outcome X” is worthless compared to the humans’ agreement on “for problem P, formulation B is better than formulation A”.</p><p>If you used ML/AI, happened to not like the outcome X, and are ready to justify Y is a better outcome, don’t argue about Y being better than X.</p><p>Rather, if X is a machine-discovered solution to problem postulated as A, find an alternate formulation, B, that yields Y or something closer to Y than to X, and then argue B, not A, is the right problem to solve.</p><p>An attempt to apply human judgement in the realm of the output of the ML/AI solution is a poor way to start an argument. When it comes to finding the best solution given the set of well-formulated constraints, the machine is better than us; and that is to be seen as the ultimate blessing, not as a source of trouble.</p><p>In order to preserve and multiply the value that ML&amp;AI brings into our, human, world, humans are best to apply their intuition and expertise at the problem statement level. This is where morals, biases, and other constraints can and should come into play. This is the level where we can have a deep, thoughtful conversation about what the right direction is, and what is to be avoided as morally or socially unacceptable.</p><p>Then, keep tweaking the constraints until the machine-generated solution fits all of them and doesn’t make eyebrows rise. And be ready to embrace the solution, no matter what your inner voice — with its own imperfections and biases — will tell you about how unconventional might it be.</p><p>May the data be with you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea588432016a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Incremental Improvements]]></title>
            <link>https://medium.com/dima-korolev/incremental-improvements-78d846c7e2f9?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/78d846c7e2f9</guid>
            <category><![CDATA[history]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Sun, 16 Apr 2017 21:16:35 GMT</pubDate>
            <atom:updated>2017-04-16T21:16:35.357Z</atom:updated>
            <content:encoded><![CDATA[<p>On this cozy planet, at some point long, long time ago, a wheel was discovered:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*EEpefQPcx8nxGvcF.jpg" /></figure><p>It was part of, or, as some may say, it gave birth to an ever accelerating technological progress, which, in mechanical engineering, has arrived to the bearing …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*2MbD4D5o_fLYwwxYD6rJZw.png" /></figure><p>.. and then, relatively quickly, upgraded the steam engine to a full-blown internal combustion engine, which we still use plenty:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*w_-JbQmjuxcAUVux59aV5g.jpeg" /></figure><p>In today’s world, circumstances must be rather strange for it to make sense to utilize the <em>roue primitive</em> again.</p><p>At the same time, we can’t argue that if a well-educated modern engineer finds themselves a few thousand years back in time, the very best they could give the ancient us would be just that: the wheel.</p><p>The idea of substituting sliding friction by rolling resistance, to be more precise. But it won’t change much in their primitive minds.</p><p>Improving the efficiency at which goods can be transported from point A to point B is something that can be shown by example, internalized in our antecedents’ reptile brains, and tremendously improve the pace at which technological progress happens.</p><p>Humans of that epoch lacked the knowledge of mathematics and physics necessary to adopt something even a tiny bit more sophisticated.</p><p>Hence, if, for a thought experiment, we assume a wheel has been introduced to our species by some superintelligent being, we may as well assume this being possessed the knowledge of not only the wheel, but also the bearing, the steam engine, the internal combustion engine, the transistor, the CPU, space travel, and many more concepts we are soon to be aware of, or, maybe, are destined to never witness.</p><p>If, in this hypothetical thought experiment, there was no means for the human race to arrive to the idea of the wheel by ourselves, I, for one, would be thankful to the imaginary superintelligent being for gifting it to us.</p><p><em>… I see human history as a long period of complete failure. Failure that is to make any progress. Our species has existed, depending on where you count it from, maybe 50 thousand years, maybe 100, 200 thousand years, but anyway, the vast majority of that time people we alive, they were thinking, they were suffering, they wanted things, nothing ever improved. Or… the slow improvements that did happen, happened so that geologists can’t distinguish the difference between the artifacts of one era to another with a resolution of, like ten thousand years. So from the point of view of a human lifetime, nothing ever improved. And generation upon generation upon generation of suffering and stasis. Then there was a slow improvement, and then a more rapid improvement, and there were several attempts to institutionalize a tradition of criticism, which I think is the key to rapid progress in the sense that we think of it. Progress discernible on the timescale of a human lifetime. And also error correction, so that regression is less likely. Ah, that happened several time and failed every time except once: in the European Enlightenment of the 17th-18th centuries …</em></p><p>— David Deutsch, on a <a href="https://www.samharris.org/podcast/item/surviving-the-cosmos">podcast</a> with Sam Harris.</p><p>Similarly, a well-educated modern philosopher would not be able to teach ancient humans much. And we won’t blame neither this philosopher nor those ancient us. The development of our species as a civilization, with respect to speech, script, logic, or moral judgement to name a few, was still a few millennia behind.</p><p>What a modern philosopher could do is introduce something that can be shown by example, internalized by our antecedents’ reptile brains, and tremendously improve the pace at which moral progress happens.</p><p>One such idea, or, perhaps, the best of such ideas, could be the concept of treating each other as Human Beings. In modern mathematical terms, this would be the idea of treating our civilization as a positive-sum game, not as a zero- or negative-sum one.</p><p>But from the standpoint of the ancient humanity it doesn’t really matter which words of which language do you use to describe the idea. What matters is that it got widely adopted, and then internalized through generations.</p><p>Now replace the wheel by Christianity, and you’ll get a point of view which, if you ask me, is pretty damn healthy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78d846c7e2f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dima-korolev/incremental-improvements-78d846c7e2f9">Incremental Improvements</a> was originally published in <a href="https://medium.com/dima-korolev">Live Long and Prosper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Current for Realtime Machine Learning]]></title>
            <link>https://medium.com/dima-korolev/current-for-realtime-machine-learning-4f04aa8ab81a?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f04aa8ab81a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Thu, 13 Apr 2017 19:54:20 GMT</pubDate>
            <atom:updated>2017-04-14T08:17:31.426Z</atom:updated>
            <content:encoded><![CDATA[<p>Since 2014, I am working on Current: The C++ framework for realtime machine learning.</p><p>Realtime adaptive behavior is the theme for backends these days. With Current, we make it easy to add AI capabilities to existing APIs. Training and applying models in early 21st century should be as simple as sending and receiving HTTP headers is since late 20th.</p><p>Any bit of data is something machine learning models will need for realtime decision making. This philosophy is the cornerstone of Current. As time went by, it became clear this vision has been the right one from day one.</p><p>To enable this on the infrastructure level, every piece of data belongs to a strongly-typed immutable event log. This event log is the single source of truth for both the event itself and the domain it contributes to.</p><p>The above approach scales naturally to the application-wide persistent publish-subscribe bus. Materialized views emerge where structured tables used to be. Isolation primitives are transactions, which in Current are native lambda functions. Following the actor model design, transactions can be replayed as part of stream replication. Atop this, we added out-of-the-box RESTful APIs, CQS support, and a convex optimization engine that uses the same code to train and apply models.</p><p>Natively, Current produces binaries that can hold thousands of HTTP QPS end-to-end off a single CPU. If someone asks whether your binary is a database or an application server, the answer will be an unambiguous “Yes”.</p><p>Think of a strongly-typed Kafka and Redis running as part of your code, add zero-downtime follower-to-master flip capabilities, blend in per-query model application, and, chances are, you’ll imagine almost exactly what Current had implemented and battle-tested as of a year ago.</p><p>Running Current in production for over a year, we have strengthened our belief the MVP we built is exactly the convex hull of what it takes to prove Current’s mission. Today, we are standing on the shoulder of this giant: the framework makes bringing machine learning into application APIs as simple it is today to configure Jenkins, plug in a Slack bot, or enabling web analytics reporting.</p><p>Our approach to realtime machine learning deserves more attention, and that’s what this post is about. While designing the system from scratch in C++ is a paradigm shift for many, training and applying models is the piece that can be used out of the box by any junior C++ developer, with no commitment to ship Current to the backend stack, and, I keep my fingers crossed, without ever having to reach out to us for help.</p><p>Credentials first. I am a data engineer. Software engineering and data infrastructure are my strengths; my passion, however, is working with models, sometimes training dozens a day, validating them offline and online, and, ultimately, shipping the best ones.</p><p>To summarize my broad experience, the large and yet unbridged gap in machine learning is the language barrier between data scientists and software engineers. Yes, the one you’ll hear of on any data science meetup. That’s exactly the field where I’m making a living for the past several years.</p><p>Data scientists are proficient at understanding the underlying data and building models atop it. And, way more often than not, we build models using the tools that do not map well to what can be launched in production. More importantly, a lion share of the work of the data scientist is massaging the data, cleaning up the features, creating the new ones, sometimes pruning rows or columns, normalizing and calibrating input, etc. Needless to say, the result of the above, while often impressive when presented on a big screen from a PowerPoint slide, is generally not ready to be shipped live, at least by the standards developers would require it to be.</p><p>Engineers, and, especially, DevOps, look at the result of the data scientists’ work from a different perspective. First, speaking of realtime machine learning, we need the model to be applied at the query time. Which means along with the carefully trained model parameters vector we need the piece of code to apply this model; the code that’s both performant to serve the traffic and robust to not crash if some division by zero or a square root of a negative occurs. Then, the model should cover all possible input cases: including the rare ones data scientists have dismissed as noise, and including the yet unseen ones as long as they fit the broader set of constraints assumed. On top of the above, we need monitoring tools: if suddenly our model misbehaves, we need to know about it ASAP. Last, but not least: we need a big red button, a downgrade switch to quickly revert back to the old model, be it just the parameters vector, or a different piece of code altogether.</p><p>Sometimes a requirement as simple as “a percentile feature” is the blocker; in a good organization where scientists work closely with engineers it would be caught quickly. Sometimes that one “not-too-but-still-quite important” feature is taken from a different data source, making it impossible to access directly at the query time. Sometimes we discover the model doesn’t perform as well because somehow what should only have been part of the test set has penetrated into the training one.</p><p>A large part of our motivation with Current has been to make sure data scientists and engineers speak the same language.</p><p>To accomplish this goal, this language is C++. Don’t panic, it’s a very small subset of the language, which reads just like JavaScript or Objective C. And even if the only language you speak is Python, x[10] * (1.0 / (1.0 + exp(-x[15])) can hardly be unintuitive.</p><p>We get a huge win right away as the result of this move: the code to compute features on the training data during the manual, batch, training phase, is the same code that runs in production at thousands of queries per second. The very semantics of the process ensures the code a data scientist writes is by itself shippable.</p><p>Not a bit less importantly, as we can use the whole power of C++, Current includes the complete implementation of the module to perform analytical differentiation as part of the convex optimization engine. In plain English it means: as long as you can make your cost function relatively low-parametric and relatively convex, you only need to implement this very function (well, and define the starting point), and the machine will do everything else for you.</p><p>For the demo, let’s take a look at the <a href="https://en.wikipedia.org/wiki/Iris_flower_data_set">Iris Flowers dataset</a>. Among many reasons to choose it are:</p><ul><li><strong>It’s low-dimensional</strong>: just four variables makes the data easy to <a href="https://en.wikipedia.org/wiki/Iris_flower_data_set#/media/File:Iris_dataset_scatterplot.svg">visualize</a>.</li><li><strong>It is dense</strong>: there are no missing values to take care of.</li><li><strong>It is calibrated</strong>: possible laymen assumptions about what Euclidian distance could mean will pass the reality check.</li><li><strong>It is labeled</strong>: classification problems are perfect for simple demonstrations.</li><li><strong>It is balanced</strong>: of three classes, there are exactly 50 examples of each.</li></ul><p>Nuff said. Here is the data plotted by the Current-based <a href="https://github.com/C5T/Current/blob/8f5c8dcdc3bb7e86df5119276d2f7670d3d5bbda/examples/Iris/step2_render.cc#L71-L91">code</a> itself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HWyse8zS5VSB140s." /></figure><p>First, let’s train a descriptive model finding an ellipsoid (with axes parallel to the four dimensions) that best captures each of three classes. Yes, there is a closed-form solution for this, but look now neat the <a href="https://github.com/C5T/Current/blob/8f5c8dcdc3bb7e86df5119276d2f7670d3d5bbda/examples/Iris/step3_optimize.cc#L158-L165">cost function</a> and the <a href="https://github.com/C5T/Current/blob/8f5c8dcdc3bb7e86df5119276d2f7670d3d5bbda/examples/Iris/step3_optimize.cc#L325-L341">optimization code</a> is. Here’s the 4D visualization of the above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T0XmseFzrV4tFZPK." /></figure><p>Ultimately, in the real world, we want to know how well the model performs the task of telling cases of one class from the cases of another. To visualize this, let’s examine the histogram of scores. The model, the descriptive one so far, outputs the probability of whether an example of class X is predicted to belong to class Y. For X == Y, the predicted class being the true class, we would expect to see the probabilities equal or very close to one; similarly, for X != Y the resulting probabilities should be zero or very close to zero.</p><p>Three possible values for X and Y make for a good 3x3 matrix of histograms. Model outputs are probabilities, so each histogram plots the frequencies of values from zero to one. Rows are the true labels of the cases/flowers, and the columns are the models applied. These two <a href="https://github.com/C5T/Current/blob/8f5c8dcdc3bb7e86df5119276d2f7670d3d5bbda/examples/Iris/step3_optimize.cc#L168-L231">pieces</a> of <a href="https://github.com/C5T/Current/blob/8f5c8dcdc3bb7e86df5119276d2f7670d3d5bbda/examples/Iris/step3_optimize.cc#L513-L544">code</a> produce the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*oUdz8XrfQwyuFwGa." /></figure><p>In the perfect world, on the main diagonal there would be three histograms which only output the scores of 1.0, a single bar on the very right. The other six histograms, in this perfect world, would show all 0.0, a single bar on the very left.</p><p>The world we live in is perfect for Setosa, the top row and the leftmost column. The Setosa flowers are easy to tell from the others. Among the other two, however, the Versicolor ones and the Virginica ones, there is indeed some room for confusion, which accidental small bars here and there show. Note how quite a few non-perfect scores are present in either direction (false positives and false negatives), and the absolute values of sub-perfect model outputs span the whole probabilities range.</p><p>In fact, when evaluating the descriptive model by the classification accuracy metric, its performance is 0.84. This is the value of the log loss objective, normalized back as the probability.</p><p>or the readers who are not data scientists:</p><ul><li>There are three ellipsoids, one per class.</li><li>Each ellipsoid is treated as a <a href="https://en.wikipedia.org/wiki/Multivariate_normal_distribution">multidimensional Gaussian</a>, with the axes parallel to the coordinate axes, for the sake of simplicity.</li></ul><p>When “grading” a new datapoint:</p><ul><li>The “coordinates” of this datapoint, “x”, the four-dimensional vector, are fed into the three Gaussians. If classes are A, B, and C, the three outputs are f_A(x), f_B(x), and f_C(x).</li><li>All three are non-negative, as they are the the values of the probability density function.</li><li>The model output, which is the final probability, is assumed to be f_T(x) / (f_A(x) + f_B(x) + f_C(x)), where T is the “truth”: the class to which this datapoint does belong.</li><li>Note that this way the output for each datapoint is by definition a three-dimensional simplex; a probability distribution over the three classes, or, in other words, three non-negative numbers that sum up to one.</li><li>Practically speaking, as Gaussians fade away quickly, for many points in our space the output would be very close to 1.0, as two of the three values of f_X() will be almost indistinguishable from zero.</li><li>The log-loss cost function is the sum of the logarithms of model outputs for the “true” class across all the points. The loss is to be minimized.</li></ul><p>To normalize this cost function to a human-readable number, note that:</p><ul><li>The number of datapoints is a constant, so optimizing the sum of the logarithms is the same as optimizing the average value of the logarithm.</li><li>The exponentiation of the average value of the logarithms of a series of numbers is their geometric mean.</li><li>Thus, the final figure, which in our case is 0.84, is the “average” probability our model would assign to the true class.</li></ul><p>Now, from Current’s standpoint, computing the cost function and optimizing it are just two different interpretations of the very same code.</p><p>Given we have the code to compute the log loss, let’s optimize for it directly, and see how better can the model get on the final accuracy metric.</p><p>The resulting ellipsoids will look as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cYWm2DJJFAxTrJUC." /></figure><p>The ellipsoids no longer “attempt” to cover the points precisely. <strong>It is because the training process of the discriminant model is not blindly trying to be close to the points of the respective class, but is aiming towards capturing the boundary between the classes.</strong> The red, Setosa, labels, have hardly changed, while the Versicolor and Virginica ones moved by quite a bit, improving the classification accuracy.</p><p>The “confusion matrix” of score histograms now is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*zhTfa5Wv9vgVtR72." /></figure><p>As we can see, there are much fewer errors, as the histograms are much closer to what they should be: “just the bar on the right” along the main diagonal, and “just the bar on the left” elsewhere. Note also that the errors, while present, are not as bad — keep in mind that for practical purposes, especially in the simplest case where the output is just the “predicted” flower type, the largest “score” would win. Simply put, if the output probabilities are 0.4, 0.3, 0.3, the 0.4 one, while being far from 1.0, would still result in no classification mistake, assuming it comes from the true class.</p><p>The final classification “accuracy”, which, for the purposes of this post, is the average log-loss exponentiated to look as a zero-to-one “probability”, is 0.96.</p><p>This is to be compared to the 0.84 value which is the “accuracy” of the descriptive model applied to solve the task of classification.</p><p>As the training process is iterative, it is also interesting to examine the training curves:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*YlKbeEpt1iSs_e1T." /></figure><p>The blue curve is the descriptive classification phase. It asymptotically flats at 0.84, the value we have seen before. This phase of the training can be optimized away, as there exists a closed-form solution for fitting a multi-dimensional Gaussian atop a set of points. Without employing the closed-form solution, the descriptive phase of learning takes about 30 iterations to converge, the last half of which can safely be skipped.</p><p>The red curve is the discriminant learning phase. It takes a little over two hundred iterations, and reaches the accuracy of around 0.96. In practice, as it can be seen from this plot, when starting the discriminant training from the point to where the descriptive one has got us to, it doesn’t take more than another twenty iterations to reach a flat at the final 0.96 value.</p><p>The code behind this post is production-ready.</p><p>If your exercise of the day is to build a service that can categorize four-parameter Irises into one of the three classes, follow the links to get just that: a 99%-complete example. An HTTP endpoint to receive the four coordinates as URL parameters of as a JSON body of a request, apply the resulting model, and return the best guess and/or the per-class probabilities, would just be a few dozen lines of code to add.</p><p>It would also serve a few thousand QPS, from a single CPU core.</p><p>If you have a machine with a C++ compiler — which, really, is any Mac or Linux today — building the binaries that spin up local web servers rendering the pictures above would not take you more than ten minutes. <a href="https://github.com/C5T/Current/tree/8f5c8dcdc3bb7e86df5119276d2f7670d3d5bbda/examples/Iris">Current/examples/Iris</a> is the link. Do not hesitate to reach out if you have any questions.</p><p>To recap, today we have learned:</p><ul><li>A huge part of convex optimization problems can be automated away.</li><li>Training and applying machine learning models can be done in the same language.</li><li>If this language is C++ and the framework is Current, after going through the exercise of training the model, the production-ready service that can serve thousands of QPS over HTTP in plain JSON out of the box comes, well, for free.</li></ul><p>Some day soon I will write about Current’s publish-subscribe capabilities, and about our RESTful storage with CQS support, to truly make the above toy example the AI-friendly blend of the database and the application server.</p><p>For more back story, check out these <a href="http://dima.ai/static/current.pdf">slides</a> from 2015 and this <a href="https://dimakorolev.quora.com/TailProduce-Design-Doc">design doc</a> from 2014.</p><p>Thanks,<br>Dima</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f04aa8ab81a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dima-korolev/current-for-realtime-machine-learning-4f04aa8ab81a">Current for Realtime Machine Learning</a> was originally published in <a href="https://medium.com/dima-korolev">Live Long and Prosper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Talk at 42]]></title>
            <link>https://medium.com/@Dima/talk-at-42-6dbc69c3c507?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dbc69c3c507</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ecole-42]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Fri, 13 Jan 2017 04:00:32 GMT</pubDate>
            <atom:updated>2017-02-02T08:09:18.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uIoZ-6-PpG-I2PBU.jpg" /></figure><p>Yesterday I had the privilege to speak at <a href="https://www.42.us.org/">42</a>.</p><p>Hello, and thank you for having me.</p><p>First and foremost, let me express my deepest appreciation to Forty Two for boldly shifting the focus of education in IT back to where it should have always been: the fundamentals of software engineering.</p><p>In late 2016, Oliver Belanger and Charles Summers gave me the opportunity to explore how the organization operates at heart. Seeing the approach from the inside only strengthened my belief in your mission. I am grateful for the chance to talk to you today, and please kindly let me say a few words in support of Forty Two’s vision.</p><p>It is becoming increasingly apparent that we live in the world of power law. The power law assumes exponential distribution of the ratio of impact to effort, with its most famous formulation being the Pareto principle: twenty percent of people drink eighty percent of champagne. In IT, the state of the art for those who fearlessly go to explore computer science and software engineering is very different from the “market” of kickstarting one’s career by landing a job as a developer. Having spent the past ten years in the industry, I myself can very well feel it requires more and more determination to stay sane in today’s polarized world, where it may seem like every other door in San Francisco is either a shop hiring bootcamp graduates, or a bootcamp itself.</p><p>Good news is: Tech is swinging back to where it came from. Having the solid foundation of the basics of how computers do what they do is cool again. And Forty Two embodies it perfectly.</p><p>I interviewed hundreds of developers in my life. Unless one’s goal really is to get an okay offer to accept as quickly as possible, there is evidence to back the claim bootcamps and hackathons do everything but the most important part: explain what computer programming really is about. And while online courses chime in to fill the gap, the desire to be mentored by seasoned engineers is, from what I can see, at all time high. Today, most junior candidates appear to cite being mentored by industry experts as the number one trait they want to see in the team they look forward to joining.</p><p>And while it does give me immense pleasure to see aspiring younger generation’s desire to understand computers the way they are, I think it is also a worrisome wake-up call.</p><p>Today, the technology stack is changing faster than ever. But, even within the IT, keeping the power law in mind, many seem to be in denial about it. Somewhat paradoxically, in many work environments neither tools, nor processes, nor mindsets change quickly enough to catch up with the pace of technology. Given these environments — often times large enterprises — happen to employ a lion share of software engineering workforce, it is indeed easy for one to look around and believe their methods and skills are up to date and will remain in demand.</p><p>But it is not the case, as one quickly discovers looking outside their comfort zone, especially considering projects or companies of even slightly different nature. Technology waves in software engineering are too powerful for us to only react retroactively. If you’re into web frontends, React is the way to go, even if you don’t necessarily need all its features. While PHP and Apache are here to stay for quite some time, most people won’t consider them for a project they are starting today. If you’re into data analytics, you will likely be using a streaming no-SQL database, or even a graph database with materialized views, although the interface to query the data could still be the good old SQL. If you claim the knowledge of certain programming language, outside BigCo-s, concepts and data structures introduced in the most recent standard would likely be in heavy use in the next shop you go to. Even functional programming, of which many could say it’s still in the “slowly dragging behind” mode, is more and more important, if not when putting new features into code, then certainly in understanding modern concepts such as immutability, persisted state, or actor model data crunching.</p><p>The fundamentals always stay the same though. I still use Linux, grep, vim, and make. My bash usage patterns remain the same, except for perhaps introducing `jq` into the mix. O(N) algorithms will keep beating the O(N*logN) ones for large inputs until the end of at least the Turing-machine computing times. And, yes, as a friend of mine jokes, any real-life problem can still be solved with a savvy combination of hash maps and the Monte Carlo method. Some fifteen years ago I stumbled upon a good line in job requirement (which by itself has probably been another twenty years old): “The candidate must possess solid understanding of Unix, and a good taste to not consider the above an accomplishment”. As of today, this motto does not seem to get old at all.</p><p>The approach I consider the right one and bless wholeheartedly is what Forty Two does. Find energized, hungry, and motivated people who have the mindset that makes you genuinely curious to get to the very bottom of what happens under the hood. Gather you guys under the same roof. Offer an intense crash course, combining careful guidance with frequently leaving you on your own. Promote communication. And observe the results.</p><p>The knowledge of LAMP or MEAN stacks does not survive the time test. The knowledge of the fundamentals sure does. And Forty Two is invested into making all of you learn the latter.</p><p>I am a firm believer in substance over form. In software engineering, it so happens that the substance is virtually always right at our fingertips, often just a few taps away. To me, as well as to the best engineers I know, it’s an embarrassment to not look into something that seems to behave unpredictably; and, trust me, we do it all the times. Yet, more and more people seem to struggle to name a Unix command line tool that would use DFS, BFS, or topological sort. And, while legitimately understanding complexity, I often get a blank stare asking how long would bubble-sorting a million integers take on a modern MacBook. It is a dangerous trend, and I’m very optimistic Forty Two is on track to reverse it.</p><p>On a closing note, I do see the world making the full circle once again. In the late 20th century, the software industry has been dominated by nerds with values close to the ones of Sheldon Cooper, and those nerds, yours truly somewhat included, were possessed by nothing but computers. Coincidentally, we happened to be those magicians corporations went out of their ways to hire. After that “era”, in the early 21st century, we lived through the correction of the market, when it turned out very many roles are perfectly fit for Howard Wolowitzes, and assuming “experimental” can be above or on par with “theoretical” is no longer a red flag when it comes to the industry. Unfortunately for all of us who prefer to get things done and not talk about them, the correction did go too far, and we’re now back to the brave new world where the knowledge of the substance of what computers do and how do they do it is again understood to be more important than the knowledge of the form of how to put together a simple web service or mobile app.</p><p>The future is brilliant. Cars already drive themselves. We will have augmented and virtual realities. We can no longer count by fingers the number of fields where AI has surpassed humans in cognition and perception. We can edit human DNA live, and our medical devices, of size that can get be implanted into our organs, are computationally more powerful than the machines our grandparents landed on other planets more than half a century ago.</p><p>It would be insane to not assume most of you guys will be working in or next to some of these fascinating industries. And it would as insane to believe having shallow hands-down experience with the MEAN stack could be made use of.</p><p>For the Knowledge we go.</p><p>Thank you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*DyUwCThZXi4ngyOX851IIw.png" /><figcaption><a href="https://www.42.us.org/">https://www.42.us.org/</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dbc69c3c507" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meganezia :: Deportation]]></title>
            <link>https://medium.com/@Dima/meganesia-deportation-a992e292ca10?source=rss-335654e24a38------2</link>
            <guid isPermaLink="false">https://medium.com/p/a992e292ca10</guid>
            <category><![CDATA[meganezia]]></category>
            <category><![CDATA[deportations]]></category>
            <dc:creator><![CDATA[Dima]]></dc:creator>
            <pubDate>Sun, 06 Nov 2016 22:00:05 GMT</pubDate>
            <atom:updated>2017-01-10T19:22:53.098Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*3ePWEmYvUBG4FJABs81CMA.png" /><figcaption><a href="https://media.mnn.com/assets/images/2016/11/authagraph.png">https://media.mnn.com/assets/images/2016/11/authagraph.png</a></figcaption></figure><p>Folks, we’ve been so amazed by a piece of Sci-Fi a few months ago that we translated it into English.</p><p><a href="https://meganesia.github.io/deportation/">Meganesia :: Deportation</a></p><ul><li><a href="https://meganezia.github.io/deportation/Deportation.pdf">PDF</a></li><li><a href="https://meganezia.github.io/deportation/Deportation.fb2">FB2</a></li><li><a href="https://meganezia.github.io/deportation/Deportation.epub">EPUB</a></li><li><a href="https://meganezia.github.io/deportation/Deportation.html">HTML</a></li><li><a href="https://meganezia.github.io/deportation/Deportation.txt">TXT</a></li></ul><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a992e292ca10" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>