<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by raver119 on Medium]]></title>
        <description><![CDATA[Stories by raver119 on Medium]]></description>
        <link>https://medium.com/@raver119?source=rss-be05fbfe8065------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-_NhNmtUgnQw0fKE.jpeg</url>
            <title>Stories by raver119 on Medium</title>
            <link>https://medium.com/@raver119?source=rss-be05fbfe8065------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:01:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raver119/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Terragrunt for multi-regional deployments]]></title>
            <link>https://raver119.medium.com/terragrunt-for-multi-regional-deployments-b5bdb3371a93?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5bdb3371a93</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[terragrunt]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Sat, 17 Aug 2024 08:03:54 GMT</pubDate>
            <atom:updated>2024-08-17T08:03:54.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*3hiSXH8HLi1byxQPR1pzYQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@tvick?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Taylor Vick</a> on <a href="https://unsplash.com/photos/cable-network-M5tzZtFCOfs?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Terraform is a major tool for Infrastructure-as-a-Code systems. Amazing flexibility, huge ecosystem. But it doesn’t answer few questions for large scale applications.</p><p>I.e. how to define a multi-tenant multi-regional deployments?</p><p>One of possible solutions would be a pair of Terragrunt and Terraform.</p><p>Terragrunt is a Terraform wrapper basically. In the proposed setup it will be used to represent regional deployments as dynamically generated Terraform deployments. Consider the following structure:</p><pre>.<br>├── terragrunt.hcl<br>├── ...<br>├── modules/<br>│   ├── .../<br>├── prod/<br>│   ├── customerA/<br>│   │   ├── us-east1/<br>│   │   │   ├── terragrunt.hcl<br>│   │   ├── us-west2/<br>│   │   │   ├── terragrunt.hcl<br>│   │   ├── customer.hcl<br>│   ├── customerB/<br>│   │   ├── us-central1/<br>│   │   │   ├── terragrunt.hcl<br>│   │   ├── customer.hcl<br>│   ├── .../<br></pre><p>Terraform state in this setup will be independent for each customer region deployment, since it can be defined on Terragrunt level using variables.</p><p>It might be something like this, defined in the root terragrunt.hcl file:</p><pre>terraform {<br>  backend &quot;gcs&quot; {<br>    bucket      = &quot;tfstate&quot;<br>    credentials = ${local.credentials_to_use}<br>    prefix = &quot;customers/${local.customer_id}-${local.region_id}&quot;<br>  }<br>}</pre><p>In this case credentials might be loaded from secure source. I.e. from SOPS-encrypted file. Other variables could be inferred from the HCL files higher in the stack:</p><pre>customer_vars = read_terragrunt_config(find_in_parent_folders(&quot;customer.hcl&quot;))</pre><p>The deployment process in this setup is managed via Terragrunt, i.e. like terragrunt run-all apply --terragrunt-working-directly prod/</p><p>Terragrunt will scan the directory tree, and build a flat list of deployments like:</p><pre>prod/customerA/us-east1/terragrunt.hcl<br>prod/customerA/us-west2/terragrunt.hcl<br>prod/customerB/us-central1/terragrunt.hcl</pre><p>Each of these deployments will be pre-processed by Terragrunt and executed as independent Terraform deployments in parallel. Parallelism can be configured or even disabled though.</p><p>Further reading: <a href="https://terragrunt.gruntwork.io/">Terragrunt</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5bdb3371a93" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[To own, or not to own]]></title>
            <link>https://medium.com/geekculture/to-own-or-not-to-own-831116c5f04e?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/831116c5f04e</guid>
            <category><![CDATA[hosting]]></category>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Tue, 29 Jun 2021 06:33:00 GMT</pubDate>
            <atom:updated>2021-06-29T06:33:00.470Z</atom:updated>
            <content:encoded><![CDATA[<p>Cloud computing is a major buzzword of the decade I think. Now its pretty common to see AWS or GCP experience as a requirement for a job.</p><p>Not a surprise, sure — everyone deploys to the cloud these days. Let’s take a closer look on clouds, and start with pros and cons of a public cloud deployments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0ZNoWr7n7sD5EMBb0tp3Q.jpeg" /></figure><p>Let’s try to be positive, and start with pros for a public cloud offerings. It’s going to be easy, since pros are really huge and can be seen immediately. But let’s name a few.</p><p><strong>Quick setup</strong></p><p>Once you start a project, it just takes a couple of clicks to setup the infrastructure for it, and that’s just incredible! With Docker or Kubernetes you can deploy your app in less than a minute. That’s definitely a major selling point for public clouds, since if you’re looking to buy own hardware it typically takes up to 4 weeks just to deliver your new shiny server to your DC.</p><p><strong>Cheap to start</strong></p><p>It doesn’t take a fortune to setup the infrastructure for a proof-of-concept or an MVP. Couple of CPU cores here, couple of gigabytes there — and your application runs in a real distributed environment. All you need to start is a credit card with like $10 available. Pretty impressive if compared to a typical mid-level 1U server price tag of $4500.</p><p><strong>No extra staff required</strong></p><p>If you don’t own the hardware, you also don’t have to maintain it. So you don’t have to hire all the personnel required to keep the hardware alive. And if your team is really aware of DevOps (and it’s not a someone’s title in your team) you’ll be okay.</p><p>Every coin has 2 sides at very least, so do public cloud offerings as well. However there’s a certain asymmetry: if pros can be seen right away, long before you start using them, cons sometimes aren’t that obvious.</p><p><strong>Vendor lock-in.</strong></p><p>Cheap block storage. Vendor-specific database systems. Container orchestration systems. Machine learning pipelines.</p><p>Every time you integrate certain proprietary infrastructure components — you’re also getting another chain attached to your leg. If your project heavily relies on a specific functionality to be feasible, or just amount of efforts required to migrate from a given proprietary component will be too high — you become physically bound to the platform of your choice.</p><p>Vendor lock-in is never a good thing, you know that. APIs might change. Legal policies might change. Pricing might change. Everything might change overnight, and if you’re already locked in, you’ll be bound to accept whatever changes they have for you. Or you’ll be pushed out of business.</p><p><strong>Performance per dollar.</strong></p><p>Major cloud infrastructure vendors often tend to sell virtual CPU cores. Sounds simple enough, but the hard truth is simple too: cores are not made enough. Say, Intel Xeon E5–2650 v4 one vendor offers in its “CPU optimized” VMs is a CPU introduced back in 2016. 5 years ago.</p><p>So, you’ll be paying a top dollar for a performance of a 5-years-old machine. That’s going to be a 2x worse performance if compared to the modern machine with modern CPU cores, given 15–25% year-to-year performance improvements Intel and AMD typically announce. Not to mention faster memory used in modern servers. So, if your average compute-bound API request takes 200ms in the public cloud, it can take only 100ms on the private hardware. Or, you can serve 2x more requests with the same number of cores.</p><p><strong>Platform limitations.</strong></p><p>Clouds only seem to be infinite. In fact it’s just the same boring datacenter filled with the old good rack mounted servers. And there’s finite number of servers in a datacenter. So there are physical limits on how many hardware resources one can rent. There are limits for the hardware options as well. You can easily bring whatever hardware you need in the average DC you own or rent, as long as it fits power and safety requirements, but it’s barely going to work with major cloud platforms.</p><p>P.S. Just search the web for the customers feedback for “quota increase _platform name_” to see how bad it can be once your project suddenly takes off and you urgently need additional resources.</p><p>There’s no silver bullet when it comes to infrastructure, and neither public or private clouds should be considered as one. Radically different approaches to infrastructure can be beneficial at different stages of your project. I, personally, try to keep mobility as a core requirement for whatever I’m working on. If it makes sense to move some part of the project, or the project as a whole onto private hardware — such a move must be technically possible. Or vice versa — if it’s cheaper to use public cloud — nothing in project design should prevent that. Thanks to Docker and Kubernetes, now it’s way easier to do than it was 10 or 15 years ago.</p><p>Plan for a change since day 0: design your systems in a way you’ll be able to stay mobile. Plus, even in a high-margin business costs do matter, and if migration to or from a public cloud can improve your costs structure — think about that in advance.</p><p>And another thing to keep in mind: cloud is just someone else’s computer you’re renting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=831116c5f04e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/to-own-or-not-to-own-831116c5f04e">To own, or not to own</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The technological dead-end is near?]]></title>
            <link>https://medium.com/swlh/the-technological-dead-end-is-near-30e72eeee6cc?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/30e72eeee6cc</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Sat, 20 Mar 2021 08:06:16 GMT</pubDate>
            <atom:updated>2021-03-20T08:06:16.387Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhkUzpUUEfBXSahyhbQ4bA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@11th_wave?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Eleventh Wave</a> on Unsplash</figcaption></figure><p>We use modern technologies every day. For better, mostly.</p><p>They became essential for us: we’re communicating, commuting, and collaborating by using cutting-edge technologies, consuming powers unthinkable 25 years ago: modern smartphone has peak performance like 10 times better than a supercomputer from the 1990s!</p><p>Additional computational power makes further advances possible. Think convolution neural networks adoption explosion — it happened mostly due to parallel computation framework &amp; devices provided by nVidia.</p><p>That, in its turn, spiked ultra-wide adoption of other machine learning algorithms, making products that were never possible before: real time speech-to-text conversion, real time translation, contextual search, etc. So yea, modern technology is everywhere now.</p><p>With all that said, this situation has a significant drawback, which is typically ignored: while complexity is growing — the reliability suffers. And this trend has been worsening for quite some time now.</p><p>Think about “tech interactions” everyone has every day:</p><p>I’m asking Siri to start playing music. Can someone guarantee that my voice command will be recognized properly? Nope. It might be, but it might not, as well.</p><p>Some time ago I’ve managed to hit the ground hard. Do you think fall detection on my shiny smartwatches was triggered properly? Nope, didn’t happen. But sometimes it gets triggered when I pet my dog.</p><p>Don’t get me wrong, I’m not saying neural networks are bad. The problem is way deeper, and has been with us for a long time!</p><p>When I play an online game, maybe there’s someone who can guarantee my next TCP packet will be delivered within a certain timeframe? No way, we have a “jitter” word for that.</p><p>Maybe, if I store a bit into your RAM, there’s a guarantee I’ll read the same value later? Haha. Bit flipping is a thing.</p><p>Our technologies these days are probabilistic, at best. Probabilistic by nature sometimes, and quite often probabilistic by design. Maybe things will work properly on the first attempt, but if not — well, try again, and hope for success. If you want more reliability — i.e. you’re building aircraft or space systems — use complex ensembles of systems coupled with sophisticated consensus algorithms, and hope for an increased chance of success. And that’s a problem waiting for a solution — you can’t stack probabilistic tech over and over again — eventually, this sandcastle will fall apart!</p><p>Human nature doesn’t help here as well: every developer knows what “tech debt” means. Every developer knows that “bugs do exist” and “edge cases” are real. All these idioms do not improve reliability, for sure.</p><p>Right now probabilistic tech causes some annoyance and jokes, at most, but I’m afraid it will become a blocker for our progress in the foreseeable future: unreliable basement of our technologies will just make further improvements too complex or way too expensive, which is the same outcome anyway: progress slowdown. I suppose we can see this happening right now if we’ll look into the silicon manufacturing process: every subsequent technological improvement significantly increases complexity and price, virtually leaving no room for competition. Without competition, the progress will be crippled to death.</p><p>It’s time to shift the paradigm before we are in the dead-end. Reliability must become front and center for engineering. Whatever your product is — make sure it works in all possible use cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30e72eeee6cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/the-technological-dead-end-is-near-30e72eeee6cc">The technological dead-end is near?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speeding up your local Docker builds]]></title>
            <link>https://raver119.medium.com/speeding-up-your-local-docker-builds-9b670a114c4c?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b670a114c4c</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Tue, 22 Sep 2020 10:21:51 GMT</pubDate>
            <atom:updated>2020-09-22T21:49:41.427Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ve6MMxFPpq4q8EpueWGTUg.jpeg" /></figure><p>Every day I build containers locally. Many of us do — Docker is a crucial tool in the developer arsenal these days. However, when you build and maintain 1–2 containers — that’s great, but when you maintain an app built from 10 containers? 15? 20? Each local build, even with cache enabled, will take a significant amount of time. Multiply that by a number of daily rebuilds — and you’ll see hours of your precious time leaking away!</p><p>Sure, there’s CI/CD tools capable of parallel building, but what about local builds? I’ve spent some time googling for a solution to this small problem, and, to my surprise, wasn’t able to find one.</p><p>So, I’ve thought I could make a useful tool for myself, plus a blog post for others :)</p><p><strong>Step 0. Let’s define the problem and requirements</strong>.</p><p>I need an app, I’ll call it Krane, that is able to build multiple Docker images in parallel. There are a few special requirements for this app:</p><ul><li>Build configuration must be persistent. I’d hate using CLI to describe 10–20–30 images. So, JSON or YAML build configuration would be good.</li><li>Krane should take care of internal dependencies: if image2 depends on image1, it will guarantee that image2 will be built only after image1 is built.</li><li>Krane should be aware of build outcome: if one of the images fails the build — all remaining images should fail too.</li><li>Krane should be 100% compatible with Minikube, which I use for local development. However, this requirement will be satisfied automatically if I just use Docker executable.</li></ul><p>Since this app needs concurrent execution, Python wouldn’t be the first tool to use for me. Golang will fit way better, I believe.</p><p><strong>Step 1. Build configuration in a file</strong>.</p><p>My new tool should be able to read build configuration from a file, passed as an argument into the app. Something like app -f configFile.yml.</p><p>Golang provides JSON support out of the box, without any external dependencies. However, since Docker/Kubernetes environment is YAML-centric, it would make sense to use YAML for build configuration as well.</p><p>Golang has no issues with YAML as well, there are multiple libraries providing YAML support. I prefer <a href="https://github.com/go-yaml/yaml">this</a> one.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03350e1d78f1d7616ce7036bd8a04716/href">https://medium.com/media/03350e1d78f1d7616ce7036bd8a04716/href</a></iframe><p>Now, since I’m able to read random YAML files, time to make sure I’m able to pass the configuration file as a CLI argument.</p><p>Golang has a builtin package for that as well: flag. Defining all the needed input arguments is really trivial:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/986c7d8446da556b3ce8d63649dff3cd/href">https://medium.com/media/986c7d8446da556b3ce8d63649dff3cd/href</a></iframe><p>Now, something like krane -f config.yml is definitely going to work :)</p><p><strong>Step 2. Find dependencies within the build task</strong>.</p><p>When you’re building a bunch of independent containers — dependencies tracking is not a problem — by definition. But if some images depend on other images within the task — it means that dependencies must be built before images that depend on them.</p><p>In other words: I have to search for internal dependencies before building anything. Thanks to Docker developers, the Dockerfile format is pretty straightforward: there’s a dedicated &lt;<strong>FROM</strong>&gt; keyword, so old good regular expressions will do the job.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14d3c19f66be3c1767f415d598b45110/href">https://medium.com/media/14d3c19f66be3c1767f415d598b45110/href</a></iframe><p>When applied to every Dockerfile in the job, it’ll get me a full map of dependencies, where the key is a container name, and the value is a slice of containers it depends on.</p><p><strong>Step 3. Organizing the build process</strong>.</p><p>The map of dependencies is good, but how can I use it to organize the build process? One of the simplest ways is to represent the build it as a sequence of sequences of independent build steps. Basically topological sort of the graph, when the outcome isn’t 1D sequence, but 2D instead, to allow parallelism.</p><p>It might sound tough, but it’s really trivial. Imagine the following algorithm:</p><ul><li>All independent containers are built first, in parallel. Let’s call it “Layer”.</li><li>Containers that depend on the previous layer are built in parallel.</li></ul><p>The last step is repeated until all containers are built.</p><p>With this approach, the executor will dispatch individual build jobs to separate goroutines on each Layer. Once all jobs dispatched, the executor will wait until all jobs are finished, before switching to the next layer</p><p><strong>Step 4. Handling the outcome</strong>.</p><p>The last requirement I have is the build state handling and transfer: if one of the jobs fails — it shouldn’t be silently swallowed. I must be aware of the problems as soon as they arise: there’s no sense waiting for the full build to finish if one of the jobs failed. So, early stopping would be a “really nice to have” feature.</p><p>Luckily, Golang has channels for communications between goroutines, so each worker will get a channel for reading build jobs, and channel for reporting. The reporting channel will be used for tracking the outcome of each build.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67c4c34d4d28528510f5f57176992e15/href">https://medium.com/media/67c4c34d4d28528510f5f57176992e15/href</a></iframe><p><strong>Final step. Comparing the apples</strong>.</p><p>It’s time to see numbers. For the performance test, I’ve mastered pretty much realistic sample deployment: 4 containers building React apps (frontend part), 2 containers building Go apps (backend part), and an ML-deployment container (almost static one). I will compare build time twice: the first run with a no-cache option, and the second run without it.</p><p>no-cache sequential build time:</p><pre>real 7m15,544s<br>user 0m5,941s<br>sys 0m7,696s</pre><p>no-cache parallel build time:</p><pre>real 2m46,595s<br>user 0m5,800s<br>sys 0m8,451s</pre><p>partially cached sequential build time:</p><pre>real 2m57,410s<br>user 0m6,304s<br>sys 0m7,320s</pre><p>partially cached parallel build time:</p><pre>real 0m17,323s<br>user 0m6,111s<br>sys 0m7,808s</pre><p>So, the relative speedup is somewhere between x2.5 and x10, which is just great for me: my typical builds are partially cached. I’ll save lots of my time using this small tool.</p><p>I hope you’ll find it useful too.</p><p>Feel free to contact me if you have any questions :) <br>As usual, the source code for this app is available <a href="https://github.com/raver119/krane">on GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b670a114c4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How hyperparameter tuning works]]></title>
            <link>https://raver119.medium.com/how-hyperparameter-tuning-works-dcdc39d346cf?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcdc39d346cf</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Sun, 26 Jul 2020 09:23:55 GMT</pubDate>
            <atom:updated>2020-09-24T17:01:17.757Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2gxQj86v8iRmo89ZVGZig.png" /><figcaption><a href="https://www.freepik.com/photos/business">Business photo created by kjpargeter — www.freepik.com</a></figcaption></figure><h3>How hyperparameters tuning works</h3><p>Typical ML model training workflow looks fairly standard these days:</p><ul><li>Prepare data for training and validation</li><li>Choose architecture for the model</li><li>Pick initial parameters that make sense</li><li>Tweak the model and its parameters until you get a result that meets your requirements</li></ul><p>Today I’d like to explore the last step: tweaking the model. There’s even a special name for this process: hyperparameters optimization, which is a relatively big field with existing tools and some competition between them.</p><p>However, if all you need is hyperparameters tuning, and you don’t need any bells or whistles on top, it’ll be fairly simple software. And still quite interesting how such a simple tool helps in daily ML tasks.</p><p>So, I’ve got an idea that writing such a framework from scratches would be good for my next Medium post, and will allow me to show how such tools work :)</p><p>First of all let me define what exactly I’m going to build: <strong>micro-framework written in Python that simplifies and speeds up optimization of certain hyperparameters of my future Keras models.</strong></p><ul><li>The framework must be written in Python</li><li>The framework must be fully compatible with Keras</li><li>The framework must be compatible with TensorFlow Dataset API</li><li>1-month+ of adoption time shouldn’t be required to start using a framework</li></ul><p>Sounds legit, but this formulation also requires me to define what’s “tuning” and what’s “certain hyperparameters”.</p><p>Each network has general architecture, say ResNet50-inspired network. That’s what defines the high-level structure of the neural network. Certain combinations of layers, specific activation functions in certain places, etc. But then, there are all other activation functions, initial parameters distributions, optimizer to use, learning rate, optional structural blocks within a neural network, and so on. So, that’s parameters that define how the neural network will learn its parameters.</p><p>Now “grid search”. This kind of means that the framework I need should accept a list of parameters, test them one by one, and tell me which parameters define the best model. So this list of parameters is a “grid”, and we’ll search for the best combination of parameters. Kind of “brute force” approach in ML/DL field.</p><p>This grid might look like the dict below:</p><pre>params = {&#39;lr&#39;: [0.001, 0.0005],<br>          &#39;loss&#39;: [&#39;mean_squared_error&#39;],<br>          &#39;act_in&#39;: [&#39;relu&#39;, &#39;swish&#39;, &#39;tanh&#39;],<br>          &#39;act_out&#39;: [&#39;tanh&#39;, &#39;sigmoid&#39;]}</pre><p>And then the framework will generate a list of all possible combinations from this structure. Frankly speaking, it’s just a nested for-loop.</p><p>From the end-user point of view the result will look like this:</p><pre>{&#39;lr&#39;: 0.001, &#39;loss&#39;: &#39;mean_squared_error&#39;, &#39;act_in&#39;: &#39;relu&#39;, &#39;act_out&#39;: &#39;tanh&#39;}<br>{&#39;lr&#39;: 0.001, &#39;loss&#39;: &#39;mean_squared_error&#39;, &#39;act_in&#39;: &#39;relu&#39;, &#39;act_out&#39;: &#39;sigmoid&#39;}<br>....<br>{&#39;lr&#39;: 0.005, &#39;loss&#39;: &#39;mean_squared_error&#39;, &#39;act_in&#39;: &#39;tanh&#39;, &#39;act_out&#39;: &#39;sigmoid&#39;}</pre><p>Once I have these combinations, I’ll be able to pass them one by one to some Python function, that generates Keras Model, with all these options applied.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d768c73074015ddff354d152c9f8cfb/href">https://medium.com/media/4d768c73074015ddff354d152c9f8cfb/href</a></iframe><p>// Data pipeline</p><p>Now, when I’m able to build all possible combinations of models it’s time to think about data pipeline. There are 2 major requirements:</p><ul><li>Each model should be trained on the same amount of data</li><li>TensorFlow/Keras expect “endless” generators.</li></ul><p>I suppose the most simple way to meet these requirements would be the use of Callable that returns Python generator+ some wrapper that will make it “endless”.</p><p>The general schema here will look like this:</p><ul><li>Python generator yields either Tuple[np.ndarray, np.ndarray] or Tuple[Dict[str, np.ndarray], Dict[str, np.ndarray]]</li><li>We need at least 1 generator: for training data.</li><li>A validation generator is optional.</li><li>Each generator made “endless”</li><li>Each “endless” generator is converted to TF DataSet</li><li>DataSets are fed into Models</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51dce221533fc52bd9ccedd2291784e8/href">https://medium.com/media/51dce221533fc52bd9ccedd2291784e8/href</a></iframe><p>// Comparing the models</p><p>To pick the best model, we need some kind of “ground truth”, something comparable. Luckily, we have various metrics that will suit different models: AUC, Accuracy, Recall, Precision, F1, and, actually, the Error. For sake of simplicity, I’ll just use the error, loss in TF terms.</p><p>How can I do that? Easy! Keras interface allows the use of callbacks for such tasks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/33f25ddc89e950ae21ab6bf82680e57e/href">https://medium.com/media/33f25ddc89e950ae21ab6bf82680e57e/href</a></iframe><p>// Tuning the model</p><p>Now, when we have all parts ready — let’s use it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a55b30fd7ba898a67580681b2a0029a7/href">https://medium.com/media/a55b30fd7ba898a67580681b2a0029a7/href</a></iframe><p>Now, when I’ll run the script, I’ll find out what are the best params for this particular model.</p><pre>Model 1 of  12...<br>...<br>Model 12 of  12...<br>Best params found: {&#39;lr&#39;: 0.0005, &#39;loss&#39;: &#39;mean_squared_error&#39;, &#39;act_in&#39;: &#39;tanh&#39;, &#39;act_out&#39;: &#39;tanh&#39;}</pre><p>Doesn’t look too complex I hope :)</p><p>Sure, there are lots of improvements possible here. I.e.:</p><ul><li>Allow data parallelism</li><li>Use multiple workers in parallel, to speed up the search process</li><li>Use some smart algorithm instead of brute force</li></ul><p>And, obviously, there are pitfalls to keep in mind:</p><ul><li>It’s very easy to get into a huge grid, where the search process will take days, months, or even years. So, you have to keep the number of possible combinations under tight control.</li><li>In real tasks, one will probably use only a tiny fraction of available data for the tuning process. So, we have to make sure that this data is really representative fraction.</li></ul><p>Thanks for reading. Feel free to contact me if you have any questions :)</p><p>Full source code is available in my GitHub repository:</p><p><a href="https://github.com/raver119/EasyTune">raver119/easytune</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcdc39d346cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Task Learning With TF.Keras]]></title>
            <link>https://medium.com/swlh/multi-task-learning-with-tf-keras-5b28dd60246e?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b28dd60246e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 18:05:55 GMT</pubDate>
            <atom:updated>2020-07-15T14:26:23.356Z</atom:updated>
            <content:encoded><![CDATA[<p>One year ago I’ve posted an <a href="https://medium.com/@raver119/lstm-and-one-basic-nlp-task-9dcfedbcb9c3">article</a> showing how to build trivial sentence breaker and tokenizer in Java with DeepLearning4J. Recently, I’ve got a need to build similar model in Python, so I’ve decided to write a follow-up post, that’ll show DeepLearning4J -&gt; Keras migration, explain some of the “DL4J vs Keras” nuances and highlight some of the issues Keras has at this moment.</p><p>Primary goal for this small project is to get model that is able to segment raw Russian/Ukrainian text into: independent sentences and independent tokens. On top of that, each individual token must get Part of Speech attribution. So, by definition it’s a multi-task learning problem: raw text as input and 3 target functions. In terms of architecture it will be a pretty trivial recurrent model, Many-to-Many setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*gYZsWp-VSEsfXr03siwOEw.png" /><figcaption>Model architecture</figcaption></figure><p><strong>Preparing the data</strong>:</p><p>Keras expects input in NTS format - [examples, timesteps, features]. Building one-hot dataset will be fairly trivial in this case: for each example in batch I’ll pick number of sentences, glue them together, and split into individual characters. These characters will be one-hot encoded and resulting tensor will be fed into neural network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bf4409514017727b6c58e119d592fe5/href">https://medium.com/media/6bf4409514017727b6c58e119d592fe5/href</a></iframe><p>So, I have 2 inputs: LTR reading,RTL reading, and exactly 3 outputs: Tokenization, Sentence breaker and Parts of Speech.</p><p><strong>Class imbalance</strong>:</p><p>Due to the design I have, there will be certain class imbalance: parts of speech distribution across the typical text isn’t uniform. Sentence breaks are even more rare events: you have exactly 1 sentence start/sentence end pair per sentence. And lots of characters between them. It’s not really possible to solve this problem with augmentation or by manually balancing datasets (since that’s natural imbalance) I had do use per-class weights.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73cc3dc2bc74be8e06232446147a1e6f/href">https://medium.com/media/73cc3dc2bc74be8e06232446147a1e6f/href</a></iframe><p>There’s a problem with this approach though: class_weight option is incompatible with tf.distribute.MirroredStrategy.</p><pre>NotImplementedError: `class_weight` is currently not supported when using tf.distribute.Strategy.</pre><p>In my case it’s not a game stopper though, since model is pretty small and I can definitely live with single GPU training, but if someone is going to train big networks on huge datasets and is going to rely on distributed training — that’s something to keep in mind.</p><p><strong>Porting masking</strong>:</p><p>In DeepLearning4J input/output masks are defined explicitly, and provided at training time as a part of DataSet/MultiDataSet. In Keras you have couple of options available: you can use Masking layer — this layer “masks” timesteps that contain only mask_value thus excluding such timesteps from gradients calculation and propagating this mask through the network. Other way would be providing mask tensor explicitly as input, and use multiplication somewhere within your model graph in order to nullify gradients of unused timesteps.</p><p>Since I’m using casual one-hot encoding, it perfectly justifies usage of Masking layer, with only one nuance: I’d also like to use Bidirectional layer to make classifications based on both LTR and RTL readings. Since we’re dealing with variable-length timeseries masks, it’s easier to to provide model with manually reversed text as second input, rather than working around issues that will arise in Keras with Bidirectional layer. Mathematically, amount of information provided to the model in this way will be equal to Bidirectional layer approach, and will cause exactly zero issues.</p><p><strong>Final model code</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7753bcbce7af1e681603d2f5ae91ac8/href">https://medium.com/media/d7753bcbce7af1e681603d2f5ae91ac8/href</a></iframe><p>After some time spent training, the model will be able to generate outputs that look like this:</p><pre>BRK: [1 0 0 0 0 0 0 0 0 0 0 0 0 0 2]<br>POS: [ 0  0  0  0  0  0  0  0 17  7  7  7  7  7 11]<br>TOK: [0 1 1 1 1 1 1 2 4 0 1 1 1 1 3]</pre><p>This allows to split raw input text into sentences and tokens, plus attribute each token with its PoS. Which is exactly what I’ve been looking for.</p><p>Thanks for reading. Feel free to reach me out, if you have any questions :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b28dd60246e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/multi-task-learning-with-tf-keras-5b28dd60246e">Multi-Task Learning With TF.Keras</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Denoising images with deep learning]]></title>
            <link>https://raver119.medium.com/denoising-images-with-deep-learning-9124d36fda0a?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/9124d36fda0a</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Thu, 02 Jul 2020 10:22:56 GMT</pubDate>
            <atom:updated>2020-07-02T10:29:32.363Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I’ve started looking into computer vision problems, and image denoising/image reconstruction is one of pretty funny problems: it has more than one viable solution. So, I’ve decided to take a stab on it, and build basic image denoiser with neural network.</p><p>Luckily, I know that U-net architectures typically serve good for this kind of tasks. U-net idea is pretty close to autoencoders: you have feature extraction part of neural network, followed by reconstruction part. And since convolution/deconvolution layers are used all the way through the network — output pixels are not independent of each others.</p><p>This is how neural network is going to look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*VHcCYGvz0poFB6AszLdBJw.png" /><figcaption>Principal schema for the neural network</figcaption></figure><p>The project itself will consist of 3 major parts that will be implemented:</p><ul><li>Build data pipeline for training and validation</li><li>Design and train neural network</li><li>Test results visually</li></ul><p>So, let’s start.</p><p><strong>Data pipeline</strong>:</p><p>Neural network I’m going to create will accept arbitrary sized noisy image as input, and will return denoised image of the same size as output. This means that I’ll need bunch of source images and I’ll need to apply some noise to these images, and that’s about it, since for this kind of tasks original (unmodified) image serves as target image.</p><p>For my own convenience, I’ve organized on-the-fly data pipeline: one big image is loaded, partitioned into bunch of 64x64 (or 128x128, or whatever) images and fed into neural network. This process will be repeated for all images I have. On-the-fly pipeline simplifies experiments: different image sizes, different strides for partitioning, different noise levels, and so on.</p><p>Here’s going this pipeline will look in Python:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d530c99ac50a25f55f96193298955469/href">https://medium.com/media/d530c99ac50a25f55f96193298955469/href</a></iframe><p>Noise is added on the fly as well: certain percentage of pixels will be filled with random “dark pixels” with size varying between 1x1 up to 2x2. Definitely not a 100% natural noise, but good enough noise for 1-evening toy problem.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/805ac19d3ce9008c015892ea700385b7/href">https://medium.com/media/805ac19d3ce9008c015892ea700385b7/href</a></iframe><p>Now, when I have 2-in-1 generator compatible with TensorFlow Datasets I can easily set it up, and use it for actual training:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/840d813a17f5f9aff03cf069b8a1ac7f/href">https://medium.com/media/840d813a17f5f9aff03cf069b8a1ac7f/href</a></iframe><p>With all data pipeline bits uncovered, it’s time to look into actual neural network.</p><p><strong>Neural network architecture</strong>:</p><p>As I’ve mentioned above, neural network for this kind of tasks is typically U-net architecture. Something one could call “Convolutional autoencoder” — I have 2-branch structure here: X branch for bigger feature maps, Y branch for smaller feature maps. Eventually they’re combined and used for final image reconstruction.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df7484c50d7d43bd6c70a986d6381c33/href">https://medium.com/media/df7484c50d7d43bd6c70a986d6381c33/href</a></iframe><p>As you can see, some arguments reference params dictionary, I’ve used this for hyperparams tuning. Here’s the example of params:</p><pre>model = build_model({&#39;lr&#39;: 0.001,<br>                     &#39;conv_activation&#39;: &#39;swish&#39;,<br>                     &#39;last_activation&#39;: &#39;tanh&#39;,<br>                     &#39;out_activation&#39;: &#39;sigmoid&#39;,<br>                     &#39;loss&#39;: &#39;mean_squared_error&#39;})</pre><p>Now, with all bits set — let&#39;s hit Run button, and approximately 4714 seconds later (using RTX 2080 Ti) one will see that epoch is over.</p><pre>76200/76200 — 4714s 62ms/step — loss: 5.0034e-04 — val_loss: 2.3591e-04 </pre><p><strong>Trying the model</strong>:</p><p>For quick experiments with trained model I’ve created minimalistic Jupyter notebook (available in the GitHub repo, link at the bottom). Jupyter is pretty good for such things: one can just edit 1 line in 1 cell, and re-run this particular cell alone — and that’s exactly what I need here for “visual validation”.</p><p>Since the model is fully-convolutional, we can feed different shapes into it. However, for me personally, it’s interesting to see how network handles different amounts of applied noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*9UsiO_0RhHXxhOdOUWtSLg.jpeg" /><figcaption>64x64 RGB, 10%, 20% and 30% noise respectively</figcaption></figure><p>I think It’s safe to say that results are not ideal (since we can see artifacts even on 10% noise level), but on the other hand — hey, it’s 100k params network, capable of denoising 256 image patches in 10–15 milliseconds! So, with some additional tuning applied, we could probably get it down to 30-50k params and 3–7 milliseconds.</p><p>Thanks for reading. Feel free to reach me out, if you have any questions :)</p><p>Link to the repo with project source code: <a href="https://github.com/raver119/denoiser">https://github.com/raver119/denoiser</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9124d36fda0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick ML model serving with Java]]></title>
            <link>https://raver119.medium.com/quick-ml-model-serving-with-java-3329bddd447d?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/3329bddd447d</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Mon, 21 Oct 2019 08:58:23 GMT</pubDate>
            <atom:updated>2019-10-21T08:58:23.859Z</atom:updated>
            <content:encoded><![CDATA[<p>The microservices architecture allows you to do something, that was not really viable, or even previously possible — you can build applications with multiple languages at once — and that’s simply amazing! But what if you are restricted to a certain tech stack and languages? Or what if all you need is MVP (minimal viable product)? Or you don’t want to build/use heavy-weight ML systems? Or you just need something to make the aforementioned MVP quickly in your given stack and language?</p><p>Let’s consider the case of an MVP for a distributed Java application that uses Apache Kafka for message passing. At some point(s) we need to apply machine learning to the incoming data, get predictions, process them, and send updated data further along the pipeline. As with my earlier posts, I’ll use <a href="https://deeplearning4j.org/">DeepLearning4J</a> framework for the code samples here.</p><p>First of all we’ll need to set up something to interact with Apache Kafka. Something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1a2454595e1fbc5740ec41add9622dc/href">https://medium.com/media/b1a2454595e1fbc5740ec41add9622dc/href</a></iframe><p>The idea is pretty trivial. We’re setting up a Kafka consumer that gets data in one Thread, executes thetransform() method to process data, and writes it back to Kafka from the other Thread. transform() is made abstract, so we can provide in-place implementations, for better readability.</p><p>Now, with Kafka set up, we can get to actual machine learning for our MVP. I see at least two approaches here:</p><p><strong>Approach 1. ML as part of the pipeline</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CpDggj0l7OFZjidqioobCQ.png" /><figcaption>Basic schema. Consumer gets House objects, and produces ProcessedHouse objects</figcaption></figure><p>If for a single data entity coming in, you have exactly one ML application to use — you might get away with a simple Kafka consumer/producer combine or Kafka Stream node. Besides obvious edge cases, like EmbeddingLayer, this approach can be scaled pretty easily to meet your throughput requirements.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dc9a41e896a0bf925c59c85176147a3/href">https://medium.com/media/3dc9a41e896a0bf925c59c85176147a3/href</a></iframe><p>The implementation is fairly obvious: we subscribe to certain Kafka topic, and as soon a message comes in — transform() method is invoked, with <strong>House</strong> object provided as argument, we can pass the featurized data to the next Kafka topic, where some another part of our application will make some use of it.</p><p>There’s a performance tweak we can do — batching: instead of processing data entries one by one, we process them in batches, using the hardware more effectively with vectorized data. This is especially important if you’re using GPUs. They typically have somewhat higher latency when executing computations, but significantly higher memory bandwidth. Therefore with bigger batches the overall system throughput will increase.</p><p>But what if you have more than one entity coming out of here? Or if you have more than a single ML model to be invoked for a given data entry, especially if models are imbalanced in terms of computational expenses? The “ML as part of the pipeline” approach won’t be efficient. Here is another more suitable approach.</p><p><strong>Approach 2. ML as standalone apps</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AYlJ0Pvg4x2irZU69fRchA.png" /><figcaption>Basic schema. ML models run in separate pods, and accessed asynchronously from thin consumer.</figcaption></figure><p>In this approach, we use something like the sidecar pattern. There’s an application that consumes data from Kafka, and sends requests to the ML models using JSON/gRPC/etc. This allows us to scale ML pods individually, as required for maximizing throughput.</p><p>First we set up a model server that will accept JSON, deserialize it into a POJO and run actual inference</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51478f4a397127d833c6ec6a23f7c4e3/href">https://medium.com/media/51478f4a397127d833c6ec6a23f7c4e3/href</a></iframe><p>Next, we set up a remote client that will be used to communicate with remote model.</p><p>If we have more than one ML model to be invoked here, we’d better use an asynchronous client and invoke the most computationally expensive model first, thus hiding at least part of computations behind execution latency.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbeee080812d6d15dbf8683aca8916f2/href">https://medium.com/media/fbeee080812d6d15dbf8683aca8916f2/href</a></iframe><p>In comparison to our original application based on Kafka nothing has changed. There is still one consumer group that reads <strong>House</strong> topic, and produces <strong>ProcessedHouse </strong>entries. But internally we use a couple of sidecar applications — one for <strong>House</strong> price prediction, and one for potential target customer group. In this way we can deploy and scale our models independently (as long as the <strong>House </strong>POJO stays the same), since all ML-related code is limited to the model serving containers without bloating your generic Java application.</p><p>Obviously, both approaches shown here can’t be considered “ultima ratio” for ML model serving and are more like ad-hoc solutions. There are lots of nuances to be covered by proper serving tool: ETL consistency, versioning system, scalability tuning for various edge cases, A/B testing etc to name a few. Some of these problems are serious enough to be worthy of dedicated posts. Stay tuned :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3329bddd447d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LSTM and One Basic NLP Task]]></title>
            <link>https://raver119.medium.com/lstm-and-one-basic-nlp-task-9dcfedbcb9c3?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dcfedbcb9c3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Mon, 07 Jan 2019 18:59:01 GMT</pubDate>
            <atom:updated>2019-01-07T18:59:01.818Z</atom:updated>
            <content:encoded><![CDATA[<p>Doesn’t matter how many NLP frameworks/libraries/models are available out there, eventually you’ll need a language model that has no .zip made publicly available by some good guys. Just because whole life is a kind of N+1 problem. So, I’ve wrote small article with basic walk-through over the process of solving such a problem in easy way.</p><p>In such situation, I think, highest chances are you’ll need sentence breaker and/or tokenizer as first step. Sometimes it might be even the only step you need to make on your own. Sure, there are universal algorithms, like whitespace tokenizer, char-level representations, sub-word embeddings etc. But what if that doesn’t work for you? What if you need some nice &amp; simple sentence breaker that’s able to distinguish the difference of point symbol used at the end of sentence, and point symbol used in abbreviation? Or tokenizer that’s able to catch specific phrase patterns? Actual requirements may vary, sure, but you get the idea: real world problem can easily have special requirements that are not covered by some awesome model you can download from the web.</p><p>Deep learning can help us here and, luckily, this kind of models are relatively easy to build &amp; train these days. For my small task sentence breaker/tokenizer had couple of size &amp; perf related requirements, so I’ve decided to start with LSTM prototype.</p><p>Disclaimer: Any framework can be used to build solution for this problem, but since that’s my article, snippets will be in <a href="https://java.com/en/">language</a>/<a href="https://deeplearning4j.org/">framework</a> of my choice. There will be no model to download etc — it’s just an overview of a problem/solution, and real solution might (and most probably <strong>will</strong>!) be different for your problem.</p><p><strong>Step 0. Architecture</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*eG1JyFccyxqVIpHHSpdTPg.png" /><figcaption>LSTM cell internals</figcaption></figure><p>Why LSTM? Because problem I’m going to solve can be easily framed as sequential prediction problem. And LSTM is rather simple, efficient for this kind of problems and has cuDNN implementation. Since in this subproject sentences will be relatively short, I won’t use optimization tricks like TBPTT, just can’t see any sense bothering with it in initial version, but in future, if this part of project will require bigger chunks of text to be processed at once — it can be added in no time.</p><p><strong>Step 1. Corpus</strong>.</p><p>From my own experience — getting training corpus for some language that has no publicly available models — is the biggest problem. But with the help of luck &amp; google — you might be able to find something suitable. I.e. PoS-tagged corpus? Even if you don’t need parts of speech, such a corpus will have tokenization applied. In worst case — you’ll have to apply tokenization manually &amp; use more of augmentation to increase corpus size.</p><p><strong>Step 2. Pipeline</strong>.</p><p>Once you have corpus ready, it’s almost time to transform it to some format suitable for training. But before that we should consider couple of things:</p><ul><li>text normalization: task-specific stuff mostly. I.e. if you don’t need numbers — replace them with something common. Or replace rare symbols with more common symbols. As I’ve already said: task-specific processes :)</li><li>augmentation: cheap way to extend your corpus. It’s often considered good idea to introduce some common typos to the corpus. Replace some words with common abbreviations, or even make use of some thesaurus vocabularies. Lots of options here.</li><li>iterators: Iterator is just a programming pattern. You organize your data as a flow of “sentences”, which are translated into a flow of “characters”, with some optional preprocessing (like mentioned above) applied on the fly.</li></ul><p>And make sure all your pipeline “improvements” do not hurt quality &amp; quantity of the target patterns you’ll be looking for in your data. I.e. if you’re going to detect medical tokens like “dichloro-6-methyl-something-3-hydrid” — make sure you don’t completely remove numbers or hyphens from your corpus.</p><p><strong>Step 3. Representation</strong>.</p><p>Since we don’t have tokenizer yet (we’re kind of building it!), we’re basically limited to sub-word representations. Easiest one is a char-level representation: we build dictionary of all characters, and feed them into neural network as one-hot encoding. Since, I’m going with LSTM, data should be represented as sequence of characters in 3D tensors: [batchSize, dictionarySize, timeSteps].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*pO6G3SHSGtbYX9DDcLSiNQ.png" /><figcaption>Basic input tensor data layout for LSTM. Order of dimensions might vary for different frameworks.</figcaption></figure><p><strong>Step 4. Model</strong>.</p><p>Model design is a tough task, tuning takes time etc, but we need something to start with, right? For the prototype I’ve decided to go with single unidirectional LSTM layer, followed by 2 independent output layers. Many-To-Many recurrent network basically. Sentence breaker output is a binary classifier, with class 0 used for regular characters, and class 1 used for sentence break. Tokenizer output uses 3 “classes”: class 0 used for regular characters, class 1 used for “token start” event, class 2 used for token end effect. Pretty trivial setup here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a204d78bb22e210ff514f581e1fcd88e/href">https://medium.com/media/a204d78bb22e210ff514f581e1fcd88e/href</a></iframe><p>So, 1 input, 1 LSTM layer, 2 outputs: one for sentence ends, other one for tokens. Sure, it’s doable in one task as well, but I’ve decided to go this way due to some “production” requirements.</p><p>The only “trick” used here is the use of weighted loss function, but please take a look at the end of next step before complaining about it.</p><blockquote>Total Parameters: 302839</blockquote><p>Pretty small network here, so it will be affordable to any kind of hardware, hopefully.</p><p><strong>Step 5. Training &amp; Evaluation</strong>.</p><p>Most simple part here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/857938d28928da0a94acaa5e53a5bbe9/href">https://medium.com/media/857938d28928da0a94acaa5e53a5bbe9/href</a></iframe><p>Some time later you’ll see something like this:</p><blockquote>AUC (Area under ROC Curve): 0.9845094309720785<br>AUPRC (Area under Precision/Recall Curve): 0.96599434700799</blockquote><p>Sure, there’s always room for improvement, but there’s a lot of articles explaining how to build better models out there.</p><p><strong>Step 6. Use in production</strong>.</p><p>This step will be different for everyone I guess. In my case — it’s pretty simple functional wrapper, that takes Java String in and:</p><ol><li>applies some minimal preprocessing</li><li>converts text to chars &amp; one-hot representation</li><li>calls model.output()</li><li>rolls over predictions to build sequence of tokens split into sentences</li><li>returns those back</li></ol><p>Once framework I’ve used will get support for native graph exports, this network may be used as part of bigger graph directly, without any Java glue, but for now I’m quite okay with this approach, since it gets me breaker/tokenizer performance I need in single core execution.</p><p>So, 6 steps later, I have something I can use for the next steps of my project. No doubts, eventually this model will be significantly improved, but for now it’s ok.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dcfedbcb9c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why blockchain isn’t a future of anything]]></title>
            <link>https://raver119.medium.com/why-blockchain-isnt-a-future-of-anything-184ffef8bfa?source=rss-be05fbfe8065------2</link>
            <guid isPermaLink="false">https://medium.com/p/184ffef8bfa</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[raver119]]></dc:creator>
            <pubDate>Sat, 18 Aug 2018 09:48:10 GMT</pubDate>
            <atom:updated>2018-08-18T09:55:46.275Z</atom:updated>
            <content:encoded><![CDATA[<p>Blockchain is simple, and simplicity makes it useful in various fields. That was the idea i believe. Now i think blockchain failed its mission in this iteration.</p><p>Reason is simple: algorithm can’t be a religion, and blockchain definitely looks like a religion now. In search of product/market fit, there were so much promises blockchain might solve this or that problem, that masses bought that. That was easy, given how big the hype was — thanks to Bitcoin prices. So people tend to believe into Bitcoin/blockchain, and spread their beliefs among communities. That’s what any religion does, regardless of actual efficiency of some ritual or technology. And usually you don’t want want your primary tool for business processes to be that kind of thing. People usually want efficient tools.</p><p>But what do we really have at the end of day?</p><ul><li>Distributed registry concept</li><li>Smart contract concept</li></ul><p>And that’s it, as far as i know. These concepts are great, no argues. But these ideas aren’t nearly enough to fulfill promises like “Project X will revolutionize everything”. Don’t get me wrong please, ideas are amazing, but there’s little to none additional benefits to the original concept of “3rd party arbiter” used for thousands years now. And both ideas basically reinvent that “arbiter”.</p><p>Additional security? Arguable. Lower costs? That’s false. Performance? I hope you’re kidding.</p><p>I’m sure there will be way more useful concepts born in this field, but probably it’ll happen in next generations. At this moment, blockchain field became too toxic: too much scam, too much hype, too much regulations are coming in, too much everything that prevents talented people from looking into this amazing tech.</p><p>So, maybe let’s talk about blockchain in 10–15 years?</p><p>P.s. Why I hadn’t ever heard anything like “ArrayList is the future of X!”? 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=184ffef8bfa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>