<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Cozannet on Medium]]></title>
        <description><![CDATA[Stories by Samuel Cozannet on Medium]]></description>
        <link>https://medium.com/@samnco?source=rss-484dbfca3cf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rlmtyzemfYP6pjPS.</url>
            <title>Stories by Samuel Cozannet on Medium</title>
            <link>https://medium.com/@samnco?source=rss-484dbfca3cf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:29:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samnco/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless Framework & Immutable Packaging: How to]]></title>
            <link>https://medium.com/@samnco/serverless-framework-immutable-packaging-how-to-9573ffdd3dd6?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9573ffdd3dd6</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Tue, 04 Aug 2020 13:30:47 GMT</pubDate>
            <atom:updated>2020-08-04T13:30:47.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IcMBx5e_aJuZyy2k5YJuiA.png" /></figure><h3>Problem Definition</h3><p>Some time ago when I started playing with the Serverless framework, a “successful” a release of a service went south (big time). <br>Every ~30 requests, lambdas would run out of memory and crash. Increasing the memory to the 3GB max just offered some breathing room, but not enough to keep the application away from a complete roll back. <br>After we rolled back to the previous version, to our surprise, we kept having the same issue. <br>We had to force the complete code package back to a previous backup we had to get back to normal.</p><p>Effectively, the code we had validated wasn’t the one that moved to production that day.</p><p>Not because the code itself was not identical, but because the build process took place at different times. A version of one of the dependencies of the AWS Python Boto3 library had been updated in the 3h window between our last release to QA and the move to production. <br>This new version was incompatible with another dependency of our code and caused a huge memory leak of 30MB per call on the lambda, exhausting our initial 1GB in about 30 calls.</p><p>Bottomline: When you run</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7274f6a9a8321598c28b0272d5fa67ce/href">https://medium.com/media/7274f6a9a8321598c28b0272d5fa67ce/href</a></iframe><p>what the service actually does is:</p><ol><li>Load its own configuration</li><li>Load the serverless.yml environment</li><li><strong>(Re)install the dependencies (python3 requirements.txt in our case) to the .serverless folder</strong></li><li>Package &amp; Ship the code (Create CF template and zip file from .serverless then ship it to S3). This depends on the stage.</li><li>Deploy the stack. Also stage dependent.</li></ol><p>Since there is a dependency download and install at every deploy, <strong>you lose immutability</strong>.</p><p>Diving into the docs, you quickly realize there is a sls package command and start to see the light, only to realize the syntax is actually sls package --stage development. Artifacts cannot be moved between stages, defeating the very purpose of the command.</p><blockquote><em>The question remains</em>: how to generate an immutable package that can safely move from development to production?</blockquote><h3>Solution</h3><p><strong>IMPORTANT NOTES</strong>:</p><ul><li>This article assumes a python application. Behavior for other languages might vary, “PRs” are welcome to present an holistic view of that problem to readers.</li><li>It uses the serverless-python-requirements which is a must-use plugin especially if you have a multi-cultural team using MacBooks and Linux OSes such as Ubuntu.</li></ul><p>The solution to this problem requires to analyze what the package actually generate, separate the immutable from the mutable, and enforce the use of the immutable between deploy stages.</p><h4>Investigating packaging by the Serverless framework</h4><p><strong><em>What is the output of </em></strong><strong><em>sls package?</em></strong></p><p>When we run</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f8ae6f6c216a51aebdc71a2517b204e/href">https://medium.com/media/5f8ae6f6c216a51aebdc71a2517b204e/href</a></iframe><p>we obtain:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a1aaf856165a9b648314b232b392426/href">https://medium.com/media/0a1aaf856165a9b648314b232b392426/href</a></iframe><p>In this folder,</p><ul><li>the 2 json files are mutable and depend on the stage.</li><li>requirements(.txt) are also eventually variable in time depending on your locking strategy for package versions</li><li>However, the zip file is actually immutable and constitute our application code ready to be deployed (including its requirements which makes the other things irrelevent. Weird.)</li></ul><p><strong>NOTE</strong>: requirements are only there for python, and if you use the plugin. You might have other equivalent stuff in there for other languages.</p><p><strong><em>Can we modify it before deploy?</em></strong></p><ul><li>If we remove the mutable assets from the folder and try to deploy we get:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7225284d638cd23e5507da2da1a29e48/href">https://medium.com/media/7225284d638cd23e5507da2da1a29e48/href</a></iframe><p>it fails dramatically with ServerlessError: No serverless-state.json file found in the package path you provided.</p><ul><li>If we remove the requirements.txt file and requirements folder which supposedly are useless (everything should be in the .zip file) we get:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45ce969b31c6c43df0dcff74588b8a28/href">https://medium.com/media/45ce969b31c6c43df0dcff74588b8a28/href</a></iframe><p>it succeeds. So it’s doing something but does not really need it at the end of the day and we can safely remove that.</p><p>At this point if we use our stack by default, we still need the zip file that contains the code, but also the stage information via the 2 json files.</p><p><strong><em>Can we use the zip file as an immutable artifact?</em></strong></p><p>The serverless.yml file lets you define artifacts you want to use prior to deployment. You can define artifacts globally at the service level, or locally at the function level. For the sake of demonstration we will take the global approach.</p><p>First of all re-generate the package from scratch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9a7bb4dc28820988f7a61f0c0192ad2/href">https://medium.com/media/c9a7bb4dc28820988f7a61f0c0192ad2/href</a></iframe><p>Now edit your serverless.yml file to add:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95951ea5247e91b76a7a6e08a781d26b/href">https://medium.com/media/95951ea5247e91b76a7a6e08a781d26b/href</a></iframe><p>Let’s generate a secondary package to see the difference with the other:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ddaa21da24c21bd6e04a521065958063/href">https://medium.com/media/ddaa21da24c21bd6e04a521065958063/href</a></iframe><p>So we can see that this time the zip file was <strong>not</strong> placed into the folder.</p><p>Now if we try to deploy without managing the package,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70cc3e856568784659f6c3c58fa879ac/href">https://medium.com/media/70cc3e856568784659f6c3c58fa879ac/href</a></iframe><p>aaand… works!</p><p>More importantly we did not have to reference the package to deploy. The deploy phase re-generated the serverless-stage.json and cloudformation-template-update-stack.json by itself, and only consumed the artifact. So yes, this is actually something that can be used in an immutable fashion.</p><p><strong><em>What files are required to deploy?</em></strong></p><p>We know that</p><ul><li>the serverless.yml file is required for any sls command</li><li>plugins must be pre-installed for the sls commands to run so package.json is also required</li><li>We don’t need anything else than the generated my-app.zip file in terms of code.</li></ul><p>Let’s check if that is enough:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c1a67889a4b0ca1c025d31c8c9d673e/href">https://medium.com/media/0c1a67889a4b0ca1c025d31c8c9d673e/href</a></iframe><p>Oups… Fail!</p><p>There we go. This app (and yours probably) is spread across several directories. From the top folder:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/480f864848ea6441d6a3a77e1d00cad6/href">https://medium.com/media/480f864848ea6441d6a3a77e1d00cad6/href</a></iframe><p>What we see with our sls deploy command is that in the absence of our src/be/../../config/sls/resource.yml file our setup fails to deploy! And that is perfectly normal. The zip file my-app.zip only contains the code of the functions, not potential add-ons to the serverless.yml file.</p><p>Let’s copy the config folder over to our serverless test folder, recreate the same environment we have in our default source tree, then try to deploy again:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f69742547ff4e320000f8f7cf5d57b5a/href">https://medium.com/media/f69742547ff4e320000f8f7cf5d57b5a/href</a></iframe><p>aaaand…. works!</p><h4><strong><em>Implementing a pipeline</em></strong></h4><p>Basically we now know that it’s possible to build once, and deploy many different stages. We just have to move around the immutable assets and potential configuration files.</p><p>Let’s take an hypothetical project with the following structure</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16e3a2c6ade76a3d175ea4f1db04abdc/href">https://medium.com/media/16e3a2c6ade76a3d175ea4f1db04abdc/href</a></iframe><p>that has 3 stages: development, qa and production.</p><p>The desired outcome is to generate an artifact, and re-use it as-is at deploy. Also if you keep the serverless.yml file with an artifact in it, it will be assumed to exist. So the first thing we need to do is change our definition of artifact in serverless.yml to something we can configure:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12e0ba8a1a7153fa7ff8357fad157fd7/href">https://medium.com/media/12e0ba8a1a7153fa7ff8357fad157fd7/href</a></iframe><p>So now, any command where you explicitly add an artifact as argument will take it. If not specified, sls will assume no artifact.</p><p>You can now follow the following sequence to generate the artifact:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d01e2a6ddda53ec7af399b4cce9737f7/href">https://medium.com/media/d01e2a6ddda53ec7af399b4cce9737f7/href</a></iframe><p>Now this (my-app.zip) is the file you want to store and distribute as an artifact. You will be able to deploy</p><ul><li><strong>if</strong> you have it: it contains the necessary files to successfully deploy regardless of the stage</li><li><strong>only if</strong> you have it: if you remove a single file from it, it will fail to deploy. It’s effectively the smallest artifact you can store (regardless of the compression method and assuming no config file is redundant)</li></ul><p>With it you can deploy to any stage, such as development</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37b74a143a524137e850a90cf7107f07/href">https://medium.com/media/37b74a143a524137e850a90cf7107f07/href</a></iframe><p>QA:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/062393e33ad39474e38a6d3f99403538/href">https://medium.com/media/062393e33ad39474e38a6d3f99403538/href</a></iframe><p>and ultimately production:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf4624eb6bb62d8f6bfabc6b210a26fa/href">https://medium.com/media/bf4624eb6bb62d8f6bfabc6b210a26fa/href</a></iframe><h3>References</h3><p>This article was largely inspired by <a href="https://medium.com/@bishwash.aryal/serverless-framework-build-immutable-package-for-ci-cd-pipeline-949b49f7df91">this post</a> that I found while initially researching how to overcome the problem. Kudos to @bishwash.aryal for sharing his findings in the first place.</p><h3>Conclusion</h3><h4>It’s mostly about immutability</h4><p>I hope this article outlines how critical it is to create and operate immutable packages across your CI/CD Pipelines.<br>The fact the major part of the article is spent explaining how to deal with artifacts in the context of the Serverless Framework should not outshine this fact.</p><p>In fact, you could very much have the same problem with Docker or any other packaging solution, and it’s very much up to you to make sure that building the target artifact happens once and only once in your pipeline.</p><h4>But it’s also about the Serverless Framework</h4><p>The Serverless Framework is the leading tool to help you operate serverless services. The fact it does not manage immutability by default and requires almost a hack to handle this basic feature is all but worrying.</p><p>On my wishlist for the framework I see this as my first request:</p><pre>$ sls --immutable --output /path/to/file.zip package</pre><p>In the meantime, some better documentation about this use case in the context of CI/CD would certainly help.</p><p><em>Originally published at </em><a href="https://gist.github.com/3061d6cc8730a86b7d9e3a404b46397f"><em>http://github.com</em></a><em>.</em></p><p>Image by <a href="https://pixabay.com/users/stux-12364/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=485222">Thanks for your Like • donations welcome</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=485222">Pixabay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9573ffdd3dd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Framework & the CloudFormation 200 resource limit: Workarounds]]></title>
            <link>https://medium.com/@samnco/serverless-framework-the-cloudformation-200-resource-limit-workarounds-c7a2081378e1?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7a2081378e1</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cloudformation]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Tue, 04 Aug 2020 09:20:53 GMT</pubDate>
            <atom:updated>2020-08-04T09:20:53.919Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifWyoYeo_p7sgh5or3TXGg.jpeg" /></figure><h3>Serverless Framework &amp; the CloudFormation 200 resource limit: workarounds</h3><p>Serverless is one of the major computing models that companies of any sizes have been adopting over the last few years. In 2019, it made the Gartner Top 10 Trends impacting Infrastructure &amp; Operations alongside AI, Edge Computing or Global Infrastructure, with a mainstream horizon between 2020 and 2022.</p><p>Serverless effectively delivers on many fancy promises it makes, such as scalability, consumption based billing, event driven model (…). <br>Being a relatively recent technology, it also brings its own hurdles, only reinforced by the fact few reference documents exist to help run Serverless applications.</p><p>This posts aims at sharing some solutions to one of the biggest problems you will encounter in your serverless journey on AWS : <strong>the dreadful 200 resource limit of CloudFormation</strong>.</p><p>It assumes that you already</p><ul><li>have some knowledge of Serverless as a computing model and a framework</li><li>are familiar with AWS and its toolbox (CLI, CloudFormation, Lambda + associated services)</li></ul><h3>Problem Definition</h3><p>If you use the Serverless framework for your AWS application, you implicitly leverage CloudFormation and are bound to <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html">the limits</a> of the service.</p><p>The most critical one (afaiu) is the 200 resource limit. Basically, in ny given CloudFormation stack, you may not declare more than</p><ul><li>200 resources such as a S3 bucket, a DynamoDB table or a Lambda function, and</li><li>60 outputs for other stacks to consume</li></ul><p>This is relatively hard to reach with non-lambda setups. It is, however, very possible in a lambda-based environment. <br>While an EC2 instance that runs a significant number of individual tasks may require 10 resources (Instance, IAM role, SG, some networking, AutoScaling Group…), a single <em>function</em> requires</p><ul><li>the Lambda Function: HelloWorldLambdaFunction</li><li>IAM Role: HelloWorldIamRoleLambdaExecution</li><li>a version: HelloWorldLambdaVersionThisIsAVeryLongId</li><li>a Log Group: HelloWorldLogGroup</li></ul><p>If this function is called via an API Gateway, you need to add</p><ul><li>the API Gateway Resource: ApiGatewayResourceHelloWorld</li><li>the execution Permission: HelloWorldLambdaPermissionApiGateway</li><li>its Methods: ApiGatewayMethodHelloWorldGet and ApiGatewayMethodHelloWorldOptions (if CORS is active)</li></ul><p>That does not account for plugins such as serverless-plugin-aws-alerts that also create CloudWatch alarms and add several objects per function, or even additional AWS resources declared directly in your serverless.yml file.</p><p>When you run sls deploy, each serverless.yml file gets packaged into a single monolithic CloudFormation template, and then deployed. As a result there is a direct relationship between the number of functions you declare and the count of CloudFormation resources.</p><p>A practical example: For a 14 Lambda service running behind an API Gateway with the metrics and alarms setup, you might have:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a09abb07b4c93df8268f146cf6d11d4a/href">https://medium.com/media/a09abb07b4c93df8268f146cf6d11d4a/href</a></iframe><p>If you add enough functions, you might reach 201 resources and deploying will result in</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd25cc1d029347a65a9a8114a66f7e8f/href">https://medium.com/media/fd25cc1d029347a65a9a8114a66f7e8f/href</a></iframe><p>That’s right, <strong>you cannot deploy anymore</strong>. So this is pretty critical.</p><blockquote>Each Lambda function requires <strong>4 to 10 resources</strong>, plus an offset of at least 10 for shared resources such as Authorizers, API Gateway Stage (…).</blockquote><blockquote>Since CloudFormation has a limit at 200 resources, you can declare <strong>a max of 27 Lambdas per stack if you run an API Service, and 48 if you don’t.</strong></blockquote><blockquote>Reaching this limit will block any further deployment of the application.</blockquote><h3>Work Arounds</h3><p>If you are serious about serverless, you <strong>will</strong> get into this situation. It’s not a <em>if</em>, it’s a <em>when</em>. Let’s see with a practical eye what solutions are available to you.</p><h4>The (sometimes) quick &amp; the (always very) dirty</h4><blockquote><strong>TL;DR</strong>: disable or degrade observability or security mechanisms to save resources<br><strong>Pros</strong>: Immediate result, simple redeploy, no architecture change<br><strong>Cons</strong>: as all trade-offs you give something away for it and grow tech debt</blockquote><p>In certain situations, you <em>need</em> to deploy and you are ready to make a lot of sacrifices to get going.</p><ul><li>If you are can live without CloudWatch Alarms (when using the serverless-plugin-aws-alerts plugin), turn that off and you’ll save at least 1 resource per Lambda, which is not too bad for a start.</li><li>You an also remove API Gateway logging via:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc55919c912923ee78cae5930cbe7945/href">https://medium.com/media/fc55919c912923ee78cae5930cbe7945/href</a></iframe><ul><li>You can degrade your CORS position by disabling it on some or all endpoints, and this will save you the Options API Gateway Method for each Lambda.</li><li>Finally, you can temporarily degrade your security position by using a single IAM role for all your Lambdas, which will save you a resource per Lambda.</li></ul><p>Everything here really must be considered as last resort options, but at least you know they exist.</p><h4>Declare IAM Roles separately</h4><blockquote><strong>TL;DR</strong>: write IAM roles separately to save a resource per lambda<br><strong>Pros</strong>: immediate result, simple redeploy, no re-architecture, scalable<br><strong>Cons</strong>: lower maintainability</blockquote><p>When declaring a function, you can either use</p><ul><li><a href="https://www.npmjs.com/package/serverless-iam-roles-per-function">a plugin</a> to declare IAM Roles per function: This will generate a new IAM resource for each lambda you create</li><li>or the default link to an IAM Role in the function definition, which allows you to SAVE one resource by removing combining with the previous section</li></ul><p>Having IAM Roles in separate documents requires developers to have a lot of discipline when they modify existing functions, especially when some permissions are removed and one forgets to update the attached role.</p><p>But again, I would not hesitate a single second if I <em>had to</em> deploy and was out of other options, or if the structure of the team and processes makes it possible.</p><h4>Remove all unnecessary assets from serverless.yml</h4><blockquote><strong>TL;DR</strong>: Update the serverless stacks to split infrastructure from application code<br><strong>Pros</strong>: fast result, no rearchitecture of the application<br><strong>Cons</strong>: blue-green required, does not scale</blockquote><p>When you start with the Serverless framework you might lean towards declaring all resources in your serverless.yml file, such as DynamoDB tables, or S3 buckets. It turns out to consume precious resources which you might be better off importing.</p><p>The framework offers multiple options to import objects from other stacks. A simple one is to declare dependencies in your custom area</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0391a5246888d6b4de3e9aaa6522d62/href">https://medium.com/media/d0391a5246888d6b4de3e9aaa6522d62/href</a></iframe><p>You can then use resources of these stacks in your functions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0518dabd9fa55444f290bd5a92f99c40/href">https://medium.com/media/0518dabd9fa55444f290bd5a92f99c40/href</a></iframe><p>By doing so, you clear up some room for what matters: functions. However, there is not necessarily a direct relation between your infrastructure size and the count of Lambdas, which means this won’t help you scale much.</p><p><strong>Note</strong>: If your poison is Terraform instead of CloudFormation, I strongly recommend <a href="https://theburningmonk.com/2019/03/making-terraform-and-serverless-framework-work-together/">this article</a> from Yan Cui (or any content he produces, everything is of great quality and very well written). It presents a simple trick to expose Terraform resources in a way that can be consumed by Serverless.</p><h4>Splitting serverless.yml</h4><blockquote><strong>TL;DR</strong>: Split the stack into smaller individual service stacks, each with their own 200 resource limit<br><strong>Pros</strong>: Great scalability, best practice, several options available<br><strong>Cons</strong>: blue-green required, significant effort, harder if not green field</blockquote><p>Since each serverless.yml file generates a single CloudFormation template, using several of them is actually the most natural way to scale.</p><p>The first low hanging fruit tactic you can apply is decoupling functions that require an API gateway from functions that are triggered by other types of events such as</p><ul><li>S3 events triggering antivirus checks</li><li>SQS/SNS Triggered Lambdas</li><li>Cron Jobs</li></ul><p>This looks trivial, and it is. But it’s also very effective…</p><p>You might still exhaust an API stack resources and need to break it down a bit further.</p><ul><li>It’s possible to point a serverless.yml file to consume an API declared in another stack. This way you can declare a “Master” stack that declares your API and overall structure, then have child stacks that add themselves to the master. There is an excellent article covering this published by GorillaStack <a href="https://www.gorillastack.com/news/splitting-your-serverless-framework-api-on-aws/">here</a>. <strong>This is by far the best and most scalable solution to the problem</strong>.</li><li>An alternative is to use several APIs, and group them under a common Custom Domain. Basically API Gateway is a CloudFront distribution which API and UI are customized to make it easy to manage APIs. When you use a Custom Domain, you can think of this as if you had a single distribution, many origins, and you create custom behaviors for each endpoint. This lowers your code maintenance overhead but requires but takes a long time to deploy the first time (40min+) and is not flexible to changes.</li><li>Of course you can combine both approaches. Custom Domains are super useful to expose an API under a meaningful URL, so don’t shy away from them because of the creation time</li></ul><h4>Using Plugins</h4><blockquote><strong>TL;DR</strong>: Benefit from others’ experience with the same problem via the plugins they wrote<br><strong>Pros</strong>: perfect if your usecase maps to theirs, low/no code — config only<br><strong>Cons</strong>: blue-green required, requires some R&amp;D to test limits of plugins, potential lock-in</blockquote><p>As you would expect, if you read this you are not the only one with the problem. Others have taken action and wrote a few plugins. As much as I am a big fan of reusing OSS components, my personal experience with these plugins is that they have been written with very specific problems in mind, and they can be difficult to adapt to your own setup, hence can generate more challenges than solutions.</p><p>Make your own mind to it, but be aware that:</p><ul><li>they will almost always require a complete redeploy. At the very least, if you have this issue, you’ll have to consider a blue-green deploy.</li><li>They are real hard to adopt if you are not in a green field situation.</li><li>Once you pick one, getting out of it might be costly as it will require a new blue-green deployment for the app.</li></ul><p><strong><em>Nested Stack</em></strong></p><p>This <a href="https://github.com/jagdish-176/serverless-nested-stack">plugin</a> converts the stack into a parent stack and 2 sub stacks:</p><ul><li>Roles &amp; logs</li><li>Functions and everything else</li></ul><p>This means each function now amounts for 5 or 6 elements, while 2 are moved towards the log stack. This increases our scale to about 35 to 40 functions.</p><p>On the cons side,</p><ul><li>We must redeploy from scratch, which can generate a downtime</li><li>The plugin can requires a rewrite of many elements of the serverless.yml file to adjust shared parameters between stacks which conflicts with the import function</li><li>up to 50% more capacity but will also fail at some point and require another batch of work to work around the 200 resource limit</li></ul><p><strong><em>Split Stack</em></strong></p><p>This <a href="https://github.com/dougmoscrop/serverless-plugin-split-stacks">plugin</a> also distributes the stack across an unlimited number of sub stacks via a configuration file. By default, it creates one stack per type of element.</p><p>This is pretty cool on paper since it essentially means we can scale indefinitely.</p><p>By default, the plugin operates a split by roles, which generates 4 stacks:</p><ul><li>The parent stack contains most of the functions</li><li>API Nested Stack contains the API resources (1 per Lambda)</li><li>Permissions Nested Stack contains the Lambda Permissions (1 per Lambda)</li><li>Versions Nested Stack contains the Lambda versions (1 per Lambda)</li></ul><p>At the time of this writing, it does:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6b7406ec24252c3a8306f2982ca27d3/href">https://medium.com/media/d6b7406ec24252c3a8306f2982ca27d3/href</a></iframe><p>The reduction is therefore 1.5x with about 50 elements moving to separate stacks. This moves 3 elements to separate stacks, thus 4 to 5 per Lambda in the main stack, allowing to grow to 45 to 50 Lambdas (by default)</p><p>This result is shadowed however by the facts that:</p><ul><li>it requires to do a blue-green deployment because many names will be re-used and can create conflicts</li><li>the root stack still has a linear relation to the number of lambdas</li></ul><p><strong><em>Custom split</em></strong></p><p>Custom Split is a plugin option that lets you define naming convention for resources, or select specific resource types to move to separate stacks.</p><p>On paper, it is possible to decouple the stacks in a custom manner:</p><pre>─ Top Level API<br> ├── /foo<br> ├── /bar<br> ├── /baz</pre><p>which is similar to splitting serverless.yml above with a single API.</p><p>In practice the plugin does not share the API Resources as the framework expects and it makes this breakdown really hard to leverage. Overall, I was not able to obtain valid results with this approach and disgarded the plugin altogether, which doesn’t mean it has not evolved and might work for you (please comment if so…)</p><p><strong><em>Additional Stacks</em></strong></p><p>A fourth <a href="https://github.com/SC5/serverless-plugin-additional-stacks">plugin</a> is available which also has the ability to orchestrate the deployment of stacks to a certain degree (before and after the main serverless stack).</p><p>This is probably the most flexible option available since it lets you organize the setup as you want.</p><p>However, if you are not greenfield, it will also require a redeployment of your application to re-wire the stacks together.</p><h4>API Gateway / Lambda Proxy</h4><p>When you define API Gateway methods, you can either define:</p><ul><li>Individual API Methods</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c72f341a3a99783bf5c3beaf6b48995/href">https://medium.com/media/9c72f341a3a99783bf5c3beaf6b48995/href</a></iframe><p>With this the API Gateway makes all routing decisions based on your explicit setup. Each { endpoint; method } couple will be wired to a single handler in your code. Said otherwise, you leverage the tooling offered by AWS in the rawer fashion available to you, at the expense of the count of resources needed (since Serverless will create AWS resources for each of the { endpoint; method } couples)</p><ul><li>Lambda-Proxy Integration:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd7ca94d939ca3c347f04131d36aad00/href">https://medium.com/media/cd7ca94d939ca3c347f04131d36aad00/href</a></iframe><p>Here you create an API resource that forwards all traffic to a single handler. That handler will have to make decisions to distribute code to sub-handlers, or can be a monolith. Instead of leveraging the tool, you write you “own” routing as you used to with traditional frameworks. Obviously, doing this will greatly reduce the number of AWS resources required.</p><p>I won’t enter the debate of which one is better than the other. I’ll just point out that some team members are better at pure code and others more agile with infrastructure. At the end of the day, this is a coding decision that depends on the team and overall development strategy.</p><h3>Conclusion</h3><p>You will get into the 200 resource limit wall one way or another.</p><p>What’s important is to be ready. With that in mind, remember wishful thinking for serverless files loses by a landslide to thoughtful approach to API design. I can only recommend reading</p><ul><li>This great presentation about <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/32713.pdf">Good API Design by Joshua Blosh</a></li><li>This awesome and practical <a href="https://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api">blog post</a> about the Enchant API</li></ul><p>Both are fantastic entries to API design that will help build a great service, and reduce the cost of facing the dreadful 200 resource limit.</p><p><em>Originally published at </em><a href="https://gist.github.com/9718fd4e729b21e43c369524e9dd9baf"><em>http://github.com</em></a><em>.</em></p><p>Image by <a href="https://pixabay.com/users/geralt-9301/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=1521348">Gerd Altmann</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=1521348">Pixabay</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7a2081378e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reading AWS SES encrypted emails with boto3]]></title>
            <link>https://medium.com/@samnco/reading-aws-ses-encrypted-emails-with-boto3-9c177f8ba130?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c177f8ba130</guid>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Fri, 02 Nov 2018 10:59:48 GMT</pubDate>
            <atom:updated>2021-10-19T14:01:45.672Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS Simple Email Service (SES) has this very neat feature that allows you to create a pipeline to process incoming emails.</p><p>Each step of the pipeline is called an Action, which can be to store the email in S3, to send an SNS notification, run a Lambda. A very simple example might be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*hMWX8NmjdJkdb6K8ZEcydA.jpeg" /><figcaption>Simple SES Email Processing Pipeline</figcaption></figure><p>When you store incoming emails in S3, you also have the option to encrypt them with an SES client-side encryption using a KMS Key.</p><p>On the bright side, resulting S3 keys will benefit from both the server-side encryption from S3, and the client-side from SES, allowing for various RBAC models on the data.</p><p>On the dark side, the documentation about decryption is… cryptic to say the least. Even Google hasn&#39;t figured out how to decrypt the data in Python, @AWS still hasn&#39;t included client-side support in the aws-encryption-sdk, and people have to combine Java and Python to get this done, the world feels empty.</p><p>There is just this one genius who actually documented code that sorts of works <a href="https://github.com/tedder/s3-client-side-encryption">here</a> about &quot;generic&quot; client-side encryption on S3. Thanks a million @tedder42 for publishing this.</p><p>Unfortunately, it does not work on emails:</p><pre>$ python get.py my-bucket incoming-emails/n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1 output.txt<br>Traceback (most recent call last):<br>  File &quot;get.py&quot;, line 63, in &lt;module&gt;<br>    decrypt_file(decrypted_envelope_key[&#39;Plaintext&#39;], dest_file, envelope_iv, int(original_size), &quot;decrypted-&quot; + dest_file)<br>  File &quot;get.py&quot;, line 29, in decrypt_file<br>    decryptor = AES.new(key, AES.MODE_CBC, iv)<br>  File &quot;/usr/lib/python2.7/dist-packages/Crypto/Cipher/AES.py&quot;, line 94, in new<br>    return AESCipher(key, *args, **kwargs)<br>  File &quot;/usr/lib/python2.7/dist-packages/Crypto/Cipher/AES.py&quot;, line 59, in __init__<br>    blockalgo.BlockAlgo.__init__(self, _AES, key, *args, **kwargs)<br>  File &quot;/usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py&quot;, line 141, in __init__<br>    self._cipher = factory.new(key, *args, **kwargs)<br>ValueError: IV must be 16 bytes long</pre><p>Looking at the code, it appears that the AES encryption used is <strong>MODE_CBC</strong>:</p><pre>...<br>def decrypt_file(key, in_filename, iv, original_size, out_filename, chunksize=16*1024):<br>    with open(in_filename, &#39;rb&#39;) as infile:<br>        decryptor = AES.new(key, AES.MODE_CBC, iv)<br>...</pre><p>And if you look at the metadata of emails encrypted by SES, you can see that the algorithm is <strong>MODE_GCM</strong> and not MODE_CBC</p><pre>$ aws s3api head-object --bucket my-bucket --key incoming-emails/n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1<br>{<br>    &quot;AcceptRanges&quot;: &quot;bytes&quot;, <br>    &quot;ContentType&quot;: &quot;application/octet-stream&quot;, <br>    &quot;LastModified&quot;: &quot;Thu, 01 Nov 2018 14:18:34 GMT&quot;, <br>    &quot;ContentLength&quot;: 38519, <br>    &quot;ETag&quot;: &quot;\&quot;a7e63f50de8711e8aaf64b406710a414\&quot;&quot;, <br>    &quot;ServerSideEncryption&quot;: &quot;AES256&quot;, <br>    &quot;Metadata&quot;: {<br>        &quot;x-amz-unencrypted-content-length&quot;: &quot;38503&quot;, <br>        &quot;x-amz-iv&quot;: &quot;bB2T1y32dE87G1e8&quot;, <br>        &quot;x-amz-cek-alg&quot;: &quot;<strong>AES/GCM/NoPadding</strong>&quot;, <br>        &quot;x-amz-wrap-alg&quot;: &quot;kms&quot;, <br>        &quot;x-amz-matdesc&quot;: &quot;{\&quot;aws:ses:message-id\&quot;:\&quot;n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1\&quot;,\&quot;aws:ses:rule-name\&quot;:\&quot;MoveToS3\&quot;,\&quot;aws:ses:source-account\&quot;:\&quot;012345678912\&quot;,\&quot;kms_cmk_id\&quot;:\&quot;arn:aws:kms:eu-west-1:012345678912:alias/aws/ses\&quot;}&quot;, <br>        &quot;x-amz-key-v2&quot;: &quot;AQIDAHghn...oraTng==&quot;, <br>        &quot;x-amz-tag-len&quot;: &quot;128&quot;<br>    }<br>}</pre><p>Here is the trick: MODE_GCM does <strong>NOT</strong> exist in the default cryptography Python library. So if you use the provided code and just replace the mode, it will still fail with</p><pre>Traceback (most recent call last):<br>  File &quot;get.py&quot;, line 63, in &lt;module&gt;<br>    decrypt_file(decrypted_envelope_key[&#39;Plaintext&#39;], dest_file, envelope_iv, int(original_size), &quot;decrypted-&quot; + dest_file)<br>  File &quot;get.py&quot;, line 29, in decrypt_file<br>    decryptor = AES.new(key, AES.MODE_GCM, iv)<br>AttributeError: &#39;module&#39; object has no attribute &#39;MODE_GCM&#39;</pre><p>However, this mode <strong>DOES </strong>exist<strong> </strong>in <a href="https://pycryptodome.readthedocs.io/en/latest/src/cipher/modern.html">PyCryptodome</a>.</p><p>Let&#39;s install the standalone library:</p><pre>$ pip install pycryptodomex</pre><p>and slightly update the <strong>get.py</strong> script to get the AES cipher from it instead of the default Crypto, then use the expected mode:</p><pre>#!/usr/bin/env python3</pre><pre># Copyright 2015, MIT license, github.com/tedder.<br># You know what the MIT license is, follow it.</pre><pre>import base64<br>import json<br><strong>from Cryptodome.Cipher import AES # pycryptodomex</strong><br>import boto3<br>import sys</pre><pre>if len(sys.argv) != 4:<br>  print(&quot;usage: get.py bucket s3_key destination_filename&quot;)<br>  sys.exit(-1)</pre><pre>bucket_name = sys.argv[1]<br>key_name = sys.argv[2]<br>dest_file = sys.argv[3]</pre><pre>def decrypt_file(key, in_filename, iv, original_size, out_filename, chunksize=16*1024):<br>    with open(in_filename, &#39;rb&#39;) as infile:<br>        decryptor = AES.new(key, <strong>AES.MODE_GCM</strong>, iv)</pre><pre>with open(out_filename, &#39;wb&#39;) as outfile:<br>            while True:<br>                chunk = infile.read(chunksize)<br>                if len(chunk) == 0:<br>                    break<br>                outfile.write(decryptor.decrypt(chunk))<br>            outfile.truncate(original_size)</pre><pre>s3 = boto3.client(&#39;s3&#39;)<br>location_info = s3.get_bucket_location(Bucket=bucket_name)<br>bucket_region = location_info[&#39;LocationConstraint&#39;]<br>object_info = s3.head_object(Bucket=bucket_name, Key=key_name)</pre><pre>metadata = object_info[&#39;Metadata&#39;]<br>material_json = object_info[&#39;Metadata&#39;][&#39;x-amz-matdesc&#39;]</pre><pre>envelope_key = base64.b64decode(metadata[&#39;x-amz-key-v2&#39;])<br>envelope_iv = base64.b64decode(metadata[&#39;x-amz-iv&#39;])<br>encrypt_ctx = json.loads(metadata[&#39;x-amz-matdesc&#39;])<br>original_size = metadata[&#39;x-amz-unencrypted-content-length&#39;]</pre><pre>kms = boto3.client(&#39;kms&#39;)<br>decrypted_envelope_key = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)</pre><pre>s3.download_file(bucket_name, key_name, dest_file)<br>decrypt_file(decrypted_envelope_key[&#39;Plaintext&#39;], dest_file, envelope_iv, int(original_size), &quot;decrypted-&quot; + dest_file)</pre><pre><strong>print(&quot;Flawless Victory!&quot;)</strong></pre><p>Suddenly it works much better:</p><pre>$ python get.py &lt;bucket&gt; prefix/n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1 output.txt<br>Flawless Victory!</pre><p>and the content of the email:</p><pre>$ cat decrypted-output.txt<br>Return-Path: &lt;<a href="/service/mailto:origin@example.com">origin@example.com</a>&gt;<br>Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [1.2.3.4])<br> by inbound-smtp.eu-west-1.amazonaws.com with SMTP id ...<br> for <a href="/service/mailto:target@acme.com">target@acme.com</a>;<br> Fri, 02 Nov 2018 10:25:11 +0000 (UTC)<br>X-SES-Spam-Verdict: PASS<br>X-SES-Virus-Verdict: PASS<br>Received-SPF: none (spfCheck: 1.2.3.4 is neither permitted nor denied by domain of example.com) client-ip=1.2.3.4; <a href="/service/mailto:envelope-from=origin@example.com">envelope-from=origin@example.com</a>; helo=mail-ed1-f54.google.com;<br>Authentication-Results: amazonses.com;<br> spf=none (spfCheck: 1.2.3.4 is neither permitted nor denied by domain of example.com) client-ip=1.2.3.4; <a href="/service/mailto:envelope-from=origin@example.com">envelope-from=origin@example.com</a>; helo=mail-ed1-f54.google.com;<br> dkim=pass <a href="/service/mailto:header.i=&lt;a href=">@example</a>-com.20150623.gappssmtp.com&quot;&gt;header.i=<a href="http://twitter.com/example">@example</a>-com.20150623.gappssmtp.com;<br> dmarc=none header.from=example.com;<br>X-SES-RECEIPT:... ; c=relaxed/simple; s=...; d=amazonses.com; t=1541154311; v=1; bh=...; h=From:To:Cc:Bcc:Subject:Date:Message-ID:MIME-Version:Content-Type:X-SES-RECEIPT;<br>Received: by mail-ed1-f54.google.com with SMTP id u12-v6so1445426eds.4<br>        for &lt;<a href="/service/mailto:target@acme.com">target@acme.com</a>&gt;; Fri, 02 Nov 2018 03:25:10 -0700 (PDT)<br>DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;<br>        d=example-com.20150623.gappssmtp.com; s=20150623;<br>        h=mime-version:from:date:message-id:subject:to;<br>        bh=...;<br>        b=...<br>X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;<br>        d=1e100.net; s=20161025;<br>        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;<br>        bh=...;<br>        b=...<br>X-Gm-Message-State: ...<br>X-Google-Smtp-Source: ...<br>X-Received: by 2002:a17:906:6c9a:: with SMTP id s26-v6mr6344779ejr.239.1541154310145;<br> Fri, 02 Nov 2018 03:25:10 -0700 (PDT)<br>MIME-Version: 1.0<br>From: Samuel Cozannet &lt;<a href="/service/mailto:origin@example.com">origin@example.com</a>&gt;<br>Date: Fri, 2 Nov 2018 11:24:59 +0100<br>Message-ID: &lt;...<a href="http://twitter.com/mail">@mail</a>.gmail.com&gt;<br>Subject: Hello World<br>To: <a href="/service/mailto:target@acme.com">target@acme.com</a><br>Content-Type: multipart/alternative; boundary=&quot;0000000000003362570579abf3c8&quot;</pre><pre>--0000000000003362570579abf3c8<br>Content-Type: text/plain; charset=&quot;UTF-8&quot;</pre><pre>I used to be an encrypted email</pre><pre>--0000000000003362570579abf3c8<br>Content-Type: text/html; charset=&quot;UTF-8&quot;</pre><pre>&lt;div dir=&quot;ltr&quot;&gt;I used to be an encrypted email&lt;/div&gt;</pre><pre>--0000000000003362570579abf3c8--</pre><p>There you go, you just successfully decrypted an email sent to an SES gateway, encrypted client-side with a KMS Key and stored to S3.</p><p>Again thanks Ted for sharing your code, it did put me on the right track! I hope this will now help others in the same situation as I was. Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c177f8ba130" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deleting Content in DynamoDB from the CLI]]></title>
            <link>https://medium.com/@samnco/deleting-content-in-dynamodb-from-the-cli-831ce5ab083c?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/831ce5ab083c</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Wed, 24 Oct 2018 15:50:00 GMT</pubDate>
            <atom:updated>2018-10-31T12:01:03.601Z</atom:updated>
            <content:encoded><![CDATA[<p>If you started developing serverless applications using AWS Lambda, chances are you also use DynamoDB as a storage backend.</p><p>If so, there are scenarios where you want to replace a table by an empty one:</p><ul><li>Load &amp; PerformanceTesting</li><li>Reset development/testing environment</li></ul><p>There are basically 2 methods to do so with their own strengths and weaknesses.</p><h3>Deletion/Creation</h3><p>A quick and dirty way to do so is to delete and recreate the table. If you want to keep the settings you had before, you will need to extract a schema matching the --generate-cli-skeleton flag.</p><p>This can be achieved with</p><pre>$ aws dynamodb describe-table \<br>    --table-name foobar | \<br>    jq &#39;.Table | del(.TableId, .TableArn, .ItemCount, .TableSizeBytes, .CreationDateTime, .TableStatus, .ProvisionedThroughput.NumberOfDecreasesToday)&#39; | \<br>    tee foobar-schema.json</pre><p>Once this is done, you can delete the table</p><pre>$ aws dynamodb delete-table --table-name foobar</pre><p>And finally recreate it with</p><pre>$ aws dynamodb create-table --cli-input-json file://foobar-schema.json</pre><p>This is perfect (and fast!) when the table does not have any streams attached. However, you might have noticed that DynamoDB stream ARNs, which you will need to process streaming data, depend on their creation date-time as in</p><pre>arn:aws:dynamodb:eu-west-1:123456789012:table/foobar/stream/2018-10-23T10:58:41.679</pre><p>Which means that if you deleted the table and recreated it, you changed the Stream ARN, thus potentially lost the connection to your lambdas. Sub optimal.</p><h3>Only deleting keys</h3><p>If you need to preserve external connectivity, or for some reason cannot tolerate to delete and recreate the table, you might want to delete all its content.</p><p>To do that cleanly, you will first need to extract the schema of the table slightly differently that we did before:</p><pre>$ export KEY_SCHEMA=&quot;$(aws dynamodb describe-table \<br>    --table-name foobar | \<br>    jq -r &#39;.Table.KeySchema[].AttributeName&#39; | \<br>    tr &#39;\n&#39; &#39; &#39;)&quot;</pre><p>Now you will need to scan your table for these keys, then use them to order deletion. The data extraction has a trick in the jq command:</p><pre>$ aws dynamodb scan \<br> --table-name foobar \<br> --attributes-to-get ${KEY_SCHEMA} | \<br> jq -r &quot;.Items[] | tojson&quot; | \<br> tr &#39;\n&#39; &#39;\0&#39;</pre><p>This returns a single, long line of JSON objects of the table items separated by null characters. We just emulated the -print0 feature of find with jq and tr. .</p><p>Now we will need to pipe that in xargs via</p><pre>... | xargs -0 -I keyItem \<br>  aws dynamodb delete-item \<br>   --table-name foobar \<br>   --key=keyItem</pre><p>The whole command being:</p><pre>$ aws dynamodb scan \<br>   --table-name foobar \<br>   --attributes-to-get ${KEY_SCHEMA} | \<br>   jq -r &quot;.Items[] | tojson&quot; | \<br>   tr &#39;\n&#39; &#39;\0&#39; | \<br>   xargs -0 -I keyItem \<br>    aws dynamodb delete-item \<br>      --table-name foobar \<br>      --key=keyItem</pre><p>If you want to debug the code, you can add -t to the xargs options, which copies the output to stdout before the actual execution.</p><p>The advantage of this method is to maintain the overall coherence of your AWS environment. However, it is slooooooowwwwww, especially if you are in underpowered development environments.</p><h3>Conclusion</h3><p>This is quite a change compared to my previous K8s work, but I must say serverless seriously rocks, but there are a number of things I wish I had known before starting working on that technology.</p><p>This is just the first of a series of tips and tricks about Serverless on AWS so others don&#39;t have to spend the time I did on them. I hope you&#39;ll enjoy them!</p><p>Any question, feedback, don&#39;t hesitate to ask in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=831ce5ab083c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Printing money with your idle hardware]]></title>
            <link>https://itnext.io/opportunistically-mining-cryptocurrencies-in-kubernetes-94934d095c4d?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/94934d095c4d</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tesla]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ubuntu]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Tue, 27 Feb 2018 17:04:31 GMT</pubDate>
            <atom:updated>2018-09-03T08:57:19.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s assume you operate a large bare metal cluster which you rent to your customers to run their workloads.</p><p>You notice that the cluster is in fact only used at 60% in average over the last months. You keep 15% capacity free in case of surges, and you need 5% of the cluster to operate the infra itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aVIqBkWlGjK3EokRKz6LKA.png" /><figcaption>Split of resource usage in the infrastructure</figcaption></figure><p>There are peaks at certain times of the day where it loads at 90%, but they are really hard to predict because they depend on your customers’ businesses and do not follow patterns of data you own.<br>Overall, you always run with about 20% of untaped capacity.</p><p>This may happen to you as a service provider or when you have a bunch of servers in your office. Either way<strong>, it&#39;s waste</strong>.</p><p>What if we could leverage that pool of resources and put it to good use? Let&#39;s see how! Yes <a href="https://medium.com/u/24413768aadb">Tesla Motors</a> this is for you ;D</p><h3>“Opportunistic AutoScaling”</h3><h4>Kubernetes and Autoscaling</h4><p>Since v1.2 of K8s, you can use the Horizontal Pod Autoscaler to autoscale an app based on CPU consumption. Cool but very limited use case.</p><p>Since the version 1.6, you may use custom metrics to autoscale an application within a cluster. Much better. It means you can expose metrics such as the number of hits on an API or its latency, then scale the serving pods according to that metric instead of the default CPU load.</p><p>Now doesn&#39;t that mean we could expose remaining capacity of the cluster (20%), and leverage it to autoscale a money printing application, so that the cluster is permanently used at its optimal capacity?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yp7zlMtuhN58NQEupb4TXQ.png" /><figcaption>What you really want as a split</figcaption></figure><h4>Opportunism</h4><p>The above behaviour is what I call &quot;<strong>Opportunistic Autoscaling</strong>&quot;. The ability for an app to leverage otherwise unused capacity of the infrastructure, may it be CPU, memory or GPUs.</p><p>The business critical app will be measured on how it performs. Your API must always answer with a low latency for example. On the other hand, your non business app can only consume <em>what’s left:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_K7n2Z7br_lF8tsN6NlvFA.png" /><figcaption>Resources available for autoscaling</figcaption></figure><p>In a nutshell, the &quot;remaining capacity&quot; behaves as:</p><ul><li>If the “business critical&quot; load on your cluster goes UP, the remaining resources go DOWN, and the opportunistic app shall scale DOWN.</li><li>If on the contrary your paid load goes DOWN, the remaining capacity goes UP, and the number of opportunist containers should go UP.</li></ul><p>The target is to have a load that is as constant as possible around a threshold you define (80% in our example), thus collecting an average of 20% unused power and monetize it.</p><h4>Printing money</h4><p>It may look a silly application but I can definitely tell you that the mining pools seem to see a load increase when office hours finish, showing that (some) business resources are definitely being used for mining at night!</p><p>In a real life scenario, crypto mining is effectively adding resources to a compute grid, hence it does also make sense beyond the hype and fun. There are also a lot of other interesting use cases. Among others:</p><ul><li>Lambda on the edges using a serverless framework (Telco / Cloud Operator)</li><li>Elastic transcoding (Media Lab / Cloud): Think of what Ikea is doing on workstations but in a compute cluster</li><li>AI on the edges (Media Lab / Cloud)</li><li>Caching (CDN)</li><li>The cool use case you’ll share in comments</li></ul><p>OK, enough talking! Let&#39;s get this done and increase revenues.</p><p><strong>DISCLAIMER</strong>: Deploying this is fairly complex and involves multiple steps. As a consequence this post is a lot longer and more technical than usual ones. If you are here because you like the use case but do not wish to dig into technical details, you can essentially skip from now to the conclusion.</p><h3>Using Reversed Custom Metrics</h3><p>In this blog, we will create a K8s cluster with a custom metrics API on bare metal.</p><p>We then create an app that exposes (among others) a metric as follows</p><blockquote>Remaining CPU = (total amount of CPU requested in cluster) - (total requested by all applications)</blockquote><p>This metrics decreases when the load of the cluster grows, and grows when the load shrinks. This effectively will make the metric a “client” of the requested load on a server.</p><p>Then we will use this metric to configure a Horizontal Pod Autoscaler (HPA) in Kubernetes. This will result in keeping the load in the cluster as high as possible.</p><h3>Full disclosure</h3><p>This blog post has been sponsored by <a href="https://www.kontron.com/">Kontron</a> who gracefully allowed me to play with a 6 nodes cluster of their latest <a href="https://symkloud.com/">SymKloud Platform</a>, an awesome piece of hardware in which you can mix and match modules to create a cluster. There are modules with GPUs, CPUs, some dedicated to storage or caching… Each 2U server can contain up to 9 &quot;sleds&quot;, effectively going up to 576 cores with dual CPU sleds, or up to 9 nVidia P4 with 288 cores. My cluster had 6 workers, 2 of them with GPUs.</p><p>In addition, my friend <a href="https://www.linkedin.com/in/ronan-delacroix/">Ronan Delacroix</a> helped me with the code and wrote all of the python needed for this experiment.</p><p>This work is currently presented at MWC in Barcelona on <a href="https://www.mobileworldcongress.com/exhibitor/kontron/">Kontron&#39;s booth</a>.</p><h3>Kubernetes Cluster</h3><p>It happens that Kontron partners with <a href="https://www.ubuntu.com/kubernetes">Canonical</a> to run Kubernetes on SymKloud. You will not be surprised to learn that this post is based on the <a href="https://www.ubuntu.com/kubernetes">Canonical Distribution of Kubernetes</a> (CDK), in its version 1.8.</p><p>If you need to replicate this in a way or another, you will need a K8s Cluster in version 1.8, with RBAC active, and an admin role.</p><p><strong>Important Note</strong>: The APIs we will be using here are very unstable and subject to big changes. I really recommend you read the K8s change log to check on them.</p><ul><li>For example, there was a <a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md">change in 1.8</a> on the name of the APIs. If you run a 1.7 cluster, this will impact you.</li><li>There are also changes in 1.9 and the custon.metrics.k8s.io moves from v1alpha1 to v1beta1.</li></ul><p>There are some details of the configuration we will see today that are done in a certain way on <a href="https://www.ubuntu.com/kubernetes">CDK</a> and may be slightly different on clusters that are self hosted. I will try to mention them whenever possible. In any case, feel free to ask questions in the Q&amp;A.</p><h4>RBAC Configuration</h4><p><strong>NOTE</strong>: this applies to CDK, and will apply to GKE when the API Aggregation is GA with K8s 1.9.</p><p>On CDK and GKE the default user is not a real admin from an RBAC perspective, so you need to update it before you can create other Cluster Role Bindings that extend your own role.</p><p>First make sure you know your identity with</p><ul><li>CDK: you are the user admin</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ac5ec1c9bfc69c2dbee347e49f3636c/href">https://medium.com/media/0ac5ec1c9bfc69c2dbee347e49f3636c/href</a></iframe><ul><li>GKE: your GCP user is the admin</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df414cc35223794fff14c668ba81bf92/href">https://medium.com/media/df414cc35223794fff14c668ba81bf92/href</a></iframe><p>Now grant yourself the cluster-admin role:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d650bdddcd1cc7bf2c741e28da9f82b/href">https://medium.com/media/7d650bdddcd1cc7bf2c741e28da9f82b/href</a></iframe><p>You can then check it out with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7865e5fc9b80b3b8bc065d9633406c5f/href">https://medium.com/media/7865e5fc9b80b3b8bc065d9633406c5f/href</a></iframe><h4>Helm Install</h4><p>In order for Helm to operate in a RBAC cluster, we need to add it as a cluster-admin as well:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34bef132a3de75948b7c7eba37608320/href">https://medium.com/media/34bef132a3de75948b7c7eba37608320/href</a></iframe><p>OK, we are done with prepping our cluster</p><p><strong>Note</strong>: By default Helm deploys without resource constraints. When trying to surcharge a cluster and maximize its usage, it means that Tiller will be part of the pods that may go away because resources are exhausted. If you do not want that to happen, you can edit the manifest and reapply it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb87d095147d8d9f64f45e76c38b02f7/href">https://medium.com/media/fb87d095147d8d9f64f45e76c38b02f7/href</a></iframe><h3>AutoScaling: Preparing the cluster</h3><h4>Introduction &amp; References</h4><p>First of all, I recommend you have a look at the <a href="https://kubernetes.io/docs/concepts/overview/extending/">documentation</a> about extending Kubernetes.</p><p>Then also look at the <a href="https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/">documentation</a> about extending Kubernetes with the Aggregation Layer.</p><p>The last theorical doc is about setting up an API Server, and you can find it <a href="https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/">here</a>.</p><p>Once the aggregation is active, we will deploy 2 custom APIs: the <a href="https://github.com/kubernetes-incubator/metrics-server">Metric Server</a> and a <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">Custom Metric Adapter</a></p><p>OK now that you are versed in what we need to do, let’s get started.</p><h4>Configuring the control plane</h4><p>In order to activate the Aggregation Layer, we must add a few flags to our API Server:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74f4790865e2b60565d8c73cf7660ec2/href">https://medium.com/media/74f4790865e2b60565d8c73cf7660ec2/href</a></iframe><p>You will note that we do not activate the flags</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/935e6b4319e25c2fe7003dbee504df38/href">https://medium.com/media/935e6b4319e25c2fe7003dbee504df38/href</a></iframe><p>This is because the proxy in CDK uses a Kubeconfig and not a client certificate.</p><p>However, we do enable the aggregator routing because the control plane of Kubernetes is not self hosted and we fall in the case “If you are not running kube-proxy on a host running the API server, then you must make sure that the system is enabled with the enable-aggregator-routing flag”.</p><p>Also we added the client-ca-file flag to export the CA of the API server in the cluster.</p><p>Now for the Controller Manager, we must tell it to use the HPA, which we do with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4a425b313e3389eef9c53b3969224e5/href">https://medium.com/media/c4a425b313e3389eef9c53b3969224e5/href</a></iframe><p>Note that the 2 last options here are really for demos to make it quick to observe the results of actions. You may not need change them for your use case (they default to 3m and 5m).</p><p>Just to make sure the settings are applied restart the 2 services with</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac8460fbb30f5e8f1b5df15b6a864bc0/href">https://medium.com/media/ac8460fbb30f5e8f1b5df15b6a864bc0/href</a></iframe><p>This will make Kubernetes create a configmap in the kube-system namespace called extension-apiserver-authentication, which contains all the additional flags we generated and their configuration. You can have a look at it via</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8b3579750427351bee2395bf4ef4c58/href">https://medium.com/media/f8b3579750427351bee2395bf4ef4c58/href</a></iframe><p>Each of API servers will now need to have an RBAC authorization to read this Config Map. Thankfully K8s will also automatically create a role for it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed81d1d85bf14cd1b0970eb85b7a9d45/href">https://medium.com/media/ed81d1d85bf14cd1b0970eb85b7a9d45/href</a></iframe><p>Last but not least, you won’t need Heapster for now, so make sure it is not there via:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32e919100f7fff0a243a1c2f4f92d783/href">https://medium.com/media/32e919100f7fff0a243a1c2f4f92d783/href</a></iframe><h4>Initial API State</h4><p>Before you start having fun with API Servers, have a look to the status of your cluster with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/089f6b98a6c2ddc4edc4253868b134a0/href">https://medium.com/media/089f6b98a6c2ddc4edc4253868b134a0/href</a></iframe><p>At the end of the next sections, you will have 3 more APIs in this list:</p><ul><li><strong>monitoring.coreos.com/v1</strong>, for the Prometheus Operator</li><li><strong>metrics.k8s.io</strong>, for the Metrics Server that collects metrics for CPU and Memory</li><li><strong>custom.metrics.k8s.io</strong>, for the custom metrics you want to expose</li></ul><h3>Adding the Metrics Server API</h3><p>There are 2 implementations of the Metrics API (metrics.k8s.io) at this stage: Heapster and the Metrics Server. At the time of this writing, the Metrics Server has a simple deployment method, while Heapster required some work on my end, and I was too lazy to write the code.</p><p>We can simply deploy it with</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b517aec38b24ec79ff34d13491a212be/href">https://medium.com/media/b517aec38b24ec79ff34d13491a212be/href</a></iframe><p>This manifest contains:</p><ul><li>the Service Account for the Metrics Server</li><li>a RoleBinding so that the Metrics Server can read the configmap above</li><li>a ClusterRoleBinding so that the Metrics Server inherits the system:auth-delegator ClusterRole (you can find documetation about that <a href="https://kubernetes.io/docs/admin/authorization/rbac/">here</a>.</li><li>a Deployment and ClusterIP Service for the Metrics Server</li><li>an APIService object, which is a registration of the new API into the API Server.</li></ul><p>Now check our APIs again:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0949832951e81b78f84d5dba19d10c32/href">https://medium.com/media/0949832951e81b78f84d5dba19d10c32/href</a></iframe><p>Awesome… But does it really work? Query the endpoint of the API via kubectl to make sure</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bee6262267376a6f31499a6f75b6f0f4/href">https://medium.com/media/bee6262267376a6f31499a6f75b6f0f4/href</a></iframe><p>Good job, NodeMetrics and PodMetrics are exposed. Look into what you can use from there:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb4c15f5ce78725b953cc077bf061977/href">https://medium.com/media/fb4c15f5ce78725b953cc077bf061977/href</a></iframe><p>And</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac1da853164108ecacba80ca3d34efa4/href">https://medium.com/media/ac1da853164108ecacba80ca3d34efa4/href</a></iframe><p>No big surprise here, <strong>you can access the CPU and memory consumption in real time</strong>. Refer to the docs for more details about how to query the API.</p><h3>Installing the Custom Metrics Pipeline</h3><p>Right before we have taken a shortcut, having a metric pipeline that is directly exposable as the aggregated API. Unfortunately, in the case of custom metrics, we must do this in 2 distinct steps.</p><p>First of all we must deploy the custom metrics pipeline, which will give us the ability to collect metrics. We use Prometheus for that part as the canonical example of metrics collection system on K8s.</p><p>Then we will expose these metrics via a specific API Server. We will use the work of Sully (@DirectXMan12) that can be found <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">here</a> for that.</p><p>Prometheus has many installation methods. My personal favorite is the <a href="https://github.com/coreos/prometheus-operator">Prometheus Operator</a>. It takes a lot of efforts to architect a piece of software using traditional solution. But crafting a software model that ties to the underlying distributed infrastructure beautifully is closer to art than to anything else.</p><p>That is essentially what the operator is. The operator models how Prometheus should be given a set of conditions, then realizes that in Kubernetes. Whaow, good job @CoreOS.</p><p>Note that you can create an Operator for anything, and that something similar is coming for <a href="https://www.slideshare.net/inwinstack/extend-the-kubernetes-api-with-crd-and-custom-api-server">Tensorflow</a> as far as I can see the APIs coming up… Anyway, let’s not get distracted.</p><p>Install the Prometheus Operator with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f38f6782074e5ca1e153c42aa126d8b/href">https://medium.com/media/0f38f6782074e5ca1e153c42aa126d8b/href</a></iframe><p>This contains:</p><ul><li>a Service Account for the operator</li><li>a ClusterRole and ClusterRoleBinding that are fairly extensive, so that the Operator can deploy Custom Resource Definitions for Prometheus (instances of), Alert Managers and Service Monitors.</li><li>a Deployment for the Operator pod.</li></ul><p>This will let the Operator add the monitoring API as well:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc762f49133f68f3c79c5be3fce24d8f/href">https://medium.com/media/bc762f49133f68f3c79c5be3fce24d8f/href</a></iframe><p>Now create an instance of Prometheus with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63bb623cb49774e9261568f55181d423/href">https://medium.com/media/63bb623cb49774e9261568f55181d423/href</a></iframe><p>The RBAC manifest will allow Prometheus to read the metrics it needs in the cluster and /metrics endpoints of any object (pod or service). The Prometheus manifest defines an instance and a service to expose it as a nodePort (so we can have a look at the UI).</p><p>What is important in this second file is the section:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d92daf92993853e18481386f9be57bda/href">https://medium.com/media/d92daf92993853e18481386f9be57bda/href</a></iframe><p>This essentially dedicates the Prometheus instance to Service Monitors with this label (or set of labels). When we will define the applications we want to monitor and how, we will need that information.</p><p>Note that this is a trivial example of deployment, with no persistent storage or any fancy thingy. If you are contemplating using this for a more production grade usage, you will need to spend some time on this.</p><p>OK, now you can connect on the UI and check that you have everything deployed correctly. It is pretty empty for now…</p><h3>Installing the Custom Metrics Adapter</h3><p>Now that we have the ability to collect metrics via our Prometheus pipeline, we want to expose them under the aggregated API.</p><p>First of all, you will need some certificates. Joy. This is all documented <a href="https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md">here</a>. Run the following commands to generate your precious:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/341dbb7c522ca46b31e9dc2658e71600/href">https://medium.com/media/341dbb7c522ca46b31e9dc2658e71600/href</a></iframe><p>In order to authenticate our extended API server against the Kubernetes API Server, we have several options:</p><ul><li>Using a client certificate</li><li>Using a Kubeconfig file</li><li>Using BasicAuth or Token authentication</li></ul><p>Adding users with certificates in CDK is a project in itself and would deserve its own blog post. If interested, ping me in questions and we can discuss this in DMs. BasicAuth and Tokens are easy, but they also require to edit <strong>/root/cdk/known_tokens.csv</strong> or <strong>/root/cdk/basic_auth.csv</strong> on all masters and restart the API server daemon everywhere.</p><p>So the solution with the least complexity is actually the Kubeconfig file. Thanks to RBAC, the only thing we need to create a new user is a service account, which will give us access to an authentication token, which we can then put into our kubeconfig.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/052af12bb7f82d6c112b3dda4fb4f938/href">https://medium.com/media/052af12bb7f82d6c112b3dda4fb4f938/href</a></iframe><p>You can then create a copy of your .kube/config file and edit the user section to add the custom-api-server:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac873ab40d1885dcb4ea0b058eef0a97/href">https://medium.com/media/ac873ab40d1885dcb4ea0b058eef0a97/href</a></iframe><p>Do not forget to also edit the contexts to map to this user instead of admin.</p><p>Now edit a cm-values.yaml file for the helm chart:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/696b18f986ae07f51bb06c794b0e7c18/href">https://medium.com/media/696b18f986ae07f51bb06c794b0e7c18/href</a></iframe><p>OK you are now ready to download the chart and install it</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6020e8203e86685bff24720abc372fdf/href">https://medium.com/media/6020e8203e86685bff24720abc372fdf/href</a></iframe><p>And we check that the new API is registered in Kubernetes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab5bcd88189b68bf4b4bcbbdecc61b89/href">https://medium.com/media/ab5bcd88189b68bf4b4bcbbdecc61b89/href</a></iframe><p>Great. Now let us check that everything works properly by querying the K8s endpoint for it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bfd7bc2681b58c2a071da78c594631a/href">https://medium.com/media/5bfd7bc2681b58c2a071da78c594631a/href</a></iframe><p>At this point of our development, if we get a 200 answer and NOT a 404 we are good and we have our autoscaling up &amp; running. If you get a 404, then it did not work properly.</p><h3>Summary</h3><p>In the long section above, we have done the following</p><ol><li>Install the new Metrics Server and add the metrics.k8s.io API to the cluster. This gave us access to an equivalent of heapster, but exposing metrics under the classic API.</li><li>Install a Custom Metrics pipeline to be able to collect any metrics. We did this via Prometheus, using the Operator to create a Prometheus Instance</li><li>Install the Custom Metrics API custom.metrics.k8s.io via the installation of a Prometheus Adapter.</li></ol><p>For each step, we validated that the cluster worked properly and as intended. Now we need to put it to good use.</p><h3>Using Custom Metrics</h3><h4>Demo Application: http_requests</h4><p>First of all we will test our set up with a very simple application written by @luxas that exposed a http_request metric on /metric. You can deploy it with</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27a6786c4c5b84c02e659b03e448198c/href">https://medium.com/media/27a6786c4c5b84c02e659b03e448198c/href</a></iframe><p>This manifest contains:</p><ul><li>the Deployment and Service so we can query the application</li><li>a Service Monitor, which will indicate to the prometheus instance that it should scrap the metrics of the application</li><li>An Horizontal Pod Autoscaler (HPA), which will consume the number of http_requests and use it to scale the application.</li></ul><p>Let us look into the HPA for a moment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfa92850e01000352bbd9fc0d51f4097/href">https://medium.com/media/bfa92850e01000352bbd9fc0d51f4097/href</a></iframe><p>As you can see, we have here</p><ul><li>a Target (our deployment), with a minReplicas and a maxReplicas.</li><li>a metrics of type pod which tries to make sure that pods get an average 500m queries (which slightly above what the standard load from Kubernetes + Prometheus is)</li></ul><p>So this means that you do not need the application to rely on its own metrics. You could potentially target any application metrics and use them to manage another application. Very powerful principles.</p><p>Let us say for example that you manage an application based on the principle of decoupled invocation, such as a chat or an order management solution. Some day, you start getting a peak of requests on the front end, and the backend does not follow. The queue fills up, and you start experimenting delays in processing of the requests. Well now you can scale the workers that process the queue based on the requests made on the front end. You create a target object that monitors the number of http_requests on the frontend, but the scale target may be your application. It is as simple as that.</p><p>Now look at how Custom API reacts to this (it may take a couple of minutes before this works)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4766f2b952082336b93afebf88c485a/href">https://medium.com/media/e4766f2b952082336b93afebf88c485a/href</a></iframe><p>And we can then query the service itself:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fe48b44866498e609924014c9089755/href">https://medium.com/media/1fe48b44866498e609924014c9089755/href</a></iframe><p>And we then look at our HPA:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05ce3e3a0de3db01065a6a0e378dc79d/href">https://medium.com/media/05ce3e3a0de3db01065a6a0e378dc79d/href</a></iframe><p>We can see here that just the requests for status account for 433m <em>per pod</em> which is a little more than a request every 400ms. Now deploy a shell app so we can create some load:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/605e97d46a5681996483d3227bf05de2/href">https://medium.com/media/605e97d46a5681996483d3227bf05de2/href</a></iframe><p>Now prepare 2 shells. In the first one, connect into your container with</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a795a5b1d527d66ee61e0ebc40254a1/href">https://medium.com/media/4a795a5b1d527d66ee61e0ebc40254a1/href</a></iframe><p>And in the second one, track the HPA with</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22c7fe96a298d33f02a52eff4259a577/href">https://medium.com/media/22c7fe96a298d33f02a52eff4259a577/href</a></iframe><p>and we have successfully triggered an up scale and downscale of an application</p><h4>Main Application: Optimizing the infrastructure</h4><p>OK! Now we have an application that can generate load in our cluster and consume resources. Now let us look at how to use the remaining resources.</p><p>So first of all let’s look at our metrics harvesting application Ronan wrote. It exposes the following metrics:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/deb5c23e05fbdc1e2187bb7c3256a5b4/href">https://medium.com/media/deb5c23e05fbdc1e2187bb7c3256a5b4/href</a></iframe><p>In addition, he created a nice UI that presents the values in real time.</p><p>This requires a Grafana installation. You can install both apps with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c870261a7296414e9a65b65ccbebb404/href">https://medium.com/media/c870261a7296414e9a65b65ccbebb404/href</a></iframe><p>These manifests contain</p><ul><li>Cluster Roles and bindings for collecting metrics</li><li>Deployments for both Grafana and the python application</li><li>Nodeports services on ports 30505 (app) and 30902 (Grafana)</li><li>Config maps to configure both</li></ul><p>What is of interest to us in this example is the “cpu_capacity_remaining”. As mentioned in the intro, I had access thanks to Kontron to a 184-core cluster. I decided to “reserve” 30 cores, or 15% of my capacity to give room for load peaks. This gave me an autoscaler looking like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56eb948f839893e7869cda64a42aeeee/href">https://medium.com/media/56eb948f839893e7869cda64a42aeeee/href</a></iframe><p>You will note I am using <a href="https://electroneum.com/">Electroneum</a> as my crypto. The reason for this is practical. It is a very new cryptocurrency, with limited mining resources allocated to it right now, which means you can directly measure your impact and see daily returns, which is cool for demos. In case you wonder, as this currency requires a Monero miner, this setup can easily be converted into something more lucrative by pointing it to a real monero pool.</p><p>To replicate this blog with your own machines, edit the src/manifest-etn.yaml file according to your own cluster then deploy with :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71b4f6b4d82b76fc0e2df78439a6493e/href">https://medium.com/media/71b4f6b4d82b76fc0e2df78439a6493e/href</a></iframe><p>This manifest contains:</p><ul><li>a Deployment of the miner</li><li>a Horizontal Pod Autoscaler as seen above</li><li>a service to expose the UI on port 30500 of nodes.</li></ul><p>Now let us check on our HPAs with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4750f16dbeb98714ed20dc8c55d3abbe/href">https://medium.com/media/4750f16dbeb98714ed20dc8c55d3abbe/href</a></iframe><p>Alright, we are all set! Now we can finally check how our application reacts to load.</p><h3>Opportunistic Load Balancer in motion</h3><p>In order to supercharge our cluster, we reuse our shell-demo application, and generate 10 hits per second on the API for 5 min. Because we are expecting only 0.5 hits, this will quickly trigger the scale out:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cc4e9ccd0f9ad42eb99ca83ad497f78/href">https://medium.com/media/5cc4e9ccd0f9ad42eb99ca83ad497f78/href</a></iframe><p>There you go, we can see the new pod coming in. Each new pod requests 4 CPU cores to the cluster. This unbalances the HPA, that tries to counter by releasing miners. Over 5 minutes, our app will scale up to 17 replicas, thus claiming 68 cores to the cluster, which will be freed from the mining app. After 5 minutes, the load is now normal and we see a scale down of the simple app from 17 pods down to its stable version at 2 replicas. The HPA for the miner will react and start harvesting the capacity.</p><p>This can be seen in the UI on the CPU capacity graph</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2J7ajJRoxUG2rqZBTbGmw.png" /></figure><p>We now have an application that is opportunistically adjusting to the load created by other applications in the cluster <em>all by itself</em>. To see a little better how the HPA behaves, we can look directly at Grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ofZPbbXd-XgIYjUdX2ry_Q.png" /><figcaption>Cluster perfectly auto adjusting to business load</figcaption></figure><p>Here you can clearly identify the peaks of load on the second graph in green, how the HPA reacts by scaling the number of API replicas. On the top graph, we can see the blue area (business load) going up, and shortly after the yellow line going down (this is the opportunistic app scaling down), the red &quot;remaining CPU cores&quot; fluctuating, while the total (yellow + blue + red) is about constant, representing the total number of cores in the system (184).</p><p>I should share a potatoe as this post was really really long.</p><h3>Some thoughts about the HPA</h3><h4>Keep the non business load low</h4><p>While creating this blog, I had a huge hard time configuring the HPA to make it stable and convergent and not completely erratic. One must understand that the HPA in K8s is, so far, pretty dumb. It is not exactly learning from the past, rather systematically repeating the same reaction patterns regardless of the fact they failed or succeeded in the past.</p><p>Let’s say a custom metric is at 150% of its target value, then the cluster will perform a 150% capacity increase. This means that if your application is creating 2% HPA resource value for 1% increase of scale, you will enter into a turbulence zone, with the HPA proving incapable of converging because it is always overreacting to the environment.</p><p>Because of that behaviour, if the opportunistic load represents the majority of your total cluster load, you have a risk of generating an ever fluctuating, sub-optimal HPA. Below is an example where the mining rig varies between 15 and 120 cores (60% of the cluster), while the business load is only ~20%. Under these conditions, the cluster takes too long to converge and, effectively, sometimes never does so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8GC0ozVUZrrSEISgvFwLGA.png" /><figcaption>Cluster in a non convergent state</figcaption></figure><p>Long story short: NEVER, EVER use an HPA that can diverge!! Experiment and learn, keep the influence of the HPA reasonable in the cluster.</p><h4>Total Failure</h4><p>So this is something I was not able to debug completely.</p><p>In the last Grafana screen above you can see that there is a longer peak of high load in the second load surge. In effect, the HPA got stuck and for some reason would never go back until you force it to.</p><p>From my experience, this only happens when an HPA fluctuates greatly then reaches its max. At this point, if that condition lasts for too long, it will then fail to downscale afterwards, thus effectively crashing.</p><p>Again, when building an HPA, do some experiments. Test your metrics, make sure they work well together.</p><h3>Conclusion</h3><p>I always dreamt of building the “<em>Opportunistic Autoscaler</em>”. For the first time of my life, thanks to Ronan, Kontron, and the awesome work done by the community on Kubernetes and Canonical on CDK, I was able to put it together. And it &quot;just works&quot;!</p><p>At the beginning of the post, we wanted to add value by either reducing costs or increase revenues. Depending on your opportunistic app, you may be in either or both use cases.</p><p>For sure, over the course of the month this was up and running, we managed to</p><ul><li>average ~25 free CPU cores, from a target of 24 in the HPA , creating random load every 30min.</li><li>Opportunistically consume an average of 30 cores, which would have been lost otherwise.</li></ul><p>Does that make money with mining cryptos? Not much. We mined about 1000 ETN while testing this, for a total value of about $100. More than nothing but not a lot.</p><p>But now think Serverless and do the math. 30 cores is about 1 sled of the system free at any time. Assuming this also translates into the same amount of RAM being free:</p><ul><li>A sled can have up to 256GB RAM.</li><li>For Lambda, AWS charges 0,00001667$/GB-s plus a bit for the invocations</li><li>There are 86400x365,25 = 31 557 600 s / yr</li></ul><p>soooo… 256 * 0,00001667 * 31 557 600 = <strong>$134 672,69</strong></p><blockquote>$134 672,69 is, if it was permanently running Lambda 100% all the time, the business value of the very sled we just used. Not bad for an &quot;unused&quot; resource.</blockquote><p>Does that give you some ideas?</p><h3>References</h3><p>I would like to shout a special thanks to @Luxas and @DirectXMan12 for inspiring this work and for the fantastic walkthrough they wrote <a href="https://github.com/luxas/kubeadm-workshop">here</a> and <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/walkthrough.md">there</a> that helped me a lot while writing this.</p><p>The code for the UI is here: <a href="https://github.com/ronhanson/crypto-miner-webui">https://github.com/ronhanson/crypto-miner-webui</a></p><p>The source and manifests for K8s are here: <a href="https://github.com/madeden/blogposts/tree/master/k8s-autoscaling">https://github.com/madeden/blogposts/tree/master/k8s-autoscaling</a></p><p><em>Originally published at </em><a href="https://gist.github.com/d8142e6cbd72b894d03d12d1f1eb8687"><em>gist.github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94934d095c4d" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/opportunistically-mining-cryptocurrencies-in-kubernetes-94934d095c4d">Printing money with your idle hardware</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fun with Kubernetes & Tensorflow Serving]]></title>
            <link>https://itnext.io/fun-with-kubernetes-tensorflow-serving-4fef8d7502b9?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fef8d7502b9</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Fri, 23 Feb 2018 17:44:59 GMT</pubDate>
            <atom:updated>2018-08-23T15:04:38.150Z</atom:updated>
            <content:encoded><![CDATA[<p>There are multiple available walkthroughs available for Tensorflow Serving, to run on K8s or otherwise. The vast majority gets you to the point where you still need to use the Tensorflow Python client to publish images or whatever you want to analyze.</p><p>While you can find data about performance optimisation of Tensorflow itself from the <a href="https://www.tensorflow.org/performance/performance_guide">docs</a>, it does not really cover some operational details which can be interesting when you intend to run the service. For an inception model, should you for example use PNG images? JPEG files? Is performance better if you use a high quality images or should you downscale them? Is accuracy better if you use a high quality images?</p><p>To start answering these questions, I teamed up with <a href="https://symkloud.com/">Kontron</a>, who provided access to hardware, and <a href="https://www.linkedin.com/in/ronan-delacroix/">Ronan Delacroix</a>, the best Python guru I have ever known.</p><p>We did :</p><ol><li>Build from source TF Serving binaries for CPU and GPU, in both optimized and non optimized fashion, with parameters taken from the docs;</li><li>Build Docker images with these fresh binaries;</li><li>Look at how to prepare the inception model to be served by TF Serving;</li><li>Build a Kubernetes manifest to deploy TF Serving with various resource constraints;</li><li>Build a web app with a simple API through which we can publish an image, run a simple pre-processing pipeline and compare the results;</li><li>Draw some early stage conclusions.</li></ol><p>And now we want to share that awesome experience! And collect your feedback to iterate on it.</p><h3>Prerequisites</h3><p>To replicate this post, you will need access to a Kubernetes 1.6+ cluster, preferably with GPUs in it.</p><p>In our example here, we used the <a href="https://www.ubuntu.com/kubernetes">Canonical Distribution of Kubernetes</a> in its version 1.8.7, deployed on a 6 worker HA cluster using the <a href="https://www.kontron.com/products/systems/cloud-systems/symkloud-ms2910.html">Kontron Symkloud MS2911</a>. 2 of the nodes were equipped with <a href="http://images.nvidia.com/content/pdf/tesla/184457-Tesla-P4-Datasheet-NV-Final-Letter-Web.pdf">nVidia Tesla P4 cards</a>.</p><h3>Preparing Tensorflow Serving</h3><p>Building TF Serving “just right” is no easy task, especially as it is a fast moving target. This exercise was based on TF 1.5, and I do not guarantee that it will be reproducible. I spent a LOT of time getting it right, and I am still not happy with the whole thing.</p><h4>Development Docker Image</h4><p>First of all we need to build a base image that we can then use to build the various versions of our model server. We could use multistage Docker images, but for the sake of understanding what we are doing, I am just do it manually here.</p><p>The base &quot;FROM&quot; is from the latest CUDA image, so that we can do both CPU and GPU builds, and you can find the source in the repo.</p><p>In the src folder for this repo, you will find a nice Dockerfile for this. Build with:</p><pre>$ cd src<br>$ docker build — rm -t ${USER}/tf-dev -f Dockerfile.tf-dev .<br>$ cd ..</pre><p>Now we need to download Tensorflow Serving. There was a big change between 1.4 and 1.5 where submodules were removed. This change was not completely taken care of in docs and build systems, so we will need to hack it a little bit here.</p><pre>$ git clone --recurse-submodules <a href="https://github.com/tensorflow/serving">https://github.com/tensorflow/serving</a><br># This will in the end NOT download some required dependencies, so<br>$ cd serving<br>$ git clone --recursive <a href="https://github.com/tensorflow/tensorflow.git">https://github.com/tensorflow/tensorflow.git</a><br>$ git clone --recursive <a href="https://github.com/tensorflow/models.git">https://github.com/tensorflow/models.git</a> tf_models</pre><p>At this point, you have a docker image ready as well as the source code to build. Now let’s run it</p><pre>$ docker run -it -name tf-dev -v $PWD/serving:/serving ${USER}/tf-dev bash</pre><h4>Building TF Serving</h4><p>Now you are in the container, to build TF Server, you have several options:</p><ol><li>GPU, non optimized:</li></ol><pre># cd /serving<br># bazel build -c opt --config=cuda \<br>    --crosstool_top=<a href="http://twitter.com/local_config_cuda">@local_config_cuda</a>//crosstool:toolchain \<br>    tensorflow_serving/model_servers:tensorflow_model_server \<br>    &amp;&amp; cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \<br> /usr/local/bin/tensorflow_model_server.gpu.standard \<br>    &amp;&amp; bazel clean --expunge</pre><p>2. GPU, optimized:</p><pre># cd /serving<br># bazel build -c opt --config=cuda \<br>    --crosstool_top=<a href="http://twitter.com/local_config_cuda">@local_config_cuda</a>//crosstool:toolchain \<br>    --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 — copt=-mfma — copt=-O3 \<br>    tensorflow_serving/model_servers:tensorflow_model_server \<br>    &amp;&amp; cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \<br> /usr/local/bin/tensorflow_model_server.gpu.optimized \<br>    &amp;&amp; bazel clean --expunge</pre><p>3. CPU, non optimized</p><pre># cd /serving<br># bazel build -c opt \<br>    tensorflow_serving/model_servers:tensorflow_model_server \<br>    &amp;&amp; cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \<br> /usr/local/bin/tensorflow_model_server.cpu.standard \<br>    &amp;&amp; bazel clean --expunge</pre><p>4. CPU, optimized :</p><pre># cd /serving<br># bazel build -c opt \<br>    --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 \<br>    tensorflow_serving/model_servers:tensorflow_model_server \<br>    &amp;&amp; cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \<br> /usr/local/bin/tensorflow_model_server.cpu.optimized \<br>    &amp;&amp; bazel clean --expunge</pre><p>This takes a LOT of time. On my Xeon E5 24 cores, 521 seconds (9min) for the non optimized CPU version, and up to <br>1427 seconds (22min) for the optimized GPU version. At the end of the process, you will have these 4 binaries in /usr/local/bin/ of your docker container.</p><p>Now we also need to build another dev binary for the purpose of serving Inception. If you have your own models already you do not need to do this step:</p><pre># cd /serving<br># bazel build -c opt \<br>    tensorflow_serving/example:inception_saved_model</pre><p>On a side note, if you run the build on tensorflow_serving/… you will build many more examples, though it will take longer. May be useful for some use cases.</p><p>On a second side note, this is <em>supposed to</em> work, but there is bug reported <a href="https://github.com/tensorflow/serving/issues/737">here</a> that prevents it from working properly. If you really need this, the good news is that the bitnami docker image <strong>bitnami/tensorflow-inception:latest</strong> contains a working binary at <strong>/opt/bitnami/tensorflow-inception/bazel-bin/tensorflow_serving/example/inception_saved_model</strong></p><h4>Extracting binaries</h4><p>All the binaries are built, you can now extract them from another shell with:</p><pre>$ mkdir bin<br>$ for type in gpu cpu; do<br>    for version in standard optimized; do<br>      docker cp tf-dev:/usr/local/bin/tensorflow_model_server.${type}.${version} ./bin/<br>    done<br>  done</pre><p>In order to save the tool that we will use to unpack the mode, we commit our docker image with:</p><pre>$ docker commit tf-dev ${USER}/tf </pre><p>You do not have to do this if you intend to use the bitnami image.</p><h3>Building TF Serving Images</h3><p>Now that you have all the binaries ready, use them to build relatively small Docker images</p><p>Again you will find the sources in the src folder of this repository. You can adapt the files to run the non optimized versions as well</p><pre>$ for type in gpu cpu; do<br>    docker build — rm -t ${USER}/tf-serving:${type} -f src/Dockerfile.${type} .<br>    docker push ${USER}/tf-serving:${type}<br>  done</pre><h3>Creating a servable inception model</h3><p>Strangely enough when you come from a more ops background, models that you download cannot be used out of the box. First you need to convert them into a frozen model, add a file for variables, then and only then can you serve them. Not very practical but well, it is what it is…</p><p>Download the Inception model from the Google Storage then extract it:</p><pre>$ cd /tmp<br>$ curl -sL <a href="http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz">http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz</a><br>$ tar xfz inception-v3–2016–03–01.tar.gz<br>$ cd -</pre><p>Now you will need to convert it into something servable by TF Serving:</p><pre>$ mkdir model-data<br>$ docker run — rm \<br>    -v $PWD/model-data:/model-data \<br>    -v $PWD/serving:/serving \<br>    -v /tmp/inception-v3:/inception \<br>    ${USER}/tf \<br>    /serving/bazel-bin/tensorflow_serving/example/inception_saved_model \<br>       --checkpoint_dir=/inception \<br>       --output_dir=/model-data</pre><p>The output of this operation is a subfolder in your model-data folder that contains a saved_model.pb files as well as a variables folder. This is what the model_server uses to serve a model.</p><p><strong>NOTE</strong>: if you got stuck in the tensorflow bug reported above, use this line instead:</p><pre>$ mkdir model-data<br>$ docker run — rm \<br>    -v $PWD/model-data:/model-data \<br>    -v /tmp/inception-v3:/inception \<br>    bitnami/tensorflow-inception \<br>    /opt/bitnami/tensorflow-inception/bazel-bin/tensorflow_serving/example/inception_saved_model \<br>      --checkpoint_dir=/inception \<br>      --output_dir=/model-data</pre><p>which will output something like:</p><pre>Welcome to the Bitnami tensorflow-inception container<br>Subscribe to project updates by watching <a href="https://github.com/bitnami/bitnami-docker-tensorflow-inception">https://github.com/bitnami/bitnami-docker-tensorflow-inception</a><br>Submit issues and feature requests at <a href="https://github.com/bitnami/bitnami-docker-tensorflow-inception/issues">https://github.com/bitnami/bitnami-docker-tensorflow-inception/issues</a><br>Send us your feedback at <a href="/service/mailto:containers@bitnami.com">containers@bitnami.com</a></pre><pre>2018-02-22 17:25:00.723742: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA<br>Successfully loaded model from /inception/model.ckpt-157585 at step=157585.<br>Exporting trained model to /model-data/1<br>Successfully exported model to /model-data</pre><h3>Developing a Python API to publish images</h3><p>Ronan developed an application that does the following:</p><ul><li>Allow to upload an image (of a cat)</li><li>resize it in several versions from fitting in a 2048x2048 box to fitting in a 128x128 box (we actually did down to 32x32 but the results were not good enough)</li><li>Transcode each version in PNG format and in JPEG format</li><li>Publish each of the versions to 3 Tensorflow servers at the same time</li><li>Display a comparison graph of the results</li></ul><p>The code for this app is <a href="https://github.com/ronhanson/tensorflow-image-resizer.git">here</a> and it is packaged as a docker image you can download with</p><pre>docker pull ronhanson/tensorflow-image-resizer </pre><h3>Deploying in Kubernetes</h3><h4>Deploying the model to nodes</h4><p>OK so I did not have the chance to have any storage available, and I did not want to have issues of bandwidth and IO for my demo. So I used hostPath to store the model (yes I know, shame on me)</p><p>You should be able to do something like</p><pre>$ scp -r model-data &lt;remote-instance&gt;:model-data</pre><p>for each of your worker nodes, then SSH into each of them and :</p><pre>remote-instance:$ sudo mv model-data /<br>remote-instance:$ exit</pre><p>OK at this stage you have your model shared on all the nodes. Obviously, if you are running in the cloud, use storage classes and publish the content into it. Message me if you are struggling with this step.</p><h4>Deploying TF</h4><p>In the src folder, you will find 3 manifests to deploy 3 Tensorflow Serving instances with a GPU, 1 CPU and finally 8 CPU cores. Adapt them to your needs (change the TF image and the constraints as needed) then deploy with</p><pre>kubectl create -f src/manifest-tensorflow-serving-gpu.yaml<br>kubectl create -f src/manifest-tensorflow-serving-1cpu.yaml<br>kubectl create -f src/manifest-tensorflow-serving-8cpu.yaml</pre><h4>Deploying the frontend</h4><p>You will find also in src a manifest to deploy the Python frontend to K8s. Note that for practical reasons we do the resizing server side in this example, which is CPU intensive. You may change the resource requests to improve response time. More on this in the conclusion.</p><p>Also note we are using a nodePort, also for practical reasons as we only run a bare metal cluster. You may change that to something more suited for your use case such as a Load Balancer for example.</p><pre>kubectl create -f src/manifest-tensorflow-image-resizer.yaml</pre><h3>Playing with our cluster</h3><p>Great!! Now you may connect on any of the nodes IP address on port 30501 (http://&lt;node-ip&gt;:30501) and look at the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xgggLy0-ZUhcHjyTbsBXxA.png" /><figcaption>analysis of a 24MP image of a cat</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1hIW365_Pu9U2Z1jCZiWg.png" /><figcaption>Analysis of a 14MP image (of another cat…)</figcaption></figure><h3>Conclusion</h3><p>To be honest, some of the conclusions we drew make sense, other are less intuitive. Note that they are findings made on the inception model, and you may have different findings on your own model. We believe this is easy enough to replicate that you can run (and share!!) your results and thoughts</p><ol><li><strong>Image size (hence bandwidth) has a very low impact on prediction accuracy</strong>. This means it is almost always beneficial to resize the image to 1024x1024 or even 512x512 before submitting it to the Tensorflow Serving Model Server. You lose less than 5% accuracy but get a 99% bandwidth saving!! <br>So, If you run a publicly available system, you may work on resizing the image client side to optimize your pipeline. This is easy enough with javascript (actually Ronan’s code has it disabled, but you can very easily switch it on)<br>If you run a datacenter or edge compute, with M2M only, then having a resizing/transcoding pipeline will improve overall performance and save bandwidth.</li><li><strong>Image Type DOES matter</strong>. You do not see that in the UI directly but you can collect the JSON output from the API, and notice that the same image with the same size is 4 to 5 times longer to analyze in PNG than it is in JPEG. This tells us that clearly compression is good, and that color space is probably not so much of a problem for the performance of the inception model. We would need more research and file types to nail this.</li><li><strong>Image size does not have so much impact on the duration of the prediction</strong>, at least once we start resizing. The raw sources we used (over 10M pixel each) would take a long time to run, but when we are lower than 4MP, the prediction duration is about constant. This one was pretty weird as I expected the prediction to be much much faster with “smaller” images. Well it is not.</li><li><strong>1x GPU (nVidia P4) is roughly equivalent to 8x CPU HT cores of the Intel Xeon</strong> once the images have been resized, but it is waaaaay better if you are looking at full scale ones. So I guess here it is a question of use case. <br>So, if you are looking for absolute precision, you will need a GPU to get a decent performance and use large images<br>If you are looking for speed at the expense of precision/accuracy, then a traditional high density approach with CPUs will probably make more sense.</li></ol><p>If you think we should run more use cases and tests, let us know (and PRs welcome).</p><p>If you are looking to run this in a real customer scenario, the below could help you improve performance:</p><ul><li>The <a href="https://github.com/tensorflow/serving/blob/master/tensorflow_serving/batching/README.md">Batch API for TF Serving</a> should be a great way to improve performance, especially with GPUs</li><li>ZenDesk released a paper about some in depth details <a href="https://medium.com/zendesk-engineering/how-zendesk-serves-tensorflow-models-in-production-751ee22f0f4b">here</a></li></ul><p>If you are looking into replicating and extending this post, note that the application has a full API which you can use to automate testing and processing the JSON output. Details are in the readme. You are welcome to test and publish:</p><ul><li>More compilation options</li><li>Other GPU performance metrics. P4 in our case was used because it has a low power requirement and a friendly form factor for the highly modular MSP platform.</li><li>K8s vs. other solutions? (in effect, all this can run outside of Docker and K8s, it was just easier for us to set it up that way)</li></ul><p>Anyway, we hope you liked this as much as we loved building it! Do not forget to clap for it!! And if you have other use cases or research, Ronan and myself are available to get you started on Kubernetes as well.</p><p>This demo will be presented at <a href="https://www.mobileworldcongress.com/exhibitor/kontron/">Mobile World Congress on Kontron&#39;s booth</a>. Come have a look and discuss!</p><h3>References</h3><p>You may find the code for this here:</p><ul><li>Python App: <a href="https://github.com/ronhanson/tensorflow-image-resizer/">https://github.com/ronhanson/tensorflow-image-resizer/</a></li><li>Blogpost and sources: <a href="https://github.com/madeden/blogposts/tree/master/k8s-tf-serving">https://github.com/madeden/blogposts/tree/master/k8s-tf-serving</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fef8d7502b9" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/fun-with-kubernetes-tensorflow-serving-4fef8d7502b9">Fun with Kubernetes &amp; Tensorflow Serving</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPUs & Kubernetes para Deep Learning — Part 3/3: Automatización del despliegue de Tensorflow]]></title>
            <link>https://medium.com/@samnco/gpus-kubernetes-para-deep-learning-part-3-3-303e448ced5c?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/303e448ced5c</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Tue, 24 Oct 2017 08:01:01 GMT</pubDate>
            <atom:updated>2018-01-19T13:10:50.829Z</atom:updated>
            <content:encoded><![CDATA[<p>Aquí estamos de nuevo. Tras haber pasado más o menos 21 minutos leyendo <a href="https://planetachatbot.com/gpus-kubernetes-para-deep-learning-part-1-3-90c506e0645e">el primer post</a> sobre cómo construir un clúster GPU Kubernetes en AWS, otros 7 minutos en <a href="https://planetachatbot.com/gpus-kubernetes-para-deep-learning-part-2-3-fe912b264875">agregar el almacenamiento EFS</a>, es hora de ver algo real, algo que verdaderamente podamos medir. Así que en este post vamos a definir, diseñar, desplegar y operar un canal completo de Deep Learning.</p><p>Entonces, ¿qué es exactamente un canal de Deep Learning? Bueno, mi definición de este concepto es el resultado de 4 pasos, con un posible bucle de retroacción, que consiste en:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*Awzt3mSWiRAYw-C9PpUNRw.png" /></figure><p><strong>Introducción de los datos:</strong> este paso es la acumulación y el procesamiento previo de los datos que se usarán para entrenar nuestro(s) modelo(s). Este paso, aunque tal vez sea el menos divertido, es uno de los más importantes. Tu modelo será tan bueno como los datos con los que lo has entrenado.</p><p><strong>Entrenamiento:</strong> esta es la parte donde juegas a ser Dios y creas la Inteligencia. A partir de tus datos, crearás un modelo que esperas sea representativo de la realidad y capaz de describirla (e incluso por qué no lo genera).</p><p><strong>Evaluación + Monitoreo:</strong> A menos que puedas demostrar que tu modelo es bueno, no vale la pena nada. La fase de evaluación tiene como objetivo medir la distancia entre tu modelo y la realidad. Esto puede ser devuelto a un ser humano para ajustar los parámetros. En las configuraciones avanzadas, esto puede ser básicamente tu prueba en CI / CD del modelo, y ayudar a sintonizar automáticamente el modelo incluso sin intervención humana.</p><p><strong>Servicio:</strong> es muy probable que desees actualizar el modelo de vez en cuando. Si deseas reconocer las amenazas en la red, por ejemplo, está claro que deberás aprender nuevas firmas de malware y patrones de comportamiento, o tendrás que cerrar tu empresa antes de lo que te gustaría. Servir es esta fase en la que expones el modelo para el consumo y asegurarte de que tus clientes siempre disfrutan del último modelo.</p><h4><strong>Ahora, ¿cuál es la mejor forma de modelar una aplicación en Kubernetes?</strong></h4><p>Un buen modelo debe ser fácil de reproducir y se reduce a como de bueno es tu ensamblaje.</p><p>No utilizarías Ubuntu si no hubiera una utilidad apreciable para instalar todas las cosas.</p><p>Tampoco Canonical gastaría dinero y esfuerzos considerables en Ubuntu Core y tus Contenedores de aplicaciones Snap si no creyera que, en última instancia, era un beneficio para la comunidad al acordar un formato de empaquetado de distribución múltiple.</p><p>Docker sería mucho menos exitoso, ya que si no resuelve un problema fundamental de compartir un terreno común entre los desarrolladores y los operadores.</p><p>El empaquetado a escala es un problema mucho más difícil que el empaquetado de SO único. En el caso de Canonical resuelven el empaquetado de la aplicación “mascota” a gran escala con Juju y charms. Y Kubernetes resuelve el problema de las aplicaciones a gran escala con Helm. ¡Hagamos esto para el aprendizaje profundo!</p><p>El marco de aprendizaje profundo que utilizaremos es Tensorflow, la solución DL de código abierto de Google. Como Google también abre fuentes Kubernetes, parece natural combinar estas 2 piezas juntas.</p><h3>El plan</h3><h4>¿Qué es lo que vamos a hacer?</h4><p>Vamos a reutilizar los bits y partes de los post anterior para centrarnos en el Deep Learning:</p><ol><li>Implementar Kubernetes con GPU. Solo tienes que leer <a href="https://planetachatbot.com/gpus-kubernetes-para-deep-learning-part-1-3-90c506e0645e">la primer parte.</a></li><li>Añadir EFS al CLuster. Lo tienes en<a href="https://planetachatbot.com/gpus-kubernetes-para-deep-learning-part-2-3-fe912b264875"> el segundo capítulo de esta trilogía</a>.</li><li>Introducir todos los datos y paquetar.</li><li>Código de entrenamiento y paquetar.</li><li>Código de evaluación y paquete.</li><li>Código y paquete de servicio.</li><li>Proceso de despliegue.</li><li>Conclusión.</li></ol><p>Así pues, puedes intuir que esto va a ser un proceso algo largo. Espero que tengas tiempo para poderlo completar. Cuando acabes el modelo resultante será algo similar a esta imagen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKhyhSFT9K1zbf6FtZm6SA.png" /></figure><h4>Requisitos</h4><p>Para llevar a cabo los pasos siguientes, es importante:</p><ul><li>Entender Kubernetes 101.</li><li>Comprender los conceptos básicos de Deep Learning y, en concreto, de Tensorflow.</li><li>Comprender o intentar entender los conceptos básicos del empaque Kubernetes con Helm.</li><li>Aunque te puede parecer sencillo, también necesitas un poco de conocimientos de escritura de Dockerfile.</li></ul><h4>Los preliminares</h4><ul><li>Asumo que has superado con éxito los pasos que expliqué en los dos post anteriores. Si no es así, revísalo o contáctame si tienes algún problema.</li><li>Asegúrate de tener instalado <a href="https://github.com/kubernetes/helm">Helm</a>. En el momento de escribir esto, estamos en 2.6.2</li></ul><pre>curl <a href="https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get">https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get</a> &gt; get_helm.sh<br>chmod 700 get_helm.sh<br>./get_helm.sh<br>helm init</pre><p>Si quieres instrucciones más detalladas puedes encontrarlas en los <a href="https://docs.helm.sh/using_helm/#installing-helm">documentos oficiales</a>.</p><ul><li>Copia este repositorio para tener acceso a todas las fuentes.</li></ul><pre>git clone <a href="https://github.com/madeden/blogposts">https://github.com/madeden/blogposts</a> ./<br>cd blogposts/k8s-dl</pre><ul><li>Necesitarás servir las <em>Charts</em> de Helm, que puedes hacer por:</li></ul><pre>cd src/charts<br>helm serve . &amp;</pre><h3>Ingest Pipeline</h3><h4>Introducción</h4><p>Toda buena historia de datos comienza con datos sin procesar, que posteriormente son cuidadosamente procesados y publicados en una ubicación conocida.</p><p>ImageNet es un conjunto de datos masivo de imágenes que se utiliza, entre otras cosas, para entrenar el modelo de Inception. Está hecho de una serie de archivos gestionados por imagenet.org, que no son compatibles con Tensorflow.</p><p>Google abrió el código fuente para descargar y procesar previamente ImageNet hace un tiempo aquí. Para automatizar la descarga, creamos una imagen de Docker simple <strong>src/docker/tf-models/Dockerfile</strong> que reutiliza la imagen oficial de tensorflow para extraer el conjunto de datos a <strong>${DATA_DIR}/imagenet</strong> .</p><p>Aquí es donde el volumen EFS que agregó anteriormente será útil, ya que esto tomará alrededor de 500 GB al final. Por cierto, en mis instancias de AWS, demoraron aproximadamente 3 días en descargarlo y procesarlo previamente, así que prepárate y paciente.</p><h4>Almacenamiento de empaque</h4><p>Para desplegar esto, necesitas:</p><ul><li>un volumen persistente (consulta el post anterior), de gran tamaño a 900 GB para acomodar los modelos;</li><li>una reclamación de volumen persistente (consulte el post anterior), sobredimensionada a 750 GB;</li><li>un constructo Kubernetes llamado Job, que es esencialmente una forma de ejecutar trabajos por lotes o cron en k8s, donde montamos nuestro PVC, y que usa la imagen descrita anteriormente.</li></ul><p>El proceso de construcción de un Chart de Helm es bastante simple: se resume toda la configuración, se colocan todos los valores en un archivo yaml y se muestran todas las claves en un conjunto de otras plantillas yaml que forman el esqueleto de Kubernetes. Agregas algunos metadatos a su alrededor y voilaaa. Puedes consultar la carpeta <strong>src/charts</strong> para tener una idea de lo que es.</p><blockquote><strong>Nota</strong>: si tuvieras que crear un Chart, podrías hacer `helm create foobar` y generarías el esqueleto.</blockquote><p>El primer cuadro que creamos es el chart EFS, que conectará nuestro almacenamiento EFS con Kubernetes. Creamos un <strong>storage.yaml</strong> archivo en src <strong>/charts/efs/templates</strong> con:</p><pre># Creating the PV first<br>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: {{ .Values.storage.name }}<br>  namespace: {{.Release.Namespace}}<br>  labels:<br>    heritage: {{.Release.Service | quote }}<br>    chart: &quot;{{.Chart.Name}}-{{.Chart.Version}}&quot;<br>    release: {{.Release.Name | quote }}<br>  annotations:<br>    &quot;helm.sh/created&quot;: {{.Release.Time.Seconds | quote }}<br>spec:<br>  capacity:<br>    storage: {{ .Values.storage.pv.capacity }}<br>  accessModes:<br>  - {{ .Values.storage.accessMode }}<br>  nfs:<br>    server: {{.Values.global.services.aws.efs.id}}.efs.{{.Values.global.services.aws.region}}.amazonaws.com<br>    path: &quot;/&quot;<br>---<br>kind: PersistentVolumeClaim<br>apiVersion: v1<br>metadata:<br>  name: {{ .Values.storage.name }}<br>  namespace: {{.Release.Namespace}}<br>  labels:<br>    heritage: {{.Release.Service | quote }}<br>    chart: &quot;{{.Chart.Name}}-{{.Chart.Version}}&quot;<br>    release: {{.Release.Name | quote }}<br>  annotations:<br>    &quot;helm.sh/created&quot;: {{.Release.Time.Seconds | quote }}<br>spec:<br>  accessModes:<br>  - {{ .Values.storage.accessMode }}<br>  resources:<br>    requests:<br>      storage: {{ .Values.storage.pvc.request }}</pre><p>Sí, reconociste las plantillas ¡Go! Ahora mira el archivo <strong>values.yaml</strong>:</p><pre>global:<br>  imagePullPolicy: IfNotPresent<br>  services:<br>    aws:<br>      region: us-east-1<br>      efs:<br>        id: fs-47cd610e</pre><pre>storage:<br>  name: tensorflow-fs<br>  accessMode: ReadWriteMany<br>  pv: <br>    capacity: &quot;900Gi&quot;<br>  pvc:<br>    request: &quot;750Gi&quot;</pre><p>Bastante fácil: reemplaza las sub secciones por puntos, aplica una convención de CamelCase en tus variables y tienes tu primera introducción a Helm. Si luego usa el Chart, previsualiza la salida con</p><pre>helm install efs --name release-name --namespace super-ns --debug --dry-run</pre><p>Generará un manifiesto en Kubernetes como:</p><pre># Creating the PV first<br>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: tensorflow-fs<br>  namespace: super-ns<br>  labels:<br>    heritage: Tiller<br>    chart: tensorflow<br>    release: release-name<br>  annotations:<br>    &quot;helm.sh/created&quot;: 12345678<br>    &quot;helm.sh/hook&quot;: pre-install<br>    &quot;helm.sh/resource-policy&quot;: keep<br>spec:<br>  capacity:<br>    storage: 900Gi<br>  accessModes:<br>  - ReadWriteMany<br>  nfs:<br>    server: fs-47cd610e.efs.us-east-1.amazonaws.com<br>    path: &quot;/&quot;<br>---<br>kind: PersistentVolumeClaim<br>apiVersion: v1<br>metadata:<br>  name: tensorflow-fs<br>  namespace: super-ns<br>  annotations:<br>    &quot;helm.sh/created&quot;: 12345678<br>    &quot;helm.sh/hook&quot;: pre-install<br>    &quot;helm.sh/resource-policy&quot;: keep<br>  labels:<br>    heritage: Tiller<br>    chart: tensorflow<br>    release: release-name<br>spec:<br>  accessModes:<br>  - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 750Gi</pre><p>¡¡Sencillo!! Si eliminas el <em>dry-run</em>, realmente lo implementarías, pero ahora no queremos hacerlo.</p><p>También puedes anular el archivo default <strong>values.yaml</strong> agregando:</p><pre>--values /path/to/my-values.yaml</pre><p>Ya puedes comenzar a construir uno propio copiando el mío y cambiando los valores para adaptarlo a tu propio entorno.</p><p>Obtienes el ejercicio Preparamos plantillas, tenemos un archivo de valor para instarlo. Helm mezcla ambos y genera un archivo YAML que funciona con Kubernetes. Cuando se instala el chart, un pequeño servidor llamado Tiller en el clúster de Kubernetes transmite el manifiesto instanciado a la API para su implementación (este servidor de Tiller se creó al ejecutar <em>helm init</em>). Puedes crear cualquier cosa que Kubernetes pueda operar. También puedes encontrar más información sobre la estructura y algunos consejos para comenzar a desarrollar en estos documentos.</p><h4>Ingestión de datos de empaque</h4><p>Creamos un chart de Helm llamado dataloader, que</p><ul><li>tienes un requisito en efs, expresado a través de un archivo requirements.yaml en la raíz del chart</li><li>Agregas un trabajo que descargará “algo” que puede configurar.</li></ul><p>Revise el contenido de <strong>src/charts/dataloader</strong> para comprender la estructura.</p><p>Ahora puedes usar Makefile en la carpeta de gráficos para crear este paquete,</p><pre>cd src/charts<br>make dataloader</pre><p>Luego puedes implementar eso en tu clúster y comenzar la descarga:</p><pre>helm install dataloader --name dataloader --values /path/to/my-values.yaml</pre><h4>¿Cómo adaptarlo tú mismo?</h4><p>La clave para personalizar esto es tu capacidad para preparar una imagen acoplable que tenga sentido en tu contexto.</p><p>Busca en el repositorio en la carpeta <strong>src/docker/tf-models</strong> un archivo llamado <strong>download.sh</strong> , y modifícalo según tus necesidades.</p><p>Constrúyelo con:</p><pre>./src/docker/tf-models<br>docker build --rm \<br>  -t &lt;username&gt;/&lt;name&gt;:&lt;version&gt; \<br>  -f Dockerfile-1.3.0 \<br>  .</pre><p>Tienes que tener en cuenta que, si es necesario, puedes usar la versión 0.11, que hice debido a algunos problemas de compatibilidad debido a la transición de transición entre 0.11 y 0.12 que aún no se han solucionado en 1.0.0 para la capacitación distribuida.</p><p>Luego empuja eso a tu repo:</p><pre>docker push &lt;username&gt;/&lt;name&gt;:&lt;version&gt;</pre><pre>storage:<br>  name: tensorflow-fs<br>  accessMode: ReadWriteMany<br>  pv: <br>    capacity: &quot;900Gi&quot;<br>  pvc:<br>    request: &quot;750Gi&quot;</pre><pre>dataLoader:<br>  service:<br>    name: dataloader<br>    command: &#39;[ &quot;/download.sh&quot; ]&#39;<br>  settings:<br>    dataset: flowers<br>    imagenetUsername: foo<br>    imagenetApiKey: bar<br>  image: <br>    repo: &lt;username&gt;<br>    name: &lt;name&gt;<br>    dockerTag: &lt;version&gt;</pre><p>Crea el EFS y todo lo que se necesita en AWS siguiendo las instrucciones de la parte anterior.</p><p>Ahora inicia un archivo my-values.yaml que contenga las secciones global, storage y dataLoader:</p><pre>global:<br>  imagePullPolicy: IfNotPresent<br>  services:<br>    aws:<br>      region: us-east-1<br>      efs:<br>        id: &lt;your EFS ID&gt;</pre><pre>storage:<br>  name: tensorflow-fs<br>  accessMode: ReadWriteMany<br>  pv: <br>    capacity: &quot;900Gi&quot;<br>  pvc:<br>    request: &quot;750Gi&quot;</pre><pre>dataLoader:<br>  service:<br>    name: dataloader<br>    command: &#39;[ &quot;/download.sh&quot; ]&#39;<br>  settings:<br>    dataset: flowers<br>    imagenetUsername: foo<br>    imagenetApiKey: bar<br>  image: <br>    repo: &lt;username&gt;<br>    name: &lt;name&gt;<br>    dockerTag: &lt;version&gt;</pre><p>Y publícalo en el cluster con:</p><pre>helm install dataloader --name dataloader --namespace default --values /path/to/my-values.yaml --debug</pre><p>Si lo has implementado anteriormente, puedes hacer lo siguiente:</p><pre>helm upgrade dataloader dataloader --values /path/to/my-values.yaml</pre><p>Eso es. Ahora se ha descargado un conjunto de datos. Puedes ir y tomar 42 cafés mientras lo hace, o continuar leyendo este blog para ir conocendo los asos que vas a tener que dar a continuación.</p><p>Por ahora, centremos en poner todos nuestros servicios en funcionamiento.</p><h3>Formación</h3><h4>Arquitectura</h4><p>La formación es a menudo la característica más anunciada de Deep Learning. Es donde ocurre la ciencia y la magia real. En todos los casos, el rendimiento puede aumentarse drásticamente si se aprovechan las GPU, que, por casualidad, hemos aprendido a agregar en un clúster de Kubernetes.</p><p>Cuando se ejecuta localmente en tu estación de trabajo, realmente no te importa la programación del código. Tensorflow intentará consumir todos los recursos disponibles. Si tienes varias GPU, puedes pedirle que las use todas o solo las que quieras.</p><p>Sin embargo, cuando comienzas a escalar tu sistema, las cosas se vuelven un poco más complicadas. Tensorflow usa 2 tipos de trabajos para distribuir computación: servidores de parámetros y workers</p><ul><li>Los servidores de parámetros almacenan y actualizan variables para otros nodos. En esencia, manejan el tráfico de “configuración”.</li><li>Los workers hacen la parte difícil y proporcionan computación para completar las tareas que reciben. A uno de los trabajadores se le otorga una función maestra específica, y está a cargo de la coordinación entre todos los trabajadores. Parece que esto siempre es predeterminado para worker-0 en lo que he visto hasta ahora.</li></ul><p>Tanto PS como los workers intentarán usar las GPU si están disponibles, y compiten por eso. Debido a que los PS tienen un uso menor de la GPU y generalmente son un recurso de alarma, tiene sentido tener cuidado al asignar recursos a las diferentes entidades de Tensorflow. En el metal de tu máquina, esto puede ser un problema. Pero los contenedores y la gestión de recursos nos ayudarán a lidiar con esto de manera elegante.</p><p>Una segunda cosa a tener en cuenta es una limitación del empaque (y probablemente mis propias limitaciones). Al construir el paquete, es imposible saber en qué workers se ejecutará cada trabajo y predecir la cantidad de GPU que estarán disponibles para Tensorflow, especialmente en los grupos asimétricos donde algunos nodos tienen 1x GPU, otros tienen 2, 8 o Más. Debido a que Kubernetes no es muy inteligente en esta etapa al exponer las GPU como si expusiera otros recursos como CPU o RAM, la única forma en que he podido hacer que el empaquetado funcione es asumir que todos los nodos son iguales.</p><p>Si seguiste el último post y recuerdas el hecho de que implementamos un p2.xlarge y un p2.8xlarge … Desafortunadamente, estaremos limitados por p2.xlarge y solo aprovecharemos 1 GPU en cada servidor.</p><p>Por último, pero no menos importante, Tensorflow no parece tener la capacidad de cambiar tu escala dinámicamente (sin reiniciar el clúster). Esto realmente hace nuestras vidas más fáciles cuando se trata de planificar la implementación.</p><p>Echemos un vistazo a las estructuras que necesitamos para diseñar nuestro cluster:</p><ul><li>La distinción entre PS y Worker requerirá una estructura condicional.</li><li>La gestión del tamaño variable requerirá iterar en una secuencia, como un bucle for. En Helm, esto usa una noción llamada rango.</li></ul><h4>Implementación — Despliegue para trabajos de Tensorflow</h4><p>Echemos un vistazo a src <strong>src/charts/tensorflow/values.yaml</strong> and <strong>src/charts/tensorflow/templates/cluster-deployment.yaml</strong> para analizar las diferentes secciones:</p><pre># Values File, section for the Tensorflow cluster<br>tfCluster:<br>  service:<br>    name: tensorflow-cluster<br>    internalPort: 8080<br>    externalPort: 8080<br>    type: ClusterIP<br>  image: <br>    repo: samnco<br>    name: tf-models<br>    dockerTag: train-1.3.0-gpu<br>  settings:<br>    isGpu: true<br>    nbGpuPerNode: 1<br>    jobs: <br>      ps: 8<br>      worker: 16</pre><p>Entonces, aquí en los valores, díganos qué tan grande será nuestro clúster (8 PS, 16 Trabajadores, 1 GPU por nodo).</p><p>En la plantilla de implementación, comenzamos por definir el archivo de configuración a través de un configMap:</p><pre># Defining a generic configuration file for the cluster<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: tensorflow-cluster-config<br>  namespace: {{.Release.Namespace}}<br>data:<br>  clusterconfig: &gt;<br>        {<br>        {{- range $job, $nb := .Values.tfCluster.settings.jobs }}<br>          {{ $job | quote }}: [<br>          {{ range $i, $e := until (int $nb) | default 8 }}<br>            &quot;{{ $job }}-{{$i}}.{{ $relname }}.svc.cluster.local:8080&quot;,<br>          {{ end }}<br>              ],<br>        {{- end }}<br>        }</pre><p>Puedes ver la sección {{range}}, donde recorreremos la sección de trabajos. Cuando instanciado, esto creará algo como:</p><pre>{<br>  &quot;ps&quot;: [<br>    &quot;ps-0.default.svc.cluster.local:8080&quot;,<br>    &quot;ps-1.default.svc.cluster.local:8080&quot;,<br>    ...<br>    &quot;ps-7.default.svc.cluster.local:8080&quot;,<br>  ],<br>  &quot;worker&quot;: [<br>    &quot;worker-0.default.svc.cluster.local:8080&quot;,<br>    ...<br>    &quot;worker-15.default.svc.cluster.local:8080&quot;,<br>  ],<br>}</pre><p>En este momento, te das cuenta que el JSON que formamos no está limpio, tenemos una última coma que Tensorflow aceptará, pero en la que debes tener cuidado.</p><p>Observa también que en la estructura de rango que iterar con un índice que comienza en 0 en una serie de valores, por lo tanto, pasamos de 0 a 7 para 8 trabajos PS.</p><p>Ahora, si vamos un poco más abajo en el archivo, podemos ver la sección condicional:</p><pre>{{ if eq $job &quot;worker&quot; }}<br>        {{ if $isGpu }}<br>        securityContext:<br>          privileged: true<br>        resources:<br>          requests:<br>            alpha.kubernetes.io/nvidia-gpu: {{ $nbGpu }}<br>          limits:<br>            alpha.kubernetes.io/nvidia-gpu: {{ $nbGpu }} <br>        volumeMounts:<br>          {{ range $j, $f := until (int $nbGpu) | default 1 }}<br>        - mountPath: /dev/nvidia{{ $j }}<br>          name: nvidia{{ $j }}<br>          {{ end }}<br>        {{ end }}<br>        {{ end }}</pre><p>si estamos construyendo un clúster de GPU, solo asignamos GPU a los procesos de Worker, y asignamos a cada worker todas las GPU a las que puede acceder en un nodo dado. Eso sí, no olvidamos compartir los dispositivos char necesarios (/dev/nvidia0 y más)</p><p>Nuestra imagen de formación se toma del tutorial de Inception que Google proporciona, que operamos a través de un simple programa de bash que lee el configMap, extrae el JSON adecuado e inicia el proceso de capacitación a partir de eso:</p><pre>bazel-bin/inception/${DATASET}_distributed_train \<br>  --batch_size=32 \<br>  --data_dir=${DATA_DIR}/${DATASET} \<br>  --train_dir=${DATA_DIR}/${DATASET}-checkpoints \<br>  --job_name=&quot;${JOB_NAME}&quot; \<br>  --task_id=${TASK_ID} \<br>  --ps_hosts=&quot;${PS_HOSTS}&quot; \<br>  --worker_hosts=&quot;${WORKER_HOSTS}&quot; \<br>  --subset=&quot;train&quot;</pre><p>Observarás que aplicamos el dir_datos al resultado de nuestra fase de ingesta, para garantizar que nuestra capacitación realmente pueda leer desde allí. El Dockerfile proporciona ideas para mejorar esta sección que están fuera del alcance de este artículo, pero puede contribuir ;)</p><h4>Implementación — Despliegue para servicios de trabajo de Tensorflow</h4><p>Para hacer que cada uno de estos trabajos sea accesible a través de Discovery Service, cada uno de ellos se asigna a tu propio Servicio Kubernetes, que está disponible en el archivo cluster-service.yaml .</p><p><strong>Implementación — Configuración</strong></p><p>La sección del archivo values.yaml es</p><pre>tfCluster:<br>  deploy: true<br>  service:<br>    name: tensorflow-cluster<br>    internalPort: 8080<br>    externalPort: 8080<br>    type: ClusterIP<br>  image: <br>    repo: samnco<br>    name: tf-models<br>    dockerTagCpu: train-0.11-0.0.8<br>    dockerTagGpu: train-0.11-gpu-0.0.8<br>  settings:<br>    dataset: imagenet<br>    isGpu: true<br>    nbGpuPerNode: 1<br>    jobs: <br>      ps: 8<br>      worker: 16</pre><p>Puedes ver en la sección de configuración cómo estamos agregando un número de GPUs previamente calculado por nodo y creando nuestros trabajos PS y Worker.</p><h4>¿Cómo adaptarlo tú mismo?</h4><p>De nuevo, esto es bastante sencillo. En este punto tendrás que preparar cualquier imagen de Docker para Tensorflow que se encargará de recibir:</p><ul><li>tu configuración de clúster desde un ENV denominado CLUSTER_CONFIG, en el formato del JSON descrito anteriormente</li><li>tu nombre de trabajo como ENV llamado POD_NAME en el formato &lt;nombre del trabajo&gt; — &lt;id de tarea&gt;</li></ul><p>Estos son los únicos requisitos realmente. Puedes preparar imágenes de CPU y GPU, publicarlas y adaptar la sección tfCluster de tu archivo values.yaml para que coincida con tu configuración.</p><h4>¿Por qué no implementamos ahora?</h4><p>¡Porque primero debemos revisar la evaluación y los procesos de servicio! ¡Ni siquiera estamos a medio camino!</p><h3>Evaluación y Monitoreo</h3><h4>Arquitectura</h4><p>La evaluación es un proceso específico, que consume GPU, y qué salida permite medir la calidad del modelo. Genera datos que pueden ser consumidos por la GUI de Tensorflow, la Tensorboard. Sin el tensorboard, es muy difícil para un ser humano entender cómo evoluciona el entrenamiento y si los parámetros están bien.</p><p>La tensor en sí es un proceso simple que expone una aplicación web para presentar los gráficos y los datos. Por lo tanto, necesitamos planificar un servicio y ser un punto de entrada para ello, para permitir que el mundo lo vea.</p><h4>Implementación</h4><p>Para mantener este blog tan breve como sea posible, te dejaré revisar el contenido de <strong>src/charts/tensorflow/eval-deployment.yaml</strong> and <strong>src/charts/tensorflow/tensorboard-service.yaml</strong> , que siguen al mismas ideas que las anteriores.</p><p>La sección values.yaml se ve así:</p><pre>evaluating:<br>  deploy: true<br>  replicaCount: 1<br>  image: <br>    repo: samnco<br>    name: tf-models<br>    dockerTagCpu: train-0.11-0.0.8<br>    dockerTagGpu: train-0.11-gpu-0.0.8<br>  service:<br>    name: eval<br>    command: &#39;[ &quot;/eval.sh&quot; ]&#39;<br>  settings:<br>    dataset: imagenet<br>    isGpu: true<br>    nbGpuPerNode: 1<br>  resources:<br>    requests:<br>      cpu: 1000m<br>      memory: 1Gi<br>    limits:<br>      cpu: 2000m<br>      memory: 4Gi</pre><pre>tensorboard:<br>  deploy: true<br>  replicaCount: 1<br>  image: <br>    repo: gcr.io/tensorflow<br>    name: tensorflow<br>    dockerTag: 1.0.0<br>  service:<br>    name: tensorboard<br>    dns: tensorboard.madeden.net<br>    type: ClusterIP<br>    externalPort: 6006<br>    internalPort: 6006<br>    command: &#39;[ &quot;tensorboard&quot;, &quot;--logdir&quot;, &quot;/var/tensorflow/imagenet-eval&quot; ]&#39;<br>  settings:<br>  resources:<br>    requests:<br>      cpu: 2000m<br>      memory: 4Gi<br>    limits:<br>      cpu: 4000m<br>      memory: 8Gi</pre><p>Notarás:</p><ul><li>La cantidad de cálculo dada a Tensorboard es alta. Incluso esto puede ser demasiado pequeño, encontré que este sistema consume mucho recursos. A menudo se cuelga o deja de responder si el modelo se mantiene entrenando durante mucho tiempo.</li><li>El hecho de que incluimos la GPU como una opción, como hicimos en el contexto del clúster.</li><li>El registro de DNS para el tensor (aquí tensorboard.madeden.net): Deberás preparar un round robin de DNS que apunte a las direcciones públicas de tus nodos de trabajo para poder acceder a esto. Mi recomendación es tener un registro genérico como workers.yourdomain.com luego un CNAME para el tensor.</li></ul><h4>¿Cómo adaptarlo tú mismo?</h4><p>Deberás preparar una secuencia de comandos eval.sh y / o una imagen de Docker específica que se encargará de la evaluación del modelo. Tendrás las siguientes propiedades:</p><ul><li>Leer archivos desde una ubicación predecible</li><li>Escribir evaluación desde una ubicación predecible</li><li>En nuestro contexto, comenzamos la evaluación con:</li></ul><pre>bazel-bin/inception/${DATASET}_eval \<br> --checkpoint_dir=&quot;${DATA_DIR}/${DATASET}-checkpoints&quot; \<br> --eval_dir=&quot;${DATA_DIR}/${DATASET}-eval&quot; \<br>    --data_dir=&quot;${DATA_DIR}/${DATASET}&quot; \<br> --subset=&quot;validation&quot;</pre><p>lo que significa que esperamos que la capacitación escriba en <strong>${DATA_DIR}/${DATASET}-checkpoints</strong>, la evaluación para escribir desde <strong>${DATA_DIR}/${DATASET}-eval </strong>y el conjunto de datos de evaluación para mentir en <strong>${DATA_DIR}/${DATASET}</strong></p><p>También necesitarás ajustar el comando Tensor de tabla para ajustar la ubicación de los registros.</p><h4>¿Por qué no implementamos ahora?</h4><p>¿Por qué eres tan impaciente? Solo necesitamos revisar el proceso de servicio y ya habremos terminado.</p><h3>Servicio</h3><h4>Arquitectura</h4><p>Servir es la capacidad de presentar el modelo capacitado para el consumo por parte de terceros usuarios. Este proceso, que se puede hacer a través de tensorflow-serving, consume el resultado del proceso de entrenamiento, que almacenamos en los puntos de control de imágenes, y permite consultarlo a través de una API.</p><p>Cualquiera puede enviar una imagen y obtener un archivo JSON como respuesta, que contiene las cosas que se han identificado y la probabilidad de que el modelo sea correcto sobre ellas.</p><p>Servir un modelo no requiere mucha potencia de procesamiento y puede ejecutarse en una instancia de CPU razonable. Sin embargo, debe poder escalar mucho horizontalmente, especialmente si el modelo se sirve a numerosos clientes. Por lo tanto, vamos a adoptar un enfoque de implementación y servir al modelo a través de un servicio y un punto de ingreso.</p><p>En el lado del estibador, esto también es un poco complicado. El servicio Tensorflow es muy nuevo y menos maduro que el resto del ecosistema TF. La imagen de docker oficial proporcionada por Google es muy intrincada, y reconstruye Bazel en la compilación de imágenes, que es subóptima. Hice una imagen secundaria que se construye a partir de la imagen de flujo de tensor oficial, agrego los binarios de bazel y luego instalo la parte de servicio. Un poco más rápido para construir, especialmente después de construir todas las demás imágenes del repositorio.</p><p>Debido a que el modelo evoluciona con el tiempo, tenemos que agregar algunos parámetros al comando de lanzamiento, que transpiran a través del ENV que establecemos:</p><pre>/serving/bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \<br>    --port=${PORT}\<br>    ${BATCHING} \<br>    --model_name=${MODEL_NAME} \<br>    --model_base_path=${MODEL_PATH}</pre><p>En este punto, la clave es la configuración enable_batching, y tendrás TF Serving para volver a cargar el modelo en una base de cron, por defecto cada 5 minutos. Como también mantenemos el valor predeterminado en la capacitación, que es exportar un nuevo punto de control cada 10 minutos, esto garantiza que siempre se sirva el último modelo.</p><h4>Implementación</h4><p>La implementación se realiza a través de los archivos <strong>src/charts/tensorflow/serving-deployment.yaml</strong> and <strong>src/charts/tensorflow/serving-service.yaml</strong>, que te dejaré revisar. También puede revisar la parte de valores de serving.yaml para comprender la configuración.</p><p>El archivo values.yaml merece una revisión:</p><pre>serving:<br>  deploy: true<br>  replicaCount: 3<br>  image: <br>    repo: samnco<br>    name: tf-models<br>    dockerTag: serving-1.3.0<br>  service:<br>    name: serving<br>    dns: inception.madeden.net<br>    type: ClusterIP<br>    externalPort: 8500<br>    internalPort: 8500<br>    command: &#39;[ &quot;/serve.sh&quot; ]&#39;<br>  settings:<br>    dataset: imagenet<br>  resources:<br>    requests:<br>      cpu: 50m<br>      memory: 256Mi<br>    limits:<br>      cpu: 200m<br>      memory: 512Mi</pre><p>aquí nuevamente tenemos un registro de DNS para administrar, que se espera que agregue como un CNAME si sigue el consejo anterior, o un turno redondo a todos los trabajadores de lo contrario.</p><h4>¿Cómo adaptarlo tú mismo?</h4><p>Deberás preparar un script serve.sh y una imagen específica que se encargará de la publicación del modelo, que puede basarse en la publicación Dockerfile-1.3.0 proporcionada. Tendrás las siguientes propiedades:</p><ul><li>Leer archivos desde una ubicación predecible</li><li>Tener un nombre de modelo predecible</li></ul><p>En nuestro contexto, comenzamos la evaluación con:</p><pre>/serving/bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \<br>    --port=${PORT}\<br>    ${BATCHING} \<br>    --model_name=${MODEL_NAME} \<br>    --model_base_path=${MODEL_PATH}</pre><p>Lo cual te da una pista de qué tienes que modificar en tu contexto.</p><p>Y sí … ¡Ahora desplegamos!</p><h3>¡Despliegue!</h3><h4>Implementación de Tensorflow</h4><p>¡¡Al final!!</p><p>Supongamos que tienes:</p><ul><li>Preparado tu archivo my-values.yaml</li><li>Agregado tensorboard.yourdomain.com y los registros de DNS serving.yourdomain.com apuntando a los trabajadores;</li></ul><p>Entonces puedes desplegar con:</p><pre>make tensorflow<br>helm install tensorflow --name tensorflow --values /path/to/my-values.yaml --debug</pre><p>y mira cómo comienza tu clúster. Después de unos minutos, puedes ir a tu tensor y deberías ver:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XIgYTuL5qf3p1PKUiPLbaQ.png" /><figcaption>Interface de Kubernetes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bkrOc9D4IJG9_Deel9vRVA.png" /></figure><h3>Unas pocas palabras antes de concluir</h3><p>Para preparar este blog trabajé en 2 modelos: Imagenet y una CNN distribuida de un taller que Google realizó el pasado mes de agosto. The Distributed CNN es bueno porque usa un pequeño conjunto de datos, por lo tanto, funciona muy bien y rápidamente OOTB.</p><p>Imagenet es el que realmente me hubiera gustado ver funcionar, y todas las imágenes están destinadas a aprovecharla. Desafortunadamente en esta etapa, todo comienza bien, pero parece que no se entrena realmente. PS y los trabajadores comienzan, luego no hacen nada sin fallar, y no generan ningún registro. Estoy trabajando en eso, pero no quería que esperes demasiado para la publicación …</p><p>Ponte en contacto conmigo en PM para discutir y si quieres experimentar con él o compartir ideas para solucionarlo, con mucho gusto mencionaré tu ayuda :)</p><p>Además, si usas las imágenes Tensorflow 1.0.0 para Imagenet, te encontrarás con un problema debido a la transición de 0.11 a 0.12, que esencialmente devuelve algo como:</p><pre>$ kubectl logs worker-0-2134621739-2qd0j<br>Extracting Bazel installation...<br>.........<br>____Loading package: inception<br>...<br>____Found 1 target...<br>____Building...<br>____[0 / 1] BazelWorkspaceStatusAction stable-status.txt<br>____Building complete.<br>...<br>I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally<br>INFO:tensorflow:PS hosts are: [&#39;ps-0.default.svc.cluster.local:8080&#39;]<br>INFO:tensorflow:Worker hosts are: [&#39;worker-0.default.svc.cluster.local:8080&#39;, &#39;worker-1.default.svc.cluster.local:8080&#39;]<br>...<br>I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: <br>name: Tesla K80<br>major: 3 minor: 7 memoryClockRate (GHz) 0.8235<br>pciBusID 0000:00:1e.0<br>Total memory: 11.17GiB<br>Free memory: 11.11GiB<br>I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 <br>I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y <br>I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0)<br>I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job ps -&gt; {0 -&gt; ps-0.default.svc.cluster.local:8080}<br>I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; localhost:8080, 1 -&gt; worker-1.default.svc.cluster.local:8080}<br>I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:221] Started server with target: grpc://localhost:8080<br>Traceback (most recent call last):<br>  File &quot;/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/inception/inception/imagenet_distributed_train.py&quot;, line 65, in &lt;module&gt;<br>    tf.app.run()<br>...<br>TypeError: __init__() got multiple values for keyword argument &#39;dtype&#39;</pre><p>Si tiene este problema, use una imagen 0.11 o anterior.</p><blockquote>Nota: desde la escritoria de este post, Tensorflow ha evoluado hasta 1.3.0. No he tenido tiempo para probar de nuevo con las ultimas versiones. Ya me diras algo si tienes suerte al despliegue!</blockquote><h3>Conclusión</h3><p>Otra publicación de blog de looong. Pero creo que valió la pena. En resumen, en este post has llevado a cabo los siguientes pasos:</p><ol><li>Implementar Kubernetes 1.7.x utilizando la Distribución Canónica de Kubernetes (CDK).</li><li>Las cargas de trabajo activadas de la GPU en él</li><li>Almacenamiento adicional</li><li>Tensorflow modelado en un Charts de Helm escalable y fácil de operar</li><li>Implementado ese Chart de Helm</li><li>Aprender a reproducir esto por ti mismo</li></ol><p>En esta etapa, deberías poder usar Kubernetes y Tensorflow para tus propios proyectos. Realmente espero que sí, y si no, solo escríbeme e intentaremos solucionarlo.</p><p>Grande, enorme gracias a Amy, quien publicó el taller inicial que inspiró este gráfico de Helm, y a las otras personas de Google, que abren el suministro de tanto contenido. Esto es increíble, y espero que ayude a otros a adoptar la tecnología.</p><p>¿Tiene una pregunta, una solicitud, un informe de error o desea operar un clúster de GPU? tuitame @SaMnCo_23 y hablemos.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fde58d5%3Fas_embed%3Dtrue&amp;url=https%3A%2F%2Fupscri.be%2Fde58d5&amp;image=https%3A%2F%2Fe.enpose.co%2F%3Fkey%3DdRXnS9Gplk%26w%3D700%26h%3D425%26url%3Dhttps%253A%252F%252Fupscri.be%252Fde58d5%252F%253Fenpose&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1342d9242d958b58363dfe521f61b02/href">https://medium.com/media/f1342d9242d958b58363dfe521f61b02/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MMX82VX3T3Ts2Sae6-toyw.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=303e448ced5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPUs & Kubernetes para Deep Learning — Part 2/3]]></title>
            <link>https://medium.com/@samnco/gpus-kubernetes-para-deep-learning-part-2-3-fe912b264875?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe912b264875</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[español]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[tecnologia]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Tue, 17 Oct 2017 08:33:20 GMT</pubDate>
            <atom:updated>2017-10-17T08:33:20.699Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AwGb23K0GhjwsLuTkNfCCA.jpeg" /></figure><p>En <a href="https://planetachatbot.com/gpus-kubernetes-para-deep-learning-part-1-3-90c506e0645e">el primer post de esta saga</a>, construimos un clúster de GPU e instalamos Kubernetes para que pudiéramos hacer un procesamiento de datos avanzado.</p><p>Y, ¿qué es lo que necesitas justo después de tener las GPU’S? Datos, Datos y más datos. Y técnicamente, si has echado un vistazo a cualquiera de los tutoriales para Tensorflow o las publicaciones recientes del blog PaddlePaddle, habrás visto que cada uno de ellos comienza con la ejecución de un trabajo de Kubernetes para descargar y preparar un conjunto de datos.</p><p>No se puede almacenar en contenedores directamente, pero hay muchas maneras de proporcionar una capa de almacenamiento a las Kubernetes pods:</p><ul><li>En las opciones locales: Ceph RDB, GlusterFS, Fibre Channel, iSCSI, vSphere Volume, volúmenes Quobyte, VMWare Photon, o simplemente NFS (y probablemente habrá más opciones que estoy olvidando).</li><li>En la nube: cualquiera de las primitivas de bloque de las nubes como AWS EBS, Discos Persistentes de Google, Almacenamiento Azure (Archivo o Disco).</li><li>Ambos: HostPath en los disco duros de cada uno de los nodos.</li></ul><p>En junio de 2016, AWS creó EFS (Elastic File System) que ya está disponible al publico. EFS es una capa de almacenamiento compatible con NFS4, que puede ser fácilmente aprovechada para proporcionar PVs multiReadWrite a pods de Kubernetes. Como es elástico, es una opción muy agradable para el almacenamiento de datasets, tamaño que no siempre se puede predecir cuándo se alojan de forma remota. Puede escalar hasta 52TB para un solo archivo. Y no requiere funky IAM Roles para conectarse a él, sólo un cliente NFS y algunas reglas que puedes aplicar después de la implementación. Por la parte negativa no es el sistema de almacenamiento el màs rapido del mundo.</p><p>TL;DR: vale la pena intersarse en EFS. Aún más en nuestro contexto de Kubernetes para la ciencia.</p><p>Vamos a ver cómo podemos agregar Elastic File System a nuestro clúster de GPU para esta parte 2, antes de llegar a los resultados finales de la parte 3. Podríamos simplemente copiar pegar un tuto aquí, pero, por lo que sé, todos lo hacen con la interfaz de usuario, que tarda siglos, y no puede ser automatizado. Quiero automatizar todas las tareas.</p><h3><strong>El plan a seguir</strong></h3><p>En este post vas a:</p><ul><li>Reutilizar el clúster que desplegamos la última vez.</li><li>Agregar mediante programación un sistema de archivos EFS y puntos de montaje a sus nodos.</li><li>Comprobar que funciona agregando un PV / PVC en k8s.</li></ul><h3><strong>Requisitos</strong></h3><p>Si quieres continuar desarrollando esto vas a necesitar:</p><ul><li>Entender Kubernetes 101.</li><li>Tener credenciales de administrador para AWS.</li><li>Tienes que <a href="https://planetachatbot.com/gpus-kubernetes-para-deep-learning-part-1-3-90c506e0645e">leer la Parte 1</a> y tener el grupo preparado al alcance de su mano.</li><li>Para comer json y para el desayuno jq. Cada día.</li></ul><h3><strong>Preliminar</strong></h3><p>No tengo que añadir nada en este punto, solo tienes que ir al post anterior y ojear los puntos en los que tengas dudas.</p><p>Solo eso. Copia la publicación la última vez, aquí está la carpeta para esta sección:</p><pre>cd ../k8s-using-efs</pre><p>¿Listo? Pues pongámonos manos a la obra.</p><h3><strong>Adición y conexión de un backend de almacenamiento EFS</strong></h3><h4><strong>Requisito previo</strong></h4><p>Para administrar EFS, vas a necesitar:</p><ul><li>La región donde se despliega. En nuestro caso, us-east-1</li><li>El VPC en el que se encuentra. Juju utiliza el VPC predeterminado de forma predeterminada, por lo que puede buscarlo en AWS GUI o si no ha implementado más VPCs con</li></ul><pre>REGION=us-east-1<br>VPC_ID=$(aws --region us-east-1 ec2 describe-vpcs \<br>    | jq --raw-output &#39;.[][].VpcId&#39;)<br># vpc-b4ce2bd1</pre><ul><li>Todas las subredes con las instancias desplegadas en</li></ul><pre>SUBNET_IDS=$(aws --region us-east-1 ec2 describe-subnets \<br>    | jq --raw-output &#39;.[][].SubnetId&#39;)<br># subnet-26300e52<br># subnet-418dea7b<br># subnet-0cc0984a<br># subnet-645b204c</pre><ul><li>El grupo de seguridad para permitir el acceso desde</li></ul><pre>SG_ID=$(aws --region us-east-1 ec2 describe-instances \<br>    | jq --raw-output &#39;.[][].Instances[] \<br>        | select( .Tags[].Value \<br>        | contains (&quot;juju-default-machine-0&quot;)) \<br>        | .SecurityGroups[1].GroupId&#39;)<br># sg-bc4101c0</pre><p>En este punto, engañamos un poco. Para cada modelo que crees, Juju creará un Grupo de Seguridad, el cual será nombrado con el formato juju- &lt;uuid&gt; donde uuid es un UUID de 32 caracteres generado aleatoriamente. Entonces cada máquina heredará un grupo de seguridad secundario con el formato juju- &lt;uuid&gt; — &lt;id&gt; , donde uuid es el mismo que el SG genérico, e id es el ID de máquina en el modelo.</p><p>Como resultado, inmediatamente después de un despliegue, cada máquina Juju tendrá sólo 2 SG. Utilizamos esta propiedad y el método de clasificación predeterminado de jq para extraer el valor correcto.</p><p>En realidad, ahora que hablamos de ello, hay una característica bastante desconocida pero útil en Juju: se puede personalizar la etiqueta de sus instancias (por lo tanto, hacer lo anterior mucho más simple)</p><p>OK, ya tenemos los datos necesarios, vamos a crear nuestro FS.</p><h4><strong>Creación del EFS</strong></h4><p>Crea y guarda el ID de EFS con:</p><pre>EFS_ID=$(aws canonical --region us-east-1 efs create-file-system --creation-token $(uuid) \<br>    | jq --raw-values &#39;.FileSystemId&#39;<br># fs-69de7c20<br># Note the full output: <br># {<br>#     &quot;CreationToken&quot;: &quot;f2513790-f2c8-11e6-a002-9735d97703bd&quot;,<br>#     &quot;LifeCycleState&quot;: &quot;creating&quot;,<br>#     &quot;OwnerId&quot;: &quot;131768076974&quot;,<br>#     &quot;FileSystemId&quot;: &quot;fs-69de7c20&quot;,<br>#     &quot;NumberOfMountTargets&quot;: 0,<br>#     &quot;PerformanceMode&quot;: &quot;generalPurpose&quot;,<br>#     &quot;CreationTime&quot;: 1487085556.0,<br>#     &quot;SizeInBytes&quot;: {<br>#         &quot;Value&quot;: 0<br>#     }<br># }</pre><p>Ahora necesitas crear puntos de montaje para cada una de las subredes en las que tienes instancias:</p><pre>for subnet in ${SUBNET_IDS}<br>do <br>    aws --region us-east-1 efs create-mount-target \<br>        --file-system-id ${EFS_ID} \<br>        --subnet-id ${subnet} \<br>        --security-groups ${SG_ID}<br>done<br># {<br>#     &quot;LifeCycleState&quot;: &quot;creating&quot;,<br>#     &quot;IpAddress&quot;: &quot;172.31.24.140&quot;,<br>#     &quot;OwnerId&quot;: &quot;131768076974&quot;,<br>#     &quot;NetworkInterfaceId&quot;: &quot;eni-8c0bb86c&quot;,<br>#     &quot;MountTargetId&quot;: &quot;fsmt-d900b590&quot;,<br>#     &quot;SubnetId&quot;: &quot;subnet-26300e52&quot;,<br>#     &quot;FileSystemId&quot;: &quot;fs-69de7c20&quot;<br># }<br># {<br>#     &quot;OwnerId&quot;: &quot;131768076974&quot;,<br>#     &quot;NetworkInterfaceId&quot;: &quot;eni-265845ce&quot;,<br>#     &quot;SubnetId&quot;: &quot;subnet-418dea7b&quot;,<br>#     &quot;FileSystemId&quot;: &quot;fs-69de7c20&quot;,<br>#     &quot;LifeCycleState&quot;: &quot;creating&quot;,<br>#     &quot;MountTargetId&quot;: &quot;fsmt-da00b593&quot;,<br>#     &quot;IpAddress&quot;: &quot;172.31.63.172&quot;<br># }<br># {<br>#     &quot;IpAddress&quot;: &quot;172.31.6.243&quot;,<br>#     &quot;NetworkInterfaceId&quot;: &quot;eni-8d930649&quot;,<br>#     &quot;SubnetId&quot;: &quot;subnet-0cc0984a&quot;,<br>#     &quot;FileSystemId&quot;: &quot;fs-69de7c20&quot;,<br>#     &quot;OwnerId&quot;: &quot;131768076974&quot;,<br>#     &quot;MountTargetId&quot;: &quot;fsmt-dc00b595&quot;,<br>#     &quot;LifeCycleState&quot;: &quot;creating&quot;<br># }<br># {<br>#     &quot;SubnetId&quot;: &quot;subnet-645b204c&quot;,<br>#     &quot;FileSystemId&quot;: &quot;fs-69de7c20&quot;,<br>#     &quot;NetworkInterfaceId&quot;: &quot;eni-17f676e5&quot;,<br>#     &quot;OwnerId&quot;: &quot;131768076974&quot;,<br>#     &quot;IpAddress&quot;: &quot;172.31.44.34&quot;,<br>#     &quot;LifeCycleState&quot;: &quot;creating&quot;,<br>#     &quot;MountTargetId&quot;: &quot;fsmt-df00b596&quot;<br># }</pre><p>Ahora sí, ya estás listo para añadir el almacenamiento en Kubernetes. Antes de eso, veamos cómo lo consumiríamos de una máquina existente o nueva que controlamos.</p><h4><strong>Consumir de un nodo diferente</strong></h4><p>Para consumir EFS desde un nodo en una de las subredes anteriores debe realizar (en el nodo):</p><pre>sudo apt install -yqq nfs-common<br>mkdir efs<br>sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 EFS_SERVICE_HOST:/ efs</pre><p>donde EFS_SERVICE_HOST es $ {FS_ID}. $ {REGION}. amazonaws.com (en nuestro ejemplo fs-69de7c20.us-east-1.amazonaws.com)</p><p>Eso es todo, no muy difícil. Veamos ahora para CDK .</p><h3><strong>Conexión de EFS y CDK</strong></h3><p>Para consumir almacenamiento como EFS en Kubernetes, se necesitan 2 primitivas:</p><ul><li><strong>Volúmenes Persistentes</strong> (PV), que definen grandes asignaciones de almacenamiento (red) que se pueden dividir en subvolúmenes separados. Piense en esto como un disco que usted podría particionar esencialmente. Los PVs son administrados por el administrador.</li><li><strong>Reivindicaciones de volumen persistente</strong> (PVC): se trata de un fragmento / partición de un PV que se solicita para ser asignado a uno o más pods. El PVC es consumido por los usuarios del cluster.</li></ul><p>Los PVC pueden tener diferentes propiedades de lectura / escritura:</p><ul><li><strong>ReadOnlyMany</strong>: como secretos, puede ser leído por muchos pods, pero nadie puede escribir.</li><li><strong>ReadWriteOnce</strong>: sólo una pod puede escribir, pero muchos pueden leer. Por ejmplo, si tienes un despliegue maestro / agente.</li><li><strong>ReadWriteMany</strong> : todo el mundo puede leer y escribir. Si tienes que intercambiar archivos entre diferentes servicios, esto es una forma sencilla de hacerlo.</li></ul><p>Un manifiesto para un PV tiene más o menos esta apariencia:</p><pre>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: efs-volume<br>spec:<br>  capacity:<br>    storage: 100Gi<br>  accessModes:<br>    - ReadWriteMany<br>  nfs:<br>    server: EFS_SERVICE_HOST<br>    path: &quot;/&quot;</pre><p>donde EFS_SERVICE_HOST es $ {FS_ID}. $ {REGION} .amazonaws.com . Aquí podemos ver que este PV utiliza la primitiva NFS para consumir EFS.</p><p>Afortunadamente en la distribution de Kubernetes de Canonical , todos los nodos de trabajo tienen nfs-common (asi tambien que ceph-client) instalado de forma predeterminada, por lo tanto, esto funciona fuera de la caja, no tiene que preocuparse por ello.</p><p>Puede crear este PV con el clásico:</p><pre>kubectl create -f src/pv.yaml</pre><p>Un PVC consumiendo esto podría ser:</p><pre>kind: PersistentVolumeClaim<br>apiVersion: v1<br>metadata:<br>  name: efs-volume<br>spec:<br>  accessModes:<br>    - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 50Gi</pre><p>Nuestro PVC solicita 50GB de los 100 que hemos provisto anteriormente. Observe el vocabulario o solicitando recursos en el PVC, mientras que el PV tiene una capacidad dada.</p><p>Puede crear este PVC con</p><pre>kubectl create -f src/pvc.yaml</pre><p>En este momento, lo que tienes en su grupo es la capacidad de los pods para reclamar este PVC, para, a continuación, acceder a leer / escribir cosas.</p><h4><strong>Consumir el PVC</strong></h4><p>Los PVC se consumen como puntos de montaje por pods. A continuación, se muestra un ejemplo de trabajo que tiene un pod consumiendo un PVC:</p><pre>apiVersion: batch/v1<br>kind: Job<br>metadata:<br>  name: data-consumer<br>spec:<br>  template:<br>    metadata:<br>      name: pi<br>    spec:<br>      containers:<br>      - name: data-consumer<br>        image: super-container:latest<br>        imagePullPolicy: Always<br>        volumeMounts:<br>        - mountPath: &quot;/efs&quot;<br>          name: efs<br>      volumes:<br>        - name: efs<br>          persistentVolumeClaim:<br>            claimName: efs-volume<br>      restartPolicy: Never</pre><p>A partir de ahí, voy a dejar ir su imaginación. Un poco de ayuda: Kubernetes tiene los pods para definir las aplicaciones, y se asegura de que las vainas se mantienen vivos si lo desea.</p><p>Pero también tiene una noción de J<em>obs</em>. Los trabajos son esencialmente la ejecución una vez de Mesos: los trabajos por lotes que se ejecutan una vez o como cron. Una práctica al escribir Paquetes Helm o talleres para Kubernetes es definir un trabajo que implementa un pod que descarga datos a su EFS.</p><p>Cuando se hace, el verdadero trabajo puede comenzar. Pero eso es para la próxima vez.</p><h3><strong>Conclusión</strong></h3><p>Este fue un post bastante corto, pero será instrumental para lo que viene en las próximas semanas.</p><p>¿Qué aprendimos?</p><ul><li>Implementar un clúster de dev simple en AWS en unos pocos comandos (nada nuevo aquí para los veteranos!)</li><li>Automatizar la creación y conexión de un volumen EFS desde la CLI. Ahora puedes escribir esto y crear nuevos en segundos.</li><li>Consumir EFS de la distribución canónica de Kubernetes (y en realidad de cualquier Kubernetes)</li></ul><p>¿Algo especial sobre esto? Sí.</p><ul><li>EFS se presenta como un punto final NFS, por lo que sigue siendo extremadamente portátil en las implementaciones. CDK en las instalaciones utiliza Ceph como un backend, que también tiene una opción NFS a través de la Rados GW. No necesitarás cambiar una línea de tu manifiesto para pasar de la nube pública a las instalaciones.</li><li>EFS no requiere que las funciones de IAM se consuman de instancias (por ejemplo, S3 requiere credenciales o un rol de IAM), por lo que es extremadamente fácil de consumir. Hay un caso de uso especialmente que es clave. No es posible agregar una función IAM a una instancia existente, sólo puedes modificar una existente. EFS evita este problema potencial y puede utilizarse en una infraestructura pre existente.</li><li>Vimos que el PV tiene un tamaño, pero no fijamos nada en la definición de EFS. EFS escalará (aproximadamente) indefinidamente, este límite es una propiedad de Kubernetes para los PVs.</li></ul><p>Por el momento, esto es todo. En un futuro próximo, vamos a mezclar este post, las GPU y empezar a investigar varios marcos de aprendizaje profundo en Kubernetes.</p><p>¡Mantente al tanto!</p><h3><strong>Derribando</strong></h3><p>Como se ve en el post anterior, siempre que lo desee, puede eliminar este clúster con:</p><pre>juju kill-controller aws/us-east-1</pre><p>Esto te pedirá confirmación y luego destruirá todo … Pero ahora, estás a solo unos comandos y un café de reconstruirlo, y un pequeño guión de añadir almacenamiento de nuevo a él. Así que no es un problema, hay soluciones.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe912b264875" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPUs & Kubernetes para Deep Learning — Part 1/3]]></title>
            <link>https://medium.com/@samnco/gpus-kubernetes-para-deep-learning-part-1-3-90c506e0645e?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/90c506e0645e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[español]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Tue, 10 Oct 2017 08:01:14 GMT</pubDate>
            <atom:updated>2017-10-10T08:01:14.016Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gR6Vk9-bHksqXyuxD5-Sng.jpeg" /></figure><p>Hace unas semanas compartí un proyecto paralelo sobre la construcción de un <a href="https://hackernoon.com/installing-a-diy-bare-metal-gpu-cluster-for-kubernetes-364200254187">clúster DYI GPU</a> para k8s para jugar con Kubernetes con un ROI adecuado contra instancias AWS g2.</p><p>Esto fue espectacularmente interesante cuando AWS se quedó atrás con las viejas tarjetas nVidia K20s (que ya no son compatibles con los últimos controladores). Pero con la adición de la <a href="https://aws.amazon.com/ec2/instance-types/#p2">serie P</a> (p2.xlarge, 8xlarge y 16xlarge) las nuevas tarjetas son K80s con 12GB de RAM, y, en consecuencia, escandalosamente más potentes que las anteriores.</p><p>Baidu acaba de publicar un post en el blog de Kubernetes sobre la configuración de PaddlePaddle, pero sólo se centraron en las CPU. Pensé que sería interesante ver una configuración de Kubernetes en AWS añadiendo algunos nodos GPU, y luego ejercer un marco de aprendizaje profundo en él. Los documentos determinan que es posible…</p><p>Este post es el primero de la trilogía: Configurar el clúster de GPU (este blog), añadir almacenamiento a un grupo de Kubernetes (justo después) y, finalmente, ejecutar un entrenamiento de aprendizaje profundo en el clúster (trabajando en él, …).</p><p><strong>El plan a seguir:</strong></p><p>En este post vamos a:</p><ol><li>Implementar k8s en AWS en un modo de desarrollo (no HA, colocating etcd, el plano de control y PKI)</li><li>Implementación de nodos 2x con GPUs (p2.xlarge y p2.8xlarge instances)</li><li>Implementación de nodos 3x con CPU solamente (m4.xlarge)</li><li>Validar la disponibilidad de la GPU</li></ol><h3><strong>Se necesita:</strong></h3><p>Para todos aquellos que quieran seguir leyendo, es importante tener en cuenta que:</p><ol><li>Se poseen conocimientos de Kubernetes 101</li><li>Tiene credenciales de administrador para AWS</li><li>Si siguió las otras publicaciones, sabrá que utilizaremos la <a href="https://www.ubuntu.com/cloud/kubernetes">distribución de Kubernetes de Canonical</a> (la empresa que publica Ubuntu), por lo tanto, algunos conocimientos sobre Ubuntu, Juju y el resto del ecosistema de Canonical ayudarán.</li></ol><h3><strong>Pasos previos</strong></h3><p>Asegúrate de que tienes instalado Juju.</p><p>En Ubuntu,</p><pre>sudo snap install juju --classic</pre><p>Para otras cuestiones te recomiendo que vayas a los <a href="https://jujucharms.com/docs/2.0/getting-started-general">documentos oficiales</a>.</p><p>A continuación, para conectarse a la nube AWS con sus credenciales, lea <a href="https://jujucharms.com/docs/2.0/help-aws">esta página</a></p><p>- Finalmente copia este código y tendrás acceso a todas las fuentes.</p><pre>git clone <a href="https://github.com/madeden/blogposts">https://github.com/madeden/blogposts</a> ./<br>cd blogposts/k8s-gpu-cloud</pre><p>¡DE ACUERDO! Comencemos con el universo de GPU-izing!</p><h3><strong>Implementación del Cluster</strong></h3><p>Como de costumbre comienza con la secuencia de arranque. Sólo tienes que tener cuidado de que las instancias de p2 sólo están disponibles en us-west-2, us-east-1 y eu-west-2, así como las regiones us-gov. A lo largo de estos años, he tenido malas experiencias ejecutando instancias de p2 en Europa, por lo tanto, te recomiendo que indiques alguna región de EE.UU.</p><pre>juju bootstrap aws/us-east-1 --credential canonical --constraints &quot;cores=4 mem=16G root-disk=64G&quot; <br># Creating Juju controller &quot;aws-us-east-1&quot; on aws/us-east-1<br># Looking for packaged Juju agent version 2.2.4 for amd64<br># Launching controller instance(s) on aws/us-east-1...<br>#  - i-0d48b2c872d579818 (arch=amd64 mem=16G cores=4)<br># Fetching Juju GUI 2.3.8<br># Waiting for address<br># Attempting to connect to 54.174.129.155:22<br># Attempting to connect to 172.31.15.3:22<br># Logging to /var/log/cloud-init-output.log on the bootstrap machine<br># Running apt-get update<br># Running apt-get upgrade<br># Installing curl, cpu-checker, bridge-utils, cloud-utils, tmux<br># Fetching Juju agent version 2.2.4 for amd64<br># Installing Juju machine agent<br># Starting Juju machine agent (service jujud-machine-0)<br># Bootstrap agent now started<br># Contacting Juju controller at 172.31.15.3 to verify accessibility...<br># Bootstrap complete, &quot;aws-us-east-1&quot; controller now available.<br># Controller machines are in the &quot;controller&quot; model.<br># Initial model &quot;default&quot; added.</pre><h4><strong>Implementación de instancias</strong></h4><p>Una vez que el controlador está listo puedes comenzar a desplegar servicios. En mis publicaciones anteriores, he utilizado paquetes que son accesos directos para implementar aplicaciones complejas.</p><p>Si ya estás familiarizado con Juju, puedes incluir esto <em>juju deploy src/k8s-gpu.yaml</em> y saltar al final de esta sección. Para los interesados ​​en entrar en detalles, esta vez nos desplegaremos manualmente, y pasaremos por la lógica del despliegue.</p><p>Kubernetes se compone de 5 aplicaciones individuales: Master, Worker, Flannel (red), etcd (base de datos para el almacenamiento del estado esperado del clúster) y easyRSA (PKI para cifrar la comunicación y proporcionar certificados x509).</p><p>En el momento del despliegue, puede conceder limitaciones a Juju, ya sea muy específicas (tipo de instancia) o laxist (# de núcleos). Con el último, Juju elegirá la instancia más barata que coincida con sus restricciones en la nube de destino.</p><p>Lo primero es desplegar las aplicaciones:</p><pre>juju deploy cs:~containers/kubernetes-master-47 --constraints &quot;cores=4 mem=8G root-disk=32G&quot;<br># Located charm &quot;cs:~containers/kubernetes-master-47&quot;.<br># Deploying charm &quot;cs:~containers/kubernetes-master-47&quot;.<br>juju deploy cs:~containers/etcd-48 --to 0<br># Located charm &quot;cs:~containers/etcd-48&quot;.<br># Deploying charm &quot;cs:~containers/etcd-48&quot;.<br>juju deploy cs:~containers/easyrsa-15 --to lxd:0<br># Located charm &quot;cs:~containers/easyrsa-15&quot;.<br># Deploying charm &quot;cs:~containers/easyrsa-15&quot;.<br>juju deploy cs:~containers/flannel-26<br># Located charm &quot;cs:~containers/flannel-26&quot;.<br># Deploying charm &quot;cs:~containers/flannel-26&quot;.<br>juju deploy cs:~containers/kubernetes-worker-52 --constraints &quot;instance-type=p2.xlarge&quot; kubernetes-worker-gpu<br># Located charm &quot;cs:~containers/kubernetes-worker-52&quot;.<br># Deploying charm &quot;cs:~containers/kubernetes-worker-52&quot;.<br>juju deploy cs:~containers/kubernetes-worker-52 --constraints &quot;instance-type=p2.8xlarge&quot; kubernetes-worker-gpu8<br># Located charm &quot;cs:~containers/kubernetes-worker-52&quot;.<br># Deploying charm &quot;cs:~containers/kubernetes-worker-52&quot;.<br>juju deploy cs:~containers/kubernetes-worker-52 --constraints &quot;instance-type=m4.2xlarge&quot; -n3 kubernetes-worker-cpu<br># Located charm &quot;cs:~containers/kubernetes-worker-52&quot;.<br># Deploying charm &quot;cs:~containers/kubernetes-worker-52&quot;.</pre><p>Aquí puedes ver una interesante propiedad de Juju que nunca antes habíamos abordado: nombrar los servicios que implementas. Desplegamos el mismo “charm” kubernetes-worker dos veces, pero dos veces con GPUs y el otro sin. Esto nos da una forma de agrupar instancias de cierto tipo, a costa de duplicar algunos comandos.</p><p>Observa también los números de revisión de los “charm” que implementamos. Las revisiones no están directamente relacionadas con las versiones del software que implementan. Si los omites, Juju elegirá la última revisión, como Docker haría en las imágenes.</p><h4><strong>Agregar el software de relaciones y exposición</strong></h4><p>Ahora que las aplicaciones están desplegadas, necesitamos decirle a Juju cómo están relacionadas entre sí. Por ejemplo, el maestro de Kubernetes necesita certificados para cifrar las comunicaciones con su API. Por lo tanto, existe una relación entre el kubernetes-master:certificates y easyrsa:client.</p><p>Podemos traducir esta relación en que una vez que se conecten las dos aplicaciones, se ejecutarán algunos scripts para consultar la API de EasyRSA con el objetivo de crear los certificados necesarios y luego copiarlos en la ubicación correcta en el k8s master.</p><p>Estas relaciones crean estados en el clúster, a los que los “charms” pueden reaccionar.</p><p>En esencia, a un nivel muy alto, veo a Juju como un pub-sub implementación de la aplicación de despliegue. Cada acción dentro o fuera del cluster publica un mensaje en un “common bus” y los charms pueden reaccionar a éstos y realizar acciones adicionales, modificando el estado general… y así sucesivamente hasta que se alcance el equilibrio.</p><p>Añadamos las relaciones:</p><pre>juju add-relation kubernetes-master:certificates easyrsa:client<br>juju add-relation etcd:certificates easyrsa:client<br>juju add-relation kubernetes-master:etcd etcd:db<br>juju add-relation flannel:etcd etcd:db<br>juju add-relation flannel:cni kubernetes-master:cni</pre><pre>for TYPE in cpu gpu gpu8<br>do <br>  juju add-relation kubernetes-worker-${TYPE}:kube-api-endpoint kubernetes-master:kube-api-endpoint<br>  juju add-relation kubernetes-master:kube-control kubernetes-worker-${TYPE}:kube-control<br>  juju add-relation kubernetes-worker-${TYPE}:certificates easyrsa:client<br>  juju add-relation flannel:cni kubernetes-worker-${TYPE}:cni<br>  juju expose kubernetes-worker-${TYPE}<br>done</pre><pre>juju expose kubernetes-master</pre><p>Nota al final los comandos de exposición .</p><p>Estas son instrucciones para que Juju abra el firewall en la nube para puertos específicos de las instancias. Algunos son predefinidos en los charm (Kubernetes Master API es 6443, los trabajadores abren 80 y 443 para entradas), pero también puedes forzarlos si es necesario (por ejemplo, cuando agregas manualmente cosas en las instancias después de la implementación)</p><h4><strong>Añadiendo CUDA</strong></h4><p>Desde la primera publicacion de este blog, la instalaciones de los drivers nVidia y de CUDA a sido completamente automatizadas en CDK (Canonical Distribution of Kubernetes). Cada 5 minutos, un proceso se encarga de descubrir el hardware de la maquinas, y, si se encontra una tarjeta nVidia, arrancar el despliegue de los drivers y de CUDA. Asi se puede empezar con maquinas sin GPUs y añadirlos después.</p><p>Disponemos de 2 opciones en el Worker de Kubernetes para controlar el despliegue y la version CUDA:</p><pre>&quot;cuda-version&quot;:<br>    &quot;type&quot;: &quot;string&quot;<br>    &quot;default&quot;: &quot;8.0.61-1&quot;<br>    &quot;description&quot;: |<br>      The cuda-repo package version to install.<br>  &quot;install-cuda&quot;:<br>    &quot;type&quot;: &quot;boolean&quot;<br>    &quot;default&quot;: !!bool &quot;true&quot;<br>    &quot;description&quot;: |<br>      Install the CUDA binaries if capable hardware is present.</pre><p>Los opciones por defecto funcionan por nuestro caso pero si necesitaras un cambio o si quisiera forzar una version, las podrías usar.</p><p>El despliegue y configuración de CUDA aparecen en el estatus de Juju y los logs.</p><pre>juju status<br>Model    Controller     Cloud/Region   Version<br>default  aws-us-east-1  aws/us-east-1  2.2.4</pre><pre>App                     Version  Status       Scale  Charm              Store       Rev  OS      Notes<br>easyrsa                 3.0.1    active           1  easyrsa            jujucharms   15  ubuntu  <br>etcd                    2.3.8    active           1  etcd               jujucharms   48  ubuntu  <br>flannel                 0.7.0    active           6  flannel            jujucharms   26  ubuntu  <br>kubernetes-master       1.7.4    active           1  kubernetes-master  jujucharms   47  ubuntu  exposed<br>kubernetes-worker-cpu   1.7.4    active           3  kubernetes-worker  jujucharms   52  ubuntu  exposed<br>kubernetes-worker-gpu   1.7.4    active           1  kubernetes-worker  jujucharms   52  ubuntu  exposed<br>kubernetes-worker-gpu8  1.7.4    active           1  kubernetes-worker  jujucharms   52  ubuntu  exposed</pre><pre>Unit                       Workload     Agent      Machine  Public address  Ports           Message<br>easyrsa/0*                 active       idle       0/lxd/0  10.0.0.122                      Certificate Authority connected.<br>etcd/0*                    active       idle       0        54.242.44.224   2379/tcp        Healthy with 1 known peers.<br>kubernetes-master/0*       active       idle       0        54.242.44.224   6443/tcp        Kubernetes master running.<br>  flannel/0*               active       idle                54.242.44.224                   Flannel subnet 10.1.76.1/24<br>kubernetes-worker-cpu/0    active       idle       4        52.86.161.22    80/tcp,443/tcp  Kubernetes worker running.<br>  flannel/4                active       idle                52.86.161.22                    Flannel subnet 10.1.79.1/24<br>kubernetes-worker-cpu/1*   active       idle       5        52.70.5.49      80/tcp,443/tcp  Kubernetes worker running.<br>  flannel/2                active       idle                52.70.5.49                      Flannel subnet 10.1.63.1/24<br>kubernetes-worker-cpu/2    active       idle       6        174.129.164.95  80/tcp,443/tcp  Kubernetes worker running.<br>  flannel/3                active       idle                174.129.164.95                  Flannel subnet 10.1.22.1/24<br>kubernetes-worker-gpu8/0*  active       idle       3        52.90.163.167   80/tcp,443/tcp  Kubernetes worker running.<br>  flannel/5                active       idle                52.90.163.167                   Flannel subnet 10.1.35.1/24<br>kubernetes-worker-gpu/0*   active       idle       1        52.90.29.98     80/tcp,443/tcp  Kubernetes worker running.<br>  flannel/1                active       idle                52.90.29.98                     Flannel subnet 10.1.58.1/24</pre><pre>Machine  State    DNS             Inst id              Series  AZ<br>0        started  54.242.44.224   i-09ea4f951f651687f  xenial  us-east-1a<br>0/lxd/0  started  10.0.0.122      juju-65a910-0-lxd-0  xenial  <br>1        started  52.90.29.98     i-03c3e35c2e8595491  xenial  us-east-1c<br>3        started  52.90.163.167   i-0ca0716985645d3f2  xenial  us-east-1d<br>4        started  52.86.161.22    i-02de3aa8efcd52366  xenial  us-east-1e<br>5        started  52.70.5.49      i-092ac5367e31188bb  xenial  us-east-1a<br>6        started  174.129.164.95  i-0a0718343068a5c94  xenial  us-east-1c</pre><pre>Relation      Provides                Consumes                Type<br>certificates  easyrsa                 etcd                    regular<br>certificates  easyrsa                 kubernetes-master       regular<br>certificates  easyrsa                 kubernetes-worker-cpu   regular<br>certificates  easyrsa                 kubernetes-worker-gpu   regular<br>certificates  easyrsa                 kubernetes-worker-gpu8  regular<br>cluster       etcd                    etcd                    peer<br>etcd          etcd                    flannel                 regular<br>etcd          etcd                    kubernetes-master       regular<br>cni           flannel                 kubernetes-master       regular<br>cni           flannel                 kubernetes-worker-cpu   regular<br>cni           flannel                 kubernetes-worker-gpu   regular<br>cni           flannel                 kubernetes-worker-gpu8  regular<br>cni           kubernetes-master       flannel                 subordinate<br>kube-dns      kubernetes-master       kubernetes-worker-cpu   regular<br>kube-dns      kubernetes-master       kubernetes-worker-gpu   regular<br>kube-dns      kubernetes-master       kubernetes-worker-gpu8  regular<br>cni           kubernetes-worker-cpu   flannel                 subordinate<br>juju-info     kubernetes-worker-gpu   cuda                    subordinate<br>cni           kubernetes-worker-gpu   flannel                 subordinate<br>juju-info     kubernetes-worker-gpu8  cuda                    subordinate<br>cni           kubernetes-worker-gpu8  flannel                 subordinate</pre><p>Al final, veamos lo que nvidia-smi nos devuelve:</p><pre>juju ssh kubernetes-worker-gpu/0 sudo nvidia-smi<br>Tue Feb 14 13:28:42 2017       <br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |<br>|-------------------------------+----------------------+----------------------+<br>| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>|===============================+======================+======================|<br>|   0  Tesla K80           On   | 0000:00:1E.0     Off |                    0 |<br>| N/A   33C    P0    81W / 149W |      0MiB / 11439MiB |     95%      Default |<br>+-------------------------------+----------------------+----------------------+<br>                                                                               <br>+-----------------------------------------------------------------------------+<br>| Processes:                                                       GPU Memory |<br>|  GPU       PID  Type  Process name                               Usage      |<br>|=============================================================================|<br>|  No running processes found                                                 |<br>+-----------------------------------------------------------------------------+</pre><p>En el 8xlarge más potente,</p><pre>juju ssh kubernetes-worker-gpu8/0 sudo nvidia-smi<br>Tue Feb 14 13:59:24 2017       <br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |<br>|-------------------------------+----------------------+----------------------+<br>| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>|===============================+======================+======================|<br>|   0  Tesla K80           On   | 0000:00:17.0     Off |                    0 |<br>| N/A   41C    P8    31W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   1  Tesla K80           On   | 0000:00:18.0     Off |                    0 |<br>| N/A   36C    P0    70W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   2  Tesla K80           On   | 0000:00:19.0     Off |                    0 |<br>| N/A   44C    P0    57W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   3  Tesla K80           On   | 0000:00:1A.0     Off |                    0 |<br>| N/A   38C    P0    70W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   4  Tesla K80           On   | 0000:00:1B.0     Off |                    0 |<br>| N/A   43C    P0    57W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   5  Tesla K80           On   | 0000:00:1C.0     Off |                    0 |<br>| N/A   38C    P0    69W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   6  Tesla K80           On   | 0000:00:1D.0     Off |                    0 |<br>| N/A   44C    P0    58W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   7  Tesla K80           On   | 0000:00:1E.0     Off |                    0 |<br>| N/A   38C    P0    71W / 149W |      0MiB / 11439MiB |     39%      Default |<br>+-------------------------------+----------------------+----------------------+<br>                                                                               <br>+-----------------------------------------------------------------------------+<br>| Processes:                                                       GPU Memory |<br>|  GPU       PID  Type  Process name                               Usage      |<br>|=============================================================================|<br>|  No running processes found                                                 |<br>+-----------------------------------------------------------------------------+</pre><p>¡Aaa y sí! Tenemos nuestras 8 GPUs como se esperaba 8x 12 GB = 96 GB de RAM de vídeo!</p><p>En esta etapa, sólo los tenemos habilitados en los hosts. Ahora vamos a añadir soporte de GPU en Kubernetes.</p><h3><strong>Añadir el soporte de GPU en Kubernetes</strong></h3><p>Cuando Juju encontra GPUs y despliega CUDA, también se encarga de configurar Kubernetes para dar la bienvenida a imágenes Docker que usan a GPUs. No hay nada que hacer.</p><p>Puede ser util saber que CDK añade un “label” <em>cuda=true</em> a los nodos con GPUs para dar indicaciones y posibilidades de forzar la ubicación de aplicaciones con CUDA.</p><h4><strong>Probar lo que hemos desarrollado</strong></h4><p>Ahora queremos saber si el clúster realmente tiene habilitada la GPU. Para validar, ejecuta un trabajo con un nvidia-smi pod:</p><pre>kubectl create -f src/nvidia-smi.yaml</pre><p>Después, espera un poco y ejecuta el comando de registro:</p><pre>kubectl logs $(kubectl get pods -l name=nvidia-smi -o=name -a)<br>Tue Feb 14 14:14:57 2017       <br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |<br>|-------------------------------+----------------------+----------------------+<br>| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>|===============================+======================+======================|<br>|   0  Tesla K80           Off  | 0000:00:17.0     Off |                    0 |<br>| N/A   47C    P0    56W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   1  Tesla K80           Off  | 0000:00:18.0     Off |                    0 |<br>| N/A   39C    P0    70W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   2  Tesla K80           Off  | 0000:00:19.0     Off |                    0 |<br>| N/A   48C    P0    57W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   3  Tesla K80           Off  | 0000:00:1A.0     Off |                    0 |<br>| N/A   41C    P0    70W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   4  Tesla K80           Off  | 0000:00:1B.0     Off |                    0 |<br>| N/A   47C    P0    58W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   5  Tesla K80           Off  | 0000:00:1C.0     Off |                    0 |<br>| N/A   40C    P0    69W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   6  Tesla K80           Off  | 0000:00:1D.0     Off |                    0 |<br>| N/A   48C    P0    59W / 149W |      0MiB / 11439MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>|   7  Tesla K80           Off  | 0000:00:1E.0     Off |                    0 |<br>| N/A   41C    P0    72W / 149W |      0MiB / 11439MiB |    100%      Default |<br>+-------------------------------+----------------------+----------------------+<br>                                                                               <br>+-----------------------------------------------------------------------------+<br>| Processes:                                                       GPU Memory |<br>|  GPU       PID  Type  Process name                               Usage      |<br>|=============================================================================|<br>|  No running processes found                                                 |<br>+-----------------------------------------------------------------------------+</pre><p>Lo que interesa aquí es que el pod vea todas las tarjetas, incluso si sólo compartimos el dispositivo / dev / nvidia0 char. En tiempo de ejecución, tendríamos problemas.</p><p>Si desea ejecutar múltiples contenedores GPU, tendrá que compartir todos los dispositivos char como lo hacemos en el segundo archivo yaml (nvidia-smi-8.yaml)</p><h3><strong>Conclusión</strong></h3><p>Alcanzamos el primer hito de nuestro viaje en tres fases: el clúster está activo y funcionando, las GPU están activadas, y Kubernetes ahora dará la bienvenida a las cargas de trabajo de la GPU.</p><p>Si eres un científico de datos o ejecutas cargas de trabajo Kubernetes que podrían beneficiarse de las GPUs, esto ya te da una forma elegante y muy rápida de la gestionar tus configuraciones. Pero normalmente en este contexto, también es necesario disponer de almacenamiento entre las instancias, ya sea para compartir el conjunto de datos o para intercambiar resultados.</p><p>Kubernetes ofrece muchas opciones para conectar el almacenamiento. En la segunda parte de esta trilogía, vamos a ver cómo automatizar la adición de almacenamiento EFS a nuestras instancias, a continuación, ponerlo a buen uso con algunos conjuntos de datos.</p><p>Mientras tanto, te invito a que te pongas en contacto conmigo si tienes un caso de uso específico en la nube para discutir los detalles operativos. Yo estaré encantado de ayudarle a configurar tu propio clúster de GPU y empezar a trabajar para la ciencia.</p><h3><strong>Derribando</strong></h3><p>Siempre que lo desee, puede derribar este grupo. Estos casos pueden ser costosos, por lo tanto, encenderlos cuando no los utilice no es una mala idea.</p><pre>juju kill-controller aws/us-east-1</pre><p>Esto te pedirá confirmación y después ya podrás eliminar todo…</p><p>Pero ahora sabes que eres solo a algunas commandos de arancarlo de nuevo en minutitos así que no sea un problema!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90c506e0645e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using the nVidia GT 1030 for CUDA workloads on Ubuntu 16.04]]></title>
            <link>https://medium.com/@samnco/using-the-nvidia-gt-1030-for-cuda-workloads-on-ubuntu-16-04-4eee72d56791?source=rss-484dbfca3cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/4eee72d56791</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Samuel Cozannet]]></dc:creator>
            <pubDate>Fri, 04 Aug 2017 09:21:42 GMT</pubDate>
            <atom:updated>2017-08-04T09:21:42.775Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently nVidia released a new low-end card, the <a href="https://www.techpowerup.com/gpudb/2954/geforce-gt-1030">nVidia GT1030</a>. Its specs are so low that it is not even listed on the <a href="https://developer.nvidia.com/cuda-gpus">official CUDA supported cards page</a>! The thing is it is also the cheapest card you can find on the market, around 70 bucks on <a href="https://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&amp;field-keywords=gt+1030">Amazon</a>, and the spec page lists 384 CUDA cores.</p><p>There are a fair number of people asking if this card can be used for CUDA workloads, and the answers are not always straightforward.</p><p>Luckily I got my hands on one of these. So… Can you or can you not run anything on it?</p><p>Short answer: yes, but not much.</p><h3>Installation hurdles…</h3><h4>The problem</h4><p>Most CUDA based systems will tell you to use the nvidia-375 driver package available from the official repos. It is the driver that is actually included in the repository version of CUDA 8.0.</p><p>If you do so with the GT1030, you will fail as the drivers do not support it. When calling <em>nvidia-smi</em> the logs with say:</p><pre>Aug  3 10:26:32 ubuntu kernel: [  682.768627] NVRM: RmInitAdapter failed! (0x26:0xffff:1097)<br>Aug  3 10:26:32 ubuntu kernel: [  682.768696] NVRM: rm_init_adapter failed for device bearing minor number 0</pre><p>and deviceQuery will report</p><pre>deviceQuery Starting...</pre><pre>CUDA Device Query (Runtime API) version (CUDART static linking)</pre><pre>cudaGetDeviceCount returned 38<br>-&gt; no CUDA-capable device is detected<br>Result = FAIL</pre><h4>The solution</h4><p>To solve the issue, we need to install CUDA as the standalone runfile, then install the latest drivers from the 381 branch as well as libcuda. You can do it on a fresh install via:</p><pre>$ wget <a href="https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_375.26_linux-run">https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_375.26_linux-run</a><br>$ chmod +x cuda_8.0.61_375.26_linux-run<br>$ sudo ./cuda_8.0.61_375.26_linux-run --silent --toolkit --samples </pre><p>This will have the CUDA toolkit installed without the drivers. Then</p><pre>$ sudo add-apt-repository -y ppa:graphics-drivers/ppa<br>$ sudo apt update<br>$ sudo apt install --no-install-recommends nvidia-381 nvidia-381-dev libcuda1-381 docker.io</pre><p>will install the drivers that will work for us. We will use the Docker package later in the post.</p><h4>The validation</h4><p>All we have to do now is run nvidia-smi and deviceQuery to check everything:</p><pre>$ nvidia-smi<br>Fri Aug  4 09:26:54 2017       <br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |<br>|-------------------------------+----------------------+----------------------+<br>| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>|===============================+======================+======================|<br>|   0  GeForce GT 1030     Off  | 0000:03:00.0     Off |                  N/A |<br>| 30%   35C    P0    12W /  30W |      0MiB /  2000MiB |      0%      Default |<br>+-------------------------------+----------------------+----------------------+<br>                                                                               <br>+-----------------------------------------------------------------------------+<br>| Processes:                                                       GPU Memory |<br>|  GPU       PID  Type  Process name                               Usage      |<br>|=============================================================================|<br>|  No running processes found                                                 |<br>+-----------------------------------------------------------------------------+</pre><p>and</p><pre>$ cd NVIDIA_CUDA-8.0_Samples/1_Utilities/deviceQuery<br>$ make<br>$ ./deviceQuery<br>./deviceQuery Starting...</pre><pre>CUDA Device Query (Runtime API) version (CUDART static linking)</pre><pre>Detected 1 CUDA Capable device(s)</pre><pre>Device 0: &quot;GeForce GT 1030&quot;<br>  CUDA Driver Version / Runtime Version          8.0 / 8.0<br>  CUDA Capability Major/Minor version number:    6.1<br>  Total amount of global memory:                 2000 MBytes (2097217536 bytes)<br>  ( 3) Multiprocessors, (128) CUDA Cores/MP:     384 CUDA Cores<br>  GPU Max Clock rate:                            1468 MHz (1.47 GHz)<br>  Memory Clock rate:                             3004 Mhz<br>  Memory Bus Width:                              64-bit<br>  L2 Cache Size:                                 524288 bytes<br>  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)<br>  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers<br>  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers<br>  Total amount of constant memory:               65536 bytes<br>  Total amount of shared memory per block:       49152 bytes<br>  Total number of registers available per block: 65536<br>  Warp size:                                     32<br>  Maximum number of threads per multiprocessor:  2048<br>  Maximum number of threads per block:           1024<br>  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)<br>  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)<br>  Maximum memory pitch:                          2147483647 bytes<br>  Texture alignment:                             512 bytes<br>  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)<br>  Run time limit on kernels:                     No<br>  Integrated GPU sharing Host Memory:            No<br>  Support host page-locked memory mapping:       Yes<br>  Alignment requirement for Surfaces:            Yes<br>  Device has ECC support:                        Disabled<br>  Device supports Unified Addressing (UVA):      Yes<br>  Device PCI Domain ID / Bus ID / location ID:   0 / 3 / 0<br>  Compute Mode:<br>     &lt; Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) &gt;</pre><pre>deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GT 1030<br>Result = PASS</pre><p>So YES, we do have CUDA on the board (as expected but not officially recognized)</p><h3>CUDA Workloads</h3><h4>Crypto Currency Mining #1: Ethereum (fails)</h4><p>A while a go I dockerized the Claymore Miner, and I keep updating the image from times to times. I usually do scale out tests on Kubernetes with it, but it’s fine with Docker alone.</p><p>Note I am not using nvidia-docker because it expects the upstream Docker and I am using the version from the Ubuntu archive.</p><pre>$ docker run -it \<br>  -v /usr/lib/nvidia-381/bin:/usr/local/nvidia/bin \<br>  -v /usr/lib/nvidia-381:/usr/lib/nvidia \<br>  -v /usr/lib/x86_64-linux-gnu:/usr/lib/cuda \<br>  -e LD_LIBRARY_PATH=&quot;$LD_LIBRARY_PATH:/usr/lib/nvidia:/usr/lib/cuda&quot; \<br>  -e POD_NAME=1nvidia \<br>  --device /dev/nvidia0:/dev/nvidia0 \<br>  --device /dev/nvidiactl:/dev/nvidiactl \<br>  --device /dev/nvidia-uvm:/dev/nvidia-uvm \<br>  samnco/claymore-miner:9.7-nvidia \<br>  /entrypoint.sh</pre><pre>-di detect is not supported anymore (obsolete)</pre><pre>****************************************************************Í»<br>*      Claymore&#39;s Dual ETH + DCR/SC/LBC/PASC GPU Miner v9.7      *<br>****************************************************************Í¼<br>ETH: 1 pool is specified<br>Main Ethereum pool is eu1.ethermine.org:4444<br>AMD OpenCL platform not found</pre><pre>Driver 368.81 is recommended for best performance and compatibility<br>Be careful with overclocking, use default clocks for first tests<br>Press &quot;s&quot; for current statistics, &quot;0&quot;..&quot;9&quot; to turn on/off cards, &quot;r&quot; to reload pools, &quot;e&quot; or &quot;d&quot; to select current pool <br>CUDA initializing...</pre><pre>NVIDIA Cards available: 1 <br>CUDA Driver Version/Runtime Version: 8.0/8.0<br>GPU #0: GeForce GT 1030, 2000 MB available, 3 compute units, capability: 6.1</pre><pre>Total cards: 1 <br>ETH: Stratum - connecting to &#39;eu1.ethermine.org&#39; &lt;46.105.121.53&gt; port 4444<br>ETH: Stratum - Connected (eu1.ethermine.org:4444)<br>ETHEREUM-ONLY MINING MODE ENABLED (-mode 1)</pre><pre>ETH: eth-proxy stratum mode<br>Watchdog enabled<br>Remote management (READ-ONLY MODE) is enabled on port 3333</pre><pre>ETH: Authorized<br>Setting DAG epoch #137...<br>ETH: 08/04/17-08:36:04 - New job from eu1.ethermine.org:4444<br>ETH - Total Speed: 0.000 Mh/s, Total Shares: 0, Rejected: 0, Time: 00:00<br>ETH: GPU0 0.000 Mh/s<br>Setting DAG epoch #137 for GPU0<br>Create GPU buffer for GPU0<br>CUDA error - cannot allocate big buffer for DAG. Check readme.txt for possible solutions.</pre><pre>GPU 0 failed<br>Setting DAG epoch #137 for GPU0<br>GPU 0, CUDA error 11 - cannot write buffer for DAG</pre><pre>GPU 0 failed<br>ETH: 08/04/17-08:36:26 - New job from eu1.ethermine.org:4444<br>ETH - Total Speed: 0.000 Mh/s, Total Shares: 0, Rejected: 0, Time: 00:00<br>ETH: GPU0 0.000 Mh/s<br>Quit signal received...<br>GPU0 t=34C fan=36%<br>WATCHDOG: GPU error, you need to restart miner :(</pre><p>OK, we have a fail. This was expected though as the DAG for ETH at this point is bigger than 2GB, so it doesn’t fit in the memory of the GPU. No ETH mining with it!</p><h4>Crypto Currency Mining #2: Monero (works)</h4><p>For this I have an image of xmrminer which I use to showcase how handling sigint in Docker images is a necessity. Monero is definitely a CPU oriented currency: you do not get more from a GPU you would from a high end CPU. A single Xeon hyperthreaded core will give you 10 to 20 hashes/sec, while a high end Radeon or nVidia Pascal card will give about 500 to 1000 H/s.</p><p>Here, with a 96x8 or a 48x16, I get</p><pre>docker run -it \<br>  -v /usr/lib/nvidia-381/bin:/usr/local/nvidia/bin \<br>  -v /usr/lib/nvidia-381:/usr/lib/nvidia \<br>  -v /usr/lib/x86_64-linux-gnu:/usr/lib/cuda \<br>  -e LD_LIBRARY_PATH=&quot;$LD_LIBRARY_PATH:/usr/lib/nvidia:/usr/lib/cuda&quot; \<br>  --device /dev/nvidia0:/dev/nvidia0 \<br>  --device /dev/nvidiactl:/dev/nvidiactl \<br>  --device /dev/nvidia-uvm:/dev/nvidia-uvm \<br>  -v $PWD/config.nvidia:/config/config.txt \<br>  samnco/xmrminer:0.1.5-nvidia \<br>  /usr/bin/xmr-stak-nvidia</pre><pre>-------------------------------------------------------------------<br>XMR-Stak-NVIDIA mining software, NVIDIA Version.<br>NVIDIA mining code was written by KlausT and psychocrypt.<br>Brought to you by fireice_uk under GPLv3.</pre><pre>Configurable dev donation level is set to 1.0 %</pre><pre>You can use following keys to display reports:<br>&#39;h&#39; - hashrate<br>&#39;r&#39; - results<br>&#39;c&#39; - connection<br>-------------------------------------------------------------------<br>[2017-08-04 08:46:11] : Connecting to pool xmr-eu.dwarfpool.com:8050 ...<br>[2017-08-04 08:46:11] : Connected. Logging in...<br>[2017-08-04 08:46:11] : Difficulty changed. Now: 50000.<br>[2017-08-04 08:46:11] : New block detected.<br>HASHRATE REPORT<br>| ID |  10s |  60s |  15m |<br>|  0 | 117.4 | (na) | (na) |<br>---------------------------<br>Totals:   117.4 (na) (na) H/s<br>Highest:  117.4 H/s</pre><p>These settings consume about 75% of the card memory, and 100% of its compute capability. I spent a short amount of time tweaking this, but do not expect to get more than 200H/s on this card. Not too bad but not great either.</p><h4>Tensorflow</h4><p>For tensorflow we will just run the default examples</p><pre>docker run -it -p 8888:8888 \<br>  -v /usr/lib/nvidia-381/bin:/usr/local/nvidia/bin \<br>  -v /usr/lib/nvidia-381:/usr/lib/nvidia \<br>  -v /usr/lib/x86_64-linux-gnu:/usr/lib/cuda \<br>  -e LD_LIBRARY_PATH=&quot;$LD_LIBRARY_PATH:/usr/lib/nvidia:/usr/lib/cuda&quot; \<br>  -e PASSWORD=admin \<br>  --device /dev/nvidia0:/dev/nvidia0 \<br>  --device /dev/nvidiactl:/dev/nvidiactl \<br>  --device /dev/nvidia-uvm:/dev/nvidia-uvm \<br>  tensorflow/tensorflow:latest-gpu</pre><p>Then we connect on the UI and run the Hello World</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*36CydD27mjWpVfJ5uHUXng.png" /><figcaption>Tensorflow Hello World</figcaption></figure><p>What we do here is that we use the Kernel menu and do a “restart and run all cells”, so that Jupyter will play the whole notebook. The result is that it consumes just about 1800MB of RAM on the card:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*YBI7XA9xudF_2KESOVmeUQ.png" /><figcaption>Memory Consumption from Tensorflow</figcaption></figure><p>So now if we go on another notebook and do the same, we’ll end up with 2 notebooks loaded at the same time, which will result in a failure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*uNXoYFZhdHJAiQGoeqeHuw.png" /><figcaption>Second notebook coming in</figcaption></figure><p>From the tensorflow logs, when we start the second notebook:</p><pre>2017-08-04 09:04:59.467813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: <br>name: GeForce GT 1030<br>major: 6 minor: 1 memoryClockRate (GHz) 1.468<br>pciBusID 0000:03:00.0<br>Total memory: 1.95GiB<br>Free memory: 138.06MiB<br>2017-08-04 09:04:59.467856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0 <br>2017-08-04 09:04:59.467866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y <br>2017-08-04 09:04:59.467885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GT 1030, pci bus id: 0000:03:00.0)<br>2017-08-04 09:04:59.468853: E tensorflow/stream_executor/cuda/cuda_driver.cc:924] failed to allocate 138.06M (144769024 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY<br>2017-08-04 09:04:59.783213: E tensorflow/stream_executor/cuda/cuda_blas.cc:365] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED<br>2017-08-04 09:04:59.783283: W tensorflow/stream_executor/stream.cc:1601] attempting to perform BLAS operation using StreamExecutor without BLAS support<br>[32m[I 09:06:31.837 NotebookApp](B[m Saving file at /2_getting_started.ipynb</pre><p>Totally uncool.</p><h3>Conclusion</h3><p>You can’t do much more than watching movies with this board unfortunately. If you want to do some deep learning but have little money to spend, then even this is not the best option.</p><p>You can find the GTX 1050 Ti, which doubles all the specs for double of the money at $150. Not too expensive considering 2GB of RAM today seems to be really the bare minimum.</p><p>If can spend up to $300, then the latest GTX 1060 6GB with extended memory bandwidth will be your best option. It’s what I used for recent posts.</p><p>That being said, the GT 1030 carries some interest if:</p><ul><li>You have a requirement on the passive cooling system</li><li>You have a single slot requirement in density environments</li></ul><p>Any question, let me know :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4eee72d56791" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>