<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Namely Labs - Medium]]></title>
        <description><![CDATA[Namely Engineering + Design - Medium]]></description>
        <link>https://medium.com/namely-labs?source=rss----8354434f0ab4---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Namely Labs - Medium</title>
            <link>https://medium.com/namely-labs?source=rss----8354434f0ab4---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 03:06:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/namely-labs" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running and Writing Gatekeeper Policies in Kubernetes — Part 3]]></title>
            <link>https://medium.com/namely-labs/running-and-writing-gatekeeper-policies-in-kubernetes-part-3-d5e307f4cde3?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/d5e307f4cde3</guid>
            <category><![CDATA[open-policy-agent]]></category>
            <category><![CDATA[kubernetes-administration]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gatekeepers]]></category>
            <dc:creator><![CDATA[Sebastian Radloff]]></dc:creator>
            <pubDate>Wed, 27 Oct 2021 15:00:57 GMT</pubDate>
            <atom:updated>2021-10-27T15:00:57.516Z</atom:updated>
            <content:encoded><![CDATA[<h3>Running and Writing Gatekeeper Policies in Kubernetes — Part 3</h3><h3>Series Overview</h3><p>This series examines an opinionated Kubernetes Gatekeeper policy ecosystem, including what Gatekeeper is, why use Gatekeeper, writing and testing policies, deploying policies to a local Kubernetes cluster, and how to make them visible to users. All code for this series resides in this GitHub repository:</p><p><a href="https://github.com/sebradloff/k8s-gatekeeper-policies-example">GitHub - sebradloff/k8s-gatekeeper-policies-example</a></p><p><a href="https://medium.com/@sebradloff/running-gatekeeper-in-kubernetes-and-writing-policies-part-1-fcc83eba93e3"><strong>Part 1: Gatekeeper Components and Architecture Overview</strong></a></p><p><a href="https://medium.com/@sebradloff/running-and-writing-gatekeeper-policies-in-kubernetes-part-2-1c49c1c683b2"><strong>Part 2: Install Gatekeeper to a Local Cluster and Iterate on Policies</strong></a></p><p>In this post, we’ll continue writing policies to demonstrate a fuller range of possibilities, such as how to use the konstraint core library, why you might not want to apply policies directly to pods, and how to write policies that require knowledge of existing resources in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*YcCtE27xakT_lSmw54Tguw.jpeg" /></figure><h3>Using the konstraint core library and default keyword</h3><p>As we’ve seen before, we can import existing rego and use it in other policies. The official OPA docs do a great job explaining <a href="https://github.com/open-policy-agent/opa/blob/27c8d75d3528dc8a04e716d5455f97ee7fb537bc/docs/content/policy-language.md#imports">imports</a> if you need a refresher. In this section, we’ll create a few policies that leverage the <a href="https://github.com/sebradloff/k8s-gatekeeper-policies-example/blob/main/policies/lib/core.rego">imported konstraint core library</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a11960c3bfcf61d8d4716f2ceddca0ed/href">https://medium.com/media/a11960c3bfcf61d8d4716f2ceddca0ed/href</a></iframe><p>The above policy (namespace_team_label_02) is functionally the same as the first one we reviewed (namespace_team_label_01), ensuring all namespaces have the ‘team’ label. We kept the same tests from our first policy since the logic should be the same. Let’s take a look at the policies side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9VE_UR2S9mOVemTfSBxFNQ.png" /></figure><p>The significant differences are the core library usage, the helper function has_team_label, and the violation rule block signature.</p><p>The core library document has predefined queries and functions that we leverage. Below are the relevant snippets of source code from the core library, which we depend on in our policy:</p><pre>package lib.core</pre><pre>default is_gatekeeper = false</pre><pre>is_gatekeeper {<br> has_field(input, &quot;review&quot;)<br> has_field(input.review, &quot;object&quot;)<br>}</pre><pre>resource = input.review.object {<br> is_gatekeeper<br>}</pre><pre>resource = input {<br> not is_gatekeeper<br>}</pre><pre>name = resource.metadata.name</pre><pre>labels = resource.metadata.labels</pre><pre>has_field(obj, field) {<br> not object.get(obj, field, &quot;N_DEFINED&quot;) == &quot;N_DEFINED&quot;<br>}</pre><pre>format(msg) = {&quot;msg&quot;: msg}</pre><pre>format_with_id(msg, id) = msg_fmt {<br> msg_fmt := {<br>  &quot;msg&quot;: sprintf(&quot;%s: %s&quot;, [id, msg]),<br>  &quot;details&quot;: {&quot;policyID&quot;: id},<br> }<br>}</pre><p>As we can see, this code is similar to what we had in the first policy (namespace_team_label_01). The resource variable is defined as input.review.object assuming the function is_gatekeeper evaluates to true. The following rule block where the resource variable is equal to input if the is_gatekeeper function evaluates to false. This overwriting style is typical with OPA policies as a form of control flow. Below is an example of how the policy could be written in python:</p><pre># imagine as python code</pre><pre>def is_gatekeeper(input):<br>    return has_field(input, &quot;review&quot;) and has_field(input[&quot;review&quot;], &quot;object&quot;)</pre><pre>def has_field(obj, field):<br>    val = obj.get(field, &quot;N_DEFINED&quot;)<br>    return val != &quot;N_DEFINED&quot;</pre><pre>if __name__ == &quot;__main__&quot;:<br>    input = {<br>        &quot;review&quot;: {<br>            &quot;object&quot;: {<br>                &quot;is_obj&quot;: &quot;nice&quot;<br>            }<br>        }<br>    }<br>    if is_gatekeeper(input):<br>        resource = input[&quot;review&quot;][&quot;object&quot;]<br>    else:<br>        resource = input</pre><p>The has_team_label helper function in our new policy is leveraging the <a href="https://www.openpolicyagent.org/docs/latest/policy-language/#default-keyword">default keyword</a>, which allows us to define a default value for documents that might evaluate to undefined. In our usage, core.labels might evaluate to “undefined” if there is a labels key on the resource under review. With this helper function, we “catch” that error and ensure the value for our helper function returns false.</p><p>The violation rule block signature hasn’t changed. The variablemsg resolves to the signature required{&quot;msg&quot;: msg, &quot;details&quot;: {}} , but the core library is doing that for us with the core.format_with_id function. It nicely formats the output with a policyID, which is useful in error messages, enabling users to look up the policy in our generated documentation easily.</p><h3>Using the konstraint pods library and the unforeseen consequences of applying policies directly on pods</h3><p>Let’s create a policy requiring all containers to have resource requests to binpack our nodes and properly avoid the noisy neighbor problem.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/936b01870da206675e097428aa35dbb5/href">https://medium.com/media/936b01870da206675e097428aa35dbb5/href</a></iframe><p>In this policy, we use the <a href="https://github.com/sebradloff/k8s-gatekeeper-policies-example/blob/main/policies/lib/pods.rego">konstraint pods library</a> to iterate through all containers. The pods.containers document returns a list of container objects, which we then iterate through using the [_] notation, and assign each container to the container variable. The <a href="https://github.com/open-policy-agent/opa/blob/27c8d75d3528dc8a04e716d5455f97ee7fb537bc/docs/content/policy-language.md#universal-quantification-for-all">OPA official documentation on universal quantification</a> does a great job explaining the iteration strategies.</p><p>There are three violation rules in this policy and three helper functions. The helper functions are simple but make the policy code more expressive. container_requests_provided confirms that the container has the resource requests CPU and memory keys defined. The container_requests_zero_memory and container_requests_zero_cpu functions run a regex match against memory and CPU resource requests to ensure a non-zero value was provided. The output message is formatted in such a way to identify the resource kind, resource name, and container name violating the policy, providing clear and actionable feedback to our users.</p><p>While this policy does what we expect it to do in theory, most of our users are not creating Pod resources directly but are instead creating Deployments or Jobs that eventually create the Pods. If they apply a Deployment with containers violating this policy, the Deployment will be successfully created, creating a new ReplicaSet that will then fail to create a Pod. That’s why it’s essential to consider how your developers interact with the cluster when writing these policies. We can easily make this policy more user friendly by adding the kinds to the @kindsannotations at the top of the src.rego file, as demonstrated below in container_deny_without_resource_request_02 .</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d3291171dd19a9d236f5d5b141977bc/href">https://medium.com/media/7d3291171dd19a9d236f5d5b141977bc/href</a></iframe><h3>Writing policies that make determinations based on other resources existing in the cluster</h3><p>There are situations in which you want to block the creation or update of a resource given the existence of another resource in the cluster. For example, let’s say that you’ve established an API gateway pattern in which routing to each service is “/api/${resource}/${subpath}” is rewritten to “/${subpath}. Consider the following ingresses:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/725b36cdd603ca74a003e220bab954b4/href">https://medium.com/media/725b36cdd603ca74a003e220bab954b4/href</a></iframe><p>The foo ingress will route all calls to/api/foozles/${subpath} to the foo service at/${subpath}. The bar ingress will route calls to /api/foozles/barzles/${subpath} to the bar service at /${subpath}, which goes against our standard pattern.</p><p>We can prevent this from happening by writing a policy and using a gatekeeper <a href="https://github.com/open-policy-agent/gatekeeper/blob/b468cf25dd7c5fd2b5918fe3ff47e003db930de2/website/docs/sync.md">Config</a> resource type. We list the Kubernetes resource types that we’d like to query as part of our policies in the config spec. Important to note that the name of the config must be config, for it to be reconciled by Gatekeeper. Below is the config we have written to query namespaces and ingresses:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2d1356a2f7ef75d6aeb10810496d9df/href">https://medium.com/media/e2d1356a2f7ef75d6aeb10810496d9df/href</a></iframe><p>Now we can query both object types through data.inventory in our policy. For cluster scoped objects like namespaces, the query format is data.inventory.cluster[&lt;groupVersion&gt;][&lt;kind&gt;][&lt;name&gt;] . For namespace scoped objects like ingresses, the query format is data.inventory.namespace[&lt;namespace&gt;][&lt;groupVersion&gt;][&lt;kind&gt;][&lt;name&gt;]. Below is the rego source defining our policy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbfe7bf2d5ff5c2183c442fc583c3133/href">https://medium.com/media/dbfe7bf2d5ff5c2183c442fc583c3133/href</a></iframe><p>At the top of the violation rule, we iterate through all namespaces existing in the cluster and each namespace’s ingresses. Then we check to see if the ingress we’ve pulled from inventory is the same as the one currently under review. This check is necessary whenever you’re applying a policy against the same resource type you’re actively querying from the cluster. For example, if we want to apply changes to the foo ingress in the foo namespace, without the not identical(other_ing, curr_ing) check, the policy would find the changes in violation due to the existing foo ingress in the foo namespace. The rest of the policy isn’t necessarily an ideal way for determining if there’s an issue, but it works for this example.</p><p>When writing tests, you’ll want to mock data.inventory. Creating helper functions to build your Kubernetes objects makes it easier to create test cases and enhances test legibility. You can see this in the test file below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f042f9b93df8b00fcb706be822fbcd1/href">https://medium.com/media/5f042f9b93df8b00fcb706be822fbcd1/href</a></iframe><p>Feel free to test the policy by applying the manifests in example-k8s-resources/ingress-policy-testing.</p><p>In the next post, we’ll take an in-depth look at structuring your CI for your policies repository and recommend strategies for applying policies that already have resources in violation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5e307f4cde3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/running-and-writing-gatekeeper-policies-in-kubernetes-part-3-d5e307f4cde3">Running and Writing Gatekeeper Policies in Kubernetes — Part 3</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authentication at Namely: Moving to the Edge]]></title>
            <link>https://medium.com/namely-labs/authentication-at-namely-moving-to-the-edge-ec912f118d1a?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/ec912f118d1a</guid>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[openresty]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Abi Srivastava]]></dc:creator>
            <pubDate>Tue, 09 Apr 2019 19:21:36 GMT</pubDate>
            <atom:updated>2019-04-09T19:26:40.371Z</atom:updated>
            <content:encoded><![CDATA[<p>In the coming year, Namely Engineering is on track to add over 100 new microservices to keep up with our technical and product growth. And as Namely grows, we need to continue adhering to best practices in security and ensuring security is provided by default. With this, we need to implement secure mechanisms to provide authentication information to services in a way which is scalable and maintainable.</p><p>Authentication is the process of verifying that you are who you say you are, and it is a fundamental concept in SaaS software. When building microservices, authentication is something about which nearly every service is concerned. A service needs to know the user making a request so that it can associate ownership with resources and check whether the user is allowed to perform the action they’re requesting. Without a centralized strategy, this can be a big overhead in development and in the footprint of each service.</p><h3>Namely Before a Common Authentication Framework</h3><p>Before introducing a standard approach, we had authentication implemented in our monolith, which required services to do the heavy lifting to authenticate themselves. To ensure that we continued to adhere to a high security standard, we decided to make security a ubiquitous concept. This makes it even easier for our engineers to meet these standards, as services would not have to care about a primary function like authentication.</p><h3>Designing a Brand New Gatekeeper</h3><p>After multiple brainstorming sessions, a few key features stood out as must-haves for the new authentication system.</p><ol><li>Scalable and highly performant</li><li>Provides a standard and uniform method to authenticate across all services</li><li>Treats authentication as a default action, a no-op</li><li>Minimizes the effort needed to use the authentication system for a new service</li></ol><p>Keeping these must-haves in mind, we decided to handle the process of identifying and authenticating a request at the outermost layer, which is the proxy server. Since all requests pass through the proxy server, we would guarantee security by default.</p><p>We call this system Authentication at the Edge.</p><h3>How does it work?</h3><p>Any request flowing through our ELBs passes through our OpenResty proxy servers to the Kubernetes cluster. Our OpenResty proxy servers authenticate the request by looking up session information. They then generate and sign a <strong>J</strong>SON <strong>W</strong>eb <strong>T</strong>oken (<strong>JWT</strong>) for the particular request. The upstream service can use this JWT to verify a user’s identity and to extract other user information. This JWT will only be visible from inside the cluster and is never exposed to the end-user, thus providing a protected layer of security.</p><p><strong>OpenResty</strong></p><p><a href="http://openresty.org/en/">OpenResty</a> is a flavor of NGINX with added Lua support which adds the ability to provide some core logic inside of the proxy server.</p><p>When a request flows into nginx, theauth_request nginx module is used to invoke a custom OpenResty Lua library which checks for a valid session in the request. If the session is invalid, it redirects the request to the user authentication page. In case of a valid session, a JWT is generated using the session variables, and the request is forwarded to the kubernetes ingress controller for service resolution with the JWT added as a header.</p><p><strong>JWT</strong></p><p><a href="https://jwt.io/">JSON Web Tokens</a> are an open source industry standard outlined in <a href="https://tools.ietf.org/html/rfc7519">RFC 7519</a>. They are used as a method for securely representing claims between parties.</p><p>A JWT consists of three base64 encoded strings representing a header field, a payload field, and a signature. The header contains information about reading the JWT (the signature algorithm, the type of token). The payload field contains a series of claims (key-value pairs). The signature is a <a href="https://en.wikipedia.org/wiki/Digital_signature">digital signature</a> of the header and payload fields.</p><p>For example, the following:</p><pre>{<br>  &quot;alg&quot;: &quot;HS256&quot;,<br>  &quot;typ&quot;: &quot;JWT&quot;<br>},<br>{<br>  &quot;sub&quot;: &quot;1234567890&quot;,<br>  &quot;name&quot;: &quot;John Doe&quot;,<br>  &quot;admin&quot;: true<br>}</pre><p>would become the encoded JWT:</p><pre>eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeFONFh7HgQ</pre><p>where the last field is the signature of the first two fields. The fields in the payload are application dependent, although some standard fields are defined in the RFC.</p><p>At Namely, we use the JWTs as the source of authentication information. With this standardized method of propagating authentication information, we do not need to maintain multiple headers or multiple calls to the session store to get the user information.</p><p><strong>Authenticated vs Unauthenticated</strong></p><p>We use a custom session library built on top of the openresty session management extension which allows us to store session information in a configurable store, such as a cookie (client side) or Redis (server side). Upon intercepting a request, the lua code checks for the presence of two properties to verify the authenticity of the request. The request, if authenticated, would need to have a valid custom session cookie or a session header. If either of these two are present, openresty calls our custom session store to validate the session using a set criteria including, but not limited to, expiration and user validity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*RvkqpQuensI-K8uBilJv2w.png" /><figcaption>Authentication Flow</figcaption></figure><p><strong>Implementation Details</strong></p><p>We used the <a href="http://nginx.org/en/docs/http/ngx_http_auth_request_module.html">auth_request</a> nginx module to authenticate requests. The catch-all block for the nginx location looks like this</p><pre>location / {<br>  auth_request /auth-service;<br>  proxy_pass $ingress_upstream;<br>}</pre><p>The auth_request module requires the /auth_service endpoint to evaluate the request and return with either a HTTP Status code 401 for unauthenticated requests or a status code 200 for authentication requests. The ingress_upstream variable points to the kubernetes ingress controller which resolves the upstream service using the ingress rules.</p><p>The auth_service location encapsulates a content_by_lua_block, which acts as a “content handler” and executes the Lua code specified. Upon interception of the request, the lua code checks for the presence of a valid custom session cookie or a session header to verify the authenticity of the request. If either of these properties are present, openresty calls our custom session store to validate the session using a set criteria including, but not limited to, expiration, user validity and other domain information.</p><pre>location /auth-service {<br>  internal;<br>  content_by_lua_block {<br>    local session_id = session.get_cookie()</pre><pre>    -- If no session cookie<br>    if not session_id or session_id == nil then<br>      ngx.status = 401<br>      ngx.exit(ngx.OK)<br>    end</pre><pre>    -- If session cookie is nil, check request headers<br>    if session_id == nil then<br>      session_id = ngx.var.http_x_shared_session_id<br>    end</pre><pre>    -- Obtain the JWT and check validity<br>    local res, err = session.get_JWT(session_id)<br>    if not jwtutil.validate_jwt(res.jwt.token) then<br>      ngx.status = 401<br>      ngx.exit(ngx.OK)<br>    end</pre><pre>    add_bearer_header(res.jwt.token)</pre><pre>    ngx.status = 200<br>    ngx.exit(ngx.OK)<br>  }<br>}</pre><h3>Secure By Default</h3><p>Secure by default means that the default configuration of our proxy server provides implicit security. By handling authentication for the default routing location /, we decided that all routes should be secure and require a valid session. However, there is a need to allow unauthenticated requests to flow through to public resources and to maintain the ability to alter the public resource behavior if the user is authenticated.</p><p>To facilitate this use case, we maintain a list a public_routes which can be accessed without an authenticated request. With these unauthenticated requests, the lua code checks for the presence of the route being accessed in the public_route and redirects the request to the downstream service.</p><p>Constructing this list of public_routes was an uphill battle. Namely did not have explicit lists of public resources. This was due to the fact that all end services were previously responsible for their own authentication. Thus, that logic was buried within those services. It would have helped if, from the start, Namely had maintained a list of public routes to allow for easier extraction into a microservice architecture.</p><h3><strong>Initial Results</strong></h3><p>We have already started seeing the benefits of the strategy of authenticating at the Edge. Now, individual service developers do not have to worry about authentication. This, in turn, means fewer programming errors and a faster turnaround time for productionalizing a new service. There has been a reduction in the tight coupling of several services by removing large blocks of authentication code from existing applications.</p><blockquote>Now, individual service developers do not have to worry about authentication. This, in turn, means fewer programming errors and a faster turnaround time for productionalizing a new service.</blockquote><h3>Future Work</h3><p>One of the challenges we will tackle later this year is around providing the same authentication mechanism for the Namely API. The API is publicly used by multiple clients, including our mobile app, and uses the OAuth flow for authentication. This is something we are planning to build as an openresty module for the API requests to follow similar internal authentication standard.</p><p>In closing, we are thrilled at the flexibility that Authentication at the Edge has provided us. Openresty has been highly performant, and placing the onus of authentication at the outermost layer made our ecosystem boundaries clean and secure. Namely engineers are now able to develop services more rapidly and grow while continuing to adhere to best security practices.</p><p><em>Interested in working on other projects like this? Check out our </em><a href="https://www.namely.com/careers/"><em>Careers Page</em></a><em>!</em></p><p><em>This would not have been possible without the help of multiple teams at Namely that enabled us to develop this efficiently and collaboratively. Additionally, I’d like to thank Nicholas Narh and Martin Kess as the other two primary contributors to this project. Finally, I would like to thank Sid Gopinath and Mike Hamrah for their immense help in writing this article!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec912f118d1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/authentication-at-namely-moving-to-the-edge-ec912f118d1a">Authentication at Namely: Moving to the Edge</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Crash Course For Running Istio]]></title>
            <link>https://medium.com/namely-labs/a-crash-course-for-running-istio-1c6125930715?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/1c6125930715</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[envoy-proxy]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <dc:creator><![CDATA[Shray Kumar]]></dc:creator>
            <pubDate>Thu, 17 Jan 2019 21:52:34 GMT</pubDate>
            <atom:updated>2019-01-17T21:52:33.971Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*36mSzheRxdoU-71LY8wvOg.png" /><figcaption>Istio Service Mesh</figcaption></figure><p>At Namely we’ve been running with Istio for a year now. Yes, that’s pretty much when it first came out. We had a major performance regression with a Kubernetes cluster, we wanted distributed tracing, and used Istio to bootstrap Jaeger to investigate. We immediately saw the potential of a service mesh as it relates to our infrastructure and decided to make an investment in the tool.</p><p>It hasn’t always been the smoothest ride, but we have learned a <em>ton</em> about how it works and how to operate it. This post — the start of series — hopes to explain how Istio integrates with Kubernetes and some operational observations we’ve made along the way. We’ll go into some technical details, but mostly keep it high level, and with additional posts to come.</p><h3>What is Istio?</h3><p>Istio is a service mesh configuration engine. It reads the state of a Kubernetes cluster and performs updates to L7 (HTTP and gRPC) proxies that are initialized as “sidecars” to Kubernetes pods. These sidecars are Envoy containers that have been setup to read configuration from the Istio Pilot API (also a gRPC service) and then route traffic based on that configuration. The powerful L7 proxy under the hood allows us to leverage features such as metrics, tracing, retry logic, circuit breaking, load balancing and canary deployments.</p><h3>Starting from the Beginning — Kubernetes</h3><p>When using Kubernetes you create a Pod using something like a Deployment, StatefulSet, or just by creating a vanilla pod without a higher-level controller. Kubernetes then does its best to maintain a “desired state” by creating the pod(s) in your cluster on some node, making sure they start, and restart if they crash. When a pod is first created Kubernetes goes through its API lifecycle and ensures that each step succeeds before actually creating the pod in the cluster.</p><p>The API lifecycle involves the following phases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0C4pDnPiOL1zmupe12PZ_Q.png" /><figcaption>Thanks Banzai Cloud for a great graphic.</figcaption></figure><p>The one piece of this graph to call out is the “Mutating Admission Webhooks”. These are a special piece of the Kubernetes lifecycle that allows customization of resources before they are committed to the etcd store, the ‘source of truth’ for Kubernetes configuration. This is where Istio gets a lot of its magic.</p><h3>Mutating Admission Webhooks</h3><p>When an individual Pod is created (either via kubectl or a Deployment resource), it goes through this same lifecycle, hitting mutating admission webhooks which modify the pod before it actually gets applied.</p><p>During the process of installing Istio the istio-sidecar-injector is added as a mutating webhook configuration resource:</p><pre>$ kubectl get mutatingwebhookconfiguration<br>NAME                     AGE<br>istio-sidecar-injector   87d</pre><p>And viewing the configuration:</p><pre>apiVersion: admissionregistration.k8s.io/v1beta1<br>kind: MutatingWebhookConfiguration<br>metadata:<br>  labels:<br>    app: istio-sidecar-injector<br>    chart: sidecarInjectorWebhook-1.0.4<br>    heritage: Tiller<br>  name: istio-sidecar-injector<br>webhooks:<br>- clientConfig:<br>    caBundle: redacted<br>    service:<br>      name: istio-sidecar-injector<br>      namespace: istio-system<br>      path: /inject<br>  failurePolicy: Fail<br>  name: sidecar-injector.istio.io<br>  namespaceSelector:<br>    matchLabels:<br>      istio-injection: enabled<br>  rules:<br>  - apiGroups:<br>    - &quot;&quot;<br>    apiVersions:<br>    - v1<br>    operations:<br>    - CREATE<br>    resources:<br>    - pods</pre><p>This tells Kubernetes to send all Pod creation events to the istio-sidecar-injector service in the istio-system namespace if the namespace has the istio-injection=enabled label. The injector service then will modify the PodSpec to include two additional containers, one temporarily for setting up the proxy rules, and another for the actual proxying. The sidecar injector service adds these two additional containers via a template; the template is found in the istio-sidecar-injector configmap. This process is otherwise known as “sidecar’ing”.</p><h3>Sidecar’ed Pods</h3><p>Istio sidecars are components that are more or less “magical”. Istio does a good job of hiding these details from you once you’ve installed the project. But knowing what they are and how they work is helpful in situations where you need to debug network requests.</p><h3>The Init and Proxy Containers</h3><p>Kubernetes allows you to initialize temporary “one-off” containers before engaging your primary processes, called “init containers”. These are great to perform tasks like bundling assets, database migrations, or in Istio’s case: setting up network rules.</p><p>Istio uses Envoy for proxying all of the requests pods receive to the correct destination. To make this a reality, Istio creates iptables rules that sends outbound / inbound traffic directly to Envoy. Envoy is able to proxy traffic to its original destination as seamlessly as possible. You can think of this traffic as taking a small “detour” to allow things such as distributed tracing, request metrics, and policy enforcement. To see how Istio creates these iptable rules, you can <a href="https://github.com/istio/istio/blob/master/tools/deb/istio-iptables.sh">view this file</a> on the Istio repository.</p><p><a href="http://twitter.com/jimmysongio">@jimmysongio</a> has an excellent flowchart demonstrating the relationship between the iptables rules and the Envoy proxy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ADBvKrIe0gkifGNhwy_Ng.jpeg" /><figcaption>Envoy to Envoy traffic</figcaption></figure><p>Because Envoy receives all inbound and outbound traffic, that means all traffic now is “Envoy to Envoy”, as shown in the graph above. The Istio Proxy is another container that is added to all pods that the Istio Sidecar Injector modifies. This container is what is running an Envoy process that is configured to receive all traffic for the pod (with some exceptions, such as traffic exiting your Kubernetes cluster).</p><p>The Envoy process is also configured to discover all of the routes via the Envoy v2 API that Istio implements.</p><h3>Envoy and Pilot</h3><p>Envoy on its own doesn’t contain the logic to discover pods and services in your cluster. Envoy acts as a <em>data plane</em> and wants a <em>control plane, </em>some kind of “puppet master”, to handle the configuration behavior. Envoy has a configuration flag, asking for a host/port of a service to receive this configuration via a gRPC API. Istio, through its Pilot service, implements (read: fulfills the requirements) for this gRPC API. Envoy connects to this API based on the sidecar’ed configuration injected via the mutating webhook. This API provides all traffic rules that Envoy needs to discover and route traffic with the cluster. This is the ‘service mesh’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*VXRosxzM2KO_HOzYx4N08g.png" /><figcaption>Pod &lt;-&gt; Pilot communication</figcaption></figure><p>Pilot is also hooked up to the Kubernetes cluster and reads the state of the cluster and listens for updates. It keeps track of Pods, Services, and Endpoints. By keeping track of these resources in your Kubernetes cluster, it can then provide the proper configuration to all of the Envoy sidecars that are connected to Pilot, acting as a bridge between Kubernetes state and Envoy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*WHtyuKzEUA7n5ntKsTk_IQ.png" /><figcaption>Pilot to Kubernetes</figcaption></figure><p>When Pods, Services, or Endpoints are updated or created in Kubernetes, those changes are sent to Pilot, which will then send the proper configuration down to every connected Envoy instance.</p><h3>What Configuration Is Being Sent?</h3><p>So now that we know how Envoy / Pilot interact at a high level, let’s talk about the type of configuration that Envoy receives from Istio’s Pilot.</p><p>Out of the box, Kubernetes handles most of your networking needs with a type called Service, which manages a type called Endpoint. You can see a list of Endpoints by running:</p><pre>kubectl get endpoints</pre><p>This is a list of IPs and Ports that exist within the cluster and the target they reference (typically a Pod, created from a Deployment). They’re also a very critical part of how Istio configures and sends routing information to Envoy.</p><h3>Services, Listeners, and Routes!</h3><p>When you create a Service resource in a Kubernetes cluster, you include a set of “match labels” that will select all pods that match those labels. When you send traffic to the Service IP, Kubernetes will then pick a Pod to send the traffic to. For example, performing:</p><pre>curl my-service.default.svc.cluster.local:3000</pre><p>Will first hit the virtual IP that has been assigned for the Service my-service in the default namespace, that IP will then forward the traffic to a pod that matches the service’s label selector.</p><p>Istio and Envoy change this logic slightly. Istio configures Envoy based on Services and Endpoints in the Kubernetes cluster it is operating within to leverage Envoy’s intelligent routing and load balancing, bypassing the Kubernetes Service. Instead of proxying through a single ip, Envoy knows and connects directly to pod IPs. <strong>Istio maps Kubernetes configuration to Envoy’s configuration to make this possible.</strong></p><p>The lingo between Kubernetes, Istio, and Envoy all vary slightly so it can be confusing to know which piece maps to what.</p><h3>Services</h3><p>A Service in Kubernetes maps to a <strong>Cluster</strong> in Envoy. An Envoy Cluster is something that contains a list of <strong>Endpoints</strong>, which are the IPs (or hostnames) of instances to handle requests. We can see a list of the Clusters that have been configured in an Istio sidecar’d pod by running istioctl proxy-config cluster &lt;pod name&gt;. Running this command shows this pod’s current understanding of the “state of the world”. This is an example from one of our environments.</p><pre>$ istioctl proxy-config cluster taxparams-6777cf899c-wwhr7 -n applications<br>SERVICE FQDN                                                              PORT      SUBSET     DIRECTION     TYPE<br>BlackHoleCluster                                                          -         -          -             STATIC<br>accounts-grpc-gw.applications.svc.cluster.local                           80        -          outbound      EDS<br>accounts-grpc-public.applications.svc.cluster.local                       50051     -          outbound      EDS<br>addressvalidator.applications.svc.cluster.local                           50051     -          outbound      EDS</pre><p>Now, if we look, we’ll see the same services exist in this namespace:</p><pre>$ kubectl get services<br>NAME                                     TYPE           CLUSTER-IP   EXTERNAL-IP        PORT(S)                         <br>accounts-grpc-gw                         ClusterIP      10.3.0.91    &lt;none&gt;             80/TCP                          <br>accounts-grpc-public                     ClusterIP      10.3.0.202   &lt;none&gt;             50051/TCP                       <br>addressvalidator                         ClusterIP      10.3.0.56    &lt;none&gt;             50051/TCP                       </pre><p>If you’re wondering how Istio determines what protocol your service uses, Istio configures protocols for service manifests by reading the name field on port entries.</p><pre>$ kubectl get service accounts-grpc-public -o yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: accounts-grpc-public<br>spec:<br>  ports:<br>  - name: grpc<br>    port: 50051<br>    protocol: TCP<br>    targetPort: 50051</pre><p>By setting your port to grpc or prefixing it with grpc-, Istio will configure the service specifically with the HTTP2 protocol. We learned the hard way on how Istio uses these port names when our proxy configs got screwed up because we didn’t use http or grpc prefixes…</p><p>Using kubectl and port-forwarding the Envoy admin page, you can observe that account-grpc-public’s endpoints are implemented by Pilot as a cluster in Envoy with HTTP2 protocol options set, validating our assumptions:</p><pre>$ kubectl -n applications port-forward otherpod-dc56885ff-dqc6t 15000:15000 &amp;<br>$ curl <a href="http://localhost:15000/config_dump">http://localhost:15000/config_dump</a> | yq r -</pre><pre>...<br>    - cluster:<br>        circuit_breakers:<br>          thresholds:<br>          - {}<br>        connect_timeout: 1s<br>        eds_cluster_config:<br>          eds_config:<br>            ads: {}<br>          service_name: outbound|50051||accounts-grpc-public.applications.svc.cluster.local<br>        http2_protocol_options:<br>          max_concurrent_streams: 1073741824<br>        name: outbound|50051||accounts-grpc-public.applications.svc.cluster.local<br>        type: EDS<br>...</pre><p>Port 15000 is the <a href="https://www.envoyproxy.io/docs/envoy/latest/operations/admin">Envoy admin page</a> that is available in each sidecar.</p><h3>Listeners</h3><p>Similarly, Listeners incorporate Kubernetes endpoints in order to allow traffic into Pods, the address validator service has one endpoint here:</p><pre>$ kubectl get ep addressvalidator -o yaml<br>apiVersion: v1<br>kind: Endpoints<br>metadata:<br>  name: addressvalidator<br>subsets:<br>- addresses:<br>  - ip: 10.2.26.243<br>    nodeName: ip-10-205-35-230.ec2.internal<br>    targetRef:<br>      kind: Pod<br>      name: addressvalidator-64885ccb76-87l4d<br>      namespace: applications<br>  ports:<br>  - name: grpc<br>    port: 50051<br>    protocol: TCP</pre><p>This translates to a single listener on an address validator pod, listening on port 50051:</p><pre>$ kubectl -n applications port-forward addressvalidator-64885ccb76-87l4d 15000:15000 &amp;<br>$ curl <a href="http://localhost:15000/config_dump">http://localhost:15000/config_dump</a> | yq r -</pre><pre>...<br>    dynamic_active_listeners:<br>    - version_info: 2019-01-13T18:39:43Z/651<br>      listener:<br>        name: 10.2.26.243_50051<br>        address:<br>          socket_address:<br>            address: 10.2.26.243<br>            port_value: 50051<br>        filter_chains:<br>        - filter_chain_match:<br>            transport_protocol: raw_buffer<br>...</pre><h3>Routes</h3><p>The Istio project does not use the standard Kubernetes Ingress object, and instead opts for a more abstract and powerful custom resource known as the VirtualService. A VirtualService allows you to match routes to upstream clusters by attaching them to a gateway. This idea can be likened to using Kubernetes Ingress along with an Ingress Controller.</p><p>In Namely’s case, we use the Istio’s Ingress-Gateway to funnel all of our internal GRPC traffic:</p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: Gateway<br>metadata:<br>  name: grpc-gateway<br>spec:<br>  selector:<br>    istio: ingressgateway<br>  servers:<br>  - hosts:<br>    - &#39;*&#39;<br>    port:<br>      name: http2<br>      number: 80<br>      protocol: HTTP2<br>---<br>apiVersion: networking.istio.io/v1alpha3<br>kind: VirtualService<br>metadata:<br>  name: grpc-gateway<br>spec:<br>  gateways:<br>  - grpc-gateway<br>  hosts:<br>  - &#39;*&#39;<br>  http:<br>  - match:<br>    - uri:<br>        prefix: /namely.address_validator.AddressValidator<br>    retries:<br>      attempts: 3<br>      perTryTimeout: 2s<br>    route:<br>    - destination:<br>        host: addressvalidator<br>        port:<br>          number: 50051</pre><p>Upon first glance, the above example may be difficult to digest. What’s occurring in the background is that there is an Istio-IngressGateway deployment that captures which endpoints to serve based upon the istio: ingressgateway selector. In this example, the IngressGateway serves traffic on all domains on port 80 on the HTTP2 protocol. A VirtualService is implemented that fulfills routes for this gateway, matches on the prefix /namely.address_validator.AddressValidator and passes them onto the upstream addressvalidator service on port 50051 with a 2 second retry rule.</p><p>Port forwarding the Istio-IngressGateway pod and viewing its Envoy configuration confirms the above VirtualService:</p><pre>$ kubectl -n istio-system port-forward istio-ingressgateway-7477597868-rldb5 15000</pre><pre>...<br>          - match:<br>              prefix: /namely.address_validator.AddressValidator<br>            route:<br>              cluster: outbound|50051||addressvalidator.applications.svc.cluster.local<br>              timeout: 0s<br>              retry_policy:<br>                retry_on: 5xx,connect-failure,refused-stream<br>                num_retries: 3<br>                per_try_timeout: 2s<br>              max_grpc_timeout: 0s<br>            decorator:<br>              operation: addressvalidator.applications.svc.cluster.local:50051/namely.address_validator.AddressValidator*<br>...</pre><h3>Random Things We Googled When Operating Istio</h3><p><strong>I’m receiving a 503 or 404 request!</strong><br>There are a variety of reasons these errors can occur, but most commonly:</p><ul><li>Your application’s sidecars cannot reach out to Pilot (confirm Pilot is running)</li><li>The Kubernetes service manifest is not configured with the correct protocol</li><li>Your VirtualService/Envoy Configuration may be capturing the route in the wrong upstream cluster. Start with an edge service where you expect ingress and study its Envoy logs, or use a tool such as Jaeger to confirm where failures are occurring.</li></ul><p><strong>What do the acronyms NR/UH/UF mean in Istio Proxy Logs?</strong></p><ul><li>NR — No Route</li><li>UH — Upstream Unhealthy</li><li>UF — Upstream Failure</li></ul><p>Read more on this topic on <a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log">Envoy’s website</a>.</p><p><strong>What are some HA considerations for Istio?</strong></p><ul><li>Adding NodeAffinity to critical Istio components to evenly distribute pods across different availability zones and increasing the minimum replica count.</li><li>Run a newer version of Kubernetes with Horizontal Pod Autoscaling enabled. This will allow mission critical pods to scale up and down as load increases/decreases.</li></ul><p><strong>My Cronjob never ends, what could be the problem?</strong><br>After the primary workload is finished, the sidecar container continues to stay up. You can work around this by disabling sidecars on cronjobs by adding the sidecar.istio.io/inject: “false” annotation to the <em>PodSpec</em>.</p><p><strong>How do you install Istio?</strong><br>We use Spinnaker for our deployments, but in general the process entails pulling down the latest Helm charts, making our in-house modifications, using helm template -f values.yml and committing those files to Github to compare changes before we apply them via kubectl apply -f -. We take this approach in order to confirm there are no CRD or API changes across versions we may not have accounted for.</p><p>Thanks to <a href="https://medium.com/u/5045186675bf">Bobby Tables</a> and <a href="https://medium.com/u/58ae22fa494">Michael Hamrah</a> for his contributions to this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c6125930715" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/a-crash-course-for-running-istio-1c6125930715">A Crash Course For Running Istio</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Following an Internal Open Source Model of Software Development]]></title>
            <link>https://medium.com/namely-labs/following-an-internal-open-source-model-of-software-development-4da7731067db?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/4da7731067db</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[engineering-culture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Abraham Heidebrecht]]></dc:creator>
            <pubDate>Fri, 28 Sep 2018 20:59:44 GMT</pubDate>
            <atom:updated>2018-09-28T20:59:44.391Z</atom:updated>
            <content:encoded><![CDATA[<p>At Namely, we have implemented an internal open source model to improve our velocity, ship more features, and fix more bugs. We use this model in a couple of ways, the most important of which is to build microservices. We also use an open source mindset in the way most people are probably familiar with: publishing and sharing libraries. For .NET, we use an internal nuget feed (Nuget is the package management system for .NET). We also have internal Ruby gems and common Go packages used for our services. For this article I’ll be talking specifically about our .NET efforts. However, this will be broadly applicable to other languages and frameworks.</p><h3>Using Internal Shared Libraries</h3><p>There are a couple problematic models for writing internal shared libraries:</p><p>1. One team writes all the shared code but doesn’t ship anything that uses it. All teams use the codebase. This method often results in a codebase that doesn’t solve the problems that product teams are facing. The code is challenging to use, as it is often over genericized or deeply flawed in ways that are only obvious once you start using it.</p><p>2. Everybody theoretically owns the shared codebase, but only a few people (many times one person) actually contribute to the repo. Like a baby open source project with one maintainer, this shared code solves the problem that the owner needed to solve. But, without input from other teams, it won’t be solving their problems and will therefore not be used. This model often devolves into separate shared code bases for different teams or a shared library that doesn’t get updated in years, especially once the one person who was working on it leaves the company.</p><p>We started our shared library with the second model above, and in fact, we started multiple separate repos for shared libraries. We eventually realized that the libraries weren’t being contributed to by enough developers, and as a consequence, they weren’t as useful as they could and should be. Furthermore, new versions of the libraries were difficult to push because developers had to package and push to our internal nuget from their local machines. We felt it was vital to fix these problems, and we looked at how successful open source projects operate. We felt by adopting some of their practices, we could make the library better and increase adoption across the organization.</p><p>We consolidated the libraries into one repo with an easy CI/CD process for deployment. Perhaps unintuitively, we also made it <em>harder</em> to get a PR merged — we added the requirement for a second reviewer and instituted a soft policy of requiring at least one reviewer from another team. We also adopted a process to drive code quality by strictly adhering to Semantic Versioning of new releases. This means we require new large features to go through a pre-release/review/iteration cycle where other teams actually use the feature and provide meaningful feedback on its use before we merge it to master.</p><p>That sounds like a lot of process — and it is. The point is to make sure our shared code is well tested, understood by other teams, and has appropriate extensibility points. Because it is a lengthy process, we typically only do this for larger features. When writing bug fixes or smaller features, a developer will submit a PR directly to master and deploy once merged. We still require sufficient test coverage and multiple code reviews, but nothing else. The point is to get code to our colleagues that is well thought out and tested, not to tick some boxes for the sake of process!</p><p>Since this is an internal open source project, all devs are allowed and encouraged to contribute to the repo. As an example, one of our Go developers was working on a feature that every service at Namely should utilize. They submitted the PR to our shared repo (one of the first times they had ever written C#!) and got some meaningful feedback from myself and a couple of other devs. Once it was addressed, they were able to deploy the package to nuget, and now all our .net services have the feature! All these changes have proven successful, as every .net team (and one Go team) has contributed at least one feature to the shared codebase.</p><h3>Using Open Source Practices to Write Microservices</h3><p>In addition to the shared library, we use an internal open source mindset to develop new microservices. At Namely, we have not one, but two large monolithic code bases that we’re actively working to break up (you all agree that is the correct architecture for a cloud platform, right?).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*M72pSyvTmowxz4gC" /></figure><p>We have faced numerous challenges as we rebuild our platform using a modern architecture, but one of the hardest questions to answer is: What do we build first?</p><p>Our product requirements drove the first few services — we had new features to implement, so instead of adding them to the monoliths, we added them to independent services. We’ve organized our services into verticals based on the business entity. For example, my team owns the service for vertical A, but we depend on data from verticals B, C, and D. Other teams own these services. In the monolith, the data we needed was in the database, so we would simply query the appropriate table. In the microservices world, each of those verticals has a service that owns the data we need.</p><p>So, we are in the monolithic world, building Service A, but we need data from other services. But they don’t exist yet (or if they do, they don’t yet have the endpoints we need). We could wait for the other vertical owners to prioritize our needs, or we could write it ourselves.</p><p>This is where the open source mindset really helps us. Instead of waiting for vertical owners to build the endpoints we need, my team can build the endpoints we need with their input. Namely uses GRPC and protobuf to define the service contract, so the first step is to define the new endpoint contract in a proto. My team will create a PR with the endpoint, and the vertical owner will review it with a critical eye towards the future state of their service. The goal is to build the service the correct way initially, not just relieve the immediate needs of our service (and accidentally create new tech debt for another team)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*st9PuXhLHAEXumrz" /></figure><p>Once we’ve merged the proto, somebody has to build the feature. If the other team has already prioritized the work (or has available bandwidth to fit it in), they’ll implement the new method. If not, my team will build the endpoints and submit PRs to the other teams’ repos. This allows the other team to ensure the code is up to their standards, as well as allowing them to understand the changes. Since they’re going to own the code after we merge, they’re invested in providing appropriate feedback on our PRs and rightfully are very strict when it comes to demanding full test coverage.</p><p>This has allowed my team to continue shipping new features for our product in the microservices world, while not requiring other teams to focus on our feature. Likewise, other teams can and do submit PRs for our services, and we don’t have to devote too many resources to unblock them! The open source mindset has allowed us to keep our velocity up without blocking, or being blocked by, other teams.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4da7731067db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/following-an-internal-open-source-model-of-software-development-4da7731067db">Following an Internal Open Source Model of Software Development</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sailing With Spinnaker: Deploying a Better CD Pipeline To Dozens of Engineers and Services]]></title>
            <link>https://medium.com/namely-labs/sailing-with-spinnaker-deploying-a-better-cd-pipeline-to-dozens-of-engineers-and-services-b21dcfa0e1a1?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/b21dcfa0e1a1</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[spinnaker]]></category>
            <dc:creator><![CDATA[Shray Kumar]]></dc:creator>
            <pubDate>Tue, 29 May 2018 19:27:10 GMT</pubDate>
            <atom:updated>2018-05-29T19:27:09.794Z</atom:updated>
            <content:encoded><![CDATA[<p>Namely is growing rapidly: we have dozens of engineers pushing out new features across a variety of services every week. We recently rolled out <a href="https://medium.com/namely-labs/spinnaker-and-kubernetes-for-continuous-delivery-561aa472a0e8">Spinnaker to help our engineers reliably deliver code with zero impact to our customers</a>. We were previously using a set of simple bash scripts powered by Jenkins and needed a better solution for visibility, rollbacks, and configuration for services running on Kubernetes. Spinnaker creates a powerful abstraction for engineers to easily understand which code is running in which environment and seamlessly move code between environments.</p><p>Migrating to Spinnaker was no small endeavor. We needed to ensure running Spinnaker was bulletproof. If our CD system fails we are unable to deliver new code and we are unable to mitigate issues with code fixes. We also need to ensure our engineers are comfortable using Spinnaker for code promotions, production deployments, rollbacks and scaling. In this post we are going to cover how we introduced Spinnaker into our ecosystem, learned to operate it reliably, and what abstractions we’ve created around Spinnaker to ease the developer experience.</p><p>Running Spinnaker in a highly available manner required us to understand its architecture at a deeper level. This is essential for an engineering organization where dozens of engineers will depend on Spinnaker for their work. Spinnaker is a system <a href="https://www.spinnaker.io/reference/architecture/">designed around microservices</a> where various components work together independently. It also connects with multiple other services (Jenkins, Kubernetes). This creates numerous opportunities for failure and micro-outages. When pipelines fail the SRE team must quickly identify whether the problem lies within Spinnaker, within Kubernetes or within the deployed service itself.</p><p>Understanding the Spinnaker architecture allows us to quickly shed light on an event such as clouddriver — the cloud-interface abstraction — losing connectivity to a Kubernetes cluster or a Redis pod failure which is used as the Spinnaker state store. Developing this understanding, designing a deployment strategy, and deciding on how engineers will use Spinnaker requires a multi-pronged approach.</p><h4>Read The Code</h4><p>Reading the code for Spinnaker’s microservices is absolutely paramount to gain operational understanding. This is a powerful yet missed tool SREs should employ when working with systems, whether developed internally or externally. For instance, understanding <a href="https://www.spinnaker.io/reference/halyard/custom/">how to override custom profiles</a> was instrumental in tuning Spinnaker to achieve expected behavior during our PoC phase and resilience testing. Spinnaker is written in Java but Namely swarms around Ruby, Go and DotNet Core. <a href="https://www.travistomsu.com/2016/12/19/configuring-spinnaker/">We had to deep-dive into into Spring profiles</a> to understand Spinnaker configuration.</p><p>With this understanding we were able to speed up the rate at which clouddriver <a href="https://github.com/spinnaker/clouddriver/blob/master/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/cache/RedisCacheConfig.groovy#L59">polls agents</a> for an overall snappier experience at our size. Here’s how we added a custom override to the clouddriver-local.yml configuration:</p><pre>redis:<br> connection: ${services.redis.baseUrl:redis://localhost:6379}<br> poll:<br> intervalSeconds: 15</pre><p>By default, disabling the previous deployment in a spinnaker pipeline can take up to 90 seconds in a `disable cluster` stage. By increasing the frequency at which orca periodically polls clouddriver can dramatically speed up your pipeline execution. Here’s how you can lower that ask to 25 seconds:</p><pre>tasks:<br> disableClusterMinTimeMillis: 25000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*0nocv2qCwtyT9pSB." /></figure><h4>HA or Get Out</h4><p>Most open source tools offer a quickstart approach to get you up and running easily. Although it gets you using the tool quickly it is never appropriate for production use. Spinnaker is no different. A problem of setting up a highly available Spinnaker system is that it requires a persistent storage backing. Out of the box Spinnaker uses a single Redis instance to hold state. On a platform such as Kubernetes this can prove to be problematic when Spinnaker’s Redis pod fails and get redeployed. We came to this realization when older builds were randomly triggering from Jenkins and several builds were running concurrently on multiple environments. We switched to a persistent Redis cluster via ElasticCache after discovering the effects of what occurs when you <a href="https://medium.com/@rizza/bobby-tables-thats-right-elasticache-would-be-a-much-better-option-bbfdb46f5261">lose Redis connectivity</a> and the circuit breaker kicks in.</p><p>Additionally you need to ensure your polling mechanism can recreate state appropriately. This is handled by the Igor service. In the event of a catastrophic failure we want to ensure old builds are not sporadically triggering in Spinnaker. In igor-local.yml:</p><pre>spinnaker:<br> build:<br> pollInterval: 30<br> pollingSafeguard:<br> # A lower value can help prevent accidental re-triggers of pipelines, but may require hands-on operations if<br> # set low.<br> # See code documentation for more details. Class [PollingSafeguardProperties] in<br> # <a href="https://github.com/spinnaker/igor/blob/master/igor-web/src/main/groovy/com/netflix/spinnaker/igor/IgorConfigurationProperties.groovy#L47">https://github.com/spinnaker/igor/blob/master/igor-web/src/main/groovy/com/netflix/spinnaker/igor/IgorConfigurationProperties.groovy#L47</a><br> itemUpperThreshold: 1000</pre><p>Any HA solution requires multiple replicas to exist for mission critical components. Spinnaker is no exception. Critical spinnaker components include CloudDriver, which handles caching of your cloud agent data and determines the state of your deployments. Orca handles orchestration of tasks across Spinnaker. It’s essential to adjust the number of replicas based upon your workload across clusters. If too few resources are allocated you will get task queueing which will delay critical operations like rollbacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-EbQiJF2WotxUVw9." /></figure><h4>Learn From Failure (and Bugs)</h4><p>In order to offer our engineers a better deployment experience via Spinnaker we ensured that any unintended behavior in Spinnaker was investigated and some sort of remediation steps were offered (whether a process change, tooling enhancement, or bug fix). This was a time consuming but it is essential to investigate flakiness or inconsistency. We must ensure unintended behaviors do not bubble up to our production environment. Like any other software, Spinnaker has bugs. Some are caused by the ever changing Kubernetes platform, others by Spinnaker itself. Finding bugs and possible workarounds to them built up our confidence up in using Spinnaker. On a related note Namely <a href="https://landing.google.com/sre/book/chapters/postmortem-culture.html">has a healthy post-mortem culture</a> so we continuously learn from failure and build resiliency into our platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3_JCZNStMObAw79H." /></figure><h4>Migrate Incrementally, Build Automation</h4><p>In our move to using Spinnaker we agreed upon devising a migration strategy that would allow us to move specific projects over that were low risk and were less likely to cause production outages to our customer experience. After moving a handful of projects over we quickly realized that more tooling was requisite to empower our developers to stand up new projects on their own. By creating automation such as our open source <a href="https://medium.com/namely-labs/spinnaker-and-kubernetes-for-continuous-delivery-561aa472a0e8">k8s-pipeliner and estuary</a> we were able to ease engineers into adopting our new infrastructure with minimal impact in their day-to-day.</p><h4>Leverage the Community</h4><p>Open source software is more than sharing code. A healthy community leverages economies of scale: you may be a small organization, but many small distributed organization create a large, healthy community of users. Staying active in the <a href="https://join.spinnaker.io/">Spinnaker community</a> is an excellent way to stay up to date on how other companies are using Spinnaker. Spinnaker is a complex system with many moving parts. Understanding how other companies are using the product was crucial in determining where we should focus our efforts. In addition, watching the GitHub repository allows us to determine what areas of the project were receiving enhancements or bug fixes.</p><h4>Automate All the Things</h4><p>Our approach began by building Spinnaker naively by employing its setup guide and halyard. This worked well initially because it allowed us to understand its internals but grew out of control as we began to add multiple clusters, authentication, and alerting. In order to make our process reproducible and to be able to blow away an existing installation at any moment, we moved to using a terraform setup using remote-exec modules, templates, and hashicorp vault for certificate management. This allows us to treat our Spinnaker infrastructure like any other infrastructure. We version control deployment changes to our Spinnaker topology, configuration changes and secrets. This allows us to reproduce our environment and test new updates easily.</p><p><em>The author would like to thank Michael Hamrah for his contributions to the article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b21dcfa0e1a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/sailing-with-spinnaker-deploying-a-better-cd-pipeline-to-dozens-of-engineers-and-services-b21dcfa0e1a1">Sailing With Spinnaker: Deploying a Better CD Pipeline To Dozens of Engineers and Services</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spinnaker and Kubernetes for Continuous Delivery]]></title>
            <link>https://medium.com/namely-labs/spinnaker-and-kubernetes-for-continuous-delivery-561aa472a0e8?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/561aa472a0e8</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[spinnaker]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Bobby Tables]]></dc:creator>
            <pubDate>Thu, 26 Apr 2018 13:04:57 GMT</pubDate>
            <atom:updated>2018-04-26T13:23:35.112Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/140/1*xZG33HNDutUB3vDy1719Ag.png" /><figcaption>Spinnaker Logo</figcaption></figure><p>Here at Namely we’ve successfully adopted Spinnaker to handle deployments for all of our container based applications across three environments (integration, staging, and production). Spinnaker is a popular open-source tool from Netflix and a great tool for continuous deployment. Matched with the popular and versatile Kubernetes orchestration platform you have an amazing combination to push out new functionality to a microservice platform.. Along the way we learned a lot about what works and what doesn’t. We now have a process and tooling that allows teams to reliably deploy using Spinnaker to Kubernetes.</p><p>We decided to go with Spinnaker because it has native Kubernetes support, supports pipeline executions for complex automatic and manual deploy steps, and has native blue/green deployments and rollbacks baked in for safe releases and recovery. Previously we had a half-baked solution: we were simply running “kubectl apply”’s in a Jenkinsfile from a Jenkins job. Every project had a templated Kubernetes manifests checked into its repository. We would just swap out the image name to the currently built image. We called this “SimpleCD”, and even though it was conceptually simple it caused a lot of headaches. Rollbacks weren’t a thing, deployment logs were scattered, there was no auditing our current state, people would modify their scripts leading to inconsistency, and the deployment task was prone to failure. Jenkins isn’t great at deploying things, especially when they are complex.</p><p>We’ve deployed Spinnaker to Kubernetes in a separate AWS environment called “ops” which has been peered with our other AWS environments. We used <a href="https://www.spinnaker.io/setup/install/halyard/">Halyard</a> to configure most of the spinnaker deployment. There are little things that don’t work via the Halyard tool such as <a href="https://www.spinnaker.io/setup/security/authentication/x509/">configuring certificate authentication</a>, so we actually patch the secrets manifest that are deployed to Kubernetes that contains all of the Spinnaker configuration. Spinnaker is then given Kubernetes credentials for three Kubernetes environments: integration, staging, and production.</p><p>The primary reason we chose Spinnaker was its pipeline functionality for deploying our applications. At first we tried using the Docker registry push as a trigger for Spinnaker pipelines but it was slow due to the polling mechanism used to fetch new images from our internal Quay registry. We quickly pivoted from that to use Jenkins jobs as triggers for our pipelines.</p><p>This was a much better solution over SimpleCDas it allowed us to parameterize our pipelines using the “Property File” configuration option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDP9EvBtyFdda2KE3hRgyw.png" /><figcaption>Jenkins config with a property file</figcaption></figure><p>We then made a simple modification to our Jenkins jobs that emitted a properties file that Spinnaker could pick up and read from:</p><pre>sh “””echo ‘docker_image = ${DOCKER_REPOSITORY}:${COMMIT_HASH}’ &gt;&gt; .ci-properties”””<br>sh “””echo ‘docker_tag = ${COMMIT_HASH}’ &gt;&gt; .ci-properties”””<br>archiveArtifacts ‘.ci-properties’</pre><p>This allowed us to configure our Spinnaker pipelines using the .ci-properties values easily. Our JSON configuration for image declaration changed to this:</p><pre>“imageDescription”: {<br> “account”: “namely-registry”,<br> “fromTrigger”: false,<br> “imageId”: “${ trigger.properties[‘docker_image’] }”,<br> “organization”: “namely”,<br> “registry”: “registry.namely.land”,<br> “repository”: “namely/platform-nginx”,<br> “tag”: “${ trigger.properties[‘docker_tag’] }”<br>},</pre><p>This works very well but it <em>does</em> break the Spinnaker UI meaning we had to deal with the raw json configuration. No one likes writing JSON by hand, so this lead us to creating a tool to handle this problem for us.</p><h3>k8s-pipeliner</h3><p>Spinnaker allows you to configure a pipeline with just JSON, which is great because it meant we can create a tool to render this JSON for us. But we needed to create a format to support this.</p><p>Since we already had our projects all defined in Kubernetes deployment manifest files, we just needed to make something that would basically glue Spinnaker pipeline configuration and Kubernetes specific semantics together. So we created the <a href="https://github.com/namely/k8s-pipeliner">k8s-pipeliner tool</a> to do just that.</p><p>The basic premise of this tool was “let’s just use Kubernetes manifests to define how a project <em>runs</em>” and “Let’s use Spinnaker to define how a project <em>deploys</em>”. The tool uses a relatively simple (but could be improved) format to reference manifests and how they fit into Spinnaker pipeline stages. After defining the pipeline.yml and manifest definition, engineers could run the tool locally to print out Spinnaker pipeline JSON, and paste that into the UI to configure an entire execution.</p><p>After you generate this JSON, you can create an entire pipeline easily:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhZmPlsTJu7Mk45zUA6czw.gif" /></figure><p>Manual isn’t great though, right? We can do better than that.</p><h3>Estuary</h3><p>Having engineers copy and paste JSON is error prone and just feels gross, so we built another service to automate this process. Since k8s-pipeliner is written in Go, this meant we could import the packages that generate this JSON in a new service we call “Estuary”.</p><p>Using GitHub webhooks, we’ve configured all of our repositories to hit an endpoint in Estuary whenever a push to the main branch (master in our case) occurs. Estuary then clones the entire repository, generates the Spinnaker pipeline JSON, and does a POST request to the Spinnaker API:</p><pre>level=info msg=&quot;handling github push event&quot; clone_url=&quot;<a href="https://github.com/namely/namely.git">https://github.com/namely/namely.git</a>&quot;<br>level=info msg=&quot;looking for pipeline&quot; total=2<br>level=info msg=&quot;found pipeline in Spinnaker&quot; pipeline_id=e2ff76c7-03af-4fce-bc58-964eb83abded<br>level=info msg=&quot;received status code&quot; response_code=200<br>level=info msg=&quot;updated pipeline&quot; application_name=hcm pipeline_id=e2ff76c7-03af-4fce-bc58-964eb83abded pipeline_name=hcm</pre><p>This means if you modify your pipeline.yml or manifest definitions, upon merge your pipeline will automatically be updated via Estuary. There are other benefits as well. For example, we can inject in configuration for <em>all</em> pods being deployed so pipeline.yml files don’t need to include it themselves, making Pipeline definitions smaller per project.</p><p>This project will be available as open source soon as well.</p><h3>Deployment Events</h3><p>Whenever a project deploys, it’s nice to know when it happens. Spinnaker has a small but powerful feature of dispatching webhooks whenever a task completes. We use this to emit DataDog events anytime a pipeline completes. This allows us to graph events over metrics so if a deploy breaks or slows something down, we can easily see it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LyOgfaJzPunsEJlCtp1nCw.png" /></figure><p>The tool that accomplishes this is a simple DataDog bridge service called <a href="https://github.com/bobbytables/spinnaker-datadog-bridge">spinnaker-dd-bridge</a>. It accepts all webhooks from Spinnaker and can render Go templated events that are sent to DataDog. The tags for application are also included in the event so we can overlay them easily in our graph dashboards per app.</p><p>All in all, Spinnaker is a great tool for deployments, but we found it needed some tooling to become useful for our organization. Manual configuration of pipelines via the UI was a nonstarter because there’s no log of who changed what. By moving pipeline config to repositories, we get this audit trail of change.</p><h3>Closing Up</h3><p>We’re pretty happy with how Spinnaker and Kubernetes work together. We’re actively watching the development of the V2 provider in Spinnaker so we can move all of our deployments to manifest only. This will allow us to also deploy things such as ConfigMaps and ingress rules from Spinnaker as well. The k8s-pipeliner tool will support this in the future as it becomes more stable.</p><blockquote>We had a great time building this infrastructure. Maybe you would too. <a href="https://www.namely.com/careers/">Checkout our careers page</a> to come help us process over $12 billion in payroll a year, provide benefits to top tier companies, and much more.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=561aa472a0e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/spinnaker-and-kubernetes-for-continuous-delivery-561aa472a0e8">Spinnaker and Kubernetes for Continuous Delivery</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Business Data Tools With Jupyter]]></title>
            <link>https://medium.com/namely-labs/creating-business-data-tools-with-jupyter-ee152f5a8f95?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/ee152f5a8f95</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[business-tools]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Brian Tenazas]]></dc:creator>
            <pubDate>Tue, 28 Feb 2017 15:46:31 GMT</pubDate>
            <atom:updated>2017-02-28T15:46:30.409Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*_7-r4KKN7EyR8xcmehDtGA.png" /></figure><p>When it comes to bulk inserting data into your database, there’s nothing worse than getting bad data. Poor formatting, mismatched IDs and other mistakes in the flat file will lead to errors that could cause the import to fail or worse, insert data you never wanted in the first place. As part of the Data Ops team at Namely, this is a challenge I faced every day. One of my first programming challenges when I joined the team was to find a way to ensure that client data about to be inserted was as clean as possible. With a lot of help from my team, we built tools that can transform messy external files into easily importable data in seconds.</p><p>In this article, I will show you how to build a sample of a Python CSV file validation tool we use at Namely. This lightweight application will be able to check each row of a CSV file and see if they adhere to your business logic such as making sure ID numbers are within a certain range or if the length of a string is too long for your database field.</p><p>*Please note that these instructions assume that you have some Python experience.</p><h4>What Is Jupyter Notebook?</h4><p>Jupyter Notebook is a powerful web application that can be described as an HTML wrapper around a terminal or command prompt. It was built with data science in mind, using notebooks as a way for data scientists to share their code, their ideas and to visualize their results. As such, it supports major data science languages such as Python and R. Our Data Ops team operates primarily in Python, which is well-suited for data processing due to a wealth of specialized libraries such as numpy and pandas. Another benefit of Jupyter is it allows us to build simple, yet effective GUIs using widgets. This allows us to deploy some of our notebooks to colleagues without requiring extensive programming knowledge to understand the input.</p><h4>Setting Up Your First Jupyter Project</h4><p>Since our notebooks serve as the interface for users via <a href="https://ipywidgets.readthedocs.io/en/latest/">widgets</a>, we try to keep them separated from the Python files to maintain a clean divide between front-end and back-end for ease of testing. We built the validator tool to be extensible for future use so our example folder will copy our repo’s folder structure, which looks something like this:</p><pre>Project<br>    | <br>    + -- Notebook Folder <br>    |          |<br>    |          + File Validator Notebook<br>    |   <br>    |<br>    + -- File Validator App Folder<br>               |<br>               + __init__.py<br>               + app.py<br>               | <br>               + File Type 1 Folder<br>                      |<br>                      + __init__.py<br>                      + type.py</pre><p>The workflow consists of the user loading a .csv file via the notebook, which is transformed into a pandas dataframe, passed into the <em>app.py</em>, which then transforms and routes the data to the appropriate file type for processing. Finally, the results are returned by the <em>app.py</em> to the notebook/user.</p><p>Now that the file structure is established, we can start building.</p><p>First, let’s install the required Python modules for this example. You can do this globally or in a virtual environment. You will need:</p><ul><li><a href="http://jupyter.org/install.html">Jupyter Notebook</a></li><li><a href="http://pandas.pydata.org/pandas-docs/stable/install.html">Pandas</a> — Python data processing library</li><li><a href="http://pyrsistent.readthedocs.io/en/latest/intro.html#installation">Pyrsistent</a> — Python functional data structure library</li></ul><p>Pandas is the most popular data processing library in Python. It borrows the dataframe concept from R (think Excel but programmatic and fast) and allows you to perform relatively large data operations such as calculating standard deviations, pivoting data or joining multiple dataframes on an index. Because it easily consumes .csv files through the read_csv() function, this was an easy choice for this application.</p><p>We chose Pyrsistent over other Python validation libraries like <a href="http://docs.python-cerberus.org/en/stable/usage.html">Cerberus</a> because it was syntactically simpler to understand, allowed for other tasks like transformations in one function and has excellent error catching.</p><h4>Creating Your First File Validation</h4><p>With your environment setup, we can now write the file validation! Open up your favorite IDE and let’s write the sample validator for numbers less than 10 and the length of a string that cannot exceed 20 characters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca3734e6f488d16d1d8cf8cb61e7c5e7/href">https://medium.com/media/ca3734e6f488d16d1d8cf8cb61e7c5e7/href</a></iframe><p>The field function is one of the best features of Pyrsistent. It’s an almost one line transformer and validator. From the example above, here’s what some of the arguments do:</p><ul><li>mandatory — This variable must be present if <strong>True</strong>. If not present, the PRecord returns an error.</li><li>factory — Attempts to transform the data being assigned to a variable to whatever type or custom transform function you write. For example, if you wanted a factory function to transform a string to capitalize the first letter, you would write</li></ul><pre>def title_factory(string):<br>    return string.title()</pre><ul><li>invariant — This is essentially where you would place the business logic of whatever you are trying to validate. It requires a function (in this case a lambda) that returns a tuple with a boolean if the invariant held or not (is x &gt; 10?) and a message if the invariant failed (‘Number exceeds range’). We’ve used list compares, length ranges and other similar methods for this argument.</li></ul><h4>Creating The Main Backend Application</h4><p>With the file type in hand, the back-end app is relatively simple to write (and you only have to write it once<em>!</em>) Since we know we are consuming a pandas dataframe from the notebook, the app needs to transform this one more time into a dictionary that can be used by our new <em>Validate</em> class. Here’s our app should look like (you can also install <a href="https://pypi.python.org/pypi/tqdm"><em>tqdm</em></a><em> </em>if you want a cool status bar):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e304277fcd26d7ac048bc8ef59c2f9c/href">https://medium.com/media/5e304277fcd26d7ac048bc8ef59c2f9c/href</a></iframe><h4>Building Your Jupyter Notebook GUI</h4><p>The backend is now complete. Now we just need a way for a user to interface with our brand new app.</p><ul><li>In your terminal, navigate to the project folder and start the notebook service by typing</li></ul><pre>$ jupyter notebook</pre><ul><li>Your default browser should pop up with the project folder like this!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hghpKTD5H8pmM3cE3GRHig.png" /></figure><p>*Note that you cannot go beyond the “home folder” where you started the service. If you need to navigate to another folder, simply kill the service (Ctrl + C on Mac), move your terminal to the new base folder and start it again.</p><ul><li>With the notebook service active, click into the Notebook folder and start a new notebook by clicking on New → Python.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*bI1C0MQ5hjopIh63VMLK6g.png" /></figure><ul><li>You will be greeted by with the blank notebook GUI and are now ready to write a simple front-end for our app with some nice widgets. These widgets will allow users to select the files and convert them into dataframe objects that our app can consume. The code looks like this:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/892ccbbd06a63ed7083138ab31c5f080/href">https://medium.com/media/892ccbbd06a63ed7083138ab31c5f080/href</a></iframe><ul><li>Once you’ve got all that code in the block, hit Shift + Enter to execute your code. The output of your notebook’s code block should now look like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*prv_g-MVNPcK68YaRH5tNg.png" /></figure><ul><li>Now you just have to pass a path to a test file. Here’s what the cells would look like:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*GMFmaSDvAwetbKidbECjDQ.png" /></figure><ul><li>Run that file path through and your notebook will look like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*PJRQywHhoIUGGQcAn4ntZg.png" /></figure><ul><li>Now go check the folder where your file is and you should have the output file with your testfilename_validated.csv and it should now have this awesome status and error message column:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*Vq2FbNvX5b3KT19-qj-KIw.png" /></figure><p>That’s it! You’ve successfully created a CSV validator! Of course, this is pretty basic but you can easily modify the type.py to accommodate any headers/data to fit your needs.</p><p>The validator you just created is one of many possible tools you can easily create and distribute via Jupyter. By creating quick tools that can be easily distributed, you can make data tasks more efficient for your organization with reduced development time until either a proper application is developed for long-term needs or until the tool has fulfilled its function and is no longer required.</p><h4>Many thanks to everyone who helped me with this post.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee152f5a8f95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/creating-business-data-tools-with-jupyter-ee152f5a8f95">Creating Business Data Tools With Jupyter</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we build gRPC Services at Namely]]></title>
            <link>https://medium.com/namely-labs/how-we-build-grpc-services-at-namely-52a3ae9e7c35?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/52a3ae9e7c35</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Bobby Tables]]></dc:creator>
            <pubDate>Thu, 23 Feb 2017 21:54:54 GMT</pubDate>
            <atom:updated>2017-02-23T21:54:53.978Z</atom:updated>
            <content:encoded><![CDATA[<h3>How We Build gRPC Services At Namely</h3><p>Over the past year, we’ve been transitioning two applications into gRPC services: a monolithic Ruby on Rails application and a .NET application. Along the way we’ve experimented with best practices, built tools to help with the process, and learned a lot about what works and what doesn’t. In this post, we’re going to talk about:</p><ul><li>Managing protocol buffer definitions</li><li>Generating the appropriate language files from the definitions</li><li>Deploying these services</li></ul><p>What we’re NOT going to talk about (so you can turn back now if you want):</p><ul><li>Setting up secure connections for our services</li><li>Request tracing</li><li>How we implement the services themselves</li></ul><h3>Why gRPC?</h3><p>In a word: structured. Although building a JSON API is simple to do, you have a higher risk of breaking service consumers because of its unstructured nature. You can achieve a variance of structure using a format such as Swagger or a <a href="https://martinfowler.com/articles/consumerDrivenContracts.html">Consumer Driven Contract</a>. But it’s not a guarantee.</p><p>gRPC and protocol buffers promise a certain structure that will be sent without exception. gRPC also defines the protocol in which the data is transferred and the libraries for it. JSON is usually sent over HTTP, but gRPC doesn’t leave this up for debate. gRPC also provides the framework for error codes, authentication, and streaming messages.</p><p>Overall, we’re in the business to deliver value to our customers, not discuss which JSON standard to use. JSON-API? HAL? Hypermedia at all (<em>shivers</em>)? <strong>We just want to ship</strong>. gRPC lets us do that easily.</p><h3>How We Manage Protocol Buffer Files</h3><p>Our first approach attempted to keep protocol buffer files (.proto) within the same project they were implemented. However, we found this to be cumbersome because whenever a new service consumer was being built, the developer would have to copy the generated files out of of the service itself into their new project.</p><blockquote>Our philosophy shifted to keeping the definitions and the implementations separate because it stays true to the idea of protocol buffers: describe what your service will do and abide your service to that contract.</blockquote><p>After realizing that we can kill two birds with one stone, we moved our protocol buffers into a separate repository on Github called the protorepo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*FvCYDhoU1saZRmmbnUG_ww.png" /></figure><p>This new approach allowed us to gain many advantages. First off, our new single repository for protocol buffers acts as an inventory of all Namely’s services using gRPC. This grants us the pleasantry of knowing what those are at a quick glance.</p><blockquote>Now that the definitions and implementations were separated, team collaboration was more easily focused on how services communicate with each other rather than getting distracted by implementation details.</blockquote><p>Developers were now only notified for definition changes, rather than spam alerts from pull requests such as “<em>move spacing 2 pixels to the right</em>”. The notifications from a chosen repository became more meaningful. This spurred more conversations with members from multiple teams chiming in on service proposals, proto changes, etc.</p><p>By having a single repository for all service definitions, it allowed us to localize all of our build tools into one place.</p><h3>How we generate our language specific files</h3><p>One of the great things about gRPC and protocol buffers is the amount of languages it supports. As of writing, it currently supports 10 languages(!). Although, a problem with this is installing the tool to generate the specific languages files from the proto definitions can be a pain, and frankly, why do something a computer can do it for you automatically?</p><p>Our folder structure in the protorepo is standardized to enable automatic generation of language specific files. For example, if you have a service called &quot;companies&quot;, you&#39;d have a folder called the same thing:</p><pre>$ pwd<br># /Sites/namely/protorepo<br>$ tree<br>├── | companies<br>│   ├── .protolangs<br>│   └── companies.proto</pre><p>Let&#39;s focus on a single file here: .protolangs. This file informs our build tools which languages need to be built for this service. For our companies service, it looks like this:</p><pre>$ cat .protolangs<br>go<br>csharp</pre><p>Each language is separated with a new line. When changes are pushed to the repository, our tool runs through each folder, reads the .protolangs file and runs the appropriate generator for all of the .proto files in the directory.</p><p>You might be thinking, “what awesome tool did you make for this?”. It’s not fancy by any means. In fact, it’s just long bash script. If you dare, <a href="https://gist.github.com/bobbytables/2fab9ac9509867b5acfe2bb5693aee71">here is a link to it</a>.</p><p>After you’ve added all of the languages you care about to the.protolangs file, you&#39;ll need to create a new repository in the format protorepo-{service-name}-{language}. So if you have a project called “heimdall” that uses the Go language, your new repository would be “github.com/namely/protorepo-heimdall-go”. These repositories are where we push all of the newly generated protocol buffer files once any commit is made to the protorepo repository.</p><p><strong>Here’s a rundown of our generation process:</strong></p><ul><li>Change into a service specific directory</li><li>Read every language the .protolangs file declares</li><li>Run a docker image relevant for each language <br>(We support most languages — if you’re interested, check out our docker-protoc project: <a href="https://github.com/namely/docker-protoc">https://github.com/namely/docker-protoc</a>)</li><li>Take the newly outputted files and move them into the language specific repository</li><li>Do a simple git diff to see if anything changed</li><li>If the project <em>did</em> change, commit the changes, and push</li><li>Rinse and repeat for every folder</li></ul><p>This process has made it incredibly easy to write a service definition using the interface definition language provided by protocol buffers and generate the files you want quickly. But where do we go from here?</p><h3>How We Deploy Our gRPC Services</h3><p>We’re huge fans of containers at Namely, specifically the Docker toolchain. We have now adopted a system that uses Kubernetes on AWS for a majority of our deployment and orchestration.</p><p>When a team is building a service, it’s required that the build process includes building a Docker image and pushing it to our private Docker repository (we use <a href="https://quay.io/">quay.io</a> internally). Whenever a commit is pushed to a project, we take the first 8 characters of the commit’s SHA and use that as the Docker image’s variant.</p><p>For example, if you push a new commit that the first 8 characters are bc79d9cb and your project is called companies, then your Docker tag will look like:</p><pre>registry.ourinternaldns.com/namely/companies:bc79d9cb</pre><p>It is natural to be thinking “Yikes, but isn’t that using a ton of space?”. Short answer: yes.</p><p>The reason we do this is because we allow developers to deploy <em>any</em> version of their code to a development Kubernetes cluster. This means any branch, feature, etc. can be tested by our QA team and run a blackbox test suite against it. Everyone gets a clean environment on demand (out of scope for this post, but it’s damn cool).</p><p>However, the process is a little different once we actually deploy to our customers. Instead, we use release tags by using the releases feature on Github.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Q2ouIcGl42jLZfPV3nlfrA.png" /></figure><p>When a tag is created on a repository, our CI pipeline builds a Docker image using the new release tag as the variant. By using the companies service above as an example, the new Docker tag would be:</p><pre>registry.ourinternaldns.com/namely/companies:v1</pre><p>Once we have our newly built image, we modify another monorepo full of Kubernetes manifests for each project (similar to our protorepo):</p><pre>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  name: companies-prod<br>  namespace: production<br>  labels:<br>    name: companies-prod<br>    component: companies<br>spec:<br>  replicas: 3<br>  template:<br>    metadata:<br>      labels:<br>        name: companies-prod<br>        component: companies<br>    spec:<br>      containers:<br>      - name: companies<br>        image: &quot;registry.ourinternaldns.com/namely/companies:v1&quot;<br>        imagePullPolicy: IfNotPresent<br>        ports:<br>        - containerPort: 50051</pre><p>This process is still manual, e.g. someone modifies these yaml files by hand. But! We <em>are</em> considering ways to improve it. For example, we’re using <a href="https://github.com/kubernetes/helm">Helm</a> for our development cluster (the one developers and QA use) but we haven’t yet switched to using it for production usage.</p><p><strong>There are many ways to organize services, and while ours is far from perfect, it has mitigated the pain and confusion around building services with protocol buffers and gRPC. There’s no longer a mystery of where definitions are stored and how the language specific files are generated. It is handled automatically for you!</strong></p><p><strong>I hope this post was interesting and would love to chat and answer any questions you might have about it. Cheers!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52a3ae9e7c35" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/how-we-build-grpc-services-at-namely-52a3ae9e7c35">How we build gRPC Services at Namely</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[API Documentation 2.0 with Swagger and Stoplight at Namely]]></title>
            <link>https://medium.com/namely-labs/api-documentation-2-0-with-swagger-and-stoplight-at-namely-38217ee0f362?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/38217ee0f362</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[swagger]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[documentation]]></category>
            <dc:creator><![CDATA[Katie Reed]]></dc:creator>
            <pubDate>Mon, 07 Nov 2016 19:33:32 GMT</pubDate>
            <atom:updated>2017-07-26T20:19:04.557Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UivtxWoxKPo1ns1KowpyFA.png" /></figure><h4>A quick Google search on how to best document your API will return a myriad of tools ranging from framework specification formats to software that will generate documentation with a slick and aesthetic UI. This article will break down our experience and experimentation with these tools, and what ultimately led to our optimal final product: <a href="http://developers.namely.com/">API Documentation 2.0</a></h4><h3>Part I: The Philosophy</h3><p>Your API is a living and breathing document — yet the API documentation itself is a contractual agreement between two consenting parties. It is a company’s moral obligation to uphold their end of the agreement by providing clear and accurate documentation.</p><p>Our team at Namely decided to adopt a philosophy moving forward where <strong>the API design drives the code </strong>. This can be broken down into two commandments:</p><ol><li><em>All new APIs must be designed first.</em></li><li><em>The API design will change, and the framework must make it possible to adapt quickly without letting the code, design, and documentation fall out of sync.</em></li></ol><p>In order to make good on this philosophy, we decided that we needed a specification framework that would act as the source of truth for our API that automatically generates documentation.</p><h3>Part II: The Specification Framework</h3><p>The leading players in the API specification framework field are: <a href="http://www.swagger.io/">Swagger</a>, <a href="http://raml.org/">RAML</a>, and <a href="https://apiblueprint.org/">API Blueprint</a>. Swagger stood out to us from the beginning because it is the most heavily adopted, with a large community of users and supporters, and has the greatest support for multiple languages.</p><p>Swagger allowed us to establish a set of rules for describing our REST API with a document that details all the URI paths for an API, query parameters, and the request and response bodies. The format is both machine and human readable. As a result, it can be used to share documentation among product managers, testers and developers, but can also be used by various tools to automate API-related processes.</p><p>An extra benefit of having our entire API documented in Swagger format in GitHub is that it incentivizes transparency and collaboration for proposed amendments to the API. This leads to improved architectural design as well as company-wide notifications when a piece of core functionality changes.</p><p>An example from the YAML Swagger file from the Namely API:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a5606b203d65bbcd89af0c30b90d12d/href">https://medium.com/media/7a5606b203d65bbcd89af0c30b90d12d/href</a></iframe><h3>Part III: The API Documentation Generators</h3><p>The next step was to figure out how to host the documentation and which developer portals play nicely with Swagger. The API documentation generators we sampled were: <a href="http://www.readme.io/">ReadMe</a>, <a href="http://www.gelato.io/">Gelato</a>, and <a href="http://www.stoplight.io/">Stoplight</a>.</p><p><strong>ReadMe</strong><br>Our API documentation was originally hosted on ReadMe and input manually from a template and text fields before we adopted Swagger. As you can imagine, this process was time intensive and caused our API documentation to regularly go out of date. ReadMe is currently in beta mode for uploading Swagger files and automating this process, and unfortunately, wasn’t at the level of reliability and functionality we felt comfortable moving forward with. Additionally, ReadMe did not offer support for a sandbox to make test API calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UysRL2TpmFdnLK6cgDZMzg.png" /><figcaption>The original API documentation hosted with Readme.io</figcaption></figure><p><strong>Gelato</strong><br>Gelato smoothly consumed Swagger files (even with error catching!) and a host of other improved features such as a markdown editor and interactive API explorer where you could utilize an interactive sandbox to make test API calls.</p><p>However, there were two major holdups with Gelato:</p><ol><li><strong>The Lack of Support for Variables</strong><br>Namely uses a subdomain for each client in the following format:</li></ol><pre>&lt;&lt;subdomain&gt;&gt;.namely.com</pre><p>The subdomain is meant to be replaced with the company’s name for their specific data, such as <em>Hooli</em> or <em>Pied Piper</em>.<em> </em>Without the functionality for variables, this would make testing the API endpoint in the sandbox ineffective.</p><p><strong>2. The Counterintuitive UI and Heavy White Space</strong><br>The documentation auto-populated the first item with the name of the path and its corresponding model definition to create the <strong>Endpoint Overview</strong>. This falsely led the user to believe that the <strong>/countries </strong>index would return only the Countries definition model, when it would actually return an object with 3 model definitions such as: <strong>countries</strong>,<strong> links</strong>, and<strong> meta.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*trLILPesyJ3p21eYMMeskA.png" /><figcaption>Namely’s API documentation hosted by Gelato</figcaption></figure><p><strong>Stoplight</strong><br>The classic idiom of “third time’s the charm” rang true once we tried out Stoplight. Stoplight not only offered all of the previously mentioned features, but also supported variables. The UI was the best we had seen among all of the contenders thus far when it came to the user experience, design and customization. Additionally, Stoplight processed the Swagger file instantly while Gelato took about 4–5 minutes for processing which slowed down the development process.</p><p>Stoplight’s newest product is called “Scenarios” and allows you to test, automate, and debug web APIs + AWS Lambda. We were able to utilize this feature to our advantage by automating the import process between our Swagger file on Github and Spotlight — <strong>so</strong> <strong>the code and documentation are never out of sync since the documentation is always regenerated. </strong>🎉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ph4-EBXgvw7_AmVsG0KZuw.png" /><figcaption>The new and improved Namely API Documentation hosted by Stoplight</figcaption></figure><h3>Conclusion</h3><p>Swagger and Stoplight definitely stood out from the pack and proved to be a winning duo for creating beautiful API documentation with ease. Check out our new API documentation at <a href="http://developers.namely.com/">developers.namely.com</a> and let us know what you think!</p><p><em>Katie Reed is a software developer at Namely. Namely is an all-in-one platform for HR, Benefits, and Payroll. We’re hiring for Engineering, Product, and Design roles — check out all of our jobs </em><a href="https://www.namely.com/careers/"><em>here</em></a><em>.</em></p><p><strong>Thanks to </strong><a href="https://twitter.com/chrisnlott"><strong>Chris Lott</strong></a><strong> at Spotlight for his responsiveness to my queries.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38217ee0f362" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/api-documentation-2-0-with-swagger-and-stoplight-at-namely-38217ee0f362">API Documentation 2.0 with Swagger and Stoplight at Namely</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A completely neutral post about containers.]]></title>
            <link>https://medium.com/namely-labs/a-completely-neutral-post-about-containers-c0f9c8cd3baf?source=rss----8354434f0ab4---4</link>
            <guid isPermaLink="false">https://medium.com/p/c0f9c8cd3baf</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Carlos Nunez]]></dc:creator>
            <pubDate>Thu, 10 Dec 2015 15:31:06 GMT</pubDate>
            <atom:updated>2015-12-10T15:31:05.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*VRKNzd7NNv4I2Bst.jpg" /></figure><p><strong><em>Edit</em></strong><em>: I mentioned below that Docker containers can only run a single process. While that is true by default, it is actually not the rule. You can use Docker Supervisor to manage multiple processes within a container. </em><a href="https://docs.docker.com/engine/articles/using_supervisord/"><em>Here</em></a><em> is how you do that.</em></p><p><em>Thanks, </em><a href="https://reddit.com/r/sysadmin/comments/3uxwwn/i_wrote_a_intro_post_for_containers_and_docker/cxiwddc"><em>/u/carlivar!</em></a></p><p>Fact: there are at least 22 conversations arguing about the usefulness/uselessness of containers happening right now. For every person arguing about them, at least three blog articles will shortly appear telling you why containers are going to be bigger than the Super Bowl or how they are the worst thing to happen to computing since <a href="https://www.youtube.com/watch?v=b2F-DItXtZs">web scale</a> became a thing.</p><p>This short post will not be any of those.</p><p>I believe in using the right tool for the job and I hate fads. I’ve also been learning about containers for the last few months and am becoming incredibly interested in figuring out how to bring them to Windows 2012 and below (Windows Server 2016+ will have them, but Microsoft will be lucky and <em>probably</em> scared if they saw anything above 20–30% uptake within its first two years), so here’s a short post describing what I’ve learned about them, how they work, and why they might or might not be useful for you and/or your business.</p><h3>Here’s A Scenario</h3><p>You’re a developer for an eCommerce website and are responsible for maintaining the order management system (let’s call it an “OMS”). The user-facing shopping cart and checkout processes are tied into the OMS, and these are updated more frequently than the service that handles order transactions. You’d like to be able to quickly test new shopping cart features to respond to market competition more quickly.</p><p>One way of doing this is to build virtual machines that mimic your production environment. Depending on the size of your technology team, either you or your system administrators (sysadmins) will set this up for you (if it isn’t already so). Once the machines are ready, either you or your fellow sysadmins will update their underlying operating systems, install and configure the sites dependencies, clone the branch in which you do your development (or somehow get the files across to your sparkling-new staging environment), and, finally, run a few incantations to get everything moving.</p><h3>Reasons This Could Suck</h3><p>This is pretty straightforward for some (many?) teams, but much easier said than done for larger ones for a few reasons:</p><ul><li><strong>Red tape</strong>: The bigger the organization is, the more likely there exists lots of complexity (and siloing to match) that makes things like creating new VMs a multi-hour/multi-day effort.</li><li><strong>Patching is…patching</strong>: Patching operating systems can take a long time and introduce unforeseen regressions that might not be part of your spot test(s).</li><li><strong>Golden images get complicated</strong>: Maintaining images gets increasingly complicated fast, especially as your site gains more moving parts.</li><li><strong>Money</strong>: Most “enterprise-” grade virtualization or cloud platforms cost money, and more instances = more compute, networking and storage = more 💰💰.</li><li><strong>Overhead</strong>: Instead of worrying about whether your new shopping cart feature works, you have to worry about things like whether <em>/etc/networks</em> is set up right or whether your host has the right Windows license applied to it.</li></ul><p>What was originally an “easy” feature test has now become this monolith of an environment fraught with overhead that you’d rather not deal with. This is one of the many reasons why testing isn’t given the due diligence that it often needs.</p><h3>Meet Containers</h3><p>A <em>container</em>, also commonly known as a <em>jail</em>, is an isolated copy of an operating system’s user space within which processes can be executed independently from its parent. What this <em>actually</em> means is that you can fork off a “copy” of Linux or Windows…within Linux or Windows. These copies use all of the same hardware as the host they live on, but as far as your network or storage is concerned, they are all independent systems. As a result, containers don’t need their own bank of memory or disk space like a virtual machine would.</p><p>With containers, you can do things like:</p><ul><li>Run a web server, a database and a message queue…without virtual machines,</li><li>Create restricted login environments for remote users that, to them, looks and feels like the actual system, and</li><li>Spin up thousands of “instances”…on a regular MacBook.</li></ul><p>If you’re thinking “wow; that’s amazing,” you are not wrong.</p><h3>Containers are not new.</h3><p>Despite the recent press (and VC-backed funding) that Docker (and, in turn, containers) have been getting, containers are far from a new concept.</p><p>The earliest, and most common, form of containers were chroots and/or BSD jails. These first appeared in <a href="https://en.wikipedia.org/wiki/Chroot">Version 7 UNIX and 4.2BSD</a> in 1979 and 1982 by way of the chroot syscall and binary and has been used on a daily basis since to restrict the scope of access that users have on UNIX-based systems when they log in. When a user logs into a chroot jail, they will appear to have root access to the entire system (and might even be able to write to anything within the root partition) but, in actuality, will only have access to a volume or directory within the actual system it lives under. This gives the user the ability to do whatever they want on a system without actually destroying anything important within it.</p><p>Additionally, hosting providers have been providing virtual private servers since well before AWS was a thing. These VPSes were either virtual machines or chroots depending on the type of system requested. Google has also been using (and are still using!) containers internally for executing, maintaining and easily failing over ad-hoc jobs and services since ~2006. They recently made parts of their implementation (called “Borg”) public; check out the <a href="http://research.google.com/pubs/pub43438.html">research paper</a> if you’re interested. (Googlers take advantage of Borg all of the time for nearly anything and all of their production services run on it. It’s unbelievably reliable and ridiculously high-scale.)</p><h3>Okay, so why Docker?</h3><p>So if containers aren’t a new concept, then why is Docker getting all of the attention and being hailed the Second Coming of computing?</p><p>What Docker excels in that <em>lxc</em> sort-of doesn’t is ease-of-use. Docker’s guiding philosophy is that its containers should be used to do a single thing and a single thing only and that containers should be able to run on any host, anywhere. This allows Docker containers to be extremely lightweight and its underlying architecture comparatively simple. This manifests itself in several ways:</p><ul><li>Every container is composed of a <em>Dockerfile</em> that describes it and a read-only <em>image</em> that defines its base. The image is kind-of like a classic operating system golden image with the big difference being that it lacks an operating system.</li><li>Images are stored in an image registry called the <a href="https://hub.docker.com/">Docker Hub</a> where they can be downloaded from anywhere and versioned to make rollbacks easy. Committing to and pulling from Docker Hub is built into the docker binary.</li><li>Docker also includes a <a href="https://docs.docker.com/engine/reference/api/docker_remote_api/">RESTful API</a> that makes it very easy to build tooling around.</li><li>Docker is FREE!</li></ul><p><strong>TL;DR</strong>: Docker makes it very easy and cheap to build up a containerized environment in very little time.</p><h3>Why not Docker (or containers)?</h3><p>While Docker is an amazing tool for getting containers up and running quickly, it doesn’t come without its limitations:</p><ul><li>Docker containers can only run one process at a time. This means that services that are typically co-located on a traditional host will have to be on separate containers with Docker and the onus for getting them to communicate with each other falls on you.</li></ul><p>For running a container with multiple processes at once, a Linux Container (or LXC container) or a VM is probably more appropriate.</p><ul><li>Since Docker virtualizes the operating system within the operating system (<a href="http://cdn.meme.am/instances/500x/60864927.jpg">yo dawg</a>), it is theoretically easier for a process running within a container <a href="http://www.projectatomic.io/blog/2014/08/is-it-safe-a-look-at-docker-and-security-from-linuxcon/">to escape out into its parent operating system</a>. A (modern) virtual machine is much less susceptible to an attack like this since the hypervisor under which it operates within a different protection ring inside of the processor, which protects rogue processes within machines from being able to access real hardware on the host.</li></ul><p>That said, most security researchers consider Docker to be pretty safe at the moment, so the likelihood of this happening is low enough to mitigate concerns.</p><ul><li>For the same reason as above, you can’t run different operating systems within containers. This means that Windows containers on a Linux host is out of the question…for now. Additionally, Docker uses Linux-specific features in its core, meaning Windows and OS X hosts need to use a virtual machine and a lightweight service called Boot2Docker to take advantage of it. Windows Server 2016 will bring support for containers to the forefront, but that’s still a few months out.</li></ul><p>It should be noted that <strong>containers are NOT replacements for virtual machines</strong>. Virtual machines are better solutions for heavy, monolithic apps or systems that are difficult to break apart; or anything that requires an entire operating system stack. However, people have been finding containers to be pretty good replacements for small apps or services within apps (or enough motivation to break those big, heavy apps down into pieces using a <a href="https://www.opengroup.org/soa/source-book/soa/soa.htm">service-oriented architecture</a>).</p><h3>Our Scenario…With Containers</h3><p>With that, let’s go back to our expedient web dev that’s eager to test that new shopping cart feature. Taking the VM approach described above might be difficult and time-consuming. If she were to use containers, the process would be a little different:</p><ul><li>Install Docker (and Boot2Docker if needed) onto their system.</li><li>Create Dockerfiles describing the containers needed to re-build a local instance of production or <a href="https://docs.docker.com/engine/articles/baseimages/">create images</a> if needed</li><li>Define your environment using <a href="https://docs.docker.com/compose/">Docker Compose</a></li><li>docker-compose up</li></ul><p>That’s mostly it! All of this can be done on localhost, and all of it is (relatively) easy.</p><h3>About Me</h3><p><em>Carlos Nunez is a site reliability engineer for </em><a href="http://www.namely.com/"><em>Namely</em></a><em>, a human capital management and payroll solution made for humans. He loves bikes, brews and all things Windows DevOps and occasionally helps companies plan and execute their technology strategies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0f9c8cd3baf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namely-labs/a-completely-neutral-post-about-containers-c0f9c8cd3baf">A completely neutral post about containers.</a> was originally published in <a href="https://medium.com/namely-labs">Namely Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>