<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Orion Delwaterman on Medium]]></title>
        <description><![CDATA[Stories by Orion Delwaterman on Medium]]></description>
        <link>https://medium.com/@o_the_del?source=rss-10e5d25d538c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bMse8XlCy2HBG4eL.jpeg</url>
            <title>Stories by Orion Delwaterman on Medium</title>
            <link>https://medium.com/@o_the_del?source=rss-10e5d25d538c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:22:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@o_the_del/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Engineering Learnings: Sagemaker Notebook Servers]]></title>
            <link>https://medium.com/in-the-weeds/data-engineering-learnings-sagemaker-notebook-servers-23a0a7227295?source=rss-10e5d25d538c------2</link>
            <guid isPermaLink="false">https://medium.com/p/23a0a7227295</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[amazon-sagemaker]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Orion Delwaterman]]></dc:creator>
            <pubDate>Mon, 13 Jan 2020 15:34:48 GMT</pubDate>
            <atom:updated>2020-01-13T15:34:48.316Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Engineering Learnings: Retaining Files in Sagemaker Notebook Servers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3KnqMxjXtkl9eeCnudcaEA.jpeg" /></figure><p>Last year at Greenhouse, we doubled the size of our Data Science team. With a larger team, we’re looking to tackle larger problems, provide more insights, and ship more machine learning related products in the near future. And as part of that expansion, the Tools and Operations team have taken a large focus on Data Engineering work. We have been building out more tools and optimizing workflows to maximize the generation of that delicious data work. This is a quick little story about a small detour we took when working with AWS Sagemaker Notebooks.</p><p>A large part of Data Science development is running various experiments with our data. These can be building simple charts, doing complicated Bayesian statistics, or even prototyping ML models for making various predictions. The preferred tool of choice these days are <a href="https://jupyter.org/">Jupyter Notebooks</a>, which provide a REPL and some nice graphical tools that can make this experimentation process faster for our Data Scientists. I won’t go into much detail about these, as this story isn’t so much about the notebooks.</p><p>As good stewards of data, we have to make sure that these experiments are done in a responsible manner. We’re dealing with real data, so we need to build a secure environment for these Notebook servers to run in; one with the correct security and auditing tools in place to keep the data safe. Being an AWS shop, we choose to work with Sagemaker Notebook servers, which got us 85% of the way there. Beyond the branding, a Notebook server is just an EC2 instance with a Jupyter Notebook server running.</p><p>We customized our Notebooks servers with a Github integration, some standard aliases, and various checks and protections to make sure no data leaks out into the world. We built all of this customization using Lifecycle Hooks, tested it all, and turned over to the Data Science Team. And that’s where they encountered the problem.</p><p>Because our Data Scientists don’t work 24 hours a day, there are many times when the notebook servers are doing nothing. These aren’t web servers, which need to be on for any customer at any time of the night. They’re dev machines, and only need to be on when crunching the numbers. To save the planet (and money) our team turns off the notebook servers when they are not needed. On Friday afternoons, we send out a slack message to the team, who then shut down their notebooks, and will boot them back on Monday morning.</p><p>But that first Monday back, something funny happened. The Data Science team started getting errors from git when they tried to commit changes they were making to their notebooks. Git would complain that our pre-commit binary, which prevents data from being checked in, was missing. I immediately opened a terminal on the instance, ran the which command, and sure enough, the binary that we installed with our hooks was just gone. I was left scratching my head.</p><p>We began by examining our lifecycle hooks, which we used to set up the server. AWS Sagemaker Notebook Servers support two types of Lifecycle hooks: <strong>Start Notebook</strong> and <strong>Create Notebook</strong>. The <strong>Create Notebook</strong> hook runs only a single time when the notebook server is created, and the <strong>Start Notebook</strong> runs every time the notebook server is booted. We set our scripts to work on the <strong>Create Notebook</strong> hook since we only needed to install the binaries once the machine was built. After all, turning off the server shouldn’t erase the file, should it? In fact, I had tested this. I had created a sample notebook on a Notebook server, turned it off and on again, and found the notebook still there. None of this made sense.</p><p>But then it occurred to me that maybe turning off was a Notebook Server wasn’t analogous to just powering down a machine. I opened a terminal on the Notebook Server and did some investigation. On Notebook Servers, all of the notebook files are stored in the /home/ec2-user/Sagemaker directory. I decided to try an experiment. I put three empty files into different directories: /, /home/ec2-user and /home/ec2-user/Sagemaker , and then rebooted the machine to see if any of those files would disappear. Sure enough, the files in / and /home/ec2-user disappeared while the file in /home/ec2-user/Sagemaker remained.</p><p>It turns out that the Sagemaker service doesn’t preserve the entire machine image when you turn off the Notebook; it only saves the notebook files which are all under /home/ec2-user/Sagemaker . This actually makes a lot of sense. If AWS wants to make a change to their machine image for Sagemaker, they don’t have to modify their user’s machine images. They just preserve the one directory (the one with the notebooks) and then they just build the whole machine fresh each time. If you want to apply an update, just reboot all the user’s machines.</p><p>The solution was simple - move all of our hooks to the <strong>Start Notebook</strong> Lifecycle hook. Now we treat every boot of a notebook server like a fresh install. It just goes to show you that no abstraction is perfect. We made the assumption that the AWS machine was like any server on their platform because that fit the mental model. But sometimes you have to dig down in the details to figure out what’s going on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23a0a7227295" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/data-engineering-learnings-sagemaker-notebook-servers-23a0a7227295">Data Engineering Learnings: Sagemaker Notebook Servers</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Remedial Data Science Engineering]]></title>
            <link>https://medium.com/in-the-weeds/remedial-data-science-engineering-424a15f8ea0c?source=rss-10e5d25d538c------2</link>
            <guid isPermaLink="false">https://medium.com/p/424a15f8ea0c</guid>
            <category><![CDATA[greenhouse]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[recruiting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Orion Delwaterman]]></dc:creator>
            <pubDate>Mon, 14 May 2018 18:06:38 GMT</pubDate>
            <atom:updated>2018-05-14T18:06:38.695Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kf2naTGgGtlwUmL7MhEg3Q.png" /><figcaption>Data scientists, hard at work!</figcaption></figure><p>At Greenhouse, we have just launched our first customer facing Machine Learning feature: <a href="https://www.greenhouse.io/blog/introducing-greenhouse-predicts-machine-learning">Greenhouse Predicts</a>. Greenhouse has always supplied meaningful data to our customers with various reports and charting, so we are quite comfortable working with data. But machine learning is new to us and there turned out to be quite a few things we would need to learn along the way.</p><p>For GH Predicts, I was responsible for the tools and systems for deploying our machine learning predictive model. I thought that this would be a simple API server that would just require me learning a few libraries to interact with our model. As it turned out there were a lot of things that our team would have to contend with. This post in the first of a series of posts I am writing about those hard lessons. We are calling the series “Remedial Data Science Engineering”, to give you a sense of what we were in for.</p><h3>Part 1 — Real Memory Constraints</h3><h4>Setup</h4><p>Typically, ML-enabled predictive services like these require two steps: training and predicting. First we take a lot of data, run it through some algorithms to train a model. Second we put that model into some api server, feed it some data from customers and it spits out predictions.</p><p>The training step is an “offline” process. We do this on some periodic basis, and store the resulting model for later use. We are not overly concerned with performance since the job is run infrequently and can take an arbitrary amount of time. We settled on doing a monthly training, and storing the resulting model as a <a href="https://docs.python.org/3/library/pickle.html">pickle file</a>, a python file format for serializing generic objects. During initial development phase, our Data Scientist was training a small model on his laptop using a Jupyter notebook and a small subset of our total data to generate the model.</p><p>The second step, predicting, is the “online” part. We decided to build a small JSON API service, Prediction API. Prediction API would boot up, pull the model pickle file from S3, and then take simple API requests to make predictions. We settled to build simple Flask application, that connected to S3.</p><p>Most of my software development experience is with web applications, so when I think about application constraints, I am usually focused on the speed of requests, speed of database queries, and limiting the amount data needed to fulfill requests. Prediction API server didn’t connect to a database, so I focused on keeping the requests times small. Turns out the speed of predictions would be the least of my problems.</p><h4>Not so smooth sailing</h4><p>The problems we encountered began when we first started using production size data. When I was first building the server, the pickle file for the model was sizable but manageable at 87Mb. When we started to use all of our production data to train the model, the pickle file jumped to 19Gb. Let that sink in for a second. Our Prediction API server needed to hold in memory a 19Gb object. The laptop I develop on only has 16 Gb. This is a memory hog!</p><p>Our Infrastructure team uses Kubernetes along with a lot of custom tooling all running on top AWS 32Gb Memory servers to deploy all of our applications. Our model was 19Gb so in theory we should be able to boot one server with this new model. The kubernetes pod would take almost all resources of a server but it was possible, right? I configured the server to use all the memory possible pointed to the large model file on S3, booted it, and got…OOMed (Out of Memory). I took a big sigh and dug in.</p><p>The problem turned out to be with how we loaded the model. We were using <a href="https://boto3.readthedocs.io/en/latest/">boto3</a> to pull down the pickle file from S3, and then pickle to load the object (this is a simplified version of our code):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2409ded503caf49083855c04fe4d51c5/href">https://medium.com/media/2409ded503caf49083855c04fe4d51c5/href</a></iframe><p>So following the logic of the code, we create an S3 object, read down the entire contents of file as binary data, store it in raw_data , and then use pickle.loads to build the object. That means that for a moment of time we were essentially storing the model twice: once as raw binary data and second as the actual object that we were building. That’s 38Gb of memory, of course we were going to OOM the box.</p><p>Given the size of this data, we could no longer naively download data willy-nilly. Fortunately the pickle library has the load method, which takes an IOStream like thing to build an object (it relies on duck-typing rather than checking the class of the object). Without diving too deep into the details, this allows pickle to read a chunk of bytes, build some part of the object, and discard the bytes (allow them to be garbage collected), then repeat the process until all the parts of the object is loaded. We could stream the data from S3 and not have to store the whole thing twice over.</p><p>Unfortunately, pickle.load requires an object which implements read and readline . The object returned by s3_object.get()[&#39;Body&#39;] , a <a href="https://github.com/boto/botocore/blob/ef1d115e2210bb9d9e9ece9ab982a4622c44a1ea/botocore/response.py">StreamingBody</a> from the botocore library, does not implement those methods (Maybe one day Amazon will fix this…maybe). So this was going to require a little more hacking.</p><p>I dug through botocore’s repo and discovered StreamingBody uses a <a href="https://github.com/boto/botocore/blob/ef1d115e2210bb9d9e9ece9ab982a4622c44a1ea/botocore/response.py#L41-L44">raw socket to pull down its data</a>. Python has a pretty good IO library, including io.BufferedReader which is ideal for reading such a large file. So with a little hacking we could transform botocore’s StreamingBody to io.BufferedReader :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cac1b0c3c225a26a546ffebc13caf80/href">https://medium.com/media/0cac1b0c3c225a26a546ffebc13caf80/href</a></iframe><p>With this code in place, and confident in my success. I booted the server with the new code and got… OverflowError: signed integer is greater than maximum. Wait…what?</p><h4>You had one job</h4><p>What in the world was I doing that would cause an overflow error. Looking through the stack trace I noticed this C level call for reading bytes:</p><pre>File &quot;/Users/orion.delwaterman/.pyenv/versions/3.6.1/Python.framework/Versions/3.6/lib/python3.6/ssl.py&quot;, line 625, in read<br>    v = self._sslobj.read(len, buffer)</pre><p>I then took a quick peek at the Python source code to find the function written in C to see this:</p><pre>//<a href="https://github.com/python/cpython/blob/master/Modules/_ssl.c#L2361-L2364">https://github.com/python/cpython/blob/master/Modules/_ssl.c#L2361-L2364</a><br>static PyObject *_ssl__SSLSocket_read_impl(PySSLSocket *self, int len, int group_right_1, Py_buffer *buffer)</pre><p>This is the C level method for reading bytes from SSL stream, and the second argument len is a signed integer. That means that pickle needed more than 2³¹-1 bytes to load a single primitive object (this turned out to be a particularly large numpy array within our probabilistic model with 5,342,077,759 bytes; yeah that’s more than 5 billion bytes). The way SSL was written it never anticipated a request that large.</p><p>I didn’t want to patch Python just to get around this issue with SSL, so I decided to write an intermediate buffer. This buffer would implement the same read method but would check the size requested. If it was greater than a signed integer, it would split the request into multiple chunks, download each chunk individually, and then join them back together at the end.</p><p>Yes…I wrote a buffer for a buffer:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98922f4d71a42e777482f9ff09820e04/href">https://medium.com/media/98922f4d71a42e777482f9ff09820e04/href</a></iframe><p>This got the server booting, and we were able to start serving predictions using production models.</p><h4>Now you know, and knowing is…</h4><p>There would be more lessons to come while working on this project. We eventually found a way to shrink the model to a reasonable size, but that’s for another blog post. From this episode, I learned that you need production size data as soon as possible. Most of the libraries and tools web developers tend to work with are optimized for “normal” sized files. When working with large datasets, you are not always going to be able to rely on these tools. So be prepared to dig in to them, learn how they operate, and hack around when necessary.</p><p>Interested in working on these types of problems or about the company? Check out our job board; we are hiring <a href="https://grnh.se/jwi0xc">https://grnh.se/jwi0xc</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=424a15f8ea0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/remedial-data-science-engineering-424a15f8ea0c">Remedial Data Science Engineering</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Onboarding — Reflect yourself]]></title>
            <link>https://medium.com/in-the-weeds/onboarding-reflect-yourself-9901b41b691e?source=rss-10e5d25d538c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9901b41b691e</guid>
            <category><![CDATA[onboarding]]></category>
            <dc:creator><![CDATA[Orion Delwaterman]]></dc:creator>
            <pubDate>Tue, 20 Feb 2018 20:05:04 GMT</pubDate>
            <atom:updated>2018-02-20T20:05:04.339Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a post I originally wrote in way back in 2015 when I first joined Greenhouse Software. Our blog has since moved to medium and I am reposting it here for all the world to see:</p><p>As I write this blogpost, it is day 5 for me at Greenhouse. I really loved the whole interview process with Greenhouse, and I am super excited to get working. But as all new employees know, before you can contribute to your company you have to go through the onboarding process. A large chunk of institutional knowledge has to be transfered from the employer to you. Everything from the individual responsibilities of each job, company culture and guidelines to where the bathrooms are, and where to grab some good food.</p><p>For engineers, we have the additional dreaded task of setting up our environment and trying to learn the codebase. Every engineer has to go through the steps of taking that brand new laptop or desktop and configuring it to run the application you are going to work on. I’ve worked for a few different companies in my time and have consequently gone through various different approaches to getting that environment going. Everything from scheduled classes and side by side walk-throughs, to being given a laptop and told to figure it out. Greenhouse gave me a laptop the first day, had a couple of coworkers help me set up some accounts and then gave me a handy wiki page to start building everything out.</p><p>As I was building my environment, I was told by many of the engineers to feel free to provide feedback. If I saw anything that was confusing or maybe could be done in a better manner to go ahead and provide feedback. I was super excited to hear this because most companies miss this opportunity. Onboarding should not only be about a new employee learning about their new company, but also their company learning about themselves.</p><p>When a new employee struggles to understand something, the company often thinks that something is wrong with the onboarding process. They add new classes or revamp how they are teaching the old ones. But sometimes when a new employee struggles to understand some new process it really means that process is too complex. The onboarding process provides clear feedback on how your company looks to an outsider, and every company should be leveraging the feedback from onboarding to fix the hidden problems within their own organization. This is especially true for the software we write. If a new dev has a really hard time understanding how the code works, or if it takes them forever to set up their environment, you just got a strong signal that something is wrong in your code and you should be investigating it.</p><p>So the next time when you onboard a new employee, pay close attention to them during the process. Here are a couple of things to keep in mind:</p><h3>Your Cognitive Bias — The Curse of Knowledge</h3><p>Imagine your sweet old grandmother calls you up for help with her email. Something is wrong with her gmail and nothing works. You tell her to push F5 to refresh the browser and she hits the ‘F’ key and then the ‘5’. You get immediately frustrated because how could anyone not know what the ‘F5’ key is. Of course, you are a good grandchild and don’t say anything about it. She’s 90 years old, you remind yourself. She hasn’t used computers for that long and its all new to her.</p><p>The initial frustration you are feeling with your poor grandmother (who you never call by the way), comes from a cognitive bias called <a href="https://en.wikipedia.org/wiki/Curse_of_knowledge">The Curse of Knowledge</a>. Basically you as a human being tend to assume that other people have the same knowledge that you do and will therefore act as you would. Your brain knows what an ‘F5’ key is so it assumes your grandmother does as well. This is the same thing for new employees. If an onboarding process or code structure seems obvious to you but confounding to a new employee, its not because the new employee is a dud. They simply do not have all the knowledge that you do. More importantly when they get really stuck, whatever knowledge they need to proceed is not easily discoverable. So when you see that smart new dev struggling to set up their environment, this is a signal that something could be wrong.</p><h3>World standards vs Company Standards</h3><p>I live in an old apartment in New York City, which means my apartment has some quirks. Specifically, for reasons I don’t understand, the cold and hot water is reversed in all the sinks and the shower in my apartment. Yes, in my apartment cold is on the left and hot is on the right. Now I have lived there for several years so I don’t even think about it anymore. However, whenever someone visits me they are quickly suprised when they turn the wrong knob and suddenly the cold glass of water they were expecting is instead luke-warm. Essentially there is a deviation between the standard US temperature controls and the temperature controls in my apartment.</p><p>This happens a lot with software and processes as well. A new employee tries to do something that follows a general standard and is surprised to find out that it doesn’t work. This is your cue as the business that you have deviated from the rest of the world. New employees are the best people to alert you to these differences, because they don’t have the biases that everyone in the company has from working there for years. Just how I no longer think about turning the left handle to get cold water, so too will you easily forget that the patterns you use at your company may not be the same as the rest of the world.</p><p>Once that happens now you have to ask yourself: “why?”.</p><h3>Reasons to Deviate: Historical vs Purposeful</h3><p>When you are providing an explanation to a new employee about why you would use a bastardized version of XML instead of the accepted world standard, be aware of the type of reasons that you are giving: Historical or Purposeful.</p><p>A purposeful reason is a change made to accomplish some goal, whereas a historical reason is a cause and effect history of how something got to the state it is. The difference here can be subtle, but it’s important. Here are two examples:</p><p><a href="https://www.elastic.co/">Elasticsearch</a> is free text search engine that uses a RESTful HTTP JSON API. Basically anytime you want to do anything with Elasticsearch you make a web request with your data as a JSON object in the body. There is one call where this differs: the bulk insert. For this one operation you must supply the data in JSON objects separated by newlines instead of in a proper JSON array. Why not an array you may ask? Because if you had 20,000 objects pased in an array you would have to parse the whole body in one go, whereas objects seperated by newlines can be streamed. Elasticsearch deviates from a standard to make their bulk processing more performant. Elasticsearch deviated from the world standard to achieve a specific goal.</p><p>At a company I previously worked at (which shall rename nameless), I had to make a url that would reply with all comments on an article in XML for the company’s search engine. Except that instead of normal XML with the starting XML tag and all the data in a root node, I had to strip it all down to just an a bunch of comment nodes all in sequence. Why was this? Well the guy who original wrote the script to import comments had built it that way and had since left the company. Nobody on his team knew how it worked and so no one ever touched it. This explanation is more about the history of the product, rather than a specific goal trying to be met.</p><p>I tend to be very suspicious of historical reasons. Typically these reasons are more about being lazy and/or unwilling to invest in refactoring code, than sound technical decisions. That doesn’t mean that historical reasons can’t be good decisions, say for instance you are Facebook and you have too much data to run migrations. But chances are more likely that your historical reasons are pretty poor. So if you hear yourself explaining something to the new dev by recalling the history of something, you should be immediately suspicious of this decision.</p><h3>Conclusions</h3><p>As you go through this process with new hires, remember that none of these signals will tell you whether something is bad or good. It’s just a good place to take some honest reflection. Maybe your company has a very good reason for WSDL files, or using non-standard XML. But the important thing with these roadbumps is to have conscious reasons for the hard parts of your apps. This honest reflection will help point to great refactoring points and should improve the quality of your code overall.</p><p>Greenhouse is smart enough to ask for this type of feedback, and I am excited to get working here. Want to join? Apply <a href="http://grnh.se/jwi0xc">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9901b41b691e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/onboarding-reflect-yourself-9901b41b691e">Onboarding — Reflect yourself</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We need to stop talking to ourselves]]></title>
            <link>https://medium.com/@o_the_del/we-need-to-stop-talking-to-ourselves-71bd5e24f89a?source=rss-10e5d25d538c------2</link>
            <guid isPermaLink="false">https://medium.com/p/71bd5e24f89a</guid>
            <category><![CDATA[democrats]]></category>
            <category><![CDATA[2016-election]]></category>
            <category><![CDATA[hillary-clinton]]></category>
            <dc:creator><![CDATA[Orion Delwaterman]]></dc:creator>
            <pubDate>Wed, 09 Nov 2016 15:17:34 GMT</pubDate>
            <atom:updated>2016-11-09T19:20:23.815Z</atom:updated>
            <content:encoded><![CDATA[<p>I am pretty depressed this morning. I am a Democrat, a liberal, and a donor to Hillary Clinton. I am shocked and stunned about the election, and I have no idea how to react. Like many of my friends I don’t understand how anyone voted for Trump because I don’t know any Trump voters. And I am beginning to realize that’s the problem.</p><p>I think the natural reaction from the left will be to go looking for someone to blame. Some people will blame the Democrats for selecting Hillary, saying that Bernie could have won. Some will blame the pollsters who completely blew their predictions. Some will blame the so called “protest” voters who possibly could have swung the election in Hillary’s favor. We’ll talk about sexism and racism. It’s tempting to tunnel that anger and frustration towards one group or person, but its ultimately unproductive. Yelling at people is unproductive.</p><p>As a white college educated liberal male who lives in New York City, I enjoy a lot of privilege. The people around me also enjoy that same privilege. I have pushed politically for minority rights and so I considered myself egalitarian. But there are people in this country that believe I scoff at them; that I mock and scoff at their ideas; that I ignore their struggles. While I would disagree with these thoughts I have to admit that I have routinely ignored them. I don’t know anybody in a rural part of the country. I know very few evangelicals. I avoid talking about politics with my more conservative parts of the family.</p><p>I think its time I met some of these people, and I stop avoiding these conversations. Maybe we can have a coffee and just get to know each other. Maybe we can listen to each other instead of talking over each other. It’s a skill that I think all Americans can begin to develop: listening.</p><p>So, my fellow liberals, the next time you hear someone spouting something you disagree with, just listen. You don’t have to agree, and you can certainly respond. But we should at least acknowledge what the other side is saying. After all, we are one country and we have to live together.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71bd5e24f89a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embrace your inner Project Manager]]></title>
            <link>https://medium.com/in-the-weeds/embrace-your-inner-project-manager-c5f574d9736f?source=rss-10e5d25d538c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5f574d9736f</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Orion Delwaterman]]></dc:creator>
            <pubDate>Tue, 27 Sep 2016 20:02:28 GMT</pubDate>
            <atom:updated>2016-09-28T14:34:46.741Z</atom:updated>
            <content:encoded><![CDATA[<p>Like any growing software company, we here at Greenhouse have technical debt. Face it: we all do. In our quest to reduce it, we recently completed an upgrade of our main software from Rails 3.2 to Rails 4.2. The upgrade was long overdue, and we still have to do the upgrade to 5.0, but we were happy to retire it.</p><p>During this upgrade we learned a lot. We are planning to write a whole series of articles regarding technical details and different upgrade strategies. But this post is about the “softer” skill here: project management.</p><h4>Project Management? I don’t do that!</h4><p>Let’s say you have decided that you really want to tackle some major technical debt in your codebase. Rewrite that ancient app that was written before your company pivoted, break your monolithic app into services, or maybe, you just want to upgrade your framework like we did.</p><p>That’s great and all, but I am going to guess that you don’t work in isolation. You need a champion! Not a hero coder who will do all the work (more on that in another blog post), but an advocate for change that can speak with the entire company. That person can lay out reasons for the change, communicate progress, and generally do all the coordination that is necessary besides just the coding.</p><p>You may be lucky enough to work in a company where your Product Manager may be your advocate, or maybe you’re even lucky enough to have a Technical Product Manager. But if you work at a smaller company, or one that is not that technically minded, this is normally not the case. Product Managers are worried about customer features; Technical Product Managers will be hired in “the future”. Chances are one of the engineers will have to take this mantle on.</p><p>For this latest upgrade that task fell to me. Along the way I think I learned a few things about it. So here are a couple of things I learned while managing this process.</p><h4>Pitching: How to get the work scheduled</h4><p>I have never met a product manager that liked upgrades or addressing technical debt. It’s not that all the product managers I know are mean; far from it. It’s just that the pain of technical debt is felt most directly by developers, not customers, and the benefits of any technical upgrade are not always easy to spot. It took time for me to learn that part of my job as a developer is to make the case for the upgrade, not just decree it.</p><p>When you are planning on asking for the resources and time to do this type of upgrade, you need to put yourself into the mind of your product managers. Devoting resources to upgrading some technical part of the platform is incredibly expensive. Not only does the company spend money on the developer time for the upgrade, but it also loses the opportunity to develop new customer-facing features. This is already a pretty high cost, and you haven’t even started your pitch.</p><p>You are going to have to explain the value, i.e. build a business case. Will this upgrade help with security? Make development faster? Will it improve the speed and reliability of your servers? When we put together a plan for the upgrade, we added a single slide to our presentation listing the benefits of doing the upgrade. We talked about technical benefits such as speed and security, but also softer values like developer happiness. You feel the pain of technical debt directly. So you are the person best equipped to talk about why reducing it makes the company stronger.</p><h4>Communication: How to keep the project going</h4><p>Once we had approval for the project, we eagerly started coding away. We had a plan going forward on how to run the upgrade, and we had developers dedicated to the cause. All was right with the world, right? Well, no.</p><p>While we had a great plan about how to proceed, we didn’t have a plan on how to communicate the project’s progress to the rest of organization. This lack of communication created a crisis of confidence for product and business. The lack of confidence first started manifesting with the head of product walking to my desk every week asking me how things were going. It culminated with the head of engineering pulling me aside to tell me that the project felt like a big black hole to management and that people were scared.</p><p>Part of the problem with these types of upgrades is demonstrating what’s been built. When we are building a customer-facing feature, its easy enough to boot up our local server and show the new widget we just added to the dashboard. With this kind of upgrade there is nothing to show except maybe some console output on your machine. So all our managers had to judge the project is what we told them.</p><p>We changed our process immediately. We shared our milestones for the project with all the stakeholders. These were pretty simple such as <em>Rails 4 server boots</em>, <em>Docker build and deploys correctly to server</em>, and <em>Tests run (not necessarily succeed)</em>. And for each milestone we came up with some sort of metric to measure our progress within the milestone. The metrics weren’t always perfect, but they didn’t have to be. As long as they gave a rough estimation for where we were, it helped management to understand our overall progress. We also began sending out weekly status emails to all the stakeholders. We would outline what we accomplished, what problems we encountered, and what we planned to do the next week.</p><p>These steps were very simple, but incredibly valuable. Not only did it reduce the number of queries that we received about the project, but it also built trust. In feedback I got from the Product team after the upgrade, everyone listed the emails as the best tool we gave them to track the progress of the project. That confidence will keep you and your team happily coding away.</p><h4>Closing: How to finish the project</h4><p>Depending on the nature of your project, this can be very simple, or (as in most cases) very very complex. For our upgrade we did a Blue-Green release that was complicated by the fact that we had to have two separate queues for background jobs. But the complexity of the deployment is also going to drive the complexity of the communication that is needed to finish the project well.</p><p>As we approached the end of the project we started talking to every department to let them know the upgrade was coming. We talked with our Customer Support, Sales, and Infrastructure teams. When we got within two weeks of deployment we had quick daily stand up meeting with a representative of every department directly involved in the release.</p><p>All this upfront communication paid dividends during the migration. When things started to go wrong (and something always goes wrong), everyone was prepared and ready to act. Nobody in the company panicked because they knew what was happening, and had clear lines of communication to check in when they needed to. This gave us a great deal of flexibility and adaptability that allowed us to react quickly to issues and fix them on the fly.</p><h4>Summary</h4><p>If you have arrived at this point, you’ll notice that all three sections are really about communication. Developers are supposedly not any good at talking to other human beings. But it’s essential if you are going to project manage your upgrade. Talk to your company, and spend the time to make sure people understand how and why you are doing that massively long project.</p><p>So go ahead and get the ball rolling. Don’t be afraid of stepping into that advocate roll. You’ll be better for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*3fUnjTQZkyzH0nPyuNzf3g.png" /></figure><p>Orion Delwaterman is a Senior Software Developer at Greenhouse Software with 10 plus years of experience in Ruby, Rails, Javascript, and other languages that go bump in the night. He’s also a stage actor who loves doing Shakespeare, and is just a little bit tall.</p><p>Work with us. <a href="http://grnh.se/jwi0xc">http://grnh.se/jwi0xc</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5f574d9736f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/embrace-your-inner-project-manager-c5f574d9736f">Embrace your inner Project Manager</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>