<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ethan Rosenthal on Medium]]></title>
        <description><![CDATA[Stories by Ethan Rosenthal on Medium]]></description>
        <link>https://medium.com/@eprosenthal?source=rss-f1ddb650e48c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7ZRebZiFIQiBpohm.jpg</url>
            <title>Stories by Ethan Rosenthal on Medium</title>
            <link>https://medium.com/@eprosenthal?source=rss-f1ddb650e48c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:06:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eprosenthal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimization in Sickness and in Health]]></title>
            <link>https://medium.com/dia-co-engineering/optimization-in-sickness-and-in-health-7d8f28f0bd45?source=rss-f1ddb650e48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d8f28f0bd45</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[integer-programming]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[weddings]]></category>
            <dc:creator><![CDATA[Ethan Rosenthal]]></dc:creator>
            <pubDate>Thu, 18 May 2017 16:01:01 GMT</pubDate>
            <atom:updated>2017-05-18T16:01:01.129Z</atom:updated>
            <content:encoded><![CDATA[<p>As a personal styling service, we at Dia&amp;Co are always looking for ways that algorithms may help us to create personalized experiences that will surprise and delight our customers. If those algorithms happen to make us more efficient in the process, well that’s called a win-win! This is often the case where optimization algorithms come into play — they help us to quickly maximize some objective function subject to real world constraints.</p><p>For example, we may deal with a problem where we want to maximize a proxy for customer delight, like the number of items that she keeps in her box, subject to the fact that we only have so much inventory. Alternatively, we may match each customer up to the stylist who best <em>gets</em> the customer’s style, but each stylist can only take on so many customers.</p><p>When I’m dealing with optimization algorithms day-in and day-out, they tend to bleed into the rest of my life. Recently, I was attempting to figure out the seating arrangements for my upcoming wedding. This turned out to be quite a difficult task. We are limited by the number of tables that we have, certain people <em>have</em> to sit together (e.g. my bride-to-be and I, obviously), and certain people do <em>not</em> want to sit together. I sat there staring at a spreadsheet for quite some time but did not get very far. Every time I placed one group of people at a table, that created an issue at a different table. This started to feel like an optimization problem…</p><h3>Optimization to the rescue</h3><p>Seating arrangements naturally lend themselves to a solution via integer programming, an optimization problem where some variables must take on integer values. As with any optimization problem, the key is to identify the objective function and the constraints. We will attempt to maximize or minimize our objective function, and the constraints define what we consider to be a valid solution.</p><p>For our problem, the objective function is quite simple. We simply want to give everybody a seat. A more advanced version of this problem may explore “good” seats versus “bad” seats, but we’ll leave that as an exercise to the reader 😏. Thus, we will give 1 “point” for every guest that gets a seat, and that’s all there is to our objective function.</p><p>The constraints are a bit more complicated. We can list them out here, and then we will write them in code, later:</p><ul><li>Every guest must get a seat</li><li>There is a minimum and maximum number of people per table</li><li>Certain people must sit together</li><li>Certain people must not sit together.</li></ul><h3>Simulating the Wedding</h3><p>Out of respect for my wedding guests, I will not walk through the “production” data here (does Aunt So-and-So want the world to know that she doesn’t want to sit with Cousin Such-and-Such? Probably not.). Rather, I will simulate a bunch of wedding guests and constraints and show you how to solve the made-up problem.</p><p>It turns out that there is a simple python package that is pip-installable called <a href="https://pypi.python.org/pypi/names/">names</a> which lets us generate a bunch of fake guests. We’ll go ahead and create a 120-person wedding.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c69829526666d4264050f29ae3815502/href">https://medium.com/media/c69829526666d4264050f29ae3815502/href</a></iframe><p>Okay cool, looks like Laura is getting married to George. Congratulations! We will now generate our “constraints” where we randomly select people who must sit together and randomly select people who must not sit together. In order to simulate naturally occurring “groups” of people, like families, one person may be paired with multiple other people. By storing each pair or anti-pair as a sorted tuple of names, we are guaranteed to only have unique pairings in our sets (beware that this <em>could</em> result in groups that are larger than the table size).</p><p>Note that we have a check in place to make sure that we do not create a conflict where two people must both sit and not sit together. Also, we should make sure the bride and groom are at the same table!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54d4e1fd30c919a5b21fe274b945654e/href">https://medium.com/media/54d4e1fd30c919a5b21fe274b945654e/href</a></iframe><h3>Defining the problem</h3><p>With our wedding guests simulated, we now have the three typical steps of solving an optimization problem. We must first define our <em>variables</em>, the unknown terms in our math problem which we would like to solve for. In our case, we will make our variables binary indicators that indicate Yes or No whether a guest is assigned to a particular table.</p><p>For the math-inclined, we can think of this as a matrix where each row represents a guest, each column represents a table, and the entries are 1 or 0. For those more comfortable with code, we can treat our variables as a dictionary, where the key is a tuple of (guest, table), and the value is a boolean True or False.</p><p>In terms of solving the actual optimization problem, we will not be writing these algorithms ourselves; thankfully, there are libraries that do this for us. We’ll use <a href="https://www.gnu.org/software/glpk/">GLPK</a>, a popular, free solver. The only issue is that it is nontrivial to convert Python code into a correct input to GLPK. That’s where APIs come in! <a href="https://github.com/coin-or/pulp">pulp</a> is a Python package that allows one to build optimization problems in familiar Python code and then converts that code to suitable input to a bunch of different optimization libraries.</p><p>In the code below, I commit the Python <a href="https://www.python.org/dev/peps/pep-0008/#imports">sin</a> of importing <em>everything</em> from the pulp library. For whatever reason, many of the optimization API libraries that I have seen (<a href="http://www.cvxpy.org/en/latest/">cvxpy</a>, <a href="https://www.gurobi.com/index">Gurobi</a>, pulp, etc…) all perform wildcard imports in all of their examples. Go figure. With that said, most of the unfamiliar functions in the code that follow likely come from pulp (LpProblem, for example).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ec7f7670fbf1feba391f58ff97f169f/href">https://medium.com/media/1ec7f7670fbf1feba391f58ff97f169f/href</a></iframe><p>With the variables defined, the next two steps are defining our objective function and defining the constraints. The objective function is fairly simple — we will give 1 “point” for each guest assigned to a table. The constraints follow and are documented in the code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f39480eec6c36d75af6a405e14009573/href">https://medium.com/media/f39480eec6c36d75af6a405e14009573/href</a></iframe><h3>Let’s be solutions oriented</h3><p>Now that the whole problem is defined, all that is left to do is solve! We call solve on our problem, and everything gets passed through to GLPK. The log file for GLPK will be stored in glpk.log should you want to peruse it later. Once solved, we then inspect our seating arrangements.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b1505c2e2bf69b27ab25320a2464a40/href">https://medium.com/media/9b1505c2e2bf69b27ab25320a2464a40/href</a></iframe><p>Awesome! Everybody has now been assigned a table. We can print out this list, but it always helps me to visualize things. Please pardon the hideousness of the following script as I wrangle matplotlib to show me the seating arrangement.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44a3e5c7df787f385774831082d560dc/href">https://medium.com/media/44a3e5c7df787f385774831082d560dc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fR1QYz8ZpquvhE0JuN05iA.png" /><figcaption>Seating table arrangements with bride and groom in red</figcaption></figure><p>If you find yourself thinking about how math and programming can help to optimize your daily life, then head on over to our <a href="https://www.dia.com/careers">careers</a> page — we’re hiring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d8f28f0bd45" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dia-co-engineering/optimization-in-sickness-and-in-health-7d8f28f0bd45">Optimization in Sickness and in Health</a> was originally published in <a href="https://medium.com/dia-co-engineering">Making Dia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Data Tips for Machine Learning in Production]]></title>
            <link>https://medium.com/dia-co-engineering/5-data-tips-for-machine-learning-in-production-360e14addeee?source=rss-f1ddb650e48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/360e14addeee</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Ethan Rosenthal]]></dc:creator>
            <pubDate>Fri, 31 Mar 2017 19:02:04 GMT</pubDate>
            <atom:updated>2017-03-31T19:02:04.343Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ClHNmgl17-w8icGhsykSRg.jpeg" /></figure><h3>1. You can’t measure that which you do not have</h3><p>This may seem like an obvious statement akin to Wayne Gretzky’s (or Michael Scott’s) “You miss 100% of the shots you don’t take.” Just like that quote, it can be helpful to continually remind oneself that one first needs to record events before they can be measured.</p><p>Let’s imagine that you have trained a logistic regression model to detect fraudulent credit cards entered into your payment processing system. You spend time optimizing your performance metrics offline, pick a threshold for your classifier, and deploy the system as an API that returns True or False dependent on whether or not the credit card is fraudulent. You record an event with the user information and whether or not the credit card was fraudulent.</p><p>Later, your model performance slips, and you wonder if your threshold should change. Unfortunately, you never tracked the raw logistic regression output (just the binary output), so you have no way to quickly check the data to validate this hypothesis.</p><h3>2. When in doubt, track it</h3><p>If you cannot measure that which you do not have, then make sure you have as much as possible! We live in an age where data storage is cheap and scalable database queries are getting easier. So just track it.</p><p>It is rare to ever regret capturing an event. On the other hand, nothing can make up for the lost data due to not tracking an event. Even if you are simply writing data to a log file, that log file <em>will</em> come in handy when things break (and things <em>will</em> break).</p><p>In the parlance of machine learning (ML), you can think of this like valuing recall over precision. If you treat the act of tracking an event as a positive classifier prediction, then useless events can be called false positives. These are fine! But, if not tacking a useful event is a false negative, then this is what we should be minimizing.</p><h3>3. Inputs matter too</h3><p>ML systems are fundamentally functions with inputs and outputs. You typically input some data and get a prediction out. Even if you have taken the previous two tips to heart and are measuring all sorts of outputs from the ML system, the inputs define the output.</p><p>What were the inputs to the model at the time the time of prediction? In the fraudulent credit card example, maybe the front-end provides the threshold to the model via an API query parameter. Were you tracking that incoming API call? If not, then at some point in the future you will not remember that threshold value.</p><h3>4. All data has a timestamp</h3><p>ML is often presented as a prediction over static features. It is not. Imagine you are predicting the lifetime value of a new user to your service, and the state the user lives in is a feature of your model. You one-hot-encode the state, and train and deploy your model.</p><p>There is an assumption here that the way the user’s state was captured never changes. However, maybe your company switched from asking the user explicitly to inferring their state from their IP address. The latter method is much noisier, and this will manifest itself in your model.</p><p>Do you know <em>when</em> the user’s state was written to the database? Later on, when you backtest or train offline, will you know which method was used for recording the user’s state? Furthermore, will you know all of the other aspects of your model — feature engineering, model parameters, <em>when </em>the model was trained, etc…?</p><p>This is by no means easy., and you will not be able to handle all assumptions. In fact, Netflix wrote a gigantic <a href="http://techblog.netflix.com/2016/02/distributed-time-travel-for-feature.html">post</a> about all of their efforts in this realm, but admits that this is “just a start”. Still, this does not mean that you should not try to either record some assumptions or at least be cognizant of them.</p><h3>5. True performance metrics are measured in production</h3><p>Even if you have tried to follow tips 1–4 and are super diligent about running unbiased, offline ML experiments — you split into training and test sets, you run K-fold cross validation on the training set, and finally test on the test set — the true measure of your model’s performance will be when it is actually performing. So, make sure that you are tracking the predictions and periodically evaluating the final performance. You may be surprised!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=360e14addeee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dia-co-engineering/5-data-tips-for-machine-learning-in-production-360e14addeee">5 Data Tips for Machine Learning in Production</a> was originally published in <a href="https://medium.com/dia-co-engineering">Making Dia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing New Office Anxiety with a New Citi Bike Dataset]]></title>
            <link>https://medium.com/dia-co-engineering/reducing-new-office-anxiety-with-a-new-citi-bike-dataset-fb469fd6f5b6?source=rss-f1ddb650e48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb469fd6f5b6</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bike-sharing]]></category>
            <category><![CDATA[commuting]]></category>
            <dc:creator><![CDATA[Ethan Rosenthal]]></dc:creator>
            <pubDate>Fri, 17 Feb 2017 20:02:00 GMT</pubDate>
            <atom:updated>2017-02-17T20:02:00.816Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dXa9LLzHSnl5PEjkb5xW-w.jpeg" /><figcaption>Macdougal St Station</figcaption></figure><p>Dia&amp;Co recently outgrew its old office and moved across town to a larger space. While this sounds like the best problem for a startup to experience (we keep growing!), I was anxious.</p><p>Was I worried that with 4x the space we might lose the tight culture that comes with close-quarters? No, we have been persistent in our commitment to <a href="https://making.dia.com/how-our-engineering-team-acts-on-our-core-values-every-day-6ce7811b015e#.xwnzcmi6e">defining and acting on our core values</a>, and these will stand the tests of a larger office.</p><p>Was I worried that 6x the conference rooms would be insufficient for our rapidly growing teams? Eh, I think there must be a physical law that says that startup conference rooms are scale invariant — they will always be booked regardless of the number of employees.</p><p>Was I worried that the increased sunlight would make it harder to concentrate? Come to NYC in the winter — I will happily welcome that problem.</p><p>No, I was worried about my new commute. As a daily user of New York City’s bike sharing program, Citi Bike, I have first-hand experience with the laws of supply and demand in a hot economy. Any rider of the Citi Bike system knows that stations are liable to be full when you need to drop a bike off, and conversely stations are often empty when you really need to check out a bike. In the evenings, I would often have to pick up my bike at a station further from the office due to the closer station being empty. “What would the situation be like at our new office?”, I wondered.</p><p>Thankfully, we here at Dia&amp;Co are well-versed in predicting demand in a hot economy. We run a reasonably lean warehouse, so forecasting inventory levels becomes especially important for satisfying a rapidly growing customer base. Let’s put our data science hats on and get to work.</p><h3>Data: When in Doubt, Get it Yourself</h3><p>Anybody who reads data science blog posts online is probably sick of hearing about analysis of Citi Bike data. Every couple months, the Citi Bike service releases public datasets of every ride taken on the system over some period of time. Unfortunately, this data is not sufficient to back out how the bike occupancy levels varied as a function of time at different stations across the city.</p><p>Thankfully, I have been collecting my own dataset for a couple months, now. Citi Bike has a public API which allows one to query the current number of available bikes and docks at each station in the city. I have a small EC2 box which has been hitting the API every 2 minutes since September 18th, 2016 and storing the data in a Postgres database. As of February 14th, there are over 70 million rows in this database. The code for generating this dataset is available <a href="https://github.com/EthanRosenthal/bike-listener">here</a>.</p><h3>Dasking the DataFrame to Save Some RAM</h3><p>I had previously dumped the whole database to a CSV and transported it to my local computer. Uncompressed, the dataset is 8.6 GB on disk. While my 16 GB MacBook Pro could hold everything in memory, between PyCharm, Slack, and any data manipulations, I was worried about running out of memory.</p><p>This is where <a href="http://dask.pydata.org/en/latest/">dask</a> comes in. Dask does a lot of things, from parallelized big data analytics to distributed task scheduling, and it all happens in Python. Likely the most familiar use-case for data scientists is to run pandas-esque operations on data which does not fit in memory. Dask integrates quite nicely with pandas, so one can reuse much of the same code.</p><p><em>Disclaimer: I should have just queried the data when it was in the Postgres database, but then I wouldn’t have had a chance to talk about dask!</em></p><p>We’ll start by reading the Citi Bike dataset into a dask DataFrame and pulling out the old and new office stations’ data as regular pandas DataFrames. We’ll call these old and new stations howard and macdougal, respectively, for the streets on which they reside. Dask evaluates computations &quot;lazily&quot; (like Spark), so the real computation does not happen until we actually pull out the relevant stations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90335934e75e52d8c27d348ae223ec14/href">https://medium.com/media/90335934e75e52d8c27d348ae223ec14/href</a></iframe><h3>Visualizing Occupancy</h3><p>With the two stations’ data loaded into memory as pandas DataFrames, we can start to play and get a feel for the data. To start, let’s look at how the number of docks available at the old, Howard bike station varied across the full time of the dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3010da080ae27552cfde52e1830aecf/href">https://medium.com/media/c3010da080ae27552cfde52e1830aecf/href</a></iframe><p>Well that looks fairly incomprehensible.</p><p>The data is currently indexed by the execution_time which is the time that I pinged the Citi Bike API. We&#39;ll go ahead and resample the time period of the data and also get rid of a bunch of columns that are not relevant to this analysis. I&#39;ll then try averaging each days&#39; data in order to get a simple picture of what a typical day looks like for the station. We&#39;ll ignore weekends (because I&#39;m not working then!), and I&#39;ll also faintly plot each days&#39; data in order to get a feeling for the variance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efd1b7b29f53bab24153211691c4deec/href">https://medium.com/media/efd1b7b29f53bab24153211691c4deec/href</a></iframe><p>We can see that the old office Howard station and new office Macdougal stations are commuter <em>destination</em> bike racks. How do we know this? It fills up during work hours (the number of available bikes increases) and empties during evening hours. Ideally, my new office would represent a reverse-commute. This looks to not be the case!</p><p>We should expect the opposite behavior from a commuter <em>origin</em> station. We can easily find one in Alphabet City, a neighborhood with few offices and a lack of subway stations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/064e603312f31254ffddf5853e87d3c0/href">https://medium.com/media/064e603312f31254ffddf5853e87d3c0/href</a></iframe><h3>Old vs. New</h3><p>Back to the problem at hand — would I run into more issues picking up a bike at the new office versus the old one? To turn this question into math, I will say that my goal is to determine the probability that I will not be able to find a bike when leaving work on a given day. There are a number of ways that we could answer this with the data. Motivated by working at a fast growing startup, I opted for a method with only enough complexity to answer my question with the necessary accuracy.</p><p>I tend to leave work sometime between 6 and 7 PM. Let’s look at what the number of available docks at the old and new office stations looked like around that time range on a single day:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3db6749e34017b03f4c2f4bdad8eafa/href">https://medium.com/media/e3db6749e34017b03f4c2f4bdad8eafa/href</a></iframe><p>This bodes well for the new office! We can see that during the entire period from 6–7 PM, there were available bikes at the new Macdougal station. Meanwhile, the old Howard station emptied out at 6:45 PM.</p><p>Now that we’ve seen what a typical day looks like, we can formulate a plan. First, I will say that my departure time is uniformly distributed between 6 and 7 PM (planning ahead is not my strongest suit). I am also going to make the assumption that the moment the station hits zero bikes available, it will stay like that for the rest of the time period from 6–7 PM. This is because the fluctuations around zero bikes are often quick and small, so my chance of grabbing a spot during this time is likely minimal and below the resolution of our measurement. Additionally, there are often spurious effects in the data when bikes and docks break (which happens often and ought to be the subject of a separate blog post).</p><p>With the above assumptions in place, we can easily calculate the probability that I will not be able to find a bike by finding all of the times between 6 and 7 PM when we assume the bike rack to be empty and dividing that by all of the times between 6 and 7 PM.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b674f3af05790867c8f05d4de628bc9/href">https://medium.com/media/7b674f3af05790867c8f05d4de628bc9/href</a></iframe><p>There you have it — my anxiety was unwarranted! I am 1.8x less likely to not be able to find a bike at the new office versus the old. <em>Score 1 for Macdougal</em>.</p><p>What about arriving in the morning? We can ask the same question of whether or not I will be able to park my bike betwween 9 and 10 AM. Unfortunately, it looks like the situation is worse at the new office.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62980a4bfc7dce623d6dd33f894466a0/href">https://medium.com/media/62980a4bfc7dce623d6dd33f894466a0/href</a></iframe><p>While a solution would be to get to work earlier, I think I’d rather park at a further station.</p><p>Do you like diving into nonstandard datasets and extracting insights? Do you have a data project that you have been working on which you are excited about? We’re <a href="https://www.dia.com/careers">hiring</a> and would love to hear about it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb469fd6f5b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dia-co-engineering/reducing-new-office-anxiety-with-a-new-citi-bike-dataset-fb469fd6f5b6">Reducing New Office Anxiety with a New Citi Bike Dataset</a> was originally published in <a href="https://medium.com/dia-co-engineering">Making Dia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scope the Solution before Solving the Machine Learning]]></title>
            <link>https://medium.com/dia-co-engineering/scope-the-solution-before-solving-the-machine-learning-7e5ddb622733?source=rss-f1ddb650e48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e5ddb622733</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[bias]]></category>
            <dc:creator><![CDATA[Ethan Rosenthal]]></dc:creator>
            <pubDate>Fri, 23 Dec 2016 21:26:01 GMT</pubDate>
            <atom:updated>2016-12-24T02:33:02.408Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20eHbYzqtMnfHU1XdHy56Q.jpeg" /></figure><p>Science encourages us to first attempt to explain new phenomena using our existing knowledge. If this does not work, then new theories must be created. This process is honed in science classes. For example, students rarely receive problems on physics exams that are exactly the same as a homework problem. The expectation is that the students will be able to translate the new exam problems into their existing understanding of physics in order to solve them.</p><p>Similarly, in data science, our first instinct is usually to convert a proposed problem into a previously known solution. This is encouraged in interviews. We are presented a case study</p><blockquote>Determine customers likely to churn</blockquote><p>and we think</p><blockquote>Ok, that means classification: did_churn = 1, did_not_churn = 0. I know how to solve this with logistic regression, random forests, etc…</blockquote><p>While this approach may get us the job, we should not be so cavalier when actually using machine learning to build business solutions. We should employ a top-down approach and spend considerable time figuring out:</p><ol><li>What is the actual end goal?</li><li>What type of output do I need to achieve this goal?</li><li>Which types of models produce this desired output?</li><li>Which types of inputs would help me produce this desired output?</li><li>Are these inputs valid?</li></ol><p>Let’s delve into a made-up example of how we would tackle each one of these bullet points.</p><h3>1. What is the actual end goal?</h3><p>Imagine we work at a driving insurance company, and somebody tells us that they would like to know who will get into an accident next month. Our first instinct may be to say,</p><blockquote>Hey, that’s just like the churn problem! I’ll build a classifier, and I’m good to go.</blockquote><p>Not so fast! Let’s ask <em>why</em> the stakeholder wants to know this. There could be many reasons:</p><ul><li>They want to know which types of customers get in accidents more often.</li><li>They want to intervene somehow in order to prevent accidents.</li><li>They want to raise these customers’ rates.</li><li>They want to forecast expected claims over the next 6 months.</li><li>Etc…</li></ul><p>Each of these scenarios could require wildly different outputs, models, and inputs. Let’s assume that they want to raise these customers’ rates. This probably raises the question of how much to raise rates, how that relates to likelihood of accident, and a bunch of other questions. For simplicity’s sake, let’s assume that another department handles this, and we must only identify these customers.</p><h3>2. What type of output do I need to achieve this goal?</h3><p>This is partially where the idea of the data “product” comes into play. We know that the stakeholders would likely to identify customers for whom to raise rates. Would they like us to give them a list of customers we think will get into an accident next month? Would they like an “accident score” calculated for each customer? Do we want to allow non-technical people to make decisions on a technical score? Should this score populate a database or be returned by an API? Should the score be from 0 to 1? And on and on and on.</p><p>Let’s say that the stakeholders want a list of customers likely to get in an accident each month.</p><h3>3. Which types of models produce this desired output?</h3><p>We need to create a list of customers who we think will get into an accident. Machine learning can often be broadly grouped into classification, regression, recommendation, and reinforcement learning. These groups are not mutually exclusive (recommendation can be solved as classification, for example).</p><p>Our scenario feels like a classification problem: we must predict whether or not a customer will get into an accident. Importantly, we do not require a probability, so are freed to use various loss functions and models without having to think about how to convert a binary outcome into a probability.</p><p>We’ll keep it simple stupid and start linear with logistic regression.</p><h3>4. Which types of inputs would help me produce this desired output?</h3><p>Almost any article expositing on machine learning will claim that feature engineering constitutes the majority of work, and I have always found this to be true. When thinking through the inputs to our model, we can use our domain knowledge to get started.</p><p>For the case of accidents, a couple inputs that may come to mind are:</p><ul><li>Customer’s number of accidents</li><li>Customer’s age</li><li>Customer’s car</li></ul><p>Once we’ve determined a couple features, we must then figure out if we have the data. Lastly, we can attempt to build the actual inputs to the model with this data, but this is where things get tricky.</p><h3>5. Are these inputs valid?</h3><p>Even if we have the data and can convert it into a numerical form for input into a machine learning model, there are many ways to do this incorrectly. Often the error comes in the form of leaking the future into the past.</p><p>For example, let’s say we have a nice table called future_customer_accidents</p><pre>| customer_id | date_month | had_accident_next_month |<br>------------------------------------------------------<br>|   1000001   |   08/2016  |       False             |<br>|   1000001   |   09/2016  |       True              |<br>|   1000001   |   10/2016  |       True              |<br>|   1000002   |   07/2016  |       False             |</pre><p>and we wish to predict has_accident_next_month for each customer_id and date_month tuple. Imagine we have a separate table called accidents:</p><pre>| customer_id |      date     | accident_type |<br>-----------------------------------------------<br>|   1000001   |   2016-10-12  | fender_bender |<br>|   1000001   |   2016-11-24  | fender_bender |<br>|   1000002   |   2016-09-03  |    totaled    |</pre><p>If we want to calculate a customer’s number of accidents, then we can simply write</p><pre>SELECT<br>  fca.customer_id,<br>  COUNT(*) AS number_of_accidents<br>FROM future_customer_accidents fca<br>LEFT JOIN accidents a ON a.customer_id = fca.customer_id<br>GROUP BY 1</pre><p>Done, right? Not quite. By grouping our accidents table and ignoring the dates, we have counted future accidents when joining back to past dates. For example, we see that customer_id 1000001 had accidents in both October and November. We will count their number of accidents as 2 when predicting in September when we would not know at that time if they have had any accident.</p><p>While this was a specific example, we should always consider the validity of our inputs especially when dealing with temporal data. Beyond temporal data, we should also think about whether or not the input is <em>ethical</em>. We may have information about a customer’s race, income, sexual orientation, etc… While using this data as a feature to a model <em>may</em> increase the models’ predictive power, is it ethical to raise insurance rates on somebody due to their race? No. Often the biases that can arise in the machine learning model are more <a href="http://www.slideshare.net/eestola/when-recommendation-systems-go-bad">subtle</a>, and other times they are blatantly baked in from the <a href="https://arxiv.org/abs/1611.04135">beginning</a>. In fact, this is an active area of <a href="https://www.whitehouse.gov/blog/2016/05/04/big-risks-big-opportunities-intersection-big-data-and-civil-rights">research</a>.</p><h3>Moving Forward</h3><p>Next time you get a request for a machine learning solution, try to think through the above 5 steps. This time investment up front can avoid headaches in the future. And, if you like scoping projects and building machine learning products, then check out our <a href="https://www.dia.co/careers">careers</a> page and come help us build!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e5ddb622733" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dia-co-engineering/scope-the-solution-before-solving-the-machine-learning-7e5ddb622733">Scope the Solution before Solving the Machine Learning</a> was originally published in <a href="https://medium.com/dia-co-engineering">Making Dia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hiring Data Scientists: A Classification Problem]]></title>
            <link>https://medium.com/dia-co-engineering/hiring-data-scientists-a-classification-problem-456affe73592?source=rss-f1ddb650e48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/456affe73592</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Ethan Rosenthal]]></dc:creator>
            <pubDate>Fri, 28 Oct 2016 19:01:02 GMT</pubDate>
            <atom:updated>2016-10-28T19:12:39.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k41nKRtLeMQHcj-2rhGkKg.jpeg" /></figure><p>It is easy as an interviewee to bash the entire interviewing process — it takes forever, it’s painful, and it can be demoralizing. Just as well, it’s easy for interviewers to lament the difficulty in finding and assessing good talent. Being in New York City, this reminds me of the contention between drivers and pedestrians. We can switch between either role but will continually complain about the opposing one!</p><p>With this in mind, we have strived to make the data science interviewing process at <a href="https://www.dia.co/">Dia&amp;Co</a> respectful of the candidate while still serving its purpose of identifying people who match our needs. A key part of this process is the Data Challenge.</p><p>Take-home tests, data challenges, data assessments, or whatever else you want to call them have become <a href="http://firstround.com/review/how-to-consistently-hire-remarkable-data-scientists/">fashionable</a> as a step in data science interviews. As the Data team at <a href="https://www.dia.co/">Dia&amp;Co</a> has experience with many other companies’ interview processes, we have built up a wealth of opinions on what works and what does not for the Data Challenge.</p><h3>What the Challenge Is</h3><p>From our side, the purpose of the Data Challenge is</p><blockquote>To assess how well the candidate translates business problems into machine learning solutions as well as get an understanding of their coding ability and style.</blockquote><p>However, the challenge also gives the candidate insights into our business and some problems that we may be thinking about. After all, interviewing goes both ways, <em>especially right now when the data science market is so hot</em>.</p><p>The challenge itself takes the form of some data and a business problem presented to the candidate. Specifically, this is fake data corresponding to realistic data that a company would have. Sometimes, companies send out data with anonymized features (e.g. “feature_1”, “feature_2”) which feels largely academic and does not test domain or business thinking. The candidate is then asked to write a script to solve this problem and include a brief writeup of their approach.</p><h3>What the Challenge is Not</h3><p>We spent considerable time thinking about what the challenge should not do as a result of being burned on poor processes from other companies in the past. Primarily, the challenge serves as the <em>minimum bar</em> that we set for moving candidates passed this stage of the interview. Because of this, we do not give extra consideration for going above and beyond the requirements of the test because we do not wish to favor candidates with infinite free time. We do not wish to assess feature engineering prowess or how accurate one can make their machine learning model because we do not want them to spend all day on it. While these can be important to the job, these easily lead to scope creep in the data challenge, and we have other skills that we would rather prioritize.</p><p>All of this is an attempt to be respectful of the candidate’s time. Consequently, we expect the challenge to be able to be completed in less than 3 hours. Many companies claim an upper bound on the time that they expect candidates to spend on their take-home tests, but rarely have we found cases where this is actually true. In fact, I once completed half of a test while taking twice the amount of time, and the company moved me on to the next stage! In this case, the test length should have been cut in half (at least).</p><h3>Classifying</h3><p>Once the challenge is completed and returned by the candidate, we have a strict rubric for grading. The rubric gives approximately equal weighting to coding, machine learning, and business know-how. The process is then somewhat similar to a classification problem. The scores on each section of the rubric are our features, we add them up as our model, and then a threshold is used to determine Pass vs. Not Pass. Of course, this threshold is a key parameter for trading off between precision and recall. We do not want to waste time on false positive candidates, nor do we want to miss out on false negatives. We even employ an “ensemble model” and have multiple people grade in order to minimize bias and variance.</p><p>Maybe the entire process can be thought of as an operations research problem — maximizing hiring of candidates who will be successful on the job while constraining against money, time, and everything else. Our data challenge classification problem serves as an input into the global optimization problem, and <a href="http://blog.ethanrosenthal.com/2016/08/30/towards-optimal-personalization/">synthesizing</a> machine learning and operations is something we are very excited about here at <a href="https://www.dia.co/">Dia&amp;Co</a>.</p><p><em>If you would like to work on this synthesis, then please check out our </em><a href="https://www.dia.co/careers"><em>careers</em></a><em> page because we’re hiring! And let me know what you think of the challenge :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=456affe73592" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dia-co-engineering/hiring-data-scientists-a-classification-problem-456affe73592">Hiring Data Scientists: A Classification Problem</a> was originally published in <a href="https://medium.com/dia-co-engineering">Making Dia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embedding Everything for Anything2Anything Recommendations]]></title>
            <link>https://medium.com/dia-co-engineering/embedding-everything-for-anything2anything-recommendations-fca7f58f53ff?source=rss-f1ddb650e48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fca7f58f53ff</guid>
            <category><![CDATA[plus-size-fashion]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ethan Rosenthal]]></dc:creator>
            <pubDate>Fri, 30 Sep 2016 16:01:02 GMT</pubDate>
            <atom:updated>2016-09-30T16:01:02.654Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://www.dia.co/">Dia&amp;Co</a>, our goal is always to find the <em>best</em> clothing to match the personal style and fit of each and every one of our customers. We excel at this due to our deep expertise and understanding of our customers’ unique needs, and we are always striving to improve. One way we improve is through our use of data.</p><p>Let’s say that we wanted to build an algorithm to find the products that a customer is most likely to love given her unique style. We could go fancy, build out a deep learning solution, and maybe this would surface the very best products for each customer. Unfortunately, we would find reduced interpretability of the model. There are often multiple stakeholders that would find <em>more</em> benefit from knowing which features are important compared to a simple prediction score.</p><p>On the other end of the model complexity spectrum are simple linear models. While these tend to perform fairly well, our needs may be more complex than the model can handle. There could be many interaction terms at play, and we might not have the time or expertise to hand-roll all of them. Linear models also make it difficult to ask recommendation-style questions of the data like “Which tank-tops go best with this pair of jeans?”</p><p>We have recently fallen in love with latent factor models for their ability to fit complex objective functions, remain reasonably interpretable, and prove flexible under recommendation-style interrogation.</p><h3>Factors FTW</h3><p>Before I explain how we use latent factor models at Dia, let me provide a brief primer on what I mean by latent factors. There are classes of machine learning algorithms that learn vector representations of the data’s features by embedding the features in a lower dimensional space whilst optimizing an objective function (e.g. logistic loss). Examples usually help when math words fail:</p><p>Imagine a feature of our products is a binary variable saying whether or not the item of clothing is a pair of jeans. The latent factor model would learn some vector, i.e. a series of numbers with direction: [0.2, 0.9, -1.4, …, 0.1] that represents this feature. The numbers inside the vector are free weights that are <em>learned</em> during the optimization process. Because these vector representations all exist in the same vector space, we can use the vectors to explore the relationship between features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*5QWbDiGV3jNVvcAlI1Rpag.png" /><figcaption>Example of feature vectors in a 2-dimensional latent space. Many jeans have frayed cuffs, so these vectors may lie close to each other as opposed to a “top” feature vector.</figcaption></figure><h3>From matrix factorization to factorization machines</h3><p>The usual place to start with latent factor models is with <a href="https://en.wikipedia.org/wiki/Matrix_decomposition">matrix</a> <a href="http://dl.acm.org/citation.cfm?id=1608614">factorization</a>. Typically, in this paradigm, one learns a single vector representation for each unique product ID and customer ID. With these vectors, you can then use a similarity measure (typically cosine similarity) to look at similarities between products, between customers, and between customers and products. Chris Johnson has an excellent implementation of implicit matrix factorization on his <a href="https://github.com/MrChrisJohnson/implicit-mf">github</a>. I’ve also previously written in more detail about explicit matrix factorization on my personal <a href="http://blog.ethanrosenthal.com/2016/01/09/explicit-matrix-factorization-sgd-als/">blog</a>.</p><p>At Dia, we ask our customers to fill out a survey before ordering clothing so that we get to know them better and are able to delight them with hand-picked clothing to match their style. We also have rich information about our clothing through our detailed taxonomy and tagging systems. Until recently, it was difficult to include this extra customer and product information (often called side information) into a latent factor model.</p><p><a href="https://github.com/lyst/lightfm">LightFM</a> is a python library from the data scientists at <a href="https://www.lyst.com/">Lyst</a>, and it incorporates this extra information in a fast and scalable manner (i.e. Cython + parallelization). Vectors are learned for the side information, and this allows one to look at relationships between features, products, and customers. The LightFM algorithm approximates products and customers as the sum of all their respective feature vectors. The prediction of the algorithm is the dot product of the summed product and customer vectors.</p><p>One can take things a step further and employ <a href="http://ieeexplore.ieee.org/document/5694074/">factorization machines</a>. This is a similar class of model but one that makes no assumptions about how feature vectors should be combined. Factorization machines learn vectors for every feature, and they predict via the dot product of every combination of features (and employ nice math tricks to make this computationally feasible). This ability to look at all possible interactions of features embedded in the same vector space make factorization machines extremely performant at modeling complex data. The Python libraries <a href="http://contrib.scikit-learn.org/polylearn/">polylearn</a> and <a href="http://ibayer.github.io/fastFM/">fastFM</a> both provide scikit-learn compatible APIs for training and using factorization machines.</p><h3>Latent factors at Dia&amp;Co</h3><p>We’ve learned these latent factor representations for both our customer and product data by training latent factor models on our historical data. While one should always pick a metric for optimization and hyperparameter tuning of machine learning models, it’s especially fun to play with trained recommendation systems to make sure that they <em>make sense</em>.</p><p>Let’s walk through an example. Say we have a matrix of products and their features called prod_features with shape (num_products, num_features). Each row is a product, and each column is a feature (like is_a_jean). We also have another <em>learned</em> matrix of latent factors for these features called prod_feature_factors with shape (num_features, num_factors).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*olPzEfjXCNA3HbniWWoX0Q.png" /><figcaption>We have a labeled matrix prod_features on the left and a <em>learned matrix prod_feature_factors on the right.</em></figcaption></figure><p>We could go the LightFM route and approximate a product as the sum of its features’ latent factors to create single vector representations for each product. To do this in Python and numpy, we simply use matrix multiplication.</p><pre>prod_representation = prod_features.dot(prod_feature_factors)</pre><p>We’ll also assume that we have paired dictionaries which map the column indices of product_features to the names of our features and vice versa as well as the row indices of our products to the products’ names and vice versa.</p><pre>print(prod_feat_dict[&#39;is_a_jean&#39;])<br># 0<br>print(reverse_prod_feat_dict[0])<br># is_a_jean<br>print(&#39;prod_index_dict[&#39;Product Name 1&#39;])<br># 42<br>print(&#39;reverse_prod_index_dict[42])<br># Product Name 1</pre><p>We can now interrogate our model by asking questions. For example, we ask our customers which types of tops they like. Using latent factors, we can look at the top products for customers who checked off that they like sleeveless tops. Assuming we have analogous customer information to our product information, this code looks like</p><pre>def cosine_similarity(mat1, mat2):<br>    sim = mat1.dot(mat2)<br>    norms = np.array([np.sqrt(np.diagonal(sim))])<br>    sim /= np.linalg.norm(mat2, axis=1)[:, np.newaxis]<br>    sim /= np.linalg.norm(mat1, axis=1)[:, np.newaxis]<br>    return sim</pre><pre>sleeveless_idx = cust_feat_dict[&#39;loves_sleeveless_tops&#39;]<br>sleeveless_vec = cust_feat_factors[sleeveless_idx][:, np.newaxis]<br>sims = cosine_similarity(sleeveless_vec, prod_representation)</pre><pre>recs = [reverse_prod_index_dict[i] for i in np.argsort(-sims)]</pre><p>Unsurprisingly, the top products for customers who like sleeveless tops are tank tops:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*3iESlLvrtXheamdHVmj3-g.jpeg" /></figure><p>Even though this is the result that we should expect, this is important to see that (1) our survey is collecting true and relevant information and (2) our algorithm is learning accordingly.</p><p>We can also ask more nuanced questions. Let’s say a customer loves the last maxi dress that we sent them (an example of which is shown on the left below) but we do not want to send them another dress. We can search for the clothing that is most similar to maxi dresses and then filter out any dress results. We find mostly wide-legged trousers which hit the lower body similarly to maxi dresses but are an entirely different category of clothing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*e36etLwj7rWSbuBpKGsiKw.jpeg" /></figure><p>As you can see, by combining our detailed customer and product information with the power of well-trained latent factors, our ability to surprise and delight our customers with personalized clothing is simply limited by our creativity in asking questions of our data.</p><p><em>If you are interested in working on problems like these or would like to learn about the myriad of other technology solutions that we are building, then visit our </em><a href="https://www.dia.co/careers"><em>careers page</em></a><em> and join us!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fca7f58f53ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dia-co-engineering/embedding-everything-for-anything2anything-recommendations-fca7f58f53ff">Embedding Everything for Anything2Anything Recommendations</a> was originally published in <a href="https://medium.com/dia-co-engineering">Making Dia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>