<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by kenneth kuhn on Medium]]></title>
        <description><![CDATA[Stories by kenneth kuhn on Medium]]></description>
        <link>https://medium.com/@kenneth.kuhn?source=rss-86e0b6401914------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vAkgz5PG63Qd9W01uy3FMw@2x.jpeg</url>
            <title>Stories by kenneth kuhn on Medium</title>
            <link>https://medium.com/@kenneth.kuhn?source=rss-86e0b6401914------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:47:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kenneth.kuhn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Science Stakeholder Management]]></title>
            <link>https://medium.com/@kenneth.kuhn/data-science-stakeholder-management-19500d7754c7?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/19500d7754c7</guid>
            <category><![CDATA[backlog]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[stakeholder-management]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 00:35:04 GMT</pubDate>
            <atom:updated>2023-07-20T00:35:04.860Z</atom:updated>
            <content:encoded><![CDATA[<p>I previously wrote about two Data Scientist pain points: <a href="https://medium.com/@kenneth.kuhn/the-thrash-4783e653f9dc">the thrash</a>, when you’re not sure what to work on next, and <a href="https://medium.com/@kenneth.kuhn/category-ii-toxicity-9f6129ee4827">the slightly toxic coworker</a>. But my slack messages and 1:1 meetings are full of complaints that are not covered by either of these scenarios.</p><p>My coworkers and I complain about challenging interactions with stakeholders. Requests and requirements that are difficult to accommodate or that just don’t make sense. Meetings. So many meetings. Awkward conversations. In meetings, via emails, or in instant messages. Incorrect interpretations of Data Science work. So this post is about all of that.</p><p>I’ll try to be helpful and add notes on the best ways I’ve found to deal with certain issues. I write from the perspective of a Data Scientist who has mainly worked on applications of Optimization (math programming), but will try to generalize.</p><h4>Request Management</h4><p>Probably the main complaint I have heard from coworkers past and present is that stakeholders are asking for too much in too little time.</p><p>“They ask for a different analysis every day and each time they want it done by the end of the day.”</p><p>“How long do they think it takes me to run the code?”</p><p>“They are asking for me to be in another recurring meeting. How will I have time to do the <em>real work</em>?”</p><p>“There is no time to understand the data, no time for research. We are accumulating massive amounts of tech debt.”</p><p>“This is not what I came here to work on. Why would the company want <em>ME</em> to work on <em>THIS</em>?”</p><p>“There are 12 business people asking for stuff at the same time. We have a 4 person Data Scientist team. And Liz just got here and Jim doesn’t know SQL. &amp;?*#!”</p><p>Documentation and active management can help. Make a backlog for your team or even for yourself if no one else has made one. Refer stakeholders to the backlog when they make a request. Get your Data Science or Project Manager involved if you feel overwhelmed. They can make and manage the backlog. They can be the guardian of your time, the filterer of requests. Have efficient meetings with a limited set of invitees and an agenda set ahead of time. Request changes and even turn down meeting requests if you find yourself in too many inefficient meetings. It is okay to say no.</p><h4>In Production</h4><p>There is often a particular crunch when Data Science models are put into production and users first start seeing the results. A small expert team can build and <em>productionalize</em> a model that can help a much larger number of end users. But the small team may not be able to respond to all the feedback generated during model use.</p><p>Train a larger team to respond to user feedback. This team doesn’t necessarily have to be experts in Machine Learning or Optimization. I would recommend that the team who built the original model develop a playbook for responding to ad-hoc requests.</p><p>I have found that the majority of <em>interesting</em> results generated by Optimization algorithms are generated by <em>interesting</em> features of input data. This is true in Machine Learning too but Optimization seems especially … not robust. Automated input and output data validation checks can help. Set these up as soon as possible.</p><p>Also consider having a team dedicated to researching model enhancement. Especially for models at the core of business operations. I would argue that the research-focused team DOES necessarily have to be experts. I’ve been a Data Scientist long enough to see models maimed or replaced by less elegant or even less functional models after the original project team turns over.</p><h4>Requirements Management</h4><p>Another of the main complaints I hear from other Data Scientists is that the requests or requirements defined by the product manager or business don’t make sense.</p><p>Typically the concern is technical in nature. <em>They</em> are asking for something that cannot be done in the way that <em>they</em> are talking about doing <em>it</em>. This is especially problematic when the requirements are not just high level guidance but include low level details about how the Data Science model should work.</p><p>Sometimes I also hear from Data Scientists with domain expertise who question stakeholder logic. This is me on transportation and logistics projects. Tech does a pretty poor job of recognizing the domain expertise of, ahem, more seasoned Data Scientists. Sometimes more seasoned Data Scientists have a difficult time adjusting to the role of a Data Scientist. Startup veterans can struggle with just how little they oversee at a large company.</p><p>One variation on this theme is when the high level requirements are missing. <em>They</em> are asking for a system to help them make this decision but I have no idea how they want me to make that decision.</p><p>In the worst cases, the high level requirements are missing but the technical details are specified. Welcome to government work! I kid.</p><p>“They want me to model price elasticity but we have only ever charged customers one set price.”</p><p>“They want me to apply my model from the other project here but we have millions of unique customers and my model can’t support that scale.”</p><p>“They want me to optimize our assortment in order to maximize sales while ignoring any operational constraints. The solution is to put all possible products in the assortment. Do I just build an algorithm for that?”</p><p>There are variations on this complaint that involve objectives which conflict with one another or with the objectives of what some other project or team is doing.</p><p>The situation is more tense if the requirements change or these types of complaints surface midway through a project.</p><p>This is especially true on Optimization projects. Changes to objectives or constraints can break problem structure with dramatic implications for runtimes. Solution algorithms or data pipelines may need to be rebuilt.</p><p>It can help to get involved in planning discussions early and to be vocal. Let the broader team know if anything sounds especially challenging or unrealistic from a Data Science perspective. Make sure your position is known. Bring your concerns up any chance you get. Talk to other Data Scientists to make sure your concerns are valid. At the same time, commit to trying to make the project work regardless of what the broader team agrees on. Oh no, am I just paraphrasing crap Amazon leadership principles now?</p><p>It can also help to show literature or results that back up your positions. The vast majority of Data Science projects involve building models similar to models which have been built before, at the very least in academic papers on toy examples.</p><h4>Research in the Real World</h4><p>The challenges described above are complicated by the need to do research. It’s often not obvious which requirements will or will not pose particular problems. Business and operations staff are used to hashing out issues during relatively quick and intense meetings. But Data Science issues often cannot be solved, or even identified, like this.</p><p>Stakeholders will want to be involved at all times and to track progress continuously. But there are often lengthy periods where the team needs to run experiments that are only interesting from a Data Science perspective or which end up being a dead end. Or to build up the code structure. Or to collect and wrangle the data, etc. This is especially true towards the start of a project, roughly between the <em>planning</em> and <em>productionalization</em> phases.</p><p>My advice here is to get business results in front of stakeholders as soon as possible. Even if you aren’t confident in those results. Do not show stakeholders the mathematical programming formulation. Do not discuss the <em>metaparameters</em> of your Machine Learning pipeline. The discussion will lose focus. You may be cajoled into making technical changes that do not make sense technically.</p><p>It’s easy for me to say this sort of thing but difficult to get those early results fast enough. If you are struggling, I’d go back to my advice above to lean on your Data Science manager. Communicate a lot during the planning phase before the research and infrastructure work. Ramp up communications as soon as the exclusively Data Science work is done and re-invest in stakeholder relationships.</p><h4>Confidence and Misinterpretation</h4><p>My advice about early results runs somewhat contrary to lots of advice you’ll see about Optimization projects specifically. If you show results to stakeholders too soon, they may misinterpret the results or see something off and lose confidence in your work.</p><p>These are totally valid concerns, fitting in nicely with the last of the common complaint archetypes.</p><p>“the business saw one result that they didn’t like and spent the whole meeting lecturing me about how the business operates”</p><p>“Jan from the ops team kept referring to the output as probability of conversion but that’s not what I am modeling at all”</p><p>I would say that the concerns will be worse if you show models in place of results or if you wait too long to show results. The results should be universally understood. If there are inconsistencies in how people view the results, it’s best to uncover these inconsistencies early. If there are troubling results, it’s best to uncover those troubling results early. A healthy Data Science environment will allow for, and even encourage, model revision.</p><p>It helps to neither be overly confident nor overly feeble when presenting results. Business and operations staff, in particular, will know details of how your company works better than you. Don’t assume that their concerns are trivial. But also don’t agree to change everything because of one result that one stakeholder didn’t anticipate.</p><p>Talk with your tech lead or manager when unsure what to do next. Consider escalation. Your Director may not realize that your model has become famous on the Ops team and that they are attempting to use the results 12 different ways. But don’t escalate unless you’ve tried speaking with your stakeholders first and are seriously convinced that the project setup is wrong or concerned that a mistake is about to be made. Pick your battles.</p><p>Consulting taught me not to take comments and questions at face value. Try to understand why a stakeholder is making a particular comment. What experiences have led them to speak up here, now? What larger point are they trying to make? How can I address the larger point, moving forward?</p><p>Was that helpful? Obvious? It’s hard to be interesting without saying anything controversial. Especially on a dry, somewhat negative topic. I’ll try to have something more upbeat and more interesting next time. Special thanks to Ethan and Alice for helping me think through some of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19500d7754c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bard, Baudrillard, and the RTO]]></title>
            <link>https://medium.com/@kenneth.kuhn/bard-baudrillard-and-the-rto-5c98862222a6?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c98862222a6</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[the-matrix]]></category>
            <category><![CDATA[baudrillard]]></category>
            <category><![CDATA[simulacra]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Wed, 01 Mar 2023 02:51:30 GMT</pubDate>
            <atom:updated>2023-03-01T03:24:02.798Z</atom:updated>
            <content:encoded><![CDATA[<p>I love a good fad and thinking about the future. I also work in an organization called Applied AI.</p><p>Naturally, I have been fascinated by the recent advances in generative AI. This feels like the most interesting thing to happen in tech in awhile.</p><p>Generative AI immediately makes me think of Baudrillard’s Simulacra and Simulation. We are making rapid advances in our ability to (mass) produce synthetic content. Such content is already supplanting human-generated content. The nature of content will change. I don’t pretend to know how.</p><p>My employer has recently begun the dreaded Return To Office (RTO). The policy and the office itself also make me think of Baudrillard. The nature of my work will change, again, much to my chagrin.</p><h4>Simulacra</h4><p>Baudrillard talks about four stages of development of simulacra. First there are images that are easily recognizable as stand ins for something real. Next we begin to use images to alter how people understand reality. The relationship between image and reality breaks, and at this point images no longer need to refer to something specific and real. Finally, we reach pure simulacra where any given simulacrum has no relationship with the real. We are now in the era of hyperreality, where we base our lives on simulations without relationship to the real.</p><p>The example Baudrillard used that stuck in my head was the American chain restaurant styled as an Irish Pub. At the time that Simulacra and Simulation was published, such establishments had started to appear in Ireland for the first time. There are four-leaf clovers on the wall, lots of wooden furniture, dim lights, Guinness on tap, and shepherd’s pie and fish and chips on the menu. The pubs are not trying to be similar to any specific real place. They do make the public feel welcome and comfortable having drinks and overpriced, bland food. These places are easily recognizable and interpretable. Maybe slightly more so in the 80’s and 90’s than today. Great, now I feel like going to <a href="https://www.yelp.com/biz/stephens-green-mountain-view-2">Stephen’s Green</a>.</p><p>Baudrillard’s theory has been influential in postmodern thought, and it has been used to critique the way that our culture produces and consumes meaning. It suggests that we need to be more aware of the ways that simulations and copies have replaced reality, and that we need to be more critical of the images and symbols that we encounter in our daily lives.</p><p>Apologies for butchering that quick intro to Simulacra and Simulation. I relied on hazy, 25 year-old memories from when I was a student, <a href="https://en.wikipedia.org/wiki/Simulacra_and_Simulation">wikipedia</a>, a few first-page google links, and, of course, <a href="https://chat.openai.com/chat">ChatGPT</a>.</p><h4>ChatGPT</h4><p>It won’t surprise many of you to learn that the first draft of much of this post was written by ChatGPT.</p><p>ChatGPT is an AI language model developed by OpenAI that uses natural language processing to understand and respond to user input. It is designed to simulate human-like conversation and provide helpful responses to a wide range of questions and topics.</p><p>ChatGPT is built on the GPT (Generative Pre-trained Transformer) architecture, which is a type of neural network that has been trained on large amounts of text data. This training allows ChatGPT to generate responses that are contextually relevant and grammatically correct.</p><p><a href="https://www.forbes.com/sites/roberthart/2023/02/23/chatgpts-biggest-competition-here-are-the-companies-working-on-rival-ai-chatbots/?sh=3a64d607216b">Competitors to ChatGPT</a> include Google’s Bard and Baidu’s Ernie. These services have been generating a ton of buzz. ChatGPT has become “<a href="https://www.theguardian.com/technology/2023/feb/02/chatgpt-100-million-users-open-ai-fastest-growing-app">the fastest-growing consumer internet app ever</a>.” We are discussing how to use Bard at work.</p><p>Chatbots are but one form of generative AI. <a href="https://www.craiyon.com/">Craiyon</a> generates images from text prompts. <a href="https://www.synthesia.io/">synthesia</a> generates videos from text prompts.</p><h4>Hyperreality</h4><p>As ChatGPT bragged to me, the service has reached the point where it can be difficult to distinguish between the responses generated by ChatGPT and those produced by a human.</p><p>Huge volumes of chatbot-generated text will soon be published on the internet, if they haven’t already. This mass production of the simulation will degrade the real. It is already commonplace for motivated actors to use chatbot text to shape our discourse.</p><p>We are moving towards the hyperreal.</p><p>The next step in our Baudrillardian voyage will be text, images, and videos that no longer resemble the creations of a human today. 01001000 01100101 01101100 01101100 01101111 00100001. I have no idea how this transformation happens or what the result will be. I don’t think that I have made a big leap in reasoning and yet I haven’t seen others write about this.</p><p>I have seen plenty of people write about how generative AI will put people out of work. This point of view seems historically illiterate to me. There will be so many opportunities for people to manage generative AI systems, predominantly in ways that don’t require a deep understanding of AI technologies and theory. There will also <strong>always</strong> be other productive ways to use human labor and thought. Disruption? Sure. Mass unemployment? I doubt it. But, yea, if I were a content creator, I’d start playing with generative AI.</p><h4>The Office</h4><p>The big news at work recently has been the closing of certain offices and the <a href="https://www.wsj.com/articles/walmart-to-close-three-tech-hubs-tells-tech-staff-to-return-to-offices-3fb131e4">upcoming Return To Office</a>. I have thoughts. You know I can’t resist drawing a weird connection or two.</p><p>Working from home has, in some respects, felt like being part of a simulation. For the last 3 years, almost all of my work communications have been via Zoom video chat and Slack instant messages. For the last 2 years, my colleagues and I have mostly kept our cameras off. It felt awkward and impersonal at first but now I wouldn’t go back. Well, I wouldn’t go back by choice, but lately it feels like the world is going back, ha! The cameras off informal policy lowered my stress levels and increased my productivity without really impacting social cohesion. But I have to say, I can imagine a chatbot connected to a voice synthesizer doing my job.</p><p>A chatbot can’t Return To Office. At least not yet.</p><p>The office is, of course, itself a simulation, a particularly vexing Irish pub. The executive class claim that RTO improves productivity but I haven’t seen any real evidence to support the claim. It definitely improves the appearance of work.</p><p>We all have stories we tell ourselves about ourselves and one of mine is that I’m good at the work but not good at generating the appearance of work. My Spanish teacher was shocked that I was a National Merit Finalist. I have gotten some of my best performance reviews while working from home and I don’t think it’s a coincidence. I am the best chatbot. I wonder how many other people feel something similar.</p><p>Tech workers skew young. Many of my colleagues have never worked in an office. They are also part of Gen Z, which famously <a href="https://ripplematch.com/insights/negative-stereotypes-about-gen-z-in-the-workplace-a2a02995/">struggles with face-to-face interactions</a>. It will be fascinating to see how they handle RTO.</p><p>Personally, I’m going to start working on an AI that can generate the appearance of being a good office worker.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c98862222a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maturity Levels of OR Projects]]></title>
            <link>https://medium.com/@kenneth.kuhn/maturity-levels-of-or-projects-78d25ed2123f?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/78d25ed2123f</guid>
            <category><![CDATA[optimization-algorithms]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-programming]]></category>
            <category><![CDATA[operations-research]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Sat, 30 Apr 2022 02:01:49 GMT</pubDate>
            <atom:updated>2022-04-30T02:01:49.218Z</atom:updated>
            <content:encoded><![CDATA[<p>We recently went through a re-org in my organization. I am now <em>horizontal</em>, responsible for raising the bar on all things Operations Research in our corner of the world. This is a pretty ideal role for me. I get to learn a lot, focus on math, and act as a force multiplier.</p><p>I previously worked for more than six years as an Operations Researcher at the RAND Corporation. My new role reminds me of that job. Somewhat disappointingly, there were no <a href="https://www.youtube.com/watch?v=pnohc2AnQtw">reverse vampires</a> or “<a href="https://www.youtube.com/watch?v=HubxWPXK3jQ">run[ning] the world in a Malthusian manner</a>.” Investigative journalism for an outlet that emphasizes not having a point of view and has a low readership would be a more apt put-down. I digress.</p><p>What does it mean to raise the bar on OR work? My <a href="https://medium.com/@kenneth.kuhn/the-magic-of-operations-research-3f39e7513b85">previous post</a> was a description of, in my humble opinion, a successful effort. But can we generalize? I have seen a number of Medium posts on <em>maturity levels</em> in <a href="https://medium.com/vespa/the-big-data-maturity-levels-8b61875032cc">Data</a> <a href="https://odsc.medium.com/the-continuing-path-of-data-science-maturity-fe500e526888">Science</a> and <a href="https://towardsdatascience.com/the-four-maturity-levels-of-ml-production-systems-94c4468e6b3">Machine</a> <a href="https://medium.com/slalom-data-ai/mlops-part-1-assessing-machine-learning-maturity-88e9cb05eca9">Learning</a>. Let’s define maturity levels for OR projects, starting with the basics but also mentioning features I rarely see in practice.</p><h4>The Basics</h4><p>The core of an OR project is a specific type of mathematical model (which I <a href="https://medium.com/@kenneth.kuhn/thoughts-on-the-vehicle-routing-problem-1-of-2-68086db61959">went over previously</a>). An OR project should have obvious objective function(s) being maximized or minimized. The objective function(s) should be consistent with business goals. The objective function(s) should also be closed-form expressions of decision variables and model parameters.</p><p>Are there any unnecessary variables or model parameters in the objective function(s)? A lot of times I see complicated expressions of input data which could be computed prior to model formulation and solution. It’s not always necessary to simplify things but it can help us to focus on the core mathematical problem and is a sign of maturity with respect to OR.</p><p>Was the objective function defined in a way that makes it amenable to optimization, for example a linear function of decision variables?</p><p>Does the project have defined constraints? Are these also consistent with what business wants or expects? Are they closed-form expressions, inequalities involving decision variables that appear in or otherwise affect the objective function(s)?</p><p>Are the constraints as simple and amenable to optimization as they can be? Could any of the constraints be merged? Could any be eliminated without affecting the optimal solution? This is surprisingly common. An example might involve a constraint that places a lower bound on a weighted summation of terms that the objective is trying to maximize. Do any constraints boil down to just fixing the values of certain decision variables? If so, it would probably be better to just fix those variables directly.</p><p>Is the overall formulation a canonical optimization problem? It typically isn’t, but I’ve noticed that mature OR projects are more likely to be based on solving a canonical problem. There are often known, high quality solution algorithms to such problems. It takes an OR expert to recognize that the project boils down to, for example, the set packing problem.</p><p>Is it clear how the overall formulation is optimized? Which commercial solver or custom algorithm is used? What are the stopping conditions for the solver / algorithm? What happens when an optimal solution has not been found after some stopping condition is met?</p><p>When I arrived at Walmart, there were multiple projects where people had defined reasonable looking objective functions but never mentioned constraints. It was interesting; they were using (some of) the language of OR but never got as far as having an actual formulation.</p><p>Some projects have objective functions, or less frequently constraints, that call on other functions. An example that seems to be getting more common is the objective function that is the output of a Machine Learning model. These projects can involve relatively sophisticated customized solution algorithms. They almost have to since you can’t pass most commercial solvers a formulation containing black box calculations. I would still classify such projects as relatively immature in terms of OR. There was likely little thought given to how the mathematical form of the ML model affects the runtimes or even the solutions of the optimization.</p><p>What are the differences, if any, between the objective functions and the metrics being used by business and managers to evaluate outputs? They don’t have to be the same. In fact, we should be cynical of results showing that an optimization-based approach performs better than an alternative algorithm on a measure that the optimization-based approach maximizes or minimizes. Of course it does! On the other hand, differences between objective functions and metrics that people care about can be a sign that the formulation needs to be refactored.</p><h4>Documentation and Project Management</h4><p>I am a firm believer in documentation. This is the area where commercial projects are least likely to be mature by my standards. It may not seem like a rational choice for a startup to invest developer time in documentation. But documentation can speed up and improve collaboration.</p><p>One of the best things we did early on in my <a href="https://medium.com/@kenneth.kuhn/the-magic-of-operations-research-3f39e7513b85">prior project</a> was to document data input and outputs requirements. Expectations were set and then met. This is probably the most important form of documentation to have.</p><p>Meaningful improvement on OR projects often comes from changes to the formulations at the heart of the projects. It is much easier to consider such changes if formulations are properly documented. A mature OR project will have its formulation(s) including objective function(s), constraints, decision variables, and model parameters described in a text document. The document will be available to everyone on the project and to teams working with the project team. The document will include both the closed-form mathematical expressions mentioned above and text explaining them.</p><p>Going a step further, it would be a good sign if some document(s) have been presented, reviewed, and published outside of the project team. Ideally in a scientific outlet. None of this is necessary, but it does demonstrate a level of maturity on a project. Doing so ensures that outside reviewers have looked at the details of a project and provided feedback. Uber <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.21872">surge</a> <a href="https://faculty.fuqua.duke.edu/ioconference/papers/2017/Castillo,%20Juan%20Camilo_Surge_07_17.pdf">pricing</a> is the best example I’ve seen here.</p><p>The documentation should motivate design decisions. This is particularly important if a custom solution algorithm was developed. Why was this type of algorithm developed? What are some alternatives? Why are they less suited to the problem at hand? If the algorithm is a heuristic, is there any proof that it would not be practical to solve the problem to optimality? Most published scientific papers in OR fail to answer some of these questions, in my humble opinion.</p><p>Time spent doing basic documentation quickly pays for itself. For example, onboarding is much slower and less effective without documentation. Managers could still not prioritize documentation if their incentives aren’t set properly. Developers could still avoid or do a terrible job at documentation, again especially if incentives aren’t set properly. Project management can help.</p><p>It is the role of the project manager to ensure that there is time set aside to document. A mature OR project will further have project management practices that account for investigation of interesting results and debugging as needed.</p><p>Ideally, there would also be time, space, and the right staff set aside to perform research on potential improvements to the formulation, code, and solution algorithms. These are all things that are difficult to plan for in the poorly managed “agile” processes that have infested data science. But without them, project methodologies become outdated and code bases become unwieldy. It was understood at Uber that there would be continual research into the matching algorithm(s).</p><h4>The Code, Automation, Etc.</h4><p>The most important part of a tech project is of course the code. The code should be efficient and relatively easy to maintain. It is particularly important on an OR project to separate the sections of the code responsible for:</p><ul><li>collection of data inputs,</li><li>model representation and storage,</li><li>model solution, and</li><li>post-processing and delivery of outputs.</li></ul><p>There can be benefits to using different methods, languages, and/or developers for these various parts of an OR project.</p><p>Let’s focus on the modeling parts of the project. Transparency should be the highest priority here, in the parts of the code that are typically the most difficult to follow. Someone familiar with the mathematical modeling language or library used should be able to quickly understand the formulation from the code.</p><p>I now use the pyomo library in python, use separate functions to define each objective function and constraint, and have separate files defining the model parameters, decision variables, objective function(s), and constraint sets. This seems to help.</p><p>One nice thing about pyomo as a modeling language is that it’s simple to change the solver. The importance of being able to swap solvers was brought up by a colleague at Walmart. I now take it as a sign of a mature OR project.</p><p>Articles on the maturity of ML projects often focus on automation. We can apply the same lens to OR projects. Is there an automated pipeline that goes through the various parts of the OR code mentioned above in sequence? Are there policies and processes in place to run the pipeline? Are inputs and outputs automatically tested? All of these would be signs of a mature OR project, although not all are necessary for all OR projects.</p><p>That’s all I can think of at the moment. Did I miss anything? Let me know.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*chkIq7AC2zSGnZv4kgIFUA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78d25ed2123f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Magic of Operations Research]]></title>
            <link>https://medium.com/@kenneth.kuhn/the-magic-of-operations-research-3f39e7513b85?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f39e7513b85</guid>
            <category><![CDATA[operations-research]]></category>
            <category><![CDATA[assortment-planning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[assortment-optimization]]></category>
            <category><![CDATA[logistics]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Wed, 26 Jan 2022 06:25:13 GMT</pubDate>
            <atom:updated>2022-01-26T06:25:13.688Z</atom:updated>
            <content:encoded><![CDATA[<p>I was recently asked to speak for 2 to 3 minutes at our All Hands meeting, to summarize progress on my main project over the past year. I looked up some metrics and made some quick comparisons. The results are … interesting.</p><h4>The Comparison</h4><p>I work on <a href="https://en.wikipedia.org/wiki/Retail_assortment_strategies#Assortment_optimization">Assortment Optimization</a>. We are developing a tool for internal use.</p><p>One year ago, I had just joined a team of 40 working on the data science bits of the backend. 10 of us worked specifically on the optimization piece. It took over 10 hours to run the main optimization script for one relatively small category. Much of the work involved repeated manual re-runs, exporting of Excel sheets, and discussions with business experts. We adjusted parameters and tried to hit certain targets. This was necessary before we could present category results to users. 4 categories were marked as “passed” as of 28 January, 2021. The script applied a set of heuristics sequentially, but it was essentially a greedy algorithm. Our part of the code was tens of thousands of lines of R code spread across a few hundred files. The team that had developed the code was cycling off the project. They were documenting all that they had done. A bunch of new features were being rolled out in a hurry and the code was breaking. It was fast becoming obvious that the few new hires who were staying wouldn’t be able to maintain the code. We were in crisis mode. I was wondering what I had signed up for.</p><p>Let’s fast forward to the present. We recently finished a run of 87 large categories in 15 minutes. We are going to try to optimize 500 categories simultaneously next. Business validations are largely automated. Many are what I’d call “automatic.” We are able to make performance guarantees relative to optimality and/or to bound computation times. We are accounting for the same “complexities” of the problem, like demand transference, operational requirements at distribution centers, regulatory and contractual requirements, etc. We do this in maybe one thousand lines of Python code in tens of files. There are about 13 people total working on this project at any given time. 3 work specifically on the optimization piece. It was me, a recent MS grad, and an MLE in India for most of the year. I actually felt good enough about the state of the project to step away and work on a sister project recently.</p><p>Over the course of the year, lots of talented engineers left the project. This includes both the entire team that built the original code and almost everyone else who I worked with one year ago.</p><p>The 5 or so people who focused on Project and Engineering Management left. A year ago, Jira tickets were detailed and people moved them across various boards. We had sprint planning, retro, and cross-team demo and planning meetings. All were well attended and felt useful. None of that is true now. We haven’t had a PM on the science part of this project for about 10 months now. There have been herculean efforts by one Data Engineer and, more recently, a new hire Director to organize us and manage communications with the (slightly crazy) front end team. We still have the demo meetings, I think, although almost no one I know attends any more. They have been abandoned by most of us as being a waste of time but somehow I think they are still attended by the most senior among us. Hmm. In any event, I don’t think I’ll offend anyone (and I certainly don’t mean to) by saying we are in a much worse position with regards to Project Management. Our documentation is poor.</p><p>And yet… I think we are in a much better place than we were a year ago. I think the top level stats reflect that.</p><p>I hope I’m not offending anyone here. I never tried too hard to figure out what went wrong in the development process previously. I have no interest in laying blame anywhere. The engineers and project managers I interacted with were all excellent.</p><h4>The Magic</h4><p>One of the main reasons why we’re in a better place is simply the magic of Operations Research (OR).</p><p>People often compare <a href="https://cloud.google.com/blog/topics/developers-practitioners/life-changing-magic-making-ml">Machine Learning</a> (ML) to magic. (See also <a href="https://www.forbes.com/sites/kalevleetaru/2019/07/08/todays-deep-learning-is-like-magic-in-all-the-wrong-ways/?sh=3db6a8ca28fc">Deep Learning</a>, <a href="https://www.sapiens.org/column/machinations/ai-as-magic/">Artificial Intelligence</a>, <a href="https://www.zignite.io/post/gpt-3-the-ai-that-does-magic">Natural Language Processing</a>, etc.) You throw data at <a href="https://scikit-learn.org/stable/">sklearn</a> and out comes a model capable of predicting <a href="https://journals.sagepub.com/doi/abs/10.1177/0361198118782272">flight delays</a> remarkably well. The model itself demonstrates previously unknown details about how systems work. I haven’t seen many people compare OR to magic. But it can be. And this project is a good example.</p><p>I put a brief introduction to OR in a <a href="https://medium.com/@kenneth.kuhn/thoughts-on-the-vehicle-routing-problem-1-of-2-68086db61959">previous post</a>. You build a (simple) mathematical model, a “formulation,” consisting of an <strong>objective function</strong> and several <strong>constraints</strong>. You then develop an algorithm or ask a specialized service to <strong>solve </strong>the model, finding the values for the <strong>decision variables </strong>that maximize or minimize the objective function while meeting all of the constraints. State-of-the-art <strong>solvers </strong>are remarkably efficient. They can also be managed, for example to limit their run times or memory use while still guaranteeing a solution within x% of optimality. They do counter-intuitive things, focusing exclusively on the underlying math of the problem at hand.</p><p>I came up with the assortment optimization formulation we are using now. I swear this post isn’t just a humble (or not so humble) brag. The formulation is quite simple. (More on this later.) My team and I coded it up. I thought the benefits would be that we’d have a relatively good solution to the underlying assortment optimization problem relatively fast. And that it would all be thanks to the <a href="https://www.ibm.com/products/ilog-cplex-optimization-studio">CPLEX solver</a> being slightly smarter than whatever we could come up with. But the benefits ended up being so much more than that.</p><p>One benefit that became apparent early on was that our code was much simpler and easier to maintain than the code it was replacing. Everything was in python. Our initial “pseudocode,” developed over about 2 weeks, was put straight into production and basically just worked. We spent countless hours debugging unexpected results and refining the flow of our script in the following months and never really had to change the core optimization code. The mathematical modeling was done using <a href="http://www.pyomo.org/">pyomo</a> and even this part looked and still looks embarrassingly simple.</p><p>We worked with business to define what a good solution looks like, and then added the minimum necessary constraints to our formulation. Optimization forces you to put everything into a precise mathematical framework, which is great when requirements are otherwise missing. This is, for me, a not obvious benefit of OR.</p><p>A few data engineers on my team guided staff on other teams to build data pipelines to feed our optimization code. A good amount of the speed up we were able to achieve on the optimization part of this project stems from work being offloaded to the data pipelines. The data pipelines can be run offline, something the data engineers recognized well before I did. They did an amazing job. Here again, OR helped in a not obvious way by convincing us to define and use relatively small, precisely defined, structured data sets.</p><p>More recently, a Distinguished Data Scientist revamped the code using <a href="https://pypi.org/project/pyspark/">pyspark</a>. This resulted in another large speed up. It helped that in OR everything is, essentially, matrix manipulations. I could write a whole separate article on the magic of pyspark but I’m still learning and probably not the person to write that article.</p><p>So things are much better on the Assortment Optimization project now and, for me, a lot of it boils down to the magic of OR.</p><h4>Side Notes</h4><p>I have a few other thoughts about why things are better now.</p><p>Large software projects are… unwieldy. The costs and expectations are often too high. Some developers may get annoyed at being unable to shape the direction of the project. Others will take no ownership. This is particularly true if people are maintaining code that they didn’t develop. Solid project management helps but excess meetings waste developer time. Have too many meetings at different times on different days and the deep thought needed to tackle the most important issue becomes impossible. The amount of time we were wasting on Assortment Optimization meetings one year ago honestly should have been red flag number one.</p><p>It is so much easier to maintain and add features to a script that takes 10 minutes rather than 10 hours to run. That is obvious right? And yet I think people vastly underestimate the costs of additional runtime. Extra runtime wastes developer time and saps morale. If your script takes over, say, 40 minutes to run, then you’re in trouble. In this regard, OR is maybe not magic enough. I had a friend tell me recently that she switched from working on OR to ML because “OR doesn’t scale.” I know exactly what she means. Want to add an extra variable to your problem? Your runtimes may explode. And look out if you want to do something non-linear.</p><p>I personally think OR formulations should be simple. That helps with the scaling. The job of OR is to solve the simple but challenging combinatorial math puzzle at the heart of your business problem. Think some type of <a href="https://en.wikipedia.org/wiki/List_of_knapsack_problems">knapsack</a> or <a href="https://en.wikipedia.org/wiki/Vehicle_routing_problem">vehicle routing problem</a>. Your problem really doesn’t have to be more complicated than this. Trust me. If it’s good enough for Amazon and Walmart, it’s good enough for you. The OR practitioner fresh out of graduate school or with a background in ML will try to fit every business concern into the model. You really don’t need or want 17 objective functions and 43 constraint sets. You can interpret the results of your optimization in 27 different ways but don’t pollute the math.</p><p>In academia, I would often be adding an additional complication to an existing formulation. Make it more “realistic” and use complex math to show how it can be done. In industry, I am often simplifying some complex and broken formulation or analysis. Make it simpler and more reliable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f39e7513b85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Category II Toxicity]]></title>
            <link>https://medium.com/@kenneth.kuhn/category-ii-toxicity-9f6129ee4827?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f6129ee4827</guid>
            <category><![CDATA[work-culture-in-start-up]]></category>
            <category><![CDATA[work-culture]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[toxic-co-workers]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Sun, 14 Nov 2021 23:59:34 GMT</pubDate>
            <atom:updated>2021-11-23T15:43:33.181Z</atom:updated>
            <content:encoded><![CDATA[<p>My last post was about one of the bugbears of data scientists, <a href="https://medium.com/@kenneth.kuhn/the-thrash-4783e653f9dc">the thrash</a>. Another hobgoblin is the somewhat toxic manager, coworker, or workplace. There’s a lot of <a href="https://www.amazon.com/Assholes-Theory-Aaron-James/dp/0804171351">content</a> <a href="https://hbr.org/2020/06/times-up-for-toxic-workplaces">out there</a> <a href="https://www.washingtonpost.com/business/2021/09/30/during-great-resignation-workers-refuse-accept-unacceptable/">already</a> <a href="https://medium.com/@alexisazria23/red-flags-of-toxic-managers-companies-f91338c51687">about toxicity</a> in the workplace and what to do about it. I’ll try to point out what’s different in data science and offer my somewhat cynical take on things.</p><p>Toxicity is much more of an immediate and concrete problem than the thrash. The issue is software engineer Steve and how you can get through the team meeting without a) quitting on the spot, b) slashing Steve’s tires in the parking lot after work, or c) feeling awful and eating a pint of <a href="https://www.yelp.com/biz/afters-ice-cream-pasadena-2">frozen hot cocoa</a> ice cream when you get home. The Steve problem is easy to explain to coworkers, friends, and family. They will empathize. They will also quickly tire of the conversation. Talking about the issue can make you look weak or like a gossip. This is especially true if whoever you are talking to likes Steve or Steve’s work. Or if Steve is not actually all that toxic. It’s complicated, at least in data science.</p><p>Let me say up front that I don’t believe I’ve worked with any truly toxic people. I believe such people are rare in data science. My partner is an <a href="https://www.aota.org/about-occupational-therapy.aspx">Occupational Therapist (OT)</a>. All of the OTs I have met have been kind. There must be some toxic OTs but it’s honestly hard to picture that. I spent most of my professional career in academia and at a think tank. Toxicity is relatively common there. <a href="https://blogs.egu.eu/divisions/sm/2021/06/01/narcissistic-abuse-in-academia/">Narcissism</a> and all. Even so, I’m not sure I worked with any truly toxic people. There was a lot of awkwardness and bad management, sometimes coming from me. (Why is there no management training in research - or data science?) I did hear awful stories about professors or collaborators I barely knew. The stories generally focused on the PhD student — advisor relationship. Codependence and a strong power imbalance breed toxicity.</p><p>Do we have such conditions in data science? It’s rare. I have noticed that junior data scientists working directly with people much higher up in the org chart, often in the C-suite, are typically the ones who experience toxicity. Truly toxic people, to the extent they exist in the field, will target those with less power. Junior data scientists will also have less opportunity to fight or flee or ignore when faced with problematic situations. The role of a junior data scientist can be similar to that of a PhD student. The fact that there are so many people trying to be junior data scientists right now doesn’t help the situation. I would say to be choosy when taking your first data science job or two, but I know lots of people don’t have that luxury.</p><p>The junior data scientist is more likely to inadvertently say or do something that will trigger an outburst. The new hire might not know when to ask for help and when not to ask, when to wait and when to push ahead with analysis, and just <a href="https://medium.com/@kenneth.kuhn/how-data-scientists-are-evaluated-832806c5ec51">how to be thought of as a high performer</a>. He or she might not have the context to know what the company has tried 37 times before without success, or what sets software engineer Steve off. The new hire is also more likely to do bad data science. The new hire is less likely to have seen <em>aggressive</em> coworkers. Insert joke about sensitive millennials here but it was shocking to me, at age 39, when I got my first tech job and ran into my first aggressive coworkers. There are no adults in the room at many tech jobs.</p><p>I don’t believe I’ve worked with any truly toxic people, but I have worked with people who took it upon themselves to organize their coworkers and to critique the efforts of others particularly during team meetings. I did work at Uber, famous for corporate values such as <a href="https://www.hrcapitalist.com/2019/06/brotastic-vs-we-care-a-quick-review-of-ubers-current-and-past-corporate-values.html">toe-stepping and principled confrontation</a>. It probably sounds like I’m basing this article on Uber, but honestly it wasn’t that different at my other tech jobs.</p><p>All teams in tech are dysfunctional to one degree or another. There is lots of bad data science being done. Both can limit the effectiveness of otherwise solid contributors and teams. Both can be infuriating. Especially when deadlines are tight and expectations are high. Especially to those trying to climb the corporate ladder quickly. Especially to those who view themselves as teachers or leaders. This is what I’ve seen in data science. People who had good intentions but who would occasionally toe step… or worse. Many of these people have been, or would be, surprised to learn that they were called toxic. Many of these people were after a promotion, or three. They essentially all got those promotions. More on this later. If I sound like an apologist, keep reading.</p><p>There are aggressive people in other domains too, of course. I can remember being shocked at how intense and fierce the <a href="https://ieor.berkeley.edu/">Berkeley IEOR</a> professors were to visiting faculty giving talks. They really did not like fuzzy logic. This is apparently common at top technical departments. Were they toxic? Not sure I’d go that far. I will say that the very smartest and most accomplished professors I have interacted with, the kind that manage to play in the symphony orchestra when they’re not teaching, have also been amongst the kindest.</p><p>The difference in data science is proximity. You, especially if you are a new hire, will have to work closely with software engineer Steve. In the before times, this was literally true. Somehow it’s even worse now. Maybe because you don’t see all of Steve now, just the side of him where he’s yelling at you. The zoom call or slack ding can be anxiety inducing.</p><p>The (slightly) toxic manager or coworker(s) problem in data science disproportionately affects women and introverts. These groups are, generally speaking, less secure and confident in the workplace. There is surely a relationship to race, ethnicity, and national origin as well. I haven’t seen anything remotely relevant on the job, thankfully, but I know the relationship is there. <a href="https://www.espn.com/nba/story/_/id/32440987/phoenix-suns-robert-sarver-allegations-racism-misogyny">The Robert Sarver story</a>, a case study on toxicity, is really a story about race and gender. There is nothing special about data science in this regard, except maybe that the demographics are different.</p><p>The (slightly) toxic coworker problem is made worse by a feeling that the coworker is part of a powerful <a href="https://blog.axcethr.com/how-employers-can-handle-cliques-at-work">clique</a> maybe even including the manager. You can see how this would be worse for the marginalized.</p><p>People in a work clique may not think of themselves as part of a clique. I remember a coworker at Uber (who was by no means toxic) sitting somewhere new in the office for one day and realizing that the crew around where he normally sits looked and sounded like <a href="https://meangirls.fandom.com/wiki/The_Plastics">The Plastics</a>. Those of us on the outside knew this already. The open plan startup office is the high school cafeteria. Want to ensure a good work environment for all? It can help to shake up the seating chart or the team meetings. On the other hand, a solid work clique is exactly what’s needed to rapidly produce solid new features and products.</p><p>That last line is exactly why the manager might be a part of a clique or support the clique. We can also go back to what I wrote earlier, that in data science there are more toe steppers and less outright toxic people. The toe steppers are trying to pull their team up. Managers often see this and appreciate it. These toe steppers are often high performers or at least thought of as high performers. They have often been in the organization a long time and are high up in the org chart. So they interact with the managers frequently.</p><p>The toxic ones are also sure to make the managers, skip levels, and other teams know how much they contribute while playing down the contributions of others. This is one of the ways I distinguish the toxic from the merely “direct.” It’s also why it’s so important to track turnover stats, perform (and care about) exit interviews, and have processes for 360 degree and anonymous or “safe” feedback.</p><p>I’ve noticed that the aggressive and toxic are often the people who rise up the corporate ranks the fastest, with the most toxic rising the quickest. This is true in all fields. It disgusted me at first and I was really annoyed with my first few employers. Now I realize it’s pretty universal and I even sort of understand why. There is for sure a value to toe stepping sometimes, and you’ll have to do more of it the higher up the ladder you go. Also toe stepping is correlated with working hard.</p><p>How big of a problem is this in data science? I’m basing this post on roughly a dozen scenarios I’ve seen on the job, literally multiple people at each job. Some were actual problem cases; some just had different personalities or came from different cultures. If you think this post is about you, sure, take it as an opportunity to reflect. But honestly you are at most one data point and the truly toxic wouldn’t be concerned about this article.</p><p>The toe steppers I have worked with, even the ones that I think crossed some lines, really did pull teams up in the short term. There was a lack of organization. People weren’t taking initiative. Deadlines were being missed. Projects were limping towards irrelevance. Then software engineer Steve rallied the troops and suddenly the team was meeting deadlines. Steve represented the project well to management. There were real benefits.</p><p>On the other hand, I have seen situations where literally everyone on the project team was desperate to leave. The main negative effect, for the company, of toxicity is that it <a href="https://www.kaporcenter.org/tech-leavers/">leads to good people leavin</a>g projects, teams, and companies. Personally, I haven’t left a job because of toxicity. But I have found myself frequently consoling despondent colleagues. Honestly, this has happened at every job I’ve had even ones where it would be a huge stretch to call the people or environment toxic. I have felt compelled to confront Steve once or twice, and to back up the testimony of new hire Nathan more times than that. I’m not going to lie to protect Steve. This is outside of my comfort zone and takes up a decent amount of energy.</p><p>So what can be done? New hire Nathan typically wants to report Steve to HR or management. My personal take on this is that it’s not a great idea unless something really bad has happened and lots of people witnessed it. Nathan will be disappointed in the results. There’s a good chance management will “take Steve’s side.” This is particularly true if Steve is a toe stepper but not all that much more. What can management really do even if there is some evidence of toxicity?</p><p>One thing I routinely see is HR or management asking Nathan and Steve to have one or multiple chats. I’m no management expert but this strikes me as a terrible idea. This is sure to stress out Nathan. It may also stress out Steve, who probably doesn’t think that he is toxic and now has appointments with someone who has already complained about them to management. Maybe it causes Steve to make small behavioral changes. From what I’ve seen, these <a href="https://doi.apa.org/doiLanding?doi=10.1037%2Fpspp0000210">don’t last long</a>. They might help for the toe steppers who don’t realize when they are close to crossing boundaries. But a quick chat with management probably could have achieved the same result.</p><p>Shaking up team ceremonies or even staffing can help. Empowering new hires can help. One important point here is that managers should not only grease the squeaky wheel. Letting one or two complainers leave will just make things worse for those that stay. Reducing meetings can help.</p><p>Changing official corporate culture can help, in the long run. Screening for personality fit during the interview process can help. According to a former coworker, Stitch Fix doesn’t suffer brilliant jerks and weeds them out in the interview process. I didn’t work with any at Scoop, which emphasized corporate culture.</p><p>I mentioned 360 reviews and anonymous feedback before. Both are helpful for evaluating managers, tech leads, and team processes. Most of the organizations that I have worked at relied on “culture surveys” which include vanilla questions about job satisfaction. This isn’t what I’m talking about. The many people, like myself, who see but are not the target of slight toxicity have little reason or opportunity to remark on what they know is a potential problem. Much better to ask more specific questions about people (not just the manager!) or how interactions are structured within the team.</p><p>Sometimes organizations rely on exit surveys as well. I’ve heard many people downplay exit surveys. Disgruntled Debbie is already leaving; why would an organization make changes to satisfy her? Why would Debbie want to risk burning bridges or start a difficult conversation when she already knows she has escaped? These are good questions but in fact the people that I know who have left because of toxicity or perceived toxicity have been pretty up front about their feelings on the way out. They want the company and the people to know and to change. When I left RAND, I definitely noted what I thought was off in corporate culture in the hopes that it would be cleaned up to benefit future Kennys. I spent 6+ years at the place; I cared about it. I didn’t call out individual people, but none were truly toxic. I would bet that there is a lot of useful information that’s going unnoticed in exit interviews and attrition statistics. Companies should look at both more. The stats tell a story in and of themselves, if we’re brave enough to read it. Definitely not unique to data science.</p><p>In my experience, most of the toxicity or perceived toxicity in data science happens during informal but high profile team meetings where big shots are present and/or important business decisions are made. Steve belittles Jennifer, or her idea, so that he looks important and his idea carries the day. This is one reason why I’m a fan of <a href="https://www.cnbc.com/2018/04/23/what-jeff-bezos-learned-from-requiring-6-page-memos-at-amazon.html">document-based meetings</a> and decision making, although I’ve rarely seen them. We are supposed to be letting the data guide us so let’s focus on the evidence.</p><p>You might be thinking that you or your project or your organization don’t have the time for any of this. Ok but do you have the time to replace the people that leave because of the toxicity or perceived toxicity?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f6129ee4827" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Thrash]]></title>
            <link>https://medium.com/@kenneth.kuhn/the-thrash-4783e653f9dc?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/4783e653f9dc</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Sat, 09 Oct 2021 19:38:29 GMT</pubDate>
            <atom:updated>2021-10-09T19:42:24.606Z</atom:updated>
            <content:encoded><![CDATA[<p>My colleague called it <strong>the thrash</strong>. It is one of the worst parts of life as a Senior Data Scientist (DS). You don’t know what to work on next. The ideas from the Product Manager (PM) are missing, vague, or technically naive. The ideas on the Quarterly Business Plan (QBP), in the backlog, or being discussed by your coworkers don’t match your areas of expertise or don’t seem as if they could ever be all that impactful. They don’t spark joy. Maybe you have some ideas of your own but they aren’t aligning with what you are hearing from your Engineering Manager (EM), Tech Lead, Director, or PM.</p><p>When you are stuck in the thrash, you probably aren’t “producing” in the usual way. You aren’t committing code. You find it hard to explain what you have been working on during sprint planning meetings or at stand up. All of this may make you or your EM or PM feel as if you aren’t being productive enough. This is especially true if you have that backlog or those ideas from your PM.</p><p>I have a lot of contacts in academia and I imagine some of you are thinking: “Welcome to my world!” A good chunk of your work life involves thinking up what you or your students should work on next. Whatever you work on has to be something really novel. This can definitely be stressful but, at least in my brief time in academia, it never felt quite as bad as the DS thrash. This might be because idea exploration was the job. It might be because no one was tracking my work on a weekly basis, had some larger ambitions that required (frequent) contributions from me, or was in any real position of authority over me.</p><p>I did once work with an experimental scientist who really struggled with something like the thrash when doing public policy work. She was used to being constrained by the time and money required to perform experiments. She couldn’t grapple with the murkiness of the thrash. She felt disappointed that she couldn’t do research and publish as fast as her ambitions. She didn’t know what to put down for Budget Justification in grant applications. “Am I supposed to ask them to give me money and time just to think? And to fund my travel around the world meeting other people for dinner? How much is enough? How much is too much?”</p><p>I steered my experimental scientist friend towards organizations funding <a href="http://www.trb.org/Projects/RequestsforProposals.aspx">relatively well defined research projects to be completed during relatively well defined time windows</a>. When she submitted proposals, she found the grants were won by the people who had submitted the project ideas and, in some cases, already done the projects. But, more than anything else, the project ideas were “lame.” This is the price you pay for skipping the thrash.</p><p>You may not have to deal with the thrash if your employer is very organized or if you are working in a more “junior” capacity, regardless of actual job title. The price you pay may include having to work on lame projects. You will also encounter the younger brother of the thrash, the feeling of not knowing how to accomplish the specific tasks which have been assigned to you. This too can feel awful, particularly if people are asking you when you’ll be done with your tasks or pressuring you to finish them quickly. From my outsider perspective, it seems like being (and learning to be) a Software Engineer (SWE) involves constant battles with the <strong>micro-thrash</strong>. Maybe this is why burnout is common.</p><p>Let’s talk about some solutions.</p><p>Speaking with the Operations (Ops) or business people at your company can be helpful. They battle with very tangible issues that can often be mitigated with a little DS magic. If nothing else, maybe you could automate, partially automate, or improve some process that takes hours and hours of busy work now. Hopefully you can improve the logic at the same time. Just speaking with the Ops and business people, you will learn about parts of the business and the industry that you would otherwise not be exposed to.</p><p>You may come to appreciate the ideas of your PM or EM more, given this new context. Or you may find out that those ideas are as bad or worse than you thought. Hopefully you find some better ideas that do spark joy, for you, for those Ops and business people, and for everyone.</p><p>I find it funny when some companies want to route all communication through the PMs. Those poor PMs. Trying to herd all the SWE and DS cats. Translating the technical gibberish the SWE and DS cats produce when they can be bothered to document or show up for stand up into business english. Now also responsible for explaining the nuances of the business to those cats. All without any real authority. When there is this resource of the Ops people readily available. One of the real revelations for me at my first tech job at Uber was just how great the Ops people were (are?). If you want to really know about transportation, about your city, about microeconomics, … talk to Uber Ops. We are just starting to interact with “business” for my projects at Walmart, but I am getting a similar vibe from these people.</p><p>Speaking with people, in your organization or others, several levels higher in the org chart than you can also be helpful. The worst part of the thrash is feeling limited or constrained in what you can accomplish. You will probably feel as if you are smart and hard working enough to do great things for your employer, but you just… can’t. You might be tempted to blame your managers, but basically you don’t know how to apply your skills. That Senior Director or Vice President probably knows about a plethora of projects and project ideas that you don’t.</p><p>My colleague told me about the thrash when I was at Scoop. My brilliant colleagues had revamped “the matcher” in a matter of weeks. The matcher was the one piece of Math Programming at Scoop. I didn’t know what to work on next. My colleague wanted us to work on revamping geospatial data at Scoop, leveraging the power of <a href="https://eng.uber.com/h3/">h3</a> and <a href="https://kepler.gl/">kepler</a>. The PMs were hostile to that idea. How would being smarter with geo data help the bottom line? Very much a P2. They wanted us to develop Machine Learning (ML) models of whether or not we could serve different customer requests. Making models of, basically, what our own backend systems were doing initially seemed like a questionable idea.</p><p>I spoke with a bunch of people at Scoop while still trying to <a href="https://medium.com/@kenneth.kuhn/how-data-scientists-are-evaluated-832806c5ec51">move tickets across the Jira board</a>. I spoke with an Ops guy (who coincidentally had come from Uber) and with one of the co-founders. I spoke with all of my DS colleagues. (We had a great team! Miss you all.) I realized that the geo data revamp could be paired with a pricing update that could produce real value for the company and our users. I realized that the ML models were just a way to make some of our services (e.g., pricing) aware of the dynamics of other of our services (e.g., the matcher). I ended up working on both of these project ideas. I learned a lot and felt both were meaningful. It’s a shame about COVID.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4783e653f9dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Data Scientists Are Evaluated]]></title>
            <link>https://medium.com/@kenneth.kuhn/how-data-scientists-are-evaluated-832806c5ec51?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/832806c5ec51</guid>
            <category><![CDATA[performance-reviews]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[faang]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Fri, 23 Jul 2021 00:51:58 GMT</pubDate>
            <atom:updated>2021-07-23T01:00:05.944Z</atom:updated>
            <content:encoded><![CDATA[<p>I joined medium to write about transportation but my post about the <a href="https://medium.com/@kenneth.kuhn/a-field-guide-to-data-science-interviews-1b41beac1879">Data Science Interview Process</a> was my most read post by far.</p><p>This is a similar post about how you might be evaluated as a Data Scientist at a tech company. As before, I’ll add in personal anecdotes, focus on what I wish I had known previously, and offer my somewhat cynical take on things.</p><h4>It’s (Hopefully) Not That Important</h4><p>The stakes are both higher and lower than you might think.</p><p>A great performance review can lead to a promotion. A terrible performance review can lead to being <a href="https://medium.com/swlh/the-most-terrifying-string-for-software-engineers-to-parse-in-their-brain-pip-afe919c0dc3d">put on a PIP</a> and/or being laid off. Furloughs and layoffs are common in tech, especially now. Being evaluated poorly, officially or unofficially, can lead to uncomfortable meetings with your Engineering Manager and your team, and just a poor work experience. So it’s worth working to make sure you are thought of as a high performer.</p><p>One thing to keep in mind is that, thankfully, the Data Science job market is hot. There is <a href="https://zacernst.medium.com/leaving-academia-for-the-private-sector-seven-years-later-fbb7849182f6">career security if not always job security</a>. I lost what I thought was my dream job and relatively rapidly got 3 new offers and a job one level higher. Yay! And I’m terrible at interviewing. If you’re aiming for a promotion and don’t get it, apply elsewhere. You may get that level up. If you do, chances are you’ll make more money switching anyways.</p><p>There are two exceptions here. If you’re just starting your Data Science journey, then it will be harder to bounce back and find that next job. If you’re on a work visa, then the stakes are higher. People know these things. Good people will want to help.</p><p>Another thing to keep in mind is the stochastic nature of all of this. It’s much easier to get that promotion or avoid the sack if your project, manager, and startup are … impactful. You can and should carefully select your work. But there will be a lot you can’t control, especially as a junior Data Scientist. When layoffs or furloughs happen, they are often planned at high levels within the organization based on data you will never see. It’s not all about you. It’s also important to recognize that <a href="https://medium.com/@kenneth.kuhn/i-sometimes-move-rocks-fd0d1d829cd4">different people will shine in different environments</a>. Get in where you fit in!</p><p>Also know that your career will be long and there aren’t that many levels in Data Science. If things don’t turn out right this year, maybe it’ll happen next year. No worries!</p><h4>Move the Jira Tickets Across the Board</h4><p>Imagine a super organized, <a href="https://en.wikipedia.org/wiki/Agile_software_development">agile</a> company. You join as a junior Data Scientist and are assigned to a project. That project is led by a Project Manager. You have a <a href="https://www.atlassian.com/software/jira">Jira</a> board full of tickets describing work that needs to be done, organized around 2 week sprints. You have daily stand up meetings where everyone talks a bit about the tickets assigned to them. You may even have backlog grooming, sprint planning, and sprint retrospective meetings. Wow, such organization!</p><p>Many Data Scientists and Engineers hate Jira. Jira is <a href="https://towardsdatascience.com/the-jira-mindset-is-damaging-your-data-science-team-66efc46aaa47">awkward for data science</a>. People will tell you the Jira board doesn’t matter. It does. Particularly for junior employees. Important people can track it. Learn how Jira works. Make sure you always have tickets and are moving your tickets across the board at a good pace relative to your peers.</p><p>Your world may not look like the one I described. At 2 of the 3 jobs I’ve had, there were long stretches when a coworker used a janky Google Sheet to track project status. I don’t mean to criticize; I didn’t really mind the janky Google Sheets. My current work is split between responding to urgent incidents, attending entirely too many meetings, and blue sky research, none of which can be tracked well in Jira. I guess here I do mean to criticize. We have all of the hassles of Project Management and none of the benefits. I’m eight months in and yet to see an <a href="https://www.fictiv.com/articles/how-to-write-an-engineering-requirements-document">PRD or an ERD</a>. Whatever. I’m moving tickets across the board, and making sure boxes get ticked off in that Google Sheet. Jira isn’t perfect, but if you aren’t moving the tickets across the board it’s often a sign that something is off.</p><h4>Commit Code, Merge PRs</h4><p>You should pay attention to Jira because it can be used to quantify your output as a worker. Welcome to <a href="https://www.vice.com/en/article/wx5nmw/amazon-introduces-tiny-zenbooths-for-stressed-out-warehouse-workers">your dystopian present</a>. Another way to quantify output is to look at activity on the corporate code repository. How many commits have you made? Pull requests merged? reviewed? Etc.</p><p>You may be saying that as a Data Scientist your job is not to sling code. I’m inclined to agree. Even for Software Engineers, it would be foolish to evaluate someone based on how many lines of code they write. That’s true too. But I am just letting you know that there are high ranking employees at tech companies who do evaluate people by looking at their github activity. I’ve seen it happen. Particularly for high ranking employees who used to be Software Engineers and Machine Learning Engineers. Sometimes they just want to see if someone has the ability to contribute to the production code base. The hardest commit to merge is the first one. Just do it! Ask for help if you need it (see below).</p><p>You may be saying that Data Scientists on your team don’t really touch production code, or that there has never been a task assigned to you where you needed to change production code. That’s how I initially felt at my first Data Science job at Uber. I worked hard for those first few months, putting together analyses in Jupyter notebooks, queries, docs, and presentations. I’d like to think I shaped the development of some key services and the team’s verbiage (ha! see below). No one ever indicated I should even think about the code. But, if I had it to do over again, I’d have worked more in the code from the start. I would find a way. Put your name on the product and, at a tech company, the code is the product.</p><h4>Ask for Guidance and Help (in the right way)</h4><p>There are two super common failure modes for new Data Scientists. One involves the new hire not learning how to contribute followed by them getting frustrated, demotivated, and disengaged. Maybe he or she doesn’t know what they should be working on or how to use git or doesn’t have the right permissions set up to access the data and doesn’t know who to ask. He or she asks the wrong person or waits too long to ask. Everyone is surprised when, four months in, Anthony on Pricing doesn’t know how to query for revenue.</p><p>Another super common failure mode involves the junior Data Scientist working hard for entirely too long all by themselves on some complex analysis or feature that their team is not all that interested in. This one is especially painful because of the disconnect between how well Evelyn thinks she’s doing and how well the rest of the team thinks she’s doing.</p><p>Both of these super common failure modes are easily avoided with proper communication. Ask for help when you get stuck, especially when you’re just starting out. Ask for feedback when you complete a task, and pay attention to that feedback. Make sure you are regularly completing tasks. Ask lots of people.</p><p>One important point: you don’t have to go exclusively to your Tech Lead. Pay attention to how your Tech Lead likes to communicate. There are Zoom people and there are Slack people. It’s 2021; there are no other acceptable methods of communication. Constantly bugging your Tech Lead via the form of communication they don’t like isn’t a great idea. How would you like it if you kept getting phone calls from some strange number?</p><p>Your relationship with your Tech Lead is very important, even if they aren’t technically your boss. Your Engineering Manager will ask your Tech Lead about your contribution to the team.</p><p>Also, your peers will often know how to help and probably aren’t as busy as your Tech Lead. Here’s a secret I’ve learned after 3 years as a Data Scientist: we’re all making it up as we go along, building upon what we have seen. It’s helpful to learn from as many people as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGC8npYiZrzDDP_hf2VjEg.jpeg" /></figure><h4>Etc.</h4><p>It is important to have good relationships with your Engineering Manager and your Tech Lead. You already know that and are probably over emphasizing how important it is to make the right impression on them. Don’t overthink it. Don’t be obsequious. But hopefully you genuinely get along. It’s easier in tech. We’re all working together. It’ll be easier to get along if you’re moving the jira tickets across the board.</p><p>It is beneficial to shape the direction and even the verbiage of your project and team. Coming from 6 years in consulting / public policy, I know I over emphasized this. I was proud when my Uber Bus team started talking about headways and passengers alighting and looking at time-space diagrams. That was my influence! But there were minimal tangible outputs. I should’ve been making Pull Requests on the existing Route Generation microservice.</p><p>It’s important to document your work. Make and distribute a brief but well written Google Doc explaining your analysis. Your Jupyter notebook that’s almost entirely code isn’t enough. Have one clean and consistent point of view. Make a Confluence page for the big task or little project that you’re working on. Solicit feedback, especially if English isn’t your first language. Missing or exceptionally poor documentation is very common and will make a bad impression on Engineering Managers — the very people who have to justify your performance review. Again coming from 6 years in consulting / public policy, I know I over emphasized documentation. But I also got it slightly wrong. Don’t explain both sides and don’t write too much. I need to follow that advice right now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=832806c5ec51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thoughts on the Vehicle Routing Problem (2 of 2)]]></title>
            <link>https://medium.com/@kenneth.kuhn/thoughts-on-the-vehicle-routing-problem-2-of-2-add0cb4cce64?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/add0cb4cce64</guid>
            <category><![CDATA[vehicle-routing-problem]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[travelling-salesman]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Sun, 13 Sep 2020 15:44:23 GMT</pubDate>
            <atom:updated>2020-09-13T15:44:23.953Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@kenneth.kuhn/thoughts-on-the-vehicle-routing-problem-1-of-2-68086db61959">last post</a>, I introduced a mathematical formulation of the Vehicle Routing Problem inspired by <a href="http://web.mit.edu/jaillet/www/general/online-routing-18.pdf">a recent paper</a> on taxi routing. The paper caught my eye initially because the authors mentioned being able “to dispatch in real time thousands of taxis serving more than 25,000 customers per hour.” Let’s go a bit deeper now and look at the authors’ solution algorithms.</p><p>This post does not have mathematical expressions in it but will definitely be easier to follow for readers with a background in Operations Research. I’ll be a bit snarky, mainly to point out areas where I think further work is warranted.</p><h3>Initial Algorithms, Results</h3><p><a href="http://web.mit.edu/jaillet/www/general/online-routing-18.pdf">The paper</a> we are discussing first compares four solution algorithms for the formulation from the <a href="https://medium.com/@kenneth.kuhn/thoughts-on-the-vehicle-routing-problem-1-of-2-68086db61959">previous post</a>.</p><p>The baseline is set by a <strong>greedy </strong>algorithm that goes through riders sequentially, in order of their earliest possible pickup time, and assigns them to the driver that can pick them up first. The authors also use an algorithm they label <strong>MIOoptimal</strong>, which calls the <a href="https://www.gurobi.com/">Gurobi commercial solver</a>. The authors next apply the well-known <a href="https://en.wikipedia.org/wiki/2-opt"><strong>2-OPT </strong>heuristic</a> algorithm. I won’t describe this heuristic in detail here but it’s fairly general and often the first non-greedy heuristic people (especially Computer Scientist types?) reach for when faced with a Vehicle Routing type problem. Finally, the authors use their own algorithm called <strong>maxflow</strong>. maxflow selects several sets of random but feasible times for rider pickups. It then assigns riders to drivers by solving a relatively simple network flow problem, for each set of pickup times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*IbG_hIFBLvmFPu_HHP95Xw.jpeg" /><figcaption>Some initial results regarding computation times</figcaption></figure><p>This chart displaying the running times of the MIOoptimal algorithm demonstrates both that it would be impractical to solve realistic-sized problems to optimality and that the length of rider pickup time windows are a key factor in determining computational complexity. Longer time windows mean more possibilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*aAwqVSh29NSsPXJeCZNRNw.jpeg" /><figcaption>Performance of initial algorithms (relative to greedy)</figcaption></figure><p>MIOoptimal is able to increase profits 15–20% above the greedy baseline in the scenarios where demand is high. You can see why there is so much interest in optimizing vehicle routing.</p><p>I really like the setup here. The use of a heuristic algorithm on any problem should be motivated by an examination of how well the algorithm performs relative to optimal on small problem instances and a demonstration of why solving to (guaranteed) optimality is impractical on larger instances.</p><h3>Snark</h3><p>The authors highlight the relatively good performance of the maxflow algorithm when demand was high and time windows were 1 minute long. However, the algorithm performs worse than the greedy algorithm in the low- and medium-demand cases when rider time windows are 6 minutes long, the longest tested. That’s surprisingly poor, to be quite honest. maxflow does worse than the 2-OPT algorithm whenever time windows are longer than a minute.</p><p>Taking a step back from the math, in what context would people make requests for rides well in advance of when they want to be picked up but also demand to be picked up within a 1 minute time window? It doesn’t make sense in New York City but it makes even less sense when you think of taxi riders in rural areas or developing countries. Moving away from literal taxi routing, it doesn’t make sense for applications involving moving freight.</p><p>Of course the maxflow algorithm does well when time windows are tiny. The selection of rider pickup times, which maxflow accomplishes in a rather… haphazard fashion is relatively unimportant in such problems. The assignment of riders to drivers is more important and much simpler here, and maxflow solves this sub-problem to optimality. It sure is handy that the problem is simpler in these instances.</p><p>I’m reminded of the Aircraft Sequencing Problem, selecting the order in which aircraft utilize a common resource such as a runway. The <a href="https://dspace.mit.edu/handle/1721.1/67911">seminal paper on this problem</a> introduced a Dynamic Programming algorithm with “Constrained Position Shifting,” requiring that no aircraft be shifted more than a few spots from where it sits in some given nominal ordering. Such a constraint is the absolute best way to mitigate the <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">curse of dimensionality</a> and make Dynamic Programming an attractive framework. I’ve heard people talk about Constrained Position Shifting as a way to ensure fairness. Sure, I guess, although it seems pretty clear to me that the math came first.</p><p>All of this is just to say that the maxflow algorithm may not make sense for your particular Vehicle Routing Problem. In its defense, the maxflow algorithm is most intriguing in the context of the algorithms the authors build on top of it for dealing with larger problems.</p><h3>Fancier Algorithms</h3><p>The authors have established that it’s impractical to solve larger problem instances to optimality. Their solution is to “sparsify the flow graph,” essentially to rule out some possibilities and to remove some of the binary variables from larger optimization problem.</p><p>They start by proposing a <strong>k-neighborhood </strong>algorithm. Driver time is our currency in the problem we are solving. We trade it in for profit. Recall that if we choose to ask driver i to pick up rider j first, then the rider’s pickup time must be larger than both the quickest i can get to j and the earliest j can be picked up. So the cost of this assignment, in driver time, is the larger of these two quantities.</p><p>If we choose to have some driver serve rider j2 immediately after j1, then the time between the two pickups must be both larger than the minimum time to drive j1 to their destination and then get to j2’s location and larger than the minimum time between the two relevant time windows. This is the cost of such an assignment.</p><p>The k-neighborhood algorithm looks at each rider and identifies both the k least costly ways to serve the rider and the k least costly next riders to serve after the rider. All of the identified interesting assignments are collected. Any potential assignments not identified are discarded with decision variables fixed at 0.</p><p>Are you with me so far? Great, let’s dive deeper into the madness.</p><p>The authors’ <strong>backbone</strong> algorithm involves applying the k-neighoboorhood algorithm, then running the maxflow algorithm on the resulting, reduced-size problem. All of the assignments that appear in any of the solutions that the maxflow algorithm produces are then stored. (Recall that the maxflow algorithm involves trying out lots of different sets of rider pickup times, each of which generates its own set of driver assignments.) Any assignments not identified as optimal in any of the solutions produced by the maxflow algorithm are discarded, with decision variables fixed at 0. Finally, the backbone algorithm solves the resulting mini-problem to optimality using the MIOoptimal algorithm.</p><p>We’ve gone through the looking glass now. But wait, the authors go slightly deeper.</p><p>The authors’ <strong>local-backbone </strong>algorithm takes as input a feasible solution for the master problem. It runs a modified version of the backbone algorithm which consistently includes the decision variables corresponding to driver assignments made in the provided feasible solution. The result of this is a new solution for the master problem. We can plug that new solution right back in and iterate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*3XpwNmpQUFjNm60wxguwEQ.jpeg" /><figcaption>Some results on relatively large problems</figcaption></figure><p>The results above are from problem instances which involved 2,700 drivers and 6,000–6,500 riders. The K parameter is the k used in the k-neighborhood algorithm. The k-neighborhood algorithm isn’t used by the 2-OPT heuristic. The MIOoptimal results shown here are the result of applying the MIOoptimal algorithm to reduced-size problems defined by the k-neighborhood algorithm. Here all algorithms had a 5 minute limit on computation time. MIOoptimal was unable to solve even the reduced-size problems to optimality when K was larger than 2. Rider time windows were 5 minutes long.</p><p>5 minute time windows still seems short to me but it’s definitely better than 1 minute time windows.</p><p>It’s again interesting to see much fancier algorithms performing worse than the greedy algorithm, here when K = 2. This points to the difficulty of the problem and the importance of knowing the holistic state of the world. The greedy algorithm must be choosing assignments for drivers which are still reasonable but which don’t appear in the driver’s top 2 options at that time and location in a counterfactual scenario where there are no other drivers.</p><p>There might be a way to improve the k-neighborhood algorithm based on what’s going on around each potential assignment. This could involve looking at the distribution of the costs considered by the algorithm. Maybe only consider k assignments plus the next n assignments which have a cost of less than x driver minutes. I don’t know. But it seems like this is where the algorithms could be improved (?).</p><h3>Wrap Up</h3><p>The authors go on to test their algorithms on online problems in a simulation, resolving updated problem instances every 30 seconds. This is where the authors can claim to serve 25,000+ customers per hour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*lbPYSmFtQW2BnZq6JQPYMQ.jpeg" /><figcaption>Some results from the simulation representing the online problem</figcaption></figure><p>I found the chart above interesting as it shows the benefits of optimizing vehicle paths and of getting information about ride requests ahead of time.</p><p>When solving Vehicle Routing Problems in real life, we don’t always get information on ride requests ahead of time. A lot depends on context. I don’t think the general public realizes the efficiency benefits unlocked by “prior request time.”</p><p>I saw that Lyft recently began offering people a “<a href="https://www.lyft.com/blog/posts/wait-and-save">Wait and Save</a>” option. Might this be an attempt to split the “extra profit” generated by prior request time?</p><p>In a section on Extensions, the authors talk about being able to forecast demand and “to route idle taxis to areas of popular demand.” This would be the next best thing to prior request time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=add0cb4cce64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thoughts on the Vehicle Routing Problem (1 of 2?)]]></title>
            <link>https://medium.com/@kenneth.kuhn/thoughts-on-the-vehicle-routing-problem-1-of-2-68086db61959?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/68086db61959</guid>
            <category><![CDATA[vehicle-routing-problem]]></category>
            <category><![CDATA[travelling-salesman]]></category>
            <category><![CDATA[vehicle-routing]]></category>
            <category><![CDATA[operations-research]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Thu, 03 Sep 2020 17:35:03 GMT</pubDate>
            <atom:updated>2020-09-03T17:35:03.659Z</atom:updated>
            <content:encoded><![CDATA[<p>I start off this post by providing a gentle introduction to a specific mathematical problem that has numerous practical applications. I then dive into some technical details and my thoughts on an interesting, recently published paper. You’ve been warned. I’ll try to make it interesting.</p><p>My dad showed me Charles ReVelle’s <a href="https://pages.jh.edu/~jhumag/0697web/revelle.html">article about the Roman Legions Problem</a> a lifetime ago, just as I was starting at Johns Hopkins. The Problem is a neat puzzle. I was hooked. I took all of Dr. ReVelle’s courses. He became my unofficial advisor.</p><h3>Intro to Operations Research</h3><p>The Roman Legions problem is an <strong>operations research</strong> problem, a special type of math problem where you <strong>minimize or maximize</strong> a given function known as the <strong>objective function </strong>by modifying certain variables known as <strong>decision variables</strong>. You are restricted from violating given mathematical relations known as <strong>constraints</strong>. The objective function and constraints often include constant values known as <strong>model parameters</strong>.</p><p>I find it fascinating how you formulate and then solve such a problem. You must select an objective function and constraints so that when the problem is <strong>solved</strong> the resulting decision variables provide useful information, like where to locate the Roman legions. Your constraints must force the variables<strong> </strong>to take on values that make intuitive sense. There is no point in a formulation that tells you to place -1 Roman legion in Constantinople.</p><p>You generally strive to make all functions as simple and intuitive as possible. You also choose a formulation that allows you to find a solution relatively quickly. It can help to survey established formulations. There are well known algorithms for rapidly finding solutions to such formulations, and many of the formulations are capable of representing large numbers of seemingly unrelated practical problems.</p><p>It is magical when you are able to quickly get the solution to what initially seemed to be a tricky problem. It is interesting when you realize the problem which you need to solve is mathematically equivalent to what initially seems like a totally unrelated issue.</p><p>I am particularly interested in operations research problems related to geography and transportation. It’s remarkable how many of the world’s best operations researchers study transportation. Perhaps there is some relationship between being <a href="https://www.amazon.com/Maphead-Charting-Weird-World-Geography/dp/1439167184">obsessed with maps</a> and with math puzzles. Is there such a thing as map-math heads?</p><h3>The Traveling Salesman Problem</h3><p>The <a href="https://en.wikipedia.org/wiki/Travelling_salesman_problem"><strong>Traveling Salesman Problem</strong></a> is the most famous operations research problem. A salesman must visit a set of locations and return home. The goal is to minimize the distance traveled. The decision boils down to selecting the order in which the locations are visited.</p><p>The distances between locations are model parameters. It’s interesting to note that, at the transportation tech companies I’ve worked at, the estimation of quantities like this is a very important issue in its own right. You’ll sometimes find a disconnect between what academics study and what’s important in practice, particularly when it comes to transportation network companies.</p><p>Most formulations use one binary decision variable<strong> </strong>for each pair of locations, representing whether or not the salesman goes directly from the first of the locations to the second. (Binary decision variables must be set to a value of 0 or 1.) If you know which of these decision variables are set to 1, you can construct the optimal path for the salesman. There are other ways to formulate the problem but these decision variables allow for a relatively compact mathematical representation of the problem. The algorithms that find the best paths for the salesman the fastest often assume a formulation that uses decision variables like these. Neat right?</p><p>A key technical challenge involves ensuring that a solution does not include a “sub-tour,” for example having the salesman go from location A to location B and from location B to location A. If your formulation isn’t quite right, such a sub-tour could trick you into thinking that both locations have been visited when neither location is linked to the rest of the salesman’s path. The end result will be a nonsensical route for the salesman.</p><h3>The Vehicle and Taxi Routing Problems</h3><p>The <a href="https://en.wikipedia.org/wiki/Vehicle_routing_problem"><strong>Vehicle Routing Problem</strong></a><strong> </strong>is a generalization of the Traveling Salesman Problem and a well known problem in its own right. You have a set of vehicles and a (larger) set of locations to visit. You must specify routes for all the vehicles so that they collectively visit all the locations. There are numerous practical applications; for example consider how Amazon delivers all of the packages we’ve been ordering lately.</p><p>A common extension involves <strong>time windows</strong> that restrict when each location can be visited. The requirement that all locations be visited is often removed. You must then choose which locations to visit, committing to visiting those locations during specific time windows.</p><p>In the <strong>online </strong>version of the problem, all the relevant information is not available initially. Requests show up over time, as vehicles operate on their routes.</p><p>The <strong>Taxi Routing Problem</strong> is a special form of an online version of the Vehicle Routing Problem with time windows. The vehicles are referred to as drivers, and the drivers serve riders. Each rider has to be taken between pickup and dropoff locations. It’s not as different from the classic Vehicle Routing Problem as you might think, as we’ll see.</p><h3>Personal History</h3><p>I first came across the Vehicle Routing Problem in 2008. I was working for <a href="https://www.aviationsystemsdivision.arc.nasa.gov/index.shtml">NASA studying aviation</a>. I had talked them into letting me work in Germany for 2 months as an exchange researcher with the German Aerospace Center (DLR). DLR was in the middle of the CARMA project, an attempt to optimize the movement of service vehicles at Hamburg Airport. It was a dream project for a map-math head like me.</p><p>I quickly became convinced that we were working on a specific type of Vehicle Routing Problem. My German manager and I published <a href="https://elib.dlr.de/60144/2/ServiceVehicles_2009-01-20.pdf">a paper exploring this idea</a>. The paper won an award at an aviation conference but was otherwise forgotten. I doubt that the algorithms I developed are used anywhere today. But I still remember being lost on the outskirts of a small town in the middle of northern Germany, obsessed with the Vehicle Routing Problem for 2 months.</p><p>Fast forward to today. We have started a <a href="https://en.wikipedia.org/wiki/Journal_club">journal club</a> at Scoop and I am organizing the next session. I notice that 2 of the most famous operations researcher professors at MIT and their PhD student recently published a <a href="http://web.mit.edu/jaillet/www/general/online-routing-18.pdf">paper on the Taxi Routing Problem</a>. The problem seems salient for transportation tech companies. The paper is remarkably well written and easy to follow. There are lots of neat little math-y tricks. My obsession with the Vehicle Routing Problem has been rekindled.</p><p>I’ll spend the rest of this article diving into technical details. The math I present is very much based on the MIT paper, although I’ve changed the notation and even the formulation slightly. I made most of the changes to simplify things. But I’ve also included a tweak to the formulation based on my own work on the CARMA project (more on this later). Hopefully this inspires some new map-math heads. At the very least, it’s helping me organize my thoughts and prepare for journal club.</p><h3>Decision Variables</h3><p>Let’s start with the decision variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*FACf5P7-NJGvmRLQUCPcdg.jpeg" /></figure><p>The first two types of decision variables are the two most intuitive. In an optimal solution, these variables will tell us which riders we pick up and when we pick them up. To relate the two and ensure our decision variables take on values consistent with what they represent, we will need to construct paths for the drivers. We do this via the third and fourth types of decision variables.</p><p>When I studied the Vehicle Routing Problem (a dozen years ago!), formulations used decision variables with 3 indices. Does vehicle i visit location k immediately after location j? It makes intuitive sense, but if you have many vehicles and locations you end up with a huge number of decision variables.</p><p>Here, all decision variables only have 2 indices. You specify which rider, if any, each driver picks up first. You also specify which rider, if any, comes after each other rider in some (any) driver’s route. If these decision variables are set, you can construct each driver’s route. One of my first thoughts here is that it’s really neat that we have a formulation likely to involve fewer decision variables.</p><p>Another thought I have, and had a dozen years ago, is that the Vehicle Routing Problem is two problems in one. We must assign the riders to specific drivers and then solve the Traveling Salesman Problem, or something like it, for each driver. Here the c and d decision variables hint at this way of splitting (<strong>decomposing</strong>), the larger problem. The new formulation also hints at an imperfect but potentially fast strategy (a <strong>heuristic</strong>), where riders are sequentially added to a set of incomplete driver routes. These are my somewhat random first thoughts.</p><h3>Model Parameters</h3><p>Next, let’s examine the model parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*7DgjSJpGXN4vHcmyhZEStg.jpeg" /></figure><p>We derive some reward for serving riders. We can think of this as the profit or revenue for our taxi firm. The relevant parameters mimic the decision variables listed previously. I’ll talk more about this when introducing the objective function.</p><p>We specify the time windows during which each rider can be picked up, as mentioned previously.</p><p>We also specify how long it takes each driver to get to each rider, and to go between riders. Here again, we are mimicking our decision variables. Note that our decision variables specify pickup times so we really care about the time between pickups — including both the time a driver spends taking passenger 1 to their destination and the time it takes the driver to then travel alone to passenger 2’s trip origin. At least for me, it now becomes clear why the Taxi Routing Problem is just a special case of the Vehicle Routing Problem. Note also that we only care about the minimum of the relevant travel times. We assume that we can add any amount of delay to any driver’s schedule as needed.</p><h3>The Objective Function</h3><p>The objective function is just the sum total of the rewards we receive for serving riders. We want to maximize this function. We only receive a reward if we serve a rider, which is indicated by one of our decision variables being 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*uqLrSJZOMD5JsI8cPM1d9g.jpeg" /></figure><p>Our rewards are based not just on whether or not we serve a rider but also on what happened immediately prior to us picking up that rider. Note in particular that this formulation allows us to penalize for dead heading, for example when a driver must travel between one rider’s drop off location and the next rider’s pick up location.</p><p>Ignoring dead heading, we could have a fixed reward for serving a rider. We could even a fixed reward for serving any rider. In that case, we’d just maximize the number of riders we serve. Our formulation allows this. In these cases, we would set groups of model parameters equal to the same value. No problem. But our formulation is more general than that.</p><h3>Constraint Set 1</h3><p>Next let’s examine the constraints that exclusively focus on the binary decision variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*ngRunnZ6FP1DFkfESCBlSA.jpeg" /></figure><p>The first constraint ensures that each driver is only assigned at most one rider to pick up first. The second constraint ensures that, for each rider j, we can only think of picking up other riders immediately after dropping off j if we have committed to serving j. Furthermore, we can only have at most 1 of these next pickups.</p><p>The third constraint ensures that our variables are consistent, that selecting to pickup rider k means we either have chosen some driver i to pickup k first or we have chosen to pickup rider k immediately after dropping off rider j. Note that this constraint together with the requirement that these decision variables be binary ensures that no solution involves picking up a rider more than once. For any rider k, at most one of the relevant c and d decision variables will be set to 1.</p><p>We have specified |I|+2*|J| mathematical relations, where |I| is the number of drivers and |J| is the number of riders. It is remarkable how efficient the formulation is with regards to the number of relations specified.</p><p>Even more importantly, the objective function together with the constraints shown above form a relatively simple sub-problem. We can remove (<strong>relax</strong>) the constraint that the decision variables be binary and let them be real valued numbers between 0 and 1. There will be an optimal solution to the resulting problem where all decision variables are 0 and 1 anyways. This allows us to use relatively fast <strong>linear programming</strong> algorithms to solve what looks like a typically harder to solve binary or <strong>integer programming</strong> problem. In particular, the problem is a type of problem called a <strong>Network Flow Problem </strong>that is relatively easy to solve, at least for small “networks.” We’ll return to this when we talk about solution strategies.</p><h3>Constraint Set 2</h3><p>There are additional constraints that focus on the timing of rider pickups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*ocfodO4MjsYj20QtqTYCaQ.jpeg" /></figure><p>The first constraint shown above ensures that each rider’s pickup time is within the relevant time window. The second constraint is more complex. Recall, from the preceding section, that for any rider k, at most one of the relevant c and d decision variables will be set to 1. All the other decision variables will be set to 0. The second constraint then specifies that if some driver i is to pickup k first, then the pickup time must be after the earliest possible time i can get to k. If k is to be picked up immediately after j, then the difference between the pickup times must be at least the fixed minimum time between these two pickups.</p><p>The second constraint shown above is my own simplified version of two separate constraints provided in the paper by the MIT authors. Those authors chose to deal with the two if statements mentioned at the end of the preceding paragraph separately. I chose to combine them. This is based on a trick I applied in the CARMA project. It worked well for me in that instance. We’ve reduced the number of constraints and, at least to me, the formulation is a little cleaner.</p><p>It’s possible I’ve missed something here as I’ve spent a few days on this and others have definitely studied this problem more intensely, for a longer period of time. The MIT authors focus on the online version of the Taxi Routing Problem, while I’ve purposefully minimized discussion of the complications that arise in this context to simplify things. Let me know if you have thoughts here.</p><h3>(Starting to Talk About) Solution Strategies</h3><p>In the CARMA project, I tried to get good solutions to the Vehicle Routing Problem by using an algorithm that first fixed the binary variables. We can imagine doing something like that here.</p><p>Note that if we have specified values for the c and d variables, then it becomes very easy to set the b variables. Focus first on the riders that some driver picks up first (as indicated by a c variable set to 1). Each of these rider’s pickup times will have to be greater than both the time for the relevant driver to reach them and the start of the relevant time window. Set each pickup time to the larger of the two relevant lower bounds. Then look at the riders picked up next. Each of these rider’s pickup times will have to be greater than both the prior rider’s pickup time plus the minimum time between these two specific pickups and the start of the relevant time window. Set each of these rider’s pickup times to the larger of the two relevant lower bounds. This might sound complicated in words, but it’s remarkably simple math and code.</p><p>The MIT paper turns my idea on its head. (I was so close and yet so far.) Imagine having specified values for the b variables (the pickup times) but not the a, c, and d variables. If the pickup time windows are relatively short, then this becomes a particularly useful way to think about the problem (more on this later).</p><p>The first of the constraints in what I’m calling Constraint Set 2 is irrelevant if we’ve chosen reasonable (<strong>feasible</strong>) b variables. In the authors’ algorithms, they actually choose the pickup times based on the given time windows.</p><p>The other constraint here is more interesting. Recall that for any rider at most one of the relevant c and d decision variables will be set to 1. Our choice boils down to picking one of these variables. For any rider, our constraint boils down to the rider’s pickup time being greater than either some known t parameter or some other rider’s pickup time plus some known v parameter. This will either be true or false. In a world where we know the pickup times, this constraint simply (and quickly) rules out certain choices. That’s all it does. Put another way, for any given rider and driver, either the driver can or cannot reach the rider before the (assumed specified) pickup time. For any given pair of riders, either the time between the pickup times respects or doesn’t respect the minimum time between pickups. This constraint forces certain c and d variables to 0. It is otherwise irrelevant.</p><p>The MIT paper describes an algorithm that randomly selects numerous sets of b variables (rider pickup times), removes no longer feasible c and d variables (see preceding paragraph), and then solves the relatively simple problem comprised of the objective function together with Constraint Set 1. This “<strong>maxflow</strong>” algorithm gets relatively high quality solutions to the Taxi Routing Problem relatively rapidly, particularly when time windows are short.</p><p>The paper gets more interesting as it builds off this idea, taking huge examples (<strong>instances</strong>) of the Taxi Routing Problem and using the maxflow algorithm to intelligently reduce the size of the problem before solving it. The authors develop an application capable of “dispatching in real time thousands of taxis serving more than 25,000 customers per hour.”</p><p>This blog post is already entirely too long so I’ll talk more about this in a follow-up post, assuming both you and I still have interest in it. I’m a little surprised if you made it this far. Congrats! Maybe you too are a map-math head.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68086db61959" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I (Sometimes) Move Rocks]]></title>
            <link>https://medium.com/@kenneth.kuhn/i-sometimes-move-rocks-fd0d1d829cd4?source=rss-86e0b6401914------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd0d1d829cd4</guid>
            <category><![CDATA[work]]></category>
            <category><![CDATA[performance-management]]></category>
            <category><![CDATA[performance-reviews]]></category>
            <category><![CDATA[workplace]]></category>
            <dc:creator><![CDATA[kenneth kuhn]]></dc:creator>
            <pubDate>Tue, 11 Aug 2020 01:30:36 GMT</pubDate>
            <atom:updated>2021-08-02T16:37:27.951Z</atom:updated>
            <content:encoded><![CDATA[<p>You never know what kind of feedback you will get when you go through an internal review. The process is especially nerve-racking when it happens just a few weeks before an important presentation to the client, on a complex project, and when one reviewer is a high-ranking researcher who has not been involved in the project.</p><p>We went through our slides slowly but largely without interruption, as the Senior Economist silently nodded and took notes. As we wrapped up, we fielded mostly predictable questions and comments from others in the room. Some we knew were totally off-base. Smile, nod, and take notes that you’ll ignore later. Eventually we asked the Senior Economist for his thoughts.</p><p>He spoke for 7 or 8 minutes straight, softly and slowly, raising issues and opportunities, pointing out potential gaps in his and our understanding, defining detailed short- and long-term options for us to edit our work, and not asking for an immediate reply on any of it.</p><p>It would be an understatement to say that I was impressed. I made a mental note to be this kind of reviewer, to be this kind of researcher. I also made a note to stop by the office of the Senior Economist for a chat. This is what I wanted to be when I grew up at this job.</p><h4>The Follow Up Meeting</h4><p>The main thing I remember from the subsequent one-on-one meeting was the Senior Economist’s quote, and I remember it only vaguely. It was something like:</p><p>“<strong><em>there are people who move rocks and people who don’t</em></strong>.”</p><p>I was asking what I should work on. He was telling me it didn’t matter.</p><p>First of all, this is a horrifying thing to hear if you have impostor syndrome. All young researchers have impostor syndrome. OK, that’s not true. There are also quite confident young researchers. These are the ones you probably avoid like the plague given half a chance. But also…</p><p>There are few things in life I have found as wrong as the sentiment expressed by the Senior Economist.</p><p>Let’s take school as an example. I had a miserable first year of graduate school. I couldn’t keep up in Stochastic Processes I or II. The <a href="https://ieor.berkeley.edu/">IEOR</a> professors at Berkeley wanted nothing to do with me.</p><p>It all turned around after that first year. I had a perfect 4.0 GPA including in several IEOR courses about stochastic processes. I was named Student of the Year one year. Yadda yadda yadda. Thank you for indulging me.</p><p>What changed? I got scared and started to work a little harder. The French students’ extensive prior training in Probability theory became less of a factor in my GPA. I found an advisor I meshed with. I came out of the closet. A lot of things changed.</p><p>I later taught students that did great in my class but terrible in other classes. And vice-versa.</p><p>I had a side hustle gig with the LA Dodgers for a few months in 2018. That was fun! I was pretty bad at that job. I wish I knew then what I know now. I would have been better.</p><p>I have written wonderful and mediocre reviews, to people I knew or later realized weren’t all that wonderful or mediocre. (Side note: be kind. You don’t regret being too kind.)</p><p>If you’ve been a researcher for any length of time, you’ll have gotten wonderful and terrible reviews back on the same piece of work.</p><p>I saw one tech blog where the author claimed that new hires are thought of as geniuses for the 6 months after they are hired and idiots for the next 6 months. Yes! And, also, sometimes the reverse.</p><p>I used to think I could get along with anyone and make almost any situation work. I was able to make close friends out of all different sorts of people. I was able to survive, as a student or visitor, in all sorts of different environments.</p><p>Now I know that I will not be outstanding as an employee in all sorts of work environments, working with all sorts of people. I still always try my hardest. And, in the right environment and working with the right people, yea, I will be outstanding. I wish I had something more profound to say. Maybe in the future I will.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd0d1d829cd4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>