<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bryan on Medium]]></title>
        <description><![CDATA[Stories by Bryan on Medium]]></description>
        <link>https://medium.com/@GilbertGravis?source=rss-7ed35fd2a965------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aRx3v2iRUxMoQssH32ib5w.png</url>
            <title>Stories by Bryan on Medium</title>
            <link>https://medium.com/@GilbertGravis?source=rss-7ed35fd2a965------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:34:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@GilbertGravis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[[Book Review] Who Gets What — and Why]]></title>
            <link>https://medium.com/@GilbertGravis/book-review-who-gets-what-and-why-e530617da3a?source=rss-7ed35fd2a965------2</link>
            <guid isPermaLink="false">https://medium.com/p/e530617da3a</guid>
            <category><![CDATA[game-theory]]></category>
            <category><![CDATA[market-design]]></category>
            <category><![CDATA[matchmaking]]></category>
            <dc:creator><![CDATA[Bryan]]></dc:creator>
            <pubDate>Wed, 09 Mar 2022 19:16:44 GMT</pubDate>
            <atom:updated>2022-03-09T19:22:58.452Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Book Review] Who Gets What — and Why</h3><p>This book, by Nobel prize winning economist Alvin Roth, is a fabulous introduction to market design. I first encountered Alvin Roth at the KDD conference in London in 2018. He gave one of the keynote talks, introducing his work and general principles of market design.</p><p>The book is structured into four parts:</p><ol><li>Markets and their ubiquity</li><li>Different ways markets can fail</li><li>Market interventions</li><li>Forbidden and “free” markets</li></ol><p>Part 2 was probably my favorite. It was filled with great examples of markets gone wrong, and I loved the introductions to various types of markets and the ways of describing them — “congested”, “unraveling”, “thin/thick”, “repugnant”, differentiating “commodity markets” from “matching markets”, finding “stable matching pairs”.</p><p>A key thrust of the book is that markets are human artifacts, not “naturally emerging”, i.e. that they are designed intentionally and have rules of operation. He depicts “free market fundamentalists” as somewhat deluded, given that even so-called free markets require basic rules for effective operation.</p><p>Throughout the book, Roth integrates explanations of a his own work, which has chiefly been on on matching markets, helping to design systems for efficient allocation of opportunities and candidates for those opportunities.</p><p>He covers a few examples in detail, the kidney exchange program, The Match, and several school admissions systems.</p><p>The Match is the algorithm used in the US to match medical school graduates to hospitals for their years of in-hospital training. The Match is based on an evolution of the <em>deferred acceptance algorithm</em>, also known as the <a href="https://en.wikipedia.org/wiki/Gale%E2%80%93Shapley_algorithm">Gale-Shapley algorithm</a>. Roth became involved when the algorithm began to fail, which happened because of spouses attempting to match to schools together. (He also references something called the <em>Iron Law of Marriage</em> in this section — “you can’t be happier than your spouse” — which I found delightful.) Roth’s work introduced changes to The Match to take into account matching pairs, the results of which is called the Roth-Peranson algorithm, which uses Gale-Shapley, then takes care of blocking pairs one-by-one.</p><p>Following his successful work on The Match, Roth worked on redesigning the admissions systems of major school systems in New York and Boston. The work on these three problems covers most of the last half of the book.</p><p>The book finishes with three chapters on other topics which seem to be outside of the purview of most of this work but are interesting nonetheless. The chapters on Signaling and Repugnant Markets were two of the most interesting to me. Roth comes out in favor of creating legal markets for organ exchange and drugs, arguing along the lines of pragmatic utilitarianism (enforcing bans does far more harm than the rates of drug addiction). At the time of the book’s publishing, legalized marijuana was in its early stages, and Roth seems excited about the prospect of change there.</p><p>All in all, I would highly recommend the book as a springboard into market design.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e530617da3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A De-biasing Framework for Interviews and Hiring]]></title>
            <link>https://medium.com/@GilbertGravis/a-de-biasing-framework-for-interviews-and-hiring-647950e998be?source=rss-7ed35fd2a965------2</link>
            <guid isPermaLink="false">https://medium.com/p/647950e998be</guid>
            <category><![CDATA[fallacy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[hiring]]></category>
            <dc:creator><![CDATA[Bryan]]></dc:creator>
            <pubDate>Tue, 03 Aug 2021 18:26:23 GMT</pubDate>
            <atom:updated>2021-08-03T18:26:23.010Z</atom:updated>
            <content:encoded><![CDATA[<p>Interviews are one of the most important processes that a company can have. They are the route by which we decide whether to make literal strangers a part of your professional neighborhood. They are high pressure for both sides. For the candidate, interviews are their time to demonstrate mastery of their craft, a capacity to think on their feet, and an ability to communicate effectively about complex topics, all in the course of a gruelling day of constant bombardment by new people and experiences. For the interviewer and hiring managers, interviews are the only chance we have to make a judgment about what it will be like to work with someone, potentially for years to come. Naturally, candidates want to do their best while also learning about future colleagues, while we naturally want to have confidence that the people we’re bringing in are competent and compatible with the ways we like to work.</p><p>With so many intense emotions at work and important decisions in the balance, interviews are one of the areas where our natural biases can be most exposed. To pull some terminology from <a href="https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow"><em>Thinking Fast and Slow</em></a>, short-term human-to-human interaction can be rife with System 1 thinking (quick judgments based on heuristics) instead of System 2 thinking (methodical deliberation). In fact, there are names for the types of biases that might emerge during an interview, including <a href="https://en.wikipedia.org/wiki/Confirmation_bias">Confirmation Bias</a>, <a href="https://en.wikipedia.org/wiki/Optimism_bias">Optimism Bias</a>, the <a href="https://en.wikipedia.org/wiki/Halo_effect">Halo Effect</a>, the <a href="https://en.wikipedia.org/wiki/Regression_fallacy">Regression Fallacy</a>, and many others. If we think back to interviews we’ve been a part of (on either side), I’m sure we can recall instances where these biases were at play.</p><p>In addition to our (completely and forgivably human) tendency towards bias, we also must recognize that interviews are an environment of high variance. When a candidate is preparing for interviews, especially technical interviews, they’re making guesses on which topics to review and how to prepare. They may have a rough idea of what will be covered, but the chance that they’ve set up time to master the specific topic of any particular interview is pretty low. Moreover, anyone’s confidence with how to proceed on a given question is subject to the specific context of that question. For example, despite the fact that a technical question ultimately just amounts to arithmetic, one’s ability to realize that might depend on how the question is set up, whether they fully understood the question as asked, and also whether they understand arithmetic. As a result, a candidate’s performance might be highly dependent on the specific circumstances of that question, of that interviewer, of that time of day, etc.</p><p>Finally, the interview process is a matter of estimation. An interview day usually involves interacting with four to six interviewers, each asking a different set of questions and potentially evaluating against a different set of expectations. So what’s the right way to combine these evaluations into an overall decision to hire or not hire? How should one balance the candidate’s stellar performance in one person’s interview against the lackluster performance in another? What is the appropriate way to weigh each interview to reach the best decision?</p><p>Finally, if we zoom out, we can see that the goal of the hiring process is<em> </em>to<em> </em><strong><em>maximize</em></strong><em> the probability of hiring highly competent people, </em>while<em> </em><strong><em>minimizing</em></strong><em> the search cost and the probability of mistakenly hiring incompetent people, all while being humane and fair.</em> This is a constrained optimization problem.</p><p>Hopefully I’ve been able to show how interviews involve challenges pertaining to <em>bias</em>, <em>variance</em>, <em>estimation, </em>and <em>optimization</em>. Naturally, this is a case for some data science thinking :) Let’s consider how we can design an interview process to take on these challenges.</p><h3>Core Skills</h3><p>First and foremost, what are we hiring for? What does it take to be good at the job? Answers to this question can often start out quite vague — a project coordinator needs “to interface effectively with stakeholders”, while a project director might need to “manage a complex project with many moving pieces”, while a software engineer needs to “write performant software”.</p><p>It’s useful to break these high level ideas into their lower-level components. To do that, think about what this person will actually be doing on the job. What does “interfacing effectively with stakeholders” really require? It sounds like it might involve persuasive and logical writing, public speaking, and the ability to negotiate effectively between competing priorities without offending anyone. Similarly, “managing a complex project” involves having technical expertise in the project area, an ability to effectively order dependent processes, and an ability to forecast and design systems of redundancy for when things go awry. Finally, “writing performant software” involves knowledge of a computer language or framework, a familiarity with software architecture, an ability to debug and troubleshoot, and an ability to implement efficient solutions to technical problems.</p><p>For each of these examples, the breakout into specific components represents an effort to identify and define the <em>core skills</em> associated with the role. For every role or title, the hiring manager should have a sense for what the core competencies are for the position (the “principle components” if you will) and design a body of interviews to test these core competencies.</p><p>Ideally, these core skills should be “orthogonal”, meaning that they are independent from each other. For example, being a persuasive writer is different from being a persuasive speaker, and any individual might exhibit a high level of competence at either, none, or both of these skills. Similarly, knowing a computer language is very different from knowing about software architecture, and while most candidates who have knowledge in the latter will also have knowledge in the former, it is possible for that not to be the case. This orthogonality is desirable because it allows for more independent measure of performance in interviews, as explained in the next section.</p><p>Core skills should also be “atomic” or “modular” in the sense that they represent a low-enough level representation of a skill that it can effectively distinguish between different roles. What that means will be different company to company. More specialized roles might have a more specialized definition of core skills. Despite a high area of overlap, a “Psychiatrist” job is different from a “Child and Adolescent Psychiatrist”, and has a slightly different set of core skills. It’s my recommendation that one not get too specific with such distinctions until the context requires it, which most frequently happens in academia and medicine, where organizations set out to hire specialists in very specific domains, but can also happen in the technology industry.</p><p>As part of identifying a definition of core skills, we should recognize that, even within the same profession, different practitioners might have varying levels of expertise across the skills associated with it. Moreover, some skills might be more essential to effective performance than others. For example, while I certainly prefer having a surgeon that is both kind and competent, it is far more important that she be competent. So, along with having a definition of core skills, we should give weights to these skills when defining a given role, and also develop a scale for measuring competency. My favorite scale is that of [Novice, Beginner, Intermediate, Advanced, Expert].</p><ul><li>Novice — Knows the existence of a concept</li><li>Beginner — Knows how to define a concept</li><li>Intermediate — Knows how to apply a concept</li><li>Advanced — Knows how to teach a concept</li><li>Expert — Knows how to adapt a concept for any scenario</li></ul><p>Below is an example of some of the core skills we might want to assess, and how we might weigh them, for the roles mentioned earlier. You don’t need to agree with the weighting shown here. Hopefully your team can develop its own definitions and systems.</p><p>Project coordinator</p><ul><li>Ability to write persuasively and logically High priority</li><li>Ability to speak persuasively and logically to a group Medium priority</li><li>Ability to adroitly negotiate tradeoffs, i.e. prioritize High priority</li></ul><p>Project director</p><ul><li>Technical expertise in the project domain High priority</li><li>Ability to plan execution and account for bottlenecks High priority</li><li>Ability to design redundant systems, and forecast Medium priority</li></ul><p>Junior software engineer</p><ul><li>Competency with a computer language High priority</li><li>Knowledge of software architecture Medium priority</li><li>Ability to debug Medium priority</li><li>Ability to design and implement efficient solutions High priority</li></ul><p>This framework is applicable to any profession, except perhaps those that don’t rely on skills but rather on one’s network or relationships. The “network effect” (knowing the right people, or having the right people vouch for you) can become a major contributor to one’s professional success. This makes sense, as the ability to recruit or lead other competent people can depend on having a good network and reputation. In the best scenarios, this is the result of building a community of advocates based on a history of competence and good judgement, but it’s easy for it to become a shortcut for making heuristic judgements, like someone being hired based only on being connected to powerful people (which is how many of the new-grad sons and daughters of politicians and celebrities get hired).</p><p>In the absence of something like the <em>core skills</em> framework, it is natural for two things to happen. First, people tend to interview only along their own areas of competence and familiarity, which is not always what the job will require. Second, in the absence of clear criteria, people tend to retreat into likeability assessments, gut feels based on how they felt during the conversation with the candidate. These types of judgements are unreliable.</p><h3>Designing Assessments and Interviews</h3><p>So we’ve done the work and identified the role we’re hiring for and defined the core skills associated with that role. Now we want to create a set of assessments that measure candidates’ competencies in these areas.</p><p>Let’s recall the overall goal of the hiring process: to<em> </em><strong><em>maximize</em></strong><em> the probability of hiring highly competent people, </em>while<em> </em><strong><em>minimizing</em></strong><em> the search cost and the probability of mistakenly hiring incompetent people, all while being humane and fair</em></p><p>Hiring competent people requires, first and foremost, that we are able to assess competence. Avoiding hiring incompetent people means that we must set criteria around those measures of competence. Minimizing costs means that we make sure people’s time is well-spent (potentially biased based on whose time is more costly). Being humane and fair means that we treat everyone with kindness and respect and that we commit to constantly improving and debiasing the process.</p><h4>Designing a panel</h4><p>Each interview in a panel should be grounded in our definition of core skills. If we find ourselves defining an interview unaffiliated with at least one core skill, then we should take a look at the interview or the core skills to find out why they don’t align. Neither the candidate or interviewers should be burdened with interviews focused on things we don’t care about, and even “behavioral interviews” should be oriented around an underlying competency we are attempting to measure.</p><p>First, an aside.</p><p>Many companies have behavioral interviews checking for “cultural fit”, a notoriously fraught and difficult-to-define concept. These interviews often end up being likeability tests or strange “belongingness” checks, which are where a lot of biases (both conscious and unconscious) can show up. The way to resolve this is not to abandon the idea of cultural fit, but rather to be explicit about what is meant by it. If fluent communication in a language is going to be important, then make it explicit. If direct delivery of bad news is important to your organization, then find a way to evaluate a candidate’s decision-making process when presented with competing incentives. If the position involves interfacing with a group of Ivy-league investors who are temperamental and judgmental, then it might make sense to check if candidates can stand up to the kind of “status checks’’ that they’ll be subjected to on the job. If profanity or high levels of informality at work is a part of your company’s culture, then define that explicitly. Depending on the position, these more amorphous qualities can in fact represent “core skills”. Making them explicit can remove much of the bias commonly associated with testing for cultural fit. On the other hand, if you find that your efforts to be explicit about cultural fit direct you to definitions based on immutable characteristics or legally protected categories (such as race, sex, age, religion, etc.), then that’s a warning sign that you probably need to adapt your company’s culture to be more inclusive.</p><p>Back to interview design.</p><p>Somewhat counterintuitively, we don’t actually want to use each interview to cover just one core skill. Ideally, each interview should cover at least two, and the whole panel should provide overlapping coverage across the entire set of core skills. The reason to do this is two fold. First, when you measure something more than once, you reduce variance in the measurement. Second, getting independent measures allows for even better estimation, and over time also allows for you to measure and control for the biases of any particular interviewer.</p><p>Another thing we want is for our interview results to contain lots of <em>information</em>, in the technical sense. Consider an interview in which the candidate was asked to find the result of a simple math problem: 2+2. The pass rate would be about 100% and a candidate passing wouldn’t tell you anything about their skills. Similarly, imagine an interview in which the candidate flipped a coin to determine whether they passed or not. The results of such an interview would be about 50%, and the results would still be meaningless.</p><p>The problem is that these assessments do not discriminate between high and low performers. And while these examples are extreme, many real-life interviews also suffer from this problem. Assessments should reveal a lot about the candidate in relation to the core skills they’re trying to measure. They should be<em> deep </em>and<em> adaptive</em>, allowing us to accurately measure a candidate’s performance on the whole spectrum of expertise (from Novice to Expert). That means that, as opposed to merely testing for basic proficiency, a good assessment should give the candidate an opportunity to impress the interviewer by pushing the boundaries of expertise in the domain. This can be done by creating questions with extensions that can be brought into play if a candidate is performing well. This makes the interview adaptive, allowing a deeper measure of the candidate’s competency in a specific domain if the candidate is ready for it. Through this, we can create assessments where “high performance” is a meaningful indicator of real expertise, and where we also have meaningful information across the gradient of low, medium, and high levels of expertise.</p><h4>Testing and calibrating</h4><p>To do this effectively, the interview panel needs to calibrate effectively on what is a good question and how that question should be graded. Creating good assessments is a challenging process. As the sections above indicate, it requires careful consideration of core skills and the design of how to test those skills in a deep and adaptive manner.</p><p>Once an assessment has been designed, it should be tested. If you have time and a team to help, the best way to do this is to do it internally, within your team, first. This involves giving the assessment to colleagues and calibrating performance expectations around their results. This is great, but many companies don’t have time for this level of process, so most calibrating happens “in the wild” with real candidates. This calibration process might be regarded as the “learning phase” of the assessment. If every candidate is knocking your interview out of the park, then you might need to add some extensions to get a better read on high-performing candidates. Similarly, if only a small percentage of candidates get even part way through your interview, then it might be better to reconsider how rapidly the interview progresses into advanced content. Remember, the goal of an assessment is to reveal information, not to stress someone out with difficult brain teasers, and not to make someone feel smart.</p><p>There’s a bit of a caveat here, because aggregate candidate performance on a given assessment will depend not only on the calibration of the interview, but also on the distribution of skills among the candidate pool. For instance, on-campus recruiting or recruiting directly from training programs will result in a candidate pool with a certain set of biases (usually great theoretical knowledge and very little practical knowledge), while targeted recruiting of people with more experience will lead to an interview pool with a significantly different distribution of skills. It’s important to be mindful about such differences, and potentially consider that when designing assessments. If you feel confident in your assessments but find there’s misalignment between the expected and actual performance of candidates, you should consider a new recruiting strategy.</p><h3>Making a decision</h3><p>So, you’ve got a definition of the role in terms of core skills, a set of assessments that you feel confident in, a group of calibrated interviewers to help conduct the interviews, and a pool of promising candidates to evaluate. You’re almost there! The question remaining is how to make a good decision about hiring or not hiring, and the associated questions of seniority, salary, and so on.</p><p>Once we have estimations of a candidate’s proficiencies across all of the core skills, we can make a decision. Recall that, in our definitions of the role in terms of core skills, we gave ourselves the leeway to weigh different skills differently. Similarly, a candidate’s performance across the skills may not be equally distributed. They might be “spikey”, where the candidate has high performance around a specific domain but lower elsewhere, or they could be more “well-rounded”, exhibiting Intermediate or Advanced proficiency across many skills, but not Expert in any. Both might be appropriate for the role under consideration, depending on how we weigh the importance of each skill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/0*OvOZSGrAy_JwF6hO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/0*8wrD-uz9-CGSNnob" /></figure><p>Despite the fact that it may seem that only two decisions (try to hire, don’t hire) exist for a candidate at this point, there are in fact many more gradations. The company can shape its decision — in terms of salary, title, leveling, etc — to account for different levels of interview performance. Another advantage of the core skill definitions is that it allows for reasonable benchmarking based on current team members.</p><p>Given the two candidates above, how do you make a decision? Different companies have very different rules on that. In the <em>The Hard Thing About Hard Things</em>, Ben Horowitz strongly recommends keeping final hiring decisions in the hands of the hiring manager. On the other hand, some companies practice majority or consensus models, where the entire hiring committee needs to be a “yes” on a candidate for an offer to be made. In most cases, I recommend something like the majority model, with some caveats.</p><p>There are a few reasons for this:</p><ul><li>First, if you’ve thought deeply about the core skills and designed a good interview process, you should be able to trust the hiring panel to make good decisions.</li><li>Second, it’s best when your colleagues are excited about a candidate joining the company, and not have resentment or misgivings. Hiring is not just about individuals, but about building teams.</li><li>Third, showing trust in your processes begets trust in your processes. Good patterns of decision-making start from the top. Even though, as the hiring manager, you may feel differently about a specific candidate than the rest of the panel, trusting your team to make good decisions will make them more likely to trust each other.</li></ul><h3>In summary (TL;DR)</h3><p>That’s it! You’ve made it. Here’s some bullet points to summarize the big takeaways.</p><ul><li>Define core skills for each role</li><li>Make those core skills <em>atomic</em> and <em>orthogonal</em></li><li>Design interviews that cover at least 2 core skills each, motivated by real-world or close-to-real-world circumstances and constraints</li><li>Interviews should not be gimmicky or arbitrarily stressful</li><li>An individual interview should aim to assess at least one core skill, ideally more than one</li><li>An panel of interviews should provide at least one, and ideally more than one, measure against each core skill</li><li>Decide on a process for making hiring decisions, and stick to it, even when it hurts</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=647950e998be" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>