<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harry Crane on Medium]]></title>
        <description><![CDATA[Stories by Harry Crane on Medium]]></description>
        <link>https://medium.com/@HarryCrane?source=rss-af8982d697ed------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Z77mH-XRvkJF3JPY.jpg</url>
            <title>Stories by Harry Crane on Medium</title>
            <link>https://medium.com/@HarryCrane?source=rss-af8982d697ed------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:03:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@HarryCrane/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Fundamental Principle of Probability]]></title>
            <link>https://medium.com/data-science/the-fundamental-principle-of-probability-resolving-the-replication-crisis-with-skin-in-the-game-1a06a45e9231?source=rss-af8982d697ed------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a06a45e9231</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[replication-crisis]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Harry Crane]]></dc:creator>
            <pubDate>Mon, 02 Jul 2018 16:56:07 GMT</pubDate>
            <atom:updated>2019-01-10T13:01:37.612Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>If you assign a probability to an outcome happening, then you must be willing to accept a bet offered on the other side (that the outcome will not happen) at the correct implied odds.</blockquote><p>If you wouldn’t knowingly invest in a hedge fund whose manager is compensated the same for generating big returns as for going insolvent, or place a bet with a bookie who doesn’t pay up when he loses, or believe a journalist who doesn’t vet his sources, why would you trust claims made by scientists who have little or nothing to lose when their assumptions are wrong, analyses are flawed, or findings are false?</p><p>The businesses of investing, bookmaking, and news reporting are by no means perfect, but at least they have built-in mechanisms to align the interests of producer (fund manager, bookie, journalist) and consumer (investor, gambler, reader). Fund managers, bookies, and journalists have to risk their own wealth, reputation, and credibility so that they have funds to manage, books to keep, and audiences to write for. The same goes for doctors, butchers, chefs, and (non-union) plumbers. But why not scientists?</p><p>Rather than being evaluated against objectively established standards (e.g., profit, honor, and truth in the above examples), professional scientists police themselves with codes of conduct, peer review, and editorial boards,<br>affording themselves a great deal of upside — in the form of job security, a comfortable salary, millions of dollars in taxpayer-funded grants, and even the possibility of fame and fortune if one of their discoveries hits the mainstream — with the potential for downside mostly limited to exceptional circumstances, such as fraud or misconduct. This asymmetry combined with academia’s ‘publish or perish’ culture leads to a perverse incentive structure in which quick-and-dirty is rewarded more than careful-and-correct, which in turn has led to the so-called scientific ‘<a href="https://thewire.in/science/replication-crisis-science">replication crisis</a>’.</p><p>Ioannidis called attention to the adverse effects of this system more than a decade ago in an essay titled <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">Why Most Published Research Findings Are False</a>. More than a decade later, there are numerous proposals on the table to reform the way scientific inquiries are conducted, evaluated, and reported, and with it improve the reliability of published research. But even with widespread interest in reform, the replication crisis shows little sign of receding, as <a href="http://science.sciencemag.org/content/349/6251/aac4716.full">one recent effort</a> successfully replicated only 37% of results published in the psychology literature.</p><p>These problems will persist until science, like investing, bookmaking, and news reporting, implements a built-in mechanism that aligns the career success of scientists with the reliability of their results. I argue here that this is much easier to achieve than it may seem, as it requires little or no change in the way scientific inquiries are currently conducted and only minor modification in how scientific conclusions are reported. The key idea is to make conclusions reported in the scientific literature more <em>scientific</em>, in the sense of being testable and falsifiable, rather than being merely transparent or reasonable, as evaluated against a rubric of best practices or a checklist of state of the art methods. So instead of shying away from the perceived pitfalls of what has been deemed a ‘results-oriented’ approach to scientific reporting, I argue for more emphasis on results, by which I specifically mean <em>actual</em> results, e.g., clearcut criteria by which to verify or falsify published claims, instead of <em>reported</em> results, e.g., statistical significance.</p><p>It is not enough to police methods, as (well-intentioned) proposals such as <a href="https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/registered-reports-a-step-change-in-scientific-publishing">Registered Reports</a> (RRs) seek to do. The determination of which methods are ‘sound’ will be made by the same editors and referees who currently decide whether a result is ‘significant’, changing not the end goal (publication) or the mechanism (peer review) but rather the means by which that end is achieved (i.e., by applying ‘sound methods’ instead of reporting ‘significant results’). So while such implementations may change specific behaviors, e.g., by eliminating P-hacking and other forms of results-oriented misconduct, they will do so without any guarantee of eliminating the replication problems caused by those behaviors.<br> <br>The importance of correct scientific <em>results </em>over sound scientific <em>methods </em>underscores the larger societal role of science, a role which is downplayed by Registered Reports and similar proposals to shift emphasis from results to methods. Unlike purely academic disciplines such as philosophy, history, and mathematics, whose immediate impacts do not extend far beyond the ivory tower, results published in the scientific literature have direct consequences on the economy, society, medicine, business, industry, and public policy. So while focusing on methods may be a prudent approach to ensure consistent science in the aggregate, the correctness of individual studies matters to consumers who base policy and business decisions on specific conclusions in the scientific literature. The fact that an errant conclusion in cancer research, education policy, or economics was obtained using `state of the art’ methods and approved by ‘expert’ peer reviewers is no consolation to the cancer patient, special needs child, or impoverished citizen affected by misguided policies based on those flawed reports. For regardless of whether an individual scientist knowingly commits fraud, unknowingly misapplies a statistical method, or unluckily draws an errant conclusion due to random error, the end result is the same: the conclusions reported in the peer-reviewed, scientific literature are less reliable, and whoever depends on the reliability of scientific research (i.e., just about everyone) is worse off as a result.<br> <br>This is not to suggest that scientists must be 100% certain of a conclusion before they report it, as trial and error is critical to scientific progress, but consumers of those conclusions, which includes fellow scientists as well as members of the general public, should be assured that what they are reading is very likely correct, not just because the scientific community vouches for the soundness of the methods or the reasonableness of the conclusions but because the scientists asserting these claims have incentives for being right and face consequences for being wrong. This same basic tenet keeps fund managers, bookies, journalists, doctors, butchers, chefs, and plumbers (except those protected by the union) on their toes, and would do the same for scientists.<br> <br>As long as the primary objective of <em>scientists</em> (i.e., publication) is at odds with the objective of <em>science</em> (i.e., the attainment of new knowledge and insights), the reliability of the scientific literature will be influenced more by sociological than by scientific factors. To align the interests of consumers and producers of science, it is not enough to merely eliminate incentives for certain kinds of misconduct or to incentivize specific best practices. The consistent production of unreliable science (intentional or not) must become an unsustainable strategy for a scientific career, and conversely the consistent production of reliable science must become the clearest path to a successful scientific career. Put another way, the academic world that scientists inhabit needs to make contact with the real world in which their conclusions have influence.</p><h4>Academia vs. Reality</h4><p>Improving the reliability of published science goes hand-in-hand with aligning the goals of <em>academic</em> science and <em>real</em> science. Real scientific conclusions can be tested against reality, while academic science can only be evaluated by other academics. Thus, testability marks the line between academic and real science, and charts the logical path toward improving the reliability of scientific literature. Because the peer review decisions that guide publication in scientific journals are based primarily on academic criteria, it should come as little surprise that some of academic science fares rather poorly when graded against reality; see, e.g., the large-scale <a href="http://science.sciencemag.org/content/349/6251/aac4716.full">replication study in psychology</a>.</p><p>A key step in distinguishing the academic from the real lies in a clearer understanding of how scientific conclusions are often assessed, by statistical methods and their accompanying probabilities, and how these probabilities provide an organic mechanism by which to evaluate scientific claims, all without any need for peer reviewed journals, expert referees, or bureaucratic oversight. Indeed, a major contributing factor to the replication crisis is that the scientific literature makes no distinction between academic and real probabilities, a distinction critical to understanding the role played by probability in scientific discourse. <br> <br> The key point is that <em>real </em>probabilities can be tested in a way that <em>academic </em>probabilities cannot be.</p><blockquote><em>Academic probabilities</em> are theoretical calculations with only a hypothetical connection to the real world.</blockquote><p>Academic probabilities are the result of armchair analyses, as when a Bayesian philosopher explains probability in terms of a hypothetical disposition toward betting, or when a scientist assumes that the relationship between diabetes and certain presumed risk factors (e.g., age, weight, diet, etc.) follows a multivariate logistic regression, or when a statistician assumes that the test statistic computed from a clinical trial follows a <em>t</em>-distribution with a certain degrees of freedom. Initially, the probabilities emerging from these analyses are <em>academic</em> (as in ‘not of practical relevance, of only theoretical interest’) because they attain their meaning by <em>fiat</em> (an arbitrary degree or pronouncement) instead of by a concrete connection to the real world. And because these probabilities are academic, so are the conclusions those probabilities support.</p><p>Conclusions drawn from academic analyses become <em>real</em> only when the hypothetical assumptions on which they are based become tied to reality.</p><blockquote><em>Real probabilities</em> are backed by something real and are about something real.</blockquote><p>For a probability to be ‘backed by something real’ it must have a tangible guarantor, i.e., something must serve as collateral to ensure that the purveyor of the probability (‘the probabilist’) is acting in good faith. For a probability to be ‘about something real’ the statement that the probability is about must be decidable, in that there must be clear criteria to determine whether or not the object of the probability statement did or did not happen, is or is not true.</p><p>With this understanding, probabilities used in stock option pricing are academic <em>until an option is bought or sold on the basis of those probabilities</em>. Once the transaction is made, the probability becomes testable through the realized profit or loss of that transaction. In the same way, probabilities and derivatives of probabilities such as P-values, confidence intervals, etc. reported in scientific literature are academic <em>unless and until those probabilities can be tested for their accuracy</em>. (N.B. Although realizing a loss on any given transaction does not indicate that the associated probability calculation is wrong (and similarly realizing a gain on any given transaction does not indicate that the associated probability calculation is right), sustained losses (gains) over a long series of transactions provide an objective way to assess systematic error (or soundness) in these calculations.)</p><h4>The Fundamental Principle of Probability</h4><p>The distinction between academic and real probabilities gives rise to what I call the <em>Fundamental Principle of Probability</em> (FPP), which establishes the connection between academic and real probabilities by tying academic probabilities to real outcomes. The FPP is a common sense concept, intuitively understood by almost every stock trader, bookie, and man on the street:</p><blockquote><em>If you assign a probability to an outcome happening, then you must be willing to accept a bet offered on the other side (that the outcome will not happen) at the correct implied odds</em>.</blockquote><p>By the FPP, when you claim that the probability of ‘<em>A</em>’ is <em>p</em>, then you are implicitly offering odds of <em>p/(1-p)</em> for a bet on ‘not-<em>A</em>’. If I choose to take you up on your offer, then you win whatever money I risked if <em>A</em> happens, and I win <em>p</em> of your dollars for every <em>1-p</em> dollars I risked if <em>A</em> does not happen.</p><p>According to the FPP, real probabilities consist of three ingredients:</p><blockquote>(I) A decidable statement <em>A </em>to which the probability statement applies.</blockquote><blockquote>(II) An exposure limit, i.e., the maximum amount of money that one is willing to risk on the probability assessment of <em>A</em>.</blockquote><blockquote>(III) A probability <em>p </em>that the assertion <em>A </em>turns out to be true.</blockquote><p>Academic probabilities often fail to satisfy (I) and almost always fail to satisfy (II). In current academic science, the exposure limit in (II) is implicitly set to 0, which according to the FPP conveys no confidence in the probability assessment, and therefore no credibility to its associated conclusion. To make scientific conclusions more credible, scientists need to up the ante on their claims. It may well be that the scientist is convinced by this own analysis and provides a compelling argument in its favor. But the probabilities backed up by these words are meaningless unless backed up by the FPP.</p><p>Technically, the probability <em>p </em>in (III) is a ‘lower probability’ for <em>A</em>, in the sense that it determines the odds <em>p/(1-p)</em> of a bet <em>against</em> <em>A. </em>(Note that it is not assumed that <em>p </em>is the price of a ‘fair bet’, as offering fair bets would expose the probabilist to potentially ruinous fluctuations in wealth without any potential long-run gain. The purpose of the FPP is to make probabilities meaningful, not to force scientists to gamble or expose themselves to ruin.) If <em>p</em> is a legitimate lower bound on the probability of <em>A</em>, then the implied odds <em>p/(1-p)</em> for a bet against <em>A </em>will be favorable to the probabilist in the long run.<br>If <em>p </em>is too optimistic, however, the implied odds will overpay for losses, causing the probabilist to suffer financial loss in the long run and providing evidence against the initial probability assessment.</p><h4>Academic probabilities and the replication crisis</h4><p>At its core, the replication crisis is a statistical and probabilistic crisis, caused by a combination of misapplied statistical methods, misinterpreted probabilities, and unchecked misconduct. The crisis is allowed to persist by confusing academic probabilities, and the conclusions they are claimed to support, for something real. The FPP provides a built-in mechanism that forces users of statistical and probabilistic methods to report more meaningful probabilities (or else report no probabilities at all), without any need for validation by editors, peer reviewers, or other so-called experts.</p><p>To appreciate how implementing the FPP would affect the reporting and evaluation of scientific conclusions, first consider the role it plays in the casino industry. With the exception of professional advantage players, it is well known that casino games are unfavorable to the gambler. (Footnote: ‘Advantage players’ are gamblers who bet with the odds in their favor. Perhaps the best known example of advantage play is card counting in blackjack. More recently, <a href="http://www.espn.com/espn/chalk/story/_/id/11665414/phil-ivey-loses-124m-battle-court-being-accused-improperly-withholding-winnings">Phil Ivey and Cheung Yin Sun</a> won more than $20 million dollars using a technique called <a href="https://en.wikipedia.org/wiki/Edge_sorting">edge sorting</a> to get an advantage while playing baccarat. Other examples of advantage play in sports betting, video poker, roulette, craps, and other ‘unbeatable’ casino games are discussed in Bob Dancer and Richard Munchkin’s `<a href="https://www.youtube.com/user/2RWM">Gambling with an Edge</a>’ podcast.) Importantly, this knowledge comes from common sense, and not from a formal statistical analysis of casino games (an analysis which the average person is ill-equipped to carry out). It is obvious that the odds offered by any major casino must favor the house by the simple facts that (1) the casino willingly accepts any bet at its stated odds and (2) casinos are profitable businesses.</p><p>With respect to the specific odds offered at specific casinos, one can ask how I know that a 7 is rolled with less than 20% frequency at the craps tables at Australia’s Crown Melbourne Casino. I claim to know this even though I’ve never visited the Crown Melbourne, much less inspected its dice, tables, or dealers to confirm that the game is (more or less) fair. And as emphasized above, I’m not basing my knowledge on a theoretical STAT 101 calculation — under the standard model of two ‘fair dice’ the probability of rolling a 7 is 16.7% — since I have no way to assess the assumptions behind that calculation. In fact, the exact probability (i.e., frequency) of rolling a 7 at the Crown Melbourne very well might be 16.7%, or 18%, or 14%. I don’t know, and I don’t care. No matter what the actual frequency is, I’m pretty sure that it must be less than 20%. And I know this because the Crown Melbourne offers odds of 4-to-1 (an implied probability of 1/(4+1)=0.20) for a one roll bet on 7 <em>and the Crown Melbourne is still in business</em>. Whether the true frequency is 16.7%, 18%, or 14%, as long as it is lower than 20%, the casino will win in the long run, and the gambler will lose.</p><p>Because of the survival mechanism built into the FPP — casinos that state bad probabilities either revise their odds or go bankrupt — and the observation that the Crown Melbourne casino is still surviving (in fact, thriving), I can be reasonably sure that the probability bound implied by its odds is legitimate without understanding how the Crowne Melbourne determines its odds and without even knowing the rules of craps. I know this all because the casino’s probabilities are real: if the implied odds were too high, the casino would go broke; if they were too low, then the casino would have too few customers.</p><p>If we don’t need inside information to determine that the casino’s odds give valid probability bounds for the outcomes of its games, why do we need editors, associate editors, and referees to evaluate the soundness of methods that scientists use to compute P-values, confidence intervals, Bayes factors, and other probabilistic measures of evidence? Just like casinos don’t offer overly generous odds to attract more customers (because they would quickly go bankrupt), wouldn’t it be nice if scientists didn’t even try to publish hacked or overly optimistic P-values or other probabilities, because doing so would put them out of the scientific business? It’s a simple idea with a simple implementation thanks to the Fundamental Principle of Probability.</p><h4>Replication and the Fundamental Principle of Probability</h4><p>In scientific terms, every roll of the dice, spin of the roulette wheel, and hand of blackjack is a test of the casino’s stated odds, and thus also the probabilities implied by those odds. Every bet against the house is an attempt to falsify the casino’s implicit probability claims. And the fact that casinos consistently profit from these games, and gamblers consistently lose, can (and should) be interpreted as compelling evidence that the odds are set correctly. (Footnote: By ‘correctly’ here I mean that the probabilities implied by those odds give a valid lower bound on the true probability (i.e., frequency) of occurrence.)</p><p>Like casino odds, bureaucratic assessment of scientific methods would become unnecessary if scientific conclusions were also backed by <em>real </em>probabilities. Overly optimistic probabilities about the reliability (i.e., replicability) of a given finding would incentivize replication attempts; and overly conservative probabilities would understate the importance and reliability of a given conclusion. Accordingly, the Fundamental Principle of Probability suggests a straightforward and objective way to report conclusions and carry out replication attempts in the scientific literature by laying out the following conditions when reporting their conclusions.</p><blockquote><strong><em>Replication Protocol 1</em></strong><em>. Describe the protocol under which a replication attempt should proceed.</em></blockquote><blockquote><strong><em>Replication Criteria 1</em></strong><em>. State the criteria on which the results of the protocol will be considered a successful replication.</em></blockquote><blockquote><strong><em>Replication Probability 1</em></strong><em>. State the probability that this procedure will result in successful replication.</em></blockquote><blockquote><strong><em>Replication Exposure 1</em></strong><em>. State an amount of money that the scientists behind these claims are willing to expose for tests of their conclusions, in accordance with item (II) of the FPP above, and put this amount of money in escrow.</em></blockquote><blockquote><em>Continue with additional Replication Protocols, Criteria, and Probabilities 2, 3, 4, etc. as desired.</em></blockquote><p>The stated protocols, criteria, and probabilities provide a concrete basis on which to test findings according to the FPP. This in turn constrains the stated replication probability by forcing the authors, their institutions (if any), the journal publishing the work (if any), and any editors, associate editors, and referees involved in the publication decision to back up their claims with something real. (Depending on the field, this collateral could be $500, $5,000, $50,000, or $500,000 per claim. The appropriate amount of ‘skin in the game’ should be roughly proportional to the cost of carrying out the research and what the involved parties stand to gain from the publication. The point is that this collateral should be sufficiently large that the authors should not be willing to lose the money in exchange for publication. So if it cost $1 million (most or all of which is usually taxpayer money) to perform the research, then it is reasonable to post an additional $500K to back up the claims of that research. If the research costs only $1K, then $500 should be sufficient. As a practical matter, at least part of the cost of the replication attempt, performed by an agreed upon third party, should be defrayed by this collateral.)</p><p>After publication, the collateral is locked in escrow for a fixed period of scrutiny (say 2 years) during which other scientists can choose to bet against replication at the odds implied by the stated replication probability. Importantly, the replication attempt must occur after the skeptic has decided to bet against replication and has placed his/her money in escrow. This replication attempt should be carried out by a trusted, neutral third-party team of scientists, for which the costs of carrying out the replication are funded in whole or in part by the initial scientific team and the journal publishing the claim. A scientific team (a ‘skeptic’) wishing to test the claimed replication probability will put the amount they are willing to risk in escrow while the replication attempt is carried out. In doing so, the skeptics agree to the terms stated in the original authors’ replication protocol and criteria, which would set in motion a replication attempt by an agreed upon neutral third party. The outcome of that replication attempt determines the outcome of the bet between the scientists making the claims (the probabilists) and those testing the claims (the skeptics). (Footnote: The precise protocol by which such third parties will be chosen and the truthfulness of the conclusions assured is a practical issue of much the same flavor as ensuring that the current peer review process is properly administered. I don’t discuss this further here.)</p><p>This process continues for the pre-specified period of scrutiny or until the escrow money runs out. If the escrow money runs out, then the authors either post more collateral or the result is archived with clear documentation of the outcome of the replication attempts along with a record of the author’s net monetary gain/loss. Authors who nevertheless stand by their claims could post more escrow money to encourage additional testing of their results, in hope of validating their initial findings. After the period of scrutiny is over, regardless of the outcome the results of the replication attempts are published along with the final tally of gains/losses at the stated replication probabilities.<br>It is important to note that while financial gains/losses throughout the replication process do not necessarily indicate that the initial claims are right/wrong, they can be reasonably interpreted as evidence for/against the initial claims, and such evidence should be documented in the published scientific record.</p><p>The authors are free to revise their probabilities (upward or downward) at any point during the period of scrutiny, and the odds at which the replication studies are offered will be revised accordingly. For example, if the initially stated replication probability of 80% is too conservative (e.g., the true replication probability should be 90%), then there may be few attempts to replicate the initial claims, because the odds against replication are unfavorable. In response to this, the authors may wish to increase their replication probability to 85%, serving two beneficial purposes from the standpoint of the original authors. First, it strengthens the original claims — because the probabilities stated by the authors are <em>real</em> in the sense of being backed by the FPP, the higher replication probability makes the original claims stronger. Second, the higher probability of replication offers a better price to anyone wishing to challenge the results (5.6-to-1 instead of 4-to-1), and so could entice more activity in trying to replicate/falsify the original results. In turn, successful replication makes the claims in the article more credible. If the authors’ assessment of the true probability at 90% is correct, these replication attempts should generate additional research funds in the long term. Furthermore, successful attempts to replicate will make the authors’ findings more credible.</p><p>Alternatively, the authors could adjust their replication probabilities downward, say from 80% to 70%, if they determine that their initial probabilities were overly optimistic. Either way, if the stated conclusions are reliable, and the probabilities are well-calibrated, then the scientists, their universities, and the journals behind good and precise science will be rewarded, both financially and academically, for drawing sound conclusions. If the conclusions are unreliable, then these scientists, the universities who support them, and the journals who publish bad science will be penalized.</p><h4>What will come of this?</h4><p>The above process leverages the built-in correction mechanism of statistical methods (through the FPP) to improve the reliability of scientific claims based on statistical and probabilistic methods: authors should not state overly conservative probabilities, or else there will be little interest in their conclusions and few attempts to replicate their findings; but they should also not state overly optimistic probabilities, or else they are offering too good of a price for replication attempts that are likely to cost them money in the long run. Such financial loss has the direct impact of depriving researchers who publish overly optimistic replication probabilities of research funding. Also, because these outcomes are documented in the final published version of the results, the failed replication attempts may also lead to a loss in credibility of the researchers involved in the failed claims. Conversely, a gain of funds from the replication process will have the opposite (positive) effect on the reputation and financial status of the scientists behind the initial claims. Most important of all, the scientific literature should become more accurate and more complete as a result.</p><p><em>Stated replication probabilities should be more conservative</em></p><p>Assuming its dice are fair, the casino knows that the true odds of rolling 7 are 5-to-1 (implied probability of 16.7%), but it offers only 4-to-1 (implied probability of 20%). In terms of the replication probabilities above, the casino is claiming that the probability of <em>not</em> rolling a 7 is 80% even though the theoretical probability is 83.3%. A casino that overstates the probability, say, at 85% (for 5.6-to-1 implied odds), would suffer financial loss as a result.</p><p>By analogy, the scientist who uses a statistical method to compute the probability of replication at 90% is incentivized to quote a more conservative probability, say 80% or 85%, in publication. These understated probabilities account for uncertainty due to assumptions of the model, approximations, and other factors beyond the scientists’ control. Scientists who publish a higher figure to convey confidence in the outcome — the academic equivalent of a bluff — risk losing the money posted as collateral when their bluff is called. This in turn will create a negative impression of their stated conclusions and deprive these scientists of resources for future research. If, however, the probability is too conservative (say, 25% instead of 80%), then the claim will be too weak to generate sufficient interest in replication, which in turn undermines the significance of the reported finding.</p><p><em>Probabilities reported in the literature will be more meaningful</em></p><p>Even ignoring the divide between academia and reality, the specific probabilities that currently appear in scientific writing are often interpreted qualitatively. Without the opportunity to test probabilities or model assumptions (according to the FPP), there is little conceptual difference between a probability of 90% or 95%. Both probabilities are high, and would usually be interpreted as evidence in support of a given claim. <br>But when bound by the FPP there is a big difference between 90% (9-to-1 odds) and 95% (19-to-1 odds). If the true probability is somewhere in between, say 92.5%, then stating 90% instead of 95% is the difference between a positive and a negative return on the funds put in escrow, giving the reported numerical probabilities a more precise meaning.</p><p><em>Increased replication attempts</em></p><p>Rewarding scientists for correctly identifying overly optimistic published claims provides an incentive to spearhead replication attempts. Aside from gaining better clarity about the published literature, researchers who call for the replication of flawed claims will be rewarded with additional funds for their own research. Contrast this to the current situation in which a number of people bemoan the fact that replication attempts are too few because there are no incentives (financial or otherwise) for replicating the results of other scientists.</p><p><em>No protection for ‘normal science’</em></p><p>Because of the way scientific results are currently evaluated, by the gatekeepers in charge of peer-reviewed journals, <a href="https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions">there is a tendency for findings that go against the<em> status quo</em> to be suppressed</a>. Shifting from academic to real probabilities, and therefore academic to real science, would eliminate the need to suppress anything that is submitted for publication in the scientific literature. The scientific establishment can easily refute outrageous and bogus claims on objective grounds by betting against replication at the implied odds. If the establishment is correct, then the scientists behind the inaccurate claims will suffer financial loss, and the establishment a financial gain; and conversely if the establishment is incorrect. In this way, controversial results that go against the scientific paradigm need not be filtered out of the literature, as is currently the norm, but rather can be put under scrutiny in a transparent and objective manner, further strengthening the predominant paradigm when these new results are refuted and expediting progress when the controversial results stand up to scrutiny.</p><h4>Investing in better science</h4><p>Some critics will cry foul that science isn’t a betting game and that the financial incentives of the above proposal will corrupt science. But while I’ve used the language of betting here for illustration, the proposal is better thought of as a mechanism for investing in good science/scientists and divesting from bad science/scientists, with the intention to improve the overall quality and reliability of published science. Such a result would clearly improve, not corrupt, the current state of science. The way it works right now is that scientists report fiat probabilities, and implicitly state odds, with few consequences for passing bad probabilities, fake probabilities, counterfeit probabilities, or fraudulent probabilities. The result is the replication crisis.</p><p>For good scientists, good journals, good editors, and good referees, the proposal here is an investment in good science. Better scientists will be rewarded with more resources to do more, better science. Bad scientists will be penalized by losing resources, so that they must do less science, or else re-focus their efforts on quality over quantity. Similarly, universities that employ good scientists and journals that consistently publish reliable science will be rewarded. Those that support bad science from bad scientists will be penalized. Instead of letting universities skim millions of dollars of taxpayer money as ‘indirect costs’ from grants, funding agencies should mandate that some or all of those indirect costs first be invested in the science conducted under that grant, as part of the escrow. Whatever share of that escrow money is left after the period of scrutiny goes to the university in indirect costs. If the escrow money is lost, then the university gets no overhead. The indirect costs instead go to the institutions of the scientists who correctly invested in attempts to falsify the bad conclusions. As a result, taxpayers who fund the research will get a better return on their investment because their tax money will be automatically allocated to more prudent, honest researchers based on results, instead of on the subjective determination of a grant panel, editorial board, or anonymous referee.</p><p>And as far as incentives are concerned: are the incentives of prestige, tenure, promotion, and money not already a factor in how scientists behave? Aren’t the arbitrary ways in which these awards are conferred at least partly to blame for rampant P-hacking, fraud, and other questionable research practices in the scientific literature? In fact, these sociological factors are often cited as the catalyst for the ‘results-oriented’ attitude that spawns the replication crisis, and which the proposal here seeks to correct. Even with the FPP, the same incentives will be present, but the FPP offers improvement by making it harder to achieve these incentives by asserting spurious, careless, or fake probabilities.</p><h4>Reforming science with real probability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*y9UPKn-SPZ5YhqFR.jpg" /><figcaption>Courtesy of <a href="http://lowres.jantoo.com/business-stat-statistician-lie-liar-statistic-01436756_low.jpg">http://lowres.jantoo.com/business-stat-statistician-lie-liar-statistic-01436756_low.jpg</a></figcaption></figure><p>When <a href="https://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics">Disraeli</a> spoke of “lies, damn lies, and statistics’’ he was referring to statistics in the absence of the FPP, or to probability without “<a href="https://en.wikipedia.org/wiki/Skin_in_the_Game_(book)">skin in the game</a>’’. The suggestion here to merge probabilities reported in the scientific literature with the FPP is a proposal to improve the reliability of the scientific literature by giving real meaning to reported probabilities, and thus real meaning to the conclusions based on them, all without imposing any extra requirements, administrative work, or other restrictions on the way research is conducted. Rather than being burdened by additional constraints, scientists will be liberated to run studies as they deem appropriate, without the need to apply a specific method in order to appease a referee or be published in a certain journal. With this freedom comes accountability for reported results, as ultimately the stated replication probability will have to stand on its own merits when tested according to the scientists’ prescribed replication protocol. This system is designed so that good scientists succeed, bad scientists fail, and conclusions reported in the literature are more reliable on the whole.</p><p>When implemented in this way, the FPP might be thought of as the Hippocratic Oath of probability, an oath not to expose others to the potential harm of statistical methods without exposing oneself to that same downside risks. Those uncomfortable taking the risks associated with their methods shouldn’t be evaluating their results using probabilities, just like those who can’t stand the sight of blood aren’t cut out for medicine and athletes with a weak chin aren’t cut out to be boxers.</p><h4>Technical notes:</h4><ol><li>In <a href="https://thejollyswagmen.com/new-blog/harrycrane">earlier versions</a>, I discussed the FPP under the name “First Rule of Probability”.</li><li>The interpretation of probabilities as prices of bets is fundamental to the ‘radical subjective Bayesian’ (RSB) philosophy of de Finetti, Savage, and others. The difference between the RSB, as a philosophical stance, and the FPP, as a practical principle, is the same difference between academia and reality. RSB probabilities are merely hypothetical degrees of belief with only an academic connection to betting, while real probabilities are actual betting quotients, as realized by the real consequences of stating those probabilities (the potential for financial loss or gain).</li><li>The FPP gives a probability statement real meaning by forcing the purveyor of probability to act as the ‘house’ for anyone wishing to test the probability assessment. The probabilist may also wish to state a probability for the contrapositive of <em>A </em>(i.e., that <em>A</em> does not happen), written -<em>A</em>.<br>In this case, the lower probability <em>p* </em>for -A offers an implied odds of <em>p*/(1-p*)</em> for a bet on <em>A</em>. Together the probabilities <em>p </em>and <em>p* </em>give a range for the actual probability of <em>A </em>and<em> -A</em>:</li></ol><blockquote>p ≤ P(A) ≤ 1-p*<br>p* ≤ P(-A) ≤ 1-p.</blockquote><p>If these are not proper intervals, meaning that <em>p+p*&gt;1</em>, then the probabilist exposes himself to sure loss, as I can bet <em>1-p</em> on <em>A</em> at odds of <em>p/(1-p)</em> and <em>1-p*</em> on -<em>A</em> at odds of <em>p*/(1-p*)</em>. If <em>A </em>happens, then I win <em>(1-p) </em>x<em> (p/(1-p))=p</em> for my bet on <em>A </em>and lose <em>1-p*</em> for my bet on -<em>A</em>, for a total gain of <em>p-(1-p*)=p+p*-1&gt;0</em>. If -<em>A</em> happens, then I win <em>(1-p*) </em>x<em> (p*/(1-p*))=p*</em> for my bet on -<em>A</em> and lose <em>1-p</em> for my bet on <em>A</em>, for a total gain of <em>p*-(1-p)=p+p*-1&gt;0</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a06a45e9231" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-fundamental-principle-of-probability-resolving-the-replication-crisis-with-skin-in-the-game-1a06a45e9231">The Fundamental Principle of Probability</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The modern-day snake oil salesman]]></title>
            <link>https://medium.com/@HarryCrane/the-modern-day-snake-oil-salesman-2ca934cc2ab?source=rss-af8982d697ed------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ca934cc2ab</guid>
            <category><![CDATA[donald-trump]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[politics]]></category>
            <dc:creator><![CDATA[Harry Crane]]></dc:creator>
            <pubDate>Fri, 02 Dec 2016 17:08:43 GMT</pubDate>
            <atom:updated>2017-04-19T01:39:37.918Z</atom:updated>
            <content:encoded><![CDATA[<p>Nate Silver, founder of <a href="http://www.fivethirtyeight.com/">FiveThirtyEigh</a>t and poster boy of data-driven election predictions, is on fire. He’s always right, even when he’s wrong.</p><p>In 2008, Silver gave Obama a 94% probability of winning. He got it right.</p><p>In 2012, Silver gave Obama a 91% probability of winning. He got it right.</p><p>In 2016, Silver gave Trump a 28% probability of winning. He got it right.</p><p>That’s right. Silver had Clinton at 72%, Trump at 28%. Trump won. And Silver was right?! That’s right.</p><p>And I suppose Silver would have also been right if Clinton had won instead? That’s right.</p><p>Can he ever be wrong? Do his (or anyone else’s) probabilities mean anything at all?</p><p>We’re told that they do. Statistics and data science programs are booming at the top universities; data scientists are a hot commodity in Silicon Valley and on Wall Street; and academic jargon like ‘big data’, ‘machine learning’, and ‘analytics’ has infiltrated our everyday vernacular. Not to mention the hundreds of millions of dollars, perhaps more, of federal taxpayer dollars spent on data-driven research each year.</p><p>All of these trends empower modern data zealotry, which casts those dubious of data-driven methods into the same (deplorable) basket as the tin foil hat wearers, moon landing skeptics and illuminati alarmists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*2HdpaQ6rVkS3I6BipIb5hg.gif" /></figure><p>How dare you question something as objective, as cut and dry, as the data, as the statistics!</p><p>Based on an argument from the theory of option pricing and stochastic calculus, <a href="http://math.stackexchange.com/questions/1911110/what-is-nassim-talebs-criticism-of-538s-election-model">Nassim Taleb</a> dared to call FiveThirtyEight “quite incompetent (and aggressively so).” But contrary to Taleb’s criticism, Silver isn’t incompetent. He’s just good. Really good.</p><p>He’s good precisely because he’s not pricing or selling options or, in Taleb’s parlance, putting any “skin in the game” to back up his predictions. He’s good because he has turned probabilities into a commodity, a consumer good, a media empire. And he’s really good because even when his predictions seem to fail, he doesn’t change a thing. He justifies why they failed — or why they didn’t — with the same logic that led him astray in the first place. And gets away with it.</p><p>As if that weren’t enough, Silver also has the public backing of some well-regarded academic statisticians. The <a href="http://simplystatistics.org/2016/11/09/not-all-forecasters-got-it-wrong/">Simply Statistics</a> blog shilled for Silver in its November 9 entry: “Statistically speaking, Nate Silver, once again, got it right,” wrote bloggers Rafa Irizarry, Roger Peng and Jeff Leek. (Unfortunately, the Simply Statistics argument seems to be largely based on the assumption that Silver’s credible intervals are 95% when they are in fact 80%. So, statistically speaking, Irizarry, Peng and Leek are wrong. Or, by their same logic, maybe that makes them right, you know, statistically speaking? I digress.)</p><p>With no skin in the game, Silver’s forecasts might still be evaluated, not based on dollars and cents, but rather based on how “calibrated” they are. Roughly, a forecaster is well-calibrated if his 10% predictions occur 10% of the time, his 20% predictions occur 20% of the time, and so on. This same idea of calibration is related to <a href="http://andrewgelman.com/2016/11/11/election-surprise-three-ways-thinking-probability/">Andrew Gelman</a>’s “slightly sophisticated view” of the election outcome and also the admonishment by <a href="http://simplystatistics.org/2016/11/09/not-all-forecasters-got-it-wrong/">Simply Statistics</a>, “Remember, if in ten election cycles you call it for someone with a 70% chance, you should get it wrong 3 times.”</p><p>By this metric, a well-calibrated forecaster is a credible forecaster since when he forecasts an event at, say, 30%, the event occurs 30% of the time in the long run. Seems reasonable? At the very least, this gives the forecasts some meaning, right?</p><p>Not so fast.</p><p>In their work on game-theoretic probability, <a href="http://www.probabilityandfinance.com/articles/08.pdf">Vladimir Vovk, Akimichi Takemura and Glenn Shafer</a> prove that a forecaster can be well-calibrated, regardless of what happens in reality and without any knowledge of future events, by using a strategy called “defensive forecasting”. To assess calibration, forecasts are compared against certain test functions. For example, the accuracy of 20% predictions is evaluated based on the performance of not just the 20% forecasts but also the 19% forecasts, 18% forecasts, 21% forecasts, and so on, with forecasts closer to 20% given a higher weight. As long as these test functions are continuous — meaning the weights do not change too abruptly as forecasts vary — defensive forecasting ensures calibration in the long run by simply choosing probabilities that correct for past deviations.</p><p>In guaranteeing well-calibrated forecasts without any knowledge of future events, defensive forecasting seems to explain how predictions, such as FiveThirtyEight’s 28% Trump probability, can be both “correct” — in terms of calibration — and meaningless — offering no substantive insight about the real world — at the same time.</p><p>To overcome this criticism, a forecaster with guts (not Silver) might offer bets based on his predictions. (Silver’s 72% probability for Clinton translates to a bet on Clinton at 0.39 to 1 odds, for instance.) But even “skin in the game” may not be enough to sniff out a charlatan forecaster. In this context, the test functions from the above game-theoretic framework become gambling strategies against the forecaster, and the requirement that these strategies are continuous acts as a constraint against erratic betting behavior. Overall, the Vovk-Takemura-Shafer theory suggests that as long as the gambler is tame — if he’s predictable, if he follows the herd, if he’s a sheep — the forecaster can exploit that tameness without any clue about the likelihood of future outcomes and, in fact, without even knowing what event is being bet on. All the forecaster needs to do is offer bets that prevent the gambler from making money, which is possible against any continuous strategy. To have any chance at making money, the gambler can’t be docile. He needs to be a little wild, a bit irregular, even volatile. Make the forecaster uncomfortable. Throw him off balance.</p><p>So, returning to my rude question: Do Silver’s probabilities mean anything at all?</p><p>Both common sense and basic mathematical intuition suggest not.</p><p>Since ancient times, people have relied on prophets, seers, fortune tellers, and psychics to foretell the future. How naïve and unsophisticated those cultures were! We’re so much more advanced now. We have Nate Silver. Instead of looking into a crystal ball, Silver looks into the data. Instead of consulting with the gods, he consults with mathematical models. He doesn’t sell snake oil, he sells probabilities. His predictions are often misinterpreted, but never wrong.</p><p>We’re left with a story comical and tragic all at once: Nate Silver is our version of the Oracle of Delphi, of Nostradamus, of The Amazing Criswell.</p><p>Prove me wrong.</p><p>Or, even better, prove Silver wrong.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ca934cc2ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>