<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Igor Šegota on Medium]]></title>
        <description><![CDATA[Stories by Igor Šegota on Medium]]></description>
        <link>https://medium.com/@igor-s?source=rss-e5f8ebca4ad8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-oBizPY2fDlRVekaj78GFA@2x.jpeg</url>
            <title>Stories by Igor Šegota on Medium</title>
            <link>https://medium.com/@igor-s?source=rss-e5f8ebca4ad8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:25:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@igor-s/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unbox the Cox: A Hidden Dark Secret of Cox Regression]]></title>
            <link>https://medium.com/data-science/unbox-the-cox-a-hidden-dark-secret-of-cox-regression-fb432137fd6?source=rss-e5f8ebca4ad8------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb432137fd6</guid>
            <category><![CDATA[survival-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[cox-regression]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Igor Šegota]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 17:26:29 GMT</pubDate>
            <atom:updated>2023-06-27T17:26:29.589Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why do perfect predictors result in a p-value of 0.93?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-pFisjQEIExfrQbn" /><figcaption>Photo by <a href="https://unsplash.com/pt-br/@pechka?utm_source=medium&amp;utm_medium=referral">Dima Pechurin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Diving into the perfect predictors</h3><p>If you have been following my previous blog posts, you might recall that logistic regression encounters a problem when trying to fit perfectly separated data, <a href="https://medium.com/towards-data-science/logistic-regression-faceoff-67560de4f492">leading to an infinite odds ratio</a><em>. </em>In Cox regression, where hazard replaces odds, you might wonder if a similar issue arises with perfect predictors. It does occur, but unlike logistic regression, it is much less apparent how this occurs here and even what constitutes “perfect predictors”. As will become more clear later, perfect predictors are defined as predictors <em>x</em> whose ranks exactly match the ranks of event times (their Spearman correlation is one).</p><p>Previously, on “Unbox the Cox”:</p><p><a href="https://towardsdatascience.com/unbox-the-cox-intuitive-guide-to-cox-regressions-c485408ae15d">Unbox the Cox: Intuitive Guide to Cox Regressions</a></p><p>we explained maximum likelihood estimation and introduced a made-up dataset with five subjects, where a single predictor, <em>x</em>, represented the dosage of a life-extending drug. To make <em>x</em> a perfect predictor of event times, here we swapped the event times for subjects C and D:</p><pre>import numpy as np<br>import pandas as pd<br>import plotnine as p9<br><br>from cox.plots import (<br>    plot_subject_event_times,<br>    animate_subject_event_times_and_mark_at_risk,<br>    plot_cost_vs_beta,<br>)<br><br>perfect_df =  pd.DataFrame({<br>    &#39;subject&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;, &#39;E&#39;],<br>    &#39;time&#39;: [1, 3, 4, 5, 6],<br>    &#39;event&#39;: [1, 1, 1, 1, 0],<br>    &#39;x&#39;: [-1.7, -0.4, 0.0, 0.9, 1.2],<br>})<br><br>plot_subject_event_times(perfect_df, color_map=&#39;x&#39;)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*HuuyVD_CbGVQBZcXGASmtw.png" /><figcaption>Image by the author.</figcaption></figure><p>To understand why these “perfect predictors” can be problematic, let’s pick up right where we left off and check the negative log-likelihood cost plotted against <em>β</em>:</p><pre>negloglik_sweep_betas_perfect_df = neg_log_likelihood_all_subjects_sweep_betas(<br>    perfect_df,<br>    betas=np.arange(-5, 5, 0.1)<br>)<br>plot_cost_vs_beta(negloglik_sweep_betas_perfect_df, width=0.1)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*RUcWfvmhmd-xZteuZ-ftrA.png" /><figcaption>Image by the author.</figcaption></figure><p>You can see right away that there is no minimum value of <em>β</em> any longer: if we use very large negative values of <em>β</em>, we end up with log-likelihood fits that are near-perfect for all events.</p><p>Now, let’s dive into the math behind this and take a look at the likelihood of event A. We’ll dig into how the numerator and denominator change as we tweak <em>β</em>:</p><figure><img alt="eq_LA_perfect.png" src="https://cdn-images-1.medium.com/proxy/1*1DgX86zpaVXL0Y8ZrCPmPg.png" /></figure><p>When <em>β</em> is high or a large positive number, the last term in the denominator (with the largest <em>x</em> of 1.2), representing the hazard of subject E, dominates the entire denominator and becomes exceedingly large. So, the likelihood becomes small and approaches zero:</p><figure><img alt="eq_LA_largebeta.png" src="https://cdn-images-1.medium.com/proxy/1*171MJfZ4C1mYctqF4GeTJQ.png" /></figure><p>This results in a big negative log-likelihood. The same thing happens for each individual likelihood because the last hazard of subject E, will always exceed any hazard in the numerator. As a result, the negative log-likelihood increases for subjects A to D. In this case, when we have high <em>β</em>, it brings down all the likelihoods, resulting in poor fits for all events.</p><p>Now, when <em>β</em> is low or a big negative number, the first term in the denominator, representing the hazard of subject A, dominates since it has the lowest <em>x</em> value. As the same hazard of subject A also appears in the numerator, the likelihood L(A) can be arbitrarily close to 1 by making β increasingly negative, thereby creating an almost perfect fit:</p><figure><img alt="eq_LA_largenegbeta.png" src="https://cdn-images-1.medium.com/proxy/1*p1gHyCG5PMIIBd8xfZgqKA.png" /></figure><p>The same deal goes for all the other individual likelihoods: negative <em>βs </em>now boost the likelihoods of all events at the same time. Basically, having a negative<em> </em>β doesn’t come with any downsides. At the same time, certain individual hazards increase (subjects A and B with negative<em> </em>x), some stay the same (subject C with<em> x = </em>0), and others decrease (subject D with positive<em> </em>x). But remember, what really matters here are ratios of hazards. We can verify this hand-waving math by plotting individual hazards:</p><pre>def plot_likelihoods(df, ylim=[-20, 20]):<br>    betas = np.arange(ylim[0], ylim[1], 0.5)<br>    subjects = df.query(&quot;event == 1&quot;)[&#39;subject&#39;].tolist()<br>    likelihoods_per_subject = []<br>    for subject in subjects:<br>        likelihoods = [<br>            np.exp(log_likelihood(df, subject, beta))<br>            for beta in betas<br>        ]<br>        likelihoods_per_subject.append(<br>            pd.DataFrame({<br>                &#39;beta&#39;: betas,<br>                &#39;likelihood&#39;: likelihoods,<br>                &#39;subject&#39;: [subject] * len(betas),<br>            })        <br>        )<br>    lik_df = pd.concat(likelihoods_per_subject)<br>    return (<br>        p9.ggplot(lik_df, p9.aes(&#39;beta&#39;, &#39;likelihood&#39;, color=&#39;subject&#39;))<br>        + p9.geom_line(size=2)<br>        + p9.theme_classic()<br>    )    <br><br>plot_likelihoods(perfect_df)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*K-D_4_W5yW86lyjWZukJFw.png" /><figcaption>Image by the author.</figcaption></figure><p>The way likelihoods are put together, as a ratio of a hazard to the sum of hazards of all subjects still at risk, means that negative <em>β</em> values make a <strong>perfect fit for the likelihood of each subject whose event time rank is greater than or equal to the predictor rank</strong>! As a side note, if <em>x</em> had a perfect <em>negative</em> Spearman correlation with event times, things would be flipped around: arbitrarily positive <em>β</em>s would give us arbitrarily good fits.</p><h3>Misaligned predictor and time ranks</h3><p>We can actually see this and show you what goes down when event time ranks and predictor ranks do not line up using a another made-up example:</p><pre>sample_df = pd.DataFrame({<br>    &#39;subject&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;, &#39;E&#39;, &#39;F&#39;, &#39;G&#39;, &#39;H&#39;],<br>    &#39;x&#39;: [-1.7, -0.4, 0.0, 0.5, 0.9, 1.2, 1.3, 1.45],<br>    &#39;time&#39;: [1, 2, 4, 3, 5, 7, 6, 8],<br>    &#39;rank_x&#39;: [1, 2, 3, 4, 5, 6, 7, 8],<br>    &#39;event&#39;: [1, 1, 1, 1, 1, 1, 1, 0],<br>})<br><br>sample_df</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*H7qTZ5oORK764wbkfh6ZuQ.png" /></figure><p>In this particular example, the time column ranges from 1 to 8, where each value represents its own rank. We also have an x_rank column, which ranks the predictors <em>x</em>. Now, here&#39;s the key observation: for subjects D and G, their x_rank is actually higher than their corresponding time rank. As a result, the likelihoods of D and G won&#39;t experience the cancellation effect between the numerator and denominator when we have large negative values of <em>β</em>:</p><figure><img alt="eq_LD_nonperfect.png" src="https://cdn-images-1.medium.com/proxy/1*NcJ6q6mdYTH8UrCcnCD9-w.png" /></figure><figure><img alt="eq_LD_nonperfect2.png" src="https://cdn-images-1.medium.com/proxy/1*OCAdvO2g2tB0mt0qmV5--w.png" /></figure><p>Their likelihoods are now maximal at some intermediate finite values of <em>β</em>. Let’s take a look at a plot of individual likelihoods to see this:</p><pre>plot_likelihoods(sample_df)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*QZM5c-wQSFR8GEty1s5M5g.png" /><figcaption>Image by the author.</figcaption></figure><p>These “misaligned” ranks between a time and a predictor play a crucial role: they stop all likelihoods from essentially collapsing into one when we have significantly negative <em>β</em>s.</p><p>In conclusion, in Cox regression, in order to obtain a finite coefficient <em>β</em> for a predictor <em>x</em>, we require at least one instance where the rank of the predictor <em>x</em> is lower than the rank of the event time.</p><h3>Perfect is indeed the enemy of the good (p-value)</h3><p>So, how do these perfect predictors actually behave in real-life scenarios? Well, to find out, let’s once again turn to the lifelines library for some investigation:</p><pre>from lifelines import CoxPHFitter<br><br>perfect_cox_model = CoxPHFitter()<br>perfect_cox_model.fit(<br>  perfect_df,<br>  duration_col=&#39;time&#39;,<br>  event_col=&#39;event&#39;,<br>  formula=&#39;x&#39;<br>)<br>perfect_cox_model.print_summary()</pre><pre>#&gt; /.../coxph_fitter.py:1586: ConvergenceWarning:<br>#&gt; The log-likelihood is getting suspiciously close to 0 and the delta is still large.<br>#&gt; There may be complete separation in the dataset.<br>#&gt; This may result in incorrect inference of coefficients.<br>#&gt; See https://stats.stackexchange.com/q/11109/11867 for more.<br><br>#&gt; /.../__init__.py:1165: ConvergenceWarning:<br>#&gt; Column x has high sample correlation with the duration column.<br>#&gt; This may harm convergence.<br>#&gt; This could be a form of &#39;complete separation&#39;.<br>#&gt; See https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression<br><br>#&gt; /.../coxph_fitter.py:1611:<br>#&gt; ConvergenceWarning: Newton-Rhaphson failed to converge sufficiently.<br>#&gt; Please see the following tips in the lifelines documentation:<br>#&gt; https://lifelines.readthedocs.io/en/latest/Examples.html#problems-with-convergence-in-the-cox-proportional-hazard-model</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*BQ2u7LfSSbOHeHOMFREjxw.png" /></figure><p>Just like in logistic regression, we’re encountering convergence warnings and getting extremely wide confidence intervals for our predictor coefficient <em>β</em>. <strong>As a result, we end up with a p-value of 0.93</strong>!</p><p>If we simply filter models based on p-values without taking this issue into account or conducting further investigation, we would overlook these perfect predictors.</p><p>To tackle this convergence problem, the lifelines library documentation and some helpful StackOverflow threads suggest a potential solution: incorporating a regularization term into the cost function. This term effectively increases the cost for large coefficient values, and you can activate L2 regularization by setting the penalizer argument to a value greater than zero:</p><pre>perfect_pen_cox_model = CoxPHFitter(penalizer=0.01, l1_ratio=0)<br>perfect_pen_cox_model.fit(perfect_df, duration_col=&#39;time&#39;, event_col=&#39;event&#39;, formula=&#39;x&#39;)<br>perfect_pen_cox_model.print_summary()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*iap-Q40aOIPxiVJSmpNjvQ.png" /></figure><p>This approach fixes the convergence warning, but it doesn’t really make a huge dent in shrinking that pesky p-value. Even with this regularization trick, the p-value for a perfect predictor still hangs around at a somewhat large value 0.11.</p><h3>Time is relative: only ranks matter</h3><p>Lastly, we’ll verify that the absolute values of event times have no impact on a Cox regression fit, using our previous example. To do this, we’ll introduce a new column called time2, which will contain random numbers in the same order as the time column:</p><pre>sample_df = pd.DataFrame({<br>    &#39;subject&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;, &#39;E&#39;, &#39;F&#39;, &#39;G&#39;, &#39;H&#39;],<br>    &#39;x&#39;: [-1.7, -0.4, 0.0, 0.5, 0.9, 1.2, 1.3, 1.45],<br>    &#39;time&#39;: [1, 2, 4, 3, 5, 7, 6, 8],<br>    &#39;rank_x&#39;: [1, 2, 3, 4, 5, 6, 7, 8],<br>    &#39;event&#39;: [1, 1, 1, 1, 1, 1, 1, 0],<br>}).sort_values(&#39;time&#39;)<br><br>np.random.seed(42)<br>sample_df[&#39;time2&#39;] = sorted(np.random.randint(low=-42, high=888, size=8))<br>sample_df</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*zn8GH_B_FAcDQX21Ejvrig.png" /></figure><p>Their fits are indeed identical:</p><pre>sample_cox_model = CoxPHFitter()<br>sample_cox_model.fit(<br>  sample_df,<br>  duration_col=&#39;time&#39;,<br>  event_col=&#39;event&#39;,<br>  formula=&#39;x&#39;<br>)<br>sample_cox_model.print_summary()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*Gz1Z6BGenohWx0ydj-hK6w.png" /></figure><pre>sample_cox_model = CoxPHFitter()<br>sample_cox_model.fit(<br>  sample_df,<br>  duration_col=&#39;time2&#39;,<br>  event_col=&#39;event&#39;,<br>  formula=&#39;x&#39;<br>)<br>sample_cox_model.print_summary()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*tL3Crwdr_-7sE-T2WVkkLQ.png" /></figure><h3>Conclusion</h3><p>What did we learn from all of this?</p><ul><li>Perfect predictors in survival models are those predictors whose ranks perfectly match the ranks of event times.</li><li>Cox regression cannot fit these perfect predictors with a finite coefficient <em>β</em>, leading to wide confidence intervals and big p-values.</li><li>The actual values of event times don’t really matter — it’s all about their ranks.</li><li>When the ranks of event times and predictors don’t align, we don’t get that handy cancellation effect for large <em>β</em> values in likelihoods. So, we need at least one case where the ranks don’t match to have a fit with a finite coefficient.</li><li>Even if we try some fancy regularization techniques, perfect predictors can still give us those annoyingly wide confidence intervals and high p-values in real-life situations.</li><li>Just like in logistic regression, if we don’t really care about those p-values, using a regularization method can still provide us with a handy model fit that gets the prediction right!</li></ul><p>If you would like to run the code yourself, feel free to use IPython notebooks from my Github: <a href="https://github.com/igor-sb/blog/blob/main/posts/cox_perfect.ipynb">https://github.com/igor-sb/blog/blob/main/posts/cox_perfect.ipynb</a></p><p>Farewell until the next post! 👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb432137fd6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/unbox-the-cox-a-hidden-dark-secret-of-cox-regression-fb432137fd6">Unbox the Cox: A Hidden Dark Secret of Cox Regression</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unbox the Cox: Intuitive Guide to Cox Regressions]]></title>
            <link>https://medium.com/data-science/unbox-the-cox-intuitive-guide-to-cox-regressions-c485408ae15d?source=rss-e5f8ebca4ad8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c485408ae15d</guid>
            <category><![CDATA[cox-proportional-hazards]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <category><![CDATA[cox-regression]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Igor Šegota]]></dc:creator>
            <pubDate>Tue, 06 Jun 2023 15:00:14 GMT</pubDate>
            <atom:updated>2023-06-06T15:00:14.033Z</atom:updated>
            <content:encoded><![CDATA[<h4>How do hazards and maximum likelihood estimates predict event rankings?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b6BhNlegBECgsJaW" /><figcaption>Photo by <a href="https://unsplash.com/fr/@csgboyer?utm_source=medium&amp;utm_medium=referral">Chris Boyer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Introduction</h4><p>The goal of Cox regression is to model the relationship between predictor variables and the time it takes for an event to happen — like events that only happen once. Let’s dive into a made-up dataset with 5 subjects, labeled A to E. During the study, each subject either experienced an event (event = 1) or not (event = 0). On top of that, each subject got assigned a single predictor, let’s call it <em>x</em>, before the study. As a practical example, if we are tracking the death events, then <em>x</em> could be the dosage of a drug we are testing to see if it helps people live longer, by affecting the time until death.</p><pre>import pandas as pd<br>import numpy as np<br><br>sample_df = pd.DataFrame({<br>    &#39;subject&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;, &#39;E&#39;],<br>    &#39;time&#39;: [1, 3, 5, 4, 6],<br>    &#39;event&#39;: [1, 1, 1, 1, 0],<br>    &#39;x&#39;: [-1.7, -0.4, 0.0, 0.9, 1.2],<br>})<br><br>sample_df</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*0USFlXIhSgkzYlJgNEEi-w.png" /></figure><p>In this dataset, subject E did not experience anything during the study, so we set event = 0 and the time assigned is basically the last moment we knew about them. This kind of data is called “censored” because we have no clue if the event happened after the study ended. To make it easier to understand, a cool “lollipop” 🍭 plot comes in handy for visualizing this type of data:</p><pre>from cox.plots import plot_subject_event_times<br>plot_subject_event_times(sample_df)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*RFJYWqRnZeZeKNN2MrUlgw.png" /><figcaption>Lines indicate the duration of time each subject experienced before an event, which is represented by a filled circle. Subject E does not have a circle since it survived the entire study.</figcaption></figure><p>(️️Plotting functions available on my <a href="https://github.com/igor-sb/blog/blob/main/posts/cox/plots.py">Github repo</a>.)</p><p>Just a heads up: I want to make the main ideas of Cox regression easier to grasp, so we will focus on data where only one event can happen at a given time (no ties).</p><h3>Hazards</h3><p>Hazard represents the instant rate (probability per unit time) at which an event can occur at a specific time — assuming the event has not occurred before that point. Since it is a rate at which events take place, it can have arbitrary units and unlike probability, hazard values can range between 0 and infinity: [0, ∞).</p><p>In Cox regression, hazard works similar to the odds in logistic regression. In logistic regression, odds = p/(1-p) takes probabilities from the range of [0, 1] and transforms them to a range of [0, ∞). Then, taking the logarithm converts the odds to log-odds, which can have values from negative infinity to positive infinity (-∞, ∞). This log-odds transformation of probabilities is done to match the possible output values to the linear combination of predictors <em>β₁x₁ + β₂x₂ + …</em>, which also can range (-∞, ∞).</p><p>Here, we start with a hazard h(t, x) that already has values ranging from 0 to infinity [0, ∞). By applying a logarithm, we transform that range to (-∞, ∞), allowing it to be fitted with a linear combination of predictors:</p><figure><img alt="eq_log_hazard.png" src="https://cdn-images-1.medium.com/proxy/1*STgfqShVTHAaMXMVElO2PA.png" /></figure><p>In Cox regression, there’s another assumption that helps make things easier. It assumes that all the time dependence of the log-hazard is packed into the intercept term:</p><figure><img alt="eq_log_hazard_with_time.png" src="https://cdn-images-1.medium.com/proxy/1*mBPWpRcccbIws3vD5y8Bmg.png" /></figure><p>The motivation behind this assumption is to simplify the fitting process significantly (we will show how in a moment). In the literature, the intercept term <em>β₀(t)</em> is usually moved to the left side of this equation and expressed as a <em>baseline hazard</em> <em>log[h₀(t)]</em>:</p><figure><img alt="eq_log_hazard_with_baseline.png" src="https://cdn-images-1.medium.com/proxy/1*OETIVumHbvqFu1cEf8PGJA.png" /></figure><p>From this equation, we can expressed the hazard <em>h(t, x)</em>:</p><figure><img alt="eq_hazard.png" src="https://cdn-images-1.medium.com/proxy/1*SbYoRL3q8B84Ma4x9dE3KA.png" /></figure><p>Now, here is the cool part: since the data from each subject only comes to hazard through predictors <em>x</em>, every subject’s hazard has the same time dependence. The only difference lies in the <em>exp(βx)</em> part, which makes hazards from different subjects proportional to each other. That’s why this model is also called a “proportional hazard” model.</p><p>We have been drawing a lot of parallels to the logistic regression. If you read my previous post on logistic regression:</p><p><a href="https://towardsdatascience.com/logistic-regression-faceoff-67560de4f492">Logistic regression: Faceoff</a></p><p>you may be wondering if Cox regression also suffers from predictors being “too good”. Stay tuned for the next post which will cover that!</p><h3>Likelihoods</h3><p>Cox models are fit using a method called Maximum Likelihood Estimation (MLE). Likelihoods are quite similar to probabilities: they share the same equation — almost like two sides of the same coin. Probabilities are functions of data <em>x</em>, with fixed model parameters <em>β</em>s, while likelihoods are functions of <em>β</em>s, with fixed <em>x</em>. It‘s like looking at the probability density of a Normal distribution, but instead of focusing on <em>x</em>, we focus on <em>μ</em> and <em>σ</em>.</p><p>The MLE fitting process begins with a rank-order of event occurrence. In our made-up data this order is: A, B, D, C, with E censored. This is the only instance where time comes into play in Cox regression. <strong>The actual numeric values of event times do not matter at all, as long as the subjects experience the events in the same order.</strong></p><p>We then move on to each subject one by one and estimate the probability or likelihood of that subject experiencing an event compared to all the other subjects who are <em>still at risk</em> of having an event. For instance, take subject A who experienced an event at t = 1. The likelihood of this happening is determined by the hazard rate at which subject A experiences the event, relative to the combined hazard rates of everyone else who is still at risk at t = 1 (which includes everyone):</p><figure><img alt="eq_likelihood_A_def.png" src="https://cdn-images-1.medium.com/proxy/1*4kDtwj9hUoX1Vu7OGVdDdg.png" /></figure><figure><img alt="eq_likelihood_A_expanded.png" src="https://cdn-images-1.medium.com/proxy/1*j3t9b8to_C_DUNvP3JdmZQ.png" /></figure><p>As you may have noticed, we did not bother defining the baseline hazard <em>h₀(t)</em> because it actually cancels out completely from the likelihood calculation.</p><p>Once we substitute the values for <em>x</em> (-1.7, -0.4, 0.0, 0.9, 1.2) for each subject, we end up with an equation that only has <em>β</em> left:</p><figure><img alt="eq_likelihood_A_final.png" src="https://cdn-images-1.medium.com/proxy/1*baK3XkBquCjVR0jt4mqprA.png" /></figure><p>From this point on, for any time after t = 1, the hazard of subject A is considered zero and is not taken into account when calculating further likelihoods. For instance, at another time, t = 3, subject B experiences an event. So, the likelihood for subject B is determined relative to the hazards of subjects B through E only:</p><figure><img alt="eq_likelihood_B.png" src="https://cdn-images-1.medium.com/proxy/1*Bb83ALB9lkmR7jXN9w6uMw.png" /></figure><p>We could keep going and calculate the likelihoods for all subjects A through D, but we will cover that in the coding part in the next section. Subject E, being censored and not experiencing an event does not have its own likelihood. Since the censored data is only used in the likelihoods of uncensored subjects, the resulting combined likelihood is often referred to as “partial likelihood”.</p><p>To summarize this process, we can create an animated lollipop plot:</p><pre>from cox.plots import animate_subject_event_times_and_mark_at_risk<br>animate_subject_event_times_and_mark_at_risk(sample_df).save(<br>    &#39;../images/cox_likelihood_fitting_sample.gif&#39;<br>)</pre><figure><img alt="Cox likelihood fitting animation" src="https://cdn-images-1.medium.com/proxy/1*m2AkyOTC3Zq5j0Pyq5Yrnw.gif" /></figure><h3>Finding β</h3><p>When events occur independently of each other, the joint probability or likelihood of observing all events can be calculated by multiplying individual likelihoods, denoted as <em>L= L(A) L(B) L(C) L(D)</em>. However, multiplying exponential expressions can lead to numerical errors, so we often take the logarithm of this likelihood. By applying the logarithm, we transform the product of likelihoods into a sum of log-likelihoods:</p><figure><img alt="eq_sum_likelihoods_def.png" src="https://cdn-images-1.medium.com/proxy/1*O1KRZAimnuZUMIfZANoDfw.png" /></figure><figure><img alt="eq_log_likelihood_A.png" src="https://cdn-images-1.medium.com/proxy/1*W745PCPUB_BaAHRo8gZsNQ.png" /></figure><p>Since the logarithm is a monotonic function, both the likelihood and the log-likelihood reach their maximum point at the same value of <em>β</em>. To facilitate visualization and enable comparison with other cost functions, we can define a cost as the negative log-likelihood and aim to minimize it instead.</p><h3>Ready, Set, Code!</h3><p>We can implement this algorithm in Python, step by step. First, we need to extract the event time and predictor <em>x</em> for each uncensored subject. This can be accomplished using the function event_time_and_x_from_subject(). Once we have the event time for a subject, we can subset our data frame to identify the rows corresponding to all subjects who are still at risk. This is achieved using the function subjects_at_risk_data(). Finally, we calculate the log-likelihood for each subject using the function log_likelihood():</p><pre>def event_time_and_x_from_subject(df, subject):<br>    subject_with_event_df = df.query(f&quot;subject == &#39;{subject}&#39; &amp; event == 1&quot;)<br>    if subject_with_event_df.empty: # Censored subjects<br>        return (np.inf, 0)<br>    return subject_with_event_df.iloc[0][[&#39;time&#39;, &#39;x&#39;]]<br><br>def subjects_at_risk_data(df, subject):<br>    time = event_time_and_x_from_subject(df, subject)[0]<br>    return df.query(f&#39;time &gt;= {time}&#39;)<br><br>def log_likelihood(df, subject, beta):<br>    x_subjects_at_risk = subjects_at_risk_data(df, subject)[&#39;x&#39;]<br>    x_subject = event_time_and_x_from_subject(df, subject)[1]<br>    at_risk_hazards = np.exp(beta * x_subjects_at_risk)<br>    return beta * x_subject - np.log(np.sum(at_risk_hazards))</pre><p>For visualization purposes, we plot the cost or negative log-likelihoods. Therefore, we need to compute these values for each subject at a specific <em>β</em> value:</p><pre>def neg_log_likelihood_for_all_subjects(df, beta):<br>    subjects = df.query(&quot;event == 1&quot;)[&#39;subject&#39;].tolist()<br>    neg_log_likelihoods = [<br>        -log_likelihood(df, subject, beta)<br>        for subject in subjects<br>    ]<br>    return pd.DataFrame({<br>        &#39;subject&#39;: subjects,<br>        &#39;neg_log_likelihood&#39;: neg_log_likelihoods<br>    })</pre><p>To find the minimum cost, we sweep through the range of <em>β</em> values:</p><pre>def neg_log_likelihood_all_subjects_sweep_betas(df, betas=np.arange(-5, 5, 0.1)):<br>    loglikelihoods_per_beta = []<br>    for beta in betas:<br>        beta_df = neg_log_likelihood_for_all_subjects(df, beta)<br>        beta_df.insert(0, &#39;beta&#39;, beta) # Add beta column<br>        loglikelihoods_per_beta.append(beta_df)<br>    return pd.concat(loglikelihoods_per_beta)<br><br>negloglik_sweep_betas_df = neg_log_likelihood_all_subjects_sweep_betas(sample_df)<br>negloglik_sweep_betas_df</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*44P3QLmYI4kICPRBQIlJsQ.png" /></figure><h3>Making sense of it all</h3><p>Rather than aggregating the data frame by summing the log-likelihoods grouped by subject, we can keep it in its current form and visualize it as stacked bar charts. In this visualization, the total height of each bar corresponds to the sum of negative log-likelihoods. Each subject is represented by a different color within the bar, indicating their individual likelihood and contribution to the overall likelihood:</p><pre>from cox.plots import plot_cost_vs_beta<br>plot_cost_vs_beta(negloglik_sweep_betas_df, width=0.1)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*pyN0kk3c-lzSCcCyh4vvHg.png" /><figcaption>Each narrow vertical colored bar represents the individual negative log-likelihood.</figcaption></figure><p>Let’s understand how to interpret this plot.</p><p>First, note that the negative log-likelihood (cost) is large when the likelihood and hazards are small. Think of the y-axis similar to -log(p-value); larger values indicate lower probability.</p><p>Second, censored subjects (like subject E) do not have their own individual likelihoods, so they don’t appear on the plot. However, their contribution is incorporated into the likelihoods of subjects A to D.</p><p>Now, consider different scenarios based on the value of <em>β</em>:</p><ul><li>If <em>β</em> is large and negative, subjects A, B, and C (with <em>x ≤ 0</em>) and their events are fitted nearly perfectly. The likelihoods of these subjects are all close to one. However, fitting such large negative <em>β</em> values for A, B, and C comes at the cost of subject D. In this range of <em>β</em>, the probability of subject D (with <em>x &gt; 0</em>) having an event is very low. As a result, the total cost is dominated by the small likelihood of subject D.</li><li>If <em>β</em> is large and positive, the hazard of subject D (with <em>x &gt; 0</em>) becomes significant compared to the other hazards. The purple section (subject D) becomes a small part of the total cost. However, since subjects A, B, and C all have <em>x ≤ 0</em>, the cost of fitting a large <em>β</em> for them is high. Consequently, these subjects dominate the total cost.</li><li>The optimal value of β, located around 2 on the plot, strikes a balance between assigning high probabilities to events of subjects A, B, and C versus subject D. This optimal value can be verified numerically:</li></ul><pre>negloglik_sweep_betas_df \<br>    .groupby(&quot;beta&quot;) \<br>    .agg(sum_neg_log_likelihood=(&#39;neg_log_likelihood&#39;, &#39;sum&#39;)) \<br>    .reset_index() \<br>    .sort_values(&#39;sum_neg_log_likelihood&#39;) \<br>    .head(1)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*VQE2kuBDFNJuZYUaVQtP_w.png" /></figure><h3>Using lifelines library</h3><p>Now that we have a better understanding of how Cox regression works, we can apply it to sample data using Python’s lifelines library to verify the results. Here is a code snippet for our made-up data:</p><pre>from lifelines import CoxPHFitter<br><br>sample_cox_model = CoxPHFitter()<br>sample_cox_model.fit(sample_df, duration_col=&#39;time&#39;, event_col=&#39;event&#39;, formula=&#39;x&#39;)<br>sample_cox_model.print_summary()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*C4KcIPbJDZPLD1Wm3EY7oA.png" /></figure><p>In the output, we can observe the coefficient (coef) value of -1.71, which corresponds to the β coefficient. Next to it, we have exp(coef), representing exp(β), as well as columns indicating standard errors and confidence intervals. The “partial log-likelihood” value is -2.64, which matches our manual result.</p><p>Lastly, it is important to mention that Cox regression implementations also offer an extension that allows the model to handle multiple events tied at the same event time. However, this goes beyond the scope of this discussion.</p><h3>Conclusion</h3><p>There was a lot to “unbox” here:</p><ul><li>Cox regression models the association between predictor variables and the rank-order of times to an event occurrence.</li><li>Actual numeric values of event times do not matter at all, as long as the subjects experience the events in the same order.</li><li>Hazards are probabilities per unit time and can have arbitrary units, while likelihoods are related the probability of events occurring.</li><li>Stacked bar charts can be used to provide insights into maximum likelihood estimation by stacking individual negative log-likelihoods and exploring how they change with predictors <em>x</em>.</li><li>By striking a balance between assigning probabilities to events for various subjects, MLE finds <em>β</em> for which the observed data is <em>most likely to happen</em>.</li></ul><h4>To be continued… 👀</h4><h3>References</h3><ul><li>Code with plots: <a href="https://github.com/igor-sb/blog/blob/main/posts/cox/plots.py">https://github.com/igor-sb/blog/blob/main/posts/cox/plots.py</a></li><li>Survival Data Analysis slides from Prof. Patrick Breheny: <a href="https://myweb.uiowa.edu/pbreheny/7210/f19/index.html">https://myweb.uiowa.edu/pbreheny/7210/f19/index.html</a></li><li>ChatGPT for text cleanup and funny hallucinations</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c485408ae15d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/unbox-the-cox-intuitive-guide-to-cox-regressions-c485408ae15d">Unbox the Cox: Intuitive Guide to Cox Regressions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic Regression: Deceptively Flawed]]></title>
            <link>https://medium.com/data-science/logistic-regression-deceptively-flawed-2c3e7f77eac9?source=rss-e5f8ebca4ad8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c3e7f77eac9</guid>
            <category><![CDATA[odds-ratio]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Igor Šegota]]></dc:creator>
            <pubDate>Tue, 23 May 2023 15:09:24 GMT</pubDate>
            <atom:updated>2023-05-23T15:09:24.290Z</atom:updated>
            <content:encoded><![CDATA[<h4>When can large odds ratios and perfectly separated data bite you?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T86UVPBEN_RVjCuQ" /><figcaption>Photo by <a href="https://unsplash.com/es/@alvannee?utm_source=medium&amp;utm_medium=referral">Alvan Nee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is a second part to a previous post on conceptual understanding of logistic regression:</p><p><a href="https://towardsdatascience.com/logistic-regression-faceoff-67560de4f492">Logistic regression: Faceoff</a></p><p>Last time we visualized and explained fitting log-losses in logistic regression. We also showed that this process cannot fit perfectly separated data. In other words, unlike linear regression with ordinary least square fit, logistic regression actually works better if the data is a little bit noisy!</p><p>In practice, does this actually matter? <em>It depends:</em></p><ul><li>It matters if our goal is to use the output for statistical inference. For example, accurately estimating model coefficients, calculating confidence intervals and to test hypotheses using p-values.</li><li>It does not matter much or at all, if our goal is to use the output of a logistic model to create a predictive classification model.</li></ul><h3>Statistical inference: statsmodels</h3><h4>Sample data</h4><p>For this part, we will use Python’s statsmodels library. Keep in mind that statsmodels and scikit-learn (used later) parametrize the probability using <em>β</em>s instead of <em>k</em> and <em>x₀</em>:</p><figure><img alt="eq_px_betas_k_x0.png" src="https://cdn-images-1.medium.com/proxy/1*q59t4rpSwIISLB3sjsXbjA.png" /></figure><p>where the relationship between <em>k</em>, <em>x₀</em> and <em>β₁</em>, <em>β₀</em> is:</p><figure><img alt="eq_betas_vs_k_x0.png" src="https://cdn-images-1.medium.com/proxy/1*bGBODQLDBi2hFUsreO18zw.png" /></figure><p>We will continue with datasets we generated in the <a href="https://towardsdatascience.com/logistic-regression-faceoff-67560de4f492">first part on logistic regression</a>, first with the “imperfect” data sample_df , using statsmodels’ formula API:</p><pre>import statsmodels.formula.api as smf<br>model = smf.logit(&#39;y ~ x&#39;, df).fit()<br>model.summary()<br><br>#&gt; Optimization terminated successfully.<br>#&gt;         Current function value: 0.230366<br>#&gt;         Iterations 8</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*HoOAhcJDS6AuWdD7uea0xg.png" /></figure><p>Our model parameters are <em>k = 3</em> and <em>x₀ = 2.5</em>, so those translate to <em>β₁ = 3</em> and <em>β₀ = -7.5. </em>We can compare those with fitted parameters by reading them out from the coef column of the bottom table:</p><figure><img alt="eq_betas_example.png" src="https://cdn-images-1.medium.com/proxy/1*9aPrdBu6kOFJHPeiT-HYRA.png" /></figure><p>We have very few data points and the seed was intentionally chosen to showcase outliers, so the fit is a little bit off, but it is still in the right ballpark. The total log-loss is here reported as “Log-Likelihood”, which is just the negative of total log-loss and equals -6.911.</p><h4>Perfectly separated data</h4><p>What happens when we run our perfectly separated dataset in statsmodels? Again, it depends!</p><pre>perfect_sep_model = smf.logit(&#39;y ~ x&#39;, perfect_sep_data).fit()<br><br>#&gt; ...<br>#&gt; PerfectSeparationError: Perfect separation detected, results not available</pre><p>In this case we got an error — no results are output. In other cases of perfect separation, we may get a warning. For example, if we use the same parameters but different random seed:</p><pre>perfect2_df = create_sample_data(k=3, x0=2.5, n_points=30, seed=154)<br>perfect_sep2_df, perfect_sep_logloss2_df = fit_data_to_logloss(<br>    perfect2_df,<br>    k=3,<br>    x0=2.5<br>)<br><br>perfect_sep_model_2 = smf.logit(&#39;y ~ x&#39;, perfect_sep2_df).fit()<br>perfect_sep_model_2.summary()<br><br>#&gt; Warning: Maximum number of iterations has been exceeded.<br>#&gt;         Current function value: 0.000000<br>#&gt;         Iterations: 35<br><br>#&gt; /.../statsmodels/base/model.py:604: ConvergenceWarning: <br>#&gt;   Maximum Likelihood optimization failed to converge. Check mle_retvals</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6Eo01u04mdnT96NjDfGwA.png" /></figure><p>Since the second model did not converge either, we can argue that it probably should have also returned an error, not an innocent warning. Logistic regression function in R, glm(..., family=binomial) does the same. To quote <a href="https://www.burns-stat.com/pages/Tutor/R_inferno.pdf">R Inferno</a>, Circle 5, Consistency:</p><blockquote>There is a problem with warnings. No one reads them. People have to read error messages because no food pellet falls into the tray after they push the button. With a warning the machine merely beeps at them but they still get their food pellet. Never mind that it might be poison.</blockquote><p>Therefore, be careful when doing inference using effect sizes and p-values! In this case the data is perfectly separable — we have a perfect predictor — while the reported p-value (P &gt; |z|) is 0.998. Ignoring or misunderstanding these warnings may get you miss some obvious features in the data.</p><p>Alternatively, consider using a different model. There is a brave new world outside of logistic regression! 🌍</p><h3>Prediction: scikit-learn</h3><h4>Sample data</h4><p>scikit-learn automates and abstracts much of the calculations while providing a simple and consistent way to run many different models. However, in doing so, it hides a number of things happening in the background. When it is not obvious what is happening under the hood, learning data science concepts from scikit-learn documentation can lead to misunderstanding.</p><p>One of these is misunderstandings is that when we run LogisticRegression() from sklearn.linear_model and use a .predict() method, it <em>makes it seem </em>as if we are running a classification model:</p><pre>from sklearn.linear_model import LogisticRegression<br><br>def sklearn_logistic_fit(df):<br>    sklearn_logistic = LogisticRegression(penalty=None).fit(<br>        df[&#39;x&#39;].values.reshape(-1, 1),<br>        df[&#39;y&#39;].values<br>    )<br>    predicted_outcomes = sklearn_logistic.predict(<br>        df[&#39;x&#39;].values.reshape(-1, 1)<br>    )<br><br>    print(&quot;beta_1 or log-odds-ratio: &quot;, sklearn_logistic.coef_[0][0])<br>    print(&quot;beta_0 or y intercept: &quot;, sklearn_logistic.intercept_[0])<br>    print(&quot;Original outcomes: &quot;)<br>    print(df[&#39;y&#39;].values)<br>    print(&quot;Predicted outcomes: &quot;)<br>    print(predicted_outcomes)<br> <br>    return predicted_outcomes<br><br>sklearn_logistic_fit(sample_df)<br><br>#&gt; beta_1 or log-odds-ratio:  2.6187462367743803<br>#&gt; beta_0 or y intercept:  -5.682896087072998<br>#&gt; Original outcomes: <br>#&gt; [0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1]<br>#&gt; Predicted outcomes: <br>#&gt; [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1]</pre><p>However, we can verify that this is still the same old logistic regression by either calling its .predict_proba() method or by digging out the model parameters from the model object, which we printed out here.</p><p>Under the hood, .predict() predicts outcomes based on the larger of two probabilities: <em>p</em> or <em>1-p</em>. If <em>p &gt; 1-p</em> then it predicts 1, otherwise it predicts 0. It does not calculate confidence intervals, does not have p-values and does not do any kind of statistical inference or hypothesis testing. It is still a regression model, just with another thresholding layer on top of it.</p><p>Note: LogisticRegression() by default uses L2 regularization, which adds an extra term to the log-loss function. To make it comparable with statsmodels, use LogisticRegression(penalty=None).</p><h4>Perfectly separable data</h4><p>The advantage of ignoring the accuracy of fitted coefficients is that even if the coefficients are not accurate, or just plain wrong, the model can still be good enough! As the saying goes, “let not the perfect be the enemy of the good”.</p><p>scikit-learn by default uses a different fitting method (called “<a href="https://en.wikipedia.org/wiki/Limited-memory_BFGS">lbfgs</a>”) than statsmodels. In my very limited set of tests, lbfgs did not error out on perfectly separated data. To see what happens here, let us run the same two datasets we used previously for statsmodels:</p><pre>sklearn_logistic_fit(perfect_sep_df)<br><br>#&gt; beta_1 or log-odds-ratio:  22.65352734156491<br>#&gt; beta_0 or y intercept:  -53.25978472434583<br>#&gt; Original outcomes: <br>#&gt; [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]<br>#&gt; Predicted outcomes: <br>#&gt; [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]<br><br>sklearn_logistic_fit(perfect_sep2_df)<br><br>#&gt; beta_1 or log-odds-ratio:  180.11386979983104<br>#&gt; beta_0 or y intercept:  -469.3714808877968<br>#&gt; Original outcomes: <br>#&gt; [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]<br>#&gt; Predicted outcomes: <br>#&gt; [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]</pre><p>This algorithm also stops at some point, giving finite values of <em>β₁ </em>(or <em>k</em>), without an error, but it still predicts the correct outcomes. These are perfectly usable model fits for classification!</p><h3>Log-odds and loose ends</h3><p>There is one last topic we did not touch too much. Coefficient <em>k</em> (or <em>β₁</em>) which multiplies <em>x</em>, has another interpretation — a <em>log odds ratio</em>. Odds ratio describes the multiplicative change in odds, when <em>x</em> increases by 1:</p><figure><img alt="eq_log_odds_ratio.png" src="https://cdn-images-1.medium.com/proxy/1*JJzdwcUSVChi7KEZ5QSUZw.png" /></figure><p>This definition states that <strong>odds ratio is a ratio of ratios of probabilities</strong>. If increasing <em>x</em> by one unit, increases the probability of <em>y = 1</em> from 0.1 (odds 0.1 / 0.9 = 0.11) to 0.2 (odds = 0.2 / 0.8 = 0.25), that is represented by odds ratio of 0.25 / 0.1 = 2.27.</p><p>When probabilities are small, large odds ratio does not reflect large absolute probabilities. If increasing <em>x</em> by one unit, increases the probability of <em>y = 1:</em></p><ul><li>from <em>p(x) = 0.0001</em> (odds = 0.0001 / (1–0.0001) ≈ 0.0001)</li><li>to <em>p(x + 1) = 0.001</em> (odds = 0.001 / (1–0.001) ≈ 0.001)</li></ul><p>then this gives us the odds ratio of 0.001/0.0001 = 10. Even though the odds ratio is 10, the final absolute probability is still very small: 0.001. The good news is that <strong>when probability is small, odds ratio becomes easier to interpret </strong>— it is approximately equal to a ratio of two probabilities <em>p(x + 1) / p(x)</em>.</p><p>Compare this to another example where probability increases by a substantial amount, say by 25% from <em>p(x) = 0.5</em> to <em>p(x + 1) = 0.75</em>: in this case odds ratio is only 3!</p><p>Therefore, high odds ratio needs to be used with a grain of salt when the probability of <em>y = 1</em> is small. There are alternatives to odds ratio, such as <a href="https://www.statology.org/interpret-relative-risk/">relative risk</a> which are more interpretable, but may not be simple to calculate.</p><h3>Conclusion</h3><p>In the last two posts, we gave an intuitive explanation to logistic regression and show how to run regression models in Python’s statsmodels and scikit-learn libraries.</p><p>Take home points:</p><ul><li>Logistic regression outputs a probability — therefore it is a regression algorithm.</li><li>Logistic regression is often used for classification. For example, by predicting the outcome with a larger probability — or by setting a custom probability threshold.</li><li>Parameters are estimated numerically using the difference between data and two <strong>crossed hockey sticks log-loss curves that serve as a cost function</strong>.</li><li>Use <strong>statsmodels </strong>to run a logistic regression when you are interested in statistical inference (care about accuracy of coefficients, hypothesis testing, p-values, ...).</li><li>Use <strong>scikit-learn</strong> to run a logistic regression when you just want to predict the outcome or when you need to run a larger model.</li><li>Perfectly separable data breaks statistical inference: best-fit coefficients do not exist or “are infinite” and p-values will be large (because of large confidence intervals). If we only care about using the model for classification/prediction, then scikit-learn’s implementation works better for perfectly separated data.</li><li><strong>Do not blindly trust large (log-)odds ratios</strong>. Consider calculating additional metrics, such as absolute probabilities.</li></ul><p>I hope this helps you in your data science journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c3e7f77eac9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/logistic-regression-deceptively-flawed-2c3e7f77eac9">Logistic Regression: Deceptively Flawed</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic regression: Faceoff]]></title>
            <link>https://medium.com/data-science/logistic-regression-faceoff-67560de4f492?source=rss-e5f8ebca4ad8------2</link>
            <guid isPermaLink="false">https://medium.com/p/67560de4f492</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[cross-entropy]]></category>
            <category><![CDATA[log-loss]]></category>
            <dc:creator><![CDATA[Igor Šegota]]></dc:creator>
            <pubDate>Thu, 18 May 2023 13:28:22 GMT</pubDate>
            <atom:updated>2023-06-24T14:34:48.852Z</atom:updated>
            <content:encoded><![CDATA[<h3>Logistic Regression: Faceoff and Conceptual Understanding</h3><h4>What do log-losses and perfectly separated data have to do with hockey sticks?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V6K6xj5AOTCjPEXi" /><figcaption>Photo by <a href="https://unsplash.com/@jerryyu?utm_source=medium&amp;utm_medium=referral">Jerry Yu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Who ordered this?</h3><p>As of this writing, Google search for “logistic regression tutorial” shows about 11.2M results. Why add another thing to this pile?</p><p>After reading a good number of articles, books and guides, I realized that most lack clear and intuitive explanations of how logistic regression works. Instead, they usually strive to be either practical, by showing how to run models, or as mathematically complete as possible, and as a consequence, basic concepts get buried underneath a forest of matrix algebra.</p><p>We will start by clearing up what seem to be common misconceptions. Logistic regression is <strong>not</strong>:</p><ul><li>linear regression but with sigmoid curve instead of a straight line</li><li>classification algorithm (but can be used for this)</li><li>sigmoid curve “fit” of a decision boundary separating two classes of points in the x-y plane</li></ul><h3>What <em>is</em> a logistic regression?</h3><p>Logistic regression is a regression model that returns a probability of a binary outcome (0 or 1), assuming that log of the odds is a linear combination of one or more inputs. Odds is a ratio between probability of outcome happening (<em>p</em>) and the probability of the outcome not happening (<em>1-p</em>). When we have one input or predictor, this starting assumption is mathematically expressed as:</p><figure><img alt="eq_log_odds.png" src="https://cdn-images-1.medium.com/proxy/1*ebX9fRL2sCplFOmQWaoz0Q.png" /></figure><p>The goal behind logistic regression is to model cases when inputs are shifting the outcome probability progressively from 0 to 1. The probability of the outcome being 1, <em>p</em>, can be derived from the previous equation and expressed as a function of inputs:</p><figure><img alt="p_x_definition.png" src="https://cdn-images-1.medium.com/proxy/1*Ig6Cg5CQCsFUU-BxHXkKVA.png" /></figure><p>In the last part we swapped from parameters <em>β₁</em> and <em>β₀</em> to <em>k</em> and <em>x₀</em>. Using <em>k</em> and <em>x₀</em> will give us a clearer picture of the model as we go along. We will also stick to a single predictor variable <em>x</em>, as opposed to marching in with an army of matrices, so we can easily visualize logistic fits.</p><h3>Logistic curve</h3><p>We will begin by plotting the logistic curve, with parameters <em>x₀ = 2.5</em> and <em>k = 3</em>, on an interval <em>x</em> between 0 and 5:</p><pre>import numpy as np<br>import pandas as pd<br>import plotnine as p9<br>from scipy.stats import uniform, bernoulli<br><br># From https://github.com/igor-sb/blog/blob/main/posts/logistic/plots.py<br>from logistic.plots import plot_naive_logistic_fit<br><br>def logistic(x, k, x0):<br>    return 1 / (1 + np.exp(-k*(x - x0)))<br><br>def create_smooth_logistic_curve_data(k, x0, n_points=100):<br>    df = pd.DataFrame({&#39;x&#39;: np.linspace(0, 5, n_points)})<br>    df[&#39;p_x&#39;] = logistic(df[&#39;x&#39;], k, x0)<br>    return df<br><br>def create_sample_data(k, x0, n_points, seed=1):<br>    np.random.seed(seed)<br>    df = pd.DataFrame({<br>        &#39;x&#39;: uniform.rvs(loc=0, scale=5, size=n_points)<br>    }).sort_values(&#39;x&#39;, ignore_index=True)<br>    p_x = logistic(df[&#39;x&#39;], k, x0)<br>    df[&#39;y&#39;] = bernoulli.rvs(p_x)<br>    return df<br><br>sample_df = create_sample_data(k=3, x0=2.5, n_points=30)<br>smooth_px_df = create_smooth_logistic_curve_data(k=3, x0=2.5)<br><br>plot_naive_logistic_fit(sample_df, smooth_px_df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*tKAqMhm4Dk9BaaXtiZ6vKQ.png" /><figcaption>Points (red for y=0 and teal for y = 1) and p(x) (black) for logistic regression.</figcaption></figure><p>This logistic curve <em>p(x)</em> is described by two parameters:</p><ul><li><em>x₀</em> is the value of a predictor <em>x</em> for which the probability is 0.5 (mid-point): <em>p(x = x₀) = 0.5</em>, so tells us about the location of the mid-point.</li><li><em>k</em> is related to the slope of the probability at mid-point: <em>(dp/dx){x = x₀} = k/4</em>, so tells us about the steepness of the curve at that mid-point. The larger the <em>k</em>, the steeper the curve in the middle.</li></ul><p>If we naively employed ordinary least squares to fit the curve <em>p(x)</em> to these points, we would find that all residuals would be less than 1 and most points on the “wrong side” of the mid-point would have residuals ~ 1. It would make more sense to assign a much larger cost to points that are large outliers.</p><h3>Log-loss fit</h3><p>Instead of trying to make ordinary least squares work to fit <em>p(x)</em> to the points, logistic regression proceeds differently:</p><ul><li>For teal points at <em>y = 1</em>, we will fit <em>-log p(x) </em>instead of <em>p(x)</em>. Negative logarithm makes <em>-log p(x)</em> progressively larger; as <em>p(x)</em> approaches zero.</li><li>For the red points at <em>y = 0</em> we can do the same by using the probability that the outcome is zero, <em>-log[1-p(x)]</em>.</li></ul><p>We call these “log-losses”. If we collapse all the points to <em>y = 0</em>, then for each point these two log-losses represent a<em> cost </em>(loss) of that point, for being some amount away from the log-loss curves. In order to utilize numpy vectorization, we will code these two together as a single log-loss function (this combo log-loss also goes by the name “cross entropy”):</p><pre>def log_loss(p_x, y):<br>    return -y * np.log(p_x) - (1 - y) * np.log(1 - p_x)</pre><p>One way to think about logistic regression is a method that simultaneously fits:<em> -log p(x)</em> for <em>y = 1</em> and -<em>log[1-p(x)]</em> for <em>y = 0</em>.</p><p>How do these two log-loss curves look?</p><p>To visualize them, we will plot the same data in the previous plot, but now with log-losses instead of probability:</p><pre>def create_smooth_logloss_data(k, x0, n_points=100):<br>    x = np.linspace(0, 5, n_points)<br>    p_x = logistic(x, k, x0)<br>    return pd.DataFrame({<br>        &#39;x&#39;: np.concatenate((x, x)),<br>        &#39;y&#39;: np.concatenate(([0] * len(x), [1] * len(x))),<br>        &#39;log_loss&#39;: np.concatenate((log_loss(p_x, 0), log_loss(p_x, 1))),<br>    })<br><br>def add_logloss(df, k, x0):<br>    p_x = logistic(df[&#39;x&#39;], k, x0)<br>    return df.assign(log_loss = log_loss(p_x, df[&#39;y&#39;]))<br><br>def fit_data_to_logloss(sample_df, k, x0):<br>    sample_fit_df = add_logloss(sample_df, k, x0)<br>    logloss_df = create_smooth_logloss_data(k, x0)<br>    return (sample_fit_df, logloss_df)</pre><pre>from logistic.plots import plot_logistic_fit<br><br>sample_fit_df, logloss_df = fit_data_to_logloss(sample_df, k=3, x0=2.5)<br><br>plot_logistic_fit(sample_fit_df, logloss_df)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*nJq6eRrsQr5-P_1k5bltmw.png" /></figure><p>Here we collapsed all points to <em>y = 0</em>, but use the colors as <em>y</em> labels, since the values of log-losses on their own represent the cost. Red points (<em>y = 0</em>) are fit to the red hockey stick curve: <em>-log[1-p(x)]</em>. Teal points (<em>y = 1</em>) are fit to the teal hockey stick curve: <em>-log p(x)</em>. Sum of the vertical dashed lines represents the total log-loss that needs to be minimized for various <em>k</em> and <em>x₀.</em></p><p>Unlike probability, log-loss curves have the property of penalizing big outliers proportionally more and they do not have residuals that cap out at 1.</p><h3>Finding the minimal log-loss</h3><p>How does changing <em>k</em> and <em>x₀</em> affect this fit? To answer this, we can run fits with various combinations of <em>k</em> and <em>x₀.</em></p><pre>def fit_parameter_combinations(sample_df, combinations):<br>    sample_df_list = []<br>    logloss_df_list = []<br>    for k, x0 in combinations:<br>        sample_fit_df, logloss_df = fit_data_to_logloss(sample_df, k, x0)<br>        sample_fit_df[&#39;k&#39;] = logloss_df[&#39;k&#39;] = k<br>        sample_fit_df[&#39;x0&#39;] = logloss_df[&#39;x0&#39;] = x0<br>        sample_df_list.append(sample_fit_df)<br>        logloss_df_list.append(logloss_df)<br><br>    return (<br>        pd.concat(sample_df_list, ignore_index=True),<br>        pd.concat(logloss_df_list, ignore_index=True)<br>    )</pre><p>Changing <em>x₀</em> moves the intersection point sideways:</p><pre># From https://github.com/igor-sb/blog/blob/main/posts/logistic/plots.py<br>from logistic.plots import plot_logistic_fit_panel<br><br>x0_dfs, x0_logloss_dfs = fit_parameter_combinations(<br>    sample_df,<br>    [(3, 1.5), (3, 2.5), (3, 3.5)]<br>)<br>plot_logistic_fit_panel(x0_dfs, x0_logloss_dfs, &#39;~x0&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*wbHuS2F_gEjqBdZnTQKxYg.png" /></figure><p>If <em>x₀</em> is chosen away from the optimal point, the log-loss increases because increasing number of points gets fitted to the rising parts of the hockey sticks.</p><p>Changing<em> k</em> affects the steepness of the log-loss curves (note the different y axes):</p><pre>k_dfs, k_logloss_dfs = fit_parameter_combinations(<br>    sample_df,<br>    [(0.5, 2.5), (3, 2.5), (7, 2.5)]<br>)<br>plot_logistic_fit_panel(k_dfs, k_logloss_dfs, &#39;~k&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*htAJnOE0KXmnrgb_l00M1g.png" /></figure><p>If <em>k</em> is too low (0.5), most points add small but significant amounts to the total log-loss. If <em>k</em> is too high (7.0), only the points on the “wrong side” contribute a significant amount to the total log-loss. In this case, it is the two teal points on the left of mid-point at <em>x₀ = 2.5</em>.</p><p>This brings up a question: what if there are no points on the “wrong side” of the mid-point, such as when the data is perfectly separated?</p><h3>Perfectly separated data</h3><p>It turns out, the logistic model cannot fit data that is perfectly separated! 😮 We can apply what we learned earlier about fitting log-losses to understand why. We start by creating perfectly separated data (with <em>k = 3</em> and <em>x₀ = 2.5</em>):</p><pre>perfect_df = create_sample_data(k=3, x0=2.5, n_points=30, seed=112)<br><br>perfect_sep_df, perfect_sep_logloss_df = fit_data_to_logloss(<br>    perfect_df,<br>    k=3,<br>    x0=2.5<br>)<br><br>print(&quot;Total log-loss: &quot;, perfect_sep_df[&#39;log_loss&#39;].sum())<br><br>plot_logistic_fit(perfect_sep_df, perfect_sep_logloss_df)<br><br>#&gt; Total log-loss:  1.4776889859527733</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*6fEIzYdcISyCqAFH5Trwnw.png" /></figure><p>Remember how changing <em>k</em> affects these hockey stick log-loss curves? When we increased <em>k</em>, the main contribution to the total log-loss was from the points in the “wrong side” of the mid-point. Now, all points are on the “correct side” of the mid-point.</p><p>That means that we can create arbitrarily good fits by continuously increasing <em>k</em>. Here is how the fits look when we set <em>k = 13</em>:</p><pre>perfect_sep_df, perfect_sep_logloss_df = fit_data_to_logloss(<br>    perfect_df,<br>    k=13,<br>    x0=2.5<br>)<br><br>print(&quot;Total log-loss: &quot;, perfect_sep_df[&#39;log_loss&#39;].sum())<br><br>plot_logistic_fit(perfect_sep_df, perfect_sep_logloss_df)<br><br>#&gt; Total log-loss:  0.050774866143719344</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*1Q6uqYR7KXkGGh2IUYjF4g.png" /></figure><p>Fitting perfectly separated data would require log-losses to have a 90-degree angle and fitted probability to have an infinite slope at the mid-point. Therefore, there is no parameter <em>k</em> for which total log-loss has a minimum. In practice, numerical algorithms stop after some number of steps and may either return the value of <em>k</em> in the last step or an error.</p><h3>To be continued</h3><p>This post covers the main part of what logistic regression conceptually does. In the next part, we will cover the somewhat unintuitive meaning of parameter k as a log-odds-ratio and show how to run, and break, logistic models in Python libraries statsmodels and scikit-learn:</p><p><a href="https://towardsdatascience.com/logistic-regression-deceptively-flawed-2c3e7f77eac9">Logistic Regression: Deceptively Flawed</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67560de4f492" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/logistic-regression-faceoff-67560de4f492">Logistic regression: Faceoff</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why and How to Adjust P-values in Multiple Hypothesis Testing]]></title>
            <link>https://medium.com/data-science/why-and-how-to-adjust-p-values-in-multiple-hypothesis-testing-2ccf174cdbf8?source=rss-e5f8ebca4ad8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ccf174cdbf8</guid>
            <category><![CDATA[false-positive]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[p-value]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Igor Šegota]]></dc:creator>
            <pubDate>Fri, 05 May 2023 03:58:14 GMT</pubDate>
            <atom:updated>2023-05-09T02:55:10.407Z</atom:updated>
            <content:encoded><![CDATA[<h4>P-values below a certain threshold are often used as a method to select relevant features. Advice below suggests how to use them correctly.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*P1LXW8ikCQilW176SdwG5A.png" /><figcaption>Photo by the author. Taken at Westfield UTC Mall, La Jolla, California.</figcaption></figure><p>Multiple hypothesis testing occurs when we repeatedly test models on a number of features, as the probability of obtaining one or more false discoveries increases with the number of tests. For example, in the field of genomics, scientists often want to test whether any of the thousands of genes have a significantly different activity in an outcome of interest. Or whether <a href="https://xkcd.com/882/">jellybeans cause acne</a>.</p><p>In this blog post, we will cover few of the popular methods used to account for multiple hypothesis testing by adjusting model p-values:</p><ol><li>False Positive Rate (FPR)</li><li>Family-Wise Error Rate (FWER)</li><li>False Discovery Rate (FDR)</li></ol><p>and explain when it makes sense to use them.</p><p>This document can be summarized in the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zfI1agLAdH-Fxhdp_LwUKQ.png" /><figcaption>Image by the author.</figcaption></figure><h3>Create test data</h3><p>We will create a simulated example to better understand how various manipulation of p-values can lead to different conclusions. To run this code, we need Python with pandas, numpy, scipy and statsmodels libraries installed.</p><p>For the purpose of this example, we start by creating a Pandas DataFrame of 1000 features. 990 of which (99%) will have their values generated from a Normal distribution with mean = 0, called a Null model. (In a function norm.rvs() used below, mean is set using a loc argument.) The remaining 1% of the features will be generated from a Normal distribution mean = 3, called a Non-Null model. We will use these as representing interesting features that we would like to discover.</p><pre>import pandas as pd<br>import numpy as np<br>from scipy.stats import norm<br>from statsmodels.stats.multitest import multipletests<br><br>np.random.seed(42)<br><br>n_null = 9900<br>n_nonnull = 100<br><br>df = pd.DataFrame({<br>    &#39;hypothesis&#39;: np.concatenate((<br>        [&#39;null&#39;] * n_null,<br>        [&#39;non-null&#39;] * n_nonnull,<br>    )),<br>    &#39;feature&#39;: range(n_null + n_nonnull),<br>    &#39;x&#39;: np.concatenate((<br>        norm.rvs(loc=0, scale=1, size=n_null),<br>        norm.rvs(loc=3, scale=1, size=n_nonnull),<br>    ))<br>})</pre><p>For each of the 1000 features, p-value is a probability of observing the value at least as large, if we assume it was generated from a Null distribution.</p><p>P-values can be calculated from a cumulative distribution ( norm.cdf() from scipy.stats) which represents the probability of obtaining a value equal to or <strong>less than</strong> the one observed. Then to calculate the p-value we calculate 1 - norm.cdf() to find the probability <strong>greater than</strong> the one observed:</p><pre>df[&#39;p_value&#39;] = 1 - norm.cdf(df[&#39;x&#39;], loc = 0, scale = 1)<br>df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*RXxSmmU6cgkwx_vAR55Nhg.png" /></figure><h3>False Positive Rate</h3><p>The first concept is called a False Positive Rate and is defined as a fraction of null hypotheses that we flag as “significant” (also called Type I errors). The p-values we calculated earlier can be interpreted as a false positive rate by their very definition: they are probabilities of obtaining a value at least as large as a specified value, when we sample a Null distribution.</p><p>For illustrative purposes, we will apply a common (magical 🧙) p-value threshold of 0.05, but any threshold can be used:</p><pre>df[&#39;is_raw_p_value_significant&#39;] = df[&#39;p_value&#39;] &lt;= 0.05<br>df.groupby([&#39;hypothesis&#39;, &#39;is_raw_p_value_significant&#39;]).size()</pre><pre>hypothesis  is_raw_p_value_significant<br>non-null    False                            8<br>            True                            92<br>null        False                         9407<br>            True                           493<br>dtype: int64</pre><p>notice that out of our 9900 null hypotheses, 493 are flagged as “significant”. Therefore, a False Positive Rate is: FPR = 493 / (493 + 9407) = 0.053.</p><p>The main problem with FPR is that in a real scenario we do not a priori know which hypotheses are null and which are not. Then, the raw p-value on its own (False Positive Rate) is of limited use. In our case when the fraction of non-null features is very small, most of the features flagged as significant will be null, because there are many more of them. Specifically, out of 92 + 493 = 585 features flagged true (“positive”), only 92 are from our non-null distribution. That means that a majority or about 84% of reported significant features (493 / 585) are false positives!</p><p>So, what can we do about this? There are two common methods of addressing this issue: instead of False Positive Rate, we can calculate Family-Wise Error Rate (FWER) or a False Discovery Rate (FDR). Each of these methods takes the set of raw, unadjusted, p-values as an input, and produces a new set of “adjusted p-values” as an output. These “adjusted p-values” represent estimates of <em>upper bounds</em> on FWER and FDR. They can be obtained from multipletests() function, which is part of the statsmodels Python library:</p><pre>def adjust_pvalues(p_values, method):<br>   return multipletests(p_values, method = method)[1]</pre><h3>Family-Wise Error Rate</h3><p>Family-Wise Error Rate is a probability of falsely rejecting one or more null hypotheses, or in other words flagging true Null as Non-null, or a probability of seeing one or more false positives.</p><p>When there is only one hypothesis being tested, this is equal to the raw p-value (false positive rate). However, the more hypotheses are tested, the more likely we are going to get one or more false positives. There are two popular ways to estimate FWER: Bonferroni and Holm procedures. Although neither Bonferroni nor Holm procedures make any assumptions about the dependence of tests run on individual features, they will be overly conservative. For example, in the extreme case when all of the features are identical (same model repeated 10,000 times), no correction is needed. While in the other extreme, where no features are correlated, some type of correction is required.</p><h4>Bonferroni procedure</h4><p>One of the most popular methods for correcting for multiple hypothesis testing is a Bonferroni procedure. The reason this method is popular is because it is very easy to calculate, even by hand. This procedure multiplies each p-value by the total number of tests performed or sets it to 1 if this multiplication would push it past 1.</p><pre>df[&#39;p_value_bonf&#39;] = adjust_pvalues(df[&#39;p_value&#39;], &#39;bonferroni&#39;)<br>df.sort_values(&#39;p_value_bonf&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*AL5D2iYccorCkP_ytue41g.png" /></figure><h4>Holm procedure</h4><p>Holm’s procedure provides a correction that is more powerful than Bonferroni’s procedure. The only difference is that the p-values are not all multiplied by the total number of tests (here, 10000). Instead, each sorted p-value is multiplied progressively by a decreasing sequence 10000, 9999, 9998, 9997, …, 3, 2, 1.</p><pre>df[&#39;p_value_holm&#39;] = adjust_pvalues(df[&#39;p_value&#39;], &#39;holm&#39;)<br>df.sort_values(&#39;p_value_holm&#39;).head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*UpPMsnpsFZFWh9kagKxPUA.png" /></figure><p>We can verify this ourselves: the last 10th p-value on this output is multiplied by 9991: 7.943832e-06 * 9991 = 0.079367. Holm’s correction is also the default method for adjusting p-values in p.adjust() function in R language.</p><p>If we again apply our p-value threshold of 0.05, let’s take a look how these adjusted p-values affect our predictions:</p><pre>df[&#39;is_p_value_holm_significant&#39;] = df[&#39;p_value_holm&#39;] &lt;= 0.05<br>df.groupby([&#39;hypothesis&#39;, &#39;is_p_value_holm_significant&#39;]).size()</pre><pre>hypothesis  is_p_value_holm_significant<br>non-null    False                            92<br>            True                              8<br>null        False                          9900<br>dtype: int64</pre><p>These results are much different than when we applied the same threshold to the raw p-values! Now, only 8 features are flagged as “significant”, and all 8 are correct — they were generated from our Non-null distribution. This is because the probability of getting even one feature flagged incorrectly is only 0.05 (5%).</p><p>However, this approach has a downside: it failed to flag other 92 Non-null features as significant. While it was very stringent to make sure none of the null features slipped in, it was able to find only 8% (8 out of 100) non-null features. This can be seen as taking a different extreme than the False Positive Rate approach.</p><p>Is there a more middle ground? The answer is “yes”, and that middle ground is False Discovery Rate.</p><h3>False Discovery Rate</h3><p>What if we are OK with letting some false positives in, but capturing more than single-digit percent of true positives? Maybe we are OK with having <em>some</em> false positive, just not that many that they overwhelm all of the features we flag as significant — as was the case in the FPR example.</p><p>This can be done by controlling for False Discovery Rate (rather than FWER or FPR) at a specified threshold level, say 0.05. False Discovery Rate is defined a fraction of false positives among all features flagged as positive: FDR = FP / (FP + TP), where FP is the number of False Positives and TP is the number of True Positives. By setting FDR threshold to 0.05, we are saying we are OK with having 5% (on average) false positives among all of our features we flag as positive.</p><p>There are several methods to control FDR and here we will describe how to use two popular ones: Benjamini-Hochberg and Benjamini-Yekutieli procedures. Both of these procedures are similar although more involved than FWER procedures. They still rely on sorting the p-values, multiplying them with a specific number, and then using a cut-off criterion.</p><h4>Benjamini-Hochberg procedure</h4><p>Benjamini-Hochberg (BH) procedure assumes that each of the tests are <em>independent</em>. Dependent tests occur, for example, if the features being tested are correlated with each other. Let’s calculate the BH-adjusted p-values and compare it to our earlier result from FWER using Holm’s correction:</p><pre>df[&#39;p_value_bh&#39;] = adjust_pvalues(df[&#39;p_value&#39;], &#39;fdr_bh&#39;)<br>df[[&#39;hypothesis&#39;, &#39;feature&#39;, &#39;x&#39;, &#39;p_value&#39;, &#39;p_value_holm&#39;, &#39;p_value_bh&#39;]] \<br>    .sort_values(&#39;p_value_bh&#39;) \<br>    .head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*0pJvl8639tDJUIMj0jY6OQ.png" /></figure><pre>df[&#39;is_p_value_holm_significant&#39;] = df[&#39;p_value_holm&#39;] &lt;= 0.05<br>df.groupby([&#39;hypothesis&#39;, &#39;is_p_value_holm_significant&#39;]).size()</pre><pre>hypothesis  is_p_value_holm_significant<br>non-null    False                            92<br>            True                              8<br>null        False                          9900<br>dtype: int64</pre><pre>df[&#39;is_p_value_bh_significant&#39;] = df[&#39;p_value_bh&#39;] &lt;= 0.05<br>df.groupby([&#39;hypothesis&#39;, &#39;is_p_value_bh_significant&#39;]).size()</pre><pre>hypothesis  is_p_value_bh_significant<br>non-null    False                          67<br>            True                           33<br>null        False                        9898<br>            True                            2<br>dtype: int64</pre><p>BH procedure now correctly flagged 33 out of 100 non-null features as significant — an improvement from the 8 with the Holm’s correction. However, it also flagged 2 null features as significant. So, out of the 35 features flagged as significant, the fraction of incorrect features is: 2 / 33 = 0.06 so 6%.</p><p>Note that in this case we have 6% FDR rate, even though we aimed to control it at 5%. FDR will be controlled at a 5% rate <em>on average</em>: sometimes it may be lower and sometimes it may be higher.</p><h4>Benjamini-Yekutieli procedure</h4><p>Benjamini-Yekutieli (BY) procedure controls FDR regardless of whether tests are independent or not. Again, it is worth noting that all of these procedures try to establish <em>upper bounds</em> on FDR (or FWER), so they may be less or more conservative. Let’s compare the BY procedure with a BH and Holm procedures above:</p><pre>df[&#39;p_value_by&#39;] = adjust_pvalues(df[&#39;p_value&#39;], &#39;fdr_by&#39;)<br>df[[&#39;hypothesis&#39;, &#39;feature&#39;, &#39;x&#39;, &#39;p_value&#39;, &#39;p_value_holm&#39;, &#39;p_value_bh&#39;, &#39;p_value_by&#39;]] \<br>    .sort_values(&#39;p_value_by&#39;) \<br>    .head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*yVMcSVPKz1sHLSTTbaZ0Hw.png" /></figure><pre>df[&#39;is_p_value_by_significant&#39;] = df[&#39;p_value_by&#39;] &lt;= 0.05<br>df.groupby([&#39;hypothesis&#39;, &#39;is_p_value_by_significant&#39;]).size()</pre><pre>hypothesis  is_p_value_by_significant<br>non-null    False                          93<br>            True                            7<br>null        False                        9900<br>dtype: int64</pre><p>BY procedure is stricter in controlling FDR; in this case even more so than the Holm’s procedure for controlling FWER, by flagging only 7 non-null features as significant! The main advantage of using it is when we know the data may contain a high number of correlated features. However, in that case we may also want to consider filtering out correlated features so that we do not need to test all of them.</p><h3>Summary</h3><p>At the end, the choice of procedure is left to the user and depends on what the analysis is trying to do. Quoting Benjamini, Hochberg (Royal Stat. Soc. 1995):</p><blockquote>Often the control of the FWER is not quite needed. The control of the FWER is important when a conclusion from the various individual inferences is likely to be erroneous when at least one of them is.</blockquote><blockquote>This may be the case, for example, when several new treatments are competing against a standard, and a single treatment is chosen from the set of treatments which are declared significantly better than the standard.</blockquote><p>In other cases, where we may be OK to have some false positives, FDR methods such as BH correction provide less stringent p-value adjustments and may be preferrable if we primarily want to increase the number of true positives that pass a certain p-value threshold.</p><p>There are other adjustment methods not mentioned here, notably a <a href="https://en.wikipedia.org/wiki/Q-value_(statistics)">q-value</a> which is also used for FDR control, and at the time of writing exists only as an R package.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ccf174cdbf8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-and-how-to-adjust-p-values-in-multiple-hypothesis-testing-2ccf174cdbf8">Why and How to Adjust P-values in Multiple Hypothesis Testing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>