<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marco Paciscopi on Medium]]></title>
        <description><![CDATA[Stories by Marco Paciscopi on Medium]]></description>
        <link>https://medium.com/@marco.paciscopi?source=rss-2b62a1409d2a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mtGpt_q3VjmY4JUeVQVkVQ.jpeg</url>
            <title>Stories by Marco Paciscopi on Medium</title>
            <link>https://medium.com/@marco.paciscopi?source=rss-2b62a1409d2a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:33:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marco.paciscopi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Survival Analysis: main approaches and their usage]]></title>
            <link>https://medium.com/eni-digitalks/survival-analysis-main-approaches-and-their-usage-b03aa24b5565?source=rss-2b62a1409d2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b03aa24b5565</guid>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[survival]]></category>
            <category><![CDATA[survival-analysis]]></category>
            <dc:creator><![CDATA[Marco Paciscopi]]></dc:creator>
            <pubDate>Wed, 07 Sep 2022 11:31:08 GMT</pubDate>
            <atom:updated>2022-09-07T11:31:08.101Z</atom:updated>
            <content:encoded><![CDATA[<p>An exhaustive overview of the main approaches of Survival Analysis will give you a better idea of their application in Enterprise Use Cases</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Z0aCj8qfVPS4GEE5.png" /><figcaption>Source <a href="https://unsplash.com/">https://unsplash.com/</a></figcaption></figure><p>Many data scientists have likely already come across with the concept of survival analysis. First of all, it’s important to define what a survival analysis is: it’s a popular data analysis approach for which the outcome of interest is the <strong>time until an event occurs</strong>.</p><p>Let’s analyze more deeply the single terms of this definition.</p><p>The<strong> Event </strong>represents any experience of interest that may happen to an individual, for example disease, incidence or recovery.</p><p>The <strong>Time</strong> simply refers to the <strong>time span from when you start to follow-up</strong> an individual <strong>until the Event occurs.</strong></p><p>We usually refer to this as <strong>Survival Time</strong> because traditionally survival analysis originated from medical research to measure lifespans of individuals of a population under examination.</p><p>Later, the same concepts have also been applied to many other fields:</p><ul><li>Predict when a client will churn in retail companies</li><li>Predict customer’s time to purchase a product</li><li>Predict the lifetime of a machine in manufacturing companies.</li></ul><h3>The Censoring Problem</h3><p>A key problem in survival analysis is dealing with <em>censored</em> observations, which occur when we don’t know exactly the survival time of individuals. There are three possible censoring schemes:</p><p><strong>Right Censoring</strong>: This happens when the follow-up period ends before the event of interest</p><ul><li>For example, a retailer wants to predict the retention time of its clients. Right censoring occurs if a client doesn’t churn before the end of the observation period.</li></ul><p><strong>Left Censoring</strong>: This happens when the period of study starts after the event of interest</p><ul><li>An example is the study of age at which children learn a new task. Left censoring occurs if some children already knew the task before the observation period.</li></ul><p><strong>Interval Censoring: </strong>The only information is that the event occurs within some interval</p><ul><li>For example, in industrial experiments periodic inspections are made for proper functioning of equipment items. If a malfunction occurs between two consecutive inspections, the real survival time of the item is only known to fall in the interval</li></ul><h3>Survival and Hazard Functions</h3><p>Let <em>T</em> be the random variable representing the time until the event of interest occurs, like the churn event of a client or the failure of a machine. Two important quantitative measures are used in Survival Analysis to describe its distribution: <strong>Survival Function </strong>and <strong>Hazard Function.</strong></p><p><strong>Survival Function</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/0*-V5mFlNE-TTQYuGb.png" /><figcaption>Formula [1] Definition of Survival Function</figcaption></figure><p>It defines the probability that the event of interest has not occurred yet at time <em>t </em>(in other terms, it is the probability of an individual surviving beyond time <em>t</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/0*0BtEf4in8zzvO4Uc.png" /><figcaption>Figure 1: Left: In theory Survival Function is a decreasing smooth curve. Right: In practice, it is a step function</figcaption></figure><p><strong>Hazard Function</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/0*lRTZeuSiwijjYvis.png" /><figcaption>Formula [2] Definition of Hazard Function</figcaption></figure><p>The numerator of <em>h(t) </em>is the probability that an individual will experience the event of interest in the interval <em>[t, t + </em>Δ<em>t) </em>given that he has survived up to time <em>t, </em>while the denominator is the width of the interval. Hence, the limit of the rate when Δ<em>t </em>goes down to zero can be defined as the instantaneous rate of occurrence of the event of interest. Keep in mind that <strong><em>h(t)</em> is a rate, not a probability!</strong></p><h3>Possible Scenarios of the Hazard Functions</h3><p>The hazard function may have different shapes according to different scenarios, its only constraint is that it must be nonnegative. The shape of hazard functions gives important insights about survival. Typical shapes are the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/0*1NmWShajQNkSlv79.png" /><figcaption>Figure 2: Typical shapes of hazard functions. From left to right: constant, increasing and decreasing</figcaption></figure><p><strong>Constant Hazard</strong></p><p>Such a shape can be expected in all the situations where the instantenous rate of occurrence of the event of interest is the same at any time. From formula [3] we notice that, when the hazard function is constant, the survival function has an exponential distribution. This kind of assumption may be valid for some high quality electronic components, for which the effect of time can be assumed to be negligible.</p><p><strong>Increasing Hazard</strong></p><p>Increasing hazard functions are the most common ones because they arise when there is a natural aging. For example, this assumption can be used to model the longevity of a vehicle or a battery, that tend to wear out over time.</p><p><strong>Decreasing Hazard</strong></p><p>This graph is expected to represent the longevity of all those products that are are worth more over time than new products. Generally speaking, a decreasing hazard shape arises when individuals have an high early risk of failure, that goes down over time. Quoting Nietzsche, <em>“That which does not kill us, makes us stronger.”</em></p><p>Let’s see now a less standard shape of hazard function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*ZfprAtaPyhHFFkAa.png" /><figcaption>Figure 3: Bathtub shaped hazard function</figcaption></figure><p>Can you think of any real case of hazard function with this kind of shape?</p><p>This is an example of “bathtub” shaped hazard cuntion. Many electronical devices, for example televisions, show this kind of hazard function because the risk of failure can be high initially, low in the center and high again at the end of life because of the increasing wearout</p><h3>Why is the Hazard Function so important?</h3><p>There are multiple reasons behind its importance. In particular:</p><ul><li>It provides an instantaneous potential of risk while the Survival Function is only a cumulative measure over time,</li><li>Most of statistical techniques model survival analysis data in terms of the hazard function.</li></ul><p>A simple mathematical relationship links the two functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/0*8Had19Av-niRHhsm.png" /><figcaption>Formula [3] Survival Function in terms of Hazard Function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/0*EbbYjeWsRP9beb9P.png" /><figcaption>Formula [4] Hazard Function in terms of Survival Function</figcaption></figure><p>The term <em>H(t)</em> is the <strong>cumulative hazard function</strong>. It represents, like the name suggests, the accumulation of the hazard over time. It can be interpreted as the total amount of risk that has been accumulated up to time <em>t. </em>So formula [3] says the higher the accumulation of hazard up to time <em>t</em>, the lower the probability of an individual surviving beyond time <em>t</em>.</p><h3>Objectives of Survival Analysis</h3><p>Survival Analysis has three objectives:</p><ul><li><strong>Estimate and interpret survival and hazard functions, </strong>given a set of survival data. There are many univariate methods of doing that:<br>- <strong>Kaplan-Meier</strong>, a non-parametric method to estimate the survival function<br>- <strong>Nelson-Aalen</strong>, a non-parametric estimator of the cumulative hazard function<br>- Parametric models to represent survival and hazard functions like Weibull, Exponential, Log-Logistic</li><li><strong>Compare survivor and/or hazard functions</strong> with statistical tests</li><li><strong>Assess the relationship between explanatory variables and survival time</strong>:<br>- Cox’s proportional hazard<strong> </strong>and other survival regression models use additional data as covariates other than the duration and censorship variables</li></ul><h3>Estimate and interpretation of survival and hazard functions with Kaplan-Meier method</h3><p>Kaplan-Meier (KM) is the standard method to estimate the survival function. The estimator is defined as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*Sfqa9cOd0EwzwIPP.png" /><figcaption>Formula [5] The Kaplan-Meier Estimator</figcaption></figure><p>where <em>dᵢ</em> is the number of events of interest at time <em>tᵢ</em> and <em>Yᵢ</em> is the number of individuals at risk before time t<em>ᵢ</em> (i.e. they have not yet had an event or been censored up to time t<em>ᵢ</em>). The ratio (<em>Yᵢ — dᵢ)/Yᵢ </em>is used to estimate the conditional probability that an individual, who survives to at least time <em>tᵢ</em>, survives past time <em>tᵢ</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/0*-VLPiUF8VOX1_uMh.png" /><figcaption>Formula [6] Conditional Probability to survive past time</figcaption></figure><p>So, the KM estimator at time <em>tᵢ</em> can be equivalently defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*LKvmwWUzkaXCW7Ek.png" /><figcaption>Formula [7] Equivalent definition of Kaplan-Meier Estimator</figcaption></figure><p>Let’s see now an example of construction of the Kaplan-Meier survival curve for an enterprise case.</p><p>Consider 10 clients <em>A, …, L</em> of a Utility Company, who activate their gas or power service at the starting time <em>t₀</em> of the observation period. Their retention times are shown in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*5kpmV5XhaJYC7znNocE71A.jpeg" /><figcaption>Figure 4: Example of retention times of a company clients</figcaption></figure><p>Clients <em>I, H, G</em> are right censored because they don’t churn within the Observation Period.</p><p>Client <em>C </em>is censored because his power service is interrupted for non-payment (technically not a churn event).</p><p>Clients <em>A, B, D, E, F, L</em> churn before within the observation period.</p><p>The KM estimates at each event of interest are then computed with formula [7]. See the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*UAEGduzfXZ8obZOT.png" /><figcaption>Table [1] Table representing survival data of Figure 5</figcaption></figure><p>The KM estimates can be reported in the following plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*LW-UKBcRmUXC3J5T.png" /><figcaption>Figure 5: Kepler-Meier estimate of survival data of Table [1]</figcaption></figure><h3>Compare survival curves with statistical tests — the Log-rank Test</h3><p>Let’s consider again the example of churn for a client base of an energy utility company. Imagine analyzing the churn phenomenon between two groups of different clients, for example gas and power clients or groups belonging to different ranges of age.</p><p>For each churn event time <em>tᵢ</em> we calculate the expected number of churn individuals under the null hypothesis that there were in reality no differences in survival between the two groups. So, at each churn event time <em>tᵢ</em>, the total churn events (<em>d₁ᵢ + d₂ᵢ</em>)<em> </em>should split between the two groups. The expected churn events for each group are computed with the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*bnEtEJ3VhPEFAM07.png" /><figcaption>Formula [8] Expected counts definition in Log-Rank Test for two groups of individuals</figcaption></figure><p>We report the survival data for this example and all the calculations in the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*h8APFxoFAHxqoJ1L.png" /><figcaption>Table [2] Example Survival Data for Log-Rank test</figcaption></figure><p>For the 2-group case, we can compute the Log-rank statistic with the following formula, that is equivalent for each one of the two groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/0*geuer94P5whEEPi6.png" /><figcaption>Formula [9] Definition of Log-rank statistic</figcaption></figure><p>where <em>Oᵢ</em> is the sum of the observed counts and <em>Eᵢ</em> s the sum of the expected counts. Under the Null Hypothesis, it is approximately a chi-square statistic with 1 degree of freedom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/0*3JFhUY5DlqHj6LKZ.png" /><figcaption>Formula [10] Approximate formula for Log-rank statistic</figcaption></figure><p>Returning to the example in Table [2], the Log-rank statistic is <em>χ = (7 -13.92)²/13.82 + (15–8.08)²/8.08 = 9.35. </em>Computing the p-value of <em>χ² </em>and considering the significance level of <em>0.05, </em>we can reject the null hypothesis. In other terms we can say that the two survival functions are not statistically equivalent.</p><h3>Assess the relationship between explanatory variables and survival time — Survival Regression with Cox proportional hazard model</h3><p>Finally, we introduce the Survival Regression Models that model the relationship between the survival time of a population and a set of explanatory variables. The most popular one is the <strong>Cox’s Proportional Hazard (PH) Model</strong>.</p><p>While the previous univariate models like Kaplan-Meier attempt to estimate the survival or the hazard functions with only the duration and censorship variables, the Cox PH Model attempts to<strong> represent the hazard rate as a function of time<em> t </em>and a set of its explanatory variables X.</strong></p><p>The formula is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/0*al1oRbMNCeS9EOvs.png" /><figcaption>Formula [11] The Proportional Hazard Assumption</figcaption></figure><p>Cox’s proportional hazard model is semi-parametric because the baseline hazard <em>hₒ(t) </em>is an unspecified function. Instead, the coefficients <em>β </em>can be estimated maximizing a likelihood function without the need of modelling the baseline hazard function. In fact, Cox’s proportional hazard model does not directly estimate the baseline hazard function, so another univariate estimator is needed to estimate it and obtain the survival probabilities.</p><p>Let’s consider again the example of churn in a utility: <strong>we could use the Cox PH Model to predict the retention time of customers</strong>, given a list of time-independent features correlated with churn, for example the residence, payment modality or the acquisition channel.</p><p>A strong limitation of Cox PH Model is that the explanatory variables must be non-time dependent, but in real use cases the features of individuals change over time. Just think of the case of churn, in which the survival of a customer can depend for example on the last received bills, that obviously change over time. Hence an <strong>extended version of Cox model </strong>can be used, but at the cost of not satisfying the Proportional Hazard Assumption.</p><h3>Conclusion</h3><p>We hope that this article gave you a deeper and clearer understanding of the main approaches of Survival Analysis. Just keep in mind that this is only the launch pad for a broader variety of topics and more complex approaches, that you can find especially in <em>Kleinbaum, D. G., &amp; Klein, M. (2011)</em> [1].</p><p>We tried to combine the theory parts with simple practical examples to help you to digest not immediate concepts and apply them to real business use cases.</p><p>Finally, if you are a python user and you want to implement all the previous approaches and the more sophisticated ones, you could try <strong>Lifelines</strong>, a complete survival analysis library written in Python [4].</p><h3>References</h3><p>[1] Kleinbaum, D. G., &amp; Klein, M. (2011). Survival Analysis: A Self-Learning Text, Third Edition</p><p>[2] <a href="https://www.theanalysisfactor.com/survival-analysis-interpreting-shapes-of-hazard-functions/">Interpreting the Shapes of Hazard Functions in Survival Analysis — The Analysis Factor</a> (https://www.theanalysisfactor.com/survival-analysis-interpreting-shapes-of-hazard-functions)</p><p>[3] Bland JM, Altman DG. The logrank test. BMJ. 2004;328(7447):1073. doi:10.1136/bmj.328.7447.1073</p><p>[4] Davidson-Pilon , <em>Lifelines: survival analysis in Python</em>, Journal of Open Source Software, 4(40), 1317, <a href="https://doi.org/10.21105/joss.01317">https://doi.org/10.21105/joss.01317</a> (2019)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b03aa24b5565" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eni-digitalks/survival-analysis-main-approaches-and-their-usage-b03aa24b5565">Survival Analysis: main approaches and their usage</a> was originally published in <a href="https://medium.com/eni-digitalks">Eni digiTALKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>