<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikolas Weissmueller on Medium]]></title>
        <description><![CDATA[Stories by Nikolas Weissmueller on Medium]]></description>
        <link>https://medium.com/@nikolas.weissmueller?source=rss-3e54d31a607d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*woCVVs_FC_qtmouF-EsfFA.jpeg</url>
            <title>Stories by Nikolas Weissmueller on Medium</title>
            <link>https://medium.com/@nikolas.weissmueller?source=rss-3e54d31a607d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:39:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikolas.weissmueller/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mind Your Metrics: Beyond FSA and LSA for Site Activation]]></title>
            <link>https://medium.com/@nikolas.weissmueller/mind-your-metrics-beyond-fsa-and-lsa-for-site-activation-2249593e049e?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2249593e049e</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[clinical-operations]]></category>
            <category><![CDATA[clinical-trials]]></category>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 22:42:36 GMT</pubDate>
            <atom:updated>2026-09-09T04:55:27.523Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3o9CNWVNlTemQUxnGvgWw.png" /><figcaption><strong>Figure 1.</strong> FSA and LSA mark endpoints; the capacity trajectory between them determines enrollment impact. The stochastic example is illustrative: country first-site activation is Gamma-distributed and subsequent-site lags are Exponential. Capacity is weighted by baseline expected PSM. At month 3, the linear ramp has 50% of capacity online versus about 65% in the stochastic example.</figcaption></figure><p>Better questions lead to better answers, and better metrics enable better questions. Clinical operations is replete with metrics: counts, dates, durations, and rates. They are often simple by design, but simplicity can come at the cost of information value. Optimizing a metric that poorly reflects the process is unlikely to improve the outcome. Site activation provides a useful example. Comparing a simple milestone metric with a more operationally meaningful alternative shows how better measurement can enable better questions, more actionable insights, and ultimately better decisions.</p><p>Teams routinely track startup using two bookend milestones: <strong>First Site Activated (FSA)</strong> and <strong>Last Site Activated (LSA)</strong>.</p><blockquote>FSA and LSA are useful milestones, but neither measures what ultimately matters for enrollment: <strong>how quickly recruitment capacity comes online</strong>.</blockquote><h3>A Better Startup Metric</h3><p>A simple alternative is the <strong>enrollment-weighted mean activation time</strong>. Let <em>Λi</em> be the uncertain post-activation enrollment rate (patients per site per month or PSM) of site <em>i</em>. At planning, use the best available expected PSM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*WlxsaJ-VwxMp35a-GxXKwg.png" /></figure><p>If site<em> i</em> activates at time <em>ai</em>, define</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*0cOhfDBMQ_ozC5uW8hbBvA.png" /></figure><p>where lower is better. <strong>This is simply the average activation time, weighted by the enrollment capacity each site was expected to contribute. </strong>A late site expected to contribute little receives little weight. A delayed site expected to contribute 10% of total capacity matters ten times more, for the same delay, than one expected to contribute 1%. That is closer to the operational objective than either FSA or LSA.</p><h3>Why It Works</h3><p>Let the mature expected enrollment rate be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*iiZEyx5onlOq1LE3AdUUQQ.png" /></figure><p>At time <em>t</em>, the fraction of baseline expected capacity online is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*Hcgk2XR7q9FVQrnyWvXOqQ.png" /></figure><p><strong>If <em>F0(t) </em>= 0.60, then 60% of the enrollment engine expected at baseline is active, whether that comes from three large centers or thirty small ones.</strong> Cumulative expected enrollment is the area under this capacity curve:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*rPjef-1i1hnrImLX4K1AxA.png" /></figure><p>where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*Y7Ci5IwBKC2RhtAh1jIpXA.png" /></figure><p><strong>Enrollment accumulates as expected capacity comes online. </strong>After all planned sites have activated, the missing area above the capacity curve is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*5Hsl5-gQtMZ9tVMz1Yfcsg.png" /></figure><p>Because the weights are fixed,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*0cOhfDBMQ_ozC5uW8hbBvA.png" /></figure><p>Therefore, the weighted mean is exactly the <strong>post-startup displacement of the enrollment</strong>. If the enrollment target <em>N*</em> is reached after all planned sites are active, then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*bAFg7Lqhxo68dfcC3N5EqQ.png" /></figure><p>The first term is how long enrollment would take if all of the expected capacity existed on day one. The second is the timeline cost of bringing that capacity online gradually. For a linear capacity ramp from zero at <em>t=0</em> to full capacity at <em>S</em>, <em>τ0</em> = <em>0.5 S, </em>where S is LSA — FSA. If the ramp starts at FSA =<em>a0</em> and lasts <em>S</em> months,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*6Eu7-QKBjcuxk5bVdwRJ3w.png" /></figure><p><strong>While useful as a rule of thumb, the weighted mean is the more general result. </strong>Importantly, <em>τ0</em> does not replace the full curve. <strong>Two activation patterns can have the same weighted mean but different early enrollment trajectories. </strong>Use <em>F0(t)</em> to understand startup as it unfolds, and <em>τ0</em> to summarize its eventual timeline penalty.</p><h3>We Do Not Know Which Sites Will Perform: no problem</h3><p>Expected PSM is a <strong>weight</strong>, not a known site property. A site expected to enroll at 4 PSM may open and deliver 0.5 PSM. That does not make the original weight arbitrary; it means the outcome was uncertain. Three views should therefore remain separate:</p><ul><li><strong>Plan:</strong> What we expected before startup.:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*YjIqEopvTHOkPkJHfyd1ag.png" /></figure><ul><li><strong>Current forecast: W</strong>hat we expect after observing new evidence:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/1*Vad9_cMnhQrDhg8Ao7z7dg.png" /></figure><ul><li><strong>Realized:</strong> What ultimately materialized:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/36/1*3ZZIznELZFQOW7nGWr6kKg.png" /></figure><p>For <strong>startup performance</strong>, we keep the baseline weights <em>μi</em> fixed. Otherwise, a delayed site could improve the startup metric simply because its PSM forecast was revised downward after opening.</p><p>For <strong>enrollment forecasting</strong>, update PSM as evidence arrives. In stochastic planning, activation time and PSM are both uncertain. For simulation <em>m</em>, calculate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*A4ma8QrTbIaeQAuEI3boPw.png" /></figure><p>and, when completion occurs after startup in that realization,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*5bLihJ7-zpPnQPSDCQsQ9A.png" /></figure><p><strong>uncertainty does not invalidate the weighted mean, but turns it into a distribution. </strong>However, we should not average the inputs first:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*fewEt52CCIAvMjE28RKuqQ.png" /></figure><p>Activation and PSM should therefore be simulated jointly when they are related.</p><h3>Applications in Planning</h3><p>The metric immediately informs practical questions:</p><ul><li><strong>How much of the enrollment timeline is startup?</strong> Compare <em>τ0</em> with <em>N*/R0</em>.</li><li><strong>Which sites or countries are worth accelerating?</strong> If site <em>i</em> is moved <em>δi &gt; 0</em> months earlier, while full planned capacity is unchanged, operational effort should follow <strong>expected site capacity at risk</strong>, not raw site counts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*sNSw-8b5BYxrJFg1ZKzw0g.png" /></figure><ul><li><strong>How uncertain is completion?</strong> Draw activation times and PSM jointly and propagate them through τ and the enrollment model rather than adding an arbitrary buffer to LSA.</li></ul><h3>Applications in Monitoring</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XbLq81z-oFsLAMuIApSL0g.png" /><figcaption><strong>Figure 2.</strong> Planned versus actual startup using fixed baseline expected PSM weights. The top panel shows the fraction of baseline expected capacity activated over time; the shaded area is the startup deficit relative to plan. The lower panel shows the integrated startup slip, D(t)_startup, in equivalent months at full planned capacity.</figcaption></figure><p>For startup execution, we retain the baseline weights <em>μi</em> and planned denominator <em>R0</em>. This prevents changing PSM forecasts from rewriting startup performance retrospectively. Let <em>P(t)</em> and <em>A(t)</em> indicate whether site <em>i</em> was planned and actually active at time <em>t</em>, and define</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*TqJLIMlgMoF18hAEeiOPEw.png" /></figure><ul><li><strong>How much expected capacity is actually online?</strong> Track <em>C_E(t)/R0</em> against <em>C_P(t)/R0</em>, as in Figure 2. This shows whether startup is delivering the enrollment capacity expected at baseline.</li><li><strong>How far behind plan is startup?</strong> Integrate the gap between planned and actual expected capacity. This expresses startup difference in equivalent months at full planned capacity.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*RLZFtKXv68c6Yi-WGYaYwQ.png" /></figure><ul><li><strong>Is the problem startup or site performance?</strong> Once site performance is observed, let realized PSM be 𝜆<em>i</em> and define</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*xL20vp_n1XvEhXlgsyz_Yw.png" /></figure><p>Then exactly,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*Y_-2pdw_KzlC4IJm4pwZSg.png" /></figure><blockquote>This separates <strong>capacity that came online late</strong> from <strong>capacity that enrolled below expectation</strong>.</blockquote><ul><li><strong>Where should intervention go?</strong> Prioritize delayed sites and countries by expected capacity at risk and the acceleration still achievable, rather than by site count alone.</li><li><strong>Has the enrollment forecast changed?</strong> Keep baseline weights fixed for measuring startup execution, but update PSM estimates separately for the current enrollment forecast as new evidence arrives.</li></ul><h3>Scope and Assumptions</h3><p>The formulation is deliberately simplified. It assumes approximately constant post-activation site rates (PSM), additive contributions, and no binding caps, competition, site closures, or patient-pool exhaustion.</p><p>If enrollment completes before all relevant capacity activates, solve the accrual curve directly</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*bAFg7Lqhxo68dfcC3N5EqQ.png" /></figure><p>With stochastic patient arrivals, that expression describes the conditional mean or fluid accrual trajectory; the actual enrollment-completion time remains stochastic and is best obtained by simulation.</p><h3>Takeaway</h3><p><strong>FSA and LSA are useful milestones, but weak metrics for managing enrollment.</strong> A better single-number startup metric is the <strong>enrollment-weighted mean activation time</strong>: the average activation time weighted by expected PSM. It does not require knowing in advance which sites will actually become high performers. Endpoints mark thve boundaries of startup. The capacity curve determines its impact. Site activation is only one example. <strong>The broader lesson is that thoughtfully designed metrics enable better questions, sharper insights, and more actionable decisions.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2249593e049e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Accelerating Clinical Trial Enrollment Has Diminishing Returns]]></title>
            <link>https://medium.com/@nikolas.weissmueller/why-accelerating-clinical-trial-enrollment-has-diminishing-returns-8e73921d46f0?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e73921d46f0</guid>
            <category><![CDATA[pharmaceutical]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[clinica-operations]]></category>
            <category><![CDATA[biotechnology]]></category>
            <category><![CDATA[strategic-planning]]></category>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 10:44:29 GMT</pubDate>
            <atom:updated>2025-04-07T10:57:18.866Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oa3IfnRbXyYP_ME-dvETEg.png" /><figcaption>Clinical trial timeline showing enrollment period versus readout acceleration.</figcaption></figure><p>In pharmaceutical development, delays can wipe out billions of dollars. Clinical programs routinely aim to accelerate trial timelines by expediting patient enrollment. Yet, there exists a mathematical reality that consistently disappoints sponsors and is not widely known: <strong>when you accelerate enrollment by time A, you only accelerate the final study readout by about A/2.</strong></p><blockquote>This “half-gain paradox” means slashing six months from your enrollment period typically delivers merely three months in actual timeline savings.</blockquote><p>This counterintuitive relationship isn’t merely an empirical observation — it’s a mathematical certainty that emerges from the fundamentals of how events accumulate in clinical trials.</p><h3>When does this happen?</h3><p>To better understand this phenomenon, let’s examine the anatomy of time-to-event trials, where statistical power hinges on observing a pre-specified number of events (such as disease progression, hospitalization, or death).</p><p>In these trials:</p><ul><li>We require a target number of events (E) to achieve statistical power</li><li>Events occur at an approximately constant hazard rate (λ) across the population</li><li>Patients enter the study sequentially during the enrollment period</li><li>The study concludes once E events have been observed</li></ul><p>When a subject enrolls, they begin accumulating “exposure time” during which they could experience an event. The cumulative exposure time across all subjects — not merely the number of subjects — determines when we reach our event target.</p><p><strong>The critical insight: patients enrolled early contribute disproportionately more exposure time than those enrolled later. </strong>A patient enrolled on day one may contribute years of follow-up, while those enrolled in the final month contribute negligible initial exposure.</p><h3>Why does this happen?</h3><p>Consider an oncology trial with a 12-month enrollment period:</p><ul><li>A patient enrolled on day 1 contributes 12 months of exposure by the end of enrollment</li><li>A patient enrolled at month 6 contributes only 6 months of exposure</li><li>A patient enrolled on the final day contributes virtually zero exposure</li></ul><p>Now, suppose we compress enrollment to 9 months instead of 12 (saving 3 months):</p><ul><li>The total sample size remains identical (N)</li><li>The enrollment rate increases proportionally (from N/12 to N/9 patients per month)</li><li>The exposure distribution shifts accordingly</li></ul><p>The mathematics dictates that for uniform enrollment, the time saved in final readout will be precisely half the time saved in enrollment — in this case, 1.5 months. Let’s derive this relationship formally for a simplified case:</p><p><strong>Assumptions:</strong></p><ul><li>Constant event hazard rate λ across time</li><li>Uniform enrollment rate within each scenario</li><li>Required number of events = E</li><li>Total sample size = N (constant between scenarios)</li><li>Original enrollment period = T</li><li>Accelerated enrollment period = T-A</li></ul><p><strong>Original Scenario:</strong> For a patient enrolled at time s, their exposure time at timepoint t is (t-s). The enrollment rate is N/T patients per unit time. The expected number of events by time t (where t &gt; T) is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*gNsrw4Gvr9U3z_HF" /></figure><p><strong>Solving:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/0*obCxjhzGRIk_ocDw" /></figure><p><strong>Accelerated Scenario:</strong> With enrollment period (T-A), the enrollment rate increases to N/(T-A).</p><p>The expected number of events by time t’ is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DrsLZqpJdMj5vdJY" /></figure><p>Solving:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T2dLWn3MNTUMEIJ_" /></figure><p><strong>Time Saved:</strong> Setting E(t) = E’(t’) = E (required events):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*YgEXZfxeHYB2A_ZC" /></figure><p>Therefore, <strong>the time saved (t — t’) equals precisely A/2</strong>.</p><h3>Hard truths about enrollment acceleration</h3><ul><li>Enrollment acceleration translates to only half as much readout acceleration</li><li>The relationship is not optional but a mathematical certainty</li><li>The phenomenon applies to any time-to-event endpoint (OS, PFS, etc.)</li><li>This relationship holds regardless of treatment effect size</li></ul><h3>Strategic implications</h3><p>Understanding this mathematical relationship enables more realistic planning:</p><ul><li>To compress a timeline by six months via speeding up enrollment only, focus on accelerating by 12 months</li><li>Accrual has a hard physical resource limit to what acceleration is possible and decreasing marginal returns along the way</li><li>To drastically accelerate study readout, consider other levers beyond the operational space, such as enrichment strategies, dynamic borrowing, adaptive designs, interim analyses, etc.</li></ul><h3>The bottom line</h3><p>The half-gain paradox represents an immutable mathematical reality in time-to-event trials. While perhaps deflating to sponsors eager for timeline acceleration via clinical trial operations, <strong>acknowledging the half-gain paradox allows for more realistic planning and more effective resource allocation</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e73921d46f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting Data Bias with Kullback-Leibler Divergence]]></title>
            <link>https://medium.com/@nikolas.weissmueller/detecting-data-bias-with-kullback-leibler-divergence-355ebe940b56?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/355ebe940b56</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Thu, 12 Sep 2024 05:23:53 GMT</pubDate>
            <atom:updated>2024-09-12T05:29:23.373Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJVbu2YuHg5NBk00Hxp3nw.png" /><figcaption>KL divergence decreases to zero for unbiased data, but converges to a non-zero value for biased data.</figcaption></figure><p>Data bias can lead to incorrect conclusions and flawed decisions. The Kullback-Leibler (KL) divergence can help identify such biases in datasets, even when unbiased data is not available. The following builds on and implements the concepts described by De Bernardis [1].</p><h4>What is Data Bias?</h4><p>Data bias occurs when the sampling process favors certain data points over others. This can happen due to non-randomized collection methods, faulty sensors, or many other sources of systematic errors.</p><h4>KL Divergence</h4><p>KL divergence measures the difference between two probability distributions. In this context, it compares the distribution of a potentially biased sample to an assumed unbiased distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ojm4j8Mxo3IxF0LCdqkfQ.png" /><figcaption>Mathematical formulation of the Kullback–Leibler divergence. Source: Wikipedia.</figcaption></figure><h4>Starting point: Use Prior Information</h4><p>You don’t need to start with unbiased data. Instead, you can:</p><ol><li>Use prior knowledge about the expected distribution.</li><li>Make educated guesses about the unbiased distribution.</li><li>Iteratively refine your assumption of the true distribution.</li></ol><h4>How KL Divergence Reveals Bias</h4><ol><li>Assumed unbiased distribution: KL divergence approaches zero as sample size increases.</li><li>Biased data: KL divergence decreases initially but plateaus at a non-zero value.</li></ol><h4>Implementation Steps</h4><ol><li>Define an assumed unbiased distribution based on prior knowledge.</li><li>Generate samples of various sizes from your actual data.</li><li>Calculate KL divergence between each sample and the assumed distribution.</li><li>Plot KL divergence against sample size.</li></ol><h4>Interpreting Results</h4><ul><li>Decreasing KL divergence: Data likely follows the assumed distribution.</li><li>KL divergence plateauing: Potential bias present or incorrect assumption about true distribution.</li></ul><h4>Iterative Refinement</h4><p>If results suggest bias:</p><ol><li>Update your assumption about the true distribution.</li><li>Recalculate KL divergence with the new assumed distribution.</li><li>Repeat until you find a distribution that minimizes KL divergence.</li></ol><h4>Applications</h4><ol><li>Bias Detection: Identify if data collection processes are biased.</li><li>Bias Quantification: Estimate the severity of bias.</li><li>Distribution Inference: Refine understanding of the true underlying distribution.</li><li>Data Adjustment: Inform statistical corrections to data or analysis methods.</li></ol><h4>Takeaway</h4><p>KL divergence, combined with prior knowledge, provides a practical method to detect and understand data biases, even when unbiased samples are not available.</p><h4>Code</h4><p>The plot at the top of the article was created as per the below implementation.</p><pre>import numpy as np<br>from scipy import stats<br>import matplotlib.pyplot as plt<br><br>def generate_sample(size, mean=0, std=1):<br>    return np.random.normal(mean, std, size)<br><br>def weighted_sample(size, weight_func, mean=0, std=1):<br>    x = np.random.normal(mean, std, size * 2)<br>    w = weight_func(x)<br>    return x[np.random.random(size * 2) &lt; w / np.max(w)][:size]<br><br>def polynomial_weight(x, a):<br>    return 1 + a * x**2<br><br>def exponential_weight(x, a):<br>    return np.exp(a * x)<br><br>def calculate_kl_divergence(p, q):<br>    # Use kernel density estimation for smoother pdf estimates<br>    kde_p = stats.gaussian_kde(p)<br>    kde_q = stats.gaussian_kde(q)<br>    x = np.linspace(min(p.min(), q.min()), max(p.max(), q.max()), 1000)<br>    return np.sum(kde_p(x) * (np.log(kde_p(x)) - np.log(kde_q(x)))) * (x[1] - x[0])<br><br>def calculate_kld_curve(sample_sizes, reference_dist, test_dist, num_runs=10):<br>    kld_values = []<br>    for size in sample_sizes:<br>        klds = []<br>        for _ in range(num_runs):<br>            p = reference_dist(size)<br>            q = test_dist(size)<br>            klds.append(calculate_kl_divergence(p, q))<br>        kld_values.append(np.mean(klds))<br>    return kld_values<br><br>def plot_kld_curves(sample_sizes, curves, labels):<br>    plt.figure(figsize=(12, 7))<br>    for curve, label in zip(curves, labels):<br>        plt.plot(sample_sizes, curve, label=label, marker=&#39;o&#39;, markersize=4)<br>    plt.xscale(&#39;log&#39;)<br>    plt.yscale(&#39;log&#39;)<br>    plt.xlabel(&#39;Sample Size&#39;)<br>    plt.ylabel(&#39;KL Divergence&#39;)<br>    plt.legend()<br>    plt.title(&#39;KL Divergence vs Sample Size&#39;)<br>    plt.grid(True, which=&quot;both&quot;, ls=&quot;-&quot;, alpha=0.2)<br>    plt.tight_layout()<br>    plt.show()<br><br># Main execution<br>n = 10000<br>sample_sizes = np.logspace(1, 5, 30).astype(int)<br><br># Define distribution generators<br>unweighted_dist = lambda size: generate_sample(size)<br>poly_weighted_dist = lambda size: weighted_sample(size, lambda x: polynomial_weight(x, 0.5))<br>exp_weighted_dist = lambda size: weighted_sample(size, lambda x: exponential_weight(x, 0.2))<br><br># Calculate KLD curves<br>unweighted_curve = calculate_kld_curve(sample_sizes, unweighted_dist, unweighted_dist)<br>poly_weighted_curve = calculate_kld_curve(sample_sizes, unweighted_dist, poly_weighted_dist)<br>exp_weighted_curve = calculate_kld_curve(sample_sizes, unweighted_dist, exp_weighted_dist)<br><br># Plot results<br>plot_kld_curves(sample_sizes, <br>                [unweighted_curve, poly_weighted_curve, exp_weighted_curve],<br>                [&#39;Unweighted&#39;, &#39;Polynomial Weighted&#39;, &#39;Exponential Weighted&#39;])</pre><h4>Sources</h4><ol><li>De Bernardis, Francesco. “Uncovering Biases in Data with the Kullback-Leibler Divergence.” The Quantastic Journal, Medium, 2 Sept. 2024, medium.com/the-quantastic-journal/uncovering-biases-in-data-with-the-kullback-leibler-divergence-57a1ee57e2ba.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=355ebe940b56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster Normal Probabilities]]></title>
            <link>https://medium.com/@nikolas.weissmueller/a-bounded-integral-for-the-normal-distribution-50a86bc40307?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/50a86bc40307</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[normal-distribution]]></category>
            <category><![CDATA[computation]]></category>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Sun, 08 Sep 2024 05:33:13 GMT</pubDate>
            <atom:updated>2024-09-08T17:39:14.365Z</atom:updated>
            <content:encoded><![CDATA[<h4>Faster but less accurate: A Bounded Integral for the Normal Distribution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zXOhKvMzM3V6WOal.png" /><figcaption>Probability density function for the Normal Distribution. Source: Wikipedia.</figcaption></figure><p>Traditionally, we calculate the probability that a standard normal random variable<em> Z</em> exceeds some cutoff <em>z</em> according to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8b0FI27LmaJnLKysZEoTA.png" /></figure><p>This formulation suffers from computational inefficiency due to its unbounded upper limit and variable lower bound.</p><p>In 1991, J. W. Craig devised a clever alternative:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSts1u9mRcaKT3JlyQvlaw.png" /></figure><p>While appearing more complex, this formulation offers superior computational properties:</p><ol><li>It has fixed, finite integration bounds (0 to π/2).</li><li>The integrand exhibits well-behaved characteristics.</li><li>It is robust across a wide range of z values.</li></ol><p>The standard form directly represents the definition of the normal distribution probability. Craig’s formula, while mathematically equivalent, is an alternative representation that introduces trigonometric functions, which can introduce small numerical errors in computation.</p><p>Let’s compare Craig’s method to the standard formulation for three z-values:</p><pre>import numpy as np<br>from scipy import integrate<br>from scipy.stats import norm<br><br>def craigs_integral(z, n=10):<br>    q, _ = integrate.fixed_quad(<br>        lambda t: np.exp(-z**2 / (2*np.sin(t)**2)) / np.pi,<br>        0.0, np.pi/2, n=n)<br>    return q<br><br>for z in [0.5, 2, 5]:<br>    craig = craigs_integral(z)<br>    actual = norm.sf(z)<br>    print(f&quot;z = {z}: Craig&#39;s = {craig:.8f}, Actual = {actual:.8f}&quot;)</pre><p>This code returns:</p><pre>z = 0.5: Craig&#39;s = 0.30858301, Actual = 0.30853754 <br>z = 2: Craig&#39;s = 0.02274966, Actual = 0.02275013 <br>z = 5: Craig&#39;s = 0.00000029, Actual = 0.00000029</pre><p>Craig’s method yields a reasonable approximation, especially for larger z values, that closely aligns with the standard normal distribution.</p><h4>Speed vs. Accuracy: The Trade-off</h4><p>Craig’s method achieves a material computational efficiency improvement over the conventional approach:</p><pre>import numpy as np<br>from scipy import integrate<br>from scipy.stats import norm<br>import time<br><br>def traditional_integral(z):<br>    return integrate.quad(lambda t: np.exp(-t**2/2) / np.sqrt(2*np.pi), z, np.inf)[0]<br><br>def craigs_integral(z, n=10):<br>    return integrate.fixed_quad(<br>        lambda t: np.exp(-z**2 / (2*np.sin(t)**2)) / np.pi,<br>        0.0, np.pi/2, n=n)[0]<br><br># Generate 100 z values between 0 and 5<br>z_values = np.linspace(0, 5, 1000)<br><br># Time the traditional method<br>start_time = time.time()<br>traditional_results = [traditional_integral(z) for z in z_values]<br>traditional_time = time.time() - start_time<br><br># Time Craig&#39;s method<br>start_time = time.time()<br>craig_results = [craigs_integral(z) for z in z_values]<br>craig_time = time.time() - start_time<br><br>print(f&quot;Traditional method time: {traditional_time:.4f} seconds&quot;)<br>print(f&quot;Craig&#39;s method time: {craig_time:.4f} seconds&quot;)<br>print(f&quot;Speedup factor: {traditional_time / craig_time:.2f}x&quot;)<br><br># Check accuracy<br>actual_results = norm.sf(z_values)<br>traditional_error = np.mean(np.abs(traditional_results - actual_results))<br>craig_error = np.mean(np.abs(craig_results - actual_results))<br><br>print(f&quot;\nMean absolute error:&quot;)<br>print(f&quot;Traditional method: {traditional_error:.2e}&quot;)<br>print(f&quot;Craig&#39;s method: {craig_error:.2e}&quot;)</pre><p>Here is the output:</p><pre>Traditional method time: 0.1697 seconds<br>Craig&#39;s method time: 0.0225 seconds<br>Speedup factor: 7.55x<br><br>Mean absolute error:<br>Traditional method: 2.62e-13<br>Craig&#39;s method: 5.89e-05</pre><h4>Takeaway</h4><p>Craig’s method, in this simple form, calculates about seven times faster than the standard approach. While it loses a small measure of precision, it gains much in speed. The method’s efficiency, therefore, makes it suitable for applications where rapid computation outweighs the need for maximum accuracy.</p><h4>References</h4><ol><li>Craig, John W. “A new, simple and exact result for calculating the probability of error for two-dimensional signal constellations.” <em>MILCOM 91-Conference record</em>. IEEE, 1991.</li><li>Cook, John D. “A better integral for the normal distribution.” John D. Cook, 31 Aug. 2024, <a href="http://www.johndcook.com/blog/2024/08/31/craigs-formula/">www.johndcook.com/blog/2024/08/31/craigs-formula/</a>. Accessed [7 Sept. 2024].</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50a86bc40307" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevating Scoring Systems]]></title>
            <link>https://medium.com/@nikolas.weissmueller/adding-probabilities-to-scoring-systems-step-by-step-cee67537b02b?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cee67537b02b</guid>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Sun, 19 May 2024 18:41:35 GMT</pubDate>
            <atom:updated>2024-06-02T04:07:36.948Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Probabilistic Approach to Risk Assessment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*EFjFuAkO9PiZ07_P" /><figcaption>The “Probability of Success by Score” plot is obtained after Bayesian updating with data to quantify the probability of success for each score.</figcaption></figure><p>Scoring systems can help make decisions based on simplified criteria and often have 3, 5, or 7 ordinal levels. Frequently, these scores serve as proxies for the likelihood of achieving a desired outcome but are rarely evolved to quantify the probability of success associated with each score. <em>Wouldn’t it be neat if you knew how likely you are to obtain the desired outcome, given the score that you have assigned?</em> This work builds on Graeme Keith’s excellent article “Transforming Scores into Probability”, which I highly recommend you read.</p><p>To estimate probabilities, you need data. The first three steps of the process focus on collecting usable data:</p><ol><li>Define the score and its levels clearly, aiming for discrete, ordinal, mutually exclusive, and cumulatively exhaustive (MECE) definitions.</li><li>Ensure intra and inter-operator consistency by training people to apply the heuristic effectively and assign scores correctly.</li><li>Record the score when it is assessed and the outcome when it becomes known.</li></ol><p>First, we quantify the data as evidence, then we convert the evidence back to probabilities. The relationship between evidence and probability can be expressed as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jn3wxKoWqS1nJD6VUKiwhQ.jpeg" /></figure><p>We will be using the evidence version of Bayes’ Rule, which simplifies updating probabilities to addition. When we look at Bayes’ Rule, from the data collected, we have all the information we need to find the posterior probability. These distributions help determine how informative each score is in predicting success or failure. By comparing the marginal distributions, we can identify scores that are more strongly associated with success or failure, which is crucial in calculating the final probabilities.</p><p>The change in evidence for each score is calculated using the marginal distributions, quantifying how much each score shifts the belief in the likelihood of success.</p><p>The baseline evidence, derived from the overall success rate, is then updated with the change in evidence for each score to obtain the updated evidence. Bayes’ rule in terms of evidence is used for this update:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5gyflGT8e_wiyOCviOn5A.png" /></figure><p>Finally, the updated evidence for each score is converted back to probabilities, providing a clear measure of the likelihood of success given a particular score.</p><p><strong>Here’s a high-level example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*e9jCDbcdGPalwtzjLPtksg.png" /></figure><h4>Building on Graeme’s Example</h4><p>Let’s now reproduce Graeme’s example and build on it. Compared to our simple example earlier, we now have five scores instead of one. Initially, all scores are assumed to have no predictive power, and the outcome is equally likely to occur irrespective of the score. We encode this using a Dirichlet prior. The Dirichlet distribution is a multivariate generalization of the Beta distribution, allowing us to model the probabilities of multiple mutually exclusive events simultaneously. In our context, it enables us to account for all scores and their corresponding success and failure counts. The process then uses Monte Carlo simulations to update the posterior distributions based on the observed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*5tgtOJrxV7DS0MfBDyvBdQ.png" /><figcaption>The heuristic has five levels, and for each score, we have the counts of successes and failures.</figcaption></figure><h4><strong>1. Change in Evidence by Score:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*OuwYFCzML5nJuWoa" /></figure><p>The “Change in Evidence by Score” plot shows how much each score shifts the belief in the likelihood of success. Positive values indicate that the score increases the belief in success, while negative values indicate that the score decreases the belief in success.</p><h4><strong>2. Marginal Distributions for Success and Failure Cases:</strong></h4><p>The “Marginal Distributions for Success and Failure Cases” plot shows the probability of observing each score given success <em>P(score|S) </em>and failure <em>P(score|F)</em>. By comparing these distributions, we can identify scores that are more strongly associated with success or failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*rZsDd1HZgnkyZddG" /></figure><h4>3. Probability of Success by Score:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*EFjFuAkO9PiZ07_P" /></figure><p>The “Probability of Success by Score” plot shows the updated probability of success for each score, taking into account the baseline probability and the change in evidence provided by each score. This simple 5-level heuristic now quantifies the likelihood of an outcome, which can be used for risk-adjustment.</p><p>Converting ordinal scores into probabilities of success using Bayesian updating is a powerful technique that can greatly enhance decision-making processes. By following the steps outlined in this article and leveraging the evidence version of Bayes’ Rule, you can transform your scoring system into a predictive tool that quantifies the likelihood of achieving desired outcomes. This approach not only improves the interpretability of scores but also enables more informed and data-driven decisions.</p><h3>Sources:</h3><p>Keith, Graeme. “Transforming Scores into Probability.” <em>Cantor’s Paradise</em>, 1 Feb. 2021, <a href="http://www.cantorsparadise.com/transforming-scores-into-probability-fb4d4be7deab">www.cantorsparadise.com/transforming-scores-into-probability-fb4d4be7deab</a>. Accessed 18 May 2024.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cee67537b02b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simplicity from complexity]]></title>
            <link>https://medium.com/@nikolas.weissmueller/simplify-and-accelerate-beware-simplistic-solutions-e1afb982973b?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1afb982973b</guid>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Tue, 16 Apr 2024 03:53:41 GMT</pubDate>
            <atom:updated>2024-04-17T06:39:04.922Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Non-Proportional Hazards Survival Analysis Example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zj-liQW1KzfXT6rfgp9MTA.png" /><figcaption>Adapted from: <a href="https://andreakihlstedt.com/pushing-past-complexity-to-discover-elegant-simplicity/">https://andreakihlstedt.com/pushing-past-complexity-to-discover-elegant-simplicity/</a></figcaption></figure><p>In today’s fast-paced world, the mantra “simplify and accelerate” is ubiquitous. The goal is clear: achieve more with less, and do it faster. As someone who enjoys optimizing and simplifying methods and processes, I find this challenge both exciting and daunting. Creating simple yet effective solutions is no easy feat, especially when dealing with complex, unsolved problems.</p><p>As Mark Twain famously said, “I didn’t have time to write a short letter, so I wrote a long one instead.” This insightful perspective on writing also applies to analytic methods. Developing a solution that is both simple and useful is hard work. In writing, we often fill pages, only to revise, cut, edit, and word-smith until our ideas are distilled and crystallized for effective communication. The methods creation process is not too dissimilar. The key difference is that if a robust solution can be found, only for some cases, it might also be simplified for broader fit-for-purpose use. Simplistic solutions are easy to find and fast to create, but lack substance and value. Instead, we strive for solutions that stand the test of time and create long-term value. As Einstein and Occam put it, good solutions are “as simple as possible, but as complex as necessary”.</p><p>To demonstrate the effort required to define a simple solution, let’s consider a difficult problem in survival analysis: when faced with non-proportional hazards, such as delayed effect, long-term plateaus, crossing curves, converging, or diverging hazards, how can we determine the probability of correctly identifying an effect if it exists? Generally, the gold standard is involved simulations. However, with some elbow grease, one can derive this succinct formula, an approximation that might be fit-for-purpose in many cases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPH9pmI9NQYWGtRsh7ZY-w.png" /></figure><blockquote>Paradoxically, simplifying is hard work, and to accelerate, you need to invest time first.</blockquote><p>While the aforementioned formula is short, it is the result of several pages of work, the application of first principles, and a few simplifying assumptions along the way. You can stop reading here and pass over the crux of this article in respectful silence, or you might read on and experience the toil firsthand by following the thought process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d6ebUsRfBCd3UPNWIzt_hw.png" /><figcaption>Survival curves for Proportional (left) and Non-Proportional (right) Hazards. NPH often arises in oncology and can be challenging to handle well.</figcaption></figure><h4>Example</h4><p>Let’s consider the problem: when hazard ratios change over time, “standard” methods don’t apply. A straightforward approach is to create a representative average and describe it with a single formula. By working through the derivation step-by-step, we can see how a simple formula emerges from the complex problem. The process involves:</p><ul><li>Breaking down the problem into manageable pieces</li><li>Making reasonable assumptions</li><li>Applying mathematical principles to arrive at the solution</li></ul><p>By following these steps, we can navigate through the complexity and reach a simple solution on the other side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBDE8-s9hW6IFtlK03odQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/0*Y6U1yY2u6sygt5D8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*PrrlX9YFtsx2p0oJ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*b8sanXnlHphAUXVC" /></figure><h4>Python implementation and example:</h4><pre>import numpy as np<br>from scipy.stats import norm<br><br>def approx_power_nph(lambdas_0, lambdas_1, durations, n_0, n_1, alpha=0.05):<br>    &quot;&quot;&quot;<br>    Approximate power calculation for non-proportional hazards <br>    using the derived weighted log-rank test.<br><br>    Parameters:<br>    -----------<br>    lambdas_0 : Hazard rates for the control group in each interval.<br>    lambdas_1 : Hazard rates for the treatment group in each interval.<br>    durations : Duration of each interval in months.<br>    n_0 :       Number of subjects in the control group.<br>    n_1 :       Number of subjects in the treatment group.<br>    alpha :     Significance level. (default=0.05)<br>        <br>    Returns:<br>    --------<br>    power : Approximate power of the weighted log-rank test.<br>    beta :  Weighted average of log hazard ratios.<br>    I :     Information.<br>    &quot;&quot;&quot;<br><br>    M = len(lambdas_0)<br>    E_d_0 = [n_0 * (1 - np.exp(-lambdas_0[m] * durations[m])) for m in range(M)]<br>    E_d_1 = [n_1 * (1 - np.exp(-lambdas_1[m] * durations[m])) for m in range(M)]<br>    weights = [(E_d_0[m] * E_d_1[m]) / (E_d_0[m] + E_d_1[m]) for m in range(M)]<br>    weights = [w / sum(weights) for w in weights]<br>    beta_m = [np.log(lambdas_1[m] / lambdas_0[m]) for m in range(M)]<br>    beta = sum(weights[m] * beta_m[m] for m in range(M))<br>    I = 1 / sum(1 / (E_d_0[m] + E_d_1[m]) for m in range(M))<br>    z_alpha_2 = norm.ppf(1 - alpha / 2)<br>    power = norm.cdf(np.sqrt(I) * np.abs(beta) - z_alpha_2)<br>    return power, beta, I<br><br># Example usage<br>l0 = np.log(2) / 12   # median of 12 months<br>l1 = 0.7 * l0         # hazard ratio of 0.7<br>lambdas_0 = [l0, l0]  # Hazard rates for the control group in each interval<br>lambdas_1 = [l0, l1]  # Hazard rates for the treatment group in each interval<br>durations = [10, 50]  # Duration of each interval in months<br>n_0 = 50  # Number of subjects in control group<br>n_1 = 50  # Number of subjects in treatment group<br><br>power, beta, I = approx_power_nph(lambdas_0, lambdas_1, durations, n_0, n_1)<br>print(f&#39;beta: {beta:.4f}&#39;)<br>print(f&#39;I: {I:.4f}&#39;)<br>print(f&#39;Average HR: {np.exp(beta):.4f}&#39;)<br>print(f&#39;Power: {power:.4f}&#39;)<br><br># Output:<br># beta: -0.2402<br># I: 29.5602<br># Average HR: 0.7865<br># Power: 0.2565</pre><h4>Side calculations for step 9:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*8v20jqsnXKHJQVHU" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*fvLMrIi1UGaLFC6k" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*dW7XQDpN4igmrofZ" /></figure><h4><strong>Sources:</strong></h4><ol><li>Schoenfeld, David. “The asymptotic properties of nonparametric tests for comparing survival distributions.” <em>Biometrika</em> 68.1 (1981): 316–319.</li><li>Anderson, Keaven. “Group Sequential Design Under Non-Proportional Hazards”. <a href="https://keaven.github.io/gsd-deming/">https://keaven.github.io/gsd-deming</a>. Accessed 15 April 2024.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1afb982973b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You need a Quant Team]]></title>
            <link>https://medium.com/@nikolas.weissmueller/if-you-invest-in-data-you-should-also-invest-in-methods-116245cfee86?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/116245cfee86</guid>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Sun, 31 Mar 2024 22:35:58 GMT</pubDate>
            <atom:updated>2024-04-02T04:56:36.005Z</atom:updated>
            <content:encoded><![CDATA[<h4>As a data-driven organization, if you invest in data, you should also invest in methods and suitable governance for both.</h4><p>Companies are increasingly turning to analytical methods as the backbone of their decision-making processes, effectively elevating these methods to the role of decision-makers. This should motivate careful method design, validation, and governance.</p><h4>The Importance of Method Quality in Data-Driven Decision-Making</h4><p>Why should we care about method quality when “all models are wrong” anyway? George Box’s famous quote, “all models are wrong, but some are useful,” is often taken out of context. (Box, 1979) The more significant message from Box’s paper is that “since all models are wrong, the scientist must be alert to what is importantly wrong.” (Box, 1976) In other words, while no model accurately captures the complexities of reality, it is crucial to ensure that models are not fundamentally flawed.</p><p>Knowing a “fundamentally flawed” from a “wrong but useful” model can be challenging, and often requires a deep theoretic and subject-specific understanding. The finance industry has long recognized the value of “Quants”, skilled mathematicians or statisticians, who custom-design algorithms and rigorous methods that are tailored to a specific business need. Other industries have followed suit, creating new departments. Netflix, for instance, has dedicated Algorithms R&amp;D and Metrics Development teams, among others (see below). Netflix attributes much of its success to its focus on algorithm quality.</p><p>Here is how Elizabeth Stone (Netflix CTO) has structured her organization — a great example of what it takes, and what can be done:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9l-Byqfi5bIGgiraf3iIxQ.jpeg" /><figcaption>Netflix Chief Technology Officer. Accessed April 1, 2024.</figcaption></figure><blockquote>“We are a centralized and globally distributed team that includes:</blockquote><blockquote>- Consumer research <br>- Experimentation and causal inference<br>- Metrics development, analytics insights, and tooling<br>- Network engineering &amp; partnerships and content delivery<br>- Optimization models and automation<br>- Research &amp; development of machine learning algorithms<br>- Software, data, security, and distributed systems engineering</blockquote><blockquote>We apply these areas of functional expertise to all areas of Netflix’s business, including the member-facing product, content &amp; studio production, games, advertising, business operations, finance, and security.” <em>— Elizabeth Stone</em></blockquote><h4>The Consequences of Using Inappropriate Methods By Example</h4><p>Using mismatched methods is self-evidently a bad idea. It is less clear, however, just how subtle the distinction between a matched and mismatched method can be, and how decision-altering its impact. Let’s work a hypothetical example to demonstrate the impact of a matched vs. mismatched method, and show how the application of fundamental theoretical principles can produce a bespoke solution rapidly that is not “importantly wrong”. The example also shows that applying a ‘stock’ method can lead us to an incorrect data-driven decision.</p><h4>Example:</h4><p>Suppose we obtained some data and would like to do the following:</p><ol><li><strong>Find the best test to examine our hypothesis, and</strong></li><li><strong>Compute the probability of detecting an effect if one exists.</strong></li></ol><p>Based on the data generation process, we know that the observations (X1, . . . , Xn) are independent and identically distributed (iid) according to a <strong>Poisson distribution</strong> with mean λ. We consider the null hypothesis H0 : λ = 1 against the alternative hypothesis H1 : λ &gt; 1.</p><p>However, we already have a formula and code that is specific for a <strong>Normal distribution</strong> with mean 0 and variance σ². When we fit the Normal distribution to the data, it’s not a terrible fit. So, we consider the null hypothesis H0 : σ = 1 against the alternative hypothesis H1 : σ &gt; 1, and move on.</p><p>Surely, using the Gaussian instead of the Poisson will be fine, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*37Qds8hfb5Mnk2KHJK3TjQ.jpeg" /><figcaption>Created at <a href="https://imgflip.com/memegenerator/166969924/Flex-Tape">https://imgflip.com/memegenerator/166969924/Flex-Tape</a></figcaption></figure><p>Let’s do the work and check. The complete solution is appended below in handwritten form. I am skipping to the solution graph here, but the interested reader is encouraged to review the derivations.</p><p>When we plot the performance of the derived best tests for the Poisson and Normal against variance, we obtain an interesting graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*kwgjfDo6Y_YCsGfK" /><figcaption>Comparison of the power curves for hypothesis tests based on the Poisson and Normal distributions (n=30, α=0.05). The yellow marks the difference in power for a given variance between using a Poisson test and a Normal test.</figcaption></figure><p>From the example, we observe that when the true underlying distribution of the data is Poisson, but a test designed for Normal data is used, several issues can arise:</p><ul><li><strong>Reduced power</strong>: The Normal test may have lower power compared to a Poisson-specific test, potentially leading to a higher risk of Type II errors. Using the appropriate test can improve power.</li><li><strong>Incorrect sizing</strong>: The Normal test may suggest a larger sample size is needed, potentially leading to oversampling. The appropriate test can optimize sample sizes for efficient decision-making.</li><li><strong>Suboptimal decisions</strong>: The Normal test may influence decision thresholds, potentially leading to suboptimal decision-making. The correct test can help set optimal decision thresholds based on the true power of the test.</li></ul><h4>Practical implications for organizations</h4><p>Most off-the-shelf methods need careful customization, or risk being “importantly wrong”. Stretching methods beyond their validity limits, or using them out of context, is a common reason why some models are “importantly wrong”. However, customization requires specialized skills and dedicated resources. Organizations that give equal priority to algorithm and data quality are well-equipped to succeed.</p><blockquote>Just as calculating the area of a rectangle depends on the lengths of sides A and B, so too does the success of models hinge on the quality of both the data and the methods.</blockquote><p>Therefore, if you invest in data, you should also invest in methods and suitable governance for both. If you use models to inform decisions, you might benefit from a dedicated Methods/Quants/Algorithms team.</p><p><strong>Code for the figure:</strong></p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy.stats import norm, chi2, ncx2<br><br>def poisson_power(n, lambda_, alpha):<br>    z_alpha = norm.ppf(1 - alpha)<br>    power = 1 - norm.cdf(np.sqrt(n / lambda_) * (1 - lambda_) + z_alpha / np.sqrt(lambda_))<br>    return power<br><br>def normal_power(n, sigma_squared, alpha):<br>    z_alpha = norm.ppf(1 - alpha)<br>    power = 1 - norm.cdf(np.sqrt(n / 2) * ((1 - sigma_squared) / np.sqrt(sigma_squared)) + z_alpha / np.sqrt(sigma_squared))<br>    return power<br><br># Set parameters<br>n = 30<br>alpha = 0.05<br><br># Generate lambda and sigma^2 values for plotting<br>variance_values = np.linspace(1, 2.5, 100)<br><br># Compute power for each lambda and sigma^2 value<br>poisson_powers = [poisson_power(n, lambda_, alpha) for lambda_ in variance_values]<br>normal_powers = [normal_power(n, sigma_squared, alpha) for sigma_squared in variance_values]<br><br># Create the plot<br>plt.figure(figsize=(8, 6))<br>plt.plot(variance_values, poisson_powers, label=&#39;Poisson Power&#39;)<br>plt.plot(variance_values, normal_powers, label=&#39;Normal Power&#39;)<br><br># Fill the area between the curves<br>plt.fill_between(variance_values, poisson_powers, normal_powers, color=&#39;yellow&#39;, alpha=0.2)<br><br>plt.xlabel(&#39;Variance (λ for Poisson, $σ^2$ for Normal)&#39;)<br>plt.ylabel(&#39;Power&#39;)<br>plt.yticks(np.arange(0, 1.05, 0.05))<br>plt.title(&#39;Power Function Curves&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><p><strong>Worked example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rs_L6UlKmYlcEzpno6ZNig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gWCKh2He6t1hJW350pFFSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uRXSgDvocR5vnp2bsGAs3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XWkUIcoA5t8BCJicSNCgoQ.png" /></figure><p><strong>Sources:</strong></p><p>Box, George EP. “Science and statistics.” <em>Journal of the American Statistical Association</em> 71.356 (1976): 791–799.</p><p>Box, George EP. “Robustness in the strategy of scientific model building.” <em>Robustness in statistics</em>. Academic Press, (1979): 201–236.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=116245cfee86" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Schoenfeld Shortcut]]></title>
            <link>https://medium.com/@nikolas.weissmueller/the-schoenfeld-shortcut-rough-and-ready-survival-analysis-study-sizing-692769c69156?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/692769c69156</guid>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Thu, 28 Mar 2024 05:14:08 GMT</pubDate>
            <atom:updated>2024-03-29T01:14:02.676Z</atom:updated>
            <content:encoded><![CDATA[<h4>Rough and Ready Survival Analysis Study Sizing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*7Ed--2p_u3j_YuIXfEATFA.png" /><figcaption>Schoenfeld Power for a range of potential Hazard Ratios and four different total number of events cases.</figcaption></figure><p>Imagine you are a clinical trial physician, and management wants to explore different design options for your upcoming survival study. They need to know how many patients you’ll require for each scenario, and they want an answer ASAP. It’s 10 PM, and the statistician (un)surprisingly is not responding to your calls or messages.</p><p>Determining the appropriate sample size for a survival analysis study can be challenging, especially when you need an estimate quickly. While you should always consult a statistician for a more accurate calculation, the Schoenfeld power approximation can provide a helpful ballpark estimate in the meantime. In the morning, of course, you follow up with your statistician to get the official numbers — and perhaps, you even earn some kudos for producing a sensible estimate under pressure.</p><h4><strong>Why care about study size?</strong></h4><p>We care about study size because we want the study to produce reliable results, while simultaneously not overextending on resources. When the study size is fixed, we might want to know the probability of detecting a real effect when it exists. The Schoenfeld power approximation can provide an estimate quickly. When a few assumptions hold, this estimate is a good approximation of the study power.</p><h4><strong>What is the Schoenfeld power approximation?</strong></h4><p>The Schoenfeld power approximation is a method for estimating the power of a survival analysis study that compares the time-to-event outcomes between an experimental treatment group and a control group. This approximation relies on the proportional hazards assumption, which states that the ratio of the hazard rates (the instantaneous risk of an event occurring) between the two groups remains constant over time. In other words, the observed treatment benefit, represented by the hazard ratio, does not change throughout the study duration.</p><p>Benefits of the Schoenfeld approximation include speed, simplicity, sample size determination for a desired power level, and rapid study design exploration. The latter pertains to exploring various design options, such as different hazard ratios, sample sizes, and randomization ratios.</p><h4><strong>How does it work?</strong></h4><p>I include the derivation of the Schoenfeld approximation to provide the reader with a deeper understanding of the method and to showcase the thought process to arrive at this useful approximation:</p><p>Suppose that a hazard ratio <em>ν </em>&lt; 1 represents a benefit of the experimental treatment over the control, and let <em>σ </em>= log(<em>ν</em>). Under the proportional hazards assumption, the distribution of the Cox model estimate is approximately normally distributed with mean and standard deviation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*wpqE6AFTwL0GOUxQ" /></figure><p>Where <em>r</em> is the randomization ratio (treatment:control) and <em>n</em> is the total number of events. Using the Cox model to estimate <em>σ</em>, the Wald test for <em>H</em>0: <em>σ </em>= 0 can be approximated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*ocn9a8Ho9NI0wkcf" /></figure><p>The Wald test<em> Zw </em>and the logrank <em>Z</em> are asymptotically efficient and equivalent. So, we can define the standardized effect size as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*Q62d09MoPY0WaSdw" /></figure><p>​Letting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/0*KOINX5zPZRnrZOeO" /></figure><p>we have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/0*RRAzZRbTUfHajkg_" /></figure><p>Thus, the standardized <em>Z</em> version of the logrank is approximately distributed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*zTfw2skl6Hiemy8q" /></figure><p>In this notation, a hazard ratio <em>ν </em>&lt; 1, negative values of <em>σ</em>, and the standardized Z-test all favor the experimental treatment over the control. Finally, the power for the logrank test is approximated by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kOHxFz-dS_gss_k0" /></figure><p>We now solve for the number of events <em>n</em> to see how many events are required to obtain a desired power:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*tqe0Cw4tD2aQ9TZ8" /></figure><p>and so (finally):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gxgDRR91Id9W_P1f" /></figure><p>where:</p><ul><li><em>β</em>: Type II error rate (1− power)</li><li><em>α</em>: Significance level (Type I error rate)</li><li><em>Z</em>: Standard normal random variable</li><li>Φ−1: Inverse of the standard normal cumulative distribution function</li><li><em>r</em>: Randomization ratio (treatment/control)</li><li><em>ν</em>: Hazard ratio</li><li><em>θ</em>: Standardized effect size (defined above)</li></ul><p>This approach allows us to approximate the number of events required to attain a predetermined level of power. To give the clinician in our story a break at 10PM, we conclude this article with two working examples.</p><h4>Example 1: Power for a fixed number of events</h4><p>Consider a study with 100 events, a hazard ratio of 0.70 (experimental treatment is better), equal randomization (r=1), and a significance level of 0.025. Using the Schoenfeld approximation, we get P[Z ≤ z] ≈ 0.43.</p><pre>from scipy.stats import norm<br><br>def schoenfeld_power(aHR, n, r, alpha):<br>    &quot;&quot;&quot;<br>    Calculates the power using Schoenfeld&#39;s formula.<br>    Parameters:<br>    - aHR (float): Anticipated hazard ratio.<br>    - n (int): Total number of events.<br>    - r (float): Randomization ratio (treatment/control).<br>    - alpha (float): Significance level.<br>    Returns:<br>    - float: Power calculated using Schoenfeld&#39;s formula.<br>    &quot;&quot;&quot;<br>    return norm.cdf(norm.ppf(alpha) - np.sqrt(n * r) * np.log(aHR) / (1 + r), loc=0, scale=1)<br><br># Example:<br>schoenfeld_power(aHR = 0.7, n=100, r=1, alpha=0.025)<br># = 0.42991551348734924</pre><h4><strong>Example 2: Size for a desired study power</strong></h4><p>Suppose we want to estimate the required number of events for a study with a hazard ratio of 0.70, a one-sided significance level (<em>α</em>) of 0.025, and a desired power (1−<em>β</em>) of 0.9:</p><pre>import numpy as np<br>from scipy.stats import norm<br><br>def schoenfeld_events(alpha, beta, r, hr):<br>    &quot;&quot;&quot;<br>    Calculates the number of events needed for Schoenfeld&#39;s approximation.<br>    <br>    Parameters:<br>    - alpha (float): Significance level (Type I error).<br>    - beta (float): Type II error (1 - power).<br>    - r (float): Randomization ratio (treatment/control).<br>    - hr (float): Hazard ratio.<br>    <br>    Returns:<br>    - float: Number of events needed for Schoenfeld&#39;s approximation.<br>    &quot;&quot;&quot;<br>    # Calculate the inverse standard normal CDF for alpha and beta<br>    qnorm_alpha = norm.ppf(1 - alpha)<br>    qnorm_beta = norm.ppf(1 - beta)<br>    <br>    # Calculate the number of events<br>    n_events = ((1 + r)**2 / (r * (np.log(hr)**2))) * ((qnorm_alpha + qnorm_beta)**2)<br>    <br>    return n_events<br><br>schoenfeld_events(alpha=0.025, beta=0.1, r=1, hr=0.70)<br># = 330.37791396380953<br><br># A: You would need approximately 331 events to achieve 90% power.</pre><p>To get a study size (number of subjects), you will need to divide the number of events by the fraction of subjects in the study who will experience an event.</p><p>Code for the cover image:</p><pre># Code for the header image<br>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br><br># Create the sample sizes DataFrame<br>tbl_samplesizes = pd.DataFrame({<br>    &#39;type&#39;: [&#39;N=50&#39;, &#39;N=100&#39;, &#39;N=200&#39;, &#39;N=400&#39;],<br>    &#39;n&#39;: [50, 100, 200, 400]<br>})<br><br># Create the hazard ratios and sample sizes combinations<br>hazard_ratios = np.arange(0, 1.6, 0.001)<br>n_values = tbl_samplesizes[&#39;n&#39;].values<br>combinations = pd.DataFrame(np.array(np.meshgrid(hazard_ratios, n_values)).T.reshape(-1, 2), columns=[&#39;hazard ratio&#39;, &#39;n&#39;])<br><br># Merge the sample sizes and combinations DataFrames<br>plt_powr_curves = pd.merge(tbl_samplesizes, combinations, on=&#39;n&#39;)<br><br># Calculate power and create the name column<br>alpha = 0.025 # Two-tailed test<br>r = 1  # Randomization ratio (treatment/control)<br>plt_powr_curves[&#39;power&#39;] = plt_powr_curves.apply(lambda row: schoenfeld_power(row[&#39;hazard ratio&#39;], row[&#39;n&#39;], r, alpha), axis=1)<br>plt_powr_curves[&#39;name&#39;] = plt_powr_curves.apply(lambda row: f&quot;{row[&#39;type&#39;]} (n = {int(row[&#39;n&#39;])})&quot;, axis=1)<br><br># Create the plot<br>plt.figure(figsize=(10, 6))<br>for name, group in plt_powr_curves.groupby(&#39;n&#39;):<br>    plt.plot(group[&#39;hazard ratio&#39;], group[&#39;power&#39;], label=name)<br>plt.xlabel(&#39;Hazard Ratio&#39;)<br>plt.ylabel(&#39;Power - Schoenfeld Approximation&#39;)<br>plt.xlim(0, 1.6)<br>plt.ylim(0, 1)<br>plt.xticks(np.arange(0, 1.7, 0.2))<br>plt.yticks(np.arange(0, 1.1, 0.1))<br>plt.legend(tbl_samplesizes[&#39;type&#39;].tolist(), title=&#39;&#39;, loc=&#39;upper right&#39;)<br>plt.grid(True, which=&#39;major&#39;, linestyle=&#39;-&#39;)<br>plt.grid(False, which=&#39;minor&#39;)<br>plt.tight_layout()<br>plt.show()</pre><h4>Sources:</h4><ol><li>Schoenfeld, David. “The asymptotic properties of nonparametric tests for comparing survival distributions.” <em>Biometrika</em> 68.1 (1981): 316–319.</li><li>Anderson, Keaven. “Survival Analysis Overview.” gsDesign Package Vignette, CRAN (Comprehensive R Archive Network), <a href="https://cloud.r-project.org/web/packages/gsDesign/vignettes/SurvivalOverview.html">https://cloud.r-project.org/web/packages/gsDesign/vignettes/SurvivalOverview.html</a>. Accessed 27 March 2024.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=692769c69156" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hard Truths About P-Values]]></title>
            <link>https://medium.com/@nikolas.weissmueller/hard-truths-about-p-values-that-you-should-know-c62d2c566ddd?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c62d2c566ddd</guid>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Wed, 20 Mar 2024 05:59:32 GMT</pubDate>
            <atom:updated>2024-03-20T16:34:19.067Z</atom:updated>
            <content:encoded><![CDATA[<h4>Significance is not importance. Insignificance is not unimportance.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*aA4u4-HSPdE9IR7DavCb_w.jpeg" /><figcaption>The image depicts a right-tailed p-value (H0: p1-p2 &gt; 0). A two-tailed test would have green shading in both tails of the distribution (H0: p1-p2 ≠ 0). Adapted from Mcleod, Saul. “P-Value and Statistical Significance: What It Is &amp; Why It Matters.” Simply Psychology, 13 Oct. 2023, <a href="http://www.simplypsychology.org/p-value.html">www.simplypsychology.org/p-value.html</a>.</figcaption></figure><p>In the world of pharmaceuticals, we start from the assumption that a new drug does not work until it is proven that it does. To go from A to B, one collects data and tests if the observed outcomes would be so surprising and unlikely under the assumption that the drug does not work, that we cannot sensibly maintain that initial assumption. So, we reject our initial assumption and conclude that the drug works. P-values help quantify the magnitude of surprise.</p><p>This concept extends to scientific research in general, where p-values are often seen as the gold standard for determining the significance of a study’s findings. Some readers might remember a p-value definition as:</p><blockquote>The probability of observing a result as extreme (or more extreme) as the one observed in a study, assuming that the null hypothesis is true.</blockquote><p>Concretely, the data we collect provide an estimate of the true effect. We use the data to compute a p-value, which helps us assess the compatibility of our data with the null hypothesis. For example, if we’re comparing a new drug to a placebo and obtain a p-value of 0.02, it means that if the drug has no effect (i.e., the null hypothesis is true), we would expect to see a difference between the groups as large or larger than what we observed in only 2% of studies. This would be a surprising outcome. Therefore, this suggests that our data are more consistent with the drug having an effect than with it having no effect, but it doesn’t definitively prove the drug works.</p><p>Neither the definition nor our simple example provides a deep appreciation of what a p-value is and is not. Below are some insights that further study in statistics will provide, summarized from Prof. Stark’s paper on p-values.</p><h3>Hard Truths About P-Values</h3><ul><li>Significance is not importance. Insignificance is not unimportance.</li><li>P-values are a tool to advance science by helping to rule out wrong assumptions (null hypotheses). This is a bit backwards, as we’re focusing on excluding wrong hypotheses rather than confirming correct hypotheses.</li><li>P-values do not measure the size or practical importance of an effect. They only help assess whether an apparent effect is real or an artifact.</li><li>A larger p-value does not mean that the null hypothesis is true. Lack of evidence against an explanation is not evidence for that explanation.</li><li>P-values are often misinterpreted as the probability that the null hypothesis is true. This is incorrect. P-values are calculated assuming the null hypothesis is true.</li><li>There is no universal threshold for what constitutes a “significant” p-value. The scientific context matters.</li><li>P-hacking, significance hunting, selective reporting, and failing to account for multiple comparisons can make p-values misleading.</li><li>Using “straw man” null hypotheses that have little to do with how the data were collected or generated can lead to unsurprisingly small p-values.</li><li>Comparing p-values to a fixed threshold (e.g., 0.05) and considering a result scientifically established if and only if the p-value is below that threshold is poor science and statistics.</li><li>P-values, confidence intervals, and other related concepts only apply when the data have a random component, such as being a random sample, resulting from random assignment of subjects to different treatment conditions, or having random measurement error. They do not apply to samples of convenience, haphazard samples, or populations.</li></ul><p>While p-values can be a useful tool for ascertaining if findings are real or merely artifacts, they should be consumed with open eyes and a healthy dose of skepticism. Understanding the limitations of p-values is crucial for making informed decisions about the validity of scientific claims and avoiding misinterpretation of research findings.</p><h4><strong>Sources</strong></h4><ol><li>Mcleod, Saul. “P-Value and Statistical Significance: What It Is &amp; Why It Matters.” Simply Psychology, 13 Oct. 2023, <a href="http://www.simplypsychology.org/p-value.html">www.simplypsychology.org/p-value.html</a>.</li><li>Stark, P.B. “Making Sense of P-Values.” Draft, 24 Nov. 2015, <a href="http://www.stat.berkeley.edu/~stark/Preprints/pValues.pdf">www.stat.berkeley.edu/~stark/Preprints/pValues.pdf</a>. Accessed 19 Mar. 2024.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c62d2c566ddd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating Uncertainty: Market Planning with Sequential Decision Analysis]]></title>
            <link>https://medium.com/@nikolas.weissmueller/navigating-uncertainty-mastering-market-planning-with-sequential-decision-analysis-c15e92e69302?source=rss-3e54d31a607d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c15e92e69302</guid>
            <dc:creator><![CDATA[Nikolas Weissmueller]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 04:51:25 GMT</pubDate>
            <atom:updated>2024-03-19T03:01:21.761Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HGsD0OMz1rf33iyIk4mQMQ.png" /><figcaption>Left: Cumulative average reward trend over 100 iterations. Right: Fluctuations in reward per iteration, indicating learning and adaptation in the model.</figcaption></figure><p>Imagine you’re the captain of a ship, navigating through treacherous waters filled with hidden reefs and unpredictable storms. Every decision you make could lead to mission success or failure. This is not unlike the challenges faced by business leaders in today’s ever-changing market landscape. How do you chart a course toward sustainable profitability when the future is uncertain? Sequential Decision Analysis (SDA) can help.</p><blockquote>Sequential Decision Analysis is an effective approach for managing a sequence of interconnected decisions over time, guiding you towards the best choices under uncertain conditions.</blockquote><h4>What is Sequential Decision Analysis?</h4><p>Sequential Decision Analysis is a systematic approach to decision-making that takes into account the interconnected nature of choices over time. It’s like playing a game of chess, where each move you make has consequences that ripple into the future. The goal of SDA is to maximize a specific objective, such as profit or customer satisfaction, over a given time period.</p><h4>The Six Components of Sequential Decision Analysis</h4><ol><li><strong>State variables</strong>: These are the indicators that define the current state of the system at each decision point.</li><li><strong>Decision variables</strong>: These are the factors that you can control and adjust to influence the outcome.</li><li><strong>Exogenous information</strong>: These are the external factors that impact the system. They are often uncertain and outside your control, such as market trends or competitor actions.</li><li><strong>Transition function</strong>: This is the set of rules that govern how the state variables change over time based on your decisions and the exogenous information.</li><li><strong>Objective function</strong>: This is the performance metric you want to optimize, such as maximizing profit or minimizing costs.</li><li><strong>Policy</strong>: In the context of Sequential Decision Analysis, a policy refers to the decision-making strategy or set of rules that determine how decisions are made at each step, based on the current state of the system and the available information. The policy aims to optimize the objective function over the given time horizon.</li></ol><h4>Real-World Application: Adaptive Market Planning</h4><p>To illustrate the power of Sequential Decision Analysis, let’s dive into a real-world example: a version of the Adaptive Market Planning model, developed by Prof. Warren Powell and his student Donghun Lee. This model is designed to optimize market planning decisions, such as determining order quantities for products or services, in the face of uncertain demand.</p><blockquote>The model is a variant of the classic news vendor problem, where a vendor must decide how many newspapers to order each day to maximize profit, given uncertain demand. If they order too few, they miss out on potential sales; if they order too many, they’re left with unsold inventory.</blockquote><p>The model takes this concept and applies it to a broader range of market planning decisions. The model that solves the news vendor problem also applies to a wide range of situations where you need to make decisions under uncertainty. Here are a few examples:</p><ul><li><strong>Inventory management</strong>: Retailers, manufacturers, and distributors can use similar models to optimize their inventory levels, balancing the costs of overstocking and understocking.</li><li><strong>Capacity planning</strong>: Service providers, such as hotels or airlines, can use these models to determine the optimal capacity to meet uncertain demand while maximizing revenue.</li><li><strong>Supply chain management</strong>: Companies can use similar models to make decisions about order quantities, stock levels, and production capacity in their supply chains.</li><li><strong>Resource allocation</strong>: Organizations can use these models to allocate limited resources, such as funds, staff, or equipment, to meet uncertain demand in various projects or departments.</li><li><strong>Pricing and revenue management</strong>: Businesses can use similar models to optimize their pricing strategies and revenue management decisions, considering factors like demand uncertainty and price sensitivity.</li></ul><h4>Under the Hood: The Mathematics of Adaptive Market Planning</h4><p>Let’s take a closer look at the mathematical underpinnings of the Adaptive Market Planning model. The model has two state variables: the order quantity (<em>q</em>), and a counter (<em>c</em>), that keeps track of how many times the derivative of the objective function changes sign. The decision variable is the step size (θ), which determines how much the order quantity is updated based on the observed outcomes. The exogenous information is the uncertain demand (D) for the product in each time period.</p><p>In our example, the transition function governs how the order quantity (a state variable) changes over time based on our decisions and the observed demand (exogenous information). The transition function updates the order quantity using the following rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1FiR0b6Kgogf4If4cY6SDg.png" /></figure><p>Here, <em>δ</em> is the derivative of the objective function with respect to the order quantity, and <em>θ</em> is the learned step size at time <em>t</em>, calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*KH3t456TFG4TCT9Y1Sd1wA.png" /></figure><p><em>θ </em>ensures that the model gets smarter as it goes, leading to better decision-making and improved performance.</p><p>The counter <em>c</em> is incremented whenever the sign of the derivative changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G7KQ1rGmC6dliyGTHoJSOA.png" /></figure><p>The objective function aims to maximize the expected profit by considering the revenue from sales and the cost of ordering and holding inventory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJv9reohkaCbpBhO8p3IMQ.png" /></figure><p>It is a balancing act, finding the sweet spot between selling enough to make money and not ordering too much that you’re stuck with excess inventory.</p><h4>Implementing the Adaptive Market Planning Model</h4><p>To help you get started with implementing the Adaptive Market Planning model, I’ve created a Jupyter notebook that walks you through the process step-by-step. The notebook, available on my GitHub repository linked below, includes the following code:</p><pre>&quot;&quot;&quot;<br>Adaptive Market Planning Model<br>by Nikolas Weissmueller (c) 2024<br><br>Adapted from Prof. Warren Powell and Donghun Lee (c) 2018<br>&quot;&quot;&quot;<br><br>import numpy as np<br>from dataclasses import dataclass<br><br>@dataclass<br>class State:<br>    order_quantity: float<br>    counter: int<br><br>@dataclass<br>class Decision:<br>    step_size: float<br><br>class AdaptiveMarketPlanningModel:<br>    &quot;&quot;&quot;Base class for the Adaptive Market Planning model.&quot;&quot;&quot;<br><br>    def __init__(self, s_0, T, reward_type, price=1.0, cost=1.0, seed=20180613):<br>        &quot;&quot;&quot;<br>        Initialize the Adaptive Market Planning model.<br><br>        :param s_0: Initial state dictionary<br>        :param T: Number of time steps<br>        :param reward_type: Type of reward (&#39;Cumulative&#39; or &#39;Terminal&#39;)<br>        :param price: Price of the product (default: 1.0)<br>        :param cost: Cost of the product (default: 1.0)<br>        :param seed: Random seed for reproducibility (default: 20180613)<br>        &quot;&quot;&quot;<br>        self.prng = np.random.RandomState(seed)<br>        self.state = State(*s_0.values())<br>        self.T = T<br>        self.reward_type = reward_type<br>        self.obj = 0.0<br>        self.past_derivative = 0.0<br>        self.cost = cost<br>        self.price = price<br>        self.t = 0<br>        self.learning_list = []<br><br>    def exog_info_fn(self, decision: Decision) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Generate exogenous information (demand) based on the decision.<br><br>        :param decision: Decision object<br>        :return: Dictionary containing the demand information<br>        &quot;&quot;&quot;<br>        return {&#39;demand&#39;: self.prng.exponential(demand_scale)}<br><br>    def transition_fn(self, decision, exog_info):<br>        &quot;&quot;&quot;<br>        Update the state based on the decision and exogenous information.<br><br>        :param decision: Decision object<br>        :param exog_info: Dictionary containing the exogenous information<br>        &quot;&quot;&quot;<br>        self.learning_list.append(self.state.order_quantity)<br>        derivative = self.price - self.cost if self.state.order_quantity &lt; exog_info[&#39;demand&#39;] else -self.cost<br>        self.state.order_quantity = max(0, self.state.order_quantity + decision.step_size * derivative)<br>        self.state.counter += (self.past_derivative * derivative &lt; 0)<br>        self.past_derivative = derivative<br><br>    def objective_fn(self, decision, exog_info):<br>        &quot;&quot;&quot;<br>        Calculate the objective function value based on the decision and exogenous information.<br><br>        :param decision: Decision object<br>        :param exog_info: Dictionary containing the exogenous information<br>        :return: Objective function value<br>        &quot;&quot;&quot;<br>        return self.price * min(self.state.order_quantity, exog_info[&#39;demand&#39;]) - self.cost * self.state.order_quantity<br><br>    def step(self, decision):<br>        &quot;&quot;&quot;<br>        Perform a single step of the Adaptive Market Planning model.<br><br>        :param decision: Decision object<br>        &quot;&quot;&quot;<br>        self.t += 1<br>        exog_info = self.exog_info_fn(decision)<br>        onestep_contribution = self.objective_fn(decision, exog_info)<br>        self.obj += onestep_contribution if self.reward_type == &#39;Cumulative&#39; else (onestep_contribution if self.t == self.T else 0)<br>        self.transition_fn(decision, exog_info)<br>    <br>    def reset(self):<br>        self.t = 0<br>        self.obj = 0.0<br>        self.learning_list = []<br><br>class AdaptiveMarketPlanningPolicy:<br>    &quot;&quot;&quot;Base class for the Adaptive Market Planning policy.&quot;&quot;&quot;<br><br>    def __init__(self, model, theta_step):<br>        &quot;&quot;&quot;<br>        Initialize the Adaptive Market Planning policy.<br><br>        :param model: Adaptive Market Planning model object<br>        :param theta_step: Step size parameter for the policy<br>        &quot;&quot;&quot;<br>        self.model = model<br>        self.theta_step = theta_step<br>        self.decision = Decision(0)<br><br>    def kesten_rule(self):<br>        &quot;&quot;&quot;<br>        Apply Kesten&#39;s rule to determine the step size.<br><br>        :return: Decision object with the calculated step size<br>        &quot;&quot;&quot;<br>        self.decision.step_size = self.theta_step / (self.theta_step + self.model.state.counter - 1)<br>        return self.decision<br><br>    def run_policy(self):<br>        &quot;&quot;&quot;<br>        Run the Adaptive Market Planning policy for the specified number of time steps.<br><br>        :return: Tuple containing the final objective value and the learning list<br>        &quot;&quot;&quot;<br>        np.vectorize(lambda _: self.model.step(self.kesten_rule()))(np.zeros(self.model.T))<br>        return self.model.obj, self.model.learning_list.copy()<br>    <br># Set up the model parameters<br>cost, price, theta_step, T, reward_type = 20, 26, 50, 24, &quot;Terminal&quot;<br>trial_size,  demand_scale = 100, 100<br>init_state = {&#39;order_quantity&#39;: 0, &#39;counter&#39;: 0}<br><br># Initialize the model and policy<br>model = AdaptiveMarketPlanningModel(init_state, T, reward_type, price, cost)<br>policy = AdaptiveMarketPlanningPolicy(model, theta_step)<br><br># Run multiple iterations of the policy<br>rewards_per_iteration = np.zeros(trial_size)<br>learning_list_per_iteration = []<br>for ite in range(trial_size):<br>    model.reset()<br>    reward, learning_list = policy.run_policy()<br>    rewards_per_iteration[ite] = reward<br>    learning_list_per_iteration.append(learning_list)<br><br># Calculate cumulative average rewards<br>nelem = np.arange(1, trial_size + 1)<br>rewards_per_iteration_cum_avg = rewards_per_iteration.cumsum() / nelem<br>if reward_type == &quot;Cumulative&quot;:<br>    rewards_per_iteration_cum_avg /= T<br>    rewards_per_iteration /= T<br><br># Calculate the optimal order quantity<br>optimal_order_quantity = -np.log(cost / price) * 100<br><br># Print the results<br>print(f&quot;Optimal order_quantity for price {price} and cost {cost} is {optimal_order_quantity:.2f}&quot;)<br>print(f&quot;Reward type: {reward_type}, theta_step: {theta_step}, T: {T} - Average reward over {trial_size} iterations is: {rewards_per_iteration_cum_avg[-1]:.2f}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*d9m9aQ8jlRgIJZDfhY7GIQ.png" /><figcaption>Comparison of order quantity over time: The model’s adaptive learning using Kesten’s Rule (θ=50) fluctuates around the analytical solution.</figcaption></figure><h4>Wrap up</h4><p>The Adaptive Market Planning model, with its ability to learn and adapt based on observed outcomes, is just one example of how Sequential Decision Analysis can be applied to real-world problems. Whether you’re managing inventory, planning capacity, or allocating resources, the principles of SDA can help you make better decisions in the face of uncertainty.</p><h4><strong>Helpful resources</strong></h4><ul><li>Free Sequential Decision Analysis book: <a href="http://tinyurl.com/sdabook">http://tinyurl.com/sdabook</a></li><li>Prof. Powell’s GitHub Repo: <a href="https://tinyurl.com/sdagithub">https://tinyurl.com/sdagithub</a></li><li>My Jupyter notebook implementation: <a href="https://github.com/nweissmueller/adaptiveMarketPlanning">https://github.com/nweissmueller/adaptiveMarketPlanning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c15e92e69302" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>