<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Debisree Ray on Medium]]></title>
        <description><![CDATA[Stories by Debisree Ray on Medium]]></description>
        <link>https://medium.com/@debisreer?source=rss-dbc85e29e884------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Z4hTUsB-5X9pjytm.jpg</url>
            <title>Stories by Debisree Ray on Medium</title>
            <link>https://medium.com/@debisreer?source=rss-dbc85e29e884------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:10:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@debisreer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Linear Regression, From the Ground Up]]></title>
            <link>https://medium.com/data-science-collective/linear-regression-from-the-ground-up-5dd85d5c5bdf?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dd85d5c5bdf</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 20:06:58 GMT</pubDate>
            <atom:updated>2026-10-31T20:35:36.754Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Why the Straight Line Works, When It Fails, and What the Residuals Reveal</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Krg243CFpzIhG3OxUzGdA.png" /></figure><p>Linear regression is the oldest tool in the modern statistical toolkit and, by a wide margin, the most frequently misapplied. Its appeal is obvious: the model is interpretable, the solution is exact, and the arithmetic is fast enough to be trivial on any dataset a practitioner is likely to encounter. Its danger is equally obvious once examined. A regression will return coefficients, standard errors, and a respectable R² whether or not the assumptions that give those numbers meaning are satisfied. The software does not complain. The output looks identical either way.</p><h3>1. The Model</h3><p>The model states that the outcome is a weighted sum of the predictors plus an unobserved error:</p><pre>y = β₀ + β₁x₁ + β₂x₂ + ... + βₚxₚ + ε</pre><p>Collecting n observations, this is written compactly as y = Xβ + ε, where y is the n×1 vector of outcomes, X is the n×(p+1) design matrix whose first column consists of ones (supplying the intercept), and β is the vector of coefficients to be estimated.</p><p>A persistent source of confusion deserves attention here. The term <em>linear</em> describes the relationship between the outcome and the parameters, not between the outcome and the raw measurements. A model containing x², log(x), or an interaction term x₁·x₂ remains a linear regression; these are simply additional columns of X. What the framework cannot accommodate is a parameter embedded inside a nonlinear function, such as y = β₀·e^(β₁x). That is nonlinear regression and requires an iterative solver.</p><p>The common objection that linear regression “cannot fit curves” is therefore mistaken. It cannot fit curves that have not been supplied as columns.</p><h3>2. The Estimator</h3><p>The coefficients are chosen to minimise the sum of squared residuals:</p><pre>RSS(β) = ‖y − Xβ‖²</pre><p>Differentiating with respect to β and setting the result to zero yields the normal equations, and from them the closed-form solution:</p><pre>XᵀX β̂ = Xᵀy        ⟹        β̂ = (XᵀX)⁻¹Xᵀy</pre><p>Two properties of this result are worth noting. First, the second derivative of RSS is 2XᵀX, which is positive semi-definite for any design matrix. The objective is therefore convex, and the single stationary point is a global minimum. There are no local optima. This is precisely why linear regression requires no learning rate, no initialisation and no training loop — a fact that separates it from nearly every model that followed it.</p><p>Second, the formula is a statement of theory rather than a recipe for computation. Forming XᵀX squares the condition number of the problem, so a design matrix that is merely awkward becomes numerically unstable. Production implementations — numpy.linalg.lstsq, scikit-learn&#39;s LinearRegression — proceed by QR or singular value decomposition instead.</p><p>It is also worth recording why squared error is the conventional loss. If the errors are assumed to follow ε ~ N(0, σ²I), the log-likelihood of the data equals a negative constant multiple of RSS plus terms independent of β. Maximising the likelihood is therefore identical to minimising the squared error. Least squares is not a neutral choice of loss function; it is a Gaussian noise assumption expressed in a different notation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EnUqtVNwtpYEiqlKDzYP2Q.png" /><figcaption><em>Figure 1: </em>Sixty simulated property sales. The vertical segments are the residuals — the gap between each observation and the fitted line. Least squares selects the unique line for which the sum of the squared lengths of these segments is smallest. The line here is fitted to floor area alone for visual clarity; the full two-predictor fit appears in Section 7.</figcaption></figure><h3>3. Residuals</h3><p>Residuals are the diagnostic instrument of regression analysis. Every assumption examined in the next section is stated in terms of errors, and every check available in practice is performed on residuals. The distinction between the two is fundamental and frequently elided.</p><h3>Error versus residual</h3><ul><li>The error, εᵢ = yᵢ − (true population line), is the deviation from the unknown true relationship. It is not observable, and never will be.</li><li>The residual, eᵢ = yᵢ − ŷᵢ, is the deviation from the fitted line. It is observable and is what every diagnostic uses.</li></ul><p>The residual serves as an estimate of the error. The gap between the two is the reason regression diagnostics are approximate rather than definitive.</p><h3>Three properties imposed by the algebra</h3><p>When an intercept is included in the model, three facts hold by construction and not as evidence of anything:</p><ol><li>The residuals sum to zero. This is forced by the column of ones. A mean residual of zero confirms the arithmetic, not the model.</li><li>The residuals are orthogonal to every predictor, that is, Xᵀe = 0. Consequently, the correlation between residuals and any included predictor is exactly zero. This is why residuals are plotted against <em>fitted values</em> and against <em>omitted</em> variables — plotting them against an included predictor can reveal nothing.</li><li>The residuals are orthogonal to the fitted values, which produces the decomposition TSS = ESS + RSS and, with it, the definition of R².</li></ol><h3>Estimating the noise</h3><p>The variance of the error term is estimated as</p><pre>s² = RSS / (n − p − 1)</pre><p>Residuals are systematically smaller than errors, because the act of fitting pulls the line toward the observed data. Dividing by n would produce a downward-biased estimate. Since p + 1 degrees of freedom were consumed in estimating the coefficients, only n − p − 1 remain. The intuition is clearest in the extreme case: when n = p + 1 the model fits perfectly, RSS is zero, and nothing whatever has been learned about σ².</p><p>The quantity s, known as the residual standard error, expresses the typical magnitude of a prediction miss in the original units of the outcome. It is closely related to RMSE and is considerably more informative than R² alone, yet it is reported far less often.</p><h3>Residuals generate the standard errors</h3><pre>Var(β̂) = σ²(XᵀX)⁻¹        SE(β̂ⱼ) = s·√[(XᵀX)⁻¹]ⱼⱼ        tⱼ = β̂ⱼ / SE(β̂ⱼ)</pre><p>with tⱼ distributed on n − p − 1 degrees of freedom. Every p-value in a regression table is therefore downstream of the residuals. This explains a pattern that recurs throughout the next section: a violated assumption about the error structure may leave the coefficients entirely intact while rendering the inference around them invalid.</p><h3>Residuals are not directly comparable</h3><p>Raw residuals have unequal variances, since Var(eᵢ) = σ²(1 − hᵢᵢ), where hᵢᵢ denotes the leverage of observation i — the i-th diagonal element of the projection matrix H = X(XᵀX)⁻¹Xᵀ. An observation positioned far from the centre of the predictor space exerts a strong pull on the fitted line and consequently exhibits an artificially <em>small</em> residual. Such observations conceal themselves from a naive residual plot.</p><p>Three adjustments follow:</p><ul><li>Standardised residuals, eᵢ / (s√(1 − hᵢᵢ)), place all observations on a common scale. Values beyond roughly ±3 warrant inspection.</li><li>Leverage averages (p+1)/n across the sample. Values exceeding two or three times that average are considered high.</li><li>Cook’s distance combines residual magnitude and leverage into a single measure of influence, answering how far the entire vector of fitted values would shift were the observation deleted.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GdHqpuJpazA8NzYKKQUCzA.png" /><figcaption><em>Figure 2: Standard diagnostics for the two-predictor model. Panel (b) is the most informative of the three: systematic curvature would indicate a misspecified functional form, and a widening spread would indicate non-constant error variance. Panel (c) matters chiefly for small-sample inference.</em></figcaption></figure><h3>4. Evaluating the Fit</h3><p>R² measures the proportion of variance in the outcome accounted for by the model. Because adding any column enlarges the subspace onto which the outcome is projected, R² can never decrease when a predictor is added — irrelevant predictors included. Adjusted R² introduces a penalty for the number of parameters and can therefore decline when a new variable contributes nothing.</p><p>Both quantities describe variance explained rather than the magnitude of error, and neither is sufficient on its own. A model may report a high R² while still producing predictions that are unacceptably far from reality in absolute terms. Sound practice is to report a variance-explained measure alongside at least one error-magnitude measure expressed in the units of the outcome.</p><p>The comparative behaviour of these metrics — MAE, MSE, RMSE, MAPE, R² and adjusted R², their respective sensitivities to outliers and to scale, and the circumstances under which each is appropriate — is treated in detail in a companion article: <a href="https://medium.com/data-science-collective/a-comparative-overview-of-regression-error-metrics-fcfd439c53b3">A Comparative Overview of Regression Error Metrics</a>. Readers evaluating a fitted regression are advised to consult it before settling on any single number.</p><h3>5. The Assumptions</h3><p>The classical assumptions are typically presented as an undifferentiated list, which creates the impression that all must hold or nothing is valid. They are more usefully understood as a hierarchy: each successive assumption purchases a stronger class of claim. Recognising which assumption supports which claim determines how seriously a given violation should be taken.</p><p>Assumption 1 — Linearity in the parameters. The conditional mean is genuinely E[y|X] = Xβ. <em>Supports:</em> any interpretation of the coefficients at all. <em>Diagnosed by:</em> residuals against fitted values; partial residual plots. <em>If violated:</em> the estimator converges to the best linear approximation of the wrong relationship. Remedied by transformations, polynomial terms, splines or interactions.</p><p>Assumption 2 — Full column rank. No column of X is an exact linear combination of the others, and n exceeds p. <em>Supports:</em> the existence of a unique solution, since XᵀX must be invertible. <em>Diagnosed by:</em> the rank of X; variance inflation factors for near-violations. <em>If violated:</em> infinitely many coefficient vectors fit equally well. Software does not raise an error — lstsq silently returns the minimum-norm solution among them.</p><p>Assumption 3 — Exogeneity, E[ε|X] = 0. The errors have zero conditional mean; no omitted variable is correlated with the included predictors. <em>Supports:</em> unbiasedness and consistency. This is the load-bearing assumption of the entire framework. <em>Diagnosed by:</em> no statistical test. It is a claim about the data-generating process, defensible only through study design or subject-matter knowledge. <em>If violated:</em> bias that does not diminish as the sample grows.</p><p>Assumption 4 — Homoscedasticity, Var(ε|X) = σ². Error variance is constant across the predictor space. <em>Supports:</em> efficiency and the validity of the σ²(XᵀX)⁻¹ covariance formula. <em>Diagnosed by:</em> a funnel or fan shape in the residual plot; the Breusch–Pagan or White test. <em>If violated:</em> the coefficients remain unbiased; the standard errors, t-statistics, and p-values do not. Remedied by heteroscedasticity-consistent (robust) standard errors, weighted least squares, or a transformation of the outcome.</p><p>Assumption 5 — Uncorrelated errors, Cov(εᵢ, εⱼ) = 0. <em>Supports:</em> the same claims as Assumption 4. <em>Diagnosed by:</em> residuals plotted in time or group order; the Durbin–Watson statistic; the autocorrelation function. <em>If violated:</em> the same consequences, typically more severe. Clustered or serially dependent data can inflate t-statistics substantially. Remedied by clustered or Newey–West standard errors, or generalised least squares.</p><p>Assumption 6 — Normally distributed errors, ε ~ N(0, σ²). <em>Supports:</em> exact t and F distributions in small samples. <em>Diagnosed by:</em> a normal Q–Q plot of the residuals. <em>If violated:</em> comparatively little, provided n is reasonably large, since the central limit theorem renders β̂ approximately normal regardless. This is the most over-emphasised assumption in the standard list.</p><p>Two conditions that are frequently but wrongly included among the assumptions should be noted. Linear regression does not require the predictors to be normally distributed; they may be binary, skewed or bounded. Nor does it require the outcome to be marginally normal. The assumption concerns the <em>conditional</em> error distribution, which is why a histogram of y conveys almost no information about whether the model is appropriate.</p><p>The Gauss–Markov theorem summarises the position: under Assumptions 1 through 5, and without any appeal to normality, ordinary least squares is the Best Linear Unbiased Estimator — minimum variance within the class of estimators that are linear in y and unbiased. The qualifiers are load-bearing. Relaxing linearity admits robust estimators that perform better under heavy-tailed errors; relaxing unbiasedness admits penalised estimators that achieve lower mean squared error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*REf9CUqEkZruU8s2XsLnXQ.png" /><figcaption><em>Figure 3: The residual plot is the principal instrument for assessing Assumptions 1, 4, and 5. Panel (a) shows the appearance of a well-specified model. Panel (b) indicates a missing term. Panel (c) indicates non-constant variance, which invalidates the standard errors while leaving the coefficients unbiased. Panel (d), plotted in observation order, indicates serial dependence.</em></figcaption></figure><h3>6. Limitations</h3><p>Sensitivity to outliers and influential observations. Because squared loss grows quadratically, an observation twice as distant contributes four times the penalty. The consequences depend critically on <em>where</em> that observation sits. An outlier with an unremarkable predictor value has a large residual but little effect on the fit. A point with unusual predictor values but consistent with the overall trend has high leverage yet remains harmless. Only an observation possessing both characteristics — high leverage combined with a large residual — materially distorts the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUubXc_6YOnnZVdJpzWV7g.png" /><figcaption><em>Figure 4: A single additional observation is introduced in each panel. In (a) and (b), the fitted slope is essentially unchanged. In ©, where high leverage coincides with a large residual, the slope falls from 1.83 to 1.08 on the strength of one observation out of forty-one. Cook’s distance is designed to detect this third case precisely.</em></figcaption></figure><p>Multicollinearity. Correlated predictors violate no assumption, but they inflate the variance of the coefficient estimates:</p><pre>Var(β̂ⱼ) = σ² / [(n − 1)·sⱼ²·(1 − Rⱼ²)]        VIF = 1 / (1 − Rⱼ²)</pre><p>where Rⱼ² is obtained by regressing predictor j on the remaining predictors. At a pairwise correlation of 0.99, the variance inflation factor approaches 50, and standard errors are roughly seven times larger than they would otherwise be. Coefficients become unstable across samples and may reverse sign. Predictive performance often survives intact; interpretation does not. Variance inflation factors above 5 to 10 are conventionally treated as a warning.</p><p>Failure when p ≥ n. With more predictors than observations, XᵀX is singular, and no unique solution exists. Regularisation is the standard remedy.</p><p>Omitted variable bias. If the true relationship is y = Xβ + Zγ + ε and only X is fitted, then E[β̂] = β + (XᵀX)⁻¹XᵀZγ. The bias is proportional to both the strength of Z’s effect on the outcome and the correlation between Z and the included predictors. Increasing the sample size does not reduce it. This is the formal basis for the standard caution that a regression coefficient represents a conditional association — the expected difference in the outcome between two observations differing by one unit in xⱼ and identical in the included covariates — rather than a causal effect.</p><p>No protection against extrapolation. A linear approximation adequate within the observed range carries no guarantee outside it, and the model possesses no mechanism for signalling that its inputs have moved beyond the data on which it was fitted.</p><p>Additive and constant effects by default. The effect of one predictor is assumed independent of the values of the others, and constant across its own range. This holds only where interaction or polynomial terms have been explicitly specified.</p><p>Practical data requirements. All predictors must be numeric and complete. Categorical variables require encoding, with one level omitted to avoid reintroducing perfect collinearity — the dummy variable trap. Missing values require imputation or deletion, each carrying its own bias.</p><h3>7. A Worked Example</h3><h4>The problem</h4><p>Consider an analyst at a property firm who has records of sixty recent residential sales. For each sale, three quantities are recorded: the price achieved, the floor area of the property, and the number of years since it was built. The analyst is asked three questions that a regression is well suited to answer.</p><p><em>How much is an additional unit of floor area worth, holding age constant?</em> <em>How much value does a property lose per year of age, holding floor area constant?</em> And <em>how large is the irreducible uncertainty — how far from the prediction should a given sale be expected to fall?</em></p><p>The quantities involved are as follows:</p><ul><li><strong>price</strong> — the outcome, in thousands of currency units</li><li><strong>area</strong> — floor area, measured in hundreds of square feet, ranging from 8 to 30 (that is, 800 to 3,000 sq ft)</li><li><strong>age</strong> — years since construction, ranging from 0 to 40</li></ul><h4>Why the data are simulated</h4><p>Real sales data would make the example more convincing and far less instructive, because the true coefficients would be unknown. There would be no way to distinguish an accurate estimate from one that merely looks precise. Simulation removes that ambiguity: the data are generated from a relationship fixed in advance, and the fitted coefficients can then be compared directly with it.</p><p>The generating process used here is:</p><pre>price = 50 + 3.2·area − 0.8·age + ε,        ε ~ N(0, 8²)</pre><p>Every term carries a concrete meaning. The intercept of 50 represents the notional value of a property with zero floor area and zero age — a quantity with no physical interpretation, included because the arithmetic requires an anchor. The coefficient of 3.2 states that each additional hundred square feet adds 3,200 to the price. The coefficient of −0.8 states that each year of age removes 800. The noise term, with a standard deviation of 8, represents everything the two predictors do not capture: location, condition, negotiating position, the state of the market on the day. Its magnitude matters for calibration — against observed prices spanning roughly 45 to 140, a typical deviation of 8 is substantial but not overwhelming.</p><p><strong>By construction, every assumption from Section 5 is satisfied.</strong> The relationship is linear in the parameters; the two predictors are drawn independently of one another and of the noise; the error variance is constant across the predictor space; the errors are independent; and they are drawn from a normal distribution. This is deliberate. The purpose of the exercise is to establish what a correctly specified regression looks like — in its coefficient table, its residual plot and its diagnostic quantities — so that the departures described in Sections 5 and 6 are recognisable when they occur in data that was not built to order.</p><h4>What a successful fit would look like</h4><p>Three outcomes would indicate that the estimator has recovered the truth. The coefficients should land near 3.2 and −0.8, within a margin consistent with their reported standard errors. The residual standard error should land near 8, which would confirm that the model has correctly separated signal from noise rather than absorbing some of the noise into the fit. And the residual plot should show no structure of any kind, since by construction there is none to find.</p><p>The code below fits the model directly from the normal equations rather than calling a library routine, constructs the inference table from the residuals, and verifies the algebraic identities established in Section 3.</p><pre>import numpy as np<br>from scipy import stats<br>from sklearn.linear_model import LinearRegression<br><br># ---------- a toy problem: property price from floor area and age ----------<br>rng  = np.random.default_rng(7)<br>n    = 60<br>area = rng.uniform(8, 30, n)          # floor area, hundreds of sq ft<br>age  = rng.uniform(0, 40, n)          # years since built<br><br># truth: price = 50 + 3.2*area - 0.8*age + noise,  sigma = 8<br>y = 50 + 3.2 * area - 0.8 * age + rng.normal(0, 8, n)<br><br>X = np.column_stack([np.ones(n), area, age])   # design matrix, intercept first<br><br># ---------- fit: solve the normal equations ----------<br>XtX  = X.T @ X<br>beta = np.linalg.solve(XtX, X.T @ y)   # solve rather than invert<br><br># ---------- residuals, and everything derived from them ----------<br>resid  = y - X @ beta<br>p, dof = X.shape[1] - 1, n - X.shape[1]<br>rss    = resid @ resid<br>s2     = rss / dof                     # unbiased estimate of sigma^2<br>se     = np.sqrt(np.diag(s2 * np.linalg.inv(XtX)))<br>t      = beta / se<br>pval   = 2 * stats.t.sf(np.abs(t), dof)<br><br>tss = ((y - y.mean()) ** 2).sum()<br>r2  = 1 - rss / tss<br><br>print(f&quot;{&#39;&#39;:&lt;10}{&#39;coef&#39;:&gt;9}{&#39;std err&#39;:&gt;9}{&#39;t&#39;:&gt;8}{&#39;p&#39;:&gt;10}&quot;)<br>for nm, b, s, tv, pv in zip([&quot;intercept&quot;, &quot;area&quot;, &quot;age&quot;], beta, se, t, pval):<br>    print(f&quot;{nm:&lt;10}{b:&gt;9.3f}{s:&gt;9.3f}{tv:&gt;8.2f}{pv:&gt;10.1e}&quot;)<br><br>print(f&quot;\nresidual std error  {np.sqrt(s2):.2f}  on {dof} df&quot;)<br>print(f&quot;R-squared  {r2:.3f}    adjusted  {1 - (1 - r2) * (n - 1) / dof:.3f}&quot;)<br><br># ---------- algebraic identities from Section 3 ----------<br>print(f&quot;\nmean residual          {resid.mean():.2e}&quot;)             # zero by construction<br>print(f&quot;max |X.T @ resid|      {np.abs(X.T @ resid).max():.2e}&quot;)  # orthogonal to X<br><br>H    = X @ np.linalg.inv(XtX) @ X.T     # projection matrix<br>lev  = np.diag(H)                       # leverage<br>cook = resid**2 / ((p + 1) * s2) * lev / (1 - lev)**2<br>print(f&quot;average leverage       {lev.mean():.3f}   (= (p+1)/n)&quot;)<br>print(f&quot;largest Cook&#39;s D       {cook.max():.3f}&quot;)<br><br>m = LinearRegression().fit(X[:, 1:], y)<br>print(f&quot;\nsklearn  {m.intercept_:.3f}  {np.round(m.coef_, 3)}&quot;)</pre><p>Output:</p><pre>coef  std err       t         p<br>intercept    46.598    3.356   13.89   6.1e-20<br>area          3.239    0.151   21.50   5.0e-29<br>age          -0.761    0.082   -9.29   5.2e-13<br><br>residual std error  7.37  on 57 df<br>R-squared  0.903    adjusted  0.900<br><br>mean residual          -4.18e-14<br>max |X.T @ resid|      2.51e-11<br>average leverage       0.050   (= (p+1)/n)<br>largest Cook&#39;s D       0.120<br><br>sklearn  46.598  [ 3.239 -0.761]</pre><h4>Reading the output:</h4><p>All three criteria set out above are met. The coefficient on area is estimated at 3.239 against a true 3.2, a discrepancy of roughly a quarter of its standard error. The coefficient on age is estimated at −0.761 against a true −0.8, within half a standard error. The residual standard error of 7.37 is a credible estimate of σ = 8, and R² of 0.903 indicates that floor area and age jointly account for about ninety per cent of the price variation — leaving the remaining tenth to the factors deliberately omitted from the generating process.</p><p>Translated back into the original question, the model reports that an additional hundred square feet is worth approximately 3,240, with a 95% interval of roughly ±300, and that each year of age costs approximately 760, with an interval of roughly ±165. Both effects are estimated with sufficient precision to be useful, which is a direct consequence of sixty observations spread evenly across a wide range of both predictors.</p><p>Several points from the preceding sections are visible simultaneously in this output. The core residual diagnostics can be reproduced directly from these quantities.</p><pre>import matplotlib.pyplot as plt<br><br>fig, ax = plt.subplots(1, 2, figsize=(11, 4))<br>ax[0].scatter(X @ beta, resid); ax[0].axhline(0)<br>ax[0].set_xlabel(&quot;fitted value&quot;); ax[0].set_ylabel(&quot;residual&quot;)<br>stats.probplot(resid, dist=&quot;norm&quot;, plot=ax[1])<br>plt.tight_layout(); plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X22AzUqhGQme7Yt4gz6nxw.png" /><figcaption>Figure 5: The output of the code above, shown unstyled as it appears when run. The left panel scatters without pattern around zero and the right tracks the reference line closely — what a correctly specified model looks like.</figcaption></figure><h3>8. Points of Frequent Confusion</h3><p>Error and residual are not synonyms. The error is the deviation from the unknown true relationship and is unobservable; the residual is the deviation from the fitted line and estimates it.</p><p>The divisor n − p − 1 is not arbitrary. Fitting consumes p + 1 degrees of freedom, and residuals are correspondingly smaller than errors. Dividing by n yields the biased maximum-likelihood estimate.</p><p>R² cannot be negative in-sample when an intercept is present, but it can be negative on held-out data, where it indicates that the model predicts less accurately than the outcome’s own mean.</p><p>R² equals the squared correlation between y and x only in simple regression. In the multiple case, it equals the squared correlation between y and the fitted values.</p><p>Normality is an assumption about the errors, not about the predictors or the outcome. Its practical importance diminishes rapidly as the sample size grows.</p><p>Heteroscedasticity and multicollinearity damage different things. The first invalidates the standard errors while leaving the coefficients unbiased. The second inflates coefficient variance while typically leaving predictive accuracy unaffected.</p><p>Statistical significance is not a causal claim. A coefficient is a conditional association given the variables that were included, and any omitted variable correlated with both predictor and outcome biases it irreducibly.</p><h3>9. Conclusion</h3><p>Linear regression rewards the attention it is given. The estimator itself is a single line of linear algebra with a unique global solution, and that simplicity is genuine rather than apparent. The difficulty lies entirely in the conditions required for its output to be meaningful — and those conditions are assessed not through the coefficient table, which always looks plausible, but through the residuals, which do not.</p><p>Two of the limitations catalogued above share a common structure. Multicollinearity inflates coefficient variance, and the case p ≥ n admits no unique solution at all. In both instances XᵀX is singular or nearly so, and an unbiased estimator pays for that condition with variance that may be arbitrarily large. The Gauss–Markov theorem establishes that ordinary least squares is optimal among <em>unbiased</em> linear estimators; it makes no claim that remaining unbiased is worth the price.</p><p>Ridge regression accepts a small, deliberate bias in exchange for a substantial reduction in variance, and in doing so restores the invertibility of XᵀX in every case. That is the subject of the next article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dd85d5c5bdf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/linear-regression-from-the-ground-up-5dd85d5c5bdf">Linear Regression, From the Ground Up</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Central Limit Theorem: Why the Normal Distribution Rules the World]]></title>
            <link>https://medium.com/data-and-beyond/the-central-limit-theorem-why-the-normal-distribution-rules-the-world-336dbd8611ec?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/336dbd8611ec</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[central-limit-theorem]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 20:34:52 GMT</pubDate>
            <atom:updated>2026-08-01T20:44:00.105Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Part 2 of a series on the two pillars of probability theory.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*8F3YfEV_RcqdffKIpnzj2A.gif" /></figure><h3>Introduction</h3><p>If the Law of Large Numbers (covered in <a href="https://medium.com/data-and-beyond/the-law-of-large-numbers-why-averages-can-be-trusted-5f66b1d06d9a">Part 1</a>) tells us <em>where</em> an average is going, the Central Limit Theorem tells us <em>how</em> it gets there. It is arguably the single most consequential result in all of statistics — the reason the bell curve appears in test scores, measurement errors, financial returns, polling margins, and quality control charts. The Central Limit Theorem (CLT) explains a minor miracle: no matter how strange, skewed, or lumpy a population is, the distribution of sample means drawn from it converges toward a normal distribution as the sample size grows.</p><h3>The Core Idea, Stated Plainly</h3><p>Let there be a population with a finite mean μ and finite variance σ². The population itself may have any shape — uniform, exponential, bimodal, wildly skewed. Suppose samples of size <em>n</em> are drawn repeatedly from this population, and the mean of each sample is computed.</p><p>The Central Limit Theorem makes three claims about the distribution of those sample means:</p><ol><li>Its shape approaches a <strong>normal distribution</strong> as <em>n</em> increases, regardless of the population’s shape.</li><li>Its center equals the population mean: E[X̄] = μ.</li><li>Its spread shrinks predictably: SD(X̄) = σ/√n, a quantity known as the <strong>standard error</strong>.</li></ol><p>The third point deserves emphasis. The √n in the denominator implies that halving the uncertainty in an estimate requires quadrupling the sample size. This law of diminishing returns governs the economics of every survey, clinical trial, and A/B test ever conducted.</p><h3>The Formal Statement</h3><p>Let X₁, X₂, …, Xₙ be independent and identically distributed (i.i.d.) random variables with E[Xᵢ] = μ and Var(Xᵢ) = σ² &lt; ∞. The sample mean is defined as:</p><p><strong>X̄ₙ = (1/n) Σ Xᵢ</strong></p><p>The Central Limit Theorem states that the standardized sample mean converges in distribution to the standard normal:</p><p><strong>√n (X̄ₙ − μ) / σ → N(0, 1) as n → ∞</strong></p><p>Equivalently, for large n, the sample mean is approximately distributed as:</p><p><strong>X̄ₙ ≈ N(μ, σ²/n)</strong></p><p>Two technical points merit attention. First, the convergence is <em>in distribution</em> — a statement about the shape of probabilities, weaker than the almost-sure convergence of the Law of Large Numbers. Second, the only requirements are independence, identical distribution, and finite variance. Nothing about the population needs to be normal. Therein lies the entire magic of the theorem.</p><h4>A Sketch of Why It Works</h4><p>A full proof employs characteristic functions (the Fourier transform of a distribution). The essence is as follows: the characteristic function of a sum of independent variables is the <em>product</em> of their individual characteristic functions. When the sum is standardized, and each characteristic function is expanded as a Taylor series, the first-order terms vanish (centering removes the mean), the second-order terms accumulate into exp(−t²/2) — precisely the characteristic function of the standard normal — and all higher-order terms are crushed by the √n scaling. The idiosyncrasies of the original distribution reside in those higher-order terms, and they die as n grows. Only the mean and variance survive the limit.</p><h3>Seeing It in Python</h3><p>Theory becomes conviction through simulation. The code below draws samples from an <strong>exponential distribution</strong> — a heavily right-skewed population that looks nothing like a bell curve — and tracks how the sample means organize themselves into a normal distribution.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy import stats<br><br>np.random.seed(42)<br><br># A deliberately non-normal population: exponential with mean 1<br>population_mean = 1.0<br>population_std = 1.0   # for Exp(1), mean = std = 1<br><br>sample_sizes = [1, 5, 30, 100]<br>n_simulations = 10_000<br><br>fig, axes = plt.subplots(1, 4, figsize=(18, 4))<br><br>for ax, n in zip(axes, sample_sizes):<br>    # Draw 10,000 samples of size n; compute each sample&#39;s mean<br>    sample_means = np.random.exponential(scale=1.0,<br>                                         size=(n_simulations, n)).mean(axis=1)<br><br>    print(f&quot;n = {n:3d} | mean of sample means = {sample_means.mean():.4f} | &quot;<br>          f&quot;SD = {sample_means.std():.4f} | &quot;<br>          f&quot;theory σ/√n = {1/np.sqrt(n):.4f} | &quot;<br>          f&quot;skew = {stats.skew(sample_means):+.3f}&quot;)<br><br>    ax.hist(sample_means, bins=60, density=True,<br>            color=&quot;steelblue&quot;, alpha=0.7, edgecolor=&quot;white&quot;)<br><br>    # Overlay the normal curve predicted by the CLT<br>    x = np.linspace(sample_means.min(), sample_means.max(), 300)<br>    se = population_std / np.sqrt(n)<br>    normal_pdf = (1 / (se * np.sqrt(2 * np.pi))) * \<br>                 np.exp(-0.5 * ((x - population_mean) / se) ** 2)<br>    ax.plot(x, normal_pdf, &quot;r-&quot;, lw=2, label=&quot;CLT prediction&quot;)<br><br>    ax.set_title(f&quot;n = {n}\nSE = {se:.3f}&quot;)<br>    ax.legend()<br><br>plt.suptitle(&quot;Sample means of an Exponential(1) population&quot;, fontsize=14)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FiWzQACwv1PXWUcs696MSw.png" /><figcaption>four-panel histograms at n = 1, 5, 30, 100 with red CLT prediction curves</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QdZi7vHbEa_57aRWj9mktg.png" /><figcaption>formatted output table — mean of sample means, SD, theory σ/√n, skewness]</figcaption></figure><p>The numbers tell the whole story before the plots do. At every sample size, the mean of the sample means sits at the population mean of 1; that is claim (2). The empirical standard deviation of the sample means tracks the theoretical σ/√n to two or three decimal places; claim (3): 0.4480 versus 0.4472 at n = 5, and 0.1009 versus 0.1000 at n = 100. Meanwhile, the skewness, the fingerprint of the exponential’s asymmetry, collapses from +1.92 (essentially the raw population, since n = 1) to +0.17 at n = 100. That is claim (1) in action: the population’s shape being averaged out of existence.</p><p>The visual progression is equally striking. At n = 1, the histogram simply reproduces the skewed exponential shape. At n = 5, a hump begins to form, but the right tail lingers. By n = 30, the folk threshold quoted in every introductory course, the histogram is nearly indistinguishable from the red normal curve. At n = 100, the agreement is essentially perfect.</p><h4>Quantifying the Convergence</h4><p>The analysis can go beyond visual inspection: the discrepancy between the sample-mean distribution and the normal distribution can be measured directly.</p><pre>from scipy import stats<br><br>for n in [2, 5, 10, 30, 100, 500]:<br>    sample_means = np.random.exponential(1.0, size=(10_000, n)).mean(axis=1)<br>    # Standardize using the CLT&#39;s predicted mean and standard error<br>    z = (sample_means - 1.0) / (1.0 / np.sqrt(n))<br>    # Kolmogorov–Smirnov distance to the standard normal<br>    ks_stat, _ = stats.kstest(z, &quot;norm&quot;)<br>    print(f&quot;n = {n:4d}  |  skew = {stats.skew(sample_means):+.3f}&quot;<br>          f&quot;  |  KS distance = {ks_stat:.4f}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQnzB8LemaU5vw7S5Mn4Fg.png" /><figcaption>formatted table — skewness and KS distance for n = 2 to 500</figcaption></figure><p>The Kolmogorov–Smirnov distance — the largest gap between the empirical distribution of standardized sample means and the true standard normal — shrinks monotonically, roughly halving each time n grows by a factor of four or five. This is no accident: the Berry–Esseen theorem guarantees that the error decays at rate C/√n, giving the CLT not merely a promise of convergence but a speed limit on it. The KS distance drops from 0.0911 at n = 2 to 0.0167 at n = 100 — a more than fivefold improvement, of the same order as the √50 ≈ 7× reduction the theorem predicts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQx5JTots3cymqeaJzyzbw.png" /><figcaption>Log-log plot of empirical KS distance against the Berry–Esseen 1/√n rate line</figcaption></figure><h3>Worked Example: The Insurance Company’s Dilemma</h3><p>Abstract theorems earn their keep through concrete decisions. An instructive case is an insurer covering 10,000 policyholders. Each claim is wildly unpredictable — most policyholders claim little or nothing in a year, while a few claim enormous amounts. The simulation below models claims with a lognormal distribution tuned to a mean of approximately $500 per policyholder:</p><pre>import numpy as np<br>from scipy import stats<br><br>np.random.seed(42)<br><br># 2,000 simulated years, 10,000 policyholders each<br>claims = np.random.lognormal(mean=4.595, sigma=1.8, size=(2_000, 10_000))<br>totals = claims.sum(axis=1)<br><br>print(f&quot;Individual claim: mean = {claims.mean():,.0f}, &quot;<br>      f&quot;SD = {claims.std():,.0f}, skew = {stats.skew(claims.ravel()):+.1f}&quot;)<br>print(f&quot;Average claim per policyholder: mean = {claims.mean(axis=1).mean():.2f}, &quot;<br>      f&quot;SD = {claims.mean(axis=1).std():.2f}&quot;)<br>print(f&quot;Theory SE = σ/√n = {claims.std()/np.sqrt(10_000):.2f}&quot;)<br>print(f&quot;Total payout: mean = {totals.mean():,.0f}, SD = {totals.std():,.0f}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M0XYVVsaukPmlYcQNts0hA.png" /><figcaption>formatted table — individual claim stats, average claim stats, theory SE, total payout stats</figcaption></figure><p>The contrast in that output is remarkable. A single claim has a skewness of <strong>+69.6</strong> — a distribution so violently asymmetric that its standard deviation ($2,444) is nearly five times its mean ($500). Individually, these claims are close to unforecastable. Yet the <em>average</em> claim across 10,000 policies has a standard deviation of just $24.33, matching the CLT’s prediction of σ/√n = 2444/√10000 = $24.44 almost exactly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5yf2u4Rw4BWxwiVZ12D0hg.png" /><figcaption>violently skewed single-claim distribution on the left, bell-shaped total payout with normal fit on the right</figcaption></figure><p>The total payout is therefore approximately normal, with a mean of $5.0 million and a standard deviation of about $243,000. The insurer can set aside the mean plus three standard errors — roughly <strong>$5.73 million</strong> in reserves — and expect solvency in about 99.9% of years. An entire industry is built on this arithmetic: individual chaos, aggregated, becomes collective predictability.</p><h3>Why the CLT Matters So Much</h3><p><strong>It is the engine of statistical inference.</strong> Every confidence interval of the form “estimate ± 1.96 × standard error” is a direct application of the CLT. Every z-test and, with a finite-sample correction, every t-test rests on the sample mean being approximately normal. Without the CLT, statistical inference would require knowing the exact population distribution in advance — which is precisely what is never known in practice.</p><p><strong>It explains why the normal distribution is everywhere.</strong> Heights, measurement errors, and manufacturing tolerances tend toward normality because each is the <em>sum of many small independent influences</em> — genes, environmental factors, micro-vibrations. The CLT is nature’s averaging mechanism. Wherever many small independent effects add up, a bell curve emerges.</p><p><strong>It justifies A/B testing and polling.</strong> When a pollster reports a “margin of error of ±3%,” that figure is 1.96 × √(p(1−p)/n) — the CLT applied to a Bernoulli population. When a product team declares an experiment significant, the underlying z-statistic assumes the CLT has already done its work on the conversion-rate averages.</p><p><strong>It underlies machine learning practice.</strong> Mini-batch gradient estimates in stochastic gradient descent are sample means; their approximately normal fluctuation around the true gradient is a CLT phenomenon, and it shapes how learning rates and batch sizes trade off against one another.</p><h3>Where the Theorem Breaks</h3><p>Intellectual honesty requires stating the boundaries. The CLT fails, or slows dramatically, when its assumptions are violated.</p><p><strong>Infinite variance.</strong> The Cauchy distribution has no defined mean or variance; averages of Cauchy variables are themselves Cauchy — no convergence ever occurs. Heavy-tailed phenomena such as financial crashes and city sizes can behave similarly, which is why naive normal approximations in risk management have caused genuine disasters.</p><p><strong>Strong dependence.</strong> The i.i.d. assumption matters. Highly correlated observations (time series, clustered data) effectively shrink the sample size, and the standard error σ/√n becomes optimistically small. Generalized versions of the CLT exist for weakly dependent data, but the practitioner must know which regime applies.</p><p><strong>Small n with extreme skew.</strong> “n ≥ 30” is a heuristic, not a law. As the tables above demonstrate, even at n = 30 the sample means retained a skewness of roughly +0.4 — visible asymmetry. For a distribution as extreme as the insurance claims (skew +69.6), convergence at small n would be far slower still. Simulation, as demonstrated throughout this article, is the honest way to check.</p><h3>Conclusion</h3><p>The Central Limit Theorem is the bridge between the unknowable and the estimable. The Law of Large Numbers promises that averages settle down; the CLT specifies the exact probabilistic law governing their fluctuations — normal, centered at μ, with spread σ/√n. That single formula converts raw data into confidence intervals, p-values, error margins, and risk reserves.</p><p>It is worth pausing on how remarkable the claim is: an infinite diversity of populations, all funneled by averaging into one universal shape. The normal distribution is not an assumption statisticians impose on the world. It is a destination the world arrives at on its own, wherever independence and aggregation are present. Understanding the CLT is understanding why statistics works at all.</p><p>For readers who wish to see these two pillars put to work making decisions under uncertainty, hypothesis testing has been covered in detail in two earlier articles by the author:</p><p><a href="https://medium.com/data-and-beyond/statistical-hypothesis-testing-part-i-the-one-tailed-test-807b21af0456">1. Statistical Hypothesis Testing Part I — The One-Tailed Test</a></p><p><a href="https://medium.com/data-and-beyond/statistical-hypothesis-testing-part-ii-the-two-tailed-test-0d207080cd09">2. Statistical Hypothesis Testing Part II — The Two-Tailed Test</a>.</p><p>Every z-statistic and p-value in those articles rests on the theorem developed here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=336dbd8611ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-and-beyond/the-central-limit-theorem-why-the-normal-distribution-rules-the-world-336dbd8611ec">The Central Limit Theorem: Why the Normal Distribution Rules the World</a> was originally published in <a href="https://medium.com/data-and-beyond">Data And Beyond</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Law of Large Numbers: Why Averages Can Be Trusted]]></title>
            <link>https://medium.com/data-and-beyond/the-law-of-large-numbers-why-averages-can-be-trusted-5f66b1d06d9a?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f66b1d06d9a</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[statistical-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 05:26:44 GMT</pubDate>
            <atom:updated>2026-07-07T05:43:07.147Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Part 1 of a series on the two pillars of probability theory</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RqBvJHr80yyJNED5tLg6VA.png" /></figure><h3>Introduction</h3><p>A casino does not know whether the next spin of the roulette wheel will result in a win or a loss. Yet it knows, with near certainty, that it will be profitable by the end of the year. An insurance company cannot predict whether a particular customer will file a claim. Yet it prices policies confidently for millions of customers at once.</p><p>How can businesses built entirely on random events be so predictable? The answer is one of the oldest and most important results in probability theory: the <strong>Law of Large Numbers (LLN)</strong>. In plain terms, it says:</p><blockquote>Individual outcomes are unpredictable, but averages of many outcomes are not. As more and more observations are collected, their average settles down to a fixed, predictable value — the true mean.</blockquote><h3>The Core Intuition: Randomness Averages Out</h3><p>Consider a fair coin, where heads count as 1 and tails as 0. The “true mean” of a single flip is 0.5.</p><ul><li>After <strong>10 flips</strong>, getting 7 heads (an average of 0.7) is completely ordinary.</li><li>After <strong>1,000 flips</strong>, an average of 0.7 would be surprising.</li><li>After <strong>1,000,000 flips</strong>, an average of 0.7 would be essentially impossible.</li></ul><p>Nothing about any individual flip has changed. The coin remains just as random on flip one million as on flip one. What changes is the <em>influence</em> of any single flip on the average. In 10 flips, one lucky head moves the average by 10 percentage points. In a million flips, it moves the average by 0.0001 percentage points.</p><p>This is the entire secret of the LLN. Randomness never disappears — it gets <strong>diluted</strong>. Each new observation carries less and less weight, so the wild swings of individual outcomes cancel each other out, and the average is pulled inexorably toward the true mean.</p><p>A common misunderstanding is worth clearing up immediately. The LLN does <strong>not</strong> say that after a streak of heads, tails become “due” to balance things out. The coin has no memory. The average converges not because past imbalances are corrected, but because the sheer volume of future flips drowns them out. This mistaken belief has a name — the <strong>gambler’s fallacy</strong> — and the LLN is often wrongly cited in its support.</p><h3>Setting Up the Mathematics</h3><p>The formal setting requires three ingredients.</p><p><strong>1. A sequence of random variables.</strong> Let X1, X2, X3, … represent the outcomes of repeated observations — coin flips, dice rolls, customer claims, measurement readings.</p><p><strong>2. Two standard assumptions.</strong> The observations are assumed to be <strong>independent and identically distributed (i.i.d.)</strong>:</p><ul><li><em>Independent</em>: one outcome does not influence another. The third coin flip does not care what the first two did.</li><li><em>Identically distributed</em>: every observation comes from the same underlying process. The coin is not swapped for a biased one halfway through.</li></ul><p><strong>3. A true mean.</strong> Each observation has an expected value, written μ= E[Xi​]. This is the theoretical long-run average — 0.5 for a fair coin, 3.5 for a fair die. It is a fixed number, but in real life, it is usually <em>unknown</em>. That is precisely why anyone collects data in the first place.</p><p>The object the LLN describes is the <strong>sample mean</strong> — the ordinary average of the first n observations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*2lo3-czY1Idtfm6-rujNJQ.png" /></figure><p>Note what kind of object this is: because it is built from random inputs, X̄n is itself random. Two people flipping the same coin 100 times will get two different values of X̄₁₀₀. The LLN is a statement about how this randomness behaves as n grows.</p><h3>The Weak Law of Large Numbers</h3><h4>The Statement</h4><p>The Weak Law of Large Numbers says that for any margin of error ε&gt;0, no matter how small:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*ARXD8Znm_ClpOyPqLNy75g.png" /></figure><h4>Reading the Statement in Plain English</h4><p>The symbols compress a very concrete idea. Unpacking them piece by piece:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzoICIndZC9_mHe0a4KHWA.png" /></figure><p>Put together: <em>pick any tolerance, however strict. With enough data, the chance that the sample average misses the true mean by more than that tolerance becomes negligible.</em> This mode of convergence is called <strong>convergence in probability</strong>.</p><h4>Why It Is True: The Proof Idea</h4><p>The proof is short, and its logic is illuminating. It rests on two facts.</p><p><strong>Fact 1: Averaging shrinks variance.</strong> If each observation has variance σ²(a measure of its spread), then the average of n independent observations has variance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*MdQwkyjUJm8SA5E5m-CFxw.png" /></figure><p>This single formula is the mathematical heart of the entire law. The spread of the average dies at a rate 1/n: average 100 observations, and the variance drops 100-fold. It is the precise expression of the “dilution” intuition — independent errors partially cancel rather than accumulate.</p><p><strong>Fact 2: Small variance means small deviation probability.</strong> <strong>Chebyshev’s inequality</strong> states that any random variable is unlikely to stray far from its mean if its variance is small:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*phDYyYhpc9sfMLHp5_n6Eg.png" /></figure><p>The right-hand side visibly goes to 0 as n→∞, and the proof is complete. The chain of logic reads naturally: <em>averaging kills variance, and a variable with vanishing variance cannot stray from its mean.</em></p><h3>The Strong Law of Large Numbers</h3><p>There is a second, stronger version of the law, due to Kolmogorov:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*GpZFf_MsS4WZBIC31Cz3Bg.png" /></figure><p>The difference from the weak law is subtle but real, and it concerns the <em>trajectory</em> of the average over time.</p><ul><li>The <strong>weak law</strong> takes a snapshot: at any single large n, the average is <em>probably</em> close to μ. It technically leaves open the possibility that the average drifts away from μ again and again forever, just at increasingly rare moments.</li><li>The <strong>strong law</strong> watches the whole movie: with probability 1, the running average eventually enters any small neighborhood of μ and <strong>stays there forever</strong>. Escaping infinitely often is an event of probability zero.</li></ul><p>An analogy: the weak law says that on any given late day of the journey, the traveler is probably near home. The strong law says the traveler eventually arrives home and never leaves again. For practical purposes, the distinction rarely matters, but conceptually, the strong law is the fuller guarantee, and remarkably, both hold as long as the true mean μ exists and is finite.</p><h3>Seeing the Law in Python</h3><p>The simulation below draws from an <strong>exponential distribution, </strong>a heavily skewed distribution often used to model waiting times, with true mean μ = 1. It tracks the running average of five independent experiments as the sample grows from 1 to 100,000.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>rng = np.random.default_rng(42)<br><br>n = 100_000          # total number of observations<br>mu = 1.0             # true mean of the exponential distribution<br><br>fig, ax = plt.subplots(figsize=(10, 5))<br><br># Five independent experiments, each tracking its running average<br>for trial in range(5):<br>    samples = rng.exponential(scale=mu, size=n)<br><br>    # running_mean[k] = average of the first k+1 observations<br>    running_mean = np.cumsum(samples) / np.arange(1, n + 1)<br><br>    ax.plot(running_mean, lw=1, alpha=0.8, label=f&#39;Experiment {trial + 1}&#39;)<br><br>ax.axhline(mu, color=&#39;black&#39;, ls=&#39;--&#39;, lw=2, label=f&#39;True mean μ = {mu}&#39;)<br>ax.set_xscale(&#39;log&#39;)                      # log scale shows early chaos clearly<br>ax.set_xlabel(&#39;Number of observations n (log scale)&#39;)<br>ax.set_ylabel(&#39;Running average&#39;)<br>ax.set_title(&#39;Law of Large Numbers: five running averages converging to μ&#39;)<br>ax.legend()<br>plt.tight_layout()<br>plt.show()</pre><p>The picture that emerges tells the whole story. On the left of the chart (small n), the five running averages swing wildly and disagree with each other — pure noise. Moving right, every trajectory is squeezed toward the dashed line at μ=1, and by n=100,000, they are all essentially indistinguishable from the truth. That squeezing <em>is</em> the Law of Large Numbers, rendered visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EeN7a3Yr6SMHsmSTyx7vGQ.png" /></figure><h3>When the Law Breaks: The Cauchy Distribution</h3><p>The LLN is a theorem, not a law of nature, and theorems have conditions. The key requirement is that a finite mean μ <strong>exists</strong>. Some distributions have tails so heavy that no mean exists at all — the most famous being the <strong>Cauchy distribution</strong>.</p><p>For Cauchy data, the average of a million observations is statistically no better behaved than a single observation. More data does not help. At all.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>rng = np.random.default_rng(7)<br>n = 100_000<br><br>samples = rng.standard_cauchy(size=n)<br>running_mean = np.cumsum(samples) / np.arange(1, n + 1)<br><br>plt.figure(figsize=(10, 4))<br>plt.plot(running_mean, lw=1)<br>plt.xscale(&#39;log&#39;)<br>plt.xlabel(&#39;Number of observations n (log scale)&#39;)<br>plt.ylabel(&#39;Running average&#39;)<br>plt.title(&#39;Cauchy data: the running average never settles down&#39;)<br>plt.tight_layout()<br>plt.show()</pre><p>The resulting plot shows a running average that lurches violently even after tens of thousands of observations, occasionally jumping when a single extreme value lands. This is not a curiosity for mathematicians only: financial returns, catastrophe losses, and viral phenomena all have heavy tails, and analysts who trust averages blindly in such domains are leaning on a theorem whose conditions may not hold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGunVCbzaOTPztkUPgtDBQ.png" /></figure><h3>Why the Law Matters</h3><p>The LLN is the quiet foundation beneath much of modern quantitative work:</p><ul><li><strong>The statistics exist because of it.</strong> Estimating an unknown population mean from a sample average is only justified because the LLN guarantees the sample average converges to the truth. In statistical language, the sample mean is a <em>consistent</em> estimator.</li><li><strong>Monte Carlo simulation works because of it.</strong> When an integral or expectation is too hard to compute analytically, one simulates thousands of random scenarios and averages them. The LLN is the guarantee that this average approaches the correct answer.</li><li><strong>Insurance and casinos are businesses because of it.</strong> A single policy or a single bet is a gamble; a million of them is arithmetic. Aggregation converts randomness into predictability — which is, in a sense, the LLN monetized.</li></ul><h3>What the Law Does Not Say</h3><p>One question the LLN leaves completely unanswered: at any <em>finite</em> sample size, <strong>how far</strong> is the average likely to be from the truth? The law promises arrival at the destination but says nothing about the size and shape of the fluctuations along the way. Is a sample of 500 enough? What error bar should accompany an estimate?</p><p>Answering that question requires the second pillar of probability theory — the <strong>Central Limit Theorem: </strong>That is the subject of an upcoming Part 2 of the series. So, stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f66b1d06d9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-and-beyond/the-law-of-large-numbers-why-averages-can-be-trusted-5f66b1d06d9a">The Law of Large Numbers: Why Averages Can Be Trusted</a> was originally published in <a href="https://medium.com/data-and-beyond">Data And Beyond</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A High-Level Overview of Web Application Development]]></title>
            <link>https://blog.stackademic.com/a-high-level-overview-of-web-application-development-1b061b4a94f2?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b061b4a94f2</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python-web-developer]]></category>
            <category><![CDATA[full-stack-developer]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Tue, 26 May 2026 07:20:03 GMT</pubDate>
            <atom:updated>2026-05-26T07:20:03.882Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Understanding the architecture, core technologies, and evolutionary trajectory of modern web software, from static documents to full-stack systems.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ociFc0-TAldpDWYAZKJSpw.gif" /></figure><h3>I. The Core Primitives: What Constitutes a Web Application</h3><p>A <strong>web application</strong> is software delivered over the Hypertext Transfer Protocol (HTTP) whose interface is rendered inside a browser runtime. Three primitives govern the entire discipline.</p><p><strong>HTTP</strong> is a stateless, request-response protocol formalized in RFC 2616. A client transmits a request; a server returns a response. Each exchange is self-contained — the server retains no memory of prior interactions. This property, called <em>statelessness</em>, is a deliberate architectural constraint identified by Roy Fielding in his 2000 doctoral dissertation introducing Representational State Transfer (REST). Statelessness simplifies server design, enables horizontal scaling, and makes each request independently cacheable.</p><p><strong>The browser</strong> is not a passive document viewer. It is a sandboxed runtime environment capable of parsing markup (Hypertext Markup Language, or HTML), applying stylesheets (Cascading Style Sheets, or CSS), and executing a general-purpose programming language (JavaScript). Modern browsers contain a rendering engine (Blink, WebKit, Gecko), a JavaScript engine (V8, SpiderMonkey, JavaScriptCore), a networking stack, and a security sandbox — collectively forming a virtual machine rivaling desktop application runtimes in capability.</p><p><strong>The server</strong> is a program on a remote machine that listens on a network port, receives HTTP requests, executes computation, and returns HTTP responses. It may be written in any language. The browser never observes the server’s source code — only its output.</p><p>The <strong>central architectural question</strong> of web development is: <em>where does computation occur?</em> The answer partitions the entire technology ecosystem into two domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jecHGaq7J-S7voeYx6pPNQ.png" /></figure><p>Every technology in web development belongs to one side or the other. Confusion between the two is the single most common source of misunderstanding among newcomers to the field.</p><h3>II. The Static Web at a Glance: Documents, Not Programs</h3><p>The World Wide Web, created by Tim Berners-Lee at CERN in 1991, was originally a <strong>document retrieval system</strong>. A server stores files. A browser requested and rendered them. Every visitor received identical content.</p><h4>How HTML Defines Structure</h4><p>HTML is a <strong>declarative markup language</strong>. It expresses <em>what elements exist and how they relate hierarchically</em>, not how they should behave.</p><pre>&lt;article&gt;<br>    &lt;h1&gt;On the Architecture of Web Applications&lt;/h1&gt;<br>    &lt;p&gt;A web application is software delivered over the HTTP protocol.&lt;/p&gt;<br>    &lt;a href=&quot;/references.html&quot;&gt;References&lt;/a&gt;<br>&lt;/article&gt;</pre><p>The browser parses HTML into a tree structure called the <strong>Document Object Model (DOM)</strong> — an in-memory representation where every element, attribute, and piece of text becomes a programmable object known as a “node.” The DOM is not the HTML source file itself; it is a live object tree that the browser constructs <em>from</em> the source. JavaScript interacts with the page by reading and manipulating this tree, not by editing the raw markup. The DOM is therefore the bridge between static documents and dynamic behavior.</p><p>For example, the HTML above produces a DOM tree with this structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fj1oAh8Xp3bWmbT2IzDXlw.png" /></figure><p>Each node in this tree is an object with properties, methods, and event listeners. The entire interactive web is built on programmatic access to this tree.</p><h4>How CSS Controls Presentation</h4><p>CSS separates presentation from structure through a <strong>pattern-matching rule system</strong>. Selectors identify DOM elements; declarations specify their visual properties. The browser’s rendering engine resolves specificity, applies inheritance, computes layout geometry, and paints the result to the screen.</p><pre>body {<br>    font-family: Georgia, serif;<br>    max-width: 700px;<br>    margin: 0 auto;<br>    line-height: 1.7;<br>}<br><br>h1 {<br>    font-size: 2rem;<br>    border-bottom: 2px solid #e0e0e0;<br>}</pre><p>CSS is also declarative. Rather than issuing sequential drawing instructions, it establishes invariants: <em>“all </em><em>h1 elements possess these properties.&quot;</em> The rendering engine enforces those invariants regardless of when or how elements enter the DOM.</p><h4>Where Static Sites Reach Their Limit</h4><p>HTML and CSS can produce responsive, multi-page websites — portfolios, documentation, publications. They cannot respond to user input beyond hyperlinks, retrieve data from a database, personalize content per visitor, or maintain state across interactions. A static site is a <strong>publication</strong>. An application requires a <strong>programming language</strong>.</p><h3>III. JavaScript in Brief: Making the DOM Interactive</h3><p>JavaScript (Brendan Eich, 1995) is the only programming language browsers execute natively. It provides programmatic access to the DOM, enabling the page to respond to events, modify its own structure, and communicate with servers — all without a full page reload.</p><pre>&lt;button id=&quot;btn&quot;&gt;Clicked 0 times&lt;/button&gt;<br><br>&lt;script&gt;<br>    let count = 0;<br>    const button = document.getElementById(&quot;btn&quot;);<br>    button.addEventListener(&quot;click&quot;, () =&gt; {<br>        count++;<br>        button.textContent = `Clicked ${count} times`;<br>    });<br>&lt;/script&gt;</pre><p>This pattern is called <strong>imperative DOM manipulation</strong>. The programmer locates nodes in the DOM by identifier, attaches event listeners, and directly mutates node properties. document.getElementById(&quot;btn&quot;) traverses the DOM tree, finds the node withid=&quot;btn&quot;, and returns a reference to it. The addEventListener method attaches a callback function that fires when the specified event occurs. The textContent property, when modified, triggers the browser&#39;s rendering engine to repaint the affected region of the screen.</p><p>For isolated interactions, the approach suffices. For large-scale applications with hundreds of interdependent elements, it becomes unmanageable — state is scattered across closures, mutations cascade unpredictably, and the relationship between data and display grows opaque.</p><p>This scalability failure motivated the creation of declarative UI frameworks.</p><h3>IV. React at a Conceptual Level: Declarative UI as a Function of State</h3><p>React (Facebook, 2013) introduced a <strong>paradigm shift</strong> in client-side development. Rather than imperatively mutating the DOM node by node, a developer <em>declares</em> what the interface should look like for any given state. React computes the minimal set of actual DOM operations required — a process called <strong>reconciliation</strong> — by comparing a lightweight in-memory copy of the DOM (the “virtual DOM”) against its previous version and applying only the differences to the real DOM.</p><h4>The Defining Abstraction</h4><p>The central concept of React is expressible in a single equation:</p><p><strong><em>UI = f(state)</em></strong></p><p>The user interface is a <strong>pure function of application state</strong>. When state changes, the function re-executes, producing a new virtual DOM tree. React diffs the new tree against the previous one and applies only the changed nodes to the real DOM. The developer never calls getElementById or assigns textContent.</p><pre>import { useState } from &quot;react&quot;;<br><br>function Counter() {<br>    const [count, setCount] = useState(0);<br><br>    return (<br>        &lt;button onClick={() =&gt; setCount(count + 1)}&gt;<br>            Clicked {count} times<br>        &lt;/button&gt;<br>    );<br>}</pre><p>The contrast with the imperative version is architectural, not cosmetic. In the imperative model, the programmer manages both <em>what the UI should be</em> and <em>how to transition between states</em>. In the declarative model, the programmer specifies only the former; the framework handles the latter.</p><p>The syntax used inside the return statement — HTML-like tags embedded in JavaScript — is called <strong>JSX</strong> (JavaScript XML). JSX is not valid JavaScript; it is a syntactic extension that a compiler (such as Babel or the TypeScript compiler) transforms into standard JavaScript function calls before the browser executes them.</p><h4>Components: The Building Blocks</h4><p>React’s structural unit is the <strong>component</strong> — a self-contained, composable function that accepts data (called <em>props</em>, short for “properties”) and returns a description of UI.</p><pre>function BookCard({ title, author }) {<br>    return (<br>        &lt;div className=&quot;card&quot;&gt;<br>            &lt;h3&gt;{title}&lt;/h3&gt;<br>            &lt;p&gt;{author}&lt;/p&gt;<br>        &lt;/div&gt;<br>    );<br>}<br><br>function Library({ books }) {<br>    return (<br>        &lt;div&gt;<br>            {books.map(b =&gt; (<br>                &lt;BookCard key={b.id} title={b.title} author={b.author} /&gt;<br>            ))}<br>        &lt;/div&gt;<br>    );<br>}</pre><p>Components are analogous to functions in procedural programming: they encapsulate logic, accept inputs, produce outputs, and compose into larger structures. This composability is what allows React to scale to applications with thousands of UI elements.</p><h4>What React Does and Does Not Do</h4><p>React is a <strong>client-side rendering library</strong>. It runs in the browser. It <em>generates</em> HTML and CSS; it does not replace them. React cannot access databases, read files from disk, or perform any server-side operation. For data, it must issue HTTP requests to a backend. React occupies one side of the network — it always requires the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cUFlCb5b9SlXTpQYGgOlLQ.png" /></figure><h3>V. Node.js in Context: JavaScript on the Server</h3><p>Until 2009, JavaScript execution was confined to browsers. <strong>Node.js</strong> (Ryan Dahl, 2009) extracted Chrome’s V8 JavaScript engine and embedded it in a standalone runtime with access to the file system, network sockets, and operating system APIs (Application Programming Interfaces).</p><p>This was architecturally significant. JavaScript — previously a client-only language — could now serve as a server-side runtime, handling HTTP requests, connecting to databases, and executing business logic. The appeal was linguistic unification: the same language on both sides of the network.</p><p><strong>Express</strong>, the dominant Node.js web framework, mirrors the route-to-handler pattern found across all server-side frameworks:</p><pre>const express = require(&quot;express&quot;);<br>const app = express();<br>app.use(express.json());<br><br>let books = [{ id: 1, title: &quot;Dune&quot;, author: &quot;Frank Herbert&quot; }];<br><br>app.get(&quot;/api/books&quot;, (req, res) =&gt; res.json(books));<br><br>app.post(&quot;/api/books&quot;, (req, res) =&gt; {<br>    const book = { id: books.length + 1, ...req.body };<br>    books.push(book);<br>    res.status(201).json(book);<br>});<br><br>app.listen(4000);</pre><p>The structural equivalence to Python frameworks is not coincidental. It reflects the universality of the underlying pattern: <strong>map URL paths to handler functions that process requests and return responses.</strong></p><h3>VI. The Evolution of Python API Frameworks: From Flask to FastAPI</h3><p>The trajectory from Flask to FastAPI illustrates a broader shift in server-side architecture: from synchronous, unvalidated, manually documented APIs to asynchronous, type-safe, auto-documented ones.</p><h4>Early Paradigm: Flask (2010) — The Synchronous Micro Framework</h4><p>Flask, created by Armin Ronacher, introduced the decorator-based routing pattern to Python. Its minimalism was its strength — and its limitation.</p><pre>from flask import Flask, jsonify, request<br><br>app = Flask(__name__)<br>books = [{&quot;id&quot;: 1, &quot;title&quot;: &quot;Dune&quot;, &quot;author&quot;: &quot;Frank Herbert&quot;}]<br><br>@app.route(&quot;/api/books&quot;, methods=[&quot;GET&quot;])<br>def get_books():<br>    return jsonify(books)<br><br>@app.route(&quot;/api/books&quot;, methods=[&quot;POST&quot;])<br>def create_book():<br>    data = request.get_json()<br>    book = {&quot;id&quot;: len(books) + 1, &quot;title&quot;: data[&quot;title&quot;], &quot;author&quot;: data[&quot;author&quot;]}<br>    books.append(book)<br>    return jsonify(book), 201</pre><p><strong>Key characteristics and limitations:</strong></p><p><strong>Synchronous execution.</strong> Flask operates on the Web Server Gateway Interface (WSGI), a synchronous protocol. Each request occupies a thread for its entire duration. If a handler performs I/O — a database query, an external API call — that thread blocks, doing nothing, until the operation completes. Under concurrent load, blocked threads accumulate, and throughput collapses.</p><p><strong>No built-in validation.</strong> The expression data[&quot;title&quot;] trusts that the client sent well-formed JSON containing a title field. If it did not, the server raises an unhandled KeyError 500 Internal Server Error in production. Validation must be bolted on through third-party libraries (Marshmallow, Cerberus), adding boilerplate and dependencies.</p><p><strong>No auto-generated documentation.</strong> API consumers (frontend developers, third-party integrators) must rely on separately maintained documentation that inevitably drifts from the actual implementation.</p><h4>Intermediate Step: Django REST Framework — The Enterprise Approach</h4><p>Django REST Framework (DRF) addressed Flask’s gaps by providing serializers for validation, viewsets for CRUD (Create, Read, Update, Delete) boilerplate, browsable APIs for debugging, and integrated authentication. However, DRF was tightly coupled to Django’s ORM (Object-Relational Mapping) and middleware stack — a full application framework, not a composable library.</p><pre>from rest_framework import serializers<br><br>class BookSerializer(serializers.Serializer):<br>    id = serializers.IntegerField(read_only=True)<br>    title = serializers.CharField(max_length=200)<br>    author = serializers.CharField(max_length=100)</pre><p>DRF solved real problems but imposed Django’s weight on projects that needed only an API layer.</p><h4>The Modern Standard: FastAPI (2018) — Async, Validated, Self-Documenting</h4><p>FastAPI, created by Sebastián Ramírez, rethought Python web APIs from first principles by leveraging two language features that did not exist when Flask was designed: <strong>type hints</strong> (PEP 484, introduced in Python 3.5, 2014) and <strong>native async/await</strong> (PEP 492, introduced in Python 3.5, 2015).</p><pre>from fastapi import FastAPI, HTTPException<br>from pydantic import BaseModel<br><br>app = FastAPI()<br><br>class BookCreate(BaseModel):<br>    title: str<br>    author: str<br><br>class Book(BookCreate):<br>    id: int<br><br>books: list[Book] = [<br>    Book(id=1, title=&quot;Dune&quot;, author=&quot;Frank Herbert&quot;),<br>]<br><br>@app.get(&quot;/api/books&quot;, response_model=list[Book])<br>async def get_books():<br>    return books<br><br>@app.get(&quot;/api/books/{book_id}&quot;, response_model=Book)<br>async def get_book(book_id: int):<br>    book = next((b for b in books if b.id == book_id), None)<br>    if not book:<br>        raise HTTPException(status_code=404, detail=&quot;Not found&quot;)<br>    return book<br><br>@app.post(&quot;/api/books&quot;, response_model=Book, status_code=201)<br>async def create_book(book: BookCreate):<br>    new_book = Book(id=len(books) + 1, **book.model_dump())<br>    books.append(new_book)<br>    return new_book</pre><p>Three architectural innovations define FastAPI and represent the current state of the art in Python API development.</p><h4>Key Innovation 1: Pydantic — Declarative, Type-Driven Validation</h4><p>In Flask, validation is procedural — the developer writes conditional checks for each field. In FastAPI, validation is <strong>declarative</strong>. The BookCreate model is a schema that specifies the exact shape of valid input. Pydantic, the data validation library FastAPI is built upon, enforces that schema at runtime.</p><pre>class BookCreate(BaseModel):<br>    title: str          # must be a string<br>    author: str         # must be a string</pre><p>If a client sends {&quot;title&quot;: 123, &quot;author&quot;: null}, Pydantic rejects the request <em>before the handler function executes</em>, returning a structured 422 (Unprocessable Entity) response:</p><pre>{<br>    &quot;detail&quot;: [<br>        {&quot;loc&quot;: [&quot;body&quot;, &quot;title&quot;], &quot;msg&quot;: &quot;Input should be a valid string&quot;, &quot;type&quot;: &quot;string_type&quot;},<br>        {&quot;loc&quot;: [&quot;body&quot;, &quot;author&quot;], &quot;msg&quot;: &quot;Input should be a valid string&quot;, &quot;type&quot;: &quot;string_type&quot;}<br>    ]<br>}</pre><p>The shift from procedural to declarative validation eliminates an entire class of bugs — malformed input never reaches business logic.</p><h4>Key Innovation 2: Async I/O — Cooperative Multitasking</h4><p>Flask’s synchronous model assigns one operating system thread per request. If a handler waits on a database for 50 milliseconds, the thread is idle for 50 milliseconds. Under 1,000 concurrent requests, 1,000 threads sit idle simultaneously — consuming memory and exhausting thread pools.</p><p>FastAPI’s async def handlers operate on an <strong>event loop</strong> — a single-threaded scheduler that manages many concurrent tasks by switching between them at I/O boundaries. When a handler reaches an await expression, it <em>yields control</em> to the event loop, which immediately begins processing another request. When the awaited I/O completes, the original handler resumes from where it paused. One thread serves thousands of concurrent connections.</p><pre>import httpx<br><br>@app.get(&quot;/api/external&quot;)<br>async def fetch_external():<br>    async with httpx.AsyncClient() as client:<br>        # During this await, the event loop serves other requests<br>        response = await client.get(&quot;https://api.example.com/data&quot;)<br>        return response.json()</pre><p>This is <strong>cooperative multitasking</strong> — the same concurrency model that underlies Node.js, Nginx, and Go’s goroutine scheduler. FastAPI runs on ASGI (Asynchronous Server Gateway Interface) servers like Uvicorn, bringing this model to Python’s web ecosystem as a first-class primitive rather than a bolted-on extension.</p><p>The performance implications are substantial. Under I/O-bound workloads, FastAPI consistently handles 5–10x the concurrent connections of Flask, with lower memory consumption per connection.</p><h4>Key Innovation 3: OpenAPI Auto-Generation</h4><p>FastAPI reads type hints at startup and generates a complete <strong>OpenAPI 3.0 specification</strong> — an industry-standard, machine-readable description of every endpoint, its parameters, request body schema, and response schema. An interactive documentation interface (Swagger UI) is served automatically at the /docs path.</p><p>This eliminates documentation drift. The specification is derived from the code itself; it cannot fall out of sync. Frontend teams can auto-generate TypeScript interfaces from the OpenAPI schema, ensuring that client and server agree on data shapes at compile time rather than discovering mismatches at runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIs_pCTswZBdFA_TfEEjgw.png" /></figure><p>This pattern generalizes beyond Python. Across all server-side ecosystems — Node.js, Go, Rust, Java — the industry is converging on the same trio of principles: <strong>async-first concurrency, type-driven validation, and machine-generated API documentation.</strong></p><h3>VII. How the Frontend and Backend Connect: A High-Level View</h3><p>Regardless of which frontend framework or backend language is chosen, the communication mechanism between client and server is invariant: <strong>the browser’s </strong><strong>fetch() API transmitting HTTP requests and receiving JSON (JavaScript Object Notation) responses.</strong></p><pre>// READ — identical whether the backend is Flask, FastAPI, Express, or Go<br>const response = await fetch(&quot;http://localhost:8000/api/books&quot;);<br>const books = await response.json();<br><br>// CREATE<br>await fetch(&quot;http://localhost:8000/api/books&quot;, {<br>    method: &quot;POST&quot;,<br>    headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>    body: JSON.stringify({ title: &quot;Neuromancer&quot;, author: &quot;William Gibson&quot; }),<br>});</pre><p>This code is identical in vanilla JavaScript, React, Vue, Svelte, or Angular. The frontend framework determines how state is managed and how the DOM is updated — not how HTTP requests are issued.</p><h4>A Note on CORS</h4><p>Browsers enforce <strong>Cross-Origin Resource Sharing (CORS)</strong> — a security mechanism that blocks requests between different origins. An “origin” is defined as the combination of protocol, domain, and port. A frontend served from http://localhost:3000 and a backend at http://localhost:8000 constitute different origins; the browser will block requests between them by default.</p><p>The server must explicitly declare which origins are permitted via response headers. In FastAPI:</p><pre>from fastapi.middleware.cors import CORSMiddleware<br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=[&quot;http://localhost:3000&quot;],<br>    allow_methods=[&quot;*&quot;],<br>    allow_headers=[&quot;*&quot;],<br>)</pre><p>In Express:</p><pre>const cors = require(&quot;cors&quot;);<br>app.use(cors({ origin: &quot;http://localhost:3000&quot; }));</pre><p>CORS is enforced by the browser, not the server. Server-to-server communication is unaffected. The mechanism exists to protect end users from malicious client-side scripts making unauthorized requests to APIs to which the user is authenticated.</p><h3>VIII. Static vs. Dynamic: A Conceptual Spectrum</h3><p>The distinction between static and dynamic web applications is defined by <strong>when and where content is generated</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3mzqMmvtOjYcdvPPKhzY7Q.png" /></figure><p>The majority of modern web applications occupy the <strong>SPA</strong> column: a client-side JavaScript application (built with React, Vue, or Svelte) communicates with a backend API. The server never generates HTML — it serves structured data. The browser constructs the entire visual interface by manipulating the DOM based on JSON received from the API.</p><h3>IX. The Technology Stack at a Glance</h3><p>Every technology in web development occupies a defined position in a layered abstraction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LttownJahMUq3KdVxpjZAQ.png" /></figure><p>React does not replace HTML — it generates it. Node.js does not compete with React — they occupy opposite sides of the network. Flask and Express are structural equivalents in different languages. The universal constant is the interface layer: <strong>HTTP requests carrying JSON payloads, interpreted by both sides independently.</strong></p><h3>X. A Minimal Working Example: The Irreducible Full-Stack Application</h3><p>The minimal complete web application consists of two programs communicating over HTTP.</p><p><strong>Server</strong> (FastAPI):</p><pre>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from pydantic import BaseModel<br><br>app = FastAPI()<br>app.add_middleware(CORSMiddleware, allow_origins=[&quot;*&quot;], allow_methods=[&quot;*&quot;])<br><br>class Item(BaseModel):<br>    text: str<br><br>items = [{&quot;id&quot;: 1, &quot;text&quot;: &quot;Learn HTTP&quot;, &quot;done&quot;: True}]<br><br>@app.get(&quot;/api/items&quot;)<br>async def get_items():<br>    return items<br><br>@app.post(&quot;/api/items&quot;, status_code=201)<br>async def add_item(item: Item):<br>    entry = {&quot;id&quot;: len(items) + 1, &quot;text&quot;: item.text, &quot;done&quot;: False}<br>    items.append(entry)<br>    return entry</pre><p><strong>Client</strong> (HTML + JavaScript):</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;body&gt;<br>    &lt;h1&gt;Items&lt;/h1&gt;<br>    &lt;ul id=&quot;list&quot;&gt;&lt;/ul&gt;<br>    &lt;input id=&quot;input&quot; /&gt;<br>    &lt;button onclick=&quot;add()&quot;&gt;Add&lt;/button&gt;<br>    &lt;script&gt;<br>        const API = &quot;http://localhost:8000/api/items&quot;;<br><br>        async function load() {<br>            const items = await (await fetch(API)).json();<br>            document.getElementById(&quot;list&quot;).innerHTML =<br>                items.map(t =&gt; `&lt;li&gt;${t.done ? &quot;✅&quot; : &quot;⬜&quot;} ${t.text}&lt;/li&gt;`).join(&quot;&quot;);<br>        }<br><br>        async function add() {<br>            const input = document.getElementById(&quot;input&quot;);<br>            await fetch(API, {<br>                method: &quot;POST&quot;,<br>                headers: {&quot;Content-Type&quot;: &quot;application/json&quot;},<br>                body: JSON.stringify({ text: input.value })<br>            });<br>            input.value = &quot;&quot;;<br>            load();<br>        }<br><br>        load();<br>    &lt;/script&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>The frontend renders the interface by manipulating the DOM. The backend persists data and enforces validation. They communicate over HTTP with JSON. This two-program, request-response architecture is the <strong>irreducible kernel</strong> of every web application. Authentication, databases, caching, containers, CDNs, and orchestration — all are refinements layered upon this foundation.</p><h3>Concluding Principle</h3><p>Frameworks are transient. The web’s durable primitives are: HTTP as transport, JSON as interchange format, HTML as structure, CSS as presentation, JavaScript as client-side computation, the DOM as the programmatic bridge between markup and behavior, and the client-server boundary as the organizing principle. Mastery of these primitives renders every framework a dialect of an already-familiar language.</p><h3>Glossary of Terms, Abbreviations, and Technical Concepts</h3><p><strong>API:</strong> (Application Programming Interface): A defined set of rules and endpoints through which one piece of software communicates with another. In web development, it typically refers to a set of HTTP endpoints that accept requests and return JSON data.</p><p><strong>ASGI: </strong>(Asynchronous Server Gateway Interface): The asynchronous successor to WSGI. A protocol specification for Python web servers and frameworks that supports async/await, WebSockets, and long-lived connections. Uvicorn is the most common ASGI server.</p><p><strong>async/await: </strong>Language keywords that enable asynchronous programming. async declares a function as a coroutine (a pausable function). await pauses execution until an asynchronous operation completes, yielding control to the event loop to handle other tasks in the meantime.</p><p><strong>CDN (Content Delivery Network): </strong>A geographically distributed network of servers that caches and delivers static content (HTML, CSS, JavaScript, images) from locations physically close to the end user, reducing latency.</p><p><strong>CORS (Cross-Origin Resource Sharing): </strong>A browser security mechanism that restricts web pages from making HTTP requests to a different origin (protocol + domain + port) than the one that served the page, unless the target server explicitly permits it via response headers.</p><p><strong>CRUD (Create, Read, Update, Delete): </strong>The four fundamental operations of persistent data storage, commonly mapped to the HTTP methods POST, GET, PUT/PATCH, and DELETE respectively.</p><p><strong>CSS (Cascading Style Sheets): </strong>A declarative language for describing the visual presentation of HTML documents. Styles are applied through a rule system of selectors and property declarations, with specificity and inheritance determining which rules take effect.</p><p><strong>DOM (Document Object Model): </strong>The browser’s in-memory tree representation of an HTML document. Every HTML element, attribute, and text node becomes a programmable JavaScript object (a “node”). The DOM is what JavaScript reads and manipulates to create dynamic, interactive web pages — it is not the HTML source file, but a live structure built from it.</p><p><strong>Event Loop: </strong>A single-threaded scheduling mechanism that manages concurrent operations by cycling through a queue of tasks. When an async operation (such as a network request) is awaited, the event loop handles other pending tasks until the operation completes, enabling high concurrency without multiple threads.</p><p><strong>Express: </strong>The most widely used web framework for Node.js. It provides a minimal routing and middleware layer for building HTTP servers in JavaScript.</p><p><strong>FastAPI: </strong>A modern, asynchronous Python web framework created by Sebastián Ramírez in 2018. Built on type hints, Pydantic validation, and ASGI, with automatic OpenAPI documentation generation.</p><p><strong>Flask: </strong>A lightweight, synchronous Python web framework created by Armin Ronacher in 2010. Known for its minimalism and decorator-based routing pattern.</p><p><strong>HTML (Hypertext Markup Language): </strong>The standard declarative markup language for structuring content on the web. HTML defines what elements exist on a page and their hierarchical relationships.</p><p><strong>HTTP (Hypertext Transfer Protocol): </strong>The application-layer protocol used for communication between web clients and servers. It follows a request-response model where the client sends a request (with a method, URL, headers, and optional body) and the server returns a response (with a status code, headers, and optional body).</p><p><strong>I/O (Input/Output): </strong>Operations involving data transfer between a program and an external system — file reads, database queries, network requests. I/O operations are typically orders of magnitude slower than in-memory computation, making them the primary target of async optimization.</p><p><strong>JSON (JavaScript Object Notation): </strong>A lightweight, text-based data interchange format consisting of key-value pairs and ordered lists. Despite its name, JSON is language-independent and serves as the dominant format for data exchange between web clients and servers.</p><p><strong>JSX (JavaScript XML): </strong>A syntactic extension to JavaScript used in React that allows HTML-like tags to be written directly within JavaScript code. JSX is not valid JavaScript; it is compiled into standard function calls before execution.</p><p><strong>Middleware: </strong>Software that sits between the incoming HTTP request and the route handler, performing cross-cutting operations such as logging, authentication, CORS header injection, or request parsing. Middleware functions typically process the request, optionally modify it, and pass it to the next handler in the chain.</p><p><strong>Node.js</strong>: A JavaScript runtime environment built on Chrome’s V8 engine that allows JavaScript to execute outside the browser, on servers and local machines. Created by Ryan Dahl in 2009.</p><p><strong>OpenAPI (formerly Swagger)</strong>: An industry-standard specification for describing RESTful APIs in a machine-readable format (JSON or YAML). Tools can generate interactive documentation, client libraries, and server stubs from an OpenAPI specification.</p><p><strong>ORM (Object-Relational Mapping): </strong>A programming technique that maps database tables to classes and rows to objects, allowing database queries to be written as method calls in the application’s programming language rather than raw SQL.</p><p><strong>PEP (Python Enhancement Proposal): </strong>A formal design document for proposing new features, conventions, or processes for the Python programming language. PEP 484 introduced type hints; PEP 492 introduced async/await syntax.</p><p><strong>Props (Properties): </strong>In React, the read-only data is passed from a parent component to a child component. Props are the input mechanism through which components receive the data they need to render.</p><p><strong>Pydantic</strong>: A Python library for data validation and parsing using type annotations. It enforces data schemas at runtime, converting and validating input automatically. FastAPI uses Pydantic as its core validation engine.</p><p><strong>REST (Representational State Transfer): </strong>An architectural style for distributed systems, defined by Roy Fielding in 2000. Key constraints include statelessness, a uniform interface based on resources identified by URLs, and manipulation of those resources through standard HTTP methods.</p><p><strong>RFC (Request for Comments)</strong>: A formal document published by Internet standards organizations (primarily the IETF) that defines protocols, procedures, and conventions for the Internet. RFC 2616 defines HTTP/1.1.</p><p><strong>SPA (Single Page Application): </strong>A web application architecture in which the browser loads a single HTML page and dynamically updates the DOM via JavaScript, rather than requesting new pages from the server. Navigation, rendering, and state management all occur client-side; the server provides data through an API.</p><p><strong>SSR (Server-Side Rendering): </strong>A technique in which the server generates complete HTML for each page request, as opposed to sending a minimal HTML shell and building the page client-side with JavaScript. Frameworks like Next.js support SSR for React applications.</p><p><strong>TCP/IP (Transmission Control Protocol / Internet Protocol): </strong>The foundational network protocol suite of the Internet. TCP provides reliable, ordered delivery of data between applications; IP handles addressing and routing of packets across networks. HTTP operates on top of TCP/IP.</p><p><strong>Type Hints: </strong>Annotations in Python source code that declare the expected data types of variables, function parameters, and return values. Introduced in PEP 484 (Python 3.5), they enable static analysis tools, IDE autocompletion, and — in frameworks like FastAPI — runtime validation and documentation generation.</p><p><strong>V8: </strong>Google’s open-source, high-performance JavaScript and WebAssembly engine, written in C++. It powers the Chrome browser and, extracted into a standalone runtime, forms the core of Node.js.</p><p><strong>Virtual DOM: </strong>A lightweight, in-memory representation of the real DOM maintained by frameworks like React. When application state changes, the framework constructs a new virtual DOM tree, diffs it against the previous version, and applies only the necessary changes to the real DOM — minimizing expensive browser reflows and repaints.</p><p><strong>WebSocket: </strong>A communication protocol that provides full-duplex (bidirectional) communication channels over a single TCP connection. Unlike HTTP’s request-response model, WebSockets allow the server to push data to the client without the client first requesting it, enabling real-time applications such as chat, live dashboards, and collaborative editing.</p><p><strong>WSGI (Web Server Gateway Interface): </strong>A synchronous protocol specification (PEP 3333) defining how Python web servers communicate with Python web applications. Flask and Django traditionally operate on WSGI. It processes one request per thread and does not natively support async operations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b061b4a94f2" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/a-high-level-overview-of-web-application-development-1b061b4a94f2">A High-Level Overview of Web Application Development</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistical Hypothesis Testing-Part II: The Two-Tailed Test]]></title>
            <link>https://medium.com/data-and-beyond/statistical-hypothesis-testing-part-ii-the-two-tailed-test-0d207080cd09?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d207080cd09</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Mon, 25 May 2026 06:52:20 GMT</pubDate>
            <atom:updated>2026-05-25T07:11:10.467Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How scientists test for an effect when its direction is unknown.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OURNIO230p2vX2QGnr08RA.gif" /></figure><blockquote><strong>This article is the second installment of a two-part series; Part I is available </strong><a href="https://medium.com/@debisreer/statistical-hypothesis-testing-part-i-the-one-tailed-test-807b21af0456"><strong>here</strong></a><strong>.</strong></blockquote><h3><strong>1. The non-directional question</strong></h3><p>Part I treated questions with a built-in direction: whether a mean is <em>higher</em> than a claimed value, or <em>lower</em>. Many questions carry no such direction. A test is needed simply to determine whether a quantity <em>differs</em> from a reference value, with departures in either direction equally relevant.</p><p>The corresponding alternative hypothesis is <strong>non-directional</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*N1bF0bepcILh4jfxarugBA.png" /></figure><p>This is the <strong>two-tailed</strong> (two-sided) test. It is the appropriate choice whenever a deviation in <em>either</em> direction would be meaningful and worth detecting. Three illustrations:</p><ul><li>A coin is examined for fairness. Bias toward heads and bias toward tails are both departures from fairness; neither is privileged.</li><li>A manufacturing line targets a precise fill weight. Both over-filling and under-filling are defects — one wastes material or risks harm, the other shortchanges the customer.</li><li>A laboratory instrument is checked for calibration against a known standard. Any drift, positive or negative, signals a fault.</li></ul><p>Because surprising outcomes can now lie at <em>either</em> extreme, the rejection region occupies <em>both</em> tails of the distribution.</p><h3>2. The relationship between the two tests</h3><p>A two-tailed test inherits its entire machinery from Part I, with two adjustments that follow directly from having two rejection regions rather than one.</p><p><strong>The error budget is split.</strong> The total significance level α must be distributed across both tails. For α = 0.05, each tail receives 2.5%. The critical value is therefore drawn further out than in a one-tailed test at the same α: for 24 degrees of freedom, it becomes ±2.06, against the one-tailed 1.71. A two-tailed test sets a <em>higher bar</em> for rejection.</p><p><strong>The p-value is doubled.</strong> Because an extreme result could have occurred in either direction, the tail probability beyond the observed statistic is counted on both sides:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*f9DY0Lp237KPCmzVh8cmGA.png" /></figure><p>The figure below shows the same 5% error budget allocated each way, for 24 degrees of freedom. The contrast is the heart of the matter: the one-tailed test pools all 5% into the predicted tail and rejects past 1.71; the two-tailed test reserves 2.5% for each tail and rejects only past ±2.06.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hZBbpvRmJQOfTs6lpXQs5w.png" /></figure><h3>3. A worked two-tailed problem</h3><p><strong><em>The setup.</em></strong><em> A pharmaceutical line fills capsules with a target of </em><strong><em>500 mg</em></strong><em> of active ingredient. Both over-filling (a toxicity risk) and under-filling (an efficacy failure) are defects, so any deviation from 500 mg matters. A quality-control sample of </em><strong><em>30</em></strong><em> capsules yields a sample mean of </em><strong><em>506 mg</em></strong><em> and a sample standard deviation of </em><strong><em>14 mg</em></strong><em>. At α = 0.05, is there evidence that the true mean fill differs from 500 mg?</em></p><p>Because deviations in either direction are defects, the test is <strong>two-tailed</strong>. The standard deviation is estimated from the sample, so a one-sample <em>t-test</em> applies.</p><p><strong>Step 1 — State the hypotheses.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*n_E_dfkPgT2u4umZPwv9Vw.png" /></figure><p><strong>Step 2 — Standard error.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*xdo5_lIUBbb_XSH15CgMrw.png" /></figure><p><strong>Step 3 — Test statistic.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*5ey8M11lATLcW4geAABd8w.png" /></figure><p><strong>Step 4 — Degrees of freedom.</strong> df = n − 1 = 29.</p><p><strong>Step 5 — Critical values.</strong> For a two-tailed test at α = 0.05, each tail holds 2.5%; the critical values for 29 degrees of freedom are <strong>±2.0452</strong>. Since |2.3474| &gt; 2.0452, the statistic falls in the (upper) rejection region.</p><p><strong>Step 6 — p-value.</strong> Doubled, because the test is two-tailed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*wwv_dSXamEC9VwUrenuwkA.png" /></figure><p><strong>Step 7 — Decision.</strong> Since p ≈ 0.026 &lt; α = 0.05, the null hypothesis is <strong>rejected</strong>: there is evidence that the mean fill differs from 500 mg. A 95% confidence interval for the true mean, (500.8, 511.2) mg, excludes 500 and tells the same story.</p><p>The figure below places the statistic against the two-tailed rejection regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlMMUQQ5dTvrT7xnOxX1nA.png" /></figure><h3>4. How to choose: the decision made before the data</h3><p>The choice between one- and two-tailed forms is not a matter of taste, and it is not made after seeing the data. It is dictated by the research question and fixed in advance. A short decision rule captures almost every case:</p><ul><li><strong>Does a deviation in only one direction carry scientific meaning, with the opposite direction simply uninteresting?</strong> If so, a <em>one-tailed</em> test in that direction is justified — for example, a regulator who cares only whether a contaminant <em>exceeds</em> a limit.</li><li><strong>Would a deviation in either direction be meaningful?</strong> If so, a <em>two-tailed</em> test is required — for example, a calibration check where any drift is a fault.</li><li><strong>When in genuine doubt, the two-tailed test is the conservative default.</strong> It demands stronger evidence and cannot be accused of having been tuned to manufacture a result.</li></ul><p>The consequences of the choice are visible by analysing identical data both ways. Recall the caffeine sample of Part I, with t = 2.22 on 24 degrees of freedom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oagif3tdn1ZMVNcswlf-Hw.png" /></figure><p>The same data produce a p-value of 0.018 under the one-tailed framing and exactly double, 0.036, under the two-tailed framing. This doubling is the source of a serious temptation — and a serious error.</p><h3>5. A necessary caution: the direction is not chosen to fit the result</h3><p>Because a one-tailed test halves the p-value, an analyst who inspects the data notices the effect points “the right way,” and only then declares the test one-tailed has effectively doubled the true false-positive rate while reporting the lower number. This practice — a form of what is broadly termed <em>p-hacking</em> — invalidates the test. The probability statements underlying a p-value are only meaningful when the hypothesis is specified independently of the data it is tested against.</p><p>The safeguard is procedural: the alternative hypothesis, including its direction, is committed to before the data are examined, ideally pre-registered. A one-tailed test is legitimate only when the directional question genuinely preceded the data. When the direction is suggested by the data themselves, the two-tailed test is the honest choice.</p><p>It is worth noting that the one-tailed test’s advantage in <strong>power</strong> — discussed in Part I — is the legitimate flip side of this temptation. Concentrating the error budget in one tail genuinely improves the chance of detecting a true effect in that direction. That benefit is real and valuable, but it is earned only by committing to the direction in advance.</p><h3>6. The two-tailed test in Python</h3><pre>import numpy as np<br>from scipy import stats<br><br>mu_0, x_bar, s, n = 500, 506, 14, 30   # null value, sample mean, sample sd, sample size<br>alpha = 0.05<br><br>se = s / np.sqrt(n)<br>t_stat = (x_bar - mu_0) / se<br>df = n - 1<br><br># TWO-tailed p-value: doubled, because either direction counts<br>p_value = 2 * stats.t.sf(abs(t_stat), df)<br>t_crit  = stats.t.ppf(1 - alpha/2, df)   # note alpha/2<br><br>print(f&quot;t-statistic        : {t_stat:.4f}&quot;)<br>print(f&quot;Critical values    : ±{t_crit:.4f}&quot;)<br>print(f&quot;p-value (two-tail) : {p_value:.4f}&quot;)<br>print(&quot;Reject H0&quot; if p_value &lt; alpha else &quot;Fail to reject H0&quot;)</pre><p>t-statistic: 2.3474<br>Critical values: ±2.0452<br>p-value (two-tail) : 0.0259<br>Reject H0</p><p>With raw data, the two-tailed test is the default behaviour of scipy — no alternative argument is needed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rrpc4-ahpmpdlfobvaEUgA.png" /></figure><p>The contrast with Part I’s code is the whole story in two lines: the p-value is multiplied by two, and the critical value uses alpha/2 rather than alpha.</p><h3>7. Conclusion of the series</h3><p>Across both parts, the architecture of a hypothesis test has remained constant: a skeptical null, a test statistic measuring departure from it, a p-value quantifying how surprising that departure would be under the null, and a significance level fixing how much surprise is tolerated. The one-tailed and two-tailed tests differ only in how the error budget is allocated — wholly into one predicted tail, or split between two — and that single difference flows through to the critical value, the p-value, and the power of the test.</p><p>The deeper lesson is that the most consequential decision is made before any computation: whether the research question is directional. Get that right, in advance and independently of the data, and the arithmetic that follows is routine. Get it wrong — or worse, decide it after the fact to secure a smaller p-value — and even flawless arithmetic produces a misleading conclusion. The p-value measures not whether a hypothesis is true, but how strange the data would be if it were not; honouring that meaning begins with an honestly chosen hypothesis.</p><h3>References</h3><ul><li>Khan Academy — <em>Significance tests (hypothesis testing)</em>, in the AP®/College Statistics course, and especially the video <em>One-tailed and two-tailed tests</em> (Inferential Statistics). Freely available at khanacademy.org; an excellent visual companion to this series.</li><li>Wasserstein, R. L., &amp; Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. <em>The American Statistician</em>, 70(2), 129–133. (Essential reading on the misuse of p-values, including selective one-sided testing.)</li><li>Fisher, R. A. (1925). <em>Statistical Methods for Research Workers.</em> Oliver &amp; Boyd.</li><li>Diez, D., Çetinkaya-Rundel, M., &amp; Barr, C. <em>OpenIntro Statistics</em> (free textbook, openintro.org).</li><li>Moore, D. S., McCabe, G. P., &amp; Craig, B. A. <em>Introduction to the Practice of Statistics.</em> W. H. Freeman.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0d207080cd09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-and-beyond/statistical-hypothesis-testing-part-ii-the-two-tailed-test-0d207080cd09">Statistical Hypothesis Testing-Part II: The Two-Tailed Test</a> was originally published in <a href="https://medium.com/data-and-beyond">Data And Beyond</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistical Hypothesis Testing -Part I: The One-Tailed Test]]></title>
            <link>https://medium.com/data-and-beyond/statistical-hypothesis-testing-part-i-the-one-tailed-test-807b21af0456?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/807b21af0456</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Mon, 25 May 2026 06:07:42 GMT</pubDate>
            <atom:updated>2026-05-25T07:08:55.243Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How scientists determine whether an effect is real or just noise.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NXe4bMjESIsANEHkpoDj3A.gif" /></figure><p>This is the first of a two-part series. Part I establishes the foundations of hypothesis testing and develops the one-tailed (directional) test in full.</p><p><a href="https://medium.com/@debisreer/statistical-hypothesis-testing-part-ii-the-two-tailed-test-0d207080cd09">Part II</a> turns to the two-tailed (non-directional) test and to the central practical question the two parts together are designed to answer: when each form is appropriate.</p><h3>1. Foundations: the logic of a hypothesis test</h3><p>A courtroom supplies the cleanest analogy for what a hypothesis test does. A defendant is presumed innocent, and the prosecution does not attempt to prove innocence — that is the standing assumption. Evidence is gathered, and a single question is posed: <em>if the defendant were genuinely innocent, how improbable would this evidence be?</em> Once the evidence grows improbable enough under the presumption of innocence, that presumption is abandoned.</p><p>A statistical test follows the same structure. It begins with a default, skeptical assumption — the <strong>null hypothesis</strong>, H₀ — which asserts that nothing of interest is occurring: no effect, no difference, no departure from a claimed value. Data are then collected, and the operative question becomes: <em>how surprising would these data be if H₀ were true?</em> When the data are surprising enough, H₀ is rejected in favour of an <strong>alternative hypothesis</strong>, H₁.</p><p>A single principle governs everything downstream: the null is never <em>proven</em> true. A test returns only “reject H₀” or “fail to reject H₀,” exactly as a jury returns “guilty” or “not guilty,” but never “innocent.” Failing to reject the null is a statement about insufficient evidence, not a certificate of its truth.</p><h3>2. The fundamental ingredients</h3><p>Every hypothesis test is assembled from the same five components:</p><ol><li><strong>The null hypothesis (H₀)</strong> — the status quo under challenge, e.g., “the true mean equals 150.”</li><li><strong>The alternative hypothesis (H₁)</strong> — the claim entertained instead, e.g., “the true mean is greater than 150.”</li><li><strong>A test statistic</strong> — a single number computed from the data that expresses how far the observation lies from what H₀ predicts, measured in units of random noise.</li><li><strong>The p-value</strong> — the probability of a test statistic at least as extreme as the one observed, <em>assuming H₀ is true</em>.</li><li><strong>The significance level (α)</strong> — a threshold fixed in advance for how much surprise is tolerated before H₀ is rejected; conventionally, 0.05.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WWcoqhnzb1TkRYRbLbHVDw.png" /></figure><h3>3. The p-value, defined precisely</h3><p>The definition below merits memorisation, word for word.</p><blockquote><strong><em>Definition.</em></strong><em> The </em><strong><em>p-value</em></strong><em> is the probability of obtaining a result </em>at least as extreme<em> as the one observed, </em><strong><em>assuming the null hypothesis is true</em></strong><em>.</em></blockquote><p><em>In symbols: </em><em>p = P(data this extreme or more | H₀ is true).</em></p><p>Stated informally: if nothing were truly happening, how often would chance alone yield a result this surprising, in the direction named by the alternative? A small p-value means “rarely,” and that rarity is what undermines confidence in the null.</p><p>The direction of conditioning must be read with care. The p-value is the probability of <strong>the data given the hypothesis</strong>, never the probability of the hypothesis given the data. It does <strong>not</strong> report the probability that H₀ is true, nor the probability that a result arose “by chance.” Confusing these two directions is the original sin of p-value interpretation.</p><h3>4. What makes a test one-tailed — and when it is appropriate</h3><p>The alternative hypothesis can be framed in two ways, and the framing determines the entire geometry of the test.</p><p>A <strong>two-tailed</strong> (non-directional) alternative asserts only that the true value <em>differs</em> from the null value, in either direction. A <strong>one-tailed</strong> (directional) alternative is more committed: it specifies a direction — that the true value is <em>greater</em> than the null value (a right-tailed test) or <em>less</em> than it (a left-tailed test), but not merely that it is different.</p><p>The one-tailed form is appropriate precisely when the research question is itself directional, and when a departure in the opposite direction would be of no scientific interest. Consider three situations in which only one direction matters:</p><ul><li>A regulator suspects a product contains <em>more</em> of a regulated substance than its label declares. A finding that it contains <em>less</em> would not constitute a violation.</li><li>An engineer asks whether a redesigned process is <em>faster</em> than the existing one. A slower result simply means the redesign failed; there is no separate hypothesis to confirm.</li><li>A pharmacologist asks whether a compound <em>lowers</em> blood pressure. An increase would be a safety concern handled separately, not the question under test.</li></ul><p>The decisive discipline is that the direction is fixed <strong>before the data are examined</strong>, on the basis of the scientific question, never chosen afterward to suit the result. The consequences of violating this rule are taken up in Part II.</p><p>The geometric consequence is that the entire region of “surprising” outcomes is concentrated in one tail of the distribution. The full error budget α is placed on a single side rather than divided between two. This concentration is what gives the one-tailed test greater sensitivity to a true effect <em>in the predicted direction</em> — at the cost of being structurally unable to detect an effect in the opposite direction.</p><h3>5. A worked one-tailed problem</h3><blockquote><strong>The setup.</strong> A coffee chain advertises that its medium latte contains <strong>150 mg of caffeine</strong> on average. A consumer-protection laboratory suspects the machines are over-dosing — that the true mean is higher than advertised. A sample of <strong>25</strong> lattes is tested, yielding a sample mean of <strong>158 mg</strong> and a sample standard deviation of <strong>18 mg</strong>. At a significance level of α = 0.05, is there evidence that the true mean caffeine content exceeds 150 mg?</blockquote><p>The suspicion is directional — concerned only with the mean being <em>higher</em> — so this is a <strong>right-tailed, one-sided test</strong>. Because the standard deviation is estimated from the sample rather than known in advance, the reference distribution is the <strong>t-distribution</strong>, and the procedure is a one-sample <em>t-test</em>.</p><p>The t-test rests on an assumption worth stating explicitly: the underlying measurements are taken to be approximately normally distributed (or the sample is large enough that the Central Limit Theorem makes the sample mean approximately normal regardless). For a continuous quantity such as caffeine content, and a sample of 25, this is a reasonable working assumption. When it fails badly — heavily skewed data, prominent outliers, or very small samples — the t-test can mislead, and one of the alternatives described below is more appropriate.</p><p><strong>Step 1 — State the hypotheses.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*gi49teGqnhlZRqHJnvk65A.png" /></figure><p><strong>Step 2 — Compute the standard error of the mean.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*nrb0TXK2LDlqLEC97F6RHw.png" /></figure><p><strong>Step 3 — Compute the test statistic.</strong> The t-statistic counts how many standard errors the observed mean lies above the null value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*LDBdMhPXUPed6mjjy1Ob2Q.png" /></figure><p><strong>Step 4 — Determine the degrees of freedom.</strong> For a one-sample t-test, df = n − 1 = 24.</p><p><strong>Step 5 — Identify the critical value.</strong> For a one-tailed test at α = 0.05, the full 5% sits in the upper tail; the critical value for 24 degrees of freedom is <strong>1.7109</strong>. Since 2.2222 &gt; 1.7109, the statistic lies inside the rejection region.</p><p><strong>Step 6 — Compute the p-value.</strong> Because the test is one-tailed, the p-value is the upper-tail area beyond the statistic alone — it is <em>not</em> doubled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*YY72fTUuJCfvCucipEnTcQ.png" /></figure><p><strong>Step 7 — Reach a decision.</strong> Since p ≈ 0.018 &lt; α = 0.05, the null hypothesis is <strong>rejected</strong>.</p><p><strong>Interpretation.</strong> If the lattes genuinely averaged 150 mg, a sample mean as high as the one observed would arise only about 1.8% of the time. That is rare enough — below the pre-set 5% threshold — to conclude that the true mean most likely exceeds 150 mg. Equivalently, a one-sided 95% confidence bound places the true mean above roughly 151.8 mg.</p><h4><strong>When the normality assumption does not hold:</strong></h4><p>Several established alternatives exist, chosen according to how the assumption is violated:</p><ul><li><strong>Nonparametric tests</strong>, which make no distributional assumption and operate on the <em>ranks</em> of the data rather than their raw values. The one-sample <strong>Wilcoxon signed-rank test</strong> is the direct counterpart to the one-sample t-test; the <strong>Mann–Whitney U test</strong> replaces the two-sample t-test, and the <strong>Kruskal–Wallis test</strong> replaces one-way ANOVA. These are the standard remedies for skewed data or ordinal measurements, at the cost of some statistical power when the data <em>are</em> in fact normal.</li><li><strong>Resampling methods</strong>, chiefly the <strong>bootstrap</strong> and <strong>permutation (randomization) tests</strong>, build the reference distribution directly from the observed data instead of assuming a theoretical one. They are flexible, intuitive, and well-suited to awkward statistics or modest samples, given adequate computing.</li><li><strong>Transformations</strong>, such as taking logarithms of right-skewed positive data, can render the distribution approximately normal so that the ordinary t-test again applies.</li><li><strong>A larger sample</strong>, which is not a test but a genuine fix: by the Central Limit Theorem, the sampling distribution of the mean approaches normality as n grows, so moderate non-normality matters far less when n is large.</li></ul><p>A practical workflow is to inspect the data first — a histogram or a Q–Q plot, optionally with a formal check such as the <strong>Shapiro–Wilk test</strong> — and reach for a nonparametric or resampling method only when normality is clearly untenable.</p><h3>6. Visualising the test</h3><p>The figure below shows the t-distribution with 24 degrees of freedom. The red region is the upper 5% tail — the rejection region — and the dashed line marks the observed statistic, t = 2.22. Its position beyond the critical value at 1.71 is the visual expression of rejecting H₀.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ju2EEqOuNVIc4qDpasmWg.png" /></figure><h3>7. Why 5%, and what changes when the threshold is tightened</h3><p>The choice of α = 0.05 carries an air of natural law, yet it is purely a convention. Its prominence traces to Ronald Fisher, who in the 1920s described a one-in-twenty result as a convenient practical marker of an outcome worth taking seriously. The value was never derived from a deeper principle; it endured because it was simple, round, and widely copied.</p><p>The significance level functions as a lever, and moving it has consequences. The level α <em>defines</em> the size of the rejection region — the tail area beyond the critical value. Lowering α drives the critical value further into the tail, shrinking the rejection region and demanding stronger evidence before H₀ can be rejected. The animation below holds the caffeine data fixed (the statistic stays at t = 2.22) while α is tightened from 5% toward 1%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*HRyVxzxDP1t-UA7dqhtRSQ.gif" /></figure><p>The lesson becomes concrete at the instant the critical line overtakes the observed statistic. At α = 5%, the critical value is 1.71, and the null is rejected. At α = 3%, it rises to 1.97 — and the null is <em>still</em> rejected, since 2.22 remains beyond it. But at α = 1%, the critical value climbs to 2.49, now to the right of the statistic, and the identical data deliver the opposite verdict: fail to reject. The decision reverses not because the evidence changed, but because the standard of evidence did. The tipping point falls precisely where α equals the p-value — here, near 1.8%.</p><h3>8. The two ways a test can err</h3><p>Because a verdict is rendered under uncertainty, a test can be wrong in two distinct ways.</p><p>A <strong>Type I error</strong> is a <em>false positive</em>: H₀ is rejected when it was in fact true, convicting an innocent defendant. Its probability is exactly α. A <strong>Type II error</strong> is a <em>false negative</em>: H₀ is retained when it was in fact false, acquitting a guilty one. Its probability is denoted β, and the complement 1 − β is the test’s <strong>power</strong>, the probability of correctly detecting a real effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PxuCHjta8yExhbe_WwjROA.png" /></figure><p>This is why tightening α is not a free improvement: reducing the Type I rate inevitably enlarges β and lowers power. It also explains a structural advantage of the one-tailed test — by concentrating the entire error budget in the predicted direction, it achieves <em>higher power</em> than a two-tailed test for the same sample size, whenever the effect truly lies in that direction.</p><p>9. The same test in Python</p><pre>import numpy as np<br>from scipy import stats<br><br>mu_0, x_bar, s, n = 150, 158, 18, 25   # null value, sample mean, sample sd, sample size<br>alpha = 0.05<br><br>se = s / np.sqrt(n)<br>t_stat = (x_bar - mu_0) / se<br>df = n - 1<br><br># ONE-tailed (upper) p-value: the upper-tail area, NOT doubled<br>p_value = stats.t.sf(t_stat, df)<br>t_crit  = stats.t.ppf(1 - alpha, df)<br><br>print(f&quot;t-statistic       : {t_stat:.4f}&quot;)<br>print(f&quot;Critical value    : {t_crit:.4f}&quot;)<br>print(f&quot;p-value (one-tail): {p_value:.4f}&quot;)<br>print(&quot;Reject H0&quot; if p_value &lt; alpha else &quot;Fail to reject H0&quot;)</pre><p>t-statistic: 2.2222<br>Critical value: 1.7109<br>p-value (one-tailed): 0.0180<br>Reject H0</p><p>With the raw measurements rather than summary statistics, the one-sided test is a single call — the alternative=&#39;greater&#39; argument is what makes it one-tailed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUL4wm5fvl6fYROqFOZ0EQ.png" /></figure><h3>References</h3><ul><li>Khan Academy — <em>Significance tests (hypothesis testing)</em>, in the AP®/College Statistics course, and the video <em>One-tailed and two-tailed tests</em> (Inferential Statistics). Freely available at khanacademy.org; widely regarded as among the clearest introductory lectures on the subject.</li><li>Fisher, R. A. (1925). <em>Statistical Methods for Research Workers.</em> Oliver &amp; Boyd. (The origin of the 0.05 convention.)</li><li>Wasserstein, R. L., &amp; Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. <em>The American Statistician</em>, 70(2), 129–133.</li><li>Diez, D., Çetinkaya-Rundel, M., &amp; Barr, C. <em>OpenIntro Statistics</em> (free textbook, openintro.org).</li><li>Moore, D. S., McCabe, G. P., &amp; Craig, B. A. <em>Introduction to the Practice of Statistics.</em> W. H. Freeman.</li></ul><h4><strong>Part II of the series, on the two-tailed test, is available </strong><a href="https://medium.com/@debisreer/statistical-hypothesis-testing-part-ii-the-two-tailed-test-0d207080cd09"><strong>here</strong></a><strong>:</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=807b21af0456" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-and-beyond/statistical-hypothesis-testing-part-i-the-one-tailed-test-807b21af0456">Statistical Hypothesis Testing -Part I: The One-Tailed Test</a> was originally published in <a href="https://medium.com/data-and-beyond">Data And Beyond</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linux 101: Get Started with Linux]]></title>
            <link>https://medium.com/codetodeploy/linux-101-get-started-with-linux-7590dd502903?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/7590dd502903</guid>
            <category><![CDATA[linuxworld]]></category>
            <category><![CDATA[linux-commands]]></category>
            <category><![CDATA[linux-os]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 05:42:57 GMT</pubDate>
            <atom:updated>2026-04-10T11:42:23.492Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>From Open-Source Foundations to Computational Control: Why Linux Dominates Modern Scientific Computing</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9rqrzPzS6sd-4W5cDNyy2Q.gif" /></figure><blockquote><strong><em>🚨 HIRING: Tech Professionals</em></strong></blockquote><blockquote>💰 $3K–$10K/Month<br> 🌍 Remote + 🏢 Onsite Opportunities</blockquote><blockquote>🎯 Open Roles:<br> Backend • Frontend • Full Stack • DevOps • Data Engineer • UI/UX • QA<br> <a href="https://optimhire.com/?ref_code=codetodeploy"><strong>👉 Apply Now</strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><p>In modern scientific and technological practice, the operating system is far more than an interface layer; it is a foundational component that shapes computational reliability, performance, and reproducibility. Among operating systems, Linux occupies a uniquely dominant position across high-performance computing (HPC), cloud infrastructure, data science platforms, and embedded systems.</p><p>Unlike consumer-oriented operating systems that prioritize abstraction and ease of use, Linux is designed around transparency, composability, and explicit control over system behavior. These properties are not incidental; they arise from its Unix heritage and its open-source development model. For scientists and engineers, this translates into an environment where computational processes are not concealed but exposed, inspectable, and controllable.</p><h3>Historical Foundations and Design Philosophy</h3><p>Linux traces its lineage to the Unix operating system, developed at AT&amp;T Bell Labs in the early 1970s by Ken Thompson and Dennis Ritchie. Unix introduced several foundational ideas that remain central to modern computing: treating virtually everything as a file, designing small and composable tools that each do one thing well, and enabling multi-user, multitasking environments.</p><p>In the 1980s, Richard Stallman launched the GNU Project with the goal of creating a fully free Unix-like operating system. The project produced critical components: compilers (GCC), text editors (Emacs), and core utilities (coreutils), but lacked a production-ready kernel, the core component responsible for managing hardware and system resources.</p><p>In 1991, Linus Torvalds, then a student at the University of Helsinki, released the first version of the Linux kernel. When combined with the existing GNU user-space tools, it formed a complete, freely available operating system commonly referred to as GNU/Linux.</p><p>This system embodies several key principles:</p><ul><li><strong>Open-source collaboration.</strong> The source code is publicly available, enabling peer review, modification, and redistribution.</li><li><strong>Modularity and extensibility.</strong> System components are designed to be independently replaceable and configurable.</li><li><strong>Hardware abstraction with minimal overhead.</strong> The kernel provides efficient access to hardware without unnecessary layers of indirection.</li></ul><p>These characteristics make Linux particularly well-suited for scientific and engineering applications, where system behavior must be predictable, auditable, and reproducible.</p><h3>Linux vs. Windows: Architectural and Philosophical Differences</h3><h3>Kernel Design</h3><p>Linux employs a <strong>monolithic kernel</strong> with loadable module support. In this architecture, core services such as process scheduling, memory management, and device drivers operate within a single address space, enabling efficient inter-component communication with minimal context-switching overhead. Modules can be loaded and unloaded at runtime, providing flexibility without sacrificing performance.</p><p>Windows uses a <strong>hybrid kernel</strong> that combines elements of monolithic and microkernel designs. This introduces additional abstraction layers that are optimized for broad hardware compatibility and general-purpose usability, but at the cost of reduced transparency and somewhat greater overhead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IqWiDsR1MQvl2bLCR-Sn6w.png" /></figure><p>Linux prioritizes explicit user control and configurability, whereas Windows emphasizes convenience and a consistent graphical experience.</p><h3>Software Distribution</h3><p>Linux systems use <strong>package managers</strong> (such as APT on Debian-based distributions, DNF on Fedora, and Pacman on Arch Linux) that resolve dependencies automatically, verify software integrity through cryptographic signatures, and maintain version consistency across the system.</p><p>Windows has traditionally relied on standalone installers, though the introduction of tools like Windows Package Manager (winget) has begun to close this gap. Nonetheless, Linux’s package management ecosystem remains more mature and more tightly integrated into routine system administration.</p><h3>Reproducibility and Automation</h3><p>Linux excels in environments that demand reproducibility. Shell scripting, pipeline composition, and containerization technologies such as Docker and Kubernetes are native to the Linux ecosystem. These capabilities are essential for scientific workflows, machine learning pipelines, and distributed computing systems where identical environments must be reliably recreated across machines and over time.</p><h3>Internal Structure of Linux:</h3><h3>Kernel Space and User Space</h3><p>Linux enforces a strict separation between two operational domains:</p><ul><li><strong>Kernel space</strong> handles hardware control, memory management, process scheduling, and device drivers. Code executing here has unrestricted access to system resources.</li><li><strong>User space</strong> encompasses applications, shells, and user-facing utilities. Programs in user space cannot directly access hardware or kernel memory.</li></ul><p>Communication between these domains occurs through <strong>system calls, </strong>a controlled interface that ensures user-space programs interact with hardware only through well-defined, permission-checked pathways.</p><h3>The Shell</h3><p>The shell (commonly Bash or Zsh) serves as a command-line interpreter, mediating between the user and the kernel:</p><pre>User → Shell → Kernel → Hardware</pre><p>Beyond simple command execution, the shell enables scripting, process control, and the composition of complex operations from simple building blocks — a paradigm directly inherited from Unix.</p><h3>File System Hierarchy</h3><p>Linux enforces a standardized directory structure defined by the <strong>Filesystem Hierarchy Standard (FHS)</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vg11vF_KScqBN3Cufd8tFw.png" /></figure><p>A key abstraction in Linux is that devices, processes, and even kernel parameters are represented as files. This uniform interface simplifies system interaction and enables powerful composition of tools.</p><h3>The Linux Security Model</h3><p>Security in Linux is not an afterthought: it is embedded into the architecture at multiple levels.</p><h3>Principle of Least Privilege</h3><p>Users operate with the minimum permissions necessary. Administrative actions require explicit privilege escalation via sudo, ensuring that routine operations cannot accidentally or maliciously alter critical system components.</p><h3>File Permissions</h3><p>Every file and directory carries a permission set governing access for three categories: the owning <strong>user</strong>, the owning <strong>group</strong>, and <strong>others</strong>. Permissions are expressed as read (r), write (w), and execute (x) flags. For example:</p><pre>-rwxr-xr--</pre><p>This notation indicates that the owner has full permissions, group members can read and execute, and all other users can only read.</p><h3>Open-Source Transparency</h3><p>Because the Linux kernel and most associated tools are open source, vulnerabilities are subject to continuous public scrutiny. This transparency enables rapid identification and resolution of security issues through community-driven review and patching.</p><h3>Trusted Package Management</h3><p>Software is installed from curated, cryptographically signed repositories, significantly reducing the risk of supply-chain attacks and the installation of malicious software.</p><h3>Mandatory Access Control</h3><p>Linux supports advanced security frameworks such as <strong>SELinux</strong> (Security-Enhanced Linux, originally developed by the NSA) and <strong>AppArmor</strong>. These systems enforce policy-based access control that goes beyond traditional discretionary file permissions, and they are widely deployed in enterprise and cloud environments.</p><h3>Process Isolation</h3><p>Linux provides robust process isolation, ensuring that a failure or compromise in one process does not propagate to others. This property is foundational to system stability and is further extended by modern containerization technologies.</p><h3><strong>Essential Linux Commands</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Th-znGJl7NqbIbpXIqIAg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fcQINNrZ-lmKrvVIgPPSoQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AktyAhHVc2O2HTNZCADuYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gdMMZzZWfrAsqivprDy71g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XwV1FVLSI2yYNJm8GCOyCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GbH5Z3q3lgmEtJc0iiCCnQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dyIZ8xY65nG_m7hB_TMzTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ta-6POcYVeEJ2_DYZWHf5Q.png" /></figure><h3>Why Linux Is the Preferred Platform for Scientists and Engineers</h3><p>Linux provides the properties that scientific and engineering work demands:</p><ul><li><strong>Deterministic environments</strong> enable the reproducibility of experiments and computational results.</li><li><strong>Scalability</strong> allows the same workflows to run on a laptop, a departmental server, or a supercomputer. As of 2024, all 500 systems on the TOP500 list of the world’s most powerful supercomputers run Linux.</li><li><strong>Transparency</strong> offers full visibility into system behavior, from kernel parameters to network configuration.</li><li><strong>Automation</strong> through shell scripting and pipeline composition eliminates manual intervention and reduces human error.</li><li><strong>Security</strong> is enforced through layered, well-audited mechanisms that protect sensitive research data and infrastructure.</li></ul><p>These properties align directly with the core requirements of scientific computing: precision, control, and reproducibility.</p><h3>Conclusion</h3><p>Linux is not designed to obscure complexity; it is designed to expose it in a structured and controllable way. For scientists and engineers, this is not a drawback but a necessity.</p><p>Mastering Linux is less about memorizing commands and more about internalizing a paradigm: that complex computational systems can be constructed by composing simple, well-defined operations. Once this principle is understood, Linux becomes more than an operating system; it becomes an extension of analytical thinking itself.</p><p>📄 <a href="https://github.com/debisree/Linux-101-Get-Started-with-Linux/raw/main/linux_cheatsheet.pdf"><strong>Download the Linux Command Cheat Sheet</strong></a>: A printable two-page reference covering all essential commands discussed in this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7590dd502903" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/linux-101-get-started-with-linux-7590dd502903">Linux 101: Get Started with Linux</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Outliers Through Boxplots]]></title>
            <link>https://medium.com/codetodeploy/outliers-in-data-understanding-the-iqr-method-0341463f86a7?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/0341463f86a7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[outlier-detection]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[box-plot]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 17:03:41 GMT</pubDate>
            <atom:updated>2026-03-14T02:58:31.806Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How boxplots visually reveal unusual observations using the Interquartile Range (IQR)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RmE-BtRiMXxW2xrRrv6evg.png" /></figure><blockquote>🔥 <strong>Top Tech Jobs Are Hiring NOW — Don’t Miss Out.</strong></blockquote><blockquote>🚀 <strong>Multiple Roles Available<br> </strong><a href="https://app.usebraintrust.com/r/codeto1/"><strong>👉 Apply &amp; Secure Your Job</strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><h3>Introduction</h3><p>In statistical analysis and machine learning, the presence of <strong>outliers, </strong>observations that deviate substantially from the majority of the data, poses both a challenge and an opportunity. From a statistical perspective, outliers can distort summary statistics, influence parameter estimation, and bias predictive models. From a practical standpoint, however, these extreme observations may represent meaningful phenomena such as rare events, system failures, experimental anomalies, or valuable signals embedded in noisy data.</p><p>Consequently, the goal of outlier detection is not merely to identify and remove unusual observations. Rather, it is to <strong>understand whether the deviation reflects noise, error, or genuine information</strong>. In applied domains such as finance, manufacturing, healthcare analytics, and behavioral data science, this distinction becomes particularly important. Removing valid extreme observations can lead to systematic bias, whereas failing to detect erroneous ones may degrade model reliability and operational decision-making.</p><p>Several statistical and algorithmic techniques have been developed to identify outliers. These range from simple rule-based approaches and parametric methods such as z-score thresholds to more sophisticated algorithms, including density-based anomaly detection and ensemble isolation techniques. Despite the development of these advanced methods, one of the most widely used and interpretable approaches remains the <strong>Interquartile Range (IQR) method</strong>, originally popularized through Tukey’s boxplot framework.</p><p>The IQR method is attractive for several reasons. It is <strong>non-parametric</strong>, meaning it does not assume any specific distributional form for the data. It is also <strong>robust to extreme values</strong>, as it relies on quartiles rather than mean-based statistics that are sensitive to skewed observations. Because of its simplicity and interpretability, the IQR approach is often used as a first diagnostic step in exploratory data analysis.</p><h3>Understanding the Boxplot</h3><p>A <strong>boxplot (box-and-whisker plot)</strong> provides a compact visual summary of a numerical distribution. It highlights key aspects of the data, such as <strong>central tendency, variability, and potential outliers,</strong> in a single visualization. Because of its simplicity and interpretability, the boxplot is widely used in exploratory data analysis and statistical reporting.</p><p>A boxplot is constructed from the <strong>five-number summary</strong> of the data: the minimum, first quartile (Q1), median (Q2), third quartile (Q3), and maximum. The following components define the structure of a boxplot (<strong>refer to the schematic diagram above</strong>).</p><p>A boxplot is a compact visual summary of a distribution. It displays the center, spread, and possible outliers of a numeric data. The following are the main components of a boxplot (ref: the picture above).</p><h4>1. The Box</h4><p>The rectangular <strong>box spans from the first quartile (Q1) to the third quartile (Q3)</strong>. This region represents the <strong>interquartile range (IQR)</strong>, which contains the <strong>middle 50% of the observations</strong> in the dataset. The size of the box provides a visual indication of the variability of the central portion of the data (<strong>see the schematic diagram above</strong>).</p><h4>2. The Median Line</h4><p>Inside the box, a horizontal line indicates the <strong>median (Q2)</strong> of the distribution. The median represents the <strong>50th percentile</strong>, meaning half of the data points lie below it and half lie above it. This line is a robust measure of central tendency because it is less sensitive to extreme values than the mean (<strong>refer to the schematic diagram above</strong>).</p><h4>3. The Whiskers</h4><p>The <strong>whiskers extend outward from the box</strong> toward the most extreme observations that still fall within the acceptable range defined by the <strong>IQR rule</strong>. Specifically, whiskers extend to the largest and smallest values within:</p><p>Q1−1.5×IQR andQ3+1.5×IQR</p><p>Importantly, whiskers <strong>do not necessarily reach the absolute minimum and maximum values</strong> of the dataset. Any observations that fall outside these boundaries are typically plotted individually as <strong>outliers</strong>, often displayed as isolated points beyond the whiskers (<strong>see the schematic diagram above</strong>).</p><h4><strong>4. Individual points beyond the whiskers</strong></h4><p>These points are plotted separately and are identified as <strong>potential outliers</strong> using the IQR rule.</p><h3>How outliers are detected from a boxplot: The Interquartile Range (IQR) Method</h3><p>The IQR method is based on <strong>order statistics</strong>, which describe the relative position of observations in a sorted dataset.</p><p>Given a numerical variable, the distribution can be summarized using three quartiles:</p><ul><li><strong>First quartile (Q1):</strong> the 25th percentile</li><li><strong>Second quartile (Q2):</strong> the median (50th percentile)</li><li><strong>Third quartile (Q3):</strong> the 75th percentile</li></ul><p>The <strong>Interquartile Range</strong> is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*Ipd6FJFdkTluD3WVxeVLLQ.png" /></figure><p>The IQR measures the spread of the <strong>central 50% of the data</strong>, making it resistant to the influence of extreme observations.</p><p>To identify outliers, Tukey proposed defining “fences” beyond which observations are considered unusual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*opJAc1o8Eex-V2VJiOjsbg.png" /></figure><p>Any observation lying outside these bounds is classified as a potential outlier. Some practitioners also define <strong>extreme outliers</strong> using a more conservative threshold:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*1vEiyo4Ryf_NaG5GZis4oA.png" /></figure><h3>Relationship Between IQR and Boxplots</h3><p>The IQR method is most commonly visualized through a <strong>boxplot</strong>.</p><p>In a boxplot:</p><ul><li>The box spans from Q1 to Q3.</li><li>The line inside the box represents the median.</li><li>The “whiskers” extend to the most extreme values within the IQR fences.</li><li>Points beyond the whiskers are plotted individually as outliers.</li></ul><p>This graphical representation provides an immediate visual summary of the distribution, highlighting skewness, dispersion, and potential anomalies.</p><h3>Advantages of the IQR Method</h3><h4>Robustness</h4><p>Because quartiles are based on rank ordering rather than magnitude, the IQR method is far less sensitive to extreme values than mean-based statistics.</p><h4>Distribution Independence</h4><p>Unlike z-score methods, the IQR technique does not assume that the underlying data follows a normal distribution.</p><h4>Interpretability</h4><p>The method provides clear, deterministic thresholds that can easily be communicated to stakeholders.</p><h4>Computational Simplicity</h4><p>IQR calculations are straightforward and computationally inexpensive, making them suitable for large datasets and real-time monitoring systems.</p><h3>Python Implementation of IQR Outlier Detection</h3><h4>Computing IQR thresholds</h4><pre>import pandas as pd<br><br>def compute_iqr_bounds(series, k=1.5):<br>    s = series.dropna()<br><br>    q1 = s.quantile(0.25)<br>    q3 = s.quantile(0.75)<br><br>    iqr = q3 - q1<br><br>    lower = q1 - k * iqr<br>    upper = q3 + k * iqr<br><br>    return lower, upper</pre><p>Flagging outliers in a dataset</p><pre>def detect_iqr_outliers(df, column, k=1.5):<br><br>    lower, upper = compute_iqr_bounds(df[column], k)<br><br>    df_out = df.copy()<br><br>    df_out[f&quot;{column}_is_outlier&quot;] = (<br>        (df_out[column] &lt; lower) |<br>        (df_out[column] &gt; upper)<br>    )<br><br>    return df_out, lower, upper</pre><p>Example usage:</p><pre>df = pd.DataFrame({<br>    &quot;value&quot;: [10, 12, 11, 13, 10, 9, 12, 11, 200]<br>})<br><br>df_out, lower, upper = detect_iqr_outliers(df, &quot;value&quot;)<br><br>print(&quot;Lower fence:&quot;, lower)<br>print(&quot;Upper fence:&quot;, upper)<br><br>print(df_out[df_out[&quot;value_is_outlier&quot;]])</pre><h3><strong>Important Nuances When Applying IQR</strong></h3><p><strong>Skewed Distributions</strong></p><p>In highly skewed data, the IQR rule may flag many observations in the long tail even though they are legitimate values.</p><p>In such cases, practitioners often:</p><ul><li>Apply logarithmic transformations</li><li>Use asymmetric thresholds</li><li>Analyze distributions within subgroups</li></ul><p><strong>Multi-Modal Data</strong></p><p>If the dataset contains multiple subpopulations, a global IQR may obscure local anomalies.</p><p>For example:</p><ul><li>Multiple machine operating modes</li><li>Different product grades</li><li>Customer segments</li></ul><p>In these situations, it is preferable to compute IQR <strong>within homogeneous groups</strong>.</p><h4>Small Sample Sizes</h4><p>With small datasets, quartile estimates become unstable. This may lead to unreliable fence thresholds. In such cases, domain knowledge or alternative robust statistics may be more appropriate.</p><h4>What Should Be Done After Outliers Are Detected?</h4><p>Identifying outliers is only the first step. The appropriate response depends heavily on the context of the data.</p><h4>Scenario 1: Data Errors</h4><p>Outliers may result from:</p><ul><li>Measurement errors</li><li>Data entry mistakes</li><li>Sensor malfunctions</li><li>Data pipeline failures</li></ul><p>In these cases, removing or correcting the observations is appropriate.</p><p>However, such actions should always be <strong>documented and reproducible</strong>.</p><h4>Scenario 2: Valid but Rare Observations</h4><p>In many domains, extreme values represent legitimate phenomena.</p><p>Examples include:</p><ul><li>High-value transactions in finance</li><li>Rare production conditions in manufacturing</li><li>Exceptional performance metrics</li></ul><p>Removing such values may introduce bias and reduce model generalizability. Instead, they should typically be retained.</p><h4>Scenario 3: Model Sensitivity</h4><p>Some statistical models, particularly linear regression models, are sensitive to extreme observations.</p><p>Possible strategies include:</p><ul><li>Winsorization (clipping values to boundary thresholds)</li><li>Transformations such as logarithms</li><li>Robust regression techniques</li><li>Tree-based models that are less sensitive to extreme values</li></ul><h4>Scenario 4: Anomaly Detection as the Objective</h4><p>In some applications, outliers are the very events we aim to detect.</p><p>Examples include:</p><ul><li>Fraud detection</li><li>Equipment failure prediction</li><li>Cybersecurity monitoring</li></ul><p>In these cases, outliers should be preserved and studied rather than removed.</p><h3>A Practical Workflow for IQR-Based Outlier Handling</h3><p>A structured workflow for handling outliers may involve the following steps:</p><ol><li>Conduct exploratory data analysis</li><li>Compute IQR thresholds for relevant variables</li><li>Flag observations exceeding the thresholds</li><li>Investigate potential causes</li><li>Decide on appropriate handling strategies</li><li>Document all decisions and transformations</li></ol><p>This approach ensures transparency and reproducibility in analytical workflows.</p><h3>Conclusion</h3><p>The Interquartile Range method provides a simple yet powerful framework for identifying potential outliers in a dataset. Its robustness, interpretability, and distribution-free nature make it a valuable tool during exploratory analysis and data quality assessment.</p><p>However, detecting outliers should not automatically lead to their removal. The interpretation and handling of these observations require careful consideration of the underlying data-generating process and the objectives of the analysis.</p><p>Ultimately, effective outlier management lies not in rigidly applying statistical rules, but in combining <strong>statistical insight with domain knowledge and thoughtful decision-making</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0341463f86a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/outliers-in-data-understanding-the-iqr-method-0341463f86a7">Understanding Outliers Through Boxplots</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Encoding in Machine Learning]]></title>
            <link>https://medium.com/@debisreer/encoding-in-machine-learning-8e7b11857daa?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e7b11857daa</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[encoding]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 07:22:21 GMT</pubDate>
            <atom:updated>2026-07-12T15:52:41.976Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A Data Pre-processing Technique for Translating Categorical Semantics into Mathematical Structure</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y9jsdYozJjz0qyrH0FIR5w.png" /></figure><p>In statistical modeling and machine learning, encoding is explicitly recognized as a <strong>data preprocessing technique</strong>. It belongs to the broader pipeline of data preparation steps, alongside cleaning, scaling, normalization, and feature engineering, that transform raw data into a model-ready format.</p><p>Real-world datasets frequently contain categorical labels, qualitative groupings, identifiers, and structured codes. These representations carry domain meaning but lack numerical form. Since most machine learning algorithms operate on numerical matrices, categorical variables must be transformed before modeling. This transformation process is known as <strong>encoding</strong>, and it is a foundational step in data preprocessing.</p><p>While often introduced as a routine task, encoding is more accurately understood as a modeling decision. The method chosen determines how categorical semantics are embedded into mathematical structure, shaping both the geometry of the feature space and the behavior of the learning algorithm.</p><h3>Encoding as a Data Preprocessing Step</h3><p>In the machine learning workflow, preprocessing typically includes:</p><ul><li>Handling missing values</li><li>Removing outliers</li><li>Scaling numerical features</li><li><strong>Encoding categorical variables</strong></li></ul><p>Encoding specifically addresses the incompatibility between categorical data and algorithms such as Linear Regression, Logistic Regression, Random Forest, and XGBoost.</p><p>These models rely on operations rooted in linear algebra, optimization theory, and impurity measures. Since such operations require numeric input, categorical variables must first be transformed into a numerical representation. Thus, encoding is structurally required in most traditional supervised learning pipelines.</p><p>Some modern gradient boosting frameworks (e.g., XGBoost) support <strong>native categorical handling</strong> when categorical columns are properly typed (e.g., pandas category) and enable_categorical=True. This can reduce the need for explicit one-hot encoding.</p><p>In such cases, the model learns the optimal partitioning of categorical splits internally, without requiring explicit one-hot encoding. That being said:</p><ul><li>Native categorical support is model-specific.</li><li>It may require specific configuration.</li><li>It does not eliminate the conceptual importance of representation choices.</li><li>In many production pipelines, explicit encoding remains preferred for transparency and reproducibility.</li></ul><p>Thus, while encoding is not universally mandatory in modern toolkits, it remains foundational in most modeling workflows.</p><h3>Major Encoding Techniques</h3><h3>1. Label Encoding</h3><p>Label encoding assigns a unique integer to each category.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ATNeJaR8KtE-8kvPzJHDA.png" /></figure><p>The Python code:</p><pre>from sklearn.preprocessing import LabelEncoder<br>import pandas as pd<br><br>df = pd.DataFrame({<br>    &quot;Category&quot;: [&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;A&quot;]<br>})<br><br>le = LabelEncoder()<br>df[&quot;Category_encoded&quot;] = le.fit_transform(df[&quot;Category&quot;])<br><br>print(df)</pre><p>Label encoding is the simplest possible mapping from symbolic categories to numeric values. It preserves uniqueness while minimizing dimensionality. It answers the question:</p><blockquote><em>“How can I convert categories into numbers without expanding the feature space?”</em></blockquote><p>By assigning integers, we implicitly place categories along a single numeric axis. Even if no order exists conceptually, the numbers impose one mathematically:</p><pre>0 &lt; 1 &lt; 2</pre><p>For linear models, this introduces artificial ordinal structure.</p><h3>When to Use</h3><ul><li>Label encoding is often acceptable for tree ensembles, but it can still impose an artificial order if the implementation treats the codes as ordered and splits by thresholds. If your tool supports native categorical splits, prefer that; otherwise, consider one-hot / target encoding depending on cardinality.</li><li>When dimensionality must remain small</li><li>When the category count is manageable</li></ul><p>Avoid linear or distance-based models unless categories are truly ordinal.</p><h3>2. One-Hot Encoding</h3><p>One-hot encoding creates a binary column for each category.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ATNeJaR8KtE-8kvPzJHDA.png" /></figure><p>Python code snippet:</p><pre>df = pd.get_dummies(df, columns=[&quot;Category&quot;], drop_first=False)<br>print(df)</pre><p>One-hot encoding solves the artificial ordering problem of label encoding. Instead of placing categories along a single axis, it assigns each category its own independent axis.</p><p>This ensures that:</p><ul><li>No category is “greater” than another.</li><li>One-hot prevents imposing an arbitrary <strong>ordering</strong> among categories. However, the notion of ‘distance’ depends on the metric: under Euclidean distance, any two distinct one-hot categories are equally far apart; under other metrics, behavior differs.</li><li>Linear models can assign independent coefficients. For linear models with an intercept, one-hot often drops one level (drop_first=True) to avoid perfect multicollinearity; regularized models can sometimes tolerate full rank deficiency.</li></ul><p>The feature space expands. Suppose there are 50 categories; 50 new dimensions are created. This increases interpretability but also increases sparsity.</p><h3>When to Use</h3><ul><li>Nominal categorical variables</li><li>Linear models</li><li>Logistic regression</li><li>Neural networks</li></ul><p>Avoid when the number of categories is very large.</p><h3>3. Ordinal Encoding</h3><p>Ordinal encoding assigns integers that reflect true order.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*A9XGwOretlXZg5zfx8wt_w.png" /></figure><p>Python code:</p><pre>from sklearn.preprocessing import OrdinalEncoder<br><br>df = pd.DataFrame({<br>    &quot;RiskLevel&quot;: [&quot;Low&quot;, &quot;Medium&quot;, &quot;High&quot;]<br>})<br><br>encoder = OrdinalEncoder(categories=[[&quot;Low&quot;, &quot;Medium&quot;, &quot;High&quot;]])<br>df[&quot;Risk_encoded&quot;] = encoder.fit_transform(df[[&quot;RiskLevel&quot;]])<br><br>print(df)</pre><p>Some categorical variables possess intrinsic ranking, and ignoring that order can discard meaningful structural information. <strong>Ordinal encoding</strong> is designed to preserve this monotonic structure while maintaining low dimensionality. Unlike label encoding, where any imposed order may be arbitrary, the ordering in ordinal encoding is deliberate and grounded in domain knowledge.</p><p>When encoded numerically, the model can interpret larger values as representing “higher” levels in the hierarchy. However, this representation introduces an implicit assumption: the encoded levels are treated as evenly spaced along a numerical scale. In many models, especially linear ones, differences are interpreted arithmetically (for example, the step from 1 to 2 is treated as equivalent to the step from 2 to 3).</p><p>In practice, real-world ordinal categories may not reflect such uniform spacing. The conceptual distance between “Medium” and “High” may not equal that between “Low” and “Medium.” Therefore, while ordinal encoding preserves order, it may still impose a linear structure that does not fully align with the underlying phenomenon.</p><h3>When to Use</h3><ul><li>True ordinal variables</li><li>When rank information is meaningful</li><li>When dimensionality should remain low</li></ul><p>Never use ordinal encoding for purely nominal data.</p><h3>4. Target Encoding</h3><p>Target encoding replaces each category with a statistic derived from the target variable (often the mean).</p><p>Python code:</p><pre>df = pd.DataFrame({<br>    &quot;Category&quot;: [&quot;A&quot;, &quot;B&quot;, &quot;A&quot;, &quot;C&quot;],<br>    &quot;Target&quot;: [1.2, 0.8, 1.1, 1.5]<br>})<br><br>mean_map = df.groupby(&quot;Category&quot;)[&quot;Target&quot;].mean()<br>df[&quot;Category_target_enc&quot;] = df[&quot;Category&quot;].map(mean_map)<br><br>print(df)</pre><p>When the category count is high, one-hot encoding becomes impractical. Target encoding compresses information by embedding categories into a statistically informed continuum. Instead of representing identity, it represents predictive power. This method introduces supervised information into features. Categories become numbers reflecting their association with the outcome.</p><p>The danger is leakage: if encoding uses full dataset statistics, the model may “peek” at the answer.</p><h3>When to Use</h3><ul><li>High-cardinality categorical variables</li><li>Large datasets</li><li>Tree-based models</li><li>When predictive performance outweighs interpretability</li></ul><p>Compute encodings using out-of-fold estimates (and apply smoothing/regularization) to reduce variance for rare categories.</p><h3>5. Frequency Encoding</h3><p>Frequency encoding replaces each category with its occurrence rate in the dataset.</p><pre>freq_map = df[&quot;Category&quot;].value_counts(normalize=True)<br>df[&quot;Category_freq&quot;] = df[&quot;Category&quot;].map(freq_map)</pre><p>Frequency often carries a signal. Rare categories may behave differently from common ones. This method preserves distributional information while avoiding dimensional expansion.</p><h3>When to Use</h3><ul><li>Tree-based models</li><li>Baseline models</li><li>Memory-constrained systems</li></ul><p>Not ideal when interpretability of individual categories is required.</p><h3>Encoding as Representation Design</h3><p>Each encoding answers a different representational question:</p><ul><li>Label encoding: How do we map identity minimally?</li><li>One-hot encoding: How do we preserve neutrality?</li><li>Ordinal encoding: How do we preserve rank?</li><li>Target encoding: How do we preserve predictive association?</li><li>Frequency encoding: How do we preserve distributional prevalence?</li></ul><p>Thus, encoding is not merely technical preprocessing. It defines how categorical semantics are embedded into numerical space.</p><p>Even with modern tools like XGBoost supporting native categorical handling, understanding encoding remains essential. Model defaults do not replace conceptual clarity.</p><h3>Concluding Thought</h3><p>Encoding is a foundational data preprocessing technique. But more importantly, it is an act of abstraction. It formalizes qualitative distinctions into quantitative representation. The choice of encoding determines how categories are geometrically arranged, statistically interpreted, and ultimately learned by the model. In rigorous machine learning practice, encoding is not just about converting strings to numbers. It is about deciding what structure we allow our models to perceive.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e7b11857daa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Permutation Importance as a Performance-Based Measure of Feature Contribution]]></title>
            <link>https://medium.com/codetodeploy/permutation-importance-as-a-performance-based-measure-of-feature-contribution-c5da64e1f16e?source=rss-dbc85e29e884------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5da64e1f16e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[permutation-importance]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[modeling]]></category>
            <dc:creator><![CDATA[Debisree Ray]]></dc:creator>
            <pubDate>Wed, 28 Jan 2026 01:28:55 GMT</pubDate>
            <atom:updated>2026-02-06T17:50:51.884Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>From resampling theory to model-agnostic interpretability in machine learning</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJ8JgtqxJqImUmmk1DNORg.png" /></figure><blockquote>🚀 Master In-Demand Tech Skills &amp; Boost Your Career</blockquote><blockquote>— <a href="https://www.dpbolvw.net/click-101641947-17135603">Start Your Free Trial Today!</a></blockquote><p>Contemporary machine learning models achieve a good predictive performance across a wide range of scientific and industrial domains. However, this success often comes at the cost of interpretability. As models increase in complexity — progressing from linear parametric forms to ensemble methods and high-capacity nonlinear learners — the relationship between input variables and model outputs becomes progressively less transparent. This opacity poses a fundamental challenge: <strong>how can we assess the contribution of individual features to a model’s predictive behavior without direct access to its internal mechanisms?</strong></p><p>Feature importance methods have emerged as a central response to this challenge. Rather than interrogating model parameters directly, these approaches seek to quantify how variations in input features influence predictive performance. Among the available techniques, <strong>permutation importance</strong> occupies a distinctive position. It is model-agnostic, grounded in statistical principles, and explicitly tied to a model’s empirical performance on unseen data. Most importantly, it reframes interpretability as a perturbation problem:</p><blockquote><em>what happens to predictive accuracy when the information carried by a feature is systematically disrupted?</em></blockquote><p>Permutation importance operationalizes this idea by deliberately breaking the association between a feature and the target variable through random reordering. The resulting degradation in predictive performance provides a direct and interpretable measure of that feature’s relevance to the model. This seemingly simple procedure reflects a deeper statistical philosophy — one rooted in resampling and randomization — that has long been used to test hypotheses and assess significance in classical statistics.</p><h3>Historical and Statistical Foundations</h3><p>The conceptual foundation of permutation importance predates machine learning by several decades. Permutation and randomization tests were developed in early 20th-century statistics (notably by Fisher and later formalized by others such as Pitman)</p><p>These tests evaluate statistical hypotheses by repeatedly shuffling data labels to generate a reference distribution under the null hypothesis of no association.</p><p>The central idea is straightforward: if a variable has no meaningful relationship with an outcome, then randomly permuting its values should not materially change any statistic of interest. Conversely, if the variable carries information, permuting it should degrade performance.</p><p>This logic entered machine learning most prominently through the work of <strong>Leo Breiman</strong>, who introduced permutation-based variable importance in the context of Random Forests. Breiman proposed measuring feature importance by permuting each variable in the out-of-bag samples and recording the resulting drop in predictive accuracy. Crucially, this approach did not rely on tree structure, split counts, or impurity measures — it was entirely performance-based.</p><p>Over time, permutation importance was recognized as a <strong>model-agnostic interpretability technique</strong>, applicable to any predictive model for which a performance metric can be computed. This generality distinguishes it from many alternative feature importance measures.</p><h3>Conceptual Overview</h3><p>At a high level, permutation importance answers the following counterfactual question:</p><blockquote>How much worse would the model perform if it could no longer use a given feature?</blockquote><p>To approximate this counterfactual, the method proceeds as follows:</p><ol><li>Train a predictive model using the original dataset.</li><li>Measure its baseline performance on a validation or test set.</li><li>Select one feature.</li><li>Randomly permute the values of that feature across samples, leaving all other features unchanged.</li><li>Re-evaluate model performance on the perturbed dataset.</li><li>Attribute the change in performance to the importance of the permuted feature.</li></ol><p>This process is repeated independently for each feature, yielding a ranked list of importance scores.</p><h3><strong>Mathematical Formulation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*HGenKoDmNOC8KlmW_IoMZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*uqXtwnkbHy5S64FCO791oQ.png" /></figure><h3>Intuitive Interpretation</h3><p>Permutation importance measures the <strong>marginal contribution of a feature to predictive performance</strong>, given that all other features remain intact.</p><ul><li>A <strong>large positive importance</strong> indicates that the feature carries information the model actively uses.</li><li>An importance <strong>near zero</strong> suggests the feature is redundant or irrelevant.</li><li>A <strong>negative importance</strong> (rare but possible) may indicate noise, sampling variability, or overfitting.</li></ul><p>Importantly, permutation importance does <strong>not</strong> measure causal influence. It quantifies how much the model relies on a feature, not whether the feature is intrinsically important in the data-generating process.</p><h3>Practical workflow: how to use permutation importance correctly</h3><h4>Use a held-out set</h4><p>Permutation importance should be computed on a <strong>validation or test set</strong>, not on the training data, because the goal is to measure reliance under generalization rather than memorization.</p><h4>Choose the right scoring metric</h4><p>The importance values depend on the scoring metric:</p><ul><li>Accuracy emphasizes overall correctness.</li><li>F1 emphasizes minority class performance.</li><li>RMSE penalizes large errors.</li><li>MAE is more robust to outliers.</li></ul><p>Since permutation importance is “performance drop,” it inherits the <em>priorities</em> of the metric.</p><h4>Use enough repeats</h4><p>With too few repeats, estimates can be noisy. In practice, 10–30 repeats are common, with higher numbers for small datasets or unstable models.</p><h4>Interpret as model reliance, not causality</h4><p>A high importance score means the model <em>uses</em> the feature; it does not imply the feature is causally responsible for the outcome.</p><h3>Worked example: Python code snippet (scikit-learn)</h3><p>Below is a minimal, reproducible example using a standard dataset and sklearn.inspection.permutation_importance.</p><pre>import numpy as np<br>import pandas as pd<br><br>from sklearn.datasets import load_diabetes<br>from sklearn.model_selection import train_test_split<br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.inspection import permutation_importance<br>from sklearn.metrics import mean_squared_error<br><br># 1) Load a real dataset (regression)<br>data = load_diabetes(as_frame=True)<br>X = data.data<br>y = data.target<br><br>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, test_size=0.25, random_state=42<br>)<br><br># 2) Fit a model<br>model = RandomForestRegressor(<br>    n_estimators=400,<br>    random_state=42,<br>    n_jobs=-1<br>)<br>model.fit(X_train, y_train)<br><br># 3) Baseline performance on test data<br>y_pred = model.predict(X_test)<br>rmse_base = mean_squared_error(y_test, y_pred, squared=False)<br>print(f&quot;Baseline RMSE: {rmse_base:.3f}&quot;)<br><br># 4) Permutation importance (use negative RMSE so &#39;higher is better&#39; for sklearn)<br>result = permutation_importance(<br>    model,<br>    X_test,<br>    y_test,<br>    n_repeats=30,<br>    random_state=42,<br>    scoring=&quot;neg_root_mean_squared_error&quot;<br>)<br><br># 5) Summarize<br>importance_df = pd.DataFrame({<br>    &quot;feature&quot;: X.columns,<br>    &quot;importance_mean&quot;: result.importances_mean,<br>    &quot;importance_std&quot;: result.importances_std<br>}).sort_values(&quot;importance_mean&quot;, ascending=False)<br><br>print(importance_df)<br><br># Note:<br># Since scoring is NEGATIVE RMSE, higher score is better.<br># So importance_mean here already represents performance DROP in that score space.<br># Larger positive values =&gt; bigger drop =&gt; more important feature.</pre><h3>How to interpret the output</h3><ul><li>importance_mean: average decrease in the model’s score after shuffling that feature.</li><li>importance_std: uncertainty across permutations.</li><li>Features near zero are weakly used (or redundant given other features).</li></ul><pre><br>import matplotlib.pyplot as plt<br><br>plt.figure(figsize=(8, 5))<br>plt.barh(<br>    importance_df[&quot;feature&quot;],<br>    importance_df[&quot;importance_mean&quot;],<br>    xerr=importance_df[&quot;importance_std&quot;]<br>)<br>plt.gca().invert_yaxis()<br>plt.xlabel(&quot;Permutation importance (mean ± std)&quot;)<br>plt.title(&quot;Permutation Feature Importance&quot;)<br>plt.tight_layout()<br>plt.show()</pre><h3>Limitations and failure modes</h3><p>Permutation importance is powerful, but it is not universally reliable. The most important limitations are structural, not cosmetic.</p><h4>1. Correlated (or redundant) features</h4><p>If two features share information (multicollinearity), permuting one may not reduce performance much because the model can recover the signal from the other.</p><p><strong>Consequence:</strong> truly relevant features may appear less important than they “should,” because importance is effectively <em>shared</em> among correlated predictors.</p><p>This is not simply a bug — it reflects a fundamental identifiability issue: when multiple predictors encode the same information, attributing unique credit is inherently ambiguous.</p><p><strong>Mitigation ideas:</strong></p><ul><li>Group correlated features and permute as a block.</li><li>Use conditional permutation strategies (more complex).</li><li>Complement with SHAP (still not fully immune, but provides additional perspective).</li></ul><h4>2. Permutation breaks the joint structure</h4><p>Permutation preserves a feature’s marginal distribution but disrupts its relationships with other features. For some problems, this creates “unrealistic” samples (feature combinations that do not occur naturally).</p><p><strong>Consequence:</strong> the measured performance drop may reflect distribution shift artifacts rather than true reliance.</p><h4>3. Metric dependence</h4><p>Because importance is defined by performance degradation, it is metric-relative:</p><ul><li>A feature important for ranking (AUC) may not be important for thresholded accuracy.</li><li>A feature important for reducing large errors (RMSE) may not be as important for MAE.</li></ul><p><strong>Best practice:</strong> report the metric explicitly and align it with business/scientific objectives.</p><h4>4. Computational cost</h4><p>Permutation importance requires repeated scoring passes:</p><ul><li>p features × K permutations × dataset inference<br> This can be expensive for large p, large n, or heavy models.</li></ul><p><strong>Mitigation ideas:</strong></p><ul><li>Use fewer repeats initially; refine later.</li><li>Compute on a representative subsample.</li><li>Parallelize scoring when possible.</li></ul><h4>5. Not causal (and can be misleading under leakage)</h4><p>Permutation importance is not causal inference. It can also be inflated by:</p><ul><li>leakage features (post-outcome signals)</li><li>target proxies</li><li>data pipeline errors</li></ul><p><strong>Best practice:</strong> Validate data lineage and leakage safeguards before trusting the importance.</p><h4>6. Structured data: time series, groups, panels</h4><p>Naïve permutation assumes exchangeability of rows. In time series or grouped data, shuffling can violate structure.</p><p><strong>Mitigation ideas:</strong></p><ul><li>Block permutation (shuffle within time blocks)</li><li>Permute within groups (e.g., within customer/account)</li><li>Use time-aware validation schemes first (walk-forward)</li></ul><h3>Best-practice checklist</h3><ul><li>Compute on <strong>held-out</strong> data (validation/test), not training</li><li>Use <strong>repeats</strong> and report uncertainty (mean ± std)</li><li>Choose the <strong>metric</strong> that matches your objective</li><li>Inspect <strong>correlations</strong>; consider grouped/block permutations when needed</li><li>Treat results as <strong>model reliance diagnostics</strong>, not causality</li><li>Pair with other interpretability tools (e.g., partial dependence, SHAP) when the stakes are high</li></ul><h3>Conclusion</h3><p>Permutation importance provides a principled, model-agnostic method for quantifying feature relevance through perturbation. Its strength lies in its direct connection to predictive performance: if destroying a feature’s information harms generalization, the model was relying on that feature. At the same time, its interpretability is conditional and context-dependent, particularly under correlated predictors, structured data, and potential leakage.</p><p>Used thoughtfully, permutation importance is a strong baseline for interpretability: transparent, empirical, and tightly coupled to the operational question that matters most in applied modeling — <em>what drives predictive performance in practice?</em></p><h3>📘 Further Reading:</h3><h4>1. Leo Breiman (2001) — Random Forests (Original Source)</h4><p>🔗 <a href="https://link.springer.com/article/10.1023/A:1010933404324">https://link.springer.com/article/10.1023/A:1010933404324</a></p><h4>2. Christoph Molnar — Interpretable Machine Learning</h4><p>🔗 <a href="https://christophm.github.io/interpretable-ml-book/permutation-feature-importance.html">https://christophm.github.io/interpretable-ml-book/permutation-feature-importance.html</a></p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/code-to-deploy-3784391b9/"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Note:</strong> This Post may contain affiliate links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5da64e1f16e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/permutation-importance-as-a-performance-based-measure-of-feature-contribution-c5da64e1f16e">Permutation Importance as a Performance-Based Measure of Feature Contribution</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>