<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adnan Mazraeh on Medium]]></title>
        <description><![CDATA[Stories by Adnan Mazraeh on Medium]]></description>
        <link>https://medium.com/@adnan.mazraeh1993?source=rss-4d6bb2142c5f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FDIg4KL7Wt6hpW-ID7zkow.jpeg</url>
            <title>Stories by Adnan Mazraeh on Medium</title>
            <link>https://medium.com/@adnan.mazraeh1993?source=rss-4d6bb2142c5f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:18:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adnan.mazraeh1993/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 10): Orthogonal Matching Pursuit (OMP)]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-10-orthogonal-matching-pursuit-omp-f7d649ec1e7a?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7d649ec1e7a</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Mon, 09 Feb 2026 19:16:33 GMT</pubDate>
            <atom:updated>2026-02-09T19:16:33.636Z</atom:updated>
            <content:encoded><![CDATA[<h4>Welcome to your comprehensive guide to <strong>Orthogonal Matching Pursuit (OMP)</strong> in Scikit-Learn. This algorithm is a cornerstone of <strong>Sparse Coding</strong> and <strong>Signal Processing</strong>. It is conceptually similar to Forward Stepwise Regression but is mathematically optimized for finding the “sparsest” possible solution (using the fewest features).</h4><h3>Part 1: The Concept (In Simple English)</h3><h3>The Problem: The Cocktail Party</h3><p>Imagine you are at a noisy party. You hear a mix of sounds (Target Signal y). You know the specific voices of 100 people at the party (Features/Atoms in a Dictionary X).</p><p><strong>The Goal:</strong> You want to recreate the sound you hear using <strong>only 3 people’s voices</strong>. You can’t use everyone. You need to pick the absolute best 3 voices and determine how loud each should speak to match the noise perfectly.</p><h3>How OMP Solves It (The “Greedy” Strategy)</h3><ol><li><strong>Listen &amp; Match:</strong> OMP looks at the target sound (y) and compares it to all 100 voices (X).</li><li><strong>Pick the Best:</strong> It picks the <em>one</em> voice that sounds most like the target. Let’s say it picks “Alice.”</li><li><strong>Subtract (Orthogonalize):</strong> This is the magic step. It calculates exactly how much of the target sound is explained by Alice. It then <strong>removes</strong> Alice’s contribution from the target. Now we have a “residual” sound (what’s left).</li><li><strong>Repeat:</strong> Now OMP looks at the <em>residual</em> sound. It compares the remaining 99 voices to this residual. It picks “Bob.”</li><li><strong>Refine (The “Orthogonal” part):</strong> Unlike simple matching, OMP now looks at <em>both</em> Alice and Bob together. It recalculates their volumes to make sure they don’t overlap or interfere. It ensures the residual is <strong>orthogonal</strong> (perpendicular) to the chosen voices.</li><li><strong>Stop:</strong> It repeats this until it hits a limit (e.g., “3 voices used”) or the error is small enough.</li></ol><h3>Part 2: The Math (The “Professional” Details)</h3><p>OMP tries to solve the linear system y=X= (where w is the weight/coefficient vector), subject to a constraint on the number of non-zero elements (L0-norm).</p><p>OMP tries to solve the linear system y=Xw (where w is the weight/coefficient vector), subject to a constraint on the number of non-zero elements (L0-norm).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*bVBTJXzp5zzsWa4LfKREkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*IEFBlHFJsWBDAWxON5Bq2Q.png" /></figure><h3>Part 3: Basic Implementation</h3><p>Let’s use OMP to reconstruct a signal. We will create a synthetic signal that is a combination of just 4 sine waves, but we will hide them inside a dictionary of 100 different waves.</p><h3>Step 1: Generate Data (Sparse Signal)’</h3><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.linear_model import OrthogonalMatchingPursuit<br>from sklearn.datasets import make_sparse_coded_signal<br><br># 1. Generate a signal<br># n_samples=1: Just one signal to reconstruct<br># n_components=100: The dictionary has 100 possible &quot;voices&quot; (atoms)<br># n_features=512: The length of the signal (time points)<br># n_nonzero_coefs=4: Only 4 &quot;voices&quot; are actually used<br>y, X, w = make_sparse_coded_signal(<br>    n_samples=1,<br>    n_components=100,<br>    n_features=512,<br>    n_nonzero_coefs=4,<br>    random_state=42<br>)<br><br># Reshape y to be a proper vector for sklearn (n_samples, )<br>y = y.flatten()<br><br># X is currently shape (100, 512). Transpose it to (n_samples, n_features) format?<br># Wait! In Signal Processing:<br># Dictionary (D) is (n_features, n_atoms).<br># Sparse Code (x) is (n_atoms, ).<br># Signal (y) = D * x<br># Sklearn expects: y = X * w<br># So we transpose X to fit sklearn&#39;s regression shape: (n_samples=512, n_features=100)<br>X_transposed = X.T<br><br>print(f&quot;Shape of Design Matrix (Dictionary): {X_transposed.shape}&quot;)<br>print(f&quot;Shape of Target Signal: {y.shape}&quot;)</pre><h3>Step 2: Fit the Model</h3><p>We know the truth is 4 non-zero coefficients. Let’s see if OMP can find them.</p><pre># Initialize OMP<br># n_nonzero_coefs=4 tells the algorithm explicitly: &quot;Find exactly 4 atoms&quot;<br>omp = OrthogonalMatchingPursuit(n_nonzero_coefs=4)<br><br># Fit<br>omp.fit(X_transposed, y)<br><br># Results<br>coef = omp.coef_<br>idx_r, = coef.nonzero()<br><br>print(f&quot;Recovered {len(idx_r)} non-zero coefficients.&quot;)<br>print(f&quot;Indices of non-zero coefficients: {idx_r}&quot;)</pre><p>Step 3: Visualization (Reconstruction)</p><pre>plt.figure(figsize=(10, 6))<br><br># 1. Plot the original clean signal (y)<br>plt.plot(y, &#39;r&#39;, label=&#39;Original Signal&#39;, linewidth=2)<br><br># 2. Reconstruct signal using found coefficients<br># prediction = X * w<br>y_reconstructed = omp.predict(X_transposed)<br>plt.plot(y_reconstructed, &#39;b--&#39;, label=&#39;OMP Reconstruction&#39;, linewidth=2)<br><br>plt.title(&quot;Sparse Signal Recovery with OMP&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()<br><br># Visualize the coefficients (Sparse Code)<br>plt.figure(figsize=(10, 4))<br>plt.stem(idx_r, coef[idx_r], basefmt=&quot; &quot;)<br>plt.title(&quot;Recovered Coefficients (Sparse Pattern)&quot;)<br>plt.xlabel(&quot;Dictionary Index&quot;)<br>plt.ylabel(&quot;Weight&quot;)<br>plt.show()</pre><h3>Part 4: OMP vs. OMP-CV (Automatic Tuning)</h3><p>In the real world, you rarely know exactly how many non-zero coefficients exist (k=4). You usually want to stop when the error is “low enough.”</p><p>Scikit-Learn provides OrthogonalMatchingPursuitCV to automatically find the best number of non-zero coefficients using Cross-Validation.</p><p><strong>Note:</strong> OMP-CV works slightly differently than LassoCV. It increases the number of active features one by one and checks the cross-validation score.</p><pre>from sklearn.linear_model import OrthogonalMatchingPursuitCV<br><br># Initialize CV model<br># cv=5: 5-fold cross validation<br># max_iter: Maximum number of features to try (don&#39;t go beyond 20 for this example)<br>omp_cv = OrthogonalMatchingPursuitCV(cv=5, max_iter=20, n_jobs=-1)<br><br># Fit<br>omp_cv.fit(X_transposed, y)<br><br>print(f&quot;Best number of features found: {omp_cv.n_nonzero_coefs_}&quot;)<br>print(f&quot;Score (R2): {omp_cv.score(X_transposed, y):.4f}&quot;)</pre><h3>Part 5: “Super Professional” Usage — Dictionary Learning</h3><p>OMP is rarely used for simple regression (like predicting house prices). It is almost exclusively used in <strong>Dictionary Learning</strong> and <strong>Image Compression</strong>.</p><p><strong>Scenario:</strong> Image Denoising.</p><p>You can treat small 8×8 patches of an image as signals. You learn a “Dictionary” of shapes (edges, textures). Then you use OMP to represent every patch using only a few dictionary atoms. Because noise is random and doesn’t fit the dictionary well, reconstructing the image using only the OMP coefficients removes the noise!</p><p><em>Note: The code below is a conceptual advanced example using the </em><em>MiniBatchDictionaryLearning class, which uses OMP internally for the coding step.</em></p><pre>from sklearn.decomposition import MiniBatchDictionaryLearning<br>from sklearn.feature_extraction.image import extract_patches_2d, reconstruct_from_patches_2d<br>from sklearn.datasets import load_sample_image<br><br># 1. Load an image (China usually comes with sklearn)<br>image = load_sample_image(&quot;flower.jpg&quot;)<br>image = image / 255.0  # Normalize<br>h, w, c = image.shape<br><br># Convert to grayscale for simplicity<br>image_gray = image.mean(axis=2)<br>print(f&quot;Image shape: {image_gray.shape}&quot;)<br><br># 2. Extract 8x8 patches<br>patch_size = (8, 8)<br>data = extract_patches_2d(image_gray, patch_size, max_patches=1000, random_state=42)<br>data = data.reshape(data.shape[0], -1) # Flatten patches<br>data -= np.mean(data, axis=0) # Remove mean<br>data /= np.std(data, axis=0) # Normalize<br><br># 3. Learn a Dictionary using OMP logic internally<br># n_components=100 atoms in dictionary<br># transform_algorithm=&#39;omp&#39; tells it to use Orthogonal Matching Pursuit<br>dico = MiniBatchDictionaryLearning(n_components=100, alpha=1, n_iter=500,<br>                                   transform_algorithm=&#39;omp&#39;, random_state=42)<br><br>dico.fit(data)<br><br># 4. Visualize the learned dictionary (The &quot;Atoms&quot;)<br>plt.figure(figsize=(8, 8))<br>for i, comp in enumerate(dico.components_[:100]):<br>    plt.subplot(10, 10, i + 1)<br>    plt.imshow(comp.reshape(patch_size), cmap=plt.cm.gray_r, interpolation=&#39;nearest&#39;)<br>    plt.xticks(())<br>    plt.yticks(())<br>plt.suptitle(&#39;Learned Dictionary Atoms (Using OMP)&#39;, fontsize=16)<br>plt.show()</pre><p><strong>Interpretation:</strong></p><p>The plot will show you 100 small squares. Some look like vertical lines, some horizontal, some diagonal. OMP uses combinations of these “basic shapes” to rebuild the entire complex image.</p><h3>Part 6: When to use OMP?</h3><p><strong>✅ USE IT WHEN:</strong></p><ol><li><strong>Sparse Signal Recovery:</strong> You know the target is a combination of a very small number of features.</li><li><strong>Compressed Sensing:</strong> You have fewer measurements than unknowns, but you know the solution is sparse.</li><li><strong>Speed:</strong> OMP is generally faster than Lasso (L1 optimization) when the number of non-zero coefficients is very small.</li></ol><p><strong>❌ DO NOT USE IT WHEN:</strong></p><ol><li><strong>Dense Data:</strong> If the target depends on <em>many</em> or <em>all</em> features (like house prices depending on 50 variables), Ridge or OLS is better.</li><li><strong>Correlated Features:</strong> If two features are identical, OMP will pick one and ignore the other completely. It is a “greedy” algorithm and can be unstable with highly correlated features.</li></ol><h3>Part 7: Summary &amp; Code Template</h3><p>Here is your clean template for using OrthogonalMatchingPursuit in a generic regression context.</p><pre>from sklearn.linear_model import OrthogonalMatchingPursuit<br>from sklearn.pipeline import make_pipeline<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import mean_squared_error<br><br># 1. Setup Data<br># X, y = load_data()...<br># X_train, X_test...<br><br># 2. Pipeline<br># Normalization is critical for OMP because it relies on dot products<br>pipeline = make_pipeline(<br>    StandardScaler(),<br>    # tol=None means &quot;don&#39;t stop based on error&quot;<br>    # n_nonzero_coefs=10 means &quot;Stop exactly after finding top 10 features&quot;<br>    OrthogonalMatchingPursuit(n_nonzero_coefs=10, fit_intercept=True, normalize=False)<br>)<br><br># 3. Fit<br>pipeline.fit(X_train, y_train)<br><br># 4. Results<br>y_pred = pipeline.predict(X_test)<br>mse = mean_squared_error(y_test, y_pred)<br>model = pipeline.named_steps[&#39;orthogonalmatchingpursuit&#39;]<br><br>print(f&quot;MSE: {mse:.4f}&quot;)<br>print(f&quot;Selected Features Indices: {model.coef_.nonzero()[0]}&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7d649ec1e7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 8): LARS Lasso]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-8-lars-lasso-a14bff4da248?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a14bff4da248</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Sun, 08 Feb 2026 06:28:49 GMT</pubDate>
            <atom:updated>2026-02-08T06:28:49.633Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scikit-learn: From Zero to Hero (Part 9): LARS Lasso</h3><h4>If you understood the previous section (LARS), you are 90% there. But that last 10% makes a huge difference in how the model behaves mathematically. Let’s go from <strong>Zero</strong> (What is the difference?) to <strong>Pro</strong> (AIC/BIC selection and path algorithms).</h4><h3>Part 1: The Concept (Simple English)</h3><h3>The Confusion</h3><p>You might ask: <em>“Wait, we have </em><em>Lasso (1.1.3) and we have </em><em>Lars (1.1.7). Why do we need </em><em>LassoLars?”</em></p><p>Here is the secret:</p><ul><li><strong>Lasso</strong> is a <strong>Problem</strong> (an objective: minimize error + penalty).</li><li><strong>LARS</strong> is an <strong>Algorithm</strong> (a specific way to calculate numbers).</li></ul><p>Usually, the Lasso command uses an algorithm called “Coordinate Descent.” It’s like guessing and checking rapidly.</p><p><strong>LARS Lasso</strong> uses the elegant geometry of LARS to solve the Lasso problem.</p><h3>The “One-Way Street” Analogy</h3><p>Imagine driving a car to pick up friends (features) to go to a party (the target).</p><ol><li><strong>Standard LARS:</strong> You pick up Friend A. You drive towards Friend B. You pick up Friend B. You never drop anyone off. Once a friend is in the car, they stay in the car.</li><li><strong>Lasso LARS:</strong> You pick up Friend A. You pick up Friend B. But, if the car turns in a certain direction and Friend A suddenly becomes “useless” or “negative” to the journey, <strong>you kick them out of the car</strong>.</li></ol><p><strong>The Key Difference:</strong></p><p>In LassoLars, a feature can enter the model, become active, and then <strong>leave</strong> the model later as the algorithm progresses. In standard Lars, features never leave once they enter.</p><h3>Part 2: The Math (The “Pro” Details)</h3><p>To be a professional, you need to understand the <strong>KKT Conditions</strong> (Karush-Kuhn-Tucker conditions), but let’s simplify them. The Lasso solution requires that for any active feature (coefficient ≠0), the correlation with the residual must be equal to the regularization parameter α.</p><p><strong>The LARS Modification:</strong></p><p>Standard LARS guarantees that correlations decrease at the same rate. However, standard LARS allows coefficients to change sign (e.g., go from positive +5 to negative −2).</p><p><strong>Lasso Penalty (∣w∣1​) Constraint:</strong></p><p>The Lasso penalty ∣w∣ has a “kink” at zero. It treats positive and negative values differently. If a coefficient in the LARS path crosses zero (tries to change sign), it violates the Lasso definition.</p><p><strong>The Fix:</strong></p><p>The <strong>LassoLars algorithm</strong> adds a check:</p><blockquote><strong><em>“If a coefficient hits zero, drop it from the active set and recompute the direction.”</em></strong></blockquote><p>This simple check ensures that the LARS algorithm produces the <strong>exact</strong> solution to the Lasso problem.</p><h3>Part 3: Basic Implementation</h3><p>Using LassoLars is almost identical to using Lasso, but it is often faster for high-dimensional data (where number of features &gt; number of samples).</p><pre>import numpy as np<br>from sklearn import linear_model<br>from sklearn.metrics import mean_squared_error, r2_score<br>from sklearn.datasets import make_regression<br>from sklearn.model_selection import train_test_split<br><br># 1. Generate Data (Sparse: only 10 features are useful out of 200)<br>X, y = make_regression(n_samples=100, n_features=200, n_informative=10, noise=10, random_state=42)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># 2. Define LassoLars<br># alpha is the regularization strength. <br># alpha=0 is Linear Regression (OLS). alpha=1 is high regularization.<br>model = linear_model.LassoLars(alpha=0.1, max_iter=1000)<br><br># 3. Fit<br>model.fit(X_train, y_train)<br><br># 4. Predict<br>y_pred = model.predict(X_test)<br><br>print(f&quot;Number of non-zero coefficients: {np.sum(model.coef_ != 0)}&quot;)<br>print(f&quot;R2 Score: {r2_score(y_test, y_pred):.4f}&quot;)</pre><h3>Part 4: The “Super Professional” Technique (AIC &amp; BIC)</h3><p>This is the main reason pros use LARS.</p><p>In standard Lasso, you usually use Cross-Validation (LassoCV) to find the best alpha. This requires training the model 5 or 10 times (k-fold).</p><p>However, because LARS calculates the <strong>entire path</strong> of solutions efficiently, we can use statistical criteria to find the best alpha in <strong>one single run</strong> without cross-validation. This is extremely fast.</p><p>We use <strong>AIC</strong> (Akaike Information Criterion) or <strong>BIC</strong> (Bayesian Information Criterion).</p><ul><li><strong>Goal:</strong> Balance the “goodness of fit” (low error) with “complexity” (number of features).</li><li><strong>Lower is better</strong> for both scores.</li></ul><pre>from sklearn.linear_model import LassoLarsIC<br>import matplotlib.pyplot as plt<br><br># We use the diabetes dataset for a real-world example<br>from sklearn.datasets import load_diabetes<br>X_real, y_real = load_diabetes(return_X_y=True)<br><br># 1. LassoLarsIC with AIC<br># This calculates the path and automatically picks the alpha that minimizes AIC<br>model_aic = LassoLarsIC(criterion=&#39;aic&#39;)<br>model_aic.fit(X_real, y_real)<br><br># 2. LassoLarsIC with BIC<br># BIC usually prefers simpler models (fewer features) than AIC<br>model_bic = LassoLarsIC(criterion=&#39;bic&#39;)<br>model_bic.fit(X_real, y_real)<br><br>print(f&quot;Alpha chosen by AIC: {model_aic.alpha_:.4f}&quot;)<br>print(f&quot;Alpha chosen by BIC: {model_bic.alpha_:.4f}&quot;)<br><br># --- Visualization of the Information Criteria ---<br># This is how you prove to your boss/professor you picked the right model<br>plt.figure(figsize=(10, 6))<br><br># Plot AIC curve<br>plt.plot(model_aic.alphas_, model_aic.criterion_, label=&#39;AIC Curve&#39;, color=&#39;blue&#39;)<br># Mark the best alpha<br>plt.axvline(model_aic.alpha_, linestyle=&#39;--&#39;, color=&#39;blue&#39;, label=&#39;Best AIC Alpha&#39;)<br><br># Plot BIC curve<br>plt.plot(model_bic.alphas_, model_bic.criterion_, label=&#39;BIC Curve&#39;, color=&#39;red&#39;)<br># Mark the best alpha<br>plt.axvline(model_bic.alpha_, linestyle=&#39;--&#39;, color=&#39;red&#39;, label=&#39;Best BIC Alpha&#39;)<br><br>plt.xlabel(&#39;Alpha (Regularization Strength)&#39;)<br>plt.ylabel(&#39;Information Criterion (Lower is Better)&#39;)<br>plt.title(&#39;AIC vs BIC for Hyperparameter Tuning&#39;)<br>plt.legend()<br>plt.show()</pre><p><strong>Interpretation:</strong></p><ul><li>BIC penalizes complex models more heavily. It usually suggests a higher alpha (fewer features).</li><li>AIC is more lenient. It usually suggests a lower alpha (more features).</li><li>Using LassoLarsIC is much faster than LassoCV on large datasets.</li></ul><h3>Part 5: Comparing the Paths (LARS vs Lasso)</h3><p>To truly see the difference between Section 1.1.7 (Lars) and 1.1.8 (LassoLars), we can compute both paths on the same data.</p><p>In most simple datasets, they look identical. But mathematically:</p><ol><li><strong>Method ‘lar’</strong>: Coefficients move, features are added.</li><li><strong>Method ‘lasso’</strong>: Coefficients move, features are added, <strong>features can be dropped</strong> (coefficient hits 0 and stops).</li></ol><pre># Compute paths manually<br># method=&#39;lasso&#39; is the logic for 1.1.8<br>alphas_lasso, _, coefs_lasso = linear_model.lars_path(X_real, y_real, method=&#39;lasso&#39;, verbose=True)<br><br># method=&#39;lar&#39; is the logic for 1.1.7<br>alphas_lar, _, coefs_lar = linear_model.lars_path(X_real, y_real, method=&#39;lar&#39;, verbose=True)<br><br>print(f&quot;Lasso Path steps: {coefs_lasso.shape[1]}&quot;)<br>print(f&quot;LARS Path steps: {coefs_lar.shape[1]}&quot;)</pre><p><em>If the number of steps is different, it means the Lasso method had to drop a variable at some point, extending the path calculations.</em></p><h3>Part 6: When to use what?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*rp334gnyT3uvp7E4lVCk-Q.png" /></figure><p><strong>Pro Tip:</strong></p><p>If your features are highly correlated, LassoLars can be numerically unstable because of the matrix inversion required in the LARS step. In that case, use standard Lasso (Coordinate Descent) or ElasticNet.</p><h3>Part 7: Summary Code Template</h3><p>Here is the robust template for using LARS Lasso with Automatic Tuning via Information Criteria (The Pro Way).</p><pre>from sklearn.linear_model import LassoLarsIC<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.pipeline import make_pipeline<br><br># 1. Prepare Data<br># X, y = load_data...<br><br># 2. Create Pipeline<br># Always scale data before Lasso/LARS!<br>pipeline = make_pipeline(<br>    StandardScaler(),<br>    # criterion=&#39;bic&#39; is safer to avoid overfitting<br>    LassoLarsIC(criterion=&#39;bic&#39;, max_iter=5000) <br>)<br><br># 3. Fit<br>pipeline.fit(X, y)<br><br># 4. Extract Results<br>model = pipeline.named_steps[&#39;lassolarsic&#39;]<br>print(f&quot;Selected Alpha: {model.alpha_}&quot;)<br>print(f&quot;Number of Features: {np.sum(model.coef_ != 0)}&quot;)<br><br># 5. Score<br>score = pipeline.score(X, y)<br>print(f&quot;R2 Score: {score:.4f}&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a14bff4da248" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 8): Least Angle Regression]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-7-least-angle-regression-af70d078c0dc?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/af70d078c0dc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Sun, 08 Feb 2026 06:10:12 GMT</pubDate>
            <atom:updated>2026-02-08T06:10:30.098Z</atom:updated>
            <content:encoded><![CDATA[<h4>This algorithm is often considered one of the most “elegant” mathematical algorithms in machine learning. While it solves a regression problem (predicting a number), the <em>way</em> it gets to the answer is completely different from standard Linear Regression. Let’s go from <strong>Zero</strong> (Conceptual) to <strong>Pro</strong> (Implementation and Math).</h4><h3>Part 1: The Concept (Simple English)</h3><p>To understand LARS, we first need to look at how other algorithms think:</p><ol><li><strong>Standard Linear Regression:</strong> Tries to find the answer all at once. It looks at all features and calculates the perfect coefficients immediately. This is slow if you have thousands of features.</li><li><strong>Forward Stepwise Selection:</strong> Picks the best feature, fully fits it. Then picks the next best, fully fits it. This is “greedy” and often too aggressive.</li></ol><h3>The “LARS” Intuition (The Walking Analogy)</h3><p>Imagine you are standing at the origin (0,0) and you want to walk towards a destination (the Truth/Target y). You have several paths (Features/Variables) you can take.</p><p><strong>How LARS walks:</strong></p><ol><li><strong>Look:</strong> Which feature is most correlated with the target? Let’s say it’s <strong>Feature A</strong>.</li><li><strong>Walk:</strong> Start walking along the direction of Feature A. Do not go all the way to the end! Just take small steps.</li><li><strong>Check:</strong> As you walk along A, the angle to the target changes. Eventually, there will come a point where <strong>Feature B</strong> becomes <em>just as correlated</em> with the remaining distance (residual) as Feature A is.</li><li><strong>Turn:</strong> Stop immediately. Now, calculate a new direction that is exactly <strong>halfway</strong> between Feature A and Feature B (the “Least Angle”).</li><li><strong>Walk Together:</strong> Walk in this new joint direction until a third feature (Feature C) joins the party.</li></ol><p><strong>Why is this “Super Professional”?</strong></p><p>Instead of guessing or recalculating everything extensively, LARS calculates the <strong>exact path</strong> to the solution efficiently. It builds the model step-by-step, adding variables one by one only when they are needed.</p><h3>Part 2: The Math (Under the Hood)</h3><p>LARS is based on the geometry of the data.</p><h3>1. The Definitions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*ZQ6SsEuNKgh2cfh_hb98zw.png" /></figure><h3>2. The Process</h3><p>The algorithm looks at the correlation between the features (X) and the current residual (r).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*GJDSxoNZCXpoa3fMvalmYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*rTCiTQ5IbNrGlHCbcplzTw.png" /></figure><h3>Part 3: Basic Implementation (Lars)</h3><p>Scikit-learn provides the Lars class. This is the pure implementation of the algorithm.</p><h3>Step 1: Setup and Data</h3><p>We will use diabetes data, a classic dataset for regression.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn import linear_model<br>from sklearn.datasets import load_diabetes<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import r2_score<br><br># Load data<br>X, y = load_diabetes(return_X_y=True)<br><br># Split data<br>X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)<br><br>print(f&quot;Features: {X.shape[1]}&quot;)</pre><h3>Step 2: Training LARS</h3><p>One specific parameter for Lars is n_nonzero_coefs. Since LARS adds features one by one, you can tell it to stop after it has picked, say, the top 5 most important features.</p><pre># Initialize LARS<br># n_nonzero_coefs=5 means &quot;Stop after finding the 5 most important features&quot;<br>lars_model = linear_model.Lars(n_nonzero_coefs=5, fit_intercept=True)<br><br># Fit the model<br>lars_model.fit(X_train, y_train)<br><br># Check which features were selected<br>print(&quot;Coefficients:&quot;, lars_model.coef_)<br><br># Make predictions<br>y_pred = lars_model.predict(X_test)<br>print(f&quot;R2 Score: {r2_score(y_test, y_pred):.4f}&quot;)</pre><p><strong>Note on Output:</strong> You will see many zeros in the coefficients. LARS found the 5 best features and left the others at 0.</p><h3>Part 4: LARS vs. Lasso (LassoLars)</h3><p>This is where it gets interesting.</p><p><strong>Lasso</strong> is a regularization method (L1 penalty).</p><p><strong>LARS</strong> is an algorithm (a way to calculate numbers).</p><p>It turns out, with a tiny modification, the LARS algorithm can solve the Lasso problem. In Scikit-learn, this is called LassoLars.</p><p><strong>Why use </strong><strong>LassoLars instead of </strong><strong>Lasso?</strong></p><ul><li>Lasso uses coordinate descent (an approximation).</li><li>LassoLars uses the LARS logic (exact math).</li><li>LassoLars is typically <strong>faster</strong> if you have many features but you expect only a few to be non-zero (High-Dimensional sparse data).</li></ul><pre># LassoLars - Using LARS algorithm to solve Lasso<br># alpha is the regularization strength<br>lasso_lars = linear_model.LassoLars(alpha=0.1, max_iter=1000)<br><br>lasso_lars.fit(X_train, y_train)<br><br>print(f&quot;LassoLars Coefficients: {lasso_lars.coef_}&quot;)<br>print(f&quot;LassoLars Score: {lasso_lars.score(X_test, y_test):.4f}&quot;)</pre><h3>Part 5: The “Super Professional” Visualization (The LARS Path)</h3><p>The true power of LARS is that it generates the entire “Regularization Path.” This means it calculates the coefficients for <strong>every possible value of alpha</strong> in one go.</p><p>We can visualize how the coefficients grow as the model becomes more complex (adding features one by one).</p><pre># Compute the regularization path<br># This function returns the alphas and the coefficients at every step<br>alphas, _, coefs = linear_model.lars_path(X, y, method=&#39;lasso&#39;, verbose=True)<br><br># Plotting the LARS Path<br>plt.figure(figsize=(10, 6))<br><br># coefs is shape (n_features, n_steps). We transpose it to plot.<br>xx = np.sum(np.abs(coefs.T), axis=1)<br>xx /= xx[-1] # Normalize to 0-1 range for x-axis<br><br>plt.plot(xx, coefs.T)<br>plt.title(&quot;LARS Path (Evolution of Coefficients)&quot;)<br>plt.xlabel(&quot;Ratio of |coef| / max|coef| (Model Complexity)&quot;)<br>plt.ylabel(&quot;Coefficients&quot;)<br>plt.grid(True)<br>plt.axis(&#39;tight&#39;)<br>plt.show()</pre><h3>How to read this chart (Pro Skill):</h3><ol><li><strong>X-axis (Left to Right):</strong> Represents the model evolution. On the left, complexity is 0 (all coefs are 0). On the right, the model is fully fit (OLS).</li><li><strong>Lines:</strong> Each colored line is a feature.</li><li><strong>Events:</strong> Notice where a line “births” (leaves 0). That is the exact moment LARS decided “This feature is important enough to include.”</li><li><strong>Kinks:</strong> You will see lines change direction sharply. That is the “Angle Split” event we discussed in Part 1. A new variable entered, changing the direction of the optimization.</li></ol><h3>Part 6: When to use LARS?</h3><p>To be a pro, you need to know when <strong>NOT</strong> to use it.</p><h3>✅ The Pros (Use it when…)</h3><ol><li><strong>High-Dimensional Data:</strong> You have more columns (features) than rows (samples) (p&gt;&gt;np &gt;&gt; np&gt;&gt;n). LARS is incredibly efficient here.</li><li><strong>Sparsity:</strong> You suspect only a small number of features actually matter.</li><li><strong>Exactness:</strong> You need the exact solution path, not an approximation.</li></ol><h3>❌ The Cons (Avoid it when…)</h3><ol><li><strong>Noise:</strong> LARS is very sensitive to noise. Because it is iterative, if the data is noisy, an “early bad step” (picking the wrong feature first) can mess up the rest of the path.</li><li><strong>Multicollinearity:</strong> If two features are highly correlated, LARS might behave erratically or pick one arbitrarily. Elastic-Net is usually better here.</li></ol><h3>Part 7: Summary &amp; Code Template</h3><p>Here is your copy-paste template for professional use of LARS with Cross-Validation (finding the best stopping point automatically).</p><pre>from sklearn.linear_model import LassoLarsCV<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.pipeline import make_pipeline<br>from sklearn.model_selection import train_test_split<br>from sklearn.datasets import make_regression<br><br># 1. Generate Data<br>X, y = make_regression(n_samples=200, n_features=500, n_informative=10, noise=0.1)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)<br><br># 2. Pipeline<br># Always scale data for LARS/Lasso<br>pipeline = make_pipeline(<br>    StandardScaler(),<br>    # cv=5: 5-fold cross-validation to find best alpha automatically<br>    LassoLarsCV(cv=5, max_iter=2000, n_jobs=-1) <br>)<br><br># 3. Fit<br>pipeline.fit(X_train, y_train)<br><br># 4. Results<br>model = pipeline.named_steps[&#39;lassolarscv&#39;]<br>print(f&quot;Best Alpha chosen: {model.alpha_}&quot;)<br>print(f&quot;Number of features kept: {np.sum(model.coef_ != 0)}&quot;)<br>print(f&quot;R2 Score: {pipeline.score(X_test, y_test):.4f}&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af70d078c0dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 7): Multi-task Elastic-Net]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-7-multi-task-elastic-net-71dc7f953b0e?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/71dc7f953b0e</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Sat, 31 Jan 2026 05:05:52 GMT</pubDate>
            <atom:updated>2026-01-31T05:05:52.810Z</atom:updated>
            <content:encoded><![CDATA[<h4>Welcome to your comprehensive guide to <strong>Multi-task Elastic-Net</strong> in Scikit-Learn! We will start from the very beginning and move toward professional implementation, covering the theory, math, code, and practical “pro tips.”</h4><h3>Part 1: The Concept (In Simple English)</h3><p>Before writing code, let’s understand <strong>what</strong> this is and <strong>why</strong> we need it.</p><h3>1. What is “Multi-task”?</h3><p>In standard regression (like Linear Regression), you usually try to predict <strong>one</strong> specific number (e.g., the price of a house).</p><ul><li><strong>Single-task:</strong> One input (X) → One output (y).</li><li><strong>Multi-task:</strong> One input (X) → <strong>Multiple outputs</strong> (Y).</li></ul><p><strong>Example:</strong> Imagine you have data about a patient’s diet and exercise (X).</p><ul><li><strong>Single-task:</strong> Predict their Blood Pressure.</li><li><strong>Multi-task:</strong> Predict their Blood Pressure, Cholesterol, <strong>and</strong> Blood Sugar simultaneously.</li></ul><h3>2. What is “Elastic-Net”?</h3><p>It is a combination of two famous regularization techniques:</p><ol><li><strong>Lasso (L1L1L1):</strong> Good at feature selection (forcing useless coefficients to become exactly zero).</li><li><strong>Ridge (L2L2L2):</strong> Good at handling correlated features (keeping coefficients small but not necessarily zero).</li></ol><p><strong>Elastic-Net</strong> mixes both to get the best of both worlds.</p><h3>3. So, what is “Multi-task Elastic-Net”?</h3><p>It is an algorithm that predicts multiple target variables at the same time using the Elastic-Net regularization.</p><p><strong>The Super Power:</strong> The most important feature of this algorithm is <strong>Joint Sparsity</strong>.</p><p>If a feature (e.g., “eating pizza”) is bad for Blood Pressure, the algorithm assumes it is likely relevant (or irrelevant) for Cholesterol and Blood Sugar too. It forces the model to select <strong>the same features</strong> for all tasks.</p><h3>Part 2: The Math (The “Professional” Details)</h3><p>To be a pro, you must understand the objective function.</p><p>In standard Elastic-Net, we minimize:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*36ChdANJUXage8AKp5_3rA.png" /></figure><h3>Part 3: Basic Implementation</h3><p>Let’s write code. We will create a dataset where we want to predict 2 distinct values (YYY has 2 columns).</p><h3>Step 1: Import Libraries and Generate Data</h3><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.linear_model import MultiTaskElasticNet<br>from sklearn.datasets import make_regression<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import r2_score, mean_squared_error<br><br># 1. Generate synthetic data<br># n_targets=2 means we are predicting 2 different things at once<br>X, Y = make_regression(n_samples=1000, n_features=20, n_informative=10, <br>                       n_targets=2, noise=0.5, random_state=42)<br><br># Split into training and testing<br>X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)<br><br>print(f&quot;Shape of X: {X.shape}&quot;) # (1000, 20)<br>print(f&quot;Shape of Y: {Y.shape}&quot;) # (1000, 2) &lt;- Note the 2 columns</pre><h3>Step 2: Define and Fit the Model</h3><p>We define the model with specific hyperparameters.</p><ul><li>alpha: Constant that multiplies the penalty terms. Higher = more regularization (simpler model).</li><li>l1_ratio: The mixing parameter. 0.50.50.5 means 50% Lasso, 50% Ridge.</li></ul><pre># Initialize the model<br># alpha=0.5: Moderate regularization strength<br># l1_ratio=0.5: Equal balance between L1 and L2<br>model = MultiTaskElasticNet(alpha=0.1, l1_ratio=0.5, random_state=42)<br><br># Train the model<br>model.fit(X_train, Y_train)<br><br># Check the coefficients<br># The shape will be (n_features, n_targets) -&gt; (20, 2)<br>print(&quot;Shape of Coefficients:&quot;, model.coef_.shape)</pre><p><strong>Step 3: Evaluate</strong></p><pre># Predict on test data<br>Y_pred = model.predict(X_test)<br><br># Evaluate using R-squared (score 1.0 is perfect)<br>score = r2_score(Y_test, Y_pred)<br>mse = mean_squared_error(Y_test, Y_pred)<br><br>print(f&quot;R2 Score: {score:.4f}&quot;)<br>print(f&quot;Mean Squared Error: {mse:.4f}&quot;)</pre><h3>Part 4: The “Pro” Analysis — Joint Feature Selection</h3><p>Why use MultiTaskElasticNet instead of just running two separate ElasticNet models (one for each target)?</p><p>The answer is <strong>Row Sparsity</strong>. Let’s visualize the coefficients to prove that the model selected the <em>same</em> features for both tasks.</p><pre># Let&#39;s look at the coefficients for the first 5 features<br>print(&quot;Coefficients for first 5 features:&quot;)<br>print(model.coef_[:, :5]) # Transposed for easier reading here<br><br># Visualization code to prove Joint Sparsity<br>plt.figure(figsize=(10, 6))<br><br># Plot coefficients for Task 1<br>plt.plot(model.coef_[0], label=&#39;Task 1 Coefs&#39;, marker=&#39;o&#39;, linestyle=&#39;--&#39;)<br><br># Plot coefficients for Task 2<br>plt.plot(model.coef_[1], label=&#39;Task 2 Coefs&#39;, marker=&#39;x&#39;, linestyle=&#39;-&#39;)<br><br>plt.title(&quot;Multi-task Elastic-Net Coefficients&quot;)<br>plt.xlabel(&quot;Feature Index&quot;)<br>plt.ylabel(&quot;Coefficient Value&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><p><strong>Interpretation of the Plot:</strong></p><p>If you run this code, you will notice something specific: whenever the blue line (Task 1) is at <strong>zero</strong>, the orange line (Task 2) is <strong>also at zero</strong>.</p><ul><li>The model decided: “Feature 3 is useless for Task 1, so I will force it to be useless for Task 2 as well.”</li><li>This creates a more robust model when tasks are related.</li></ul><h3>Part 5: Automatic Tuning with Cross-Validation</h3><p>A professional never guesses alpha and l1_ratio. We use <strong>Cross-Validation</strong> to find the optimal numbers. Scikit-learn provides MultiTaskElasticNetCV for this exact purpose.</p><pre>from sklearn.linear_model import MultiTaskElasticNetCV<br><br># Define a list of l1_ratios to try<br>l1_ratios = [0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1]<br><br># Initialize the CV model<br># cv=5 means 5-fold cross-validation<br>cv_model = MultiTaskElasticNetCV(l1_ratio=l1_ratios, cv=5, random_state=42, n_jobs=-1)<br><br># Fit (this takes longer as it trains many models)<br>cv_model.fit(X_train, Y_train)<br><br>print(f&quot;Best Alpha found: {cv_model.alpha_}&quot;)<br>print(f&quot;Best L1 Ratio found: {cv_model.l1_ratio_}&quot;)<br><br># Predict using the best model automatically<br>y_pred_cv = cv_model.predict(X_test)<br>print(f&quot;tuned R2 Score: {r2_score(Y_test, y_pred_cv):.4f}&quot;)</pre><h3>Part 6: Important Parameters &amp; Attributes Reference</h3><p>Here is a breakdown of the MultiTaskElasticNet class arguments for fine-tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*arCuACw62Wqn0Dxr1rIHjw.png" /></figure><h3>Part 7: When to use this? (Pro Tips)</h3><p>Do not use Multi-task Elastic-Net blindly.</p><p><strong>✅ USE IT WHEN:</strong></p><ol><li><strong>Correlated Targets:</strong> The things you are predicting (Y1,Y2Y_1, Y_2Y1​,Y2​) are related (e.g., Temperature and Humidity).</li><li><strong>Shared Features:</strong> You suspect that the subset of relevant features is the same for all tasks.</li><li><strong>High Dimensionality:</strong> You have many features (columns), perhaps more than samples (n_features&gt;n_samplesn\_features &gt; n\_samplesn_features&gt;n_samples).</li><li><strong>Collinearity:</strong> Your input features XXX are correlated with each other (Elastic-Net handles this better than Lasso).</li></ol><p><strong>❌ DO NOT USE IT WHEN:</strong></p><ol><li><strong>Targets are Independent:</strong> If Task A has nothing to do with Task B, treating them together introduces noise and bias. Use separate Single-task models.</li><li><strong>Different Features Matter:</strong> If Feature 1 drives Task A, but Feature 2 drives Task B, the Multi-task constraint (forcing rows to zero) will hurt performance.</li></ol><h3>Summary Code Snippet (The Template)</h3><p>Here is a clean template for your projects:</p><pre>from sklearn.linear_model import MultiTaskElasticNetCV<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.pipeline import make_pipeline<br><br># 1. Prepare Data<br># X_train, Y_train...<br><br># 2. Create Pipeline (Scaling is crucial for ElasticNet)<br>pipeline = make_pipeline(<br>    StandardScaler(),<br>    MultiTaskElasticNetCV(cv=5, l1_ratio=[.1, .5, .7, .9, .95, .99], n_jobs=-1)<br>)<br><br># 3. Train<br>pipeline.fit(X_train, Y_train)<br><br># 4. Results<br>best_model = pipeline.named_steps[&#39;multitaskelasticnetcv&#39;]<br>print(f&quot;Best alpha: {best_model.alpha_}&quot;)<br>print(f&quot;Score: {pipeline.score(X_test, Y_test)}&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71dc7f953b0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 6): Elastic-Net regression and classification]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-6-elastic-net-regression-and-classification-49a28876eead?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/49a28876eead</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Sat, 31 Jan 2026 04:35:41 GMT</pubDate>
            <atom:updated>2026-01-31T04:35:41.618Z</atom:updated>
            <content:encoded><![CDATA[<h4>Welcome back! In our previous sessions, we explored:</h4><ul><li><strong>Linear Regression:</strong> Our baseline model that tries to fit a line by minimizing the sum of squared errors.</li><li><strong>Ridge Regression (L2 Regularization):</strong> A modification that adds a penalty to prevent coefficients from becoming too large, which helps reduce model complexity and overfitting. It shrinks coefficients towards zero but never makes them <em>exactly</em> zero.</li><li><strong>Lasso Regression (L1 Regularization):</strong> Another modification that adds a penalty. This penalty is special because it can shrink some coefficients to be <em>exactly</em> zero, effectively performing automatic feature selection.</li></ul><p><strong>Now, let’s look at the limitations of Ridge and Lasso to understand why Elastic-Net was created.</strong></p><ul><li><strong>Lasso’s Weakness:</strong> When you have a group of features that are highly correlated (e.g., a person’s height in centimeters and height in inches), Lasso tends to be a bit unstable. It might arbitrarily pick one of these features and set the coefficients of the others to zero. This isn’t always ideal, as the other correlated features might still contain useful, albeit redundant, information.</li><li><strong>Ridge’s Weakness:</strong> Ridge will shrink the coefficients of correlated features together, which is good. However, it will never eliminate any features. If you have 10,000 features and only 100 are actually useful, Ridge will keep all 10,000, giving the useless ones very small coefficients. This doesn’t help in simplifying the model.</li></ul><p>This is where Elastic-Net comes in. It’s the diplomatic solution that combines the strengths of both Ridge and Lasso.</p><h3>1. What is Elastic-Net? The Core Idea</h3><p>In simple terms, <strong>Elastic-Net is a linear regression model that is trained with both L1 and L2 regularization penalties.</strong></p><p>Think of it like a manager hiring for a team:</p><ul><li><strong>Linear Regression:</strong> Hires every single applicant, no matter how unqualified. The team is bloated and performance is mediocre.</li><li><strong>Lasso (L1):</strong> A ruthless manager. If two applicants have very similar skills, the manager hires one and fires the other immediately. This is efficient but might discard someone who was almost as good.</li><li><strong>Ridge (L2):</strong> A cautious manager. Hires everyone but gives them all a very low salary (shrinks coefficients). Nobody gets fired, but the impact of each individual is reduced.</li><li><strong>Elastic-Net:</strong> A balanced manager. It first uses the Lasso approach to fire the completely unqualified applicants (feature selection). Then, for the remaining group of qualified applicants who have similar skills, it uses the Ridge approach to balance their importance, making them work together as a group.</li></ul><p><strong>This combined approach allows Elastic-Net to handle datasets with a large number of features, especially when some of those features are correlated with each other. It performs feature selection (like Lasso) while also stabilizing the model when faced with correlated predictors (like Ridge).</strong></p><h3>2. The Math Behind Elastic-Net (Simplified)</h3><p>To truly understand Elastic-Net, we need to look at its cost function. The goal of the model is to find the coefficients (w) that minimize this cost.</p><p>Remember the cost functions for Ridge and Lasso:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*yWpOqjDLAcfvC6tcVG-A7A.png" /></figure><p><strong>This looks complicated, but it’s very logical. Let’s break down the new parameters:</strong></p><p>By tuning both alpha and l1_ratio, you have fine-grained control over your model’s regularization.</p><h3>3. Scikit-learn Implementation: From Zero to Pro</h3><p>Let’s build an Elastic-Net model step-by-step.</p><h4>Step 1: Setup and Data Creation</h4><p>We’ll create a synthetic dataset perfect for showing Elastic-Net’s strengths. We will create a dataset with several features, where some are highly correlated. (y_train): {y_train.shape}&quot;)el and fit it to our training data. We need to provide values for alpha and l1_ratio.</p><pre># Instantiate the ElasticNet model<br># alpha controls the overall strength of the penalty.<br># l1_ratio controls the mix between L1 and L2. l1_ratio=0.5 means a 50/50 mix.<br>elastic_model = ElasticNet(alpha=0.1, l1_ratio=0.5, random_state=42)<br><br># Train the model on the training data<br>elastic_model.fit(X_train, y_train)</pre><h4>Step 3: Inspecting the Model and Making Predictions</h4><p>Now that the model is trained, let’s see what it learned. We’ll look at the coefficients and evaluate its performance.</p><pre># Get the learned coefficients<br>coefficients = elastic_model.coef_<br><br># Count how many coefficients were shrunk to exactly zero<br>zero_coeffs = np.sum(coefficients == 0)<br>non_zero_coeffs = np.sum(coefficients != 0)<br><br>print(f&quot;Total features: {len(coefficients)}&quot;)<br>print(f&quot;Number of coefficients shrunk to zero: {zero_coeffs}&quot;)<br>print(f&quot;Number of coefficients kept: {non_zero_coeffs}&quot;)<br><br># Make predictions on the test set<br>y_pred = elastic_model.predict(X_test)<br><br># Evaluate the model<br>mse = mean_squared_error(y_test, y_pred)<br>print(f&quot;\nMean Squared Error on the test set: {mse:.2f}&quot;)<br><br># You can also inspect the intercept<br>print(f&quot;Model Intercept: {elastic_model.intercept_:.2f}&quot;)</pre><p>Notice how Elastic-Net successfully performed feature selection by setting many coefficients to zero, just like Lasso!</p><h4>Step 4: The Professional Way — Hyperparameter Tuning with ElasticNetCV</h4><p>Choosing alpha and l1_ratio by hand is just guessing. A professional approach is to let the data tell us the best values. Scikit-learn provides a fantastic tool for this: ElasticNetCV. The “CV” stands for Cross-Validation.</p><p>This class performs an exhaustive search over a grid of alpha and l1_ratio values and finds the best combination using cross-validation.</p><pre># Import ElasticNetCV<br>from sklearn.linear_model import ElasticNetCV<br><br># Define the l1_ratios to test. Let&#39;s try a few values to see the effect.<br>l1_ratios_to_try = [0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1]<br><br># Instantiate ElasticNetCV<br># It will automatically find the best alpha from a list of possible values.<br># We provide it the list of l1_ratios to try.<br># cv=5 means 5-fold cross-validation.<br>elastic_cv_model = ElasticNetCV(l1_ratio=l1_ratios_to_try, cv=5, random_state=42, max_iter=10000)<br><br># Train the model<br>elastic_cv_model.fit(X_train, y_train)<br><br># Get the best parameters found by the cross-validation search<br>best_alpha = elastic_cv_model.alpha_<br>best_l1_ratio = elastic_cv_model.l1_ratio_<br><br>print(f&quot;Best alpha found: {best_alpha:.4f}&quot;)<br>print(f&quot;Best l1_ratio found: {best_l1_ratio}&quot;)<br><br># Evaluate the tuned model<br>y_pred_cv = elastic_cv_model.predict(X_test)<br>mse_cv = mean_squared_error(y_test, y_pred_cv)<br>print(f&quot;\nMean Squared Error from the CV-tuned model: {mse_cv:.2f}&quot;)<br><br># Inspect the coefficients of the tuned model<br>tuned_coeffs = elastic_cv_model.coef_<br>print(f&quot;Number of non-zero coefficients in tuned model: {np.sum(tuned_coeffs != 0)}&quot;)</pre><p>Using ElasticNetCV is the standard and most robust way to use Elastic-Net in practice. It takes the guesswork out of hyperparameter tuning.</p><h3>4. Summary: Pros, Cons, and When to Use</h3><h4>Pros of Elastic-Net</h4><ul><li><strong>Best of Both Worlds:</strong> Combines the feature selection capability of Lasso with the correlated feature handling of Ridge.</li><li><strong>Handles Correlated Features:</strong> Unlike Lasso, it tends to select groups of correlated features together rather than arbitrarily picking one.</li><li><strong>Very Flexible:</strong> By tuning the l1_ratio, you can smoothly transition between a Ridge-like model and a Lasso-like model.</li><li><strong>Excellent for High-Dimensional Data:</strong> It’s a go-to model when the number of features (ppp) is much larger than the number of samples (nnn).</li></ul><h4>Cons of Elastic-Net</h4><ul><li><strong>Two Hyperparameters:</strong> You need to tune both alpha and l1_ratio, which makes the tuning process more complex and computationally expensive than for Ridge or Lasso alone (though ElasticNetCV makes this easy).</li><li><strong>Interpretability:</strong> While it performs feature selection, the shrunken, non-zero coefficients can still be less interpretable than those from a simple linear regression.</li></ul><h4>When should you use Elastic-Net?</h4><ul><li><strong>As a default choice for regularization:</strong> When you’re unsure whether Ridge or Lasso will perform better, Elastic-Net is a safe and powerful bet. By tuning the l1_ratio, you let the data decide which penalty is more important.</li><li>When you have <strong>many features</strong> and you suspect some are redundant or correlated.</li><li>When you want to perform <strong>feature selection</strong> but are worried about the instability of Lasso with correlated data.</li><li>In fields like <strong>bioinformatics</strong> (gene analysis) or <strong>finance</strong>, where datasets often have more features than samples.</li></ul><p>That’s it for our deep dive into Elastic-Net! You now have a very powerful tool in your regression toolkit.</p><p><strong>Next up:</strong> We will start looking at a completely different family of models: <strong>Classification Models</strong>, starting with Logistic Regression.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49a28876eead" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 5): Multi-task Lasso regression and classification]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-5-multi-task-lasso-regression-and-classification-c8cb2c0c38e3?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8cb2c0c38e3</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 06:39:11 GMT</pubDate>
            <atom:updated>2026-01-29T06:39:11.189Z</atom:updated>
            <content:encoded><![CDATA[<h4>The “Why?” — Beyond a Single Prediction</h4><p>So far, all the models we’ve looked at (OLS, Ridge, Lasso) have been designed for a single task. We have a set of features X and we want to predict a single target variable y.</p><ul><li><strong>Example:</strong> Predict a house’s price (y) using features like square_footage, num_bedrooms, etc. (X).</li></ul><p>But what if you want to predict <strong>multiple, related things at the same time</strong> using the same set of features?</p><ul><li><strong>Example:</strong> Imagine you are a materials scientist. You have a set of features for a new alloy (X), like its composition and processing temperature. You want to simultaneously predict its hardness, tensile_strength, and conductivity (let’s call them y1, y2, y3).</li></ul><p><strong>The Simple Approach:</strong> You could just train three separate Lasso models:</p><ol><li>Lasso model for hardness: X -&gt; y1</li><li>Lasso model for tensile_strength: X -&gt; y2</li><li>Lasso model for conductivity: X -&gt; y3</li></ol><p><strong>The Problem with the Simple Approach:</strong> This method completely ignores the fact that hardness, tensile_strength, and conductivity are likely related. They are all outcomes of the same underlying physical properties of the alloy. By training separate models, you lose the opportunity to <strong>share information</strong> between the tasks. One model might decide feature_5 is important, while another might ignore it due to random noise in its specific target data. This can lead to inconsistent and less robust feature selection.</p><p><strong>The Core Idea:</strong> Multi-task Lasso assumes that if a feature is important for one task, it’s likely important for the other related tasks as well. It enforces a rule: <strong>a feature is either selected for ALL tasks, or it’s dropped for ALL tasks.</strong> It performs joint feature selection across multiple regression problems.</p><h4>Part 2: The Core Idea of Multi-task Lasso</h4><p>Multi-task Lasso modifies the Lasso cost function to work with a matrix of coefficients W (where each column corresponds to a task) and a matrix of targets Y.</p><p>The penalty is the crucial part. It’s no longer the sum of the absolute values of each individual coefficient. Instead, it uses what is called a <strong>mixed L2,1 norm</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*tMqfXG1VdwaFHxyYSs0qkw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*8iD0EbFskK3TL7JcG-g3bw.png" /></figure><p><strong>The consequence of this penalty:</strong> The L1 part of the penalty encourages sparsity (setting things to zero), just like in regular Lasso. But what is being set to zero? Not individual coefficients, but the <strong>entire group (the L2 norm) of coefficients for a specific feature.</strong> If the model decides that feature j is not important, the only way to make its penalty term zero is to make <strong>all of its coefficients across all tasks (Wj1,Wj2,…) equal to zero simultaneously.</strong></p><p>This enforces <strong>group sparsity at the feature level.</strong></p><h4>Part 3: Multi-task Lasso in Scikit-Learn</h4><p>Let’s prove this with code. We will create a scenario with 20 features, but only 2 of them are useful for predicting our 3 target variables.</p><p><strong>Step 1: Setup and Data Creation</strong></p><p>Let’s prove this with code. We will create a scenario with 20 features, but only 2 of them are useful for predicting our 3 target variables.</p><p><strong>Step 1: Setup and Data Creation</strong></p><pre># Import necessary libraries<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.linear_model import MultiTaskLassoCV, LassoCV<br><br># Set a seed for reproducibility<br>np.random.seed(0)<br><br># --- Create Synthetic Data ---<br>n_samples, n_features, n_tasks = 100, 20, 3<br><br># Generate random features<br>X = np.random.randn(n_samples, n_features)<br><br># Create a &quot;true&quot; coefficient matrix W.<br># Only the first 2 features are relevant for all 3 tasks.<br># The other 18 features are useless (all their coefficients are 0).<br>W_true = np.zeros((n_features, n_tasks))<br>W_true[0, :] = [1.5, 0.5, -1.0] # Coefficients for feature 0 across 3 tasks<br>W_true[1, :] = [-2.0, 1.0, 0.8] # Coefficients for feature 1 across 3 tasks<br><br># Generate the three target variables Y using the same features + noise<br># Y = X @ W_true does the matrix multiplication<br>Y = X @ W_true + 0.5 * np.random.randn(n_samples, n_tasks)<br><br>print(f&quot;Shape of X: {X.shape}&quot;)<br>print(f&quot;Shape of Y: {Y.shape}&quot;)<br>print(f&quot;Shape of True Coefficient Matrix W: {W_true.shape}&quot;)</pre><p><strong>Step 2: Train a Multi-task Lasso and Separate Lasso Models</strong></p><p>We will compare the MultiTaskLassoCV model against training three separate LassoCV models.</p><pre># --- Train the Multi-task Lasso Model ---<br># It finds the best alpha using cross-validation.<br>mt_lasso_cv = MultiTaskLassoCV(cv=5, random_state=42)<br>mt_lasso_cv.fit(X, Y)<br><br># --- For comparison, train 3 separate Lasso models ---<br>lasso_models = []<br>for i in range(n_tasks):<br>    lasso_cv = LassoCV(cv=5, random_state=42)<br>    lasso_cv.fit(X, Y[:, i]) # Fit on the i-th column (task) of Y<br>    lasso_models.append(lasso_cv)</pre><p><strong>Step 3: Compare the Resulting Coefficient Matrices</strong></p><p>Let’s see what features each approach selected.</p><pre># Get the learned coefficient matrix from Multi-task Lasso<br>W_mt_lasso = mt_lasso_cv.coef_<br><br># Build the coefficient matrix from the 3 separate Lasso models<br>W_separate_lasso = np.zeros_like(W_true)<br>for i in range(n_tasks):<br>    W_separate_lasso[:, i] = lasso_models[i].coef_<br><br># --- Visualize the results ---<br>fig, axes = plt.subplots(1, 3, figsize=(15, 5), sharey=True)<br><br># Plot True Coefficients<br>im1 = axes[0].imshow(W_true, aspect=&#39;auto&#39;, cmap=&#39;viridis&#39;)<br>axes[0].set_title(&quot;True Coefficients&quot;)<br>axes[0].set_xlabel(&quot;Tasks&quot;)<br>axes[0].set_ylabel(&quot;Features&quot;)<br><br># Plot Multi-task Lasso Coefficients<br>im2 = axes[1].imshow(W_mt_lasso.T, aspect=&#39;auto&#39;, cmap=&#39;viridis&#39;) # Transposed for easier access<br>axes[1].set_title(&quot;Multi-task Lasso Coefficients&quot;)<br>axes[1].set_xlabel(&quot;Tasks&quot;)<br><br># Plot Separate Lasso Coefficients<br>im3 = axes[2].imshow(W_separate_lasso, aspect=&#39;auto&#39;, cmap=&#39;viridis&#39;)<br>axes[2].set_title(&quot;Separate Lasso Coefficients&quot;)<br>axes[2].set_xlabel(&quot;Tasks&quot;)<br><br>fig.colorbar(im1, ax=axes.ravel().tolist(), shrink=0.7)<br>plt.suptitle(&quot;Comparison of Coefficient Matrices&quot;)<br>plt.show()<br><br># Print the number of non-zero features selected by each model for the first task<br>print(&quot;--- Feature Selection Comparison (for Task 0) ---&quot;)<br>print(f&quot;Features selected by Multi-task Lasso: {np.sum(mt_lasso_cv.coef_[:, 0] != 0)}&quot;)<br>print(f&quot;Features selected by Separate Lasso: {np.sum(W_separate_lasso[:, 0] != 0)}&quot;)</pre><p><strong>Expected Output:</strong></p><ul><li>The visualization for “True Coefficients” will show non-zero values only in the first two rows.</li><li>The “Multi-task Lasso Coefficients” plot will look very similar. It will have correctly identified that only the first two features are important and will have set all other 18 rows to zero. <strong>The sparsity pattern will be clean and consistent across all tasks.</strong></li><li>The “Separate Lasso Coefficients” plot might be messy. Due to noise, one of the models might have mistakenly selected a useless feature (e.g., feature 10) for one of the tasks. The sparsity pattern will not be uniform across tasks.</li></ul><h4>Part 4: When to Use Multi-task Lasso? (Real-World Applications)</h4><p>Multi-task Lasso is not for every problem, but it’s a superstar in specific domains where you have related outputs:</p><ol><li><strong>Bioinformatics / Genomics:</strong> Predicting multiple gene expression levels or disease markers from the same set of genetic data (SNPs). The underlying biological pathways are shared, so you expect the same genes to be relevant for related outcomes.</li><li><strong>Neuroimaging (fMRI):</strong> In magnetoencephalography (MEG), you measure brain signals from hundreds of sensors over time. A common task is to identify which brain regions (features) are active. The signal at time t and t+1 are two related tasks. Multi-task Lasso can be used to find a sparse set of active regions whose activity persists over time.</li><li><strong>Economics:</strong> Predicting a set of related economic indicators (e.g., GDP for different countries in the EU) from a common set of global economic features.</li></ol><h3>Summary of sklearn.linear_model.MultiTaskLasso</h3><ul><li><strong>What it is:</strong> An extension of Lasso for problems with multiple, related target variables.</li><li><strong>Main Purpose:</strong> To perform <strong>joint feature selection</strong> across several regression tasks simultaneously, assuming they share the same underlying sparse set of important features.</li><li><strong>Key Parameter:</strong></li><li>alpha: The regularization strength. A larger alpha leads to more features being eliminated across all tasks.</li><li><strong>How to Choose Alpha:</strong> Use MultiTaskLassoCV for automatic selection via cross-validation.</li><li><strong>Key Advantage:</strong> By “borrowing strength” across tasks, it can achieve better feature selection and model robustness than training independent models, especially when the number of samples is limited.</li><li><strong>The “Magic”:</strong> Uses a mixed L2,1 norm penalty that forces entire rows of the coefficient matrix (all coefficients for a single feature) to be zero all at once.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8cb2c0c38e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 3):Ridge regression and classification]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-3-ridge-regression-and-classification-78c7c07cdba9?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/78c7c07cdba9</guid>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 06:19:19 GMT</pubDate>
            <atom:updated>2026-01-29T06:26:56.461Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scikit-learn: From Zero to Hero (Part 4): Lasso regression and classification</h3><h4>The “Why?” — A Limitation of Ridge Regression</h4><p>In our last lesson, we saw that Ridge Regression is fantastic for making models more stable and handling multicollinearity. It does this by adding a penalty( Eq. 1) that shrinks coefficients towards zero.</p><figure><img alt="Eq. 1" src="https://cdn-images-1.medium.com/max/84/1*QcTaB1fR7yiJUF4Nf81MHw.png" /><figcaption>Eq. 1</figcaption></figure><p>However, Ridge has one key characteristic: it <strong>shrinks</strong> the coefficients but <strong>never makes them exactly zero</strong> (unless alpha is infinite). This means that even if a feature is completely useless and has no relationship with the target variable, Ridge Regression will still assign it a very small, non-zero coefficient. <strong>The Question:</strong> What if we have a dataset with hundreds or thousands of features, and we suspect that many of them are completely irrelevant? We want a model that can not only shrink coefficients but also perform <strong>feature selection</strong> by eliminating useless features entirely. This is where Lasso comes in.</p><h4>Part 2: The Core Idea of Lasso Regression</h4><p>Lasso looks almost identical to Ridge, but with one critical change in the penalty term. The Lasso model’s goal is to minimize this cost function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*y3Eh_UvvU1jsWbmlt6Fp_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*MbugKpFqLTPHpUfHYozZsA.png" /></figure><p>This might seem like a small change, but it has a profound mathematical and practical consequence. The L1 penalty is structured in a way that it is able to force some of the coefficients to be <strong>exactly zero</strong>.</p><p><strong>The Simple Analogy:</strong></p><p>Imagine you have a budget for your model’s coefficients.</p><ul><li><strong>Ridge (L2)</strong> is like saying, “Try to spend as little as possible on all features, but every feature must get at least a tiny fraction of a cent.” It distributes the budget.</li><li><strong>Lasso (L1)</strong> is like saying, “Spend your budget only on the most important features and completely ignore the rest (spend zero on them).” It concentrates the budget.</li></ul><p>By setting some coefficients to zero, Lasso effectively removes those features from the model, creating what is called a <strong>sparse model</strong>. A sparse model is one where many of the coefficients are zero. This makes the model simpler and easier to interpret.</p><h4>Part 3: Lasso in Scikit-Learn</h4><p>Let’s demonstrate Lasso’s feature selection capability. We will create a dataset where only 2 out of 10 features are actually useful.</p><p><strong>Step 1: Setup and Data Creation</strong></p><p>We’ll create data where y depends only on X1 and X2. The other 8 features will just be random noise.</p><pre># Import necessary libraries<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import pandas as pd<br>from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV<br><br># Set a seed for reproducibility<br>np.random.seed(42)<br><br># --- Create Synthetic Data with Useless Features ---<br>n_samples = 100<br>n_features = 10<br><br># Generate all features from a random distribution<br>X = np.random.randn(n_samples, n_features)<br><br># The &quot;true&quot; coefficients. Only the first two are non-zero.<br>true_coefficients = np.array([1.5, -2.5, 0, 0, 0, 0, 0, 0, 0, 0])<br><br># y is a linear combination of ONLY the first two features, plus some noise<br>y = X @ true_coefficients + 0.5 * np.random.randn(n_samples)<br><br># Note: X @ true_coefficients is a fast way to do matrix multiplication for:<br># y = X[:,0]*1.5 + X[:,1]*-2.5 + ... + X[:,9]*0 + noise</pre><p><strong>Step 2: Train OLS, Ridge, and Lasso Models</strong></p><p>We will train all three models to see how each one handles the useless features. We’ll use RidgeCV and LassoCV to let them find their own best alpha.</p><pre># --- Train OLS Model ---<br>ols_reg = LinearRegression()<br>ols_reg.fit(X, y)<br><br># --- Train Ridge Model (with Cross-Validation) ---<br># We provide a range of alphas for RidgeCV to test<br>ridge_cv = RidgeCV(alphas=np.logspace(-6, 6, 13))<br>ridge_cv.fit(X, y)<br><br># --- Train Lasso Model (with Cross-Validation) ---<br># LassoCV will also find the best alpha automatically<br>lasso_cv = LassoCV(cv=5, random_state=42)<br>lasso_cv.fit(X, y)</pre><p><strong>Step 3: Compare the Coefficients</strong></p><p>This is where Lasso’s power becomes obvious.</p><pre># Print the results<br>print(&quot;--- True Coefficients ---&quot;)<br>print(true_coefficients)<br>print(&quot;\n--- OLS Coefficients ---&quot;)<br>print(np.round(ols_reg.coef_, 2))<br>print(&quot;\n--- RidgeCV Coefficients (best alpha={:.4f}) ---&quot;.format(ridge_cv.alpha_))<br>print(np.round(ridge_cv.coef_, 2))<br>print(&quot;\n--- LassoCV Coefficients (best alpha={:.4f}) ---&quot;.format(lasso_cv.alpha_))<br>print(np.round(lasso_cv.coef_, 2))<br><br># Visualize the coefficients<br>plt.figure(figsize=(12, 6))<br>plt.plot(ols_reg.coef_, &#39;o-&#39;, label=&#39;OLS&#39;, markersize=8)<br>plt.plot(ridge_cv.coef_, &#39;s-&#39;, label=&#39;Ridge&#39;, markersize=8)<br>plt.plot(lasso_cv.coef_, &#39;x-&#39;, label=&#39;Lasso&#39;, markersize=8)<br>plt.plot(true_coefficients, &#39;k--&#39;, label=&#39;True coefficients&#39;)<br>plt.xlabel(&quot;Coefficient Index&quot;)<br>plt.ylabel(&quot;Coefficient Value&quot;)<br>plt.title(&quot;Comparison of Coefficients from OLS, Ridge, and Lasso&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><p><strong>Expected Output:</strong></p><ul><li><strong>OLS Coefficients:</strong> The first two coefficients will be close to 1.5 and -2.5, but the other “useless” features will have small, non-zero coefficients. The model is trying to use the noise to explain the target.</li><li><strong>Ridge Coefficients:</strong> All 10 coefficients will be non-zero. The first two will be shrunk a little towards zero, and the coefficients for the noisy features will be even smaller than in OLS, but still not exactly zero.</li><li><strong>Lasso Coefficients:</strong> The first two coefficients will be close to the true values (though also shrunk a bit). Critically, the coefficients for the 8 useless features will be <strong>exactly 0.0</strong>. Lasso has successfully performed feature selection!</li></ul><h4>Part 4: Ridge vs. Lasso — When to Use Which?</h4><p>This is a common interview question and a practical consideration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*GcqehG1j2bwYKYFBSNH2hQ.png" /></figure><p><strong>A Glimpse into the Future:</strong> What if you want a bit of both? A model that can handle correlated features well (like Ridge) but also perform some feature selection (like Lasso)? That’s where our next model, <strong>Elastic Net</strong>, comes in. It’s a hybrid that combines both L1 and L2 penalties.</p><h3>Summary of sklearn.linear_model.Lasso</h3><ul><li><strong>What it is:</strong> A linear regression model that includes an L1 regularization penalty.</li><li><strong>Main Purpose:</strong> To create a <strong>sparse model</strong> by performing automatic feature selection, which can improve interpretability and reduce overfitting.</li><li><strong>Key Parameter:</strong></li><li>alpha: The regularization strength. A larger alpha leads to more coefficients being set to zero.</li><li><strong>How to Choose Alpha:</strong> Use LassoCV to find the optimal alpha using cross-validation.</li><li><strong>Strengths:</strong></li><li>Automatic feature selection is its superpower.</li><li>The resulting models are simple and easy to explain.</li><li><strong>Weaknesses:</strong></li><li>Can struggle with highly correlated features, sometimes picking one at random.</li><li>If the number of features is greater than the number of samples, Lasso can only select at most n_samples features.</li></ul><p>You’ve now learned the three foundational linear models in regression: OLS, Ridge, and Lasso. Understanding the trade-offs between them is a core skill in machine learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78c7c07cdba9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 3):Ridge regression and classification]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-3-ridge-regression-and-classification-6e2408d226b2?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e2408d226b2</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 11:07:58 GMT</pubDate>
            <atom:updated>2025-10-13T11:09:00.407Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2: The “Why?” — Problems with Ordinary Least Squares (OLS)</h4><p>In our last session, we learned that OLS finds the line that minimizes the sum of squared errors. It does a fantastic job, but it has two major weaknesses:</p><p><a href="https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-2-ordinary-least-squares-88f3b937f897">https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-2-ordinary-least-squares-88f3b937f897</a></p><ol><li><strong>High Sensitivity to Data (High Variance):</strong> OLS can be <em>too</em> sensitive to the training data. If we slightly change our training data (e.g., add or remove a few points), the best-fit line found by OLS can change dramatically. This means the model might fit the training data perfectly but perform poorly on new, unseen data. This problem is called <strong>overfitting</strong>. The model learns the <em>noise</em> in the training data, not just the underlying trend.</li><li><strong>Multicollinearity:</strong> This is a big one. As mentioned before, if two or more of your input features are highly correlated (e.g., ‘house age’ and ‘year built’), OLS becomes unstable. The math behind it can “blow up,” leading to ridiculously large coefficient values (w1,w2,…). A large coefficient suggests that a tiny change in a feature leads to a massive change in the output, which is usually a sign of an unstable, overfitted model.</li></ol><p><strong>The Core Problem:</strong> OLS only cares about one thing: minimizing the error on the training data. It has no mechanism to keep the model’s coefficients “sensible” or “small.”</p><h4>Part 2: The Core Idea of Ridge Regression</h4><p>Ridge Regression starts with the same goal as OLS — to minimize the sum of squared errors — but it adds a crucial <strong>penalty</strong>.</p><p>Think of it like a parent telling a child, “You can have dessert (minimize the error), but you also have to finish your vegetables (keep the coefficients small).”</p><p>The Ridge model’s goal is to minimize a new cost function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*oPvwe-DB6n-tJKu_HiB3DQ.png" /></figure><p>Let’s break down that new part:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*jUtjeE7tgTbvF7j-5CNWbw.png" /></figure><p><strong>In simple terms:</strong> Ridge Regression “shrinks” the coefficients towards zero. This makes the model less sensitive to the noise in the training data and more stable in the presence of multicollinearity. It trades a little bit of bias (the fit on the training data might be slightly worse than OLS) for a big decrease in variance (the model will perform much better on new data). This is known as the <strong>bias-variance tradeoff</strong>.</p><h4>Part 3: Ridge Regression in Scikit-Learn</h4><p>Let’s see this in action. We’ll create a situation where OLS struggles and Ridge shines: a dataset with multicollinearity.</p><p><strong>Step 1: Setup and Data Creation</strong></p><p>We’ll create two features, X1 and X2, that are highly correlated with each other.</p><pre># Import necessary libraries<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import pandas as pd<br>from sklearn.linear_model import LinearRegression, Ridge<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import mean_squared_error<br><br># Set a seed for reproducibility<br>np.random.seed(0)<br><br># --- Create Synthetic Data with Multicollinearity ---<br># We&#39;ll create 100 data points and 2 features.<br>n_samples = 100<br>X1 = 2 * np.random.rand(n_samples, 1)<br><br># X2 will be very similar to X1, plus a tiny bit of noise. This creates high correlation.<br>X2 = X1 + 0.1 * np.random.randn(n_samples, 1)<br><br># Combine them into our feature matrix X<br>X = np.c_[X1, X2] # np.c_ is a handy way to concatenate arrays column-wise<br><br># y will be our target variable. Let&#39;s say the true relationship is y = 2 + 1*X1 + 3*X2 + noise<br>y = 2 + 1*X1 + 3*X2 + np.random.randn(n_samples, 1)<br><br># Let&#39;s check the correlation<br>df_for_corr = pd.DataFrame(data=X, columns=[&#39;X1&#39;, &#39;X2&#39;])<br>print(&quot;--- Correlation Matrix ---&quot;)<br>print(df_for_corr.corr())<br>print(&quot;\nNotice the very high correlation between X1 and X2 (close to 1.0).\n&quot;)</pre><p><strong>Step 2: Train OLS and Ridge Models</strong></p><p>Now, let’s train both a standard LinearRegression (OLS) model and a Ridge model on this data and compare their coefficients.</p><pre># Split the data<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># --- Train OLS Model ---<br>ols_reg = LinearRegression()<br>ols_reg.fit(X_train, y_train)<br><br># --- Train Ridge Model ---<br># We need to choose a value for alpha. Let&#39;s start with alpha=1.<br>ridge_reg = Ridge(alpha=1)<br>ridge_reg.fit(X_train, y_train)<br><br># A Ridge model with a very large alpha<br>ridge_reg_high_alpha = Ridge(alpha=100)<br>ridge_reg_high_alpha.fit(X_train, y_train)</pre><p><strong>Step 3: Compare the Coefficients</strong></p><p>This is where the magic happens. Let’s look at the coefficients found by each model. Remember, the true coefficients we set were [1, 3].</p><pre>print(&quot;--- Model Coefficients ---&quot;)<br>print(f&quot;True Coefficients used to generate data: [1.0, 3.0]&quot;)<br>print(f&quot;OLS Coefficients: {ols_reg.coef_[0]}&quot;)<br>print(f&quot;Ridge (alpha=1) Coefficients: {ridge_reg.coef_[0]}&quot;)<br>print(f&quot;Ridge (alpha=100) Coefficients: {ridge_reg_high_alpha.coef_[0]}&quot;)</pre><p><strong>What you will see:</strong></p><ul><li>The <strong>OLS coefficients</strong> will likely be large and potentially far from the true values of [1, 3]. For example, you might get something like [-10.5, 14.7]. The model is unstable due to multicollinearity. It’s assigning a large negative weight to one feature and a large positive weight to the other to compensate.</li><li>The <strong>Ridge (alpha=1) coefficients</strong> will be much smaller and closer to the true values, for example, [1.3, 2.9]. The penalty term has “shrunk” the coefficients, making the model more stable and realistic.</li><li>The <strong>Ridge (alpha=100) coefficients</strong> will be even smaller, for example, [0.4, 0.6]. The very high penalty has forced the coefficients to be very close to zero.</li><li><strong>Step 4: Finding the Best Alpha (Hyperparameter Tuning)</strong></li><li>How do we know if alpha=1 or alpha=10 or alpha=0.1 is the best? We can’t just guess. This is a classic <strong>hyperparameter tuning</strong> problem.</li><li>Scikit-learn provides an incredibly useful model called RidgeCV which performs <strong>Cross-Validation</strong> to automatically find the best alpha for you.</li><li><strong>How it works:</strong> RidgeCV will test a range of alpha values. For each alpha, it splits the training data into several “folds” (e.g., 5). It trains the model on 4 folds and tests it on the 5th fold, rotating which fold is the test set. It calculates the average error across all folds for that alpha.</li><li><strong>The result:</strong> RidgeCV automatically selects the alpha that produced the best average performance and then retrains a final model on all the training data using that best alpha.</li></ul><pre>from sklearn.linear_model import RidgeCV<br><br># Create a list of alphas to test<br>alphas_to_test = [0.01, 0.1, 1, 10, 100]<br><br># Instantiate RidgeCV. cv=5 means 5-fold cross-validation.<br># &#39;neg_mean_squared_error&#39; is a common scoring metric. Sklearn uses negative because its functions always try to maximize a score.<br>ridge_cv = RidgeCV(alphas=alphas_to_test, scoring=&#39;neg_mean_squared_error&#39;, cv=5)<br><br># Fit the model. It will automatically find the best alpha.<br>ridge_cv.fit(X_train, y_train)<br><br># --- Results from RidgeCV ---<br>print(&quot;\n--- RidgeCV Results ---&quot;)<br>print(f&quot;Best alpha found by RidgeCV: {ridge_cv.alpha_}&quot;)<br><br># The coefficients of the final model trained with the best alpha<br>print(f&quot;Coefficients from RidgeCV model: {ridge_cv.coef_[0]}&quot;)<br><br># Let&#39;s check the performance on the test set<br>y_pred_cv = ridge_cv.predict(X_test)<br>rmse_cv = np.sqrt(mean_squared_error(y_test, y_pred_cv))<br>print(f&quot;RMSE on test set with best alpha: {rmse_cv}&quot;)</pre><p>Using RidgeCV is the standard and professional way to use Ridge Regression.</p><h3>Regarding “Ridge Classification”</h3><p>This is an excellent question that often causes confusion.</p><p>Strictly speaking, <strong>there is no “Ridge Classification” in the same way there is Ridge Regression.</strong> The Ridge penalty (L2 regularization) is a concept applied to a cost function that involves minimizing squared errors, which is inherently a regression problem (predicting a continuous value).</p><p>However, the <strong>idea</strong> of applying an L2 penalty to combat overfitting and multicollinearity is used everywhere in machine learning, including in classification models.</p><p>The most direct and popular example is <strong>Logistic Regression</strong>.</p><ul><li>sklearn.linear_model.LogisticRegression is a model for classification (predicting categories like “Yes/No” or “Cat/Dog/Bird”).</li><li>By default, this scikit-learn model <strong>already includes L2 regularization</strong> just like Ridge!</li></ul><p>Let’s look at its parameters:</p><pre>from sklearn.linear_model import LogisticRegression<br><br># Check the default parameters<br>log_reg = LogisticRegression()<br>print(log_reg)</pre><p>Of course! This is the perfect next step after understanding Ordinary Least Squares (OLS). We will now explore **Ridge Regression**, a powerful variant of linear regression. It’s an excellent example of a concept called “regularization.”</p><p>After we master Ridge Regression, I’ll clarify the “Ridge Classification” part of your request.</p><p>You will see something like LogisticRegression(C=1.0, penalty=&#39;l2&#39;, ...).</p><ul><li>penalty=&#39;l2&#39; means it’s using the same sum-of-squares penalty as Ridge Regression.</li><li>The parameter C is the <strong>inverse of the regularization strength</strong>. It’s the reciprocal of alpha (C=1/αC = 1/\alphaC=1/α).</li><li>A <strong>small </strong><strong>C</strong> means a <strong>large alpha</strong> (strong regularization).</li><li>A <strong>large </strong><strong>C</strong> means a <strong>small alpha</strong> (weak regularization).</li></ul><p>So, when people say “Ridge Classification,” they are almost always referring to <strong>Logistic Regression with an L2 penalty</strong>. It’s the classification counterpart to Ridge Regression. It uses regularization to create a more robust and generalizable classification model.</p><h3>Summary of sklearn.linear_model.Ridge</h3><ul><li><strong>What it is:</strong> A linear regression model that includes an L2 regularization penalty.</li><li><strong>Main Purpose:</strong> To create a more robust, stable linear model that is less prone to overfitting and performs better in the presence of multicollinearity.</li><li><strong>Key Parameter:</strong></li><li>alpha: The regularization strength. Controls the amount of coefficient shrinkage.</li><li><strong>How to Choose Alpha:</strong> Don’t guess! Use RidgeCV to find the optimal alpha using cross-validation.</li><li><strong>Strengths:</strong></li><li>All the interpretability of linear regression.</li><li>More robust and stable than OLS, especially with correlated features.</li><li>Reduces model variance, leading to better generalization on unseen data.</li><li><strong>Weaknesses:</strong></li><li>Shrinks coefficients towards zero but <strong>never makes them exactly zero</strong>. This means it doesn’t perform feature selection (it will keep all features in the model, just with small coefficients). This is a key difference from our next topic, Lasso Regression.</li><li>Still assumes a linear relationship between features and the target.</li></ul><p>This lesson covered a fundamental concept: regularization. Understanding how Ridge uses a penalty to improve upon OLS is a major step toward thinking like a machine learning professional.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e2408d226b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit-learn: From Zero to Hero (Part 2):Ordinary Least Squares]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/scikit-learn-from-zero-to-hero-part-2-ordinary-least-squares-88f3b937f897?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/88f3b937f897</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 08:58:14 GMT</pubDate>
            <atom:updated>2025-10-13T09:03:51.036Z</atom:updated>
            <content:encoded><![CDATA[<h4>Your simple, step-by-step guide to mastering machine learning. We make the complex easy.</h4><h3>Ordinary Least Squares (OLS)</h3><p>Imagine you have a scatter plot of data points, like the price of a house versus its size. You can see a general trend: as the size increases, the price tends to increase.</p><p><strong>Ordinary Least Squares</strong> is a method to find the “best-fitting” straight line that goes through these data points.</p><p>But what does “best-fitting” mean?</p><ul><li>For every data point, we can measure the vertical distance between the point and our line. This distance is called the <strong>“residual”</strong> or <strong>“error”</strong>.</li><li>Some points will be above the line (positive residual), and some will be below (negative residual). To avoid these canceling each other out, we square each residual. This makes them all positive and also penalizes larger errors more heavily.</li><li><strong>OLS finds the one unique line that makes the sum of all these squared residuals as small as possible.</strong> That’s why it’s called “Ordinary Least Squares” — we are minimizing the sum of squares.</li></ul><p>In mathematical terms, we are trying to find the coefficients (slope and intercept) for a line that minimizes this cost function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*_kpr8AJ1znIjQWa9I6jxZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*YH8IIXHAlduKy3LGQLq3rA.png" /></figure><h4>Part 2: OLS in Scikit-Learn</h4><p>In scikit-learn, the OLS model is implemented in the LinearRegression class. Let’s walk through a complete example.</p><p><strong>Step 1: Setup and Data Creation</strong></p><p>First, we need to import the necessary libraries and create some sample data. We’ll use numpy for numerical operations and matplotlib for plotting.</p><pre># Import necessary libraries<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LinearRegression<br>from sklearn import metrics<br><br># Set a seed for reproducibility, so we all get the same random numbers<br>np.random.seed(42)<br><br># --- Create Synthetic Data ---<br># Let&#39;s create data that looks like a line with some random noise.<br># We&#39;ll create 100 data points.<br># X will be our feature (e.g., &#39;Years of Experience&#39;). It&#39;s a number between 0 and 10.<br>X = 2 * np.random.rand(100, 1) # 100 rows, 1 column<br><br># y will be our target variable (e.g., &#39;Salary&#39;).<br># Let&#39;s say the true relationship is y = 4 + 3*X + some random noise.<br>y = 4 + 3 * X + np.random.randn(100, 1)<br><br># Let&#39;s visualize our data to see what it looks like<br>plt.figure(figsize=(10, 6))<br>plt.scatter(X, y)<br>plt.title(&quot;Synthetic Data for Linear Regression&quot;)<br>plt.xlabel(&quot;Feature (e.g., Years of Experience)&quot;)<br>plt.ylabel(&quot;Target (e.g., Salary)&quot;)<br>plt.grid(True)<br>plt.show()</pre><p>When you run this code, you will see a scatter plot of points that roughly form an upward-sloping line. Our goal is to use LinearRegression to find that line.</p><p><strong>Step 2: Splitting Data into Training and Testing Sets</strong></p><p>A crucial step in machine learning is to evaluate your model on data it has never seen before. To do this, we split our dataset into two parts:</p><ul><li><strong>Training Set:</strong> The model “learns” from this data (about 70–80% of the total).</li><li><strong>Testing Set:</strong> We use this data to see how well the model performs on unseen data (the remaining 20–30%).</li></ul><p>Scikit-learn has a handy function train_test_split for this.</p><pre># Split the data into training (80%) and testing (20%) sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br>print(f&quot;Shape of X_train: {X_train.shape}&quot;)<br>print(f&quot;Shape of X_test: {X_test.shape}&quot;)<br>print(f&quot;Shape of y_train: {y_train.shape}&quot;)<br>print(f&quot;Shape of y_test: {y_test.shape}&quot;)</pre><p><strong>Step 3: Creating and Training the Model</strong></p><p>This is the core of the process. It’s surprisingly simple in scikit-learn.</p><ol><li><strong>Instantiate the model:</strong> Create an object of the LinearRegression class.</li><li><strong>Train the model:</strong> Call the .fit() method on the training data. This is where scikit-learn does the math to find the best-fitting line (i.e., calculates the optimal coefficients and intercept).</li></ol><pre># 1. Create an instance of the LinearRegression model<br>lin_reg = LinearRegression()<br><br># 2. Train the model using the training data<br># The .fit() method is the &quot;learning&quot; step<br>lin_reg.fit(X_train, y_train)<br><br># The model is now trained!<br>print(&quot;Model training complete.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*eRAa434zl-5E_urmDep-hA.png" /></figure><pre># The intercept (w_0)<br>print(f&quot;Intercept found by the model: {lin_reg.intercept_[0]}&quot;)<br><br># The coefficient for our single feature X (w_1)<br>print(f&quot;Coefficient found by the model: {lin_reg.coef_[0][0]}&quot;)</pre><p>You should see that the intercept is very close to 4 and the coefficient is very close to 3. The small difference is due to the random noise we added to the data.</p><p><strong>Step 5: Making Predictions</strong></p><p>Now that the model is trained, we can use it to make predictions on new data. We use the .predict() method. Let’s use it on our X_test data, which the model has never seen.</p><pre># Make predictions on the test set<br>y_pred = lin_reg.predict(X_test)<br><br># Let&#39;s look at a few predicted values vs actual values<br># We create a simple DataFrame with pandas for better visualization (optional but recommended)<br>import pandas as pd<br>df_results = pd.DataFrame({&#39;Actual&#39;: y_test.flatten(), &#39;Predicted&#39;: y_pred.flatten()})<br>print(&quot;\nComparing actual vs. predicted values:&quot;)<br>print(df_results.head()) # Print the first 5 rows</pre><p><strong>Step 6: Visualizing the Results</strong></p><p>A picture is worth a thousand words. Let’s plot our original data points along with the regression line that our model found.</p><pre>plt.figure(figsize=(10, 6))<br><br># Plot the training data points in blue<br>plt.scatter(X_train, y_train, color=&#39;blue&#39;, label=&#39;Training Data&#39;)<br><br># Plot the testing data points in green<br>plt.scatter(X_test, y_test, color=&#39;green&#39;, label=&#39;Testing Data&#39;)<br><br># Plot the regression line found by our model in red<br># We use X_test for predictions to show how the line fits the test data<br>plt.plot(X_test, y_pred, color=&#39;red&#39;, linewidth=3, label=&#39;Regression Line&#39;)<br><br>plt.title(&quot;Ordinary Least Squares - Model Fit&quot;)<br>plt.xlabel(&quot;Feature&quot;)<br>plt.ylabel(&quot;Target&quot;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><p>You will see a red line cutting through the data points, representing the best fit that the OLS model could find.</p><p><strong>Step 7: Evaluating the Model</strong></p><p>How do we quantify “how good” our model is? There are several standard metrics for regression problems.</p><ul><li><strong>Mean Absolute Error (MAE):</strong> The average of the absolute differences between the actual and predicted values. It’s easy to understand.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*kRKkp9L6Dnd1uQvyslLWBw.png" /></figure><p><strong>Mean Squared Error (MSE):</strong> The average of the squared differences. This penalizes larger errors more than MAE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*mCn0Z2ZmCCo-3f4fgEinUA.png" /></figure><p><strong>Root Mean Squared Error (RMSE):</strong> The square root of the MSE. This is useful because its units are the same as the target variable (e.g., if you’re predicting salary in dollars, the RMSE will also be in dollars).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*kBmiZEo8InkbzyukjGpUjw.png" /></figure><p>Scikit-learn provides these in the sklearn.metrics module.</p><pre># Calculate and print evaluation metrics<br>print(&quot;\n--- Model Evaluation ---&quot;)<br>mae = metrics.mean_absolute_error(y_test, y_pred)<br>mse = metrics.mean_squared_error(y_test, y_pred)<br>rmse = np.sqrt(mse) # or metrics.root_mean_squared_error(y_test, y_pred)<br>r2 = metrics.r2_score(y_test, y_pred)<br><br>print(f&quot;Mean Absolute Error (MAE): {mae}&quot;)<br>print(f&quot;Mean Squared Error (MSE): {mse}&quot;)<br>print(f&quot;Root Mean Squared Error (RMSE): {rmse}&quot;)<br>print(f&quot;R-squared (R2) Score: {r2}&quot;)</pre><p>An R-squared score close to 1 (like 0.7–0.8 for this noisy data) indicates a good fit.</p><h4>Part 3: Limitations and Assumptions of OLS</h4><p>LinearRegression is powerful and fast, but it’s not a silver bullet. It works best when certain assumptions about the data are met:</p><ol><li><strong>Linearity:</strong> The relationship between the features and the target is linear. If the true relationship is a curve, a straight line won’t be a good fit.</li><li><strong>Independence of Errors:</strong> The errors (residuals) should not be correlated with each other. (e.g., the error for one data point doesn’t tell you anything about the error for another).</li><li><strong>Homoscedasticity:</strong> The errors should have constant variance at every level of the features. In a plot of residuals, you should see a random cloud of points, not a cone or fan shape.</li><li><strong>Normality of Errors:</strong> The errors are assumed to be normally distributed.</li><li><strong>No Multicollinearity:</strong> The input features should not be highly correlated with each other. If you have two features like “house size in square feet” and “house size in square meters”, they are perfectly correlated and will cause problems for the model.</li></ol><p>Scikit-learn’s LinearRegression model has no parameters to tune, which makes it simple but also less flexible. It simply finds the mathematically optimal line based on minimizing the sum of squared errors.</p><h3>Summary of sklearn.linear_model.LinearRegression</h3><ul><li><strong>What it is:</strong> Implements Ordinary Least Squares linear regression.</li><li><strong>Main Purpose:</strong> To find a linear relationship between one or more features (X) and a continuous target variable (y).</li><li><strong>Key Methods:</strong></li><li>.fit(X, y): Trains the model.</li><li>.predict(X): Makes predictions on new data.</li><li>.score(X, y): Returns the R-squared score for the given data.</li><li><strong>Key Attributes (after fitting):</strong></li><li>.coef_: The calculated coefficients (slopes) for the features.</li><li>.intercept_: The calculated intercept.</li><li><strong>Strengths:</strong></li><li>Very fast and computationally efficient.</li><li>Easy to use and interpret. The coefficients directly tell you the impact of each feature.</li><li><strong>Weaknesses:</strong></li><li>Can be sensitive to outliers.</li><li>Assumes a linear relationship, which might not hold true for complex data.</li><li>Performs poorly when its underlying assumptions are violated.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88f3b937f897" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Core Idea of Linear Models (2)]]></title>
            <link>https://medium.com/@adnan.mazraeh1993/the-core-idea-of-linear-models-2-227b69305ec3?source=rss-4d6bb2142c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/227b69305ec3</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Adnan Mazraeh]]></dc:creator>
            <pubDate>Mon, 06 Oct 2025 12:03:52 GMT</pubDate>
            <atom:updated>2025-10-06T12:03:52.693Z</atom:updated>
            <content:encoded><![CDATA[<p>At their heart, all linear models make predictions using a simple linear equation. You absolutely know this from middle school math, even if you don’t realize it!</p><p>Remember the equation for a line?</p><p>$$ y = mx + b $$</p><p>* $y$: What we want to predict.<br>* $x$: Our input feature.<br>* $m$: The **slope** or **weight**. It tells us how much $y$ changes for a one-unit increase in $x$.<br>* $b$: The **intercept**. It’s the value of $y$ when $x$ is zero.</p><p>**Machine learning just gives these terms new names and expands the idea to more features.**</p><p>$$ \text{prediction} = (\text{weight}_1 \times \text{feature}_1) + (\text{weight}_2 \times \text{feature}_2) + \dots + (\text{weight}_n \times \text{feature}_n) + \text{intercept} $$</p><p>Or, using mathematical notation:</p><p>$$ \hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b = \sum_{i=1}^{n} w_i x_i + b $$</p><p>* $\hat{y}$ (y-hat): The predicted value.<br>* $x_i$: The value of the $i$-th feature (e.g., house size, number of bedrooms).<br>* $w_i$: The **coefficient** or **weight** for the $i$-th feature. This is the crucial part! The model *learns* these weights from the data.<br>* $b$: The **intercept**.</p><p>**The “Learning” Process:** The goal of a linear model during the `.fit()` process is to find the best possible values for the weights ($w_1, w_2, \dots$) and the intercept ($b$) that make the predictions ($\hat{y}$) as close as possible to the true values ($y$) in the training data.</p><p>**The “Linear” Part:** The model is “linear” because the prediction is always a simple weighted sum of the input features. It can’t learn complex interactions like “if feature 1 is high *and* feature 2 is low, then something special happens.” It only learns direct, additive relationships. This makes them:<br>* **Fast:** Calculations are simple additions and multiplications.<br>* **Interpretable:** You can look at the learned weights! A large positive weight ($w_i$) means that increasing feature $i$ strongly increases the prediction. A weight near zero means that feature has little effect.</p><p>— -</p><p>### Part 2: The Models in the “Linear Models” Toolbox</p><p>This section of the scikit-learn docs presents several models. They all use the same linear equation, but they differ in *how* they find the “best” weights and if they add any penalties or constraints.</p><p>#### 1.1.1. `LinearRegression` (Ordinary Least Squares — OLS)</p><p>* **Type:** Regression.<br>* **Analogy:** The “Eager Student.”<br>* **How it learns:** Its only goal is to find the weights that minimize the **sum of the squared differences** between the predicted values and the actual values. This difference is called the “residual” or “error”. The formula it minimizes is $\sum (y_{true} — \hat{y}_{pred})²$.<br>* **When to use it:** Your first choice for a simple regression task. A great baseline to see if there’s any linear relationship in your data at all.<br>* **Strengths:** Simple, no hyperparameters to tune, highly interpretable.<br>* **Weaknesses (Very Important!):**<br> 1. **Sensitive to Outliers:** One data point that is wildly off can pull the entire line towards it, skewing the results.<br> 2. **Suffers from Multicollinearity:** If you have two features that are highly correlated (e.g., `house_size_sqft` and `house_size_sqm`), the model becomes unstable. It won’t know how to assign weights, so it might give one a huge positive weight and the other a huge negative weight. The model can become wildly complex and overfit.</p><p>#### 1.1.2. `Ridge` Regression</p><p>* **Type:** Regression.<br>* **Analogy:** The “Cautious, Well-Rounded Student.”<br>* **How it learns:** It does the same thing as `LinearRegression` (minimizes the sum of squared errors) BUT with an added condition: **it also tries to keep the weights as small as possible.** It adds a “penalty” proportional to the sum of the *squares* of the weights ($w_1² + w_2² + \dots$).<br>* **The Hyperparameter `alpha`:** This controls the strength of the penalty.<br> * `alpha=0`: Same as `LinearRegression`. No penalty.<br> * Small `alpha` (e.g., 0.1, 1): A small penalty. The model is slightly cautious.<br> * Large `alpha` (e.g., 10, 100): A huge penalty. The model is very cautious and will make the weights very small, leading to a “simpler” model that is less likely to overfit.<br>* **When to use it:** This should be your default choice for regression, especially if you have many features or suspect your features might be correlated. It’s more robust than simple `LinearRegression`.<br>* **Strengths:** Protects against multicollinearity and overfitting. Makes the model more stable.<br>* **Weaknesses:** Shrinks weights *towards* zero, but never makes them *exactly* zero. So, it doesn’t perform feature selection.</p><p>#### 1.1.3. `Lasso` Regression (Least Absolute Shrinkage and Selection Operator)</p><p>* **Type:** Regression.<br>* **Analogy:** The “Minimalist, Decisive Student.”<br>* **How it learns:** It’s similar to Ridge, but its penalty is different. It adds a penalty proportional to the sum of the *absolute values* of the weights ($|w_1| + |w_2| + \dots$).<br>* **The Magic of Lasso:** This different penalty style has a profound effect. For features it deems unimportant, it can shrink their weights to be **exactly zero**.<br>* **When to use it:** When you believe many of your features are irrelevant and you want your model to automatically perform **feature selection**. It simplifies the model by telling you which features to ignore.<br>* **Strengths:** Automatic feature selection, which creates simpler and more interpretable models.<br>* **Weaknesses:** Can be a bit unstable in selecting features if you have a group of highly correlated features; it might arbitrarily pick one from the group and discard the others.</p><p>#### 1.1.11. `LogisticRegression`</p><p>* **Type:** Classification.<br>* **Analogy:** The “Line-Drawing Judge.”<br>* **How it works (The Trick):**<br> 1. It starts by calculating a linear equation just like `LinearRegression`: $z = w_1 x_1 + w_2 x_2 + \dots + b$. The output, $z$, can be any number from $-\infty$ to $+\infty$.<br> 2. It then feeds this output $z$ into a special function called the **Sigmoid** or **Logistic** function: $\sigma(z) = \frac{1}{1 + e^{-z}}$.<br> 3. This sigmoid function squashes any input value into a range between 0 and 1.<br> 4. This 0-to-1 output can now be interpreted as a **probability**. If the output is &gt; 0.5, the model predicts Class 1. If it’s &lt; 0.5, it predicts Class 0.<br>* **What it’s really doing:** It’s finding a linear boundary (a line, or plane) that best separates the classes.<br>* **When to use it:** The absolute baseline model for any classification task. It’s fast, interpretable, and surprisingly effective. If a simple `LogisticRegression` does poorly, your problem is likely not “linearly separable.”<br>* **Hyperparameters:** It also has regularization (like Ridge/Lasso) controlled by the `penalty` (`’l1&#39;` for Lasso, `’l2&#39;` for Ridge) and `C` (which is the *inverse* of `alpha`, so a small `C` means strong regularization) parameters.</p><p>— -</p><p>### Part 3: Practical Code Example</p><p>Let’s use the Boston Housing dataset (a classic) and see these models in action. We want to predict house prices (`MEDV`).</p><p>```python<br>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.linear_model import LinearRegression, Ridge, Lasso, LogisticRegression<br>from sklearn.metrics import mean_squared_error</p><p># — — Data Loading and Preparation — -</p><p># Scikit-learn 1.2 removed the Boston dataset. We load it from an online source.<br>data_url = “<a href="http://lib.stat.cmu.edu/datasets/boston">http://lib.stat.cmu.edu/datasets/boston</a>&quot;<br>raw_df = pd.read_csv(data_url, sep=”\s+”, skiprows=22, header=None)<br>data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])<br>target = raw_df.values[1::2, 2]<br>feature_names = [‘CRIM’, ‘ZN’, ‘INDUS’, ‘CHAS’, ‘NOX’, ‘RM’, ‘AGE’, ‘DIS’, ‘RAD’, ‘TAX’, ‘PTRATIO’, ‘B’, ‘LSTAT’]<br>X = pd.DataFrame(data, columns=feature_names)<br>y = pd.Series(target, name=’MEDV’)</p><p># Split data into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</p><p># IMPORTANT: Linear models are sensitive to feature scale. Always scale your data!<br>scaler = StandardScaler()<br>X_train_scaled = scaler.fit_transform(X_train)<br>X_test_scaled = scaler.transform(X_test)</p><p># — — Model Training and Evaluation — -</p><p># 1. Linear Regression (The Eager Student)<br>lr = LinearRegression()<br>lr.fit(X_train_scaled, y_train)<br>lr_pred = lr.predict(X_test_scaled)<br>lr_mse = mean_squared_error(y_test, lr_pred)<br>print(f”Linear Regression MSE: {lr_mse:.2f}”)</p><p># 2. Ridge Regression (The Cautious Student)<br># alpha=1.0 is a good starting point<br>ridge = Ridge(alpha=1.0)<br>ridge.fit(X_train_scaled, y_train)<br>ridge_pred = ridge.predict(X_test_scaled)<br>ridge_mse = mean_squared_error(y_test, ridge_pred)<br>print(f”Ridge Regression MSE: {ridge_mse:.2f}”) # Should be very similar or slightly better than LR</p><p># 3. Lasso Regression (The Minimalist Student)<br># alpha=0.1 is a good starting point for Lasso<br>lasso = Lasso(alpha=0.1)<br>lasso.fit(X_train_scaled, y_train)<br>lasso_pred = lasso.predict(X_test_scaled)<br>lasso_mse = mean_squared_error(y_test, lasso_pred)<br>print(f”Lasso Regression MSE: {lasso_mse:.2f}”)</p><p># — — Interpreting the Coefficients — -</p><p># Let’s create a DataFrame to compare the learned weights (coefficients)<br>coeffs = pd.DataFrame({<br> ‘Feature’: feature_names,<br> ‘LinearReg’: lr.coef_,<br> ‘Ridge (alpha=1)’: ridge.coef_,<br> ‘Lasso (alpha=0.1)’: lasso.coef_<br>}).set_index(‘Feature’)</p><p>print(“\n — — Learned Coefficients — -”)<br>print(coeffs)</p><p># Notice how Lasso sets some coefficients to exactly 0!<br>zero_coeffs_lasso = coeffs[coeffs[‘Lasso (alpha=0.1)’] == 0].index.tolist()<br>print(f”\nFeatures eliminated by Lasso: {zero_coeffs_lasso}”)<br>```</p><p>#### Analysis of the Code Output:</p><p>1. **MSE Scores:** You’ll notice the Mean Squared Error (MSE) for all three models is quite similar. This tells you that for this dataset, the regularization isn’t dramatically changing the overall prediction accuracy, but it is changing *how* the model gets there.<br>2. **Coefficients Table:** This is the most insightful part!<br> * **LinearReg:** It will have some non-zero coefficient for every feature.<br> * **Ridge:** The coefficients will be generally smaller (shrunken towards zero) compared to `LinearRegression`, making the model less complex and more robust.<br> * **Lasso:** This is the star. You will see that it has set the coefficients for some features (like `INDUS` or `CHAS`) to **exactly 0.0**. It has decided these features are not useful for predicting the house price and has discarded them, effectively performing automatic feature selection. This gives you a simpler, more interpretable final model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=227b69305ec3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>