<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Umer on Medium]]></title>
        <description><![CDATA[Stories by Umer on Medium]]></description>
        <link>https://medium.com/@umerfarooq230?source=rss-4bee6e00ed0b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ynvxBVO1mUXjDIzwXFBAHg.jpeg</url>
            <title>Stories by Umer on Medium</title>
            <link>https://medium.com/@umerfarooq230?source=rss-4bee6e00ed0b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:27:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@umerfarooq230/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding the Math Behind Fluid Simulation by Coding in Python]]></title>
            <link>https://towardsdev.com/understanding-the-math-behind-fluid-simulation-by-coding-in-python-a21c530dfcf0?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a21c530dfcf0</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fluid-dynamics]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 20:26:01 GMT</pubDate>
            <atom:updated>2026-10-25T20:26:01.691Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fefzHuAa5Ty50a-sX_fnA.jpeg" /><figcaption>Source: code execution screenshot</figcaption></figure><p>I’ve always thought of optimization as an abstract technique. For me, whenever the backend was laggy or responding late, I figured it was because of sequential table scans in the DB or bad memory allocation, and I was able to solve them with some existing techniques.</p><p>Still, the idea of “optimization” stuck with me and I couldn’t grasp it properly. What does it mean to optimize a system?</p><h3>The Bigger Idea</h3><p>That was until I made a fluid simulation in Python. Here’s the high-level idea:</p><ol><li>Make the particles. Each particle is considered a drop.</li><li>The liquid particles have to stay close, but not too close (they would become a solid), and far, but not too far (they would become a gas).</li><li>When a liquid particle is disturbed from one place to another, the close ones rush in to fill that space immediately.</li></ol><p>Even though this is the intuitive idea, there are deeper things here.</p><p>To simulate properly, we need the following calculations:</p><ul><li>Density</li><li>Pressure</li><li>Pressure Force</li><li>Velocity</li><li>Position</li></ul><h3>Smoothed Particle Hydrodynamics (SPH)</h3><p>Now we bring in the science behind our points above. In the SPH paper, the concept of a smoothing radius was introduced, which defines how far a particle can influence other particles when calculating properties such as density, pressure and velocity.</p><p>For two particles Xi and Xj, with a smoothing radius h = 2.0, we calculate the distance r = |Xi − Xj|. If r &lt; h, the neighboring particle contributes to the calculation. Otherwise, it doesn’t.</p><p>The whole point of the smoothing kernel is to assign more weight to the nearby particles and less to the distant ones.</p><h3>Density</h3><p>Think of density as how crowded the particles are around a particular particle.</p><p>If you have ten particles packed into a small space, the density will be higher than if those same ten particles are spread across a larger space. But how do we calculate this for a single particle when there are no fixed boxes or containers around it?</p><p>This is where the smoothing kernel comes in.</p><p>For a particle i, we calculate its density by looking at its neighboring particles, considering their mass and how far away they are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*V2fxD7FMShX2t8Lm9RCXBg.png" /><figcaption>source: Latex converted image</figcaption></figure><p>Here, mⱼ is the mass of a neighboring particle, and W is the smoothing kernel we discussed above.</p><p>The equation basically says: take each neighboring particle’s mass, multiply it by its influence, and add everything together.</p><p>A nearby particle contributes more, while a distant particle contributes less. If there are more particles crowded around our particle, the calculated density increases.</p><p>The interesting part is that we are estimating density without dividing a container into fixed regions. Each particle calculates its density from its surroundings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*7e8m7p_ME-A5xaAA_Of-XA.jpeg" /><figcaption>Source: <a href="https://www.slideserve.com/gloria/igert-sph-and-free-surface-flows">IGERT: SPH and Free Surface Flows</a></figcaption></figure><h3>Pressure</h3><p>Now that we have density, we can start understanding pressure.</p><p>Imagine pushing your hand into a bucket of water. The water resists being compressed. In a fluid simulation, we need a way to represent this behavior mathematically.</p><p>One common approach is to compare the current density with a target density, also called the rest density.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*pw88MhP5ShL1qX7pQsUBIg.png" /><figcaption>source: Latex converted image</figcaption></figure><p>Here, ρᵢ is the density we just calculated, ρ₀ is the target density, and k controls how strongly pressure responds to changes in density.</p><p>If the particles become too crowded, their density rises above the target, and pressure increases. This pressure helps push them apart.</p><p>Think of the target density as the crowding level we want the fluid to maintain. The simulation uses pressure to respond when particles deviate from that level.</p><p>Of course, this is a simplified pressure equation. More advanced simulations can use different equations of state, but the intuition remains similar.</p><h3>Pressure Force</h3><p>We now know how crowded the particles are and how much pressure they have. But how does this actually make them move?</p><p>Let’s imagine two regions of water. One is crowded, and the other has more room. The difference in pressure creates a force that tends to push the fluid away from the higher-pressure region.</p><p>This is where the pressure gradient comes in.</p><p>A gradient tells us how quickly something changes across space and in which direction it increases. In our case, it tells us how pressure changes around a particle.</p><p>The pressure force acts in the opposite direction to the pressure gradient, pushing fluid from higher-pressure regions toward lower-pressure regions.</p><p>In SPH, we calculate this using the pressures and densities of neighboring particles, along with the gradient of the smoothing kernel.</p><p>We don’t need to memorize the full equation to understand what is happening. Each neighbor contributes to the force on our particle, and the simulation adds those contributions together.</p><p>The important thing is that pressure is not just a number attached to a particle. Differences in pressure between neighboring particles are what help determine how the fluid moves.</p><h3>Velocity</h3><p>Now we have forces acting on our particles. What happens next?</p><p>Newton’s second law tells us that force produces acceleration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/82/1*DjENI1Nf4j1rUR6D7--_eQ.png" /><figcaption>source: Latex converted image</figcaption></figure><p>So, once we calculate the forces acting on a particle, we can calculate its acceleration.</p><p>But acceleration and velocity are not the same thing. Acceleration changes velocity over time.</p><p>For a small time step, we can update velocity using:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*hHZCXuvF-quWEGz8-ANxCw.png" /><figcaption>source: Latex converted image</figcaption></figure><p>Here, Δt is the small interval of time for which we are simulating the motion.</p><p>For example, if a particle is moving at 2 m/s and experiences an acceleration of 3 m/s² for 0.1 seconds, its velocity becomes 2.3 m/s.</p><p>The same idea applies to every particle in the simulation. Pressure, gravity, and other forces change their velocities over time.</p><h3>Position</h3><p>Finally, we use the velocity to update the position of each particle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*n_PoRfsD-7o8-JDmnNiZTw.png" /><figcaption>source: Latex converted image</figcaption></figure><p>If a particle is moving at 2.3 m/s for 0.1 seconds, it moves approximately 0.23 meters during that time step.</p><p>This gives us the particle’s new position.</p><p>But remember, moving one particle changes its distance from the others. That means the smoothing kernel weights can change, which affects density, pressure, and the forces calculated in the next step.</p><p>And this is where everything connects.</p><h3>Bringing It All Together</h3><p>We started with a collection of particles and wanted to make them behave like a fluid.</p><p>To do that, we estimate how crowded each particle’s surroundings are, calculate pressure from density, and use pressure differences to determine forces. Those forces change velocity, and velocity changes position.</p><p>Then we repeat the process.</p><p>Position → Density → Pressure → Force → Velocity → Position</p><p>Each iteration changes the state of the simulation, and each new state influences the next one.</p><p>This is the part I find interesting. What looks like a fluid moving naturally on the screen is actually the result of repeatedly calculating local interactions between particles.</p><p>We aren’t explicitly telling the water how to flow around every obstacle or how to fill every gap. Instead, we define the rules governing the particles and let their interactions produce the motion.</p><p>There is still a lot more to fluid simulation, including viscosity, boundary conditions, and numerical stability. But understanding these basic calculations gives us a foundation for exploring them.</p><p>And for me, this is where the abstract idea of simulation starts to make sense: a complex system can emerge from a relatively small set of rules, applied repeatedly.</p><h3>Making in Python</h3><p>The theory above is clean. The code is where it gets messy, so here are the functions that actually matter, and what each one does to the fluid you see on the screen.</p><p>One honest note first. The code is not the loop from the sections above, line by line. The classic version calculates a pressure force, turns it into velocity, and then moves the particle. My first version did exactly that and ran at around 10 FPS. The problem is that pressure forces are touchy. If the time step is too big, particles overshoot, push each other even harder, and the whole thing explodes. So you are forced to take tiny steps, which means a lot of calculations for every frame.</p><p>The version I ended up with is called Position Based Fluids. It skips the force. It predicts where every particle is about to land, moves the crowded ones apart until the density looks right, and then works out the velocity from how far each one moved. Same idea, different order. Think of it as walking someone to the right seat instead of shoving them and hoping they land there.</p><h3>The smoothing kernel and density</h3><pre>def smoothing_kernel(radius, distance):<br>    volume = math.pi * radius ** 8 / 4.0<br>    value = np.maximum(0.0, radius * radius - distance * distance)<br>    return value ** 3 / volume</pre><p>This is the “more weight to nearby particles, less to distant ones” idea as code. The middle line is the clever one. radius² − distance² is large when a neighbor is close and shrinks to zero right at the smoothing radius, and np.maximum(0, ...) makes sure anything beyond it gets exactly zero influence. Cubing it makes the weight fade out smoothly instead of switching off like a light, which keeps the fluid from twitching every time a particle crosses the edge of someone&#39;s neighborhood. Dividing by volume keeps the total weight consistent, so changing the radius doesn&#39;t change the density on its own.</p><p>The radius is a real trade-off. Make it too small and particles barely see each other, so the fluid turns into loose sand. Make it too big and everything blurs together, and the simulation slows down because every particle has more neighbors to check.</p><pre>def calculate_density(positions, sample_point, smoothing_radius, particle_mass):<br>    distances = np.linalg.norm(positions - sample_point, axis=1)<br>    return float(np.sum(particle_mass * smoothing_kernel(smoothing_radius, distances)))</pre><p>This is the density equation from earlier, almost word for word. The first line measures how far every neighbor is. The second weights each one with the kernel, multiplies by mass, and adds it all up. The result is a crowdedness meter, and everything else in the simulation reacts to this one number. In my app it powers the “density under the cursor” readout. The solver itself does the same sum inline, inside a function we will meet in a moment.</p><h3>The line above every function</h3><pre>@njit(parallel=True, cache=True, fastmath=True)<br>def solve_lambda(...):<br>    for particle in prange(count):</pre><p>Normal Python reads a loop one instruction at a time, which is painfully slow when the loop runs over thousands of particles, several times per frame. @njit (from a library called Numba) translates the function into machine code the first time it runs, which is why the first launch takes a few seconds. prange goes one step further. Each particle&#39;s calculation doesn&#39;t depend on the others, so the work is split across every CPU core at the same time.</p><p>No formula changed. The fluid behaves exactly the same, it just gets there much sooner. It is the cheapest optimization in the whole project: one line, and no change to the maths.</p><h3>Finding neighbors without checking everyone</h3><p>Everything above needs one thing, a particle’s neighbors. The naive way is to measure the distance from every particle to every other particle. With 9,000 particles, that is about 81 million checks per pass, and we do several passes per frame.</p><pre>cell_x = min(max(int(pred[particle, 0] * inv_cell_size), 0), grid_cols - 1)<br>cell_y = min(max(int(pred[particle, 1] * inv_cell_size), 0), grid_rows - 1)<br>cell = cell_y * grid_cols + cell_x</pre><p>Think of cutting the screen into squares, each exactly as wide as the smoothing radius. These three lines just work out which square a particle is standing in. Then comes the useful part:</p><pre>for cell_row in range(max(cell_y - 1, 0), min(cell_y + 2, grid_rows)):<br>    for cell_col in range(max(cell_x - 1, 0), min(cell_x + 2, grid_cols)):<br>        cell = cell_row * grid_cols + cell_col<br>        for slot in range(cell_start[cell], cell_start[cell + 1]):<br>            neighbor = order[slot]</pre><p>Anyone further away than the smoothing radius has zero influence anyway, so a particle’s neighbors can only be in its own square or the eight around it. Instead of checking thousands of particles, each one checks a few dozen. The answer is identical and the fluid looks the same. The work just dropped enormously.</p><p>If this sounds familiar, it is the same idea as a database index. A sequential table scan reads every row to find the few that matter. The grid jumps straight to them.</p><h3>Gravity and the prediction</h3><pre>vel_y = vel[particle, 1] + gravity * dt<br>...<br>pred[particle, 0] = pos_x + vel_x * dt<br>pred[particle, 1] = pos_y + vel_y * dt</pre><p>The first line is gravity. Every step, every particle’s downward velocity goes up a little. Take it out and the fluid floats like it is in space. Make it bigger and the same liquid feels heavy.</p><p>The last two lines are the prediction, and they are the part I find most interesting. We don’t move the particle yet. We write down where it would end up if nothing else interfered: position + velocity × time step. Think of it as a rough draft of the next frame. The rest of the calculation edits this draft, and only then do we make it real. Because the density is measured on the draft instead of the old positions, crowding gets fixed before it ever shows up on the screen, rather than after a crowd has already formed.</p><p>The same function also handles the mouse, where grabbed particles are steered toward the cursor, and a speed limit at the end stops one violent frame from launching particles across the screen.</p><h3>How crowded, and how hard to push</h3><pre>density_error = density * inv_rest_density - 1.0      # &gt; 0 means crowded<br>if density_error &gt; 0.0:<br>    lagrange[particle] = -density_error / (<br>        neighbor_grad_sq_sum + grad_sum_x * grad_sum_x<br>        + grad_sum_y * grad_sum_y + softening)<br>else:<br>    lagrange[particle] = 0.0</pre><p>This is the pressure idea from earlier wearing a different outfit. density_error compares a particle&#39;s density with the rest density. A value of 0.2 means the particle is 20% too crowded.</p><p>The if is a deliberate choice. A particle that isn&#39;t crowded gets no push at all, so we only resist squashing, never stretching. If we also pulled stretched fluid back together, the fluid would clump into blobs. Leaving it out lets a splash scatter into droplets.</p><p>The number we store is how hard this particle must be pushed to fix its crowding. The more crowded it is, the bigger the push, and the negative sign means away. The long bottom part scales the push by how sensitive the density is to movement, so a particle squeezed in a tight spot isn’t over-corrected. And softening is a small safety cushion so this division can never blow up and fire a particle across the screen.</p><h3>Pushing the particles apart</h3><pre>overlap = smoothing_radius - distance<br>grad_scale = spiky * overlap * overlap * mass * inv_rest_density / distance<br>pair_lagrange = own_lagrange + lagrange[neighbor]<br>move_x += pair_lagrange * grad_scale * offset_x<br>move_y += pair_lagrange * grad_scale * offset_y</pre><p>This is where the pressure gradient from the earlier section finally shows up. grad_scale is the gradient of the smoothing kernel. It decides how hard two particles shove each other: a lot when they are almost touching, fading to nothing at the edge of the smoothing radius. Multiplying by offset, the direction from the neighbor to me, turns that into a push along the line between the two. Add up the push from every neighbor and each particle ends up with one move.</p><p>Using own_lagrange + lagrange[neighbor] keeps the push fair. Particle A moves away from B by exactly as much as B moves away from A, a bit like Newton&#39;s third law, so a crowd spreads out evenly instead of drifting to one side.</p><pre>if move_length &gt; max_correction:<br>    move_x *= max_correction / move_length<br>    move_y *= max_correction / move_length</pre><p>This cap says no particle may move more than a fraction of the smoothing radius in one pass. Without it, two particles that start almost on top of each other would catapult one another across the screen.</p><h3>Turning movement into velocity</h3><pre>vel[particle, 0] = (pred[particle, 0] - pos[particle, 0]) * inv_dt<br>vel[particle, 1] = (pred[particle, 1] - pos[particle, 1]) * inv_dt</pre><p>Velocity is distance divided by time. After all that pushing, the draft positions have moved away from where the particles actually were, and these two lines measure how far, divided by the time step. This is how pressure turns into motion. A particle pushed upward by a crowd now carries upward velocity into the next frame, and that carried-over momentum is what makes a splash. Without these lines, the pushes would only shuffle particles around, and none of that motion would continue into the next frame.</p><h3>Back to optimization</h3><p>So, what does it mean to optimize a system?</p><p>Looking back, none of the optimizations changed what the simulation was trying to do. The grid didn’t change a single density value, it just stopped me from checking particles that could never matter. Numba didn’t change a formula, it changed how fast, and on how many cores, the formulas run. Even the firmness dial is a decision about how much work is worth doing.</p><p>For me, that is the answer. Optimizing a system isn’t about making everything faster. It is about finding the work that never needed to happen, and then the work that can happen at the same time. The grid is basically a database index, and the parallel loop is the same idea as spreading requests across workers. I had met these ideas on the backend before, but I never saw them this clearly until I could watch the frame rate change.</p><p>This article was inspired by <a href="https://www.youtube.com/c/SebastianLague">Sebastian Lague’s Youtube channel</a> and his video on <a href="https://www.youtube.com/watch?v=rSKMYc1CQHE&amp;list=PLFt_AvWsXl0dJrPelwdVSMVC_MDTYPJWn">Coding Adventure Simulating Fluids</a></p><p>Follow on <a href="http://www.linkedin.com/in/umer-farooq230">Linkedin</a> and <a href="https://www.instagram.com/gradientindecent">Instagram</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a21c530dfcf0" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/understanding-the-math-behind-fluid-simulation-by-coding-in-python-a21c530dfcf0">Understanding the Math Behind Fluid Simulation by Coding in Python</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an API for My Own Problem (and how you can do it too)]]></title>
            <link>https://medium.com/@umerfarooq230/building-an-api-for-my-own-problem-and-how-you-can-do-it-too-3cfe64b07cf6?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cfe64b07cf6</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[side-hustle]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[saas]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 05:52:47 GMT</pubDate>
            <atom:updated>2026-08-06T05:52:47.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6HyWssMGay0E6B2r" /><figcaption>Photo by <a href="https://unsplash.com/@guerrillabuzz?utm_source=medium&amp;utm_medium=referral">GuerrillaBuzz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I recently started to dive deep in building a system rather than just ML models. My thinking shifted from straight solutions to complete systems and how a problem goes through many stages before becoming a whole end to end solution. So I wanted to take a real problem from start to finish.</p><p>From this article I hope to give you a blueprint how I think and tackle problems, hopefully you can add some steps into your way of thinking too</p><p><strong>Try My API for Free: </strong><a href="https://rapidapi.com/umerfarooq230/api/email-urgency-scoring-api/playground/apiendpoint_cd398009-c9b2-4f79-a9ec-5c99cf3e457f">Email Scoring API</a></p><h4>The Problem</h4><p>Many of my friends complained about their flooded inboxes with spam, useless subscriptions, events updates, older email lists, etc. And important emails vanishes between them, that’s a bigger problem for business owners as they have hundreds and thousands of emails monthly, and reading each one completely to determine its agenda is not practical</p><p>And that was the problem to solve systematically</p><h4>The Approach</h4><p>Like most side projects, this one did not start with a startup idea or a business plan. It started with a small problem that kept repeating itself. I wanted a simple way to decide which emails actually needed my attention.</p><p>At first I thought about building something just for myself. Before writing any code, I tried to understand the decision I was making every time I opened my inbox.</p><p>It looked something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*Bp3xyQQtzlp-WmGqjobzpQ.jpeg" /><figcaption>Personal Miro Board</figcaption></figure><p>Around this time I also watched a Fireship video about APIs. That eventually led me to Code with Ania Kubów’s API tutorial. Those videos did not give me the solution, but they helped me understand how an API should expose functionality instead of focusing on the interface</p><h4>What Didn’t Work</h4><p>This step had two levels of execution. The first version was only for me. Hard coded values, trigger words, fixed scoring system and simple thresholds. If an email contained words like “refund” or “cancel”, it received extra points. Simple and clean</p><p>The problem appeared when I imagined someone else using it. My definition of an important email is probably different from a customer support team or an online store.</p><p>Second step was to make it adaptable to scale for different owners, businesses and thinking like an engineer. So no hard coded values or rules, just a simple question: “How can someone else define what matters to them?”. Answering this question will eventually solve the problem.</p><h4>The Pipeline</h4><p>Here’s what happens when an email hits the system — no black box, just a straight line from text in to a number out.</p><p><strong>1. The email comes in</strong><br> Just three plain things: subject, body, sender. That’s it. This shape is defined in schemas.py .One file that says &quot;an email looks like this, nothing more.&quot;</p><p><strong>2. It gets checked against a list of words</strong><br> A dictionary of phrases that usually mean trouble (“refund,” “chargeback,” “legal action”) and phrases that usually don’t (“thanks,” “just wondering”). Each word carries a weight. Lives in triggers.py, and the core function is about as plain as code gets:</p><pre>find_matches(text, triggers) → list of (word, weight) pairs found</pre><p><strong>3. The highest match wins, not the total</strong><br> My first instinct was to add everything up. Wrong move, five polite words shouldn’t outscore one real “chargeback.”</p><pre>base_score = highest weight among all matches</pre><ul><li>One bad word is enough to flag something serious</li><li>That’s how people actually read an inbox, you react to the worst part, not the average</li></ul><p>This lives in scoring.py, the heart of the system.</p><p><strong>4. A few small signals nudge the score</strong><br> Each one is its own tiny function does one job, nothing else:</p><ul><li>ALL CAPS subject → small bump</li><li>Multiple exclamation marks → small bump</li><li>Sender is marked VIP → bump</li><li>Thread’s been sitting unanswered for days → capped bump</li></ul><p>None of these decide anything alone. They just adjust a score the words already set.</p><p><strong>5. The number becomes a label</strong><br> 8+ is Critical. 5–7 is Needs Attention. Below that, Low Priority. This mapping sits in its own file, categories.py — so changing what counts as &quot;Critical&quot; means editing one small file, not touching the scoring logic.</p><p><strong>6. Every account gets its own rules</strong><br> A bakery and a logistics company don’t care about the same words. So before scoring anything, the system asks one question: “<em>what are this account’s rules?” </em>and scores against whatever comes back. Nothing is hardcoded per business.</p><p><strong>7. Everything gets written down</strong><br> Every score, every match, every timestamp, saved. Not for the sake of storing data, but so that six months from now, “why did this get flagged?” has an answer sitting right there. One file, storage.py, is the only part of the system allowed to touch the database. Everything else just asks it politely.</p><p><strong>8. No language model, anywhere in this loop</strong><br> That was the point from day one:</p><ul><li>No waiting on a response</li><li>No unpredictable cost per email</li><li>No guessing why a score came out the way it did</li></ul><p>Just words, weights, and a handful of small, honest files — each one doing exactly one job.</p><p>An email goes in one end. A number and a reason come out the other. Nothing in between is a mystery.</p><h4>Why an API</h4><p>At one point I considered building a web application or even a browser extension. Both would have solved my own problem. The issue was that they solved it in only one place.</p><p>An API felt like the better foundation. A web application can call it. A browser extension can call it. A CRM can call it. Even a small automation script can use the same endpoint.</p><h4>The V1</h4><p>The current version does what I originally wanted it to do. It accepts an email, evaluates it, and returns structured information that another application can use. It is containerized with Docker, documented, and simple enough to integrate into other projects.</p><p>There is still plenty to improve. I want users to define their own scoring rules without touching the code. Better history and analytics would also make the API more useful over time.</p><p><strong>Try My API for Free: </strong><a href="https://rapidapi.com/umerfarooq230/api/email-urgency-scoring-api/playground/apiendpoint_cd398009-c9b2-4f79-a9ec-5c99cf3e457f">Email Scoring API</a></p><p>Follow me on <a href="https://www.linkedin.com/in/umer-farooq230/">Linkedin</a></p><p>For queries: umerfarooq230@gmail.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cfe64b07cf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I made a System Calculator for Local LLMs (with Source Code)]]></title>
            <link>https://medium.com/@umerfarooq230/i-made-a-system-calculator-for-local-llms-with-source-code-3bcbebeeaed3?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bcbebeeaed3</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[ai-solution]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 20:53:36 GMT</pubDate>
            <atom:updated>2026-04-28T20:53:36.685Z</atom:updated>
            <content:encoded><![CDATA[<p>I coded the solution of my own problem</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xhEQQWNB7F81Wqu_" /><figcaption>Photo by <a href="https://unsplash.com/@solenfeyissa?utm_source=medium&amp;utm_medium=referral">Solen Feyissa</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Ever since I have discovered local LLMs from huggingface and Ollama, I have downloaded and ran many of the models, but each time the same problem occurred</p><h3>The Problem</h3><p>Every time I wanted to run a model, My process looked like this:</p><ol><li>Find a model</li><li>Get excited about its benchmarks</li><li>Scroll reddit for reviews and system reqs</li><li>Ask GPT</li><li>Try anyway</li></ol><p>… and it still crashes after 3 prompts.</p><p>The question was same every time. “What should be my system’s specifications to run this smooothly?” but my approach was wrong</p><h3><strong>The Solution</strong></h3><p>I made a simple web app which ask you two simple things about your system. <strong>GPU (VRAM) and RAM. </strong>And other things like Number of Tokens and Quantization</p><p>and It tells you</p><ul><li>Which models you can run</li><li>Which ones might run but within limits</li><li>Which ones will not just run</li></ul><p>That’s it. No Overengineering. Thinking in “System” rather than “Substance”</p><h3>How It Works</h3><p>The app estimates how much VRAM a model needs based on its parameter size, then adjusts that based on quantization.</p><p>A 7B model in full precision is very different from the same model in 4-bit, so that reduction is applied directly. It also includes context length in the calculation, since longer token windows increase memory usage and are often the hidden reason models fail to run.</p><p>Once it has this estimate, it compares it with your available VRAM and gives a practical result. Based on the headroom I have added different categories</p><ul><li><strong>SAFE</strong>: Uses ≤85% GPU VRAM</li><li><strong>TIGHT: </strong>Fits in GPU VRAM (tight, up to 100%)</li><li><strong>OFFLOAD:</strong> Requires CPU/RAM offloading (slower)</li><li><strong>WON’T RUN</strong>: Exceeds GPU + system RAM</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UkQfYDOoGdI_ZdnC" /><figcaption>Photo by <a href="https://unsplash.com/@galka_nz?utm_source=medium&amp;utm_medium=referral">Galina Nelyubova</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Thinking In Systems</h3><p>As I am becoming more mature with coding and taking Infrastructure seriously rather than adding more features, my perspective about developing is also changing</p><p>Simple solutions always beats complex engineering. This is how old me would have thought about the solution.</p><p>Make an AI wrapper -&gt; give RAG docs -&gt; Track the system reqs -&gt; Give Solution</p><p>But when I stopped thinking about the hype of training a model and started thinking about building a system, things changed, for good.</p><h4><strong>Code:</strong></h4><p>You can check the <a href="https://github.com/umer-farooq230/Can-My-GPU-Run-It"><strong>Repo here</strong></a></p><p>Would love to see your creativity here modifying the code</p><p>Feel free to connect with me on <a href="https://www.linkedin.com/in/umer-farooq230/">Linkedin</a>, <a href="https://github.com/umer-farooq230/">Github</a> or <a href="http://umerfarooq230@gmail.com/">Email</a>. Can talk about anything</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bcbebeeaed3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Building Boring Machine Learning Projects; Mistakes and Solutions]]></title>
            <link>https://medium.com/@umerfarooq230/stop-build-boring-machine-learning-projects-mistakes-and-solutions-b16b40cc69dd?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b16b40cc69dd</guid>
            <category><![CDATA[machine-learning-projects]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[end-to-end]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Mon, 07 Jul 2025 19:59:53 GMT</pubDate>
            <atom:updated>2025-07-30T07:05:33.480Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to create projects which helps in your learning, looks good on resume, and provide real life value?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VNjQEVnoz6ojFQw1" /><figcaption>Photo by <a href="https://unsplash.com/@hdbernd?utm_source=medium&amp;utm_medium=referral">Bernd 📷 Dittrich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As I am out of the beginner phase of ML (and assuming you’re past your initial learning and know about gradient descent, optimizer and knows about basic models) I realized how my projects added close to zero value in my skill and resume.</p><p>This is how my project looked like.</p><ol><li>Search for new project idea (no original idea I could think of)</li><li>Look if there is any tutorial on it</li><li>Open VS code and make a jupyter file</li><li>Data cleaning + visualization + modeling</li></ol><p>…….That’s it</p><p>If you’re just like me, then this will help you</p><h3>The Problem With Most Beginner Projects</h3><p>Most ML projects from learners fall into one of the following traps:</p><ul><li>A single Jupyter notebook with everything crammed in</li><li>Trained on clean toy datasets like Iris or Titanic</li><li>No real structure or maintainability</li><li>No automation or data pipeline</li><li>No interface or way to interact with the model</li></ul><p>These may teach you syntax or how to fit a model — but not how to actually build machine learning systems.</p><h3>The “Original” Idea</h3><p>We all have been there. Open Google and type “unique python ML ideas” or do the same in ChatGPT.</p><p><strong>But all the answers are also the same</strong>. Sentiment analysis, clustering, fake news detection, fraud detection, churn prediction, Image classification or face detection</p><p>Let’s change that.</p><p>Instead of chasing “unique” ideas from a list, focus on building <strong>personal</strong> ideas — projects that come from your own problems, interests, or daily life. Try to make a mix of your interests in different things.</p><p>For example:</p><ul><li>I like philosophy so I fine tuned a model that will take user’s input and tell them which philosopher they are.</li><li>I also tried to train a customer support model for my local cookie store that will be able to give humor answers as well as serious answers (that taught me how to combine different datasets), because the store has that type of vibe (not to mention, I failed miserably)</li></ul><p>First thing is you can ask questions to yourself and that “what if” will take you to places. And the key here is not to think that this project is too difficult. Now what are the benefits.</p><ol><li>Motivation. At the end of project you will say, “yes I did this and this was my idea”.</li><li>You will push your limits and learn things you never knew existed.</li><li>You will be forced to complete this project yourself and not with AI (The dopamine at the end will be worth it).</li></ol><h3>Out of the One File System</h3><p>Like I said we have been tuned by the tutorial hell that creating one file is enough. But it’s not. If you have never saw another file instead of the “main.py” you don’t know what is config file and to be honest I also didn’t.</p><p>So take my advice and create different folders for</p><ol><li>Data (Raw and Processed)</li><li>notebooks (EDA or extras)</li><li>Source (Preprocess, Train, Evaluate)</li><li>Models (yes we will now save our models)</li><li>Config/Configuration</li><li>APIs</li></ol><p>So this how your parent folder should look like</p><pre>my-ml-project/<br>├── data/<br>│   ├── raw/<br>│   ├── processed/<br>├── notebooks/<br>│   └── EDA.ipynb<br>├── src/<br>│   ├── data_loader.py<br>│   ├── preprocess.py<br>│   ├── train.py<br>│   ├── evaluate.py<br>├── models/<br>├── config/<br>│   └── config.yaml<br>├── scripts/<br>│   └── run_pipeline.sh<br>├── api/<br>│   └── app.py<br>├── requirements.txt<br>├── README.md</pre><p>Using this system you will know your code and will not spend days looking for a specific indentation error and will be easy to navigate. You can also follow any other system (This is just an example to get you out of the one file system)</p><h3>The End-to-End Pipeline</h3><p>As most of aims for the high paying jobs in companies (yes we aim for high paying ), the technical recruiters will not look how good you can train the model, that is just one part. But your chances increases as they know you can build the app from start to finish and knows what you do.</p><p>The best lesson I learned: the model is just one part of the system. In one of my projects, I spent 90% of the time on everything <em>except</em> training: cleaning text, handling missing data, building a CLI tool, packaging the app, writing docs.</p><p>That’s what real ML work looks like.</p><p>A simple, working pipeline beats a fancy model every time.</p><p>Here’s what I aim for now:</p><ul><li>Ingest raw data (e.g., API, CSV, or must learn scraping)</li><li>Make proper directories (folders)</li><li>Visualize raw data</li><li>Clean the data and preprocess it</li><li>Visualize clean data</li><li>Train a model</li><li>Try to tweak the model and see results</li><li>Visualize the model</li><li>Save the model properly (with versioning)</li><li>Serve via API or Streamlit</li><li>Make a repository, using bash commands (YES PLEASE LEARN GIT AND GITHUB)</li></ul><h3>Final Thoughts and Next Steps</h3><p>Your next steps would be learning MLflow and docker</p><p>If you’re trying to break into ML, the best thing you can do is <strong>build like you’re already on the job</strong>. Not just fitting models — but shipping systems.</p><p>Start small. Use real data (don’t always use the the kaggle). Organize your code like it matters — because it does.</p><p>Find me one <a href="https://www.linkedin.com/in/umer-farooq230/"><em>LinkedIn</em></a> and <a href="https://github.com/umer-farooq230"><em>GitHub</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b16b40cc69dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Cool Guide to Encryption; Hide Messages in an Image]]></title>
            <link>https://medium.com/@umerfarooq230/a-cool-guide-to-encryption-hide-messages-in-an-image-f6a080589672?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6a080589672</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hacking]]></category>
            <category><![CDATA[steganography]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Fri, 29 Nov 2024 19:21:20 GMT</pubDate>
            <atom:updated>2024-12-21T14:28:53.474Z</atom:updated>
            <content:encoded><![CDATA[<h4>Source code at the end (and a secret)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f8Duunv3L8ho9KlUzyL03w.png" /><figcaption>Photo by Author (Canva)</figcaption></figure><h3><strong>Introduction</strong></h3><p>Do you want to hide your secrets in another file or format? A hate email to your boss encoded into a secret sentence? Or into an image or audio file Got embarrassing karaoke audio you want to keep private?</p><p>For all of the above, you use steganography and there are types of steganography you use based on your needs</p><ol><li>Text</li><li>Image</li><li>Video</li><li>Audio</li><li>Network</li></ol><h4><strong>What to expect?</strong></h4><p>This is a hybrid-style piece that serves both as a tutorial and a conceptual blog, where I explain an encryption technique. The code snippets are non-sequential, but the complete source code is provided at the end.</p><p>If you’re here to understand the concept, feel free to skip the <strong>“Understanding the Logic”</strong> section specifically designed for fellow programmers.</p><p>Hope you have a fun journey :)</p><h3>The Why?</h3><p>Now the question is why I took this project? Mostly because I am always up for something new. While searching for a university project, I came across the idea of Steganography (the encryption technique). I dove headfirst into research. Understanding the principles of steganography was my first step.</p><p>But that turned out to be more stressful than I had imagined.</p><h3><strong>The How</strong></h3><p>First, let me tell you that there are more complex and effective methods for this but I have used the most common and easiest method</p><h4>Least Significant Bit (LSB) Encoding Explained</h4><p>LSBs are the most important concepts in steganography</p><p>1 Bit = 8 bytes</p><p>Right most are Least Significant and as the name suggests they will provide the <strong>least significance </strong>to a Byte</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*BSENzDWt9IHbQs3GGpH5aA.png" /><figcaption>Photo By Author (Canva)</figcaption></figure><p>In digital images, each pixel is represented by multiple bits:</p><ul><li>For an RGB image, each pixel has three color channels (Red, Green, Blue)</li><li>Typically, each channel uses 8 bits (0–255)</li><li>Example: A pixel might be (255, 128, 64)</li></ul><p>so visually the LSBs in RGBs we can look at this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*4Ql649z6kH4nTEkoXnF-CQ.png" /><figcaption>Photo by Author (Canva)</figcaption></figure><h4>How LSB Encoding Works</h4><ul><li>Each pixel’s color values have 8 bits. The rightmost bit (least significant bit) has the smallest impact on color</li><li>By changing this bit, we can hide information with minimal visual change</li></ul><h4>Encoding Process Step-by-Step</h4><ol><li>Take a pixel value: 11111111 (decimal 255)</li><li>Want to hide bit: 0</li><li>Modify the pixel: 11111110 (decimal 254)</li></ol><p>Visual impact:</p><ul><li>255 (pure white)</li><li>254 (almost pure white)</li></ul><p>Difference to the human eye? impeccable</p><h4>Understanding the Logic</h4><pre>def encode_text(text, encoding=&#39;utf-8&#39;, errors=&#39;replace&#39;):<br>    bits = bin(int.from_bytes(text.encode(encoding, errors), &#39;big&#39;))[2:]<br>    return bits.zfill(8 * ((len(bits) + 7) // 8))<br><br><br>def encode_in_image(filename, text_message):<br>    try:<br>        input_im = Image.open(filename, &#39;r&#39;).convert(&quot;RGB&quot;)<br>        image_shape = np.asarray(input_im).shape<br>        flat_array = np.asarray(input_im).flatten()<br><br>        encoded_text = encode_text(text_message + &quot;&lt;STOP&gt;&quot;)<br>        #list comprehnesion<br>        encoded_array = [<br>            (0b11111110 &amp; value) | int(encode_bit) if ix &lt; len(encoded_text) else value<br>            for ix, (encode_bit, value) in enumerate(zip(encoded_text.ljust(len(flat_array), &#39;0&#39;), flat_array))]<br><br>        encoded_im = np.array(encoded_array).reshape(image_shape)<br>        return Image.fromarray(np.uint8(encoded_im)).convert(&#39;RGB&#39;)<br>    except Exception as e:<br>        raise ValueError(f&quot;Error encoding message into image: {str(e)}&quot;)</pre><p>This code encodes a given text into a binary representation. It first converts the text to bytes using the specified encoding and handles errors as specified. Then, it converts the bytes to a binary string, ensuring each byte is represented as 8 bits by adding zero padding to the left if necessary and no, the padding will not affect the final values. This padding ensures that all binary values align to full bytes.</p><p>The second function embeds a text message into an image by modifying its pixel values. It starts by opening the image in RGB mode and converting it into a NumPy array to work with its pixel data. The function then encodes the message (with a “&lt;STOP&gt;” marker) into binary using the earlier encode_text method.</p><p>Using list comprehension, it alters the least significant bit of each pixel’s value to store the binary data of the text. If the text is shorter than the pixel data, it pads the remaining binary with zeros. Finally, the modified array is reshaped to match the original image’s dimensions and converted back into an image. Error handling is included to raise an informative message if anything goes wrong during the process.</p><h4>Decoding Pixel Data</h4><p>Now you have a secret message embedded into an image, that you can send to anyone. <em>hehe. </em>But if someone sent you that image, you want to know what is inside. let’s work on that</p><h4><strong>Read each pixel’s color channel values</strong></h4><p>Let’s suppose you have this situation where you have 11111111 for each RGB value</p><ul><li>For an RGB image, every pixel has three channels (Red, Green, and Blue). Each channel holds an 8-bit value (e.g., 255 = 11111111 in binary).</li><li>Extract the binary value of each channel sequentially.</li></ul><h4><strong>Identify the Least Significant Bit (LSB)</strong></h4><p>From each channel’s binary representation, focus on the rightmost bit, as it carries the hidden information.</p><h4>Extract the Hidden Bits</h4><p>Once the LSB of each channel is identified, the next step is to collect these bits to reconstruct the encoded data:</p><ul><li>Go through the image pixel by pixel, channel by channel.<br><em>(Example: For a pixel with RGB values of (254, 127, 65), the binary values are </em><em>11111110, </em><em>01111111, and </em><em>01000001. The extracted bits are </em><em>0, </em><em>1, and </em><em>1.)</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*yUO0yXMe48UYQiIz4w35Cw.png" /><figcaption>Source: Generated from Claude</figcaption></figure><h4>From Bits to Bytes and Information</h4><p>Remember that our extracted data is still in bits and we must convert it into bytes. And you are not a computer so you cannot read bytes (<em>obviously</em>). So you want to have these bytes transformed into meaningful words</p><p>If the hidden data represents text, use the ASCII encoding standard to convert each byte into a character.<br><em>(Example: </em><em>01100001 → &#39;a&#39;, </em><em>01100010 → &#39;b&#39;, etc.)</em></p><p>Continue this process until all LSBs from the image have been read.</p><h4>Understanding the Logic</h4><pre>def decode_text(bits, encoding=&#39;utf-8&#39;, errors=&#39;replace&#39;):<br>    n = int(bits, 2)<br>    return n.to_bytes((n.bit_length() + 7) // 8, &#39;big&#39;).decode(encoding, errors)<br><br><br>def extract_from_image(filename):<br>    try:<br>        encoded_im = np.asarray(Image.open(filename, &#39;r&#39;).convert(&quot;RGB&quot;))<br>        extracted_bits = [str(0b00000001 &amp; value) for value in encoded_im.flatten()]<br>        extracted_bits = &#39;&#39;.join(extracted_bits)<br>        return decode_text(extracted_bits, errors=&#39;replace&#39;).split(&#39;&lt;STOP&gt;&#39;)[0]<br>    except Exception as e:<br>        raise ValueError(f&quot;Error extracting message from image: {str(e)}&quot;)</pre><p>decode_text(bits, encoding=&#39;utf-8&#39;, errors=&#39;replace&#39;)</p><p>This function converts a binary string back into text. It starts by interpreting the binary string as a large integer. Then, it converts this integer into bytes, ensuring that the number of bytes matches the data size. Finally, it decodes these bytes into text using the specified encoding (default is UTF-8), handling errors if the data can’t be perfectly translated.</p><p>extract_from_image(filename)</p><p>This function retrieves a hidden message from an image. It opens the image in RGB mode, converts it to a NumPy array, and flattens it to work with all pixel values as a single list. The least significant bit (LSB) of each pixel is extracted and stored as a string of binary digits. These binary bits are then combined into a single string, and decode_text translates them back into readable text.</p><p>The function looks for the &lt;STOP&gt; marker in the decoded message and trims everything beyond it, returning only the hidden message. If any issue arises during the process, an error is raised with a clear message.</p><h3>Links</h3><p>Want to test it? Click here: <a href="https://imagepass.streamlit.app/"><strong>Image Pass</strong></a></p><p>Source code at <a href="https://github.com/umer-farooq230/Imgae-Steganography"><strong><em>Image Steganography</em></strong></a></p><h4>Conclusion</h4><p>LSB encoding is like the digital version of whispering a secret so softly that no one else notices — but instead of your lips, it’s pixels doing the talking! By tweaking the tiniest bits of color data, it hides messages in plain sight with <strong>impeccable stealth</strong>.</p><p>If you’re looking for expert help with your machine learning projects, Check out my services on Fiverr.</p><p><a href="https://www.fiverr.com/s/5rN4pX1"><strong><em>Click here</em></strong></a></p><p>Let’s be friends</p><p><a href="https://www.linkedin.com/in/umer-farooq-34b0401b6/"><strong><em>Linkedin</em></strong></a></p><p><strong><em>umerfarooq230@gmail.com</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6a080589672" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ultimate Beginner to Advance guide to Machine learning]]></title>
            <link>https://pub.towardsai.net/the-ultimate-beginner-to-advance-guide-to-machine-learning-b4dd361aefbb?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4dd361aefbb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Wed, 02 Oct 2024 14:33:16 GMT</pubDate>
            <atom:updated>2026-03-01T01:13:45.524Z</atom:updated>
            <content:encoded><![CDATA[<h4>let’s learn Machine Learning</h4><p>Edit 01–03-2026: Added the 4th phase of Deployment and Testing</p><p>You probably clicked on this story because you wanted to learn machine learning but couldn&#39;t find a way to get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EAcIND3boaW2ihVf" /><figcaption>Photo by <a href="https://unsplash.com/@goran_ivos?utm_source=medium&amp;utm_medium=referral">Goran Ivos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Six months ago, I was in your shoes. Despite countless free roadmaps out there, I struggled to find a clear path for advancing in ML. After much trial and error, I’ve cracked the code to match everyone&#39;s learning order. Stick around to unlock the secrets I’ve uncovered on this fascinating journey.</p><h3>Phases</h3><p>There are Four Phases in this article. Complete them in the same order and then move forward.</p><h3>Phase 1: Foundations</h3><h3>Python</h3><p><a href="https://www.python.org/downloads/">Install Python</a> (get the latest)</p><p><a href="https://code.visualstudio.com/">Install Vscode</a></p><p>I mean this is an obvious one but still, some people just hear about the new trend of ML and want to make ChatGPT. This is not possible. You have to learn Python and get a good grip on its fundamentals like loops, functions, lists, tuples, dictionaries, list comprehension.</p><p>First of all, learn the syntax from <a href="https://www.w3schools.com/python/default.asp"><strong>W3school</strong></a></p><p>Then to projects. Get Ideas for beginner projects from <a href="https://www.geeksforgeeks.org/python-projects-beginner-to-advanced/"><strong>Geeks for Geeks</strong></a></p><p>(Don’t listen to the “must have projects in resume” on YouTube)</p><p>Spend at least 3 months. The first month will be just learning and learning. Then the rest of the two months will be projects and more projects. You can adjust according to your level of understanding. <strong>You will be good when you can use your logic instead of looking at the code or tutorial.</strong></p><p>Directories are a common term for folders. To use datasets(explained below) your Python file and dataset should be in the same Directory</p><h4>When are you ready for the next phase?</h4><p>You are ready for the next phase when you are able to:</p><ul><li>Write Python code without tutorials.</li><li>Successfully completed at least three small projects.</li></ul><h3>Phase 2: Exploring Libraries</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p-jGOYJFdVBgf-fX" /><figcaption>Photo by <a href="https://unsplash.com/@hishahadat?utm_source=medium&amp;utm_medium=referral">Shahadat Rahman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Now that you have basic knowledge of Python. There are some libraries that will make your coding life easier. Libraries are already written pieces of code that you can take and use it as your own.</p><p>To install the libraries, open the terminal and write <em>pip install [library name] or !pip install [library name] (for </em><a href="https://colab.research.google.com/"><em>Google collab</em></a><em> only)</em></p><h4>Pandas</h4><p>This library will help you manage large files AKA datasets easily. Learn from <a href="https://www.w3schools.com/python/pandas/default.asp"><strong>W3school</strong></a><strong> </strong>or<strong> </strong><a href="https://youtu.be/tRKeLrwfUgU?feature=shared"><strong>Nicholas</strong></a><strong>. </strong>This will be easy to learn.</p><h4>Numpy</h4><p>The most common library to do matrix multiplication. Because everything will be in Matrices. Learn basics from <a href="https://www.w3schools.com/python/numpy/default.asp"><strong>W3school</strong></a><strong>.</strong></p><h4>Matlplotlib</h4><p>As a Machine Learning engineer, you are always making calculations and manipulating numbers. Matplotlib will help you turn your numbers into visual graphs and help you better understand what are you doing. Something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*hMUD9bEDjQhyrvCyMjZBRA.png" /><figcaption>Source: from <a href="https://www.w3schools.com/python/python_ml_confusion_matrix.asp">W3school</a></figcaption></figure><p>Learn Basic syntax from <a href="https://www.w3schools.com/python/matplotlib_intro.asp"><strong>W3School</strong></a></p><p>To make your graphs more visually appealing <a href="https://www.practicaldatascience.org/notebooks/class_5/week_5/46_making_plots_pretty.html">Read This</a></p><h4>Kaggle and Datasets</h4><p>Datasets are collections of big data converted into rows and columns which are mostly in Excel files. <a href="https://www.kaggle.com/"><strong>Kaggle</strong></a> is a site for already-built datasets. <strong>Practice Pandas and Numpy</strong> (at least 2 months).</p><p>You can practice on this <a href="https://www.kaggle.com/datasets/uciml/iris"><strong>dataset</strong></a><strong> </strong>(Hit download). This is a dataset on Iris flower and has columns like shape and length etc.</p><p>Make sure that after learning the basic syntax of Numpy and Pandas, you should be able to clean the dataset, remove empty rows, and calculate the mean.</p><h4>When are you ready?</h4><p>You are ready for the next phase when you can:</p><ul><li>Download and clean datasets.</li><li>Manipulate and reshape data using <strong>Pandas</strong>, demonstrating your ability to handle real-world data challenges.</li><li>Slice arrays and find mean, mode iterate, and rearrange lists with <strong>NumPy</strong>.</li><li>You can create basic visualizations that communicate your data insights.</li></ul><h3>Phase 3: Machine Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s325KlkjLojWY8dl" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You are probably saying <strong>Finally!</strong> But this is just the start. Now comes the hard part.</p><p>I started with TensorFlow. There is a whole discussion on which is better. TensorFlow or PyTorch. They are 99% the same for starting. After just learning the basics of one you can easily adapt to the other</p><h4>how I learned Beginner ML</h4><p>To be honest I learned ML the hard way. I remember hearing the word “Neural Networks” for the first time and it seems very interesting that a computer can mimic human behavior and you can train it.</p><p>Note: Always start with Machine Learning then go to Deep Learning (like TensorFlow or PyTorch)</p><p>Then I came across <a href="https://youtu.be/w8yWXqWQYmU?feature=shared">Neural Network from Scratch</a>. Because I knew intermediate Python I thought how hard it could be. That was pretty hard.</p><p>I remember hitting pause every few seconds, trying to figure out what was going on. Each time, I’d stop and ask ChatGPT about every single line of code and function. Then I’d open a new tab, search YouTube again, and dive into another concept. It was a rabbit hole</p><p>I watched the whole series of<strong> </strong><a href="https://www.youtube.com/watch?v=aircAruvnKk&amp;list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi">3Blue1Brown Neural Network</a> and wrote the concepts on paper. Then onto <a href="https://www.youtube.com/@mrdbourke">Daniel Bourke</a> and <a href="https://karpathy.ai/">Andrej Karpathy</a> and now I&#39;m learning NLP and computer vision</p><p><strong>TensorFlow or PyTorch? </strong>This is the most asked question with no perfect answer. They are almost the same with the same functions and documentation. Some people say that PyTorch is more user-friendly. But I chose Tensorflow because I liked its logo(yeah that&#39;s it).</p><ol><li><a href="https://youtu.be/tpCFfeUEGs8?feature=shared">Daniel Boruke’s </a><a href="https://youtu.be/ZUKz4125WNI?feature=shared">YouTube</a> <a href="https://youtu.be/tpCFfeUEGs8?feature=shared">TensorFlow course part-1</a></li><li><a href="https://youtu.be/ZUKz4125WNI?feature=shared">Daniel Bourke’s YouTube Tensorflow course part-2</a></li><li><a href="https://karpathy.ai/">Andrej Karpathy Website</a></li><li><a href="https://www.tensorflow.org/api_docs/python/tf">Tensorflow Documentation</a> <strong>(Most important)</strong></li><li><a href="https://www.youtube.com/watch?v=p1hGz0w_OCo&amp;list=PLcWfeUsAys2k_xub3mHks85sBHZvg24Jd">ML algorithms from scratch Assembly AI</a> (Free) <strong>(important)</strong></li><li><a href="https://youtu.be/w8yWXqWQYmU?feature=shared">Neural Network from Scratch</a> (Free) <strong>(important)</strong></li></ol><h4>Why These?</h4><p>I was looking for someone who could explain every line, function, and variable of ML code, and Daniel Bourke and Andrej Kaparthy did it <strong>Perfectly. </strong>Most importantly, they always give you a reference to where to find anything and learn yourself</p><h4>Other Resources</h4><ol><li><a href="https://www.coursera.org/learn/machine-learning">Andrew Ng’s Machine Learning on Coursera</a> (Free)</li><li><a href="http://bit.ly/dlcoursera">Andrew Ng’s deeplearning.ai course on Coursera</a> (Paid)</li><li><a href="http://www.fast.ai/">Fast.ai Deep Learning Course (part 1)</a> (Free)</li><li><a href="http://www.fast.ai/">Fast.ai Deep Learning Course</a> (part 2) (Free)</li></ol><h4>Books</h4><p>Personally, I am not a book guy for studying ML but if you are, Then the most popular ones are</p><ol><li><a href="https://www.amazon.com/gp/product/1491962291/ref=as_li_qf_asin_il_tl?ie=UTF8&amp;tag=mrdbourke-20&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=1491962291&amp;linkId=513b2171e7770c3e0bcb4697a698ed10">Hands-On Machine Learning with Scikit-Learn and TensorFlow</a></li><li><a href="https://amzn.to/2StIqrn">Artificial Intelligence: A Modern Approach</a></li></ol><h4>Maths</h4><p>You might be wondering why I never mentioned Maths till now. Well, you don’t need it before reaching here. Now my advice is:</p><blockquote>Learn Math as you do machine-learning code. This way you will know what you are writing.</blockquote><p><strong>For Probability and Statistics:</strong></p><ul><li><a href="https://www.youtube.com/watch?v=XZo4xyJXCak&amp;list=PL0o_zxa4K1BVsziIRdfv4Hl4UIqDZhXWV&amp;pp=iAQB">The Organic Chemistry Tutor</a></li></ul><p><strong>For Neural Networks:</strong></p><ul><li><a href="https://www.youtube.com/watch?v=aircAruvnKk&amp;list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi">3Blue1Brown Neural Network</a></li><li><a href="https://www.youtube.com/watch?v=OmtkvAp2OL0&amp;list=PLSQl0a2vh4HDgW9TkGULy1EWRE7rpEwjY">Khan Academy Intro to Artificial Intelligence</a></li></ul><p><strong>For Linear Algebra:</strong></p><ul><li><a href="https://www.youtube.com/watch?v=fNk_zzaMoSs&amp;list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab">3Blue1Brown Linear Algebra</a></li></ul><p>3Blue1Brown was a door opened to me. I never hated maths but never liked it also because I always thought, <strong>When in life will I use this? </strong>His videos were the answer.</p><h3>Phase 4: Deployment and Testing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YBt3u_bQtbJoc6r9" /><figcaption>Photo by <a href="https://unsplash.com/@wocintechchat?utm_source=medium&amp;utm_medium=referral">Christina @ wocintechchat.com M</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Till now you’ve done the ML part clearly, preprocessing data, feature engineering, model training. When you’ll go to an interview and they ask: “How do you handle model drift in production?” or “How can other people use your model?” you’ll be be blank.</p><p><strong>Note: Deployment is a practice-first concept and not course-first. You can just take a crash course and learn from the documentation while making a project (It’s the most Intuitive way)</strong></p><p>One thing to must note is different techniques have different purposes. They combine to make one full system</p><pre>| Technique  | What It Actually Is       |<br>| ---------  | ------------------------- |<br>| API        | Serving method            |<br>| Docker     | Packaging method          |<br>| MLflow     | Lifecycle management tool |<br>| Cloud      | Infrastructure location   |</pre><p>Now, as this is a more software engineering concept, you either have no experience with the tools and techniques or you have experience with the fundamentals of these concepts not specifically in ML. So I’ll target both and give resources accordingly</p><p>Some of the different techniques of Deployment are:</p><h4>1. API Deployment</h4><p>APIs can serve as a midpoint between users and an application (containing your model).</p><p><strong>A simple API architecture:</strong> Client → API → Model → Response</p><p>If you’re new to APIs:</p><ol><li><a href="https://youtu.be/5atR70lV1fs?si=W6jZwJ7lCSwDHxP9">APIs 101</a> (A crash course) (YouTube)</li><li><a href="https://roadmap.sh/api-design">Roadmap to API Development</a></li><li><a href="https://www.datacamp.com/courses/introduction-to-apis-in-python">Introduction to APIs in Python</a> (Datacamp Course) (Free)</li><li><a href="https://www.udemy.com/course/fastapi-the-complete-course/">FastAPI Complete Course 2026</a> (Udemy) (Paid)</li></ol><p>After you’ve got the basics then it’s easy and apply these techniques in ML</p><p>If you’ve experience with APIs:</p><ol><li><a href="https://youtu.be/5PgqzVG9SCk?si=F904MkC8OgQKHl_h">Creating APIs For Machine Learning Models</a> (A Crash Course) (YouTube)</li><li><a href="https://youtu.be/0sOvCWFmrtA?si=Q_V0YGd6fEGup7Pr">Python API Development</a> (19 hour course) (YouTube)</li><li><a href="https://www.udemy.com/course/nlp-with-bert-in-python/">Deploy ML models In Production with FastAPI and Docker</a> (Paid)</li></ol><h4>2. Containerization</h4><p>Containerization means a <strong>sealed box</strong> for your project. Inside the box, you put:</p><ol><li>Your model</li><li>Code</li><li>All libraries</li><li>The exact Python version</li></ol><p>When someone runs that box on their computer, it works exactly the same way it worked on yours. No “it works on my machine” problems.</p><p>Docker is a platform that helps us containerize the code (there are many others).</p><p>I think this one is most fun and easy to use. Because here you go from just a personal project to production-grade.</p><ol><li><a href="https://youtu.be/DQdB7wFEygo?si=Zxd_1b7hdz-psZOQ">The Only Docker Tutorial You Need To Get Started</a> (Youtube) (Best Crash Course)</li></ol><p>2. <a href="https://www.youtube.com/watch?v=zQVYjmGFvv0">Docker for Machine Learning, AI, and Data Science</a> (YouTube)</p><p>3. <a href="https://github.com/veggiemonk/awesome-docker">Awesome Docker</a> (GitHub)</p><p>Honestly only these can be enough, but for course learners there are:</p><p>4. <a href="https://www.coursera.org/learn/virtualization-docker-kubernetes-data-engineering">Virtualization, Docker, and Kubernetes for Data Engineering</a> (Coursera) (Duke University)</p><p>5. <a href="https://www.coursera.org/learn/deploy-applications-docker-build-real-world-projects">Deploy Applications with Docker: Build Real-World Projects</a> (Coursera) (EDUCBA)</p><h4>3. Experiment Tracking &amp; Testing</h4><p>The concept of <strong>Experiment Tracking</strong> may look too advance but its rather simple and very useful.</p><p>Suppose you ran a model to house prices, first you used Linear Regression with some evaluation metrics and visualizations. Now you changed the model to Random Forest with different metrics and visualizations. How do you compare both? Probably not by undoing the code and running again (hilariously maybe you’re doing it).</p><p>To test and track your models like above we use <strong>MLFlow </strong>(there are other options) which is a platform that provides us with a good interface of different runs of a model. This concept alone, separates a one-file project with the complete end-to-end project</p><ol><li><a href="https://mlflow.org/docs/latest/ml/">MlFlow Documentation</a></li><li><a href="https://youtu.be/6ngxBkx05Fs?si=VFpkR3djNxUE4fLN">MLFlow Tutorial</a> (YouTube) (Crash Course)</li><li><a href="https://youtu.be/wb-ZxtIwSTA?si=p3Vl_hPeLLlFyqMM">MLFlow: Infrastructure for a Complete Machine Learning Lifecycle</a> (YouTube)</li><li><a href="https://www.udemy.com/course/mlflow-course/?srsltid=AfmBOoptqPpt_E1wZIePoH89LQwGssWqKelB2kI6Rzen_CQ8_DnrRc4k">MLFlow in Action</a> (Udemy) (Paid)</li><li><a href="https://www.coursera.org/learn/mlops-mlflow-huggingface-duke">MLOps Tools: MLflow and Hugging Face</a> (Coursera) (Paid)</li></ol><h3>Projects</h3><p>Learning new tools will get you nothing if you cannot test those tools and knowledge. When you are following courses and tutorials, you are in your comfort zone. So when you learn something new, test it out, even if it is small.</p><p><a href="https://github.com/practical-tutorials/project-based-learning?tab=readme-ov-file#python"><strong>Find Projects for ML here</strong></a></p><p><strong>Other Resources</strong></p><ol><li><a href="https://huggingface.co/">HuggingFace </a>(Built-in Models, Datasets)</li><li><a href="https://github.com/">GitHub</a> (Source codes, datasets, community)</li><li><a href="https://colab.research.google.com/notebooks/">Google Colab</a> (will help when you run codes with computational load)</li></ol><h4>When are you ready?</h4><p>You are ready for the next Phase when you can:</p><ul><li>Build simple models like regression models and classification Models.</li><li>Understand Hyperparameters like learning_rate and activations</li><li>Built models like <strong>decision trees</strong> and <strong>KNN</strong>, and you understand how to tune these models.</li><li>Explain the difference between <strong>training accuracy</strong> and <strong>test accuracy</strong> and know when a model is overfitting.</li><li>Deploy and test models. At least monitoring them through MLFlow</li></ul><h3>Phase 4: Advanced Resources</h3><p>Now I assume you know basic to intermediate Machine Learning, but if you are confused as to where to go next, Follow this:</p><h4>Natural Language Processing (Currently doing)</h4><p>NLP is a great start to advanced ML. Don’t get too lost searching for perfection, and think you can make ChatGPT yourself at one go. learn with patience and practice on pre-trained models, which you can find on <a href="https://huggingface.co/">HuggingFace</a></p><ul><li><a href="https://keras.io/guides/keras_nlp/getting_started/">Keras NLP Documentation</a></li><li><a href="https://www.youtube.com/watch?v=VMj-3S1tku0&amp;list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&amp;pp=iAQB">Andrej Karpathy Neural Networks: Zero to Hero</a> (free)</li><li><a href="https://www.coursera.org/specializations/natural-language-processing">NLP Specialization by Andrew Ng on Coursera</a> (paid)</li></ul><h4>Generative AI</h4><p>This is more complex than NLP as it requires complex architecture and deeper mathematical understanding. It requires massive data and images, but in the end, the result is worth all the effort.</p><ul><li><a href="https://youtu.be/2IK3DFHRFfw?feature=shared">Generative AI in a nutshell</a> (free) <strong>(start here)</strong></li><li><a href="https://youtu.be/mEsleV16qdo?feature=shared">FreeCodeCamp Generative AI full course</a> (free)</li></ul><h4>Computer Vision</h4><p>This is one of the most demanding and easy-to-learn branches of ML. It seems like it is difficult, but in reality, it requires practice on OpenCV and image classification.</p><ul><li><a href="https://youtu.be/IA3WxTTPXqQ?feature=shared">FreeCodeCamp Deep Learning for Computer Vision</a> (Free) (start here)</li><li><a href="https://www.geeksforgeeks.org/computer-vision/">Geeks for Geeks Computer Vision Tutorial</a> (Free)</li><li><a href="https://www.youtube.com/watch?v=M6jukmppMqU&amp;list=PLgNJO2hghbmg0-6EbYR3MJkHJcneXEmwx&amp;pp=iAQB">OpenCV Basics by Nicholas Renotte</a></li></ul><h3>Why Am I learning from online Resources?</h3><p>Learning to learn is a skill gained through experimentation. I aim to teach the right way clearly so that a beginner doesn&#39;t go around every course on ML.</p><p>Another reason is <strong>Flexibility. </strong>I can learn any ML/DL, whether it is some beginner concept I forgot or an advanced course, I can just google it and have multiple resources to learn from</p><p>On top of that by <strong>learning Machine learning in early stages</strong>, I can network with people in this field and learn from them. And when you learn something unexpectedly early in your life, and show what you know, people around you start noticing you and give referrals. This is how you can grow your network</p><h4>Is this It?</h4><p>No. This article will keep getting updates for you to learn. I will try my best to gather the best resources in even the most advanced ML concepts and as I learn myself, I will teach you too</p><p>Let me know what you think</p><p>Let’s connect through <a href="https://www.linkedin.com/in/umer-farooq-34b0401b6/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4dd361aefbb" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/the-ultimate-beginner-to-advance-guide-to-machine-learning-b4dd361aefbb">The Ultimate Beginner to Advance guide to Machine learning</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is BackPropagation: Intelligence for Everyone (Part-4)]]></title>
            <link>https://medium.com/@umerfarooq230/what-is-backpropagation-intelligence-for-everyone-part-4-00e2c9dfc068?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/00e2c9dfc068</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[backpropagation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Tue, 01 Oct 2024 19:39:04 GMT</pubDate>
            <atom:updated>2024-10-01T19:39:04.994Z</atom:updated>
            <content:encoded><![CDATA[<h3>Recap</h3><p>In the last article of this series, we learned about Gradient Descent, which is the process by which a Neural Network determines the cost function, a function that helps minimize the errors and makes the Neural Network make good decisions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyaOdOGOg0USKCh90zRyrA.png" /><figcaption>Source: By Mbali on <a href="https://www.analyticsvidhya.com/blog/2023/01/gradient-descent-vs-backpropagation-whats-the-difference/">Analytics Vidhya</a></figcaption></figure><h3>Feedback and Improve</h3><p>Remember how the cost function works? We take the average of all the cost functions (predicted value - True Value) and add them. In the BackPropagation we take all the cost values of a layer and feed it back to the previous layer which then assigns new weights and biases. Let’s make it simpler by only taking one example for now</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*689yOC_jyMzQNHdN8ekFOw.png" /></figure><p>Suppose we are at that stage where our Neural Network is giving random guesses and each output Neuron has a random value. The thing here is we can <strong>only control weights and biases. </strong>We want to tune our Network to increase the value of number 9 and decrease every other value.</p><p><strong>(Remember: Weights are the random values which are the strengths of the connections between Neurons and biases are the values given to limit the activations till a certain value is reached)</strong></p><p>Suppose the activation of 9 is 0.2 and 5 is also 0.2</p><p>We need to control and tell our Network that increasing the activation of 9 is more important than decreasing the same activations of other numbers. Let’s focus on number 9 having an activation of 0.2</p><pre>0.2 = sigmoid (W0A0 + W1Aa + ... + Wn-1An-1 + B)</pre><p>Now we have 3 different nudges we can change to increase the activation of 9.</p><ul><li>Increase the Bias</li><li>Increase the Weights</li><li>Change Activation (from the previous layer)</li></ul><p>Each weight has a different influence on the change of activation. The brightest (more valuable) ones can have the biggest effect since they are multiplied by the larger value</p><p>When we talked about Gradient Descent, we didn&#39;t just make the values lower and higher randomly. We care about which ones have the strongest impact on the values</p><p>Talking about the activations, To increase the activation of our number 9, we can see which Neurons have the bigger values and which ones have lower values. Then we can decrease the ones with lower values and increase the ones with higher values. That is for the activations of the previous layer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*mEAM5NthfeqEAykugyyicg.png" /><figcaption>Source: <a href="https://www.geeksforgeeks.org/backpropagation-in-neural-network/">GeeksForGeeks</a></figcaption></figure><h3>BackPropagtion (Propagating to the previous layer)</h3><p>Let’s zoom out a bit and notice that we were talking about only one number 9. Each number has its own set of weights and biases that decide which neurons should increase their value and which should decrease</p><p>Looping through the Neural Network, adjusting the weights and biases of every Neuron, and feeding them to the previous layer to adjust the activations is called <strong>BackPropagation. </strong>Because we are propagating in the backward direction</p><p>Taking the average of each number’s neuron value and adding them up is the value of the negative of the Gradient.</p><h3>Stochastic Gradient Descent</h3><p>But doing all the BackPropagation one by one on all the numbers requires great computational power. To make this process fast we do Stochastic Gradient Descent. In this process, we shuffle the data and make mini-batches of it, so that the computer reads and manipulates each batch at a time.</p><p>It is like stepping down a hill with no clue and taking small steps. Some steps are wrong, some are right. If we don&#39;t shuffle and mini-batch the data, It is like our Network is stepping down the hill in the right direction very slowly and carefully</p><p>Let’s summarize everything</p><h4><strong>Calculating the Error for Each Neuron in the Output Layer:</strong></h4><ul><li>The error is the difference between the predicted value and the true value (the cost function). Using this error, we compute how much each weight and bias contributed to the error in that specific neuron. This contribution is known as the <strong>gradient</strong> (from <strong>Gradient Descent</strong>).</li></ul><h4><strong>Propagating the Error Backward:</strong></h4><ul><li>Once we calculate how much each weight and bias in the output layer contributed to the error, we move backward to the previous layer. The idea is to <strong>adjust the weights and biases in the previous layer</strong> so that they contribute more effectively to reducing the error. Doing this for all the Neurons of each number (0 to 9) and adding them is called <strong>BackPropagation</strong></li></ul><h4>Adjusting Weights and Biases:</h4><ul><li>Now we update the new weights and biases and go for the next pass.</li></ul><p>The same process is repeated for every layer, propagating the error backward through the network, all the way to the input layer. This ensures that the entire network is “learning” how to adjust itself to minimize the cost function.</p><h3>Is this It?</h3><p>As we have learned about the architecture of a Neural Network, this is the beginning of Deep Learning. Next, we will learn how we can make a simple Neural Network in Python and learn about optimizers, learning rates, and different techniques that build upon these foundations to create powerful models.</p><p>— Umer Out</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00e2c9dfc068" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 4 Questions To Be Your Best Version]]></title>
            <link>https://medium.com/@umerfarooq230/the-4-questions-to-be-your-best-version-15b6e6769af2?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/15b6e6769af2</guid>
            <category><![CDATA[living-with-purpose]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[improvement]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 19:35:54 GMT</pubDate>
            <atom:updated>2024-09-29T19:35:54.809Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a breakdown of the video by <a href="https://youtu.be/-oXigF0frLE">Clark Kegley</a> and my addition to these questions</p><p>In this world, although we have an equal amount of time some people have really different results. The results that everyone wants. This is not about time management and how to make a calendar and color code. Nope</p><p><strong>Productivity. </strong>The go-to word for every influencer. Clark says that most productivity tips just tell us how to be busy. Real Productivity doesn’t have to be busy. It’s about getting the work done which really matters and affects your life.</p><h3>Your Life</h3><p>This is the graph Clark talks about in the video. This is a normal life chart and how normal people traverse through this period of life leads to misery and guilt</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DRqRBGVHa4pV4z0bJkWK5w.jpeg" /></figure><p>How a normal person will go through life is this:</p><p>Soak up every piece of knowledge they can in the <strong>young phase</strong>. Do that trending course or degree, Go to college (if they are smart), Do that internship in a big company. Get hired by that company. And then be ready for the <strong>Adult Phase. </strong>Now from here climb the cooperate ladder and do robotic work for the next 40 years to save up for the family and retirement. After entering the <strong>Old Phase,</strong> it is just wait and wait till someone buries your body</p><p>(Living a normal life isn’t bad but being a burden on yourself is)</p><h4>Living or Preparing to Live</h4><p>Watching videos about Productivity or reading books isn’t living but preparing to live. There should be a point in your life where you go out there, have some failures and Live</p><p>Most people have this idea in mind “When I retire, I will have money and freedom and only then I can truly live the life”.</p><p>The truth is, if you don&#39;t enjoy the Adult phase, you are missing life, because in that part of life, you have optimistic health and time and freedom</p><h3>The 4 Questions</h3><p>Clark gives these amazing questions and if you answer these 4 questions honestly, it will change your perspective of seeing things</p><h4>1. What would you do first if you were fearless and could not fail?</h4><p>Most of the time we don&#39;t start something because we are scared. Remember a time when you had a great idea but somehow you just shot it down? If you have seen someone starting a business and you have an idea of how to do that business in a better way, a flood of negative thoughts will be in your mind. “It&#39;s a waste of time”. “I am very busy for that”. “This is not going to work”.</p><p>That is because of fear of failure. So answer this question with honesty and see what results you get</p><h4>2. What will I be most proud of 1 year from today?</h4><p>Clark is talking about being proud of your character. Making plans and everything is one thing but giving them your time and energy is another thing and this is what you have to focus. Think of this concept in reverse order.</p><p>Will you be proud that you fixed your health? Or getting out of that crappy job? Or starting that business and taking that leap?</p><p>And this <strong>Does not mean </strong>that you start to hate talk to yourself in the mirror and force yourself to do all that work. This will burn you out in no time and you will have done nothing.</p><h4>3. What are the 3 things that I need to stop doing?</h4><p>Achieving success is not only pursuing it with all your might. It is also about keeping track of the things that come in the way and distract you. These distractions are taking your time and energy that you don&#39;t even know about. And this can demotivate you out of your goals</p><p>Can’t start that business? Try to stop spending on useless things. Can’t lose the weight? Track your calories and stop binge eating.</p><p>This is the most important question because we often neglect that there are some habits that we have to leave to become the better self</p><h4>4. What advice would I give someone if they were in my situation?</h4><p>Humans are great at giving advice but horrible when it comes to act upon it. If you are lost and don&#39;t know what to do, take a step back and put someone else in your shoes.</p><p>Where did they go wrong? Which was the wrong step that lead to this situation? Now it has been done, what now?</p><p>Human mind is one incredible thing and we know it when something is wrong. So not being you for sometime can bring solutions to your mind you would have never thought were possible</p><p>Hope you learned something. Let me know your thoughts :)</p><p>— Umer out</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15b6e6769af2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Gradient Descent (Intelligence for Everyone Part-3)]]></title>
            <link>https://towardsdev.com/what-is-gradient-descent-intelligence-for-everyone-part-3-8686bdebfe64?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8686bdebfe64</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 11:34:43 GMT</pubDate>
            <atom:updated>2024-09-24T11:34:43.395Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*el0CtVdEy3nXoey8A5-fbg.jpeg" /><figcaption>Image Credits: O’Reilly Media</figcaption></figure><p>Let’s understand what gradient descent is in Machine Learning and how it works</p><p>This is the most important step in any Neural Network because in this step we will see the math behind a machine “learning” how to adapt to different inputs</p><p>We will use MNIST dataset in which there are thousands of examples of handwritten numbers from 0 to 9</p><h4>Cost/Loss Functions</h4><p>We always split our data into 2 separate sets</p><ol><li>Training Data</li><li>Testing Data</li></ol><p>suppose our desired number is 9</p><h4>Structure of Neural Network</h4><p>Suppose there are 4 total layers in this network where 2 are hidden layers and the other two are input and output layers respectively. The network in the first hidden layer will fetch the edges of the images and pass the information to the next hidden layer where it does the same on the structure of the input letter like the circle at the top of number 9 and a straight line below.</p><p>While training we feed our neural network with training data and make sure to shuffle it (so each value has an even probability of occurrence). As our given values have a totally random order, it is very possible that our Neural Network will be horrible at predicting the number at first (referring to the handwritten numbers data). It will give each neuron some value whereas we only want to give value to the neuron that represents our desired number</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*pOTqngRhsIqBnHZ9iNpSJQ.png" /><figcaption>MNIST dataset</figcaption></figure><p>So whenever our model predicts a number wrong we have to tell it somehow that “No this prediction is wrong!” How do we do it?</p><p>Mathematically speaking we develop a <strong>Cost Function </strong>in which we take the predicted wrong value, subtract it from the true values, square them, and finally add them to a single value. A Python code doing a cost function looks like this</p><pre>import math<br><br>predicted_value = [.01102, .05796, .09402, .07745, .08499]<br>true_value = [0.00, 0.00, 0.00, 1.00, 0.00]<br><br>sum_squared = 0<br><br>for i,j in zip(predicted_value, true_value):<br><br>        k = (i-j)**2<br>        sum_squared += k<br><br>print(sum_squared)</pre><p>Let’s look into a graph. Another name for gradient descent is minima. A minima is the last point on a curve given any value. We start at a point anywhere in the range from 0 to 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1JhPIwGaLDO_MWN2fNWILw.jpeg" /><figcaption>By <a href="https://commons.wikimedia.org/w/index.php?curid=102907088">Serge Helfrich</a> — Own work, CC BY-SA 4.0,</figcaption></figure><p>and Then after finding the slope for each of our test inputs, we can find out whether our value should go on the positive side or negative side to find the lowest point. <strong>Minimizing the cost function</strong> <strong>is the goal of gradient descent</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqgl08Z9BoaHMRknnh43cQ.jpeg" /><figcaption>By <a href="https://commons.wikimedia.org/w/index.php?curid=102907088">Serge Helfrich</a> — Own work, CC BY-SA 4.0</figcaption></figure><h3>Gradient Descent</h3><p>Consider two inputs X and Y on a plane, instead of finding the slope multiple times our question should be which is the fastest way we should step in so that our output decreases? just like stepping downhill</p><p>In multivariable calculus, the <strong>gradient</strong> is a vector that points in the direction of the steepest ascent of a multivariable function. Simply put it points the direction uphill. Taking the negative of this function will give us the direction of the downhill</p><pre>Gradient = ∇f(x,y) <br>Gradient_Descent = -∇f(x,y)</pre><p>The algorithm of a gradient descent is:</p><ol><li>Calculate ∇f</li><li>A small step in -∇f</li><li>repeat</li></ol><p>This repetition of small steps to -∇f is called the <strong>Gradient Descent.</strong></p><p>These <strong>small steps </strong>are also called<strong> the learning rate.</strong> The lower the learning rate the more efficient our network is and will learn accurately. Suppose we take big steps walking downhill it is not safe and good for us rather we take small and easy steps. It may take longer but we will reach there safe and sound. Learning Rate works exactly like that. We will see more about it in the next part</p><p>Think of gradient descent as trying to find the lowest point while walking downhill. For each step, the network adjusts its weights to move closer to this “minimum” value.</p><p>Another way of putting this into visualization is Gradient Descent tells us for each value of the vector which one should increase or decrease to find its minima and which variable impacts the function most. For example, if we have a function like</p><pre>x=1<br>y=1<br><br>∇f(1,1) = [3,1]</pre><p>We can say that X has more impact than Y. That is how our Network grows with each step and minimizes the cost function.</p><p>Gradient Descent and cost functions are applied in <strong>Backward Propagation </strong>which we will cover in the next part but this alone is amazing how any machine if trained correctly can decide with great accuracy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8686bdebfe64" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/what-is-gradient-descent-intelligence-for-everyone-part-3-8686bdebfe64">What is Gradient Descent (Intelligence for Everyone Part-3)</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[But what is Forward Propagation? (Part-2)]]></title>
            <link>https://medium.com/@umerfarooq230/but-what-is-forward-propagation-part-2-5bbed9fba106?source=rss-4bee6e00ed0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bbed9fba106</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[forward-propagation]]></category>
            <dc:creator><![CDATA[Umer]]></dc:creator>
            <pubDate>Wed, 21 Aug 2024 14:46:37 GMT</pubDate>
            <atom:updated>2024-10-01T11:25:27.930Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Forward Propagation? Intelligence for Everyone(Part-2)</h3><p>This article continues my series, “Intelligence for Everyone.”</p><p>In the previous article, I explained what Neural Networks are, how each layer is responsible for the next layer’s inputs, and the types of Neural Networks. But in the previous article, I said that there are certain Knobs and Dials we have to turn to get certain output. Let’s look into that.</p><h3>What is Forward Propagation?</h3><p>Forward propagation refers to the flow of information from the input layer, through the hidden layers, to the output layer of a neural network. During this process, the input data is transformed by the parameters (weights and biases) of the network to produce the final output.</p><h3>Weights and Biases</h3><p>Each Neuron of every layer contains some information or a certain number between 0 and 1. These numbers are called <strong>Activations, </strong>which are also our input.</p><p>Suppose our model’s second layer detects the edges like in the image below. The parameters we adjust and experiment with to enhance how the model detects edges are known as <strong>Weights</strong>.<strong> </strong>These Weights may change many times before getting the right output. To see if there is an edge we also add some negative weights with the surrounding pixels such that the middle pixels (positive weights) will be green and the outer pixels (negative weights) will be red</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGRNM9EY_VM7BOqD0EMfkw.jpeg" /></figure><p>Let’s say we have Activations A1, A2, and A3... An. weights W1,W2,W3 . . .Wn.</p><p>Now calculate the dot product of the corresponding weights and activations because the weights have multiple dimensions (rows and columns) while Activations are in one column. The dot product will result in a weighted sum</p><p>We also need to add a limit to our neurons so they will only be activated when we need them. Let’s say we only want our neuron to be activated when the weighted sum is greater than 10. For this, we will add a <strong>Bias </strong>for <em>inactivity. </em>So add a “-10&quot; to the end of our weighted sum</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y0_REX8BftR6NLW1cVixEw.png" /><figcaption>source: 3Blue1Brown (YouTube)</figcaption></figure><p>We also need some function that will squish the real number line and our weighted sum into the range of 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k3TfibHbJalfMpT5DqVTow.png" /><figcaption>source: 3Blue1Brown (YouTube)</figcaption></figure><p>For this, we use non-linear functions like Sigmoid or Relu (Rectified Linear Unit). Remember how the model changes the <strong>Weights</strong> itself, These functions help change them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*1CLVA-S4urVGiG4mOV_SIg.png" /></figure><p>So after calculating the weighted sum, adding the bias, and applying the Sigmoid function, this is how the final equation would look like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8X9lLFUyHOBcx716cUrWtg.jpeg" /></figure><p>This Python code will help you better understand how Forward Propagation works</p><pre>import numpy as np<br><br># Initialize weights, activations, and bias<br>weights = np.array([0.5, -0.2, 0.3, 0.8, -0.5])<br>activations = np.array([0.1, 0.4, 0.7, 0.3, 0.9])<br>bias = -10<br><br># Define the sigmoid function<br>def sigmoid(x):<br>    return 1 / (1 + np.exp(-x))<br><br># Forward propagation (weighted sum + bias)<br>weighted_sum = np.dot(weights, activations) + bias<br><br># Apply the sigmoid activation function<br>output = sigmoid(weighted_sum)<br><br># Print the output<br>print(f&quot;Weighted Sum: {weighted_sum}&quot;)<br>print(f&quot;Output after Sigmoid: {output}&quot;)</pre><p>Often this code is happening behind the model. However, it is crucial to know the math behind your model.</p><h3>Recap</h3><ol><li><strong>Activations </strong>are some number between 0 and 1 also known as inputs which are stored in a neuron</li><li><strong>Weights</strong> are the parameters that change consistently to adjust the learning capabilities of any model</li><li>Add a <strong>Bias</strong> for inactivity, which restricts neuron activation beyond a certain threshold.</li><li>We take the dot product of Weights and Activations and add them which results in a weighted sum</li><li>Squish the weighted sum using a non-linear function like Sigmoid or ReLu</li></ol><h4><strong>What’s Next</strong></h4><p>The above explanation was just one iteration in the Forward Propagation. Next, I will explain how Back Propagation works simultaneously with multiple forward propagation</p><p>Stay tuned because</p><h4>Intelligence is for everyone</h4><p>If you have any questions leave a comment. I also post on my <a href="https://www.linkedin.com/in/umer-farooq-34b0401b6/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bbed9fba106" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>