<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Winnie W. on Medium]]></title>
        <description><![CDATA[Stories by Winnie W. on Medium]]></description>
        <link>https://medium.com/@winniew314?source=rss-9cebad599e29------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YJYtWLZoGzcCSsND88h00w.png</url>
            <title>Stories by Winnie W. on Medium</title>
            <link>https://medium.com/@winniew314?source=rss-9cebad599e29------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:48:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@winniew314/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Looking at Artemis — PINNs]]></title>
            <link>https://medium.com/@winniew314/looking-at-artemis-pinns-2d5932965634?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d5932965634</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pinn]]></category>
            <category><![CDATA[differential-equations]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 03:09:13 GMT</pubDate>
            <atom:updated>2026-09-28T03:48:18.309Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Originally published at </em><a href="https://livnlearns.com/looking-at-artemis-i-pinns/"><em>https://livnlearns.com</em></a><em> — I would recommend reading there, cause latex didn’t import and I couldn’t be bothered :)</em></p><p>In <a href="https://livnlearns.com/looking-at-artemis/">Looking at Artemis</a> we built Artemis I’s trajectory from scratch: Earth, J2, the Moon, the Sun, solar radiation pressure, lunar J2, and one integrator. After four days it missed JPL Horizons by 7,426 km. Not because the physics was wrong, but because Orion isn’t a passive rock. It fires its engines, and a forward propagator has no way of hearing about it.</p><p>In <a href="https://livnlearns.com/differential-equations-v-physics-informed-neural-networks-pinns/">Differential Equations V</a> we went through the theory of physics-informed neural networks (PINNs): a network that fits data, but gets penalized every time it breaks a differential equation.</p><p>This post puts the two together. The question for the proof of concept is small on purpose:</p><p>If we hide four hours of Orion’s trajectory, can a neural network that knows gravity fill the gap better than one that doesn’t?</p><p>This post covers what worked, what didn’t, and what I’d change. You can find the code <a href="https://github.com/w-winnie/MLsandbox/tree/main/astro_sandbox/notebooks/artemis">here</a></p><h3>Picking a clean arc</h3><p>My first attempt used the full four-day Horizons window, and it went badly: the PINN did worse than a plain neural network with no physics at all, and its learned correction term grew to about 100× the Moon’s pull. A likely culprit was the trajectory correction burns. The physics loss assumes Orion is coasting, and the four-day window contains several thruster firings, not all of which were in my list of burn windows. Wherever the network was asked to obey coasting physics during a burn, it was being trained against the wrong equation.</p><p>So for the proof of concept we pick a 20 hour arc that sits between the first two Outbound Trajectory Correction burns: it starts about 45 minutes after OTC-1 and stops about 15 minutes before OTC-2. Every sample in it is a coast.</p><p>The data is Horizons’ state vectors for Orion (target -1023) relative to Earth’s centre, in the Earth-equator J2000 frame, every 10 minutes. That gives 121 samples. Over the arc Orion climbs from about 91,000 km to about 214,000 km from Earth.</p><h3>Simulating a blackout</h3><p>To test gap filling we hide the middle 20% of the arc (hours 8.0 to 11.8, 24 samples) and then train on every second sample outside it. That leaves 49 training points, about 40% of the arc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_D7idU62PiuXYMnB.png" /></figure><p><em>Figure 1. The 20 hour coast arc in the XY plane, with the hidden gap in red (left), and Orion’s distance from Earth over the arc (right)</em></p><h3>Physics, in Torch</h3><p>The gravity functions are the same ones from the first Artemis post, rewritten as batched Torch functions so gradients can flow through them. For this arc the physics prior is Earth’s point mass plus J2.</p><p>Propagating Earth + J2 for one hour from the Horizons initial state lands 0.07 km from Horizons. The physics code from the first Artemis post carries over fine.</p><h3>From a network to an acceleration</h3><p>The network takes normalized time τ = t / t_scale, which runs from 0 to 1 over the 20 hours, and returns a position. Velocity and acceleration come from differentiating the network with autograd, with the chain rule converting back from τ to seconds:</p><p>Because the loss needs a second derivative of the network, the activation has to be smooth. We use SiLU rather than ReLU, whose second derivative is zero almost everywhere.</p><h3>Baking in the initial state</h3><p>So a standard Neural Network for our data would look something like:</p><p>i.e. a smooth function in tau</p><p>the problem is if we initialize it randomly and evaluate g_theta(0)​ we get whatever the random weights the network arbitrarily produces, not necessarily zero, not r0, not anything meaningful</p><p>A randomly initialized network, at tau = 0 returns whatever its weights happen to produce. Rather than adding a loss term to make it start in the right place, we build the starting position and velocity into the architecture</p><h3>Simple MLP</h3><p>We usea simple MLP as a baseline, i.e. a simple non boundary corrected mlp scaled to original units</p><p>Approximates trajectory by directly mapping normalized time and space</p><h3>Centered MLP</h3><p>To give an MLP baseline since its already at a disadvantage not knowing ICs — we create a centered mlp which predicts averages instead of raw number</p><h3>Initial Conditions (Position)</h3><p>Now we use PINNs, by implementing a simple boundary constraint</p><p>spatial trajectory (r) as a function of normalized time (tau)</p><p>Where N_theta(tau) is the raw output of the mlp</p><p>r_scale is a scalar used to non-dimensionalize the spatial domain and prevents vanishing/exploding gradients</p><p>At tau = 0, the second term tau . N_theta(tau) strictly evaluates to 0, forcing r(0) = r_0 regardless of the network’s weights</p><p>Evaluating at tau = 0:</p><p>The initial velocity relies entirely on the untrained, raw output of the network N_theta(0). The optimizer must waste epochs learning this value, creating severe stiffness at t=0</p><h3>Initial State (position and velocity)</h3><p>Now we proceed to next step — to have the true velocity at tau=0 as well</p><p>To force r(0)=r0, we needed the free term to vanish to zeroth order at tau=0 hence we multiply by tau¹ now to additionally force r’(0)=v0 we multiply by tau²</p><p>you can think of this as just 2nd term of the Taylor series</p><p>Constrain both the initial position r_0 and initial velocity v_0:</p><h3>Residual</h3><p>Now we apply the same idea but to accelaration,</p><p>The equation of motion is dominated by a term we couldn’t compute exactly — Newtonian gravity, plus known perturbation models (harmonics, third body). Real trajectories still deviate slightly from what that physics predicts, and Delta_a(tau) is the network’s job — absorb whatever acceleration is left over once the known physics has been subtracted</p><p>here we are trying to model Delta_a = f(tau)</p><p>i.e. residual accelaration as a function of normalized time — this is a non parametric trajectory spline fitter for a single, fixed arc (doesnt take into account anything else)</p><h3>The loss</h3><p>The loss is a data term on the 49 training points plus a physics term on 512 collocation times, redrawn every epoch. Collocation times that land inside a burn window are rejected, so the network is never asked to obey coasting physics while an engine is firing. Each term is normalised: positions by r_scale (the largest coordinate on the arc, 205,232 km) and accelerations by a_char = μ / r_scale² (9.46 × 10⁻⁶ km/s²).</p><p>In the MLP we compute a Normalized Data Loss: Computes the Mean Squared Error (MSE) scaled by r_scale</p><p>In our PINN, we have:</p><p>Where the last term is only in residual Network where we want to model the missing physics</p><p>Cosine annealing shrinks the learning rate over training so the optimiser can settle into the narrow valleys that competing data and physics terms create, and gradient clipping stops an occasional wild second derivative from knocking training off course.</p><p>To know whether the physics is doing anything, the PINN needs rivals. Each rung of the ladder tries one approach, and every model that learns from data sees the same 49 training points:</p><p>1. <strong>Curve fits (1a, 1b).</strong> A cubic spline and a degree-7 polynomial through the training points. No physics, no network, just fitting the curve — i actually wrote a bit more about spline <a href="https://livnlearns.com/beyond-min-max-the-art-of-continuum-normalization-for-stellar-spectroscopy/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RTlHyriyKL6ay6CO.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9-qLAOmeucW3dxk9.png" /></figure><p>2. <strong>Data-only networks (2a, 2b).</strong> The same MLP size as the PINN, trained on positions alone. 2b centres and scales the outputs to give it a fairer start. This is data only networks — no physics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8LMMV5arzh61eOmz.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fb8iTVMLKPAbg8kL.png" /></figure><p>3. <strong>Head start only (3).</strong> The PINN’s exact architecture with r₀ and v₀ built in, but with the physics weight set to zero. This helps us see how effective is the PINN architecture is on its own (ignoring the physics loss we use in PINNs)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o3fUjENEXGrmgcF0.png" /></figure><p>Model 3 knows where the craft starts and how fast it’s going, and after that it’s just a curve-fitter. In the gap it has no data, so it draws a plausible smooth curve and misses by about 88 km.</p><p>4. <strong>PINNs (4a, 4b, 4c).</strong> 4a builds in only r₀ and has to work out v₀ itself. 4b builds in both. 4c adds a second small network δa(τ) that learns a correction acceleration for whatever Earth + J2 leaves out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NscvSw15rWjOsbnD.png" /></figure><p>In 4a the network has to work out Orion’s starting velocity itself. Getting within 2.2 m/s of 2,595 m/s (0.08%) sounds excellent. But a velocity error turns into position error linearly with time, so errors that small add up fast:</p><ul><li><strong>Rule of thumb:</strong> 1 m/s ≈ 3.6 km per hour.</li><li>2.2 m/s × 10 h (the middle of the gap) ≈ 79 km. The gap error is 88 km.</li></ul><p>A trajectory with a wrong v₀ misses those late points by more and more, so the data <em>should</em> force v₀ to the right value. It doesn’t, because the data term is weighted so weakly</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FB7DTXfdSl1mGxcj.png" /></figure><p>Model 4b has the same starting point but is also told “whatever curve you draw must accelerate the way gravity says — i.e. the PINNs loss” That rule reaches into the gap even though no data does, and it pins the curve to within about 5 km.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0AQmQ_BM6_dcwbGq.png" /></figure><p>5. <strong>Classical propagation (5).</strong> No learning at all: solve_ivp integrates Earth + J2 forward from the true r₀ and v₀.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p5xjfSJFFNZuJJP2.png" /></figure><h3>Overall</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EfICbIrNK60TmcRX.png" /></figure><p><em>Figure 3. Position error over the whole arc for every model (log scale). The hidden gap is shaded red.</em></p><p><strong>The physics term does real work.</strong> Model 3 and model 4b have identical architectures, identical data and the same built in r₀ and v₀. The only difference is whether the physics loss is switched on. Gap RMSE goes from 87.7 km to 5.5 km, 16 times better. That is the headline result of the proof of concept: the mechanics of a PINN (autograd accelerations, hard constraints, burn masked collocation) all work on real mission data, and the physics term measurably helps.</p><p>The data loss only ever looks at the 49 training times. The physics loss is checked at 512 random times per epoch, spread across the whole 20 hours, gap included. So the physics loss is the only part of training that supervises the gap at all. “Bending” is acceleration (the second derivative), which is why the network is differentiated twice.</p><p><strong>The PINN beats the propagator in the gap.</strong> Classical propagation (solve_ivp) uses the same physics and the same start state and gets 9.0 km in the gap. The PINN gets 5.5 km, so the training points are adding information that pure propagation doesn’t have. Model 5 and model 4b have the same physics and the same start. The solve ivp knows the start, applies the physics, step forward, and never looks at the data again. The physics is missing the Moon and Sun, so a small error builds up steadily. The PINN starts the same way, but it has also seen dots out to hour 20. Those late dots act like a faint tug on the far end of the curve. That tug straightens some of the drift in the middle.</p><p><strong>It can recover what it isn’t told.</strong> Model 4a is never given the starting velocity, and it works it out from the data and the physics to within 2.2 m/s of the true 2,595 m/s, an error of 0.08%. Because the position error from a velocity error grows in a straight line with time: 2.2 m/s over 10 hours is about 80 km of drift, which is close to 4a’s 88 km gap error. Orbital mechanics is unforgiving about velocity. A a wrong v0 makes the curve miss the hour 12–20 dots by more and more, so the data should correct it. It barely does, because the data term is too weak</p><p><strong>The curve fits win.</strong> A cubic spline (1.55 km) and a degree-7 polynomial (1.11 km) both beat the PINN in the gap, and neither knows gravity exists.</p><p>Over 4 hours, the path is extremely smooth, very close to a parabola. There are points every 20 minutes right up to both edges of the gap. A cubic spline is built out of cubic pieces, so it can represent a near-parabola almost perfectly, its error comes from the tiny part of the curve’s shape inside the gap that a cubic can’t reproduce. The degree 7 polynomial does even better because it’s fitted to the whole arc, so it “knows” the global shape of the curve. The spline isn’t clever. The test is simply easy for it — dense, clean, smooth data and a short gap.</p><p><strong>Adding a learned correction made things worse.</strong> Model 4c, the PINN with δa, goes from 5.5 km to 41.2 km. And δa is not learning the physics that’s missing. The notebook checks it against the Moon + Sun acceleration that Earth + J2 leaves out, The two diagnostic numbers:</p><ul><li>Size, 2.8× too big. δa isn’t carrying the Moon + Sun pull (1.9 × 10⁻⁸). It’s soaking up the network’s own wobble, which is about 10⁻⁷, so it lands in between (5.4 × 10⁻⁸).</li><li>Direction cosine +0.21. This measures whether two arrows point the same way: +1 is the same direction, 0 is at right angles (unrelated), −1 is opposite. +0.21 means about 78° apart, so δa is mostly unrelated to the real Moon + Sun pull.</li></ul><p>Once δa absorbs the disagreement, the physics stops constraining the curve, and 4c slides back towards the no-physics mode</p><p>4c sits between the other two, and its train error is basically model 3’s. That’s the evidence the physics got partly switched off.</p><p><strong>The data-only networks miss their own training points.</strong> Models 2a and 2b are 118 to 359 km off on the very points they were trained on.</p><p>The final loss of 10⁻⁶ looks tiny. But positions are divided by 205,232 km, so 10⁻⁶ means 0.1% per coordinate, and 0.1% of 205,000 km is about 200 km, the hard-IC architecture gets the starting point and starting velocity exactly right for free, so the network has less to learn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nuMPAge4BCS4ZlZ5.png" /></figure><p><em>Figure 4. Zoom on the gap with a straight-line trend removed from each axis, so the curvature of the arc is visible. The curve fits and the PINN (4b) sit on the truth at this scale; the data-only networks and 4a are tens to hundreds of km off.</em></p><h3>1. The PINN is a propagator in disguise</h3><p>Once again there’re 2 ways to fill a gap:</p><ul><li><strong>Interpolating:</strong> use data on <em>both</em> sides of the gap and bridge between them. A spline does this. Its errors are near zero at data points and largest in the gap.</li><li><strong>Propagating:</strong> start at t = 0 and step forward using physics. solve_ivp does this. Its errors start at zero and grow steadily with time, whether or not there&#39;s data.</li></ul><p>A PINN is <em>supposed</em> to interpolate using physics as the bridge. This one behaves like a propagator.</p><p>Look at Figure 3 again. From about hour 15 onwards, the PINN’s error curve lies on top of the classical propagation curve, and both end the arc near 40 km (38.0 km for the PINN, 40.9 km for solve_ivp). A blindfolded walker’s error grows smoothly with time. solve_ivp is the pure blindfolded walker. From about hour 15, the PINN&#39;s error line merges with it, and both end near 40 km. If the PINN were really using the data at hours 12-20, its error there would stay small, because those are points it was trained on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/0*bk-u3JzmdhjblQdo.png" /></figure><p><em>Figure 5. Train RMSE (hollow) against gap RMSE (filled). An interpolator should do worse on points it has never seen; every physics-driven model does the opposite. (The original MLP is left out: its train error is dominated by a 975 km miss at the very first sample.)</em></p><p>For an interpolator, error on the hidden points should be worse than on the training points. Every physics-driven model shows the reverse. That is the signature of a model integrating forward from t = 0: its error grows with time, and the gap happens to sit earlier in the arc than half the training points. The PINN starts from the exact r0 and v0, integrates Earth + J2 forward, and the training points at the far end barely pull it back.</p><p>The reason is in the loss scales. Each term is normalised by the size of its <em>signal</em>, not the size of its expected <em>error</em>. A 1 km position error costs about 8 × 10⁻¹² in data loss; the PINN’s final data loss of 2.6 × 10⁻⁹ corresponds exactly to its 18.2 km training RMSE. Now, the two ways the network could disagree with something:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A5AJoFgCSLRyvgeJ.png" /></figure><p><em>Figure 6. The two ways the network can disagree with something, priced in the loss units.</em></p><p>The loss adds two different kinds of error: position error (km) and acceleration error (km/s²). Wecan’t add km to km/s² directly, so the normalization quietly sets an exchange rate between them. Dividing positions by 205,232 km and accelerations by 9.46 × 10⁻⁶ km/s² means:</p><p>1 km of position error costs the same as 4.6 × 10⁻¹¹ km/s² of acceleration error (9.46 × 10⁻⁶ ÷ 205,232 = 4.6 × 10⁻¹¹.)</p><p>Now price the Moon and Sun at that rate. Their pull is 1.9 × 10⁻⁸ km/s², and 1.9 × 10⁻⁸ ÷ 4.6 × 10⁻¹¹ ≈ 420 km.</p><p>So as far as this loss is concerned, “admit the Moon exists” is as expensive as missing the data by about 420 km. The alternative, ignoring the Moon and following Earth + J2, only misses the data by about 19 km. The optimizer takes the deal. It isn’t malfunctioning, it is minimizing the loss we gave it.</p><p>A cleaner way to set the weights: dividing a squared residual by σ² is a Gaussian log-likelihood with standard deviation σ. So the current loss implicitly claims positions are trustworthy to ±205,000 km and the gravity model to ±9.5 × 10⁻⁶ km/s². More honest numbers would be about 1 km for positions and about 2 × 10⁻⁸ km/s² for Earth + J2 (the size of what it leaves out) i.e. λ_p ≈ 6 × 10⁻⁶ rather than 1, a shift of more than five orders of magnitude towards the data.</p><h3>2. The prior keeps the small term and drops the big one</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NRebtrZCmJ37BE3r.png" /></figure><p><em>Figure 7. Acceleration budget on the arc. Earth terms are evaluated at mid-arc (about 160,000 km); Moon + Sun and the network quantities are medians or RMS values from the notebook; solar radiation pressure uses the notebook’s Orion area, reflectivity and mass.</em></p><p>At 91,000 to 214,000 km from Earth, J2 is tiny, around 10⁻¹⁰ to 10⁻¹¹ km/s². The Moon and Sun, which the prior leaves out, are 50 to 1,500 times larger along the arc. A constant 1.9 × 10⁻⁸ km/s² acting for 20 hours displaces Orion by ½at² ≈ 50 km. That is the same size as classical propagation’s 40.9 km miss, and because of that, the PINN inherits that bias. More is explained in the <a href="https://livnlearns.com/looking-at-artemis/">Artemis I </a>post.</p><p>It is tempting to call the Moon negligible for the first day because its pull is tiny next to Earth’s. Next to a km-level accuracy target, it isn’t. N-Body acceleration would make this better</p><p>One more number in that chart, the PINN’s own physics residual (RMS about 1.0 × 10⁻⁷ km/s²) is larger than the Moon + Sun signal it is missing. So the physics loss isn’t yet converged tightly enough to tell the missing physics apart from its own fitting noise, which brings us to the optimizer.</p><h3>3. A free correction term makes the physics optional</h3><p>If δa can be any function of time, the network can always set it to exactly the leftover. Then the check passes for any curve, right or wrong. That’s what “δa = r̈ − a_model(r) satisfies the physics loss perfectly” means: r̈ is the curve’s own acceleration, and δa just fills in whatever is missing.</p><p>Ways to make δa identifiable: put the known Moon and Sun into the prior first so δa only has to find leftovers at the solar radiation pressure scale (a few × 10⁻¹² km/s²), as mentioned above as well the Nbody accelaration, we used in first notebook — I’ll follow up on that one in a separate final post.</p><h3>4. Adam alone stalls</h3><p>In PINNs, the two loss terms pull in different directions, and those terms differ in size by orders of magnitude (data around 10⁻⁹, physics around 10⁻⁵).</p><p>The data-only networks’ losses flatten out by epoch 1,000 to 1,500, while the cosine schedule still has the learning rate at half to three-quarters of its starting value, so they have stalled rather than run out of step size.</p><p>The centred MLP’s loss changes by 1% over its last 500 epochs, and the PINN’s physics loss improves by less than 3× over its last 2,000.</p><p>The data-only networks miss their own training points by 118 to 359 km, while a degree-7 polynomial fitted by least squares misses them by 0.36 km.</p><p>The original MLP’s error has equal lobes of about 400 km that cross zero near hours 1.7, 8.5 and 17.5: the network has captured the smooth, low-order shape of the arc and stalled there.</p><p>In the <a href="https://livnlearns.com/differential-equations-v-physics-informed-neural-networks-pinns/">PINNs post</a> I described a two-phase optimisation: Adam to find the right valley, then L-BFGS to settle to the bottom of it. This notebook only runs the first phase. Adding L-BFGS is probably the cheapest improvement on the list.</p><p>The model isn’t the only thing with problems. A few things about the test setup limit what these results can say.</p><p><strong>This is the spline’s home turf.</strong> The data is noise-free and smooth, there’s a training point every 20 minutes, and the gap is only 4 hours. Horizons isn’t raw tracking data either; it’s already a smooth trajectory product. Every one of those conditions favors a spline. The places where a spline struggles are exactly where physics should help:</p><ul><li><strong>Noise.</strong> A spline passes through every point, so it copies the noise and wiggles. Gravity doesn’t wiggle, so a physics loss smooths the noise out.</li><li><strong>Sparse data and long gaps.</strong> A spline’s error across a hole grows roughly with the fourth power of the hole’s length. A well-weighted physics bridge grows with the square. Short gaps favour the spline; long gaps should favour physics.</li><li><strong>Forecasting.</strong> Past the last data point a spline has nothing to go on. Physics keeps going.</li></ul><p>In other words, this proof of concept tested the PINN where it’s least needed.</p><p><strong>The PINN is handed the exact starting state.</strong> Models 3, 4b, 4c and 5 are given Horizons’ r0 and v0 exactly. In a real blackout you’d only know the starting state as well as the tracking before the blackout allowed. Model 4a shows what that costs: 2.2 m/s of velocity error is about 80 km by the middle of the gap.</p><p><strong>Where the gap sits matters.</strong> A propagator’s error grows with time since the start. A gap at 40–60% of the arc catches the PINN while its error is still 4–9 km. The same gap at 60–80% would land where its error is 9–25 km. A spline doesn’t care where the gap is. Sliding the gap across the arc would make the comparison fair.</p><p><strong>One run, one seed.</strong> Every number here comes from a single training run. Neural networks start from random weights, and a different start can give a different answer, so there are no error bars. On top of that, the 4b cell doesn’t reset the random seed the way the other runs do, so models 3 and 4b didn’t start from identical weights. A 16× difference is unlikely to be luck alone, but several seeds per model would settle it.</p><p><strong>The classical ceiling isn’t one.</strong> Classical propagation was given the perfect starting state and never looked at the data. That isn’t the best a classical method can do with the same physics and data, and the PINN already beats it in the gap. The fair rival is batch least-squares orbit determination. It asks: which six numbers (r₀ and v₀), propagated with my physics, best match all 49 points? Because it uses points at both ends, the error from the missing physics gets spread across the arc instead of piling up at the end. That’s what model 4a was trying to do with a neural network; orbit determination does it directly. It’s also the closed-loop approach flight dynamics teams actually use, as we saw at the end of the first Artemis post.</p><ol><li>Rebalance the loss. Sweep λ_p across 1, 10⁻², 10⁻⁴ and 10⁻⁶, and try the value from the exchange-rate argument (about 6 × 10⁻⁶). Too much physics weight gives a propagator that inherits the model’s bias. Too little gives a data-only network that can’t bridge the gap. The best value sits in between.</li><li>N-body propagator as mentioned previously. It should remove the 40–50 km bias from the missing third bodies.</li><li>Run Adam first, then L-BFGS on a fixed set of collocation points. The points need to stay fixed because L-BFGS works out the shape of the valley from its last few steps, and resampling the points every step would keep reshaping the valley under it.</li><li>Adding a batch least-squares orbit determination, with the same physics and the same 49 points, as the baseline to beat.</li><li>Add position noise (1 km and 10 km), thin the data to one point per hour, widen the gap from 20% to 60%, forecast the last 20% of the arc, and give v₀ a realistic error. Each of these targets a spline weakness, or the head start of knowing the exact state. This would even out the spline advantage.</li><li>Running at least five seeds per model, and sliding the gap across the arc, get error bars.</li><li>Use a constrained δa (for example a constant vector) on top of an n-body prior, with burn windows from both the public burn list and the Δv-residual detector. On this 20-hour arc, once the Moon and Sun are in the prior, only about 15 m of solar radiation pressure is left for δa to find. The burns are where δa has something real to learn.</li></ol><p>I guess my goal was applying PINNs to this problem which i discussed before <a href="https://livnlearns.com/looking-at-artemis/">here </a>— what looked like a simple curve — and here is what i learnt:</p><p>As a proof of concept, the pieces work. A neural network can be differentiated twice into an acceleration and checked against gravity. The starting state can be built into the architecture instead of requested through the loss. Collocation can skip over burn windows. And switching the physics term on makes the same network fill the gap 16 times better. On real mission data, that isn’t nothing.</p><p>But it isn’t yet what a PINN is meant to be: an interpolator that uses physics to bridge the gap. In its current form it’s a propagator, an expensive numerical integrator of an incomplete force model. Four things put it there:</p><ul><li>The loss weights make ignoring the data about 480 times cheaper than ignoring the Moon.</li><li>The prior keeps J2, which barely matters this far out, and drops the Moon and Sun, which do.</li><li>The correction term has enough freedom to make the physics optional.</li><li>Adam stops before reaching the bottom of the valley.</li></ul><p>On clean, dense data with a short gap, a cubic spline beats all of it.</p><p>The good news is that every one of those problems is specific and testable, and a couple come with numbers attached. Next time: rebalance the loss, add the Moon and Sun, finish the optimisation, and race orbit determination instead of a blind propagator. Then move to where a spline can’t follow: noisy data, sparse samples, long gaps, forecasting, and eventually the burns themselves.</p><p><a href="https://livnlearns.com/looking-at-artemis/">Looking at Artemis — Simulating the Artemis I Trajectory from Scratch</a> <a href="https://livnlearns.com/differential-equations-v-physics-informed-neural-networks-pinns/">Differential Equations V — Physics Informed Neural Networks (PINNs)</a> <a href="https://ssd-api.jpl.nasa.gov/doc/horizons.html">JPL Horizons API</a> <a href="https://github.com/w-winnie/MLsandbox/tree/main/astro_sandbox/notebooks/artemis">Code — MLSandbox — Winnie</a></p><p><em>Originally published at </em><a href="https://livnlearns.com/looking-at-artemis-i-pinns/"><em>https://livnlearns.com</em></a><em> on September 28, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d5932965634" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flood Prediction III — UNet Baseline]]></title>
            <link>https://medium.com/@winniew314/flood-prediction-iii-unet-baseline-73d9b28e8e5d?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/73d9b28e8e5d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[floods]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[unet]]></category>
            <category><![CDATA[disaster-preparedness]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 15:51:31 GMT</pubDate>
            <atom:updated>2026-07-07T18:21:59.495Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ve sourced the multi-modal satellite data, cleaned it and created a tensor pipeline, and dissected the mathematical morality of imbalanced loss functions. Now we dive into the core.</p><h3>Load Data &amp; Config</h3><p>We load in the data we prepared in the previous post and bind different channels, create tensors, split data in traing test and val, here is a sample of how the data looks for one file</p><p>The following image shows our two S1 bands (primary feature — radar image), the GFM labels (also displayed, but not used are the manually created S1 labels)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7h7qeuHT9lihNM-C.png" /></figure><p>Here is another example with more layers visualized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7L0fh42KlWPD3eFu.png" /></figure><p>For more details please refer to data acquisition and data preparation pipelines.</p><p>Here is the config and layers we used in this run:</p><pre>INCLUDE_ENV = {<br> &quot;env_elevation&quot;: False,<br> &quot;env_slope&quot;: True,<br> &quot;env_hand&quot;: True,<br> &quot;env_water_occ&quot;: True,<br>}<br>INCLUDE_S1_LAGS = {<br> &quot;s1_t-1&quot;: True, <br> &quot;s1_t-2&quot;: True<br>}<br>INCLUDE_WEATHER = {<br> &quot;precip_1d&quot;: True,<br> &quot;precip_3d&quot;: True,<br> &quot;precip_7d&quot;: True,<br> &quot;soil_moisture&quot;: True,<br> &quot;temperature_2m&quot;: True,<br> &quot;total_evaporation&quot;: False,<br>}<br>INCLUDE_ENV_LANDCOVER = False<br>INCLUDE_HYDRO = False<br>INCLUDE_GLOFAS = False</pre><h3>Create Patches</h3><p>Full satellite scenes are massively high-resolution, far too large to feed into a U-Net at once without triggering instant Out-Of-Memory (OOM) errors on the GPU. Training requires fixed spatial windows (in my case, PATCH_SIZE, 256×256 pixels)</p><p>However, because floods are rare anomalies (often making up &lt;1% of a scene), a naive sliding window approach would generate thousands of patches containing nothing but dry land. The network would overfit to the background class immediately. So, there is a little bit of thought involved in this patching process.</p><p>Our patching pipeline is two phased:</p><p><strong>The Manifest (Planning Phase):</strong> A spatial cartographer that calculates precise (y, x) bounding box coordinates to guarantee a specific statistical mix of flooded vs. dry land. so i can save it over the runs for consistency without bloating the space with both patches and scenes.</p><p><strong>The Dataset (Loading Phase):</strong> The PyTorch DataLoader that physically crops the tensors at those coordinates during the training loop. Hence patches essentially created when the loader is called.</p><p>Firstly we build a patch manifest — this is essentially the planning phase, we pick (y, x) top-left corners per scene so the batch mix matches PATCH_QUOTAS (mostly “some flood”, some dry).</p><h3>Config</h3><p>Here is our patch config for the run:</p><pre>PATCH_TARGET = &quot;gfm_mask&quot;<br>PATCH_SIZE = 256<br>PATCH_VALID_FRACTION = 0.90<br>PATCH_MAX_IOU = 0.50<br>TRAIN_PATCHES_PER_SCENE = 100<br>PATCH_QUOTAS = (<br> (0.7, 0.01, None), # 70% of patches MUST have at least 1% flood<br> (0.1, 0.015, None), # 10% of patches MUST have at least 1.5% flood<br> (0.2, 0.00, 0.00), # 20% of patches MUST be completely dry (0% flood)<br>)</pre><p>We define a Patch Quota, with a per-scene patch mix with share, min_flood_frac, max_flood_frac. The shares should sum to 1 and flood fraction = flooded valid pixels / all valid pixels in the patch, we use max_flood_frac=None for no upper ca[ and use max=0.0 with min=0.0 for dry-only patches</p><p>For each scene, we fill each quota bucket by anchoring on label pixels (np.argwhere), then placing a PATCH_SIZE window around a random anchor. Flood buckets anchor on flooded valid pixels; the dry bucket anchors on valid non-flood pixels (uniform random fallback if none).</p><ul><li>For every scene — we do lazy loading in RAM, load the labels and only work with valid data</li><li>We get the spatial bounds of the scene and flooded, not flooded pixel coordinates</li><li>Then we proceed to full fill our set patch quotas</li><li>We reject a patch if</li><li>too many invalid pixels (PATCH_VALID_FRACTION),</li><li>it overlaps an already chosen window (PATCH_MAX_IOU),</li><li>its flood fraction (flooded valid pixels ÷ all valid pixels) is outside the bucket’s min/max</li></ul><h3>Patch Search</h3><p>For every scene,</p><ul><li>We take in the scene (pool), h and w and size</li><li>Reject creating patches if no flood pixels in the scene</li><li>Uniformly samples exactly one pixel coordinate pair from the valid pool to act as the anchor `(cy, cx)`</li><li>Valid patch range for the anchor <br> — `bottom = cy — patch_size + 1` <br> — `top = min(cy, h — patch_size)`<br> — `left = cx — patch_size + 1` <br> — `right = min(cx, h — patch_size)`</li><li>We reject if image is smaller than desired patch</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nuo5FI4-FrO1Kdcm.png" /></figure><h3>Quota</h3><p>Then we check quota validity, by evaluating the actual pixel contents of the proposed window to ensure it isn’t filled with “nodata” padding and that it matches the flood percentage required by the current quota bucket :</p><ul><li>We take in the full array (labels, valid pixels) &amp; min-max flood threshold</li><li>If no valid pixels in the window less than minimum threshold i.e. `PATCH_VALID_FRACTION`, reject the patch</li><li>Create label patch window from top left anchors `(y,x)`</li><li>Isolate only the valid pixels using boolean indexing</li><li>Calculate the mean of valid labels in the patch</li><li>Rejects if mean or flood fraction of patch defines bounds of min-max flood threshold</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MiiYMPg8J4hUWbea.png" /></figure><h3>Overlap</h3><p>Then we proceed to evaluate spatial overlap for our patches and reject patches which overlap more than desired fraction, to guarantee that the model does not train on the same geographic features repeatedly, we do this by calculating the Intersection over Union (IoU) between a proposed bounding box and all previously selected boxes</p><ul><li>Take candidate coordinates `(y, x)` and a list of already selected chosen coordinates</li><li>Iterate over every top-left coordinate pair previously approved for the current scene `(py, px)`</li><li>Calculate the height and width intersection (ih, iw) — `Overlap = Bottom — Top = [min(y, py) + patch_size] — max(y, py)`</li><li>Calculate area of intersect — `ih * iw`</li><li>Apply the IOU threshold — Area_Intersection / Area_Union = intersection / ( patch1(h*w) + patch2(h*w) — intersection)</li></ul><p>Then we define another dataset function which is essentially the loading phase, we crop inputs/labels at those corners and return (spatial, weather, label, valid), then we just wrap the dataset in a PyTorch DataLoader.</p><p>Here is the patch summary of the data we used and the config we described:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zom64TEKjF8V9cUY.png" /></figure><h3>Features / Input</h3><p>Above is the patch summary and the split details, each patch has multiple spatial channels:</p><p>spatial input: (32, 9, 256, 256)</p><p>And because of the nature of the model we’re using ( <strong>Film UNet</strong> described in the next section) — we provide weather as linear modulation layer</p><p>weather vec: (32, 5)</p><p>Here is how a patch can look like:</p><pre>0 VV shape=(256, 256) min=0 max=1 mean=0.399873<br> 1 VH shape=(256, 256) min=0 max=1 mean=0.352138<br> 2 s1_t-1_VV shape=(256, 256) min=0 max=1 mean=0.460086<br> 3 s1_t-1_VH shape=(256, 256) min=0 max=1 mean=0.384164<br> 4 s1_t-2_VV shape=(256, 256) min=0 max=1 mean=0.437939<br> 5 s1_t-2_VH shape=(256, 256) min=0 max=1 mean=0.37158<br> 6 env_slope shape=(256, 256) min=0 max=0.236625 mean=0.0237011<br> 7 env_hand shape=(256, 256) min=0 max=1 mean=0.290433<br> 8 env_water_occ shape=(256, 256) min=0 max=0.97 mean=0.232344</pre><h3>Model</h3><p>If we were only looking at satellite imagery, a standard Convolutional Neural Network (CNN) like a U-Net would be perfectly sufficient. U-Nets are the gold standard for semantic segmentation because their encoder-decoder structure (with skip connections) captures both deep, abstract context and high-resolution spatial boundaries.</p><p>Convolutional Neural Networks (CNNs) like the U-Net love 2D grids (images). But weather data like 3 day precipitation accumulation or soil moisture is coarser and better as a 1D vector.</p><p>The naive approach is to simply tile the 1-dimensional weather data across the entire 256×256 image grid and concatenate it as an extra channel. However, this is computationally wasteful, heavily inflates the parameter count of the first convolution layer, and forces a spatial network to learn global context through a localized lens.</p><p>Instead of forcing the weather into the image, we use <strong>FiLM</strong>. Developed originally for visual reasoning tasks, FiLM allows a network to modulate its spatial feature maps based on external conditioning data.</p><p>I wrote about it before <a href="https://livnlearns.com/unet-linear-modulation/">here</a>, but I’ll discuss it briefly in this post as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*5DBgmKAd_7NWE8rV.png" /></figure><p>While the spatial input goes in through a standard input layer, in our case we have a batch of 32 and we chose to go ahead with only 9 channels right now, not 17, the bulk was landcover OHE channels which were not helping our model, so I decided to remove them for noe and inspect later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nz5ih1SPjiQscocf.png" /></figure><p>Then we have a standard double convolution block, you might be familiar with it already, 4 of these make the encoder part of our UNet</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/0*szEoGnGQN1JCs8E-.png" /></figure><p>Before the weather data ever touches the image, it passes through its own isolated network branch. The 6-channel weather vector is fed into a simple Multi-Layer Perceptron (MLP). This “tabular encoder” compresses the raw meteorological values into a dense embedding, capturing the non-linear relationship between things like 7-day cumulative rainfall and current soil moisture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*fx7Iu6n2EH-4CfUW.png" /></figure><p>Think of it like a volume knob. Based on how heavily it rained (the weather vector), a Multi-Layer Perceptron (MLP) generates two scalars: gamma (scale) and beta (shift). It then applies an affine transformation to the spatial feature maps coming out of the U-Net convolutions</p><p>And although a bit rough, this is how the data flows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aqCBTBodiorgWOQW.png" /></figure><p>Since floods are extremely rare &lt; 1% pixels, we experimented with different loss functions you can read about them in detail <a href="https://livnlearns.com/bce-focal-dice-and-tversky-flood-prediction-loss-deep-dive/">here</a>. We use the Focal Tversky Loss, you can even find the implementation in the mentioned post.</p><p>Training a model on extreme class imbalance with a rational loss function like Focal-Tversky is notoriously unstable. For memory, we run the data using PyTorch’s Automatic Mixed Precision (AMP), running the forward pass in 16-bit floats. The limited numerical range and small loss values for detecting &lt; 1% data’s boundary detection, we wrapped the loss in a grad scaler, mathematically inflating the loss before back prop.</p><p>For optimizer, we used adam with cosine annealing LR since we have Tversky coupled in the loss function, making it non convex, we need to have a cosine LR scheduler which allows the learning rate to start high (to escape bad initialization pockets) and smoothly decay to a specified minimum, so that we don’t get stuck in a plateau when predicting zeroes everywhere and can find a global minima</p><p>Instead of tracking loss, I evaluated the network at the end of every epoch by computing a continuous Precision-Recall (PR) Curve. The validation loop calculates the exact F1 score across a linear grid of 99 different probability thresholds. The Early Stopping callback is strictly tied to the optimal F1 score instead of val loss or val accuracy.</p><p>To begin with, we just started with 20 epochs and got:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*6f7CpF5KfsGjV3bK.png" /></figure><p>I say <em>begin with</em> above but believe me i already tried a bunch of patching strategies, loss functions, training loops with different hyperparameters, as well as model configs. What I mean is after reaching this level there is still scope to improve model but I didn’t want to do that before investigating and even getting some more data (geographically diverse), which is a process that’s on hold for now.</p><h3>Evaluation</h3><p>Evaluating semantic segmentation models on extreme anomalies requires an entirely different methodology than standard classification. If a scene is 99% dry land, a model that simply predicts “dry” everywhere will achieve 99% pixel accuracy. We did try our best to balance the data before training and for validation.</p><p>To properly grade this U-Net, I discarded pixel accuracy as a primary metric and built an evaluation suite centered around Intersection over Union (IoU), Precision-Recall (PR), and looked at dynamic thresholding.</p><p>Standard binary classification uses a default probability threshold of 0.5. In extreme imbalance scenarios, this is almost always suboptimal. Instead of guessing, I wrote a vectorized threshold search that scans every probability cutoff from 0.01 to 0.99 and calculates the exact F1 score across all pooled validation pixels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/0*XRn7-srkjQ6CvTah.png" /></figure><p>Based on that the best threshold is ~0.85. Because our Focal-Tversky loss was aggressively penalizing False Negatives, the model learned to be highly sensitive, generating a “bloated” probability map. Raising the decision threshold forced the model to be strictly confident before classifying a pixel as water, perfectly balancing precision and recall.</p><p>If we look closely at the data, the 0.85 threshold gives a mathematically higher F1 Score (0.58 vs 0.45) and IoU (0.40 vs 0.29) because it pushes Precision way up to 0.75.</p><p>But look at what happens to Recall: At 0.85, Recall drops to 0.47. At 0.50, Recall was 0.66</p><p>At 0.85, the model becomes so conservative that it misses 53% of the actual flood pixels (Recall = 0.47). In the real world, failing to predict a flood (False Negative) is catastrophic. Over-predicting a flood (False Positive) is simply an inconvenience.</p><p>In flood prediction (and disaster response in general), Recall is king. Missing a flood (False Negative) costs lives and resources; over-predicting a flood (False Positive) just means someone checks a map twice. The mathematical “optimal” threshold of 0.85 choked the model, making it too conservative.</p><p>So, for my scenario, I chose to leave the threshold at 0.5, applying this threshold to the unseen Test:</p><pre>--- Global Evaluation Report (GFM_MASK) @ threshold=0.500 ---<br> precision recall f1-score support</pre><pre> False 1.00 1.00 1.00 46551056<br> True 0.34 0.66 0.45 33890</pre><pre> accuracy 1.00 46584946<br> macro avg 0.67 0.83 0.73 46584946<br>weighted avg 1.00 1.00 1.00 46584946</pre><pre>Global IoU @ threshold: 0.2910<br>Global pixel accuracy @ threshold: 0.9988</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__03khcGhun9tjAxTVlgnQ.png" /></figure><p>The ROC curve carries the mathematical illusion caused by the 99% dry land in the dataset, the Precision-Recall Curve (middle). This is the bitter truth, the Average Precision (AP) is 0.6558.</p><p>Precision is calculated as TP / (TP + False Positives).</p><p>Notice what is missing from that equation? True Negatives. The PR curve completely strips away the safety net of the dry land pixels. It forces the model to be judged only on the pixels it flagged as water. It shows that as you try to capture more of the flood (increasing Recall), your model starts making mistakes (dropping Precision).</p><p>Above we calculates pixel wise stats, but how does the model do scene wise:</p><p>When we break the metrics down scene-by-scene, a glaring geographical disparity emerges. The model achieved a very respectable 0.64 F1 score on the Gatineau scene, but completely collapsed with a 0.17 F1 in Ottawa. Why? Ottawa is a dense urban center. Radar imagery struggles in cities due to “double-bounce” scattering (which makes flooded streets appear bright) and tall building shadows (which mimic the dark voids of deep water). The U-Net and the GFM heuristic labels themselves failed to reliably untangle urban geometry from actual floodwater</p><p>Back to our pixel level stats evaluated for patches, while a 99.88% pixel accuracy looks incredible on paper, the Global IoU of 0.29 reveals the model is struggling to perfectly map the micro-boundaries of the water. But metrics only tell you that a model struggled, visuals help you see and analyze better.</p><p>To truly evaluate the U-Net, we have to look at the physical predictions overlaid against the raw Sentinel-1 radar and the GFM ground truth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ibCY5UlBkD8VHjx-.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yParEq9ifl4njtpt.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yeg8O_ecT5vI-KLj.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*42mA7o7kaz8dIrp6.png" /></figure><p>When you look at these overlays, the reality of Earth Observation data becomes clear. The architecture works, but the ground truth is flawed.</p><ol><li>The U-Net Learned Physics: The model is successfully identifying dark, specular radar reflections (water) and tracking the organic morphology of the river systems. It is finding the physical flood.</li><li>The GFM Label Noise: The Copernicus Global Flood Monitoring (GFM) targets we trained and evaluated on are not hand-drawn masks; they are generated by heuristic algorithms. If you look closely at the prediction maps, the U-Net is actively predicting flood waters in valid, dark SAR region. Because our evaluation metric treats the GFM mask as absolute “truth,” the model gets heavily penalized (False Positives) for finding water that the heuristic labels ignored</li></ol><h3>Conclusion</h3><p>This isn’t the final product but this iteration was promising, so far,</p><ul><li>Built a multi-modal raster pipeline fusing Sentinel-1 SAR with 1D weather vectors and other spatial features</li><li>Implemented an efficient virtual tiling strategy (patches) to dynamically feed a PyTorch DataLoader</li><li>Engineered a FiLM-conditioned U-Net to modulate spatial convolutions based on historical meteorology</li><li>Leveraged a custom Focal-Tversky loss, mixed precision training, and threshold calibration to navigate extreme class imbalance</li></ul><p>And the architecture worked. In the context of Earth Observation, extracting an F1 score of up to 0.64 (as seen in the Gatineau scene) on a dataset where the positive class makes up less than 1% of the pixels is a massive signal. It proves that the network successfully bypassed the 99% background noise and learned the actual physical signature of floodwaters.</p><p>But the final system boundary we hit wasn’t a failure of code or architecture it was the data taxonomy itself. You cannot train a neural network to achieve perfect physical accuracy using heuristically flawed labels. By forcing the network to optimize against automated GFM data, it wasn’t learning the pure fluid dynamics of a flood; it was just learning to approximate another algorithm’s blind spots.</p><p>The U-Net was actively finding floodwaters in valid radar regions, but was mathematically penalized because the automated GFM algorithm had missed them.</p><p>Furthermore, the massive disparity between our Gatineau and Ottawa results proved that standard radar approaches inherently struggle with the geometry of dense urban environments.</p><p>Clever architectures cannot magically fix fundamental data problems. To map disasters with the precision required for emergency response, the next iteration of this project requires a pivot:</p><ol><li>Abandoning Heuristic Labels: We need to transition away from automated masks and rely exclusively on hand-annotated, human-validated crisis data (like Sen1Floods11 or Copernicus EMS Rapid Mapping).</li><li>Exploring Physics-Informed Neural Networks (PINNs): Instead of relying purely on visual pixel segmentation, future iterations should explore embedding the 2D Shallow Water Equations directly into the loss function to force the model to respect the physical laws of fluid dynamics.</li></ol><p>Sometimes, the most valuable outcome of a highly engineered ML experiment isn’t a perfect 99% accuracy score (which we already discussed can be quite misleading) it’s proving exactly where, and why, the current data paradigm breaks and where it succeeds.</p><p>Though in my humble opinion and the number of iterations it took to get to this iterations, I would say for this data, this score, prediction and accuracy is not bad at all, remember our flood fraction for a scene is &lt;&lt; 1% the combination of patching strategies, linear modulation of weather, the water body masks provided as spatial input along with the radar data, the loss function, training loop — all helped significantly.</p><p><em>Originally published at </em><a href="https://livnlearns.com/flood-prediction-iii-unet-baseline/"><em>https://livnlearns.com</em></a><em> on July 7, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73d9b28e8e5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flood Prediction II — The Data Prep]]></title>
            <link>https://medium.com/@winniew314/flood-prediction-ii-the-data-prep-1f04e880b678?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f04e880b678</guid>
            <category><![CDATA[disaster-preparedness]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 19:59:08 GMT</pubDate>
            <atom:updated>2026-06-30T20:39:20.275Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s been a minute since my last post on <a href="https://livnlearns.com/flood-prediction-i-nose-diving-into-data-sources/">data acquisition</a>. Between a much needed vacation — trip to Italy and wrapping up my tap dance recital (because life exists outside of PyTorch!), sitting down to write about this project took a backseat, as this whole process is quite time consuming, believe it or not, and the charm kinda fades away after you’ve done the project, oh no my model is not near as accurate as I’d like but it met my first iteration standards and then kinda got pushed in the background as you probably saw in my <a href="https://livnlearns.com/?s=differential">differential equations series</a>. Now that I’m finally getting around to documenting the model architecture, I realized I completely skipped the prequel — the data prep, which I actually built way back at the start of the project, so, well a while ago.<br>Before we get into U-Nets and loss functions, let’s quickly summarize how we wrangle the raw satellite data</p><h3>Setup</h3><p>I have described the sources available and provided their overview in the first post, in this post we assume we have the data downloaded.</p><h3>Loading</h3><p>We open our GeoTIFFs and <strong>crop </strong>it to the <strong>GAUL administrative boundary</strong>.</p><ul><li>Reproject the boundary to the raster CRS if needed</li><li>Keep only polygons overlapping the raster bounds</li><li>Apply a tiny geometry buffer (0.0001°) to avoid GDAL edge precision bugs</li><li>Mask with all_touched=True so edge pixels are included</li><li>Convert nodata to NaN and cast to float32</li></ul><p>For S1, crop_to_aoi=True restricts the scene to the region of interest. The returned <strong>profile</strong> (transform, CRS, height, width) becomes the master grid.</p><h3>Image Alignment</h3><p>We force all disparate geospatial datasets to share the exact same pixel grid, dimensions, and Coordinate Reference System (CRS) as our master S1 imagery.</p><p>Every other raster is warped onto the S1 grid via rasterio.warp.reproject</p><p>We use a Resampling Algorithm based on the data, <strong>Nearest neighbor</strong> for env/gfm to protect <strong>categorical </strong>landcover classes which grabs the exact value of the single closest pixel without averaging and <strong>Bilinear Interpolation</strong> which calculates a weighted average of the 4 nearest pixels and creates smooth transitions for <strong>continuous </strong>data like rainfall or radar backscatter.</p><p>Additionally values below -9000 are treated as nodata and set to NaN</p><h3>Global Stats</h3><p>Before normalization, we scan a <strong>random subsample </strong>of ENV, WEATHER, and HYDRO GeoTIFFs (default 10 files, up to 100k pixels per band per file) and computes per-band 1st percentile, 99th percentile, and mean.</p><p>These stats help inspect the data and drive percentile-based scaling for elevation and hydro. They are computed once per region and reused for all dates.</p><p>Results are saved to stats_{REGION}.json (or stats.json for single-date tests).</p><ul><li>ENV band 4 (land cover) is skipped</li><li>ENV elevation ignores values below -100 m (sensor noise)</li></ul><h3>Sanitization &amp; Normalization</h3><p>We handle each data source separately, because they’re obtained from different sources, measure different values and operate in different scales</p><p>For <strong>S1</strong>, we firstly remove the sensor noise, pixels below <strong>-50 dB</strong> (SAR noise floor), then filled as <strong>0</strong> after scaling. We use strict physical bounds instead of dataset stats because backscatter has well-known ranges:</p><p>Water acts as a specular reflector, bouncing radar pulses away from the sensor, resulting in extremely low backscatter (dark pixels, near -25 dB). Urban areas and rough vegetation cause double-bounce or volume scattering, resulting in high backscatter (bright pixels, near 0 dB)</p><p>For <strong>Weather</strong>, we have a few selected bands, which we scale as follows:</p><p>For GloFAS, even though we don’t use it, discharge clipped to [0, 1] by dividing by 2500 m³/s (empirical flood spike cap).</p><p>For <strong>GFM </strong>which is the <strong>primary supervised label</strong>, we binarize the data into flood and no flood:</p><p>Cleaning up the <strong>Environment </strong>is multi step process, for the <strong>Continuous </strong>bands:</p><p><strong>Land cover</strong> (band 5) is <strong>one-hot encoded</strong> into 11 channels (World Cover classes), essentially giving each World cover class its own binary channel.</p><p>For Hydro, its a single band scaled with global 1st/99th percentiles from scene stats. Again we don’t use this</p><h3>Temporal Alignment</h3><p>Environment is a static file but data like S1, GFM and weather is available on different timestamps, the details of which you can find in the data acquisition post.</p><p>We have already downloaded daily data, but we need to align these different modalities as they might not be for the same date</p><p>For GFM, we find all GFM tiles matching the date, merges them with rasterio.merge, writes a temporary GeoTIFF, aligns to master grid, then deletes the temp file.</p><p>For weather, we load the pre-aggregated GEE weather stack for the target date.</p><p>Then we match dates and align the data</p><h3>Tiling</h3><p>Instead of physically shattering the rasters into thousands of tiny files which causes massive disk I/O bottlenecks I implemented a virtual tiling strategy. The pipeline exports the full aligned scene as a single .npz file and generates a mathematical grid of 256×256 coordinates. Later, during the PyTorch DataLoader phase, the dataset dynamically crops these coordinates in memory and applies the 1% target rule, discarding unnecessary dry patches before they ever hit the GPU</p><p>We don’t tile/patch our data here exactly, but we create an outline for it, I wrote a helper function, build_tile_index, that acts as a mathematical cartographer. It iterates over the master height and width, stepping by a defined stride (256 pixels), and calculates the exact (x, y, w, h) bounding boxes for every possible non-overlapping patch in the scene.</p><h3>S1 Labels — Optional</h3><p>I optionally build a secondary label on S1 using otsu threshold and lee median filtering, it didn’t look that great so, i haven’t used it, and trained the model with GFM</p><p>It’s basically an auxiliary water mask built from raw S1 dB values (again not used as the primary training target — or anything right now — not unless I improve it):</p><ol><li>Mask valid pixels (both bands &gt; -50 dB)</li><li><strong>Lee filter</strong> (7×7) to reduce speckle while preserving edges</li><li><strong>Otsu threshold</strong> on high-variance edge pixels (top 10% local variance) for VV and VH, with fallbacks at -17.0 and -23.0 dB</li><li>Water if VV ≤ threshold <strong>or</strong> both bands are borderline below threshold</li><li><strong>3×3 median filter</strong> for spatial cleanup</li></ol><p>Also produces s1_label_valid (where SAR data exists) and metadata: Otsu thresholds, positive pixel count, pos_weight for class imbalance.</p><p>Known limitations — Otsu fails on unimodal scenes, misses wind-roughened water, misclassifies radar shadows and smooth tarmac, and ignores double-bounce flooding in urban/forested areas.</p><pre>--- S1 Label vs GFM Benchmark IoU ---<br>Valid Evaluation Pixels : 41132045<br>Intersection : 429243<br>Union : 24371725<br>IoU Score : 0.0176</pre><h3>Putting it Together</h3><h3>Pipeline</h3><p>Then we sequence all our steps and create a pipeline, the final pipeline looks somewhat like:</p><p>For each target date, the pipeline:</p><ol><li>Loads and normalizes <strong>S1</strong>; establishes master grid</li><li>Builds <strong>S1 labels</strong></li><li>Loads <strong>static</strong> ENV + HYDRO (cached after first date in a region)</li><li>Loads and normalizes <strong>weather</strong> and <strong>GloFAS</strong></li><li>Builds <strong>GFM mosaic</strong> and normalizes to binary mask</li><li>Loads <strong>S1 temporal lags</strong> (s1_t0 = current, s1_t-1, s1_t-2 from prior acquisitions in the sorted date inventory; zeros if unavailable)</li><li><strong>Spatial masking:</strong> multiplies ENV, hydro, weather, glofas, and GFM tensors by s1_label_valid so invalid SAR pixels are zeroed consistently</li><li>Returns (tensors, tensors_selected, master_profile)</li></ol><h3>Export</h3><p>I also have an export function, which writes data in a specific format:</p><p>Write two files per sample:</p><p><strong>selected_{REGION}_{YYYY-MM-DD}.npz</strong></p><ul><li>Float32 for continuous tensors</li><li>Uint8 for masks, labels, and validity channels</li><li>ENV continuous bands stored as separate keys (env_elevation, ...) not a merged stack</li></ul><p><strong>selected_{REGION}_{YYYY-MM-DD}.json</strong></p><p>Metadata includes:</p><ul><li>shape_by_key - channel/spatial dimensions per array</li><li>crs, transform - georeferencing from master S1 profile</li><li>tile_grid - 256×256 non-overlapping tile index for downstream patching</li><li>pos_weight, positive_ratio, gfm_missing</li><li>include_env_continuous - optional band filtering at export time</li></ul><p>After running the notebook, the <strong>prepared dataset</strong> lives under EXPORT_DIR</p><p>Each pair is one <strong>georeferenced scene</strong> at S1 resolution (~6399 × 12333 pixels in the Ottawa example), with:</p><ul><li>All modalities aligned and normalized</li><li>GFM binary flood mask as the supervised target</li><li>Validity masks for SAR and GFM coverage</li><li>Three timesteps of normalized S1 (current + two lags)</li><li>JSON metadata for tiling, class weights, and provenance</li></ul><h3>Batch</h3><p>I run this function in batch using a ThreadPoolExecutor, I have a small sample to this basic multithreading works for me at this stage.</p><p>The production run loop:</p><ol><li>For each region in REGIONS</li><li>Compute and save stats_{REGION}.json</li><li>Inventory all S1 dates; optionally filter by cutoff date</li><li>Build S1 lag path maps (build_s1_lag_maps)</li><li>Process the <strong>first date sequentially</strong> to populate the static ENV/HYDRO cache</li><li>Process <strong>remaining dates in parallel</strong> (ThreadPoolExecutor, default 4 workers) with the shared static cache</li></ol><p>Each worker calls process_date_pipeline → export_prepared_sample, a log looks somewhat like:</p><pre>==================================================<br>Executing Pipeline for Target Date: 2019-05-07<br>==================================================<br> [OK] S1 loaded and normalized. Shape: (2, 6399, 12333)<br> [OK] S1 Labels generated (Otsu VV: -10.04)<br> [OK] Static ENV &amp; HYDRO context loaded (from cache)<br>Processing weather for 2019-05-07 : D:\Projects\datasets\raw\Ottawa\WEATHER\WEATHER_Ottawa_weather_1km_2019-05-07.tif<br> [OK] Weather loaded and normalized. Shape: (6, 6399, 12333)<br> [WARN] GloFAS missing for 2019-05-07; returning empty tensor.<br> [SKIP] GloFAS dummy initialized.<br>Building GFM mosaic for 2019-05-07 : 4 tiles<br>[EXPORT] Saved selected_Ottawa_2019-04-13.npz and .json</pre><h3>Analysis &amp; Checks</h3><p><strong>valid mask</strong>: s1_label_valid &amp; gfm_valid, coverage=34.46% of grid (5452658 px)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/0*Ev-ms92sVS8WReVa.png" /></figure><p>Look at the distribution of the GFM supervised target mask. Even after applying spatial subsetting, the massive spike at 0.0 (dry land) completely dwarfs the microscopic blip at 1.0 (flood water). This single plot is a stark visual warning of what&#39;s to come.</p><p><strong>s1</strong>: tensor (2, 2522, 6274)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*ahuYPJJSinPybDdA.png" /></figure><p>You can clearly see the data has been cleanly bound between 0.0 and 1.0. By physically clipping the VV backscatter to a [-25, 0] dB range prior to scaling, we preserved the natural bimodal spread of the radar returns (separating the dark, specular water reflections from the bright, double-bounce urban reflections) while entirely eliminating the sensor noise floor.</p><p><strong>env_elevation</strong>: tensor (1, 2522, 6274)</p><p><strong>env_water_occ</strong>: tensor (1, 2522, 6274)</p><p><strong>env_slope</strong>: tensor (1, 2522, 6274)</p><p><strong>env_hand</strong>: tensor (1, 2522, 6274)</p><p><strong>env_landcover</strong>: tensor (11, 2522, 6274)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rP2FHJtEq5YK_Mqe.png" /></figure><p><strong>weather</strong>: tensor (6, 2522, 6274)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gGFlSddnWvLvcH2K.png" /></figure><p>The continuous environmental and weather bands (Elevation, Slope, Precipitation, Soil Moisture) are all operating on completely different physical scales (degrees, millimeters, Kelvin). Our percentile-based and static normalization pipelines successfully squeezed these disparate metrics into a uniform [0, 1] tensor space without destroying the long-tail anomalies like the extreme precipitation spikes you can see in the 1-day and 3-day weather lags.</p><p>The purpose of above plots is to check shape of tensors and normalization, the final list of features will be discussed briefly in my next post</p><p>Below is the statistical summary of our data (1 .npz file):</p><pre>==================================================<br>TENSOR STATISTICAL SUMMARY<br>==================================================<br>s1_raw | Shape: (2, 6399, 12333) | Valid: 52.1% | Min: -65.26 | Max: 29.05 | Mean: -16.35<br>s1 | Shape: (2, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.23<br>s1_label | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.31<br>s1_label_valid | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.52<br>env_raw | Shape: (5, 6399, 12333) | Valid: 42.1% | Min: 0.00 | Max: 173.00 | Mean: 33.70<br>env_elevation | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.23<br>env_slope | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 0.64 | Mean: 0.02<br>env_hand | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.24<br>env_water_occ | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 0.99 | Mean: 0.02<br>env_landcover | Shape: (11, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.05<br>hydro_raw | Shape: (1, 6399, 12333) | Valid: 52.1% | Min: 0.00 | Max: 193e+9| Mean: 779e+8<br>hydro | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.21<br>weather_raw | Shape: (6, 6399, 12333) | Valid: 52.0% | Min: -0.00 | Max: 274.39 | Mean: 77.02<br>weather | Shape: (6, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 0.48 | Mean: 0.15<br>glofas | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 0.00 | Mean: 0.00<br>gfm_raw | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 255.00 | Mean: 46.40<br>gfm_mask | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.01<br>gfm_valid | Shape: (1, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.52<br>s1_t0 | Shape: (2, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 1.00 | Mean: 0.23<br>s1_t-2 | Shape: (2, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 0.00 | Mean: 0.00<br>s1_t-1 | Shape: (2, 6399, 12333) | Valid: 100.0% | Min: 0.00 | Max: 0.00 | Mean: 0.00</pre><p><em>Originally published at </em><a href="https://livnlearns.com/flood-prediction-ii-the-data-prep/"><em>https://livnlearns.com</em></a><em> on June 30, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f04e880b678" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Looking at Artemis — Simulating the Artemis I Trajectory from Scratch]]></title>
            <link>https://medium.com/@winniew314/looking-at-artemis-e28e6d000d49?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/e28e6d000d49</guid>
            <category><![CDATA[differential-equations]]></category>
            <category><![CDATA[astronomy]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 16:56:15 GMT</pubDate>
            <atom:updated>2026-06-30T20:06:47.090Z</atom:updated>
            <content:encoded><![CDATA[<p>Did you know NASA JPL has a service called Horizons which lets you look at mission statistics, here we try to predict the orbit and compare with Artemis I stats.</p><p><a href="https://github.com/w-winnie/MLsandbox/blob/main/astro_sandbox/notebooks/artemis/artemis_orbital_sandbox1.ipynb">https://github.com/w-winnie/MLsandbox/blob/main/astro_sandbox/notebooks/artemis/artemis_orbital_sandbox1.ipynb</a></p><p>Orbital Mechanics is.. complicated, atleast to model with one equation, here we break the trajectory into multiple parts, first being the launch i.e. the Trans Lunar Injection (TLI) when the spacecraft is actually launched from the base with the rocket and the boosters, in this stage, there is a thrust generated by the engines, there is fuel depletion, different stages are let go gradually, the main forces are earth’s gravitation and atmospheric resistance and drag</p><p>Then there is 4 day trajectory to the moon, which is mostly coasting with little controlled thrust applied to control i.e. Thrust Control maneuvers (TCM)</p><p>In this project we focus on the Coasting part that is simulating the orbit under well a relatively stable and constrained environment.</p><h3>Setup</h3><p>We start with simple equations and simulate the trajectory, then compare to the actual data recorded and provided by <a href="https://ssd.jpl.nasa.gov/horizons/">JPL Horizons</a></p><p>We take some of our mission specific constants, like mass, area, engine thrust parameters from <a href="https://www.nasa.gov/humans-in-space/space-launch-system/reference-guide/">official website</a>, and some of them from the engine manufacturer specification, as we said, to keep it simple we only consider the coasting phase right now and the engine used in that phase was <a href="https://www.l3harris.com/sites/default/files/2024-07/l3harris-ar-rl10-spec-sheet.pdf">ICPS — Interim Cryogenic Propulsion Stage</a> which was developed by L3Harris, for planetary constants we use SPICE kernels, how they work is, we can put in the ID of specific planetary bodies and get their constant — this includes that of Earth (399), Moon (301), Sun and even of Artemis (-1023)! We get mission data (actual observed trajectory of Artemis — our ground truth) from JPL Horizons.<br>*JPL Horizons is in UTC but Spice is in Ephemeris Time<br>*Artemis I data in JPL is only during the TLI burn window</p><p>Here is how we get our observed Artemis Trajectory:</p><pre>Retrieved 577 data points<br>Time span: A.D. 2022-Nov-16 08:47:00.0000 to A.D. 2022-Nov-20 08:47:00.0000</pre><pre># Fetch Artemis I data during TLI burn <br>horizons_data_raw = fetch_horizons_vectors(<br>    target_id=&#39;-1023&#39;,<br>    center=&#39;@399&#39;,  # Earth-centered<br>    start_time=&#39;2022-11-16T08:47:00&#39;, # Post separation<br>    stop_time=&#39;2022-11-20T08:47:00&#39;,  # 4 days of cislunar coast<br>    step=&#39;10m&#39; <br>) # =&gt; retuens time series of state vectors (position and velocities)<br>horizons_data = add_moon_distance_to_data(horizons_data_raw)<br>print(f&quot;Retrieved {len(horizons_data[&#39;times_utc&#39;])} data points&quot;)<br>print(f&quot;Time span: {horizons_data[&#39;times_utc&#39;][0]} to {horizons_data[&#39;times_utc&#39;][-1]}&quot;)</pre><p>Now that we have, the observed data points , the first data point of the data we pulled being the closest index to flight ignition:</p><pre>ECO index: 0 (t = A.D. 2022-Nov-16 08:47:00.0000)<br>Flight start index: 0 (t = A.D. 2022-Nov-16 08:47:00.0000)<br>Flight end index: 576 (t = A.D. 2022-Nov-20 08:47:00.0000)</pre><p>we can check the velocity and position vectors it provides</p><pre>Initial State (Earth-centered J2000):<br> Position: [-3171.296, 9061.162, 5591.822] km<br> Velocity: [-7.763256, 2.222213, 2.122515] km/s<br> |r| = 11109.917 km (altitude: 4731.8 km)<br> |v| = 8.349338 km/s (8349.3 m/s)<br>Total velocity change due to gravity during 4-day window: -8083.8 m/s</pre><p>That is well above our Lower Earth Orbit (LEO) — and quite fast — between the velocity expected at LEO and the escape velocity (7.8–11.2km/s)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yYEBYKlJVq-jtF_k.png" /></figure><p>We see in our Plot 1 (top-left) — altitude steadily increasing as it moves from earth -&gt; moon, the curve looks steep before but then flattens as it slows down</p><p>And in Plot 2 (top right) — after TLI burn, Orion is at ~8.3km/s, out of earth’s gravity, sun is probably pulling it back and its slowing down (the bump on 82hrs is another burn performed for trajectory correction)</p><p>Plot 3 (bottom left) — shows J2000 reference frame — top down view and Plot 4 (bottom right) shows cumulative velocity.</p><p>Now that we have the true observed data for the TLI window and mission constants, we try our basic physics on Artemis I.</p><h3>Two Body Integrator — Earth and Object</h3><p>We start with a 2 body integrator — i.e. if gravity was the only force acting between the two bodies (our point mass of Orion and the Earth)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bmgE1GcqOO-4r5rU.png" /></figure><p>r: Position vector [km] (3,) <br>mu: Gravitational parameter [km³/s²]</p><p>For now we’ll use traditional methods to solve this like the solve_ivp from scipy, we use Dormand Prince, the details you can see <a href="https://livnlearns.com/differential-equations-iv-numeric-solutions/">here</a></p><p>The equation we’re solving is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*J55qzI2Fx0YmHL6g.png" /></figure><p>How it works with the data we fetched is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/0*H5PaVALHgjDFkbq1.png" /></figure><p>This is essentially the equation and data we’re providing to our solver so it can predict the next position and velocity (helping us plot out the trajectory)</p><p>And here is the implementation of solve_ivp method:</p><p>Remember our state vector s = [r_x, r_y, r_z, v_x, v_y, v_z]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mjlS3b6-mdx1iTVt.png" /></figure><p>Start at position s0, assume only Earth’s gravity exists, integrate forward in time for dt_test seconds</p><pre>s0: Test Coordinates (first position &amp; velocity vector):<br>[-3.17129585e+03 9.06116222e+03 5.59182213e+03 -7.76325581e+00 2.22221349e+00 2.12251486e+00]<br>dt_test: delta t between first 2 data points: 600.0000001192093</pre><pre>def accel_two_body(r, mu=MU_EARTH):<br>    r_mag = np.linalg.norm(r)<br>    a = -mu * r / (r_mag ** 3) #[km/s^2] (3,)<br>    return a</pre><pre>Two-body baseline propagation (1-minute Post-TLI coast):<br> Position error: 448.85 m<br> Velocity error: 1.1021 m/s</pre><pre>def propagate_two_body(state0, t_span, t_eval=None, rtol=1e-10, atol=1e-12):<br>    def equations_of_motion(t, state):<br>        r = state[0:3]<br>        v = state[3:6]<br>        a = accel_two_body(r)<br>        return np.concatenate([v, a])<br>    <br>    sol = solve_ivp( # Solve Initial Value Problem<br>        equations_of_motion, <br>        t_span, <br>        state0, <br>        method=&#39;DOP853&#39;, #Runge-Kutta (Dormand-Prince 4th and 5th order)<br>        t_eval=t_eval,<br>        rtol=rtol, <br>        atol=atol,<br>        dense_output=True #spline<br>    )<br>    return sol</pre><p>so in about 60 seconds (1 minute) we missed our target by almost half a km! if we use our Model 0 equation that just accounts for earth’s gravity treating the problem like a 2 body problem.</p><p>When we run this over 4 days (i.e.e our entire set of data points) for the whole coast duration, we end up being off by ~9747.82 km</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0DfK5by6gbtU7J7V.png" /></figure><p>Where the plot on left shows the top down 2D view of different trajectories, showing divergence of our simplistic 2 body trajectory calculated (red dashed line — made of s_pred) with the one executed(green — horizon’s data)</p><p>and plot on the right, shows positional error over time, at 96hrs (4 days) it is ~10,000 km off</p><h3>J2 Oblateness — Earth</h3><p>The first and most significant gravitational perturbation comes from the shape of the Earth itself, because the Earth spins on its axis, centrifugal force flattens the poles and flings material outward at the equator, creating an equatorial bulge.</p><p>This uses Laplacian in spherical coordinates which you can find <a href="https://livnlearns.com/vector-calc-review-laplace-in-spherical-coordinates/">here</a> and spherical harmonics, more about that later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4D9OLHGWmSV-7Re3.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6vUTerO6G0tKKpZK.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LU4RQM2B8Q1DAe2e.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k2F1i5s9yHYuyDDU.png" /></figure><p>Hence our equation accounting for modified gravitation potential becomes:</p><p>In python it’d look like:</p><p>We use the same solver DOP853 and test for the initial point again and get:</p><pre>Acceleration comparison at initial position:<br> Two-body: 3.229358 m/s² (3229.358 mm/s²)<br> J2: 1.559640 mm/s²<br> J2/2body ratio: 0.0483%</pre><pre>def accel_j2(r, mu=MU_EARTH, j2=J2_EARTH, r_eq=R_EARTH):<br>    x, y, z = r <br>    # z=0, spacecraft directly above equator, if z is really large, its probably over the poles<br>    r_mag = np.linalg.norm(r)<br>    r2 = r_mag ** 2<br>    r5 = r_mag ** 5<br>    <br>    z2_r2 = (z ** 2) / r2<br>    <br>    # coeff = -1.5 * j2 * mu * (r_eq ** 2) / r5<br>    coeff = 1.5 * j2 * mu * (r_eq ** 2) / r5<br>    <br>    a_x = coeff * x * (5 * z2_r2 - 1)<br>    a_y = coeff * y * (5 * z2_r2 - 1)<br>    a_z = coeff * z * (5 * z2_r2 - 3)<br>    <br>    return np.array([a_x, a_y, a_z])<br><br>def accel_two_body_j2(r):<br>    return accel_two_body(r) + accel_j2(r)</pre><p>The standard, point mass gravity of the Earth is pulling backward on the spacecraft at 3.23m/s². This is the primary engine of orbital mechanics, keeping the capsule trapped in a curved trajectory rather than flying off into a straight line into deep space.</p><p>The extra gravitational pull caused by the Earth’s fat equatorial bulge ( J2) is applying an extra acceleration of 1.56 mm/s^2 roughly 0.05% of primary force (discarding the deviation or Earth&#39;s shape from a perfect sphere) i.e. if we keep everything else identical but ignore the planet&#39;s shape, the model will drift by exactly 74.38 km after one day</p><p>Now when we run for the whole trajectory (i.e. the data points observed), and compare Euclidean vector distance between our previous model and this new model that accounts for earth’s oblateness and plot it over 24 hours (left) and we see an error of 74.8 km</p><p>While our position divergence increases with time, our plot of the J2 force magnitude plummets off a cliff, dropping to near zero within the first 5 hours. Because J2 decays at a brutal spatial power law of 1/r⁴, the shape of the Earth matters immensely while the spacecraft is close to home, but it quickly becomes a mathematical ghost as Artemis escapes into deep space on its way to the Moon</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-_0vfckTO4AL9R_2.png" /></figure><h3>Third Body Perturbations — Sun and Moon</h3><p>As the spacecraft leaves earth’s orbit, we’re forced to account for the gravitational effects of other bodies i.e. the moon and the sun, but because our entire tracking coordinate system is anchored directly to the center of the Earth, we cannot just add a standard Newtonian gravity vector, so, we use relative third body acceleration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qhBkVFSN7nArtprD.png" /></figure><p>Now putting this in code:</p><p>Running the test for first point:</p><pre>Third-body accelerations at TLI ignition:<br> Moon: 1.506 nm/s² (direction: [-0.943, 0.282, 0.175])<br> Sun: 0.653 nm/s²<br> Moon/J2 ratio: 0.0966%</pre><pre>Moon distance from Earth: 402232 km</pre><pre>def accel_third_body(r_sc, r_body, mu_body):<br>    # Direct Term: Acceleration of Spacecraft toward Third Body<br>    r_body_to_sc = r_sc - r_body<br>    a_direct = -mu_body * (r_body_to_sc / np.linalg.norm(r_body_to_sc)**3)<br>    # Indirect Term: Acceleration of Earth toward Third Body<br>    a_indirect = mu_body * (r_body / np.linalg.norm(r_body)**3)<br>    # Relative Acceleration: Spacecraft minus Earth<br>    return a_direct - a_indirect<br>    <br>def accel_moon(r_sc, et):<br>    r_moon = get_body_position_spice(301, 399, et)  # Moon wrt Earth<br>    return accel_third_body(r_sc, r_moon, MU_MOON)<br><br>def accel_sun(r_sc, et):<br>    r_sun = get_body_position_spice(10, 399, et)  # Sun wrt Earth<br>    return accel_third_body(r_sc, r_sun, MU_SUN)</pre><p>At the moment the rocket engines ignite, the gravitational pull of the Moon acts as a literal whisper, registering at just 1.506 nm/s² and the Sun is even weaker, pulling at 0.653 nm/s², and at the start of the lunar voyage, the subtle asymmetry of Earth’s equatorial bulge (J2) is over 1000 times more powerful than the direct gravitational pull of the Moon</p><p>Now running for four days:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HlS3TuJvCcEPtzK_.png" /></figure><p>The left plot tracks the absolute strength of the Moon’s gravity acting on Orion as the hours tick by. For the first 40 hours of the trip, the line remains low and flat. Earth’s heavy mass is the dominant factor, but as the capsule puts more distance between itself and Earth, the curve suddenly reaches an inflection point and skyrockets, ramping all the way up to 14 mm/s² by hour 96. Lunar Sphere of Influence (SOI). The deep space whisper we detected at ignition has officially scaled into the dominant tractor beam that will determine whether the capsule circles the Moon or gets flung hopelessly out into deep solar space.</p><p>The right plot illustrates the compounding tracking penalty if you choose to pretend the Moon and Sun do not exist. For the first two days, the error barely registers. But as the capsule climbs higher into the cislunar landscape, the omission of third-body forces catches up with the model with a vengeance. By the end of the 4-day window, our Earth only prediction drifts off course by a massive 2,527.79 kilometers.</p><p>Radius of the Moon is roughly 1,737 km, Artemis I was targeting a highly precise flyby altitude of just 130 km above the lunar surface, if a mission planner used a simple Earth-only Two-Body model and ignored the Moon and Sun, their math would put them 2,500 km off target.</p><h3>Solar Radiation Pressure</h3><p>Photons streaming from the Sun carry momentum. When they smash into the reflective surfaces and solar panels of the Orion capsule, they transfer that momentum, acting like a continuous, ultra-faint cosmic wind pushing the spacecraft directly away from the Sun.</p><p>However, before we can calculate this photon push, we have to solve another problem: The Eclipse. If the Earth blocks the line of sight between the spacecraft and the Sun, the spacecraft plunges into darkness, and the solar pressure drops instantly to zero.</p><p>To track this, we write a vector based cylindrical shadow function</p><p>If our capsule passes the shadow test, we can calculate the Solar Radiation Pressure acceleration vector using the radiation pressure coefficient. At a standard distance of 1 Astronomical Unit (1 AU) from the Sun, the solar constant delivers a radiation pressure of roughly 4.56e-6 N/m². Our script scales this base pressure inversely with the square of our dynamic distance from the Sun</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zvJC0ZMIUBsUz4Hz.png" /></figure><p>Implementing it:</p><p>Using the same solver:</p><p>Running it on the first point to test:</p><pre>SRP acceleration at TLI: 6.596 pm/s²<br>SRP/Moon ratio: 0.4380%<br>Shadow factor: 1.0</pre><pre>def accel_srp(r_sc, mass, et, cr=CR_SRP, area=ORION_AREA):<br>    # Sun position wrt Earth<br>    r_sun_earth = get_body_position_spice(10, 399, et)<br>    <br>    # Spacecraft to Sun vector<br>    r_sc_to_sun = r_sun_earth - r_sc<br>    r_sc_sun_mag = np.linalg.norm(r_sc_to_sun)<br>    sun_hat = r_sc_to_sun / r_sc_sun_mag<br>    <br>    # Check shadow<br>    shadow = is_in_shadow(r_sc, r_sun_earth)<br>    if shadow == 0:<br>        return np.zeros(3)<br>    <br>    # SRP pressure at 1 AU: P = Phi/c<br>    # Phi = 1367 W/m^2, c = 299792458 m/s<br>    # P_1AU = 1367 / 299792458 = 4.56e-6 N/m^2<br>    P_1AU = SOLAR_FLUX / (C_LIGHT * 1000)  # N/m^2 (c in m/s)<br>    <br>    # Scale by distance<br>    r_sc_sun_au = r_sc_sun_mag / AU_KM<br>    P = P_1AU / (r_sc_sun_au ** 2)<br>    <br>    # Acceleration: a = Cr * P * A / m<br>    # Units: a [m/s^2] = Cr * P [N/m^2] * A [m^2] / m [kg]<br>    a_mag = cr * P * area / mass  # m/s^2<br>    a_mag_km = a_mag / 1000.0  # km/s^2<br>    <br>    # Direction: away from Sun<br>    # SRP acceleration [km/s^2]<br>    return -a_mag_km * sun_hat * shadow</pre><p>At launch, the photon pressure acting on Orion registers at a mind bogglingly tiny 6.596 picometers per second squared. A picometer is one trillionth of a meter. This force is so incredibly weak that it makes the Moon’s gravity look like a powerhouse by comparison, accounting for a mere 0.4380% of the lunar pull.</p><p>When we run it for the whole trajectory (we actually run twice, with SRP and without to compare the effect):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qUl0_i_wCZfVi6PA.png" /></figure><p>Our first plot (left) shows that as Artemis sails toward the Moon, the absolute magnitude of the solar pressure acceleration stays relatively stable, hovering right around the 0.0057 nm/s² range. Because a 4 day trip to the Moon barely changes our total distance relative to the massive 150-million-kilometer orbit of the Sun, the solar wind acts as a completely steady, relentless push.</p><p>Our second plot (right) tracks the absolute spatial divergence between our gravity only model and our photon-powered model, over 4 days this tiny affect actually moves the spacecraft ~0.45 km</p><h3>J2 Oblateness — Moon</h3><p>The Moon itself isn’t a perfect sphere either. The Moon has its own equatorial bulge, quantified by its own J2 value. We can’t really use our previous J2 approach that we applied to Earth because the J2000 coordinate system is anchored directly to Earth’s center of mass, and its vertical z-axis points directly out of Earth’s North Pole. Because Earth’s rotational axis stays fixed relative to this grid, our coordinate framework lines up with the shape of the planet. But the Moon is orbiting, drifting, and tilted relative to Earth. The Moon’s personal north pole is pointing in a completely different direction than Earth’s.</p><p>To figure out how the Moon’s shape is pulling on the spacecraft, we must step out of the static, inertial universe and jump into a rotating frame:</p><ol><li><strong>Step 1:</strong> We calculate the relative distance vector from the Moon to the spacecraft in the standard inertial frame (J2000)</li><li><strong>Step 2:</strong> We query SPICE to get a dynamic rotation transformation matrix (spice.pxform) that matches the exact microsecond of our flight time. We use this matrix to rotate our position vector into the Moon&#39;s body-fixed, rotating frame (IAU_MOON). In this frame, the z-axis points squarely out of the Moon&#39;s true rotation axis.</li><li><strong>Step 3:</strong> Now that our coordinates are properly aligned with the Moon’s geometry, we pass them through our standard accel_j2 function using lunar mass and lunar radius</li><li><strong>Step 4:</strong> This gives us an acceleration force vector mapped in the Moon’s frame. To feed this force back into our main tracker, we use a reverse rotation matrix to shift that acceleration vector straight back out into the J2000 inertial system</li></ol><p>The left plot tracks the raw acceleration magnitude exerted on Orion by the Moon’s equatorial bulge (J2) over the final 24 hours leading up to its close approach, as the clock ticks past hour 20, the spacecraft gets close to its target. In the final two hours, the force curves upward like a vertical cliff face.<br>The spatial proximity factor has officially taken complete control of the simulation. Because the spacecraft is reaching its absolute closest point to the lunar surface (perilune), the non-linear 1/r⁵ power law of the J2 multiplier explodes, utterly overwhelming the latitude variations and creating a massive gravitational spike</p><p>The right plot tracks the spatial divergence ( div_super_km) between the model with and without Lunar J2. The y-axis scale is incredibly tiny, topping out at around 0.000035 km (3.5 cm). You noted that at the end of the day, the final displacement is only 3.2 centimeters.</p><p>The unique, wavy shape of that specific plot is caused by the geometric focusing pocket of the Moon’s central gravity well pulling the two paths together before they snap apart.</p><p>Now you’ll ask why do we see that wavy thing on the right plot and not on the left, its because left plot is snapshot of instant of time, doesn’t care about trajectory history, right plot measures the accumulated path distance error over time. This distance gap is a dynamic entity. It is being pulled on by <em>t</em> wo different forces simultaneously: the lumpy Lunar J2 perturbation, and the massive, baseline point mass gravity of the Moon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jyrnjOBp9akv5V_H.png" /></figure><p>We sliced a 4 day simulation at the 72 hour mark to capture the final 24 hours. Because our previous simulation already having minor rounding drifts and solar radiation pressure shifts, the initial state vector already carried a small baseline velocity and position mismatch relative to the Moon</p><p>Hour 0–12 — As the simulation kicks off, the initial baseline velocity mismatch causes the two virtual spacecraft to drift apart, climbing up to 1.8 centimeters. At this distance from the Moon (hundreds of thousands of kilometers out), the local Lunar J2 force (Left Plot) is weak, so it isn’t contributing much yet. This initial hill is just the standard kinetic drift from initial conditions</p><p>Hour 12–20- Because the Moon’s core central gravity pulls everything directly toward its center of mass, it acts as a geometric lens. It physically pinches the two independent trajectories closer together. On right-hand plot, this gravitational focusing causes the distance error to shrink, dropping from 1.8 cm down to a local trough of 1.3 cm</p><p>But right at Hour 22, the spacecraft slams directly into the Lunar J2 acceleration wall seen on the left plot. At a close approach of just a few hundred kilometers, the Moon’s squashed, asymmetrical shape exerts a sudden, violent torque on the velocity vectors. Because the two spacecraft are at slightly different latitudes, this asymmetric bulge pulls on one capsule harder than the other. This gravitational shear completely overpowers the central focusing funnel, violently tearing the two paths apart in a sharp terminal hook that shoots up to 3.2 centimeters right at perilune</p><p>And well after all this dissection, the fact is, the impact of moon’s oblateness as you see in centimeters, so we’re gonna ignore this right now.</p><h3>Conclusion</h3><p>What we have so far is:</p><pre># Setup exact timestamps and ground truth data<br>t_eval_exact = horizons_data[&#39;times_et&#39;] - t_start_et<br>h_pos = horizons_data[&#39;positions&#39;]<br>h_vel = horizons_data[&#39;velocities&#39;]<br>time_hours = t_eval_exact / 3600.0<br><br># Extract Moon distance<br>moon_dist = horizons_data.get(&#39;moon_distances&#39;)<br><br># Propagate all intermediate models at the exact Horizons timestamps<br>sol_2body_exact = propagate_two_body(state0_test, (0, dt_test), t_eval=[dt_test])<br>sol_j2_exact = propagate_earth_j2(state0_coast, (0, t_eval_exact[-1]), t_eval_exact, accel_two_body_j2)<br>sol_nbody_exact = propagate_nbody(state0_coast, (0, t_eval_exact[-1]), t_eval_exact, t_start_et)<br>sol_ult_exact = propagate_srp(state0_coast, (0, t_eval_exact[-1]), t_eval_exact, t_start_et, COASTING_MASS, include_srp=True)<br><br># Calculate position errors against Horizons for each model<br>error_2body = np.linalg.norm(sol_2body_exact.y[0:3, :].T - h_pos, axis=1)<br>error_j2    = np.linalg.norm(sol_j2_exact.y[0:3, :].T - h_pos, axis=1)<br>error_nbody = np.linalg.norm(sol_nbody_exact.y[0:3, :].T - h_pos, axis=1)<br>error_ult   = np.linalg.norm(sol_ult_exact.y[0:3, :].T - h_pos, axis=1)<br><br>velmag_ultimate = np.linalg.norm(sol_ult_exact.y[3:6, :], axis=0)<br>velmag_horizons = np.linalg.norm(h_vel, axis=1)</pre><p>And we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DnXtaS9rPGnjv7ao.png" /></figure><p><strong>Plot Top-Left</strong></p><p>In a 2D XY trajectory overlay, our green dashed line ( Ultimate Sim) and the black line ( JPL Horizons) map out an identical path. They appear perfectly matched for the most part and deviate slightly. This kinda also demonstrates a dangerous visual trap in deep space mapping (or well anything on such astronomical scales): planetary scales mask massive operational drifts<strong>.</strong> Because cislunar space spans hundreds of thousands of kilometers, a tracking mismatch of several thousand kilometers looks like a minor pixel variation. We have to dive into the mathematical residuals to find the true story. (or plot these differences on a logarithmic scale)</p><p><strong>Plot Top-Right</strong></p><p>This tracks our predicted velocity magnitude against reality. Both tracks capture the expected kinetic deceleration as Orion climbs out of Earth’s heavy gravity well, dropping from 8.3 km/s down toward a 1–2km/s coast.</p><p>However, notice the sudden separation around the 82-hour mark (highlighted by the red vertical dotted line). The real-world tracking telemetry registers a distinct speed profile shift that our passive simulator misses entirely. Because a natural force like gravity does not suddenly switch gears like an automatic transmission, this is our first clear sign of a propulsive system actively firing. (TCM maneuver)</p><p><strong>Plot Bottom-Left</strong></p><p>This uses a logarithmic scale to show the cumulative position error over time for all four configurations. It clearly maps the hierarchy of cislunar forces:</p><ul><li><strong>Model 1: Pure Two-Body (Red Line):</strong> Explodes off the charts almost instantly, crossing the 1,000 km error mark within the first 12 hours. It finishes completely broken, nearly 10,000 km off target.</li><li><strong>Model 2: Earth J2 (Orange Line):</strong> Marginally safer during the first few hours, but follows the exact same catastrophic curve once the capsule leaves low Earth altitudes.</li><li><strong>Model 3: N-Body Gravity (Blue Line):</strong> The moment the moving gravitational wells of the Moon and Sun are introduced via SPICE, the error profile drops by orders of magnitude. The error line flattens out, illustrating how n body gravity stabilizes our system.</li><li><strong>Model 4: Ultimate with SRP (Green Line):</strong> Adding the faint push of solar radiation pressure strips away even more residual drift, yielding the lowest baseline error profile of any model.</li></ul><p>Despite matching reality with exceptional precision for the first half of the journey, our ultimate model hits a hard wall in the final leg of the transit, concluding with a final tracking error of 7,426.285 km.</p><p><strong>Plot Bottom-Right</strong></p><p>Our final panel tracks our straight-line distance to our target destination: the Moon. It maps our steady descent down into the lunar realm, highlighting a gray baseline of 130 km, NASA’s targeted close approach flyby altitude.</p><p>Because our passive simulation missed by over 7,400 km, our virtual version of Orion completely overshot this insertion window, sailing right past the lunar sphere into deep solar space.</p><p><strong>Why did our meticulously designed simulator miss by thousands of kilometers?</strong></p><p>Our ultimate model represents a universe where Artemis I is a totally passive, drifting rock. We don’t account for fuel consumption ( <strong>change in mass</strong>), <strong>TCM </strong>(active steering) etc.</p><p>To better encapsulate the known physics, we need to do the above and in addition, use<strong> higher order harmonics</strong> for earth, earth’s <strong>tidal fluctuations</strong> can also affect earth’s gravity (the bulge can vary a lot in mass and radius), the <strong>J2 we derived isn’t static</strong>.</p><p>Even <strong>moon</strong> has weird <strong>mass concentrations</strong> causing lumpy gravity which we did not consider here, NASA apparently uses another <strong>equation for lunar gravity</strong> derived from their past mission observations</p><p>And well Newtonian gravity is an approximation — we need to use <strong>GR corrections for our massive spinning Earth</strong></p><p>Even the SRP interacts causing <strong>secondary light and heat</strong> on the spacecraft, earth and the moon ( <strong>earthshine and moonshine</strong>) although those effects are usually very small in nanometer — picometer scale — but they’re variable as well.</p><p>In our equations we also didn’t account for our <strong>atmospheric drag</strong>, even in LEO the atmospheric drag is significant — however it is 0 in the cis lunar coast, but significant at launch and re entry.</p><p>Here in this post we tried to use our physics intuition and used numerical solvers for the differential equations we framed to describe the trajectory, then we compared it with the observed data of the mission.</p><p>What we built in Python is aForward Propagator. We gave it a starting point, hit “go,” and watched where the physics took it. But NASA doesn’t just guess and wait, they use a continuous, closed loop feedback system that mathematically stitches the simulation to real world data</p><p>They don’t have a database of positions perfectly handed to them. They have to measure the spacecraft from Earth using the Deep Space Network, once they have the noisy radio data, they feed it into their enterprise software (like MONTE). But instead of just plotting it, they run an <strong>Extended Kalman Filter (EKF)</strong> or a <strong>Batch Least Squares</strong> estimator:</p><p>This is where the magic happens:</p><ol><li>The software runs the <a href="https://docs.poliastro.space/en/stable/examples/Propagation%20using%20Cowells%20formulation.html">Cowell’s Formulation physics</a> simulation forward by one step.</li><li>It predicts what the DSN radio signal <em>should</em> look like based on that physics model.</li><li>It compares the prediction to the actual noisy radio data received from the spacecraft to generate a residual (like our error plots)</li><li>Using complex matrix calculus (Jacobians), the filter calculates exactly how much it needs to tweak the spacecraft’s velocity, position, and assumed forces to make the error drop to zero</li></ol><p>In the next post we will see how PINNs fare with this problem. Stay Tuned</p><h3>Read More</h3><p><a href="https://proceedings.scipy.org/articles/Majora-629e541a-009">MONTE — JPL/SciPy</a> <a href="https://docs.poliastro.space/en/stable/examples/Propagation%20using%20Cowells%20formulation.html">Cowell’s Formulation</a></p><p><em>Originally published at </em><a href="https://livnlearns.com/looking-at-artemis/"><em>https://livnlearns.com</em></a><em> on June 29, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e28e6d000d49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Equations V — Physics Informed Neural Networks (PINNs)]]></title>
            <link>https://medium.com/@winniew314/differential-equations-v-physics-informed-neural-networks-pinns-afc5376132a2?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/afc5376132a2</guid>
            <category><![CDATA[differential-equations]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[calculus]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 02:19:09 GMT</pubDate>
            <atom:updated>2026-06-30T20:08:58.724Z</atom:updated>
            <content:encoded><![CDATA[<p>How to teach Physics to an AI model</p><p>Neural Networks are AI</p><p>In our previous posts, we reached the limits of both the classical and computational worlds. Traditional numerical solvers, like Finite Element Methods (FEM) or LSODA are rigorous, but they are computationally crippling. Simulating the hypersonic fluid dynamics of a re-entering spacecraft or forecasting global spatiotemporal weather anomalies can take supercomputers days or weeks.</p><p>We need the simulations to be faster.</p><p>For the last decade, the tech industry’s answer to speed has been Machine Learning. We train massive Deep Neural Networks to look at past data and instantly predict the future — they act almost as <strong>Universal Function Approximators</strong>. But when applied to deep physical sciences, standard Artificial Intelligence has a fatal flaw: <strong>it is completely physically illiterate.</strong></p><p>If you train a standard neural network on atmospheric telemetry, it kind of acts like a highly advanced curve fitter. It might predict a weather front or an orbital trajectory that explicitly violates the conservation of mass, energy, or momentum, simply because it doesn’t “know” those laws exist, it could predict that mass is being spontaneously created out of nowhere, or that water is flowing uphill, simply because it found a statistical shortcut.</p><p>To model the future of the cosmos, we cannot just rely on pure data, nor can we rely on pure computation. We have to merge the two.</p><p>And that is the basis of PINNs — Physics Informed Neural Networks</p><p>Instead of forcing a computer to blindly calculate a physical grid, or blindly memorize a dataset, a PINN is a neural network that has the laws of physics mathematically baked directly into its brain.</p><h3>The Idea</h3><p>In traditional Machine Learning, when a model overfits sparse data, we use regularization (like L1 or L2 penalties) to artificially restrict the network’s weights and keep the predictions smooth.</p><p>Instead of arbitrarily penalizing large weights like that, a PINN uses the actual differential equations of the physical system (the ODEs or PDEs) as the ultimate regularization term. By encoding the laws of physics directly into the network, we inject a massive, undeniable <strong>prior</strong>. We forcefully restrict the space of admissible solutions. The network is no longer allowed to guess anything that violates the conservation of mass, energy, or momentum</p><p>Let’s take an example of the heat equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/0*GjGBrHInqOG2trSa.png" /></figure><p>Abstracting the operator and simplifying notations for ML math:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*xK-Div_ubJIU5J4O.png" /></figure><p>Now with Neural networks, instead of finding exact<strong> u(x,t)</strong> we <strong>approximate </strong>it using learned weights and biases of the network ( <strong>theta</strong>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/0*_aXz_NedIuHGoUIp.png" /></figure><p>Now we define a <strong>residual function</strong> for the network which measures how much our network’s output violates the PDE</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*coz_9Gv9ofnhp2wq.png" /></figure><p>The purpose of a Neural Network model is to find specific network parameters (weights and biases — theta) that minimize loss (L)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*KCHFM-pfo509Ik7b.png" /></figure><p>Here we <strong>define our loss as a combination of data loss and physics loss</strong> (Which is driven by our residual function which is essentially modelled as a PDE) — hence directly baking in the physics into a neural network.</p><h3>Working</h3><p>We will walk through how a PINN works using a general case</p><h3>Point Generation</h3><p>In traditional numerical solvers (like FDM or FEM), discretizing the universe is an agonizing process. You must build a rigid, perfectly interconnected grid or mesh. Generating a mesh that seamlessly wraps around without creating mathematical singularities can sometimes take an engineer days and in high dimensional physics, the number of grid points required explodes exponentially, a curse of dimensionality.</p><p>Because Physics Informed Neural Networks use Automatic Differentiation, they don’t care about grids. They evaluate spatial and temporal derivatives continuously. However, to compute the Physics Loss, we still have to tell the network where to check the physics.</p><p>We do this by generating <strong>Collocation Points</strong>, empty coordinates in space and time where we enforce the differential equation</p><p>Boundary / Initial Points which are derived from BVP and IVP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/0*IuLP44F2LpnXyVjs.png" /></figure><p>Collocation Points which are points generated by statistical sampling of random points inside the domain</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/0*6gQ3nEHs3b-_fLLK.png" /></figure><p>If we just scatter them completely randomly (Standard Monte Carlo), we risk leaving massive blind spots in our physical domain where the network can essentially cheat and ignore the physics. One of the most popular algorithms to do this statistical sampling is Latin Hypercube Sampling (LHS) which ensures our points are maximally spread out with equal probability bin.</p><p>There are also some techniques of Adaptive Sampling. As the neural network trains, we constantly monitor the physical residual (f) at all of our collocation points. If the network calculates a massive physics loss in a specific sub-region (indicating it is struggling to understand the math there), the algorithm dynamically generates thousands of new collocation points specifically clustered around that high-error zone.</p><p>All these values values for collocation points are stored as tensors</p><h3>Forward Pass</h3><p>We feed the neural network raw continuous coordinates: space (x) and time (t). The network acts as our universal approximator and spits out a predicted physical state, u_theta(x,t)</p><p>The neural network is no longer trying to find a correlation in a dataset. The neural network has literally become the mathematical function<strong>.</strong> It is a parameterized surrogate for the true, undiscovered analytical equation</p><p>In case of a simple multilayer perceptron (fully connected layers)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kyCiYKm0Mbr2hJYl.png" /></figure><p>The output of each layer is stored and parallel matrix dot product computed simultaneously across all points.</p><h3>Auto Differentiation</h3><p>To calculate the first and second derivatives required in our residual function, we use backward chain rule on the composite function</p><p>For instance, if the network had only one hidden layer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/0*UXJkt5YLYrEwPlfi.png" /></figure><p>Because PINNs use exact derivatives of activation functions, we tend to use smooth functions like tanh</p><p>In this step we are traversing back the computation graph and using a Jacobian Matrix to create tensors of first and second derivatives required in our equation.</p><p>Because of Autodiff, PINNs completely bypass the need to chop space and time into discrete meshes. They are mesh free. They evaluate the differential equation in continuous, smooth space, eliminating the grid based instabilities that plague traditional solvers.</p><h3>Loss Computation</h3><p>Standard neural networks only care about matching predictions to ground-truth data. A PINN cares about matching predictions to reality.</p><p>Now that we have our prediction (output of a forward pass), we calculate the loss function — which is a combination of PDE residual and our standard data loss.</p><p>Let’s start with data loss, the network must anchor itself to the known universe. We provide it with a set of boundary points or initial conditions (Nb), these are the specific coordinates where we have actual sensor data or known physical constraints (u_true)</p><p>In this case we’re taking a Mean Squared Error (MSE) between the predicted and true values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q8dgb2-vq2suTVrq.png" /></figure><p>If we stopped here, we would just have a standard, data hungry neural network. It would perfectly memorize the boundary points but hallucinate wildly everywhere else.</p><p>Now we introduce the governing physical law. In this case, we use the 1D Heat Equation as our residual, which dictates how temperature diffuses over space and time. In a perfect physical simulation, this equation must exactly equal zero.</p><p>We evaluate this residual across a massive set of Collocation Points (Nf). These are data free coordinates randomly sampled from inside our physical domain. The network has no ground truth data here, but we force it to evaluate its own continuous derivatives using Automatic Differentiation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/0*yETOOUHMY0RAjqXt.png" /></figure><p>The Data Loss and the Physics Loss operate on completely different scales and have completely different geometric landscapes.</p><p>Sometimes, the physics equation is incredibly complex to calculate, but memorizing the 5 sensor data points is easy. The network’s gradients will heavily prioritize dropping the Data Loss to zero, entirely ignoring the laws of physics. Other times, the network might find a trivial solution to the physics (like guessing the temperature is exactly 0 everywhere, making the PDE 0, satisfying the residual), ignoring the actual sensor data entirely.</p><p>Hence we parameterize total loss using specific weights</p><p>We can set them to our desired behavior or could even adjust them dynamically, such that at every epoch, we force the network to respect the boundary data and the physical laws equally, guiding it safely to a converged, physically accurate simulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/0*lt3LD8O03WRk-82z.png" /></figure><h3>Optimization</h3><p>To train the network, we must update our weights and biases (theta) such that the total loss goes to zero (L_theta). We do this by computing the gradient (g_t) of the loss function with respect to the weights .</p><p>This is standard in training any ML model, however for PINNs the nature of loss function is quite different warranting a different optimization approach</p><p>We decide to go two phased here, first using Adam (ADAptive Moment estimation) which is a first order optimizer, looking only at immediate scope of landscape.</p><p>Adam uses momentum and velocity, where <strong>momentum </strong>is exponentially decaying average of past gradients, physically, if the algorithm has been rolling down a steep hill for 100 steps, it builds up mathematical momentum. If it suddenly hits a small uphill bump (a local minimum), the momentum carries it right over the bump so it doesn’t get stuck.</p><p><strong>Velocity </strong>is average squared of gradients. This acts as an adaptive brake. If the gradients suddenly become massive and chaotic (indicating the algorithm just walked off a cliff), v spikes, automatically shrinks the step size (eta) so the algorithm doesn’t fly off into infinity. <br>If the landscape flattens out, v shrinks, allowing the algorithm to confidently take massive strides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cwA2LbVE0LmZK_05.png" /></figure><p>Adam is phenomenal at rapidly exploring rough terrain and finding the deepest valley. But once it gets to the bottom of that valley, it struggles to find the exact, pinpoint minimum. It bounces and oscillates violently around the bottom. And here comes the part 2 of our two phased approach.</p><p>Once Adam has safely navigated us to the bottom of the valley, we pause training and hand the neural network’s weights over to <strong>L-BFGS (Limited Memory Broyden Fletcher Goldfarb Shanno</strong>)</p><p>Where Adam only looks at the slope, L-BFGS looks at the entire landscape, it is a second order function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*qSuq1omyee0L-gGp.png" /></figure><p>If a neural network has 10k weights, the Hessian is 10k x 10k, which means 100M second derivatives at every time step, which gets very compute intensive, to solve for memory we usually only look at last m steps, (usually 50 or 100) and uses those past gradients to efficiently approximate the inverse Hessian Matrix.</p><p>And then we iterate this process till we get desirable results or reach our number of epochs or early stopping conditions.</p><h3>Conclusion</h3><p>In <strong>Parts I, II, and III</strong>, we explored the classical analytical engine. We used characteristic equations, Laplace transforms, and state space matrices to perfectly decode isolated, linear systems. We saw how brilliant minds mathematically reverse engineered the physical world.</p><p>In <strong>Part IV</strong>, we confronted the chaotic reality of the real universe. When the physics became violently nonlinear, we abandoned exact equations and learned to discretize reality, using numerical solvers like RK45 and Finite Element Methods to chop time and space into computable, simulated fragments, leveraging the power of computers to solve complicated physics.</p><p>And now, with <strong>Physics Informed Neural Networks</strong>, we no longer have to choose between the rigorous, deterministic truth of classical calculus and the blazing, predictive speed of modern AI. By forcing a neural network to act as a continuous surrogate model, anchoring it with sparse data, and punishing it for violating the differential equations using Autodiff and two-phase optimization, we achieve something profound. We stop treating machine learning as a blind curve fitter. We can teach the machine the laws of thermodynamics, fluid dynamics, and orbital mechanics.</p><p>Whether we are attempting to forecast spatiotemporal Earth Observation data, model the thermal dynamics of a re entering spacecraft, or filter the noisy astrophysics of the cosmos, we now have the ultimate toolkit.</p><p>We have finally built a machine that sees the geometry of change exactly the way we do.</p><p>I wrote these series as a preface to some projects I am working on the side and as a reference for what’s to come next, hope it was detailed and comprehensive enough to outline the whole landscape and improve understanding.</p><p><em>Originally published at </em><a href="https://livnlearns.com/differential-equations-v-physics-informed-neural-networks-pinns/"><em>https://livnlearns.com</em></a><em> on June 20, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afc5376132a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Equations IV — How Computers Solve Differential Equations]]></title>
            <link>https://medium.com/@winniew314/differential-equations-iv-how-computers-solve-differential-equations-9e8b2409e62a?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e8b2409e62a</guid>
            <category><![CDATA[math]]></category>
            <category><![CDATA[differential-equations]]></category>
            <category><![CDATA[calculus]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 18:46:14 GMT</pubDate>
            <atom:updated>2026-06-19T18:49:02.674Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous posts, we marveled at the elegance of the classical toolkit. We used characteristic equations, matrix exponentials, and integral transforms to perfectly decode the trajectories of physical systems.</p><p>But as we established, the classical engine has a fatal flaw: it demands a cooperative universe.</p><p>When say an engineer models a spacecraft re entering Earth’s atmosphere, they aren’t dealing with a polite, linear system. They are dealing with hypersonic fluid dynamics, extreme thermal gradients, ablation, and chaotic atmospheric drag. The exact, analytical math shatters under the weight of this nonlinearity.</p><p>To survive the real world, we have to step away from the approach of trying to find the perfect equation, or perfect technique that transforms our equation to an easily solvable one, because those methods only get us so far. Instead, we take the data, we discretize reality. We chop time and space into microscopic fragments and allow computers to calculate the future, one tiny frame at a time.</p><h3>Numerical</h3><p>When we cannot find a continuous function y(t) to describe our system, we use time stepping. We take our current, known state, look at the derivative (the slope) at this exact second, and use it to predict where we will be a fraction of a second from now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UctOl7zc3hf-Gi5e.png" /></figure><h4>Euler</h4><p>If you know your current position and your current velocity, you simply project that velocity forward over a small, constant time step (h)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*_hCjJ-vZ0n0Hq3z9.png" /></figure><p>It is a single step approach, explicitly based on the current state. It doesn’t account for history or takes preemptive test steps, the single step approach can be thought of as almost like cutting the Taylor series at 1</p><p>For example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*UChtsDPCDLYulx1c.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/0*HJzCq1MP7gEss3mm.png" /></figure><p>here ‘h’ needs to be microscopic to avoid truncation error under the assumption of constant slope in h timestep</p><p>While computationally cheap, Euler’s method is highly inaccurate for chaotic systems. Because it strictly uses the slope at the <em>start</em> of the step, any curvature in the true physics is completely ignored until the next step. Errors accumulate rapidly, causing simulated orbits to artificially decay or spiral out of control.</p><h4>RK — Runge Kutta</h4><p>Instead of blindly jumping forward based on a single slope, Runge-Kutta methods act like a scout. The standard RK4 method calculates the slope at the beginning of the step, takes a few trial steps to estimate the slope at the midpoint, and checks the slope at the end. It takes a weighted average of these trajectories before committing to the actual step.</p><p>There is RK1, RK2, RK3, <strong>RK4 </strong>and so on.. where the numeral represents the order, RK4 is the most common (fourth order) — we don’t go any higher as the marginal accuracy doesn’t usually justify the extra compute.</p><p>Even though it has the additional power of test steps, its still a single step, explicit and a fixed size step method</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3-hmPrV5cHEeNN6H.png" /></figure><p>In modern physics , we use adaptive solvers like <strong>RK45 </strong>( <strong>DO</strong>rmand <strong>P</strong> rince). RK45 calculates both a 4th order and a 5th order approximation simultaneously. By comparing the difference between the two, the algorithm estimates its own error. If the physical system is currently smooth, it lengthens the time step to save compute power. If the system enters a sharp curve (like a satellite whipping closely around a moon), it dynamically shrinks the time step to maintain high fidelity accuracy.</p><p>It features an adaptive time step (varying steps) and acceptable error tolerance, while remaining a single step explicit method.</p><p>It helps reduce the chances of incorrect initialization of timestep h in RK4, however it computes 7 slopes at every steps making it a bit compute heavy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*u34WVVyjRyuU5RdQ.png" /></figure><p>Another alternative is RK23 (Bogacki Shampine), which is a lite version of RK45, using RK2 and RK3, i.e. second and third order approximations, calculating 4 slopes and significantly decreasing compute time (and also accuracy) but it offers a faster alternative in cases where accuracy is not that big of concern.</p><h4>Heun’s Method</h4><p>If Euler’s method is walking forward with your eyes closed, Heun’s method is taking a quick peek before you step. It is a two stage process. First, it uses Euler to <em>predict</em> where the system will be. Then, it evaluates the slope at that future prediction and averages it with your current slope to <em>correct</em> the trajectory.</p><p>You can either look at it as Predictor and Corrector, where predictor is Euler on current step and corrector is the future step or you can simply look at it as <strong>RK2</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*zK1SWgMr44dFcsue.png" /></figure><h4>Adams Bashforth</h4><p>Runge-Kutta methods are single-step, they throw away all past data and calculate everything from scratch at each new step. But physical systems have momentum. The Adams Bashforth method is a multi-step explicit solver. It looks at the history of your previous slopes to predict the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*bXZZzQtfKMzf8-ii.png" /></figure><p>Its fixed multi step explicit process, which accounts for slope at current point n, and previous points n-1, n-2 and so on. Above is an example of a 4th order. It calculates only 1 slope per point and is computationally very fast.</p><p>If you are simulating a massive, smooth orbital trajectory, calculating the physics from scratch every millisecond is a waste of compute power. Adams Bashforth recycles the slopes from the last few steps to extrapolate the next one, making it highly efficient for smooth, well-behaved physical systems.</p><h4>BDF — Backward Differentiation Formulas</h4><p>Some physical systems are stiff, that is some quantities change way faster than other. This occurs when incredibly fast dynamics and incredibly slow dynamics happen simultaneously — like a rocket’s slow orbital decay paired with the microsecond high frequency vibrations of its thrusters. Explicit solvers (like RK45 or Adams Bashforth) choke on stiffness. They will wildly overshoot the true physics unless you make the time-step microscopically small, which stalls the simulation.</p><p>Here we employ a multi step implicit method with adaptive steps. Instead of using the current state to calculate the future, an implicit method sets up an algebraic equation where the future state depends on the <em>derivative of the future state</em>.</p><p>In first order, it can be considered as backward Euler. Because the future is trapped inside its own equation, BDF requires us to run a root-finding algorithm (like Newton Raphson) at every single time step just to move forward. It is computationally expensive per step, but it allows us to take massive time steps through stiff physics without the simulation exploding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/0*XfG-ZThjT3R7hMY4.png" /></figure><p>So we are essentially using a massive Jacobian to solve for y n+1 at every step</p><h4>LSODA — Livermore Solver for ODE</h4><p>Now the question arises, how do we know if a physical simulation is going to be stiff before we hit run? Often, we don’t. A spacecraft’s trajectory might be smooth for three days (non stiff) and then undergo a violent atmospheric re entry (stiff).</p><p>LSODA acts like an automatic transmission. It constantly monitors the physics in real time. When the system is smooth, it cruises using the highly efficient Adams Bashforth (explicit) method. The exact millisecond it detects physical stiffness, it drops the clutch and seamlessly shifts into the heavy duty BDF (implicit) mode, preventing the simulation from crashing.</p><p>It uses Adams Bashforth, calculates and tracks Lipschitz constant and the changing slope, checks if Adams and h are stable if not resorts to BDF, does periodic checks on stiffness and then resorts back to Adam when needed.</p><h4>FDM — Finite Difference Method</h4><p>When modeling the fluid dynamics of an atmosphere or the thermal distribution across a satellite’s solar panel, we must also discretize physical space.</p><p>A rigid grid is the oldest and most intuitive way to handle space. FDM shatters the continuous universe into a rigid, rectangular grid of points.</p><p>It replaces abstract continuous partial derivatives with discrete algebraic approximations using Taylor series. The derivative at any given point is simply calculated by looking at the difference between its immediate neighbors</p><p>the idea is to go from continuous data points -&gt; integer grid<br>From PDEs -&gt; Algebra</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*4smODRi73QvtBk0N.png" /></figure><p>From Algebraic Equation -&gt; Matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/0*ePLRHy3OQUcNq-MT.png" /></figure><p>Under the FDM umbrella, we have two critical schemes for stepping through time:</p><ul><li><strong>FTCS (Forward Time, Central Space):</strong> This is the explicit approach. You look at your neighbors right now (Central Space) to predict your exact state in the future (Forward Time). It is incredibly fast to compute, but notoriously fragile. If your time step is even slightly too large compared to your spatial grid, FTCS will violently destabilize.</li><li><strong>BTCS (Backward Time, Central Space):</strong> This is the implicit approach. Like BDF, it evaluates the spatial neighbors at the <em>future</em> time step. This forces you to solve a massive system of linear equations simultaneously, but in exchange, it is unconditionally stable. The simulation will never blow up, no matter how large your time step is.</li></ul><h4>SOR — Successive Over-Relaxation</h4><p>Systems like BTCS, generate a massive matrix representing thousands of interconnected spatial points, inverting a 1,000,000 x 1,000,000 matrix directly is very expensive, hence, we use iterative methods. We make a wild guess at the entire temperature grid, and then iteratively update each point based on its neighbors.</p><p>Now what is ‘over relaxation’ — SOR deliberately <em>overshoots</em> the correction based on a relaxation factor (omega), hence the name.</p><p>By mathematically “leaning” into the direction of the correction, SOR dramatically accelerates the convergence, allowing us to solve massive atmospheric or thermal grids in a fraction of the computing time</p><p>I haven’t looked into it in too much detail yet but i’d recommend going through — <a href="https://primer-computational-mathematics.github.io/book/c_mathematics/numerical_methods/6_Solving_PDEs_SOR.html">https://primer-computational-mathematics.github.io/book/c_mathematics/numerical_methods/6_Solving_PDEs_SOR.html</a></p><h4>FEM — Finite Element Method</h4><p>In FDM, we discretize a continuous space into rigid rectangular grid, but that doesn’t work for all situations where our space is possibly curved or asymmetical. FEM instead is based on geometry, it instead decomposes the space into geometric elements — a flexible mesh — could be triangles, tetrahedron, line segments</p><p>Then it uses integration by parts to convert the differential equation into an integral over volume problem, this process is called weak formulation, where instead of trying to satisfy the differential equation exactly at every point, it multiplies the physics by “test functions” and integrates across the area of each triangle. By assembling these thousands of tiny local integrals into one massive global matrix, FEM perfectly captures complex, real-world geometries that FDM could never handle.</p><p>Then it assumes basic shape functions (e.g. Linear Interpolation for line segments geometry) say N1 and N2</p><p>Then calculates integrals using the basic shape functions above to create a Local stiffness matrix (Ke)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/0*XyKv_uFydWjsqOCS.png" /></figure><p>Then it combines the local matrices to a global one</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*7sT2iLivOn3GqoBA.png" /></figure><p>and then applies the boundary condition</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*p0IIy1NnwDwMJzFh.png" /></figure><h3>Conclusion: The Computational Bottleneck</h3><p>In the real world, the universe does not hand us neat, analytically solvable equations. To model the chaotic realities of aerodynamic drag, orbital decay, or atmospheric thermodynamics, we have to discretize reality.</p><p>By stepping carefully through time with algorithms like RK45 and LSODA, and by shattering continuous space into grids and meshes using FDM and FEM, we have successfully taught computers how to simulate the physical universe. These traditional numerical methods are the undisputed workhorses of modern engineering. They are the software engines that currently guide our satellites and predict our weather.</p><p>But they have a fatal flaw: they are incredibly computationally expensive.</p><p>When you scale these methods up to simulate the fluid dynamics of an entire planet or the real time aerodynamics of a rocket launch, the matrices become so massive that even supercomputers choke. Sometimes, simulating a complex weather system using traditional numerical grids takes longer than the actual weather takes to arrive.</p><p>We have reached the limits of pure, brute force computation. To push the boundaries of Earth Observation and astrophysics forward, we cannot just keep building finer grids and taking smaller time steps. We need a fundamental paradigm shift.</p><p>We need to stop forcing computers to calculate physics from scratch, and start <em>teaching</em> them the laws of physics.</p><p>In <strong>Part V: Scientific Machine Learning</strong>, we will cross the final frontier. We will explore how to merge the raw predictive speed of Artificial Intelligence with the absolute deterministic rigor of classical calculus, resulting in the architecture that is actively reshaping research in the field: <strong>Physics Informed Neural Networks (PINNs)</strong>.</p><h3>References</h3><p><a href="https://math.libretexts.org/Courses/Monroe_Community_College/MTH_225_Differential_Equations/03%3A_Numerical_Methods/3.03%3A_The_Runge-Kutta_Method">LibreTexts — RK</a> <a href="https://www.cambridge.org/core/journals/journal-of-the-australian-mathematical-society/article/on-rungekutta-processes-of-high-order/40DFE501CAB781C9AAE1439B6B8F481A">On Runge-Kutta processes of high order — Cambridge University press</a> <a href="https://primer-computational-mathematics.github.io/book/c_mathematics/numerical_methods/5_Runge_Kutta_method.html">Primer Computational Math — Imperial College London</a></p><p><em>Originally published at </em><a href="https://livnlearns.com/differential-equations-iv-numeric-solutions/"><em>https://livnlearns.com</em></a><em> on June 19, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e8b2409e62a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Equations III — Systems, Signals, and State Space]]></title>
            <link>https://medium.com/@winniew314/differential-equations-iii-systems-signals-and-state-space-live-learn-5e10284964a5?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e10284964a5</guid>
            <category><![CDATA[differential-equations]]></category>
            <category><![CDATA[calculus]]></category>
            <category><![CDATA[math]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 14:50:53 GMT</pubDate>
            <atom:updated>2026-06-19T18:49:31.670Z</atom:updated>
            <content:encoded><![CDATA[<p>In last part, we learned how to solve isolated differential equations, a single pendulum swinging, a single spring oscillating. But physical reality is rarely isolated, and it is rarely smooth.</p><p>What happens when a physical system experiences a violent, instantaneous shock, like a sudden thruster burn on a satellite? The continuous, well behaved polynomials we’ve relied on shatter. Furthermore, what happens when variables are inextricably linked? The pitch, roll, and yaw of a spacecraft tumbling in orbit cannot be solved one at a time, they are a coupled system of equations where every movement bleeds into the next.</p><p>To survive this level of complexity, we have to change our mathematical perspective a bit.</p><h3>Analytical — Integrals</h3><p>When the time domain becomes too chaotic to navigate, we don’t force our way through it. Instead, we use integral transforms to teleport the entire differential equation into a different mathematical dimension, solve it there using basic algebra, and then teleport the solution back.</p><h4>Fourier</h4><p>Imagine taking a chaotic, noisy signal plotted on a standard x-y graph (amplitude over time). The Fourier Transform takes that time based wire and physically wraps it around a circle. The speed at which you wrap the wire around the circle is the frequency (omega)</p><ul><li>If your winding frequency omega does <em>not</em> match any frequency hidden inside the signal, the peaks and valleys cancel each other out. The wire winds symmetrically, and the center of mass stays perfectly dead-centered at the origin (0,0)</li><li>But, if your winding frequency exactly matches a hidden frequency embedded in the noise, the peaks align. The wire bulges heavily to one side, and the center of mass spikes away from the origin.</li></ul><p>The Fourier Transform is simply a mathematical radar sweeping through every possible winding frequency. When the center of mass spikes, you have detected a pure wave hidden inside the chaos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*rAEm4RgZVo0NCw4_.png" /></figure><p>Mathematically, the engine that drives this circular winding is the complex exponential e^(-i*omega*t) <br>By multiplying our time domain signal by this winding engine and integrating across all of time, we calculate that center of mass and hence convert the math from the time domain (t) into the frequency domain (omega)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/0*TwFgjePu11-Cy0gE.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*djDwzBIrP7a8u2m5.png" /></figure><p>Because in the frequency domain, the complex derivatives of a Partial Differential Equation (like that of the Heat Equation or wave equations) collapse. By decomposing our physical reality into a pure spectrum of simple sine and cosine waves, we can solve the math for each individual frequency using basic algebra, and then add them all back together.</p><p>Once again, here is an excellent visualization of how this works by 3Blue1Brown:</p><p><a href="https://www.youtube.com/watch?v=r6sGWTCMz2k&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=4">But what is a Fourier series? From heat flow to drawing with circles | DE4</a></p><p>Following is the solution of heat equation using Fourier transforms:</p><h4>Laplace</h4><p>Laplace also helps us to transform our equation from a time domain (t) to a frequency domain (s).</p><p>Imagine a satellite silently coasting through the vacuum of space. Suddenly, a micrometeorite strikes its hull, or an onboard computer fires a 0.5-second thruster burst to correct its orbit. In the time domain (t), these events are mathematical nightmares.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*jl_nUJn4d7ZuKY92.png" /></figure><p>By integrating the differential equation alongside a decaying exponential e^(-st), the Laplace transform converts complex derivatives into simple algebraic multiplication.</p><p>The following video by 3 blue 1 brown gives an excellent visual intuition of what it does, how it does it and how it really works, definitely check it out :</p><p><a href="https://www.youtube.com/watch?v=j0wJBEZdwLs&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=6">But what is a Laplace Transform?</a></p><p>How it works is:</p><p>Instead of tracking <em>when</em> things happen, the s-domain (s = sigma + i*omega) tracks the <em>anatomy</em> of the system’s behavior specifically, its exponential decay and sinusoidal frequencies.</p><p>Unlike Fourier which just winds the signal around a circle, Laplace introduces a modifier — a real number exponential decay factor (e^-sigma*t i.e. geometrically — a spiraling, shrinking vortex) and hence it takes the exploding signal and forcibly squashes it down so that it shrinks as time goes forward and Only <em>after</em> the signal has been safely squashed does the machine wind it around the circle</p><p>Because s is made of two dimensions (the squashing factor sigma on the x axis, and the winding frequency omega on the y axis), the output of the Laplace Transform isn’t a simple 2D graph but a 3D topographical map over the complex plane, on which we look for Poles i.e. points where the map shoots up into infinitely high spikes.</p><ul><li>The y axis (omega) tells you how fast the system naturally oscillates or vibrates.</li><li>The x axis (sigma) tells you the system’s survival. If a pole lands on the left side of the map (negative sigma), the system naturally decays and stabilizes. If a pole lands on the right side of the map (positive sigma), the system grows exponentially and will physically destroy itself.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/0*go2d7mqtBEb4xYqi.png" /></figure><p>There is another video by 3 blue 1 brown summarizing these applications way better:</p><p><a href="https://www.youtube.com/watch?v=FE-hM1kRK4Y&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=7">Why Laplace transforms are so useful</a></p><p>Here is an example of how this method can be applied to solve equations:</p><h3>Analytical — Systems</h3><p>In classical physics, we often model systems as if they exist in a vacuum, focusing on a single variable changing over time. But reality is entangled.</p><p>When you model the orbital dynamics of a multi-body system, or the pitch, roll, and yaw of a satellite, the rate of change of one variable depends directly on the current state of all the others. We are no longer dealing with a single equation, we are dealing with a matrix of differential equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rZsilfiwUWgRk7uR.png" /></figure><h4>Substitution</h4><p>If we have a simple system of two coupled variables, say, the coupled velocities of two masses on a vibrating string, the most classical approach is simply to eliminate one of them to see the whole picture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*pmB73P0DnpKELHig.png" /></figure><p>By isolating one variable in the first equation and substituting its derivative into the second, we can crush a system of two first order ODEs into a single, uncoupled second order ODE. We then solve this using the classical characteristic equations we learned in Part II. It works perfectly for simple systems. But as you add a third, fourth, or fiftieth dimension, this brute force algebraic substitution collapses under its own weight.</p><h4>Eigen Value &amp; Eigen Vector</h4><p>Let’s go back to our satellite tumbling out of control in orbit. From the outside, the chaotic mixture of pitch, roll, and yaw looks incredibly complex to model. But if you shift your perspective to look down the exact right angles, the principal axes of rotation, the motion suddenly makes perfect sense. The satellite is just spinning cleanly around these invisible lines.</p><p>This is exactly what Eigenvectors (v) and Eigenvalues (lambda) do for our system of differential equations. The Eigenvectors act as those fundamental axes. They represent the specific directions in our multidimensional space where the coupled system behaves as if it were completely uncoupled. The corresponding Eigenvalues dictate exactly what is happening along that specific axis, whether the motion decaying, growing, or oscillating.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*JGwhBhQdoV9zPiHe.png" /></figure><p>By breaking the equation into eigenvalues and eigenvectors, we end up converting our equation into independent solvable threads of exponential growth and decay</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*cF7zweVOyza395wF.png" /></figure><h4>Matrix Exponential</h4><p>In Part I, we learned that the fundamental blueprint for a 1D linear differential equation (x’ = ax) is the exponential function: x(t) = C e^(at). But what if a isn’t a single number? What if A is a massive matrix containing the physical constraints of an entire spacecraft.</p><p>We can still use the same math, but challenge here is how to interpret e raised to an entire matrix, <br>we use the infinite Taylor series expansion, or we use our Eigenvectors to diagonalize the matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/0*SNfQHzmYZXNvwr3m.png" /></figure><p>Another 3Blue1Brown video explaining this beautifully:</p><p><a href="https://www.youtube.com/watch?v=O85OWBJ2ayo&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=8">How (and why) to raise e to the power of a matrix | DE6</a></p><h3>Conclusion</h3><p>With integral transforms and state-space mechanics, we have reached the absolute peak of the exact mathematical world, for the most part at least. We have learned how to untangle massive matrices of coupled variables which could potentially predict a spacecraft’s orientation, and we have learned how to map explosive, discontinuous physical shocks into clean, algebraic frequency domains.</p><p>These analytical tools are monumental achievements. They are the mathematical engines that put humanity on the moon and allow us to filter the noise of the cosmos.</p><p>All of the elegant methods we have discussed, from characteristic equations and matrix exponentials to Laplace and Fourier transforms, rely heavily on the assumption that the underlying physical system is linear, or at least polite enough to be approximated as such.</p><p>But the real universe is rarely polite.</p><p>When we introduce extreme nonlinear chaos, like the unyielding fluid dynamics of atmospheric entry, the unpredictable N-body gravitational tug-of-war, or the massive spatiotemporal datasets of modern Earth Observation, our exact equations shatter. The matrices become too large, the integrals become unsolvable, and the classical math breaks down.</p><p>To survive the bleeding edge of modern physics and engineering, we hand the geometry over to the machines.</p><p>In <strong>Part IV</strong>, we will explore the Numerical Methods that actually run inside modern mission control simulators, culminating with the Physics-Informed Neural Networks (PINNs)</p><h3>References</h3><p><a href="https://www.youtube.com/watch?v=r6sGWTCMz2k&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=4">But what is a Fourier series? From heat flow to drawing with circles | DE4</a> <a href="https://www.youtube.com/watch?v=j0wJBEZdwLs&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=6">But what is a Laplace Transform?</a> <a href="https://livnlearns.com/vector-calc-review-laplace-in-spherical-coordinates/">Laplace in Spherical Coordinates</a> <a href="https://www.youtube.com/watch?v=FE-hM1kRK4Y&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=7">Why Laplace transforms are so useful</a> <a href="https://www.youtube.com/watch?v=O85OWBJ2ayo&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=8">How (and why) to raise e to the power of a matrix | DE6</a></p><p><em>Originally published at </em><a href="https://livnlearns.com/differential-equations-iii-systems-signals-and-state-space/"><em>https://livnlearns.com</em></a><em> on June 19, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e10284964a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Equations II — The Classical Mechanics of Calculus]]></title>
            <link>https://medium.com/@winniew314/differential-equations-ii-the-classical-mechanics-of-calculus-live-learn-2e552482d735?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e552482d735</guid>
            <category><![CDATA[differential-calculus]]></category>
            <category><![CDATA[differential-equations]]></category>
            <category><![CDATA[calculus]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 12:43:36 GMT</pubDate>
            <atom:updated>2026-06-19T18:49:47.857Z</atom:updated>
            <content:encoded><![CDATA[<p>In Part I, we explored the geometric blueprints of differential equations. We looked at first order systems and how we can occasionally use clever substitutions to bypass the curse of nonlinearity.</p><p>But the physical universe is rarely satisfied with first order changes. Gravity, electromagnetism, and orbital mechanics all rely on <em>acceleration</em> — the rate at which the rate of change is changing. To model these, we need second order (and possibly higher) differential equations.</p><p>When the universe hands us a linear, higher order system, we don’t have to guess blindly. We have a pristine, toolkit designed to tear these equations apart and solve them analytically. Let’s look at some of these methods.</p><h3>Analytical — Higher Order</h3><p>When we deal with linear higher-order ODEs, we divide our physical reality into two states: <strong>Homogeneous</strong> (unforced systems operating in a vacuum g(x) = 0) and <strong>Non-Homogeneous</strong> (systems being pushed or pulled by an external force, g(x) != 0)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*uVL3nWAXcNf6eMvR.png" /></figure><h4>Characteristic Equation</h4><p>Its a simple straightforward strategy for solving higher order homogenous equations (systems with no external forces acting on it) with constant coefficients (P(x) and Q(x) are constants). The intuition is to substitute an exponential such that the equation can be solved like a simple quadratic equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/0*K30Wpd7VnbYk8NT6.png" /></figure><p>The physical intuition is to solve an equation where a function, its velocity, and its acceleration all balance out to zero, we use a function that is completely immune to calculus,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/0*ttDNFGiy4jj0ZWb2.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/0*5OQxHRMAbnx93Q7_.png" /></figure><h4>Method of Undetermined Coefficients</h4><p>Now lets move to the case of constant coefficients but a non homogenous situation (i.e. g(x) != 0)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*f3EMLtc2cuG-vAUn.png" /></figure><p>This strategy relies on the assumption that a system will eventually mimic the force acting upon it, If the forcing function g(x) is relatively simple (a polynomial, an exponential, or a sinusoid), we simply guess that our final solution will have that exact same shape, plug it into the ODE, and solve for the missing coefficients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*bLeNkUdnMWmbBXF5.png" /></figure><h4>Variation of Parameters</h4><p>Now we move onto the case where our function is not a simple one that we can guess a solution to, i.e. f(x) here isn’t a simple function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*UUl7UVcfZ46UJ4ww.png" /></figure><p>This is a bit more complicated than the previous ones, what we do here is, instead of guessing a rigid shape, we take the constants (C1, C2) from our homogeneous solution and allow them to “breathe”. We transform them into evolving functions of time: u1(x) and u2(x). By using a mathematical construct called the <strong>Wronskian</strong> (which checks the fundamental independence of our solutions), we can integrate these parameters to find the exact trajectory of the system, no matter how weird the forcing function is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*lF64N6Bh7qCvympu.png" /></figure><h4>Cauchy Euler</h4><p>The cases we discussed so far involved only constant coefficients, now we move to equations where P(x) and Q(x) aren’t necessarily constant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*eZNpDjTnrMOHJIVx.png" /></figure><p>These kind of equations can model spherical or cylindrical symmetry, like calculating the gravitational potential deep inside the layers of a star, or the heat distribution expanding across a circular radar dish.</p><p>Because the power of the x coefficient exactly matches the order of the derivative here, our trusty exponential guess doesn’t work, so instead we guess a polynomial, and substitute to transform the equation into a solvable algebraic equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*YS8Owfms7Gv9O_3h.png" /></figure><h3>Analytical — Series</h3><p>What happens when the coefficients are not constants, the symmetry is broken, and the universe hands us a completely unsolvable mathematical mess?</p><p>We stop trying to find a neat, closed form equation. Instead, we build the solution from scratch, piece by piece, using infinite series</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/0*TS9PSq8ZsKfNFTn9.png" /></figure><p>In this case for instance if we let y = e^rx or y = x^r, the x acting as a coefficient of y creates a problem.</p><p>Hence we leverage infinite series like Taylor series and the Power series to build up the solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u4dcII12KgLfJkq3.png" /></figure><h4>Ordinary Points</h4><p>This method is applicable only to systems with analytic function as the coefficients (i.e. P(x) and Q(x) are smooth and defined at 0)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*WEUc2LvpGtxWhoyU.png" /></figure><p>Because the space is smooth, we assume our solution can be perfectly described by an infinite Power Series, We differentiate this infinite sum, plug it back into our ugly differential equation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/0*oUlYBavwiApzUYph.png" /></figure><p>To solve this, we shift the indices until all the powers of x align. This gives us a “recurrence relation” a master algorithm that computes every single coefficient (c0, c1, c2..) one by one, building the true physical curve with as much precision as we have the patience to calculate</p><h4>Frobenius</h4><p>This method is applicable to systems with non analytic function as the coefficients (i.e. P(x) and Q(x) are not defined at 0)</p><p>Imagine calculating the electric field exactly at the center of a point charge, or modeling the physics at the event horizon of a black hole. At these locations, the physical variables blow up to infinity. In differential equations, this mathematical tear in the fabric is called a Singular Point</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*2faiVOGcO_V73vsn.png" /></figure><p>We modify our infinite series by multiplying the entire thing by an extra parameter of x^r</p><p>By finding the roots of the “indicial equation,” we adapt the geometry of our math to navigate around the singularity, allowing us to find valid physical solutions even when the universe tries to break the rules</p><h3>Conclusion: The Limits</h3><p>Classical analytical methods and infinite series are monumental achievements of human intellect. They allow us to peer into the mechanics of the universe, decoding the exact trajectories of pendulums, the vibrations of satellite antennas, and the intense physics warping around singular points.</p><p>But as powerful as this classical engine is, it has its limits.</p><p>What happens when a physical system experiences a violent, instantaneous shock, like a sudden thruster burn on a spacecraft, or a lightning strike on a power grid? The continuous, smooth geometry we’ve relied on shatters. Polynomials and infinite series struggle to describe sudden, violent discontinuities.</p><p>Furthermore, series solutions are fundamentally <em>local</em>. They build reality step-by-step from a single starting point, but they don’t easily give us a global view of a massive, multi-variable system.</p><p>To handle the true chaos of modern engineering and astrophysics, we have to entirely change our mathematical perspective. We have to stop looking at time and space directly, and instead transform our equations into the domains of frequency and complex variables. And when even those advanced analytical tools fail us against the crushing weight of nonlinearity, we have to teach machines to solve the physics for us.</p><p>In <strong>Part III</strong>, we will cross that final frontier. We will explore Integral Transforms (Laplace and Fourier) and finally, the Numerical Methods and Physics Informed Neural Networks (PINNs) that power modern computational research.</p><p><em>Originally published at </em><a href="https://livnlearns.com/differential-equations-ii-the-classical-mechanics-of-calculus/"><em>https://livnlearns.com</em></a><em> on June 19, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e552482d735" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Equations I — Intuition & Analytical Solutions]]></title>
            <link>https://medium.com/@winniew314/differential-equations-i-intuition-analytical-solutions-75ba7905e50b?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/75ba7905e50b</guid>
            <category><![CDATA[differential-calculus]]></category>
            <category><![CDATA[calculus]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Thu, 18 Jun 2026 19:36:25 GMT</pubDate>
            <atom:updated>2026-06-19T12:41:52.054Z</atom:updated>
            <content:encoded><![CDATA[<p>“Change is the only constant in life”</p><p>That’s also how the universe works in a way and how we study it, how something changes with respect to another and differential equations is what we use to describe this.</p><p>The universe is not a static photograph it is a continuously evolving engine. Whether we are modeling the orbital decay of a satellite, the fluid dynamics of a hurricane, or the heat dissipating across a spacecraft’s hull, we are studying how one physical state changes with respect to another. We use them to model almost physical behavior from pendulums and projectiles, celestial objects, air resistance and heat equation.</p><p>Now going into the terminologies and symbolism of these equation, we broadly categorize differential equations into two types</p><h3>Types — The Anatomy of Change</h3><p>If the changes are with respect to a single variable for example change in 1D, 2D or even 3D space with respect to time, we call these Ordinary Differential Equations ( <strong>ODEs</strong>)</p><p>Imagine tracking the trajectory of a satellite in Low Earth Orbit (LEO). Its position and velocity change as time marches forward. The equation simply asks: <em>Given your exact coordinates and momentum right now, where will you be in the next instant?</em> It traces a single, deterministic thread through space.</p><p>If the system is governed by multiple inputs like temperature change in a rod that changes with both distance and time, we formulate them as Partial Differential Equations ( <strong>PDEs</strong>)</p><p>The temperature depends not just on <em>when</em> you measure it, but exactly <em>where</em> on the 3D surface you measure it. You are no longer tracking a single thread; you are modeling the entire fabric of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oXQrtXvFh5hoWtSf.png" /></figure><h4>Phase Space &amp; Stability Analysis: Seeing the Future</h4><p>In the real world, finding a clean, exact mathematical solution is rare. Before we even attempt to calculate a specific answer, we look at the geometry of the equation to determine the overall <em>stability</em> of the system. We do this by mapping the equation into a <strong>Phase Plane</strong>.</p><p>In a phase plane, we don’t plot position against time. We plot the state variables against each other, for example, position (x) on one axis and momentum (v) on the other.</p><p>When you map a differential equation into phase space, it generates a vector field. This field acts like a fluid flow of arrows, dictating the exact destiny of the system. By analyzing this geometry, we can identify critical <strong>equilibrium points</strong>:</p><ol><li><strong>Attractors (Stable):</strong> If a satellite is in a perfectly stable, repeating orbit, the phase space vector field will show all nearby arrows spiraling inward toward a closed loop or a single point. No matter where the system starts nearby, it will be pulled into this stable configuration.</li><li><strong>Repellers &amp; Saddle Points (Unstable):</strong> If a satellite is experiencing atmospheric drag and its orbit is decaying, the vector field reveals an unstable geometry. Even a microscopic perturbation a slight change in solar wind or atmospheric density will cause the trajectories to diverge wildly, pushing the system into chaos or forcing the satellite to burn up in the atmosphere.</li></ol><p>Stability analysis allows us to understand the fate of a physical system simply by looking at the geometry of its derivatives, without ever solving the equation.</p><p>The following 3 blue 1brown video has a very nice visualization of phase space towards the end of the video, of how these phase spaces look like for some of the popular and even completely arbitrary differential equations</p><p><a href="https://www.youtube.com/watch?v=p_di4Zn4wz4&amp;list=PLZHQObOWTQDNPOjrT6KVlfJuKtYTftqH6&amp;index=1">Differential equations, a tourist’s guide | DE1</a></p><p>Now you must be asking, i geometric intuition is so powerful, why do we need dense analytical methods or massive computational solvers?</p><p>The answer lies in the difference between linear and nonlinear realities.</p><p>When a differential equation is <strong>linear</strong>, its vector field is neat, predictable, and symmetrical. It obeys the principle of superposition, you can break the system into smaller, manageable pieces, solve them individually, and seamlessly add them back together.</p><p>But the universe is rarely that accommodating.</p><p>When systems become <strong>nonlinear</strong>, the geometry becomes entangled. The vector fields twist, fold, and exhibit extreme sensitivity to initial conditions. For a long time, this was a mathematical dead end.</p><p>However, we don’t always have to immediately surrender to heavy computers. For certain, highly specific first-order nonlinear systems, mathematicians have discovered brilliant analytical loopholes. These analytical techniques act as mathematical sleight of hand. They even allow us to temporarily warp the geometry of a nonlinear equation, disguising it as a linear one just long enough to solve it cleanly by hand. (We will explore these exact analytical tricks in the next section).</p><p>The true “curse” of nonlinearity only reveals itself when systems scale up.</p><p>When we move beyond isolated first-order equations into massive, coupled nonlinear systems, like the N-body problem governing the gravitational tug-of-war between satellites and moons, or the Navier-Stokes equations dictating disaster-scale weather patterns, our analytical intuitions only get us so far. To survive in that chaotic geometry, we are forced to abandon exact human calculations and approximate reality using heavy numerical solvers and even Physics-Informed Machine Learning. (We will explore these computational architectures in later posts.</p><h3>Solving Differential Equations</h3><p>There are several ways we can solve a differential equation, there are a couple of Analytical as well as Numerical methods, lets first discuss analytical ways we can solve these equations, it can be as simple as solving a linear equation or can be a Laplacian or involve integrating by parts, the way to solve depends on the type of differential equations.</p><h3>Analytical Methods — First Order</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*h1ykUtPP0ldP6PVA.png" /></figure><h4>Separation Of Variables</h4><p>Probably the simplest most intuitive way to solve differential equations, a number of differential equations — both ODEs and PDEs of first order</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/0*AFv-4s5JNDi8XFUv.png" /></figure><p>The intuition is to separate the two variables and integrate separately.<br>Imagine trying to untangle two different colored strings that have been tightly knotted together. Separation of variables is the mathematical equivalent of pulling the red string to the left and the blue string to the right until the knot dissolves. The intuition is that if the physical forces acting on a system are independent of each other, their mathematics should be independent too</p><p>This is so vital in physics and you must have used this while solving for heat equation or exponential decay separating temporal and spatial changes and integrating them 1 by 1</p><p>The only limitation of this simplistic strategy is — it only works when the function is not very complex and is easily separable — but as you’ll see that’s not always the case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*ZtwNzndDiGdfqihW.png" /></figure><h4>Integrating Factor</h4><p>A universal approach to solving first order linear equations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*v7x-jsuMw5bCnpS4.png" /></figure><p>The intuition is to multiply both left and right hand side by a factor which would help resolve the derivative using a product rule, so separation of variables and integral is possible.</p><p>This is one of the examples of mathematical elegance showing how adding a seemingly complex functions can beautifully simplify the equation.</p><p>This “magic trick” however requires the differential equation to be strictly linear. If your dependent variable (y) is squared, or trapped inside a sine wave, the integrating factor shatters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/0*CVFmkaoe9RGiYOd9.png" /></figure><h4>Exact Equations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/0*TGHml6lNF1Ud8IhW.png" /></figure><p>Can only be used if equation is exact</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*i4aEdM78qO5_zq9k.png" /></figure><p>There is a general way to solve these kind of equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*UH938chehGYwGCks.png" /></figure><p>This method is deeply geometric. We use exact equations when our 1D differential equation is actually a shadow cast by a 2D topographical map, M and N are actually partial derivatives of the parent function F, hence, we are integrating the pieces we have to reconstruct the hidden parent function.</p><p>In physics, “exactness” often translates to conservative fields. If you are calculating the work done moving a satellite through Earth’s gravitational field, the energy required depends only on the start and end points, not the path taken. That gravitational field is an “exact” mathematical system</p><h4>Bernoulli Substitution</h4><p>Used for first order non linear equations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/0*oV7Zr4ipWPfmud7q.png" /></figure><p>The intuition is to substitute using exponential of 1-n such that the equation can be converted into linear equation, this works for all values of n other than 0, 1</p><p>This is another example of mathematical elegance where we are basically disguising the system to look linear so we can solve it easily!</p><p>A general way of solving such equation would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/0*0aW5f-L2feltfcWV.png" /></figure><h4>Homogenous Substitution</h4><p>What homogenous here means that in this particular form, both numerator and denominator on the Right Hand Side have a degree of 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/0*CXPvRJebG0GUxreI.png" /></figure><p>We actually work with these quite often assuming a linear relationship between x and y and converting the complex function such that the whole equation becomes separable</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*pcalqE9lKOFaok92.png" /></figure><p>Which basically means that the whole system is scaling proportionally, Because the <em>ratio</em> between the variables is the only thing that matters, we stop trying to solve for y directly</p><p>This is highly relevant in fluid dynamics and thermodynamics, where the behavior of a system often depends on the ratio of properties (like density over pressure) rather than their absolute values</p><h4>Riccati Substitution</h4><p>These are again used when our equations are first order and non linear (containing squared variable and given a known solution) a very specific use case and a notoriously difficult equation e.g.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*ktjraNp-RRQTpt7I.png" /></figure><p>Like all substitution, the purpose is to simplify the equation to an easily solvable form, the specific substitution choice we make to this possible is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/0*KYneNyOcmz_A-rGZ.png" /></figure><p>When a satellite is tumbling and needs to stabilize its attitude using reaction wheels, the onboard computer is actually continuously solving a matrix version of the Riccati equation to figure out the optimal torque to apply without overshooting, Riccati equations also frequently pop up in 1D gas dynamics and thermodynamics, specifically when modeling the flow of compressible gases through nozzles</p><h3>Conclusion</h3><p>Geometry allows us to see the shape of the universe. By looking at phase spaces and vector fields, we can predict whether a physical system will stabilize into a beautiful orbit or spiral into chaos. We can see the destiny of the system before we ever calculate a single number.</p><p>But eventually, we need exact coordinates.</p><p>Seeing that a satellite’s orbit is decaying is not enough, a mission control engineer needs to know <em>exactly</em> when it will hit the atmosphere. Knowing that a heat shield will cool down isn’t enough, we need the precise temperature at T+40 seconds. Geometric intuition gives us the blueprint, but to actually build the physical reality, we have to roll up our sleeves and do the math.</p><p>When the universe is kind enough to give us linear systems or when we can use clever substitutions to disguise nonlinear ones we rely on the elegant, exact tools of classical calculus.</p><p>In <strong>Part II</strong>, we will open up that classical toolkit. We will move beyond first-order velocities and dive into the higher-order equations that govern acceleration, gravity, and orbital mechanics, exploring how brilliant minds mathematically reverse-engineered the physical world.</p><p><em>Originally published at </em><a href="https://livnlearns.com/differential-equations/"><em>https://livnlearns.com</em></a><em> on June 18, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75ba7905e50b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Loss Functions for Extreme Class Imbalance]]></title>
            <link>https://medium.com/@winniew314/bce-focal-dice-and-tversky-deep-dive-47dfa811e280?source=rss-9cebad599e29------2</link>
            <guid isPermaLink="false">https://medium.com/p/47dfa811e280</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Winnie W.]]></dc:creator>
            <pubDate>Mon, 18 May 2026 20:21:41 GMT</pubDate>
            <atom:updated>2026-06-30T20:10:19.244Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Diving deep into — BCE, Focal, Dice, and Tversky loss</blockquote><h3>What is a Loss Function?</h3><p>As an intuitive understanding loss would be predicted value — actual value i.e. how much off is prediction to our actual target, most of you might be familiar with R² loss in high school which is what we often compute while fitting points to a line.</p><p><strong>Loss function is what quantifies the discrepancy between what the model <em>thinks</em> reality is, and what reality <em>actually</em> is.</strong></p><p>For a single training example i.e. a single data point, it outputs a scalar representing that specific prediction’s error. When we average this error across an entire dataset or a mini batch during optimization, it becomes the <strong>Cost Function</strong> or <strong>Objective Function</strong> that the model tries to minimize.</p><p>Now why are Loss function important in ML?</p><p>Just like a physical system works on the principle of Least action ( <a href="https://livnlearns.com/?s=lagrangian">Lagrangian</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/0*G7gx-RngFAn-NyrM.png" /></figure><p>What it means is, nature basically evaluates the Lagrangian (difference of kinetic energy and potential energy) — and defines the path of the particle — which is the path of least action.</p><p>That is how our ML models also learn — following the path of least action i.e. decreasing loss function — to get to a better and better understanding of the inherent relations in the data to either output those or to output a prediction.</p><p>There are a ton of loss functions out there — some of the most common ones like RMSE, MAE, MSE etc. for regression like problems and CE, BCE, Log Loss, Hinge Loss etc. for classification like problems, we will probably cover those in detail some other time and you could try to get familiar with the basics here if you want:<br><a href="https://www.datacamp.com/tutorial/loss-function-in-machine-learning">https://www.datacamp.com/tutorial/loss-function-in-machine-learning</a></p><p>In this post I will cover some loss functions I’ve been using in my current iteration on a project involving <strong>imbalanced classification</strong> on an image (pixel level classification).</p><p>Pointwise functions would be those, that take into account single prediction and true value, compute loss and decide how much it should weigh in overall fitting / learning trajectory.</p><h3>Binary Cross Entropy (BCE)</h3><p>The essence of this function lies in framing the problem as a series of independent coin flips, what that means is — we calculate the loss (log loss actually) of each pixel independently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/0*ZCS9vJYBONXAZkQf.png" /></figure><p>Which if you think about it, will <strong>heavily penalize confidently incorrect predictions (using a log scale!)</strong></p><h4>Walkthrough</h4><p>Now to demonstrate how it works, lets work with this example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*sFF3WADmfiNs6lZz.png" /></figure><p>We have this image with 9 pixels in which only 1/9 is flooded (our target class)</p><p>Now our first model generates a series of predictions p1, which completely misses our target, which is common in a binary classification if a class imbalance isn’t handled properly, what it means is, if the model trains on data thats 99% dry (target 0) — classifying all pixels as class 0 is a guaranteed way to optimize accuracy.</p><p>in this example, our loss would come out to be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*D7TgZsl3ag_sXAHF.png" /></figure><p>Now lets look at p2, the predictions of another model (or second iteration of this one), the model correctly classifies the flooded pixel and in a way localizes the neighborhood as well, i.e. some pixels next to the flooded pixel also get classified as flooded, in this case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mmBE27XVPURxn2pz.png" /></figure><p>So if we evaluate these two iterations with BCE we would assume that p1 is superior to p2</p><h4>Notes</h4><ul><li>Measures distance between predicted and true probability distributions</li><li>Best for balanced binary classification problems</li><li>It is a convex function — smooth gradient</li><li>Probabilistic — so helps callibrate for max likelihood</li><li>could have runtime errors with log(0) -&gt; -inf</li><li>Pytorch implementation han ln and also has BCEWithLogitsLoss (Sigmoid + BCE) implementation</li></ul><h3>Cross Entropy</h3><p>Cross entropy is for multi class mutually exclusive labels</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/0*X_xRNo_0HOZGjmle.png" /></figure><p><strong>Feature</strong> <strong>Binary Cross-Entropy (BCE)</strong> <strong>Cross-Entropy (CE)</strong></p><h3>Focal</h3><p>Focal Loss was an improved, dynamically modified BCE, that was <a href="https://arxiv.org/pdf/1708.02002">introduced by Facebook</a> originally for a R-CNN framework.</p><p>The essence of this function is that unlike BCE that treats all errors equally, Focal loss introduces weights, and down weights the well classified examples <strong>forcing the network to mine for hard examples</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*AeKo-dKYmU16XC0_.png" /></figure><p>It has two hyperparameters alpha and gamma, gamma term makes for the modulating factor dictating how much should samples who are easily classified weigh and alpha helps tune loss function to the inherent imbalance of the dataset.</p><p>Intuition is that,</p><ul><li>gamma controls the steepness of the penalty for being wrong</li><li>alpha roughly reflects the inverse frequency of classes, but must be dampened if using a high gamma</li></ul><p>Usually we start with gamma = 2.0 and alpha = 0.25 (which roughly corresponds to the true target being 25% of the values)</p><h4>Walkthrough</h4><p>Now to demonstrate how it works, lets work the same example as above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*HSia6IvP6Xs39aXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*O5yKfKRLGoV9HHz6.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/0*Tb-6h2i2-jWq1R6z.png" /></figure><p>Here we see if a pixel is classified easily, like p = 0.95 (i.e. for the void pixels of prediction 0.05), the modulating factor with gamma, reduces their contribution to overall loss function significantly.</p><h4>Notes</h4><ul><li>A good potential solution for highly imbalanced datasets (especially the highly sparse ones)</li><li>Makes network focus on its representational capacity</li><li>Mines for rare / hard samples</li><li>Potentially very sensitive to noise — minor noise could be treated as extremely rare samples and dictate the optimization direction</li><li>Computationally its significantly more expensive than BCE</li><li>Numerical stability needs handling when p approaches 0 or 1</li><li>Performance highly dependent on hyperparameter selection (alpha and gamma) — adds another layer of tuning/selection</li></ul><p>Instead of pixel level attribution, these evaluate overall overlaps and use the True Positives, False Negatives, false Positives etc. in their calculations, letting you optimize for the desired outcome directly.</p><p>These are called geometric or boundary functions because they focus on preserving the decision boundary.</p><h3>Dice</h3><p>It is used to calculate overlap between predicted mask and ground truth, or as we call Intersection Over Union (IoU). The essence is the function <strong>ignores the background altogether and cares only about true and predicted values of target class</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/0*KcclGeql3PT9KetD.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*yL0IDQYPAHLdkZFc.png" /></figure><h4>Walkthrough</h4><p>To explain how this works, we will consider this example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*JpMwscXeK_uv7msZ.png" /></figure><p>Here we have an image of 100 pixels, in which 10 are flooded (i.e. our target class) and the rest 90 are class 0.</p><p>Now, lets say our model predicts 15 of the 100 pixels to be flooded, in which only 8 are truly flooded (True Positives).</p><p>This means we have 7 false alarms (False Positives), and we missed 2 true floods (False Negatives) and the rest 83 were non flooded which we guessed correctly (True Negatives)</p><p>As you can see even though TN make about 83% of our data — we don’t really care for them in our loss function.</p><h4>Notes</h4><ul><li>Blind to TN</li><li>It directly optimizes for IoU which is the metric we care about</li><li>Its good when preserving boundaries is important (Medical Imaging / SAR)</li><li>Its inherently balanced since intersections are relative</li><li>because of the mentioned scale invariance — can swing wildly from image to image i.e. missing 100 pixels in 10k pixels is a way smaller penalty than missing all 10 in 10</li><li>By itself — its a non convex function and would be challenging to find a global minima — causing unstable gradients</li><li>Its blind to the types of errors — our use case cares about FP more than FN or vice versa — this loss function does not accommodate that calibration</li><li>Needs the term epsilon added — to prevent division by 0 during runtime (prone to vanishing gradients)</li></ul><h3>Tversky</h3><p>Basically modified Dice with two parameters alpha and beta added to tune the FP and FN weight as per use case requirement. Th idea is that <strong>not all mistakes carry same penalty</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*tRF0eT5XbTbFjC0Y.png" /></figure><p>alpha = beta = 0.5 =&gt; we have dice loss function</p><p>alpha &lt; beta =&gt; Penalizing FN (Optimizing Recall) — Missing a true label is more expensive</p><p>alpha &gt; beta =&gt; Penalizing FP (Optimizing Precision) — Being confident in true labels is critical</p><h4>Walkthrough</h4><p>To explain how this works, we will consider the same example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mlIxwrKKvU7N01wD.png" /></figure><p>Here we have an image of 100 pixels, in which 10 are flooded (i.e. our target class) and the rest 90 are class 0.</p><p>Now, lets say our model predicts 15 of the 100 pixels to be flooded, in which only 8 are truly flooded (True Positives).</p><p>This means we have 7 false alarms (False Positives), and we missed 2 true floods (False Negatives) and the rest 83 were non flooded which we guessed correctly (True Negatives)</p><p>Now, if the model instead made the opposite error (2 False Positives and 7 False Negatives), the denominator would become 8 + (0.3 * 2) + (0.7 * 7) = 13.5, resulting in a Tversky Index of 0.59 and a higher Loss of 0.41. The math explicitly punishes the network harder for missing the target than for over predicting it</p><h4>Notes</h4><ul><li>pretty much same as dice, completely blind to true negatives</li><li>great for real world usecases / risks — lets optimize penalties for FP, FN</li><li>Like focal loss — sensitive to parameter selection — adds another layer of hyperparameter tuning and selection</li><li>Because of hyperparameter weighting — can be really sensitive to noise — if say FN penalized highly i.e. beta = 0.9, model would overindex and bloat target label predictions</li><li>Can optimize better for Recall than Dice</li><li>like dice non convex — can lead to unstable gradients if used by itself</li><li>Same issues as Dice for scale invariance — can fluctuate wildly from image to image</li><li>Same issues as Dice needs the term epsilon added — to prevent division by 0 during runtime (prone to vanishing gradients)</li></ul><h3>Conclusion</h3><p>There are a number of widely tested loss functions which are better suited to a specific problem and model, I would recommend trying with the most basic ones for the first prototype and then researching the ones best suited to your use case, you might need to define a custom one yourself too! But I hope this post helps give some insight into few of the ones covered, I’ll follow up with a deep into some other ones.</p><h3>Implementation</h3><p>In my case I’ve tried to implement a combination of pointwise and overall loss functions</p><p>Focal + Tversky:</p><pre>class FocalTverskyLoss(nn.Module):<br><br>  def __init__(<br>    self,<br>    focal_alpha: float = 0.25,      # focal class weight on positive pixels<br>    focal_gamma: float = 2.0,      # down-weight easy pixels: (1 - pt)^gamma<br>    tversky_alpha: float = 0.3,    # FP cost in Tversky denominator<br>    tversky_beta: float = 0.7,     # FN cost (β &gt; α → recall-friendly)<br>    smooth: float = 1e-5,          # epsilon - to prevent division by 0<br>    tversky_weight: float = 1.0,<br>  ):<br>    super().__init__()<br>    # focal loss<br>    # alpha indicates class weight to positive class, gamma is the modulating factor<br>    self.focal_alpha, self.focal_gamma = focal_alpha, focal_gamma<br>    # tversky loss<br>    # beta = 0.8, alpha = 0.2  - penalizes FN 4x times more than FP<br>    self.tversky_alpha, self.tversky_beta = tversky_alpha, tversky_beta<br>    # smooth - epsilon - to prevent division by 0<br>    self.smooth, self.tversky_weight = smooth, tversky_weight<br><br>  def forward(self, logits, targets, valid_mask=None):<br>    #extract valid area to ignore void pixels<br>    m = torch.ones_like(targets) if valid_mask is None else valid_mask.float()<br>    #total number of pixels (valid)<br>    n = m.sum() + self.smooth  <br><br>    # --- Focal: batch-level pos_weight in BCE, then (1-pt)^gamma focusing ---<br>    # with torch.no_grad():<br>      # TP count<br>      # pos = (targets * m).sum()<br>      # TN count<br>      # neg = ((1 - targets) * m).sum()<br>      # dynamic ratio of positive to negative  (capped at 50)<br>      # pos_w = (neg / (pos + self.smooth)).clamp(1.0, 50.0)  # rare-flood upweight<br>    # calculate BCE <br>    # bce = F.binary_cross_entropy_with_logits(logits, targets, reduction=&quot;none&quot;, pos_weight=pos_w)<br>    bce_unweighted = F.binary_cross_entropy_with_logits(logits, targets, reduction=&quot;none&quot;)<br>    pt = torch.exp(-bce_unweighted)<br>    # static alpha balancing map<br>    alpha_t = self.focal_alpha * targets + (1.0 - self.focal_alpha) * (1.0 - targets)<br>    # Focal formula<br>    # focal = (w * (1 - pt).pow(self.focal_gamma) * bce_unweighted * m).sum() / n<br>    focal_map = alpha_t * (1.0 - pt).pow(self.focal_gamma) * bce_unweighted<br>    focal = (focal_map * m).sum() / n<br><br>    # --- Tversky: soft tp/fp/fn per sample, then mean(1 - index) ---<br>    # Converts raw logits to probabilities and zeroes out invalid pixels<br>    p = (torch.sigmoid(logits) * m).flatten(1)<br>    # Flatten Ground Truth<br>    t = (targets * m).flatten(1)<br>    tp = (p * t).sum(1)<br>    fp = (p * (1 - t)).sum(1)<br>    fn = ((1 - p) * t).sum(1)<br>    # Tversky Formula<br>    tversky = 1 - (tp + self.smooth) / (tp + self.tversky_alpha * fp + self.tversky_beta * fn + self.smooth)<br>    # Averages the Tversky scores across the batch and adds the Focal loss<br>    return focal + self.tversky_weight * tversky.mean()</pre><p>BCE + DICE loss (assuming its for more or less balanced datasets):</p><pre>class BCEDiceLoss(nn.Module):<br>    def __init__(self, smooth: float = 1e-5):<br>        super().__init__()<br>        self.smooth = smooth<br>        self.bce = nn.BCEWithLogitsLoss(reduction=&quot;none&quot;)<br><br>    def forward(self, logits: torch.Tensor, targets: torch.Tensor, valid_mask: torch.Tensor = None) -&gt; torch.Tensor:<br>        <br>        if valid_mask is None:<br>            valid_mask = torch.ones_like(targets)<br><br>        valid_mask = valid_mask.float()<br>        <br>        # BCE Loss (masked)<br>        bce_map = self.bce(logits, targets)<br>        bce_loss = (bce_map * valid_mask).sum() / (valid_mask.sum() + self.smooth)<br><br>        # Vectorized Dice Loss<br>        probs = torch.sigmoid(logits)<br>        <br>        # Flatten spatial dimensions to [Batch, Features]<br>        batch_size = probs.shape[0]<br>        p_flat = (probs * valid_mask).view(batch_size, -1)<br>        t_flat = (targets * valid_mask).view(batch_size, -1)<br>        <br>        inter = (p_flat * t_flat).sum(dim=1)<br>        p_sum = p_flat.sum(dim=1)<br>        t_sum = t_flat.sum(dim=1)<br>        <br>        dice_score = (2.0 * inter + self.smooth) / (p_sum + t_sum + self.smooth)<br>        dice_loss = (1.0 - dice_score).mean()<br><br>        return bce_loss + dice_loss</pre><h3>Resources</h3><p>Basic Intuition:</p><ul><li><a href="https://www.datacamp.com/tutorial/loss-function-in-machine-learning">https://www.datacamp.com/tutorial/loss-function-in-machine-learning</a></li><li><a href="https://www.kaggle.com/code/nghihuynh/understanding-loss-functions-for-classification">https://www.kaggle.com/code/nghihuynh/understanding-loss-functions-for-classification</a></li></ul><p>Relatively Comprehensive Landscape:</p><ul><li><a href="https://arxiv.org/html/2504.04242v1">https://arxiv.org/html/2504.04242v1</a></li></ul><p>Focal Loss:</p><ul><li><a href="https://arxiv.org/pdf/1708.02002">https://arxiv.org/pdf/1708.02002</a></li></ul><p><em>Originally published at </em><a href="https://livnlearns.com/bce-focal-dice-and-tversky-flood-prediction-loss-deep-dive/"><em>https://livnlearns.com</em></a><em> on May 18, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47dfa811e280" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>