<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Somil Singh on Medium]]></title>
        <description><![CDATA[Stories by Somil Singh on Medium]]></description>
        <link>https://medium.com/@thesomilsinghofficial?source=rss-959694b93e8a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XGiDu969VFihz6Ga61rpDw.png</url>
            <title>Stories by Somil Singh on Medium</title>
            <link>https://medium.com/@thesomilsinghofficial?source=rss-959694b93e8a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:06:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thesomilsinghofficial/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep Learning #14: A correct sampler can still give the wrong learning signal]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@thesomilsinghofficial/deep-learning-14-a-correct-sampler-can-still-give-the-wrong-learning-signal-1a6f3405f779?source=rss-959694b93e8a------2"><img src="https://cdn-images-1.medium.com/max/1760/1*imI1M7vMNk3Z-3_0eqjf-Q.png" width="1760"></a></p><p class="medium-feed-snippet">Derive the energy-model likelihood gradient, then measure what a short local chain leaves out on 61 states.</p><p class="medium-feed-link"><a href="https://medium.com/@thesomilsinghofficial/deep-learning-14-a-correct-sampler-can-still-give-the-wrong-learning-signal-1a6f3405f779?source=rss-959694b93e8a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@thesomilsinghofficial/deep-learning-14-a-correct-sampler-can-still-give-the-wrong-learning-signal-1a6f3405f779?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a6f3405f779</guid>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 13:12:35 GMT</pubDate>
            <atom:updated>2026-10-19T17:10:45.683Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Deep Learning #13: Lower energy still doesn’t tell you the probability]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@thesomilsinghofficial/deep-learning-13-lower-energy-still-doesnt-tell-you-the-probability-20b449862c3f?source=rss-959694b93e8a------2"><img src="https://cdn-images-1.medium.com/max/2340/1*ok3xkfANCgntctsMGVKd2w.png" width="2340"></a></p><p class="medium-feed-snippet">Build an energy-based model you can normalize exactly, check its learning gradient and see where a good optimizer still leaves a bad fit.</p><p class="medium-feed-link"><a href="https://medium.com/@thesomilsinghofficial/deep-learning-13-lower-energy-still-doesnt-tell-you-the-probability-20b449862c3f?source=rss-959694b93e8a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@thesomilsinghofficial/deep-learning-13-lower-energy-still-doesnt-tell-you-the-probability-20b449862c3f?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/20b449862c3f</guid>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Thu, 15 Oct 2026 17:50:56 GMT</pubDate>
            <atom:updated>2026-10-15T17:50:56.112Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Deep Learning #12: A flow can reverse perfectly and still learn the wrong thing]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@thesomilsinghofficial/deep-learning-12-a-flow-can-reverse-perfectly-and-still-learn-the-wrong-thing-4dd4b823bc5f?source=rss-959694b93e8a------2"><img src="https://cdn-images-1.medium.com/max/1849/1*HuaSmQJ7kWGVbVDc0Jv9ew.png" width="1849"></a></p><p class="medium-feed-snippet">Two reversible layers, one likelihood and a gradient check. Part 12 builds the chain on CPU, with full code and measured output.</p><p class="medium-feed-link"><a href="https://medium.com/@thesomilsinghofficial/deep-learning-12-a-flow-can-reverse-perfectly-and-still-learn-the-wrong-thing-4dd4b823bc5f?source=rss-959694b93e8a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@thesomilsinghofficial/deep-learning-12-a-flow-can-reverse-perfectly-and-still-learn-the-wrong-thing-4dd4b823bc5f?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dd4b823bc5f</guid>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 15:18:40 GMT</pubDate>
            <atom:updated>2026-10-14T15:18:40.061Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Normalizing flows: bend a distribution without losing the way back]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@thesomilsinghofficial/normalizing-flows-bend-a-distribution-without-losing-the-way-back-7fa69ccacd6b?source=rss-959694b93e8a------2"><img src="https://cdn-images-1.medium.com/max/2340/1*OALGiloyMZITJbS4fJSNEQ.png" width="2340"></a></p><p class="medium-feed-snippet">A curved cloud, a reversible map and a density check. Part 11 fits a tiny flow on CPU, with full code and actual output.</p><p class="medium-feed-link"><a href="https://medium.com/@thesomilsinghofficial/normalizing-flows-bend-a-distribution-without-losing-the-way-back-7fa69ccacd6b?source=rss-959694b93e8a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@thesomilsinghofficial/normalizing-flows-bend-a-distribution-without-losing-the-way-back-7fa69ccacd6b?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fa69ccacd6b</guid>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 13:06:27 GMT</pubDate>
            <atom:updated>2026-10-14T13:09:20.085Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Why DDIM can change the sampling path without changing the noise task]]></title>
            <link>https://medium.com/@thesomilsinghofficial/why-ddim-can-change-the-sampling-path-without-changing-the-noise-task-fb04e7d2c260?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb04e7d2c260</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 17:02:54 GMT</pubDate>
            <atom:updated>2026-10-11T17:02:54.223Z</atom:updated>
            <content:encoded><![CDATA[<h3>How the DDIM update shares its noise budget, with a derivation and three runnable checks.</h3><p>How can the same noise predictor follow a different path back to data?</p><p>Somil Singh · Inside AI</p><p><strong>Deep Learning Models · Part 10</strong></p><p>Part 09 kept the predictor fixed and tried shorter DDIM schedules. Four clusters still appeared at 20 steps. Their spread changed though.</p><p>Lemme unpack the update behind that result. We can check its variance budget, recover the adjacent DDPM mean and measure what happens when the predicted noise is slightly wrong. No new training run needed for this one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9Yb7MggM3wHGV-5aUMGJA.png" /></figure><p><em>Schematic: the training pair stays at one noise level while the sampler chooses how to move between levels.</em></p><h3>1. A noisy level is a distribution, not a whole trajectory</h3><p>Start with the familiar corruption equation:</p><p>x_t = sqrt(A_t) x_0 + sqrt(1 - A_t) epsilon.</p><p>Here A_t is the cumulative product of the per-step signal factors. A_0 = 1. Later noisy times have smaller A_t. Epsilon is standard Gaussian noise. These are the DDPM forward marginals. <a href="https://arxiv.org/html/2006.11239v2">DDPM paper, equations 4 and 14</a>.</p><p>Condition on a particular clean value x_0. Then the mean is sqrt(A_t) x_0 and the variance is 1 - A_t. That&#39;s all a single-time marginal tells us. It doesn&#39;t specify how noise at two times is correlated.</p><p>Our scalar check takes x_0 = 1.2 and A_t = 0.25. Its noisy mean is 0.6 and its variance is 0.75. There are many ways to connect that distribution to an earlier level with A_s = 0.64.</p><h3>2. The variance has a budget</h3><p>DDIM constructs alternative joint paths while retaining those single-time marginals. Its reverse update replaces the unknown clean value with a prediction. <a href="https://arxiv.org/html/2010.02502v4">DDIM paper, equations 7, 9 and 12</a>.</p><p>Use t for the current noisy time and s for an earlier time. Write a = A_t and b = A_s, so 0 &lt; a &lt; b &lt;= 1.</p><p>The predicted clean value is:</p><p>x0_hat = (x_t - sqrt(1 - a) epsilon_hat) / sqrt(a).</p><p>Then move to s:</p><p>x_s = sqrt(b) x0_hat + sqrt(1 - b - sigma²) epsilon_hat + sigma z.</p><p>The signal term gets sqrt(b). The old residual gets a direction coefficient. Fresh independent Gaussian noise z gets sigma.</p><p>For this diagnostic only, suppose we know x_0 and the actual epsilon that corrupted it. The clean estimate is exact. The two noise terms have total variance (1 - b - sigma²) + sigma² = 1 - b. Same budget for every admissible sigma. Their dependence on the old state changes.</p><p><strong>Knowing the corruption noise is stronger than having an optimal predictor.</strong> At generation time the network sees x_t and t. It doesn’t see the hidden x_0 or the individual noise draw.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NXON9eV4Vx2jxYm5PZjfsw.png" /></figure><p><em>Schematic: three eta choices share a target conditional mean of 0.96 and variance of 0.36. The experiment below checks sampled values.</em></p><h3>3. The objective still asks for predicted noise</h3><p>The usual simplified loss samples one clean example, one time and one corruption noise:</p><p>L = E[w_t ||epsilon - f_theta(x_t, t)||²].</p><p>For an unrestricted function, its best prediction is the conditional mean E[epsilon | x_t, t]. Expand the square around that mean: the expected error becomes the unavoidable conditional variance plus the squared distance from the mean. The cross term vanishes. A positive weight at that time doesn’t change this pointwise minimizer.</p><p>A finite network shares parameters across times. It can trade accuracy at one level for accuracy at another. Different loss weights can therefore change its fitted solution. The DDIM paper’s weighted-objective argument includes a separate-parameters condition; don’t turn it into a promise that every finite shared network has identical optima.</p><p>The shared marginal noise task makes a fitted DDPM predictor usable in the DDIM sampler. It doesn’t certify perfect samples on every schedule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJ9Ac-4WZ99CD6f7tMzmDg.png" /></figure><p><em>Schematic: the target noise is available during corruption. The clean value is hidden from the predictor.</em></p><h3>4. Eta chooses the fresh noise and changes the coefficients</h3><p>Use the generalized variance from the authors’ implementation:</p><p>sigma² = eta² (1 - b)/(1 - a) (1 - a/b).</p><p>Eta = 0 removes fresh step noise. Eta = 1 with adjacent original times recovers the DDPM posterior variance. <a href="https://github.com/ermongroup/ddim/blob/main/functions/denoising.py">Authors’ sampler code</a>.</p><p>Let’s put numbers into the update. Take x_t = 0.9 and epsilon_hat = 0.6 with a = 0.25 and b = 0.64. The clean estimate is about 0.760770. At eta = 0 the direction coefficient is 0.6. At eta = 1 some of that noise budget moves into sigma z. The output block gives both values with z fixed to zero.</p><p>Now perturb only epsilon_hat by delta while holding x_t, the times, eta and z fixed. Substitution gives:</p><p>Delta x_s = [sqrt(1 - b - sigma²) - sqrt(b/a) sqrt(1 - a)] delta.</p><p>The coefficient measures local sensitivity to a prediction error. Our central finite difference checks it against the implemented update. Later steps will see a changed state and make changed predictions too, so this one coefficient isn’t a full trajectory error bound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F64rLcqkkNpt7NJTTRTvtQ.png" /></figure><p><em>Schematic: the numeric inputs are fixed. The runnable check reports the resulting output and derivative.</em></p><h3>5. Deterministic steps still start from random noise</h3><p>At eta = 0 a fixed predictor, fixed schedule and fixed initial state produce a fixed output. The initial noise is still random. That’s where the generated distribution comes from.</p><p>Training chooses noisy levels from the original schedule. Generation can choose a subset of those levels and compute the correct a and b for each jump. Skipping a level doesn’t mean applying an adjacent-step coefficient ten times further away.</p><p>The last jump uses b = A_0 = 1. Both residual coefficients vanish and the update returns the predicted clean value. Don’t evaluate the predictor at the clean endpoint; evaluate it at the current noisy time then perform that final jump.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_0sBqL2i4OvpiH-bek6RA.png" /></figure><p><em>Schematic: this lesson checks the algebra with an analytic toy predictor. It doesn’t train an image model.</em></p><h3>6. What these checks can and can’t catch</h3><p>A negative square-root argument often points to reversed time ordering, the wrong cumulative coefficient or an invalid variance. Check those before clipping. The code allows tiny roundoff error only after asserting the variance budget.</p><p>The known-noise marginal check verifies a conditional construction. Generation uses a predicted conditional mean instead. Those are different tests.</p><p>An adjacent eta = 1 identity checks the algebra against a DDPM posterior mean with zero fresh noise. It doesn’t prove a skipped eta = 1 schedule reproduces all original DDPM samples.</p><p>Finally, the four-Gaussian experiment reuses the analytic predictor from Part 09. It isolates finite schedule behaviour. Cluster counts and distance to centres are diagnostics here. Lower distance alone can mean the clusters have become too narrow (the target within-cluster mean squared radius is about 0.0288).</p><p><strong>Keep these three ideas</strong></p><ol><li>A single-time noise distribution leaves room for different correlations between times.</li><li>The DDIM update splits a fixed variance budget between the reused residual and fresh noise.</li><li>A deterministic schedule can be repeatable and still change the output when its time grid changes.</li></ol><h3>7. Run the three checks</h3><p>Use Python with NumPy and Matplotlib. Install those packages if needed with python -m pip install numpy matplotlib. Run the three blocks in order in one script or notebook. The last block writes the measured plot to the current directory.</p><p><strong>Step 1. Check conditional marginals.</strong> Here we supply the actual corruption noise and known clean value. Eta changes their coupling while the target mean and variance stay fixed.</p><pre>import numpy as np<br>import matplotlib<br>matplotlib.use(&quot;Agg&quot;)<br>import matplotlib.pyplot as plt<br>import platform<br>rng = np.random.default_rng(31)<br>def coefficients(a, b, eta):<br>    assert 0 &lt; a &lt; b &lt;= 1 and 0 &lt;= eta &lt;= 1<br>    variance = eta**2 * (1-b)/(1-a) * (1-a/b)<br>    direction2 = 1-b-variance<br>    assert direction2 &gt;= -1e-12<br>    return np.sqrt(max(variance, 0)), np.sqrt(max(direction2, 0))<br>def update(x, eps, a, b, eta=0, z=0):<br>    sigma, direction = coefficients(a, b, eta)<br>    clean = (x-np.sqrt(1-a)*eps)/np.sqrt(a)<br>    return np.sqrt(b)*clean + direction*eps + sigma*z<br>a, b, x0 = 0.25, 0.64, 1.2<br>n = 100000<br>eps, z = rng.normal(size=n), rng.normal(size=n)<br>xt = np.sqrt(a)*x0 + np.sqrt(1-a)*eps<br>print(f&quot;Python {platform.python_version()} | NumPy {np.__version__} | CPU | seed 31&quot;)<br>print(f&quot;known x0={x0} | conditional draws={n} | no training&quot;)<br>marginals = []<br>for eta in (0, 0.5, 1):<br>    xs = update(xt, eps, a, b, eta, z)<br>    sigma, direction = coefficients(a, b, eta)<br>    exact_var = direction**2 + sigma**2<br>    assert abs(exact_var-(1-b)) &lt; 1e-12<br>    marginals.append((eta, xs))<br>    print(f&quot;eta={eta:.1f} | mean={xs.mean():.6f} | variance={xs.var():.6f}&quot;)<br>print(f&quot;target mean={np.sqrt(b)*x0:.6f} | variance={1-b:.6f}&quot;)</pre><p><strong>Step 2. Check one-step sensitivity and the adjacent DDPM identity.</strong> The finite difference holds the input state fixed. The DDPM comparison uses zero fresh noise so the means can be compared directly.</p><pre># Hold x, time, eta and z fixed. Perturb only the predicted noise.<br>x, e, h = 0.9, 0.6, 1e-6<br>clean = (x-np.sqrt(1-a)*e)/np.sqrt(a)<br>for eta in (0, 1):<br>    sigma, direction = coefficients(a, b, eta)<br>    gain = direction-np.sqrt(b/a)*np.sqrt(1-a)<br>    fd = (update(x, e+h, a, b, eta)-update(x, e-h, a, b, eta))/(2*h)<br>    assert abs(fd-gain) &lt; 1e-8<br>    print(f&quot;eta={eta} | clean={clean:.6f} | next={update(x,e,a,b,eta):.6f} | gain={gain:.6f}&quot;)<br>T = 200<br>beta = np.linspace(0.0001, 0.05, T)<br>A = np.r_[1., np.cumprod(1-beta)]<br>t, s = 100, 99<br>alpha = A[t]/A[s]<br>ddpm_mean = (xt-(1-alpha)*eps/np.sqrt(1-A[t]))/np.sqrt(alpha)<br>error = np.max(np.abs(update(xt,eps,A[t],A[s],eta=1)-ddpm_mean))<br>assert error &lt; 1e-12<br>print(f&quot;adjacent eta=1 DDPM mean max error={error:.2e}&quot;)</pre><p><strong>Step 3. Compare finite deterministic schedules.</strong> This predictor is the exact conditional mean for the specified mixture. It isn’t an oracle for each example’s hidden noise. All schedules start from the same 4096 vectors.</p><pre># The exact conditional-mean predictor for a four-Gaussian toy.<br>centres = np.array([[-1.,-1.],[-1.,1.],[1.,-1.],[1.,1.]])<br>v0 = 0.12**2<br>initial = rng.normal(size=(4096,2))<br>def predictor(x, t):<br>    a = A[t]<br>    v = a*v0 + 1-a<br>    delta = x[:,None,:]-np.sqrt(a)*centres[None,:,:]<br>    logits = -(delta**2).sum(axis=2)/(2*v)<br>    w = np.exp(logits-logits.max(axis=1,keepdims=True))<br>    w /= w.sum(axis=1,keepdims=True)<br>    return np.sqrt(1-a)*(x-np.sqrt(a)*(w@centres))/v<br>def sample(calls):<br>    times = np.rint(np.linspace(T,1,calls)).astype(int)<br>    assert len(np.unique(times)) == calls<br>    x = initial.copy()<br>    for t,s in zip(times,np.r_[times[1:],0]):<br>        x = update(x,predictor(x,int(t)),A[t],A[s])<br>    return x<br>results = {k:sample(k) for k in (200,20,5)}<br>assert np.array_equal(sample(20),sample(20))<br>print(&quot;same initial noise + same eta=0 schedule: identical=True&quot;)<br>print(f&quot;terminal A={A[-1]:.6f} | samples={len(initial)}&quot;)<br>for calls,x in results.items():<br>    d2 = ((x[:,None,:]-centres[None,:,:])**2).sum(axis=2)<br>    counts = np.bincount(d2.argmin(axis=1),minlength=4)<br>    print(f&quot;calls={calls:3} | centre MSE={d2.min(axis=1).mean():.6f} | counts={counts.tolist()}&quot;)<br>print(f&quot;200 vs 20 paired max difference={np.abs(results[200]-results[20]).max():.6f}&quot;)<br>fig, axes = plt.subplots(1,3,figsize=(12,3.9),sharex=True,sharey=True)<br>for ax,(calls,x) in zip(axes,results.items()):<br>    ax.scatter(x[:,0],x[:,1],s=2,alpha=.35,c=&quot;#087F8C&quot;)<br>    ax.scatter(centres[:,0],centres[:,1],marker=&quot;x&quot;,c=&quot;#C66B2B&quot;,s=60)<br>    ax.set(title=f&quot;{calls} evaluations&quot;,xlabel=&quot;x coordinate&quot;,xlim=(-2.4,2.4),ylim=(-2.4,2.4))<br>    ax.set_aspect(&quot;equal&quot;)<br>axes[0].set_ylabel(&quot;y coordinate&quot;)<br>fig.suptitle(&quot;Inside AI · Deep Learning Models · Part 10\nSame initial noise, same predictor, different finite schedules&quot;)<br>fig.tight_layout()<br>fig.savefig(&quot;ddim-foundations-output.png&quot;,dpi=180)<br>print(&quot;Saved ddim-foundations-output.png | measured toy, no image benchmark&quot;)</pre><p><strong>Actual output, 11 October 2026.</strong> CPU run in this workspace. No neural training. The empirical moments fluctuate with the seed and sample count; the algebraic assertions use stated tolerances.</p><pre>Python 3.11.5 | NumPy 1.23.5 | CPU | seed 31<br>known x0=1.2 | conditional draws=100000 | no training<br>eta=0.0 | mean=0.957714 | variance=0.361039<br>eta=0.5 | mean=0.956966 | variance=0.361523<br>eta=1.0 | mean=0.957024 | variance=0.361133<br>target mean=0.960000 | variance=0.360000<br>eta=0 | clean=0.760770 | next=0.968616 | gain=-0.785641<br>eta=1 | clean=0.760770 | next=0.764500 | gain=-1.125833<br>adjacent eta=1 DDPM mean max error=8.88e-16<br>same initial noise + same eta=0 schedule: identical=True<br>terminal A=0.006122 | samples=4096<br>calls=200 | centre MSE=0.027672 | counts=[1072, 1031, 999, 994]<br>calls= 20 | centre MSE=0.020672 | counts=[1072, 1031, 999, 994]<br>calls=  5 | centre MSE=0.229895 | counts=[1072, 1031, 999, 994]<br>200 vs 20 paired max difference=0.446624<br>Saved ddim-foundations-output.png | measured toy, no image benchmark</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhSLZKrCePhDgFUvt014xw.png" /></figure><p><em>Measured output from Step 3: same initial noise and analytic predictor, 200, 20 and 5 evaluations. Orange crosses mark the four mixture centres.</em></p><p>The fixed 20-step run repeats exactly here. Its output differs from the 200-step run. The conditional marginal checks stay close to their target moments and the two algebraic checks pass their tolerances. These results cover this implementation and toy distribution. They aren’t image-quality scores, a speed benchmark or proof of exact finite-step sampling. The terminal mixture is only approximately standard Gaussian because A_T is small but nonzero.</p><p>For the sampler comparison that led here, read <a href="https://x.com/Skywalkerlyzv/article/2108911108371501112">Part 09 on X</a>, <a href="https://medium.com/@thesomilsinghofficial/how-ddim-takes-fewer-steps-through-the-same-noise-prediction-problem-0ed236bc649c">Medium</a> or <a href="https://thesomilsingh.substack.com/p/how-ddim-takes-fewer-steps-through">Substack</a>.</p><p>Next up: what changes when generation is learned as a velocity field?</p><p>Follow Inside AI for the next model pair. Full equations, runnable checks and the bits that break.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb04e7d2c260" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How DDIM takes fewer steps through the same noise prediction problem]]></title>
            <link>https://medium.com/@thesomilsinghofficial/how-ddim-takes-fewer-steps-through-the-same-noise-prediction-problem-0ed236bc649c?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ed236bc649c</guid>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 13:34:32 GMT</pubDate>
            <atom:updated>2026-10-10T13:34:32.038Z</atom:updated>
            <content:encoded><![CDATA[<p>What happens to the samples when 200 reverse evaluations become 20, 5 or just 1?</p><p>Somil Singh · Inside AI</p><p><strong>Deep Learning Models · Part 09</strong></p><p>Part 08 gave the noise prediction a job: it determines the reverse mean. Now there’s another choice to make. How often do we need to ask the predictor for a new answer?</p><p>Let’s hold that predictor fixed and change the sampling route. We’ll use a four cluster toy where the best noise estimate can be calculated exactly. That lets us inspect the sampler without mixing in a network’s training error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sqrQmC87_ywv2e1bTSkPjQ.png" /></figure><p><em>Schematic. The shorter route uses coefficients calculated for the selected pair of times. It doesn’t reuse an adjacent step coefficient for a larger jump.</em></p><h3>1. The time tells us how noisy the point is</h3><p>Our clean points sit around four centres: (−1, −1), (−1, 1), (1, −1) and (1, 1). Each coordinate has a within cluster standard deviation of 0.12.</p><p>At time t, write the noisy point as:</p><p><strong>xₜ = √Aₜ x₀ + √(1 − Aₜ) ε</strong></p><p>Here Aₜ is the product of the one step α values up to t. It’s the retained signal fraction. At the clean endpoint A₀ = 1. The code uses a 200 step linear beta schedule and gives A₂₀₀ ≈ 0.006122.</p><p>This marginal description tells us how to make a training pair at any chosen time. It doesn’t by itself force generation to visit every time.</p><h3>2. Keep the predictor, choose the route</h3><p>DDIM stands for Denoising Diffusion Implicit Models. Song, Meng and Ermon describe a family of sampling processes that can reuse a DDPM noise predictor. Their construction keeps the noisy marginals used by the training objective while changing the process between them. See <a href="https://arxiv.org/html/2010.02502v2">the original paper, sections 4.1 and 4.2</a>.</p><p>The model still gets the current point and its actual time. If the route jumps from 200 to 190, the next evaluation uses 190. Passing the loop counter would ask the predictor the wrong question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*afY4O47mKtRm73n1ah7m9w.png" /></figure><p><em>Schematic. Aₜ is cumulative. Our runnable example uses the exact conditional noise mean for its specified mixture, replacing a learned predictor only for this toy.</em></p><h3>3. The noise prediction objective stays familiar</h3><p>The prediction task is the same squared error job:</p><p><strong>E[‖ε − εθ(xₜ, t)‖²]</strong></p><p>For a fixed noisy point and time, the best squared error answer is the conditional mean of the noise. It isn’t necessarily the particular noise draw that produced the point.</p><p>In this toy, each noisy component is Gaussian with variance vₜ = Aₜ × 0.0144 + 1 − Aₜ. The distances to its four noisy centres give posterior component weights. If m is their weighted clean centre, the exact noise mean is:</p><p><strong>ε̂(x, t) = √(1 − Aₜ) × (x − √Aₜ m) / vₜ</strong></p><p>We’ll use that calculation for every route. No network is trained in this experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y94bQ_2QGHJJL7iKfO4J4g.png" /></figure><p><em>Schematic. This article changes the sampling job. A trained network would still need to learn useful noise predictions over its noise range.</em></p><h3>4. Build the jump from its two endpoints</h3><p>First convert the current noise estimate into a clean estimate:</p><p><strong>x̂₀ = (xₜ − √(1 − Aₜ) ε̂) / √Aₜ</strong></p><p>Then choose an earlier time s and combine three terms:</p><p><strong>xₛ = √Aₛ x̂₀ + √(1 − Aₛ − σ²) ε̂ + σz</strong></p><p>The fresh draw z is standard Gaussian. For the usual interpolation:</p><p><strong>σ² = η² × (1 − Aₛ)/(1 − Aₜ) × (1 − Aₜ/Aₛ)</strong></p><p>At η = 0, the fresh noise term disappears. The initial draw still varies across samples. Holding that draw, predictor and route fixed makes the sampler deterministic. It doesn’t make all generated points equal or every route produce the same points.</p><p>At s = 0, Aₛ = 1. Both noise coefficients vanish and the last step returns x̂₀. The code follows this endpoint directly. The <a href="https://github.com/ermongroup/ddim/blob/main/functions/denoising.py">authors’ sampler implementation</a> uses the same clean estimate, variance and direction structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2z3vInI80TBf0J_ajqYJHg.png" /></figure><p><em>Schematic of the generalized jump. The code checks its adjacent η = 1 mean against the DDPM posterior mean, using zero added noise for that identity check.</em></p><h3>5. Training and generation take different routes</h3><p>A learned version draws clean points, times and noise during training. It learns to predict the noise from those noisy pairs. Generation starts with a noise draw and repeatedly asks for a new prediction as the state changes.</p><p>Our experiment skips learning because the distribution is specified. We compare 200, 50, 20, 10, 5, 2 and 1 evaluations of that same exact predictor. Every run starts with the same 4,096 noise draws. Every route ends at time zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2b05h4jqzMGl03djLL5WUg.png" /></figure><p><em>Schematic. Retained times are selected backwards. Predictions must be recalculated at the current state rather than cached from the first point.</em></p><h3>6. Four visible clusters aren’t the whole result</h3><p>Nearest centre counts tell us whether a quadrant gets neglected. They miss how much spread remains around those centres.</p><p>For this target, the expected squared distance to its generating centre is 2 × 0.0144 = 0.0288. Nearest centre assignment gives almost the same value because the clusters are well separated. A value below that isn’t automatically better. It can mean the samples are too concentrated.</p><p>The one call route produces points close to the middle. Quadrant counts still look balanced because tiny positive and negative coordinates get assigned to different centres. That count alone hides a serious distribution failure.</p><p><strong>Keep these three ideas</strong></p><ol><li>The predictor’s job and the sampler’s route are separate choices. Keep the actual noise level attached to every evaluation.</li><li>A DDIM jump uses both endpoint signal fractions. A shorter route needs the corresponding coefficients.</li><li>Fewer calls can change spread even when every cluster gets a similar count. Measure the distribution you wanted.</li></ol><h3>7. Run the sampler comparison</h3><p>Use Python with NumPy and Matplotlib: python -m pip install numpy matplotlib. Put the following three steps in one script and run it. Step 2 depends on Step 1 and Step 3 depends on both.</p><p><strong>Step 1. Define the mixture and its exact noise predictor.</strong> The stable weights avoid underflow from large negative logits. Normal starting noise approximates the terminal noisy marginal; A₂₀₀ is small but not zero.</p><pre>import numpy as np<br>import matplotlib<br>matplotlib.use(&quot;Agg&quot;)<br>import matplotlib.pyplot as plt<br>import platform, time<br>rng = np.random.default_rng(23)<br>T = 200<br>beta = np.linspace(0.0001, 0.05, T)<br>A = np.r_[1.0, np.cumprod(1 - beta)]<br>centres = np.array([[-1., -1.], [-1., 1.], [1., -1.], [1., 1.]])<br>v0 = 0.12 ** 2<br>def epsilon_exact(x, t):<br>    a = A[t]<br>    v = a * v0 + 1 - a<br>    delta = x[:, None, :] - np.sqrt(a) * centres[None, :, :]<br>    logits = -(delta ** 2).sum(axis=2) / (2 * v)<br>    weights = np.exp(logits - logits.max(axis=1, keepdims=True))<br>    weights /= weights.sum(axis=1, keepdims=True)<br>    mean_centre = weights @ centres<br>    return np.sqrt(1 - a) * (x - np.sqrt(a) * mean_centre) / v<br>def real(n):<br>    return centres[rng.integers(4, size=n)] + np.sqrt(v0) * rng.normal(size=(n, 2))<br>initial = rng.normal(size=(4096, 2))<br>reference = real(len(initial))<br>print(f&quot;Python {platform.python_version()} | NumPy {np.__version__} | CPU | seed 23&quot;)<br>print(f&quot;T={T} | terminal A={A[-1]:.6f} | samples={len(initial)} | no training&quot;)</pre><p><strong>Step 2. Implement the jump and check two properties.</strong> An adjacent η = 1 update should match the DDPM mean. A fixed η = 0 run should repeat exactly. These checks verify implementation identities, not sample quality.</p><pre>def ddim_step(x, t, s, eta=0.0, noise=None):<br>    assert 0 &lt;= s &lt; t &lt;= T and 0 &lt;= eta &lt;= 1<br>    at, previous = A[t], A[s]<br>    eps = epsilon_exact(x, t)<br>    clean = (x - np.sqrt(1 - at) * eps) / np.sqrt(at)<br>    variance = eta ** 2 * (1 - previous) / (1 - at) * (1 - at / previous)<br>    direction2 = 1 - previous - variance<br>    assert variance &gt;= -1e-12 and direction2 &gt;= -1e-12<br>    result = np.sqrt(previous) * clean + np.sqrt(max(direction2, 0)) * eps<br>    if variance &gt; 0:<br>        assert noise is not None<br>        result = result + np.sqrt(variance) * noise<br>    return result<br>def sample(calls):<br>    assert 1 &lt;= calls &lt;= T<br>    ts = np.rint(np.linspace(T, 1, calls)).astype(int)<br>    assert len(np.unique(ts)) == calls<br>    x = initial.copy()<br>    for t, s in zip(ts, np.r_[ts[1:], 0]):<br>        x = ddim_step(x, int(t), int(s))<br>    return x<br># An adjacent eta=1 update must recover the DDPM posterior mean.<br>t, s = 100, 99<br>x = initial[:16]<br>eps = epsilon_exact(x, t)<br>alpha = A[t] / A[s]<br>mean_ddpm = (x - (1 - alpha) * eps / np.sqrt(1 - A[t])) / np.sqrt(alpha)<br>identity_error = np.max(np.abs(ddim_step(x, t, s, eta=1, noise=np.zeros_like(x)) - mean_ddpm))<br>assert identity_error &lt; 1e-12<br>assert np.array_equal(sample(20), sample(20))<br>print(f&quot;adjacent eta=1 mean error={identity_error:.2e} | eta=0 repeat identical=True&quot;)</pre><p><strong>Step 3. Compare routes and draw their actual outputs.</strong> Report centre distances alongside counts. Timings include the toy predictor and update, are single CPU runs and aren’t a hardware benchmark.</p><pre>def diagnostics(x):<br>    d2 = ((x[:, None, :] - centres[None, :, :]) ** 2).sum(axis=2)<br>    assigned = d2.argmin(axis=1)<br>    counts = np.bincount(assigned, minlength=4)<br>    mse = d2.min(axis=1).mean()<br>    return counts, mse<br>counts, ref_mse = diagnostics(reference)<br>print(f&quot;reference centre MSE={ref_mse:.6f} | expected near {2*v0:.4f}&quot;)<br>samples = {}<br>for calls in (200, 50, 20, 10, 5, 2, 1):<br>    start = time.perf_counter()<br>    samples[calls] = sample(calls)<br>    elapsed = time.perf_counter() - start<br>    counts, mse = diagnostics(samples[calls])<br>    print(f&quot;calls={calls:3d} | centre MSE={mse:.6f} | counts={counts.tolist()} | seconds={elapsed:.3f}&quot;)<br>fig, axes = plt.subplots(1, 4, figsize=(15, 4.3), constrained_layout=True)<br>for ax, label, points in zip(axes, (&quot;Target draws&quot;, &quot;DDIM 200 calls&quot;, &quot;DDIM 20 calls&quot;, &quot;DDIM 1 call&quot;), (reference, samples[200], samples[20], samples[1])):<br>    ax.scatter(points[:1000, 0], points[:1000, 1], s=3, alpha=.38, color=&quot;#087F8C&quot;)<br>    ax.scatter(centres[:, 0], centres[:, 1], s=35, marker=&quot;x&quot;, color=&quot;#C66B2B&quot;)<br>    ax.set(xlim=(-2, 2), ylim=(-2, 2), xlabel=&quot;x&quot;, ylabel=&quot;y&quot;, title=label)<br>    ax.set_aspect(&quot;equal&quot;)<br>fig.suptitle(&quot;Inside AI · Deep Learning Models · Part 09 | Actual CPU samples, exact noise predictor&quot;, color=&quot;#153449&quot;)<br>fig.savefig(&quot;ddim-output.png&quot;, dpi=160)<br>fig.savefig(&quot;ddim-output.svg&quot;)<br>print(&quot;Saved ddim-output.png | toy sampler comparison, no image benchmark&quot;)</pre><p><strong>Actual output, 10 October 2026</strong></p><pre>Python 3.11.5 | NumPy 1.23.5 | CPU | seed 23<br>T=200 | terminal A=0.006122 | samples=4096 | no training<br>adjacent eta=1 mean error=4.44e-16 | eta=0 repeat identical=True<br>reference centre MSE=0.028815 | expected near 0.0288<br>calls=200 | centre MSE=0.027681 | counts=[1028, 1021, 1000, 1047] | seconds=0.156<br>calls= 50 | centre MSE=0.024686 | counts=[1028, 1021, 1000, 1047] | seconds=0.038<br>calls= 20 | centre MSE=0.020822 | counts=[1028, 1021, 1000, 1047] | seconds=0.016<br>calls= 10 | centre MSE=0.030669 | counts=[1028, 1021, 1000, 1047] | seconds=0.008<br>calls=  5 | centre MSE=0.223984 | counts=[1028, 1021, 1000, 1047] | seconds=0.004<br>calls=  2 | centre MSE=1.707935 | counts=[1028, 1021, 1000, 1047] | seconds=0.002<br>calls=  1 | centre MSE=1.757858 | counts=[1028, 1021, 1000, 1047] | seconds=0.001<br>Saved ddim-output.png | toy sampler comparison, no image benchmark</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ox3fcylY2kEtdjKCyWHGw.png" /></figure><p><em>Actual CPU output. Each panel shows the first 1,000 of 4,096 points. Crosses mark the prescribed centres. All DDIM panels use the same initial draws and exact conditional noise predictor with η = 0.</em></p><p>The 200 call route gives a centre MSE of 0.027681, close to the target draw’s 0.028815. The 20 call route gives 0.020822, which is smaller because its spread is too narrow. With 5 calls the value jumps to 0.223984. With one call it’s 1.757858.</p><p>All routes have the same quadrant counts here. The symmetric toy and fixed initial draws preserve those assignments while point locations change. That does not imply stable semantic content for other data.</p><p><strong>Limits and the next question</strong></p><p>This is an original sampler experiment on an analytic two dimensional mixture. We haven’t trained a new network, rerun Part 07’s learned predictor, measured image quality or tested another seed. Starting with normal noise rather than the exact finite time marginal adds a small initial approximation. The centre statistic isn’t a general image quality metric. Smaller isn’t inherently better.</p><p>The sampler can use fewer evaluations, but the route still needs checking. Part 10 will examine why the noisy marginals let us change that route and what the deterministic update approximates.</p><p>For the reverse mean argument, see <a href="https://x.com/Skywalkerlyzv/article/2108560064126841315">Part 08 on X</a>, <a href="https://medium.com/@thesomilsinghofficial/why-predicting-noise-gives-a-diffusion-model-a-reverse-step-87b84ff625b0">Medium</a> or <a href="https://thesomilsingh.substack.com/p/why-predicting-noise-gives-a-diffusion">Substack</a>.</p><p>Follow Inside AI for the companion. We’ll keep the assumptions beside the code and results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ed236bc649c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why predicting noise gives a diffusion model a reverse step]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@thesomilsinghofficial/why-predicting-noise-gives-a-diffusion-model-a-reverse-step-87b84ff625b0?source=rss-959694b93e8a------2"><img src="https://cdn-images-1.medium.com/max/600/1*80xlIZ8os_LJ5quKYNc0TA.png" width="600"></a></p><p class="medium-feed-snippet">The model never sees the clean point during generation. So where does its next step come from?</p><p class="medium-feed-link"><a href="https://medium.com/@thesomilsinghofficial/why-predicting-noise-gives-a-diffusion-model-a-reverse-step-87b84ff625b0?source=rss-959694b93e8a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@thesomilsinghofficial/why-predicting-noise-gives-a-diffusion-model-a-reverse-step-87b84ff625b0?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/87b84ff625b0</guid>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 14:08:13 GMT</pubDate>
            <atom:updated>2026-10-09T14:08:13.214Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How a diffusion model learns to turn noise into four clusters]]></title>
            <link>https://medium.com/@thesomilsinghofficial/how-a-diffusion-model-learns-to-turn-noise-into-four-clusters-4d3492034acd?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d3492034acd</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 14:41:07 GMT</pubDate>
            <atom:updated>2026-10-08T14:41:07.858Z</atom:updated>
            <content:encoded><![CDATA[<p>If training starts with real data, how can generation start with noise?</p><p>Somil Singh · Inside AI</p><p><strong>Deep Learning Models · Part 07</strong></p><p>Take four little clouds of points. Add enough noise and the clouds blend together. The original structure becomes hard to see.</p><p>Now start with a fresh cloud of noise. How do we get four clusters back when there’s no original point to recover?</p><p>Lemme walk through the missing piece: a model learns a noise estimate at many noise levels. A sampler uses that estimate repeatedly to move toward the learned data distribution. We’ll train a small example, generate 4,096 points and check where they land.</p><figure><img alt="Known data plus deliberate noise gives a supervised noise target" src="https://cdn-images-1.medium.com/proxy/0*6TEzP4EAazFsnipa.png" /></figure><p><em>Concept hero. The training target is known noise. This is a schematic, not generated output.</em></p><h4>1. The noisy point is still the same kind of object</h4><p>A GAN maps a hidden code into a sample. Here, every state x_t has the same number of coordinates as the data. For an image those coordinates could be pixels. For our example they’re just two numbers.</p><p>x_0 means a clean data point. x_t means the point at noise level t. A <strong>denoising diffusion probabilistic model</strong>, or DDPM, learns a sequence of reverse distributions that can produce clean samples from a simple starting distribution.</p><p>The forward process is fixed in this example. It scales the current point slightly and adds independent Gaussian noise. The learned reverse process takes a noisy point and its timestep, then predicts the mean of the next, less noisy state.</p><p>That timestep matters. The same coordinate might be almost clean at one level and mostly noise at another.</p><h4>2. There are two directions with different jobs</h4><p>The forward transition is:</p><p><strong>x_t = sqrt(alpha_t) x_(t−1) + sqrt(beta_t) z</strong>, with z drawn from N(0,I).</p><p>beta_t is the variance of noise added at that step. alpha_t = 1 − beta_t is the amount of variance retained from the previous state. In multiple dimensions, I means we use independent noise with unit variance in each coordinate before scaling it.</p><figure><img alt="Four forward noise levels progressively obscure four point clusters" src="https://cdn-images-1.medium.com/proxy/0*LC5NpO-CPCpr5doG.png" /></figure><p><em>Simulated forward corruption using the published toy schedule and seed 29. The same base noise is used across panels for comparison; these are marginal samples, not one sampled Markov trajectory.</em></p><p>The reverse model doesn’t receive the original x_0 at generation time. It receives x_t and t. Its noise prediction controls a reverse Gaussian mean, while the sampler adds the chosen reverse noise variance except at the final step.</p><p>Think of this as learning what a plausible next state looks like. It isn’t undoing a saved random seed or restoring one uniquely determined original.</p><h4>3. Training gets a target without a human label</h4><p>We create a noisy point ourselves, so we know exactly which noise we added. The network epsilon_theta(x_t,t) tries to predict that noise.</p><p>The simplified training objective averages the squared difference between known noise epsilon and the network’s prediction over clean data, timesteps and random noise.</p><figure><img alt="Simplified noise prediction objective with target, model and error roles" src="https://cdn-images-1.medium.com/proxy/0*PaGJ8IPKDYTH6Tqp.png" /></figure><p><em>Objective schematic. The code averages over both the batch and the two coordinates.</em></p><p>The target is epsilon. theta means the learned parameters. The model must use both the noisy point and timestep to reduce the error. We sample timesteps uniformly in this toy run.</p><p>This is the simplified noise prediction objective. It is not the full variational likelihood bound with every weighting and endpoint term preserved. <a href="https://arxiv.org/html/2006.11239v2">Ho, Jain and Abbeel’s DDPM paper</a> gives the original formulation and connects the parameterization to the reverse process.</p><p>A low average training error still doesn’t prove the sampler works. Some noise levels are easier than others, and generation feeds the network its own previous outputs.</p><h4>4. The shortcut lets us train without walking the whole chain</h4><p>Define alpha_bar_t as the product alpha_1 × alpha_2 × … × alpha_t. Then we can draw a noisy point directly:</p><p><strong>x_t = sqrt(alpha_bar_t) x_0 + sqrt(1 − alpha_bar_t) epsilon.</strong></p><p>Instead of applying 150 forward steps to train on timestep 150, we use one clean point, one Gaussian draw and that cumulative coefficient.</p><figure><img alt="Numeric direct noising example with cumulative alpha 0.64" src="https://cdn-images-1.medium.com/proxy/0*-YD8lML_wAyjnvSa.png" /></figure><p><em>One coordinate: x_0 = 1, alpha_bar_t = 0.64 and epsilon = −0.5 give x_t = 0.5.</em></p><p>The square roots scale amplitudes. Mixing up alpha_t and alpha_bar_t changes the noise level. Mixing variance with standard deviation changes it again.</p><p>This shortcut supplies the correct marginal distribution at the selected timestep. It doesn’t make sampling a one step operation. Our reverse sampler still visits every timestep.</p><h4>5. One training update and one generated sample take different routes</h4><p>Training samples a real point, a timestep and noise. It creates x_t directly, predicts the known noise and updates the network.</p><p>Generation starts with x_T drawn from N(0,I). At each timestep the sampler uses:</p><p><strong>mean_t = (x_t − beta_t / sqrt(1 − alpha_bar_t) × epsilon_theta(x_t,t)) / sqrt(alpha_t).</strong></p><p>Then it draws the next state around this mean. Here the variance is beta_t × (1 − alpha_bar_(t−1)) / (1 − alpha_bar_t). Set alpha_bar_0 = 1. At the last step, the code adds no noise.</p><figure><img alt="Training uses one selected noise level while generation traverses all 200 steps" src="https://cdn-images-1.medium.com/proxy/0*GVx1Z_cjn6QeWCyo.png" /></figure><p><em>Two routes through the same noise predictor. Training has a target; generation has repeated reverse updates.</em></p><p>Our schedule uses 200 steps with beta increasing from 0.0001 to 0.05. Terminal alpha_bar is about 0.006122, so a small amount of data signal remains. Starting at a standard Gaussian is an approximation. These are toy settings, not the original paper’s image training configuration.</p><p>The Python arrays use indices 0 through 199. Index 0 corresponds to mathematical timestep 1. Keeping that mapping explicit prevents an easy off by one error.</p><h4>6. A falling loss can hide a weak noise level</h4><p><strong>Noise estimates vary by timestep.</strong> In the fresh run below, the earliest timestep’s held-out noise MSE is above 1, while the last timestep’s is much smaller. The scalar training loss hides that difference. A zero noise prediction would have expected MSE 1 under our per-coordinate reduction.</p><p><strong>Four regions can still have the wrong proportions.</strong> Count the nearest centre assignments, then inspect distances and plots. Every point gets assigned to a centre even if it is nowhere near one, so assignment counts alone are weak evidence.</p><p><strong>A terminal schedule can leave too much signal.</strong> Check alpha_bar_T before treating the terminal forward distribution as standard Gaussian. Data scaling, the schedule, network capacity and reverse variance all affect the result.</p><p><strong>Changing the sampler changes the experiment.</strong> Removing all reverse noise or skipping timesteps is a different sampling procedure. Don’t assume this trained predictor automatically makes every faster sampler correct.</p><h4>Keep these three ideas</h4><ol><li><strong>The noise is a label we create.</strong> Known corruption gives a supervised learning target without an external annotation.</li><li><strong>The timestep tells the model what noise level it sees.</strong> One direct forward draw can train a selected timestep.</li><li><strong>Generation is repeated learned reverse sampling.</strong> It creates a plausible new sample, not a guaranteed recovery of a specific original.</li></ol><h4>7. A tiny diffusion model you can actually run</h4><p>Install NumPy and Matplotlib with pip install numpy matplotlib. Save the three blocks in order as one script and run it with Python. No dataset download, pretrained weights or GPU is needed.</p><p>We use a two hidden layer tanh network with width 64. Its input is the two noisy coordinates plus eight sine/cosine timestep features. Gradients and Adam updates are written in NumPy so every update stays visible. This is an educational implementation, not an efficient production framework.</p><p><strong>Step 1. Build the data, schedule and noise predictor.</strong> The real distribution is an equal mixture centred at (−1,−1), (−1,1), (1,−1) and (1,1), with independent coordinate standard deviation 0.12. The initial model doesn’t know those centres.</p><pre>import numpy as np<br>import matplotlib<br>matplotlib.use(&quot;Agg&quot;)<br>import matplotlib.pyplot as plt<br>import platform<br>rng = np.random.default_rng(7)<br>T = 200<br>beta = np.linspace(0.0001, 0.05, T)<br>alpha = 1 - beta<br>abar = np.cumprod(alpha)<br>previous = np.r_[1.0, abar[:-1]]<br>posterior_var = beta * (1 - previous) / (1 - abar)<br>centres = np.array([[-1., -1.], [-1., 1.], [1., -1.], [1., 1.]])<br>def real(n):<br>    return centres[rng.integers(4, size=n)] + 0.12 * rng.normal(size=(n, 2))<br>def features(x, t):<br>    angle = (t[:, None] + 1) / T * np.array([1, 2, 4, 8]) * np.pi<br>    return np.c_[x, np.sin(angle), np.cos(angle)]<br>sizes = [10, 64, 64, 2]<br>params = []<br>for a, b in zip(sizes[:-1], sizes[1:]):<br>    params.extend([rng.normal(size=(a, b)) * np.sqrt(2 / (a + b)), np.zeros(b)])<br>moment = [np.zeros_like(p) for p in params]<br>velocity = [np.zeros_like(p) for p in params]<br>def predict(x, t, cache=False):<br>    f = features(x, t)<br>    h1 = np.tanh(f @ params[0] + params[1])<br>    h2 = np.tanh(h1 @ params[2] + params[3])<br>    out = h2 @ params[4] + params[5]<br>    return (out, (f, h1, h2)) if cache else out<br>print(f&quot;Python {platform.python_version()} | NumPy {np.__version__} | CPU | seed 7&quot;)<br>print(f&quot;T={T} | terminal alpha_bar={abar[-1]:.6f}&quot;)</pre><p><strong>Step 2. Train on random noise levels.</strong> Each update uses 256 new synthetic points. We run 6,000 updates at learning rate 0.001. The manual backward pass differentiates the mean squared error through both tanh layers. After training, fresh draws test four fixed noise levels.</p><pre># Run immediately after Step 1 in the same script or notebook.<br>for step in range(1, 6001):<br>    x0 = real(256)<br>    t = rng.integers(T, size=len(x0))<br>    eps = rng.normal(size=x0.shape)<br>    a = abar[t, None]<br>    xt = np.sqrt(a) * x0 + np.sqrt(1 - a) * eps<br>    estimate, (f, h1, h2) = predict(xt, t, cache=True)<br>    error = estimate - eps<br>    # Mean over BOTH samples and coordinates.<br>    d = 2 * error / error.size<br>    dw3, db3 = h2.T @ d, d.sum(axis=0)<br>    d2 = (d @ params[4].T) * (1 - h2 * h2)<br>    dw2, db2 = h1.T @ d2, d2.sum(axis=0)<br>    d1 = (d2 @ params[2].T) * (1 - h1 * h1)<br>    dw1, db1 = f.T @ d1, d1.sum(axis=0)<br>    gradients = [dw1, db1, dw2, db2, dw3, db3]<br>    for i, gradient in enumerate(gradients):<br>        moment[i] = 0.9 * moment[i] + 0.1 * gradient<br>        velocity[i] = 0.999 * velocity[i] + 0.001 * gradient ** 2<br>        m = moment[i] / (1 - 0.9 ** step)<br>        v = velocity[i] / (1 - 0.999 ** step)<br>        params[i] -= 0.001 * m / (np.sqrt(v) + 1e-8)<br>    if step in (1, 6000):<br>        print(f&quot;step {step:4d} | sampled training MSE {np.mean(error ** 2):.6f}&quot;)<br># Fresh synthetic draws, not the final training batch.<br>checks = real(4096)<br>for fixed_t in (0, 49, 99, 199):<br>    eps = rng.normal(size=checks.shape)<br>    xt = np.sqrt(abar[fixed_t]) * checks + np.sqrt(1 - abar[fixed_t]) * eps<br>    predicted = predict(xt, np.full(len(xt), fixed_t))<br>    mse = np.mean((predicted - eps) ** 2)<br>    print(f&quot;held-out timestep {fixed_t + 1:3d} | noise MSE {mse:.6f}&quot;)</pre><p><strong>Step 3. Generate without a clean input.</strong> Start 4,096 fresh Gaussian points, apply all 200 reverse steps and measure the result. The centres are used for evaluation only, never inside the reverse update. The script also saves a plot and raw sample arrays.</p><pre># Each network call predicts noise at the CURRENT timestep.<br>generated = rng.normal(size=(4096, 2))<br>for t in reversed(range(T)):<br>    eps_hat = predict(generated, np.full(len(generated), t))<br>    mean = (generated - beta[t] / np.sqrt(1 - abar[t]) * eps_hat) / np.sqrt(alpha[t])<br>    generated = mean<br>    if t &gt; 0:<br>        generated += np.sqrt(posterior_var[t]) * rng.normal(size=generated.shape)<br>distance = np.linalg.norm(generated[:, None, :] - centres[None, :, :], axis=2)<br>nearest = np.argmin(distance, axis=1)<br>print(&quot;nearest-centre counts:&quot;, np.bincount(nearest, minlength=4).tolist())<br>print(f&quot;fraction within radius 0.35: {np.mean(distance.min(axis=1) &lt; 0.35):.4f}&quot;)<br>print(&quot;generated shape:&quot;, generated.shape)<br>reference = real(4096)<br>fig, axes = plt.subplots(1, 2, figsize=(10, 4.5), sharex=True, sharey=True)<br>for ax, data, title in zip(axes, (reference, generated), (&quot;Fresh reference draws&quot;, &quot;Generated after 200 reverse steps&quot;)):<br>    ax.scatter(data[:, 0], data[:, 1], s=3, alpha=0.35)<br>    ax.set(title=title, xlabel=&quot;coordinate 1&quot;, ylabel=&quot;coordinate 2&quot;, xlim=(-2, 2), ylim=(-2, 2))<br>    ax.set_aspect(&quot;equal&quot;)<br>fig.suptitle(&quot;Inside AI · Part 07 · actual CPU run · seed 7&quot;)<br>fig.tight_layout()<br>fig.savefig(&quot;ddpm_output.png&quot;, dpi=180)<br>np.savez(&quot;ddpm_samples.npz&quot;, reference=reference, generated=generated, beta=beta, abar=abar)</pre><p><strong>Actual output, 8 October 2026.</strong> Fresh CPU training with Python 3.11.5, NumPy 1.23.5 and seed 7. Printed training losses use sampled batches before the update. Software and linear algebra differences can change results.</p><pre>Python 3.11.5 | NumPy 1.23.5 | CPU | seed 7<br>T=200 | terminal alpha_bar=0.006122<br>step    1 | sampled training MSE 0.928866<br>step 6000 | sampled training MSE 0.210487<br>held-out timestep   1 | noise MSE 1.094112<br>held-out timestep  50 | noise MSE 0.439607<br>held-out timestep 100 | noise MSE 0.288743<br>held-out timestep 200 | noise MSE 0.017920<br>nearest-centre counts: [1052, 935, 1102, 1007]<br>fraction within radius 0.35: 0.9753<br>generated shape: (4096, 2)</pre><figure><img alt="Reference and generated point distributions from the fresh CPU run" src="https://cdn-images-1.medium.com/proxy/0*OwbabAHJanzP2IlO.png" /></figure><p><em>Actual output. Left: fresh draws from the known data distribution. Right: points generated by the trained noise predictor and 200 step sampler. Identical axes, no rejected samples or selected best seed.</em></p><p>The plot recovers four visible regions. About 97.53% of generated points land within 0.35 of a centre, and the counts are not perfectly balanced. That radius is a chosen toy diagnostic, not an image quality score. It doesn’t establish exact distribution matching, image generation performance or robustness across seeds.</p><p>The held-out points are fresh samples from the same known synthetic distribution. They test this toy setup, not generalization to a new dataset. The earliest noise prediction result also gives us a real limitation to investigate rather than hiding it behind the final sample plot.</p><p>The operation to remember is <strong>predict noise at the current timestep, then apply the reverse update</strong>. A clean input supplies the training problem; it doesn’t need to be present when we generate.</p><p>Next, Part 08 will derive the reverse posterior and explain why noise prediction can parameterize its mean. It isn’t published yet. Part 07 is the working overview, and the next notes will make the coefficients earn their place.</p><p>If yall are building your foundations alongside me, keep the code and the checks together. Follow <a href="https://x.com/Skywalkerlyzv">Inside AI on X</a> for the next part, or read along on <a href="https://medium.com/@thesomilsinghofficial">Medium</a> and <a href="https://thesomilsingh.substack.com/">Substack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d3492034acd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why a GAN can reject a fake and still teach almost nothing]]></title>
            <link>https://medium.com/@thesomilsinghofficial/why-a-gan-can-reject-a-fake-and-still-teach-almost-nothing-5a991b0cf2e5?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a991b0cf2e5</guid>
            <category><![CDATA[gans]]></category>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 13:13:36 GMT</pubDate>
            <atom:updated>2026-10-07T13:13:36.334Z</atom:updated>
            <content:encoded><![CDATA[<p>Two losses want the same outcome. Why does one give a rejected fake almost no gradient?</p><p>Somil Singh · Inside AI</p><p><strong>Deep Learning Models · Part 06</strong></p><p>A discriminator rejects a generated sample with confidence. That sounds like a useful lesson for the generator. But the lesson can arrive as an almost flat gradient.</p><p>The missing detail is the generator objective. Let’s trace its local signal, derive what an ideal discriminator knows and check the numbers without training another GAN. Before reading on, try a prediction: can the generator learn while the discriminator’s weights stay fixed?</p><p><a href="https://thesomilsingh.substack.com/p/how-a-gan-learns-to-turn-noise-into">Part 05 explains the GAN training loop and its recorded samples</a>. These foundations unpack the losses behind that example.</p><figure><img alt="A rejected fake has two different local gradient strengths" src="https://cdn-images-1.medium.com/proxy/1*dHUQem2Z-zXJvU_QlfOkwA.png" /></figure><p><em>Figure 1. Schematic preview of the calculated logit derivatives. Neither number is a full network parameter gradient.</em></p><h4>1. A probability is not yet a learning signal</h4><p>Imagine a sample that the discriminator considers clearly generated. Let its raw score be the logit a and write D(x) = sigmoid(a(x)). Negative logits mean low estimated probability of being real. The generator maps noise z to a sample G(z).</p><p>The two distributions matter too: p_data describes real samples and p_g describes generated samples. A discriminator learns from their relative frequencies. It does not receive an independent certificate of beauty or usefulness.</p><p>For a fixed generator and equally weighted real and generated groups, the ideal classifier is D*(x) = p_data(x) / (p_data(x) + p_g(x)). To see why, look at a two outcome example.</p><h4>2. What the ideal discriminator knows</h4><p>Give real outcomes A and B equal mass: p_data = (0.5, 0.5). Let the generator favor A: p_g = (0.9, 0.1).</p><p>At A, generated samples are more common than real ones, so D* = 0.5 / 1.4 = 0.357143. At B they are scarce, so D* = 0.5 / 0.6 = 0.833333. The score reflects a frequency mismatch.</p><figure><img alt="Real and generated masses determine the ideal classifier at each outcome" src="https://cdn-images-1.medium.com/proxy/1*C7N-RHQ6socoN0cNlRBuBw.png" /></figure><p><em>Figure 2. Schematic finite distribution example. The masses are specified inputs, not learned estimates.</em></p><p>We obtain D* by maximizing p log D + q log(1-D) at each location. Its derivative is p/D — q/(1-D). Set that to zero and solve. The second derivative is negative when p and q are positive, so the interior solution is a maximum.</p><p>This needs densities or masses with respect to a common measure and an unrestricted discriminator. Where both masses vanish, the objective leaves D unconstrained. Where just one has mass, the optimum is at a boundary. Unequal class weighting would change the ratio.</p><h4>3. One game gives the generator two choices</h4><p>The original GAN game maximizes this value over D and minimizes it over G:</p><pre>V(D,G) = E_data[log D(x)] + E_g[log(1-D(x))]<br>min_G max_D V(D,G)</pre><pre>L_minimax(a) = log(1-sigmoid(a)) = -softplus(a)<br>L_non_saturating(a) = -log sigmoid(a) = softplus(-a)<br>softplus(a) = log(1 + exp(a))</pre><p>The discriminator rewards accepting real samples and rejecting generated ones. For the generator, both listed objectives encourage accepting a fake. The non saturating form changes the local gradient strength. Part 05 uses it.</p><figure><img alt="The discriminator game and the generator loss choices" src="https://cdn-images-1.medium.com/proxy/1*SOrLnDK5FP_RcFivy-5hkA.png" /></figure><p><em>Figure 3. Objective schematic. Expectations over real and generated groups have equal weight.</em></p><p>Binary cross entropy with fake target one implements the non saturating generator loss. A stable logit loss is useful for numerical reliability. It cannot repair every optimization failure.</p><p>Part 05 adds a mean over real examples to a mean over fake examples. A single mean over their concatenation is half that value when the batches are equal. Keep the reduction convention explicit before comparing losses.</p><h4>4. Follow the gradient all the way back</h4><p>Differentiate with respect to the fake logit:</p><pre>dL_minimax/da = -sigmoid(a)<br>dL_non_saturating/da = sigmoid(a) - 1</pre><p>At a = -8 the two values are -0.000335 and -0.999665. Gradient descent subtracts the derivative, so both try to increase a. The first gives an almost flat local signal to a confidently rejected fake. At a = 8 their strengths reverse. At a = 0 both equal -0.5.</p><figure><img alt="The local derivative multiplies two more derivatives before reaching generator parameters" src="https://cdn-images-1.medium.com/proxy/1*P8wS0DMkEUKHgECbhg0nGQ.png" /></figure><p><em>Figure 4. Chain rule schematic. Freezing discriminator weights during a generator update still permits differentiation through its input.</em></p><p>The full path is dL/dtheta_G = (dL/da)(da/dx)(dx/dtheta_G). A large first factor cannot guarantee that the other factors are useful. The answer to our opening prediction is yes: G can learn through D while D’s weights stay fixed. Detaching G(z) during that generator step would break the path. Detaching it for D’s separate update serves a different purpose.</p><h4>5. The ideal game and the actual training loop</h4><p>Let m = (p_data + p_g)/2. Substituting D* into the original game gives:</p><pre>JS(p_data || p_g) = 0.5 KL(p_data || m) + 0.5 KL(p_g || m)<br>V(D*,G) = -log 4 + 2 JS(p_data || p_g)</pre><p>For our discrete example, substitute D* = p/(p+q). Then log D* = log(p/m) — log 2 and log(1-D*) = log(q/m) — log 2. Sum with weights p and q. Each distribution sums to one, leaving the two KL terms and -2 log 2.</p><figure><img alt="Ideal density calculation and finite alternating optimization" src="https://cdn-images-1.medium.com/proxy/1*1XrbzcsXSSkwcjIB0hnNKg.png" /></figure><p><em>Figure 5. Two routes. The identity uses the optimal D and original minimax value. Finite training usually follows another generator loss.</em></p><p>Matching distributions make JS zero. With natural logarithms JS is at most log 2. This identity does not turn every non saturating generator update against a finite discriminator into an exact JS gradient step. The distinction is part of the result, not a detail to omit.</p><h4>6. Why a familiar loss value can fool us</h4><p>At the ideal matched distribution D* = 0.5. Under the sum of the two group means, the discriminator loss is 2 log 2 and the non saturating generator loss is log 2.</p><p>A finite network can produce similar values because it has failed to distinguish samples. So an uncertain discriminator is not proof of a good generator. Inspect diversity and coverage alongside the losses. Part 05’s cluster counts ask a narrower, more useful question: where did the generated samples land?</p><p>Oscillation also needs context. Each player’s objective changes with its opponent. Check samples over time and across seeds. If coverage falls while the loss looks comfortable, do not treat that scalar as a quality certificate.</p><p><strong>Keep these three ideas</strong></p><p>1. The ideal discriminator measures relative density under specified class weighting.<br>2. A rejected fake needs a useful gradient through the whole chain, not just the logit.<br>3. The ideal minimax identity explains an equilibrium. It does not certify a finite training run.</p><h4>7. Check the gradients and identity yourself</h4><p>This example needs only Python 3’s standard library. Save the three steps in one file and run it with `python example.py`. It uses fixed inputs, so no random seed is needed. No model is trained.</p><p><strong>Step 1. Compare analytic derivatives with central finite differences.</strong></p><p>The stable softplus form avoids computing a large exponential. The tolerance checks derivative agreement at three logits.</p><pre>import math<br>import platform<br>from pathlib import Path</pre><pre>def sigmoid(a):<br>    return 1.0 / (1.0 + math.exp(-a))</pre><pre>def softplus(a):<br>    return max(a, 0.0) + math.log1p(math.exp(-abs(a)))</pre><pre>print(f&quot;Python {platform.python_version()} | standard library | CPU | deterministic, no seed needed&quot;)<br>h = 1e-5<br>for a in (-8.0, 0.0, 8.0):<br>    minimax = -sigmoid(a)<br>    non_saturating = sigmoid(a) - 1.0<br>    fd_minimax = (-softplus(a + h) + softplus(a - h)) / (2*h)<br>    fd_non = (softplus(-a - h) - softplus(-a + h)) / (2*h)<br>    assert abs(minimax - fd_minimax) &lt; 1e-9<br>    assert abs(non_saturating - fd_non) &lt; 1e-9<br>    print(f&quot;Logit {a:4.0f} | minimax {minimax:.6f} | non saturating {non_saturating:.6f}&quot;)</pre><p><strong>Step 2. Derive the ideal classifier for two distributions and check the JS identity.</strong></p><p>All masses here are positive. This simple code does not implement the zero mass conventions needed for arbitrary distributions.</p><pre>data = [0.5, 0.5]<br>for name, generated in [(&quot;balanced&quot;, [0.5, 0.5]), (&quot;biased&quot;, [0.9, 0.1])]:<br>    optimal_d = [p/(p+q) for p, q in zip(data, generated)]<br>    mixture = [(p+q)/2 for p, q in zip(data, generated)]<br>    js = sum(0.5*(p*math.log(p/m) + q*math.log(q/m))<br>             for p, q, m in zip(data, generated, mixture))<br>    game = sum(p*math.log(d) + q*math.log(1-d)<br>               for p, q, d in zip(data, generated, optimal_d))<br>    assert abs(game - (-math.log(4) + 2*js)) &lt; 1e-12<br>    print(f&quot;{name}: D* {[round(d, 6) for d in optimal_d]} | JS {js:.6f} | game {game:.6f}&quot;)</pre><p><strong>Step 3. Export the calculated gradient curves.</strong></p><p>The SVG is editable and opens in a browser. It shows mathematical local derivatives across fixed logits, not training progress.</p><pre># Export the computed local derivative curves as an editable SVG.<br>xs = [-10 + 20*i/400 for i in range(401)]<br>def curve(fn):<br>    return &quot; &quot;.join(f&quot;{100 + 50*(a+10):.2f},{560 - 400*fn(a):.2f}&quot; for a in xs)<br>svg = f&#39;&#39;&#39;&lt;svg xmlns=&quot;http://www.w3.org/2000/svg&quot; width=&quot;1200&quot; height=&quot;700&quot; viewBox=&quot;0 0 1200 700&quot;&gt;<br>&lt;rect width=&quot;1200&quot; height=&quot;700&quot; fill=&quot;#FFF9EF&quot;/&gt;<br>&lt;g font-family=&quot;Arial,sans-serif&quot; fill=&quot;#153449&quot;&gt;<br>&lt;text x=&quot;70&quot; y=&quot;42&quot; font-size=&quot;22&quot;&gt;Inside AI · Deep Learning Models · Part 06&lt;/text&gt;<br>&lt;text x=&quot;70&quot; y=&quot;88&quot; font-size=&quot;32&quot;&gt;Computed magnitude of the local logit gradient&lt;/text&gt;<br>&lt;path d=&quot;M100 160 V560 H1100&quot; fill=&quot;none&quot; stroke=&quot;#153449&quot; stroke-width=&quot;2&quot;/&gt;<br>&lt;polyline points=&quot;{curve(sigmoid)}&quot; fill=&quot;none&quot; stroke=&quot;#C66B2B&quot; stroke-width=&quot;4&quot;/&gt;<br>&lt;polyline points=&quot;{curve(lambda a: 1-sigmoid(a))}&quot; fill=&quot;none&quot; stroke=&quot;#087F8C&quot; stroke-width=&quot;4&quot;/&gt;<br>&lt;text x=&quot;45&quot; y=&quot;170&quot; font-size=&quot;22&quot;&gt;1.0&lt;/text&gt;&lt;text x=&quot;45&quot; y=&quot;565&quot; font-size=&quot;22&quot;&gt;0.0&lt;/text&gt;<br>&lt;text x=&quot;88&quot; y=&quot;595&quot; font-size=&quot;22&quot;&gt;-10&lt;/text&gt;&lt;text x=&quot;594&quot; y=&quot;595&quot; font-size=&quot;22&quot;&gt;0&lt;/text&gt;<br>&lt;text x=&quot;1085&quot; y=&quot;595&quot; font-size=&quot;22&quot;&gt;10&lt;/text&gt;<br>&lt;text x=&quot;380&quot; y=&quot;630&quot; font-size=&quot;24&quot;&gt;Discriminator logit a for a fake sample&lt;/text&gt;<br>&lt;text x=&quot;310&quot; y=&quot;675&quot; font-size=&quot;23&quot; fill=&quot;#C66B2B&quot;&gt;Minimax&lt;/text&gt;<br>&lt;text x=&quot;650&quot; y=&quot;675&quot; font-size=&quot;23&quot; fill=&quot;#087F8C&quot;&gt;Non saturating&lt;/text&gt;<br>&lt;text x=&quot;120&quot; y=&quot;140&quot; font-size=&quot;22&quot;&gt;|dL/da|&lt;/text&gt;<br>&lt;/g&gt;&lt;/svg&gt;&#39;&#39;&#39;<br>Path(&quot;computed-gradient-curves.svg&quot;).write_text(svg, encoding=&quot;utf-8&quot;)<br>print(&quot;Finite difference checks and JS identity: PASS&quot;)</pre><p><strong>Actual output · 7 October 2026 · CPU · deterministic inputs</strong></p><pre>Python 3.12.14 | standard library | CPU | deterministic, no seed needed<br>Logit   -8 | minimax -0.000335 | non saturating -0.999665<br>Logit    0 | minimax -0.500000 | non saturating -0.500000<br>Logit    8 | minimax -0.999665 | non saturating -0.000335<br>balanced: D* [0.5, 0.5] | JS 0.000000 | game -1.386294<br>biased: D* [0.357143, 0.833333] | JS 0.101749 | game -1.182796<br>Finite difference checks and JS identity: PASS</pre><figure><img alt="Computed local gradient magnitude over fake sample logits" src="https://cdn-images-1.medium.com/proxy/1*gmiiVNx0TGjUIRVNZ_BYRg.png" /></figure><p><em>Figure 6. Fresh calculation on 7 October 2026, Python 3.12.14, standard library, CPU. Curves use the fixed logit grid in Step 3. This is not a GAN training experiment.</em></p><p>The rejected fake at -8 receives almost no minimax logit signal. The biased distribution has positive JS and a game value above -log 4. Those results check the calculations. They do not show that a neural generator will converge, cover every mode or create useful images.</p><p>One operation to remember: follow the derivative through the discriminator input before deciding what the generator can learn.</p><p>Next comes a different question: what if we learn to remove noise instead of learning against an opponent? Part 07 will introduce diffusion. For now, return to the verified Part 05 companion above and mark which update detaches the fake sample.</p><p>Come learn alongside me for projects and interview foundations. Follow Inside AI on <a href="https://medium.com/@thesomilsinghofficial">Medium</a> or <a href="https://thesomilsingh.substack.com/">Substack</a> and find the other channels in <a href="https://x.com/Skywalkerlyzv/status/2105897288749253054">my pinned X post</a>. If this helped, share it or leave the gradient you predicted. A clap on Medium is always welcome ❤️</p><p>Sources: <a href="https://arxiv.org/abs/1406.2661">original GAN paper</a>, <a href="https://docs.pytorch.org/docs/2.14/generated/torch.nn.BCEWithLogitsLoss.html">PyTorch logit loss documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a991b0cf2e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How a GAN learns to turn noise into samples]]></title>
            <link>https://medium.com/@thesomilsinghofficial/how-a-gan-learns-to-turn-noise-into-samples-a07797c62726?source=rss-959694b93e8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a07797c62726</guid>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Somil Singh]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 12:59:50 GMT</pubDate>
            <atom:updated>2026-10-07T13:39:07.729Z</atom:updated>
            <content:encoded><![CDATA[<p>A generator has no paired target, so where does its learning signal come from?</p><p>Somil Singh · Inside AI</p><p><strong>Deep Learning Models · Part 05</strong></p><p>An autoencoder starts with an image and tries to reconstruct it. A GAN can start with random noise and produce a new sample without being given the correct answer for that particular noise draw.</p><p><strong>If there is no paired target, what tells the generator how to improve?</strong></p><p>A second network learns to distinguish real data from generated samples. Its feedback gives the generator a direction to follow. The useful part is understanding which network learns at each step and where the gradient travels.</p><p>These are my visual notes for Part 05 of Inside AI: Deep Learning Models. We will trace the two updates, then inspect a small four cluster experiment. <a href="https://medium.com/@thesomilsinghofficial/the-vq-vae-gradient-trick-what-learns-when-a-code-is-selected-b7f4a54db78d">Part 04</a> followed gradients through a discrete code choice. Here the learning signal comes from a changing critic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mP5LARxB_RrMtsp4ds2IIA.png" /></figure><p><em>Figure 1. Two learning routes. This is a schematic, not output from a trained model.</em></p><h4>1. The hidden code is a random starting point</h4><p>Imagine a page with four small clouds of points, one near each corner. We want a program that can draw new points with the same pattern.</p><p>An ordinary reconstruction task would start from a real point. A GAN instead samples a hidden input <strong>z</strong> from a noise distribution, often a standard normal. It transforms that input into a generated point <strong>G(z)</strong>.</p><p>The generator G is a neural network with learned parameters. Different noise inputs can produce different samples, but a coordinate of z is not guaranteed to control a meaningful property such as colour or position.</p><p>The resulting distribution is <strong>implicit</strong>: we can draw noise and run G to sample it. We do not need a tractable formula for the density of each output. A basic GAN also has no encoder that sends an input back to its code.</p><h4>2. A second network supplies the feedback</h4><p>The <strong>discriminator D</strong> sees both real samples x and generated samples G(z). It learns a binary classification task: did this input come from the data or the generator?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b-_OXZ5thPCJIm5Zd46VdQ.png" /></figure><p><em>Figure 2. The generator makes samples and the discriminator learns to separate their sources. Schematic.</em></p><p>When the discriminator detects a difference, its gradient can tell the generator how to change its output. Both networks keep changing, so the feedback is a moving target rather than a fixed reconstruction error.</p><p>D(x) is a probability under this training setup. It is not a universal realism score for every possible input. A classifier that is uncertain because it learned poorly is different from one that cannot distinguish genuinely matched distributions.</p><h4>3. The losses give the networks different jobs</h4><p>The <a href="https://arxiv.org/abs/1406.2661">original GAN paper</a> describes a minimax game. D tries to separate the sources while G tries to defeat that distinction.</p><pre>min_G max_D E_real[log D(x)] + E_z[log(1 - D(G(z)))]</pre><p>For D, our code minimizes the negative of those two terms. Real samples receive label one and generated samples label zero. This rewards correct source classification.</p><p>For G, we use the paper’s common <strong>non saturating</strong> alternative: minimize the negative log probability that D assigns to a generated sample being real.</p><pre>L_G = -E_z[log D(G(z))]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5EzSyTnc0E6yVO7oAzx7vw.png" /></figure><p><em>Figure 3. Two minimization objectives with different jobs. D loss adds two batch means; G uses one. Schematic.</em></p><p>The generator uses label one during its update, but that does not give it a paired real target. It is being asked to change a classifier’s response to its own generated sample.</p><p>Our D returns <strong>logits</strong>, the values before a sigmoid. <a href="https://docs.pytorch.org/docs/2.14/generated/torch.nn.BCEWithLogitsLoss.html">PyTorch’s BCEWithLogitsLoss documentation</a> describes the combined stable calculation. Do not apply another sigmoid before this loss.</p><p>These objectives provide a training signal. Neither loss is a certificate that the samples have enough fidelity or diversity.</p><h4>4. The gradient must reach the right network</h4><p>For the D update, generate a fake batch and <strong>detach it</strong>. D learns to classify that batch without accumulating gradients in G through the fake branch.</p><p>For the G update, produce a fresh fake batch. Freeze D’s parameters but keep its computation differentiable with respect to the generated input. The gradient still travels through D, through the sample and back into G.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PO9OdDXTWTj61WOXIdRvfg.png" /></figure><p><em>Figure 4. Frozen parameters do not mean a stopped input gradient. Values are an analytical example, not a new training experiment.</em></p><p>Take a generated sample with D(G(z)) = 0.1. The generator loss is −log(0.1), about <strong>2.303</strong>. With respect to D’s input logit, the non saturating loss has derivative <strong>0.1 − 1 = −0.9</strong>. The original minimax generator term has derivative <strong>−0.1</strong> at that same probability.</p><p>That comparison explains why the alternative can supply a stronger early signal. The complete gradient still depends on the discriminator and generator Jacobians, so it does not guarantee stable training.</p><p><strong>An easy mistake:</strong> putting D’s forward pass inside no_grad during G training breaks the path G needs. Freezing D’s parameters and disabling differentiation through the whole computation are different operations. Clear each optimizer’s gradients before its backward pass because PyTorch accumulates them.</p><h4>5. Training and generation take different routes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHj9hoGR_wWLGmld3G1eEQ.png" /></figure><p><em>Figure 5. The generator stays. The discriminator is a training companion rather than a required sampling component. Schematic.</em></p><p>During training, we draw real data and noise, update D, then draw fresh noise and update G through D. This example performs one update of each per iteration. Other update ratios change the dynamics.</p><p>During generation, sample new noise and run the trained generator. No real input, encoder or discriminator call is required for this basic sampling route.</p><p>That resolves the opening puzzle: G can learn without a paired target because D supplies differentiable feedback during training. Once G has learned, the noise prior supplies new starting points.</p><h4>6. Good looking losses can hide missing samples</h4><p><strong>Many noise inputs produce similar outputs.</strong> This can be mode collapse: the generator misses regions of the data distribution. Compare sample diversity and coverage rather than trusting a single loss.</p><p><strong>Generated points fall between the real clusters.</strong> Balanced nearest centre counts can hide this. Every point has a nearest centre even when it is far from all four. Check distances and the scatter plot too.</p><p><strong>Updates swing back and forth.</strong> The networks are changing one another’s learning problem. Inspect samples over time, learning rates and update balance. One falling curve is not a reliable quality ranking.</p><p>Loss reductions matter. Our D loss adds a real batch mean and a fake batch mean. If the distributions matched perfectly with D = 0.5, that convention would give <strong>2 log 2 ≈ 1.386</strong> for D and <strong>log 2 ≈ 0.693</strong> for G. Averaging the two D terms changes its numerical reference value. Losses near these values do not by themselves establish the ideal state.</p><p>These are checks rather than complete diagnoses. Data, architecture, capacity and optimization also matter. A small point experiment does not establish performance on unseen images.</p><h4>Keep these three ideas</h4><ol><li><strong>G makes samples.</strong> It maps noise to outputs without a paired real target for each noise draw.</li><li><strong>D supplies a training signal.</strong> Freeze its parameters during the G update while preserving its input gradient.</li><li><strong>Sampling and evaluation are separate.</strong> G alone generates; samples and coverage reveal things losses can hide.</li></ol><h4>7. A small GAN you can actually inspect</h4><p>The real distribution has four equally likely Gaussian clusters centered at (−1, −1), (−1, 1), (1, −1) and (1, 1). Each coordinate has standard deviation 0.12 within a cluster. Two dimensional data makes missing regions and unwanted bridges visible without a downloaded dataset or GPU.</p><p>Install PyTorch and Matplotlib with pip install torch matplotlib. Save the three blocks in order as one Python script and run it. Both networks have two hidden layers of width 64. The script trains for 3001 iterations using batches of 128, Adam learning rate 0.0005 and betas (0.5, 0.999). These are the settings of this toy check, not a universal recipe.</p><p><strong>Step 1. Build the data source and both networks.</strong> Sample a real cluster uniformly, then add Gaussian noise. The discriminator’s last layer returns one logit rather than a sigmoid probability.</p><pre>&quot;&quot;&quot;A small non saturating GAN on a four mode 2D mixture.&quot;&quot;&quot;<br>from pathlib import Path<br>import platform<br>import torch<br>from torch import nn<br>from torch.nn import functional as F<br>import matplotlib<br>matplotlib.use(&quot;Agg&quot;)<br>import matplotlib.pyplot as plt</pre><pre>torch.manual_seed(7)<br>torch.set_num_threads(1)<br>out = Path(__file__).resolve().parent / &quot;images&quot;<br>out.mkdir(exist_ok=True)<br>print(f&quot;Python {platform.python_version()} | PyTorch {torch.__version__} | CPU | seed 7&quot;)<br>centers = torch.tensor([[-1.0, -1.0], [-1.0, 1.0], [1.0, -1.0], [1.0, 1.0]])<br>def real_batch(n):<br>    return centers[torch.randint(4, (n,))] + 0.12 * torch.randn(n, 2)<br>generator = nn.Sequential(nn.Linear(2, 64), nn.LeakyReLU(0.2), nn.Linear(64, 64), nn.LeakyReLU(0.2), nn.Linear(64, 2))<br>discriminator = nn.Sequential(nn.Linear(2, 64), nn.LeakyReLU(0.2), nn.Linear(64, 64), nn.LeakyReLU(0.2), nn.Linear(64, 1))<br>g_opt = torch.optim.Adam(generator.parameters(), lr=0.0005, betas=(0.5, 0.999))<br>d_opt = torch.optim.Adam(discriminator.parameters(), lr=0.0005, betas=(0.5, 0.999))</pre><p><strong>Step 2. Alternate the updates.</strong> Detach generated samples for D. For G, freeze D’s parameters while keeping its input path active. Each printed loss uses that update’s sampled batch, with the D update preceding the G update.</p><pre>history = []<br>for step in range(3001):<br>    real = real_batch(128)<br>    fake = generator(torch.randn(128, 2)).detach()<br>    d_loss = (F.binary_cross_entropy_with_logits(discriminator(real), torch.ones(128, 1))<br>              + F.binary_cross_entropy_with_logits(discriminator(fake), torch.zeros(128, 1)))<br>    d_opt.zero_grad()<br>    d_loss.backward()<br>    d_opt.step()<br>    for parameter in discriminator.parameters():<br>        parameter.requires_grad_(False)<br>    fake = generator(torch.randn(128, 2))<br>    g_loss = F.binary_cross_entropy_with_logits(discriminator(fake), torch.ones(128, 1))<br>    g_opt.zero_grad()<br>    g_loss.backward()<br>    g_opt.step()<br>    for parameter in discriminator.parameters():<br>        parameter.requires_grad_(True)<br>    history.append((d_loss.item(), g_loss.item()))<br>    if step in (0, 1500, 3000):<br>        print(f&quot;Step {step:4d} | D loss {d_loss.item():.4f} | G loss {g_loss.item():.4f}&quot;)</pre><p><strong>Step 3. Generate and inspect coverage.</strong> Use G alone to draw 2000 new points. Count their nearest centres, then count only points within radius 0.4. Save the sample plot and loss history for inspection.</p><pre>with torch.no_grad():<br>    real = real_batch(2000)<br>    generated = generator(torch.randn(2000, 2))<br>    distances = (generated[:, None] - centers[None]).square().sum(-1).sqrt()<br>    nearest_distance, mode = distances.min(1)<br>    counts = torch.bincount(mode, minlength=4)<br>    valid = nearest_distance &lt; 0.4<br>    valid_counts = torch.bincount(mode[valid], minlength=4)<br>print(&quot;Nearest mode counts:&quot;, counts.tolist())<br>print(&quot;Counts within radius 0.4:&quot;, valid_counts.tolist())<br>print(f&quot;Fraction within radius 0.4: {valid.float().mean().item():.3f}&quot;)<br>assert torch.isfinite(generated).all()<br>torch.save({&quot;real&quot;: real, &quot;generated&quot;: generated, &quot;history&quot;: history}, out.parent / &quot;result.pt&quot;)<br>fig, axes = plt.subplots(1, 2, figsize=(10, 4.5))<br>for ax, points, title, color in zip(axes, [real, generated], [&quot;Real samples&quot;, &quot;Actual GAN samples&quot;], [&quot;#168a87&quot;, &quot;#d77d42&quot;]):<br>    ax.scatter(points[:, 0], points[:, 1], s=4, alpha=0.35, c=color)<br>    ax.set(xlim=(-2, 2), ylim=(-2, 2), title=title, xlabel=&quot;Coordinate 1&quot;, ylabel=&quot;Coordinate 2&quot;)<br>    ax.set_aspect(&quot;equal&quot;)<br>fig.tight_layout()<br>fig.savefig(out / &quot;actual-output.png&quot;, dpi=180)<br>plt.close(fig)<br>fig, ax = plt.subplots(figsize=(8, 4.5))<br>smooth = torch.tensor(history).unfold(0, 100, 100).mean(-1).numpy()<br>ax.plot(torch.arange(len(smooth)).numpy() * 100, smooth[:, 0], label=&quot;D loss&quot;, color=&quot;#168a87&quot;)<br>ax.plot(torch.arange(len(smooth)).numpy() * 100, smooth[:, 1], label=&quot;G loss&quot;, color=&quot;#d77d42&quot;)<br>ax.set(xlabel=&quot;Training step&quot;, ylabel=&quot;Mean loss over 100 steps&quot;, title=&quot;Actual training losses are diagnostics, not quality scores&quot;)<br>ax.legend(frameon=False)<br>fig.tight_layout()<br>fig.savefig(out / &quot;actual-losses.png&quot;, dpi=180)<br>plt.close(fig)</pre><p><strong>Recorded output from 2 October 2026:</strong> Python 3.12.14, PyTorch 2.14.1+cpu, CPU and seed 7. This is the archived successful run, not a fresh training run on 6 October. Values can vary with software versions or hardware.</p><pre>Python 3.12.14 | PyTorch 2.14.1+cpu | CPU | seed 7<br>Step    0 | D loss 1.3789 | G loss 0.7235<br>Step 1500 | D loss 1.3359 | G loss 0.7461<br>Step 3000 | D loss 1.3790 | G loss 0.7290<br>Nearest mode counts: [526, 494, 482, 498]<br>Counts within radius 0.4: [505, 465, 465, 468]<br>Fraction within radius 0.4: 0.951</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HhAZEFCDCg5aW4NEG_jYkA.png" /></figure><p><em>Figure 6. Actual output from the recorded 2 October 2026 run: 2000 real and 2000 generated points. Axes are data coordinates, not latent coordinates.</em></p><p>In that run, <strong>1903 of 2000 generated points</strong>, or <strong>95.15 percent</strong>, fell within the chosen radius. The printed fraction rounds to 0.951. Close counts cover all four centres but are not perfectly balanced. Thin bridges remain between clusters in the generated plot.</p><p>Radius 0.4 is a teaching diagnostic chosen for clusters with standard deviation 0.12, not a standard generative modelling metric. It tests neither the exact within cluster distribution nor unseen image quality. One seed and one tiny distribution do not establish reliability. The finite output assertion is only a numerical sanity check.</p><p>The operation to remember is <strong>generator(torch.randn(2000, 2))</strong>. Fresh noise goes in and a learned sample comes out; the training critic is absent from that line.</p><p>Next in Deep Learning Models: why does the ideal discriminator estimate a density ratio and why do generator objectives produce different gradients? <a href="https://medium.com/@thesomilsinghofficial/why-a-gan-can-reject-a-fake-and-still-teach-almost-nothing-5a991b0cf2e5">Part 06 now connects that mathematics to this update loop.</a> For a contrasting generation route, <a href="https://medium.com/@thesomilsinghofficial/how-vaes-learn-to-generate-new-images-9ebfd7a6a508">Part 01</a> starts with a VAE prior and decoder.</p><p>I’m sharing these study notes as I upskill in AI, machine learning, deep learning, computer vision and robotics. Come learn alongside me and connect the equations to foundations for projects and interview preparation ❤️</p><p>Follow me on all my socials to stay up to date with every post and the fun. Find me on <a href="https://medium.com/@thesomilsinghofficial">Medium</a> and/or <a href="https://thesomilsingh.substack.com/">Substack</a>. For my other links, check out my pinned post on <a href="https://x.com/Skywalkerlyzv">X</a>.</p><p>If something clicked, share it with someone who is learning, support the notes with a clap or leave a question. Which failure would you check first if the losses looked reasonable but one cluster was missing?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a07797c62726" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>