<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nevan Sujit on Medium]]></title>
        <description><![CDATA[Stories by Nevan Sujit on Medium]]></description>
        <link>https://medium.com/@nsthettayil?source=rss-3bdca9bd450a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Eh0O1diau4BXFFDj</url>
            <title>Stories by Nevan Sujit on Medium</title>
            <link>https://medium.com/@nsthettayil?source=rss-3bdca9bd450a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:47:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nsthettayil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[VAEs problems]]></title>
            <link>https://medium.com/@nsthettayil/vaes-problems-5bd30dc7c59e?source=rss-3bdca9bd450a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bd30dc7c59e</guid>
            <dc:creator><![CDATA[Nevan Sujit]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 01:41:03 GMT</pubDate>
            <atom:updated>2026-08-24T01:41:03.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCeNKaZ-TDIMdX_D57A_4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uOv48U7JUty8bAGD25ZDTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u2FRXQWcD6MHJLhRLtkqrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*15CecX9R9g0YIi9jfqFbJA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bd30dc7c59e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Variational Auto-encoders]]></title>
            <link>https://medium.com/@nsthettayil/variational-auto-encoders-ddf221e2b075?source=rss-3bdca9bd450a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddf221e2b075</guid>
            <dc:creator><![CDATA[Nevan Sujit]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 01:24:08 GMT</pubDate>
            <atom:updated>2026-08-24T01:24:08.719Z</atom:updated>
            <content:encoded><![CDATA[<p>Auto-encoders, as prev. seen, provide an intuitive method to encode data. They are good at compressing a sample, to then decode it for later retrieval or interpret its features. But if you were to choose an arbitrary point in the latent space, you likely wouldn’t be able to generate meaningful new samples because the encoder-decoder mappings are deterministic, or zero variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8VXhVTegls8gLnwXJgqLRQ.png" /><figcaption>This autoencoder’s latent space is meant to compress data about cats, and it does that well, but is not capable of generating new cat samples</figcaption></figure><p>It would be interesting if we could create a probability distribution around our encoder-decoder mappings to allow us to sample data rather than just compress it. This is what <em>variational</em> auto-encoders allow us to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u2co1kgFNKNzO4mF_XHK3g.png" /><figcaption>Latent space mapping for autoencoders vs variational autoencoder</figcaption></figure><h4>Variational Auto-encoder Architecture</h4><p>The way a variational auto-encoder is able to do this is by encoding data into a (latent) mean and variance to create a mixture of Gaussians it can sample from, rather than directly encoding the data in the latent space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JukbSUfzd3SXuz-MEdP8dA.png" /></figure><h4>Loss function</h4><p>The loss function of a VAE is the combination of reconstruction loss and KL Divergence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*KBBgyuSDCnJgfhMam10a4A.png" /></figure><p>KL Divergence is a way of calculating how likely a given probability distribution will have samples similar to another. Note that KL Divergence is not commutative between P and Q, and is non-negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*aHfCJqpzAj7Qp-z-CSe7_Q.png" /></figure><p>KL Divergence is used in the VAE loss function as a regularization term, by penalizing the latent distribution for being different than P(z) = N(0, I). The idea is that the reconstruction loss pushes our latent distribution towards better reconstruction while KL divergence pulls the distribution towards behaving more Gaussian. This regularization is important to make sure the variance does not collapse to 0 (distribution stays probabilistic).</p><p>Evidence Lower Bound is the additive inverse of our loss function, it is important in variational statistics as what we should maximize instead of the intractable integral P(x). We won’t go over the full derivation of the ELBO function, but because our VAE uses underlying Gaussian probability distributions we can simplify it to the following form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GY78bz6gK4MlEd_qj31VHg.png" /></figure><p>The expectation term is just our log likelihood. The tr(Σ/I) describes the ratio of the magnitude of the variance of Q(X|Z) and P(Z)=N(0,I). We subtract J, the number of dimensions in latent space, from it to normalize it by its number of dimensions. The term ln(det(Σ)) takes into account how similar in orientation the variances of the two distributions are. And ||μ-0||² describes how far away the two distributions are, using the Mahalanobis metric.</p><h4>Re-parameterization Trick</h4><p>Once we have a loss function, next we need to determine our optimizer. Gradient descent works to optimize the decoder’s parameters but doesn’t work for the encoder’s parameters because we’d need to take the gradient of a random variable which we are sampling. So we re-parameterize Z.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*unJGSQWic3YnCLJkDp8D5A.png" /><figcaption>We can take the gradient of decoder’s parameters</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*OntZMoKUvWSGP8TE8aMC7A.png" /><figcaption>We cannot take gradient of a function we are sampling over</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*SPU3uwUXEq_gKFcpsoIwbg.png" /><figcaption>Reparameterize to normal distribution</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*0lqfzzqFolv7niBmcOtIyQ.png" /><figcaption>With this reparameterization, we are able to take the gradient</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*nsVkIbchOqrW1s4Wx6iwwQ.png" /><figcaption>Re-parameterization visually</figcaption></figure><h4>Applications and Conclusions</h4><p>VAEs are in important step towards generative neural network techniques. In addition to being used for generating new data samples, they are also very applicable in anomoly detection by finding data inputs which are outliers in the latent probability distribution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddf221e2b075" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compressing data with Autoencoders]]></title>
            <link>https://medium.com/@nsthettayil/compressing-data-with-autoencoders-5bb8bd434da0?source=rss-3bdca9bd450a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bb8bd434da0</guid>
            <category><![CDATA[math]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nevan Sujit]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 00:52:12 GMT</pubDate>
            <atom:updated>2026-08-24T01:45:45.111Z</atom:updated>
            <content:encoded><![CDATA[<p>We previously looked at compressing data using principal component analysis, and the natural step forward from this is to build a model for compressing data non-linearly into latent space.</p><p>Take, for example, a dataset of points meant to model a spiral, seen below. A spiral can be parameterized into 1D by the parametric eq: (t, sin t, cos t). It would not be possible for PCA to model this kind of transformation because PCA is a linear compression technique. We must utilize autoencoders instead to learn how to compress the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FroPpffRvEMKrxhBOfsirA.png" /></figure><h4><strong>Autencoder Architecture</strong></h4><p>Autencoders transform data from the <em>data space</em> to the <em>latent space. </em>Autencoders use a neural network architecture as a universal function approximator, combined with a loss function and optimizer, to learn the nonlinear transformation from data.</p><p>Since we don’t have access directly to the latent dimension from our data, we must build an encoder (E) and decoder (D) matrix and minimize the loss of W = ED matrix to reconstruct the data.</p><p>It is important that the number dims in latent space be less than in the input space (n&gt;p), because otherwise the neural network might learn some for of the identity function (E=I, D=I) which would not be useful when compressing and reconstructing new data, not in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzhtB0Kzs6-2QUEmmBz-BA.png" /></figure><pre>class Autoencoder(nn.Module):<br>    def __init__(self, input_dim=n, latent_dim=p):<br>        super().__init__()<br><br>        # Encoder: compress input into a smaller latent representation<br>        self.encoder = nn.Sequential(<br>            nn.Linear(input_dim, latent_dim),<br>            nn.ReLU(),<br>        )<br><br>        # Decoder: reconstruct the input from the latent representation<br>        self.decoder = nn.Sequential(<br>            nn.Linear(latent_dim, input_dim)<br>            nn.Sigmoid()  # assumes input is normalized to [0, 1]<br>        )<br><br>    def forward(self, x):<br>        z = self.encoder(x)<br>        x_recon = self.decoder(z)<br>        return x_recon<br><br>    def encode(self, x):<br>        return self.encoder(x)<br><br>    def decode(self, z):<br>        return self.decoder(z)<br><br><br># --- Example training loop ---<br>if __name__ == &quot;__main__&quot;:<br>    model = Autoencoder(input_dim=784, latent_dim=32).to(device)<br>    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)<br>    error = nn.MSELoss()<br>    # data = <br><br>    for epoch in range(10):<br>        optimizer.zero_grad()<br>        output = model(data)<br>        loss = error(output, data)<br>        loss.backward()<br>        optimizer.step()</pre><p>The above implementation is for a single layer autencoder, but it is recommended to use multiple layers.</p><p>Additionally, the latent space of the autoencoder can either make the autoencoder:</p><ul><li>undercomplete: p&lt;n</li><li>overcomplete: p&gt;n</li></ul><p>Undercomplete autoencoders are useful when trying to compress data. Overcomplete autoencoder become useful when the input data have complex relationships between features that would require an overcomplete latent space to disentangle/represent. It is important to utilize regularization techniques like dropout and weight decay because otherwise overcomplete autencoders could lead to overfitting.</p><p>It is important that the number dims in latent space be less than in the input space (n&gt;p), because otherwise the neural network might learn some for of the identity function (E=I, D=I) which would not be useful when compressing and reconstructing new data, not in the dataset.</p><h4>Feature Extraction</h4><p>It is often possible to identify unseen patterns in data using autoencoders by looking at how it represents the data in latent space. For example, autencoders make it easy to find data points that are outliers by extracting features into a latent space.</p><h4><strong>The close relationship between autoencoders and PCA.</strong></h4><p>It may not be hard to believe, but an autoencoder with no nonlinear activation function simplifies to a PCA. The following proof will demonstrate this.</p><p>Two important remarks first.</p><ol><li>We mentioned W=ED, which shows that there is a unique W to minimize the loss, but there are infinite E and D solutions because we can write W = E* C^-1 * C * Dfor any invertible matrix C.</li><li>Also, notice that if there were no rank restrictions for E and D (no restriction on p / information bottleneck), the best solution for W would just be ordinary linear regression.</li></ol><p>Our loss function (MSE) is below. It is a convex function, as indicated by it’s quadratic nature, meaning it has a single global mimimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*9xuIpLUrJnlL-J-k-nY8cA.png" /></figure><p>We can take partial derivatives of the loss function with respect to E and D to get these equations for solving them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*Qxum5qCFliMwFPggGfSICA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*3U-kSbMkzKWgYG1-veFNZQ.png" /></figure><p>Note the introduction of our sample’s covariance matrices. We now introduce the eigenvalues / vectors of our sample’s covariance matrix below, and we can see how this allows us to rewrite our loss function in terms of its eigenvalues. It is clear then that best way to minimize the loss function is to choose the top p eigenvectors with the largest eigenvalues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*p8T9kCW0Q8wOkhVV-vquTA.png" /></figure><p>A linear autencoder will not project to the eigenbasis space, but rather some linear combination of it (C is not identity), which is why this connection is not obvious experimentally until mathematically shown. Additionally, because PCA uses orthonomal projections, the encoder has minimum norm. Autoencoders don’t generalize like this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bb8bd434da0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My deep-dive in Principal Component Analysis]]></title>
            <link>https://medium.com/@nsthettayil/my-deep-dive-in-principal-component-analysis-2881471ca4e4?source=rss-3bdca9bd450a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2881471ca4e4</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[autoencoder]]></category>
            <dc:creator><![CDATA[Nevan Sujit]]></dc:creator>
            <pubDate>Tue, 21 Jul 2026 21:15:53 GMT</pubDate>
            <atom:updated>2026-07-21T21:15:53.467Z</atom:updated>
            <content:encoded><![CDATA[<h3>My deep-dive in PCA</h3><p>I recently took the time to revisit PCA. I’ve been exploring advanced encoder techniques recently, and wanted to compared these techniques in the most basic level of lossy data compression.</p><p>Relearning it reminded me that you can always have a richer understanding of any topic until it becomes entirely intuitive.</p><h4>The formal definition of principal component analysis.</h4><p>The goal of principal component analysis is to identify the principal components of a dataset.</p><p><em>Principal components</em> then are formally defined as:</p><blockquote>the set of n-dimensional unit vectors in a collection of sample points in an Rⁿ coordinate system, where the iᵗʰ vector is the direction of best fit orthogonal to first i-1 vectors</blockquote><p><em>Best fit</em> is formally defined as:</p><blockquote>minimizing the mean squared error, or equivalent, maximizing variance</blockquote><p>While it might make sense why the best fit minimizes mean squared error, the reason that is equivalent to maximizing error is best seen using Pythagorean Theorem. A sample point in a dataset can be represented by a vector, and our goal is to find the direction <em>v</em> that minimizes MSE (geometrically described as distance between <em>v</em> and sample point).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7mT3hOWpwgjgU1UdVD_bQ.png" /></figure><h4>From definition to Lagrange Multipliers.</h4><p>The definition comes off as an optimization problem (maximize variance) under constraint (unit i-th vector + orthogonal to i-1 vectors). Lagrange multipliers immediately come to mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqyXBvt9UkO5yZ5OlVQkOQ.png" /></figure><p>Apply Lagrange Multipliers and Rayleign Quotient lead to an amazing result: the principal components are the eigenvectors of the covariance matrix XᵀX</p><h4>Aside: Singular Value Decomposition.</h4><p>Computers do principal component analysis using singular value decomposition (SVD).</p><p>We all know matrices represent linear transformations, but a rectangular matrix is a special type of transformation: one that transforms vectors across dimensions. Rectangular matrices are characterized by their SVD, three matrices that decompose the original matrix. Each describes a different part of the transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c7NnG2KpHvLp4_cN2NcEDg.png" /></figure><ul><li>Vᵀ: transforms vector to eigenbasis</li><li>Σ: scales vector by singular values √λ and reduces/adds dimension</li><li>U: transforms vector from eigenbasis to standard basis</li></ul><h4>Applying SVD to PCA.</h4><p>We can use SVD in PCA by noticing the connection between a dataset and its covariance matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*cRhCJVF2ts5WOG6aE9ETsg.png" /></figure><h4>Solving for the eigenface.</h4><p>PCA allows us to compress our dataset by finding which directions (which covariance eigenvectors) represent the most variance in the data. In a sense, <em>linear regression is just finding the largest principal component</em> (and finding the bias). In machine learning communities, the largest principal component is often referred to as the eigen-(data). We will solve for the eigenface of a face dataset by implementing SVD in C++.</p><p>The maximum eigenvector is calculated using the power series, which takes inspiration from the Rayleigh Quotient method described previously.</p><pre>vector&lt;vector&lt;double&gt;&gt; power_iter(vector&lt;vector&lt;double&gt;&gt; A, int iters) {<br>    int N = A.size();<br>    vector&lt;vector&lt;double&gt;&gt; v(N, {1.0});<br>    v = normalize(v);<br><br>    for (int i = 0; i &lt; iters; i++) {<br>        v = matrix_mult(A, v);<br>        v = normalize(v);<br>    }<br>    return v;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kzUzVm_Cpdar7Hr-QvqbTg.png" /></figure><p>The result when running a dataset through this is the “eigenface”.</p><p>Full code available: <a href="https://github.com/tsnevan4204/pimi/tree/main/coursework/knowledge/pca">github.com/tsnevan4204/pimi/tree/main/coursework/knowledge/pca</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2881471ca4e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>