<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vicky Suman on Medium]]></title>
        <description><![CDATA[Stories by Vicky Suman on Medium]]></description>
        <link>https://medium.com/@vickysuman?source=rss-eb8447b276a8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3k9-medgyeLZMzn7.</url>
            <title>Stories by Vicky Suman on Medium</title>
            <link>https://medium.com/@vickysuman?source=rss-eb8447b276a8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:49:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vickysuman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Evolution of Text Representation From Topic to Vectors]]></title>
            <link>https://vickysuman.medium.com/the-evolution-of-text-representation-from-topic-to-vectors-8ed3e3c51966?source=rss-eb8447b276a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ed3e3c51966</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[topic-clusters]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <dc:creator><![CDATA[Vicky Suman]]></dc:creator>
            <pubDate>Sat, 18 Apr 2026 14:12:27 GMT</pubDate>
            <atom:updated>2026-04-18T14:34:28.106Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-bHy4Y20ynB38Igk1b5x0w.png" /></figure><p>A deep comparison of two clustering philosophies — and why the choice is really about how you represent language, not just which algorithm you pick.</p><h4>Text clustering sounds simple on paper, take a collection of documents, group similar ones together, and use those groups to power product features. In practice, it is one of those engineering problems that becomes much harder the moment we move from experiments to real systems.</h4><p>A clustering approach that looks elegant in a notebook can fail badly in production. It may merge unrelated items, split the same event into several clusters, struggle with streaming updates, or become too expensive to run at scale. The challenge becomes even sharper when the goal is not generic topic discovery, but something operationally useful such as grouping related news stories, detecting trends, or consolidating signals from real-time systems.</p><p>In this post, I want to compare two very different but highly instructive approaches to text clustering:</p><ol><li><strong>LDA-based story clustering approach </strong>— a classical probabilistic approach</li><li><strong>A modern embedding-based trend clustering pipeline using Agglomerative Clustering </strong>— a modern semantic pipeline</li></ol><p>Both approaches are trying to solve a similar high-level problem: identify groups of text items that belong together. But they are built on very different assumptions, different representations of text, and different production philosophies.</p><p>This comparison is useful because it shows that choosing a clustering system is not only about picking an algorithm. It is about deciding how you want to represent language, how clusters should evolve, and what kind of product outcome you want the system to support.</p><h3>A brief history of clustering</h3><p>Before we dive into the two approaches, it helps to understand where clustering comes from. The ideas underpinning modern text clustering did not appear overnight, they evolved across seven decades of computer science, statistics, and linguistics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*quBdQdVZgOzE96A5x5rawA.png" /></figure><ul><li><strong>Hierarchical clustering emerges (1950s–1960s)</strong><br>The earliest clustering algorithms came from taxonomy and biology. Researchers needed ways to group species by similarity. Dendrogram’s tree structures that visualize merge history became the primary tool. Ward’s minimum-variance linkage in 1963 extended these ideas into a more general framework that is still used today.</li><li><strong>K-means the algorithm everyone knows (1967s)</strong><br>MacQueen introduced k-means, a deceptively simple algorithm. Assign points to the nearest centroid, recompute centroids, repeat. Despite its limitations fixed k, spherical cluster assumption, sensitivity to initialisation k-means became the default clustering workhorse for decades and remains heavily used in NLP pipelines.</li><li><strong>Linkage strategies are formalized (1970s)<br></strong>Single linkage, complete linkage, and average linkage were rigorously defined, giving practitioners a toolkit for controlling the shape and compactness of clusters in hierarchical methods. Average linkage the strategy most relevant to this blog emerged as the most balanced choice for real-world applications.</li><li><strong>DBSCAN density beats distance (1996)<br></strong>Ester, Kriegel, Sander, and Xu introduced DBSCAN (Density-Based Spatial Clustering of Applications with Noise). Unlike k-means, DBSCAN can find clusters of arbitrary shapes and explicitly labels outliers as noise. It does not require specifying the number of clusters in advance a significant practical advantage. HDBSCAN, a hierarchical extension, appeared in 2013.</li><li><strong>LDA latent topic structure for text (2003)<br></strong>Blei, Ng, and Jordan’s Latent Dirichlet Allocation transformed how practitioners thought about text. Instead of raw word counts, LDA represented each document as a mixture of latent topics. Each topic was a probability distribution over words. This gave NLP systems a compact, interpretable representation and directly informed the first approach discussed in this blog.</li><li><strong>The Embedding Era (2013 — Present)<br></strong>Word2Vec (2013) introduced dense word vectors learned from large corpora. BERT (2018) brought full contextual embeddings. Sentence-BERT (SBERT, 2019) made sentence-level semantic similarity practical at scale. These models fundamentally changed text clustering, instead of learning representations from scratch, engineers now start with pre-trained models that already encode rich semantic knowledge</li></ul><p><strong>Why this history matters: </strong>The two approaches compared in this blog are not isolated inventions. The LDA pipeline stands on 50 years of statistical NLP. The embedding pipeline stands on 70 years of clustering theory <em>plus</em> a decade of deep learning. Understanding the lineage helps you reason about what each approach can and cannot do.</p><h3>Why text clustering is hard in the real world</h3><p>Before comparing the two approaches, it helps to define what makes text clustering difficult. For structured data, clustering can sometimes be straightforward because the dimensions are explicit and stable. Text is different. Human language is messy, high-dimensional, ambiguous, and constantly changing.</p><ul><li><strong>High dimensionality</strong>: raw bag-of-words or TF-IDF vectors can be extremely sparse and large.</li><li><strong>Lexical variation</strong>: two documents can describe the same event using different wording.</li><li><strong>Semantic ambiguity</strong>: two documents may use similar words but refer to different real-world incidents.</li><li><strong>Unknown number of clusters</strong>: in most production cases, we do not know in advance how many groups exist.</li><li><strong>Streaming behaviour</strong>: new documents keep arriving, so clusters must evolve.</li><li><strong>Operational constraints</strong>: latency, compute cost, scheduling, monitoring, and downstream consumers all matter.</li></ul><p>Because of this, a strong clustering system is never just a mathematical algorithm. It is a full pipeline decision</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9XDdNVDkWvEHBthW0uOJQ.png" /></figure><h3>Approach 1: LDA-based story clustering</h3><p>Imagine grouping news articles about the same event — written by different publishers with different wording, style, and emphasis. The pipeline looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CW8gPmpDKv4SA2ULY5Q2xw.png" /></figure><p>At the heart of this design is Latent Dirichlet Allocation (LDA). Instead of representing a document as a huge sparse vector of raw words, it represents the document as a mixture of latent topics. A topic, in turn, is a probability distribution over words.</p><p>For example, a topic might emphasize words like <em>election, campaign, ballot, voters</em>. Another might emphasize <em>earthquake, magnitude, rescue, aftershock</em>. A document is then represented as a weighted combination of these topics something like “this article is 60% politics, 25% policy, 15% international relations.”</p><blockquote><strong>Why LDA was a strong design<br></strong>Bag-of-words vectors are too high-dimensional for efficient direct similarity search. LDA provides a compact, semantic, lower-dimensional representation. Two stories about the same event may not share many exact words, but they may still occupy similar positions in topic space.</blockquote><h3>Strengths of this approach</h3><ol><li>Unsupervised — no labeled pairs required</li><li>Interpretable — topics can be inspected by engineers and stakeholders</li><li>Dimensionality reduction — compresses via semantic structure, not mechanical compression</li><li>Scalable retrieval — approximate nearest-neighbor search is manageable in topic space</li><li>Fits long-form text — topic mixtures work well when documents are substantial</li></ol><h3>The core limitation</h3><blockquote>Topic similarity is not the same as event similarity.</blockquote><p>Two articles about different floods in different countries may share a very similar topic mixture — but from a product perspective, they should not be clustered into the same story. This is where LDA begins to show its limits: it has no mechanism to distinguish between “these documents are about the same type of event” and “these documents are about the exact same incident.”</p><h3>Approach 2: Embedding-based trend clustering with Agglomerative Clustering</h3><p>The modern pipeline shifts the philosophy entirely. Instead of learning topic mixtures, it starts with embedding vectors designed to place semantically similar texts close to one another in vector space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3sXTM73mJm06Hjn_9yHtw.png" /></figure><p>Consider these two texts:</p><ul><li>“Main tranist hub closed after electrical failure”</li><li>“Power outage shuts down central station”</li></ul><p>A bag-of-words model sees almost no overlap. A good embedding model places them very close together, because they describe the same event semantically, not lexically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kj0bWK629iBJzUnJ5QUqmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U-Nru_5RvP_QgrEqUPt4jw.png" /></figure><h3>Why Agglomerative Clustering fits this setup</h3><p>Agglomerative Clustering starts by treating each item as its own cluster, then progressively merges the closest clusters according to a distance metric and linkage strategy. It is attractive for text because it does not require a predefined number of clusters, works naturally with similarity thresholds, and supports variable cluster sizes.</p><h3>Why not k-means or DBSCAN?</h3><p>K-means assumes a fixed number of clusters and roughly spherical structure neither is realistic for real text streams. DBSCAN and HDBSCAN can be difficult to tune in high-dimensional embedding spaces. they often over-fragment data or become highly sensitive to parameter settings. Agglomerative clustering with average linkage tends to be the more stable choice for semantically meaningful cluster formation.</p><h3>The real strength: evolving clusters</h3><p>In a real trend-detection system, new signals arrive continuously. The system cannot afford to recompute everything from scratch each time. A practical pipeline ingests new signals, generates embeddings, compares them against existing cluster representations, merges or creates as needed, updates centroids, and publishes mature clusters. This turns clustering into a <em>living system</em> rather than a one-time experiment.</p><h3>Side-by-side comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gxreia5I-Owr18XI6O-Q3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmzPCXf3VaEwK_zs-JjdYg.png" /></figure><h3>A deeper reasoning framework for choosing between them</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ylXSHoIHZhTeVNiYTjw26g.png" /></figure><h3>When each approach is the better choice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iRTxDODxz_NaSOWkyPqeyw.png" /></figure><h3>These are two different philosophies</h3><p>At first glance, these approaches may look like two alternative clustering algorithms. They are not. They are really two different philosophies of how to build a text clustering system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xk2ZBC0uBP4UWHrDrUIZsA.png" /></figure><blockquote>This distinction is more important than the names of the specific algorithms. The choice is not “old versus new.” It is really: <em>topic-space organization</em> versus <em>semantic event/trend formation</em>.</blockquote><h3>Final takeaway</h3><blockquote>Both approaches are valid. Both are intelligent. But they optimize for different realities.</blockquote><blockquote>The <strong>LDA-based approach</strong> is a strong classical solution for interpretable, unsupervised thematic clustering. It shines when the problem is broad document grouping and the system benefits from transparent topic structure.</blockquote><blockquote>The <strong>embedding + Agglomerative Clustering approach</strong> is a stronger modern solution for real-world semantic clustering, especially when data arrives continuously and clusters must evolve into usable product objects such as trends, summaries, or alerts.</blockquote><blockquote>If your problem is closer to editorial organization, corpus exploration, or explainable thematic grouping, LDA still deserves consideration.</blockquote><blockquote>If your problem is closer to trend detection, incident grouping, signal consolidation, or production intelligence systems, embeddings with hierarchical clustering are usually the better design.</blockquote><blockquote>In the end, the best clustering system is not the one with the most sophisticated algorithm name. It is the one whose assumptions match the shape of your data, the behavior of your users, and the operational demands of your product.</blockquote><blockquote>That is the real engineering lesson.</blockquote><p><strong>Note</strong>: This post compares two real-world clustering philosophies: a classical LDA-based story clustering pipeline and a modern embedding-based semantic trend pipeline. The goal is not to declare a universal winner, but to understand where each approach fits best and why.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ed3e3c51966" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Translation Model (en_to_hi) Error by Google and IBM]]></title>
            <link>https://vickysuman.medium.com/language-translation-model-en-to-hi-error-by-google-and-ibm-6ffdf28f9750?source=rss-eb8447b276a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ffdf28f9750</guid>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[gｏｏｇｌｅ]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vicky Suman]]></dc:creator>
            <pubDate>Sat, 17 Apr 2021 10:28:49 GMT</pubDate>
            <atom:updated>2021-04-17T10:31:33.513Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently there was viral news regarding the google translation model <strong>English to Hindi</strong> translation “<a href="https://www.timesnownews.com/the-buzz/article/googles-epic-translation-of-unworried-is-viral/737247"><strong><em>Google’s epic translation of ‘unworried’ is viral</em></strong></a>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jkvwpTh2x3D_N2mOA31sjw.jpeg" /></figure><p>then google corrected their mistake and rectified the model but still, it’s having the wrong prediction in complex cases for e.g<br><strong>Input Text:</strong> <em>I am unmarried and even though I have a son who is unworried about this<br></em><strong>Google Translation text output: </strong><em>मैं अविवाहित हूं और भले ही एक बेटा है जो इस बारे में अविवाहित है<br></em><strong>IBM Translation text output:</strong> <em>मैं अविवाहित हूँ और यहां तक कि मैं एक बेटा है जो इस बारे में चिंतित है<br></em><strong>Correct Translation text:</strong> <em>मैं अविवाहित हूं और भले ही मेरा एक बेटा है जो इस बारे में निश्चिन्त है</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ROgkVUg_4W43UO0_dK66Gg.png" /><figcaption><a href="https://translate.google.co.in/?hl=en&amp;tab=TT"><strong>Google Translation Demo Link</strong></a></figcaption></figure><p>I have also checked out the <strong>IBM Watson</strong> Language Translation model and “unworried” was translated correctly, which google failed to do but the overall translation was <strong><em>lacking for complex contextual learning</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DyZNSOyJfo6avBl4JFsXzg.png" /><figcaption><a href="https://www.ibm.com/demos/live/watson-language-translator/embedded/home"><strong>IBM Watson Translation Demo Link</strong></a></figcaption></figure><p>Now, you might be <strong><em>wondering and curious</em></strong> to know <strong><em>if</em></strong> <strong><em>Google updated or rectified</em></strong> <strong><em>the model</em></strong>, <strong><em>then why it is still predicting wrong output (Translation), what’s the plausible explanation for it?</em></strong></p><ul><li>In the corpus, we may have less complex contextual examples with tokens “<strong><em>unworried</em></strong>” rather than “<strong><em>not worried</em></strong>”(which is commonly used)</li><li>Most likely if they correct the word embedding mapping layer of tokens <strong><em>unworried</em></strong> (en_encoder) to <strong><em>निश्चिन्त</em></strong> (hi_decoder) and <strong><em>unmarried</em></strong> (en_encoder) to <strong><em>अविवाहित</em></strong> (hi_decoder) and for simple sentences, it works perfectly.</li><li>If they update the <strong><em>encoder and decoder transformer</em></strong>, i.e. if they retrain the model, then though it is possible to do this by <strong><em>freezing</em></strong> most of the <strong><em>weights</em></strong> and <strong><em>fine-tuning</em></strong> with some specific examples, it is less likely but is also evident, that it has not been completely fixed and still shows up in sentences.</li><li>From this example, we can state some limitations or drawbacks of large and complex transformer model architectures.</li></ul><p>I am curious to learn and understand the underlying <strong><em>Hybrid Model Architecture</em></strong>. These are my thoughts and what you guys think, if you agree or disagree or have something to add on, then let me know in the comments. Will be happy to have the discussion. 🙂 <a href="https://medium.com/u/4f3f4ee0f977">Google Cloud</a>, <a href="https://medium.com/u/b21de552c2c5">Inside IBM Research</a>, <a href="https://medium.com/u/b4aa94b34fc1">IBM Cloud Stories</a>, <a href="https://medium.com/u/b21de552c2c5">Inside IBM Research</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ffdf28f9750" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>