<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Siddharth Kshirsagar on Medium]]></title>
        <description><![CDATA[Stories by Siddharth Kshirsagar on Medium]]></description>
        <link>https://medium.com/@siddharth1?source=rss-cfc1f01d5bce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d72O1D9Zb81oQvQ-DxYkGA.jpeg</url>
            <title>Stories by Siddharth Kshirsagar on Medium</title>
            <link>https://medium.com/@siddharth1?source=rss-cfc1f01d5bce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:17:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@siddharth1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Hidden Patterns Behind User Behaviour]]></title>
            <link>https://siddharth1.medium.com/the-hidden-patterns-behind-user-behaviour-e3dac8d65199?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3dac8d65199</guid>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 08:32:27 GMT</pubDate>
            <atom:updated>2026-08-10T08:32:27.209Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CZ9CrWp21DxNmtej" /><figcaption>Photo by <a href="https://unsplash.com/@elladon?utm_source=medium&amp;utm_medium=referral">ELLA DON</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When I first encountered eigenvectors, they felt disconnected from real work. There were matrices, arrows, and equations, but no obvious link to a customer, a product decision, or a business outcome.</p><p>The connection became clearer when I saw how eigenvectors are used in <strong>Principal Component Analysis (PCA)</strong>. PCA helps businesses find the most meaningful patterns in large, messy datasets. To make this concrete, consider an online game.</p><p><strong>The Formal Definition, in Plain Language</strong></p><p>An <strong>eigenvector</strong> is a special direction that stays pointing in the same direction after a transformation is applied to it. The transformation can stretch or shrink the vector, but it does not rotate it away from that direction.</p><p>The <strong>eigenvalue</strong> is the amount of stretching or shrinking associated with that eigenvector.</p><p>In symbols:</p><pre>Av=λv</pre><p>Where:</p><ul><li><em>A</em> is a transformation, often represented by a matrix.</li><li><strong>v</strong> is an eigenvector.</li><li><em>λ</em> is the eigenvalue.</li></ul><p>For data analysis, the intuition matters more than the notation,</p><blockquote>An eigenvector represents a major pattern or direction in the data. An eigenvalue tells us how important that pattern is.</blockquote><p>The larger the eigenvalue, the more variation in the data that direction explains.</p><h3>The Real-World Problem: Understanding Game Players</h3><p>Suppose an online game records these features for every player:</p><pre>Aggression<br>Exploration<br>Accuracy<br>Movement speed<br>Loot usage<br>Death rate</pre><p>Each player can be represented as a row of data. Here, each score is scaled from 0 to 1, where a higher number means more of that behaviour:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rm6LxYtwEglf5sTQvjzBpQ.png" /></figure><p>At first, this looks useful. But there is a problem: these features are often correlated.</p><p>For example, players who are highly aggressive may also move quickly, die more often, and spend less time exploring. If we inspect every metric separately, we can miss the broader behavioural pattern.</p><p>That is exactly the kind of problem PCA is designed to solve.</p><h4>How PCA Uses Eigenvectors</h4><p>PCA looks at how the features vary together and asks a simple question:</p><blockquote>What is the strongest overall pattern in player behaviour ?</blockquote><p>The answer becomes the first principal component. Mathematically, it is an eigenvector of the data’s covariance matrix.</p><p>The next component answers a second question:</p><blockquote>What is the next most important pattern that is independent of the first one?</blockquote><p>PCA continues this process until it has found the useful patterns in the data.</p><p>For this game, the first component might capture the difference between players who spend more time in fast combat and players who spend more time exploring the world. It gives every player a score along that spectrum.</p><p>The second component looks for another meaningful difference that the first one did not already explain. For example, among players with a similar combat-versus-exploration score, it might identify whether they tend to rely on accuracy or on movement and speed. The key point is that PCA does not start by naming groups of players. It looks at the data first and reveals the behaviours that tend to move together.</p><h4><strong>Why Eigenvalues Matter</strong></h4><p>Each principal component has an eigenvalue. In PCA, that value tells us how much of the overall variation in player behaviour the component captures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKNZHcDh62P1Z4f90NUlMQ.png" /></figure><p>In other words, PC1 alone summarises 42% of what makes one player’s behaviour different from another’s. PC1 and PC2 together summarise 65%. Rather than working with six overlapping raw metrics, we can represent each player with two compact scores:</p><pre>[Combat-Exploration Score, Accuracy-Movement Score]</pre><h4>Turning Player Patterns Into Business Impact</h4><p>PCA does not decide what a game should do. It creates cleaner, more useful behavioural features that other systems and teams can use.</p><p>Consider two players:</p><pre>Player 1<br>Combat-Exploration Score =  0.92<br>Accuracy-Movement Score  = -0.40<br><br>Player 2<br>Combat-Exploration Score = -0.70<br>Accuracy-Movement Score  =  0.85</pre><p>The first player is closer to the fast-combat end of the first spectrum. The game might prioritise combat-heavy missions, shorter time-to-action, aggressive characters, or kill-based challenges.</p><p>The second player is closer to the exploration end of that spectrum. The game might prioritize hidden areas, collectibles, exploration-heavy maps, lore, and treasure-hunting missions.</p><p>This can support practical business outcomes such as:</p><ul><li><strong>More relevant in-game recommendations</strong></li><li><strong>Better player segmentation and targeted live-ops campaigns</strong></li><li><strong>More balanced matchmaking</strong></li><li><strong>Dynamic difficulty adjustments</strong></li><li><strong>Earlier churn-risk detection</strong></li><li><strong>Higher engagement and retention through more relevant experiences</strong></li></ul><p>The point is not that an eigenvector magically creates a recommendation. Its value is that it exposes the patterns hidden across many related signals, making downstream decisions more informed.</p><h4>A Small Python Example</h4><pre>import pandas as pd<br>from sklearn.decomposition import PCA<br>from sklearn.preprocessing import StandardScaler<br><br>players = pd.DataFrame({<br>    &quot;aggression&quot;: [9, 8, 2, 3, 9, 1],<br>    &quot;exploration&quot;: [2, 3, 9, 8, 1, 9],<br>    &quot;accuracy&quot;: [7, 6, 8, 7, 5, 9],<br>    &quot;movement_speed&quot;: [9, 8, 3, 4, 9, 2],<br>    &quot;loot_usage&quot;: [3, 4, 9, 8, 2, 9],<br>    &quot;deaths&quot;: [8, 7, 2, 3, 9, 1],<br>})<br><br># Scale features so a larger numerical range does not dominate PCA.<br>X_scaled = StandardScaler().fit_transform(players)<br><br>pca = PCA(n_components=2)<br>player_scores = pca.fit_transform(X_scaled)<br><br>print(&quot;Eigenvalues:&quot;)<br>print(pca.explained_variance_)<br>print()<br>print(&quot;Variance explained:&quot;)<br>print(pca.explained_variance_ratio_)<br><br>components = pd.DataFrame(<br>    pca.components_,<br>    columns=players.columns,<br>    index=[&quot;PC1&quot;, &quot;PC2&quot;],<br>)<br>print()<br>print(&quot;Principal directions:&quot;)<br>print(components)<br><br>player_scores_df = pd.DataFrame(<br>    player_scores,<br>    columns=[&quot;PC1_score&quot;, &quot;PC2_score&quot;],<br>)<br>print()<br>print(&quot;Player scores:&quot;)<br>print(player_scores_df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7vyUfiBzrJy9G0KOFaCqxg.png" /></figure><p>Start with Variance explained. Here, PC1 explains 68% of the differences between players and PC2 explains another 18%. Together, these two scores retain 86% of the information in the original six metrics, so they are a useful compact summary.</p><p>Next, read across each row of Principal directions:</p><ul><li><strong>PC1</strong> has positive values for aggression, movement speed, and deaths, and negative values for exploration and loot usage. It represents a spectrum from exploration and loot on one end to fast, aggressive combat on the other.</li><li><strong>PC2</strong> is dominated by accuracy. It represents a separate difference in how precisely players perform, after we account for the first pattern.</li></ul><p>The positive or negative sign is not inherently good or bad. It only tells us which end of that component’s spectrum a player is closer to. PCA can reverse all the signs and still describe the same pattern, so interpret the signs by looking at the feature weights, not in isolation.</p><p>Finally, use Player scores to place individual players on those patterns:</p><ul><li>Player 0 has a high positive PC1 score, so they are closer to the fast-combat end of the first spectrum. A product team could test combat-heavy missions, short time-to-action, or action-oriented recommendations for players with similar scores.</li><li>Player 2 has a strongly negative PC1 score and a high PC2 score. They are closer to the exploration-and-loot end while also showing strong accuracy. A game could test exploration content, collectibles, or precision-oriented challenges for this group.</li></ul><p>In practice, do not make a decision from one player score alone. Use the PCA scores as inputs to clustering, recommendation, retention, or experimentation systems. For example, compare retention and purchase rates for players with similar scores, then test whether tailored content improves those outcomes.</p><h4>Key Takeaway</h4><p>Eigenvectors are not only an abstract linear algebra topic. In PCA, they help uncover the dominant patterns in customer or player behaviour. Eigenvalues tell us which of those patterns matter most.</p><p>For a business, that means less time interpreting scattered metrics and more ability to make decisions based on meaningful behavioural patterns.</p><p>That is the moment eigenvectors stopped feeling abstract to me: they are a way to turn complex data into signals that real teams can use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3dac8d65199" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Physics of Your Product: Understanding the Mathematical Drivers of User Growth]]></title>
            <link>https://pub.towardsai.net/the-physics-of-your-product-understanding-the-mathematical-drivers-of-user-growth-e0b8c0c8677d?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0b8c0c8677d</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[product-analysis]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 12:24:31 GMT</pubDate>
            <atom:updated>2026-03-02T04:55:43.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*122h5dX5YxvXsOKe" /></figure><p>Imagine you are a restaurant owner watching a new regular walk through the door. After their first week of consistent takeout orders, a high-stakes question inevitably crosses your mind: <em>“How much is this customer actually worth to us over the next year?”</em> Will they remain a loyal power-user, or will their order frequency sharply decline after the “honeymoon phase”? At what exact point will their spending plateau?”</p><p>I’ve been fascinated by these exact questions recently. It turns out, predicting a customer’s Lifetime Value or forecasting the saturation point of their spend isn’t just guesswork it’s a problem perfectly solved by curve fitting.</p><p>If you’ve ever tried to learn this on your own, you already know it can feel like an arcane, mathematically dense dark art that takes serious time to decipher and implement.</p><p>That is exactly why I decided to write this guide. I wanted to translate this complex concept into a practical, accessible toolkit. And while it’s an absolute superpower for food delivery analytics, curve fitting is used everywhere to predict the future: pharmacologists use it to model how fast a drug is absorbed into the bloodstream, marketing teams use it to find the point of diminishing returns on ad spend, and engineers use it to forecast exactly when a lithium-ion battery will degrade past the point of usefulness.</p><p>To make this guide as hands-on as possible, I’ve also built an interactive website where you can play with the mathematical parameters and watch the graphs update in real-time. Let’s dive in.</p><p>I have created a Interactive lab to go along with this article which can be found here.</p><p><a href="https://curve-fit-labs.netlify.app/">The Curve Fitting Laboratory: From Linear to Richards &amp; Weibull</a></p><h3>1. Why Curve Fitting?</h3><p>Imagine you just launched a massive user acquisition campaign for a new app or game. Your Daily Active Users (DAU) spike on Day 1, drop sharply by Day 7, and then begin a slow, bumpy descent. Your product manager walks over and asks: <em>“Where does our baseline DAU stabilize in 6 months?”</em></p><p>You can’t just guess, and calculating a simple average of those wildly fluctuating early days will give you a terribly inaccurate forecast. This is exactly where curve fitting shines. By taking the noisy, scattered points of your early DAU data and mathematically finding the function that best “hugs” that trend, you achieve two critical things:</p><ul><li><strong>Reliable Extrapolation:</strong> You can project the curve forward in time to forecast exactly where your user base will bottom out and stabilize, allowing for accurate server and budget planning.</li><li><strong>Deep Interpretability: </strong>Unlike a “black box” machine learning model that just spits out a number, a fitted curve gives you a concrete equation. You can point to specific parameters to explain exactly how fast your users are churning versus how many are sticking around.</li></ul><p>In short, curve fitting cuts through the daily noise of your metrics to reveal the underlying physics of your product’s growth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w_Fkg6V0L2AF6Bcv" /></figure><h3>2. The Curve Catalog</h3><p>Choosing the right “family” of curves is the most important decision you will make in this process. If the underlying physics of your data is exponential, a linear fit will fail spectacularly no matter how much data you feed it.</p><p>To help you choose, let’s break down the 9 essential models, their mathematical parameters, and how tweaking those parameters physically changes the shape of your forecast.</p><h4>Part 1: The Basic Curves</h4><p>These are your standard mathematical workhorses, ideal for simpler relationships or isolated segments of a user’s lifecycle.</p><ol><li><strong>Linear</strong></li></ol><ul><li><strong>Equation:</strong> y = ax + b</li><li><strong>Parameters</strong>: <em>a</em> (Slope): Controls the constant rate of change. A higher <em>a</em> means steeper growth. <em>b</em> (Intercept) The starting value at day zero.</li><li><strong>When to use it: </strong>When your metric is growing at a perfectly steady, constant rate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PWLZ3-UVDkqdl9CrLXkvbQ.png" /></figure><p>2. <strong>Polynomial (Quadratic, Cubic, etc.)</strong></p><ul><li><strong>Equation: </strong>y = ax³ + bx² + cx + d (Cubic)</li><li><strong>Parameters: </strong><em>a</em> (Leading Coefficient): Controls the extreme “tails” of the curve. Because it is cubed, this parameter dominates over time, dictating whether the forecast ultimately shoots up to positive infinity or plunges to negative infinity. <em>b</em> and <em>c,</em> These parameters control the “humps” of the curve the local peaks and valleys where the data temporarily changes direction as well as the inflection point where the curve’s bend flips. <em>d</em> (Intercept), The baseline starting value at day zero.</li><li><strong>When to use it: </strong>Cubics are highly flexible and fantastic for interpolation when your historical data changes direction twice (for example, an initial spike in takeout orders, a mid-month slump, followed by a month-end surge). However, they come with a massive warning label: <strong>do not use them for extrapolation</strong>. Because of that <em>x³</em> multiplier, the moment you try to forecast beyond your observed data window, the curve will violently fly off to unphysical extremes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgit5Wq0rkeJthBk0F_CqQ.png" /></figure><p>3.<strong> Exponential</strong></p><ul><li><strong>Equation: </strong>y = a · e^(bx)</li><li><strong>Parameters:<em> </em></strong><em>a </em>Your starting baseline. <em>b</em> (Growth/Decay Rate) If <em>b</em> is positive, the curve shoots upwards toward infinity (viral growth). If <em>b</em> is negative, it rapidly decays toward zero (radioactive decay).</li><li><strong>When to use it: </strong>Modeling compound growth, like early viral loops or unchecked organic acquisition.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kUao4Dj3uxH3Fbj6C2zeHg.png" /></figure><p>4.<strong> Logarithimic</strong></p><ul><li><strong>Equation: </strong>y = a + b · ln(x)</li><li><strong>Parameters:<em> </em></strong><em>a</em> the baseline starting point.<em> b </em>The rate of change. A higher <em>b </em>means the curve rises more before flattening.</li><li><strong>When to use it: </strong>Modelling compound growth, like early viral loops or unchecked organic acquisition.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZIoZlQ90lZyd9AZWl0wfw.png" /></figure><p>5. <strong>Power Law</strong></p><ul><li><strong>Equation: </strong>y = a · x^b</li><li><strong>Parameters:<em> </em></strong><em>a</em> The initial scale or starting point. <em>b</em> (Shape parameter) Dictates the speed of decay or growth.</li><li><strong>When to use it: </strong>Modelling compound growth, like early viral loops or unchecked organic acquisition.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTeiCMnA-ncZ52VI-VM0XQ.png" /></figure><h4><strong>Part 2: The S-Shaped (Sigmoidal) Curves</strong></h4><p>In business, nothing grows forever. Things start slow, accelerate, and eventually plateau. Sigmoidal curves capture this perfectly and are the ultimate tool for forecasting Cumulative Revenue, LTV, and Market Penetration.</p><p>6. <strong>Logistic</strong></p><p><strong>Equation:</strong> <em>y = L / (1 + exp(-k * (x — x0)))</em></p><p><strong>Parameters:</strong> <em>L</em> (Ceiling) The maximum possible value (the plateau). <em>k</em> (Steepness) How fast the metric grows during its rapid growth phase. x0 (Midpoint) The exact time or day when growth hits its absolute peak before slowing down.</p><p><strong>When to use it:</strong> When you expect symmetric saturation. The journey from 0% to 50% looks exactly like the journey from 50% to 100%. Great for standard market penetration or balanced LTV accumulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-_YfJSxKMlHZQ5_7CDXNA.png" /></figure><p>7. <strong>Gompertz</strong></p><p><strong>Equation:</strong> <em>y = a * exp(-b * exp(-c * x))</em></p><p><strong>Parameters:</strong> <em>a</em> (Asymptote) The maximum limit or ceiling. <em>b</em> (Displacement) Sets the starting point along the x-axis. <em>c</em> (Growth Rate) Controls how quickly the curve approaches the ceiling.</p><p><strong>When to use it:</strong> When growth is asymmetric. It models rapid early growth that takes a very, very long time to finally reach the ceiling. Perfect for modeling scenarios where early adopters spend fast, but the late majority trickles in slowly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CjjFD7qvfzsXuXs0PZ45Ng.png" /></figure><p><strong>8. Richards (Generalized Logistic)</strong></p><p><strong>Equation:</strong> <em>y = A + (K — A) / (1 + Q * exp(-B * x))^(1 / v)</em></p><p><strong>Parameters:</strong> <em>A</em> (Lower Asymptote) The starting baseline (e.g., Day 1 spend). <em>K</em> (Ceiling) The upper limit or maximum LTV. <em>B</em> (Velocity) The base growth rate. <em>Q</em> (Initial state) Shifts the curve near the origin. <em>v</em> (Asymmetry) The magic parameter that dictates whether the fastest growth happens early in the lifecycle (whales) or late in the lifecycle (subscriptions).</p><p><strong>When to use it:</strong> The “Master Curve” for LTV forecasting. By tweaking the ‘v’ parameter, the Richards curve can literally shape-shift into a Logistic or Gompertz curve, making it the most robust and adaptable tool in your catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVYmOV8szdF8rtDHgReRkg.png" /></figure><p><strong>9. Weibull</strong></p><p><strong>Equation:</strong> <em>y = c — (c — a) * exp(-(x / b)^d)</em></p><p><strong>Parameters:</strong> <em>c</em> (Ceiling) The maximum value. <em>a </em>(Starting Value) The baseline or Day 0 value. <em>b </em>(Scale) Stretches or compresses the curve along the timeline. <em>d</em> (Shape) The critical parameter that controls how the curve bends and dictates the saturation speed.</p><p><strong>When to use it:</strong> Highly flexible. Originally used in reliability engineering to predict machine failure, it’s excellent for modeling user churn over time or highly variable retention behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCIqaqN4q2-mjHVTYBOorQ.png" /></figure><h4>3. The Optimization Engine: Finding the “Best Fit”</h4><p>Now that you have selected the perfect mathematical curve from the catalog, how do you actually find the specific parameters (the <em>a</em>, <em>b</em>, <em>c</em>, etc.) that make the curve “hug” your data perfectly?</p><p>This is an optimization problem. The goal of an optimizer is to guess a set of parameters, measure how far off the resulting curve is from your actual data (usually calculating the <strong>Sum of Squared Residuals</strong>), and then intelligently adjust those parameters until the error is minimized.</p><p>In Python, the workhorse for this is scipy.optimize.curve_fit. When you run this function, you are handing your data over to one of three powerful algorithms under the hood. Choosing the right one is the secret to avoiding errors and getting a reliable forecast.</p><p><strong>1. Levenberg-Marquardt (LM)</strong></p><ul><li><strong>What it is:</strong> The default algorithm in curve_fit (if you don&#39;t specify boundaries). It acts like a hybrid between two other mathematical methods (Gradient Descent and Gauss-Newton), making it incredibly fast and efficient.</li><li><strong>The Catch:</strong> It is strictly for <em>unconstrained</em> problems. You cannot tell LM that a parameter “must be greater than zero.”</li><li><strong>When to use it:</strong> When your data is clean, the mathematical model is simple (like a basic Exponential or Polynomial curve), and you don’t care if the algorithm explores negative parameter values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VxXJPll3Q7pMrceQepXFrA.png" /></figure><p><strong>2. Trust Region Reflective (TRF)</strong></p><ul><li><strong>What it is:</strong> A robust, large-scale optimizer that solves the problem by defining a “trust region” around its current guess and systematically exploring within that boundary.</li><li><strong>The Catch:</strong> It requires you to set physical bounds for your parameters.</li><li><strong>When to use it:</strong> <strong>This is your daily driver for business metrics.</strong> If you are forecasting LTV, you know the ceiling ($K$) cannot be infinite, and the starting value ($A$) cannot be negative. TRF allows you to pass a bounds argument to enforce these physical realities, preventing the model from outputting mathematically valid but logically absurd results.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P4bTVsRXGn6dxOQ7sOXwuA.png" /></figure><p><strong>3. Dogbox</strong></p><ul><li><strong>What it is:</strong> Another bounded optimization algorithm. It operates similarly to TRF but uses a slightly different mathematical approach (operating within a “box” of constraints).</li><li><strong>The Catch:</strong> It struggles with very large, complex datasets or highly correlated parameters.</li><li><strong>When to use it:</strong> When you have a small dataset with strict boundaries and TRF is failing to converge or getting stuck in a local minimum. It’s a great fallback option.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DZ8xqyCI3vzkGic-JGMpiw.png" /></figure><h4>How to Apply Them in Practice</h4><p>Using these in Python is as simple as passing the method argument to your curve_fit function. Here is the mental checklist:</p><ol><li>Need to restrict your LTV to positive numbers? Create your boundaries and use <strong>TRF</strong>.</li><li>Doing a quick, simple physics calculation with no bounds? Let it default to <strong>LM</strong>.</li><li>TRF throwing an error on your bounded problem? Swap the method to <strong>Dogbox</strong> and try again.</li></ol><h4>4. Hands-On: Predicting Cumulative App Users</h4><p>Let’s put the theory into practice. Imagine you launched a new community app 60 days ago. You have been tracking the <strong>cumulative registered users</strong> every day. The growth was explosive in month one, but it is starting to slow down. Your investors want to know: <em>“What is the natural ceiling of our user base with this current product, and where will we be at Day 365?”</em></p><p>Because cumulative users follow a classic “slow start -&gt; viral spike -&gt; market saturation” lifecycle, this is a textbook use case for an S-shaped curve. We will use the <strong>Richards (Generalized Logistic)</strong> curve and the <strong>TRF</strong> optimizer to ensure our forecast stays within logical physical boundaries.</p><p><strong>The Python Implementation</strong></p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy.optimize import curve_fit<br><br># 1. Generate some synthetic &quot;observed&quot; data (First 60 days of launch)<br>np.random.seed(42)<br>days_observed = np.arange(1, 61)<br># Simulating a viral launch that is starting to slow down<br>true_curve = 50000 / (1 + 15 * np.exp(-0.1 * days_observed))**(1/0.5)<br>users_observed = true_curve + np.random.normal(0, 500, len(days_observed))<br><br># 2. Define the Richards Curve<br>def richards_curve(x, A, K, B, Q, v):<br>    &quot;&quot;&quot;<br>    A: Lower asymptote (Starting baseline)<br>    K: Upper asymptote (Maximum market saturation)<br>    B: Growth rate / Velocity<br>    Q: Initial state offset<br>    v: Asymmetry parameter<br>    &quot;&quot;&quot;<br>    return A + (K - A) / (1 + Q * np.exp(-B * x))**(1 / v)<br><br># 3. Set Initial Guesses (p0) and Physical Bounds<br># Guessing: Start at 0, Ceiling at 100k, 10% growth rate, offset 10, v=1<br>initial_guesses = [0.0, 100000.0, 0.1, 10.0, 1.0]<br><br># Bounds: [Lower bounds for (A, K, B, Q, v)], [Upper bounds]<br># We know users can&#39;t be negative, and market size can&#39;t exceed e.g., 1 million<br>lower_bounds = [0.0, max(users_observed), 0.001, 0.1, 0.01]<br>upper_bounds = [1000.0, 1000000.0, 1.0, 100.0, 5.0]<br><br># 4. Fit the curve using Trust Region Reflective (TRF)<br>popt, pcov = curve_fit(<br>    richards_curve, <br>    days_observed, <br>    users_observed, <br>    p0=initial_guesses, <br>    bounds=(lower_bounds, upper_bounds), <br>    method=&#39;trf&#39;<br>)<br><br># Extract the fitted parameters<br>A_fit, K_fit, B_fit, Q_fit, v_fit = popt<br>print(f&quot;Predicted Market Ceiling (K): {int(K_fit):,} users&quot;)<br><br># 5. Extrapolate to Day 365<br>days_future = np.arange(1, 366)<br>users_forecast = richards_curve(days_future, *popt)<br><br># Plotting the results<br>plt.figure(figsize=(10, 6))<br>plt.scatter(days_observed, users_observed, label=&#39;Observed Users (Days 1-60)&#39;, color=&#39;black&#39;, s=15)<br>plt.plot(days_future, users_forecast, label=&#39;Richards Forecast (Day 365)&#39;, color=&#39;red&#39;, linestyle=&#39;--&#39;, linewidth=2)<br>plt.axhline(K_fit, color=&#39;blue&#39;, linestyle=&#39;:&#39;, label=f&#39;Saturated Ceiling: {int(K_fit):,}&#39;)<br>plt.title(&quot;Cumulative App Users Forecast&quot;)<br>plt.xlabel(&quot;Days Since Launch&quot;)<br>plt.ylabel(&quot;Total Registered Users&quot;)<br>plt.legend()<br>plt.grid(True, alpha=0.3)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qLVqAAMkWo0xzbKFVE5NMA.png" /></figure><h4>5. Key Takeaways</h4><p>You now have the theory, the equations, and the Python code to start predicting the future of your product’s metrics. However, as the saying goes: <em>“All models are wrong, but some are useful.”</em> Before you push your first curve-fitting script into production to forecast a multi-million dollar ad budget, keep these battle-tested rules in mind to avoid the most common traps.</p><p><strong>1. Your Initial Guesses (</strong><strong>p0) Matter More Than the Algorithm</strong></p><p>Optimisation algorithms are essentially blindfolded hikers trying to find the lowest point in a valley (the point of minimum error). If you drop them in the wrong valley, they will find the bottom of <em>that</em> valley, but entirely miss the deeper valley right next door. This is known as getting stuck in a <strong>local minimum</strong>.</p><p>If you are forecasting user LTV, and you know the industry average is around $15, do not leave the initial guess at 0. Set your p0 starting values close to reality. Spend 60 seconds eyeballing a scatter plot of your data before you write any code—it will save you hours of debugging.</p><p><strong>2. The Math Doesn’t Know Your Business (Always Set Bounds)</strong></p><p>An unconstrained optimization algorithm will happily tell you that your user base will drop to negative 5,000 next month, or that your app will reach 14 billion users by Tuesday. The math checks out, but the physics do not. Always use the <strong>TRF</strong> or <strong>Dogbox</strong> methods and pass explicit lower and upper boundaries to enforce physical and business realities (e.g., revenue cannot be negative, DAU cannot exceed the population of Earth).</p><p><strong>3. Beware of Parameter Bloat (Overfitting)</strong></p><p>The Richards curve is a masterpiece, but it requires calculating 5 separate parameters. If you only have 7 days of launch data, feeding it into a 5-parameter model will result in massive overfitting. The curve will twist itself into knots to hit every single noisy data point perfectly, resulting in an extrapolation that looks like a rollercoaster.</p><ul><li><strong>The Rule of Thumb:</strong> If you have very little data, use a simpler model (like a 3-parameter Logistic curve or a basic Power Law). Graduate to the Richards curve only when your data matures.</li></ul><p><strong>4. Visualize Your Residuals</strong></p><p>The R-squared metric is not enough to tell you if your fit is good. Always plot your residuals (the actual data points minus the curve’s predicted points).</p><ul><li><strong>Good Fit:</strong> The residuals look like random static or white noise scattered around zero.</li><li><strong>Bad Fit:</strong> The residuals follow a clear pattern (like a wave or a smile). If you see a pattern, it means your mathematical model is systematically missing a trend in the data. You chose the wrong curve catalog!</li></ul><p><strong>5. Extrapolation is a Hypothesis, Not a Prophecy</strong></p><p>Curve fitting is exceptionally good at interpolation (understanding the data you already have) and making short-to-medium-term forecasts. However, projecting a 30-day trend out to 5 years is incredibly risky. Use these models to make informed, directional business decisions — like when to cut ad spend or when to upgrade servers — but never treat them as absolute gospel.</p><h3>Conclusion</h3><p>Curve fitting is an incredible capability to add to your toolkit. It bridges the gap between raw, messy data and human-readable, actionable business strategy. By moving away from “black box” models and embracing interpretable parameters, you can not only predict <em>where</em> your metrics are going, but explain <em>why</em> they are going there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0b8c0c8677d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/the-physics-of-your-product-understanding-the-mathematical-drivers-of-user-growth-e0b8c0c8677d">The Physics of Your Product: Understanding the Mathematical Drivers of User Growth</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas 3.0 is here and here are some of the significant changes you should care.]]></title>
            <link>https://siddharth1.medium.com/pandas-3-0-is-here-and-here-are-some-of-the-significant-changes-you-should-care-aec887cc69dd?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/aec887cc69dd</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Wed, 28 Jan 2026 05:39:04 GMT</pubDate>
            <atom:updated>2026-01-28T05:39:04.151Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pTPOp-aSJh2wCY9H" /><figcaption>Photo by <a href="https://unsplash.com/@charlesdeluvio?utm_source=medium&amp;utm_medium=referral">charlesdeluvio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Key Improvements</h4><p><strong>Dedicated String Data Type by Default</strong></p><ol><li>Columns with text are now inferred as a new str dtype instead of generic object dtype.</li><li>This brings better type safety and more efficient memory usage.</li></ol><pre># Before (object dtype)<br>ser = pd.Series([&quot;a&quot;, &quot;b&quot;])<br>print(ser.dtype)  # object<br><br># After (new default)<br>ser = pd.Series([&quot;a&quot;, &quot;b&quot;])<br>print(ser.dtype)  # str</pre><p><strong>Consistent copy/view behaviour with Copy-on-Write (CoW)</strong></p><ol><li>Copy-on-Write is now the <strong>default and only mode</strong> in pandas 3.0. All indexing and selection operations behave <strong>as if they return a copy</strong>, making behavior more predictable and eliminating silent mutations.</li></ol><p>2. The most impactfull change is that <strong>chained assignment will no longer work</strong>. As a result, the SettingWithCopyWarning is also removed and defensive .copy() calls to silence the warning are no longer needed.</p><pre># Old behavior (pandas &lt; 3.0) — chained assignment<br>df = pd.DataFrame(<br>    {<br>        &quot;user_id&quot;: [101, 102, 103, 104],<br>        &quot;score&quot;: [45, 82, 91, 60],<br>    }<br>)<br><br>df[&quot;score&quot;][df[&quot;score&quot;] &gt;= 80] = 100<br># Might update df, or might fail silently (unpredictable)<br><br># New behavior (pandas 3.0) — explicit, single-step assignment<br>df.loc[df[&quot;score&quot;] &gt;= 80, &quot;score&quot;] = 100<br># Always works and is unambiguous</pre><p><strong>New </strong><strong>pd.col() Syntax</strong></p><ol><li>Pandas 3.0 introduces <strong>pd.col()</strong>, a simplified and more readable way to reference columns when creating expressions in methods like DataFrame.assign().</li><li>It removes the need for lambda functions and makes column transformations easier to scan and compose.</li></ol><pre># Before — using lambda<br>df.assign(<br>    total_price=lambda x: x[&quot;unit_price&quot;] * x[&quot;quantity&quot;]<br>)<br><br># After — using pd.col()<br>df.assign(<br>    total_price=pd.col(&quot;unit_price&quot;) * pd.col(&quot;quantity&quot;)<br>)</pre><h3>Conclusion</h3><p>Pandas 3.0 is a <strong>foundational release</strong> focused on consistency, clarity, and long-term performance.<br> The new string dtype, Copy-on-Write semantics, and expression-style APIs like pd.col() remove long-standing ambiguities and footguns, while encouraging more explicit and readable code.</p><p>While some updates require small code changes, they result in <strong>more predictable behavior, cleaner patterns, and better performance</strong> — making pandas easier to reason about at scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aec887cc69dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Deterministic AI Agents with Function Calling]]></title>
            <link>https://siddharth1.medium.com/building-deterministic-ai-agents-with-function-calling-c30107a0ac3f?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/c30107a0ac3f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Mon, 03 Feb 2025 12:24:21 GMT</pubDate>
            <atom:updated>2025-02-03T12:24:21.293Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FDQF6zMF3xwqgKFLAzyYg.jpeg" /></figure><h3>Introduction</h3><p>Large language models (LLMs) have transformed many industries by offering sophisticated language understanding and generation capabilities. However, when integrating these models into downstream processes — such as healthcare diagnostics, financial data analysis, or customer support systems — deterministic, structured outputs are often essential. Traditional prompt engineering can be used to control model responses, but it may not always yield reliable or consistent results. <strong>Function calling</strong> offers a robust alternative, enabling the creation of deterministic AI agents that can interact with external tools to perform specific tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFrVC1byPXUcW6rom02bxQ.jpeg" /></figure><h3>What is Function Calling?</h3><p><strong>Function calling</strong> is a technique that extends the capabilities of an LLM by defining and invoking external tools or functions. These tools can be pre-defined functions that the model can call to retrieve structured data or perform computations beyond its inherent abilities. This approach helps in ensuring that the output is not only relevant but also formatted in a deterministic way, which is critical for many downstream applications.</p><h3>How Function Calling Works</h3><ol><li><strong>Input Provision:</strong> The language model is provided with both a user prompt and a set of tools.</li><li><strong>Decision Making:</strong> Based on the prompt, the LLM determines whether to use one of the provided tools.</li><li><strong>Parameter Extraction:</strong> If a tool is selected, the LLM extracts the necessary parameters from the prompt.</li><li><strong>Tool Invocation:</strong> The extracted parameters are passed to the external tool or API, which executes the desired function.</li><li><strong>Response Processing:</strong> The LLM receives and processes the tool’s response, integrating it into the final output.</li></ol><h3>Implementation Example: Nutritional Information Lookup</h3><h3>Step 1: Creating the Mock Function</h3><p>First, we create a Python function that simulates looking up nutritional information from a database. This function returns random nutritional facts for known food items and handles cases where the food item is not available.</p><pre>import random <br>def get_nutrition_info(food_item):<br>    items = [<br>        &quot;apple&quot;, &quot;banana&quot;, &quot;orange&quot;, &quot;strawberry&quot;, &quot;blueberry&quot;,<br>        &quot;spinach&quot;, &quot;kale&quot;, &quot;lettuce&quot;, &quot;broccoli&quot;, &quot;cauliflower&quot;,<br>        &quot;chicken&quot;, &quot;beef&quot;, &quot;pork&quot;, &quot;salmon&quot;, &quot;tuna&quot;,<br>        &quot;rice&quot;, &quot;pasta&quot;, &quot;quinoa&quot;, &quot;bread&quot;, &quot;oats&quot;,<br>        &quot;milk&quot;, &quot;cheese&quot;, &quot;yogurt&quot;, &quot;butter&quot;, &quot;cream&quot;,<br>        &quot;olive oil&quot;, &quot;coconut oil&quot;, &quot;avocado&quot;, &quot;almonds&quot;, &quot;walnuts&quot;,<br>        &quot;tomato&quot;, &quot;cucumber&quot;, &quot;carrot&quot;, &quot;onion&quot;, &quot;garlic&quot;,<br>        &quot;potato&quot;, &quot;sweet potato&quot;, &quot;pumpkin&quot;, &quot;corn&quot;, &quot;peas&quot;,<br>        &quot;lemon&quot;, &quot;lime&quot;, &quot;grapefruit&quot;, &quot;mango&quot;, &quot;pineapple&quot;,<br>        &quot;chocolate&quot;, &quot;honey&quot;, &quot;maple syrup&quot;, &quot;sugar&quot;, &quot;salt&quot;,<br>        &quot;pepper&quot;, &quot;cinnamon&quot;, &quot;cumin&quot;, &quot;paprika&quot;, &quot;oregano&quot;,<br>        &quot;eggs&quot;, &quot;tofu&quot;, &quot;tempeh&quot;, &quot;lentils&quot;, &quot;chickpeas&quot;,<br>        &quot;mushroom&quot;, &quot;zucchini&quot;, &quot;eggplant&quot;, &quot;bell pepper&quot;, &quot;jalapeno&quot;,<br>        &quot;coconut&quot;, &quot;peanut butter&quot;, &quot;almond butter&quot;, &quot;jam&quot;, &quot;jelly&quot;,<br>        &quot;soy sauce&quot;, &quot;vinegar&quot;, &quot;mustard&quot;, &quot;ketchup&quot;, &quot;mayonnaise&quot;,<br>        &quot;bacon&quot;, &quot;ham&quot;, &quot;sausage&quot;, &quot;turkey&quot;, &quot;duck&quot;,<br>        &quot;shrimp&quot;, &quot;crab&quot;, &quot;lobster&quot;, &quot;squid&quot;, &quot;octopus&quot;,<br>        &quot;basil&quot;, &quot;rosemary&quot;, &quot;thyme&quot;, &quot;sage&quot;, &quot;parsley&quot;,<br>        &quot;ginger&quot;, &quot;turmeric&quot;, &quot;cardamom&quot;, &quot;nutmeg&quot;, &quot;cloves&quot;,<br>        &quot;watermelon&quot;, &quot;cantaloupe&quot;, &quot;honeydew&quot;, &quot;grapes&quot;, &quot;peach&quot;<br>        ]   <br><br>    if food_item.lower() in items: <br>        return {<br>            &quot;item_name&quot;: food_item,<br>            &quot;calories&quot;: random.randint(50, 300),<br>            &quot;carbohydrates&quot;: random.randint(0, 30),<br>            &quot;fiber&quot;: random.randint(0, 5),<br>            &quot;sugar&quot;: random.randint(0, 20),<br>            &quot;protein&quot;: round(random.uniform(0, 25), 1),<br>            &quot;fat&quot;: round(random.uniform(0, 15), 1),<br>        }<br>    else:<br>        return {&quot;response&quot;:&quot;Food Item Not Available&quot;}</pre><h3>Step 2: Defining the Tool Schema</h3><p>Next, we define a tool schema that tells the LLM how to use this function. The schema includes a name, a description, and an input schema that specifies the expected parameters.</p><pre>nutrition_extraction_tool = {<br>    &quot;name&quot;: &quot;nutritional_info&quot;,<br>    &quot;description&quot;: &quot;Search for a food item by the item name and return the nutritional information of that item&quot;,<br>    &quot;input_schema&quot;: {<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>            &quot;item&quot;: {<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;description&quot;: &quot;The name of the food item to be extracted&quot;,<br>            }<br>        },<br>    },<br>}</pre><h3>Step 3: Integrating with the Language Model</h3><p>We now simulate a conversation where the LLM uses our defined tool. For example, if a user asks, “How many calories does an apple have?”, the LLM will decide to use the nutritional_info tool to extract the food item from the prompt.</p><pre>response = client.messages.create(<br>    model=&#39;claude-3-haiku&#39;, <br>    messages=[<br>        {<br>            &quot;role&quot;:&quot;user&quot;, <br>            &quot;content&quot;: &quot;How much calories does Apple have&quot;<br>        }<br>    ], <br>    max_tokens=1000, <br>    tools=[nutrition_extraction_tool]<br>)<br><br>response</pre><pre>Message(id=&#39;msg_vrtx_01AFNha3XWcfmxv1pwEA59EN&#39;, content=[ToolUseBlock(id=&#39;toolu_vrtx_017iPLTE3wTD1YmNzHNzPrCK&#39;, input={&#39;item&#39;: &#39;Apple&#39;}, name=&#39;nutritional_info&#39;, type=&#39;tool_use&#39;)], model=&#39;claude-3-haiku-20240307&#39;, role=&#39;assistant&#39;, stop_reason=&#39;tool_use&#39;, stop_sequence=None, type=&#39;message&#39;, usage=Usage(input_tokens=352, output_tokens=54))</pre><p>Check if the tool was used and then pass the parameters to the function</p><pre>if response.stop_reason == &#39;tool_use&#39;: <br>    tool_use = response.content[-1] <br>    tool_name = tool_use.name <br>    tool_input = tool_use.input <br><br>    result = get_nutrition_info(tool_input) <br>    print(result)<br><br>    <br>{&#39;item_name&#39;: &#39;Apple&#39;, &#39;calories&#39;: 164, &#39;carbohydrates&#39;: 9, &#39;fiber&#39;: 1, &#39;sugar&#39;: 15, &#39;protein&#39;: 23.2, &#39;fat&#39;: 14.3}</pre><h3>Conclusion</h3><p>Function calling provides a powerful method to extend the capabilities of large language models by allowing deterministic, structured output. By defining clear tool schemas and integrating them into the model’s workflow, developers can build AI agents that reliably interact with external systems. Whether you’re designing a nutritional information lookup, a financial data retrieval system, or a customer support assistant, function calling offers a flexible and robust solution to meet the needs of modern applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c30107a0ac3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building AI-Powered Applications with CockroachDB Vector Search: From Theory to Practice]]></title>
            <link>https://pub.towardsai.net/building-ai-powered-applications-with-cockroachdb-vector-search-from-theory-to-practice-baa891ef9a08?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/baa891ef9a08</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[cockroachdb]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Sat, 09 Nov 2024 15:38:36 GMT</pubDate>
            <atom:updated>2024-11-12T10:11:14.296Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mL6twIP3m2lmPEdGsqh9lA.png" /></figure><h4>What are Vector Databases ?</h4><p>In today’s AI-driven landscape, vector databases have emerged as a critical technology for managing and querying high-dimensional data in ways that mirror human understanding. These specialized database systems store data as mathematical vectors — numerical representations that capture the semantic essence of text, images, audio, or other content — enabling applications to perform similarity searches based on meaning rather than exact matches. Unlike traditional databases that excel at storing and retrieving structured data, vector databases are specifically optimized for calculating distances between vectors, making them indispensable for modern applications like semantic search, recommendation systems, image recognition, and natural language processing. Their ability to efficiently process and compare millions of high-dimensional vectors has made them a fundamental component of AI infrastructure, allowing organizations to build more intelligent, context-aware applications that can understand and process information in ways that were previously impossible with conventional database systems.</p><h4>What is CockroachDB ?</h4><p>CockroachDB is a distributed SQL database designed for resilience and scalability. At its core, it operates on a transactional and strongly-consistent key-value store, offering horizontal scalability and robust ACID compliance. One of its defining features is its ability to survive various failures with minimal disruption while maintaining data consistency across distributed environments. This is achieved through automated sharding, multi-region deployments, and its implementation of the Raft consensus algorithm. The database provides a familiar SQL API, making it particularly accessible for teams already working with PostgreSQL, while its cloud-native architecture makes it ideal for globally distributed applications that require both scale and transactional integrity.</p><h4>Why use CockroachDB as Vector Database ?</h4><p>The recent introduction of Vector Search capabilities in CockroachDB 24.2 has positioned it as a compelling solution for AI-driven applications and vector similarity search use cases. By implementing pgvector compatibility, CockroachDB now offers efficient storage and querying of high-dimensional vectors within its distributed architecture. This integration eliminates the need for a separate vector database while leveraging CockroachDB’s existing strengths in high availability and horizontal scalability. The ability to parallelize similarity searches across multiple nodes makes it particularly effective for large-scale vector operations in applications like semantic search, recommendation systems, and natural language processing tasks. While specialized vector databases might offer more optimized performance for pure vector search use cases, CockroachDB’s solution is particularly attractive for organizations looking to consolidate their infrastructure while maintaining robust vector search capabilities alongside their operational data.</p><h4>CockroachDB’s Vector Capabilities ?</h4><p>CockroachDB’s vector extension, introduced in version 24.2, brings comprehensive vector processing capabilities to its distributed SQL platform through a VECTOR data type that handles fixed-length arrays of floating-point numbers. This pgvector-compatible implementation supports essential distance metrics including cosine distance (&lt;=&gt;), Euclidean distance (&lt;-&gt;), and negative inner product (&lt;#&gt;), making it suitable for various AI applications. The cosine similarity metric excels in semantic search by comparing directional similarity regardless of magnitude, while Euclidean distance provides intuitive spatial measurements, and the L1 (Manhattan) distance offers an alternative for cases where absolute differences between vector components are more relevant.</p><h4>Implementing CockroachDB vector search using CRDB Serverless.</h4><p>Before implementing the vector database, we will need to create a CRDB Serverless cluster the process is very simple. <br><strong>1.</strong> Login/Signup to: <a href="https://cockroachlabs.cloud/">https://cockroachlabs.cloud</a></p><p><strong>2.</strong> Click on Create Cluster button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlgKVDWI6ilKo6yzk4CCjQ.png" /></figure><p><strong>3.</strong> Select Basic Plan</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D_1I9on8LhfYrDm7sAiB7g.png" /></figure><p><strong>4.</strong> Select Respective region</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IkJYQDKA8OtjNegkXNZL6A.png" /></figure><p><strong>5.</strong> Select Capacity</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8mwcAJcGriOD5WrJFNP2A.png" /></figure><p><strong>7.</strong> Select Appropriate Cluster Name</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wHIq8hK1omFMI79F21UP-A.png" /></figure><p>8. Wait for the cluster to be created then create a appropriate SQL user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbzhbP-ykAet-7Ng81Nbsw.png" /></figure><p>9. Select the connection option as python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*DmsrDIIv9XGTxNdTcB4kVA.png" /></figure><p>Post that, copy the Download CA Cert command based on your operating system and paste it in terminal, If you are using windows make sure you paste it in PowerShell.</p><p>10. For Connecting we will be using SQL Alchemy. Select the same in dropdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*bTEAcMomn3_LeYRgggcFjw.png" /></figure><h4>Testing the connection in python</h4><ol><li>Install latest version of SQL alchemy package. (When I tested I was running python 3.11 so if error persists with higher versions try downgrading to python 3.11)</li></ol><pre>pip install sqlalchemy-cockroachdb <br>pip install pandas<br>pip install psycopg2-binary <br></pre><p>2. Test the connection</p><p>To copy the connection string copy everything user export DATABASE_URL which is in quotes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*GaUAgFgEYG_jx_mxx_j3mw.png" /></figure><pre>import os <br>os.environ[&#39;DATABASE_URL&#39;] = &#39;&lt;Your Connection String&gt;&#39;<br><br>from sqlalchemy import create_engine, text <br>engine = create_engine(os.environ[&#39;DATABASE_URL&#39;]) <br>conn = engine.connect() <br><br>response = conn.execute(text(&#39;SELECT now()&#39;)).fetchall()<br>print(response)</pre><pre>[(datetime.datetime(2024, 11, 6, 7, 28, 50, 688678, tzinfo=datetime.timezone.utc),)]</pre><h4>Testing a very simple vector search process.</h4><p>We will create a dummy database for simple cloths dataset.</p><p>And to test we will create a sample clothing item and test the similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*E-EOPrOQ2llF_iIpgMsQfA.png" /></figure><ol><li>Create a Table to store the data</li></ol><pre>## Create a table with vector column <br>create_table_query = text(&quot;&quot;&quot;<br>    CREATE TABLE IF NOT EXISTS product_embeddings (<br>        id SERIAL PRIMARY KEY,<br>        name TEXT,<br>        embedding VECTOR(3) -- Using 3 dimensional vectors for simplicity<br>    )<br>&quot;&quot;&quot;)<br><br>with engine.connect() as conn:<br>    conn.execute(create_table_query)<br>    conn.commit()</pre><p>2. Insert Data</p><pre># Insert sample products with embeddings <br>insert_query = text(&quot;&quot;&quot;<br>    INSERT INTO product_embeddings (name, embedding) VALUES<br>        (&#39;Red Shirt&#39;,   &#39;[1.0, 0.0, 0.0]&#39;),<br>        (&#39;Blue Shirt&#39;,  &#39;[0.9, 0.1, 0.0]&#39;),<br>        (&#39;Green Pants&#39;, &#39;[0.0, 1.0, 0.0]&#39;),<br>        (&#39;Blue Pants&#39;,  &#39;[0.0, 0.9, 0.1]&#39;)<br>&quot;&quot;&quot;)<br><br>with engine.connect() as conn:<br>    conn.execute(insert_query)<br>    conn.commit()</pre><p>3. Perform Similarity Search</p><pre># Perform vector similarity search<br>search_query = text(&quot;&quot;&quot;<br>    SELECT <br>        name, <br>        embedding, <br>        1 - cosine_distance(embedding, &#39;[1.0,0.0,0.0]&#39;) as similarity <br>    FROM product_embeddings <br>    ORDER BY 1 - cosine_distance(embedding, &#39;[1.0,0.0,0.0]&#39;)<br>&quot;&quot;&quot;)<br><br>with engine.connect() as conn:<br>    result = conn.execute(search_query)<br><br>result.fetchall()<br><br></pre><pre>[(&#39;Green Pants&#39;, &#39;[0,1,0]&#39;, 0.0),<br> (&#39;Blue Pants&#39;, &#39;[0,0.9,0.1]&#39;, 0.0),<br> (&#39;Blue Shirt&#39;, &#39;[0.9,0.1,0]&#39;, 0.9938837488013375),<br> (&#39;Red Shirt&#39;, &#39;[1,0,0]&#39;, 1.0)]</pre><p>This was a simple illustration now let’s understand how to use vector databases to solve real world problems.</p><h4>Smart Content Search: Building an AI-Powered Marketing Library That Understands Context, Not Just Keywords</h4><p><strong>Problem Statement</strong><br>Marketing teams waste countless hours searching through past campaigns, often missing valuable content because traditional search tools only match exact keywords. When a marketer searches for “professional workspace campaign,” they might miss highly relevant content labeled as “office environment promotion” or “corporate culture initiative.” This keyword limitation means:</p><p>1. Lost Time: Teams spend 30–40% of their time searching for existing content and campaign references<br>2. Duplicate Work: Similar campaigns are recreated because existing ones aren’t found<br>3. Missed Opportunities: Valuable insights from past campaigns remain undiscovered<br>4. Inconsistent Messaging: Teams can’t easily find and reference similar audience targeting strategies</p><p>We’re solving this by building a vector database that:<br><em>- Understands the meaning behind marketing content, not just keywords<br>- Finds similar campaigns based on context and audience targeting<br>- Connects related content even when they use different terminology<br>- Makes our entire marketing library searchable by concept and intent</em></p><p><strong>Simply put:</strong> We’re transforming our marketing database from a basic filing cabinet into an intelligent library that understands what our content means, not just what it says.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*OC8qod4-nTE4nirayxAoXg.png" /><figcaption>Content Search Illustration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*Z3oD9Rgv3MtEgTiNRhe2Qg.png" /><figcaption>Audience Search Illustration</figcaption></figure><p>Step 1 : Necessary Libraries Installation</p><p>Sentence Transformers are a type of machine learning model designed to transform sentences into high-dimensional vectors, capturing their semantic meaning. These vectors can be used for various natural language processing tasks, such as semantic search, clustering, and classification. The sentence-transformers library provides pre-trained models that can be easily integrated into your projects. To install it, you can use the following command:</p><pre>pip install sentence-transformers</pre><p>Step 2: Import Necessary Libraries</p><pre>import os <br>from datetime import datetime <br>import numpy as np <br>from sentence_transformers import SentenceTransformer <br>from sqlalchemy import create_engine, text <br>from typing import Dict, List <br>from tqdm import tqdm <br>engine = create_engine(os.environ[&#39;DATABASE_URL&#39;])</pre><p>Instantiating the SentenceTransformer model from sbert.net</p><pre>model = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)<br>model</pre><pre><br><br>SentenceTransformer(<br>  (0): Transformer({&#39;max_seq_length&#39;: 256, &#39;do_lower_case&#39;: False}) with Transformer model: BertModel <br>  (1): Pooling({&#39;word_embedding_dimension&#39;: 384, &#39;pooling_mode_cls_token&#39;: False, &#39;pooling_mode_mean_tokens&#39;: True, &#39;pooling_mode_max_tokens&#39;: False, &#39;pooling_mode_mean_sqrt_len_tokens&#39;: False, &#39;pooling_mode_weightedmean_tokens&#39;: False, &#39;pooling_mode_lasttoken&#39;: False, &#39;include_prompt&#39;: True})<br>  (2): Normalize()<br>)</pre><p>Step 3: Function to Create table to store marketing content</p><pre># Create table using raw SQL<br><br>def create_marketing_content(): <br>    create_table_query = text(&quot;&quot;&quot;<br>    CREATE TABLE IF NOT EXISTS marketing_content (<br>        id SERIAL PRIMARY KEY, <br>        title TEXT NOT NULL, <br>        content_type TEXT NOT NULL, <br>        description TEXT NOT NULL, <br>        target_audience TEXT, <br>        key_messaging TEXT, <br>        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, <br>        content_embedding VECTOR(384), <br>        audience_embedding VECTOR(384) <br>    ); <br>    &quot;&quot;&quot;)<br><br>    with engine.connect() as conn: <br>        conn.execute(create_table_query)<br>        conn.commit()</pre><p>Sample Data:</p><pre>MARKETING_CONTENT = [<br>    {<br>        &quot;title&quot;: &quot;Summer Fitness Challenge&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;30-day fitness challenge promoting our premium workout gear. Includes social media content, email sequences, and influencer partnerships.&quot;,<br>        &quot;target_audience&quot;: &quot;Health-conscious millennials, aged 25-35, interested in fitness and wellness, active on Instagram&quot;,<br>        &quot;key_messaging&quot;: &quot;Transform your summer with premium workout gear. Join our 30-day challenge for exclusive rewards.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Business Professional Collection&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Launch campaign for our new line of professional business attire. Focus on quality materials and modern designs.&quot;,<br>        &quot;target_audience&quot;: &quot;Corporate professionals, ages 30-50, fashion-conscious, looking for high-quality business wear&quot;,<br>        &quot;key_messaging&quot;: &quot;Elevate your professional wardrobe with timeless pieces designed for modern success.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Eco-Friendly Initiative&quot;,<br>        &quot;content_type&quot;: &quot;brand_campaign&quot;,<br>        &quot;description&quot;: &quot;Sustainability campaign highlighting our transition to 100% recycled materials and carbon-neutral manufacturing.&quot;,<br>        &quot;target_audience&quot;: &quot;Environmentally conscious consumers, all ages, prioritize sustainable and ethical products&quot;,<br>        &quot;key_messaging&quot;: &quot;Join us in creating a sustainable future. Every purchase makes a difference.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Student Smart Start&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Back-to-school campaign featuring student discounts, bundle deals, and campus essentials.&quot;,<br>        &quot;target_audience&quot;: &quot;College students, ages 18-22, budget-conscious, looking for quality basics and study gear&quot;,<br>        &quot;key_messaging&quot;: &quot;Start the semester right with student-exclusive deals and essential gear for success.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Winter Sports Collection&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Premium winter sports gear launch featuring innovative materials and cutting-edge designs for snow sports enthusiasts.&quot;,<br>        &quot;target_audience&quot;: &quot;Winter sports enthusiasts, ages 25-45, passionate about skiing and snowboarding&quot;,<br>        &quot;key_messaging&quot;: &quot;Conquer the slopes with gear that combines performance and style.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Mother&#39;s Day Appreciation&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Curated gift collections and special promotions celebrating mothers, featuring luxury accessories and self-care items.&quot;,<br>        &quot;target_audience&quot;: &quot;Gift shoppers, ages 25-55, looking for premium gifts for mothers&quot;,<br>        &quot;key_messaging&quot;: &quot;Celebrate the special women in your life with thoughtfully curated gifts.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Urban Commuter Series&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Launch of versatile clothing line designed for urban professionals who bike or walk to work.&quot;,<br>        &quot;target_audience&quot;: &quot;Urban professionals, ages 25-40, environmentally conscious, active commuters&quot;,<br>        &quot;key_messaging&quot;: &quot;Style meets function for the modern urban commuter.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Digital Nomad Collection&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Versatile travel-friendly clothing and accessories designed for remote workers and digital nomads.&quot;,<br>        &quot;target_audience&quot;: &quot;Remote workers, ages 25-40, frequent travelers, tech-savvy professionals&quot;,<br>        &quot;key_messaging&quot;: &quot;Work from anywhere in style with our travel-ready essentials.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Wellness Wednesday&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;Weekly wellness content series featuring workout tips, mindfulness practices, and healthy living products.&quot;,<br>        &quot;target_audience&quot;: &quot;Health and wellness enthusiasts, ages 25-45, interested in holistic health&quot;,<br>        &quot;key_messaging&quot;: &quot;Make wellness a priority with weekly inspiration and premium gear.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Holiday Gift Guide&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Comprehensive holiday shopping guide featuring curated gift collections for different personalities and budgets.&quot;,<br>        &quot;target_audience&quot;: &quot;Holiday shoppers, all ages, looking for meaningful and quality gifts&quot;,<br>        &quot;key_messaging&quot;: &quot;Find the perfect gift for everyone on your list.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Sustainable Basics&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Launch of everyday essentials made from organic and recycled materials, focusing on minimal environmental impact.&quot;,<br>        &quot;target_audience&quot;: &quot;Environmentally conscious consumers, ages 20-40, interested in sustainable fashion&quot;,<br>        &quot;key_messaging&quot;: &quot;Everyday essentials that feel good and do good.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Adventure Photography Series&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;Content series featuring outdoor photographers wearing our gear in spectacular locations.&quot;,<br>        &quot;target_audience&quot;: &quot;Outdoor enthusiasts and photographers, ages 25-45, interested in adventure and photography&quot;,<br>        &quot;key_messaging&quot;: &quot;Capture life&#39;s adventures in gear designed for the journey.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Spring Renewal Collection&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Fresh, vibrant spring collection featuring lightweight fabrics and nature-inspired colors.&quot;,<br>        &quot;target_audience&quot;: &quot;Fashion-forward consumers, ages 25-45, looking to refresh their wardrobe&quot;,<br>        &quot;key_messaging&quot;: &quot;Embrace the season with fresh styles and renewed energy.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Tech-Smart Workwear&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Innovation-focused workwear featuring smart fabrics and tech-friendly design elements.&quot;,<br>        &quot;target_audience&quot;: &quot;Tech professionals, ages 25-45, interested in innovative and functional clothing&quot;,<br>        &quot;key_messaging&quot;: &quot;Where technology meets style in the modern workplace.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Summer Festival Series&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;Festival-ready fashion collection with bohemian influences and practical features.&quot;,<br>        &quot;target_audience&quot;: &quot;Festival-goers, ages 18-35, music and fashion enthusiasts&quot;,<br>        &quot;key_messaging&quot;: &quot;Express yourself in style at this summer&#39;s hottest festivals.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Mindful Movement&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;Yoga and meditation-focused campaign promoting comfortable, sustainable activewear.&quot;,<br>        &quot;target_audience&quot;: &quot;Yoga practitioners and mindfulness enthusiasts, ages 25-55&quot;,<br>        &quot;key_messaging&quot;: &quot;Move mindfully in comfort and style.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Black Friday Preview&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Early access campaign for biggest sale of the year, featuring exclusive deals and member benefits.&quot;,<br>        &quot;target_audience&quot;: &quot;Deal-seeking shoppers, all ages, looking for premium products at great values&quot;,<br>        &quot;key_messaging&quot;: &quot;Get early access to our biggest savings of the year.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Athletic Performance Line&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;High-performance athletic wear featuring moisture-wicking technology and ergonomic design.&quot;,<br>        &quot;target_audience&quot;: &quot;Serious athletes, ages 20-40, focused on performance and quality&quot;,<br>        &quot;key_messaging&quot;: &quot;Push your limits in gear designed for peak performance.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Capsule Wardrobe Challenge&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;30-day challenge promoting minimalist fashion and sustainable consumption.&quot;,<br>        &quot;target_audience&quot;: &quot;Minimalism enthusiasts, ages 25-45, interested in sustainable fashion&quot;,<br>        &quot;key_messaging&quot;: &quot;Do more with less. Build your perfect capsule wardrobe.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Weekend Warrior Collection&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Versatile outdoor gear designed for weekend adventures and casual outdoor activities.&quot;,<br>        &quot;target_audience&quot;: &quot;Casual outdoor enthusiasts, ages 25-45, seeking versatile weekend wear&quot;,<br>        &quot;key_messaging&quot;: &quot;From city to trail, gear for every weekend adventure.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Professional Petites&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Tailored professional wear designed specifically for petite frames.&quot;,<br>        &quot;target_audience&quot;: &quot;Professional women with petite frames, ages 25-50, seeking well-fitted workwear&quot;,<br>        &quot;key_messaging&quot;: &quot;Professional style perfectly proportioned for petite frames.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Father&#39;s Day Tech Gear&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Curated collection of tech-friendly clothing and accessories perfect for tech-savvy dads.&quot;,<br>        &quot;target_audience&quot;: &quot;Gift shoppers buying for tech-interested fathers, ages 25-55&quot;,<br>        &quot;key_messaging&quot;: &quot;Innovative gifts for the tech-savvy dad.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Autumn Outdoor Living&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Fall collection focused on outdoor lifestyle and layered fashion for changing weather.&quot;,<br>        &quot;target_audience&quot;: &quot;Outdoor lifestyle enthusiasts, ages 25-45, interested in fall fashion&quot;,<br>        &quot;key_messaging&quot;: &quot;Embrace the outdoors in style this fall.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Smart Casual Revolution&quot;,<br>        &quot;content_type&quot;: &quot;brand_campaign&quot;,<br>        &quot;description&quot;: &quot;Campaign redefining smart casual wear for the modern workplace.&quot;,<br>        &quot;target_audience&quot;: &quot;Professional workers in casual offices, ages 25-45&quot;,<br>        &quot;key_messaging&quot;: &quot;Redefining workplace style for the modern professional.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Summer Beach Essentials&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Beach-ready collection featuring UV protection and quick-dry materials.&quot;,<br>        &quot;target_audience&quot;: &quot;Beach and summer lifestyle enthusiasts, ages 18-45&quot;,<br>        &quot;key_messaging&quot;: &quot;Make waves this summer with essential beach gear.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Luxury Loungewear&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Premium comfort wear featuring high-end materials and sophisticated designs.&quot;,<br>        &quot;target_audience&quot;: &quot;Luxury consumers, ages 30-55, valuing comfort and quality&quot;,<br>        &quot;key_messaging&quot;: &quot;Elevate your downtime with luxury comfort wear.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Active Kids Collection&quot;,<br>        &quot;content_type&quot;: &quot;product_launch&quot;,<br>        &quot;description&quot;: &quot;Durable, comfortable clothing line designed for active children and youth.&quot;,<br>        &quot;target_audience&quot;: &quot;Parents of active children, ages 30-45, seeking quality kids&#39; wear&quot;,<br>        &quot;key_messaging&quot;: &quot;Built to keep up with active kids.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Virtual Fashion Week&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;Digital showcase of new collections through virtual runway shows and interactive content.&quot;,<br>        &quot;target_audience&quot;: &quot;Fashion enthusiasts, ages 20-45, interested in digital fashion experiences&quot;,<br>        &quot;key_messaging&quot;: &quot;Experience fashion&#39;s future through immersive digital showcases.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Winter Wellness&quot;,<br>        &quot;content_type&quot;: &quot;campaign&quot;,<br>        &quot;description&quot;: &quot;Winter health and wellness campaign featuring cold-weather workout gear and recovery essentials.&quot;,<br>        &quot;target_audience&quot;: &quot;Winter fitness enthusiasts, ages 25-45, committed to year-round wellness&quot;,<br>        &quot;key_messaging&quot;: &quot;Stay active and well through the winter months.&quot;<br>    },<br>    {<br>        &quot;title&quot;: &quot;Graduation Collection&quot;,<br>        &quot;content_type&quot;: &quot;seasonal_campaign&quot;,<br>        &quot;description&quot;: &quot;Professional wear and accessories curated for recent graduates entering the workforce.&quot;,<br>        &quot;target_audience&quot;: &quot;Recent graduates, ages 21-25, preparing for professional life&quot;,<br>        &quot;key_messaging&quot;: &quot;Start your professional journey with confidence and style.&quot;<br>    }<br>]</pre><p>Step 4: Insert Content</p><p>Before that let’s write a function which converts numpy vectors to pgvectors</p><pre>def numpy_vector_to_pg_vector(vector: np.array) -&gt; str:<br>   &quot;&quot;&quot;<br>   Convert a numpy array to a PostgreSQL vector string format.<br>   <br>   Args:<br>       vector (np.array): Input numpy array to be converted to PostgreSQL vector format<br>       <br>   Returns:<br>       str: String representation of vector in PostgreSQL format [x1,x2,...,xn]<br>       <br>   Example:<br>       &gt;&gt;&gt; arr = np.array([1, 2, 3]) <br>       &gt;&gt;&gt; numpy_vector_to_pg_vector(arr)<br>       &#39;[1,2,3]&#39;<br>       <br>   Notes:<br>       - Flattens multi-dimensional arrays to 1D<br>       - Vector values should be numeric (int/float)<br>       - Returns string format compatible with PostgreSQL vector type<br>   &quot;&quot;&quot;<br>   return json.dumps(vector.flatten().tolist())</pre><p>Function to insert content</p><pre>def insert_content(data_content: Dict[str, str], batch_size=100):<br>   &quot;&quot;&quot;<br>   Batch insert marketing content with generated embeddings into database.<br><br>   Args:<br>       data_content (Dict[str, str]): Dictionary containing marketing content data with keys:<br>           - title: Title of the marketing content<br>           - content_type: Type of content (campaign, ad, blog etc)<br>           - description: Content description<br>           - target_audience: Target audience description <br>           - key_messaging: Key message of the content<br>       batch_size (int, optional): Number of records to insert in each batch. Defaults to 100.<br><br>   Notes:<br>       - Uses sentence transformer model to generate embeddings from content and audience text<br>       - Content embedding combines description and key messaging<br>       - Audience embedding generated from target audience text<br>       - Performs batched inserts for better performance<br>       - Automatically commits each batch<br>       <br>   Example:<br>       &gt;&gt;&gt; content = {<br>           &quot;title&quot;: &quot;Summer Campaign&quot;,<br>           &quot;content_type&quot;: &quot;campaign&quot;,<br>           &quot;description&quot;: &quot;Summer sale promotion&quot;, <br>           &quot;target_audience&quot;: &quot;Young adults&quot;,<br>           &quot;key_messaging&quot;: &quot;Get summer ready&quot;<br>       }<br>       &gt;&gt;&gt; insert_content([content], batch_size=100)<br>   &quot;&quot;&quot;<br>   insert_query = text(<br>       &quot;&quot;&quot;<br>         INSERT INTO marketing_content (<br>             title,<br>             content_type, <br>             description,<br>             target_audience,<br>             key_messaging,<br>             content_embedding,<br>             audience_embedding<br>           )<br>         VALUES (<br>             :title,<br>             :content_type,<br>             :description,<br>             :target_audience,<br>             :key_messaging,<br>             :content_embedding,<br>             :audience_embedding<br>           )<br>&quot;&quot;&quot;<br>   )<br><br>   for i in range(0, len(data_content), batch_size):<br>       batch = data_content[i : i + batch_size]<br>       batch_parameters = []<br><br>       for content in tqdm(batch):<br>           content_text = f&quot;{content[&#39;description&#39;]} {content[&#39;key_messaging&#39;]}&quot;<br>           content_embedding = model.encode(content_text)<br>           audience_embedding = model.encode(content[&quot;target_audience&quot;])<br><br>           batch_parameters.append(<br>               {<br>                   &quot;title&quot;: content[&quot;title&quot;],<br>                   &quot;content_type&quot;: content[&quot;content_type&quot;],<br>                   &quot;description&quot;: content[&quot;description&quot;],<br>                   &quot;target_audience&quot;: content[&quot;target_audience&quot;],<br>                   &quot;key_messaging&quot;: content[&quot;key_messaging&quot;],<br>                   &quot;content_embedding&quot;: numpy_vector_to_pg_vector(content_embedding),<br>                   &quot;audience_embedding&quot;: numpy_vector_to_pg_vector(audience_embedding),<br>               }<br>           )<br>       <br>       with engine.connect() as conn:<br>           conn.execute(insert_query, batch_parameters)<br>           conn.commit()</pre><p>Function to perform vector search</p><pre>def search_marketing_content(query: str, search_type: str = &#39;content&#39;, limit: int = 5):<br>   &quot;&quot;&quot;<br>   Search marketing content using semantic similarity based on vector embeddings.<br><br>   Args:<br>       query (str): Search query text to find similar content<br>       search_type (str, optional): Type of search to perform. Can be either:<br>           - &#39;content&#39;: Search based on content description and messaging<br>           - &#39;audience&#39;: Search based on target audience description<br>           Defaults to &#39;content&#39;.<br>       limit (int, optional): Maximum number of results to return. Defaults to 5.<br><br>   Returns:<br>       List[Row]: List of matching records sorted by similarity score, containing:<br>           - id: Record ID<br>           - title: Content title <br>           - content_type: Type of content<br>           - description: Content description<br>           - target_audience: Target audience<br>           - key_messaging: Key message<br>           - created_at: Creation timestamp<br>           - similarity_score: Cosine similarity score (0-1, higher is more similar)<br><br>   Notes:<br>       - Uses sentence transformer model to convert query to vector embedding<br>       - Calculates cosine similarity between query vector and stored embeddings<br>       - Higher similarity scores indicate better matches<br>       - Returns results sorted by similarity in descending order<br><br>   Example:<br>       &gt;&gt;&gt; # Search content<br>       &gt;&gt;&gt; results = search_marketing_content(&quot;sustainable products&quot;, search_type=&quot;content&quot;)<br>       &gt;&gt;&gt; <br>       &gt;&gt;&gt; # Search audience<br>       &gt;&gt;&gt; results = search_marketing_content(&quot;young professionals&quot;, search_type=&quot;audience&quot;)<br>   &quot;&quot;&quot;<br>   embedding_column = (<br>       &quot;content_embedding&quot; if search_type == &quot;content&quot; else &quot;audience_embedding&quot;<br>   )<br><br>   search_query = text(<br>       f&quot;&quot;&quot;<br>       SELECT <br>   id, <br>   title, <br>   content_type, <br>   description, <br>   target_audience, <br>   key_messaging, <br>   created_at, <br>   1 - cosine_distance({embedding_column}, :search_embedding) as similarity_score <br><br>FROM marketing_content<br>ORDER BY 1 - cosine_distance({embedding_column}, :search_embedding) DESC <br>LIMIT :limit <br>&quot;&quot;&quot;<br>   )<br><br>   with engine.connect() as conn:<br>       result = conn.execute(search_query, {<br>           &#39;search_embedding&#39;: numpy_vector_to_pg_vector(model.encode(query)),<br>           &quot;limit&quot;: limit<br>       })<br>   return result.fetchall()</pre><p>Execute the whole code to perform vector search</p><pre>def main():<br>   # Initialize model<br>   print(&#39;Initialize the model&#39;)<br>   model = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)<br><br>   # Create table and insert data<br>   create_marketing_content()<br>   print(&#39;Table Created&#39;)<br>   <br>   insert_content(data_content=MARKETING_CONTENT)<br><br>   # Perform content similarity search<br>   print(&#39;\nPerform content search&#39;)<br>   content_search = &quot;sustainable eco-friendly products environmental impact&quot;<br>   result = search_marketing_content(content_search, search_type=&#39;content&#39;)<br>   for res in result:<br>       print(f&quot;Title: {res[1]}, Content Type: {res[2]}&quot;)<br><br>   # Perform audience similarity search  <br>   print(&#39;\nPerforming audience search&#39;)<br>   audience_search = &quot;young professionals interested in luxury brands and fashion&quot;<br>   result = search_marketing_content(audience_search, search_type=&#39;audience&#39;)<br>   for res in result:<br>       print(f&quot;Title: {res[1]}, Content Type: {res[2]}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>   main()</pre><pre>Initialize the model<br>Table Created<br>100%|██████████| 30/30 [00:00&lt;00:00, 47.55it/s]<br><br>Perform content search<br>Title: Eco-Friendly Initiative, Content Type: brand_campaign<br>Title: Sustainable Basics, Content Type: product_launch<br>Title: Capsule Wardrobe Challenge, Content Type: campaign<br>Title: Mindful Movement, Content Type: campaign<br>Title: Tech-Smart Workwear, Content Type: product_launch<br><br>Performing audience search<br>Title: Luxury Loungewear, Content Type: product_launch<br>Title: Spring Renewal Collection, Content Type: seasonal_campaign<br>Title: Tech-Smart Workwear, Content Type: product_launch<br>Title: Business Professional Collection, Content Type: product_launch<br>Title: Sustainable Basics, Content Type: product_launch</pre><p>Conclusion:</p><p>In this comprehensive exploration of vector databases and CockroachDB, we’ve journeyed from fundamental concepts to practical implementation, unveiling the powerful capabilities of modern database technology in AI-driven applications. Our investigation has covered several crucial aspects:</p><h3>Key Takeaways</h3><p><strong>Vector Database Evolution</strong></p><ul><li>We’ve seen how vector databases have become essential infrastructure for AI applications, enabling sophisticated similarity searches and semantic understanding</li><li>The integration of vector capabilities into traditional databases like CockroachDB represents a significant step forward in database technology</li></ul><p><strong>CockroachDB’s Vector Implementation</strong></p><ul><li>The introduction of vector search capabilities in CockroachDB 24.2 demonstrates how established databases are evolving to meet modern AI requirements</li><li>The pgvector compatibility provides a robust foundation for vector operations while maintaining CockroachDB’s core strengths in scalability and reliability</li></ul><p><strong>Practical Applications</strong></p><ul><li>Through our simple product search example, we demonstrated the basic principles of vector search implementation</li><li>Our detailed marketing library case study showcased a real-world application, illustrating how vector databases can transform content discovery and audience targeting</li></ul><p><strong>Future Implications</strong></p><ul><li>The integration of vector capabilities into distributed SQL databases like CockroachDB suggests a trend toward consolidated database solutions</li><li>This convergence of traditional and vector database capabilities opens new possibilities for applications requiring both structured data and semantic search</li></ul><h3>Real-World Impact</h3><p>Our marketing library implementation demonstrated how vector databases can solve tangible business problems:</p><ul><li>Reducing time spent searching for relevant content</li><li>Preventing duplicate work through better content discovery</li><li>Enabling context-aware search that understands meaning beyond keywords</li><li>Improving content reusability and consistency across marketing initiatives</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baa891ef9a08" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/building-ai-powered-applications-with-cockroachdb-vector-search-from-theory-to-practice-baa891ef9a08">Building AI-Powered Applications with CockroachDB Vector Search: From Theory to Practice</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Semantic Analysis Integrating BERTopic and LLMs for Customer Intent Recognition]]></title>
            <link>https://blog.stackademic.com/advanced-semantic-analysis-integrating-bertopic-and-llms-for-customer-intent-recognition-b1f9ff0405d7?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1f9ff0405d7</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Sun, 21 Jul 2024 12:57:15 GMT</pubDate>
            <atom:updated>2024-07-24T00:22:48.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*vNxTFFW6IgcHoyLJL7mqYw.png" /></figure><p>In the fast-paced world of AI and natural language processing, a groundbreaking technique has emerged, promising to revolutionize how we uncover hidden themes in vast seas of text. Enter BERTopic, a topic modeling marvel that’s taking the data science community by storm.</p><p>Imagine you’re a data scientist at Tail Waggers, the pet product powerhouse known for “Spreading joy, one wag at a time.” Your desk is piled high with customer feedback, product reviews, and social media chatter. Hidden within this mountain of text are crucial insights that could shape the future of your furry-friend-focused business. But how do you extract these golden nuggets of information efficiently?</p><p>This is where our journey into the world of BERTopic begins. In this article, we’ll embark on an exhilarating expedition through the inner workings of this cutting-edge technique. We’ll peel back the layers, exploring its components and uncovering the magic that makes BERTopic tick.</p><p>But we won’t stop at theory. Oh no, dear reader. We’ll roll up our sleeves and dive into a real-world scenario, applying BERTopic to solve a pressing challenge at Tail Waggers. By the end of our adventure, you’ll not only understand the power of this AI-driven topic analysis tool but also see its practical application in action.</p><p>So, fasten your seatbelts and get ready to wag your analytical tails. It’s time to unleash the full potential of BERTopic and transform the way we understand our data.</p><p>Jupyter Notebook Link: <a href="https://www.kaggle.com/code/rudrasing/bert-topic">https://www.kaggle.com/code/rudrasing/bert-topic</a></p><p>Github Repo Link: <a href="https://github.com/SiddharthKshirsagar/BERTTopicModelling/tree/main">https://github.com/SiddharthKshirsagar/BERTTopicModelling/tree/main</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbJ0r8fzs8I2p5-BOkC52w.jpeg" /></figure><h4>Senario</h4><p>Welcome to the bustling digital headquarters of Tail Waggers, where the virtual aisles are teeming with more activity than a puppy playpen at lunchtime. Our online presence has suddenly become the hottest spot in town, with user traffic skyrocketing faster than a cat up a Christmas tree. But amidst the excitement, a crucial question lingers: What’s really driving this influx of paw-some visitors?</p><p>As Tail Waggers’ ace data scientist, you’ve been tasked with a mission more important than finding the perfect squeaky toy — decoding the mysterious intentions behind each click, scroll, and woof… err, page view.</p><p>Picture yourself seated at your desk, surrounded by screens flickering with streams of data. Your challenge? To transform this digital deluge into actionable insights that could make tails wag all the way to the bank. It’s time to unleash the power of AI on two critical fronts:</p><ol><li><strong>User Intent Unleashed</strong>: Your first task is to sniff out the true purpose behind each user’s online journey. Are they hunting for the latest in chew toy technology? Prowling for the purr-fect cat tower? Or perhaps they’re on a quest for the holy grail of pet care tips? By analyzing their digital pawprints — from the pages they visit to how they interact with the site — you’ll uncover the driving forces behind their virtual expeditions.</li></ol><p>Armed with a treasure trove of data served up fresh by the engineering team, you’re poised to embark on this analytical adventure. The dataset before you is a veritable smorgasbord of user behavior, packed with more features than a Swiss Army knife for pets.</p><p>As you roll up your sleeves and prepare to dive into this ocean of information, one thing becomes clear: This isn’t just about crunching numbers. It’s about understanding the heartbeat of Tail Waggers’ online community, deciphering the desires of pet parents everywhere, and ultimately, ensuring that every virtual visitor leaves with a waggy tail and a full shopping cart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zrHrvYeZANjLkTB-7bKyBg.jpeg" /></figure><p><strong>Data Provided by the Engineering Team</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NzaMf7vKbCQVkVnWNFZeJA.png" /></figure><pre><strong>Description of data: <br><br>event:</strong> The type of user interaction event. <br><strong>context_page_title:</strong> Page title describing the content or purpose of the visited page. <br><strong>timestamp:</strong> The date and time when the event occured. <br><strong>user_id:</strong> A unique identifier for each user. <br><strong>website_path:</strong> The URL path of pages visited. <br><strong>website_query_parameters:</strong> Parameters included in the URL to provide additional context about page. <br><strong>session_id:</strong> A unique idetifier for each user session.</pre><p>Our challenge at Tail Waggers is to make sense of a vast amount of unstructured user data. While traditional clustering methods could work, BERTopic offers a cutting-edge solution using topic modeling. This open-source tool excels at uncovering hidden patterns in text data. Before we dive in, let’s quickly grasp what topic modeling is and why it’s so powerful for our pet-centric analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVFA0U_8XfDrPZPozgcPFQ.jpeg" /></figure><h4>Topic Modelling:</h4><p>Topic modelling is a technique used to discover hidden topics or themes in a large collection of text data, allowing for the identification of underlying patterns and trends. It works by analyzing the frequency of words and phrases in each document to identify clusters of related words that reveal the underlying topics. By doing so, topic modeling enables the summarization and categorization of large volumes of text data, making it a powerful tool for data analysis and insight generation.</p><p>In this context we will use a advanced version of topic modelling called BERTTopic. But what is BERTopic ?</p><h4>BERTopic:</h4><p>BERTopic is a topic modelling technique that leverages embeddings and c-TF-IDF to create dense clusters allowing for easy interpretability of topics while keeping important words in the topic descriptions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qSIoG34xcgyQXliMOxbOgA.png" /></figure><p>The above diagram gives a brief about how the algorithm works.</p><p>To begin our adventure into AI-powered topic modeling, we’ll start with the basics: installation. Just as you’d prepare a new toy for a curious puppy, we need to set up our analytical playground. Here’s how to get BERTopic ready for action:</p><pre>pip install bertopic</pre><p>Now lets load the session data.</p><pre>import pandas as pd <br><br># Create a DataFrame from the provided data<br>df = pd.read_csv(&#39;https://raw.githubusercontent.com/SiddharthKshirsagar/BERTTopicModelling/main/session-data-info.csv&#39;)<br><br># Combine relevant columns into a single &#39;merged_information&#39; column<br>df[&#39;merged_information&#39;] = (<br>    df[&#39;event&#39;].fillna(&#39;&#39;) +<br>    df[&#39;context_page_title&#39;].fillna(&#39;&#39;) +<br>    df[&#39;website_path&#39;].fillna(&#39;&#39;) +<br>    df[&#39;website_query_parameters&#39;].fillna(&#39;&#39;)<br>)<br><br># Select only the &#39;session_id&#39; and &#39;merged_information&#39; columns<br>session_information = df[[&#39;session_id&#39;, &#39;merged_information&#39;]]<br><br># Group by &#39;session_id&#39; and concatenate &#39;merged_information&#39; for each session<br>session_information = (<br>    session_information.groupby(&#39;session_id&#39;)[&#39;merged_information&#39;]<br>    .apply(&quot;,&quot;.join)<br>    .reset_index()<br>)</pre><p>Imagine each row in our dataset as a unique pet adventure at Tail Waggers. The session_id is like a collar tag, identifying each customer’s journey through our digital pet shop. Attached to this tag is the merged_information — a detailed diary of their online escapade.</p><p>This merged_information is a treasure trove, chronicling every sniff, paw, and tail wag (or in human terms, every click, view, and interaction) during their visit. It’s as if we’ve compiled a play-by-play of each customer’s exploration through our virtual aisles of chew toys, kibble, and cat trees.</p><p>With this structure, we’re poised to unleash BERTopic’s analytical prowess. It will dive into these adventure logs, sniffing out patterns and grouping similar journeys, helping us understand what really makes our customers’ tails wag in excitement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*QwqNschnJ3S7XhZTWWrNHQ.png" /></figure><p>Now lets perform topic modelling</p><pre>rom bertopic import BERTopic<br>from sklearn.feature_extraction.text import CountVectorizer <br><br># Create a CountVectorizer model for text preprocessing<br># min_df=3 removes rare words, ngram_range=(1,3) considers up to 3-word phrases<br>vectorizer_model = CountVectorizer(min_df=3, ngram_range=(1,3)) <br><br># Create a BERTopic model<br>model = BERTopic(<br>    vectorizer_model=vectorizer_model,  # Use our custom CountVectorizer<br>    language=&#39;english&#39;,                 # Set language for preprocessing<br>    embedding_model=&#39;all-MiniLM-L12-v2&#39;, # Choose a pre-trained embedding model<br>    verbose=True,                       # Enable detailed progress output<br>)<br><br># Fit the BERTopic model and transform the data<br># This step discovers topics in our merged session information<br>topics, probs = model.fit_transform(session_information[&quot;merged_information&quot;])<br><br># Generate a visual representation of the discovered topics<br>model.visualize_topics()</pre><p>This code unleashes BERTopic’s analytical power on Tail Waggers’ user data. It’s like giving our AI a pair of x-ray goggles to peer into customer behavior. BERTopic handles the heavy lifting: it transforms text into BERT embeddings, compresses this information with UMAP, clusters similar sessions using HDBSCAN, and distills key topics through c-TF-IDF.</p><p>The result? A map of hidden patterns in our pet lovers’ online journeys. By visualizing these topics, we’re about to uncover the secret desires and intentions driving traffic to our virtual pet emporium. Let’s dive in and see what nuggets of insight BERTopic has dug up from our data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*dpR8dkVe_F1PF7U9_d-6jw.png" /></figure><p>While BERTopic’s initial results are intriguing, we’ve uncovered more topics than a pet store has treats. To make our insights more manageable, we need to narrow it down. HDBSCAN, the default clustering method, doesn’t let us set a specific number of topics. But fear not — we can teach our AI new tricks.</p><p>Enter K-means clustering: the perfect tool to leash our runaway topics. By using K-means, we can tell BERTopic exactly how many topic “bins” to sort our data into. It’s like organizing our pet supplies into a set number of shelves — everything gets a place, and we control how many places there are.</p><p>Let’s adjust our BERTopic model to use K-means, allowing us to specify our desired number of topics. This way, we’ll distill our user insights into a more focused and actionable set of themes, making it easier to understand and act on our customers’ needs.</p><pre>from sklearn.cluster import KMeans <br>from sentence_transformers import SentenceTransformer<br>from sklearn.feature_extraction.text import CountVectorizer <br>from bertopic import BERTopic<br><br># Create a CountVectorizer for text preprocessing<br>vectorizer_model = CountVectorizer(min_df=3, ngram_range=(1,3))<br><br># Set up KMeans for clustering, specifying 13 clusters<br>cluster_model = KMeans(n_clusters=13)<br><br># Initialize the SentenceTransformer model for creating embeddings<br>sentence_model = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;)<br><br># Generate embeddings for our session information<br>embeddings = sentence_model.encode(session_information[&quot;merged_information&quot;])<br><br># Create the BERTopic model with our custom components<br>model = BERTopic(<br>    vectorizer_model=vectorizer_model,  # Custom text vectorizer<br>    language=&#39;english&#39;,                 # Specify language<br>    verbose=True,                       # Enable detailed output<br>    hdbscan_model=cluster_model         # Use KMeans instead of HDBSCAN<br>)<br><br># Fit the model and transform the data<br>topics, probs = model.fit_transform(<br>    session_information[&quot;merged_information&quot;], <br>    embeddings=embeddings<br>)<br><br># Visualize the resulting topics<br>model.visualize_topics()</pre><p>Lets Visualize the topics</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*7xoTtNNOt-QhvAu0RTMlog.png" /></figure><p>This visualization is like a map of our customers’ interests at Tail Waggers. Each circle represents a topic, with larger circles indicating more frequent themes. The distances between circles show how related the topics are — closer circles mean more closely connected interests.</p><p>Imagine it as a bird’s-eye view of our digital pet store. We can see clusters of related products or services, perhaps showing that customers browsing dog food often explore treats too, or that cat toy shoppers frequently check out scratching posts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*i6CUGGblInulzJaMjcm-Jg.png" /></figure><p>Now, let’s dive deeper into how our Tail Waggers’ customers distribute across these topics. We’ll use UMAP (Uniform Manifold Approximation and Projection) to condense our multi-dimensional topic data into a visual format we can easily interpret.</p><p>Think of this as creating a heat map of our virtual pet store. UMAP will help us see where our customers cluster, showing us which areas of our site are bustling with activity and which might need more attention.</p><pre>from umap import UMAP<br><br># Perform dimensionality reduction on our embeddings<br># This condenses our high-dimensional data into a 2D representation<br>reduced_embeddings = UMAP(<br>    n_neighbors=10,    # Balance local versus global structure in the data<br>    n_components=2,    # Reduce to 2 dimensions for visualization<br>    min_dist=0.0,      # Minimum distance between points in the projection<br>    metric=&#39;cosine&#39;    # Use cosine similarity for measuring distances<br>).fit_transform(embeddings)<br><br># Visualize how documents (user sessions) are distributed across topics<br>model.visualize_documents(<br>    session_information[&quot;merged_information&quot;], <br>    reduced_embeddings=reduced_embeddings<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KKldwtZIXcD36EYqjNhvhQ.png" /></figure><p>Visualizing the the topics without any dimensionality reduction</p><pre>model.visualize_documents(session_information[&quot;merged_information&quot;], embeddings=embeddings)</pre><p>Now lets look at all the topics in a dataframe</p><pre>model.get_topic_info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovJ3o0WaiU7x7SoiDnNpZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S138hRd9dk2D2EfrUPAbaA.jpeg" /></figure><h4>Generative AI</h4><p>Interpreting topic models can be like decoding a pet’s body language — tricky without expertise. But we’ve got a new trick up our sleeve: large language models. Instead of manually puzzling over each topic, we’re calling in Google’s Gemini, an AI powerhouse, to be our interpreter.</p><p>Think of Gemini as our multilingual pet whisperer. We’ll feed it the topics BERTopic discovered, and it’ll translate them into clear, actionable insights about our Tail Waggers customers. This approach combines the pattern-spotting prowess of BERTopic with the contextual understanding of a large language model.</p><p>By leveraging Gemini, we’re not just identifying topics — we’re unlocking their meaning and potential impact on our pet-centric business. It’s like having an AI assistant that not only spots trends in our data but also explains why they matter for Tail Waggers.</p><p>Let’s set up Gemini and see how it interprets the voice of our furry friends’ owners. Follow this aricle to setup the API.</p><p><a href="https://siddharth1.medium.com/from-novice-to-user-first-steps-with-the-gemini-api-1c7287bcc26f">From Novice to User: First Steps with the Gemini API</a></p><p>Alright, let’s gear up for our AI-powered topic interpretation adventure. First, we’ll fetch the tools we need:</p><pre><br>import os<br>import json <br>import time<br>from tqdm.notebook import tqdm <br>import google.generativeai as genai<br><br># Set the Gemini API key as an environment variable<br>os.environ[&#39;GOOGLE_GEMINI_API_KEY&#39;] = &#39;YOUR_API_KEY&#39;<br><br># Configure the Gemini API with the key<br>genai.configure(api_key=os.environ[&#39;GOOGLE_GEMINI_API_KEY&#39;])</pre><p>As we’ve discovered, our topics_info dataframe is a treasure trove of insights about Tail Waggers’ customer behavior. Each row represents a distinct topic, complete with keywords that capture its essence and representative documents that illustrate it in action.</p><p>Imagine this dataframe as a series of snapshots from our digital pet store. The keywords are like labels on the shelves, giving us a quick idea of what each section contains. The representative docs, on the other hand, are like customer interactions we’ve overheard in each aisle.</p><p>For example, one topic might have keywords like “collar,” “leash,” and “harness,” with representative docs showing customer queries about sizing or material preferences. Another could focus on “nutrition,” “diet,” and “allergies,” with documents reflecting discussions about specialized pet foods.</p><pre>topic_information = model.get_topic_info()</pre><p>Absolutely. Let’s focus on decoding one topic to showcase how Gemini can transform our raw data into actionable insights for Tail Waggers.</p><p>We’ll select a single topic from our BERTopic results, complete with its keywords and representative documents. Then, we’ll craft a prompt for Gemini that asks it to interpret this information and create a concise, meaningful description of the topic.</p><pre># Define the Gemini model ID<br>MODEL_ID = &quot;gemini-1.5-flash&quot;  <br><br># Initialize the Gemini model with a low temperature for more focused outputs<br>genai_model = genai.GenerativeModel(<br>    model_name=MODEL_ID, <br>    generation_config={&quot;temperature&quot;: 0.3}<br>)<br><br># Craft the prompt template for topic interpretation<br>prompt = &quot;&quot;&quot;<br>Given a set of representative documents: {documents}<br>And a list of keywords describing the topic: {keywords}<br><br>Please generate:<br>1. A concise topic label (2-3 words)<br>2. A brief description (1-2 sentences)<br><br>Return the results in JSON format as follows:<br>{{<br>    &quot;topic_label&quot;: &quot;Your concise label here&quot;,<br>    &quot;description&quot;: &quot;Your brief description here&quot;<br>}}<br>&quot;&quot;&quot;<br><br># Extract information for the first topic<br>topic_representation = topic_information.iloc[0][&#39;Representation&#39;]<br>topic_representative_docs = topic_information.iloc[0][&#39;Representative_Docs&#39;] <br><br># Generate the interpretation using Gemini<br>response = genai_model.generate_content(<br>    prompt.format(<br>        documents=topic_representative_docs, <br>        keywords=topic_representation<br>    ), <br>    generation_config={&#39;response_mime_type&#39;: &#39;application/json&#39;}<br>)<br><br># Print the interpreted topic information<br>print(json.loads(response.text))Output: </pre><pre>{&#39;topic_label&#39;: &#39;Policy &amp; Sorting&#39;,<br> &#39;description&#39;: &#39;This topic covers viewing and understanding policies, potentially related to data or terms. It also involves sorting and filtering products or information based on various criteria like price, popularity, and shipping options.&#39;}</pre><p>Now that we’ve successfully decoded one “aisle” of our virtual Tail Waggers store, it’s time to apply this AI-powered insight to our entire product range. We’ll scale up our approach, using Gemini to interpret every topic BERTopic discovered in our customer data.</p><pre>def generate_topic_descriptions(documents, keywords):<br>    &quot;&quot;&quot;Generate topic descriptions using Gemini model.&quot;&quot;&quot;<br>    response = genai_model.generate_content(<br>        prompt.format(documents=documents, keywords=keywords),<br>        generation_config={&#39;response_mime_type&#39;: &#39;application/json&#39;}<br>    )<br>    return json.loads(response.text)<br><br># List to store all topic descriptions<br>topic_descriptions = []<br><br># Iterate through each row in the topic_information dataframe<br>for row in tqdm(topic_information.itertuples()):<br>    # Extract keywords and representative documents for each topic<br>    keywords = row.Representation<br>    documents = row.Representative_Docs<br>    <br>    # Generate description for the current topic<br>    topic_descriptions.append(generate_topic_descriptions(documents, keywords))<br>    <br>    # Pause briefly to avoid hitting API rate limits<br>    time.sleep(2)</pre><p>Generated Topic Descriptions:</p><pre>[{&#39;topic_label&#39;: &#39;Shopping &amp; Sorting&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on online shopping experiences, particularly the use of sorting and filtering options to refine product searches.&#39;},<br> {&#39;topic_label&#39;: &#39;E-commerce Sorting&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on sorting and filtering options available on e-commerce websites, including parameters like price, popularity, and newest arrivals.&#39;},<br> {&#39;topic_label&#39;: &#39;Shopping &amp; Sorting&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on online shopping and product sorting, analyzing user interactions with features like sorting options, filters, and coupon codes.&#39;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on website features related to sorting, filtering, and other navigation elements, often used for online shopping or information retrieval.&#39;},<br> {&#39;topic_label&#39;: &#39;E-commerce Sorting&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on the sorting and filtering options available on e-commerce websites, allowing users to refine their search results based on various criteria.&#39;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on website navigation features, including sorting, filtering, and other user interface elements that help users find and purchase products.&#39;},<br> {&#39;topic_label&#39;: &#39;E-commerce Sorting&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on sorting and filtering options available on an e-commerce website, including parameters like price, popularity, and newest arrivals.&#39;},<br> {&#39;topic_label&#39;: &#39;Online Shopping Features&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on various features commonly found in online shopping platforms, including sorting options, filtering, and coupon codes.&#39;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on website navigation features, including sorting, filtering, and other options that help users find products and information.&#39;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on website features related to sorting, filtering, and browsing products, as well as checkout and order tracking.&#39;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on user interactions with a website, including sorting, filtering, and using coupons or gift cards.&#39;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &quot;This topic focuses on website features related to sorting, filtering, and product selection, indicating a user&#39;s journey through an online store.&quot;},<br> {&#39;topic_label&#39;: &#39;Website Navigation&#39;,<br>  &#39;description&#39;: &#39;This topic focuses on user interactions with a website, including sorting, filtering, and applying coupons or gift cards.&#39;}]</pre><p>With our AI assistant having sniffed out the essence of each topic in Tail Waggers’ digital domain, it’s time to organize these insights into our data kennel. We’re about to attach these newly minted labels and descriptions to our topic dataframe, creating a comprehensive map of our customers’ interests.</p><pre># Add AI-generated labels and descriptions to the topic_information dataframe<br>topic_information[&#39;topic_label&#39;] = [item[&#39;topic_label&#39;] for item in topic_descriptions]<br>topic_information[&#39;description&#39;] = [item[&#39;description&#39;] for item in topic_descriptions]<br><br># Get detailed information about each document (user session)<br>document_information = model.get_document_info(session_information[&quot;merged_information&quot;])<br><br># Merge the topic labels and descriptions with the document information<br>document_information = document_information.merge(<br>    topic_information[[&#39;Topic&#39;, &#39;topic_label&#39;, &#39;description&#39;]], <br>    on=&#39;Topic&#39;<br>)</pre><p>This is how document information dataframe looks which can be used for further downstream analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3vVhbMVCIPu3wvNTeqEffg.png" /></figure><h4>Summary:</h4><p>“AI-Powered Topic Analysis: Advanced Techniques with BERTopic” takes readers on an exhilarating journey through the world of cutting-edge text analysis, using Tail Waggers, a fictional pet supply company, as our playground.</p><p>We began by unleashing BERTopic on Tail Waggers’ user session data, allowing this AI-driven tool to sniff out hidden patterns in customer behavior. Like a well-trained bloodhound, BERTopic grouped similar user sessions into topics, giving us our first glimpse into the underlying themes driving website traffic.</p><p>However, faced with an overwhelming number of topics, we taught our AI new tricks. By introducing K-means clustering, we corralled our insights into a manageable number of groups, creating a more focused view of customer interests.</p><p>Our analysis took flight with visualizations that mapped the landscape of user behavior. We explored intertopic distances and user session distributions, creating a bird’s-eye view of how pet lovers navigate the Tail Waggers site.</p><p>The crowning achievement came when we enlisted Google’s Gemini, a large language model, to interpret our discovered topics. This AI assistant acted as a multilingual pet whisperer, translating complex data patterns into clear, actionable insights about customer intentions and preferences.</p><p>Finally, we integrated these AI-generated descriptions back into our dataset, creating a comprehensive map of customer behavior. This enhanced data, rich with context and meaning, empowers Tail Waggers to understand not just what their customers are doing, but why they’re doing it.</p><p>This article showcases the power of combining advanced topic modeling techniques with large language models. It demonstrates how businesses can leverage AI to gain deep, nuanced understandings of their customers, paving the way for improved user experiences and data-driven decision making.</p><p>For Tail Waggers, this analysis promises to revolutionize how they cater to their pet-loving clientele, ensuring every virtual visitor leaves with a wagging tail and a full shopping cart. More broadly, it illustrates the exciting possibilities that emerge when we harness the full potential of AI in understanding human behavior and preferences.</p><p>Jupyter Notebook Link: <a href="https://www.kaggle.com/code/rudrasing/bert-topic">https://www.kaggle.com/code/rudrasing/bert-topic</a></p><p>Github Repo Link: <a href="https://github.com/SiddharthKshirsagar/BERTTopicModelling/tree/main">https://github.com/SiddharthKshirsagar/BERTTopicModelling/tree/main</a></p><h3>Stackademic 🎓</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://stackademic.com/"><strong>Stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1f9ff0405d7" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/advanced-semantic-analysis-integrating-bertopic-and-llms-for-customer-intent-recognition-b1f9ff0405d7">Advanced Semantic Analysis Integrating BERTopic and LLMs for Customer Intent Recognition</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Novice to User: First Steps with the Gemini API]]></title>
            <link>https://siddharth1.medium.com/from-novice-to-user-first-steps-with-the-gemini-api-1c7287bcc26f?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c7287bcc26f</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Sun, 21 Jul 2024 09:31:54 GMT</pubDate>
            <atom:updated>2024-07-21T09:38:34.254Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_6EL1Le8kEICvkj_5SF0Q.png" /><figcaption>created by meta.ai</figcaption></figure><p>This guide demonstrates how to integrate Google’s Gemini API into your Python projects. We’ll begin with the essential first step: installing the Python SDK.</p><pre>pip install google-generativeai</pre><p>Confirm this by running the command</p><pre>import google.generativeai as genai </pre><p>Obtaining an API Key</p><ol><li>Visit <a href="https://aistudio.google.com/app/apikey">https://aistudio.google.com/app/apikey</a></li><li>Log in to your Google account if prompted.</li><li>Look for the option to generate an API key.</li><li>Click to create a new API key.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*huCDe1He6e_LL07np1jZ6A.png" /></figure><p>Generate and copy your API key, then set it in your Python environment.</p><pre>import os <br><br>os.environ[&#39;GOOGLE_GEMINI_API_KEY&#39;] = &#39;YOUR_API_KEY&#39;</pre><p>List all the models available</p><pre>for model in list(genai.list_models()): <br>    if &#39;generateContent&#39; in model.supported_generation_methods: <br>        print(model.name)</pre><pre>Output: <br><br>models/gemini-1.0-pro<br>models/gemini-1.0-pro-001<br>models/gemini-1.0-pro-latest<br>models/gemini-1.0-pro-vision-latest<br>models/gemini-1.5-flash<br>models/gemini-1.5-flash-001<br>models/gemini-1.5-flash-latest<br>models/gemini-1.5-pro<br>models/gemini-1.5-pro-001<br>models/gemini-1.5-pro-latest<br>models/gemini-pro<br>models/gemini-pro-vision</pre><p>Using Gemini Flash to Generate a Joke</p><p>To test the API, we’ll use Gemini Flash to generate a simple joke:</p><pre>model = genai.GenerativeModel(model_name=&#39;gemini-1.5-flash&#39;)<br>prompt = &#39;Tell me a joke&#39; <br>response = model.generate_content(prompt) <br><br>print(response.text)</pre><p>Output:</p><pre>&quot;Why don&#39;t scientists trust atoms? \n\nBecause they make up everything! \n&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c7287bcc26f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Master Column Selection in Pandas]]></title>
            <link>https://python.plainenglish.io/master-column-selection-in-pandas-fef107bdc4e7?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/fef107bdc4e7</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[regex]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 05:56:46 GMT</pubDate>
            <atom:updated>2024-01-12T04:12:27.467Z</atom:updated>
            <content:encoded><![CDATA[<p>I will demonstrate how to streamline column selection in pandas using .filter() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WkMFRFoLRQ5pgWiNu2v6_w.png" /><figcaption>Created by ChatGPT</figcaption></figure><p>Why this method is so useful ?</p><p>I was recently working on structuring and cleaning response from a REST API which was huge and deeply nested json response with nearly 1000 columns and I wanted to select columns based on certain keywords for which I was using complex for loop and regex but I wanted something simple and out of the box. After some searching I stumbled upon the .filter <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.filter.html">function</a>.</p><p>There are 3 parameters in .filter namely items , like , and regex lets start first by defining these.</p><pre>DataFrame.filter(items=None, like=None, regex=None, axis=None)<br><br><br>items: List of labels to look for.<br>like: A string representing a label to search for.<br>regex: A regex pattern to search for labels.<br>axis: Specify whether to filter by rows or columns, with 0 for rows and 1 for columns.</pre><p>Let’s start with examples before that lets define a dataframe.</p><pre>import pandas as pd<br><br># Sample DataFrame<br>data = {<br>    &#39;year_2020&#39;: [1, 2, 3],<br>    &#39;year_2021&#39;: [4, 5, 6],<br>    &#39;month_Jan&#39;: [7, 8, 9],<br>    &#39;month_Feb&#39;: [10, 11, 12],<br>    &#39;sales&#39;: [13, 14, 15],<br>    &#39;profit&#39;: [16, 17, 18]<br>}<br><br>df = pd.DataFrame(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*Yg6eT6yd1VU8MtK5DtKGqA.png" /></figure><ol><li>Using <em>items</em> parameter</li></ol><p>This is very simple parameter in case you want to select columns based on their names.</p><pre># select columns named sales and profit <br><br>filtered_df = df.filter(items = [&#39;sales&#39;,&#39;profit&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*-XA54NyovpRxHku4QZPR6g.png" /></figure><p>2. Using regex parameter</p><p>This parameter allows to filter columns using python regex.</p><pre># Select columns starting with word `month`<br>filtered_df = df.filter(regex = &#39;^month&#39;)</pre><pre># Select columns that ends with 2021<br>filtered_df = df.filter(regex = &#39;2021$&#39;)</pre><pre># Select columns that contains number 20<br>filtered_df = df.filter(regex = &#39;20&#39;)</pre><pre># Select columns that starts with year_ and ends with a number <br>filtered_df = df.filter(regex = &#39;year_\d+&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*tXOl4Nb_Ac5XqxA-l195ow.png" /></figure><p>3. Using like parameter</p><pre># filter columns that contain feb<br>filtered_df = df.filter(like = &#39;feb&#39;)</pre><pre># filter columns that contain number 20 <br>filtered_df = df.filter(like = &#39;20&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*-W88pDA0R9IQGLObvu0b0A.png" /></figure><p>Conclusion:</p><p>In this comprehensive guide, I delve into advanced pandas techniques, specifically focusing on efficient data processing in pandas through the powerful .filter() method. Ideal for Python data cleaning and handling large datasets in Python, this tutorial demonstrates the art of pandas column selection and dataframe filtering techniques. Whether you&#39;re dealing with complex data structures from a REST API response or streamlining data analysis in pandas, our examples, including pandas regex filtering and dataframe column manipulation, offer practical insights. This Python pandas tutorial is not just about theory; it&#39;s about applying these methods in real-world scenarios, making it a must-read for anyone looking to master column selection with pandas filter in their data science toolkit.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fef107bdc4e7" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/master-column-selection-in-pandas-fef107bdc4e7">Master Column Selection in Pandas</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy any LLM model in just 10 mins anywhere in the world.]]></title>
            <link>https://blog.devops.dev/deploy-ml-models-in-10mins-9d5dc472bcfa?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d5dc472bcfa</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Thu, 11 May 2023 15:34:33 GMT</pubDate>
            <atom:updated>2024-05-01T09:07:33.492Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqRopoG5JUcujqrhMCkJCA.jpeg" /><figcaption>Image created by Bing AI.</figcaption></figure><p>I will demonstrate how to use Google Cloud run to host a serverless image to text model from huggingface.</p><p>Building a ML model is simple by serving it to end users is very complicated we need to keep all the dependencies in check and look at the infrastructure and scale it according to the requests to the ml api. But this can be done easily using Docker and Google Cloud run.</p><p>Create a new python virtual environment and install all the dependencies.</p><pre>python -m venv env <br><br># to activate the environment in linux <br>source env/bin/activate <br><br># to activate the environmemnt in windows <br>env/Scripts/activate <br><br># we will be using cpu only so install transformers with torch <br>pip install transformers[torch]<br><br># check the installation of transformers <br>python -c &quot;from transformers import pipeline; print(pipeline(&#39;sentiment-analysis&#39;)(&#39;we love you&#39;))&quot; <br><br># install flask to create the API <br>pip install Flask <br><br># install google-cloud-storage so that we can save the uploaded images. <br>pip install google-cloud-storage</pre><p>Now that the environment is created create a directory api_developement</p><p>create a file utils.py</p><p>There are 2 important functions in this file.</p><ol><li>image_to_text This uses huggingface api and `Salesforce/blip-image-captioning-base` model to infer image from text.</li><li><a href="https://huggingface.co/Salesforce/blip-image-captioning-base">Salesforce/blip-image-captioning-base · Hugging Face</a></li><li>upload_to_blob This function uploads the user input image to flask api to gcs bucket with a unique uuid.</li></ol><pre># utils.py <br>from PIL import Image<br>import io<br>from datetime import datetime<br>import uuid<br>from transformers import BlipProcessor, BlipForConditionalGeneration<br>from google.cloud import storage<br><br># Initializing the processor and model for image captioning<br>processor = BlipProcessor.from_pretrained(&quot;Salesforce/blip-image-captioning-base&quot;)<br>model = BlipForConditionalGeneration.from_pretrained(<br>    &quot;Salesforce/blip-image-captioning-base&quot;<br>)<br><br><br># Defining a function to convert image bytes to text using the processor and model<br>def image_to_text(image_bytes):<br>    # Opening the image and converting it to RGB format<br>    raw_image = Image.open(io.BytesIO(image_bytes)).convert(&quot;RGB&quot;)<br>    # Processing the image and returning a tensor<br>    inputs = processor(raw_image, return_tensors=&quot;pt&quot;)<br>    # Generating captions for the image<br>    out = model.generate(**inputs)<br>    # Decoding the captions and skipping the special tokens<br>    return processor.decode(out[0], skip_special_tokens=True)<br><br><br># Defining a function to create a unique uuid for each image<br>def create_unique_uuid():<br>    return uuid.uuid4().hex<br><br><br># Defining a function to get the current timestamp<br>def get_timestamp():<br>    return datetime.now().isoformat()<br><br><br># Defining a function to upload the image bytes to Google Cloud Storage<br>def upload_to_blob(image_byte):<br>    try:<br>        # Setting the bucket name<br>        bucket_name = &quot;&lt;bucket_name&gt;&quot;<br>        # Creating a storage client object<br>        storage_client = storage.Client()<br>        # Getting the bucket object<br>        bucket = storage_client.bucket(bucket_name)<br>        # Creating a blob object with a unique name and path<br>        blob = bucket.blob(f&quot;image_to_text/{create_unique_uuid()}&quot;)<br>        # Uploading the image bytes to the blob<br>        blob.upload_from_string(image_byte)<br>        # Setting the value to success message<br>        value = &quot;upload success&quot;<br>    except Exception as e:<br>        # Setting the value to exception message in case of error<br>        value = str(e)<br>    # Returning a dictionary with the upload message<br>    return {&quot;upload_message&quot;: value}</pre><p>Next, we create our main.py file through which we will be serving the flask api and also uploading the input images to cloud storage.</p><pre># Importing the required modules<br>from flask import Flask, request, jsonify<br><br># Importing the datetime module for timestamp<br>from datetime import datetime<br><br># Importing the custom functions from utils.py<br>from utils import image_to_text, upload_to_blob<br><br># Creating a Flask app object<br>app = Flask(__name__)<br><br><br># Defining the route for the index page<br>@app.route(&quot;/&quot;, methods=[&quot;GET&quot;, &quot;POST&quot;])<br>def index():<br>    &quot;&quot;&quot;This function handles the requests to the index page and returns a JSON response with the prediction and upload message.&quot;&quot;&quot;<br>    # Checking if the request method is POST<br>    if request.method == &quot;POST&quot;:<br>        # Getting the file from the request<br>        file = request.files.get(&quot;file&quot;)<br>        # Checking if the file is valid<br>        if file is None or file.filename == &quot;&quot;:<br>            # Returning an error message if no file<br>            return jsonify({&quot;error&quot;: &quot;no file&quot;})<br><br>        try:<br>            # Reading the file as bytes<br>            image_bytes = file.read()<br>            # Calling the upload_to_blob function to upload the image to cloud storage<br>            upload_message = upload_to_blob(image_bytes)<br>            # Calling the image_to_text function to extract text from the image<br>            data = image_to_text(image_bytes)<br>            # Returning a JSON response with the prediction and upload message<br>            return jsonify({&quot;prediction&quot;: data, &quot;upload_message&quot;: upload_message})<br>        except Exception as e:<br>            # Returning an error message with the exception<br>            return jsonify({&quot;error&quot;: str(e)})<br>    # Returning a welcome message if the request method is GET<br>    return &quot;Welcome&quot;<br><br><br># Running the app in debug mode if this file is executed directly<br>if __name__ == &quot;__main__&quot;:<br>    app.run(debug=True)</pre><p>Test the api by running the main.py file</p><pre>python main.py</pre><p>and use the following code to verify if the api is responding.</p><pre>import requests <br><br>response = requests.post(&quot;http://localhost:5000/&quot;, files = {&#39;file&#39;: open(&#39;image_name.jpg&#39;,&#39;rb&#39;)})<br><br>print(response.text)</pre><p>when the api is working locally use pip freeze to copy all the dependencies.</p><pre>pip freeze &gt; requirements.txt </pre><p>Now we need to dockerize the application. I took the template from cloud run documentation. create a file with name Dockerfile and copy the following.</p><pre><br># Use the official lightweight Python image.<br># https://hub.docker.com/_/python<br>FROM python:3.11-slim<br><br># Allow statements and log messages to immediately appear in the Knative logs<br>ENV PYTHONUNBUFFERED True<br><br># Copy local code to the container image.<br>ENV APP_HOME /app<br>WORKDIR $APP_HOME<br>COPY . ./<br><br># Install production dependencies.<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Run the web service on container startup. Here we use the gunicorn<br># webserver, with one worker process and 8 threads.<br># For environments with multiple CPU cores, increase the number of workers<br># to be equal to the cores available.<br># Timeout is set to 0 to disable the timeouts of the workers to allow Cloud Run to handle instance scaling.<br>CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app</pre><p>The directory containing all the files should look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*VgXufZ_e8o_jpk7WSKm6eA.png" /></figure><p>Now I hope gcloud is initialized in your local laptop. if not use the gcloud shell and copy the whole directory to gcloud shell.</p><p>Open a terminal and make sure you are in api_developement directory.</p><p>Run the following commands and the image will be deployed on cloud run. In a serverless environment. In my case the service name is imagetotext.</p><p>In this case choose your &lt;service name&gt;</p><pre>gcloud builds submit - tag gcr.io/&lt;your project name&gt;/&lt;service name&gt; <br>gcloud run deploy - image gcr.io/&lt;your project name&gt;/&lt;service name&gt;- platform managed - memory 1024Mib</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QR-kaPfrFQDRDWyVPvP6Q.png" /></figure><p>Testing the api.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*elFjAcdH7_kjUuDfQeW83g.png" /></figure><p>Find the complete code at</p><p><a href="https://github.com/kshirsagarsiddharth/api_developement.git">GitHub - kshirsagarsiddharth/api_developement</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d5dc472bcfa" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/deploy-ml-models-in-10mins-9d5dc472bcfa">Deploy any LLM model in just 10 mins anywhere in the world.</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[E-commerce Segmentation and Journey Prediction]]></title>
            <link>https://siddharth1.medium.com/e-commerce-segmentation-and-journey-prediction-937ebf5d1c1?source=rss-cfc1f01d5bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/937ebf5d1c1</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[customer-service]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Siddharth Kshirsagar]]></dc:creator>
            <pubDate>Mon, 17 Apr 2023 08:49:00 GMT</pubDate>
            <atom:updated>2023-04-17T08:49:00.659Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQxuH0gs7PxpAzLvT3JpkA.jpeg" /><figcaption>Image by Bing AI</figcaption></figure><blockquote>Goal is to Utilize customer segmentation to generate a Sequence model that predicts future trends within each identified segment.</blockquote><h4>1. Project Architecture and Control flow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*goHDmtmFW_eHzz6e" /></figure><h4>2. E-Commerce Customer Segmentation</h4><p>Customer segmentation is the process of grouping customers according to how and why they buy. It allows organizations to create more specific sales and marketing strategies for customer groups. Customer segmentation is important because it helps businesses deliver more relevant customer experiences, understand the needs and motivations of customers, and launch more impactful and cost-effective campaigns.</p><p>Code in below notebook</p><p><a href="https://www.kaggle.com/code/rudrasing/onlineretailanalysis-customer-segmentation?kernelSessionId=125615446">OnlineRetailAnalysis - Customer Segmentation</a></p><h4>3. Journey Prediction</h4><p>Customer journey prediction refers to the process of using data and analytics to anticipate the path or steps that a customer may take in interacting with a business, from the initial awareness stage to making a purchase decision, and beyond. It involves predicting the customer’s behavior and needs at each touchpoint in order to optimize the customer experience and improve business outcomes.</p><p>Each customer has a unique journey in the segments. The starting point can be anything and the ending point can be anything. According to our data there are various combinations a customer can take.</p><p>Predicting customer journeys in the defined segments is important because we can use the insights to improve customer experience, optimize business, enhance marketing and sales strategy, and easily detect customer churn.</p><p><a href="https://www.kaggle.com/code/rudrasing/onlineretailanalysis-journey-prediction?kernelSessionId=126108033">OnlineRetailAnalysis - Journey Prediction</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=937ebf5d1c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>