<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[AI Product Navigator - Medium]]></title>
        <description><![CDATA[Cracking the Code of AI-Driven Products
 
 Welcome aboard! If you’re a product manager, AI enthusiast, or just someone who loves to see how technology shapes the future, this is your spot.
 
 We decode the complexities of AI in product management - Medium]]></description>
        <link>https://medium.com/ai-product-navigator?source=rss----0e2b8d223b02---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>AI Product Navigator - Medium</title>
            <link>https://medium.com/ai-product-navigator?source=rss----0e2b8d223b02---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:47:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/ai-product-navigator" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Natural Language Processing(Twitter)]]></title>
            <link>https://medium.com/ai-product-navigator/nlp-ff9bce856ef2?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/ff9bce856ef2</guid>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 18:11:48 GMT</pubDate>
            <atom:updated>2025-07-31T02:53:05.482Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Setting the working directory</h3><h3>2. Installing and Loading Required Packages</h3><p><strong>Installing packages</strong> = Getting tools from R’s app store (like downloading Excel add-ins).</p><p><strong>Loading libraries</strong> = Activating those tools so you can use them.</p><h3>3. Connecting R with Twitter API</h3><p>You give R the <strong>login keys</strong> (API keys) to connect with Twitter. This way, R is allowed to pull tweets from your account.</p><p>Example:</p><p>api_key &lt;- “NynzEtSE9h3xKkNBbDPzZfkA6”<br>api_secret &lt;- “kaK5gWedRPIXvnnSaSyCNLvt1z0KSnx9Hd4IHpLGDb4glVK1iW”<br>access_token &lt;- “141566165-BvVk09i8nQltpGRTqY6c04oYXVbZc1ylcwQtpkbZ”<br>access_token_secret &lt;- “Y1JZSpHYhhr3TUCC6WLQpPABPTY7tpivjCwB5xYRNctIE”</p><p>setup_twitter_oauth(api_key, api_secret, access_token, access_token_secret)</p><h3>4. Downloading Tweets</h3><p>You are <strong>searching Twitter</strong> for the keyword &quot;IPL2022&quot; and downloading up to <strong>10,000 tweets</strong>.</p><p>tweets = searchTwitter(“IPL2022”, n=10000)<br>tweets</p><p>searchTwitteR()</p><p>rd&lt;- userTimeline(“narendramodi”,n=100)<br>rd</p><p>This pulls the <strong>last 100 tweets from @narendramodi’s timeline.</strong></p><h3>5. Saving and Reading Tweet Data</h3><p>You <strong>convert tweets to a table</strong> (data.frame) and <strong>save it as a CSV file</strong>.</p><p>tweets.df &lt;- twListToDF(tweets)<br>write.csv(tweets.df,”tweet-IPL.csv”)<br>tweets = read.csv(“tweet-IPL.csv”)<br>tweets<br>View(tweets)<br>getwd()</p><h3>6. Preparing Tweets for Analysis (Text Cleaning)</h3><p>A <strong>corpus</strong> is a collection of text (like a book of tweets). You need it to clean and analyze the words.</p><p>corpus = Corpus(VectorSource(tweets$text))<br>corpus[[15]]$content<br>inspect(corpus[15])</p><h3><strong>7.Now we clean it/</strong>Prepare the Tweets (Text Preprocessing)<strong>:</strong></h3><p>This is like washing vegetables before cooking. You’re removing clutter to make tweets easier to analyze.</p><p><strong>#Lower Case</strong><br>corpus = tm_map(corpus, content_transformer(tolower))<br>corpus[[5]]$content</p><p><strong>Then Remove:</strong></p><ul><li><strong>URLs</strong></li><li><strong>Punctuations</strong></li><li><strong>Numbers</strong></li><li><strong>Stopwords</strong> like “the”, “is”, “at”</li><li><strong>Extra spaces</strong></li><li>And finally do <strong>stemming</strong> (e.g., “playing” becomes “play”)</li></ul><p>Why? So we focus only on meaningful words.</p><h3>8. Frequent word and associations</h3><p>Think of tweets like conversations — they’re full of words. But some words appear more often than others, especially around a specific topic (like “IPL”, “win”, or “team”).</p><p>So, when we <strong>find the most frequent words</strong>, we are trying to:</p><ol><li><strong>Understand what people are talking about</strong> the most.</li><li><strong>Identify themes or topics</strong> in the tweets.</li><li><strong>Filter out noise</strong> and focus only on meaningful, repeated words.</li></ol><p>For example, if 5,000 people tweet about IPL and most tweets mention “win”, “match”, and “Dhoni” — those are your <strong>key themes</strong>.You’re finding the words that appear most often across all tweets (like “match”, “win”, etc.).</p><p>A <strong>DTM (Document-Term Matrix)</strong> is like a big table that shows <strong>which words appear in which tweet</strong> and <strong>how many times</strong>.</p><p><strong>a) You create a matrix that tells: which tweet contains which word and how many times.</strong><br>frequencies = DocumentTermMatrix(corpus)</p><p>frequencies<br>dim(frequencies)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*ueJNIV77xeVG9GVPYTvmWg.png" /></figure><p><strong>b)This adds up all word counts across tweets.</strong></p><h3>9. <strong>Plot Word Frequencies</strong></h3><p><strong>#Inspect Frequent Words</strong></p><p><strong>This turns frequent words into a bar chart — helping you visually see which terms are popular.</strong><br>a) This gives you a list of words that appear at least 30 times: freq.terms = findFreqTerms(frequencies, lowfreq=30)<br>freq.terms<br>as.matrix(frequencies)</p><p>b) Convert the matrix to usable format:This counts how many times <strong>each word</strong> appears in total across all tweets.<br>term.freq = colSums(as.matrix(frequencies))</p><p>c) Filter to only those with frequency ≥ 30.You’re trimming the list to keep only frequently used words.<br>term.freq = subset(term.freq, term.freq&gt;=30)</p><p>d) Convert to a data frame for plotting.Creating a table with two columns:<br>df = data.frame(term=names(term.freq), freq=term.freq)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*y7xDRLInE6K_08mkvlgIFw.png" /></figure><p>e) Plot it using ggplot2</p><p>ggplot(df, aes(x = term, y=freq)) + geom_bar(stat=”identity”) + xlab(“Terms”) + ylab(“Count”) + coord_flip()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*aBADERyyvbqAJkMbHvtw8A.png" /></figure><h3>10. <strong>Find Word Associations</strong></h3><p>Shows what other words often appear with “win”, like “match”, “team”, etc.</p><p>findAssocs(frequencies, “win”, 0.2)</p><h4>What does 0.2 mean?</h4><ul><li>The number 0.2 is the <strong>correlation threshold</strong>.</li><li>It means: “Show me all words that appear with ‘win’ and have at least <strong>20% strength of association</strong>.”</li></ul><p>In simple terms:</p><p><em>“If a word appears alongside ‘win’ in tweets </em><strong><em>20% or more of the time</em></strong><em>, include it in the result.</em></p><h3>11. <strong>Create a Word Cloud</strong></h3><p>myDTM = DocumentTermMatrix(corpus, control = list(minWordLength=60))<br>m = as.matrix(myDTM)<br>v = sort(colSums(m), decreasing=TRUE)<br>myNames = names(v)<br>myNames<br>d = data.frame(word=myNames, freq=v)</p><p>wordcloud(d$word, d$freq, min.freq=30, random.color=TRUE, ordered.colors=TRUE)<br>wordcloud(names(term.freq), term.freq, min.freq=5, random.color=TRUE, ordered.colors=TRUE)</p><p>wordcloud(corpus, colors=rainbow(27),max.words=300)</p><h3>12. Sentimental Analysis</h3><p>It is loading <strong>positive and negative word lists</strong> into R — so you can compare words from tweets against them to judge whether a tweet sounds positive, negative, or neutral.</p><p>pos = scan(‘pos.words.txt’, what=’character’, comment.char=’;’)<br>pos<br>neg = scan(‘neg.words.txt’, what=’character’, comment.char=’;’)<br>neg</p><p>It opens and reads the file pos.words.txt which contains a <strong>list of positive words</strong>, one word per line (like good, happy, love etc.)</p><ul><li>what=&#39;character&#39; tells R that it should expect words (text), not numbers.</li><li>comment.char=&#39;;&#39; means any line starting with ; is treated as a comment and ignored (many word lists include notes or headers with ;).</li></ul><p><strong>Output</strong></p><p>pos: [1] “abundance” “accomplish” “achievement” “active” “admire” “adventure”</p><p>negative: [1] “abandon” “abnormal” “abolish” “abuse” “accident” “ache”</p><p><strong>So now </strong><strong>pos is a vector (list) of all positive words you can compare against tweet text.</strong></p><h3>13. Scoring</h3><p># making score function</p><p>Example<strong>: “Wow! What a fantastic match. India totally destroyed the opponents! #win”</strong><br> <br><strong> a)Clean the text</strong></p><p>clean up sentences with R’s regex-driven global substitute, gsub():<br> <br> sentence = gsub(‘[[:punct:]]’, ‘’, sentence)</p><p><strong>Removes punctuation:“Wow What a fantastic match India totally destroyed the opponents win”</strong></p><p>sentence = gsub(‘[[:cntrl:]]’, ‘’, sentence):<strong>No change here — there are no tabs or control characters in this tweet.</strong><br> sentence = gsub(‘\\d+’, ‘’, sentence):<strong>Removes digits — not applicable here, but if the tweet had “won by 10 runs”, it would remove the “10”.</strong><br> <br><strong>b)# and convert to lower case:</strong><br> sentence = tolower(sentence):<strong>”wow what a fantastic match india totally destroyed the opponents win”</strong></p><p><strong>c)Now the tweet becomes a list of words:</strong><br> # split into words. str_split is in the stringr package<br> word.list = str_split(sentence, ‘\\s+’)<br><strong>O/P:</strong> [“wow”, “what”, “a”, “fantastic”, “match”, “india”, “totally”, “destroyed”, “the”, “opponents”, “win”]</p><p>d) # sometimes a list() is one level of hierarchy too much<br> words = unlist(word.list)<br> <br>e) # compare our words to the dictionaries of positive &amp; negative terms<br> pos.matches = match(words, pos.words)<br> neg.matches = match(words, neg.words)</p><p>Assume:</p><ul><li>Your pos.words contains: &quot;wow&quot;, &quot;fantastic&quot;, &quot;win&quot;</li><li>Your neg.words contains: &quot;destroyed&quot; (in a negative context like &quot;ruined&quot;)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*T_0eyshp1T43d4xb2bHfKg.png" /></figure><p># match() returns the position of the matched term or NA<br> <br><strong>f. Convert to TRUE/FALSE</strong></p><p># we just want a TRUE/FALSE:<br> pos.matches = !is.na(pos.matches)<br> neg.matches = !is.na(neg.matches)</p><p><strong>Now becomes:</strong></p><ul><li>pos.matches → [TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE] → 3 positive words</li><li>neg.matches → [FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE] → 1 negative word</li></ul><p># and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():</p><p><strong>g.</strong> <strong>Calculate sentiment score</strong></p><p>score = sum(pos.matches) — sum(neg.matches)<br> return(score), }, pos.words, neg.words, .progress=.progress )</p><p><strong>Score = 3–1 = 2</strong><br> <br><strong>h) scores.df = data.frame(score=scores, text=sentences)</strong><br> <br> return(scores.df)<br>}<br>score.sentiment</p><p><strong>Output for this tweet will look like:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Kho9aRqKYUUwSctiOfNvzQ.png" /></figure><h4>In summary (super simple):</h4><ol><li><strong>Clean the tweet</strong></li><li><strong>Break it into words</strong></li><li><strong>Compare each word to a dictionary</strong></li><li><strong>Count positives and subtract negatives</strong></li><li><strong>Assign a sentiment score</strong></li></ol><p><strong>Now</strong></p><h3>Stage 1: Score Tweets and Save</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*5Yx8Khb4jC__x3wjPPLZdg.png" /></figure><ul><li><strong>Positive tweets</strong> have scores &gt; 0</li><li><strong>Negative tweets</strong> &lt; 0</li><li><strong>Neutral tweets</strong> = 0</li></ul><p>You’re saving this data to a <strong>CSV file</strong> (like Excel) and then reading it back into R for further processing.</p><h3>Stage 2: Plotting Tweet Sentiment Scores</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*csHC3v2mcf5s8pRe-mSX4A.png" /></figure><h3>Stage 3: Document-Term Matrix (DTM)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*MbH7n86zk-tUGY6sjMQGTw.png" /></figure><h3>Stage 4: Decision Tree(Model)</h3><p>This <strong>decision tree</strong> shows the logic used to predict sentiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*iz5_ErA5yE9F48NLvp_KrA.png" /></figure><h3><strong>Stage 5: Prediction Table</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*0BtDCEzYtFq3jTUCPCvOfA.png" /></figure><h3>Stage 6:Clustering(Hierarchial)</h3><h3>Stage 7: K-Means Clustering</h3><p>sparse_t = t(sparse)<br>distMatrix = dist(scale(sparse_t))<br>fit = hclust(distMatrix, method=”ward.D”)<br>plot(fit)<br>rect.hclust(fit, k=3)<br>k=3<br>kmeansResults = kmeans(sparse, k)<br>round(kmeansResults$centers, digits=3)</p><p>Tweets are grouped into 3 clusters — like topics: “positive”, “player talk”, “complaints”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*GZrG4OJ5F4ek9rxxpLR6Xw.png" /></figure><h3>Stage 8: Social Network Analysis (Word Graph)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*mz4ysEgbpbzzPLcjepoJGQ.png" /></figure><h4>What is an Adjacency Matrix?</h4><p>An <strong>adjacency matrix</strong> is a <strong>table that shows which items are connected to each other</strong>.</p><ul><li>In text mining, the <strong>items are words</strong>.</li><li>In social networks, the items can be <strong>people</strong>, <strong>tweets</strong>, or <strong>hashtags</strong>.</li></ul><p><strong>In your case:</strong></p><p>You’re building a word-to-word matrix that shows:</p><p><em>“Which words appear </em><strong><em>together in the same tweet</em></strong><em>?”</em></p><p>Let’s say these are your frequent words from tweets:</p><p>[“match”, “win”, “team”, “player”]</p><p>If <strong>“match”</strong> and <strong>“win”</strong> appear in the same tweet often, then the matrix might look like:</p><ul><li>1 means those two words appeared <strong>together</strong> in at least one tweet.</li><li>0 means they didn’t.</li><li>The diagonal is usually blank or marked - because a word can’t co-occur with itself</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*Nx8WTyFypF3YsY0zCCy2MQ.png" /></figure><h4>Why is this matrix important?</h4><p>1. Foundation of Social Graphs</p><ul><li>You <strong>convert this matrix into a graph</strong> (nodes and edges).</li><li>Helps visualize how words are <strong>connected</strong> in conversations.</li></ul><p>2. Reveals word relationships</p><ul><li>Words that <strong>often co-occur</strong> probably belong to <strong>the same topic</strong> or context.</li><li>E.g., “win” ↔ “match” ↔ “team” might relate to performance</li><li>“boo” ↔ “umpire” ↔ “mistake” might show complaints</li></ul><p>3. Used in algorithms</p><ul><li>Clustering</li><li>Community detection</li><li>Centrality (e.g., most “influential” words)</li></ul><p>4. Helps identify central or bridge words</p><ul><li>A word that connects multiple topics (e.g., “MS Dhoni”) might appear with both “win” and “captain” clusters.</li></ul><h4>In simple terms:</h4><blockquote><em>An adjacency matrix is like a </em><strong><em>friendship table for words</em></strong><em> — it shows who appears with whom in the tweets. You use this to </em><strong><em>build word networks</em></strong><em>, </em><strong><em>find themes</em></strong><em>, or </em><strong><em>visualize connections</em></strong><em>.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff9bce856ef2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/nlp-ff9bce856ef2">Natural Language Processing(Twitter)</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifier: Neural Nets]]></title>
            <link>https://medium.com/ai-product-navigator/classifier-neural-nets-255bd1563840?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/255bd1563840</guid>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 18:18:49 GMT</pubDate>
            <atom:updated>2025-07-28T02:52:18.317Z</atom:updated>
            <content:encoded><![CDATA[<p>Blackbox algo</p><p>Highly successful in terms of predictive accuracy</p><p>But very difficult in terms of<strong> interpretability.</strong> <strong>classification&lt;&lt;logistics&lt;&lt;neural nets</strong></p><p>Overfitting is a major danger with Neural networks</p><p>Can Model Complex relationships as the relationships are learned from data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*f5mpkxOQNuGoFQUPa7Cu2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*DkiLiptPnr3xfbVqosdFvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*A9HntkbZstvdZL9riAZrGQ.png" /></figure><p><strong>Output of each hidden node is a function of the weighted sum of inputs.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*ZmVEUULk60cA6fXiIbbT4w.png" /></figure><p>The weights q (theta) and w are typically initialized to random values in the range -0.05 to +0.05</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*jcS0X0en7RuQwrGIbXVmPw.png" /></figure><p><strong>Supervised Training set(with labels): Above is a training data set with 7 rows/observations with their labels/results. First/initially the results might go wrong because we take random weights and theta/bias but eventually/second time the results will start correcting itself and match with acceptance, perfecting the model.</strong></p><h3><strong>The Input Layer</strong></h3><p>• For input layer, <strong>input = output</strong></p><p>E.g., for record #1:</p><ul><li>Fat input = output = 0.2</li><li>Salt input = output = 0.9</li><li>Output of input layer = input into hidden layer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*aiBVUUKRhSXtXMijTmffTQ.png" /></figure><h3><strong>How we got the o/p of hidden layer as 0.43?/Activation function</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*9iblsvPPfS-0hOx4fhsbDw.png" /></figure><h3>→Theta means Bias</h3><h3>Output Layer</h3><p>The output of the last hidden layer becomes input for the output layer .</p><p>Uses same function as above, i.e. a function g of the weighted average</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*CwX0DyoMOQxPmURSmPdjhQ.png" /></figure><h3><strong>Mapping the output to a classification</strong></h3><p><strong>• Output = 0.506</strong></p><p>• If cutoff for a observation “1” is 0.5, then we classify as “1”</p><p><strong>How to select threshold as 0.5?</strong></p><p><strong>Weights are updated after each record is run through the network</strong></p><ul><li>Completion of all records through the network is one epoch (also called sweep or iteration)</li></ul><p>• After one epoch is completed, return to first record and repeat the process</p><h3>Error</h3><p>0.506(1–0.506)(1–0.506)</p><p>Error=0.123</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*KPuWca7GMW_6obR2UlHWkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*BQjT6aHpiRdUnO4xWKH-2A.png" /></figure><p>l=0.5</p><p><strong>Hidden layer Theta</strong></p><p>Theta(new)= Theta(old) + l(error)</p><p>= (-0.3) + (0.5 * 0.123)</p><p>= -0.2385</p><p><strong>Theta at last/output layer</strong></p><p>-0.015 + (0.5 * 0.123)</p><p>= 0.046</p><h4>What is a Neural Network?</h4><p>Imagine you’re trying to teach a machine how to <strong>make decisions</strong> — for example, to recognize whether a photo has a cat or a dog.</p><p>A <strong>neural network</strong> is just a clever system that <strong>learns from examples</strong>, kind of like how humans learn by seeing and practicing.</p><h4>How Does It Work? Think of it Like a Factory:</h4><p>Let’s say we’re building a “pet sorter machine.” Here’s how it works:</p><ol><li><strong>Input layer — The Sensors</strong><br> The machine looks at a picture. It reads <strong>features</strong> like:</li></ol><ul><li>Are there pointy ears?</li><li>Is there a tail?</li><li>What’s the fur pattern?</li></ul><p>These features go into the machine — like how ingredients go into a kitchen.</p><ol><li><strong>Hidden layers — The Kitchen Staff</strong><br> These are a bunch of tiny workers (called <strong>neurons</strong>) inside the machine.<br> They take the features, mix them in different ways, and pass the result forward.</li></ol><p>Some workers may say:</p><ul><li>“Hmm… this animal has pointy ears and a short tail — could be a cat.”</li><li>Others might think: “Wait, the body looks big — maybe a dog?”</li></ul><p>These workers are learning from <strong>many, many past examples</strong> to get better at guessing.</p><p><strong>3. Output layer — The Decision</strong><br> Finally, the machine says:</p><ul><li>“I’m 90% sure this is a cat.”</li><li>“I’m 10% sure it’s a dog.”</li></ul><p>The class with the highest score wins. So the machine calls it a <strong>cat</strong>.</p><h4>How Does It Learn?</h4><p>At first, the machine might be terrible — it guesses wrong a lot.</p><p>But here’s the trick: <strong>each time it makes a mistake</strong>, it learns by adjusting how much importance it gives to different features.<br> For example:</p><ul><li>If it wrongly said “dog” for a photo of a cat, it learns, “Ah! I gave too much importance to body size. Maybe I should focus more on ear shape.”</li></ul><p>This adjusting process happens <strong>again and again</strong>, like thousands of practice rounds — and the machine gets better over time.</p><p><strong>So What Is Classification?</strong></p><p>Classification is just a fancy word for <strong>sorting things into categories</strong>.</p><p>Examples:</p><ul><li>Is this email spam or not spam?</li><li>Is this voice command saying “Play” or “Pause”?</li><li>Is this transaction fraud or normal?</li></ul><p>Neural networks are great at this kind of task — <strong>they can figure out patterns even when humans can’t easily see them</strong>.</p><h4>Real-Life Example</h4><p>Let’s say you’re teaching a child to recognize fruits.</p><p>You show them apples, bananas, and grapes. At first, they get confused.</p><p>But over time, they notice:</p><ul><li>Apples are round and red</li><li>Bananas are long and yellow</li><li>Grapes are small and green or purple</li></ul><p>Eventually, they can tell what’s what just by looking.</p><p>A neural network does exactly that — just with numbers instead of eyes!</p><h3>But how are these weights chosen at the start?</h3><h4>The Input Layer — Sensors That Read Features</h4><p>Imagine a camera feeding inputs to the machine:</p><ul><li><strong>Pointy ears</strong> → value = 0.8</li><li><strong>Tail length</strong> → value = 0.3</li><li><strong>Fur pattern (spots/stripes)</strong> → value = 0.9</li></ul><p>These are <strong>normalized values between 0 and 1</strong>, like intensity levels — just like sensors give electric signals.</p><p>These values are the <strong>input layer neurons</strong>, say:</p><ul><li>x1 = 0.8 (ears)</li><li>x2 = 0.3 (tail)</li><li>x3 = 0.9 (fur)</li></ul><p>At <strong>first</strong>, they are randomly generated — usually small values like between -1 and 1.</p><p>Why? Because the model has no clue yet — it hasn’t learned anything. It’s like hiring new workers with no training.</p><p><strong>How It Learns (Backpropagation)</strong></p><p>When it guesses wrong (calls a cat a dog), it adjusts its weights slightly using a formula that considers:</p><ul><li>How wrong it was</li><li>What weights contributed to the mistake</li></ul><p>This is guided by <strong>learning rate (α)</strong> — which is like telling the workers how fast to improve.</p><p><strong>Summary in Human Words</strong></p><ul><li><strong>Input values</strong> = features from photo (ears, tail, fur…)</li><li><strong>Weights</strong> = how important each feature is for a neuron (initially random)</li><li><strong>Bias</strong> = personal tendency of each neuron to “lean” one way</li><li><strong>Activation</strong> = how strongly a neuron fires</li><li><strong>Learning</strong> = adjusting weights after mistakes</li></ul><h4>Factory Analogy Refresher (Simple View)</h4><ul><li><strong>Input layer</strong>: Like a scanner reading pet features (e.g., ear shape, fur length).</li><li><strong>Hidden layers</strong>: Like tiny chefs experimenting with combinations.</li><li><strong>Output</strong>: Like a final chef shouting, “90% cat, 10% dog.”</li></ul><h4>What Are Weights and Biases?</h4><ul><li><strong>Weights</strong> = importance the machine gives to each feature.<br> E.g., “pointy ears” might have a weight of +2, meaning it strongly signals <strong>cat</strong>.</li><li><strong>Bias</strong> = an extra push or offset. Think of it like adjusting a recipe even if ingredients are perfect.</li></ul><p>At first, <strong>weights and biases are random</strong> — like chefs guessing wildly at recipes.<br> (Usually between -1 and 1, or small numbers near 0.)</p><h4>So How Does It Learn?</h4><ol><li><strong>Initial Guess</strong>: With those random weights, it says:</li></ol><ul><li>“This might be 60% dog, 40% cat”</li></ul><ol><li><strong>Error Check</strong>: Oops! The actual answer was <strong>cat</strong>.</li></ol><ul><li>The machine sees it was wrong.</li></ul><ol><li><strong>Fixing Mistake</strong> (Backpropagation):</li></ol><ul><li>It goes backwards and says:</li><li>“I trusted ‘short tail’ too much. Maybe reduce that weight.”<br> “I ignored ‘pointy ears’. Let’s increase its weight!”</li></ul><h3>Enter: Learning Rate (Between 0 and 1)</h3><p>The <strong>learning rate</strong> decides <em>how much</em> the weights change during learning.</p><ul><li>If it’s too <strong>high</strong> (e.g., 0.9): it jumps around too wildly and might <strong>miss the best answer</strong>.</li><li>If it’s too <strong>low</strong> (e.g., 0.0001): it <strong>learns very slowly</strong>, like a stubborn chef.</li><li>Usually values like <strong>0.01 to 0.1</strong> work well — just enough to improve steadily.</li></ul><h3>Think of learning rate like aiming a dart at a bullseye.</h3><ul><li><strong>High learning rate (0.5 or 1)</strong>:<br> You throw the dart <em>very hard</em>. You might overshoot the target — one time you go too far right, next time too far left. The model may <strong>never settle</strong> or just miss the point entirely (divergence).</li><li><strong>Low learning rate (e.g., 0.001)</strong>:<br> You throw <strong>gently</strong> and adjust slowly. It might take <strong>many throws</strong> to get close, but you move in the right direction <strong>without overshooting</strong>.</li></ul><h3>So why not always go slow?</h3><ul><li>Too slow = training takes forever.</li><li>Too fast = model becomes unstable.</li></ul><p>Use tools like <strong>learning rate decay</strong> or <strong>adaptive optimizers</strong> (e.g., Adam) to adjust automatically over time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=255bd1563840" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/classifier-neural-nets-255bd1563840">Classifier: Neural Nets</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mining Relations across Records: Association and Collaborative]]></title>
            <link>https://medium.com/ai-product-navigator/mining-relations-across-records-association-and-collaborative-3ce673b0c0a5?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/3ce673b0c0a5</guid>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 11:53:30 GMT</pubDate>
            <atom:updated>2025-07-27T16:20:34.282Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Data Set(Grocery)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*PLRInYQyTbbICBF3nDFToQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*OLwvyfCvDjSCaDW9HmbJ4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*uvi5TKnhAVF7ZTbEMNpcpA.png" /></figure><h3>Step 1: Load &amp; Prepare Data</h3><p>a) Then you add a column called purchase with value 1 — this helps in reshaping the data later:</p><p>data$purchase = 1, data$purchase = as.integer(data$purchase)</p><p>b) You arrange the data by Member_number and Date to group all purchases per person per day:<br><strong>data = arrange(data, Member_number, Date)</strong><br>data = data[,c(1:3,8)] <strong>#Keep only Member_number, Date, itemDescription, purchase</strong><br>str(data)</p><h3>Step 2: Reshape Data for Mining</h3><p>a) You want a format like: <strong>basis Mumber number and Date</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*EUbEWo4HFJnf-PdaH4KVoA.png" /></figure><p>data.wide &lt;- pivot_wider(data, names_from = itemDescription, values_from = purchase, values_fn = list(purchase=min), values_fill = list(purchase=0))</p><p><strong>b) Now remove </strong><strong>Member_number and </strong><strong>Date since you&#39;re focusing only on the items:</strong></p><p>data.new = data.wide[,-c(1:2)] #Remove first two column<br>str(data.new)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*GFp7cGRshCcxGMFZLrOOpw.png" /></figure><p><strong>c) Convert this wide binary table to a format that </strong><strong>arules can understand:</strong><br>data.matrix = as.matrix(data.new)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*tCRG9ol8L8sQXkiChDSEIw.png" /></figure><p><strong>d) data.trans = as(data.matrix, “transactions”)</strong></p><p># convert the binary incidence matrix into a transactions database</p><p>inspect(data.trans): View the transactions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*7CAoOk1L6CvfctyTrgcW4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*li84Q2V8JHd6WcjOTN3bbw.png" /></figure><h3>Step 3: Apply Association Rule Mining</h3><p><strong>a) # when running apriori(), include the minimum support, minimum confidence, and target as arguments.</strong><br> <br>rules = apriori(data.trans, parameter = list(supp = 0.001, conf = 0.1, target = “rules”))</p><p>This means:</p><ul><li><strong>support ≥ 0.001</strong> (at least 0.1% of transactions have this combo)</li><li><strong>confidence ≥ 0.1</strong> (10% of the time this rule is true)</li><li><strong>target = rules</strong>: You&#39;re asking for &quot;if X, then Y&quot; rules.</li></ul><p><strong>b) Now you look at the top rules sorted by “lift” (how strong the rule is beyond chance):</strong></p><p>inspect(head(sort(rules, by = “lift”), n = 6)) #inspect the first six rules, sorted by their lift<br>inspect(sort(rules, by = “lift”)) #inspect the rules, sorted by their lift<br> <br><strong>c) You can also target rules where the right-hand side is “whole milk” or “yogurt”:</strong></p><p>rules = apriori(data.trans, parameter = list(minlen=2, maxlen = 6, supp=0.001, conf=0.1, target =”rules”), appearance = list(rhs= c(“whole milk”,”yogurt”), default=”lhs”), control = list(verbose=F))<br>rules.sorted = sort(rules, by=”lift”)<br>inspect(rules.sorted)</p><h3>Step 4: Prune Redundant Rules</h3><p>Sometimes, one rule is just a broader version of another. You identify such <strong>redundant rules</strong>:</p><p>subset.matrix = is.subset(rules.sorted, rules.sorted)<br>subset.matrix[lower.tri(subset.matrix, diag=T)] = FALSE<br>redundant = colSums(subset.matrix, na.rm=T) &gt;= 1<br>which(redundant)</p><p>#Remove redundant rules<br>rules.pruned = rules.sorted[!redundant]<br>inspect(rules.pruned)</p><p><strong>Now you have a clean set of strong, non-overlapping rules.</strong></p><h3>Step 5: Visualize the Rules</h3><p>You use various graph methods to make it visual and understandable:</p><p>a) plot(rules)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*nQd-yw6x4bjDVYhXfvr6xA.png" /></figure><p>b) plot(rules.pruned)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*wviB1nq0fOlL2Gtatd_L_g.png" /></figure><p>c) plot(rules, method=”graph”, control=list(type=”items”))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*TGOT99G_1r46fBlqXJok7A.png" /></figure><p>d) plot(rules.pruned, method=”graph”, control=list(type=”items”))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*nTZNb46gZXnJEG6oo3KOQQ.png" /></figure><p>e) plot(rules.pruned, method=”paracoord”, control=list(reorder=TRUE))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*rLzrjepDT1uUd_dcX-fsKA.png" /></figure><p>f) plot(rules.pruned, method=”grouped”)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*I455IRg9r1m9hUEge6Duog.png" /></figure><h3><strong>Imagine you’re a store owner. This code helps you discover:</strong></h3><ul><li>“People who buy <strong>eggs and milk</strong>, often buy <strong>bread</strong>.”</li><li>“Whole milk is the most common product bought with other items.”</li><li>Use this to <strong>bundle products</strong>, <strong>place items together</strong>, or <strong>run offers</strong>.</li></ul><h3>Summary</h3><ol><li>Loads grocery purchase data</li><li>Reshapes it into transaction format</li><li>Mines interesting patterns of items bought together</li><li>Removes noise (redundant rules)</li><li>Visualizes those rules for business insight</li></ol><h3>Collaborative filtering</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FFmtorg_dmM0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFmtorg_dmM0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FFmtorg_dmM0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/06fc0fadcb41d4d85e0e14f3fc2c9970/href">https://medium.com/media/06fc0fadcb41d4d85e0e14f3fc2c9970/href</a></iframe><p><strong>Collaborative filtering</strong> is a popular data science concept used heavily in <strong>recommender systems</strong>. The basic idea? If people have shown similar preferences in the past, there’s a good chance they’ll have similar tastes in the future too. So, if two users liked the same kind of stuff earlier, we assume they’re similar — and what one liked can be recommended to the other. It’s like how you might enjoy a show your friend recommended because you both usually like the same kind of content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*oxueJ2zBiDLT2icOTqmiCQ.png" /></figure><p>Let’s break this down with an example using a fictional video platform called <strong>Statflix</strong>, which streams shows about statistics.</p><p>Imagine we have three users — U1, U2, and U3 — and eight shows. Users rate shows on a scale from 1 to 5, but not all shows get rated by everyone, so we have some gaps in the rating matrix. Now, we want to recommend a new show to <strong>User 1 (U1)</strong>.</p><h3>The Problem and Two Approaches</h3><p>U1 has rated five out of the eight shows but hasn’t watched Show 4, 5, or 6 yet.</p><p><strong>1. Simple Average (Basic Method):</strong><br> You could just average out the ratings from other users for the shows U1 hasn’t seen.</p><ul><li>For <strong>Show 4</strong>, U2 gave it a 2 and U3 gave it a 5 → average = 3.5</li><li>For <strong>Show 5</strong>, same story → average = 3.5</li><li><strong>Show 6</strong> has no data → can’t say anything.</li></ul><p>The problem? This method doesn’t consider whether U1 is more like U2 or U3 — so we might end up recommending something U1 won’t like.</p><p><strong>2. Collaborative Filtering (Smarter Method):</strong><br> Here, we first figure out which users are most similar to U1, based on their shared ratings.</p><ul><li><strong>U1 vs U2:</strong> Their ratings are quite aligned, just minor differences. So, if U2 likes something, chances are U1 will too.</li><li><strong>U1 vs U3:</strong> Their ratings differ a lot. If U3 loves a show, U1 might not feel the same.</li></ul><h3>Measuring Similarity with Cosine Similarity</h3><p>To mathematically measure how similar users are, we use <strong>cosine similarity</strong>.</p><ul><li><strong>What is it?</strong> It’s the cosine of the angle between two users’ rating vectors.</li><li><strong>Smaller angle = more similar preferences</strong>, so the cosine value gets closer to 1.</li><li><strong>Bigger angle = less similarity</strong>, so the cosine value gets smaller.</li></ul><p><strong>In our example:</strong></p><ul><li>Cosine similarity between U1 and U2 (S12) = <strong>0.99</strong> → super high similarity</li><li>Cosine similarity between U1 and U3 (S13) = <strong>0.57</strong> → not that similar</li></ul><h3>Estimating Ratings Using a Weighted Average</h3><p>Now, to guess how U1 would rate a show they haven’t seen, we calculate a <strong>weighted average</strong>. Here, the weights are the similarity scores.</p><p><strong>Formula:</strong><br> Estimated rating = (S12 × rating by U2 + S13 × rating by U3) / (S12 + S13)<br> This step normalizes the influence, because S12 and S13 don’t sum to 1.</p><p><strong>Estimated Ratings:</strong></p><ul><li><strong>Show 4:</strong> R14 = 3.1 → lower than the simple average because U2 (who’s similar to U1) rated it poorly (2).</li><li><strong>Show 5:</strong> R15 = 3.9 → higher than the simple average because U2 liked it a lot.</li></ul><p>So, <strong>Show 5</strong> is the better recommendation for U1.</p><h3>What Do the Terms Really Mean?</h3><ul><li><strong>Filtering</strong> = predicting what a user might like</li><li><strong>Collaborative</strong> = using input from <em>other users</em> to make those predictions</li></ul><h3>Real-World Challenges with Collaborative Filtering</h3><p>Although collaborative filtering is intuitive, real-world systems face a few hurdles:</p><ol><li><strong>Sparsity:</strong><br> With tons of users and content, most users don’t rate much. That leads to a sparse rating matrix, which makes it hard to find enough overlap to identify similar users.</li><li><strong>Scalability:</strong><br> As the number of users or items grows, calculations like cosine similarity become expensive and slow. You need optimized algorithms and smart engineering to scale it up.</li><li><strong>Gray Sheep / Black Sheep Problems:</strong><br> Not all users fit neatly into clusters:</li></ol><ul><li><strong>Gray Sheep:</strong> Users who don’t align clearly with any group — hard to match them with similar users.</li><li><strong>Black Sheep:</strong> Outliers with completely unique tastes — nearly impossible to recommend reliably based on others.</li></ul><p>These aren’t problems limited to collaborative filtering — they’re challenges for any kind of recommender system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ce673b0c0a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/mining-relations-across-records-association-and-collaborative-3ce673b0c0a5">Mining Relations across Records: Association and Collaborative</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Clustering: Image Processing(Part- 2)]]></title>
            <link>https://medium.com/ai-product-navigator/ml-clustering-image-processing-part-2-9df614bd838a?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/9df614bd838a</guid>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 10:12:00 GMT</pubDate>
            <atom:updated>2025-07-27T10:12:00.616Z</atom:updated>
            <content:encoded><![CDATA[<h3>Step 1. Load &amp; View the Healthy Brain Image</h3><p>healthy = read.csv(“healthy.csv”, header=FALSE)<br>str(healthy)<br>healthyMatrix = as.matrix(healthy)<br>str(healthyMatrix)</p><p><strong>Plot image</strong><br>image(healthyMatrix,axes=FALSE,col=grey(seq(0,1,length=256)))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*EeLgX-91Cy5sKwDW3KZKIQ.png" /></figure><ul><li>Loads a CSV file with pixel brightness values (like 0 to 1 or 0 to 255) for a <strong>healthy brain MRI</strong>.</li><li>Converts it to a matrix → think of it as a <strong>grid of pixel intensities</strong>.</li><li>image() then <strong>displays the image</strong> in <strong>grayscale</strong> using grey(seq(0,1,256)).</li></ul><p><strong>Imagine</strong>: Each cell in the matrix = a small tile in the MRI image.</p><h3>Step 2: Prepare for Clustering</h3><p>healthyVector = as.vector(healthyMatrix)<br>distance = dist(healthyVector, method = “euclidean”)</p><ul><li>Converts 2D matrix → 1D vector (like flattening a photo into a list of pixel values).</li><li>Tries to compute <strong>Euclidean distances</strong> between these values.</li></ul><h3>Step 3: Kmeans clustering</h3><p>k = 5</p><p># Run k-means<br>set.seed(1)<br>KMC = kmeans(healthyVector, centers = k, iter.max = 1000)</p><p>str(KMC)<br>KMC$centers</p><ul><li>We want to group pixel values into <strong>5 clusters</strong> (e.g., dark areas, light areas, etc.).</li><li>kmeans() groups <strong>similar pixel intensities together</strong>.</li><li>Useful to find <strong>regions with similar texture or brightness</strong> (like tissues vs tumor).</li></ul><h3>Step 4:Visualize the Clustered Image</h3><p><strong># Extract clusters</strong><br>a) healthyClusters = KMC$cluster<br>healthyClusters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*K9rf8er4jk0RsjVzGRgkug.png" /></figure><p>KMC$centers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*E2xOZhdmNBd5LMdyvAZxwA.png" /></figure><p><strong>b) # Plot the image with the clusters</strong><br>dim(healthyClusters) = c(nrow(healthyMatrix), ncol(healthyMatrix))</p><ul><li>We reshape the 1D cluster labels back to the <strong>original image shape</strong>.</li><li>image() now shows clusters as colorful regions.</li><li>Each color = a group of similar pixels.</li></ul><p>Even if the MRI was grayscale, now we see <strong>distinct regions</strong> with different colors.</p><p><strong>c) image(healthyClusters, axes = FALSE, col=rainbow(k))</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*GjHLh7OKQ-UzCPub_SvSWw.png" /></figure><p><strong>d) image(healthyClusters, axes = FALSE)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*7zZDeG3SQ_FsYlRRXw55Sg.png" /></figure><h4>Layman Analogy:</h4><p>Imagine you have a coloring book of the brain:</p><ul><li>The <strong>first image</strong> is like filling each region with <strong>different colors</strong>, so it’s easy to tell them apart.</li><li>The <strong>second image</strong> is like keeping it black and white or just shades — still the same picture, but not easy to differentiate.</li></ul><h3>Step 5:Apply Clusters to Tumor Image</h3><h4>Load Tumor Image</h4><p>tumor = read.csv(“tumor.csv”, header=FALSE)<br>str(tumor)<br>tumorMatrix = as.matrix(tumor)<br>tumorVector = as.vector(tumorMatrix)</p><p>Same process as before: loading and flattening the tumor image.<br>str(tumorVector)</p><h3>Step 6: Predict Cluster Labels for Tumor Image- new Test data/Tumor data</h3><p>install.packages(“flexclust”)<br>library(flexclust)</p><p>a) KMC.kcca = as.kcca(KMC, healthyVector)<br><strong>KMC: </strong>A <strong>sorting machine</strong> that has learned how to group brain pixels into <strong>k = 5 clusters</strong> — based on the <strong>healthy brain image</strong>.</p><p>It knows:</p><ul><li>What each cluster looks like (from the healthy brain),</li><li>And which pixel types belong to which cluster.</li></ul><h4>Think of KCCA as:</h4><blockquote><em>Wrapping your </em><em>KMC </em><strong><em>sorting machine</em></strong><em> in a </em><strong><em>prediction-friendly toolbox</em></strong><em> (called </em><em>kcca) from the </em><em>flexclust package.</em></blockquote><p>This toolbox makes it easy to:</p><ul><li><strong>Apply the same clustering logic</strong> to <strong>new images</strong> (like tumor brains).</li><li>Predict which cluster each new pixel should belong to</li></ul><p><strong>Converts the </strong><strong>KMC model into a form (</strong><strong>kcca) that can predict on new data.</strong></p><p><strong>Applies the healthy brain’s cluster logic to the tumor image.</strong></p><p><strong>This is like saying: <em>“Use what we learned from the healthy brain to analyze the tumor image.”</em></strong></p><p><strong>b) tumorClusters = predict(KMC.kcca, newdata = tumorVector)</strong></p><p><strong>It’s like saying:</strong></p><blockquote><em>“Hey, here’s a </em><strong><em>new brain image (tumor)</em></strong><em>. Use what you learned from the </em><strong><em>healthy brain</em></strong><em> to </em><strong><em>sort each pixel</em></strong><em> into the closest cluster.”</em></blockquote><p>So you’re <strong>not retraining</strong> — just <strong>reusing the cluster logic</strong> learned from the healthy image.</p><h4>Layman Analogy: Sorting Apples and Oranges</h4><ol><li>🍏 You train your eyes using a basket of <strong>only apples</strong>.</li><li>You learn there are 5 types of apples — red, green, yellow, bruised, small.</li><li>That’s your KMC model.</li></ol><p>Now:<br> 4. 🍎🍊 You get a new mixed basket with <strong>apples + oranges</strong>.<br> 5. You don’t want to relearn sorting.<br> 6. So you use your trained apple-sorting brain (KMC.kcca) to <strong>classify items in the new basket</strong>.</p><p>Of course, oranges might not fit perfectly — but you’ll still categorize based on what you know about apples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*eEmm3N3soXlxuhSEFi3EmQ.png" /></figure><h3>Step 7:Visualize the clusters</h3><p><strong>You’ll now see how the tumor image is broken into regions/clusters. This can help identify unusual regions (like potential tumors) based on how different they are from healthy clusters.</strong><br>a) dim(tumorClusters) = c(nrow(tumorMatrix), ncol(tumorMatrix))</p><p>b) image(tumorClusters, axes = FALSE, col=rainbow(k))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*ZxjGBmLUL0KC7NpKvEBSIw.png" /></figure><p>c) image(tumorClusters, axes = FALSE)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*ty3w0vgdiN6wS223AaEvCg.png" /></figure><h3>Step 8 :Optional- View Original Tumor Image in Grayscale</h3><p>healthy = read.csv(“tumor.csv”, header=FALSE)<br>str(healthy)<br>healthyMatrix = as.matrix(healthy)<br>str(healthyMatrix)</p><p><strong># Plot image</strong><br>image(healthyMatrix,axes=FALSE,col=grey(seq(0,1,length=256)))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*OBIRn8NmcH0y3T9EKa4H8g.png" /></figure><h3>SUMMARY</h3><ol><li>Load healthy brain image.</li><li>Use <strong>K-Means Clustering</strong> to divide it into 5 color-coded zones based on pixel brightness.</li><li>Train a model based on these healthy zones.</li><li>Then apply that model to a <strong>tumor image</strong> to <strong>highlight unusual zones</strong> (possibly showing the tumor).</li></ol><p>It’s a simple way to <strong>segment brain regions</strong> and spot the difference between healthy and suspicious areas — using just pixel color patterns.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9df614bd838a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/ml-clustering-image-processing-part-2-9df614bd838a">ML Clustering: Image Processing(Part- 2)</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML: Clustering(Image Processing)]]></title>
            <link>https://medium.com/ai-product-navigator/ml-clustering-image-processing-b4782069326e?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/b4782069326e</guid>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 09:21:35 GMT</pubDate>
            <atom:updated>2025-07-27T09:23:40.355Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re taking a <strong>grayscale image of a flower</strong> (values between 0 and 1) and using <strong>clustering</strong> to group the pixels into 3 meaningful segments — like background, flower petals, and maybe the stem — <strong>based on brightness</strong>.</p><p>These values are <strong>between 0 and 1</strong>, which is typical for <strong>normalized grayscale images</strong>, where:</p><ul><li>0 = black</li><li>1 = white</li></ul><p>You only have <strong>2 dimensions</strong> (rows and columns).</p><ul><li>A <strong>color (RGB) image</strong> would need <strong>3 dimensions</strong>: width × height × 3 (for R, G, B channels).</li><li>Since there’s <strong>no third dimension</strong>, it’s <strong>not</strong> color.</li></ul><h3>Step 1: <strong>Preliminary Work — Performing Clustering on a tiny image</strong></h3><p>setwd(“D:/ADM/practice ADM/Session 11 and 12 — Clustering and Association Mining/Clustering/Image Processing”)<br>getwd()</p><p><strong><em>Flower Data set: No variables, only data points(50 rows *50 col)=2500 points</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*MFlnSlyFj-qylCcQE6vPMA.png" /></figure><p>flower = read.csv(“flower.csv”, header=FALSE)<br>str(flower)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*t6TQ1pg4O7nFlSP0nfo07w.png" /></figure><h3>Step 2:Change the data type to matrix</h3><p>flowerMatrix = as.matrix(flower)<br>str(flowerMatrix)<br><strong>image(flowerMatrix, axes = FALSE): </strong>Converts the image into a matrix (think of it like an Excel grid), and then plots it. axes=FALSE just hides the tick marks.</p><p>In R, the image() function <strong>automatically applies a color scale</strong>, usually a <strong>rainbow</strong> or <strong>heatmap-like</strong> palette, <strong>unless you manually define it using the </strong><strong>col parameter</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*QZtAlLriCp4nqvnji-Vcdw.png" /></figure><h4>Why did you get a colorful flower image, even though the data is grayscale?</h4><p>R showed you a <strong>colorful image</strong> not because your original data had color — <br> but because <strong>R chose to color it automatically</strong> to make it look nicer or easier to understand.</p><p>Your actual image data:</p><p>Your file (flower.csv) contains only <strong>numbers between 0 and 1</strong>.</p><ul><li>These numbers represent <strong>brightness</strong> — like how light or dark each pixel is.</li><li>0 means <strong>black</strong>, 1 means <strong>white</strong>, and values in between are <strong>shades of grey</strong>.</li></ul><p>This is what we call a <strong>grayscale image</strong>. It’s like a black-and-white photo — no color at all.</p><p>What happened when you used image(flowerMatrix)?</p><p>The image() function in R says:<em>“Hey, I see numbers… I’ll give each unique number a different color to help you </em><strong><em>see the difference</em></strong><em> more clearly.</em></p><p>So instead of showing shades of grey, it says:</p><ul><li>“Let me make 0 red”</li><li>“Maybe 0.5 will be orange”</li><li>“And 1 will be yellow”</li></ul><p>That’s why you saw a <strong>colorful flower</strong> — not because the flower was colored, but because <strong>R painted</strong> it that way on your screen to help you understand it better.</p><p>So it’s not real color — it’s “pseudo-color” (fake color for visual effect).</p><p>Just like a <strong>weather map</strong> shows temperatures in red and blue — not because the air is red or blue, but to help us read it more easily.</p><h3>Step 3:Turn matrix into a vector</h3><p>flowerVector = as.vector(flowerMatrix)<br>str(flowerVector) :<strong> Flattens the 2D image matrix (50x50 = 2500 pixels) into a long 1D list of 2500 numbers. Now each pixel is a single data point.</strong></p><p>→ num [1:2500] 0.0991 0.0991 0.1034 0.1034 0.1034 …</p><h3>Step 4: Compute distances</h3><p>a. distance = dist(flowerVector, method = “euclidean”) :<strong> Measures how different each pixel is from every other pixel using the Euclidean formula (i.e., &quot;as-the-crow-flies&quot; distance). This helps group similar-looking pixels.</strong><br>b. str(distance)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*--WE7psS6Nx_ObO090qHLQ.png" /></figure><h3>Step 5: Hierarchical clustering</h3><p>clusterIntensity = hclust(distance, <strong>method=”ward.D”</strong>)</p><p><strong># Plot the dendrogram</strong><br>plot(clusterIntensity)</p><p><strong>Creates a hierarchical clustering tree — like a family tree of pixel brightnesses. The </strong><strong>ward.D method tries to form tight clusters.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*2YvAy-7sswh146LHALyAOQ.png" /></figure><h3>Step 6: Select 3 clusters</h3><p>a) rect.hclust(clusterIntensity, k = 3, border = “red”) :<strong> Draws red boxes to split the tree into 3 groups (clusters). Each pixel is assigned to one of the 3 clusters.</strong><br>b) flowerClusters = cutree(clusterIntensity, k = 3)<br>c) flowerClusters</p><p><strong>Why 3 clusters?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*hbic09pvgcuKUsBWlSNoeA.png" /></figure><h3>Step 7: Find mean intensity values</h3><p>a) tapply(flowerVector, flowerClusters, mean): <strong>Shows the average brightness level in each cluster. Useful to know how light/dark each group is.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*ZnYWDefLfN9zABgQz83-sg.png" /></figure><p>b) table(flowerClusters) — <strong>number of pixels in each cluster</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/1*s-2stixENxMaA1xiNFvCRA.png" /></figure><h3>Step 8: Plot the image and the clusters</h3><p>a) dim(flowerClusters) = c(50,50): <strong>Reshapes the cluster assignments back into image form (50x50) so you can see the clustered image.</strong><br>b) image(flowerClusters, axes = FALSE)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*g5JKTJPxMGvHOT3OcDJMfQ.png" /></figure><h3>Step 9: Original image</h3><p>image(flowerMatrix, axes = FALSE)<br>image(flowerMatrix,axes=FALSE,col=grey(seq(0,1,length=256)))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*NxXlE6yMYPNUTAbYW02XRw.png" /></figure><p>These two lines replot the <strong>original grayscale image</strong>, using a range of grey shades for better visualization</p><p>Imagine taking a black-and-white photo of a flower:</p><ol><li>You turn it into numbers based on brightness.</li><li>You use <strong>clustering</strong> to group parts of the image that look similar.</li><li>You end up with a <strong>segmented version</strong> of the image, where each part is labeled — like “dark background”, “bright flower”, etc</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4782069326e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/ml-clustering-image-processing-b4782069326e">ML: Clustering(Image Processing)</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unsupervised ML: Clustering]]></title>
            <link>https://medium.com/ai-product-navigator/unsupervised-ml-clustering-3a87cc1e48bf?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/3a87cc1e48bf</guid>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 08:11:48 GMT</pubDate>
            <atom:updated>2025-07-27T08:45:19.764Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>i) Kmeans</strong></p><p><strong>ii) Kmedian</strong></p><p><strong>iii) Hierarchial</strong></p><p><strong>iv) Image processing</strong></p><p><strong>Cluster analysis</strong> is a set of multivariate techniques mainly used to <strong>group similar items or objects</strong> based on the characteristics they share.</p><p>In simpler terms, it tries to find <strong>natural groupings or patterns</strong> in data — for example, grouping customers by behavior, cities by climate, or products by features.</p><p><strong>Multivariate</strong>: Imagine you’re analyzing people — not just by <strong>age</strong>, but also by their <strong>income</strong>, <strong>education</strong>, <strong>spending habits</strong>, and <strong>location</strong>. That’s <strong>multiple variables</strong>.</p><p>So instead of looking at one thing (like just age), you’re looking at <strong>many things together</strong> to find patterns</p><p>It’s also known by other names like:</p><ul><li><strong>Q analysis</strong></li><li><strong>Typology construction</strong></li><li><strong>Classification analysis</strong></li><li><strong>Numerical taxonomy</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*ybN-uK_k_SJigetWqt9gHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*XomUoS65pHTm3VoysSkAAw.png" /></figure><p><strong>MARKET SEGMENTATION FOR AIRLINES</strong></p><p>Market segmentation is a strategy that divides a broad target market of customers into smaller, more similar groups, and then designs a marketing strategy specifically for each group. Clustering is a common technique for market segmentation since it automatically finds similar groups given a data set. In this problem, we’ll see how clustering can be used to find similar groups of customers who belong to an airline’s frequent flyer program. The airline is trying to learn more about its customers so that it can target different customer segments with different types of mileage offers.</p><p>The file AirlinesCluster.csv contains information on 3,999 members of the frequent flyer program. This data comes from the textbook “Data Mining for Business Intelligence”</p><p>There are seven different variables in the dataset, described below:</p><p>· <em>Balance</em> = number of miles eligible for award travel</p><p>· <em>QualMiles</em> = number of miles qualifying for TopFlight status</p><p>· <em>BonusMiles</em> = number of miles earned from non-flight bonus transactions in the past 12 months</p><p>· <em>BonusTrans</em> = number of non-flight bonus transactions in the past 12 months</p><p>· <em>FlightMiles</em> = number of flight miles in the past 12 months</p><p>· <em>FlightTrans</em> = number of flight transactions in the past 12 months</p><p>· <em>DaysSinceEnroll</em> = number of days since enrolled in the frequent flyer program</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*bfHgeyOxPBgtyyWbPVkYig.png" /></figure><p><strong>PREPARING THE DATASET</strong></p><p>In this problem, we will normalize our data before we run the clustering algorithms. Why is it important to normalize the data before clustering?</p><h3>Step 1: Read data</h3><p>setwd(“D:/ADM/practice ADM/Session 11 and 12 — Clustering and Association Mining/Clustering/Market Segmentation”)<br>getwd()</p><p>data = read.csv(“Airlines.csv”)<br><strong>str(data)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*stizLiIl5bcYaIhGbdweLw.png" /></figure><p><strong>Summary(Data): It tells me the outliers</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*D8ObcGwKyhgmJt6SvwqWsw.png" /></figure><h3><strong>Step 2 : Normalization</strong></h3><p><strong>Standard deviation is 1 after scaling.By subtracting the mean and dividing by the standard deviation for each data point, you are effectively centering the data around 0 and adjusting its scale. The result is a distribution with a mean of 0 and a standard deviation of 1.This transformation is beneficial in machine learning because it helps algorithms that are sensitive to the scale of features converge faster and perform better.</strong></p><p>data1 = scale(data)##min max scaling , z score normalization, robust scaling<br>data1 = as.data.frame(data1)<br>str(data1)<br>summary(data1)<br>sd(data1$BonusMiles)</p><h3>Step 3: Hierarchial Clustering</h3><p>i) distances = dist(data1, <strong>method=”euclidean”</strong>)<br>clusterdata = hclust(distances, <strong>method=”ward.D”</strong>)<br>plot(clusterdata)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*_e_Qf2ByWIfNXVHHDgvuaQ.png" /></figure><p>ii) clusterGroups = cutree(clusterdata, <strong>k=2</strong>) <strong>#2clusters assumed</strong></p><p><strong># change it to k=3 i.e 3 clusters and check</strong><br>clusterGroups<br>clusterGroups[10] = 1<br>clusterGroups[7] = 1<br>clusterGroups[3100] = 2<br>clusterGroups[1:8] = 1 1 1 1 1 1 1 1<br>data2 = cbind(data,clusterGroups) #<strong>ClusterGroups is another column or variable added you can say to existing data(classified in to 1 or 2)</strong><br>data2 <strong>#3888 rows present</strong><br>str(data2)</p><p><strong>table(data2$clusterGroups)</strong> # Two clusters are present 1 and 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/115/1*yjg4PHBvS4zE4fLHLIttpQ.png" /></figure><p>iii)<strong> It gives me mean of balances of rows which belong to cluster 1 and cluster 2 respectively</strong></p><p>tapply(data$<strong>Balance</strong>, clusterGroups, mean)</p><p><strong>Cluster 1 mean balanace→ 89830.8</strong></p><p><strong>Cluster 2 mean balanace→ 42318.6</strong></p><p>tapply(data$QualMiles, clusterGroups, mean)<br>tapply(data$BonusMiles, clusterGroups, mean)<br>tapply(data$BonusTrans, clusterGroups, mean)<br>tapply(data$FlightMiles, clusterGroups, mean)<br>tapply(data$FlightTrans, clusterGroups, mean)<br>tapply(data$DaysSinceEnroll, clusterGroups, mean)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*QV1q5pGgS0MswemDa7K3Mg.png" /></figure><h3>Step 4: Kmeans Clustering</h3><p><strong>i) Set clusters</strong></p><p>k=4 (4clusters assumption)<br>set.seed(1000)<br>data1 = scale(data)<br>data1 = as.data.frame(data1)<br>str(data1)</p><p><strong>ii) Running Kmeans algo</strong></p><p>KMC = kmeans(data1, centers=4, iter.max=1000)<br><strong>KMC</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*zRzIcMuK42O-Ul3uDLhfkg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*NVD_xHFOLrrQuOb8uObZXw.png" /></figure><p><strong>iii)Explore clustering results</strong></p><p>KMC$cluster: clustering vector(Above image)<br>KMC$size: 699 1463 1694 143<br>KMC$centers: Above images<br>KMC[[1]]</p><p><strong>iv) Add cluster info back to original data</strong></p><p>Now you attach the cluster number as a <strong>new column</strong> in your original dataset. So every customer now has a cluster tag.<br>data3 = cbind(data,KMC$cluster): Add column” cluster” to existing data<br>str(data3)</p><p><strong>v) Understand each cluster’s behavior</strong></p><ul><li>This checks the <strong>average value</strong> of each feature for each cluster.</li><li>Example: “What’s the average balance of customers in cluster 1?” — helps you interpret what each group is like.</li><li><em>It’s like creating </em><strong><em>profiles for each group</em></strong><em> based on their behavior</em></li></ul><p><strong>It gives me mean of balances of rows which belong to cluster 1 and cluster 2 respectively</strong></p><p>tapply(data3$Balance, KMC$cluster, mean)<br>tapply(data3$QualMiles, KMC$cluster, mean)<br>tapply(data3$BonusMiles, KMC$cluster, mean)<br>tapply(data3$BonusTrans, KMC$cluster, mean)<br>tapply(data3$FlightMiles, KMC$cluster, mean)<br>tapply(data3$FlightTrans, KMC$cluster, mean)<br>tapply(data3$DaysSinceEnroll, KMC$cluster, mean)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*Seee09dpZsqb5RTXEkCa4Q.png" /></figure><p><strong>vi) Visualize clusters</strong></p><ul><li>You plot customers based on their <strong>Balance and BonusMiles</strong>.</li><li>Each point is <strong>colored based on its cluster</strong>.</li></ul><p>Code: plot(data[c(“Balance”,”BonusMiles”)], col = KMC$cluster)</p><p>· <em>Balance</em> = number of miles eligible for award travel</p><p>· <em>BonusMiles</em> = number of miles earned from non-flight bonus transactions in the past 12 month</p><p><strong>Balance (Award Miles):</strong></p><p>This is the <strong>total number of miles</strong> a person has in their frequent flyer account that they can use to <strong>book free or discounted flight tickets</strong>.</p><p>Think of it like your wallet:</p><ul><li>If you have 50,000 miles in your balance, you might be able to get a free flight.</li></ul><p><strong>BonusMiles (Non-flight Miles):</strong></p><p>These are <strong>extra miles you earned without flying</strong> — like a <strong>reward</strong> for doing other things:</p><ul><li>Swiping your credit card,</li><li>Shopping online</li><li>Staying at partner hotels,</li><li>Dining at partner restaurants.</li></ul><p>So even if you didn’t fly much, you could <strong>still earn BonusMiles</strong> through offers and promotions.</p><p>Example:</p><p><em>You flew once and earned 5,000 miles.<br> You used your airline credit card for shopping and earned 10,000 bonus miles.<br> Your </em><strong><em>Balance</em></strong><em> = 15,000 miles<br> Your </em><strong><em>BonusMiles</em></strong><em> = 10,000 miles (just the non-flight part)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*wwX29rnlnZ5b3L4ozM4vdQ.png" /></figure><p>The reason we chose to visualize the clusters using only <strong>Balance</strong> and <strong>BonusMiles</strong> is mainly for simplicity and interpretability. These two variables are often highly informative in customer datasets — <strong>Balance</strong> might reflect how much a customer is spending or saving with the service, and <strong>BonusMiles</strong> could indicate engagement or rewards activity. Plotting them against each other allows us to get a quick, intuitive sense of how the customers are grouped based on two important dimensions.</p><p>Another practical reason is that we’re limited to 2D plots in basic scatterplots. Since K-Means works in multiple dimensions (here maybe 5 or 6 variables), we can’t easily visualize all dimensions at once. So we pick two that are likely to show good separation between clusters. If other variable pairs (like DaysSinceEnroll vs BonusTrans) show more interesting separation, we can also visualize those. But starting with Balance and BonusMiles is common because they often represent monetary value and customer engagement — two core aspects of clustering in loyalty or airline datasets.</p><p>vi)</p><ul><li>Adds the <strong>cluster centers</strong> (the “average” point of each group) to the plot.</li><li>pch=8 shows them as star symbols, and cex=2 makes them bigger.</li></ul><blockquote><em>So the plot shows </em><strong><em>how the groups are separated</em></strong><em>, and where their centers are.</em></blockquote><p>points(KMC$centers[,c(“DaysSinceEnroll”,”BonusMiles”)], col=1:3, pch=8, cex=2)</p><p>Star mark at the bottom left corner of axis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*Ce2R7tvoJ0hXK8N89HEp9w.png" /></figure><h3><strong>Interpretaion</strong></h3><p>Cluster 1: High spenders, long-time customers<br>Cluster 2: Low mileage, low balance, newer members</p><h4>How to interpret this plot:</h4><p><strong>i) Clusters (colors)</strong></p><ul><li>Each color represents a different <strong>cluster</strong> found by k-means.</li><li>Even though you may have used more variables to create the clusters, you’re <strong>visualizing only two</strong> (Balance and BonusMiles), so this is just a 2D slice of the clustering.</li></ul><p><strong>ii) Patterns:</strong></p><ul><li>Most of the data points are clustered in the bottom-left — meaning <strong>low balance and low bonus miles</strong>.</li><li>A few outliers (dots far to the right or top) have very <strong>high balance or high bonus miles</strong>, and they might form separate small clusters.</li><li>Some points may seem to overlap in color or look indistinct — that’s normal since k-means works in <strong>multi-dimensional space</strong>, but you’re projecting only 2D.</li></ul><p><strong>iii) The large stars or symbols (✳)</strong></p><ul><li>These usually represent the <strong>centroids</strong> (the center points) of each cluster. They show the average balance and bonus miles for customers in each cluster.</li></ul><p><strong>Why </strong><strong>Balance and </strong><strong>BonusMiles were used here:</strong></p><ul><li>These are <strong>continuous variables</strong> and often provide a good spread for visual interpretation.</li></ul><p>The person who wrote this might have used them because they visually show the customer types:</p><ul><li><strong>Low-value frequent flyers</strong></li><li><strong>High-spending occasional users</strong></li><li><strong>Premium customers with high both</strong></li></ul><p>If you had used other combinations (like FlightMiles12mo vs FlightTrans12), the graph would look different and might reveal other behaviors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a87cc1e48bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/unsupervised-ml-clustering-3a87cc1e48bf">Unsupervised ML: Clustering</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI, ML, NLP & Deep Learning: A Friendly Guide to Cutting-Edge Algorithms]]></title>
            <link>https://medium.com/ai-product-navigator/ai-ml-nlp-deep-learning-a-friendly-guide-to-cutting-edge-algorithms-a047c110de53?source=rss----0e2b8d223b02---4</link>
            <guid isPermaLink="false">https://medium.com/p/a047c110de53</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificalintelligence]]></category>
            <category><![CDATA[product-manager]]></category>
            <dc:creator><![CDATA[Smruti @Product]]></dc:creator>
            <pubDate>Mon, 17 Mar 2025 17:12:55 GMT</pubDate>
            <atom:updated>2025-03-17T17:12:55.675Z</atom:updated>
            <content:encoded><![CDATA[<p>Artificial Intelligence (AI) is everywhere — powering search engines, recommending movies, driving cars, and even helping businesses make smarter decisions. But AI isn’t just one thing — it’s an umbrella term that includes <strong>Machine Learning (ML), Natural Language Processing (NLP), Deep Learning, and even non-ML AI algorithms</strong> like rule-based systems and search optimization.</p><p>Let’s break it down in a way that makes sense, whether you’re a tech enthusiast, a business leader, or someone just curious about AI’s potential.</p><h3>1. AI: The Big Picture</h3><p>AI is the concept of machines simulating human intelligence. Some AI systems learn from data (<strong>ML</strong>), while others rely on predefined rules, logic, or even nature-inspired techniques.</p><h3>2. Machine Learning (ML): Teaching Computers to Learn from Data</h3><p>Machine Learning enables computers to recognize patterns and make decisions without being explicitly programmed.</p><h3>Supervised Learning (Learning from labeled examples)</h3><ul><li><strong>Linear Regression</strong> — Predicts numbers (e.g., housing prices).</li><li><strong>Logistic Regression</strong> — Helps with yes/no decisions (e.g., spam detection).</li><li><strong>Decision Trees &amp; Random Forest</strong> — Like a flowchart for decision-making.</li><li><strong>SVM (Support Vector Machines)</strong> — Draws a boundary between categories.</li></ul><h3>Unsupervised Learning (Finding hidden patterns)</h3><ul><li><strong>K-Means Clustering</strong> — Groups similar things (e.g., customer segmentation).</li><li><strong>PCA (Principal Component Analysis)</strong> — Reduces data complexity.</li></ul><h3>Reinforcement Learning (Learning by trial and error)</h3><ul><li><strong>Q-Learning &amp; Deep Q Networks (DQN)</strong> — AI for gaming, self-driving cars.</li></ul><h3>Collaborative Filtering (Recommendation Systems)</h3><ul><li><strong>User-Based Collaborative Filtering</strong> — Finds similar users (Netflix, Spotify).</li><li><strong>Item-Based Collaborative Filtering</strong> — Suggests similar products (Amazon).</li><li><strong>Matrix Factorization (SVD, ALS)</strong> — Used in YouTube and music recommendations.</li></ul><h3>3. Natural Language Processing (NLP): Teaching AI to Understand Text &amp; Speech</h3><h3>Text Understanding &amp; Representation</h3><ul><li><strong>TF-IDF</strong> — Finds important words in documents.</li><li><strong>Word2Vec</strong> — Converts words into meaningful numbers.</li><li><strong>BERT</strong> — Google’s AI model for understanding search queries.</li></ul><h3>Language Generation</h3><ul><li><strong>GPT (Generative Pre-trained Transformer)</strong> — AI that writes like a human.</li><li><strong>T5 (Text-to-Text Transfer Transformer)</strong> — Versatile NLP model for multiple tasks.</li></ul><h3>Other NLP Capabilities</h3><ul><li><strong>Named Entity Recognition (NER)</strong> — Extracts names, places, and dates.</li><li><strong>Sentiment Analysis</strong> — Detects emotions in text (positive, negative, neutral).</li></ul><h3>4. Deep Learning: Neural Networks That Learn Like the Human Brain</h3><p>Deep Learning is a subset of ML that mimics how humans learn through layered neural networks.</p><h3>For Images &amp; Vision (Image Processing)</h3><ul><li><strong>CNNs (Convolutional Neural Networks)</strong> — Used in facial recognition, object detection.</li><li><strong>ResNet, VGG, AlexNet</strong> — Advanced image classification models.</li><li><strong>YOLO (You Only Look Once)</strong> — AI for real-time object detection.</li><li><strong>U-Net</strong> — Used in medical imaging (e.g., detecting tumors in scans).</li></ul><h3>For Sequential Data &amp; Speech Recognition</h3><ul><li><strong>RNNs (Recurrent Neural Networks)</strong> — Used in voice assistants (Siri, Alexa).</li><li><strong>LSTMs (Long Short-Term Memory)</strong> — Helps AI remember long-term dependencies.</li></ul><h3>For Creativity &amp; AI-Generated Content</h3><ul><li><strong>GANs (Generative Adversarial Networks)</strong> — AI that creates art, deepfakes, music.</li><li><strong>VAEs (Variational Autoencoders)</strong> — Generates realistic synthetic images.</li></ul><h3>5. AI Algorithms That Are NOT Machine Learning</h3><p>AI is not just ML — some AI algorithms don’t learn from data but still perform intelligent tasks.</p><h3>Expert Systems (Rule-Based AI)</h3><ul><li><strong>Dendral</strong> — Early AI for chemical analysis.</li><li><strong>MYCIN</strong> — AI for medical diagnosis.</li></ul><h3>Search Algorithms (Finding the Best Path)</h3><ul><li><em>A (A-Star) Algorithm</em>* — Finds the shortest path (GPS navigation).</li><li><strong>Dijkstra’s Algorithm</strong> — Optimizes routes in networks (Google Maps).</li><li><strong>Minimax Algorithm</strong> — AI for strategy games like chess.</li><li><strong>PageRank Algorithm</strong> — Google’s search ranking algorithm.</li><li><strong>BM25 (Best Matching 25)</strong> — Used in document retrieval.</li></ul><h3>Logic-Based AI (Formal Reasoning)</h3><ul><li><strong>Propositional Logic &amp; First-Order Logic (FOL)</strong> — Used in AI chatbots.</li><li><strong>Fuzzy Logic</strong> — Helps AI handle uncertainty (used in home appliances).</li></ul><h3>Evolutionary Algorithms (AI Inspired by Nature)</h3><ul><li><strong>Genetic Algorithms (GA)</strong> — Used in optimization (e.g., engine design).</li><li><strong>Swarm Intelligence (Ant Colony, Particle Swarm Optimization)</strong> — AI inspired by nature.</li></ul><h3>Planning Algorithms (AI that Thinks Ahead)</h3><ul><li><strong>STRIPS (Stanford Research Institute Problem Solver)</strong> — AI for task automation.</li><li><strong>Monte Carlo Tree Search (MCTS)</strong> — AI for board games like Go, chess.</li></ul><h3>Final Thoughts: AI is More Than Just Machine Learning</h3><p>AI is shaping the future — whether it’s through self-learning models (<strong>ML, NLP, Deep Learning</strong>) or rule-based and search-driven techniques (<strong>Expert Systems, Logic-Based AI</strong>).</p><p>From <strong>Netflix recommendations</strong> to <strong>autonomous vehicles</strong>, from <strong>Google Search</strong> to <strong>AI art generators</strong>, these algorithms are powering the world around us.</p><p>If you’re in <strong>tech, business, or just AI-curious</strong>, understanding these foundations will help you navigate and leverage AI’s potential.</p><p><strong>What’s your favorite AI application? Or which of these algorithms interests you the most? Let’s discuss in the comments!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a047c110de53" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-product-navigator/ai-ml-nlp-deep-learning-a-friendly-guide-to-cutting-edge-algorithms-a047c110de53">AI, ML, NLP &amp; Deep Learning: A Friendly Guide to Cutting-Edge Algorithms</a> was originally published in <a href="https://medium.com/ai-product-navigator">AI Product Navigator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>