<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jo Chen on Medium]]></title>
        <description><![CDATA[Stories by Jo Chen on Medium]]></description>
        <link>https://medium.com/@jo.ym?source=rss-9adbf55f1d0e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O-W5Cf_K_3vKa2rJB9E9Xg.png</url>
            <title>Stories by Jo Chen on Medium</title>
            <link>https://medium.com/@jo.ym?source=rss-9adbf55f1d0e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:13:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jo.ym/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Information Retrieval, Recommendation Systems & AI Agents: Three Solutions to the Same Problem]]></title>
            <link>https://blog.gopenai.com/information-retrieval-recommendation-systems-ai-agents-three-solutions-to-the-same-problem-5f708a36d9ee?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f708a36d9ee</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Tue, 20 Jan 2026 07:29:23 GMT</pubDate>
            <atom:updated>2026-01-21T06:44:16.487Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why Understanding IR is the Foundation for Building Better Agents</strong></p><blockquote><strong><em>Note:</em></strong><em> If you’re already well-versed in Information Retrieval (IR) or recommendation systems, you can skip this article. This piece is written for PMs, engineers, and anyone curious about AI who wants to understand the shared foundations behind search, recommendations, and AI agents — without getting lost in academic jargon.</em></blockquote><blockquote><em>We’ll use real-world products you already know — Spotify, Netflix, Tinder, Uber Eats, Google Maps — to explain the concepts. Of course, the actual systems behind these products are far more complex, but the simplified versions capture the essential ideas.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BUayL503JlqnjcRo" /><figcaption>Photo by <a href="https://unsplash.com/@trnavskauni?utm_source=medium&amp;utm_medium=referral">Trnava University</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Think about what you did online today:</p><ul><li>You Googled “best ramen near me”</li><li>Netflix showed you a row of shows “Because you watched…”</li><li>You asked ChatGPT to explain a concept</li></ul><p>These three products look completely different. But underneath, they’re solving the exact same problem:</p><blockquote><strong><em>Finding what you need from a massive pile of information.</em></strong></blockquote><p>The only difference? <strong>How you tell the system what you want.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*6RrcoRxRZTlsWOwzzjjD2Q.png" /></figure><p>This article argues that search, recommendations, and AI agents are siblings — born from the same parents, just wearing different clothes. If you’re building AI agents (especially RAG systems), understanding Information Retrieval (IR) will save you from reinventing wheels and falling into well-known traps.</p><h3>The Core Insight: Three Interfaces, One Problem</h3><p>Here’s the key realization: all three systems follow the same fundamental flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*ey9HD9h_jMQNIv-rwXW8PA.png" /></figure><p>Let’s map each system to this framework:</p><ul><li><strong>Search</strong>: Query → Retrieve matches → Rank by relevance → Show list</li><li><strong>Recommendation</strong>: Behavior history → Infer preferences → Rank by likelihood → Show list</li><li><strong>Agent/RAG</strong>: Conversation → Understand intent → Retrieve context → Generate answer</li></ul><h4>Modern Analogies</h4><p><strong>The Spotify Example:</strong></p><ul><li><strong>Search</strong> = You type “aespa” in the search bar</li><li><strong>Recommendation</strong> = Your “Discover Weekly” playlist — you didn’t ask for it, Spotify just knows</li><li><strong>Agent</strong> = You tell ChatGPT “Give me upbeat songs for a morning run” and it curates a list</li></ul><p><strong>The Dating App Example:</strong></p><ul><li><strong>Search</strong> = Searching for a specific Instagram handle</li><li><strong>Recommendation</strong> = Tinder automatically showing you potential matches</li><li><strong>Agent</strong> = Telling an AI “I like people who are into playing music and hate small talk” and getting personalized suggestions</li></ul><p>Same underlying problem. Different user interfaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*kRFA6TSm1mdvxi79WtbuJA.png" /></figure><h3>The Four Pillars</h3><p>Four core technologies power all three systems. Let’s break them down without the jargon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*-73nbggsAIUAwM2GZ8c7Aw.png" /></figure><h4>Embedding: Turning Things into Coordinates</h4><p><strong>In plain terms:</strong> Convert text, images, or products into a set of numbers (coordinates). Similar things end up with similar coordinates.</p><blockquote><strong><em>Technical term:</em></strong><em> These coordinates are called </em><strong><em>vectors</em></strong><em>, and the space they live in is called </em><strong><em>embedding space</em></strong><em> or </em><strong><em>latent space</em></strong><em>. When we say two items are “similar,” we’re measuring distance using metrics like </em><strong><em>cosine similarity (</em></strong>This is just one of many methods.)</blockquote><p><strong>Analogy: Spotify’s Hidden Music Map</strong></p><p>Imagine Spotify has a secret map of all music:</p><ul><li>Hip-hop songs cluster in one region</li><li>Classical music sits in another corner</li><li>Taylor Swift might be somewhere between “Pop” and “Country”</li><li>When you play a song, Spotify finds other songs nearby on this map</li></ul><p><strong>Instagram Version:</strong></p><ul><li>Food bloggers cluster together</li><li>Fitness influencers occupy another zone</li><li>Someone who does both? They’re in the middle</li></ul><p><strong>Why it matters:</strong> Computers don’t understand “meaning,” but they understand “distance.” Once everything has coordinates, we can calculate “who’s similar to whom.”</p><h4>ANN Search: Finding Neighbors Among Millions</h4><p><strong>In plain terms:</strong> When you have 100 million coordinate points, how do you quickly find the 10 closest ones?</p><blockquote><strong><em>Technical term:</em></strong><em> This is called </em><strong><em>Approximate Nearest Neighbor (ANN)</em></strong><em> search. Popular implementations include </em><strong><em>FAISS</em></strong><em> (Facebook), </em><strong><em>HNSW</em></strong><em>, </em><strong><em>ScaNN</em></strong><em> (Google), and </em><strong><em>Annoy</em></strong><em> (Spotify). The “approximate” part means we trade a tiny bit of accuracy for massive speed gains.</em></blockquote><p><strong>Analogy: Uber Eats Finding Nearby Restaurants</strong></p><p>When you open Uber Eats, it doesn’t calculate the distance to every restaurant in the country. That would take forever.</p><p>Instead:</p><ol><li>It first checks which zone you’re in</li><li>Then only calculates distances within that zone</li><li>This is “smart indexing” — narrow down first, then compute precisely</li></ol><p><strong>Why it matters:</strong> Without this, every search would take minutes (comparing against 100 million items one by one). With it, you get results in milliseconds.</p><h4>Ranking &amp; Reranking: Rough Cut + Fine Selection</h4><p><strong>In plain terms:</strong> First use a cheap method to grab a batch of candidates, then use an expensive method to pick the best ones.</p><blockquote><strong><em>Technical term:</em></strong><em> The first stage is called </em><strong><em>retrieval</em></strong><em> or </em><strong><em>candidate generation</em></strong><em> (optimizing for </em><strong><em>recall</em></strong><em> — not missing good items). The second stage is </em><strong><em>ranking</em></strong><em> or </em><strong><em>reranking</em></strong><em> (optimizing for </em><strong><em>precision</em></strong><em> — putting the best items at the top). Tools like </em><strong><em>Cohere Rerank</em></strong><em>, </em><strong><em>cross-encoders</em></strong><em>, or </em><strong><em>LLM-as-judge</em></strong><em> are commonly used for reranking.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGIuJU57eanuaNiLHZnLPg.png" /></figure><p><strong>Analogy: The Tinder Workflow</strong></p><ul><li><strong>Round 1 (Recall):</strong> Quickly swipe left on obvious no’s — fast, low effort</li><li><strong>Round 2 (Rank):</strong> From the right-swipes, decide who gets the Super Like — slower, more thoughtful</li></ul><p>You can’t deeply evaluate every single person. So you do it in stages.</p><p><strong>Restaurant Version:</strong></p><ul><li><strong>Rough filter:</strong> Google Maps — “4+ stars, within 1 km”</li><li><strong>Fine selection:</strong> Read reviews, look at photos, decide where to eat tonight</li></ul><p><strong>Why it matters:</strong> Balancing speed and quality. You can’t use your most sophisticated evaluation method on 100 million items.</p><h3>Query Understanding: Figuring Out What You Actually Want</h3><p><strong>In plain terms:</strong> Does “Apple” mean the fruit or the company? How near is “nearby”?</p><blockquote><strong><em>Technical term:</em></strong><em> This field is called </em><strong><em>Query Understanding (QU)</em></strong><em> or </em><strong><em>Natural Language Understanding (NLU)</em></strong><em>. It includes techniques like </em><strong><em>query expansion</em></strong><em> (adding synonyms), </em><strong><em>query rewriting</em></strong><em> (reformulating the question), </em><strong><em>intent classification</em></strong><em>, and </em><strong><em>entity recognition</em></strong><em>. In LLM-based systems, this often happens implicitly during prompt processing.</em></blockquote><p><strong>Analogy: Group Chat Dynamics</strong></p><p>You text your friends: “What should we eat later?”</p><p>A good friend automatically considers:</p><ul><li>What time is it?</li><li>Where is everyone?</li><li>What did we eat yesterday?</li><li>Who’s vegetarian?</li></ul><p>They don’t reply with “Food” — that would be useless.</p><p><strong>Another Example:</strong></p><p>You say “Send me that thing.”</p><p>Your friend instantly knows you mean the meme you were laughing about 5 minutes ago.</p><p>Systems need this same “understanding context” ability.</p><p><strong>Why it matters:</strong> Literal meaning ≠ actual intent. Misunderstand the question, and everything downstream is wasted effort.</p><h3>How They Differ: A Comparison</h3><p>Now that we see the similarities, let’s clarify the differences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0cLanrGB-0luNDbwBwKHA.png" /></figure><p>The key insight: <strong>personalization intensity increases as you move from search to recommendation to agents.</strong> Search can work without knowing who you are. Recommendation fundamentally depends on it. Agents are currently somewhere in between but moving toward heavy personalization.</p><h3>Why Agent Developers Should Care About IR</h3><p>Let’s bring this back to the practical question: why should someone building AI agents study Information Retrieval?</p><h4>RAG is Just Search in Disguise</h4><p>RAG stands for <strong>Retrieval-Augmented Generation</strong>.</p><p>That first word — Retrieval — is exactly what IR has been perfecting for decades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D90u2seZARvkMkrsMPgJZQ.png" /></figure><p>When your RAG pipeline produces bad answers, the problem is often not the LLM. It’s that you retrieved the wrong context. The LLM is just confidently summarizing garbage.</p><p><strong>Debugging tip:</strong> Before blaming model, check what your retriever actually returned.</p><h4>Personalized Agents Are the Trend</h4><p><strong>Just my own opinion.</strong></p><p>The agents of tomorrow will remember:</p><ul><li>Your preferences</li><li>Your past conversations</li><li>Your working style</li></ul><p>This is exactly what recommendation systems do. The techniques for modeling user preferences, handling cold-start problems, and balancing exploration vs. exploitation — all developed in the recommendation domain — will become essential for agent builders.</p><p>Memory + Personalization = The next frontier for agents.</p><h4>Common Pitfalls Are Shared</h4><p>The same mistakes plague all three systems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cR_ibHj79fqdK_pdGgql2g.png" /></figure><p>If you’ve solved these in one domain, you’ve half-solved them in the others.</p><h3>Practical Takeaways</h3><p>Here’s what you can actually do with this knowledge:</p><h4>1. Learn Embeddings First</h4><p>This is foundation-level knowledge. Understand how text becomes coordinates, why some models work better for certain domains, and how to evaluate embedding quality.</p><h4>2. Understand the Recall vs. Precision Trade-off</h4><p>More results ≠ better. Recall means “did we find all relevant items?” Precision means “are the items we returned actually relevant?” You usually can’t maximize both. Know which matters more for your use case.</p><h4>3. Don’t Rely Solely on Vector Search</h4><p>Hybrid search (combining vector similarity with keyword matching) almost always outperforms pure vector search. That exact product ID or technical term? Vector search might miss it. Keywords won’t.</p><h4>4. Reranking Offers Great ROI</h4><p>Adding a reranker on top of your initial retrieval is often the highest-impact improvement you can make. It’s cheaper than fine-tuning embeddings and faster to implement.</p><h4>5. Query Understanding Is Underrated</h4><p>Most tutorials skip this. But in production, understanding what the user actually wants — expanding abbreviations, resolving ambiguity, injecting context — can double your system’s effectiveness.</p><h3>Conclusion</h3><p>Let’s recap the core message:</p><p><strong>Search, recommendation, and AI agents are three interfaces to the same underlying problem</strong> — finding relevant information from a large corpus.</p><p>The technologies that power them overlap significantly:</p><ul><li><strong>Embeddings</strong> turn content into comparable coordinates</li><li><strong>ANN search</strong> makes large-scale similarity search fast</li><li><strong>Ranking</strong> separates good results from great ones</li><li><strong>Query understanding</strong> bridges what users say and what they mean</li></ul><p>If you’re building agents, especially RAG-based ones, the IR community has decades of hard-won lessons waiting for you. You don’t need to reinvent approximate nearest neighbor algorithms or figure out why vector-only search misses obvious matches.</p><p>Personalization is also becoming non-negotiable. The techniques from recommendation systems — user modeling, preference learning, contextual bandits — will increasingly show up in agent architectures.</p><p><strong>Next time you see Google search results, a Netflix recommendation, or a ChatGPT answer, remember:</strong></p><blockquote><em>They’re all doing the same thing — just with different interfaces.</em></blockquote><p>Understanding one helps you understand all three. And if you’re building the next generation of AI agents, that understanding isn’t optional — it’s foundational. Again, this is just my own opinion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f708a36d9ee" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/information-retrieval-recommendation-systems-ai-agents-three-solutions-to-the-same-problem-5f708a36d9ee">Information Retrieval, Recommendation Systems &amp; AI Agents: Three Solutions to the Same Problem</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond One-Size-Fits-All RAG: Why Different Knowledge Sources Need Different Retrieval Strategies]]></title>
            <link>https://blog.gopenai.com/beyond-one-size-fits-all-rag-why-different-knowledge-sources-need-different-retrieval-strategies-355f4fe7897e?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/355f4fe7897e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Tue, 06 Jan 2026 07:54:14 GMT</pubDate>
            <atom:updated>2026-01-07T05:40:35.777Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Building a production RAG system that handles knowledge bases, product catalogs, and compliance rules with tailored approaches</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yfi6Rkf8OLNXUlYM" /><figcaption>Photo by <a href="https://unsplash.com/@alinnnaaaa?utm_source=medium&amp;utm_medium=referral">Alina Grubnyak</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Imagine you’re building a librarian, a product recommender, and a compliance officer — all in one AI system.</p><p>The librarian needs to find specific paragraphs from lengthy brand guidelines and product manuals. The product recommender needs to suggest “related reading” from hundreds of blog posts and product pages. The compliance officer needs to catch forbidden phrases like “guaranteed cure” while understanding context.</p><p>When I started building an AI-powered product, I made a classic mistake: I treated all these knowledge sources the same way. Chunk everything into pieces, embed with the same model, retrieve by similarity. Done.</p><p><strong>It didn’t work.</strong></p><p>The librarian kept returning irrelevant chunks that had lost their context. The product recommender suggested articles based on shallow keyword overlap rather than true semantic relevance — or worse, <strong>hallucinated</strong> articles that didn’t even exist. The compliance officer missed obvious violations because vector similarity doesn’t care about exact keyword matches.</p><p>This article shares what I learned after rebuilding the retrieval system. The key insight? <strong>Different knowledge sources deserve different retrieval strategies. </strong>What works for a 50-page PDF doesn’t work for compliance rules.</p><p>I’ll walk you through:</p><ul><li>Why the standard RAG approach falls short for heterogeneous knowledge sources</li><li>Three tailored strategies we implemented for knowledge bases, platform content, and compliance</li><li>Anti-hallucination techniques that work across all sources</li><li>The cost/performance trade-offs we navigated</li></ul><p>Let’s dig in!</p><h3>Overview: RAG and the Hallucination Problem</h3><h4>What is RAG?</h4><p><strong>RAG (Retrieval-Augmented Generation)</strong> is like giving an open-book exam instead of relying on memory.</p><p>Without RAG, you ask an LLM: “What’s our return policy?” and it confidently invents one. With RAG, the system first retrieves your actual return policy document, then asks the LLM to answer based on that document.</p><p>The basic flow is simple:</p><pre>User Query → Embed → Search Vector DB → Retrieve Documents → LLM Generates Answer</pre><p>Think of it as the difference between asking someone who “read the manual once” versus someone who has the manual open in front of them.</p><h4>The Hallucination Problem</h4><p>LLMs are confident liars. They don’t know what they don’t know.</p><p>In our product, I’ve seen the model:</p><ul><li>Invent URLs that return 404s (e.g., `/blog/ultimate-guide-to-xyz` that doesn’t exist)</li><li>Cite statistics that sound plausible but are completely fabricated</li><li>Describe product features that products don’t have</li><li>Create internal links to articles that were never written</li></ul><p><strong>RAG helps — but it’s not a silver bullet.</strong> Even with retrieved documents, models can:</p><ul><li>Misinterpret the retrieved content</li><li>Blend retrieved facts with hallucinated details</li><li>Ignore retrieved content when it contradicts their “beliefs”</li></ul><h4>The Evolution of RAG</h4><p>RAG has evolved through several generations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*SOnr5Fo6x2pC19tp0m_jYA.png" /></figure><p>Most tutorials teach Naive RAG: split documents into chunks, embed everything the same way, and use cosine similarity to retrieve. It works for demos but struggles in production.</p><p><strong>Our approach is Modular RAG with source-specific strategies.</strong> Each knowledge source gets its own retrieval pipeline optimized for its unique characteristics.</p><h3>The Three Knowledge Sources</h3><p>Before diving into specifics, here’s the landscape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rbSy_Cuzpa64syLvinXMw.png" /></figure><h4>Why Not One Strategy for All?</h4><p>Think about how you’d search differently in these scenarios:</p><ol><li><strong>Finding a quote in a book</strong>: You need precision. You scan chapters, look for keywords, read surrounding context.</li><li><strong>Recommending a Netflix show</strong>: You need relevance. “Similar to X” isn’t about keywords — it’s about themes, vibes, audience.</li><li><strong>Checking legal compliance</strong>: You need both. “Contains prohibited substance” requires exact keyword matching, but “implies medical benefits” requires understanding.</li></ol><p>Same principle applies to RAG. Let’s see how.</p><h3>Knowledge Base: Contextual Retrieval</h3><h4>The Problem: Lost in Chunks</h4><p>When you chunk a 50-page brand guide into 500-token pieces, each chunk becomes an orphan. Consider this chunk:</p><blockquote>“Our return policy allows 30-day returns for unused items with original packaging.”</blockquote><p>Useful, right? But which document is this from? Is it the 2023 policy or the 2024 update? Does it apply to all products or just electronics?</p><p>The chunk has lost its context. And when the LLM retrieves it, it might apply this policy to products that have different rules.</p><p>This is the “<strong>lost in the middle</strong>” problem — and it gets worse with longer documents.</p><h4>The Solution: Contextual Retrieval</h4><p>Anthropic introduced <a href="https://www.anthropic.com/news/contextual-retrieval">Contextual Retrieval</a> to address this. The idea is simple: <strong>before embedding each chunk, prepend a context prefix that explains where this chunk comes from</strong>.</p><p>Instead of embedding:</p><blockquote>“Our return policy allows 30-day returns…”</blockquote><p>We embed:</p><blockquote>“This chunk is from the 2024 Brand Guidelines, Section 4: Customer Policies, covering return procedures for electronics. Our return policy allows 30-day returns…”</blockquote><p>Here’s how we implemented it:</p><pre>@property<br>def text_for_embedding(self) -&gt; str:<br>    &quot;&quot;&quot;Combine context prefix with content for embedding.&quot;&quot;&quot;<br>    return f&quot;{self.context_prefix}\n{self.content}&quot;</pre><p>The context prefix is generated by an LLM in batch (15 chunks per call to manage costs):</p><pre># Prompt for generating context prefixes<br>&quot;&quot;&quot;<br>Generate brief context prefixes (under 30 words each).<br>Focus on: document source, section/topic covered.<br>Example: &quot;This chunk from Brand Guide covers logo usage guidelines.&quot;<br>&quot;&quot;&quot;</pre><h4>Smart Chunking</h4><p>Not all chunk boundaries are equal. Cutting mid-sentence creates nonsensical chunks. We implemented smart boundary detection:</p><pre>@staticmethod<br>def __find_split_point(text: str, target: int, search_range: int) -&gt; int:<br>    &quot;&quot;&quot;Find the best split point near target position.<br><br>    Priority: sentence end (.) &gt; newline (\n) &gt; hard cut<br>    &quot;&quot;&quot;<br>    best_point = target<br>    best_priority = 0<br><br>    for i in range(target, max(0, target - search_range), -1):<br>        char = text[i - 1] if i &gt; 0 else &quot;&quot;<br>        if char in &quot;.!?&quot; and i &lt; len(text) and text[i] == &quot; &quot;:<br>            return i  # Best: end of sentence<br>        elif char == &quot;\n&quot; and best_priority &lt; 1:<br>            best_point = i<br>            best_priority = 1  # Good: paragraph break<br><br>    return best_point  # Fallback: hard cut</pre><h4>Hybrid Search</h4><p>Pure vector search misses keyword matches. Pure keyword search misses semantic similarity. We use both:</p><pre>WITH scores AS (<br>    SELECT<br>        c.*,<br>        (1 - (c.embedding &lt;=&gt; $2::vector)) AS semantic_score,<br>        ts_rank(<br>            to_tsvector(&#39;english&#39;, c.content),<br>            plainto_tsquery(&#39;english&#39;, $3)<br>        ) AS keyword_score<br>    FROM knowledge_chunks c<br>    WHERE c.document_id IN (SELECT id FROM docs WHERE project_id = $1)<br>)<br>SELECT *,<br>    (0.7 * semantic_score + 0.3 * keyword_score) AS combined_score<br>FROM scores<br>ORDER BY combined_score DESC<br>LIMIT 10</pre><p>The 70/30 split was tuned empirically — semantic similarity matters more, but keywords provide precision.</p><h4>Trade-off Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoIyu9YL7s5mZ04zch1-TQ.png" /></figure><p><strong>Why no LLM reranking?</strong> Knowledge base searches typically return 5–10 results. Running an LLM to rerank such a small set costs more than it improves quality. We save the LLM budget for where it matters — platform content.</p><h3>Platform Content: LLM Summary + Rerank</h3><h4>The Problem: Whole-Document Relevance</h4><p>The blog platform returns structured data: title, tags, handle, HTML body. When recommending “related articles,” we need whole-document relevance, not paragraph-level matching.</p><p>Consider: a user writes an article about “sustainable packaging.” The internal linking system should suggest:</p><ul><li>An article about “eco-friendly shipping materials” (thematically related)</li><li>A product page for “biodegradable mailers” (relevant product)</li><li>NOT an article that happens to mention “packaging” once in passing</li></ul><p>This requires understanding the document as a whole, not matching keywords.</p><h4>The Solution: No Just Chunking, LLM Summary</h4><p>Instead of chunking, we generate a summary of each article/product and embed that:</p><pre># For articles<br>text_for_embedding = f&quot;{title} {summary} {tags}&quot;<br><br># For products (include product_type for category relevance)<br>text_for_embedding = f&quot;{title} {summary} {tags} {product_type}&quot;</pre><p>But what about long articles? We use tiered summarization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgDlANuzYeg0Xcx1yHoB7g.png" /></figure><p>This keeps LLM input manageable while preserving document meaning.</p><h4>LLM Reranking</h4><p>Here’s where we spend the LLM budget. When vector search returns candidates, an LLM reranks them for “related reading” quality:</p><pre>if use_rerank and len(results) &gt; settings.search_rerank_threshold:<br>    results = await self.__llm_rerank(query, results, top_k)</pre><p>The rerank prompt focuses on what matters:</p><pre>prompt = f&quot;&quot;&quot;You are a content relevance ranker.<br><br>Consider:<br>- Semantic relevance to the query<br>- For articles: would this be good &quot;related reading&quot;?<br>- For products: would this be a relevant product recommendation?<br><br>Search query: &quot;{query}&quot;<br><br>Candidates:<br>{formatted_candidates}<br><br>Return the {top_k} most relevant items as JSON:<br>[{{&quot;index&quot;: 0, &quot;reason&quot;: &quot;Directly addresses sustainable packaging&quot;}}, ...]<br>&quot;&quot;&quot;</pre><h4>Why Rerank Here But Not Knowledge Base?</h4><p>Two reasons:</p><ol><li><strong>SEO impact</strong>: Internal linking quality directly affects SEO rankings. It’s worth the cost.</li><li><strong>Judgment complexity</strong>: “Related reading” requires understanding themes and audience intent — something embeddings don’t capture well.</li></ol><p>The cost compared to the SEO value of good internal linking, it’s a bargain.</p><h3>Compliance: Hybrid Search + LLM Judgment + Cache</h3><h4>The Problem: Keywords AND Semantics</h4><p>Compliance is the trickiest source. Consider checking if the content violates, for example, health product regulations:</p><p><strong>The content says</strong>: “This supplement can help manage blood sugar levels effectively.”</p><p><strong>The rule says</strong>: “Products cannot claim to cure, treat, or prevent any disease.”</p><p>A pure vector search might match this to diabetes-related rules (good!) but miss the exact phrase “cure” or “treat” (bad!). Meanwhile, “manage blood sugar” is a softer claim — does it violate the rule or not?</p><p>We need:</p><ul><li><strong>Keyword precision</strong>: Catch exact forbidden phrases like “cure,” “treat,” “prevent”</li><li><strong>Semantic understanding</strong>: Understand that “manage blood sugar” might imply treatment</li><li><strong>Contextual judgment</strong>: Determine if the specific context violates the rule</li></ul><h4>The Solution: Three-Layer Approach</h4><p><strong>Layer 1: Hybrid Search</strong></p><p>Find potentially relevant rules using both semantic similarity and keyword matching:</p><pre># 70% semantic + 30% keyword<br>relevant_rules = await repository.hybrid_search(<br>    project_id=project_id,<br>    query_text=content,  # The content to check<br>    query_embedding=content_embedding,<br>    top_k=20,<br>    semantic_weight=0.7,<br>)</pre><p>Why hybrid? Pure vector search for “can cure diabetes” might match “diabetes treatment guidelines” but miss rules that specifically forbid the word “cure.”</p><p><strong>Layer 2: LLM Judgment</strong></p><p>Each potentially relevant rule is judged by an LLM:</p><pre>async def check_compliance(self, project_id: UUID, content: str):<br>    # 1. Find relevant rules<br>    relevant_rules = await self.__hybrid_search(content)<br><br>    # 2. Filter low-confidence matches<br>    relevant_rules = [r for r in relevant_rules if r.score &gt;= 0.4]<br><br>    # 3. LLM judges each rule (parallel, max 10 concurrent)<br>    violations = await self.__judge_violations_parallel(content, relevant_rules)<br><br>    return violations</pre><p>The LLM evaluates three rule types differently:</p><ul><li><strong>PROHIBIT</strong>: Is the forbidden element present?</li><li><strong>REQUIRE</strong>: Is the required element missing?</li><li><strong>CAUTION</strong>: Does the cautionary condition apply?</li></ul><p><strong>Layer 3: Redis Caching</strong></p><p>LLM judgments are expensive. Caching saves ~80% of costs:</p><pre># Cache key format<br>cache_key = f&quot;compliance:judgment:{rule_id}:{content_hash[:16]}&quot;<br><br># Cache value<br># No violation: &quot;null&quot;<br># Violation found: JSON with details<br>{<br>    &quot;rule_id&quot;: &quot;...&quot;,<br>    &quot;violation_text&quot;: &quot;can cure diabetes&quot;,<br>    &quot;severity&quot;: &quot;high&quot;,<br>    &quot;explanation&quot;: &quot;Claims curative properties, violating health regulations&quot;<br>}</pre><p><strong>TTL</strong>: 24 hours (rules don’t change often)</p><p><strong>Invalidation</strong>: When a rule is deleted, we clear all cached judgments for that rule:</p><pre>async def delete_document(self, document_id: UUID):<br>    rule_ids = await self.__repository.get_rule_ids_by_document(document_id)<br>    for rule_id in rule_ids:<br>        await self.__judgment_cache.invalidate_rule(rule_id)</pre><h4>Cost Analysis</h4><p>Let’s do the math:</p><pre>Without cache:<br>- 20 rules × $0.01/judgment = $0.20 per compliance check<br>- 100 articles/day × $0.20 = $20/day<br><br>With cache (80% hit rate):<br>- 4 uncached rules × $0.01 = $0.04 per check<br>- 100 articles/day × $0.04 = $4/day<br><br>Monthly savings: ~$480</pre><p>Caching isn’t optional for LLM-heavy operations. It’s essential.</p><h3>Anti-Hallucination Strategies</h3><p>Retrieval alone doesn’t prevent hallucinations. We implemented five layers of defense:</p><h4>Strategy 1: Grounding</h4><p><strong>Principle</strong>: The LLM can only use facts from retrieved sources.</p><p>The prompt explicitly states:</p><pre>You MUST only include information that comes from the provided sources.<br>Do NOT add any facts, statistics, or claims not found in the sources.<br>If the sources don&#39;t contain enough information, say so.</pre><p>This doesn’t eliminate hallucinations, but it gives us something to check against.</p><h4>Strategy 2: Index-Based References</h4><p><strong>Problem</strong>: LLMs invent URLs. They’ll confidently link to `/blog/ultimate-guide-to-skincare` even if that page doesn’t exist.</p><p><strong>Solution</strong>: The LLM uses indices, not URLs.</p><pre>&lt;!-- LLM writes this --&gt;<br>For more information, see our guide on sustainable packaging [1]<br>and eco-friendly materials [2].<br><br>&lt;!-- Tool resolves indices to actual URLs --&gt;<br>[1]: /blog/sustainable-packaging-guide<br>[2]: /products/eco-friendly-materials</pre><p>The tool only resolves indices that map to real, retrieved content. Invented references fail validation.</p><h4>Strategy 3: JSON Schema Validation</h4><p>Every tool defines a strict JSON schema. Invalid output is rejected:</p><pre>@property<br>def parameters(self) -&gt; dict:<br>    return {<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>            &quot;title&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;maxLength&quot;: 60},<br>            &quot;meta_description&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;maxLength&quot;: 160},<br>            &quot;focus_keyword&quot;: {&quot;type&quot;: &quot;string&quot;},<br>            &quot;internal_links&quot;: {<br>                &quot;type&quot;: &quot;array&quot;,<br>                &quot;items&quot;: {<br>                    &quot;type&quot;: &quot;object&quot;,<br>                    &quot;properties&quot;: {<br>                        &quot;index&quot;: {&quot;type&quot;: &quot;integer&quot;},  # Must be valid index<br>                        &quot;anchor_text&quot;: {&quot;type&quot;: &quot;string&quot;}<br>                    },<br>                    &quot;required&quot;: [&quot;index&quot;, &quot;anchor_text&quot;]<br>                }<br>            }<br>        },<br>        &quot;required&quot;: [&quot;title&quot;, &quot;meta_description&quot;, &quot;focus_keyword&quot;]<br>    }</pre><p>If the LLM returns invalid JSON or wrong types, the tool call fails and the agent retries.</p><h4>Strategy 4: Tool Validation Layer</h4><p>Beyond schema validation, tools validate business logic:</p><ul><li><strong>`validate_content_score`</strong>: Checks the score of generated content with specified business logic, keyword density, meta tag lengths</li><li><strong>`check_compliance`</strong>: Verifies no regulation violations</li><li><strong>`link_validator`</strong>: Confirms all internal links resolve to real pages</li><li><strong>`structure_validator`</strong>: Ensures required sections (intro, conclusion) exist</li></ul><p>Each validation can reject the content and request fixes.</p><h4>Strategy 5: Prompt Engineering</h4><p>We structure prompts with explicit rules:</p><pre>[system]<br>content = &quot;&quot;&quot;<br>## CRITICAL RULES (MUST FOLLOW)<br><br>1. Internal links MUST use index references [1], [2], etc.<br>   - CORRECT: &quot;See our guide [1] for details&quot;<br>   - WRONG: &quot;See our guide (https://example.com/guide)&quot;<br><br>2. Statistics MUST come from provided sources<br>   - CORRECT: &quot;According to the provided research, 73% of users...&quot;<br>   - WRONG: &quot;Studies show that 73% of users...&quot; (no source cited)<br><br>3. Product features MUST match the product database<br>   - CORRECT: Features listed in the product data<br>   - WRONG: Invented features not in the data<br>&quot;&quot;&quot;</pre><p>We also include quantified thresholds:</p><pre>[variables]<br>min_seo_score = &quot;80&quot;<br>max_keyword_density = &quot;2.5&quot;<br>prefer_existing_category_threshold = &quot;0.8&quot;  # 80% similarity</pre><h3>Lessons Learned: Challenges and Pitfalls</h3><h4>Pitfall 1: Pure Vector Search Misses Keywords</h4><p>I spent some time debugging why compliance checks missed obvious violations. The content said “guaranteed to cure acne,” and the rule said “cannot claim to cure any condition.”</p><p>Vector similarity? 0.72 — decent, but below our threshold.</p><p>The problem: “cure” as a keyword wasn’t weighted enough. The semantic embedding focused on “acne” and “skin conditions” rather than the forbidden verb.</p><p><strong>Solution</strong>: Hybrid search with 30% keyword weight. Now “cure” matches “cure” directly.</p><h4>Pitfall 2: LLM Costs Add Up Fast</h4><p>Early prototype: every retrieval triggered an LLM rerank. Every compliance rule got an LLM judgment.</p><p><strong>Solutions we implemented</strong>:</p><ul><li>Rerank threshold: Only rerank when candidates &gt; 10</li><li>Model selection by task: Cheap and fast model for simple tasks, Pro model for complex ones</li><li>Aggressive caching: 24-hour TTL for compliance judgments</li><li>Batch operations: Context prefixes generated 15 at a time</li></ul><h4>Pitfall 3: Context Loss in Chunking</h4><p>First version: 500-token chunks, no overlap, no context.</p><p>Result: Chunks like “The above policy applies to all categories” — useless without knowing what “above policy” refers to.</p><p><strong>Solution</strong>: Contextual Retrieval prefixes + 200-char overlap + smart boundary detection.</p><h4>Pitfall 4: One Strategy Doesn’t Fit All</h4><p>I tried to build a “universal retrieval pipeline” that handled everything. It was mediocre at everything.</p><ul><li>Knowledge base got irrelevant chunks</li><li>Platform content got keyword-matched but thematically unrelated articles</li><li>Compliance missed violations</li></ul><p><strong>Solution</strong>: Accept the complexity. Three sources, three pipelines, shared infrastructure (embeddings, vector DB), specialized logic.</p><h3>Improvements and Future Work</h3><p>This system works, but there’s room to improve:</p><h4>Cross-Encoder Reranking for Knowledge Base</h4><p>Currently, we skip LLM reranking for knowledge base results. A cross-encoder model (lighter than an LLM) could improve precision without the cost.</p><h4>Multi-Language Support</h4><p>Our hybrid search uses English-optimized `ts_rank()`. For multilingual content, we’d need language-specific tokenizers or rely more heavily on semantic search.</p><h4>Incremental Embedding Updates</h4><p>Currently, updating a document means re-embedding all chunks. Incremental updates (only changed sections) would reduce costs and latency.</p><h4>Confidence Scores for Anti-Hallucination</h4><p>Instead of binary pass/fail, add confidence scores to each fact. Low-confidence statements could be flagged for human review.</p><h4>Adaptive Retrieval</h4><p>Let the system learn which strategy works best for different query types. A meta-model could route queries to the optimal pipeline.</p><h3>Conclusion</h3><p>Building a production RAG system taught me that the “standard” approach — chunk everything, embed the same way, retrieve by similarity — is a starting point, not a solution.</p><p><strong>Key takeaways</strong>:</p><ol><li><strong>Different knowledge sources deserve different strategies.</strong> A brand guideline PDF, a product catalog, and compliance rules have fundamentally different retrieval needs.</li><li><strong>Hybrid search often beats pure vector.</strong> Semantic similarity misses keyword precision. Keyword matching misses semantic relevance. Use both.</li><li><strong>Caching is essential for LLM-heavy operations.</strong> Without it, costs spiral. With it, you get money savings.</li><li><strong>Anti-hallucination requires defense in depth.</strong> No single technique is enough. Combine grounding, validation, schema enforcement, and careful prompting.</li><li><strong>Complexity is sometimes necessary.</strong> A universal pipeline sounds elegant but performs poorly. Accept the complexity when it’s justified.</li></ol><p>If you’re building a RAG system, ask yourself: Am I treating all knowledge sources the same? If yes, you might be leaving quality (and cost savings) on the table.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=355f4fe7897e" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/beyond-one-size-fits-all-rag-why-different-knowledge-sources-need-different-retrieval-strategies-355f4fe7897e">Beyond One-Size-Fits-All RAG: Why Different Knowledge Sources Need Different Retrieval Strategies</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI for Bass Music Creation: Let There Be More Bands]]></title>
            <link>https://medium.com/@jo.ym/ai-for-bass-music-creation-let-there-be-more-bands-311065d773f9?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/311065d773f9</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[thinking]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Sat, 28 Dec 2024 16:38:49 GMT</pubDate>
            <atom:updated>2026-01-06T06:05:10.596Z</atom:updated>
            <content:encoded><![CDATA[<h4>How AI Can Solve the Bassist Shortage Problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HSfRnJDKPnDYlyC9" /><figcaption>Photo by <a href="https://unsplash.com/@seththr?utm_source=medium&amp;utm_medium=referral">Seth Reese</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article is part of a series exploring “AI for Bass Music Creation”. In the <a href="https://medium.com/@jo.ym/list/ai-for-bass-music-creation-41e8eeb6b74f">previous three pieces</a>, we focused on coding experiments and technical implementations. For this final piece, we’re stepping back to share some ideas and open up a discussion.</p><h3>Introduction</h3><p>Bass players are often the backbone of a band, holding down the rhythm and adding depth to the music. But let’s face it — good bassists are hard to find. This shortage can make it tough for new bands to get started (at least that’s how it is where I live). So, can AI step in and help? In this article, we’ll talk about how AI tools could make it easier to learn bass, write cool basslines, and spark new musical ideas.</p><h3>AI as a Tool for Learning Bass</h3><p><strong>Breaking Down Basslines</strong> Learning bass can be tricky, especially when it comes to understanding how it fits into a song. AI could make this easier by analyzing famous basslines and breaking them down step-by-step. Imagine tools that:</p><ul><li><strong>Show Patterns Visually:</strong> Highlight rhythms and notes to make them easier to follow.</li><li><strong>Find Repeating Motifs:</strong> Spot patterns in songs to help you understand what makes a groove work.</li><li><strong>Give Custom Practice Exercises:</strong> Adapt to your skill level and help you practice what you need most.</li></ul><p><strong>Interactive AI Tutors</strong> Picture this — you type in, “Show me a funky groove in E minor,” and AI instantly plays a sample. Not only that, but it also explains how the groove works and gives you tips to improve your playing. These AI tutors could even give feedback on your timing and tone.</p><h3>AI Tools for Creating Music</h3><p><strong>AI-Generated Bass Grooves</strong> AI isn’t just for learning — it can help you create, too. Here’s how:</p><ul><li><strong>Quick Groove Ideas:</strong> AI can generate basslines based on your mood or style preferences, so you’re never stuck for ideas.</li><li><strong>Custom Variations:</strong> Want something with more swing or a heavier feel? AI can tweak grooves until they match your vibe.</li><li><strong>Seamless DAW Integration:</strong> AI tools can work inside your music software to suggest basslines as you write songs.</li></ul><p><strong>Making It Easier to Start a Band</strong> AI tools could also make it less intimidating for beginners to start making music. If you don’t have a bassist, no problem — AI can fill in until you find one (or maybe you’ll decide to stick with AI altogether, but as a bass player, I still hope you find a bass player ^^). This lowers the barrier to creating songs and encourages more people to jump in.</p><h3>The Future: AI Platforms for Learning and Creating</h3><p><strong>AI-Powered Learning Hubs</strong> Imagine an all-in-one platform where you can:</p><ul><li>Learn bass basics with AI tutors.</li><li>Browse a huge library of basslines for ideas.</li><li>Generate your grooves and get feedback as you play.</li><li>Jam with other users online, with AI acting as the session bassist.</li></ul><p><strong>From Learning to Creating — All in One Place</strong> AI could connect the dots between learning and writing music. For example, you could:</p><ol><li>Study a classic groove.</li><li>Use AI to remix it or create variations.</li><li>Turn your ideas into full songs. This would make learning and creating music feel like one continuous process.</li></ol><h3>Wrapping Up</h3><p>Bassists might be in short supply, but AI could help change that. By making it easier to learn and create basslines, AI tools could inspire more people to pick up the bass — or at least experiment with bass sounds in their music. Looking ahead, AI has the potential to make music education and production more accessible, helping more bands get started and more creators bring their ideas to life.</p><p>With AI, we can keep that heartbeat strong and inspire a whole new generation of musicians. BTW, a hot topic many people talk about: Is AI about enablement or replacement? <strong>I choose enablement.</strong></p><p>That being said, I still hope everyone keeps working on improving their musical sense, writing songs, and finding real-life music partners. I’m working hard on it too. QQ…</p><p>To wrap up with some casual talk — do you know why I like the bass? It’s because I love its role. It’s easy to overlook but absolutely essential. But there’s actually another reason. Back in the day, there was an anime called <em>K-On</em>, and Mio was my first 2D crush, hahaha. Nowadays, most young people probably only know Bocchi the Rock? But I also like Bocchi chan XD</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=311065d773f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI for Bass Music Creation: Generating Music]]></title>
            <link>https://medium.com/@jo.ym/ai-for-bass-music-creation-generating-music-26ca6bdbe6ed?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/26ca6bdbe6ed</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[notes]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[music]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 16:02:02 GMT</pubDate>
            <atom:updated>2024-12-28T16:39:17.908Z</atom:updated>
            <content:encoded><![CDATA[<h4>Meet Latent Spaces and Diffusion Models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jEjNhIogoXa3apst" /><figcaption>Photo by <a href="https://unsplash.com/@marius?utm_source=medium&amp;utm_medium=referral">Marius Masalar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Continuing from the <a href="https://medium.com/@jo.ym/ai-for-bass-music-creation-extracting-music-styles-and-emotional-labels-4c182888eabe">previous article</a>, we are going to generate music. (Although we didn’t use the data from the previous article XD)</p><p><em>Spoiler Alert:</em> The generated music in this experiment turned out… quite disappointing 😅. But let’s not dwell on the outcome — this is about the process, the learning, and perhaps some constructive feedback from you! Feel free to share your thoughts or tips in the comments.</p><h3>Setting the Stage: Tools and Frameworks</h3><p>Before we dive into the code and concepts, let me clarify the tools we used:</p><ul><li><strong>PyTorch</strong> — A flexible machine-learning library for building and training neural networks.</li><li><strong>Librosa</strong> — A Python library for audio processing.</li><li><strong>Diffusers</strong> — A framework for building diffusion-based generative models.</li><li><strong>Scikit-learn (TSNE)</strong> — For visualizing high-dimensional data.</li></ul><p>If you’re unfamiliar with installing PyTorch or enabling GPU acceleration, I recommend checking the <a href="https://pytorch.org/">official PyTorch documentation</a> or following tutorials tailored to your system.</p><h3>The Models and Their Roles</h3><h4>VAE (Variational Autoencoder)</h4><p><strong>What is it?</strong> Imagine you’re an artist who sketches rough outlines (encoding) and then refines those sketches into detailed paintings (decoding). A <strong>VAE</strong> is like that artist — it compresses input data into a smaller, latent representation (sketch) and learns to reconstruct it back (painting).</p><p><strong>Why use it?</strong> We use VAEs to map complex data, like audio features, into a compact latent space. Once compressed, we can tweak the latent variables to generate variations or explore new combinations.</p><p><strong>Quick Note:</strong> Libraries like <a href="https://github.com/AntixK/PyTorch-VAE">PyTorch-VAE</a> offer ready-to-use implementations, but we built our own for flexibility and experimentation.</p><h4>Diffusion Models and UNet2DModel</h4><p><strong>What are they?</strong> Diffusion models work like digital sculptors — they start with noise and gradually refine it until it resembles the target data. <strong>UNet2DModel</strong>, specifically, handles image-like data (e.g., spectrograms).</p><p><strong>Why use it?</strong> VAEs alone may produce blurry outputs. Diffusion models, on the other hand, add a layer of refinement to create sharper, more realistic results. Think of it as polishing the rough sketches provided by the VAE.</p><h3>Turning Features Back into Sound</h3><p>Generated data from our models aren’t ready-to-play audio files. Instead, they exist as <strong>Mel spectrograms</strong> — visual representations of frequency patterns over time. To hear the output, we need to reverse this process.</p><h4>Griffin-Lim Algorithm (Simple, Quick)</h4><p><strong>What is it?</strong> Imagine trying to rebuild a melody from sheet music without knowing the exact timing of notes. Griffin-Lim reconstructs the missing timing information by iteratively guessing and improving it.</p><p><strong>Why use it?</strong> It’s straightforward and doesn’t require pre-trained models, making it ideal for quick experiments like ours.</p><h4>Neural Network Vocoders (High-Quality, Advanced)</h4><p><strong>What are they?</strong> These are AI tools designed specifically to generate audio waveforms from spectrograms. Examples include:</p><ul><li><strong>MelGAN</strong> — Real-time audio generation.</li><li><strong>WaveGlow</strong> — High-quality speech synthesis.</li><li><strong>HiFi-GAN</strong> — Optimized for music and speech.</li></ul><p><strong>Why not use them here?</strong> While these tools produce better quality, they require additional setup and training. I’ll definitely revisit this for future improvements! 😃</p><h3>Visualizing the Latent Space with t-SNE</h3><p><strong>What is it?</strong> T-SNE reduces a high-dimensional maze into a 2D map, helping us see clusters and patterns.</p><p><strong>Why use it?</strong> We use t-SNE to visualize how our latent variables group together. Similar sounds should cluster closely, indicating meaningful patterns in the data.</p><p><strong>Our Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MFdd0DYzfSwz-frS4-FDMw.png" /></figure><p><strong>Interpretation:</strong> Sadly, the plot shows no clear clusters — likely a sign that our latent space lacks structure. This suggests we need to revisit preprocessing or improve model complexity.</p><h3>Code</h3><p><a href="https://github.com/jo-ym/BassAI-Lab/tree/main/stage3_train_and_gen">https://github.com/jo-ym/BassAI-Lab/tree/main/stage3_train_and_gen</a></p><h3>Next Steps</h3><p>While this project yielded disappointing results (so far), it laid the groundwork for further exploration. Here’s what’s next on my to-do list:</p><ol><li>Improve data preprocessing (cleaner datasets often yield better models).</li><li>Test neural vocoders like HiFi-GAN for higher-quality audio.</li><li>Optimize hyperparameters and experiment with larger datasets.</li><li>For example, use a filter or another tool to isolate the pure bass sound first.</li></ol><p>I hope this post sparks ideas or constructive feedback. If you’ve tried similar experiments, I’d love to hear how it went for you!</p><p>Don’t go just yet! There’s <a href="https://medium.com/@jo.ym/ai-for-bass-music-creation-let-there-be-more-bands-311065d773f9">one more post</a> in this series where I’ll share some ideas and thoughts — no implementation this time.</p><p><strong>Hi, I’m Jo!</strong><br>I’m currently working hard to land a role as a data engineer or data scientist at some media-tech / start-up company in the EU (my goal at the time of writing this article). Let’s connect on <a href="https://www.linkedin.com/in/jo-yun-qiao-chen?utm_source=share&amp;utm_campaign=share_via&amp;utm_content=profile&amp;utm_medium=android_app">LinkedIn </a>and share ideas.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26ca6bdbe6ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI for Bass Music Creation: Extracting Music Styles and Emotional Labels]]></title>
            <link>https://medium.com/@jo.ym/ai-for-bass-music-creation-extracting-music-styles-and-emotional-labels-4c182888eabe?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c182888eabe</guid>
            <category><![CDATA[emotions]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[audio]]></category>
            <category><![CDATA[librosa]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 15:59:02 GMT</pubDate>
            <atom:updated>2026-01-06T06:05:19.431Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Quick Analysis with LibROSA</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H08Ci0uWpn85urtx" /><figcaption>Photo by <a href="https://unsplash.com/@jasonrosewell?utm_source=medium&amp;utm_medium=referral">Jason Rosewell</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article, I’ll continue where we left off in the <a href="https://medium.com/@jo.ym/ai-for-bass-music-creation-preparing-audio-and-audio-features-84c3201733b1">previous post</a>. This time, we’ll explore how to extract and visualize emotional and stylistic labels from music data.</p><h3>Why This Approach?</h3><p>After surveying the available tools and methods, I found that many frameworks, like the well-known <a href="https://essentia.upf.edu/"><strong>Essentia</strong></a>, can handle these tasks effectively. However, for my current goal — quickly testing and analyzing music samples that largely belong to a single genre — I decided to take a more straightforward approach. BTW, it’s very troublesome to install Essentia on Windows @@</p><p>Rather than training a custom model or relying on pre-trained classifiers, I referenced values from research papers and implemented calculations using <strong>LibROSA</strong> to extract and analyze musical features.</p><h3>Code Implementation</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e63185b79890a04b4062766e358ebb44/href">https://medium.com/media/e63185b79890a04b4062766e358ebb44/href</a></iframe><h3>Visualization Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ookYm-uc1PmW3xaFpbGK7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*chmWKa-DMsSTqTeVaRTOKQ.png" /></figure><h3>Theory Behind the Numbers</h3><p>The numeric thresholds and calculations in this project are based on the <strong>Valence-Arousal Emotional Model</strong> — a well-established psychological framework for describing emotions along two dimensions:</p><h4>1. Valence (Pleasantness):</h4><p>Represents whether an emotion is positive or negative.</p><ul><li><strong>High Valence:</strong> Joyful, happy, and satisfying.</li><li><strong>Low Valence:</strong> Sad, melancholic, and negative.</li></ul><h4>2. Arousal (Energy):</h4><p>Measures the intensity or energy level of an emotion.</p><ul><li><strong>High Arousal:</strong> Excited, energetic, and active.</li><li><strong>Low Arousal:</strong> Calm, relaxed, and peaceful.</li></ul><h3>Calculating Valence and Arousal</h3><h4>1. Valence Calculation</h4><ul><li><strong>Feature Used:</strong> <strong>Chroma</strong> — captures harmonic and pitch stability.</li><li><strong>Scaling:</strong> Chroma values typically range from 0 to 1, so I scaled them by 10 to map the range to 0–10, aligning with psychological studies that often use a 0–10 emotional scale.</li></ul><p><strong>Thresholds:</strong></p><ul><li><strong>&gt; 6:</strong> High valence, associated with positivity and relaxation.</li><li><strong>&lt; 4:</strong> Low valence, indicating darker or more negative emotions.</li><li><strong>4–6:</strong> Neutral, balanced emotional state.</li></ul><h4>2. Arousal Calculation</h4><ul><li><strong>Feature Used:</strong> <strong>Tempo (BPM)</strong> — closely tied to perceived energy levels in music.</li><li><strong>Scaling:</strong> Tempo values are normalized by dividing by 200 to fit within a 0–1 range.</li></ul><p><strong>Thresholds:</strong></p><ul><li><strong>&gt; 0.6:</strong> High energy, suitable for upbeat and dance-like tracks.</li><li><strong>&lt; 0.4:</strong> Low energy, ideal for meditation or ambient moods.</li><li><strong>0.4–0.6:</strong> Moderate energy, a transition state.</li></ul><h3>Next Step</h3><p>Now that we’ve covered feature extraction and visualization, the next step is training and generating music.</p><p>We’ll move into latent space modeling using Variational Autoencoders (VAEs) and Diffusion Models to compress and generate bass music segments — allowing us to experiment with feature manipulation and style control.</p><p>Stay tuned for the next post, where we dive into generative AI for music creation!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c182888eabe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI for Bass Music Creation: Preparing Audio and Audio Features]]></title>
            <link>https://medium.com/@jo.ym/ai-for-bass-music-creation-preparing-audio-and-audio-features-84c3201733b1?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/84c3201733b1</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[ytd-video-downloader]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[librosa]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 15:57:02 GMT</pubDate>
            <atom:updated>2026-01-06T06:05:29.492Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I Collected and Processed Audio Features for AI Music Generation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IUIx8AhDQevcos2L" /><figcaption>Photo by <a href="https://unsplash.com/@karishea?utm_source=medium&amp;utm_medium=referral">Kari Shea</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As someone who loves the sound of the bass, I was recently browsing YouTube when I stumbled upon a channel dedicated to uploading bass performances. The best part? The genre of music was exactly what I enjoyed. Being an engineer by trade, my curiosity got the best of me. Then, I came across a phrase that sparked an idea (at least that’s how it is where I live).:</p><blockquote><em>If one more person learns to play the guitar, the world gets another guitarist; if one more person learns to play the bass, the world gets another band.</em></blockquote><p>Now, this isn’t to say guitar players aren’t important :)— I’ve played guitar myself too — but this quote perfectly captures the rarity and unique role of the bass. It made me think: <em>Why not try to use AI to fill in the gap?</em></p><p>So, I’ve decided to break down my exploration into a series of four articles:</p><ol><li>Current article: In this first article, I will discuss</li></ol><ul><li>How I downloaded music as training data, but for copyright reasons, I won’t be revealing exactly which tracks I downloaded.</li><li>To extract audio features, I used the famous audio feature extraction library, <a href="https://github.com/librosa/librosa">Librosa</a>. I’ll explain the various features and how I’ve come to understand them — again,<strong> feel free to correct me if I misunderstood something!</strong></li></ul><p>2. <a href="https://medium.com/@jo.ym/ai-for-bass-music-creation-extracting-music-styles-and-emotional-labels-4c182888eabe">Extracting Music Styles and Emotional Labels</a></p><p>3. <a href="https://medium.com/@jo.ym/ai-for-bass-music-creation-generating-music-26ca6bdbe6ed">Generating Music using Latent Spaces and Diffusion Models</a></p><p>4. <a href="https://medium.com/@jo.ym/ai-for-bass-music-creation-let-there-be-more-bands-311065d773f9">Some Thinking</a></p><p><strong>But here’s the punchline: <em>I failed</em>.</strong></p><p>So actually this series of articles is more like my personal experimental notes.</p><p>I’m aware that there are already many well-developed tools available to help me accomplish this task more quickly and successfully. However, in this case, I’d like to roll up my sleeves and tackle it myself first, as a way to build a solid foundation in this area.</p><p>You can also skip straight to my final reflections to see if you want to incorporate them into your process.</p><p>The music I generated was awful. I realized I might need to do more research and gather additional insights, so <strong>I’m opening the floor for suggestions or corrections</strong> — if you see where I’ve gone wrong, please do share! I would appreciate it.</p><h3>Downloading Music Using yt-dlp</h3><p>As part of my experiment, the first step was to download music from a YouTube channel that focuses on bass performances in the genre I like. To respect copyright, I won’t be revealing the specific channels or tracks I used. Instead, I’ll focus on the technical details of how I approached the task.</p><p>To collect the music, I used the yt-dlp tool, which is great for downloading YouTube videos or audio files in bulk. Below is the code I used to download the content:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a837a13f88dc52cbb79271aead6d069d/href">https://medium.com/media/a837a13f88dc52cbb79271aead6d069d/href</a></iframe><h3>Extracting Features with Librosa</h3><p>Once I had the music files downloaded, I needed to extract audio features to train my AI model. For this, I turned to <a href="https://github.com/librosa/librosa">Librosa</a>, a well-known library for audio analysis.</p><p>Now let’s get to know these signals:</p><h4>MFCC (Mel Frequency Cepstral Coefficients)</h4><p>Imagine you’re listening to a song, and you hear different instruments like guitar, piano, and drums. MFCCs try to capture the “timbre” of these sounds. It’s like when you hear different instruments in a song and distinguish their unique sounds. MFCCs help us understand the texture and style of the music.</p><p>It can be used for music classification, music recommendation, or identifying musical styles. It helps differentiate between different types of music (e.g., rock, jazz, electronic) or extract the timbres of specific instruments.</p><h4>Chroma Features</h4><p>Imagine you’re listening to a song, and you notice the melody changing from high to low. These changes often appear in the Chroma features. It helps us understand the harmonic structure of a song.</p><p>It can be used for chord progression analysis, which helps understand a song’s structure or arrangement style. It’s also useful for automatic music generation and chord recognition.</p><h4>Spectral Centroid</h4><p>Imagine you’re listening to music, and the sound feels either bright or dull. The spectral centroid captures this feeling. For instance, high-pitched sounds, like guitar or piano, often feel bright, while low-pitched sounds are perceived as darker. This feature helps capture that “brightness” sensation.</p><p>It can be used for music emotion analysis, and style classification, and helps distinguish between bright and dull music, which is useful in music recommendation systems and emotion analysis.</p><h4>Spectral Contrast</h4><p>Imagine you’re listening to a sound with both high and low pitches, and the energy difference between them is significant. Spectral contrast helps us understand this “high-low” contrast, e.g., a guitar might have sharp high frequencies, while a bass has a deep, thick low-end sound.</p><p>It can be used to analyze a music’s dynamic range and understand its emotional tension. It can be used in music retrieval, classification, and automated music style recognition.</p><h4>Tempo (Beats Per Minute)</h4><p>When you’re listening to a song, you naturally tap your foot or nod your head to the beat. The tempo is what defines the “pulse” of the music, determining how fast or slow the music feels.</p><p>It can be used in music classification, emotion analysis, and dynamic music generation. Music with strong tempos can be categorized as dance music, rock, etc., which is important for music recommendation systems.</p><h4>Zero-Crossing Rate</h4><p>When there are fast percussion sounds (e.g., drum beats or bass hits), the zero-crossing rate increases. It’s a way to quantify the “impact” or “punch” of the music.</p><p>It can be used to analyze the percussive nature or rhythm patterns in music. It’s useful for analyzing rhythm-heavy music like dance or rock.</p><h4>RMS (Root Mean Square Energy)</h4><p>RMS is like measuring the “power” or “volume” of the music. When you hear a loud bass hit or a loud drum sound, the RMS value captures these intensity changes.</p><p>RMS can be used to analyze the dynamic range of music, helping determine whether a piece of music has intense dynamics or stable bass, which is useful for emotion analysis, music mixing, and volume normalization.</p><h4>Mel Spectrogram</h4><p>A Mel spectrogram is like a “map” of the audio, showing the change of frequencies over time. High and low frequencies are visible on this map. It’s like drawing the “mountains” and “valleys” of a song’s frequency range.</p><p>Mel spectrograms can be used for music analysis, emotion analysis, and even music generation. It’s a commonly used feature in audio processing, helping us understand the changes in time and frequency within music.</p><h4>Spectral Bandwidth</h4><p>Think of spectral bandwidth as a measure of the sharpness or smoothness of the sound. For example, high-pitched sounds like cymbals or hissing sounds have a wider bandwidth, while bass sounds are generally smoother and have a narrower bandwidth.</p><p>Spectral bandwidth can be used to analyze the texture of the sound, whether it’s bright or dull. It’s useful for music quality analysis, particularly in music mixing, audio effects processing, and understanding the dynamics of different genres like bass music and electronic music.</p><h4>Pitch Tracking</h4><p>Pitch tracking helps us identify the “notes” in music. For example, in bass music, the notes played by the bass guitar are captured here, revealing the melody and variations of the bass line throughout the track.</p><p>Pitch tracking can be used in music creation, melody generation, chord recognition, and even automatic music arrangement. It is particularly useful for automatic analysis and generation of music.</p><h4>Harmonic-to-Noise Ratio (HNR)</h4><p>If you hear a clear, pure sound, it means the harmonic components (the musical tones) are strong, and noise (unwanted sounds) is minimal. HNR describes how pure or clear the sound is, such as in vocals or music.</p><p>HNR can be used for quality analysis of music, speech clarity evaluation, and emotion analysis in music. It is important for assessing sound quality and analyzing the clarity of a track.</p><h4>Onset Detection</h4><p>Onset detection helps pinpoint when a musical note or beat begins. For example, when you hear a bass hit or a drum kick, onset detection will tell us exactly when that sound starts, giving us insights into the song’s rhythm and beats.</p><p>Onset detection can be used in rhythm analysis, beat extraction, and structural analysis of dance music. It’s also useful for music generation algorithms that need to detect key rhythm points.</p><h4>Harmonic/Percussive Separation</h4><p>Some music contains melodies (like guitar or piano) and rhythms (like drums or percussion). This feature helps separate them. For example, when you listen to music, you might want to isolate the drums from the melody, and this feature helps with that.</p><p>It can be used in music mixing, music production, and audio separation. For example, separating percussion and melody parts is crucial for post-production audio effects or automatic arrangement.</p><h4>Spectral Roll-off</h4><p>Think of spectral roll-off as measuring the “sharpness” or “dullness” of sound. For instance, bright high-pitched sounds gradually diminish, and spectral roll-off helps us understand the intensity of those high-frequency components.</p><p>It’s useful for analyzing the brightness or clarity of music, helping with style classification, emotional analysis, and sound quality analysis.</p><h4>Magnitude</h4><p>Think of magnitude as a measure of the “loudness” or “intensity” of the music. For example, in a piece of bass music, where the bass provides strong, deep sounds, the magnitude mean will reflect the power of those low-frequency elements, helping to characterize the dynamic range of the track.</p><p>It can be used to analyze the energy distribution in the music, important for emotional analysis, dynamic range measurement, and music mixing. It provides insight into the overall loudness and energy fluctuations of a track.</p><h4>Formant Analysis</h4><p>Imagine formant analysis as looking at the “color” of a sound. Every instrument, including the bass, has its characteristic sound color, and formants help us identify these unique frequency features. For instance, the bass guitar has certain resonant frequencies that give it its characteristic low sound, which formant analysis can highlight.</p><p>Formant analysis can help us with timbre analysis, identifying specific instruments, and understanding the emotional texture of the music. It can also be useful for music generation, sound design, and audio processing, especially for applications like EQ tuning.</p><p>In the <a href="https://github.com/MDFahimAnjum/TurboLPC">TurboLPC</a> package, the arburg_vector function is used to compute LPC coefficients, which provide insights into the frequency spectrum&#39;s resonant features. By finding the roots of these coefficients (specifically the real parts), we can extract the formant frequencies that define the characteristic sounds of the instrument, such as bass.</p><p>Below is the code I used to extract and save the audio features as JSON files:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/591302a6830788d82742e4d2670cb120/href">https://medium.com/media/591302a6830788d82742e4d2670cb120/href</a></iframe><p>Once the features were extracted, I saved them in a JSON file for further analysis or model training.</p><p>In addition, I also quickly made some visual attempts, such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*zrKivPZcJD3llt1mLT5OgA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*lSbnlAcalX8EBI5C5-uTWw.png" /></figure><p>If you are interested, I put the code <a href="https://github.com/jo-ym/BassAI-Lab/blob/main/stage1_feature_extraction_and_visualization/visualization.ipynb">here</a>, but I only did a quick try and haven’t studied this seriously yet. (There is always too much to research)</p><h3>Next Step</h3><p>Now that I’ve collected the data, the next steps will involve training the AI model and analyzing the results. Stay tuned for the next article, where I’ll dive deeper into the training process and discuss the challenges I encountered.</p><p>Feel free to share your thoughts or suggestions. I’m eager to hear feedback on this project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84c3201733b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the Coinbase AI Agent]]></title>
            <link>https://medium.com/@jo.ym/exploring-the-coinbase-ai-agent-99a1d7605e49?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/99a1d7605e49</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[coinbase]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Wed, 18 Dec 2024 11:18:40 GMT</pubDate>
            <atom:updated>2026-01-06T06:05:39.898Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Survey of Features, Workflows, and Potential Extensions for Wen3 Automation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gSbSkylBfI51XTNM" /><figcaption>Photo by <a href="https://unsplash.com/@onefifith?utm_source=medium&amp;utm_medium=referral">Shubham Dhage</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Recently, I explored Coinbase’s <strong>CDP Agentkit</strong> (GitHub: <a href="https://github.com/coinbase/cdp-agentkit">cdp-agentkit</a>), a framework that simplifies integrating AI agents with on-chain blockchain operations using Python. Beyond its core functionality, I find it particularly interesting as it introduces a practical <strong>“agent-based approach”</strong> to managing tasks.</p><p>Coinbase also provides a <a href="https://docs.cdp.coinbase.com/learn/docs/based-ai-agents">Replit template</a> for users to quickly set up and experiment with the tool. While their official documentation provides step-by-step guidance, this post offers a <strong>condensed summary</strong> to help you quickly grasp the <strong>key features</strong>, the <strong>framework architecture</strong> from GitHub, and the underlying structure of their <strong>template implementation</strong>.</p><p>If you’re looking for a quick way to understand how to build and utilize AI agents for blockchain operations, this post is for you.</p><h3><strong>Architecture, Class Diagram, and Flowchart</strong></h3><h4>Class Diagram of Replit Template</h4><figure><img alt="Class Diagram of the Replit Template" src="https://cdn-images-1.medium.com/max/1024/1*p8fQLZY-S-aCIcirnOhq4g.png" /></figure><p><strong>Agent (Parent Class)</strong>:</p><ul><li>Base class for defining agents. Contains attributes like name, instructions, and a list of callable functions.</li></ul><p><strong>BasedAgent (Agent Implementation)</strong>:</p><ul><li>Implements specific blockchain-related functions like create_token, transfer_asset, deploy_nft, and register_basename.</li><li>Extends Agent and integrates with blockchain operations.</li></ul><p><strong>Swarm (Runtime Environment)</strong>:</p><ul><li>Provides a runtime for executing agent tasks and facilitating communication.</li></ul><p><strong>Wallet</strong>:</p><ul><li>Handles blockchain wallet operations like token deployment, NFT management, and asset transfers.</li><li>Returns instances of Contract and Transaction for further actions.</li></ul><p><strong>TwitterBot</strong>:</p><ul><li>Adds social interaction capabilities, allowing the agent to interact with Twitter via tweepy.</li></ul><p><strong>OpenAI</strong>:</p><ul><li>Provides integration with OpenAI APIs for generating text or images.</li></ul><p><strong>Contract and </strong><strong>Transaction</strong>:</p><ul><li>Represent deployed contracts and transaction states. These are returned by Wallet operations.</li></ul><h4>Flowchart of Chat Mode of Template</h4><figure><img alt="Flowchart of Chat Mode of Template" src="https://cdn-images-1.medium.com/max/1024/1*E__mnjK-28vSSH81OQvJaQ.png" /></figure><p>The global prompt of Based Agent is in the instructions of the initialization of Agent instructions:</p><pre>based_agent = Agent(<br>    name=&quot;Based Agent&quot;,<br>    instructions=(<br>        &quot;You are a helpful agent that can interact onchain on the Base Layer 2 using the Coinbase Developer Platform SDK. &quot;<br>        &quot;You can create tokens, transfer assets, generate art, deploy NFTs, mint NFTs, register basenames, and swap assets (on mainnet only). &quot;<br>        &quot;If you ever need to know your address, it is {agent_wallet.default_address.address_id}. &quot;<br>        &quot;If someone asks you to do something you can&#39;t do, you can say so, and encourage them to implement it themselves.&quot;<br>    ),<br>    functions=[<br>        create_token,<br>        transfer_asset,<br>        get_balance,<br>        ...<br>    ]<br>)</pre><h4>Flowchart of Auto Mode of Template</h4><figure><img alt="Flowchart of Auto Mode of Template" src="https://cdn-images-1.medium.com/max/1024/1*SiinSBWcWcq5szHdBlgQZw.png" /></figure><p>The prompt is in run_autonomous_loop()</p><pre>thought = (<br>    &quot;Be creative and do something interesting on the Base blockchain. &quot;<br>    &quot;Don&#39;t take any more input from me. Choose an action and execute it now. &quot;<br>    &quot;Choose those that highlight your identity and abilities best.&quot;<br>)</pre><h4>Flowchart of Two-agent Mode of Template</h4><figure><img alt="Flowchart of Two-agent Mode of Template" src="https://cdn-images-1.medium.com/max/1024/1*vctWdYHkd2nfb2MoOwhHmw.png" /></figure><p>The prompt is in run_openai_conversation_loop():</p><pre>openai_messages = [{<br>    &quot;role&quot;: &quot;system&quot;,<br>    &quot;content&quot;: (<br>        &quot;You are a user guiding a blockchain agent through various tasks on the Base blockchain. &quot;<br>        &quot;Engage in a conversation, suggesting actions and responding to the agent&#39;s outputs. &quot;<br>        &quot;Be creative and explore different blockchain capabilities. Options include creating tokens, &quot;<br>        &quot;transferring assets, minting NFTs, and getting balances. You&#39;re not simulating a conversation, &quot;<br>        &quot;but you will be in one yourself.&quot;<br>    )<br>}]</pre><h4>GitHub</h4><h3>Key Points of Parsing and Function Invocation Logic</h3><ol><li><strong>Input Parsing (Natural Language Processing)</strong></li></ol><ul><li>Use <strong>Natural Language Processing (NLP)</strong> to extract the intended action and parameters from the user’s input.</li><li>Example: Input &quot;Create a token named MyToken&quot; is parsed into:</li></ul><pre>{<br>    &quot;function&quot;: &quot;create_token&quot;,<br>    &quot;parameters&quot;: {<br>        &quot;name&quot;: &quot;MyToken&quot;,<br>        &quot;symbol&quot;: &quot;MTK&quot;,<br>        &quot;initial_supply&quot;: 1000000<br>    }<br>}</pre><ul><li>This step ensures the system understands the user’s request and converts it into a structured format for further processing.</li></ul><p><strong>2. Function Matching</strong></p><ul><li>Based on the parsing result, match the extracted function name with the functions defined in the functions list.</li><li>If no matching function is found: The agent provides a helpful message to the user, suggesting possible alternatives or inviting them to implement a new function.</li><li>This step ensures that only valid, predefined functions are invoked.</li></ul><p><strong>3. Function Invocation and Execution</strong></p><p>Once the matching function is identified:</p><ul><li>The agent invokes the function, passing the extracted parameters as arguments.</li><li>For example:</li></ul><pre>create_token(name=&quot;MyToken&quot;, symbol=&quot;MTK&quot;, initial_supply=1000000)</pre><p>The function executes its logic (e.g., interacting with the blockchain, performing calculations).</p><p>After execution:</p><ul><li>The result is returned to the user or another agent for further processing.</li><li>Example result: &quot;Token MyToken created successfully.&quot;</li></ul><h3>Features</h3><h4><strong>Automated Asset Management</strong></h4><ul><li>Automated management of blockchain assets (e.g., checking balances, transferring assets).</li></ul><p><strong>Why</strong>:</p><ul><li>Simplifies asset operations for users by reducing manual complexity.</li><li>Useful for large-scale transfers or automated payments.</li></ul><p><strong>How</strong>:</p><ul><li>Utilizes agent_wallet for core functionalities:</li><li><strong>Check balance</strong>: agent_wallet.balance(asset_id)</li><li><strong>Transfer assets</strong>: agent_wallet.transfer(amount, asset_id, destination_address)</li></ul><h4><strong>Gas Optimization</strong></h4><ul><li>Automatically selects optimal gas fees and supports gasless transactions for certain assets.</li></ul><p><strong>Why</strong>:</p><ul><li>Reduces transaction costs and improves efficiency.</li><li>Especially useful for micro-transactions and high-frequency trading.</li></ul><p><strong>How</strong>:</p><ul><li><strong>Gasless Transfer</strong>: Enabled for USDC on the Base Mainnet</li><li><strong>Gas Optimization</strong>: Automatically adjusts gas fees or bundles multiple transactions to reduce costs.</li></ul><h4>Token and NFT Operations</h4><ul><li>Supports creating ERC-20 tokens and ERC-721 NFTs, as well as performing related operations.</li></ul><p><strong>Why</strong>:</p><ul><li>Enables users to deploy tokens or NFTs quickly for project launches, collectibles, or other use cases.</li></ul><p><strong>Operations:</strong></p><ul><li>Token Operations</li><li>Create tokens</li><li>NFT Operations</li><li>Deploy NFT contract</li></ul><h4><strong>Basename Registration</strong></h4><ul><li>Supports registering .base.eth or .basetest.eth domains on the Base blockchain.</li></ul><p><strong>Why</strong>:</p><ul><li>Simplifies wallet address management by allowing users to use human-readable domain names.</li><li>Provides on-chain identity, suitable for branding or personal identification.</li></ul><p><strong>How</strong>:</p><ul><li>Register a domain</li><li>Underlying logic: Calls a smart contract to bind the domain to a wallet address.</li></ul><h4><strong>Automated Workflows</strong></h4><ul><li>Supports automated execution of blockchain operations without user intervention.</li></ul><p><strong>Why</strong>:</p><ul><li>Improves efficiency, especially for scheduled tasks or event-driven scenarios.</li></ul><h4><strong>Social Media Operations (Twitter)</strong></h4><ul><li>Can post tweets, read mentions, and reply to messages.</li></ul><p><strong>Why</strong>:</p><ul><li>Integrates on-chain and off-chain capabilities, suitable for social marketing or automated interactions.</li></ul><p><strong>How</strong>:</p><ul><li>Defined operations <strong>(but not integrated into the main program)</strong></li></ul><h4>Asset Swapping</h4><ul><li>Supports token swaps on decentralized exchanges.</li></ul><p><strong>Why</strong>:</p><ul><li>Simplifies asset trading for users, applicable in DeFi applications.</li></ul><h4>LangChain integration</h4><p>Seamless integration with <a href="https://python.langchain.com/docs/introduction/">LangChain</a> for easy agentic workflows.</p><h3>Conclusion</h3><p>Coinbase’s CDP Agentkit is a foundational framework that demonstrates how AI agents can interact with blockchain environments effectively. By providing essential tools for automated asset management, NFT operations, token deployment, and more, it opens the door for developers to explore innovative applications in Web3. The integration with LangChain further enhances its potential for agentic workflows, offering flexibility and modularity.</p><p>While the framework is promising, it’s essential to acknowledge that it serves as a conceptual starting point rather than a fully production-ready solution. Developers looking to deploy it in real-world scenarios will need to account for additional development, particularly when handling complex tasks like DeFi operations or multi-step workflows.</p><p>From a broader perspective, the CDP Agentkit introduces a robust approach to building AI agents — not just for blockchain but for other domains requiring automation and integration. By leveraging tools like the Replit template, developers can quickly prototype and test ideas, paving the way for more accessible and streamlined blockchain interactions.</p><p>Looking forward, the framework’s potential integration with platforms like Twitter and partnerships in DeFi holds promise for advancing user experience and operational efficiency. However, ensuring regulatory compliance and obtaining user consent will be crucial steps to maximize its applicability while maintaining trust and integrity.</p><p>Overall, the Coinbase AI Agentkit provides a glimpse into the future of AI-driven automation in Web3, serving as both a toolkit and an inspiration for developers to innovate beyond traditional paradigms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99a1d7605e49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI to Find Instrument Practice Rooms: Automating Bass Space Search with GPT and Playwright]]></title>
            <link>https://medium.com/@jo.ym/ai-to-find-instrument-practice-rooms-automating-bass-space-search-with-gpt-and-playwright-67c4ee761404?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/67c4ee761404</guid>
            <category><![CDATA[serpapi]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[google-places-api]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 16:37:35 GMT</pubDate>
            <atom:updated>2026-01-06T06:04:21.724Z</atom:updated>
            <content:encoded><![CDATA[<h4>Combining Google Places API and SerpAPI for smarter discovery</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DfmbVeMdE38HJewx" /><figcaption>Photo by <a href="https://unsplash.com/@rangel?utm_source=medium&amp;utm_medium=referral">David Rangel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Recently, I decided to start learning the bass! (you know, the instrument that looks like a guitar), but I quickly realized how frustrating it was to find a practice room!</p><p>The resources are scattered, and there’s no unified way to organize the information, which makes searching and verifying options incredibly time-consuming.</p><p>So, why not create a small tool to streamline the process of finding practice rooms? What I didn’t expect was how much more complex this task would turn out to be.</p><h3>Overview of the Process</h3><ol><li><strong>Gather Potential Locations</strong>: Using the <strong>Google Places API</strong>.</li><li><strong>Filter Irrelevant Locations</strong>: By applying <strong>keywords</strong> and leveraging <strong>SerpAPI</strong>.</li><li><strong>Extract Info from Places and Details API</strong>: Using <strong>Place Details API</strong>, processing business hours, and then saving the data.</li><li><strong>Extract Practice Room Details</strong>: Utilizing <strong>Playwright</strong> and <strong>LLM (GPT)</strong>.</li></ol><h3><strong>Gather Potential Locations</strong>: Using the <strong>Google Places API</strong></h3><ol><li><strong>Apply for a Google Places API Key</strong>: Follow the official <a href="https://developers.google.com/maps/documentation/places/web-service/get-api-key">documentation here</a></li><li><strong>Install the Python Google Maps Package</strong>: Use the google-maps-services-python library. You can find it <a href="https://github.com/googlemaps/google-maps-services-python">here</a>.</li><li><strong>Use the Google Places API (New) — Text Search</strong>:</li></ol><ul><li><strong>Code Example</strong>:</li></ul><pre>import googlemaps<br><br><br>gmaps = googlemaps.Client(key=&#39;&lt;Your API Key&gt;&#39;)<br>query = f&#39;&lt;Your City/Region&gt; Practice Room Band Room Space Rental Venue Rental Piano Room&#39;<br>places_result = gmaps.places(query=query)</pre><pre>import googlemaps<br><br><br>gmaps = googlemaps.Client(key=&#39;&lt;YOUR KEY&gt;&#39;)<br>query = f&#39;&lt;你的城市/行政區&gt; 練習室 練團室 空間租借 場地租借 琴房&#39;<br>places_result = gmaps.places(query=query)</pre><ul><li><strong>Why not Nearby Search?</strong> It requires specifying coordinates and a range, but I’m not great with geography, so I don’t know what range to set. :D</li><li><strong>If you prefer using Nearby Search</strong>: You can use the <strong>Geocoding API</strong> (make sure to enable its permissions, just like for the Places API) to get the approximate latitude and longitude of a city center by providing a city name.</li><li><strong>Code Example</strong>:</li></ul><pre># Get the latitude and longitude of the target city <br>geocode_result = gmaps.geocode(&quot;台中市&quot;)<br>location = geocode_result[0][&#39;geometry&#39;][&#39;location&#39;]<br>latitude = location[&#39;lat&#39;]<br>longitude = location[&#39;lng&#39;]<br><br># Set keywords<br>keywords = [&#39;空間租借&#39;, &#39;場地租借&#39;, &#39;練團室&#39;, &#39;練習室&#39;, &#39;琴房&#39;]<br>keyword_str = &#39; OR &#39;.join(keywords)<br><br>places_result = gmaps.places_nearby(<br>    location=(latitude, longitude),<br>    radius=5000,<br>    # type=&#39;music_school&#39;, # or store<br>    keyword=keyword_str<br>)</pre><h3><strong>Filter Irrelevant Locations</strong>: By applying <strong>keywords</strong> and leveraging <strong>SerpAPI</strong></h3><p>Since not all search results necessarily offer practice rooms, we need to filter them (this step took a lot of time, and there may still be mistakes — feedback and better ideas are welcome!).</p><p>Here’s the approach I ended up using:</p><ol><li><strong>Place Names</strong>: Filter by keywords. However, be cautious! Those that don’t meet the criteria should still be given another chance to move to the next step.</li><li><strong>Using </strong><a href="https://serpapi.com/"><strong>SerpAPI</strong></a>: Retrieve the <strong>description</strong> and <strong>Google Search results</strong> via <a href="https://serpapi.com/"><strong>SerpAPI</strong></a>. Then filter by keywords within the <strong>search results’ snippets</strong>. However, I found that giving too many results a chance tends to loosen the criteria, leading to irrelevant places. Therefore, I only allow the <strong>top result</strong> to pass the filter (let’s trust Google Search’s algorithm for now…).</li></ol><ul><li><strong>Please note the encoding of the response from SerpAPI!</strong></li></ul><p><strong>Code Example:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8baf9f9949cefc85c309e9827e474132/href">https://medium.com/media/8baf9f9949cefc85c309e9827e474132/href</a></iframe><h3><strong>Extract Info from Places and Details API</strong></h3><ul><li><strong>(Optional) Prepare a Retry and Rate Limit Control Class</strong>: It’s also helpful to have a CSV management class ready. You can refer to the beginning of the “Step 2. Retrieve data via Spotify Web API” section in <a href="https://link.medium.com/UsNuWZetnPb">this article</a>.</li><li><strong>Get detailed info like opening hours</strong>: Using <strong>Place Details API</strong>.</li><li><strong>Process the opening hours</strong>: Storing business hours in your preferred structure. In my case, I decided to store each day of the week (Monday through Sunday) in separate columns. However, <strong>be mindful of exceptions: some places may only provide opening times without corresponding closing times</strong>.</li></ul><p><strong>Code Example:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/188d5934a91e0538b996b33ad616bcec/href">https://medium.com/media/188d5934a91e0538b996b33ad616bcec/href</a></iframe><h3><strong>Extract Practice Room Details</strong>: Utilizing <strong>Playwright</strong> and <strong>LLM (GPT)</strong></h3><p>This is the most challenging part! It’s like building a custom “SearchGPT” similar to ChatGPT’s capabilities.</p><p>Although I haven’t fully completed the entire script yet (I’ll share it once it’s ready), here are the steps I’ve successfully implemented so far:</p><ol><li><strong>Use Playwright to scrape all nodes and their names</strong>: Collect all the links and associated titles from the target webpage.</li><li><strong>Feed the data to an LLM</strong>: Pass the collected information to the LLM, asking it to identify which node will most likely contain the desired information.</li><li><strong>Scrape content from the selected page</strong>: Visit the page recommended by the LLM and extract the webpage content.</li><li><strong>Request structured data from the LLM</strong>: Ask the LLM to organize the extracted content into a structured format.</li></ol><p><strong>Some Code Examples:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f3aec9ada9d8a5a5f30b3e473cf887a/href">https://medium.com/media/6f3aec9ada9d8a5a5f30b3e473cf887a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41b74d4f0d0bf1260156dab3b5ab05cc/href">https://medium.com/media/41b74d4f0d0bf1260156dab3b5ab05cc/href</a></iframe><h3>Lessons Learned</h3><p><strong>Challenges and Pitfalls</strong>:</p><ul><li><strong>Filtering by </strong><strong>type doesn’t work</strong>: The types provided by the Places API doesn’t align with what Google Maps displays, as Google does additional processing in between.</li><li><strong>Not all places have practice rooms</strong>: Even after filtering, it’s not 100% accurate. To ensure a location truly has a practice room, I’d need to continue refining the process to fully extract and verify the information. For now, I’ve managed to filter out a list of candidates.</li><li><strong>LINE Official Accounts are a hassle</strong>: Many of these locations like to put critical information in their LINE official accounts, which makes things super inconvenient 😓 because I can’t scrape that data (BTW, I’m in Taiwan).</li></ul><p><strong>Improvements and Future Work</strong>:</p><ul><li>Fully implement the SearchGPT-like functionality.</li><li>Create a cleaned and filtered table.</li><li>Build a complete tool.</li><li>A dream project (but might be complicated due to commercial considerations): Develop a reservation tool that all businesses can use! The data is just too scattered right now!</li></ul><h3>Conclusion</h3><p>This exploration is just the beginning — a journey from scattered resources to automated organization, gradually piecing together a viable solution.</p><p>Although it’s not yet perfect to extract all the information, I’ve already established a clear process and direction. In the future, I’ll continue improving the practice room information extraction and hope to eventually make this tool available to anyone who needs it.</p><p>Alright, time to go learn bass! 🎉 Playing an instrument has been the most therapeutic thing for me — I’ve discovered how deeply I can immerse myself in it. After learning keyboard and guitar, I’m finally about to dive into the instrument I’ve been most excited about: bass! I love its role and its unique tone. 🎶</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67c4ee761404" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[After Hours Explorations: Unveiling K-pop with Spotify’s API]]></title>
            <link>https://medium.com/@jo.ym/after-hours-explorations-unveiling-k-pop-with-spotifys-api-028a60ace5a3?source=rss-9adbf55f1d0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/028a60ace5a3</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[spotify-api]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[k-pop-industry]]></category>
            <dc:creator><![CDATA[Jo Chen]]></dc:creator>
            <pubDate>Sat, 07 Dec 2024 10:23:54 GMT</pubDate>
            <atom:updated>2026-01-06T06:05:49.897Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discovering trends, “hardest-working” champs, and a slice of fun through data. (Bonus: Tips for Recreating Spotify Wrapped)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ao79HIz-S63QW-to" /><figcaption>Photo by <a href="https://unsplash.com/@hyleee7?utm_source=medium&amp;utm_medium=referral">Yeon Lee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>K-pop</strong>’s global takeover is undeniable, but this isn’t a groundbreaking exposé — it’s just a curious fan’s playful data dive inspired by Spotify Wrapped. Combining my love for K-pop and interest in <strong>Spotify’s Web API</strong>, I embarked on a journey to answer lighthearted questions like:</p><ul><li>Who are the “hardest-working” idols in K-pop?</li><li>Which companies release the most albums?</li></ul><p>As a bonus, I’ll share some tips for creating your version of Spotify Wrapped (though you might need infinite patience for the data to load).</p><p>The process involves:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2q4Oxn1Z7IhkdoR8WO1J1w.png" /><figcaption>flowchart of this project</figcaption></figure><p>I will first explain the process and share some tips for each step. If you’re interested, you can try it out on your own before considering my approach (which might not necessarily be the best method).</p><h3>Step 1. Scrape data from Wikipedia</h3><p>Since Spotify’s metadata wasn’t enough to meet my needs, I turned to Wikipedia for richer metadata, including debut years, agencies, and debut dates. The process involved scraping artist lists organized by decades and extracting data from individual artist pages.</p><h4>Key Challenges and Tips:</h4><ol><li><strong>Footnote Noise:</strong> Remove unwanted footnotes to keep your data clean.</li><li><strong>Varying Structures:</strong> Some lists are single-column, others two-column. Check the source code for consistency.</li><li><strong>Redirects:</strong> Use <strong>MediaWiki API</strong> to handle artist URLs that redirect.</li><li><strong>Multiple Active Periods:</strong> For artists with two activity periods, use the first and last dates.</li><li><strong>Ambiguous Company Names:</strong> Standardize names but be aware of subsidiaries and variations.</li></ol><p>Therefore, our steps will be structured as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-cbvTyrpJQr9-cFw-z0Mfg.png" /><figcaption>flowchart of step 1</figcaption></figure><p><strong>Scrape different decades’ list</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqJQlx56k0P-DXGV_YVAuQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a364178b119e9be0d717d1b41dc98b17/href">https://medium.com/media/a364178b119e9be0d717d1b41dc98b17/href</a></iframe><p><strong>Scrape list of groups for each decade</strong></p><p>There are two tips:</p><ul><li>Remember to remove the footnotes.</li><li>The list page structure of each era may be different. For example, some are listed in one column, and some pages are divided into two columns. Please open the source code to observe the details.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zc5YxK_BG-uDtm0nHCflqg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I35MOKHDKCpKK2UfTvJEgg.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/611538eb2ae18905ed480f56b3d6ddbb/href">https://medium.com/media/611538eb2ae18905ed480f56b3d6ddbb/href</a></iframe><p><strong>Scrape information from artist pages</strong></p><p>Tips:</p><ul><li>Some artist request URLs will be redirected. We need to check this with <strong>MediaWiki API</strong>, otherwise the data will not be read.</li><li>Some artists may have two periods of active time. Here we directly take the first and last periods. Of course, sometimes there are null values.</li><li>Some artists’ companies will write down several names, so here will just pick the first one.</li><li>We need to try a few more artist pages at first because some pages have different structures, so you finally write the following function</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Cm6GU-gzeq4Bq1in6FmXQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97dcde04ae57eb9ded77e1a246b08b2d/href">https://medium.com/media/97dcde04ae57eb9ded77e1a246b08b2d/href</a></iframe><h3>Step 2. Retrieve data via Spotify Web API</h3><p><strong>Context Recap</strong></p><ul><li>Before using the Spotify API, you need to apply for a developer client_id and client_secret. For more details, refer to the <a href="https://developer.spotify.com/documentation/web-api">Spotify for Developers documentation</a>.</li><li>Spotify does not provide raw play counts but instead offers a popularity metric for both artists and tracks. This metric is a composite score. Spotify says: <em>The popularity of the track. The value will be between 0 and 100, with 100 being the most popular.<br>The popularity of a track is a value between 0 and 100, with 100 being the most popular. The popularity is calculated by algorithm and is based, in the most part, on the total number of plays the track has had and how recent those plays are. Generally speaking, songs that are being played a lot now will have a higher popularity than songs that were played a lot in the past. Duplicate tracks (e.g. the same track from a single and an album) are rated independently. Artist and album popularity is derived mathematically from track popularity. </em><strong><em>Note</em></strong><em>: the popularity value may lag actual popularity by a few days: the value is not updated in real time.</em></li><li>We’ll use the Spotify Python library: <a href="https://github.com/spotipy-dev/spotipy">spotipy</a></li></ul><p><strong>The data we want and the endpoints we will use:</strong></p><ul><li>Information about each artist: /search</li><li>All albums of each artist: /artists/{id}/albums</li><li>All tracks of each album: /album/{id}/tracks</li><li>Information of each track: /tracks</li></ul><p><strong>The most important thing to note: request limit</strong></p><p>OK, let’s start!</p><p>Firstly, we need to create a connector.</p><pre>import spotipy<br>from spotipy.oauth2 import SpotifyClientCredentials<br><br><br>sp = spotipy.Spotify(<br>    auth_manager=SpotifyClientCredentials(<br>        client_id=&quot;your client id&quot;,<br>        client_secret=&quot;your client secret&quot;<br>    ),<br>    requests_timeout=5<br>)</pre><p>Tips:</p><ul><li>If you want to get user information, there will be different ways to write it. The Bonus part will be mentioned later.</li><li>Maybe you would like to know more about <a href="http://Customized token caching">the cache part</a></li></ul><p>Here, since we want to control the execution speed and implement retries for the API calls, we wrote a separate executor class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/642bb642144f305773f614d1950ab0ab/href">https://medium.com/media/642bb642144f305773f614d1950ab0ab/href</a></iframe><p>(By the way, I also wrote a version that allows using multiple clients for execution. However, I don’t encourage doing this “bad thing&quot;, so I won’t share the implementation here.)</p><p>Additionally, to prevent data loss during interruptions, we opted to save the data as it is being fetched. To achieve this, we wrote some utility functions. Here, we provide a method for saving locally as CSV. However, this approach involves high I/O operations and is not ideal. Here encourage everyone to improve it by implementing a more efficient version.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8466921214e1bc875261903815f86d5/href">https://medium.com/media/a8466921214e1bc875261903815f86d5/href</a></iframe><p><strong>Request artists’ data</strong></p><p>We are really going to start requesting data from Spotify now…</p><p>To proceed with the next steps, we need to obtain the artists’ IDs on Spotify, which requires using the search functionality. However, there’s a tricky issue here: artists with the same name or partially matching names.</p><p>To <strong>minimize</strong> this problem, we need to use certain parameters to get the results we want:</p><ul><li><strong>Limit the </strong><strong>market to </strong><strong>&#39;KR&#39;</strong> (South Korea)</li><li><strong>Ensure it’s related to </strong><strong>&#39;k-pop&#39; by using the </strong><strong>genre parameter</strong></li></ul><p>Even with these measures, we might still retrieve data that isn’t relevant to our needs. Therefore, we’ll need to clean the data afterward.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d365dfc07caa536144436f6c07b4782/href">https://medium.com/media/7d365dfc07caa536144436f6c07b4782/href</a></iframe><p>Unfortunately, this process doesn’t account for solo releases by group members. This is something that could be explored in future iterations.</p><p><strong>Request artists’ all albums</strong></p><p>There are several Albums types:</p><ul><li>album</li><li>single</li><li>appears_on</li><li>compilation</li></ul><p>Here we only use album &amp; single</p><p><strong>Note</strong> that here will be an <strong>album with the same name but it will be released in different markets</strong>. Please see if you want to exclude it based on the requirements.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5c45d642c414db868e9e79bcc3f504b/href">https://medium.com/media/e5c45d642c414db868e9e79bcc3f504b/href</a></iframe><p><strong>Request tracks of each album</strong></p><p>Please notice the request limit.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de79bd83009bed842e2010c537c4373e/href">https://medium.com/media/de79bd83009bed842e2010c537c4373e/href</a></iframe><p><strong>Request tracks’ detailed data</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d00f60ecbfdebe5004f88c897b9d2026/href">https://medium.com/media/d00f60ecbfdebe5004f88c897b9d2026/href</a></iframe><h3>Step 3. Process and clean data</h3><p>At this point, you can clean or create derived fields in the dataset based on your analysis needs. It’s <strong>important to first identify what you want to analyze</strong> and then apply the necessary transformations.</p><p>Here are a few recommended steps:</p><ul><li><strong>Clean company names</strong>: Some companies are the same, while others are subsidiaries. My approach here isn’t perfect — I only standardized names for major top-tier companies or obvious cases, such as those with suffixes like Entertainment or Music.</li><li><strong>Filter </strong><strong>genres by </strong><strong>k-pop</strong>: As mentioned earlier, even specifying the genre parameter doesn’t guarantee accurate results. So, I directly filtered the data to include only entries containing <strong>k-pop</strong>.</li><li><strong>Add group type column</strong>: Determine whether it is a girl group or a boy group based on <strong>genres</strong>. But it failed because in the end it was found that the 5th generation group did not have the label girl or boy.</li><li><strong>Add album count column</strong>: Calculate the artists’ album number and add them to artist data.</li><li><strong>Merge data</strong>: Merge artist data from wiki and spotify, and merge track info data.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b905524c618317abc981e578cf1c6d81/href">https://medium.com/media/b905524c618317abc981e578cf1c6d81/href</a></iframe><h3>Step 4. Visualization of Some Questions</h3><ul><li><strong>Currently Active Groups by Company</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*-APwTU9f2nECD3YnkR0WMA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2243609e52894f3f3079e12dd22a454/href">https://medium.com/media/b2243609e52894f3f3079e12dd22a454/href</a></iframe><ul><li><strong>Albums Released by Year and Top Company (2001–2024)</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlhq6c_m4Mh8YjF28iGRnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GgumYRN9VtSeDCI4vRWa4w.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ed73e542095da6ee0c5dfe56cc43fc1/href">https://medium.com/media/3ed73e542095da6ee0c5dfe56cc43fc1/href</a></iframe><ul><li><strong>Albums per Active Group by Top Company (2020–2024)</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bGLVEc3yvJSAqVEtHrOcng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*itWDt5vp-QzxoW42LKI06w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3PUFDA3nwJkH6bqfu656IA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ebe1ba2fe2e0940f74af31bf55993c7/href">https://medium.com/media/5ebe1ba2fe2e0940f74af31bf55993c7/href</a></iframe><ul><li><strong>Top 20 “Hardworking” Groups</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y2qIk9dJrYX2k6urdM-Iig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YLo6oIXLSPByPQbme7HccQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b78caf31004f4d967735d7f7d836646/href">https://medium.com/media/3b78caf31004f4d967735d7f7d836646/href</a></iframe><ul><li><strong>Top 100 Tracks Group Distribution</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*hgg0i4MEz-ZWc8_EaItHHQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10c075909a03511a667c177a82f0a181/href">https://medium.com/media/10c075909a03511a667c177a82f0a181/href</a></iframe><ul><li><strong>Top 100 Tracks by Top Companies</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*fJqrhiB5StN7aayQvGTj7A.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1c5850b9186cc6b4cb157b6d4424b9e/href">https://medium.com/media/d1c5850b9186cc6b4cb157b6d4424b9e/href</a></iframe><ul><li><strong>Top 20 Artitists by Followers and affiliated companies</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*kmqBNqRcdn1acfDpIVWGQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*2t87pBPli6bQxv17m-Ui-Q.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04fb1a91251961498eb9d53791e330e7/href">https://medium.com/media/04fb1a91251961498eb9d53791e330e7/href</a></iframe><h3>Bonus: Tips for Recreating Spotify Wrapped</h3><ul><li>In the Spotify developer backend, add your account as a user</li><li>Remember to add the scope required on the corresponding document when creating the SpotifyOAuth</li><li>Endpoints: /me/top/{type} and /me/player/recently-played</li></ul><h3>Closing Thoughts</h3><p>It’s a pity that Spotify recently deprecated some highly useful endpoints, like analysis and features. These could have added a lot more depth to this project.</p><p>In fact, release and activity information can also be obtained from Wikipedia or schedules compiled by major fanbases, which might provide a more accurate way to evaluate “hard-working” artists than relying solely on Spotify data. <strong>BTW, I hope every hardworking artist receives fair compensation and treatment.</strong></p><p>As for the market parameter, there’s a lot of creative potential to explore!</p><p>That said, this was just a fun exploration to poke a little fun and satisfy some curiosity — definitely not meant to be a serious analysis. As mentioned at the beginning, this was more about diving into the Spotify Web API and sharing some tips along the way. 😊</p><p>If you have any thoughts or ideas, I’d love to hear from you! Let’s connect and discuss.</p><p>The full code for this project is available here: <a href="https://github.com/jo-ym/K-Pop-Analytics">GitHub Repository</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=028a60ace5a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>