<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Darío Andrés Muñoz Prudant on Medium]]></title>
        <description><![CDATA[Stories by Darío Andrés Muñoz Prudant on Medium]]></description>
        <link>https://medium.com/@prudant?source=rss-98dbc610ea82------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JimyKm_JcKJYaubWBVuXjw.png</url>
            <title>Stories by Darío Andrés Muñoz Prudant on Medium</title>
            <link>https://medium.com/@prudant?source=rss-98dbc610ea82------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:08:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prudant/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond the Workflow: Agentic Architecture as Small, Honest Programs]]></title>
            <link>https://ai.plainenglish.io/beyond-the-workflow-agentic-architecture-as-small-honest-programs-24e6b2facd3a?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/24e6b2facd3a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[workflow-automation]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 01:35:01 GMT</pubDate>
            <atom:updated>2026-04-01T01:50:45.241Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6NMg2igikUmwgOq3lTFwRw.png" /></figure><h3>Beyond the workflow: agentic Architecture as small, honest programs</h3><h4>How we rebuilt our AI workflow engine around agents and tools that own their behavior — and why “who answers the user” became a first-class design question.</h4><p>When a single orchestrator swells into thousands of lines of intertwined logic, the system does not fail all at once. It fails quietly: every new feature tugs against every other; tests become integration marathons; and the line between “platform capability” and “one team’s business rule” disappears.</p><p>We took a different bet: <strong>each capability exposed to the workflow should be self-contained</strong> — prompts, schemas, and domain logic living together — while the runtime only knows how to <em>invoke</em> that capability, not how to <em>be</em> it.</p><p>This post is about that shift: delegation, <strong>multi-level</strong> subagents, pass-through, and why explicit boundaries between orchestrators, specialists, and tools matter more than one central brain re-summarizing every layer.</p><h3>From “one brain” to “many specialists”</h3><p>The old picture is familiar: a generic agent absorbs routing, retrieval, prompting, and tool glue. It scales until it doesn’t.</p><p>The new picture is <strong>specialized agents and tool surfaces</strong>. In the newer stack, each workflow-facing capability tends to have a thin <strong>adapter</strong> that plugs into the engine (memory, UI, configuration, reporting) and a <strong>core</strong> that encodes what that capability is allowed to do and say. The adapter translates world state into inputs; the core runs the actual orchestration loop, generation step, branch, retrieval pass, or editing action.</p><p>That split keeps <strong>platform plumbing</strong> and <strong>product behavior</strong> from melting into one another. You can reason about “how we schedule agents and tools” separately from “what this agent is contractually allowed to output.”</p><h3>Delegation as a design principle</h3><p>Delegation here is simple to state and hard to get right in practice:</p><ul><li>The workflow invokes a <strong>named method</strong> backed by a registered capability.</li><li>The <strong>adapter</strong> turns runtime context into something the specialist understands.</li><li>The specialist <strong>executes</strong> — perhaps a single LLM call, perhaps a persistent tool loop with planning — and returns a result the parent can route.</li></ul><p>Nothing in that story requires a god-object agent that owns every prompt in the company. Each specialist <strong>owns</strong> its prompts and schemas; the platform <strong>carries</strong> conversation state, execution metadata, and delivery rules. That ownership is what makes incremental migration possible: you swap or extend one capability at a time without rewriting the entire orchestration layer.</p><h3>Subagents: nested intelligence with explicit contracts</h3><p>When an orchestrator agent <strong>calls another agent</strong> as a tool-like step, you effectively have a <strong>subagent</strong>: same stack, nested boundary. The hard part is not the extra LLM call — it is <strong>agreeing on how the nested run may end</strong>.</p><p>We treat that agreement as part of the contract:</p><ul><li>The parent exposes only the <strong>modes</strong> the child actually supports (full orchestration vs. terminal generation vs. pass-through routes).</li><li>The <strong>plan</strong> that drives the orchestrator must name a valid terminal path: either a local synthesizer owned by the parent orchestrator, or a single terminal step into a child whose surface allows explicit terminal routing.</li><li>Implicit “we are probably done” is rejected; <strong>invalid or silent closures fail fast</strong>, which is preferable to duplicated streams or ambiguous final answers.</li></ul><p>Subagents therefore are not silent black boxes — they are <strong>peers with a declared finishing grammar</strong>.</p><h3>Deep stacks: parent → child → grandchild</h3><p>Real workflows rarely stop at one hop. An orchestrator may call a specialist, which in turn invokes a finer-grained subagent — <strong>parent, child, grandchild</strong>, and sometimes deeper. That is where paper-thin abstractions break: if every layer feels obliged to “say something final,” you get a chain of <strong>re-synthesis</strong>.</p><p>Each extra summarization steals detail: citations get dropped, qualifiers disappear, and the conversational <strong>thread</strong> no longer matches what the bottom layer actually concluded. The user’s mental model (“the retrieval agent said X”) drifts from what upper layers paraphrase (“the workflow says something like X”).</p><p><strong>Pass-through</strong> is the countermove. It means the layer that truly <strong>finished</strong> the work hands its <strong>terminal payload</strong> upward <strong>without</strong> asking another model pass to rewrite it for politeness, brevity, or faux consistency. Intermediate parents act as <strong>carriers</strong>: they honor routing, logging, and policy, but they do not become accidental editors of the answer. In a deep stack, that is how you preserve <strong>semantic fidelity</strong> — verbatim where it matters, stable control metadata (already streamed, final text, route) so no level double-speaks to the UI.</p><p>Deep nesting is therefore not only a technical nesting of calls; it is a test of whether your architecture <strong>trusts</strong> lower specialists to own their wording and evidence, or forces every ancestor to re-narrate.</p><h3>Generate vs. pass-through: two different jobs</h3><p>In conversational systems, the hardest questions are often social, not algorithmic: <strong>whose turn is it to finish the thought?</strong> We split that into two ideas that are easy to conflate but serve different purposes.</p><h3>Why generate is valuable on its own</h3><p><strong>Generate</strong> means: *this specialist owns the resolution of the terminal payload for that boundary* — often through substantive model work such as grounding, structure, tone, citations, or tool use internal to that agent, but sometimes through a capability-specific path that still produces the final payload without asking the parent to author it. The value is not “another label on the same LLM call.” It is <strong>separation of concerns</strong>:</p><ul><li><strong>Who authors</strong> stays with the agent that has the right context, tools, and prompts for the job (the leaf or mid-level specialist), not with a distant orchestrator that would otherwise compress intent into a second, lossy pass.</li><li><strong>Where the answer is delivered</strong> (end user vs. calling orchestrator) can be decided by <strong>policy and explicit terminal requests</strong> without forcing the parent to <strong>re-generate</strong> content it did not produce. The parent coordinates; it does not have to impersonate the specialist.</li></ul><p>When nothing above it is orchestrating a nested run, <strong>generate</strong> still defaults sensibly — typically toward the user — so standalone runs behave predictably. When there <em>is</em> a parent, the same mode keeps <strong>one</strong> locus of generation honest while routing remains explicit.</p><h3>Why pass-through matters especially under depth</h3><p><strong>Pass-through</strong> (to the UI or upstream) means: *carry this outcome along the ancestry chain <strong>as the outcome</strong>*, not reinterpret it. Next to <strong>generate</strong>, pass-through is what stops a three-layer stack from becoming three passes of “make it sound final.” It protects <strong>thread continuity</strong> (what was said, by which step, with which evidence) and avoids the <strong>telephone game</strong> that generic “wrap everything in a summarizer” designs invite.</p><p>Together: <strong>generate</strong> answers <em>who creates the substance</em>; <strong>pass-through</strong> answers <em>who is allowed to touch that substance on the way out</em>. Framing both explicitly avoids silent double-speak — if a subagent <strong>already streamed</strong> the definitive answer, the parent should not stream it again — and keeps <strong>terminal ownership</strong> a matter of contract, not accident.</p><p>For the <strong>tool-driven orchestrator agent</strong>, <strong>orchestrate</strong> remains another mode: it may keep planning and close locally — <strong>or</strong> delegate a terminal leg to a specialized subagent when that specialist advertises compatible terminal behavior. The planner cannot accidentally finish; <strong>explicit terminal steps</strong> keep the system honest.</p><h3>Tools: internal, external, and the workflow boundary</h3><p><strong>Internal tools</strong> belong to the orchestrator’s own loop — planning persistence, human handoff, local synthesis — things that shape <em>how</em> the agent thinks, not arbitrary RPC into every subsystem.</p><p><strong>External tools</strong> surface <strong>other workflow-backed capabilities</strong>: retrieval, branching, editing, third-party automations. They participate in terminal delivery only when the target capability declares that it can honor the parent’s terminal routing request. That keeps the tool inventory honest: not every hammer is allowed to be the last word to the user.</p><p>The vocabulary is unified at the boundary: agents consume a <strong>tool registry</strong>; and, at the external-tool boundary, child executions are projected into <strong>normalized outcomes</strong> (value, error, control) so parents do not have to parse ad hoc payloads from every specialist.</p><h3>Two front doors, one idea</h3><p>Some specialists also expose <strong>design-time helpers</strong> in the product UI — suggested examples, configuration assists — beside <strong>run-time execution</strong> in the live workflow. Same conceptual split — adapter plus core — but different transport. The value is not that editor-time helpers are always the exact same behavior as runtime execution; it is that they can live under the same capability ownership model instead of becoming a second disconnected intelligence stack.</p><h3>What this buys you</h3><p><strong>Clarity of responsibility.</strong></p><p>If behavior lives with the agent or tool surface that defines it, ownership of bugs and prompts is obvious.</p><p><strong>Testability in slices.</strong></p><p>Orchestration policy, tool contracts, and individual specialists can be tested without simulating the entire platform.</p><p><strong>Evolvable orchestration.</strong></p><p>Migrated capabilities register with declared modes; the graph can grow without a single file becoming a chronicle of every domain.</p><p><strong>Safer nesting and deep stacks.</strong></p><p>Parent and child agree on <strong>how</strong> a nested run may end — not only <strong>what</strong> it returns. At parent → child → grandchild depth, <strong>pass-through</strong> limits re-synthesis so detail, citations, and conversational thread survive the walk back up the tree.</p><h3>Honest limits and where we are headed</h3><p>No architecture erases complexity; it <strong>relocates</strong> it.</p><p><strong>Migration is a long slope.</strong></p><p>Legacy execution paths run in parallel until the last consumer moves. That controls risk, but it requires a single coherent catalog of registered methods so operators — and introspection — see one picture.</p><p><strong>Uniformity vs. specialization.</strong></p><p>Some agents are thin generation wrappers; others run long-horizon tool loops with persisted plans. The balance is to share <strong>neutral machinery</strong> (the kind that does not encode a product story) while keeping prompts and rules local to each specialist.</p><p><strong>Observability and narrative.</strong></p><p>As graphs deepen, teams need clearer stories: what was planned, what tools ran, what was shown to whom. Rich execution steps and reporting are the natural next layer so humans — and agents in the loop — share the same timeline.</p><p><strong>Governance at the boundary.</strong></p><p>External tools and nested agents multiply the surface for policy: what may be invoked, from where, and with what memory visible. Configuration and contracts beat burying rules inside opaque prompts.</p><h3>Closing thought</h3><p>Agentic architecture is not only about “models that use tools.” It is about <strong>who decides, who speaks, and who owns the outcome</strong> when something completes — or fails.</p><p>Agents that delegate cleanly, tools that declare how results may pass through, and subagents that finish under an explicit contract turn those questions into <strong>design</strong> — reviewable, testable, and incremental — instead of leaving them to emerge from a monolith’s hidden corners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24e6b2facd3a" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/beyond-the-workflow-agentic-architecture-as-small-honest-programs-24e6b2facd3a">Beyond the Workflow: Agentic Architecture as Small, Honest Programs</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Splitting with purpose: Semantic chunking for advanced RAG systems]]></title>
            <link>https://ai.plainenglish.io/splitting-with-purpose-semantic-chunking-for-advanced-rag-systems-dbbdb73fe421?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbbdb73fe421</guid>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Mon, 31 Mar 2025 21:00:05 GMT</pubDate>
            <atom:updated>2025-04-07T01:20:18.676Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wYEXMDQx_Dg3J_ExTAolw.png" /></figure><p>💡 <strong>Elevate your RAG implementation with semantic chunking!</strong> 🚀 Check out the full code in our GitHub repository: <a href="https://github.com/puppetm4st3r/semantic_chunking">https://github.com/puppetm4st3r/semantic_chunking</a>. If you find this article valuable, please show your support with some claps 👏 and follow us for more content on cutting-edge NLP techniques. Together, let’s push the boundaries of what’s possible with language models! ✨</p><h3>Introduction</h3><p>Retrieval-Augmented Generation (RAG) has become the standard approach for grounding large language models in factual information and reducing hallucinations. However, not all RAG implementations are created equal. The difference between a basic RAG system and an advanced one often comes down to how effectively we process and chunk our documents before storing them in a vector database.</p><p>This article explores a sophisticated approach to RAG that leverages semantic chunking, hybrid search methods, and ranking algorithms to deliver significantly improved results for Spanish language texts. Let’s dive into what makes this implementation special and how it addresses common challenges in traditional RAG systems.</p><h3>Table of contents</h3><ol><li>The limitations of naive RAG approaches</li><li>Semantic chunking: Breaking text with meaning</li><li>Hybrid search: Combining vectors and keywords</li><li>Result fusion with Reciprocal Rank Fusion</li><li>Advanced text processing for Spanish language</li><li>System architecture and implementation details</li><li>Deployment flexibility with OpenAI or local models</li><li>Future improvements: Knowledge graphs and ontologies</li><li>Conclusion and key takeaways</li></ol><h3>The limitations of naive RAG approaches</h3><p>Most basic RAG implementations follow a simple pattern: split documents into chunks of fixed size (often by character count), embed these chunks, store them in a vector database, and retrieve them based on similarity to the query. While functional, this approach suffers from several significant drawbacks:</p><p>First, arbitrary chunking based on character count frequently breaks semantic units. A paragraph discussing a single concept might be split in the middle, or multiple unrelated paragraphs might be combined into a single chunk. This fragmentation disrupts the coherence of information and reduces retrieval quality.</p><p>Second, simple vector-based retrieval alone can miss relevant content that uses different terminology but covers the same concepts. This is particularly problematic in technical or specialized domains where the same concept might be described using various terms.</p><p>Third, naive ranking based solely on vector similarity fails to capture the nuanced relevance of documents to complex queries. A document might be semantically close to a query overall but missing specific key information requested.</p><h3>Semantic chunking: breaking text with meaning</h3><p>The core innovation in our approach is semantic chunking. Rather than splitting text at arbitrary character counts, our system analyzes the semantic structure of documents to identify natural breakpoints where topics change. Here’s how it works:</p><p>Our SemanticSplitGenerator calculates the cosine similarity between the embeddings of consecutive text segments. When this similarity drops below a threshold (typically the 95th percentile of all similarity scores), we identify a potential semantic boundary. This approach ensures that chunks represent coherent ideas or topics rather than arbitrary divisions.</p><p>The system further refines these chunks by:</p><ul><li>Preserving paragraph and sentence boundaries</li><li>Identifying natural semantic transitions using embedding distance patterns</li><li>Analyzing the flow of ideas through clustering similar sections</li><li>Joining lines where the semantic flow continues (e.g., when a new line doesn’t start with an uppercase letter)</li></ul><p>The result is a set of chunks that each contain semantically coherent information, making retrieval more precise and contextually appropriate. When a user asks a question, the system can retrieve complete ideas rather than fragments.</p><h3>Hybrid search: combining vectors and keywords</h3><p>While vector similarity is powerful for capturing semantic relatedness, keyword matching still offers advantages for precise terminology. Our system implements a hybrid search approach that combines the best of both worlds:</p><p>For vector search, we:</p><ul><li>Convert the query to a vector embedding</li><li>Use pgvector’s distance operator to find semantically similar documents</li><li>Calculate a normalized similarity score from 0–100</li></ul><p>For keyword search, we:</p><ul><li>Extract key terms from the query after removing Spanish stopwords</li><li>Use PostgreSQL’s websearch_to_tsquery for language-aware text search</li><li>Apply custom weighting to prioritize important document sections</li></ul><p>The hybrid approach catches matches that either method alone would miss. For example, if a user asks about “climate change impacts on agriculture,” the vector search might find documents discussing the relationship between changing weather patterns and farming, while the keyword search would find documents specifically mentioning “climate change” and “agriculture.”</p><h3>Result fusion with Reciprocal Rank Fusion</h3><p>Having results from both search methods raises a new challenge: how do we combine and rank them effectively? Our system implements a modified version of the Reciprocal Rank Fusion (RRF) algorithm that:</p><ol><li>Calculates a fusion score for each document using the formula: 1/(k + rank) where k is a constant (default: 60)</li><li>Weights the components based on original relevance scores from each method</li><li>Applies a 20% boost to documents found by both search methods</li><li>Normalizes final scores to a 0–100 range</li></ol><p>This sophisticated ranking approach ensures that documents strongly relevant in either search method rise to the top, while those with moderate relevance in both methods also receive appropriate recognition.</p><h3>Advanced text processing for Spanish language</h3><p>Working with Spanish text presents unique challenges that our system specifically addresses:</p><p>The normalize_spanish_text() function removes accents and normalizes special characters through character mapping. This ensures consistent text representation regardless of accent variations common in Spanish language texts.</p><p>Our keyword extraction process is tailored to Spanish, filtering out Spanish stopwords using NLTK or a fallback list when NLTK is unavailable. The BM25 ranking algorithm identifies the most relevant terms in each text fragment while accounting for Spanish language characteristics.</p><p>The database schema includes dedicated text search indices specifically configured for Spanish language processing, with PostgreSQL’s to_tsvector(&#39;spanish&#39;, content) ensuring proper language-aware tokenization and stemming.</p><h3>System architecture and implementation details</h3><p>The complete RAG pipeline consists of several integrated components:</p><ol><li><strong>Text Processing and Normalization</strong>: Specialized functions handle Spanish text, format it appropriately, and prepare it for further processing.</li><li><strong>Semantic Text Chunking</strong>: The SemanticSplitGenerator identifies natural breakpoints and creates semantically coherent chunks.</li><li><strong>Keyword Extraction with BM25</strong>: The system uses the Okapi BM25 algorithm to identify relevant keywords from each text fragment, parallelizing the process for efficiency.</li><li><strong>Vector Database Implementation</strong>: PostgreSQL with the pgvector extension stores and retrieves vector embeddings, with optimized indices for both vector and text search.</li><li><strong>Hybrid Search</strong>: The hybrid_search() function combines vector similarity and keyword matching for comprehensive retrieval.</li><li><strong>Result Fusion and Ranking</strong>: The reciprocal_rank_fusion() function combines and ranks results from both search methods.</li><li><strong>Answer Generation</strong>: The system provides both synchronous and streaming answer generation based on retrieved context.</li></ol><p>All these components work together to create a sophisticated RAG system capable of handling complex queries with high accuracy and relevance.</p><h3>Deployment flexibility with OpenAI or Local models</h3><p>The implementation is designed to work with either OpenAI’s API or local language models through compatible interfaces like vLLM, providing flexibility for various deployment scenarios:</p><p>For OpenAI integration, you simply provide your API key and point to the OpenAI endpoints.</p><p>For local deployment with vLLM, you can run models locally by starting the vLLM server with an OpenAI-compatible API and pointing the system to your local endpoints.</p><p>This flexibility allows you to balance cost, privacy, and performance considerations based on your specific needs, while maintaining the same interface and functionality.</p><h3>Future improvements: knowledge graphs and ontologies</h3><p>While our current implementation significantly improves upon naive RAG approaches, several exciting avenues for further enhancement exist:</p><p><strong>Knowledge Graphs</strong>: By extracting entities and relationships from text and organizing them into a structured graph, we could enable more precise retrieval based on relationships rather than just content similarity. For example, if a user asks about “climate policies in European countries,” a knowledge graph could directly identify relevant countries and their specific policies.</p><p><strong>Domain Ontologies</strong>: Integrating domain-specific ontologies would allow the system to understand hierarchical relationships between concepts. For instance, in a medical context, it would recognize that “hypertension” is a type of “cardiovascular condition,” improving retrieval for broader category queries.</p><p><strong>Temporal Reasoning</strong>: Adding capabilities to understand and reason about time would enhance the system’s ability to handle queries with temporal aspects, such as “How have climate policies evolved over the past decade?”</p><p><strong>Multi-document Reasoning</strong>: Enhancing the system to synthesize information from multiple documents would enable more comprehensive answers for complex queries that span several knowledge areas.</p><p>These improvements would transform the RAG system from a sophisticated retrieval mechanism into a true knowledge reasoning engine capable of handling increasingly complex information needs.</p><h3>Conclusion and key takeaways</h3><p>The semantic chunking approach to RAG presented here offers several significant advantages over naive implementations:</p><ol><li><strong>Semantic coherence</strong>: By chunking text based on meaning rather than arbitrary character counts, we ensure that retrieval returns complete, contextually appropriate information.</li><li><strong>Comprehensive retrieval</strong>: The hybrid search approach combines the strengths of vector similarity and keyword matching, capturing relevant documents that either method alone would miss.</li><li><strong>Sophisticated ranking</strong>: The modified Reciprocal Rank Fusion algorithm intelligently combines and ranks results from multiple search methods, optimizing relevance.</li><li><strong>Language-specific processing</strong>: Specialized handling for Spanish text ensures accurate processing, normalization, and keyword extraction.</li><li><strong>Deployment flexibility</strong>: The system works with both cloud-based and local LLM infrastructure, offering options for different deployment scenarios.</li></ol><p>By addressing the common limitations of naive RAG approaches, this implementation delivers more accurate, relevant, and coherent results for complex queries. As we continue to enhance RAG systems with knowledge graphs, ontologies, and other advanced techniques, we move closer to the goal of truly intelligent knowledge access systems.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://differ.blog/inplainenglish"><strong>Differ</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbbdb73fe421" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/splitting-with-purpose-semantic-chunking-for-advanced-rag-systems-dbbdb73fe421">Splitting with purpose: Semantic chunking for advanced RAG systems</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The lean machine: crafting production-grade user intent detection and content moderation AI with…]]></title>
            <link>https://blog.cubed.run/the-lean-machine-crafting-production-grade-user-intent-detection-and-content-moderation-ai-with-d261d9f28f2a?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/d261d9f28f2a</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[inteligencia-artificial]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Sun, 14 Jul 2024 01:49:47 GMT</pubDate>
            <atom:updated>2024-07-15T01:17:18.788Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SnJ_zssQmVI258QGbGSgA.png" /></figure><h3>The lean machine: crafting production-grade multilingual user intent detection and content moderation AI with minimal data</h3><p>First of all, thank you for taking the time to read this article. If you found it helpful and interesting, please give it a few claps 👏 and follow me 🚀 for more insights on efficient machine learning techniques and natural language understanding innovations.</p><h3>Intro</h3><p>Last week I wrote about [<a href="https://medium.com/@prudant/turbocharge-your-user-intent-classification-nlu-pipeline-with-efficient-machine-learning-and-fast-d83ef2251f1f">here</a>] an approach to machine learning for spotting user intentions and filtering content. That method laid the groundwork for training robust ML models without needing intense computing power or GPU resources. However, it still required a sizable dataset to produce production grade results.</p><p>Today, I am here to tell you how to create a more efficient, performing and production ready solution using transformers and a smart two-step training process, In the lab I’ve train a model that achieves production-grade results with small -sized datasets and limited GPU power (just one 3090 &#39;s GPU and 2 hours for the full traning and test process).</p><p>Here’s what’s new: I use a two-step training process. First, I apply contrastive learning to train high-quality multilingual foundation embeddings model (<a href="https://huggingface.co/intfloat/multilingual-e5-small">you can try with this</a>). Then, I fine-tune a classification head for specific tasks. This method needs way less data but still gives top-notch results for figuring out what users want and checking if content is appropriate.</p><p>In this article, I’ll break down how I did it. You’ll understand how I trained the model, what results I got, and why this matters for real-world AI use. Whether you’re deep into NLP or just curious about AI, you’ll find some interesting stuff here about doing more with less data.</p><h3>Training process</h3><h4>Contrastive learning for embeddings training</h4><p>In the first step of my process, I used contrastive learning to align the embeddings. This technique is powerful because it allows the model to learn meaningful representations from a relatively small and labeled dataset (supervised training).</p><p>Here I used about approximately 780 samples (of synthetic and non-syntethic text). This might sound like a small number, but here’s where it gets interesting: through the contrastive learning process, these 780 samples were transformed into nearly 304,000 constrastive unique training pairs.</p><p>How does this work? Contrastive learning involves creating pairs of similar and dissimilar examples. The model learns to push similar items closer together in the embedding vector space while pushing dissimilar items further apart. With 780 samples, we can create many different combinations of pairs for each classification class, exponentially increasing our effective training data.</p><p>This approach is particularly effective because it forces the model to learn nuanced distinctions between samples, resulting in high-quality embeddings that capture the essence of the data.</p><h4>Classification head training</h4><p>After obtaining these model with rich refined embeddings, the next step was to train a classification head. You can do this manually with pytorch or you can use the <a href="https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForSequenceClassification">AutoModelForSequenceClassification</a> to initialize a Transformer Classification model from the previous step and the resulting enriched embeddings model.</p><p>An important detail here is that I froze the weights of the embedding model during this step. This means that the parameters of the pre-trained embedding model remain unchanged, while only the newly added classification head is trained. This approach preserves the high-quality representations learned during the contrastive learning phase.</p><p>The <a href="https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForSequenceClassification">AutoModelForSequenceClassification</a> added the classification head on top of the embedding model, resulting in an architecture of type XLMRobertaForSequenceClassification. This choice was dictated by the architecture of the underlying embedding model (XLM-RoBERTa) and the sequence classification task.</p><p>Depending on the specific problem, the classification head for the output layer would be:</p><ul><li>For non-binary classification tasks, a softmax output layer should be used.</li><li>For binary classification tasks, a sigmoid output layer, which is ideal for yes/no decisions.</li></ul><p>This two-stage process — first training embeddings via contrastive learning, then freezing those embeddings and training only the classification head — allows for efficient use of the initial data and computational resources while still achieving high performance on the target task.</p><h4>Multilingual capability</h4><p>A key feature of this approach is its multilingual capability. I achieved this by using a high-performance pre-trained multilingual foundation embedding model. This choice allowed the final model to perform well across multiple languages without requiring separate training for each language.</p><h4>Validation and testing</h4><p>To ensure the model’s robustness and generalization ability, I conducted extensive testing. The validation set consisted of 120,000 samples (from syntethic uncensored LLM and a lot of non syntethic human data), providing a comprehensive test of the model’s performance across various inputs and languages.</p><h3>Results</h3><p>The results of this approach are particularly impressive considering the limited amount of training data and computational resources used. Compared to traditional approaches that might require thousands of samples and extensive GPU time, this method achieves comparable or superior results with just 780 initial samples and just one GPU for 2 hours of computing power.</p><p>Here are the key metrics from the testing phase with 120k samples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yd_jPxQJE5vHG0DgnHk6_A.png" /><figcaption>Image from training / testing process</figcaption></figure><p>And now with the magic of the dimensionality reduction we will see on human readable 2D scatter plot of how the foundation embeddings model was fitting across the training process over the embedding vectorial space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jsxOKR-FPl4Xpu48r4tpKQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OjH_GO3Y12cAOYUnuS43oA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d4cCrYcsp9UEMAvoGm3L7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5LjPbmQ1co1LC6OYZGXOWw.png" /><figcaption>Finally the enrich embeddings are fit!</figcaption></figure><p>These t-SNE plots showcase both the training and evaluation embeddings, the final figure at step 36.000 out of 38.074 shows at the train sub-plot (left) well-separated clusters for each class, demonstrating that the model has learned to effectively differentiate between various intents like affirmation, censorship, greetings, etc. This clear separation is a strong indicator of the model’s ability to distinguish between different classes, even with a limited initial dataset. The evaluation embeddings sub-plot (right) shows a similar pattern, albeit with fewer points, suggesting that the learned representations generalize well to unseen data. Notably, even categories that might seem conceptually close (like ‘greeting’ and ‘farewell’) are clearly distinguished in the embedding space. This visual evidence supports the effectiveness of the contrastive learning approach in creating robust, discriminative features from a relatively small dataset, which is crucial for the subsequent classification task. The clear clustering and separation in both training and evaluation sets suggest that the model has captured meaningful semantic relationships, setting a strong foundation for high-performance intent detection and content moderation tasks.</p><h3>Final thoughts</h3><p>Experimenting with this efficient, high-performance model with minimal data has been an exciting journey. The results demonstrate that with clever use of contrastive learning and strategic model architecture, we can push the boundaries of what’s possible in low-resource scenarios.</p><p>This approach opens up new possibilities for organizations and researchers working with limited datasets or computational resources. It’s particularly promising for specialized domains where large amounts of labeled data are hard to come by, or for applications requiring quick adaptation to new languages or tasks.</p><p>Looking ahead, I see several potential directions for further research:</p><ol><li>Exploring the limits of this approach: How far can we push performance with even smaller datasets?</li><li>Applying this method to other NLP/NLU tasks beyond intent detection and content moderation.</li><li>Investigating ways to further optimize the contrastive learning process for even greater efficiency.</li></ol><p>I’m excited to continue refining this approach and to see how others in the field might adapt and build upon it. As AI continues to evolve, techniques like this that maximize performance with minimal resources will be crucial in making advanced NLP/NLU capabilities more accessible and widely applicable.</p><p>Lastly, I’m thrilled to announce that I’ll soon be launching a space where you can freely test this model (and maybe others). Stay tuned for more details on how you can interact with and explore the capabilities of this lean, efficient AI firsthand.</p><h3>Cubed</h3><p><em>Thank you for being a part of the community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a> | <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://venturemagazine.net/"><strong>Venture</strong></a> | <a href="https://cubed.run/"><strong>Cubed</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d261d9f28f2a" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/the-lean-machine-crafting-production-grade-user-intent-detection-and-content-moderation-ai-with-d261d9f28f2a">The lean machine: crafting production-grade user intent detection and content moderation AI with…</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turbocharge your user intent classification NLU pipeline with efficient machine learning and fast…]]></title>
            <link>https://blog.cubed.run/turbocharge-your-user-intent-classification-nlu-pipeline-with-efficient-machine-learning-and-fast-d83ef2251f1f?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/d83ef2251f1f</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[nlu]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 07:05:25 GMT</pubDate>
            <atom:updated>2024-07-15T01:17:14.676Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HnzrsJRj24mFGec4eQvZaQ.png" /><figcaption>Img by the author</figcaption></figure><h3>Turbocharge your user intent classification NLU pipeline with efficient machine learning and fast embeddings (no GPU required)</h3><p>First of all, thank you for taking the time to read this article. If you found it helpful and interesting, please give it a few claps 👏 and follow me 🚀 for more insights on efficient machine learning techniques and natural language understanding innovations.</p><h3>Introduction</h3><p>In the ever-evolving world of natural language understanding (NLU), classifiers for intent detection play a crucial role in numerous applications, from chatbots to virtual assistants. However, large language models (LLMs) that excel at this task are often slow, costly, and require substantial computational resources. While these models offer the advantage of not needing training or fine-tuning, their performance is typically optimal only with a top-of-the-line or proprietary solutions (and a huge amount of VRAM).</p><p>My approach seeks to bridge this gap by leveraging machine learning (ML) classifiers combined with embeddings as a proxy for NLU. By doing so, I achieve a user intent classifier that runs over 20 times faster on a CPU and more than 100 times faster on a GPU compared to a sufficiently powerful LLM. This method maintains competitive accuracy without the hefty resource demands, making it an attractive alternative for efficient and fast intent classification.</p><p>This approach is also useful and more practical than training a BERT or SBERT classifier, as the embeddings and this ensemble do not require significant computational power. By leveraging pre-trained embeddings and an efficient ensemble of traditional machine learning models, we achieve high accuracy without the need for extensive computational resources.</p><h3>The motivation</h3><p>Conversational AI pipelines for retrieve-augmented generation (RAG) are pivotal. These pipelines are designed to enhance conversational interactions by retrieving relevant information and generating coherent responses. Detecting user intent is crucial for effective RAG, particularly in stages such as censorship, intent detection, and even sub-intent detection. These tasks often necessitate multiple calls to large language models (LLMs) and sophisticated prompt engineering, which can be both time-consuming and costly.</p><p>My motivation stems from the need to drastically reduce the time and number of calls to LLMs within these pipelines. By implementing a fast and efficient solution, I can achieve a high classification rate without the resource-intensive overhead of LLMs. This solution, although currently focused on classifying requests and questions, lays the foundation for future models that can support multi-turn conversations and chat history.</p><p>To illustrate the utility of my approach, I have developed an ensemble model specifically designed for the <strong>Spanish</strong> language, using embeddings from<strong> </strong><a href="https://huggingface.co/jinaai/jina-embeddings-v2-base-es"><strong>jinaai/jina-embeddings-v2-base-es</strong></a>. This model efficiently classifies the following intents: censorship, others, lead (for potential comercial leads), contact (when user required contact information), directions (when a user needs to know how to get somewhere), meet, negation, affirmation, and casual chat.</p><p>At the end of the article, I will share the training dataset link, the training and evaluation notebook and the trained model, making it accessible for those interested in exploring and utilizing this approach. By focusing on these specific intents, my model ensures a streamlined and cost-effective solution for intent classification in Spanish-speaking contexts. This not only improves the efficiency of RAG pipelines but also sets the stage for more advanced conversational AI systems in the future.</p><h3>The model</h3><p>To address the challenge of efficient and accurate user intent classification, I leveraged a combination of three powerful machine learning algorithms: support vector machine (SVM), logistic regression, and k-nearest neighbors (k-NN). These algorithms are particularly well-suited for working with high-dimensional text embeddings due to their distinct strengths and complementary characteristics.</p><p><strong>Support vector machine (SVM):</strong> SVM is renowned for its effectiveness in high-dimensional spaces and its robustness in scenarios with limited data. Given the synthetic nature of the dataset generated using a propietary LLM, which is not excessively large, the SVM’s ability to perform well with fewer data points was crucial. Its kernel trick allows it to operate effectively in high-dimensional spaces, making it an excellent candidate for handling the text embeddings produced by the<strong> </strong><a href="https://huggingface.co/jinaai/jina-embeddings-v2-base-es"><strong>jinaai/jina-embeddings-v2-base-es</strong></a> model. Moreover, SVMs are known for their generalization capabilities, which helped achieve good results despite the dataset’s size.</p><p><strong>Logistic regression:</strong> Logistic regression is a fundamental and interpretable algorithm that excels in binary and multiclass classification tasks. Its linear decision boundary works well with text embeddings, capturing the underlying patterns in the data. Logistic regression’s simplicity and efficiency make it a reliable choice for quick predictions, especially when computational resources are a concern. By calibrating the probabilities, I ensured that the model’s output probabilities were well-calibrated, enhancing its reliability in real-world applications.</p><p><strong>k-Nearest neighbors (k-NN):</strong> k-NN is a non-parametric algorithm that makes predictions based on the closest training examples in the feature space. Its intuitive approach of looking at the nearest neighbors makes it robust to outliers and noise, which can be beneficial when dealing with synthetic data. k-NN’s ability to capture local patterns in the data complements the global perspective offered by SVM and logistic regression. By calibrating its probabilities using isotonic regression, I further improved its probabilistic predictions.</p><p><strong>Ensemble model:</strong> Combining these three classifiers into an ensemble model using a soft voting mechanism significantly boosted the overall performance. Each algorithm contributed its strengths: SVM’s robustness with limited data, logistic regression’s efficiency and interpretability, and k-NN’s local pattern recognition. The ensemble model aggregated the calibrated probabilities from each classifier, leading to a more accurate and reliable final prediction. This approach reduced the variance and improved the generalization of the model, resulting in superior performance compared to any single classifier.</p><p>The synthetic training/testing dataset was generated with a propietary LLM providing a diverse range of examples, allowing the ensemble model to learn and generalize well. By sharing the training dataset and pre-trained model, I aim to enable others to explore and benefit from this efficient approach to user intent classification.</p><h3>Dataset</h3><p>To give you a better understanding of the dataset used for training the model, I will include an image below that illustrates the size and composition of the dataset. This visual representation will provide insight into the distribution of different intents and the overall structure of the data, highlighting how the synthetic examples generated with the LLM played a crucial role in building an effective classification system.<br>It’s important to note that the dataset has an intentional class imbalance.</p><p>Some intents have fewer linguistic expressions to convey their meaning, necessitating fewer samples. For example, the “censorship” intent requires a larger and more diverse set of samples covering various questionable topics to ensure comprehensive detection. This imbalance was carefully designed to reflect the real-world scenario where certain intents are more complex and varied than others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*8zLP4dPYvbIkTWSatvoMVg.png" /></figure><h3>Classifier performance</h3><p>The performance of the ensemble classifier, combining logistic regression, SVM, and k-NN, demonstrates remarkable accuracy across various intents.</p><p>Below, you will find the classification report and confusion matrix which detail the model’s precision, recall, and f1-score for each intent. This comprehensive evaluation highlights the model’s capability to accurately classify a diverse set of user intents, even with the intentional class imbalance designed to reflect real-world scenarios.</p><p>Below is the detailed classification report and confusion matrix for further interpretation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*Wt8cwTEzSiC2usHVrb8r6A.png" /><figcaption>Img by the author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*DF_meP_CWzF6qW4SjNGbqA.png" /><figcaption>Img by the author</figcaption></figure><p>The class &quot;Desconocido&quot; was programmatically injected for all test samples that got less than a .5 prob score on the classification process.</p><blockquote>Some manual shots generated for testing by me</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*z4Z2dVC6OnZ-7w08POYm8w.png" /><figcaption>Img by Google</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*kgCQt-hzwJDroJdZFnSItw.png" /><figcaption>Img by the author</figcaption></figure><h3>Quality of the embeddings</h3><p>To evaluate the quality of the embeddings used in my model, I employed UMAP (Uniform Manifold Approximation and Projection) to reduce the high-dimensional embeddings to a 2D space. This visualization technique allows us to see how well the embeddings capture the underlying structure of the data.</p><p>The UMAP visualization below shows that the embeddings separate the different classes quite distinctly, indicating that the <a href="https://huggingface.co/jinaai/jina-embeddings-v2-base-es"><strong>jinaai/jina-embeddings-v2-base-es</strong></a> model is highly effective in capturing semantic nuances. This clear separation of classes in a reduced dimensional space suggests that the embeddings are of high quality, significantly contributing to the overall performance of the classifier.</p><p>The distinct clusters corresponding to each intent class affirm the embeddings’ robustness, providing a solid foundation for the SVM, logistic regression, and k-NN classifiers to perform their tasks effectively. The embeddings’ ability to delineate classes with minimal overlap is crucial for the success of the ensemble model, leading to the impressive accuracy and reliability observed in our results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*8Se2ltjBougXlroW2mINNw.png" /><figcaption>Img by the author</figcaption></figure><h3>Conclusions</h3><p>While I have achieved a model that classifies user intents with high accuracy, there is always room for improvement, particularly in the training dataset. Enhancing the dataset to better generalize could further improve the model’s performance, making it more robust to a wider range of linguistic variations. Although extensive quality tests were not conducted, it is possible that certain areas of the language are not fully generalized between the embeddings and the training samples.</p><p>Despite these limitations, I have laid out a strategy and model architecture that is both cost-effective and holds significant potential. By leveraging an ensemble of logistic regression, SVM, and k-NN, calibrated for probability accuracy, this system runs significantly faster and is more resource-efficient compared to large language models, event with your <strong>notebook CPU.</strong></p><p>This approach demonstrates that high-performing intent classification can be achieved without the heavy computational burden of LLMs or other expensive Transformers. The solution provides a foundation for further enhancements, such as supporting multi-turn conversations and integrating more comprehensive language data.</p><h3>Repositories and final words</h3><p>I encourage you to clone the repository, test the model, and join me in improving the dataset. Together, we can refine and enhance this model, making it even more robust and generalizable. Let’s collaborate to push the boundaries of what’s possible in user intent classification and create a more powerful, efficient system.</p><p>Feel free to reach out with your feedback, suggestions, or contributions. Let’s work together to make this model the best it can be. Thank you, and happy coding!</p><p>Git link:<strong> </strong><a href="https://github.com/puppetm4st3r/intentclassification"><strong>https://github.com/puppetm4st3r/intentclassification</strong></a></p><p>HF link: <a href="https://huggingface.co/prudant/es_intent_classification"><strong>https://huggingface.co/prudant/es_intent_classification</strong></a></p><p>From Latam with ❤️</p><h3>Cubed</h3><p><em>Thank you for being a part of the community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a> | <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://venturemagazine.net/"><strong>Venture</strong></a> | <a href="https://cubed.run/"><strong>Cubed</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d83ef2251f1f" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/turbocharge-your-user-intent-classification-nlu-pipeline-with-efficient-machine-learning-and-fast-d83ef2251f1f">Turbocharge your user intent classification NLU pipeline with efficient machine learning and fast…</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering LLaMA Index: Beyond the documentation to practical insigths application]]></title>
            <link>https://blog.cubed.run/mastering-llama-index-rag-system-optimization-secrets-e98634c40a3c?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/e98634c40a3c</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[llamaindex]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Mon, 08 Apr 2024 21:26:11 GMT</pubDate>
            <atom:updated>2024-04-09T22:12:05.651Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cNq8Rt7-YIF0kgKIADM-6w.png" /><figcaption>Image from the author</figcaption></figure><h3>Mastering LLaMA Index: Beyond the documentation to practical insigths application (Part 1)</h3><h3>Introduction</h3><p>LLaMA Index plays a crucial role in enhancing Retrieval Augmented Generation (RAG) systems, equipped with a suite of features aimed at optimizing data retrieval. While its comprehensive documentation serves as a valuable starting point, the real mastery of LLaMA Index is achieved through practical application and tailored customization to meet the diverse demands of RAG systems.</p><p>In this article, we delve into the strategic application of LLaMA Index, focusing on conceptual understanding and practical insights. We will explore how to navigate the complexities of LLaMA Index, offering guidance on adapting its functionalities on RAG scenarios.</p><p>Through discussing real-world examples and sharing experiences, this post aims to equip readers with the knowledge to effectively leverage LLaMA Index in their projects, enhancing retrieval processes and efficiency.</p><p>If you want to go directly to more advanced advices and insights go to <strong>Practical Insights </strong>section, next sections is for mostly beginers on LLaMA Index framework.</p><h3>[TL;DR] Decoding LLaMA index: key components and their interplay</h3><h4>Large language model</h4><p>Let&#39;s talk about the LLM, in the quest for the optimal Large Language Model (LLM) to integrate into our RAG system, the journey was extensive and insightful. With a setup powered by three RTX 3090 GPUs, I embarked on testing a spectrum of open-source models, ranging from 7 billion to 70 billion parameters, including various moe’s and merged configurations.</p><p>Each model offered unique strengths and areas for improvement. While open-source models have their appeal, particularly in terms of accessibility and community support, they often fell short in key aspects necessary for a productive RAG system. Critical accuracy on functionalities like user intent detection, filtering inappropriate queries, and precise function calling were areas where these models struggled or do not meet the speed throughput that a production system requiered in a balance between cost and benefit.</p><p>After months of rigorous testing, the choice became clear: GPT-3.5 Turbo from OpenAI stood out for its robust performance and reliability. Despite the allure of even more advanced models like GPT-4 Turbo and Claude 3 Opus, the cost-to-benefit ratio made GPT-3.5 Turbo the most viable option for our use case, striking a balance between capability, cost, and practical utility in a production RAG environment.</p><h4>The embeddings model</h4><p>A few months ago the landscape of embedding models had a significant limitation: the available open-source options were confined to processing contexts of merely 512 or 1024 tokens. This constraint imposed a major challenge for text chunking strategies, often resulting in a problematic lack of context. Although there were methods to mitigate this issue, none struck me as particularly robust.</p><p>As time progressed, the landscape evolved with the advent of open-source models capable of handling 4k context windows, aligning closer to proprietary counterparts like those from OpenAI or Cohere. However, these models were primarily tailored for English, posing a barrier for applications in my native Spanish and its prevalent use in my environment.</p><p>Today, the breakthrough came with the introduction of more sophisticated multi-language models (including english), featuring crosslingual capabilities and extending context windows up to 8k tokens. This development shattered the previous 512-token limitation and the only english high accuray models, heralding a new era for productive RAG solutions. Now, with these advanced models, the issue of limited context windows and the resultant context loss has been significantly alleviated, paving the way for more effective and comprehensive RAG applications.</p><h4>The rerank process and model</h4><p>The reranking postprocess emerges as a pivotal component, especially for refining the outcomes derived from naive retrieval based on cosine similarity between embeddings. Initially, reranking faced similar multilingual and context length challenges as embedding models, with limitations in handling diverse languages and extended context sizes.</p><p>However, today we have models capable of reranking with small queries against texts up to 8k tokens in context length. The impact of incorporating a rerank model in a RAG system is substantial, leading to a marked improvement in accuracy compared to other common strategies. While methods like index selection agents, nested index strategies, trees, and knowledge graphs have their merits, they often require more complex implementation and might not always yield proportional benefits.</p><p>Reranking, on the other hand, addresses at least 80% of the problem with relative simplicity and ease of implementation. This straightforward approach not only enhances the effectiveness of the retrieval process but also streamlines the workflow, making it a preferred choice for optimizing RAG systems.</p><h4>The vector database</h4><p>Turning our focus to the vector database component, my approach has been guided by the principle of simplicity, which has yielded impressive results. Among the various options like document-based indices, knowledge graphs, and others, I’ve leaned towards vector indices for their straightforwardness and effectiveness. My tool of choice has been pgvector, an extension for PostgreSQL that integrates vector repository capabilities seamlessly into the database. While there are undoubtedly more complex solutions available, the integration of pgvector with PostgreSQL has provided a robust and efficient platform for managing vector indices, aligning perfectly with the enhanced capabilities of modern embedding and reranking models we’ve discussed earlier. This choice underscores my belief in the power of simplicity and efficiency in building high-performance RAG systems.</p><blockquote>Now we delve into the heart of the matter, where the practical magic of RAG systems comes to life</blockquote><h3>Practical insights</h3><h4>The text chunking and the knowledge pipeline</h4><p>This is not just about slicing and dicing text; it’s an art form that significantly influences the effectiveness of our retrieval process. In the following sub-sections, we’ll unpack the strategies and insights that have transformed text chunking from a mere technical necessity into a powerful tool for enhancing RAG system accuracy and performance.</p><p><strong>Text chunking</strong> and <strong>&quot;the knowledge pipeline&quot;</strong> represents a crucial phase in refining the RAG process, particularly with the advancements in LLaMA Index and the expansion of context window capabilities. The transition from the restrictive 512-token windows to the expansive 8k and 16k tokens offered by modern models has significantly reduced the complexity of fine-tuning chunk sizes for indexing, mitigating the ‘forget in the middle’ phenomenon in LLMs while avoiding the excessive use of their maximum context capacity.</p><ol><li><strong>Selective text and metadata: </strong>Ensuring that text chunks exclude unwanted metadata significantly impacts the quality of the embeddings. In my experience, omitting the default metadata entirely has enhanced the output, this must be done at text node level object in LLaMA Index with the <em>excluded_embed_metadata_keys</em> and <em>excluded_llm_metadata_keys</em> properties.</li><li><strong>Metadata replacement + node sentence window: </strong>This technique has been successful in generating node texts of 1024 tokens, which can then be expanded to 4096 tokens through Metadata Replacement. It balances detailed content extraction with manageable chunk sizes. This technique uses the &#39;window&#39; metadata key, so you have to considerate it in the previous advice (1). More on <a href="https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo/">https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo/</a></li><li><strong>Post-chunking reranking: </strong>With the advent of advanced rerank models, it’s feasible to apply reranking after applying the Metadata Replacement + Node Sentence Window strategy (2), further refining the retrieval quality. I recommend the <a href="https://huggingface.co/jinaai">JINA AI</a> embeddings models, the new family has an 8k token context length.</li><li><strong>Cut-off threshold implementation: </strong>Setting a threshold (typically around 50 points or 0.5) to filter out reranked nodes ensures that only the most relevant chunks are forwarded to the LLM. This strategy is pivotal in maintaining the quality of the input, as it prevents the dilution of the content with non-essential or low-relevance chunks. In my approach, I’ve found it more beneficial to inform the User that no relevant information was found rather than forwarding subpar reranked content. This prevents the LLM from ‘hallucinating’ or generating responses based on poor-quality input, maintaining the integrity and reliability of the RAG system’s output.</li><li><strong>Prompt engineering: </strong>The art of crafting clear, concise, and non-redundant prompts is vital for steering the LLM towards generating relevant and accurate responses. In my experience, prompt engineering goes beyond mere text crafting; it’s about structuring the prompt to guide the model effectively. Using delimiters like ‘ — — — — — — — — — — ‘ helps in segregating different sections such as context information, user queries, and response instructions, which significantly enhances LLM performance. Investing time in prompt engineering has been crucial, as well-crafted prompts ensure the model understands the task at hand, leading to better and more precise outputs. Clear instructions, devoid of repetition and redundancy, optimize the model’s response, making prompt engineering a critical element in the success of RAG systems.</li><li><strong>ChatTemplate sensitivity: </strong>In working with open-source LLM models, the structure of ChatTemplates is of paramount importance. The sensitivity of these models to the ChatTemplate design cannot be overstated; even an extra newline or unnecessary spaces can drastically alter the outcome. Through rigorous experimentation, I’ve learned that the model’s performance is significantly influenced by whether it was trained for multi-turn conversations or single-turn instructions. Properly structured ChatTemplates, mindful of the model’s training and capabilities, can dramatically improve the generation process. Ensuring the ChatTemplate aligns with the model’s training background is essential for eliciting the desired response, highlighting the critical role of ChatTemplate design in optimizing LLM interactions.</li><li><strong>Leveraging function calling in RAG systems:</strong> Incorporating function calling into Retrieval Augmented Generation systems is a strategy that should not be overlooked, crucial for ensuring deterministic responses that are vital for actions or decision-making processes. While user intent detection serves as a prominent example, illustrating how function calling can direct a RAG system to differentiate between commercial inquiries and technical support, its utility extends far beyond. This feature empowers the LLM to make informed decisions, structuring responses in specific formats like JSON, and navigating through a wide array of operational pathways. Whether it’s categorizing queries, selecting appropriate data indices, or executing complex decision trees, function calling enriches the RAG’s capability to deliver nuanced and contextually relevant responses. Its application is manifold, ranging from content personalization to operational automation, proving to be an essential component in crafting a sophisticated, user-centric RAG system.</li><li><strong>Users Can’t Explain What They Want:</strong> Facing the challenge of vague user queries in RAG systems, I implement a strategy combining <a href="https://en.wikipedia.org/wiki/Query_expansion">query expansion</a> and fallback reranking. By function calling, I refine the user’s initial query, creating variations to broaden the context. I then retrieve relevant text nodes for each query, removing duplicates to ensure uniqueness. In reranking process, these text nodes are evaluated against the original query using a cut-off threshold; if no node meets this, I proceed to rerank against the expanded queries in an iterative loop until finding sufficiently relevant nodes. These selected nodes are then used to inform the LLM in crafting a response aligned with the original query, or if no appropriate nodes are found, I conclude that no relevant information is available.</li></ol><h4>The bigger lessons from my Open-Source journey</h4><p>In the early stages of my journey, I relied on the Hugging Face Transformers library, which performed admirably, especially with quantized models. However, had I found an open-source model that met my requirements, I would have faced a challenge, as much of the ecosystem, including LLaMA Index, is designed for seamless integration with the OpenAI protocol specification. This is why inference servers built for scale have adopted APIs compliant with the OpenAI standard. From this experience, I learned it’s crucial to start with an inference server like <a href="https://github.com/huggingface/text-generation-inference">Hugging Face’s TGI </a>or the <a href="https://github.com/PygmalionAI/aphrodite-engine">Aphrodite Engine</a>. These are highly efficient, maximize GPU performance, and align with industry standards. If you ever need to switch to a model like GPT-3.5 Turbo, the transition is smooth. This applies equally to text embeddings and reranking models. Therefore, for those considering open-source solutions, I recommend using a server that supports a standard protocol in the industry, ensuring compatibility and future-proofing your setup.</p><h3>Final words</h3><p>Throughout this enriching journey, the most instrumental factor in implementing the various strategies and improvements discussed has been leveraging the abstractions provided by LLaMA Index. The true value of LLaMA Index lies in the robust and usable core its developers have meticulously crafted. While utilizing these abstractions might require a moderate to advanced level of programming, it doesn’t preclude beginners from engaging with the more straightforward classes derived from these abstractions, which are already quite powerful on their own. These base classes offer a solid starting point, though they may lack the finer customization options available through direct interaction with the core abstractions.</p><p>This journey through LLaMA Index is a testament to the developers’ effort in creating a tool that not only serves the needs of seasoned programmers but also remains accessible to those who are just beginning to flex their coding muscles. I encourage everyone, regardless of their programming proficiency, to dive into the world of LLaMA Index. It’s a challenge worth embracing, offering a rewarding experience that combines learning, growth, and the opportunity to contribute to and benefit from this evolving technological marvel.</p><p>Before we part ways, don’t forget to <strong>follow me</strong> for more insights and updates on this fascinating NLP journey. If you found value in this article, please show your support with a <strong>clap</strong>. Until next time, keep exploring and pushing the boundaries of what’s possible!</p><p>DM</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e98634c40a3c" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/mastering-llama-index-rag-system-optimization-secrets-e98634c40a3c">Mastering LLaMA Index: Beyond the documentation to practical insigths application</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enabling Function Calling with gorilla-llm/gorilla-openfunctions-v2 Using the OpenAI Protocol]]></title>
            <link>https://blog.cubed.run/enabling-function-calling-with-gorilla-llm-gorilla-openfunctions-v2-using-the-openai-protocol-355492d0587d?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/355492d0587d</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[functioncall]]></category>
            <category><![CDATA[local]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 03:50:29 GMT</pubDate>
            <atom:updated>2024-06-12T10:42:12.125Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBjuVP2Jr6Z8fZgenGHo6A.png" /><figcaption>Image from the author</figcaption></figure><h3>Enabling local function calling with gorilla-openfunctions-v2 7B LLM using the openAI protocol</h3><h3>Introduction</h3><p>In the rapidly evolving landscape of natural language processing and conversational AI, the ability to perform function calling within language models has emerged as a powerful tool for extending their capabilities. OpenAI’s API has been at the forefront of this development, providing a standardized protocol for function calling. However, what if you want to leverage the power of function calling with a local language model like <a href="https://huggingface.co/gorilla-llm/gorilla-openfunctions-v2">gorilla-llm/gorilla-openfunctions-v2</a> ?In this article, we’ll explore a Python implementation that bridges the gap, allowing you to use gorilla-llm/gorilla-openfunctions-v2 with the OpenAI function calling protocol.</p><h3>The challenge</h3><p>While OpenAI’s API offers seamless function calling capabilities, integrating this functionality with a local language model like <a href="https://huggingface.co/gorilla-llm/gorilla-openfunctions-v2">gorilla-llm/gorilla-openfunctions-v2</a> presents a challenge. The local model may not natively support the OpenAI protocol, making it difficult to leverage the benefits of a good system/code maintainability by not following a standard adopted by business solutions and the community. This is where our Python implementation comes into play.</p><h3>The solution</h3><p>To address this challenge, I have developed a Python wrapper that acts as an intermediary between the OpenAI API client and the <strong>gorilla-llm/gorilla-openfunctions-v2</strong> local language model. By serving <strong>gorilla-llm/gorilla-openfunctions-v2</strong> with an engine that mimics the OpenAI chat completion protocol (such as TGI Hugging Face, Vllm, Aphrodite engine, or others), we can intercept the input and output of the local model and manipulate the data to append/re-format the engine response with <strong><em>our magic sauce of the function calling inference from the great Gorilla Open Functions V2.</em></strong></p><blockquote>When a user interacts with the wrapped client, the wrapper first processes the user’s input and injects the necessary function specifications and a special model&#39;s tags. This modified input is then passed to the local engine for processing. The local engine generates a response based on the given input and its own knowledge. Upon receiving the generated response, the wrapper analyzes it, looking for any function calls that match the provided function specifications. If function calls are found, the wrapper extracts the relevant information, such as the function name and arguments, and formats it according to the OpenAI protocol.</blockquote><p>Finally, the wrapper returns the adapted response, which includes the original response content along with any identified function calls and their corresponding arguments. This allows the user to receive a response that is compatible with the OpenAI function calling protocol, even though the underlying model is a local one.</p><h3>GitHub repository</h3><p>To dive deeper into the implementation and explore the code, visit our GitHub repository: <a href="https://github.com/puppetm4st3r/local_function_calling">Local gorilla-openfunctions-v2 with OpenAI Function Calling Protocol</a>. The repository contains the Python implementation, there is a simple example because the solution is really simple.</p><p>While the current implementation provides a simple and efficient solution, there is room for improvement in the parser of the LLM output. The existing structures of the output of the engine used (TGI, Vllm, Aphrodite, or others) could be reused since they already mimics the OpenAI protocol, instead of creating the response object completely from scratch. This is an area where contributions from the community are welcome.</p><h3>Conclusion</h3><p>By leveraging the power of this Python simple implementation, developers can now seamlessly integrate function calling capabilities into the <a href="https://huggingface.co/gorilla-llm/gorilla-openfunctions-v2">gorilla-llm/gorilla-openfunctions-v2</a> local language model, opening up new possibilities for extending its functionality. We invite you to explore the repository, experiment with the code, and contribute to its development.</p><p>Stay tuned for more exciting developments in the world of natural language processing and conversational AI! Don&#39;t forget to follow and your clap in the little hand =)</p><p>Best regards,</p><p>DM</p><p><em>Thinking of starting a blog? Try </em><a href="https://differ.blog/"><strong><em>Differ</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=355492d0587d" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/enabling-function-calling-with-gorilla-llm-gorilla-openfunctions-v2-using-the-openai-protocol-355492d0587d">Enabling Function Calling with gorilla-llm/gorilla-openfunctions-v2 Using the OpenAI Protocol</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing multilingual Embeddings and Re-ranking to your local/dev environment with BAAI bge-m3 —…]]></title>
            <link>https://blog.cubed.run/bringing-multilingual-embeddings-and-re-ranking-to-your-local-dev-environment-with-baai-bge-m3-4ef464041a94?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ef464041a94</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[reranking]]></category>
            <category><![CDATA[aiba]]></category>
            <category><![CDATA[servers]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Thu, 22 Feb 2024 19:54:39 GMT</pubDate>
            <atom:updated>2024-07-11T00:59:04.942Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTD0nCMnu2B6Ep-aowP6Wg.png" /><figcaption>Image from the author</figcaption></figure><h3>Bringing multilingual Embeddings and Re-ranking to your local/dev environment with BAAI bge-m3 — 8k context length model</h3><h3>Introduction</h3><p>In the vast expanse of natural language processing (NLP), finding a robust, multilingual model that can effectively handle both embeddings and re-ranking can be akin to searching for a needle in a haystack. Today, I’m thrilled to share a solution that not only addresses this challenge but also offers a swift, easy-to-deploy server for FAST local development and testing purposes.</p><h3>The problem</h3><p>The demand for multilingual NLP solutions has skyrocketed. Yet, the availability of models that can accurately understand and process multiple languages, especially for embeddings and re-ranking tasks in the Open Source world, remains very limited. This gap significantly hinders the development of global, inclusive applications. After testing numerous models, the scarcity of efficient, multilingual options became evident, propelling me to search for a better solution.</p><h3>The solution</h3><p>Enter the <a href="https://huggingface.co/BAAI/bge-m3">BAAI’s M3</a> — 8k context length model, a beacon of hope in the multilingual NLP landscape. The model’s proficiency in understanding and processing a wide array of languages is unparalleled. Recognizing its potential, I’ve developed a simple, fast server using FastAPI to serve embeddings and re-ranking services powered by this model. This server is designed for local development and testing, offering a straightforward way to integrate advanced NLP capabilities into your projects.</p><h3>How it works</h3><p>The server leverages FastAPI to expose endpoints for generating embeddings and re-ranking, making it incredibly user-friendly for developers. With asynchronous processing and batch handling, it’s built to manage multiple requests efficiently, ensuring that even in a local development environment, you experience optimal performance.</p><p>Despite its simplicity, this server is packed with powerful features designed to maximize efficiency and performance for local development and testing. Here are a few highlights:</p><ul><li><strong>Asynchronous Processing</strong>: Utilizing FastAPI’s asynchronous capabilities, the server efficiently handles multiple requests simultaneously, ensuring rapid response times without blocking operations.</li><li><strong>Dynamic Timeout Handling</strong>: To prevent long waits and potential server hang-ups, a dynamic timeout mechanism is in place. This ensures that if a request cannot be processed within a specified timeframe, it’s gracefully handled, and an appropriate response is returned to the client.</li><li><strong>Intelligent Batching</strong>: The server smartly accumulates incoming requests into batches, optimizing the utilization of the M3 model for embeddings and re-ranking tasks. This batching mechanism significantly improves throughput, especially when dealing with high volumes of requests.</li><li><strong>Flush Mechanism for Batches</strong>: A smart flush mechanism ensures that batches are not only processed at regular intervals but also when they reach a certain size. This dual-condition approach guarantees that the server remains responsive and that requests are processed in a timely manner, enhancing the overall efficiency of the model’s utilization.</li><li><strong>GPU Utilization and Semaphore Lock</strong>: A semaphore lock ensures that GPU access in a secure manner, preventing concurrent access issues.</li></ul><p>These features, combined with the multilingual capabilities of the model, make this server a robust solution for developers looking to integrate advanced NLP functionalities into their applications with ease and a few lines of code.</p><h3>Repository and final words</h3><p>All the code, along with detailed instructions for setting up and running the server, is available in this <a href="https://github.com/puppetm4st3r/baai_m3_simple_server">GitHub repository</a>. Whether you’re a seasoned developer or just starting out, you’ll find the repository equipped with everything you need to get started.</p><p>Exploring the capabilities of the <a href="https://huggingface.co/BAAI/bge-m3">BAAI’s bge-m3</a> model has been an enlightening journey, and I’m excited for you to embark on this adventure. The server is a testament to the model’s versatility and effectiveness across languages, and I believe it will be a valuable tool in your NLP toolkit.</p><p>In plain English, if you find this server as useful as I have, don’t forget to give some claps and follow me for more insights into the world of artificial intelligence and machine learning. Your support motivates me to continue exploring and sharing solutions that can make our digital world more inclusive and intelligent.</p><p>Happy coding, and let’s break the language barriers together!</p><p>DM</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ef464041a94" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/bringing-multilingual-embeddings-and-re-ranking-to-your-local-dev-environment-with-baai-bge-m3-4ef464041a94">Bringing multilingual Embeddings and Re-ranking to your local/dev environment with BAAI bge-m3 —…</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High quality very long text summarization with powerful 7B LLM’s on high grade consumer GPU]]></title>
            <link>https://blog.cubed.run/introducing-efficient-and-simplified-long-text-summarization-with-powerful-7b-llms-and-high-grade-d56343bb2c5d?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/d56343bb2c5d</guid>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[long-texts]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Sun, 10 Dec 2023 19:01:50 GMT</pubDate>
            <atom:updated>2024-06-12T10:25:13.786Z</atom:updated>
            <content:encoded><![CDATA[<h3>High quality very long text summarization with powerful 7B LLM&#39;s on high grade consumer GPU</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9zPAHy0eYgLQ2m0LNUv3pQ.png" /><figcaption>Image by the author</figcaption></figure><h3>Introduction</h3><p>In the fast-evolving landscape of natural language processing, the ability to efficiently summarize extensive texts has become increasingly vital. Today, I am excited to share a breakthrough approach that simplifies this complex task, leveraging the power of Large Language Models (LLMs) in a unique and efficient manner.</p><p>As a data scientist working in the realm of AI and biomathematical models, I constantly seek ways to harness cutting-edge technology for practical and impactful solutions. This journey led me to explore the capabilities of a 7-billion parameter LLM, specifically optimized for local GPU deployment with AWQ 4-bit quantization, making it not only powerful but also accessible for a wide range of applications.</p><p>In this post, I will unveil an straightforward method to perform extensive text summarization in English (but also supported for multilenguaje), highlighting its simplicity and effectiveness. This method is not just a testament to the advancements in machine learning models but also to the art of prompt engineering and resource-efficient computing.</p><p>So, let’s dive into the world of AI-driven text summarization, where complexity meets simplicity, and efficiency becomes the key.</p><h3>Hands-On with simplified but powerful text summarization using LLMs</h3><h4>The core concepts</h4><p>In the realm of text summarization, the main challenge lies in processing large volumes of data without sacrificing the quality and contextual relevance of the output. The solution I propose hinges on the utilization of a Large Language Model (LLM) — specifically, a 7-billion parameter model known as <a href="https://huggingface.co/TheBloke/OpenHermes-2.5-neural-chat-7B-v3-2-7B-AWQ"><strong>OpenHermes-2.5-neural-chat-7B-v3–2–7B</strong></a> (quant AWQ version by <a href="https://huggingface.co/TheBloke"><strong>TheBloke</strong></a>). This model stands out due to its impressive performance on various language generation benchmarks, even surpassing models with up to 70 and 30 billion parameters.</p><h4>The approach</h4><p>The basics of this method is its simplicity and efficiency. By leveraging the model’s capabilities, combined with intelligent text splitting and recursive summarization, we can handle extensive documents with remarkable ease. Here’s how it unfolds:</p><ol><li><strong>Model and tokenizer initialization</strong>: We begin by initializing the model and tokenizer, ensuring they are optimized for local GPU execution. This step is crucial for handling the extensive computational requirements of LLMs.</li><li><strong>Recursive summarization function</strong>: At the heart of this process is a custom function, recursive_summarization, designed to split the document into manageable chunks. Each chunk is then summarized independently. If the document is split into multiple parts, these summaries are combined and summarized again, recursively, ensuring a comprehensive and coherent final summary.</li><li><strong>Prompt engineering:</strong> A key aspect of this method is the carefully crafted prompt that guides the model to produce summaries in English (and spanish). This prompt engineering is essential for ensuring the summaries are not only accurate but also linguistically sound.</li><li><strong>Parameters for text generation</strong>: Fine-tuning the text generation process involves adjusting parameters like sampling, temperature, top_p, and repetition penalty. These parameters play a pivotal role in dictating the style and diversity of the generated text, allowing us to align the output closely with the desired context and quality.</li><li><strong>Document processing</strong>: The final step involves processing the documents using SimpleDirectoryReader and applying the summarization function. This systematic approach ensures each document is treated with the same level of precision and care.</li></ol><p>The model’s 4-bit quantization format is a game-changer, making it viable for use on local GPUs (with 24GB VRAM).</p><p><strong><em>Although the code focuses mainly on the summary in English, I also tested it on Spanish texts with a Spanish prompt engineer with results of the same quality, this shows the versatility of LLMs in handling different languages with a high command.</em></strong></p><h3>Let&#39;s code 🚀</h3><h4>Environment setup</h4><p>Before we embark on the text summarization part, it’s essential to set up our environment. This involves importing the necessary libraries and ensuring they are installed in your Python environment.</p><p>Here’s what you need to install (if you don´t already installed ):</p><pre>pip install transformers tqdm llama_index<br><br>git clone https://github.com/casper-hansen/AutoAWQ<br>cd AutoAWQ<br>pip install -e .</pre><p>Now we start by importing the required libraries from transformers, tqdm, and llama_index. These libraries provide us with the tools for model handling, progress tracking, and text processing.</p><pre>from transformers import AutoTokenizer, AutoModelForCausalLM, AwqConfig<br>from tqdm.notebook import tqdm<br>from llama_index.text_splitter import TokenTextSplitter<br>from llama_index import Document<br>from llama_index.schema import Node<br>from llama_index import SimpleDirectoryReader</pre><p>We begin by loading up our model and tokenizer. This step is crucial for preparing our model to process and generate text.</p><pre># Initialize the model and tokenizer from huggingface repo<br>model_name_or_path = &quot;TheBloke/OpenHermes-2.5-neural-chat-7B-v3-2-7B-AWQ&quot;<br>q_conf = AwqConfig(fuse_max_seq_len=1024, do_fuse=True)<br>model = AutoModelForCausalLM.from_pretrained(model_name_or_path, config=q_conf, trust_remote_code=True, device_map=&#39;cuda:0&#39;)<br>tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True, use_fast=True)</pre><p>The model is loaded in AWQ quant 4-bits format, more information at <a href="https://github.com/mit-han-lab/llm-awq">https://github.com/mit-han-lab/llm-awq</a></p><p>Now, this is where the magic happens with our custom function recursive_summarization, it will takes a document and breaks it down into smaller chunks. Each chunk is then summarized, and if necessary, the process is repeated for a comprehensive summary.</p><pre>def recursive_summarization(document, tokenizer, model, generate_kwargs, chunk_size, summary_kind):<br>    &quot;&quot;&quot;<br>    Recursively summarize a text using an LLM and NodeParser with TokenTextSplitter.<br>    <br>    :param document: Document object with text to summarize.<br>    :param tokenizer: Tokenizer of the LLM model.<br>    :param model: LLM model.<br>    :param generate_kwargs: Arguments for the model&#39;s text generation.<br>    :param chunk_size: Maximum token size per chunk.<br>    :return: Final summary of the document.<br>    &quot;&quot;&quot;<br>    def recursive_summarization_helper(nodes):     <br>        summarized_texts = []<br>        for node in tqdm(nodes):<br>            prompt = f&quot;&quot;&quot;### System:<br>You are an expert agent in information extraction and summarization.<br>### User:<br>Read the following context document:<br>---------------<br>{node.get_content()}<br>---------------<br><br>Your tasks are as follows:<br>1.- Write an extensive, fluid, and continuous paragraph summarizing the most important aspects of the information you have read.<br>2.- You can only synthesize your response using exclusively the information from the context document.<br>### Assistant:<br>According to the context information, the summary in English is: &quot;&quot;&quot;<br>            <br>            inputs = tokenizer(prompt, return_tensors=&#39;pt&#39;).to(&quot;cuda&quot;)<br>            tokens = model.generate(**inputs, **generate_kwargs)<br><br>            # Decode the response<br>            completion_tokens = tokens[0][inputs[&quot;input_ids&quot;].size(1):]<br>            response_text = tokenizer.decode(completion_tokens, skip_special_tokens=True)<br>            summarized_texts.append(response_text)<br><br>        if len(nodes) == 1:<br>            return summarized_texts[0]<br>        else:<br>            combined_text = &#39; &#39;.join(summarized_texts).strip().strip(&#39;\n&#39;)<br>            new_nodes = node_parser.get_nodes_from_documents([Document(text=combined_text)])<br>            return recursive_summarization_helper(new_nodes)<br><br>    node_parser = TokenTextSplitter(chunk_size=chunk_size, chunk_overlap=0)<br>    initial_nodes = node_parser.get_nodes_from_documents([document])<br>    return recursive_summarization_helper(initial_nodes)</pre><p>This code reflects the innovative blend of advanced AI capabilities and user-friendly design, making the complex task of text summarization surprisingly straightforward. With this function, we’re not just processing text; we’re reshaping it into a concise and meaningful form, tailor-made for efficiency and effectiveness.</p><h3>Results</h3><p>After delving into the technical aspects of our text summarization approach, it’s vital to see it in action. To demonstrate the efficacy of this method, let’s take a classic piece of literature, “The Great Gatsby” by F. Scott Fitzgerald, and run it through our summarization process.</p><h4>The process:</h4><ol><li>Time and Iterations: The entire process took 1 minute and 57 seconds (on a RTX-4090 24gb VRAM), a testament to the efficiency of the model and our method, especially considering the complexity and length of the text.</li><li>Recursions: The text was processed through 4 recursive iterations. This shows the depth of analysis and the ability of our method to distill extensive information into concise summaries.</li></ol><h4>The outcome:</h4><p>Now, let’s examine the output generated by this process.</p><blockquote>The story tells the lives of several characters in New York during the early 20th century, focusing on the narrator, Gatsby and even more importantly Daisy. The environment presented is marked by underground tension, relationships that are difficult to decipher, and I will seriously seek joy. They all participate in exposed social parties where highly charged emotions are revealed along with many secrets kept silent. Gatsby, a mysterious but charming guy, wants to win the heart of Daisy, who for him symbolizes perfection in beauty and happiness. His aspiration to achieve happiness and love leads to creating an opulent life in West Egg, Long Island, and fighting against adversaries such as Tom Buchanan, another suitor of Daisy. As the plot evolves, we show how everyone is involved in distressing and problematic situations, reflecting humanity and ethics in our behavior. The story shows us the central tragicomedy in Gatsby, passionate about Daisy, married to Tom Buchanan; both experiencing tensions and intense feelings, but unable to shape their relationship due to the control exerted by Tom. Their romantic and work experiences generate conflict and drama including betrayals, resignations, very emotional conversations and fatal endings. I also highlight the continuous search for happiness, commitment and personal self-construction, as well as the difficulty of managing the consequences of our decisions and actions. Likewise, it highlights human complexity and interactions between people, as well as the influence of external factors on our life journey.</blockquote><p>This summary demonstrates not just the model’s ability to grasp and condense key themes and narratives, but also its linguistic prowess, particularly in maintaining the essence of the original text in a summarized form.</p><h4>Hardware requirements</h4><p>An important aspect to consider when working with this text summarization method is the hardware requirement. Specifically, the efficiency and speed of this process are significantly enhanced when executed on a GPU with at least 24GB of VRAM. This is due to the large model size and the intensive computational demands of processing extensive text data with a Large Language Model.</p><p>However, it’s worth noting that there are alternatives for those who may not have access to such high-end GPU resources. Backends like llama-ccp enable the execution of this code on a CPU. While this makes the method more accessible, it&#39;s crucial to understand that the trade-off is in processing speed. Running these tasks on a CPU will result in a much slower performance compared to a powerful GPU setup.</p><p>This information is vital for anyone planning to implement this text summarization approach, as it helps in setting realistic expectations regarding the resources required and the consequent processing speed.</p><h4>Exploring free GPU options: Google colab and Kaggle</h4><p>In addition to the hardware requirements, it’s encouraging to know that there are accessible alternatives for those without a high-end GPU access, Platforms like Google Colab and Kaggle offer free GPU resources, albeit with less memory compared to the ideal 24GB VRAM setup. While these GPUs may not match the power of more advanced setups, they still provide a viable option for running our text summarization process.</p><p>To adapt to the memory constraints of these platforms, one can adjust the chunk_size parameter in the summarization function. It&#39;s important to note that changing this parameter will impact the summarization results. A smaller chunk size may require more iterations and contextual and semantic information degradation but can make the process feasible on GPUs with lower memory.</p><p>Kaggle, in particular, offers an interesting opportunity with its provision of free access to multiple GPUs, each with around 15GB of VRAM. This multi-GPU environment can be leveraged to distribute the computational load, thus enhancing the feasibility and efficiency of the process.</p><p>These platforms not only democratize access to powerful computational resources but also open the door for experimentation and innovation in AI and language processing, even for individuals and organizations without access to high-end hardware.</p><h3>Conclusion</h3><p>As we wrap up this exploration into the realm of efficient text summarization using Large Language Models, it’s important to reflect on what we’ve covered. This journey has taken us through the intricacies of a powerful, yet surprisingly simple approach to handling extensive text data. From the initialization of a robust 7-billion parameter model to the intricacies of recursive summarization, we’ve seen how cutting-edge AI can be harnessed in practical, resource-efficient ways.</p><p>The blend of advanced AI capabilities, thoughtful prompt engineering, and efficient use of computational resources underscores a crucial point: sophisticated tasks like text summarization don’t always require complex solutions. Sometimes, simplicity, combined with the right tools and techniques, can lead to equally, if not more, effective outcomes.</p><p>Platforms like Google Colab and Kaggle further democratize this technology, making it accessible even to those without high-end hardware. This opens up a world of possibilities for individuals and organizations across the globe, allowing them to leverage AI for a myriad of applications, regardless of their hardware limitations.</p><p>As we conclude, I hope this post has enlightened you on the potentials and practicalities of AI-driven text summarization. If you found this exploration insightful, I would greatly appreciate your claps and follows. Your support motivates me to continue sharing valuable insights and breakthroughs in the field of AI and data science.</p><p>Thank you for joining me on this journey, and I look forward to bringing more innovative and practical AI solutions to your attention. Keep exploring, keep innovating!</p><p>DM</p><p><em>Are you thinking of starting a blog? Try </em><a href="https://differ.blog/"><strong><em>Differ</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d56343bb2c5d" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/introducing-efficient-and-simplified-long-text-summarization-with-powerful-7b-llms-and-high-grade-d56343bb2c5d">High quality very long text summarization with powerful 7B LLM’s on high grade consumer GPU</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stack Navigator: Expert in programming and IT solutions rated on Stack Overflow]]></title>
            <link>https://blog.cubed.run/stack-navigator-expert-in-programming-and-it-solutions-rated-on-stack-overflow-744546647752?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/744546647752</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[assistant]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[stackoverflow]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Fri, 01 Dec 2023 14:40:59 GMT</pubDate>
            <atom:updated>2024-03-25T08:45:56.851Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*be77VW8eS2a2BC4b5eQicQ.png" /></figure><h3><strong>🚀</strong>Stack Navigator AI Agent</h3><h4>GPT Assistant, expert in programming and IT solutions rated on Stack Overflow</h4><h3>Introduction</h3><p><a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a>, powered by the advanced capabilities of GPT-4, is a versatile AI tool designed to simplify programming and IT queries in any language. Utilizing Stack Overflow’s robust API, it navigates through threads to pinpoint the most pertinent and highly-rated answers. This process, enhanced by GPT-4’s multilingual proficiency, allows <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> to offer reliable, accurate solutions across various languages. It meticulously compiles information, including topic, author, and a summary of top-voted answers, along with code snippets. <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> then synthesizes <strong><em>(in your spoken language) </em></strong>a proposed solution based on all the information obtained from the Stack Overflow API. This makes <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> an invaluable asset for tech professionals worldwide, seeking to boost their efficiency and problem-solving accuracy in their preferred language.</p><h3>Why?</h3><p><a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> stands out for its ability to streamline the search for programming solutions, offering several key benefits:</p><ol><li>Accurate and Valued Responses: By exclusively selecting answered questions from Stack Overflow, it ensures the quality and accuracy of each response.</li><li>Time-Efficient: Reduces the time spent trawling through forums. Users receive concise, top-rated answers quickly, allowing more time for actual coding.</li><li>Customized Solutions: Tailors responses to specific queries, providing summaries and relevant code snippets from the most voted answers.</li><li>Powered by Chat-GPT User-Friendly Interface: Designed for easy use, it allows even those with minimal programming background to find high-quality answers effortlessly.</li><li>Continuous Learning: Regularly updated to include the latest solutions and trends in programming, ensuring users have access to the most current information.</li></ol><p>These features make <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> not just a tool but a vital resource, saving time and enhancing productivity for programmers and IT professionals.</p><h3>Technical Integrity and Privacy: Your Trust, My Priority</h3><p><a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> sets a new standard in technical excellence and privacy. It operates under Stack Exchange’s content licensing, ensuring responsible use of community insights. Importantly, I guarantee that any information sent to Stack Navigator is not used by OpenAI for model improvement purposes. Your data remains strictly confidential, aligning with our steadfast commitment to data security and user trust.</p><h3>Give Stack Navigator a Try: Your Coding Companion Awaits</h3><p>Dive into a world where coding challenges are met with ease and confidence. <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> is more than just a solution — it’s your trusted partner in the coding journey. Friendly, intuitive, and designed with your needs in mind, it’s here to revolutionize how you approach programming. Curious to see the magic? Head over to <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> and experience the joy of hassle-free coding. Join our community of satisfied users and discover why <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> is the go-to choice for programmers worldwide. Your coding breakthrough starts here! 🚀</p><h3>Test</h3><p>In order to not make this post too long, I will attach a few test shots:</p><p>My prompt was:</p><p>I have the next error un my console log: [error] [Firestore in:Get reclassified events] Error: 4 DEADLINE_EXCEEDED please generate an elaborate final with code samples in order to guide me in fixing this bug.</p><p>GPT Response:</p><p>Based on the information from Stack Overflow, here are three relevant discussions about the DEADLINE_EXCEEDED error in Firestore and potential solutions. Note that all content is licensed under <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a>.</p><p><strong>Topic 1: Firestore: Error: 4 DEADLINE_EXCEEDED: Deadline exceeded</strong></p><ul><li>Author of the answer: Saumil Shah</li><li>Summary of the most upvoted answer (19 votes): The issue was resolved by ensuring that the Firebase API call inside an async function was awaited by the caller of that function.</li></ul><p><strong>Topic 2: Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded</strong></p><ul><li>Author of the answer: AndreJacomeSilva</li><li>Summary of the most upvoted answer (1 vote): This error is triggered when a webhook call exceeds the maximum wait time. In DialogFlow, the response must occur within 10 seconds for Actions on Google applications or 5 seconds for all other applications. Exceeding this results in a timeout. There’s an option in Dialogflow to set a pre-defined message for when the fulfillment request exceeds the available time.</li></ul><p><strong>Topic 3: Firebase Cloud Functions: “4 DEADLINE_EXCEEDED: Deadline exceeded”</strong></p><ul><li>Author of the answer: Tarik Huber</li><li>Summary of the most upvoted answer (3 votes): The error can occur due to writing too much data to Firestore at once. It also happens with bad internet connections or bugs in the code. It’s recommended to limit writes to no more than 500 elements or even less, and check for slow connections or potential code bugs.</li></ul><p><strong>Proposed Solution:</strong></p><blockquote>Based on the above discussions, here are a few steps to consider:</blockquote><blockquote>Ensure Proper Async Handling: If you are making asynchronous calls (like Firebase API calls), make sure the calling function properly awaits the async function.</blockquote><blockquote>Check Webhook Time Limits: If using webhooks, ensure your response times are within the limits (10 seconds for Google Actions, 5 seconds for others).</blockquote><blockquote>Limit Data Write Volume: Try to limit Firestore write operations to no more than 500 elements at a time.</blockquote><blockquote>Check Network and Code Issues: Ensure you have a stable internet connection and review your code for potential bugs that could be causing the error.</blockquote><p><strong>Example Code:</strong></p><pre>// Async function handling example<br>async function fetchData() {<br>  // Firestore fetch operation<br>}<br><br>async function callerFunction() {<br>  await fetchData();<br>}<br><br>// Firestore write limit example<br>const MAX_WRITE_LIMIT = 500; // set write limit<br>let dataToWrite = []; // your data array<br><br>if (dataToWrite.length &lt;= MAX_WRITE_LIMIT) {<br>  firestore.collection(&#39;yourCollection&#39;).add(dataToWrite);<br>} else {<br>  console.error(&#39;Exceeding write limit&#39;);<br>}</pre><p><a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator">Give it a try!</a></p><h3>Embracing a New Era in Coding</h3><p>Remember that <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> isn’t just a tool; it’s a step towards a more efficient and empowered coding future. It’s designed with a deep understanding of your programming needs, offering reliable, streamlined solutions to your coding queries.</p><p>Thank you for joining on this exciting journey. I can’t wait for you to experience the difference <a href="https://chat.openai.com/g/g-jdlMEJJf1-stack-navigator"><strong>Stack Navigator</strong></a> makes. Happy coding, and here’s to many breakthroughs and successes in your projects!</p><p>Dont forget to clap and follow!</p><p>Best, DM</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=744546647752" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/stack-navigator-expert-in-programming-and-it-solutions-rated-on-stack-overflow-744546647752">Stack Navigator: Expert in programming and IT solutions rated on Stack Overflow</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dotty: Enhancing .NET Core Development with a Customized GPT Assistant]]></title>
            <link>https://blog.cubed.run/enhancing-net-core-development-with-a-customized-gpt-assistant-3963fa19269e?source=rss-98dbc610ea82------2</link>
            <guid isPermaLink="false">https://medium.com/p/3963fa19269e</guid>
            <category><![CDATA[net-core]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Darío Andrés Muñoz Prudant]]></dc:creator>
            <pubDate>Wed, 29 Nov 2023 23:06:37 GMT</pubDate>
            <atom:updated>2024-07-15T01:17:24.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F74SFOV4QNgEtQ1r29NaWQ.png" /><figcaption>image from DALL-E 3</figcaption></figure><h3>Dotty AI Agent</h3><h4>Enhancing .NET Core Development with a Customized GPT Assistant</h4><h3>Introduction</h3><p>In recent years, we have witnessed a revolution in the field of artificial intelligence, driven largely by advances in machine learning and natural language processing. The culmination of these efforts has led to the creation of incredibly sophisticated AI models, capable of understanding and responding to our queries with astonishing accuracy. One of the most exciting developments in this space is the introduction of customizable GPTs (Generative Pre-trained Transformers) by OpenAI.</p><p>As an industry professional and passionate about software development, I have always looked for ways to leverage emerging technology to solve complex problems and improve work efficiency. This search led me to explore the potential of custom GPTs, which inspired me to create my custom GPT, a GPT specialized in .NET Core. My goal was to develop a virtual assistant capable of not only understanding the nuances of development in .NET Core, but also providing solutions and practical guidance to real problems in this framework.</p><h3>Features, capabilities and privacy</h3><p>In the vast world of customizable GPTs, I am pleased to present a version focused on .NET Core, designed to assist and enrich the work of developers and professionals in this area. Unlike a generic GPT, this model has been enriched with a careful selection of free documents and eBooks, specifically focused on .NET Core. This personalization process allows you to carry out a kind of internal “rag” (retrieval-augmented generation), improving your ability to provide relevant and updated answers and solutions. Additionally it also has the characteristic of image generation / interpretation, and information search through BING.</p><p>What sets this GPT apart is its ability to understand and answer complex questions about .NET Core, providing practical and precise assistance. From solving programming questions to offering tips for optimizing code, this model can act as an informed and efficient development partner.</p><p>Privacy is a priority: interactions with this GPT are confidential and do not contribute to the improvement of other OpenAI models. Thus, we guarantee the security of the data and the exclusivity of the queries.</p><h3>Access and colaboration</h3><p>For all those interested in exploring the capabilities of this custom GPT in .NET Core, you can easily access it through the following link: <a href="https://chat.openai.com/g/g-cbL1ZH7wR-dotty"><strong>Dotty</strong></a><strong>.</strong></p><p>I invite .NET Core developers, professionals, and enthusiasts to use<strong> </strong><a href="https://chat.openai.com/g/g-cbL1ZH7wR-dotty"><strong>this tool</strong></a>, experiment with its capabilities, and see for yourself how it can make your work easier and better.</p><p>Collaboration and feedback are essential for the growth and continuous improvement of <a href="https://chat.openai.com/g/g-cbL1ZH7wR-dotty"><strong>Dotty</strong></a>. Therefore, I encourage all users to share their experiences, suggestions and any ideas that may contribute to its evolution. Together, we can move toward a future where custom GPTs become an integral part of software development, opening up new possibilities and boosting creativity and efficiency in our industry.</p><h3>Dotty&#39;s Test</h3><p>To test <a href="https://chat.openai.com/g/g-cbL1ZH7wR-dotty"><strong>Dotty</strong></a>, I asked for the next promt: &quot;can you write the snake game in a .net core console application? &quot;</p><p>And the result from the agent UI was:</p><pre>using System;<br>using System.Collections.Generic;<br>using System.Threading;<br><br>namespace SnakeGame<br>{<br>    class Program<br>    {<br>        static void Main(string[] args)<br>        {<br>            int boardWidth = 40;<br>            int boardHeight = 20;<br>            int score = 0;<br>            bool gameOver = false;<br>            ConsoleKey direction = ConsoleKey.RightArrow;<br><br>            // Snake starting position<br>            List&lt;Point&gt; snake = new List&lt;Point&gt;<br>            {<br>                new Point(20, 10), // head<br>                new Point(19, 10), // body<br>                new Point(18, 10)  // tail<br>            };<br><br>            // Food position<br>            Point food = GetRandomFoodPosition(boardWidth, boardHeight);<br><br>            while (!gameOver)<br>            {<br>                if (Console.KeyAvailable)<br>                {<br>                    var key = Console.ReadKey(true).Key;<br>                    switch (key)<br>                    {<br>                        case ConsoleKey.UpArrow:<br>                            if (direction != ConsoleKey.DownArrow)<br>                                direction = key;<br>                            break;<br>                        case ConsoleKey.DownArrow:<br>                            if (direction != ConsoleKey.UpArrow)<br>                                direction = key;<br>                            break;<br>                        case ConsoleKey.LeftArrow:<br>                            if (direction != ConsoleKey.RightArrow)<br>                                direction = key;<br>                            break;<br>                        case ConsoleKey.RightArrow:<br>                            if (direction != ConsoleKey.LeftArrow)<br>                                direction = key;<br>                            break;<br>                    }<br>                }<br><br>                // Update snake position<br>                MoveSnake(snake, direction);<br><br>                // Check collision<br>                gameOver = IsGameOver(snake, boardWidth, boardHeight);<br><br>                // Check if snake ate the food<br>                if (snake[0].Equals(food))<br>                {<br>                    score++;<br>                    snake.Add(new Point(-1, -1)); // add new segment<br>                    food = GetRandomFoodPosition(boardWidth, boardHeight);<br>                }<br><br>                // Drawing game board<br>                Console.Clear();<br>                DrawBoard(snake, food, boardWidth, boardHeight, score);<br><br>                // Game speed<br>                Thread.Sleep(100);<br>            }<br><br>            Console.SetCursorPosition(boardWidth / 2 - 7, boardHeight / 2);<br>            Console.WriteLine(&quot;Game Over!&quot;);<br>            Console.WriteLine($&quot;Score: {score}&quot;);<br>            Console.ReadKey();<br>        }<br><br>        static void DrawBoard(List&lt;Point&gt; snake, Point food, int width, int height, int score)<br>        {<br>            for (int y = 0; y &lt; height; y++)<br>            {<br>                for (int x = 0; x &lt; width; x++)<br>                {<br>                    if (x == 0 || y == 0 || x == width - 1 || y == height - 1)<br>                    {<br>                        Console.Write(&quot;#&quot;);<br>                    }<br>                    else if (snake[0].X == x &amp;&amp; snake[0].Y == y)<br>                    {<br>                        Console.Write(&quot;O&quot;); // Snake head<br>                    }<br>                    else if (snake.Exists(s =&gt; s.X == x &amp;&amp; s.Y == y))<br>                    {<br>                        Console.Write(&quot;o&quot;); // Snake body<br>                    }<br>                    else if (food.X == x &amp;&amp; food.Y == y)<br>                    {<br>                        Console.Write(&quot;*&quot;); // Food<br>                    }<br>                    else<br>                    {<br>                        Console.Write(&quot; &quot;);<br>                    }<br>                }<br>                Console.WriteLine();<br>            }<br><br>            Console.WriteLine($&quot;Score: {score}&quot;);<br>        }<br><br>        static void MoveSnake(List&lt;Point&gt; snake, ConsoleKey direction)<br>        {<br>            for (int i = snake.Count - 1; i &gt; 0; i--)<br>            {<br>                snake[i] = snake[i - 1];<br>            }<br><br>            switch (direction)<br>            {<br>                case ConsoleKey.UpArrow:<br>                    snake[0] = new Point(snake[0].X, snake[0].Y - 1);<br>                    break;<br>                case ConsoleKey.DownArrow:<br>                    snake[0] = new Point(snake[0].X, snake[0].Y + 1);<br>                    break;<br>                case ConsoleKey.LeftArrow:<br>                    snake[0] = new Point(snake[0].X - 1, snake[0].Y);<br>                    break;<br>                case ConsoleKey.RightArrow:<br>                    snake[0] = new Point(snake[0].X + 1, snake[0].Y);<br>                    break;<br>            }<br>        }<br><br>        static bool IsGameOver(List&lt;Point&gt; snake, int width, int height)<br>        {<br>            for (int i = 1; i &lt; snake.Count; i++)<br>            {<br>                if (snake[0].Equals(snake[i]))<br>                    return true;<br>            }<br><br>            return snake[0].X &lt; 1 || snake[0].X &gt;= width - 1 || snake[0].Y &lt; 1 || snake[0].Y &gt;= height - 1;<br>        }<br><br>        static Point GetRandomFoodPosition(int width, int height)<br>        {<br>            Random random = new Random();<br>            return new Point(random.Next(1, width - 1), random.Next(1, height - 1));<br>        }<br><br>        struct Point<br>        {<br>            public int X { get; }<br>            public int Y { get; }<br><br>            public Point(int x, int y)<br>            {<br>                X = x;<br>                Y = y;<br>            }<br><br>            public override bool Equals(object obj)<br>            {<br>                if (obj is Point)<br>                {<br>                    Point p = (Point)obj;<br>                    return X == p.X &amp;&amp; Y == p.Y;<br>                }<br>                return false;<br>            }<br><br>            public override int GetHashCode()<br>            {<br>                return X ^ Y;<br>            }<br>        }<br>    }<br>}</pre><figure><img alt="The Snake in action!" src="https://cdn-images-1.medium.com/max/298/1*ZDSlCT7em6631rgXOHTitQ.png" /><figcaption>Image from my linux terminal</figcaption></figure><p>The Snake game, implemented as a .NET Core console application, functions flawlessly, operating ‘like a charm’. It encapsulates all the quintessential features of the classic Snake game, including food collection, and automatic growth of the snake upon consuming food. The game effectively handles boundary collisions and self-collisions, which are the fundamental game-ending conditions. Its simple yet responsive control mechanism using arrow keys provides an engaging and nostalgic gaming experience.</p><p>By the other side when asked to then plain chat gpt 4 for the same prompt, the answer was:</p><blockquote>Creating a complete Snake game in a .NET console application involves several steps and a substantial amount of code. Here’s a high-level outline of what you would need to do:</blockquote><blockquote>Setup the Console Application: Start by creating a new .NET console application. This will be your project base.</blockquote><blockquote>Game Loop: Implement a game loop that continuously runs, updating the game state and rendering the game to the console.</blockquote><blockquote>Snake Representation: Create a class or structure to represent the snake. This could include properties for the snake’s position, direction, and length.</blockquote><blockquote>Game Field: Define the play area. This could be a simple grid or array representing the game field.</blockquote><blockquote>Snake Movement: Implement logic to move the snake based on user input. The snake should move one cell at a time in the specified direction.</blockquote><blockquote>Food Generation: Randomly place food items on the game field that the snake can eat.</blockquote><blockquote>Collision Detection: Implement logic to detect collisions with the game field boundaries or the snake itself.</blockquote><blockquote>Score and Growth: Keep track of the score, and increase the length of the snake each time it eats food.</blockquote><blockquote>Game Over Condition: Implement the game over logic, which occurs when the snake collides with the game field boundaries or itself.</blockquote><blockquote>User Input: Handle user input to control the direction of the snake.</blockquote><blockquote>Rendering: Display the game state in the console, including the snake, food, and score.</blockquote><blockquote>Game Restart: Allow the player to restart the game after a game over.</blockquote><blockquote>Due to the complexity and length of the code required, it’s not feasible to provide the complete implementation here. However, I can help you get started with a simple example or assist with specific parts of the implementation. Let me know how you would like to proceed!</blockquote><h3>Final Thoughts</h3><p>I sincerely hope that <a href="https://chat.openai.com/g/g-cbL1ZH7wR-dotty"><strong>Dotty</strong></a> proves to be a valuable tool for your development endeavors. Your feedback is highly valuable to me; it not only helps in refining this tool but also guides future developments. If you find <a href="https://chat.openai.com/g/g-cbL1ZH7wR-dotty"><strong>Dotty</strong></a> useful, feel free to express your support with claps or by following my profile. This way, you’ll stay updated with the latest advancements and contribute to a community of like-minded professionals and enthusiasts. Together, we can explore the full potential of AI in software development and create a more efficient, innovative future.</p><p>See you, DM</p><h3>Cubed</h3><p><em>Thank you for being a part of the community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a> | <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://venturemagazine.net/"><strong>Venture</strong></a> | <a href="https://cubed.run/"><strong>Cubed</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3963fa19269e" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/enhancing-net-core-development-with-a-customized-gpt-assistant-3963fa19269e">Dotty: Enhancing .NET Core Development with a Customized GPT Assistant</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>