<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Quang Ha (Alex) Ngo on Medium]]></title>
        <description><![CDATA[Stories by Quang Ha (Alex) Ngo on Medium]]></description>
        <link>https://medium.com/@ngoquangha?source=rss-14d94ec43f2c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lOxdv6s8b2mLFYWQP0hvWQ.jpeg</url>
            <title>Stories by Quang Ha (Alex) Ngo on Medium</title>
            <link>https://medium.com/@ngoquangha?source=rss-14d94ec43f2c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:06:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ngoquangha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RAG Evaluation with a Local LLM: Comparing Vector Index and Knowledge Graph]]></title>
            <link>https://medium.com/@ngoquangha/rag-evaluation-with-a-local-llm-comparing-vector-index-and-knowledge-graph-ed21a604d7be?source=rss-14d94ec43f2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed21a604d7be</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <dc:creator><![CDATA[Quang Ha (Alex) Ngo]]></dc:creator>
            <pubDate>Sun, 10 May 2026 01:01:00 GMT</pubDate>
            <atom:updated>2026-05-10T01:01:00.605Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgO7cuU7mrAiCcNdsLhPqA.png" /></figure><p><em>A demo to understand RAG evaluation metrics and see Vector Index vs Knowledge Graph retrieval in practice</em></p><h3>The Starting Point</h3><p>I built a demo to find out:</p><ol><li><strong>What metrics actually exist for evaluating RAG</strong> — and what each one measures.</li><li><strong>How Vector Index and Knowledge Graph retrieval compare</strong> on the same document, scored by the same LLM. In the webapp, you upload a PDF and a set of ground-truth Q&amp;A pairs. The backend ingests the document into both pipelines, runs every question through both, and scores each answer with RAGAS and DeepEval simultaneously.</li></ol><h3>The Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iELmLPbULfzQ4Z5SXsPA1A.png" /></figure><p>Three screens:</p><ol><li><strong>Upload</strong> — drag a PDF and a JSON file of ground-truth Q&amp;A pairs. The backend ingests the document into both pipelines at the same time.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6f1REhpzbeleUK-vjaGoKQ.png" /></figure><p><strong>2. Knowledge Base</strong> — explore the extracted knowledge graph as an interactive SVG force-graph. Entities are nodes, relationships are edges. You can inspect confidence scores and source pages before running any evaluation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_5tXR2nU7KB9C4Jo8Vs1w.png" /></figure><p>3. <strong>Metrics</strong> — run every ground-truth question through both pipelines, score with both frameworks, and view a 2×2 result matrix: pipeline (Vector / Graph) × framework (RAGAS / DeepEval).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xIKXRHtCefBKeH0AcR2Ww.png" /></figure><p>One rule: <strong>both pipelines use the same LLM for generation</strong>. Retrieval is the only variable.</p><h3>The Two Pipelines</h3><h3>Vector Index</h3><pre>PDF → chunk → embed → ChromaDB<br>Question → embed → cosine search → cross-encoder [1] rerank → top chunks → LLM</pre><h3>Knowledge Graph</h3><pre>PDF → spaCy NER → entities + co-occurrence edges + optional LLM triples → NetworkX graph<br>Question → entity linking → BFS [2] subgraph expansion → confidence filter → top edges → LLM</pre><p>Starting from entities matched in the question, retrieval walks the graph outward — up to 2 hops, capped at 50 nodes — then serialises the strongest edges as context for the LLM:</p><p>Nodes carry type, confidence, and provenance. For example,</p><pre>graph.nodes[&quot;BERT&quot;] == {<br> &quot;type&quot;: &quot;MODEL&quot;,<br> &quot;confidence&quot;: 0.91,<br> &quot;chunk_ids&quot;: [&quot;paper.pdf::page4&quot;, &quot;paper.pdf::page7&quot;],<br> &quot;text_spans&quot;: [{&quot;chunk_id&quot;: &quot;paper.pdf::page4&quot;, &quot;text&quot;: &quot;We train on BooksCorpus…&quot;}],<br>}</pre><p>Edges carry the relationship and a ranking score:</p><pre>graph.edges[&quot;BERT&quot;, &quot;BookCorpus&quot;] == {<br> &quot;predicate&quot;: &quot;pretrained_on&quot;,<br> &quot;confidence&quot;: 0.85,<br> &quot;composite_weight&quot;: 0.65,<br> &quot;chunk_ids&quot;: [&quot;paper.pdf::page4&quot;],<br>}</pre><p>At retrieval time, the question is first scanned for <em>entity </em>mentions that match nodes in the graph. Those matched nodes become seeds for BFS expansion. The strongest edges from the resulting subgraph are then sorted by weight and serialised into plain text for the LLM:</p><pre>Transformer - [introduced_by] → Vaswani et al. [source: paper.pdf::page1; weight=0.85; &quot;Attention is all you need…&quot;]<br>BERT - [pretrained_on] → BookCorpus [source: paper.pdf::page4; weight=0.65; &quot;We train on BooksCorpus…&quot;]<br>RLHF - [applied_in] → GPT-4 [source: paper.pdf::page7; weight=0.72]</pre><p>Edges between entities can be built from co-occurrence [3] on the same page, or from typed triples [4] extracted by the LLM — each producing a structurally different graph.</p><h3>The Metrics</h3><p>Both frameworks share a core set of metrics, though they approach each one differently under the hood. For shared metrics, the formula shown is RAGAS’s implementation. Some metrics are unique to one framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*zrRbPql7BHX3s7Zhba3ldQ.png" /></figure><p><em>† RAGAS Context Precision does not require ground truth; DeepEval’s implementation does.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CoBn-_vEmXRfXrpDLvJTiA.png" /></figure><p>Scores from a 136-page non-fiction book with 20 ground-truth questions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*xMYbpY1v0kVzr4iRNzp-GQ.png" /></figure><h3>Lessons Learnt</h3><p>All scores below were produced using <strong>llama3.1:8b</strong> running locally via Ollama as the judge model.</p><p><strong>Caveats:</strong></p><ul><li>Models below ~7B tend to return off-format JSON, which the frameworks silently parse asNone; in this demo,llama3.1:8b was the smallest that produced relatively consistent, structured output.</li><li>The Knowledge Graph here is built for processing speed, not semantic richness — a more carefully tuned graph would likely shift the scores.</li><li>A stronger judge model would shift them further.</li></ul><p><strong>What the scores show:</strong></p><ul><li>Vector led on most metrics. Context Recall was the closest (0.79 vs 0.75 on RAGAS), suggesting Graph retrieved most of the relevant content even when it scored lower elsewhere.</li><li>Hallucination was high for both pipelines (0.61 Vector, 0.81 Graph) — likely the local model generating content it cannot fully ground, rather than a retrieval problem.</li><li>The biggest framework divergence: Graph’s Answer Correctness was 0.35 on RAGAS and 0.66 on DeepEval — a concrete illustration of how differently the two frameworks define “correct.”</li></ul><p><strong>Running the evaluation:</strong></p><ul><li>The first runs used an OpenAI API key — fast, but credits burn quickly with two pipelines and two frameworks per question. Switching to Ollama cuts costs, at the expense of slower inference and lower reasoning capability.</li><li>Concurrent judge calls against Ollama stalled threads; serialising them fixed it.</li><li>With 20 questions and 4 sequential judge calls each, a full run takes 1–2 hours. The tool checkpoints after every question so a cancelled run can be resumed.</li></ul><p><strong>Source code</strong>: <a href="https://github.com/qhngo/rag-comparison-benchmark">https://github.com/qhngo/rag-comparison-benchmark</a></p><h3>Notes</h3><p><strong>[1] Cross-encoder rerank</strong> — A cross-encoder is a model that takes a (question, chunk) pair as a single input and outputs one relevance score. Unlike an embedding model — which encodes question and chunk separately and then compares vectors — the cross-encoder sees both texts together in the same attention window, letting it directly match question words against chunk words. It is slower (must run once per candidate at query time) but more precise. Here it scores all 10 candidates from the initial cosine search and keeps the top 5.</p><p><strong>[2] BFS (Breadth-First Search)</strong> — A graph traversal strategy that starts from one or more seed nodes and expands outward layer by layer: first all direct neighbours (1 hop), then neighbours of neighbours (2 hops), and so on. In this demo, BFS stops after 2 hops and is capped at 50 total nodes to avoid returning the entire graph.</p><p><strong>[3] Co-occurrence edges</strong> — Two entities are connected by a co-occurrence edge if they appear on the same page. No deeper relationship is inferred — the edge only records proximity. The more pages they appear on together, the higher the edge weight and confidence.</p><p><strong>[4] LLM triples</strong> — Typed, directional facts extracted from text by the LLM, in the form (subject, predicate, object). Example: (Dale Carnegie, born_in, Maryville). The predicate names the actual relationship rather than just recording co-occurrence, making the graph semantically richer. Each triple is grounded by a verbatim text span from the source page.</p><p><strong>[5] Reverse questions</strong> — To score Answer Relevancy, the evaluator generates N questions that the answer could plausibly be responding to (“What question does this answer?”), then computes how similar those reverse questions are to the original question via cosine similarity. If the answer is on-topic, the reverse questions will resemble the original. The final score is the mean cosine similarity across all N reverse questions — not the max, so a single lucky match does not inflate the score.</p><p><strong>[6] Position-weighted (Context Precision)</strong> — Relevant chunks that appear earlier in the retrieved list contribute more to the score than the same relevant chunks appearing later. This rewards retrieval systems that surface the most useful content at the top of the ranking.</p><p><strong>[7] Factual F1</strong> — The LLM judge decomposes both the generated answer and the ground-truth answer into individual statements, then computes an F1 score over the overlap: how many statements from the ground truth appear in the generated answer, and how many statements in the generated answer are supported by the ground truth. Answer Correctness in RAGAS combines this factual F1 with the embedding-based answer similarity score.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed21a604d7be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Interface, Multiple Agents: A Small Local Demo of Multi-Agent Routing]]></title>
            <link>https://medium.com/@ngoquangha/one-interface-multiple-agents-a-small-local-demo-of-multi-agent-routing-75cc0183e1fc?source=rss-14d94ec43f2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/75cc0183e1fc</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <dc:creator><![CDATA[Quang Ha (Alex) Ngo]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 11:14:47 GMT</pubDate>
            <atom:updated>2026-03-01T11:14:47.117Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently read <em>Building Applications with AI Agents: Designing and Implementing Multiagent Systems</em> by <strong>Michael Albada</strong>.<br>One idea that stayed with me is that strong agentic applications are not just “one model call,” but systems that combine planning, tools, memory, and multi-step execution.</p><p>That perspective motivated this demo: instead of building a single prompt-response app, I wanted to explore a practical <strong>multi-agent routing</strong> pattern behind one unified chat interface.</p><p>Disclaimer: This is not a production framework. It is a local prototype for learning and experimentation :)</p><h3>Codebase</h3><p><a href="https://github.com/qhngo/Multi-agent-routing-orchestration-demo/tree/master">https://github.com/qhngo/Multi-agent-routing-orchestration-demo/</a></p><h3>Why this demo exists</h3><p>My main motivation is to build a <strong>conceptual multi-agent platform</strong> that can grow over time.</p><p>Commercial platforms like <strong>Salesforce Agentforce</strong>, <strong>Microsoft Foundry Agent Service</strong>, and <strong>Google Vertex AI Agent Builder</strong> show where enterprise agent orchestration is heading.<br>On the open-source side, projects like <strong>OpenClaw</strong>, <strong>CrewAI</strong>, <strong>Microsoft AutoGen</strong>, and <strong>LangGraph</strong> show a similar direction in modular agent design and routing patterns.</p><p>This demo is my small local version of that idea: a foundation where we can keep adding task-specific agents for jobs a generic LLM may not handle well, while preserving a simple user experience through one unified chat interface.</p><h3>Conceptual architecture</h3><p>The flow is intentionally simple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNdfTKltbf5m3qWzlkoIeA.png" /></figure><ol><li>User sends a message in one <strong>Web Chat Interface</strong>.</li><li>A <strong>Routing Agent</strong> decides which path should handle the request.</li><li><strong>Task-oriented Agents</strong> run specialized logic.</li><li>A <strong>Local LLM / Response Synthesizer</strong> supports reasoning and final answer shaping.</li><li>A single response returns to the same interface.</li></ol><p>So the user sees one assistant, while the backend coordinates specialized components.</p><h3>Local LLM interface</h3><p>This project is designed for local experimentation.<br>To run it as intended, start a local model endpoint from:</p><p><a href="https://github.com/qhngo/Local-LLM-Interface">https://github.com/qhngo/Local-LLM-Interface</a></p><p>That endpoint is then used as shared reasoning support for routing/planning and response generation in this prototype.</p><h3>How to start the web app</h3><p>After your local LLM interface is running, start this demo app:</p><p>Create and activate a virtual environment (PowerShell)</p><pre>python -m venv .venv .\.venv\Scripts\Activate.ps1</pre><p>Install dependencies</p><pre>pip install -r requirements.txt</pre><p>Configure .env in the project root (example)</p><pre>WEB_APP_URL=http://127.0.0.1<br>WEB_APP_PORT=8000<br>APP_LOG_LEVEL=INFO<br>LOG_RETENTION=7<br>LAST_INTERACTION_THRESHOLD=7<br>SQL_PROVIDER=sqlite<br>LLM_PROVIDER=generic<br>LOCAL_API_URL=http://127.0.0.1:8081/answer<br>LOCAL_API_TIMEOUT_SECONDS=20<br>REMOTE_API_URL=http://127.0.0.1:8081/answer<br>REMOTE_API_TIMEOUT_SECONDS=20<br>KB_PROVIDER=in_memory<br>TOOLSET=default</pre><p>Run the backend</p><pre>python run.py</pre><p>Open the app in your browser</p><ul><li><a href="http://127.0.0.1:8000/">http://127.0.0.1:8000/</a></li><li>Register a user, then log in to start chatting.</li></ul><p>Then you will see this UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*3mANOsXwMCTe6SZpNbBGCg.png" /></figure><p>At the moment, the platform has two agents registered:</p><ol><li><strong>generic</strong> (fallback)<br>The default general-purpose path. It handles requests that do not clearly match a specialized agent.</li><li><strong>special:arxiv</strong> (research-paper search)<br>A specialist agent for paper discovery workflows: extracting keywords, querying arXiv, and returning ranked results</li></ol><p>You can start by sending a query, for example: “Generate a plan to help me lose weight in 6 months.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*zz-29DoZ3fA8otXDGTkpBQ.png" /></figure><p>At the bottom of each response, you will see the handling agent’s name, processing time (in seconds), and total token cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*6WSfabGzwGAiEHNai4R33g.png" /></figure><p>Another example: “<em>Search for five research papers tagged with the keyword &#39;AI’</em>”. This time, the handling agent is “<strong>special:arxiv</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*wUVgN4PJS489YwExRNu3aw.png" /></figure><h3>Where routing happens in code</h3><p>The core handoff path is explicit:</p><pre># src/app/api/routes/backend_routes.py<br>@router.post(&quot;/chat&quot;)<br>def chat(...):</pre><pre># src/app/services/chat_service.py<br>def process_for_user(...):</pre><pre># src/app/runtime/routed_runtime.py<br>def run(...):</pre><pre># src/app/runtime/agent_router.py<br>def route_with_context(...):</pre><p>In short:</p><p><em>/chat -&gt; ChatService -&gt; RoutedRuntime -&gt; AgentRouter -&gt; selected agent flow -&gt; final response</em></p><h3>What this demo demonstrates</h3><p>This prototype is mainly useful for validating ideas:</p><ul><li>Multi-agent internals can stay hidden behind one clean UI.</li><li>Routing can map user intent to specialized handling paths.</li><li>A fallback/synthesis layer can keep responses coherent.</li><li>Local-first architecture helps fast iteration.</li></ul><h3>Scope and boundaries</h3><p>This is intentionally not positioned as:</p><ul><li>a production-scale platform,</li><li>a complete security/scalability blueprint,</li><li>or a final “best practice” architecture.</li></ul><p>It is a focused learning built to test one question:<br><strong>Can multi-agent orchestration feel simple to the end user?</strong></p><h3>Closing</h3><p>Inspired by Michael Albada’s framing of agent systems, this project is my attempt to move from single-call demos to a more composable pattern.<br>If users can interact naturally without thinking about which agent is working underneath, the demo has achieved its purpose.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75cc0183e1fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Local GPU-Powered LLM API: A Demo from CLI to Web Service]]></title>
            <link>https://medium.com/@ngoquangha/designing-a-local-gpu-powered-llm-api-a-demo-from-cli-to-web-service-52916818f5a2?source=rss-14d94ec43f2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/52916818f5a2</guid>
            <category><![CDATA[local-llm]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <dc:creator><![CDATA[Quang Ha (Alex) Ngo]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 14:00:56 GMT</pubDate>
            <atom:updated>2026-02-15T14:39:34.549Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been experimenting with local LLMs on my PC to make better use of my old GPU. This is mainly a hands-on experiment, and the idea is that basic LLM testing can run locally instead of paying for commercial APIs.</p><p>At first, I was running inference the usual way: command line, manual prompts, repeat. It works, but it quickly feels limiting once you want to integrate with other tools or test workflows end-to-end. So I built a small demo that wraps a local model with FastAPI and adds a simple queue for request handling.</p><p>This is not a production system. It’s a practical demo to learn the moving parts.</p><h3>Why I built this demo</h3><p>My goals were straightforward:</p><ul><li>Run a local model with GPU acceleration</li><li>Expose an API endpoint for app integration</li><li>Avoid manually starting inference processes each time</li><li>Explore how to handle multiple incoming requests safely</li></ul><p>I’m using:</p><ul><li>SeaLLM-7B-v2.q8_0.gguf as the model</li><li>CUDA-enabled llama-server.exe from llama.cpp. GPU is NVIDIA GeForce RTX 2080 Ti.</li><li>FastAPI as the wrapper layer</li></ul><p>Local LLM API architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YxwiTDdsU92dnhG4ceBk7A.png" /></figure><h3>How the local model server is called</h3><p>The core runtime command is:</p><pre>tools/llama-b7966-bin-win-cuda-12.4-x64/llama-server.exe -m models/SeaLLM-7B-v2.q8_0.gguf -ngl 99 -c 4096 -t 8 -b 512</pre><p>A few flags matter a lot here:</p><ul><li>-ngl 99: offload as many layers as possible to the GPU</li><li>-c 4096: context window size</li><li>-t 8: CPU threads</li><li>-b 512: batch size for inference throughput tradeoffs</li></ul><p>In my app, this process is started/stopped by the API lifecycle, so I don’t have to launch it manually every session.</p><h3>Why I wrapped it with a web API</h3><p>I expose an answer endpoint (GET/POST), normalise input (text or JSON), and return a consistent response structure. Internally, the wrapper calls the local llama-server at:</p><pre>http://127.0.0.1:8080/v1/chat/completions</pre><p>This keeps model execution local while giving me an interface that feels like a normal service.</p><h3>Queue-based handling for multiple requests</h3><p>This was one of the most useful parts of the experiment.</p><p>Instead of letting concurrent API requests hit the model all at once, I enqueue each request and let a worker process them in order. Each request waits on an event and gets released once its answer is ready.</p><pre>REQUEST_QUEUE: Queue[Dict[str, Any]] = Queue()<br>REQUEST_RESULTS: Dict[str, Dict[str, Any]] = {}<br>REQUEST_EVENTS: Dict[str, Event] = {}<br>REQUEST_LOCK = Lock()<br><br>def request_worker() -&gt; None:<br>    while True:<br>        item = REQUEST_QUEUE.get()<br>        try:<br>            start_time = time.perf_counter()<br>            answer = process_request(item)<br>            elapsed_s = round(time.perf_counter() - start_time, 2)<br><br>            with REQUEST_LOCK:<br>                result = REQUEST_RESULTS.get(item[&quot;id&quot;])<br>                if result is not None:<br>                    result[&quot;answer&quot;] = answer<br>                    result[&quot;processing_time_s&quot;] = elapsed_s<br>                    REQUEST_EVENTS[item[&quot;id&quot;]].set()<br>        finally:<br>            REQUEST_QUEUE.task_done()<br><br>def _enqueue_and_wait(request_id: str, input_type: str, received):<br>    event = Event()<br>    with REQUEST_LOCK:<br>        REQUEST_RESULTS[request_id] = {<br>            &quot;received&quot;: received,<br>            &quot;input_type&quot;: input_type,<br>            &quot;answer&quot;: None,<br>            &quot;processing_time_s&quot;: None,<br>        }<br>        REQUEST_EVENTS[request_id] = event<br><br>    REQUEST_QUEUE.put({&quot;id&quot;: request_id, &quot;input_type&quot;: input_type, &quot;received&quot;: received})<br>    event.wait()<br><br>    with REQUEST_LOCK:<br>        result = REQUEST_RESULTS.pop(request_id)<br>        REQUEST_EVENTS.pop(request_id, None)<br>    return result</pre><p>It’s a simple model, but it gives predictable behaviour and avoids request pileups, causing unstable performance.</p><h3>What I learned from this build</h3><p>A few practical takeaways stood out:</p><ul><li>Local GPU inference is quite practical for prototyping. In this demo, a short request (around <strong>200 tokens</strong>) takes roughly <strong>5 seconds</strong> end-to-end, which is acceptable for interactive testing, with clear room for improvement through inference-parameter and queue tuning. The tradeoff is acoustic: when the GPU gets busy, the cooling fans sound like they’re preparing for takeoff.</li><li>Putting inference behind an API immediately improves developer experience and makes integration easier.</li><li>Queueing becomes important once multiple clients or browser tabs send requests concurrently.</li><li>Reliability fundamentals like startup checks, retries, and timeouts are necessary even in small demos.</li></ul><h3>References</h3><ul><li>SeaLLM-7B-v2 (GGUF): <a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q8_0.gguf">https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q8_0.gguf</a></li><li>llama.cpp: <a href="https://github.com/ggml-org/llama.cpp">https://github.com/ggml-org/llama.cpp</a></li><li>Demo source code: <a href="https://github.com/qhngo/Local-LLM-Interface">https://github.com/qhngo/Local-LLM-Interface</a></li></ul><h3>What’s next</h3><p>If I continue this demo, I may focus on:</p><ul><li>streaming responses</li><li>better observability and queue metrics</li><li>smarter concurrency strategies</li><li>optional multi-model routing</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52916818f5a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>