<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mahadev Gaonkar on Medium]]></title>
        <description><![CDATA[Stories by Mahadev Gaonkar on Medium]]></description>
        <link>https://medium.com/@mngaonkar?source=rss-453d3a9e4752------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9SHV5ToWLNpnYC5xe5AKIQ.jpeg</url>
            <title>Stories by Mahadev Gaonkar on Medium</title>
            <link>https://medium.com/@mngaonkar?source=rss-453d3a9e4752------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:01:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mngaonkar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We Pointed Vulkan at a 2017 Radeon]]></title>
            <link>https://medium.com/@mngaonkar/we-pointed-vulkan-at-a-2017-radeon-8ea1da5760e5?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ea1da5760e5</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[llama-cpp]]></category>
            <category><![CDATA[macos]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 03:12:40 GMT</pubDate>
            <atom:updated>2026-11-16T06:14:30.407Z</atom:updated>
            <content:encoded><![CDATA[<p>A hacking note. Not a setup guide. On one iMac, llama.cpp through MoltenVK ran a 9B model on a Radeon Pro 580. Here is what the machine did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SE8xlGC2MomunBzZZdJdw.jpeg" /></figure><p>This started as a question, not a plan: is there anything left to do with the GPU in an eight-year-old iMac? The box is an iMac18,3. Quad-core i7–7700K, 64 GB of RAM, a 5K panel, macOS 13.7, and an AMD Radeon Pro 580 with 8 GB of GDDR5. We were not trying to turn it into a workstation. We wanted to see if a current llama.cpp could be talked into using that chip at all.</p><p>The usual paths did not apply to this particular computer. No CUDA. The PyTorch we could install today does not target an Intel Mac on this OS. A stock llama.cpp Metal build detected the card and then left the real work on the CPU. So we tried the odd route: Vulkan, with MoltenVK translating it into Metal. One machine, one afternoon, one prebuilt driver library. What follows is the lab notebook.</p><h3>First try: Metal listed the GPU and skipped the math</h3><p>llama.cpp on macOS builds with Metal unless you tell it not to. Ours did. The device list looked promising.</p><pre>Metal: AMD Radeon Pro 580 (8192 MiB, 8191 MiB free)<br>simdgroup reduction   = false<br>simdgroup matrix mul. = false<br>has unified memory    = false</pre><p>The card was there. The matmul was not. In this llama.cpp tree, the matrix multiplies go to Metal only when the GPU reports simdgroup reduction, which this part does not. It reports MTLGPUFamilyCommon3. The same gate covers norms, softmax, and flash attention. The device line was a decoy. The CPU did the arithmetic.</p><p>That was the end of the Metal experiment. We left that build where it was and started a second one.</p><h3>The hack: Vulkan, then MoltenVK, then a prebuilt dylib</h3><p>macOS has no Vulkan driver of its own. MoltenVK is a Vulkan implementation on top of Metal, and llama.cpp already has a Vulkan backend. Stacking those two was the whole idea.</p><p>Homebrew would not compile MoltenVK here. The formula asked for a full Xcode.app, and this Mac only has the Command Line Tools. We sidestepped that and dropped in the Khronos prebuilt of MoltenVK 1.4.2, which is a universal binary and includes x86_64. The Vulkan loader and glslc came from Homebrew. glslc only runs while the shaders are being compiled. We do not think compiling MoltenVK ourselves would have changed the timings. We did not try.</p><p>llama.cpp was a current checkout, 9fba51d, configured with Metal off and Vulkan on, in its own directory. An ICD file under /usr/local/share/vulkan/icd.d points the loader at libMoltenVK.dylib. After that, the line we were hoping for showed up:</p><pre>ggml_vulkan: 0 = AMD Radeon Pro 580 (MoltenVK)<br>  fp16: 1 | warp size: 32 | int dot: 0 | matrix cores: none</pre><p>warp size was the nervous part. There is an open MoltenVK report for this device ID, 0x67df, on macOS 13.7, where subgroupSize comes back as 0 and llama.cpp divides by it while loading shaders. That report was against MoltenVK 1.4.1. On 1.4.2 we got 32. That is not the chip&#39;s wavefront of 64. It is also not zero. The process lived, so we kept going.</p><h3>Two models, mostly to see if the words were real</h3><p>Eight gigabytes, and the 5K panel uses some of them. We stayed inside that and did not try to be clever with system RAM. A layer that spills off this GPU crosses PCIe on every token. We wanted to know if the GPU path itself worked.</p><p>Q4_K_M, text only, flash attention off.</p><p>The small model was a sanity check. Qwen2.5 0.5B Instruct, asked for the capital of France, said Paris. Against the CPU on a 128-token prompt it was 429 tokens per second versus 167, and 61 versus 40 on generation. The gap is small because the model is small. Launch overhead eats the win. We only cared that the sentence was a sentence.</p><p>Then Qwen3.5 9B, the Unsloth Q4_K_M, 5.28 GiB, no vision projector. Same question, same answer: Paris. llama-bench, all layers offloaded, flash attention off:</p><pre>Prompt  Generation<br>--------------------------  -------  ----------<br>Qwen2.5 0.5B, 128 tokens    429 t/s      61 t/s<br>Qwen3.5 9B, 128 in, 64 out   90 t/s      15 t/s</pre><p>A later prompt pass of a couple of thousand tokens measured 121 tokens per second. A one-line question looks much slower, because a dozen tokens never get out of startup. Fifteen tokens per second of generation is what this hack produced. We are not dressing that up as a target for anyone else.</p><p>We also flipped flash attention on for the tiny model. The sentence was still coherent, and generation was slower, so we left it off for the rest of the runs. Other people have seen garbage output from flash attention on this GPU family. We hit neither a miracle nor a disaster. We just stopped touching that flag.</p><h3>Stretching the context until it fell over</h3><p>The model card talks about a context in the hundreds of thousands of tokens. The hybrid attention keeps the cache smaller than a plain 9B, which is the only reason an 8 GB hack seemed worth trying. We filled the cache with llama-bench and measured the prompt pass, then generation with that much context already resident.</p><pre>Context   Prompt  Generation after that much context<br>-------  -------  ----------------------------------<br>8,192    117 t/s                            13.7 t/s<br>16,384   114 t/s                            13.2 t/s<br>32,768   106 t/s                            11.1 t/s<br>49,152    filled                             1.0 t/s<br>65,536    filled                             0.9 t/s</pre><p>Up through 32K the numbers barely moved. Past that, the model still loaded and still read the prompt, and then each new token took about a second. The working set had run out of room next to the display. We stopped. The interesting result was the cliff, not a recommended context length. The exact llama-bench lines are in the transcript below.</p><h3>Temperature, because we were curious</h3><p>Idle, the GPU’s own sensors read 45 to 46°C, 19 watts, and a 300 MHz core clock.</p><p>Under the 9B run the clocks went to the caps this iMac’s firmware exposes, 1200 MHz core and 1695 MHz memory, and stayed there. Activity sat around 100 percent. VRAM in use was about 6.2 GB. The prompt pass drew 219 to 232 watts. Steady generation sat around 195 to 211 watts. Temperature climbed into the mid-70s and leveled off at 75 to 78°C. It did not keep rising. A few dips in clock lined up with the gap between the prompt pass and token generation, not with the temperature.</p><p>When the benchmark exited, power was back at 19 watts on the next sample. Temperature was 63°C and falling. The GPU sensor reports no fan. This iMac cools the chip with the chassis fans, and we did not instrument those.</p><p>The run those watts bought, on this one afternoon, was 121 tokens per second reading a 2,048-token prompt and 15.5 tokens per second generating 1,536 tokens.</p><h3>Commands, as run on the iMac</h3><p>Everything below was typed on that iMac, in a home directory. Paths are written as $HOME. Output is trimmed where MoltenVK printed its full extension list or the shell printed a progress bar. Nothing here is a cleaned-up installer.</p><p>Homebrew already had CMake. It could install the Vulkan build tools. It could not build MoltenVK, because only the Command Line Tools are installed:</p><pre>export PATH=&quot;/usr/local/bin:$PATH&quot;<br>brew install molten-vk vulkan-headers vulkan-loader glslang shaderc</pre><pre>Error: molten-vk: An unsatisfied requirement failed this build.<br>A full installation of Xcode.app 11.7 is required to compile<br>this software. Installing just the Command Line Tools is not sufficient.<br><br>vulkan-headers  1.4.363.0<br>vulkan-loader   1.4.363.0<br>glslang         16.6.0<br>shaderc         2026.4</pre><p>So MoltenVK came from the Khronos release tarball instead. The dylib inside it is a universal binary, x86_64 and arm64.</p><pre>mkdir -p $HOME/moltenvk<br>curl -fL -o /tmp/MoltenVK-macos.tar \<br>  https://github.com/KhronosGroup/MoltenVK/releases/download/v1.4.2/MoltenVK-macos.tar<br>tar -xf /tmp/MoltenVK-macos.tar -C $HOME/moltenvk</pre><pre>$HOME/moltenvk/MoltenVK/MoltenVK/dynamic/dylib/macOS/libMoltenVK.dylib<br>$HOME/moltenvk/MoltenVK/MoltenVK/dynamic/dylib/macOS/MoltenVK_icd.json</pre><p>file on the dylib reported Mach-O universal binary with 2 architectures: [x86_64] [arm64].</p><p>llama.cpp went in its own directory, next to the older Metal tree at $HOME/llama.cpp, which we did not touch.</p><pre>git clone --depth 1 https://github.com/ggml-org/llama.cpp.git $HOME/llama.cpp-vulkan<br>cd $HOME/llama.cpp-vulkan<br>git log -1 --oneline</pre><pre>9fba51d convert: fix ripple dflash2 NoPE metadata (#32363)</pre><pre>export PATH=&quot;/usr/local/bin:$PATH&quot;<br>cmake -B build-vk \<br>  -DGGML_METAL=OFF \<br>  -DGGML_VULKAN=ON \<br>  -DGGML_BLAS=ON \<br>  -DGGML_BLAS_VENDOR=Apple \<br>  -DCMAKE_BUILD_TYPE=Release<br>cmake --build build-vk -j 8 --target llama-cli llama-bench llama-server</pre><p>The configure step found the loader and the shader compiler, then turned Vulkan on. OpenMP was not found. Accelerate was.</p><pre>-- Found Vulkan: /usr/local/lib/libvulkan.dylib (found version &quot;1.4.363&quot;) found components: glslc glslangValidator<br>-- Vulkan found<br>-- Including Vulkan backend<br>-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)<br>-- Accelerate framework found<br>-- Generating done<br>-- Build files have been written to: $HOME/llama.cpp-vulkan/build-vk</pre><p>The build finished with Built target llama-cli and Built target llama-bench.</p><p>The stock ICD uses a relative library path, so we wrote an absolute one where the loader looks by default:</p><pre>mkdir -p /usr/local/share/vulkan/icd.d<br>cat &gt; /usr/local/share/vulkan/icd.d/MoltenVK_icd.json &lt;&lt;&#39;EOF&#39;<br>{<br>    &quot;file_format_version&quot; : &quot;1.0.0&quot;,<br>    &quot;ICD&quot;: {<br>        &quot;library_path&quot;: &quot;$HOME/moltenvk/MoltenVK/MoltenVK/dynamic/dylib/macOS/libMoltenVK.dylib&quot;,<br>        &quot;api_version&quot; : &quot;1.4.0&quot;,<br>        &quot;is_portability_driver&quot; : true<br>    }<br>}<br>EOF</pre><p>Before that file existed, the same binary only saw the GPU if VK_ICD_FILENAMES pointed at the ICD that shipped inside the tarball. After it existed, a bench run with that variable unset still printed the Radeon.</p><pre>$HOME/llama.cpp-vulkan/build-vk/bin/llama-cli --list-devices</pre><pre>[mvk-info] MoltenVK version 1.4.2, supporting Vulkan version 1.4.357.<br>[mvk-info] GPU device:<br>        model: AMD Radeon Pro 580<br>        type: Discrete<br>        vendorID: 0x1002<br>        deviceID: 0x67df<br>        GPU memory available: 8192 MB<br>Available devices:<br>  Vulkan0: AMD Radeon Pro 580 (8192 MiB, 8192 MiB free)<br>  BLAS: Accelerate (0 MiB, 0 MiB free)</pre><p>The two GGUF files landed in $HOME/models. The small one was a correctness check. The 9B file is the one the rest of the numbers come from.</p><pre>mkdir -p $HOME/models<br>cd $HOME/models<br>curl -fL -o qwen2.5-0.5b-instruct-q4_k_m.gguf \<br>  https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf<br>curl -fL -o Qwen3.5-9B-Q4_K_M.gguf \<br>  https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q4_K_M.gguf</pre><pre>-rw-r--r--  469M  qwen2.5-0.5b-instruct-q4_k_m.gguf<br>-rw-r--r--  5.3G  Qwen3.5-9B-Q4_K_M.gguf</pre><p>A short reply, reasoning turned off so the 9B model would not spend the whole budget thinking:</p><pre>$HOME/llama.cpp-vulkan/build-vk/bin/llama-cli \<br>  -m $HOME/models/Qwen3.5-9B-Q4_K_M.gguf \<br>  -ngl 99 -fa off -c 4096 -n 80 \<br>  -p &quot;What is the capital of France? Answer in one short sentence.&quot; \<br>  -st --no-display-prompt --reasoning off</pre><pre>build      : b1-9fba51d<br>model      : $HOME/models/Qwen3.5-9B-Q4_K_M.gguf<br>ftype      : Q4_K - Medium<br>modalities : text<br><br>&gt; What is the capital of France? Answer in one short sentence.<br>The capital of France is Paris.<br><br>[ Prompt: 7.5 t/s | Generation: 12.5 t/s ]</pre><p>The prompt rate there is a dozen tokens and is not the measurement. llama-bench is. Same binary, flash attention off, one repetition, no warmup.</p><pre>$HOME/llama.cpp-vulkan/build-vk/bin/llama-bench \<br>  -m $HOME/models/Qwen3.5-9B-Q4_K_M.gguf \<br>  -ngl 99 -fa 0 -r 1 --no-warmup \<br>  -p 128 -n 64</pre><pre>ggml_vulkan: Found 1 Vulkan devices:<br>ggml_vulkan: 0 = AMD Radeon Pro 580 (MoltenVK) | uma: 0 | rebar: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none<br>| model                          |       size |     params | backend    | threads |  fa |            test |                  t/s |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |           pp128 |         89.63 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |            tg64 |         14.88 ± 0.00 |<br>build: 9fba51d (1)</pre><p>Prompt length and context depth were separate runs. -n 0 skips generation. -p 0 -d N fills N tokens of cache and then generates.</p><pre>$HOME/llama.cpp-vulkan/build-vk/bin/llama-bench \<br>  -m $HOME/models/Qwen3.5-9B-Q4_K_M.gguf \<br>  -ngl 99 -fa 0 -r 1 --no-warmup --progress \<br>  -p 8192,16384,32768 -n 0<br><br>$HOME/llama.cpp-vulkan/build-vk/bin/llama-bench \<br>  -m $HOME/models/Qwen3.5-9B-Q4_K_M.gguf \<br>  -ngl 99 -fa 0 -r 1 --no-warmup --progress \<br>  -p 0 -n 32 -d 8192,16384,32768,65536<br><br>$HOME/llama.cpp-vulkan/build-vk/bin/llama-bench \<br>  -m $HOME/models/Qwen3.5-9B-Q4_K_M.gguf \<br>  -ngl 99 -fa 0 -r 1 --no-warmup --progress \<br>  -p 0 -n 32 -d 49152</pre><pre>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |          pp8192 |        116.79 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |         pp16384 |        114.39 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |         pp32768 |        106.49 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |    tg32 @ d8192 |         13.69 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |   tg32 @ d16384 |         13.21 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |   tg32 @ d32768 |         11.12 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |   tg32 @ d49152 |          1.03 ± 0.00 |<br>| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | Vulkan,BLAS |       4 |   0 |   tg32 @ d65536 |          0.88 ± 0.00 |</pre><p>The longer sustained run, the one the temperature samples were taken against, was the same bench with a 2,048-token prompt and 1,536 tokens of generation. It printed pp2048 at 120.95 t/s and tg1536 at 15.52 t/s. While it ran, the live counters came from the accelerator entry in the I/O Registry, not from a separate monitoring tool:</p><pre>ioreg -r -c AMDRadeonX4000_AMDEllesmereGraphicsAccelerator -d 1</pre><p>Inside PerformanceStatistics, idle read Temperature(C)=45, Total Power(W)=19, Core Clock(MHz)=300, GPU Activity(%)=0. Under generation those fields sat around Temperature(C)=76, Total Power(W)=208, Core Clock(MHz)=1200, Memory Clock(MHz)=1695, GPU Activity(%)=100. The highest sample during the prompt pass was 232 W and the highest temperature was 78°C.</p><h3>What we did not prove</h3><p>We did not survey other GPUs, other MoltenVK versions, or other operating systems. On 1.4.1 this same device ID has a reported crash at shader load. We happened to use 1.4.2, saw warp size: 32, and got output. A zero in that field would have ended the experiment in a divide-by-zero. We did not go looking for that bug again.</p><p>We did not try diffusion. A Python stack was the wrong tool on this OS, and we never built stable-diffusion.cpp. Any guess about SDXL or Flux would be a guess.</p><p>We also did not try to make this pleasant to live with. No service, no startup item, no claim that 15 tokens per second and a 32K cliff is a good idea. It was a hack to see if the silicon would answer. It did, up to a point, and then the context ran it out of VRAM.</p><p><em>One iMac18,3, November 2026. i7–7700K, 64 GB, Radeon Pro 580 8 GB, macOS 13.7.8. llama.cpp </em><em>9fba51d, MoltenVK 1.4.2, Qwen3.5-9B Q4_K_M from the Unsloth quant. Temperature and power are the GPU&#39;s own performance counters, not an external meter.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ea1da5760e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tiny agent harness you can actually understand]]></title>
            <link>https://medium.com/@mngaonkar/a-tiny-agent-harness-you-can-actually-understand-af27314df515?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/af27314df515</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 19:24:08 GMT</pubDate>
            <atom:updated>2026-10-18T19:24:08.931Z</atom:updated>
            <content:encoded><![CDATA[<p>Most agent demos bury the interesting part under frameworks, dashboards, and magic. I wanted the opposite: a laptop agent small enough to read in one sitting, where you can see exactly how the model, its tools, and its instructions fit together.</p><p>That’s <a href="https://github.com/mngaonkar/alt-agent-harness">alt-agent-harness</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H-sT95GhypFKrq6IOYo9oQ.png" /></figure><h3>What it is</h3><p>A skill-based agent that runs locally: an OpenAI-compatible model, a local tool loop, and Anthropic-style <strong>Skills</strong> as folders on disk. It’s the laptop successor to my earlier <a href="https://github.com/mngaonkar/esp32s3-ai-agent">esp32s3-ai-agent</a> work — same shape, stripped down until every moving part is visible.</p><p>The whole design rests on three words:</p><ul><li><strong>Skills</strong> — how to do a job in <em>this</em> workspace (instructions, plus optional scripts)</li><li><strong>Tools</strong> — raw capability (files, scripts, shell, HTTP, web search)</li><li><strong>The model</strong> — decides what to do next</li></ul><p>That split is the point. A new skill is new instructions over the same tools, so extending the agent usually means writing a Markdown file, not changing harness code.</p><h3>The core idea: progressive disclosure</h3><p>The model never sees full skill bodies up front. The system prompt lists only each skill’s name and one-line description. When a request matches, the agent must call load_skill first, pull in the full SKILL.md, and only then start using tools.</p><p>Loading happens in three levels:</p><ol><li><strong>Name + description</strong> — always in the system prompt</li><li><strong>Full </strong><strong>SKILL.md</strong> — loaded when the model calls load_skill</li><li><strong>Bundled scripts</strong> — run via read_file / run_script when the skill calls for them</li></ol><p>This gating rule is deliberate. If the model jumps straight to tools, it skips the procedure the skill exists to teach — which checks to run, in what order, and what “done” looks like. Less context noise up front, clearer procedure when it matters.</p><p>A skill is just a directory:</p><pre>skills/sysinfo/<br>    SKILL.md          # YAML frontmatter: name, description<br>    scripts/<br>        rss_watch.py  # optional bundled helpers</pre><p>The model talks to a virtual filesystem rooted at /, so /skills/sysinfo/SKILL.md is skills/sysinfo/SKILL.md in the repo. Writes are allowed only under /skills/ and /tmp/, so a bad tool call can’t overwrite the harness itself.</p><p>Three skills ship in the box:</p><ul><li><strong>sysinfo</strong> — host diagnostics (host_status plus an RSS watcher script)</li><li><strong>websearch</strong> — when to search the live web, and how to phrase queries</li><li><strong>write-skill</strong> — how to author new skills, so the harness can extend itself</li></ul><p>That last one is easy to miss and worth pausing on: the agent can write its own skills. The harness guards this — write_file refuses a loose .py under /skills/, a bundled file with no SKILL.md, and a manifest whose name doesn’t match its directory. Scripts reuse existing tools through a tool(name, args) helper instead of inventing imports that would fail at runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nKQYLixExa36Z3q_RfKM2g.png" /></figure><h3>How it’s put together</h3><pre>you  ── TUI / REPL / (later: web, Discord) ──►  Session<br>                                                   │<br>                                                   ▼<br>                                              agent loop<br>                                                   │<br>                                    ├─ skills (instructions on disk)<br>                                    ├─ tools  (files, scripts, search)<br>                                    └─ HTTPS ──►  xAI / OpenRouter / OpenAI / local</pre><p>Front-ends live in interfaces/ and talk only to Session. The harness never imports them, so a web or Discord adapter can be added later without touching the loop.</p><p>Tools are boring on purpose: load_skill, list_skills, list_dir, read_file, write_file, delete_file, run_script, run_bash, host_status, http_get, and tavily_search when a Tavily key is set. Search is only registered when the key exists, so the model is never offered a tool that is guaranteed to fail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKFYoPGFsDrPo6q20rjhdg.png" /></figure><p>If you want to read the code, it maps cleanly:</p><ul><li>src/agent/loop.py — prompt → tools → observations → answer</li><li>src/agent/session.py — interface-agnostic ask, slash commands, events</li><li>src/agent/tools.py / skills.py / paths.py — tools, disclosure, virtual FS</li><li>src/agent/config.py / setup.py / llm.py — config, first-run, model client</li><li>src/interfaces/ — Textual TUI, plain REPL, configuration TUI</li></ul><h3>Providers: bring your own model</h3><p>The harness speaks the OpenAI-compatible API, so the provider is a configuration choice, not a code change:</p><ul><li><strong>xAI</strong> — grok-4.6 (the default), grok-2-latest</li><li><strong>OpenRouter</strong> — one key, many models (Claude, Llama, Gemini, DeepSeek, GPT…)</li><li><strong>OpenAI</strong> — gpt-4o, gpt-4o-mini, o3-mini, o1</li><li><strong>Custom endpoint</strong> — any OpenAI-compatible server: Ollama (http://localhost:11434/v1), vLLM, LM Studio</li><li><strong>Tavily</strong> — optional key that turns on live web search</li></ul><p>Switching providers can be as small as exporting a different key — set OPENROUTER_API_KEY and the harness infers the endpoint and model.</p><h3>Configuration without editing JSON</h3><p>There are three ways to configure it, in increasing order of comfort:</p><ol><li><strong>Environment / </strong><strong>.env</strong> — shell exports win; otherwise a gitignored .env next to config.json is loaded.</li><li><strong>First-run setup</strong> — with no key anywhere, the TUI opens the configuration screen instead of failing; the REPL falls back to a short terminal prompt. Settings are written to config.json (also gitignored).</li><li><strong>Configuration TUI</strong> — the visual way, any time:</li></ol><pre>uv run alt-agent-harness --config</pre><p>Inside the running chat TUI, press Ctrl+O or type /config. The screen lets you pick a provider, enter keys (masked), choose or type an exact model string, set the endpoint, add an optional Tavily key, and tune reasoning effort, temperature, and max tokens. <strong>Test Connection</strong> verifies the setup before you commit, and <strong>Save</strong> hot-reloads the live session — provider, model, and tools — with no restart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k6vABxDClJA5EO3z2Xuxtw.png" /><figcaption>Configuration TUI — pick provider, key, model, test, save.</figcaption></figure><p>Under the hood it’s still just config.json. The keys that matter most: provider, base_url, model, reasoning_effort (default low — reasoning models can otherwise spend their whole token budget thinking and return nothing), max_tokens, max_tool_iterations, and history_limit.</p><h3>Try it</h3><p>You’ll need Python 3.10+ and <a href="https://docs.astral.sh/uv/">uv</a>.</p><pre>git clone https://github.com/mngaonkar/alt-agent-harness.git<br>cd alt-agent-harness<br>uv sync<br>uv run alt-agent-harness</pre><p>No key configured? The configuration screen opens first — pick a provider, paste a key, test, save. Prefer files? Drop a .env in place instead:</p><pre>OPENROUTER_API_KEY=...<br># or XAI_API_KEY=... / OPENAI_API_KEY=...<br># optional: TAVILY_API_KEY=...   # enables live search</pre><p>Once you’re in the TUI, try this sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rqOJXBJfAMhZEbX2c6msQ.png" /></figure><ol><li>/skills — the catalog the model sees up front (names + descriptions only)</li><li>Ask: how&#39;s this machine doing? — the agent should load_skill on sysinfo, then call tools</li><li>/tools — the native tool list</li><li>Ctrl+O — change provider or model, save, and keep chatting</li><li>/quit when you’re done</li></ol><p>One-shot, no UI:</p><pre>uv run alt-agent-harness -c &quot;how&#39;s this machine doing?&quot;</pre><p>Prefer a plain prompt for pipes or logging? uv run alt-agent-harness --repl. Working in another checkout? --workspace /path/to/project makes that directory the root for skills, config, and /tmp.</p><p>Slash commands are for <strong>you</strong> — Session handles them before the model ever sees the line:</p><ul><li>/config — configuration TUI (Ctrl+O)</li><li>/skills, /tools — catalogs for humans</li><li>/info — raw host snapshot</li><li>/reset, /help, /quit</li></ul><p>Don’t confuse /skills (a catalog dump for you) with load_skill (the model tool that pulls a skill into the current turn). Different audiences, same word family.</p><h3>A macOS app, if you want one</h3><p>On a Mac you can wrap the exact same TUI in a native app bundle:</p><pre>./scripts/build_macos_app.sh</pre><p>It installs Alt Agent Harness.app into /Applications — launch it from Spotlight, Launchpad, or the Dock. Under the hood it’s still the same Textual TUI, workspace, config, and skills; the bundle just gives it a front door.</p><h3>What’s next</h3><p>More front-ends — web, Discord, whatever the day needs — without touching the loop. That was always the bet: keep the surface small, load skills on demand, let tools stay boring, and make the model swappable.</p><p>Clone it, run /skills, and watch the disclosure happen. That one interaction teaches the whole design.</p><p>Repo: <a href="https://github.com/mngaonkar/alt-agent-harness">github.com/mngaonkar/alt-agent-harness</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af27314df515" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An AI Agent That Runs On an ESP32-S3 (Not On Your Laptop)]]></title>
            <link>https://medium.com/@mngaonkar/an-ai-agent-that-runs-on-an-esp32-s3-not-on-your-laptop-176524f626ca?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/176524f626ca</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[esp32]]></category>
            <category><![CDATA[esp32-s3]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 03:01:13 GMT</pubDate>
            <atom:updated>2026-08-11T14:40:30.980Z</atom:updated>
            <content:encoded><![CDATA[<h3>An AI Agent That Runs On an ESP32-S3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlDKkDjzuvAWuhBy2o8gpw.png" /></figure><p><em>Chat with a board on your LAN. It can blink its LED, check its own health, take a photo, log temperature as a text chart, and grow new skills on its own flash — all in MicroPython.</em></p><p>Most “ESP32 + AI” projects put the brain on a PC or cloud worker and use the microcontroller as a dumb sensor. This one flips that around.</p><p><strong>esp32s3-ai-agent</strong> is an AI agent that runs <strong>on</strong> a Seeed XIAO ESP32S3 Sense. It calls an OpenAI-compatible API for reasoning, but the tool loop, skills, web UI, and hardware control all live on the board.</p><p><strong>Repo:</strong> <a href="https://github.com/mngaonkar/esp32s3-ai-agent">https://github.com/mngaonkar/esp32s3-ai-agent</a></p><h3>The idea in one picture</h3><pre>Phone / laptop browser  ──HTTP──►  ESP32-S3 agent<br>                                      │<br>                                      ├─ skills (instructions on flash)<br>                                      ├─ tools (LED, camera, files, search…)<br>                                      └─ HTTPS ──►  OpenAI-compatible API</pre><p>You open a page on your local network. You type a message. The board:</p><ol><li>Builds a system prompt that lists available <strong>skills</strong> (name + short description only)</li><li>Calls the model with <strong>tools</strong></li><li>Runs tool calls on real hardware (and may <strong>retry</strong> on failure within a large tool-round budget)</li><li>Returns a reply as <strong>text</strong> the web UI renders (markdown, code fences, inline photos)</li></ol><p>The split between the three layers is the whole design:</p><p><strong>Skills</strong> are how to use <em>this</em> board — which pin the LED is on, how to report free heap, how to draw a plot in text.</p><p><strong>Tools</strong> are raw capability — led_set, board_status, take_photo, analyze_photo, write_file, delete_file, run_script, web_search, and friends.</p><p><strong>The model</strong> decides what to do next.</p><p>New skills are just folders with a SKILL.md in them on the board&#39;s filesystem. No reflash to add a procedure. The agent can also <strong>author skills itself</strong>, either when you describe one or when it finishes a multi-step task worth remembering.</p><h3>What it looks like</h3><p>The web UI is a small Matrix-green chat page served <strong>from the board</strong>. There is no cloud frontend.</p><p>Here is a real turn — the model loads the sysinfo skill, calls board_status, then answers in plain language:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*NKS2Q4Ui18EgMRCV.png" /></figure><p>That two-step is deliberate. Progressive skill loading follows the Anthropic-style skills pattern: don’t stuff every instruction into the prompt up front, let the model pull in the full text only when it decides the skill is relevant. On a device with a few hundred KB of usable prompt budget, that isn’t an elegance argument — it’s the difference between working and not.</p><p>Configuration lives on the device too:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mxTpnVH3sz6X87Yr.png" /></figure><p>A few details worth calling out:</p><ul><li>Secrets stay on the board. Saved fields show “stored — leave blank to keep”.</li><li><strong>Model</strong> becomes a dropdown, populated from GET {base_url}/models when the key works. If that call fails, it falls back to a plain text field with a hint.</li><li>The CA bundle path is read-only in the UI. Change it through a deployed config.json.</li><li>Wi-Fi settings and the web port need a restart. Almost everything else applies live.</li><li><strong>Max tool rounds</strong> defaults to <strong>50</strong>, so skill authoring and fix-and-retry loops have room to actually finish.</li></ul><h3>Hardware</h3><p>The board is a <strong>Seeed XIAO ESP32S3 Sense</strong>. It works for this because it has 8 MB of octal <strong>PSRAM</strong>, a camera (an OV3660 on my unit), a user LED on GPIO21 wired active-low, and Wi-Fi.</p><p>The runtime is <strong>MicroPython</strong>, specifically a camera-capable build from <a href="https://github.com/cnadler86/micropython-camera-API">cnadler86/micropython-camera-API</a>.</p><p>PSRAM is the part that matters most. Conversation history, a TLS session, and JSON parsing of tool-call responses are not things you fit in 512 KB of SRAM. With 8 MB behind them, they’re routine.</p><p>Two honest limitations. The onboard <strong>PDM microphone is not supported</strong> — MicroPython has no PDM RX, so audio recording was tried and then removed rather than left half-working. And the <strong>trust model is a trusted LAN</strong>: the web UI and the chat API have no authentication at all. Do not put this on the public internet.</p><h3>Deploying It</h3><p>You need a XIAO ESP32S3 Sense (or a similar ESP32-S3 with PSRAM), a USB cable that actually carries data, Python 3, Wi-Fi, and an OpenAI-compatible API key.</p><p><strong>1. Clone and install dependencies.</strong></p><pre>git clone https://github.com/mngaonkar/esp32s3-ai-agent.git<br>cd esp32s3-ai-agent</pre><pre>python3 -m venv .venv<br>source .venv/bin/activate   # Windows: .venv\Scripts\activate<br>pip install -r requirements.txt</pre><p>The pinned set is small: esptool to talk to the ROM bootloader, mpremote for day-to-day file pushes, and littlefs-python to build the board&#39;s filesystem on your machine.</p><p><strong>2. Build the image.</strong></p><pre>python3 tools/build_image.py</pre><p>This packs two things into one flashable binary in dist/: the camera-capable MicroPython firmware, and a LittleFS image containing main.py, the agent/ modules, every skill under skills/, and the CA bundle. config.json is deliberately left out, so the image carries no credentials and is safe to hand to someone else.</p><p><strong>3. Flash it.</strong></p><p>Put the board in download mode first: hold <strong>BOOT</strong>, tap <strong>RESET</strong> (or hold BOOT while plugging in USB), keep BOOT held for about three seconds, then release. macOS shows it as “USB JTAG_serial debug unit” when it’s ready.</p><pre>tools/flash.sh                    # newest image in dist/<br>tools/flash.sh dist/esp32s3-agent-1.0.0.bin   # or a specific one</pre><p>That erases the chip and writes the whole image at offset 0 — firmware and filesystem in a single pass. A new board needs nothing else: no per-file copying, no Python tooling on the device side.</p><p><strong>4. First-time setup over serial.</strong></p><p>Setup is still console-based; there’s no captive portal yet. Power-cycle <em>without</em> holding BOOT, then:</p><pre>tools/console.sh</pre><p>On first boot the board asks only for what it cannot guess: Wi-Fi SSID, Wi-Fi password, and an API key. Model name and a Tavily key for web search are optional. Everything else has a default and can be changed later from the web Config tab once you’re online.</p><p>When setup finishes you should see something like this:</p><pre>[wifi] connected, ip=10.0.0.x<br>[skills] loaded …<br>[boot] web chat at http://10.0.0.x:80/</pre><p><strong>5. Open the chat.</strong></p><p>From a phone or laptop on the same Wi-Fi, open that URL.</p><p>tools/start.sh resets the board and prints the address again if you lost it.</p><p>Serial and web share a single agent — a small console reader thread keeps the web server responsive while you’re typing at the REPL.</p><p>After that, day-to-day code changes don’t need a reflash:</p><pre>tools/deploy.sh              # push code + skills; leaves /config.json alone<br>tools/deploy.sh --config     # also overwrite board config from local config.json<br>tools/start.sh</pre><h3>Things worth asking it</h3><p><strong>“How is the board doing?”</strong> loads sysinfo and calls board_status — heap, PSRAM, filesystem, die temperature, Wi-Fi signal, uptime.</p><p><strong>“Blink the LED three times.”</strong> loads led, then runs a blink script through run_script.</p><p><strong>“Take a photo.”</strong> loads camera and calls take_photo. The web UI shows the image inline.</p><p><strong>“Analyze what you see from camera”</strong> calls analyze_photo. Photos land on flash as 24-bit BMP, which no vision endpoint accepts, so the board converts BMP to PNG and base64-encodes it into a data URL — a small pure-MicroPython PNG encoder, because there&#39;s no PIL out here.</p><p><strong>“Monitor MCU temperature for one minute.”</strong> loads temp-monitor-plot, which samples the die temperature into a CSV under /tmp and prints a text plot of the trace.</p><p><strong>“What’s the weather in Fremont?”</strong> calls web_search, if a Tavily key is configured.</p><p><strong>“Teach the board a skill that…”</strong> loads write-skill, then drafts, tests, iterates on its own errors, and saves the result under /skills/.</p><p>Or the same agent over REST:</p><pre>curl -s -X POST &quot;http://&lt;board-ip&gt;/api/chat&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;message&quot;:&quot;In one sentence, what board are you on?&quot;}&#39;</pre><p>There is one shared conversation on the device. Web, serial, and REST all append to the same history until you reboot or reset it.</p><h3>How skills work</h3><p>A skill is a directory on the board:</p><pre>/skills/led/SKILL.md<br>/skills/led/scripts/blink.py</pre><p>It loads in three levels. <strong>Level 1</strong> is the name and description, always present in the system prompt. <strong>Level 2</strong> is the full SKILL.md body, pulled in when the model calls Skill(&quot;name&quot;). <strong>Level 3</strong> is the scripts and data files, reached through read_file and run_script.</p><p>Eight skills ship in the repo:</p><ul><li><strong>sysinfo</strong> — heap, PSRAM, filesystem, die temperature, Wi-Fi, uptime.</li><li><strong>led</strong> — onboard LED and general GPIO guidance, including the fact that this LED is single-colour and active-low.</li><li><strong>camera</strong> — take photos, list them, show them in the web UI, describe them with the vision model.</li><li><strong>netscan</strong> — Wi-Fi survey, channel congestion, reachability checks.</li><li><strong>websearch</strong> — when and how to use Tavily.</li><li><strong>temp-monitor-plot</strong> — sample MCU die temperature into CSV plus a text plot.</li><li><strong>tui-plot</strong> — redraw a CSV as a text-mode chart for chat or serial.</li><li><strong>write-skill</strong> — how the agent creates and updates skills.</li></ul><h3>The agent can grow its own skill list</h3><p>his is the part I find most interesting to watch.</p><p>Describe a skill and the agent plans it, writes /skills/&lt;name&gt;/SKILL.md, and puts any scripts under /tmp first, then moves them into the skill folder once they run. After a multi-step task that worked and looks likely to recur, it may save a skill on its own initiative without being asked.</p><p>When something fails, it’s expected to read the actual error, change something, and retry — which is why the tool-round budget defaults to 50 rather than the usual handful. delete_file lets it remove a failed skill directory under /skills/ and reload the catalog, so a bad draft isn&#39;t permanent.</p><p>Filesystem writes are fenced to /skills/, /tmp/, /photos/, and /audio/. The agent cannot overwrite firmware paths, however creative it gets.</p><p>One thing to set expectations on: there is no curses or LVGL TUI on this image. “Graphs” are formatted text in code fences, which the web UI renders as monospaced markdown. If you want a real plot, take the CSV to a host machine.</p><h3>What this is not</h3><p>It is not a cloud agent that merely <em>talks about</em> an ESP32. It is not phone-only first-boot setup — serial is still required once. It is not onboard PDM mic recording, a Discord gateway, NFS, or H.264 streaming. And it is not secure on an untrusted network, because the web UI has no login.</p><p>What it <em>is</em>: a small on-device agent loop with real tools, progressive skill loading, LAN chat plus REST, an OpenAI-compatible backend, and a skill library that can grow on flash.</p><h3>Links</h3><ul><li><strong>GitHub:</strong> <a href="https://github.com/mngaonkar/esp32s3-ai-agent">mngaonkar/esp32s3-ai-agent</a></li><li><strong>Camera MicroPython upstream:</strong> <a href="https://github.com/cnadler86/micropython-camera-API">cnadler86/micropython-camera-API</a></li></ul><h3>Closing</h3><p>If you already own an ESP32-S3 Sense and an API key, you can go from clone → flash → serial setup → chat in a single sitting. The interesting part is “Agent on microcontroller” — it’s that <strong>skills and tools live next to the metal</strong>, so the model can operate <em>this</em> board and teach itself new procedures without a reflash.</p><p>Clone it, flash it, answer three questions on the console, open the green terminal on your LAN, and ask the board how it’s doing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=176524f626ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web based agent with WebLLM and LangGraph]]></title>
            <link>https://medium.com/@mngaonkar/web-based-agent-with-webllm-and-langgraph-9d92fb6e23c2?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d92fb6e23c2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Mon, 20 Oct 2025 02:00:32 GMT</pubDate>
            <atom:updated>2025-10-20T05:04:38.395Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hXrZbSq7gkWOOv5BFFfwcg.png" /></figure><p>This article demonstrates how to run an agent completely offline and local in browser using Large Language Model (technically Small Language Model)</p><h3>Why Run Agent in Browser</h3><p>With advent of powerful local GPUs mainly from Apple and Nvidia its feasible to run LLM workload completely local to perform meaningful tasks. Though it&#39;s not a match with powerful cloud offerings from OpenAI, Anthropic but for solving domain specific use case it is a promising path. Picture this, you are developing an agentic use case solving a specific problem for a domain like retail, energy, telecom etc. You can solve this use case by using powerful cloud model, but in some cases it might be an expensive option when compared with the value delivered by the solution. As it is a specific use case, you can get very close performance to the cloud model by using small language model that is fine-tuned for the specific use case. Now, running these light weight models locally makes much more sense from both feature and cost perspective.</p><p>One of the best and more accessible option to run these model is browser. This is possible with the advancement in the technologies namely WebGPU, WASM and WebLLM. The next sections will provide a brief explanation for these technologies.</p><h3>What is WebGPU</h3><p>WebGPU is a modern web API designed to provide developers with low-level, high-performance access to a device’s graphics processing unit (GPU) directly from web browsers. It’s the successor to WebGL, offering a more flexible and powerful interface for rendering graphics and performing general-purpose computations (GPGPU) in web applications. WebGPU is built to leverage the capabilities of modern GPUs, enabling complex tasks like 3D rendering, machine learning inference, and scientific simulations, all within the browser.</p><p>WebGPU is supported in modern browsers like Chrome, Edge, and Safari (with ongoing adoption in Firefox), making it accessible for web developers.</p><h3>What is WASM</h3><p>WASM or WebAssembly is a binary instruction format that allows near-native performance for code running in the browser. It’s a low-level, portable compilation target for languages like C++, Rust, or Go, enabling complex computations that JavaScript alone would perform slowly. It executes in sandbox environment in browser for security and is optimized for CPU bound tasks. It supports deterministic, high-performance execution for tasks like agent logic.</p><h3>What is WebLLM</h3><p>WebLLM is based on WebAssembly (WASM) and WebGPU. WebLLM, developed by the MLC LLM team, leverages WebAssembly to enable large language models (LLMs) to run directly in the browser. WebLLM allows high-performance execution of code, often tapping into the device’s GPU for acceleration, which makes it possible to perform AI inference client-side without server dependencies. This is a core feature of WebLLM, enabling offline operation and privacy by keeping all computation in the browser.</p><p>WASM is used by WebLLM to handle parts of the AI model’s preprocessing, orchestration, and non-GPU computations. WebGPU is a web API that provides low-level access to a device’s GPU for rendering graphics and performing general-purpose computations (GPGPU). It’s designed for tasks like 3D rendering and machine learning inference.</p><h3>Putting all the Technologies Together</h3><p>Now we understand the foundational technologies for running LLM in browser, lets build a simple AI agent that runs completely offline in browser.</p><p>All the code is located at <a href="https://github.com/mngaonkar/ai-wasm">mngaonkar/ai-wasm</a></p><p>In this article, we’ll break down the concept step by step, explore how it works under the hood, and then dive into a high-level explanation of the code in each file. Whether you’re a developer curious about browser-based AI or just intrigued by offline tech, this repo offers a blueprint for the future of web apps.</p><h3>Main Web Page</h3><p>Location — <a href="https://github.com/mngaonkar/ai-wasm/blob/master/index.html">https://github.com/mngaonkar/ai-wasm/blob/master/index.html</a></p><p>The front end web page is a simple interface developed in HTML, JavaScript and CSS (Bootstrap). It consist of a 3D character (just added for fun and is optional) built using awesome three.js and main chat section to interact with agent. Additional features include voice input and output using Web Speech API and basic configuration to select voice profile, speed and pitch.</p><p>The GUI looks something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDNtsUsKjaRioIzVMitxXw.png" /></figure><h3>Application Logic</h3><p>Location — <a href="https://github.com/mngaonkar/ai-wasm/blob/master/app.js">https://github.com/mngaonkar/ai-wasm/blob/master/app.js</a></p><p>All the fun starts in init() function. It initialize 3D character, WebLLM, speech recognition, speech synthesizer and sets up basic event listeners.</p><p>For WebLLM initialization, SLMPhi-3.5-mini-instruct-q4f16_1-MLC</p><p>is hardcoded for demo and can be replaced with any suitable model. WebLLM initialization code creates an instance of LangGraph agent (explained in next section) and passes the WebLLM engine as an argument.</p><p>3D character initialization code creates a basic graphics by literally drawing the shapes and adding few funky math operation for animation. This part of code has a good scope of improvement to load a realistic graphical character for elevated entertainment value.</p><h3>Agent Logic</h3><p>Location — <a href="https://github.com/mngaonkar/ai-wasm/blob/master/langgraph-agent.js">https://github.com/mngaonkar/ai-wasm/blob/master/langgraph-agent.js</a></p><p>Agent logic primarily implements 2 classes namely WebLLMAdapter() to call underlying LLM and generate text and LangGraphAgent() to encapsulate agent Graph logic.</p><p>WebLLMAdapter implements only method named invoke() that calls this.engine.chat.completions.create() for interacting with LLM for text completion.</p><p>LangGraphAgent() defines a simple Graph having nodes namely processInput(), callLLM() and validateResponse(). Note, these are kind of dummy function except callLLM() and provide stubs for developer for more creative pre and post processing.</p><h3>Build and Run</h3><p>Application build and run configuration is based on NPM and follows standard process as follows:</p><ul><li>npm install</li><li>npm run dev (development build) OR</li><li>npm run build (production build)</li></ul><h3>Gotchas and Future Improvements</h3><p>The code provided is completely running code with model inference executed locally in browser leveraging GPU. But here are few notes for developers:</p><ul><li>Experiement with const MAX_TOKENS = 512 variable in <a href="https://github.com/mngaonkar/ai-wasm/blob/master/langgraph-agent.js">https://github.com/mngaonkar/ai-wasm/blob/master/langgraph-agent.js</a>. This is performance trade-off more the token generated, more time model will take to respond.</li><li>Application is tested on macOS M2, Safari and Chrome. Surprising, I tested WebLLM on my older iMac with Radeon Pro 580 8 GB and works perfectly great!</li><li>Experiment with other cooler quantized (preferrably 4 bits) models like SmolLM, Gemma and Llama for better performance</li><li>This uses a web variant of LangGraph langchain/langgraph/web that is compatible to run within browser</li><li>No tools support for Agent</li><li>System prompt is hardcoded but good scope to experiment for accurate results with models.</li><li>Models are stored in browser local storage one time and typically takes time first time (Look for Application-&gt;Cache storage in Chrome developer tools)</li><li>Voice quality is acceptable but scope for improvement with better text to speech implementation</li></ul><h3>Summary</h3><p>The code repository showcases a fusion of WebLLM, LangGraph, Three.js, and web APIs to create an offline AI chat app that’s private, interactive, and extensible. By running everything in the browser, it eliminates server reliance, making AI accessible on any device. Developers can learn from its modular design to build similar tools, while users enjoy a fun, animated companion.</p><p>If you’re inspired, clone the repo, fire up a local server, and experiment!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d92fb6e23c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Agentic AI That Lives in Your Browser]]></title>
            <link>https://medium.com/@mngaonkar/building-an-agentic-ai-that-lives-in-your-browser-0a5149cf6828?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a5149cf6828</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[chrome-extension]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 05:15:18 GMT</pubDate>
            <atom:updated>2025-09-17T05:21:43.654Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve ever wished you could ask a web page to “summarize yourself,” “extract the key steps,” or even “draw me a quick sequence diagram of what’s happening here,” this project is exactly that vibe (pun intended). <strong>ai-browser-assistant</strong> is a Chrome extension that injects an AI panel into any page, analyzes what you’re looking at, and lets you chat with the page context—plus a LangGraph-powered, multi-agent workflow to structure its reasoning. (<a href="https://github.com/mngaonkar/ai-browser-assistant">GitHub</a>)</p><h3>What it does</h3><ul><li>Slides in a <strong>right-side assistant panel</strong> on any page.</li><li><strong>Analyzes</strong> the current page and gives a quick summary, key topics, and insights.</li><li>Lets you <strong>chat</strong> with the page (“What does this API do?”, “Give me a TL;DR”).</li><li>Adds a <strong>LangGraph agent</strong> to classify your query (summary vs. search vs. explanation), keep conversation history, and guide the flow.</li><li>Can <strong>generate diagrams</strong> (like component/sequence/flow/class) from page content via a Mermaid tool sample.</li></ul><h3>Why it’s interesting</h3><p>Most “AI sidebars” are just chat boxes. This one leans into <strong>agentic patterns</strong>:</p><ul><li>It <strong>classifies your intent</strong> (search / analysis / summary / explanation / general) and adjusts the response style.</li><li>It <strong>tracks threads and context</strong> across messages, so your follow-ups stay grounded in the same page.</li><li>It keeps <strong>privacy boundaries</strong> by limiting how much page content gets shipped to the model (trimmed to ~5k chars) and storing API keys in Chrome’s sync storage.</li></ul><h3>Quickstart</h3><ol><li>npm install</li><li>npm run build</li><li>In Chrome: chrome://extensions → <strong>Developer mode</strong> → <strong>Load unpacked</strong> → choose dist/</li><li>Click the extension icon → add your OpenAI key (or compatible base URL) → <strong>Save</strong> → <strong>Test</strong>. (<a href="https://github.com/mngaonkar/ai-browser-assistant">GitHub</a>)</li></ol><blockquote><em>Uses Chat Completions (defaults shown in the repo are GPT-3.5-turbo + temperature 0.7</em></blockquote><h3>How the pieces fit (architecture)</h3><p><strong>Extension anatomy</strong>:</p><ul><li>content.js – Injects the assistant UI into every page and brokers page content.</li><li>background.js – Central router: receives messages, calls the model, manages state/threads.</li><li>popup.js – Tiny control surface for toggling the panel, analyzing the page, and storing config.</li><li>langgraph-agent.js – The agent brain: query classification, state machine, error handling, page-aware prompts.</li><li>manifest.json, content.css, popup.html, webpack.config.js – scaffolding &amp; build. (<a href="https://github.com/mngaonkar/ai-browser-assistant">GitHub</a>)</li></ul><p><strong>High-level flow</strong></p><ol><li>You open the popup → click <strong>Open Assistant</strong> (or <strong>Analyze Page</strong>).</li><li>Popup sends a message to the content script (toggle the panel, scrape safe content).</li><li>Content script packages page context (title, URL, sampled text) → sends to background.</li><li>Background hands it off to the <strong>LangGraph agent</strong>, which:</li></ol><ul><li>classifies your query,</li><li>builds the prompt with page context,</li><li>calls the LLM</li><li>returns a structured response (summary/steps/explanation/diagram).</li></ul><h3>1) Content script (content.js): inject &amp; listen</h3><p>A simple message bus connects popup → content → background. In the content script, a handler toggles the assistant UI and triggers page analysis:</p><pre>// Content script (simplified)<br>class BrowserAssistant {<br>  constructor() {<br>    this.isOpen = false;<br>    this.assistantPanel = null;<br>    this.init();<br>  }<br>  init() {<br>    chrome.runtime.onMessage.addListener((request, sender, sendResponse) =&gt; {<br>      if (request.action === &#39;toggleAssistant&#39;) {<br>        this.toggleAssistant();<br>        sendResponse({ success: true });<br>      } else if (request.action === &#39;analyzePage&#39;) {<br>        this.analyzeCurrentPage();<br>        sendResponse({ success: true });<br>      }<br>      return true; // keep channel open if needed<br>    });<br>    this.createAssistantPanel();<br>  }<br>  // ...createAssistantPanel(), toggleAssistant(), analyzeCurrentPage() etc.<br>}<br>new BrowserAssistant();</pre><p>This illustrates the pattern used by the project’s content layer: <strong>receive commands</strong>, <strong>inject UI</strong>, <strong>collect page text</strong> (safely), and <strong>forward it</strong> to the background for AI processing.</p><h3>2) Background script (background.js): the router</h3><p>The background worker receives messages like analyzePage, chatMessage, setApiKey etc. The message handling is part of following code:</p><pre>async handleMessage(request, sender, sendResponse) {<br>switch (request.action) {<br>// more code<br>case &#39;chatMessage&#39;:<br>          const response = await this.processChatMessage(request.message, request.pageData, request.threadId);<br>          sendResponse({ response: response });<br>          break;<br><br>}<br>// more code<br>}</pre><p>Following code routes the request to LangGraph agent.</p><pre>  async processChatMessage(message, pageData, threadId = null) {<br>    if (!this.apiKey) {<br>      return &#39;Please configure your API key in the extension settings.&#39;;<br>    }<br><br>    try {<br>      // Get or create agent for this thread<br>      let agent = this.agents.get(threadId);<br>      if (!agent) {<br>        // Import LangGraphAgent (assuming it&#39;s available)<br>        if (typeof LangGraphAgent === &#39;undefined&#39;) {<br>          // Fallback to simple response if LangGraphAgent not available<br>          return await this.simpleChatResponse(message, pageData);<br>        }<br>        <br>        console.log(&#39;LangGraphAssistant: Creating agent with params:&#39;, {<br>          apiKey: this.apiKey ? &#39;***&#39; + this.apiKey.slice(-4) : &#39;missing&#39;,<br>          baseUrl: this.baseUrl,<br>          langsmithApiKey: this.langsmithApiKey ? &#39;***&#39; + this.langsmithApiKey.slice(-4) : &#39;missing&#39;,<br>          langsmithProject: this.langsmithProject<br>        });<br>        <br>        agent = new LangGraphAgent(this.apiKey, this.baseUrl, this.langsmithApiKey, this.langsmithProject);<br>        await agent.initialize(pageData, threadId);<br>        this.agents.set(threadId, agent);<br>        console.log(&#39;LangGraphAssistant: Created new agent for thread:&#39;, threadId);<br>      }<br><br>      // Process message using the agent<br>      const response = await agent.processMessage(message);<br>      return response;<br>    } catch (error) {<br>      console.error(&#39;Error processing chat message:&#39;, error);<br>      return &#39;Sorry, I encountered an error processing your message. Please try again.&#39;;<br>    }<br>  }</pre><h3>3) The LangGraph agent (langgraph-agent.js): brains &amp; guardrails</h3><p>The LangGraph agent is implemented in langgraph-agent.js. The setupAgent() function creates 2 agents namely summerizeAgent and mermaidDiagramAgent with MermaidDiagramTool(). The agents are added in StateGraph as follows.</p><pre>this.workflow = new StateGraph(MessagesAnnotation)<br>  .addNode(&quot;mermaid_diagram_agent&quot;, this.mermaidDiagramAgent)<br>  .addNode(&quot;summerize_agent&quot;, this.summerizeAgent)<br>  .addEdge(&quot;__start__&quot;, &quot;summerize_agent&quot;)<br>  .addEdge(&quot;summerize_agent&quot;, &quot;mermaid_diagram_agent&quot;)<br>  .addEdge(&quot;mermaid_diagram_agent&quot;, &quot;__end__&quot;);<br><br>this.supervisor = this.workflow.compile();<br>console.log(&#39;LangGraphAgent: Supervisor created successfully&#39;);</pre><p>The user message is processed with processMessage() function as follows.</p><pre>async processMessage(userMessage) {<br>// more code<br><br>// Create the message with page context<br>const contextualMessage = this.createContextualMessage(userMessage);<br><br>// more code<br>if (this.supervisor) {<br>  try {<br>    // First try the standard invoke method<br>    const invokeOptions = {<br>      messages: [new HumanMessage(contextualMessage)]<br>    };<br>    <br>    // Add tracer if available<br>    if (this.tracer) {<br>      invokeOptions.callbacks = [this.tracer];<br>    }<br>    invokeOptions.verbose = true;<br>    <br>    console.log(&#39;LangGraphAgent: Invoking agent with options:&#39;, invokeOptions);<br>    result = await this.supervisor.invoke(invokeOptions);<br>  } catch (invokeError) {<br>    console.warn(&#39;LangGraphAgent: Standard invoke failed, trying alternative method:&#39;, invokeError);<br>  }<br>} else {<br>  console.error(&#39;LangGraphAgent: Using LLM directly (agent is null)&#39;);<br>}</pre><h3>Mermaid diagram tool</h3><p>A neat trick is using Mermaid to turn natural language → <strong>component/sequence/flow/class</strong> diagrams generated from the current page’s content. For example, if you ask:</p><blockquote><em>“Make a sequence diagram of the login flow described on this page.”</em></blockquote><p>…the agent can extract the relevant bits and output:</p><pre>sequenceDiagram<br>    participant User<br>    participant UI<br>    participant EventStream<br>    participant DataProcessor<br>    participant Database<br>    User-&gt;&gt;UI: Input event<br>    UI-&gt;&gt;EventStream: Publish event<br>    EventStream-&gt;&gt;DataProcessor: Stream event<br>    DataProcessor-&gt;&gt;Database: Aggregate data<br>    Database--&gt;&gt;DataProcessor: Acknowledge aggregation<br>    DataProcessor--&gt;&gt;EventStream: Trigger updates<br>    EventStream--&gt;&gt;UI: Push updates<br>    UI--&gt;&gt;User: Display real-time data</pre><h3>Security &amp; privacy notes</h3><ul><li><strong>Keys</strong> live in <strong>Chrome sync storage</strong>;</li><li>Only <strong>OpenAI (or compatible)</strong> endpoints receive data;</li><li>Page content is <strong>trimmed/limited</strong> (≈ 5k chars) to reduce exposure.</li></ul><h3>Dev tips</h3><ul><li>Use the <strong>Troubleshooting</strong> section (panel not opening, API errors, storage issues) and <strong>Debug mode</strong> (DevTools console + Network tab) when hacking.</li><li>Packaged via Webpack; npm run package builds a ZIP for distribution.</li></ul><h3>Where to take it next</h3><ul><li><strong>Task plugins</strong>: add tools like “extract table → CSV,” “find dates/addresses,” or “autorun form fills.”</li><li><strong>Richer retrieval</strong>: vector-search the page’s DOM (or multi-page sessions) for better grounding.</li><li><strong>Model-agnostic</strong>: wire up Azure/OpenRouter/self-hosted gateways and add a UI toggle.</li><li><strong>Voice</strong>: mic in, TTS out, for hands-free reading of long docs.</li></ul><h3>Closing thought</h3><p>ai-browser-assistant is a clean, pragmatic example of <strong>agentic UX in the browser</strong>—not just a chat box, but a <strong>guided reasoning flow</strong> that understands what you asked, why you asked, and what the page can offer. Clone it, point it at your model endpoint, and make the web feel a little more… conversational.</p><p>The source code is available at <a href="https://github.com/mngaonkar/ai-browser-assistant">https://github.com/mngaonkar/ai-browser-assistant</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a5149cf6828" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing a Simple Recurrent Neural Network (RNN) from Scratch Using MLX]]></title>
            <link>https://medium.com/@mngaonkar/implementing-a-simple-recurrent-neural-network-rnn-from-scratch-using-mlx-3cea24b2e754?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cea24b2e754</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[apple-silicon]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 04:56:30 GMT</pubDate>
            <atom:updated>2025-07-30T04:56:30.167Z</atom:updated>
            <content:encoded><![CDATA[<p>Recurrent Neural Networks (RNNs) are a cornerstone of sequence modeling in machine learning, powering applications like language generation, time-series forecasting, and speech recognition. Unlike feedforward networks, RNNs maintain a “memory” of previous inputs through hidden states, making them ideal for handling sequential data. In this article, we’ll dive into a from-scratch implementation of a simple RNN using Apple’s MLX framework, as demonstrated in the attached Jupyter notebook. This implementation trains the RNN on Arthur Conan Doyle’s <em>The Sign of the Four</em> to predict the next word in a sequence, showcasing core concepts like forward propagation, backpropagation through time (BPTT), and text generation.</p><p>We’ll break down the code step by step, explaining the key classes and functions. By the end, you’ll understand how to build and train a basic multi-layer RNN without relying on high-level libraries like PyTorch or TensorFlow. Note: This RNN uses a vanilla architecture, which can suffer from vanishing gradients, but it’s perfect for educational purposes.</p><h3>Setting Up the Environment</h3><p>The notebook starts by importing MLX, a lightweight deep learning framework optimized for Apple silicon:</p><pre>import mlx.core as mx<br>import mlx<br>mx.set_default_device(mx.gpu)</pre><p>MLX provides array operations similar to NumPy but with GPU acceleration. We also define a custom softmax function for numerical stability:</p><pre>def softmax(x):<br>    e_x = mx.exp(x - mx.max(x, axis=0, keepdims=True))<br>    return e_x / e_x.sum(axis=0, keepdims=True)</pre><p>This subtracts the maximum value in each column to prevent overflow during exponentiation.</p><h3>Preparing the Data: The Corpus Class</h3><p>Data preparation is crucial for RNNs, as they operate on sequences. The Corpus class loads text from a file (e.g., ‘sign_of_four.txt’), builds a vocabulary, and provides utilities like one-hot encoding and word sampling.</p><p>Key features:</p><ul><li><strong>Vocabulary Building</strong>: Splits the text into words and creates a unique set (self.vocab).</li><li><strong>One-Hot Encoding</strong>: Converts words to one-hot vectors (e.g., one_hot(word) returns a vector with 1 at the word’s index).</li><li><strong>Batching</strong>: get_batched_data(batch_size) yields sequences for training.</li><li><strong>Sampling</strong>: sample_word(y_hat, temperature) samples the next word from predicted probabilities, with temperature controlling randomness (lower values make it more deterministic).</li><li><strong>Cooccurrence Matrix</strong> (optional, computed on demand): Builds a matrix for word embeddings based on context windows, though it’s not heavily used in training here.</li></ul><p>Example usage:</p><pre>corpus = Corpus(file_name=&#39;sign_of_four.txt&#39;)<br>print(len(corpus.vocab))  # Vocabulary size: 9202<br>one_hot_vec = corpus.one_hot(&quot;Holmes&quot;)</pre><p>The vocabulary size (9202 unique words) determines the input and output dimensions of the RNN.</p><h3>Building the RNN Cell</h3><p>The RNNCell class implements a single RNN unit. It handles forward passes for sequences, backward passes for gradients, and weight updates.</p><h3>Initialization</h3><p>Weights are initialized using Xavier/Glorot method for stability:</p><pre>scale = mx.sqrt(1.0 / (input_size + output_size))<br>self.Wh = mx.random.normal(shape=(hidden_size, hidden_size), scale=scale)<br>self.Wx = mx.random.normal(shape=(hidden_size, input_size), scale=scale)<br>self.Wy = mx.random.normal(shape=(output_size, hidden_size), scale=scale)<br>self.bh = mx.zeros((hidden_size, 1))<br>self.by = mx.zeros((output_size, 1))</pre><ul><li>Wh: Hidden-to-hidden weights.</li><li>Wx: Input-to-hidden weights.</li><li>Wy: Hidden-to-output weights.</li><li>Biases: bh and by.</li></ul><h3>Forward Pass</h3><p>Processes a sequence of inputs, updating hidden states with tanh activation:</p><pre>def forward(self, x):<br>    for t, xt in enumerate(x):<br>        self.x_t[t] = xt<br>        if t &gt; 0:<br>            self.h_t[t] = self.tanh(self.Wx @ self.x_t[t] + self.Wh @ self.h_t[t-1] + self.bh)<br>        else:<br>            self.h_t[t] = self.tanh(self.Wx @ self.x_t[t] + self.bh)<br>        y_hat_temp = self.Wy @ self.h_t[t] + self.by<br>        self.y_hat[t] = softmax(y_hat_temp)<br>    return self.y_hat</pre><p>This computes hidden states (h_t) and softmax outputs (y_hat) for each time step.</p><h3>Backward Pass (BPTT)</h3><p>Computes gradients using backpropagation through time, with clipping to prevent exploding gradients:</p><pre>def backward(self, y, top_layer=False):<br>    for t in reversed(range(len(self.y_hat))):<br>        self.dy = self.y_hat[t] - y[t]<br>        self.dWy += self.dy @ self.h_t[t].T<br>        self.dby += self.dy<br>        self.dh = self.Wy.T @ self.dy + self.dh_next<br>        self.dh_raw = (1 - self.h_t[t] ** 2) * self.dh  # Tanh derivative<br>        self.dWx += self.dh_raw @ self.x_t[t].T<br>        if t &gt; 0:<br>            self.dWh += self.dh_raw @ self.h_t[t-1].T<br>        self.dbh += self.dh_raw<br>        self.dh_next = self.Wh.T @ self.dh_raw<br>    self.clip_gradients(max_norm=5)<br>    return self.dWx, self.dWh, self.dWy, self.dbh, self.dby</pre><p>Gradients are accumulated and clipped between -5 and 5.</p><h3>Weight Update and Loss</h3><p>Simple gradient descent:</p><pre>def step(self, learning_rate=0.01):<br>    self.Wh -= learning_rate * self.dWh<br>    # ... (similar for other weights)</pre><p>Cross-entropy loss for sequences:</p><pre>def sequence_loss(self, y_hat, y):<br>    loss_value = 0<br>    for y_i, y_i_hat in zip(y, y_hat):<br>        loss_value += mx.sum(-1 * y_i * mx.log(y_i_hat + 1e-12))<br>    return loss_value / len(y_hat)</pre><h3>Stacking Layers: The RNN Class</h3><p>The RNN class creates a multi-layer RNN by stacking RNNCell instances:</p><pre>class RNN:<br>    def __init__(self, input_size, hidden_size, output_size, num_layers=1):<br>        self.cells = [RNNCell(input_size, hidden_size, output_size) for _ in range(num_layers)]<br>    <br>    def forward(self, x):<br>        for cell in self.cells:<br>            cell.zero_state()<br>            x = cell.forward(x)<br>            x = [x[t] for t in range(len(x))]  # Outputs become inputs for next layer<br>        return self.cells[-1].y_hat.values()<br>    <br>    def backward(self, y):<br>        self.cells[-1].zero_grad()<br>        dWx, dWh, dWy, dbh, dby = self.cells[-1].backward(y, top_layer=True)<br>        for i in reversed(range(len(self.cells) - 1)):<br>            self.cells[i].zero_grad()<br>            self.cells[i].backward(self.cells[i+1].x_t)</pre><p>Forward passes chain outputs between layers. Backward starts from the top layer and propagates errors downward.</p><p>In the notebook, we initialize a 6-layer RNN:</p><pre>NUM_LAYERS = 6<br>rnn = RNN(input_size=corpus.get_dim(), hidden_size=512, output_size=corpus.get_dim(), num_layers=NUM_LAYERS)</pre><h3>Training the Model</h3><p>Training uses sequence length 8 and processes batches:</p><pre>SEQ_LENGTH = 8<br>batch_data = corpus.get_batched_data(batch_size=SEQ_LENGTH)<br><br># In the loop (simplified):<br>train_data = next(batch_data)<br>xt = [corpus.one_hot(x).T for x in train_data[:SEQ_LENGTH-1]]<br>y_hat = rnn.forward(xt)<br>y = [corpus.one_hot(word).T for word in train_data[1:SEQ_LENGTH]]<br>loss_value = rnn.sequence_loss(y_hat, y)<br>rnn.zero_grad()<br>rnn.backward(y)<br>rnn.step(learning_rate=0.005)</pre><p>The notebook runs multiple epochs, tracking loss and plotting it with Matplotlib. Losses decrease over time, indicating learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*2ttF4VUYk60wDOZoDiq8Qw.png" /></figure><h3>Generating Text</h3><p>After training, the model generates text by sampling predictions:</p><pre>input_seq = [&quot;It&quot;, &quot;is&quot;, &quot;of&quot;]<br>for i in range(10):<br>    xt = [corpus.one_hot(word).T for word in input_seq]<br>    y_hat = rnn.forward(xt)<br>    index = corpus.sample_word(mx.array(list(y_hat)[-1]).squeeze(), temperature=0.1)<br>    next_word = corpus.ix_to_word[index]<br>    input_seq.append(next_word)</pre><pre>[&#39;It&#39;, &#39;is&#39;, &#39;of&#39;, &#39;delay&#39;, &#39;tongue,&#39;, &#39;dips&#39;, &#39;house.”&#39;, &#39;Watson!&#39;, &#39;allusion&#39;, &#39;made!&#39;, &#39;meekly&#39;, &#39;dogs&#39;, &#39;utter.&#39;]</pre><p>The generated text is coherent but nonsensical, typical for a simple RNN on limited data. Temperature values is important for RNN to create more meaningful text.</p><h3>Conclusion</h3><p>This from-scratch RNN implementation highlights the mechanics of recurrent networks: hidden state updates, sequence loss, and gradient flow through time. Using MLX keeps it efficient on GPUs, while the modular design (Corpus, RNNCell, RNN) makes it easy to extend — perhaps to LSTMs or GRUs for better performance.</p><p>Challenges include vanishing gradients (mitigated somewhat by tanh and clipping) and the small dataset, leading to limited generation quality. Experiment by increasing layers, hidden size, or using a larger corpus like Shakespeare’s works (commented in the notebook).</p><p>For the full code and experimentation, check the Jupyter notebook here <a href="https://github.com/mngaonkar/micrograd/blob/main/rnn-from-scratch.ipynb">https://github.com/mngaonkar/micrograd/blob/main/rnn-from-scratch.ipynb</a>.</p><p>If you’re new to RNNs, this is a great hands-on project to solidify your understanding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cea24b2e754" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommendation Engine based on AI Embeddings]]></title>
            <link>https://medium.com/@mngaonkar/recommendation-engine-based-on-ai-embeddings-27447f280c98?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/27447f280c98</guid>
            <category><![CDATA[embedding-model]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aritificial-intelligence]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Sat, 13 Jul 2024 22:10:09 GMT</pubDate>
            <atom:updated>2024-07-14T17:56:01.921Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSyOrdf33-9q3AmuUNrdIg.png" /></figure><p>This is a weekend project to build a movie recommendation engine using text embedding models. The goal is to build a website where a user can provide a brief plot description of the movies and the engine will show matching movies.</p><p><strong>Here is link to the final website </strong><a href="https://movies.altbox.one/"><strong>https://movies.altbox.one</strong></a><strong>.</strong> Note, this website will be up for limited time for demonstration purpose.</p><p><strong>The source code is available at — </strong><a href="https://github.com/mngaonkar/flix-finder"><strong>https://github.com/mngaonkar/flix-finder</strong></a></p><h3>Data Preparation</h3><p>As any AI project, everything starts with quality data. I wanted movie data that is comprehensive and with permissive licensing. After a bit of research on dataset available from IMDB, TMDB and Kaggle, I settled on Wikipedia as source of movie database.</p><p>Wikipedia does not provide movie database directly hence it was necessary to get a dump of entire wikipedia and extract movie related articles.</p><p>The Wikipedia dump is available at — <a href="https://dumps.wikimedia.org/enwiki/latest/">https://dumps.wikimedia.org/enwiki/latest/</a></p><p>Download <a href="https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2">https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2</a>. This is a compressed XML file having all Wikipedia articles.</p><p>To extract only movies article, we will need index file located at —<a href="https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream-index.txt.bz2"> </a><a href="https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream-index.txt.bz2">https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream-index.txt.bz2</a>. This files contains offset in above compressed Wikidump for a specific article.</p><p>Run <strong>tools/parse_movie_index.py</strong> to get movie_index.txt that has only movie related article indices. Note, change WIKI_INDEX_FILE to point to extracted index file.</p><p>Run <strong>tools/process_wiki_dump.py </strong>to create movies.csv dataset. Note, change WIKI_DUMP_FILE and WIKI_INDEX_FILE to point to correct compressed dump and movie_index.txt file</p><p>If everything goes well, you should have the final movie dataset <strong>movies.csv</strong>. This dataset has movie id, title, cast, plot and poster link. The movie dataset has around 15000 movies information.</p><h3>Embedding Model</h3><p>An embedding model maps discrete objects (like words, items, or nodes) into continuous vector spaces, preserving the semantic relationships among them. This process is known as embeddings. Embedding models are available for text, image and audio data. For movie recommendation, we are going to use text embedding model as we will find similar movies based on movies plots and cast.</p><p>One of the high quality model available for text embedding is sentence-transformers/all-mpnet-base-v2 (<a href="https://huggingface.co/sentence-transformers/all-mpnet-base-v2">https://huggingface.co/sentence-transformers/all-mpnet-base-v2</a>). This model is provided by <a href="https://www.sbert.net/">https://www.sbert.net/</a></p><p>Here is quick model comparison — <a href="https://www.sbert.net/docs/sentence_transformer/pretrained_models.html#original-models">Original Models</a></p><p>In summary, <a href="https://huggingface.co/sentence-transformers/all-mpnet-base-v2"><strong>all-mpnet-base-v2</strong></a><strong> </strong>offers the best quality and <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2"><strong>all-MiniLM-L6-v2</strong></a><strong> </strong>offers the best performance.</p><p><strong>backend/database.py</strong> has the embedding model selection code as follows:</p><pre>class Database():<br>    def __init__(self, db_name = &quot;milvus_demo.db&quot;, collection_name = &quot;milvus_demo&quot;):<br>        &quot;&quot;&quot;Create a vector database with a single text.&quot;&quot;&quot;<br>        self.db_name = db_name<br>        self.collection_name = collection_name<br>        model_name = &quot;sentence-transformers/all-mpnet-base-v2&quot;<br>        model_kwargs = {&#39;device&#39;: &#39;cpu&#39;}<br>        encode_kwargs = {&#39;normalize_embeddings&#39;: False}<br>        self.embeddings = HuggingFaceEmbeddings(<br>            model_name=model_name,<br>            model_kwargs=model_kwargs,<br>            encode_kwargs=encode_kwargs<br>        )</pre><p>Update model_name to select appropriate embedding model.</p><h3>Vector Database</h3><p>A vector database is a specialized database designed to efficiently store, index, and query high-dimensional vectors. These vectors are often used to represent complex data, such as text embeddings, image features, and other forms of numerical representations derived from machine learning models. Vector databases are particularly useful for tasks like similarity search, clustering, and nearest neighbor search.</p><p>There are different vector databases to choose from. Few of the notable vector database are Milvus, Pinecone, FAISS, Chroma.</p><p>For movie recommendation engine, we will use Milvus. The database related code is in <strong>backend/database.py and backend/vectorstore.py</strong></p><p>On start of the application, movies.csv file is read and each entry is converted to an vector embedding using selected embdeding model. The final embeddings are saved in <strong>movies.db</strong> as persistent vector database. The conversion to embedding is done once and next time the vector database is initialized from the movies.db.</p><p>Note, depending on compute resources, embedding creation and loading in vector database can take considerable time. Also, I have observed that movies.db is not really a portable database that we can move around the machines. May be it’s dependent on specific version Milvus libraries hence I would recommend to generate movies.db afresh on new setup.</p><h3>AI Pipeline</h3><p>For building the entire pipeline, we will be using Langchain for integrating embedding models from HuggingFace, CSVloader for loading movie dataset and streamlit as our graphical user interface.</p><p>All frontend code is in <strong>frontend/gui.py</strong> and backend code is in <strong>backend/database.py</strong>, <strong>backend/vectorstore.py</strong> and <strong>backend/loader.py</strong></p><h3>Deployment</h3><p>Clone the repository on Linux (tested on Ubuntu 20.x) or macOS (tested on Sonoma 14.5 — Apple Silicon M2) machine.</p><p>It is highly recommend to use Python virtual environment like Conda for deployment. The code is tested on <strong>Python 3.12.4</strong></p><p>Run following to install dependencies:</p><pre>pip install -r requirements.txt</pre><p>Note, you may need minor modification especially regarding package version upgrade/downgrade depending on Python environment.</p><p>Run following to start the application:</p><pre>./run.sh</pre><pre>/root/miniconda3/bin/conda run -n langchain streamlit run --ui.hideTopBar True --client.toolbarMode viewer main.py --server.port 443 --server.sslCertFile /etc/letsencrypt/live/movies.altbox.one/fullchain.pem --server.sslKeyFile /etc/letsencrypt/live/movies.altbox.one/privkey.pem</pre><p>You may want to use following version of run.sh if not planning to use HTTPS endpoint.</p><pre>/root/miniconda3/bin/conda run -n langchain streamlit run --ui.hideTopBar True --client.toolbarMode viewer main.py --server.port 80</pre><p>Note, you can use awesome and free <a href="https://letsencrypt.org/">https://letsencrypt.org/</a> for setting up HTTPS endpoint.</p><p>Feel free to send me feedback or suggestions on Twitter handle @mngaonkar</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27447f280c98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning on tiny hardware]]></title>
            <link>https://medium.com/@mngaonkar/machine-learning-on-tiny-hardware-77fa93524192?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/77fa93524192</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-processing-unit]]></category>
            <category><![CDATA[ai-hardware]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Mon, 01 Jul 2024 08:59:26 GMT</pubDate>
            <atom:updated>2024-07-27T05:49:23.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GB4-f6sXKtJU1_ew.jpg" /></figure><p>Here is my weekend experiment to run a tiny machine learning model with 20 dollar hardware.</p><h3>Luckfox Pico Max Setup</h3><p>Following steps will need a Linux x86 machine or WSL on Windows. I have tested on Ubuntu WSL on Windows 11.</p><p>Download Luckfox SDK</p><p><a href="https://github.com/LuckfoxTECH/luckfox-pico">GitHub - LuckfoxTECH/luckfox-pico: luckfox-pico sdk</a></p><blockquote>git clone <a href="https://github.com/LuckfoxTECH/luckfox-pico.git">https://github.com/LuckfoxTECH/luckfox-pico.git</a></blockquote><p>Setup the build environment variables:</p><blockquote>cd {SDK_PATH}/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/<br>source env_install_toolchain.sh</blockquote><p>Setup hardware:</p><blockquote>sudo ./build.sh launch</blockquote><p>Remember to choose the correct board. For latest Ultra and Max board its RV1106 and for older board RV1103. Mine was Pico Max hence I selected “[4] RV1106_Luckfox_Pico_Pro_Max”. <strong>Also select build root configuration and NOT Ubuntu</strong>. This is important as Ubuntu does not support NPU and CSI camera as of this writing.</p><p><strong><em>Pitfall:</em></strong></p><p><em>If you select older board RV1103 ethernet port won’t work as it has no ethernet hardware.</em></p><p>Build the firmware with following command:</p><blockquote>sudo ./build.sh</blockquote><p>The output images are generated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*XO_J6oGo9iy7iPse-uV4Kw.png" /></figure><p><strong><em>Pitfall:</em></strong></p><p><em>The root file system image in firmware is default to 6G. Update highlighed size in config file to fit your micro SD card size.</em></p><p>luckfox-pico/project/cfg/BoardConfig_IPC/BoardConfig-SD_CARD-Ubuntu-RV1106_Luckfox_Pico_&lt;board version&gt;-IPC.mk</p><p>export RK_PARTITION_CMD_IN_ENV=”32K(env),512K@32K(idblock),256K(uboot),32M(boot),512M(oem),256M(userdata),<strong>32G</strong>(rootfs),-(media)”</p><p>Download Luckfox firmware flashing tool here. <a href="https://files.luckfox.com/wiki/Luckfox-Pico/Software/SocToolKit.zip">https://files.luckfox.com/wiki/Luckfox-Pico/Software/SocToolKit.zip</a></p><p>Use SDTool to burn firmware on micro SD card. First you have to select all firmware files from output/image folder except update.img then click Create SD to flash the firmware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*7L7AC2R2rPZQPlSlJIpS_g.png" /></figure><p><strong><em>Pitfall:</em></strong></p><p><em>You may not able to see the micro SD card listed in Usb Disk. In that case, remove the USB cable and re-connect. Also, make sure you have correct driver installed as described here </em><a href="https://wiki.luckfox.com/Luckfox-Pico/Luckfox-Pico-prepare">https://wiki.luckfox.com/Luckfox-Pico/Luckfox-Pico-prepare</a></p><p>Put the micro SD card in Luckfox hardware, connect LAN cable and power up with USB port. It should get an IP address from DHCP and you should be able to login with username pico and password luckfox.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AM3k6i7A4wppbVowOZgYXQ.png" /></figure><p>The above process sometime may throw unanticipated issue, refer <a href="https://wiki.luckfox.com/Luckfox-Pico/Luckfox-Pico-quick-start">https://wiki.luckfox.com/Luckfox-Pico/Luckfox-Pico-quick-start</a> for solutions to common challenges.</p><h3>RKNN Setup</h3><p><strong>Note, following steps to be run on a Linux x86 machine and not on Luckfox Pico device. I have tested following with WSL on Windows 11.</strong></p><p>Download RKNN toolkit</p><blockquote>git clone <a href="https://github.com/rockchip-linux/rknn-toolkit2">https://github.com/rockchip-linux/rknn-toolkit2</a></blockquote><p>Install dependencies as follows. Pick up the right requirements file based on installed Python version.</p><blockquote>pip3 install -r rknn-toolkit2/packages/requirements_cp311–2.0.0b0.txt</blockquote><p><strong><em>Pitfall:</em></strong></p><p><em>You may see issues related to package dependencies. Remove/re-install appropriate package versions. I had to downgrade NumPy to 1.23.x version from 2.x.</em></p><p>Install RKNN toolkit as follows:</p><blockquote>pip install rknn_toolkit2–2.0.0b0+9bab5682-cp311-cp311-linux_x86_64.whl</blockquote><p>If all installation goes well, you should be able to import RKNN package as follows:</p><blockquote>Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux<br>Type “help”, “copyright”, “credits” or “license” for more information.<br>&gt;&gt;&gt; from rknn.api import RKNN<br>&gt;&gt;&gt;</blockquote><h3>Model Setup</h3><p>Cloen model zoo repository as follows:</p><blockquote>git clone <a href="https://github.com/airockchip/rknn_model_zoo.git">https://github.com/airockchip/rknn_model_zoo.git</a></blockquote><p>Download Yolo model.</p><blockquote>cd rknn_model_zoo/examples/yolov5/model</blockquote><blockquote>./download_model.sh</blockquote><p>The above script downloads a default model. If you want a specific Yolo model, you can download directly from <a href="https://github.com/airockchip/rknn_model_zoo/tree/main?tab=readme-ov-file#model-support">https://github.com/airockchip/rknn_model_zoo/tree/main?tab=readme-ov-file#model-support</a></p><blockquote>wget <a href="https://ftrg.zbox.filez.com/v2/delivery/data/95f00b0fc900458ba134f8b180b3f7a1/examples/yolov8/yolov8n.onnx">https://ftrg.zbox.filez.com/v2/delivery/data/95f00b0fc900458ba134f8b180b3f7a1/examples/yolov8/yolov8n.onnx</a></blockquote><p>Convert model to RKNN format.</p><blockquote>mahadev@Mahadev-PC:~/rknn_model_zoo/examples/yolov5/python$ python3 ./convert.py ../model/yolov5n.onnx rv1106<br>W __init__: rknn-toolkit2 version: 1.6.0+81f21f4d<br> → Config model<br>done<br> → Loading model<br>W load_onnx: It is recommended onnx opset 19, but your onnx model opset is 12!<br>W load_onnx: Model converted from pytorch, ‘opset_version’ should be set 19 in torch.onnx.export for successful convert!<br>Loading : 100%|████████████████████████████████████████████████| 136/136 [00:00&lt;00:00, 72907.12it/s]<br>done<br> → Building model<br>W build: found outlier value, this may affect quantization accuracy<br>const name abs_mean abs_std outlier value<br>model.0.conv.weight 2.44 2.47 -17.494<br>model.22.cv3.2.1.conv.weight 0.09 0.14 -10.215<br>model.22.cv3.1.1.conv.weight 0.12 0.19 13.361, 13.317<br>model.22.cv3.0.1.conv.weight 0.18 0.20 -11.216<br>GraphPreparing : 100%|█████████████████████████████████████████| 161/161 [00:00&lt;00:00, 14893.76it/s]<br>Quantizating : 100%|██████████████████████████████████████████████| 161/161 [00:02&lt;00:00, 70.71it/s]<br>W build: The default input dtype of ‘images’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘318’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘onnx::ReduceSum_326’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘331’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘338’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘onnx::ReduceSum_346’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘350’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘357’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘onnx::ReduceSum_365’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>W build: The default output dtype of ‘369’ is changed from ‘float32’ to ‘int8’ in rknn model for performance!<br> Please take care of this change when deploy rknn model with Runtime API!<br>done<br> → Export rknn model<br>done</blockquote><h3>Build Code</h3><p>Luckfox RKNN SDK supports inference with C/C++ only. No Python support as of this writing.</p><p>Here are the steps to build Yolov5 test code.</p><p>Set GCC compiler path as per Luckfox SDK installation directory.</p><blockquote>export GCC_COMPILER=/home/mahadev/luckfox-pico/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf</blockquote><p>Change to model zoo directory and run build command.</p><blockquote>cd /home/mahadev/rknn_model_zoo</blockquote><blockquote>./build-linux.sh -t rv1106 -a armv7l -d yolov5</blockquote><p>This will build the binaries in following folder.</p><p>/home/mahadev/rknn_model_zoo/install/rv1106_linux_arm71/rknn_yolov5_demo</p><blockquote>mahadev@Mahadev-PC:~/rknn_model_zoo/install/rv1106_linux_arm71/rknn_yolov5_demo$ ls<br>lib model rknn_yolov5_demo</blockquote><h3>Run Code</h3><p>Copy the entire folder rknn_yolov5_demo onto Luckfox Pico max board and run.</p><blockquote>[root@luckfox rknn_yolov5_demo]$ ./rknn_yolov5_demo model/yolov5.rknn model/bus.jpg<br>load lable ./model/coco_80_labels_list.txt<br>model input num: 1, output num: 3<br>input tensors:<br> index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922<br>output tensors:<br> index=0, name=output0, n_dims=4, dims=[1, 80, 80, 255], n_elems=1632000, size=1632000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003917<br> index=1, name=343, n_dims=4, dims=[1, 40, 40, 255], n_elems=408000, size=408000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922<br> index=2, name=345, n_dims=4, dims=[1, 20, 20, 255], n_elems=102000, size=102000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922<br>input_attrs[0].size_with_stride=1228800<br>model is NHWC input fmt<br>model input height=640, width=640, channel=3<br>origin size=640x640 crop size=640x640<br>input image: 640 x 640, subsampling: 4:2:0, colorspace: YCbCr, orientation: 1<br>scale=1.000000 dst_box=(0 0 639 639) allow_slight_change=1 _left_offset=0 _top_offset=0 padding_w=0 padding_h=0<br>src width=640 height=640 fmt=0x1 virAddr=0x0xa5e8c000 fd=11<br>dst width=640 height=640 fmt=0x1 virAddr=0x(nil) fd=3<br>src_box=(0 0 639 639)<br>dst_box=(0 0 639 639)<br>color=0x72<br>rga_api version 1.10.1_[0]<br>rknn_run<br>person @ (209 243 287 510) 0.808<br>person @ (110 233 203 528) 0.806<br>person @ (483 223 564 513) 0.628<br>bus @ (118 144 561 452) 0.525<br>write_image path: out.png width=640 height=640 channel=3 data=0xa5e8c000</blockquote><p>The processed image with object detected is created in current folder as out.png</p><blockquote>[root@luckfox rknn_yolov5_demo]$ ls<br>lib model <strong>out.png</strong> rknn_yolov5_demo</blockquote><p>Great learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77fa93524192" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design patterns in Go — 1]]></title>
            <link>https://medium.com/@mngaonkar/design-patterns-in-go-1-1f12340af82d?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f12340af82d</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Tue, 09 Nov 2021 03:53:46 GMT</pubDate>
            <atom:updated>2021-11-09T04:13:39.437Z</atom:updated>
            <content:encoded><![CDATA[<h3>Design patterns in Go — 1</h3><p>Before we jump the coding, it is always good to visualize the problem statement from the glasses of a design pattern. It helps build more structured and maintainable code. Go is not strictly object oriented language but has powerful features to implement most of the design patterns. This article is kind of reference notes to developer including myself on widely used design patterns. Here are few popular design patterns in Go programming to get started.</p><h3>Visitor</h3><p>Visitor pattern allows excellent code decoupling where a logical piece of code can evolve independently. This is my favorite and mostly used pattern for structuring the code. In the example, <em>Player </em>and <em>Scorer</em> are different code structures that are loosely coupled. <em>Player</em> can make use of different <em>Scorer</em> for scoring strategy. Note, <em>Scorer </em>can<em> </em>be changed or extended independent of <em>Player</em>. The <em>Scorer</em> is passed to <em>Player</em> at runtime as per required scoring strategy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b9631e276c278cc27dc756de7683665/href">https://medium.com/media/8b9631e276c278cc27dc756de7683665/href</a></iframe><h3>Observer</h3><p>Observer is a publisher-subscriber design pattern. This pattern is widely used in asynchronous message communication where events are sent from publisher to subscriber. In the following example, <em>EventSource</em> is code structure that maintains a list of registered <em>Consumers</em> and communicate messages to these <em>Consumers</em>. In micro-services architecture, instead of directly calling a <em>Consumer</em> function it typically sends the message over a channel connected to a <em>Consumer</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6193f153ad563acb6b48df2e7591f31b/href">https://medium.com/media/6193f153ad563acb6b48df2e7591f31b/href</a></iframe><h3>Adapter</h3><p>This is one of the simplest design pattern that matches one interface to another. Here is an example that demonstrates Adapter pattern by coding real world electric adapters for the US and Europe. This pattern simply converts one interface to another with optional processing if required.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3cedf0df67daf1c016fd7656b95b778/href">https://medium.com/media/c3cedf0df67daf1c016fd7656b95b778/href</a></iframe><p>Hope this helps to understand basic design patterns in Go.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f12340af82d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metaverse Investing]]></title>
            <link>https://medium.com/@mngaonkar/metaverse-investing-a9f434127526?source=rss-453d3a9e4752------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9f434127526</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Mahadev Gaonkar]]></dc:creator>
            <pubDate>Sun, 24 Oct 2021 01:06:14 GMT</pubDate>
            <atom:updated>2021-10-25T04:55:36.599Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r8KqgmLueOyRwUBo" /><figcaption>Photo by <a href="https://unsplash.com/@ibrahimboran?utm_source=medium&amp;utm_medium=referral">Ibrahim Boran</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the introductory article of <a href="https://medium.com/@mngaonkar/metaverse-e2e098e105ec">Metaverse</a> I discussed the basic idea of Metaverse and in this article I aim to provide more details on companies that are actively working on this technology. The content here would be more business oriented than technology and if you are interested in investing in this new technology, you are at right place, so read on.</p><p>The idea of Metaverse is closely related to key technologies like Internet, Blockchain, and Gaming. Hence to explore the investing opportunities, we need to have a look at the companies in this space. Let’s have a look at an exchange traded fund (ETF) called <a href="https://www.roundhillinvestments.com/etf/meta/">META</a> by Roundhill Investments. This ETF was launched in June 2021 and has total 41 companies invested in as of this writing. The categories of companies are in the area of compute, network, virtual platform, payment, interchange standards, content and asset services and hardware. This gives us a good idea that a successful implementation of Metaverse relies on not only on just one technology but other enabling platforms as well.</p><p>In compute or hardware category, NVIDIA (NVDA) is on top, as it is a GPU manufacturing company and leader in graphics hardware. Other companies in hardware area are, Taiwan Semiconductor Manufacturing (TSM) and Qualcomm (QCOM). With an industry shift towards ARM based computing, TSM and QCOM are important providers of ARM chips. These hardware companies are expected to bring in revolution with their new chips both in CPU, GPU as well as Machine learning processing units that will provide the required compute for Metaverse platform. In gaming category, new age companies like Roblox (RBLX), Unity (U) and Activision Blizzard (ATVI) are changing the landscape of immersive gaming and virtual reality software. These companies are already having the gaming or virtual reality engines that can be used as a foundation software stack for building immersive applications. Facebook (FB) with massive hold on social platform is well positioned to make a disruption in the way we will consume the content in future. In Blockchain space, Coinbase (COIN) has more to offer than being crypto exchange services. Payment companies like Square (SQ) and Paypal (PYPL) are planning to offer next generation services and will accelerate crypto payment integration with other platforms. This list is by no means complete but definitely a promising one if you want to get started in investing Metaverse technology.</p><p>On the other side of investing, we have crypto space. And here platforms like Ethereum (ETH), Cardano (ADA) and Solana (SOL) are well positioned with their respective Blockchain offering. It is also important to understand, we already have got good start on Decentralized Finance (DeFi) on these platforms. In future, we can anticipate some form of integration of already existing DeFi application within Metaverse. It is for sure that Metaverse economy will be driven by crypto and not traditional fiat currency. At the same time, these Blockchain platforms offer the required trust-less foundation for integrating different components of Metaverse. When it comes to interoperability between different Blockchain platforms technologies like Polkadot (DOT) and Chainlink (LINK) are solving the challenges in interconnected network of blockchains and tamper-proof data sources.</p><p>And the last Metaverse is incomplete without an effective delivery platform. At this point of time, it is still not clear how end user is going to consume Metaverse. If we stick to traditional way of mobile computing then companies like Google (GOOGL), Apple (AAPL) and Microsoft (MSFT) will still have a significant share in Metaverse. Or it could be the new generation virtual/augmented or mixed reality companies like Oculus VR, HTC that will take over mobile computing. I believe it would be the established mobile computing companies that would transform themselves to adopt new way of consuming digital content and I would not be surprised if there are significant acquisitions of niche player in this field.</p><p>One thing note here is, there will not be a single company that dominate Metaverse as initial vision is based on more open and collaborative platform than a vendor locked-in technology. The key to success here is, how well the above mentioned technologies get integrated with each other to provide a seamless experience to end users.</p><p>I hope this gives a better idea where are the investing opportunities for Metaverse. I am not a financial advisor and this article is for educational purpose only, so I would encourage to do your own research when it comes to investing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9f434127526" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>