<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stephen Cow Chau on Medium]]></title>
        <description><![CDATA[Stories by Stephen Cow Chau on Medium]]></description>
        <link>https://medium.com/@stephencowchau?source=rss-9add418f5830------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2u3B2ch_FsvT7vI2shDXrA.png</url>
            <title>Stories by Stephen Cow Chau on Medium</title>
            <link>https://medium.com/@stephencowchau?source=rss-9add418f5830------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:29:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stephencowchau/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GLiNER — Quick notes on fine-tuning the model on Chinese data]]></title>
            <link>https://stephencowchau.medium.com/gliner-quick-notes-on-fine-tuning-the-model-on-chinese-8de402e68bec?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/8de402e68bec</guid>
            <category><![CDATA[gliner]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 15:56:14 GMT</pubDate>
            <atom:updated>2026-10-13T15:56:50.792Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>Lately I am trying to discover capable models on doing some NER task, like personal identifiable information, addresses,…</p><p>I have been using Qwen3 0.6B or Qwen3.5 0.8B as base model and perform supervised fine-tuning with different NER data. But even with 0.6B-0.8B in size, they are still needing a GPU to infer (tired OpenVino convert, still the improvement on Xeon CPU is limited, but iGPU is reasonable).</p><p>And lately came across the GLiNER, the model is small and capable for NER task (as well as other task they mentioned in tutorial in their gliner2 code base, like JSON extraction). They do great, run fast in even CPU, but only 1 problem, they do not perform well in CJK (Chinese, Japanese, Korean) languages (all models are mainly English or Latin multi-language)</p><h3>What I did wrong</h3><p>I didn’t think much ahead, and setup the desired dataset format, including multi-language data and found that the CJK languages underperform.</p><p>After digging a bit further, the reason is the model backbone deberta-v3 is using sentencepiece as embedding and per-trained with English, and the tokenizer is by default cater for English or Latin based selected language (for their multi-language variant)</p><h3>What are some options possible</h3><ol><li>Fully fine-tune — this might suffer from catastrophic forgetting</li><li>Change of backbone to modernBert/mBERT/T5 — this need a complete retrain or surgical connect and retrain (freeze the language backbone?)</li></ol><h3>What might work?</h3><p>I checked a huggingface repo — <a href="https://huggingface.co/a5566qq123/gliner2-tw-pii">https://huggingface.co/a5566qq123/gliner2-tw-pii</a>, this one is finetuning GLiNER (the v2 pii multi-language variant) for Taiwan PII, their way is to set the model use a character level splitter, seems like that might work.</p><p>I am also trying to train mine using the same idea, let’s see how it work out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8de402e68bec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running my Medium blog post as embedding regarding my job hunting resume build up]]></title>
            <link>https://faun.pub/running-my-medium-blog-post-as-embedding-regarding-my-job-hunting-resume-build-up-b26d84180bc6?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/b26d84180bc6</guid>
            <category><![CDATA[gemini-embedding-2]]></category>
            <category><![CDATA[gemma-4]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 07:37:42 GMT</pubDate>
            <atom:updated>2026-04-30T09:27:21.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>I have been working on using LLM / agent to update my resume for job hunting. Apart from my pass career experience, I also would like to include relevant technical blog post reference in my resume to cover experience that are not in my working experience.</p><h3>What this is and what this is not</h3><p>The objective is to pull relevant technical post and put it as reference in resume, this is not a RAG use case, so the focus is more on the relevancy of the embedding.</p><p>The full ingest-query process is still work in progress, I am still only focusing on the embedding process and the quality, the query part is not yet fully think through.</p><h3>The setup</h3><p>I have exported my Medium posts, it is a zip of HTML files, including all posts and responses (comments), I put them into a folder (the technical_posts)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zuN1gRThVzvQGpyHKrbxwQ.png" /></figure><p>Each HTML file is self contained, except the images are referencing back to medium.com, most of the medium header and footer are trimmed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmSZxcSH1_h8ZADeIK9keQ.jpeg" /></figure><h3>The embedding process</h3><p>There are 2 ways I want to try building the embedding, the first is the article text (using HTML tag extract), the second is to run the HTML in browser and read the content (image and text) as embedding.</p><p>The 2nd way is a little bit tricky, by the time I want to test, the Gemini Embedding v2 is released and it support PDF or image embedding, but with a limitation, 6 pages max for the PDF, the limit does not count number of pages only (it also assume the dimension of page, which means one cannot create a very long 1 page PDF to hack the number of page limit), the mechanism is the embedding API would take the PDF, render each page into image, and then pass to embedding, so I am guessing the image embedding model failed with I pass a very long 1 page PDF (thus a very long image — in weirded dimension).</p><p>So I use a different approach, to run Gemma-4 31B over the browser rendered page as images, and generate a summary text as embedding input, and Gemma-4 does not have the limit of 6 images and so we can get a summary text, and the pass that into Gemini Embedding v2 (the same model we use for original text embedding)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-23LneaxjW4hCvGvSHUAQQ.jpeg" /></figure><h3>The quality / agreement of the 2 ways of embedding</h3><p>After having the embedding, I would like to see the distribution of the embeddings.</p><p>The first approach is using UMAP to project the high dimension vector to 2D and 3D.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mLsH6HStlWwGSTipvKoj9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybYS6iRJ9JaV90G6OSNP9Q.png" /></figure><p>The full text (illustrated as blue dots in above diagram) and visual summary (illustrated as orange dots in above diagram) are distributed quite differently.</p><p>some data point could still be close to each other, while some others are further away:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*kKEmt7FQn4VKI9ORb4MVGA.png" /><figcaption>Close pair between full text vs visual summary</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*G5cbyuy36AVdRfKmpvj70g.png" /><figcaption>far apart pair between full text vs visual summary</figcaption></figure><p>Another analysis is to see pair-wise cosine similarity, it’s observed that the articles using 2 embedding would have pair similarity between 0.58–0.83, which they would not disagree with each other, but some of them could be quite different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gSLuYktU2g0f6tPw-bWo1w.png" /></figure><p>With further analysis, one observation is the different in original article length, in which similar pair typically have shorter length (so original text and summary are closer in length), that might have several hypothesis</p><ol><li>The summary using Gemma 4 for longer original text is lossy, so the relatively shorter summary does not capture the full meaning</li><li>The longer the text, sometimes include code snippets and those, with summary in Gemma 4, might be ignored (this hypothesis consider visual summary embedding more relevant as code snippet from original text might be misleading for embedding)</li><li>The images are the distraction or difference that leads to the distribution misalignment</li></ol><h3>Conclusion</h3><p>I am still considering the HTML blog, the content, the image and might be misleading code block, might not be best approach to provide the “index” for searching relevant articles.</p><p>With me only ave handfuls of blog post, I am still thinking letting LLM to read all documents and generate and INDEX.md might be easier for my use case (this was the original approach I used, just want to explore if embedding could help better, but I guess a more sophisticated index-query pipeline is needed)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RsEAtnazOXjNhoUdeMCXDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*1aIG3rYi26OOENJE.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN.dev() &amp; get similar stories in your inbox each week for free!</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b26d84180bc6" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/running-my-medium-blog-post-as-embedding-regarding-my-job-hunting-resume-build-up-b26d84180bc6">Running my Medium blog post as embedding regarding my job hunting resume build up</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Qwen 3 on Triton Inference Server (container) with TensorRT-LLM backend]]></title>
            <link>https://pub.towardsai.net/deploying-qwen-3-on-triton-inference-server-container-with-tensorrt-llm-backend-bd325b88c7d4?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd325b88c7d4</guid>
            <category><![CDATA[triton-inference-server]]></category>
            <category><![CDATA[tensorrt-llm]]></category>
            <category><![CDATA[tensorrt]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 05:34:36 GMT</pubDate>
            <atom:updated>2026-03-31T05:19:41.917Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>When it comes to serving LLM models, there are different platform / library / format for them, I would like to get some hands on experience on that lately.</p><p>I have used Ollama / LM Studio / llama.cpp (the former 2 just leverage the llama.cpp but provide better user experience) which run using GGUF that can serve the LLM with limited GPU VRAM (thanks to layer offloading for me) with a trade of inference speed.</p><p>I also tried vllm and vllm-omni (both in container) to serve models, which directly use safetensors format, which I can only work on smaller models as vllm, unlike llama.cpp, need the full model weight (and KV cache) in GPU VRAM.</p><p>Then I came across TensorRT, which is highly optimized towards NVIDIA hardware, like NVFP4 and inference graph / kernel optimization which leads to performance gain, so I decided to give it a try.</p><h3>How to serve</h3><p>Then I came across 2 things: <strong>Triton Inference Server</strong> and <strong>TensorRT-LLM</strong>. Long story short, Triton Inference Server is a layer on top of back-end layer like TensorRT-LLM, vLLM, python back-end...</p><p>The benefit of using Triton Inference Server, as NVIDIA claimed, is it allow some feature like dynamic batching, rate limiting, support different library and framework…</p><p>I do have my question as some of the feature like dynamic batching should be supported by vLLM and TensorRT-LLM back-end (this point is pretty ambiguous when reading document and checking containers folder structure and package installed, as Triton Inference Server container have a TensorRT-LLM back-end, while TensorRT-LLM also have a triton back-end, and I believe this triton back-end is referring to triton inference server)</p><p>I decided to use Triton Inference Server, with the official NVIDIA NGC Catalog container (nvcr.io/nvidia/tritonserver:26.02-trtllm-python-py), note that this container is only bundled with minimal dependencies, mainly the Triton Inference Server v26.02 and TensorRT-LLM runtime ( which some scripts are missing for prepare model )</p><h3>The Process</h3><p>The high level process would be:</p><ol><li>Download the HuggingFace model</li><li>Convert the HuggingFace model to checkpoint</li><li>Build the Checkpoint into TensorRT engine</li><li>Configure the Triton Inference Server to load the engine</li></ol><h3>What are the hiccups</h3><h4>1. The script to prepare the loading of TensorRT engine with Triton Inference Server is missing</h4><p>As mentioned, the container image is bundled with minimal dependencies, the TensorRT-LLM runtime is included, but the necessary scripts is missing.</p><p>The solution would be download the TensorRT-LLM github repo (part of the repo, as the full repo is 1GB+ and balloon the image size a lot), and we use the below docker file to create a local/tritonserver:26.02-trtllm-python-py3 image for use</p><pre># Docker File for including TensorRT-LLM repo<br>FROM nvcr.io/nvidia/tritonserver:26.02-trtllm-python-py3<br><br>RUN git clone --filter=blob:none --no-checkout --branch v1.1.0 https://github.com/NVIDIA/TensorRT-LLM.git &amp;&amp; \<br>pushd TensorRT-LLM &amp;&amp; \<br>git sparse-checkout init --cone &amp;&amp; \<br>git sparse-checkout set triton_backend benchmarks &amp;&amp; \<br>git checkout &amp;&amp; \<br>popd</pre><pre>services:<br>  triton:<br>    build:<br>      context: .<br>      dockerfile: Dockerfile<br>    image: local/tritonserver:26.02-trtllm-python-py3<br>    container_name: triton-serve<br>    environment:<br>      - HF_TOKEN=${HF_TOKEN}<br>    volumes:<br>      - ~/.cache/huggingface:/root/.cache/huggingface<br>      - ./models:/models<br>    deploy:<br>      resources:<br>        reservations:<br>          devices:<br>            - driver: nvidia<br>              capabilities: [gpu]<br>    command: bash<br>    stdin_open: true<br>    tty: true</pre><h4>2. The Triton Inference Server config missing config variable replacement</h4><p>First off, the Triton Inference Server run with pipelines, there is a template folder structure in the TensorRT-LLM repo’s subfolder triton_backend/all_models, there are following subfolders:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*PD3Zbc1UM-tPIRylVs-Cwg.png" /></figure><p>The <strong>preprocess</strong> and <strong>postprocess</strong> is responsible for tokenizing the string input and decode the result token ID back to string output</p><p>The <strong>tensor_llm</strong> is the main place that we would put our built TesnroRT engine file into</p><p>The <strong>tensor_llm_bls</strong> is for custom business logic, which I have not touch yet. I am only guessing, it should involve creating draft model(s) [speculative decoding?] and managing LoRA (guess it related to different LoRA for different business logic or domain?)</p><p>The <strong>ensemble</strong> is the pipeline that chain preprocess -&gt; tensor_llm -&gt; postprocess</p><p>Each subfolder have a config.pbtxt that have some variables to be replaced specific to our model, we can run a readily made fill_template.py in the TensorRT-LLM github repo’s relative sub path: triton_backend/tools/fill_template.py</p><p>There are examples online or in the github repo, but when we run the triton inference server to serve the model, it keep complaining some variable is missing.</p><p>The missing one for my Qwen3 model is <strong>prompt_embedding_table_data_type</strong>, which should be TYPE_FP16 for my case ( I am only guessing it’s the target data type that we build the TensorRT target model)</p><p>My script to prepare the serving:</p><pre># bash - preapre.sh<br><br>FILL=/opt/tritonserver/TensorRT-LLM/triton_backend/tools/fill_template.py<br># this is where we copy the templates<br>MODEL_TEMPLATE_DIR=/models/qwen3_0.6b_tensorrt/trial1<br>TARGET_ENGINE_PATH=$MODEL_TEMPLATE_DIR/tensorrt_llm/1<br><br>HF_MODEL=/root/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca<br># I only have 1 GPU<br>TARGET_DEVICE_IDS=&quot;0&quot;<br># this should match trtllm-build max_batch_size switch<br>MAX_BATCH=4<br><br>python3 $FILL -i ${MODEL_TEMPLATE_DIR}/ensemble/config.pbtxt            triton_max_batch_size:4,logits_datatype:TYPE_FP32<br>python3 $FILL -i ${MODEL_TEMPLATE_DIR}/preprocessing/config.pbtxt       triton_max_batch_size:4,tokenizer_dir:${HF_MODEL},preprocessing_instance_count:1<br>python3 $FILL -i ${MODEL_TEMPLATE_DIR}/postprocessing/config.pbtxt      triton_max_batch_size:4,tokenizer_dir:${HF_MODEL},postprocessing_instance_count:1<br>python3 $FILL -i ${MODEL_TEMPLATE_DIR}/tensorrt_llm/config.pbtxt triton_max_batch_size:4,decoupled_mode:False,logits_datatype:TYPE_FP32,triton_backend:tensorrtllm,max_tokens_in_paged_kv_cache:2560,max_attention_window_size:2560,kv_cache_free_gpu_mem_fraction:0.5,exclude_input_in_output:False,enable_kv_cache_reuse:True,batching_strategy:inflight_fused_batching,max_queue_delay_microseconds:0,encoder_input_features_data_type:TYPE_FP16,engine_dir:${TARGET_ENGINE_PATH},gpu_device_ids:${TARGET_DEVICE_IDS},max_queue_size:64,prompt_embedding_table_data_type:TYPE_FP16<br>python3 $FILL -i ${MODEL_TEMPLATE_DIR}/tensorrt_llm_bls/config.pbtxt triton_max_batch_size:4,decoupled_mode:False,logits_datatype:TYPE_FP32,bls_instance_count:1,prompt_embedding_table_data_type:TYPE_FP16</pre><h3>My full steps</h3><pre># [Host machine] Start the container<br>docker compose up -d<br><br># [Host machine] Get into the BASH of the container interactively<br>docker exec -it triton-serve bash<br><br># [In Container] download the huggingface model<br>hf download Qwen/Qwen3-0.6B<br><br># [In Container] Convert HF model into checkpoint<br>## Note that the snapshot subfolder might be different <br>##  at the time you download the model<br>python3 -c &quot;<br>import tensorrt_llm<br>from tensorrt_llm.models import QWenForCausalLM<br><br>model = QWenForCausalLM.from_hugging_face(<br>    &#39;/root/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca&#39;,<br>    dtype=&#39;bfloat16&#39;<br>)<br><br>checkpoint_dir = &#39;/models/qwen3_0.6b_ckpt&#39;<br>os.makedirs(checkpoint_dir, exist_ok=True)<br>model.save_checkpoint(checkpoint_dir)<br>print(f&#39;Successfully saved checkpoint to {checkpoint_dir}&#39;)<br>&quot;<br><br># [In Container] Build the TensorRT engine<br>## Note that the output would be a rank0.engine and a config.json in<br>##   /models/qwen3_engine folder<br>trtllm-build --checkpoint_dir /models/qwen3_0.6b_ckpt \<br>             --output_dir /models/qwen3_engine \<br>             --gemm_plugin bfloat16 \<br>             --max_batch_size 4<br><br><br># [In Container] Prepare the template<br>## note that we are putting the setup into /models/qwen3_0.6b_tensorrt/trial1 folder<br>cp TensorRT-LLM/triton_backend/all_models/* /models/qwen3_0.6b_tensorrt/trial1 -r<br><br># [In Container] Run the prepare script<br>## make sure to change the target folder path in the prepare.sh script<br>bash /path/to/prepare.sh<br><br><br># [In Container] Serve it for verification<br>tritonserver --model-repository=/models/qwen3_0.6b_tensorrt/trial1/<br><br># [In Container] List the models being run<br>## Note: Expected result as follow:<br>## [<br>##   {&quot;name&quot;:&quot;ensemble&quot;,&quot;version&quot;:&quot;1&quot;,&quot;state&quot;:&quot;READY&quot;},<br>##   {&quot;name&quot;:&quot;postprocessing&quot;,&quot;version&quot;:&quot;1&quot;,&quot;state&quot;:&quot;READY&quot;},<br>##   {&quot;name&quot;:&quot;preprocessing&quot;,&quot;version&quot;:&quot;1&quot;,&quot;state&quot;:&quot;READY&quot;},<br>##   {&quot;name&quot;:&quot;tensorrt_llm&quot;,&quot;version&quot;:&quot;1&quot;,&quot;state&quot;:&quot;READY&quot;},<br>##   {&quot;name&quot;:&quot;tensorrt_llm_bls&quot;,&quot;version&quot;:&quot;1&quot;,&quot;state&quot;:&quot;READY&quot;}<br>## ]<br>curl -X POST http://localhost:8000/v2/repository/index<br><br># [In Container] Query the model to verify if it response<br>curl -X POST http://127.0.0.1:8000/v2/models/ensemble/generate \<br>  -H &#39;content-type: application/json&#39; \<br>  -d &#39;{<br>      &quot;text_input&quot;: &quot;hi&quot;,<br>      &quot;parameters&quot;: {<br>      &quot;max_tokens&quot;: 16,<br>      &quot;bad_words&quot;:[&quot;&quot;],<br>      &quot;stop_words&quot;:[&quot;&quot;]<br>    }<br>  }&#39;</pre><h3>Some additional thoughts</h3><p>I think my understanding on TensorRT is, it work on tensor level, thus we need a pipeline step like preprocess and postprocess to chain together, while if we use vLLM as backend, because vLLM would take string as input and string as output, so the preprocess and postprocess is not needed.</p><p>I am seeing triton inference server could create custom business logic and ensemble (at least for TensorRT-LLM backend), which we can chain up tensors operation “graph” (e.g. preprocess output being tensor, chain to the LLM, and then the postprocess) without needing to pass the intermediate output between RAM and VRAM, which could be nice in production setup (in dev setup we can run python script to keep everything in VRAM but we might need to take care other stuff like dynamic batching), but I have not come up a chain idea to test with yet.</p><h3>More to explore</h3><ol><li>dynamic batching — I did run trtllm-bench but that’s the lower layer on TensorRT-LLM, I would like to explore how I can benchmark the dynamic batching using variable length data</li><li>multiple model deployment — this is tough as I only got a 32GB VRAM GPU, so I can only run multiple small models</li><li>mix of models and ensemble — I am guessing a pipeline to do RAG processing with embedding models and LLM (e.g. hierarchical embedding =&gt; document summary embedding and individual document / chunk embedding)</li><li>VLLM as backend for Triton Inference Server</li></ol><h3>Conclusion</h3><p>I would say I am still puzzle about triton inference server and compare it with pure vLLM deployment to see in what scenario I should use which one.</p><h3>References</h3><ul><li><a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver">NGC Catalog for Triton Inference Server</a></li><li><a href="https://nvidia.github.io/TensorRT-LLM/overview.html">TensorRT-LLM official document</a></li><li><a href="https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/README.html">Triton Inference Server with TensorRT-LLM backend official document</a></li><li><a href="https://softwaremill.com/triton-inference-server/">Concept article about Triton Inference Server by softwaremill.com</a></li><li><a href="https://zhuanlan.zhihu.com/p/707976929">Chinese (on Zhihu) article serving VLM (Llava) on Triton Inference Server</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd325b88c7d4" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/deploying-qwen-3-on-triton-inference-server-container-with-tensorrt-llm-backend-bd325b88c7d4">Deploying Qwen 3 on Triton Inference Server (container) with TensorRT-LLM backend</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick note on running litellm on Raspberry Pi]]></title>
            <link>https://stephencowchau.medium.com/quick-note-on-running-litellm-on-raspberry-pi-f679c5f780d9?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/f679c5f780d9</guid>
            <category><![CDATA[raspberry-pi-4]]></category>
            <category><![CDATA[arm64]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[litellm]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 16:26:11 GMT</pubDate>
            <atom:updated>2026-03-24T16:26:11.108Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>Lately I am planning to run my LLM agent on my spared Raspberry Pi 4 (4GB RAM).</p><p>As I am container first person, so like my other Ubuntu device (amd64), I ran it with the image ghcr.io/berriai/litellm:main-latest, there are problems running them up even the container have an arm64 build, also tried ghcr.io/berriai/litellm:main-stable.</p><h3>Inspection</h3><p>There are several attempts trying to make those official container image run but with no luck, the images either gave error like “docker: Error response from daemon: unable to find user root: no matching entries in passwd file.” or “exec /usr/bin/sh: input/output error”</p><p>Looking at github issue, people say it’s the prisma library, but I am not sure.</p><h3>Solution</h3><p>Given it might already need rebuilding the image, so I work with creating a dockerfile with starting from python:3.11-slim-bookworm and pip install litellm and litellm[proxy] (which is the API server extension including fastapi, and prisma I believe?)</p><p>And that worked.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f679c5f780d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Exercise Of Using Coding Agent to Try Reverse Engineer a Java Program]]></title>
            <link>https://pub.towardsai.net/an-exercise-of-using-coding-agent-to-try-reverse-engineer-a-java-program-30fb9b048e9b?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/30fb9b048e9b</guid>
            <category><![CDATA[coding-agents]]></category>
            <category><![CDATA[reverse-engineering]]></category>
            <category><![CDATA[codex-cli]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 16:32:32 GMT</pubDate>
            <atom:updated>2025-12-23T03:39:37.437Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsBH2C3gOeR-kzO1NvhAUw.jpeg" /></figure><h3>Background</h3><p>I am exploring a method to try rejuvenate a legacy system (without source code), the target system might be a Java based system, so I am trying to prepare how to do it with coding agent’s help.</p><p>I ran the exercise on a Java program that I have written myself, which I know in detail:</p><ol><li>The system design and mechanism.</li><li>The database schema.</li><li>how to explain the system to others (as lately I have been performing a skill transfer to my teammate).</li></ol><p>Regarding the skill transfer, I have also been using coding agent to read the code, prepare the summary of system understanding, put it into documentation and presentation deck (using Notebook LM to read the documentation).</p><p>Regarding to the JAR, it’s written in Kotlin JVM with Ktor (API serving), Koin (dependency injection), Exposed (database access) and other libraries, using Java 23, using Ktor’s build fat JAR gradle task to build.</p><p>I am using Codex CLI v0.77 with ChatGPT 5.2 to run through the exercise (in VS code, instead of IntelliJ).</p><h3>The assumption of the exercise</h3><ol><li>Assuming no source code, only provide the built JAR file (which is a fat JAR).</li><li>Assuming having access to target system database.</li></ol><h3>The idea / design of the exercise</h3><ol><li>De-compile the JAR into code (download the <a href="https://www.benf.org/other/cfr/">CFR</a> de-compiler).</li><li>Let the coding agent to read the code and plan about how to analyze the code.</li><li>Expect to create BOTH the analysis of the code as well as a strategy file so we can take the lesson learnt from this exercise to another system ( to battle test the strategy and then hoping this still work to the target system).</li></ol><h3>The experience and observation of agent behavior</h3><ol><li>In the first place, I asked the agent provided the instruction of what library and command to perform the de-compilation, and so the agent suggested the library (the de-compiler) and how to execute, then the JAR got de-compiled.</li><li>The agent would start by analyzing the entry point by reading the META-INF’s MANIFEST.MF, and figured out the main class is from Ktor’s engine</li><li>Given the de-compiled fat JAR include all packages, so the dependencies like ktor, koin or others would also be included, that would cloud the agent’s focus, so another instruction would be asking the agent to identify supporting packages (dependencies, especially well known like ktor, koin, exposed…) vs main inspection target package(s), and create a list of focused package so the agent would keep track of throughout the process.</li><li>The agent, then read the targeted package and use heuristic to identify the app main module, the ktor routes as well as koin dependency injection (defined in app main module)</li><li>The agent then map out the package folder structure (which define the package structure), afterwards it do a depth first analysis to inspect ktor route and services one by one</li><li>For each ktor route, it first do a shallow analysis on the signature (GET/POST, the function name, request and response structure), then after all routes are inspected, it perform the inner logic inspection per route one by one the get the “depth”</li><li>In terms of “logic”, the next “depth” the agent focus on the data operation (main MySQL persistence data or Redis cache, message queue)</li><li>Regarding the MySQL database, the implementation use a mix of ORM and SQL statement (in text), the agent can identify BOTH, also including the query result plus the calculation around it</li><li>The agent also able to identify the message passing logic (for message queue) and the Redis cache update logic and structure</li><li>After the de-compiled code inspection, I also ask the agent to run a custom skill (instead of using MCP, which is on purpose, and the skill is also being generated using Codex) to query the MySQL database to verify the code inspected result with the database schema and run the actual query to make sure the identified query does run without error.</li></ol><h3>The result</h3><p>I found that the analysis and the strategy document generated is pretty solid, I tried taking the analysis document to Notebook LM to generate a presentation deck, and some slide as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BqvA751CQdm8l3NqRGgCvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnsVaLQLaKTdWoaHnUVb3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A5a-cF1Gf3p5Badi2tSW2A.png" /></figure><h3>Conclusion and follow up</h3><p>The total summary is only several hours, even the project itself is not a huge one, it show this kind of exercise have its potential (and token used is not a whole lot, gut feeling is about 30M input and &lt;10k output)</p><p>The exercise show the power of coding agent in terms of rejuvenate a legacy project, to understand it and then follow up with a new spec (which I have not done) and then spec driven development.</p><p>And for follow up experiment, I would take the strategy summary file (of the inspection exercise), and try it with another JAR and see if that still work as expected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30fb9b048e9b" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/an-exercise-of-using-coding-agent-to-try-reverse-engineer-a-java-program-30fb9b048e9b">An Exercise Of Using Coding Agent to Try Reverse Engineer a Java Program</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Claude Skill to use Zep memory (for agent to use)]]></title>
            <link>https://stephencowchau.medium.com/build-a-claude-skill-to-use-zep-memory-for-agent-to-use-cab819cca7ff?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/cab819cca7ff</guid>
            <category><![CDATA[claude-agent]]></category>
            <category><![CDATA[agent-memory]]></category>
            <category><![CDATA[zep]]></category>
            <category><![CDATA[claude-skills]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Fri, 19 Dec 2025 17:46:55 GMT</pubDate>
            <atom:updated>2025-12-19T17:46:55.594Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>I have been exploring building LLM backed agents, I tried different libraries and approaches, like when agent orchestration frameworks like AutoGen, CrewAI, SmolAgent, or more workflow-like framework like LangGraph or Dify.</p><p>None of them were fulfilling experience, I guess the main reason is the LLM back then was not as smart as nowadays, and it’s very difficult to control agent orchestration to achieve goal without side track or so.</p><p>After gaining more experience working with coding agents, I felt the passion to work on agent again.</p><h3>Claude Agent</h3><p>I have been using mostly Claude Code, I found Claude Code features fascinating, and they have make Claude Code programmable through SDK, so I decided to give it a try.</p><p>So what’s the difference between Claude agent and other framework? First off, Claude agent is a single agent, even it support subagents, it’s not designed for orchestration. The Claude model, with the system prompt and agent loop worked in a comparatively controllable manner, and it’s designed to be interactive first (and sure it can automate), so it can fit both chatbot use case and automated task execution.</p><h3>Skill</h3><p>This article’s focus is not on the agent SDK itself, but I do need to let the agent to have some tools to perform the targeted task.</p><p>There are several options, back then it’s tool, and later, we have MCP. MCP was good until when there are too many of them, they can pollute the context (and Claude have 200k context only).</p><p>When skill comes in, I have been thinking about how I can use skill to let Claude Code smart or follow a particular workflow.</p><p>I tried the simple single file SKILL.md to tell the agent to follow some step and use a database MCP to read schema, data and also code to come up with documentation, I think that work nice. And then I read about skill can execute code or script, that bridge the autonomous nature of agent and the deterministic program code logic.</p><h3>Zep</h3><p>There are a lot of agent memory library out there, Zep caught my attention when I see it try to process the text information into graph, and more importantly, having the temporal invalidation (information change over time, and some old information are stored but have a mark for invalid since a certain time), I think that’s important.</p><h3>What I try to build</h3><p>I would build skill that support storing, querying and summarize the user (consider he/she as the chatbot user) medical information.</p><p>The journey was partially nightmare when I have not enough know how on how to build the skill with code support. Most examples are in python and the SKILL.md only mention about how to call the python functions and how to setup environment.</p><p>Here comes something I felt insecure, the skill code run on local (not in a sandbox, and it would install the package according to setup instruction).</p><p>My code base is on JavaScript, so the node_module made isolation of environment, but still I have no idea how to make Claude the agent understand how to use the code. I believe LLM learn enough python, but it don’t know about JavaScript.</p><p>Ended up, the solution is some wrappers, like creating a CLI file, so the SKILL.md can explain the use case as calling “node cli.js …”., my SKILL.md as below:</p><pre>---<br>name: medical-info-skill<br>description: Store and retrieve medical information in a persistent Zep knowledge graph with automatic temporal tracking. Use when users share medical diagnoses, symptoms, medications, lab results, vital signs, or ask about medical history.<br>---<br><br># Medical Information Manager<br><br>This skill stores and retrieves medical information using Zep&#39;s temporal knowledge graph with automatic entity extraction and temporal tracking.<br><br>## Usage<br><br>Execute the CLI from `.claude/skills/medical-info-skill/`:<br><br>```bash<br># Store medical information<br>node cli.js addMedicalInfo &lt;userId&gt; &quot;&lt;medical content&gt;&quot;<br><br># Query medical history<br>node cli.js queryMedicalHistory &lt;userId&gt; &quot;&lt;question&gt;&quot;<br><br># Get complete medical context<br>node cli.js getMedicalContext &lt;userId&gt;<br><br># Clear user data<br>node cli.js clearUserGraph &lt;userId&gt;<br>```<br><br>## Examples<br><br>### Store a patient&#39;s annual physical<br>```bash<br>node cli.js addMedicalInfo persona_001 &quot;Annual physical on 2022-02-10. Diagnoses: pre-diabetes, borderline hypertension (BP 138/90), overweight (BMI 29.0), mild intermittent asthma. Vitals: HR 82 bpm, weight 79 kg, height 165 cm.&quot;<br>```<br><br>### Query medical history<br>```bash<br>node cli.js queryMedicalHistory persona_001 &quot;What diagnoses do I have?&quot;<br>```<br><br>### Get complete medical context<br>```bash<br>node cli.js getMedicalContext persona_001<br>```<br><br>## Functions<br><br>- **addMedicalInfo(userId, content)**: Stores medical information with automatic temporal tracking<br>- **queryMedicalHistory(userId, question)**: Searches medical history with natural language<br>- **getMedicalContext(userId)**: Retrieves complete medical profile<br>- **clearUserGraph(userId)**: Deletes all medical data for a user<br><br>## Temporal Tracking<br><br>Dates in medical content are automatically tracked:<br>- &quot;Started medication on 2024-01-15&quot; → valid_at = 2024-01-15<br>- &quot;Stopped medication on 2024-06-15&quot; → invalid_at = 2024-06-15<br>- &quot;Lab result from 2024-12-01&quot; → stored with that date</pre><h3>The Result</h3><p>I only use Claude Code (the CLI) to run the skill, not yet tie it to my Agent SDK code.</p><p>I fabricated some client medical visit report / summary, one example as:</p><pre># Visit Summary – Right Ankle Sprain<br><br>- **Date:** 2023-09-02<br>- **Provider:** Urgent Care Clinician<br><br>Thank you for coming in today after twisting your right ankle while jogging.<br><br>## Reason for Visit<br><br>You reported that you **twisted your right ankle** while jogging yesterday. Since then you have had pain on the outside of the ankle and some swelling. You are able to walk with a limp but have pain when putting weight on the ankle.<br><br>## Key Findings<br><br>- There was swelling and tenderness over the outer part of the right ankle.<br>- You were able to move the ankle, but it was painful at the extremes of movement.<br>- There was no obvious deformity.<br>- An **ankle X-ray** was done to check for fracture.<br><br>## Ankle X-ray Result (Summary)<br><br>- The X-ray did **not** show any broken bones.<br>- The findings were consistent with a **mild to moderate sprain** of the ligaments around the ankle.<br><br>## Diagnosis<br><br>- **Right ankle sprain (no fracture seen).**<br><br>## Treatment Plan<br><br>- Follow the **RICE** approach:<br>  - **Rest:** Limit weight-bearing on the right leg for the next few days.<br>  - **Ice:** Apply ice packs for 15–20 minutes at a time, several times a day.<br>  - **Compression:** Use an elastic bandage as instructed.<br>  - **Elevation:** Keep the ankle raised above the level of your heart when possible.<br>- Use **over-the-counter pain relievers** (such as ibuprofen) as needed, unless another doctor has told you to avoid them.<br>- You were given guidance on gentle range-of-motion exercises to start after the initial pain and swelling begin to improve.<br>- You should gradually return to jogging only after you can walk without a limp and perform daily activities without pain.<br><br>## When to Seek Care Sooner<br><br>- Increasing pain or swelling that does not improve with rest and ice.<br>- Inability to bear weight at all.<br>- Numbness, tingling, or changes in skin color in the foot.<br><br>Please keep this summary and your ankle X-ray report with your records. They help document this injury and the fact that no fracture was seen.</pre><p>I tell Claude to use the tool (yes, it’s not able to automatically detect yet), the conversation as below (up to 2nd record):</p><pre>&gt; Hi, I am Alicia, I went to doctor and they gave me this report:                                                                                  <br>  @client/persona_001/patient_reports/encounter_2022-02-10_annual_physical_patient.md                                                              <br>  ⎿  Read client/persona_001/patient_reports/encounter_2022-02-10_annual_physical_patient.md (39 lines)                                            <br>                                                                                                                                                   <br>● Hi Alicia! I&#39;ve reviewed your annual physical report from Dr. Lee dated February 10, 2022.                                                       <br>                                                                                                                                                   <br>  Here&#39;s a summary of what the report shows:                                                                                                       <br>                                                                                                                                                   <br>  Current Status:                                                                                                                                  <br>  - Blood pressure slightly elevated (138/90 mmHg)                                                                                                 <br>  - Overweight (BMI 29.0)                                                                                                                          <br>  - Pre-diabetes indicators (fasting glucose at 108 mg/dL, HbA1c at 6.0%)                                                                          <br>  - Borderline high cholesterol                                                                                                                    <br>  - Mild asthma (well controlled)                                                                                                                  <br>  - Heart and lungs are normal                                                                                                                     <br>                                                                                                                                                   <br>  Recommended Actions:                                                                                                                             <br>  - Exercise: Aim for 150+ minutes per week of moderate activity like brisk walking                                                                <br>  - Diet: Reduce sugary drinks and refined carbs; increase vegetables, lean proteins, and fiber                                                    <br>  - Monitoring: Check blood pressure at home 1-2 times weekly                                                                                      <br>  - Follow-up: Repeat blood tests in 6-12 months                                                                                                   <br>                                                                                                                                                   <br>  How can I help you with this report? Are you looking to:                                                                                         <br>  - Track your progress toward the lifestyle goals?                                                                                                <br>  - Organize and monitor your health metrics over time?                                                                                            <br>  - Get help understanding any specific findings?                                                                                                  <br>  - Plan next steps?                                                                                                                               <br>                                                                                                                                                   <br>  Let me know what would be most helpful!                                                                                                          <br>                                                                                                                                                   <br>&gt; can you help me use medical-info-skill to record this?                                                                                           <br>                                                                                                                                                   <br>● /medical-info-skill                                                                                                                              <br>                                                                                                                                                   <br>● Bash(cd /home/witti/git/personal/md_mem/.claude/skills/medical-info-skill &amp;&amp; node cli.js addMedicalInfo persona_001 &quot;Annual physical on          <br>      2022-02-10. Diagnoses: pre-di…)                                                                                                              <br>  ⎿  Tool use rejected with user message: I want to record under Alicia instead of personal_001                                                    <br>                                                                                                                                                   <br>● Got it! Let me record this under your name, Alicia:                                                                                              <br>                                                                                                                                                   <br>● Bash(cd /home/witti/git/personal/md_mem/.claude/skills/medical-info-skill &amp;&amp; node cli.js addMedicalInfo Alicia &quot;Annual physical on 2022-02-10.   <br>      Diagnoses: pre-diabete…)                                                                                                                     <br>  ⎿  [dotenv@17.2.3] injecting env (3) from .env -- tip: 🛠️  run anywhere with `dotenvx run -- yourcommand                                     <br>     {                                                                                                                                             <br>       &quot;episodeId&quot;: &quot;unknown&quot;,                                                                                                                     <br>     … +3 lines (ctrl+o to expand)                                                                                                                 <br>                                                                                                                                                   <br>● Perfect! Your annual physical exam from February 10, 2022 has been recorded in the medical-info-skill system under your name (Alicia).           <br>                                                                                                                                                   <br>  The skill has automatically extracted and stored:                                                                                                <br>  - Diagnoses (pre-diabetes, borderline hypertension, overweight, mild asthma)                                                                     <br>  - Vital signs (BP, weight, height, BMI)                                                                                                          <br>  - Lab results (glucose, HbA1c, cholesterol panel)                                                                                                <br>  - Exam findings (normal heart/lungs, well-controlled asthma)                                                                                     <br>  - Date context (2022-02-10)                                                                                                                      <br>  - Recommendations (exercise, diet, monitoring plan)                                                                                              <br>                                                                                                                                                   <br>  Now you can query this information anytime. For example, you could ask things like:                                                              <br>  - node cli.js queryMedicalHistory Alicia &quot;What are my risk factors?&quot;                                                                             <br>  - node cli.js queryMedicalHistory Alicia &quot;What were my lab results?&quot;                                                                             <br>  - node cli.js getMedicalContext Alicia (to see your complete medical profile)                                                                    <br>                                                                                                                                                   <br>  Would you like to add any other health records or check what&#39;s been stored?                                                                      <br>                                                                                                                                                   <br>&gt; I want you to help me to record another:                                                                                                         <br>  &#39;/home/witti/git/personal/md_mem/client/persona_001/patient_reports/encounter_2022-11-05_bronchitis_visit_patient.md&#39;                            <br>                                                                                                                                                   <br>● Read(~/git/personal/md_mem/client/persona_001/patient_reports/encounter_2022-11-05_bronchitis_visit_patient.md)                                  <br>  ⎿  Read 45 lines                                                                                                                              <br>                                                                                                                                                   <br>● Great! I&#39;ve read your bronchitis visit report from November 5, 2022. Now let me record this in the medical-info-skill under your name (Alicia):  <br>                                                                                                                                                   <br>● Bash(node cli.js addMedicalInfo Alicia &quot;Visit on 2022-11-05 for cough evaluation. Chief complaint: 7-10 days of cough worse at night, chest      <br>      discomfort with coughing,…)                                                                                                                  <br>  ⎿  [dotenv@17.2.3] injecting env (3) from .env -- tip: ⚙️  override existing env vars with { override: true }                                 <br>     {                                                                                                                                             <br>       &quot;episodeId&quot;: &quot;unknown&quot;,                                                                                                                     <br>     … +3 lines (ctrl+o to expand)                                                                                                                 <br>                                                                                                                                                   <br>● Perfect! Your bronchitis visit from November 5, 2022 has been recorded in the medical-info-skill system. The system has stored:                  <br>                                                                                                                                                   <br>  - Visit date: 2022-11-05                                                                                                                         <br>  - Chief complaint: 7-10 day cough, worse at night, chest discomfort, low-grade fever                                                             <br>  - Exam findings: Stable vitals, coarse breath sounds, no pneumonia signs                                                                         <br>  - Imaging: Chest X-ray negative for pneumonia, fluid, or masses                                                                                  <br>  - Diagnosis: Acute bronchitis (viral), mild asthma stable                                                                                        <br>  - Treatment: Rest, fluids, OTC cough remedies, optional inhaler if symptoms worsen                                                               <br>  - Provider: Dr. Lee                                                                                                                              <br>                                                                                                                                                   <br>  You now have both encounters recorded:                                                                                                           <br>  1. February 2022: Annual physical (pre-diabetes, hypertension risk)                                                                              <br>  2. November 2022: Bronchitis visit (viral respiratory infection)                                                                                 <br>                                                                                                                                                   <br>  Would you like to add more medical records, or would you like me to help you query your medical history?                                         <br>                                                                                                                                                   <br>&gt; help me record another &#39;/home/witti/git/personal/md_mem/client/persona_001/patient_reports/encounter_2023-03-15_exercise_start_patient.md&#39;</pre><p>I check the graph being created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*fRqt3oVqb3nvc7-62rNgnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*eBg3OTDB9l4W37E64V5VnA.png" /></figure><p>I spin up another Claude session and ask:</p><pre>&gt; hi, I am Alicia, can you use medical-info-skill to retrieve my medical summary?                                                                  <br>                                                                                                                                                   <br>● /medical-info-skill                                                                                                                              <br>                                                                                                                                                   <br>● I&#39;ll retrieve your medical summary using the medical-info-skill. Let me query your medical context.                                              <br>                                                                                                                                                   <br>● Bash(cd /home/witti/git/personal/md_mem/.claude/skills/medical-info-skill &amp;&amp; node cli.js getMedicalContext alicia)                               <br>  ⎿  Error: Exit code 1                                                                                                                            <br>     Error in getMedicalContext: Status code: 404                                                                                                  <br>     Body: {                                                                                                                                       <br>       &quot;message&quot;: &quot;user not found&quot;                                                                                                                 <br>     }                                                                                                                                             <br>                                                                                                                                                   <br>     [dotenv@17.2.3] injecting env (3) from .env -- tip: 🔄 add secrets lifecycle management: https://dotenvx.com/ops                              <br>     {                                                                                                                                             <br>       &quot;success&quot;: false,                                                                                                                           <br>       &quot;error&quot;: &quot;Status code: 404\nBody: {\n  \&quot;message\&quot;: \&quot;user not found\&quot;\n}&quot;,                                                                 <br>     … +5 lines (ctrl+o to see all)                                                                                                                <br>                                                                                                                                                   <br>● It looks like there&#39;s no medical information stored for you yet (user &quot;alicia&quot;). The system couldn&#39;t find any existing medical records.          <br>                                                                                                                                                   <br>  Would you like to:                                                                                                                               <br>  1. Store medical information - I can add your medical diagnoses, medications, vitals, or other health information to your medical profile        <br>  2. Query existing information - If you&#39;ve already stored information, I can search it with natural language questions                            <br>                                                                                                                                                   <br>  If you&#39;d like to add your medical information, just let me know what details you&#39;d like to store (diagnoses, medications, lab results, vitals, etc.), and I can save them with automatic temporal tracking.<br>                                                                                                                                                   <br>&gt; my name is Alicia, not alicia                                                                                                                    <br>                                                                                                                                                   <br>● Bash(cd /home/witti/git/personal/md_mem/.claude/skills/medical-info-skill &amp;&amp; node cli.js getMedicalContext Alicia)                               <br>  ⎿ [dotenv@17.2.3] injecting env (3) from .env -- tip: ✅ audit secrets and track compliance: https://dotenvx.com/ops                          <br>    {                                                                                                                                              <br>      &quot;userSummary&quot;: &quot;The user&#39;s email address is Alicia@example.com. The user&#39;s weight was 76.5 kg and height was 165 cm as of 2024-02-01. He     <br>    … +33 lines (ctrl+o to expand)                                                                                                                 <br>                                                                                                                                                   <br>                                                                                                                                                   <br>● Perfect! Here&#39;s your medical summary, Alicia:                                                                                                    <br>                                                                                                                                                   <br>  Your Medical Profile                                                                                                                             <br>                                                                                                                                                   <br>  Personal Information:                                                                                                                            <br>  - Email: Alicia@example.com                                                                                                                      <br>  - Height: 165 cm                                                                                                                                 <br>  - Weight: 76.5 kg (as of Feb 1, 2024)                                                                                                            <br>                                                                                                                                                   <br>  Health Conditions:                                                                                                                               <br>  - Pre-diabetes                                                                                                                                   <br>  - Borderline hypertension (BP: 128/82 mmHg as of Feb 1, 2024)                                                                                    <br>                                                                                                                                                   <br>  Healthcare Team:                                                                                                                                 <br>  - Primary Care Provider: Dr. Lee (most recent follow-up: Feb 1, 2024)                                                                            <br>  - Registered Dietitian (consulted May 20, 2023)                                                                                                  <br>                                                                                                                                                   <br>  Current Health Goals:                                                                                                                            <br>  - Physical Activity: Aim for 150+ minutes per week of moderate activity                                                                          <br>    - Short-term: 7,000 steps daily                                                                                                                <br>    - Long-term: 8,000-10,000 steps daily                                                                                                          <br>  - Blood Pressure Monitoring: 1-2 times weekly                                                                                                    <br>  - Sleep Improvement: Work on better sleep hygiene (currently averaging ~6.5 hours nightly)                                                       <br>                                                                                                                                                   <br>  Dietary Management:                                                                                                                              <br>  - Reduce: sugary drinks, sweets, pastries, refined carbohydrates, large portions                                                                 <br>  - Increase: vegetables, beans, lentils, whole grains                                                                                             <br>                                                                                                                                                   <br>  Activity Level:                                                                                                                                  <br>  - Exercise: Walking/jogging several times weekly                                                                                                 <br>  - Sleep: ~6.5 hours nightly (often going to bed late)                                                                                            <br>                                                                                                                                                   <br>  Is there anything specific about your medical information you&#39;d like to update or discuss?                                                       <br>                                                                                                                                                   <br>&gt; I forgot when did I had my ankle problem? can you query my medical record?                                                                       <br>                                                                                                                                                   <br>● Bash(node cli.js queryMedicalHistory Alicia &quot;When did I have an ankle problem?&quot;)                                                                 <br>  ⎿ [dotenv@17.2.3] injecting env (3) from .env -- tip: 🔐 prevent building .env in docker: https://dotenvx.com/prebuild                        <br>    {                                                                                                                                              <br>      &quot;context&quot;: &quot;\n\n# This is the user summary\n&lt;USER_SUMMARY&gt;\nThe user&#39;s email address is Alicia@example.com. As of 2024-02-01, the user&#39;s     <br>    … +20 lines (ctrl+o to expand)                                                                                                                 <br>                                                                                                                                                   <br>                                                                                                                                                   <br>● Based on your medical records, I found information about your ankle problem:                                                                     <br>                                                                                                                                                   <br>  Ankle Injury Details:                                                                                                                            <br>  - Date: September 1, 2023 (yesterday from the initial recording)                                                                                 <br>  - What happened: Your right ankle was twisted                                                                                                    <br>  - Context: The injury occurred during your exercise routine                                                                                      <br>                                                                                                                                                   <br>  The injury appears to have been a short-term issue (recorded as valid from Sept 1-2, 2023), suggesting it may have been a minor twist. If you&#39;d like to add more recent information about your ankle or any ongoing issues, just let me know!</pre><p>I am kind of happy about the result on the query did retrieve the detail, the date of report is 2 Sep 2023 and the yesterday is 1 Sep 2023. I need more testing (craft of data) to confirm if this Zep graph is doing the correct job (e.g. what if I add multiple ankle issue which are alike and confusing, and also I want to add more test case like remembering medicine record and invalidate some medicine prescription after some specific time and see if the query still work correctly).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cab819cca7ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Unsloth On a Jetson AGX Orin Device With Jetson-Containers]]></title>
            <link>https://pub.towardsai.net/running-unsloth-on-a-jetson-agx-orin-device-with-jetson-containers-e21f86e4a7cd?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/e21f86e4a7cd</guid>
            <category><![CDATA[jetsons]]></category>
            <category><![CDATA[unsloth]]></category>
            <category><![CDATA[jetson-containers]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 06:23:40 GMT</pubDate>
            <atom:updated>2025-12-01T02:44:24.657Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yXTahZg2MojBtgQJWN2Odg.png" /></figure><h3>Background</h3><p>I have been for a long time container first, mainly because of the ease of environment isolation (including but not limited to system package, programming library package…). For Jetson it’s a little more handy as flashing a Jetson to test new setup or revert to image is pretty tedious (just my opinion).</p><p>Running container in Jetson device have several constraints, including at least:</p><ol><li>Most package are included in Jetpack, which is carefully build packages regarding the specially build Tegra Linux (a variant of Ubuntu) and the Jetson hardware, it’s most stable when the container OS align with the host OS as container sometimes need support from host on lower level operations</li><li>If we are working on GPU related workload, we would need CUDA support and NVIDIA container toolkit, and especially the container is highly preferred to use the same CUDA version that the Jetson host installed (mostly come with Jetpack, but allow upgrade, need to check supported version, see <a href="https://docs.nvidia.com/cuda/cuda-for-tegra-appnote/index.html#upgradable-package-for-jetson">reference</a> )</li></ol><p>So here come the great dusty-nv’s jetson-containers project (See <a href="https://github.com/dusty-nv/jetson-containers/">Github</a>), which one can use it to build or run prebuilt containers that serve different machine learning or LLM purpose.</p><h3>My environment and objective</h3><p>So happen I have a Jetson AGX Orin 64GB, the great thing about it is the 64GB unified memory, which can load a larger model, and so I am trying to use it to try finetuning a LLM model (which unsloth’s Google Colab with free T4 GPU with 16GB VRAM might not fit in)</p><p>The detail of the Jetson:</p><pre>cat /etc/nv_tegra_release<br># R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan  8 01:49:37 UTC 2025<br># KERNEL_VARIANT: oot<br># ...<br><br>uname -a<br># Linux ubuntu 5.15.148-tegra #1 SMP PREEMPT Tue Jan 7 17:14:38 PST 2025 aarch64 aarch64 aarch64 GNU/Linux<br><br>nvcc --version<br># nvcc: NVIDIA (R) Cuda compiler driver<br># Copyright (c) 2005-2024 NVIDIA Corporation<br># Built on Wed_Aug_14_10:14:07_PDT_2024<br># Cuda compilation tools, release 12.6, V12.6.68<br># Build cuda_12.6.r12.6/compiler.34714021_0</pre><p>So we have Jetson R36.4.3, and it’s Jetpack 6.2, with CUDA 12.6 and based on a Ubuntu 22.04 (jammy) (ref: <a href="https://developer.technexion.com/docs/embedded-software/linux/nvidia-jetpack/usage-guides/jetpack620/">https://developer.technexion.com/docs/embedded-software/linux/nvidia-jetpack/usage-guides/jetpack620/</a>)</p><h3>What failed</h3><h4>1. Using official docker container from unsloth docker hub</h4><p>This fail fast, as the docker containers are all built with linux/amd64, while a Jetson is ARM device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*BT-CuO0XNzRPYerwfLvw9w.png" /><figcaption>all images are built for linux/amd64</figcaption></figure><h4>2. Build and run jetson-container with unsloth package</h4><p>Even it’s not on home page of jetson-containers, the package is actually added under package “llm”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6v4Xa7KrNl9qK3hZSS3P1g.png" /><figcaption>the LLM section not showing unsloth</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O7oC-liYaGbBiGo3rcBp7g.png" /><figcaption>the package already include unsloth in the repo</figcaption></figure><p><strong><em>Note:</em></strong><em> Please read the </em><a href="https://github.com/dusty-nv/jetson-containers/blob/master/docs/build.md"><em>document</em></a><em> before build</em></p><p>I tried build with:</p><pre>jetson-containers build unsloth</pre><p>Multiple trials was attempted, with autotag (auto discover information based on host Jetson device hardware and software), or specify different Python, PyTorch, CUDA version.</p><p>All trials failed at certain point (step), the commonality is building take quite a lot of time, because it would build from the base Ubuntu image and compile and build all necessary package, including CUDA, cuDNN, PyTorch…</p><h3>What worked</h3><h4>3. Locate a prebuilt container image to install unsloth</h4><p>This is the pathway I found successful eventually but got quite a lot of failures throughout the path.</p><p><strong>3.1 Locating the image</strong></p><p><a href="https://hub.docker.com/u/dustynv">dustynv’s docker hub repo</a> have pre-built containers, but not all container is a good image to start with.</p><p>The idea would be the closest to unsloth requirement is the best, so starting from unsloth <a href="https://github.com/unslothai/unsloth/blob/main/pyproject.toml">pyproject.toml</a>, we can see it require torch, bitsandbytes, transformers, xformers, possibly fast attention…</p><p>So we can go check the containers tags</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tajF9wep2eD87eGLoe3R5w.png" /><figcaption>bitsandbytes containers, even have r36 image, but it’s for CUDA 12.8</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I4YRLsvgBMLqxK3sB079bQ.png" /><figcaption>xformers containers, all targeted r35</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wExtlmn64_vx7gvFzLQe8g.png" /><figcaption>the l4t-pytorch image have a r36.4.0 image</figcaption></figure><p>We can see only the more foundation l4t-pytorch container image match our r36.4, and further check the container build, the CUDA also is the expected CUDA 12.6</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nnKhd0xLuEiVGBGoNEdxA.png" /><figcaption>the CUDA in build step is targeting 12.6</figcaption></figure><p><strong>3.2 Run the container and install unsloth</strong></p><p>follow the <a href="https://github.com/dusty-nv/jetson-containers/blob/master/docs/run.md">document</a> on running jetson containers using image tag, we can do:</p><pre>docker run --runtime nvidia -it --rm --network=host dustynv/l4t-pytorch:r36.4.0</pre><p>It would land us into the bash shell, from there we would try running pip install unsloth but immediately we face a timeout issue.</p><p>The reason is the Python package are also specially built and rely on a specific pypi repo instead of the usual one, it’s defined in environment variable PIP_INDEX_URL=”<a href="http://jetson.webredirect.org/jp6/cu126">http://jetson.webredirect.org/jp6/cu126</a>&quot;, the redirect point to pypi.jetson-ai-lab.<strong>dev</strong>, but the working repo is at pypi.jetson-ai-lab.<strong>io</strong>.</p><p>So we can do the following to override the env variable:</p><pre>export PIP_INDEX_URL=https://pypi.jetson-ai-lab.io/jp6/cu126</pre><p><strong>3.3 Tackling python package mismatch</strong></p><p>The following is just struggling with the Python package dependency system, which breaking changes between version make the code not able to run as well as some missing dependency when installing unsloth (wheel)</p><p>There are a few logic and hints to check.</p><p>First, checking the custom pre-built python wheels at <a href="https://pypi.jetson-ai-lab.io/jp6/cu126">https://pypi.jetson-ai-lab.io/jp6/cu126</a></p><p>we can see some key packages: unsloth (2025.7.9), torch (2.8.0), bitsandbytes (0.47.0.dev0), triton (3.4.0), torchvision (0.23.0), xformers (0.0.32+8ed0992.d20250724), flash-attn (2.8.2)</p><p>Then checking unsloth pyproject.toml requirement dependency, it state that some cudat version is using some pytorch version.</p><p><strong><em>3.3.1 Different failures</em></strong></p><p>The first failure attempt is to install unsloth 2025.7.9 from prebuilt wheel, and install unsloth_zoo. The problem we face would be some mismatch between the pytorch 2.4.0 come with the container image and the transformers library requirement (My aim was finetuning Gemma 3N, which require transformer 4.53.0, and that’s the version introduce the breaking change that unsloth 2025.7.9 is unexpected).</p><p>So the next meaningful trial is to install unsloth from git, multiple failures, again due to the mismatch on PyTorch version (2.4.0 from container image), transformers version…</p><p>The working setup being:</p><pre>export PIP_INDEX_URL=https://pypi.jetson-ai-lab.io/jp6/cu126<br>pip install &quot;torch==2.8.0&quot;<br>pip install &quot;torchvision==0.23.0&quot;<br>pip install bitsandbytes<br>pip install xformers<br>pip install &quot;flash-attn==2.8.2&quot;<br>pip install &quot;unsloth[huggingface] @ git+https://github.com/unslothai/unsloth.git&quot;<br>pip install unsloth-zoo<br>pip install timm<br>pip install juypterlab</pre><p>After testing the installation sequence, docker container:</p><pre>FROM dustynv/l4t-pytorch:r36.4.0<br><br>ENV PIP_INDEX_URL=https://pypi.jetson-ai-lab.io/jp6/cu126<br><br>RUN pip install &quot;torch==2.8.0&quot; &amp;&amp; \<br>pip install &quot;torchvision==0.23.0&quot; &amp;&amp; \ <br>pip install bitsandbytes &amp;&amp; \<br>pip install xformers &amp;&amp; \<br>pip install &quot;flash-attn==2.8.2&quot; &amp;&amp; \<br>pip install &quot;unsloth[huggingface] @ git+https://github.com/unslothai/unsloth.git&quot; &amp;&amp; \<br>pip install unsloth-zoo &amp;&amp; \<br>pip install timm &amp;&amp; \<br>pip install --index-url https://pypi.jetson-ai-lab.io/root/pypi jupyterlab ipywidgets<br><br>CMD [&quot;sleep&quot;, &quot;infinity&quot;]</pre><p>Note that running the initialization would complain about the xformer is not setup properly, it complain about our python version being 3.10, and the wheel is built for 3.12, but checking the wheel, it should be supporting 3.9–3.12</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9UcHDZBW1r47Rn6ayXJmTg.png" /><figcaption>prebuilt wheel of xformers is targeting python ≥ 3.9</figcaption></figure><p>Anyway, so I added the flash attention library and seems the running od Gemma 3N finetune and inference is about the same speed compare to the Colab notebook unsloth provide when run on a T4 (which is still slow on inference)</p><p>Side note: compare T4 and Jetson AGX Orin (<a href="https://technical.city/en/video/Tesla-T4-vs-Jetson-AGX-Orin-64-GB">reference</a>)</p><h3>Conclusion</h3><p>Hope this help someone (including future me).</p><p>The key is not following what version allow us to make it work, but the logic on what to check and what to try to potentially make things work</p><p>I have also been posting on the github <a href="https://github.com/dusty-nv/jetson-containers/issues/994">issue</a>, some detail not being in this article might be found there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e21f86e4a7cd" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/running-unsloth-on-a-jetson-agx-orin-device-with-jetson-containers-e21f86e4a7cd">Running Unsloth On a Jetson AGX Orin Device With Jetson-Containers</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure Cloudflare WARP setup so it work with existing VPN client]]></title>
            <link>https://faun.pub/configure-cloudflare-warp-setup-so-it-work-with-existing-vpn-client-dcd18b2ef03a?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcd18b2ef03a</guid>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[cloudflare-warp]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 08:24:49 GMT</pubDate>
            <atom:updated>2025-11-10T12:02:17.232Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>Following my <a href="https://medium.com/faun/access-internal-server-via-cloudflare-zero-trust-tunnel-for-linux-user-ef52de86fad3">previous post</a>, I have successfully configure my machine using WARP client to connect to cloudflare Zero Trust tunnel to my device inside enterprise network.</p><p>I have been using ProtonVPN, and I found that the previous setup making both client conflict with each other and I have to turn off one of them each time.</p><h3>Solution</h3><p>The solution is simple, we can make our WARP only include the private IP founded:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*gg2rjj99y7MCFfdG.png" /></figure><p>Then at the device split tunnels setting, change from exclude to include mode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*gSdDLYeEvBlgDeJk.png" /></figure><p>Then configure so that it only include the private IP you need.</p><p>One might need to wait a very short time or disconnect and connect the WARP client, that should take effect and both VPN and WARP should be able to serve their purpose side by side without needing to turn one of them off (and forget to switch on the other after use)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_nBq6GW--wIFC0ey.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcd18b2ef03a" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/configure-cloudflare-warp-setup-so-it-work-with-existing-vpn-client-dcd18b2ef03a">Configure Cloudflare WARP setup so it work with existing VPN client</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Access internal server via Cloudflare Zero Trust tunnel (for Linux user)]]></title>
            <link>https://faun.pub/access-internal-server-via-cloudflare-zero-trust-tunnel-for-linux-user-ef52de86fad3?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef52de86fad3</guid>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[cloudflare-warp]]></category>
            <category><![CDATA[tunneling]]></category>
            <category><![CDATA[cloudflare-zero-trust]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 07:53:07 GMT</pubDate>
            <atom:updated>2025-11-10T12:02:24.526Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>This is a supplement / update information of the article:</p><p><a href="https://www.kaumnen.com/p/how-to-secure-server-access-via-cloudflare">How To Secure Server Access Via Cloudflare Zero Trust &amp; WARP</a></p><p>My intention is to add some detail while I have been following the above article for my future reference.</p><p>My remote target server is a Jetson AGX Orin sit inside a enterprise network, I used to use <a href="https://github.com/fatedier/frp">FRP</a> (fast reverse proxy) a lot for proxying / exposing (with a AWS EC2 server as FRP server, and the edge compute are being FRP client).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*HqH07bIthCFLGHMSk4fH9A.png" /></figure><p>Problem is the EC2 and the network traffic need cost (not much), and the speed is not very good as my EC2 is not in the same region for my remote target and my PC (which sit in the same region).</p><p>There are yet other problems, but with knowing Cloudflare provide a tunneling technology, I am going to give it a try.</p><p>Most of the resources online point to surrender the domain on Cloudflare (which from reading the document, seems we can use DNS to route but not surrender domain is obvious and easiest way as Cloudflare can manage their own DNS record and settings without us to manage)</p><p>Also my aim is not to use hostname to access the resources, my aim only want to use a tunnel to “penetrate into” the enterprise network (which I do not have management access)</p><h3>Setup</h3><h4>1. Create the tunnel and configure the remote machine</h4><ol><li>Arrive at the Cloudflare Dashboard, on left navigation, under Protect and &amp; Connect, click Zero Trust</li><li>At Zero Trust Home, on left navigation, expand Networks, click Tunnels, and click “+ Create a tunnel”</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDO-Wb_arq1yVjPp9wQ2Xw.png" /></figure><p>3. Click “Select Cloudflared”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*APAI3sxoAyrpVWPXeHGfDA.png" /></figure><p>4. Key in a name for tunnel, then land on the configure page, choose the environment for instruction</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RyHdC_k98HZD3cSEPKdO8w.png" /></figure><p>5. In here, we would get into (when one is in the enterprise network or for me using the existing FRP proxy network) the remote target machine, and run the command accordingly.</p><p>6. Locate the Private IP of the connected tunnel (I am using my other tunnel as example below), click the connector ID link</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Hjn3H5FOKq0z9KuOoyQlA.png" /></figure><p>Then locate the private IP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v4CseYMZlFbhmorMVpAB9A.png" /></figure><p>6. Finally, go to Routes from Cloudflare Zero Trust dashboard to add the CIDR (for me I use 192.168.0.1/16, it need to align with the private IP so there is a route to the private IP)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aapr2BR8gnGeTwHU4w6KBg.png" /></figure><p>7. to prepare the remote machine able to connect with the WARP client, we also need to create the file ~/.cloudflared/config.yml, with content:</p><pre>warp-routing:<br> enabled: true</pre><p>Then use systemctl to restart cloudflared service</p><pre>sudo systemctl restart cloudflared.service</pre><h4>2. Setup the local machine</h4><p>Now, it’s the part that I am not following the article I referenced as the step to download the WARP client is behind the new Cloudflare payment page which you can select a free plan but you need to key in the payment method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjJXnEetVthzhDlfCHUEwA.png" /></figure><p>So I am using another way, the command line tool.</p><ol><li>Install the WARP client CLI</li></ol><p>Add the APT repository (I am using Ubuntu) and install the cloudflare-warp, following: <a href="https://pkg.cloudflareclient.com/">https://pkg.cloudflareclient.com/</a> (You can also locate it in Settings &gt; Resources)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PbJsnrLdsc8omZN4Nr_5tQ.png" /></figure><p>2. Register the warp client to connect to the cloudflare team</p><p>You would first need to locate the team name, at Zero Trust Home, you can see it under Manage your account (masked in below screen)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCbPhmdrFfCvUVcC9yq2DA.png" /></figure><p>Then open terminal (I am using Ubuntu as my client PC), use the cloudflare-warp installed, run:</p><pre>warp-cli registration new &lt;your teamname&gt;</pre><p>One would go through the login and some device registration page.</p><p>Got pass all setup, it would pop up an app (which in future is no longer needed or for my case, not usable), you can push the switch to connect</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*F5KFzgw83hiHiAIM4gYt0g.png" /></figure><p>In future, one can use the command (more reference: <a href="https://developers.cloudflare.com/warp-client/get-started/linux/">https://developers.cloudflare.com/warp-client/get-started/linux/</a>):</p><pre># To connect<br>warp-cli connect<br># Do you ssh to your target private IP<br># ...<br># to disconnect after use<br>warp-cli disconnect</pre><p>One thing might not work is the 192.168.0.1/16 might be excluded (the original document mentioned that but again, it point us to Settings &gt; WARP Client)</p><h4>3. Remove exclusion for private address block</h4><p>during the device registration pages, there is a step to manage the split tunnel exclusion, but the way to got back after registration could be tough.</p><p>The way to go back and manage is to go to devices, click on the device profile, and click the “edit” button on the drawer opened on the right</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xiO_h4UtY2Ubqj77Wh7ZoA.png" /></figure><p>Then scroll down to the split tunnels section and click “Manage”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NiPxflJ-cYjqNouUC9LZ-g.png" /></figure><p>Delete your private IP target if found (mine is deleted)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEP1QBq6nrdxgB8VuqTaag.png" /></figure><h3>Conclusion</h3><p>Again, the original article is well covering the process, just Cloudflare made some change in UI and my guide is just to mention them, and as a Linux user perspective to cover the steps for the WARP client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_nBq6GW--wIFC0ey.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef52de86fad3" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/access-internal-server-via-cloudflare-zero-trust-tunnel-for-linux-user-ef52de86fad3">Access internal server via Cloudflare Zero Trust tunnel (for Linux user)</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing llm-awq Tiny Chat NVILA demo with switch media in chat]]></title>
            <link>https://faun.pub/enhancing-llm-awq-tiny-chat-nvila-demo-with-switch-media-in-chat-335bd8684aee?source=rss-9add418f5830------2</link>
            <guid isPermaLink="false">https://medium.com/p/335bd8684aee</guid>
            <category><![CDATA[jetson-agx-orin]]></category>
            <category><![CDATA[tiny-chat]]></category>
            <category><![CDATA[jetsons]]></category>
            <category><![CDATA[nvila]]></category>
            <category><![CDATA[llm-awq]]></category>
            <dc:creator><![CDATA[Stephen Cow Chau]]></dc:creator>
            <pubDate>Fri, 16 May 2025 12:20:37 GMT</pubDate>
            <atom:updated>2025-05-19T10:10:43.769Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>This is a follow up of the <a href="https://medium.com/@stephencowchau/running-nvila-on-nvidia-jetson-agx-orin-64gb-development-kit-with-tinychat-d442ff56814b">last article</a>, I have been working on using NVILA on Jetson AGX Oring 64GB Development Kit, while I have been success is using llm-awq’s tiny chat to run NVILA, but the come with nvila_demo.py assume a media per chat session and require loading model every time launching the chat process, so I want to enhance it so that it can allow me load the model once and change the media and start new chat session in the process.</p><h3>Solution Code</h3><p>We reuse most of the code, except adding a new input to allow user to change media or continue with existing media or quit</p><pre>import argparse<br>import os<br><br># from termcolor import colored # Not used in the final refactored version, can be removed if not needed elsewhere<br><br>import torch<br>from transformers import AutoConfig<br><br># LLaVA and TinyChat specific imports<br>import llava<br>from llava import conversation as clib # clib not directly used, but conversation imported<br>from llava.media import Image, Video<br>from llava.utils.media import extract_media # extract_media not directly used, but good to keep if model uses it internally<br><br>from awq.quantize import fake_quant #,smooth_lm # smooth_lm imported specifically where needed<br>from tinychat.utils.load_quant import load_awq_model<br>from tinychat.utils.llava_image_processing import (<br>    load_images, # load_images not directly used, Image/Video classes are<br>    vis_images,<br>)<br>from tinychat.utils.tune import (<br>    device_warmup,<br>    tune_all_wqlinears, # Not used in original main, but part of imports<br>    tune_llava_patch_embedding,<br>)<br>from tinychat.utils.prompt_templates import (<br>    get_prompter,<br>    get_stop_token_ids,<br>    get_image_token, # Used for getting &lt;image&gt; token<br>)<br>import tinychat.utils.constants<br>from tinychat.stream_generators.NVILA_stream_gen import NVILAStreamGenerator<br>from tinychat.utils.conversation_utils import gen_params, stream_output, TimeStats<br><br><br># Function to skip certain initializations for speed, as in the original code<br>def skip(*args, **kwargs):<br>    pass<br><br># Ensure CUDA device is set (as in original)<br># Consider making this an argument if flexibility is needed.<br>os.environ[&quot;CUDA_VISIBLE_DEVICES&quot;] = &quot;0&quot;<br><br><br>def initialize_model(args):<br>    print(&quot;Initializing model... This may take a moment.&quot;)<br>    # Accelerate model initialization (from original main)<br>    setattr(torch.nn.Linear, &quot;reset_parameters&quot;, lambda self: None)<br>    setattr(torch.nn.LayerNorm, &quot;reset_parameters&quot;, lambda self: None)<br>    torch.nn.init.kaiming_uniform_ = skip<br>    torch.nn.init.kaiming_normal_ = skip<br>    torch.nn.init.uniform_ = skip<br>    torch.nn.init.normal_ = skip<br>    tinychat.utils.constants.max_seq_len = args.max_seq_len<br><br>    from tinychat.models.nvila_qwen2 import NVILAQwen2<br>    from tinychat.models.qwen2 import Qwen2ForCausalLM # Needed for model.llm<br><br>    config = AutoConfig.from_pretrained(args.model_path)<br>    config.resume_path = args.model_path<br>    if args.quant_llm or args.all:<br>        model = NVILAQwen2(config, False).half()<br>    else:<br>        model = NVILAQwen2(config, True).half()<br><br>    if args.smooth_VT or args.all:<br>        from awq.quantize import smooth_lm # Import specifically here<br>        act_scales = torch.load(args.act_scale_path)<br>        smooth_lm(model.vision_tower, act_scales, 0.3)<br><br>    if args.quant_llm or args.all:<br>        from tinychat.modules import (<br>            make_quant_norm, make_quant_attn, #make_fused_mlp, make_fused_vision_attn, # These were imported but not used in model.llm block<br>        )<br>        model.llm = Qwen2ForCausalLM(model.llm_cfg).half()<br>        model.llm = load_awq_model(model.llm, args.quant_path, 4, 128, args.device)<br>        make_quant_attn(model.llm, args.device, True)<br>        make_quant_norm(model.llm)<br>        model.llm.cpu() # Original code moved llm to CPU before moving the whole model to CUDA<br>        model.llm.resize_token_embeddings(len(model.tokenizer))<br><br>    if args.quant_VT or args.all:<br>        from tinychat.modules import QuantSiglipEncoder<br>        if args.fakequant_VT:<br>            fake_quant(model.vision_tower.vision_tower.vision_model.encoder)<br>        else:<br>            model.vision_tower.vision_tower.vision_model.encoder = QuantSiglipEncoder(<br>                model.vision_tower.vision_tower.vision_model.encoder<br>            )<br>    <br>    model = model.to(args.device).eval() # Move to device and set to eval mode<br>    device_warmup(args.device)<br>    tune_llava_patch_embedding(model.vision_tower, device=args.device)<br>    print(&quot;Model initialized successfully.&quot;)<br>    return model<br><br>def prepare_chat_environment(args):<br>    stop_token_ids = get_stop_token_ids(args.model_type, args.model_path)<br>    short_prompt_flag = args.max_seq_len &lt;= 1024<br><br>    if args.empty_prompt:<br>        input_indicator = &quot;Input: &quot;<br>        output_indicator = &quot;Generated: &quot;<br>    else:<br>        input_indicator = &quot;USER: &quot;<br>        output_indicator = &quot;ASSISTANT: &quot;<br>    return stop_token_ids, short_prompt_flag, input_indicator, output_indicator<br><br>def load_media_for_chat(model, media_paths, vis_image_flag, args):<br>    if not media_paths:<br>        return None, None, &quot;&quot;, 0<br><br>    print(f&quot;Loading media: {media_paths}&quot;)<br>    media_objects_for_prompt_construction = []<br>    media_files_for_vis = []<br>    # Determine media placeholder based on the first media item (assuming consistency if multiple)<br>    # or make it more robust if mixed media types need different placeholders in one prompt.<br>    media_placeholder_str = get_image_token(model=model, model_name=args.model_path)<br><br>    for media_path in media_paths:<br>        media_files_for_vis.append(media_path)<br>        if any(media_path.lower().endswith(ext) for ext in [&quot;.jpg&quot;, &quot;.jpeg&quot;, &quot;.png&quot;]):<br>            media_obj = Image(media_path)<br>            if not media_placeholder_str: media_placeholder_str = get_image_token() # Use utility<br>        elif any(media_path.lower().endswith(ext) for ext in [&quot;.mp4&quot;, &quot;.mkv&quot;, &quot;.webm&quot;]):<br>            media_obj = Video(media_path)<br>            if not media_placeholder_str: media_placeholder_str = &quot;&lt;vila/video&gt;&quot; # As per original code<br>        else:<br>            raise ValueError(f&quot;Unsupported media type: {media_path}&quot;)<br>        media_objects_for_prompt_construction.append(media_obj)<br><br>    num_media_items = len(media_objects_for_prompt_construction)<br><br>    if vis_image_flag and media_files_for_vis:<br>        print(&quot;=&quot; * 50)<br>        print(&quot;Visualizing Input Media:&quot;)<br>        vis_images(media_files_for_vis) # vis_images expects list of paths<br>    <br>    # This conversation structure is specifically for model.prepare_media<br>    conversation_for_media_prep = [{&quot;from&quot;: &quot;human&quot;, &quot;value&quot;: media_objects_for_prompt_construction}]<br>    prepared_media, prepared_media_cfg = model.prepare_media(conversation_for_media_prep)<br>    <br>    print(&quot;Media loaded and prepared.&quot;)<br>    return prepared_media, prepared_media_cfg, media_placeholder_str, num_media_items<br><br><br>def run_interactive_chat_session(model, args, stop_token_ids, short_prompt_flag, <br>                                 input_indicator, output_indicator,<br>                                 current_media_objects, current_media_cfg, <br>                                 media_placeholder, num_media):<br>    <br>    print(&quot;\n--- New Chat Session ---&quot;)<br>    if current_media_objects and media_placeholder:<br>        print(f&quot;Using {num_media} media item(s) (placeholder: {media_placeholder}). Type your query.&quot;)<br>    else:<br>        print(&quot;Mode: Text-only chat. Type your query.&quot;)<br><br>    model_prompter = get_prompter(<br>        args.model_type, args.model_path, short_prompt_flag, args.empty_prompt<br>    )<br>    <br>    count = 0<br>    start_pos = 0<br>    session_time_stats = TimeStats() # Stats for this specific session<br><br>    while True:<br>        print(&quot;=&quot; * 50)<br>        user_input_string = input(f&quot;{input_indicator}(Type &#39;cm&#39; to change media, &#39;quit&#39; to exit program, or your prompt): &quot;)<br>        print(&quot;-&quot; * 50)<br><br>        if user_input_string.lower() == &quot;quit&quot;:<br>            print(&quot;Exiting program...&quot;)<br>            session_time_stats.show()<br>            return &quot;quit_program&quot; <br><br>        if user_input_string.lower() == &quot;cm&quot;:<br>            print(&quot;Ending session to change media...&quot;)<br>            session_time_stats.show()<br>            return &quot;change_media&quot;<br><br>        input_prompt_text = user_input_string<br>        if user_input_string == &quot;&quot;:<br>            if count == 0 and current_media_objects: # Empty prompt on first turn with media<br>                input_prompt_text = &quot;Describe the content in detail.&quot; # Default action<br>                print(f&quot;{input_indicator}{input_prompt_text} (default action)&quot;)<br>            else: # Empty prompt not allowed otherwise, or no media to default describe<br>                print(&quot;Please provide a prompt, or type &#39;cm&#39; or &#39;quit&#39;.&quot;)<br>                continue<br>        <br>        # --- Construct the prompt for the model ---<br>        media_input_for_stream_gen = None<br>        media_cfg_for_stream_gen = None<br>        <br>        final_input_to_prompter = input_prompt_text<br>        if count == 0: <br>            if current_media_objects and media_placeholder:<br>                if media_placeholder not in input_prompt_text: # Avoid duplicating placeholder<br>                    final_input_to_prompter = (media_placeholder * num_media) + &quot; &quot; + input_prompt_text<br>                media_input_for_stream_gen = current_media_objects<br>                media_cfg_for_stream_gen = current_media_cfg<br>            model_prompter.insert_prompt(final_input_to_prompter)<br>        else: <br>            model_prompter.insert_prompt(input_prompt_text)<br>            if args.chunk_prefilling: <br>                media_input_for_stream_gen = None<br>                media_cfg_for_stream_gen = None<br>            # Else, for non-chunk_prefilling, media is typically not resent after first turn either.<br>            # NVILAStreamGenerator will use what&#39;s in model_prompter.model_input.<br><br>        stream_gen_instance = NVILAStreamGenerator(<br>            model,<br>            gen_params, <br>            model_prompter.model_input, <br>            media_input_for_stream_gen, <br>            media_cfg_for_stream_gen,<br>            start_pos,<br>            device=args.device,<br>            stop_token_ids=stop_token_ids,<br>            chunk_prefilling=args.chunk_prefilling,<br>            quant_llm=args.quant_llm or args.all,<br>        )<br>        <br>        print(output_indicator, end=&quot;&quot;, flush=True)<br>        <br>        turn_outputs = &quot;&quot;<br>        turn_total_tokens = 0<br>        # Pass session_time_stats for the first generation of this session if not chunk_prefilling<br>        if count == 0 and not args.chunk_prefilling:<br>            turn_outputs, turn_total_tokens = stream_output(stream_gen_instance, session_time_stats)<br>        else:<br>            turn_outputs, turn_total_tokens = stream_output(stream_gen_instance)<br><br>        if args.chunk_prefilling:<br>            start_pos += turn_total_tokens<br>        <br>        if not args.single_round and args.max_seq_len &gt; 512:<br>            model_prompter.update_template(turn_outputs, args.chunk_prefilling)<br>        <br>        count += 1<br><br><br>def main_loop(args):<br>    model = initialize_model(args)<br>    stop_token_ids, short_prompt_flag, input_indicator, output_indicator = prepare_chat_environment(args)<br><br>    active_media_paths = args.media <br>    <br>    while True:<br>        current_media_data = None<br>        current_media_config = None<br>        media_placeholder_token = &quot;&quot;<br>        num_active_media = 0<br><br>        if active_media_paths:<br>            try:<br>                current_media_data, current_media_config, \<br>                media_placeholder_token, num_active_media = \<br>                load_media_for_chat(model, active_media_paths, args.vis_image, args)<br>            except ValueError as e:<br>                print(f&quot;Error loading media &#39;{&#39; &#39;.join(active_media_paths)}&#39;: {e}&quot;)<br>                print(&quot;Proceeding without media for this session. You can try &#39;cm&#39; with valid paths.&quot;)<br>                active_media_paths = None <br>                current_media_data, current_media_config, media_placeholder_token, num_active_media = None, None, &quot;&quot;, 0<br>        else:<br>            # This handles both initial start without media and choosing text-only after &#39;cm&#39;<br>            print(&quot;No media specified. Starting in text-only mode.&quot;)<br>            current_media_data, current_media_config, media_placeholder_token, num_active_media = None, None, &quot;&quot;, 0<br><br>        session_result = run_interactive_chat_session(<br>            model, args, stop_token_ids, short_prompt_flag,<br>            input_indicator, output_indicator,<br>            current_media_data, current_media_config,<br>            media_placeholder_token, num_active_media<br>        )<br><br>        if session_result == &quot;quit_program&quot;:<br>            break <br>        <br>        elif session_result == &quot;change_media&quot;:<br>            new_paths_str = input(&quot;Enter new media path(s) (space-separated), or press Enter for text-only, or type &#39;quit&#39; to exit: &quot;).strip()<br>            if new_paths_str.lower() == &quot;quit&quot;:<br>                print(&quot;Exiting program...&quot;)<br>                break <br>            elif not new_paths_str: <br>                active_media_paths = None # User wants text-only for the next session<br>            else:<br>                active_media_paths = new_paths_str.split()<br>        # Loop continues for a new session<br>    print(&quot;Program finished.&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    parser = argparse.ArgumentParser(description=&quot;Interactive chat with NVILA model, allowing media changes.&quot;)<br>    parser.add_argument(<br>        &quot;--model_type&quot;, type=str, default=&quot;LLaMa&quot;, help=&quot;Type of the model (e.g., LLaMa, Qwen)&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--model-path&quot;, type=str, required=True, help=&quot;Path to the pretrained model checkpoint&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--quant_path&quot;, type=str, help=&quot;Path to the quantized model checkpoint (AWQ)&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--act_scale_path&quot;, type=str, help=&quot;Path to activation scales for quantization (smoothquant)&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--media&quot;, type=str, nargs=&quot;+&quot;, help=&quot;Initial multi-modal input (image or video paths)&quot;<br>    )<br>    parser.add_argument(&quot;--device&quot;, type=str, default=&quot;cuda&quot;, help=&quot;Device to run the model on (e.g., cuda, cpu)&quot;)<br>    parser.add_argument(&quot;--max_seq_len&quot;, type=int, default=2048, help=&quot;Maximum sequence length for the model&quot;)<br>    parser.add_argument(<br>        &quot;--single_round&quot;, action=&quot;store_true&quot;, help=&quot;If set, do not memorize previous conversation turns&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--vis-image&quot;, action=&quot;store_true&quot;, help=&quot;Visualize the input image(s) during chat&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--empty-prompt&quot;, action=&quot;store_true&quot;, help=&quot;Use an empty prompt template (minimal system messages)&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--flash_attn&quot;, action=&quot;store_true&quot;, help=&quot;Enable Flash Attention (if supported by the model)&quot; # Note: Original code didn&#39;t use this flag in main logic.<br>    )<br>    parser.add_argument(<br>        &quot;--chunk_prefilling&quot;, action=&quot;store_true&quot;, help=&quot;Optimize context stage by not recalculating history tokens&quot;<br>    )<br>    # Quantization and Smoothing Options<br>    parser.add_argument(&quot;--quant_llm&quot;, action=&quot;store_true&quot;, help=&quot;Apply AWQ quantization to the LLM&quot;)<br>    parser.add_argument(&quot;--quant_VT&quot;, action=&quot;store_true&quot;, help=&quot;Apply quantization to the Vision Tower&quot;)<br>    parser.add_argument(&quot;--smooth_VT&quot;, action=&quot;store_true&quot;, help=&quot;Apply activation smoothing to the Vision Tower before quantization&quot;)<br>    parser.add_argument(<br>        &quot;--fakequant_VT&quot;, action=&quot;store_true&quot;, help=&quot;Use fake quantization for the Vision Tower instead of real quantization&quot;<br>    )<br>    parser.add_argument(&quot;--all&quot;, action=&quot;store_true&quot;, help=&quot;Shorthand to enable all main quantization and smoothing options (quant_llm, quant_VT, smooth_VT)&quot;)<br><br>    args = parser.parse_args()<br><br>    # If --all is specified, enable the relevant individual flags<br>    if args.all:<br>        args.quant_llm = True<br>        args.quant_VT = True<br>        args.smooth_VT = True<br>        # args.fakequant_VT is intentionally left separate as it&#39;s an alternative to real quant_VT.<br><br>    # Validate that required paths for quantization/smoothing are provided if those options are selected<br>    if args.quant_llm and not args.quant_path:<br>        parser.error(&quot;--quant_llm requires --quant_path to be specified.&quot;)<br>    if args.smooth_VT and not args.act_scale_path:<br>        parser.error(&quot;--smooth_VT requires --act_scale_path to be specified.&quot;)<br>    <br>    main_loop(args)</pre><h3>Result</h3><p>We can see from below, the model is load once, while when I type “cm” (change media) when I want to change to a new media, it start a new session</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ylb8bWJTDCEzrxfAMnC8iw.png" /></figure><p>If I do not use “cm” but continue chatting, it would have the conversation continue (as below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFQvAQIhJMjBimOrqag_rQ.png" /></figure><h3>Conclusion</h3><p>From the result, I start thinking the benchmark’s <strong>T</strong>ime <strong>t</strong>o <strong>F</strong>irst <strong>T</strong>oken (TTFT), is it <strong>AFTER</strong> the image processing? I can see 0.276s for TTFT, but…I don’t think that’s their intention? I have to read the benchmark code in order to find out</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*rBrrQUK7_REGLs60.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=335bd8684aee" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/enhancing-llm-awq-tiny-chat-nvila-demo-with-switch-media-in-chat-335bd8684aee">Enhancing llm-awq Tiny Chat NVILA demo with switch media in chat</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>