How dogfooding cljgo with a seven-language agent library turned into a Clojure port with zero reader conditionals — and a portability seam called koine that keeps the ugly parts somewhere else.

Building a language host(cljgo) is easy to fool yourself about. It runs your test suite, it runs your toy programs, and you conclude it works. So I went looking for something big enough to actually hurt.

I already had one: toolnexus, a library I maintain that gives any LLM real tool-calling — MCP servers, agent skills, your own functions, sub-agents — already ported across six languages (JavaScript, Python, Go, Java, C#, Elixir), all pinned to one behavioral contract. Porting it to Clojure would give Clojure developers something they didn’t have. Running that same port on cljgo would tell me whether cljgo is real.

The rule that made it interesting: no reader conditionals

The obvious way to write code for two hosts is #?(:clj ... :cljgo ...) sprinkled wherever they differ. I refused to do that, for a selfish reason and a principled one.

The selfish one: every conditional is a place where the two hosts can silently drift apart, and drift is precisely the bug this project exists to prevent.

The principled one: toolnexus should not know cljgo exists. A library about LLM tool-calling has no business containing opinions about which Clojure you run. If it does, then every future host — Babashka, ClojureCLR, whatever comes next — means editing the library.

So the host differences went somewhere else: koine, a small portability seam. File I/O, subprocesses, HTTP, JSON, environment, time, string handling above the BMP — anything the two hosts disagree about lives there and nowhere else. koine is where the conditionals go; toolnexus stays pure Clojure.

The result is one source tree with zero reader conditionals, verified in five execution modes — JVM main, JVM REPL, cljgo AOT, cljgo interpreted, cljgo REPL — with a CI gate that fails if the five disagree by a single assertion. Currently 395 tests, 1,614 assertions, five identical verdicts.

That means the whole dependency story for a Clojure developer is one line:

;; deps.edn — koine comes along with it
{:deps {net.clojars.muthuishere/toolnexus {:mvn/version "0.13.0"}}}

Any Clojure developer can use this and never think about cljgo. And the one cljgo developer — me, so far — gets the same library, unmodified. That’s the arrangement I wanted.

So what is toolnexus?

It is not a framework, and there is no runtime to deploy. It is a library built on one observation:

An MCP server tool, an agent skill, a function you wrote, an HTTP endpoint, a shell command, and a remote agent are the same thing to an LLM — a named, described, schema’d callable.

toolnexus unifies all of those behind one Tool interface, emits the schema in OpenAI / Anthropic / Gemini formats, and ships a client with the tool-calling loop already written: parallel and chained calls, hooks, retries, conversation memory, human-in-the-loop suspension, observability.

Concretely, you get:

• an MCP host — point it at an mcp.json and every server's tools (stdio and streamable-HTTP) join your registry
• agent skills — the open SKILL.md standard, loaded on demand rather than stuffed into the prompt
• your own functions, HTTP endpoints, and ten built-in tools (bash, read, write, edit, grep, glob…)
• sub-agents with scoped tool views and hard budgets, and remote A2A agents

All seven ports run the same examples/ fixtures and must produce the same behavior. The Clojure port is held to the same full conformance tier as the other six — it is not the junior member.

Enough architecture. Here is the thing running.

One agent, three tool sources, your terminal

Let’s build the thing every framework promises and rarely shows: an interactive agent you keep asking questions, backed by three different tool sources at once —

• an HTTP tool — one declaration that fetches the official Clojure events page
• an MCP server — @modelcontextprotocol/server-filesystem, spawned over stdio; all fourteen of its tools join the registry automatically
• an agent skill — a SKILL.md telling the model how to report events, loaded on demand

(Skills are the open SKILL.md standard — markdown instructions the model loads when the task matches. The catalog goes into the system prompt; the full text loads only on request. Progressive disclosure, not prompt stuffing.)

The skill — skills/clojure-events/SKILL.md:

---
name: clojure-events
description: Use when asked about Clojure community events, meetups or conferences. How to fetch and report them.
---
# How to report Clojure events

1. Fetch the official events page with your `clojure-events-page` tool — never
   answer from memory, events go stale.
2. List each event you find as: **name** — date — location (or "online").
3. Only report events that are actually on the page. If the page lists none,
   say so.
4. Close with the source link: https://clojure.org/community/events
5. Keep it a plain list, no marketing tone.

The program — src/chat.cljc, the whole thing:

(ns chat
  (:require [clojure.string :as str]
            [koine.env :as env]
            [koine.json :as json]
            [toolnexus.core :as tn]
            [toolnexus.client :as client]
            [toolnexus.http :as http]))

(def events-page
  (http/http-tool
   {:name        "clojure-events-page"
    :description "Fetches the official Clojure community events page and returns it as text."
    :method      :get
    :url         "https://clojure.org/community/events"
    :result-mode "text"}))

(def mcp-config
  {:mcpServers
   {:files {:type    "local"
            :command ["npx" "-y" "@modelcontextprotocol/server-filesystem" "."]
            :timeout 30000}}})

(defn -main [& _]
  (let [tk (tn/build {:skills   "skills"
                      :builtins false
                      :mcp      mcp-config
                      :tools    [events-page]})
        c  (client/create-client
            {:base-url "https://openrouter.ai/api/v1"
             :style    "openai"
             :model    (or (env/get-env "TN_MODEL") "openai/gpt-4o-mini")
             :api-key  (env/get-env "OPENROUTER_API_KEY")})]
    (loop []
      (print "you> ") (flush)
      (let [line (read-line)]
        (cond
          (or (nil? line) (#{"exit" "quit"} (str/trim line)))
          (do (println "bye.") (tn/shutdown! tk)) 
           (str/blank? line) (recur)
          :else
          (let [r (client/ask c line {:toolkit tk :id "cli" :on-event trace})]
            (println (str "agent> " (:text r)))
            (recur)))))))

Note client/ask with an :id — that is conversation memory. Every question you type continues the same conversation, so follow-ups like "which of those is soonest?" just work.

Run it:

$ cd clojure/examples && task clj-ex1

It greets you with the full inventory — one HTTP tool, fourteen files_* MCP tools, the skill loader, the clojure-events skill — suggests a few questions, and waits at you>.

Recorded with vhs, unedited.

The same file on cljgo — no build step

$ task cljgo-ex1

That runs the same chat.cljc through cljgo run — Clojure hosted on Go, interpreted directly, no JVM anywhere. The startup banner is worth reading once: cljgo prints net.clojars.muthuishere/toolnexus 0.13.0 — 16 namespace(s) with no Java interop and notes it pruned org.clojure/clojure ("cljgo IS the Clojure implementation") — that is the static no-interop check happening for real, on the same artifact from Clojars. If you want the 20 ms cold start, cljgo build will also AOT it to a self-contained native binary — but nothing here requires it.

A real session on cljgo, unedited:

Both projects ship in the repo under clojure/examples (clj-ex1 and cljgo-ex1) with the Taskfile; MODEL= overrides the OpenRouter model per run.

The parts the example didn’t show

The demo used three tool sources. The rest of the library is there too, in all seven ports, because parity is the product:

• Sub-agents — an agent is a tool: a registry of agents with scoped toolkit views, teams, and hierarchical budgets (:max-tokens, :max-turns, :max-wall-ms…) that stop loudly, never silently
• Personas — from-dir composes an identity from files (SOUL.md, MEMORY.md…); a memory tool lets an agent edit its own durable notes; start-agent gives it a heartbeat
• Human in the loop — a tool can suspend with a question; the run comes back pending and resumes when your host answers. MCP servers that elicit input mid-call are bridged onto the same mechanism
• Compaction — a :before-llm hook that keeps a long-lived agent inside its context window
• Remote agents — expose your toolkit as an A2A agent with serve, or consume someone else's; either way it is just more tools
• Built-in tools — bash, read, write, edit, grep, glob and friends, with per-tool toggles

The honest numbers

We benchmarked all seven ports plus a dozen competitor frameworks on one machine in one sitting, and published the table as measured. The Clojure port is currently the slowest port over MCP — ~5.5 ms p50 per request against Go’s 0.49 — and we know exactly why: ~2 ms per stdio round-trip in the JSON-RPC path, already logged as the top optimization target. With native tools it runs the same scenario in 1.7 ms, mid-table, ahead of LangChain.js and Mastra. And the number I actually care about: the two hosts agree to within 0.2 ms from the same source file. The parity claim survives being measured under load.

If a library only tells you its winning numbers, it’s an ad. Full table: performance page.

Three gotchas so you don’t find them the hard way

1. The API-key env chain prefers OPENAI_API_KEY for openai-style clients. If that's set and you're calling OpenRouter, pass :api-key explicitly (as the demo does) — otherwise your OpenAI key goes to OpenRouter and you get a confusing 401.
2. cljgo run evaluates top-level forms and doesn't call -main. Add a two-line entry file that requires the ns and calls it — cljgo-ex1's src/run_chat.cljc is exactly that — or build a binary.
3. Write non-ASCII characters directly in source, not as \uXXXX surrogate escapes — the JVM reader assembles the pair, cljgo's doesn't. (Yes, we found that one the interesting way.)

Try it, break it, tell me

If you are a Clojure developer, you can use toolnexus today on the JVM and never think about any of the above:

{:deps {net.clojars.muthuishere/toolnexus {:mvn/version "0.13.0"}}}

If you are curious whether Clojure-on-Go is real, clone the examples and run task cljgo-ex1. That is the same question I was asking, and running it is a better answer than anything I can write here.

• toolnexus — repo · docs, all seven languages · Clojars
• The examples in this article — clojure/examples
• cljgo — repo · Clojure hosted on Go
• koine — repo · the portability seam

The problem

Every LLM project starts the same way. You wire up a couple of MCP servers. You want “agent skills” — the SKILL.md progressive-disclosure pattern from Claude and opencode, where the model loads an instruction file on demand. You expose a few of your own functions as tools, and maybe a REST endpoint or two. Then you write the tool-calling loop: send the tools, get a tool call, run it, feed the result back, repeat.

Then you do it again for the next project — in a different language, against a different model provider, with a slightly different glue layer. The frameworks that exist are either single-vendor, single-language, or a whole platform when you wanted a library.

The insight

To an LLM, an MCP server tool, an agent skill, a native function, and an HTTP endpoint are the same thing: a named, described, schema’d callable. So toolnexus unifies all four behind one Tool interface, aggregates them in a Toolkit, and emits the schema in OpenAI / Anthropic / Gemini shape. A unified client drives any endpoint with them.

MCP servers ─┐
 Agent skills ─┤
 Native fns  ─┼─▶  one uniform Tool[] registry  ─▶  any LLM (OpenAI / Anthropic / Gemini)
 HTTP / REST ─┘        + the host loop

Three steps to an agent

1. An mcp.json — your MCP servers (local stdio + remote HTTP, ${ENV} headers for auth).
2. A skills/ folder — <name>/SKILL.md files; a skill tool loads them on demand and the catalog is injected into the system prompt.
3. One call — agent.run("…", { toolkit }). The model already sees every MCP tool + the skill tool and drives them itself.

Prefer your own loop? Use toOpenAI() / toAnthropic() / toGemini() for the schema and execute(name, args) to run a call.

What makes it different

• Five languages, one behavior. JS/TS, Python, Go, Java, C# — pinned by a shared SPEC.md so they stay byte-identical (the skill-loader output is literally byte-for-byte). First-class Go and C#, where most agent tooling doesn't reach.
• Built on the real SDKs. Each port uses that ecosystem’s most popular MCP SDK — nothing reinvented.
• A library, not a framework. À la carte: want only an MCP host? Use just the loader. Skills, native/HTTP tools, and the loop are opt-in on top.
• Vendor-neutral. A plain base URL + "openai" or "anthropic" style. No lock-in.
• Batteries verified. Hooks, streaming, retries/backoff, conversation memory, a Go CLI — and every port’s examples run live against OpenRouter.

Install

npm i toolnexus                                   # JS / TypeScript
pip install toolnexus                             # Python
go get github.com/muthuishere/toolnexus/golang    # Go
dotnet add package Toolnexus                       # C#  (Maven Central for Java: coming)

Try it

const tk    = await createToolkit({ mcpConfig: "./mcp.json", skillsDir: "./skills" })
const agent = createClient({ baseUrl: "https://openrouter.ai/api/v1", style: "openai", model: "openai/gpt-4o-mini" })
const { text } = await agent.run("Refund order 1234 for the customer.", { toolkit: tk })

That’s the whole thing.

→ https://github.com/muthuishere/toolnexus

It’s not the right answer for me, most days.

What I actually need on a regular evening is much smaller. Look at a dev secret. Maybe change it. Drop a JSON service-account key next to it. Add a regression fixture that mirrors prod shape. Share it with a friend or a teammate. Push it to GitHub Actions and GCP Secret Manager so my VPS deploys aren’t a separate dance.

That’s the whole job. And every tool I tried either over-solved it, under-solved it, or made me commit something to git I shouldn’t have.

So I wrote down every specific pain I had. Fixed each one. Called the result vsync.

The pains, in order

1. The .env file is the friendliest file format we have. Don't make me give it up.

Every backend language reads .env either with dotenv or taskfile

@Value("${DATABASE_URL}") String dbUrl;     // Spring Boot

dbURL := os.Getenv("DATABASE_URL")          // Go + godotenv

db_url = os.environ["DATABASE_URL"]         // Python + python-dotenv

const dbUrl = process.env.DATABASE_URL      // Node + dotenv

Most “secret managers” want me to replace dotenv.config() with their SDK call. No thank you. The whole reason .env works is that it's a one-line bootstrap every language already understands.

Fix: Vsync uses .env as-is. vsync use dev symlinks ./.env into your vault. Your app keeps calling dotenv.config() with no path argument. Nothing in your bootstrap changes.

2. Sharing .env files is awful

DM the file in teams. Teammate pastes it. App crashes because their copy is stale. You DM again. Then they need the JSON service-account key too. You attach that to MS Teams. Plaintext. You should delete it later. You don’t.

Fix: vsync export dev mints a .share file + a passphrase. Send the file on one channel (MS Teams), the passphrase on a different channel (SMS, password manager). Intercept one alone — you have nothing. Teammate runs vsync import dev <file> and vsync pull dev. About thirty seconds, soup to nuts.

3. File paths inside .env break on every other laptop

GCP_SA_KEY_FILE_PATH=/Users/muthu/Downloads/gcp-sa.json
SSH_KEY_PATH=/Users/muthu/.ssh/id_ed25519

That /Users/mrmut/… doesn't exist on your teammate's machine. Now they need both the .env and the file it points to. More MS Teams DMs.

Fix: Vsync ships the whole vault folder, not just the env file. Drop your JSON keys, certs, regression fixtures into infra/vault/<env>/ right next to the env file. Use relative paths inside .env:

GCP_SA_KEY_FILE_PATH=./infra/vault/dev/gcp-sa.json
SSH_KEY_PATH=./infra/vault/dev/keys/id_ed25519

After vsync pull dev, those files are on every teammate's disk at the same paths. No more "send me the JSON too."

4. Encrypting-in-git is dangerous

I lived on git-crypt for a year. Good tool, but:

• The encrypted blob lives in the same repo. Forget the .gitattributes filter on a new file — you commit plaintext. That actually happened to me. Twice.
• Doesn’t play well with git worktree. Keys live in .git/git-crypt/ and don't follow you to a new worktree. Every fresh worktree was a manual git-crypt unlock before anything worked. Forget that and your app boots with placeholder strings that look like real values. Hardest kind of bug to debug.
• Onboarding a teammate meant adding their GPG key, re-encrypting everything, re-pushing. A five-minute job I always procrastinated.

Every one of those was “a separate piece of work I had to do.” Individually fine. Together — friction I stopped wanting to pay.

Fix: Vsync’s encrypted bundle lives on S3, not in your repo. infra/vault/ is gitignored. The plaintext physically cannot enter your application repo by accident — there's no filter to forget. Worktrees work because the AES key is per-machine in the OS keychain, not per-checkout.

5. Manual sync to GitHub Actions and GCP

Every VPS deploy used to be: edit the secret in my .env, then re-paste it in GitHub Actions secrets, then update GCP Secret Manager too. Always one of the three drifted.

Fix: vsync sync dev gh|gcp|all reads .env.<env> and ships every key/value to GitHub Actions / GCP Secret Manager via their CLIs. One vault. Three destinations. One command.

It even handles the JSON-key case: GCP_SA_KEY_FILE_PATH=./infra/vault/dev/gcp-sa.json in your env file → vsync reads the file at sync time and ships the contents as GCP_SA_KEY to the remote store.

6. Zero audit — who saw what, when?

With .env + MS Teams, I have no idea who pulled which secrets, from which laptop, on which date. If a prod password leaks tomorrow, I can't narrow down which teammate's machine to suspect first.

Fix: Every push / pull / import / export appends a row to s3://<bucket>/<repo>/<env>/audit.csv. Columns: ts, action, version_ts, hostname, local_ip, os_user, git_email, vsync_version, bun_version, meta.

$ vsync audit dev

TS                        ACTION  VERSION          USER     HOST              NOTE
------------------------  ------  ---------------  -------  ----------------  ------------------
2026-05-16T11:39:49.145Z  pull    20260516-113343  alice    alice-mbp.local   ticket-AUDIT-123
2026-05-16T11:33:45.684Z  push    20260516-113343  alice    alice-mbp.local   prod hotfix
2026-05-16T08:14:02.211Z  pull    20260515-201410  bob      bob-thinkpad      —

Not legal-grade tamper-evidence — anyone with bucket-write could edit the CSV(You can add policies too, but i havent tried !). But enough to answer “who pulled prod last Wednesday at 3 AM?” in one command, which is mostly what I want.

7. Existing solutions are paid SaaS or a control plane to run

Doppler, Infisical, dotenvx — all good tools. Some have free tiers, some have paid tiers, all of them eventually want you on their cloud. Or you self-host Infisical / Vault and now you’re running a Postgres + Redis + control-plane on top of whatever you’re already running.

Fix: Vsync is MIT. Totally free. No SaaS sign-up. No per-seat pricing. No control plane. BYO S3-compatible bucket — you probably already have one for backups. Works with AWS S3, Hetzner Object Storage, MinIO, Cloudflare R2, Backblaze B2.

If you outgrow it — Infisical / Doppler / Vault are real upgrade paths. Vsync sits one tier below them on purpose.

What it looks like

bun install -g @muthuishere/vsync
vsync init dev                 # generate key + config + vault folder
# drop secrets into infra/vault/dev/
vsync push dev                 # AES-256-GCM → your bucket
vsync export dev               # → .share file + passphrase for a friend

Friend / teammate:

vsync import dev ./received.share
vsync pull dev
vsync use dev                  # ./.env → infra/vault/dev/.env.dev

That’s the whole loop.

The docs site covers the rest — the two-halves security model (bucket creds on disk + AES key in OS keychain, both required to decrypt anything), crypto envelopes, manifest pointer-seal, audit append protocol with ETag-conditional retry. If any of that interests you, the docs go deep. If it doesn’t — you can use vsync without ever reading any of it. That was the point.

What I gave up

• Bun ≥ 1.2.21 on PATH. Bun.secrets gives me cross-platform OS keychain access without shipping native modules. If Bun is a dealbreaker — I get it.
• Tamper-evident audit. The CSV is transparency for honest people, not legal forensics.
• No GUI, no daemon, no sidecar. A CLI you run when you need it (or give your agent the docs it can do on its own).

Try it

• muthuishere.github.io/vsync
• github.com/muthuishere/vsync
• npm

APIs come together. Flows start working. The UI looks complete. Very quickly it feels like the system is 80% done. Everyone relaxes.

Then the second half begins.

In 1985, Tom Cargill of Bell Labs gave this pattern its name:

“The first 90% of the code accounts for the first 90% of the development time. The remaining 10% accounts for the other 90% of the development time.”

— Communications of the ACM, Programming Pearls (Jon Bentley), 1985

Software doesn’t get built linearly. It gets built in two phases that feel nothing alike.

Phase 1 is fast and visible. Features appear. The demo works. Momentum is real.

Phase 2 is slow and invisible. Edge cases appear. Assumptions break. Real usage arrives. This is where systems either harden or fall apart.

What LLMs changed

Phase 1 has been compressed. With tools like Claude, you can move from idea to running code in an afternoon. You hit “80% done” faster than was possible a year ago.

Phase 2 is exactly as hard as it ever was.

Because the last 20% was never about writing code. It’s about uncovering hidden assumptions, identifying failure paths, understanding real usage, and validating decisions under pressure.

Before LLMs, the bottleneck was “can we build this?” Now the bottleneck is “did we think about this deeply enough?”

The scarce resource is questions

Answers are easy now. Anyone can generate code, architecture, or docs on demand. What’s become rare is the ability to ask the right questions — the uncomfortable ones:

• What happens when this fails in production?
• Who controls this input?
• What assumption are we making here?
• Is this solving the real problem, or a version of it?

These questions decide whether a system survives first contact with users.

But decisions don’t stay in one room

A decision gets made. Then it moves.

A developer explains it to another developer. Then to a manager. The CTO explains it to the CEO. Each retelling strips something — the tradeoff, the edge case, the assumption nobody wrote down.

The intent gets diluted. The risks get softened. The wrong thing starts sounding right.

LLMs give a clean answer — framed in one voice.

The question isn’t just “what’s the right answer?” It’s “can this decision survive every room it enters?”

Good systems don’t only fail because of bad code. They fail because the thinking didn’t survive translation.

They don’t come from one voice

A backend engineer sees failure modes. A security engineer sees blast radius. A product manager sees the value metric. An architect sees capacity math. A tech writer makes the decision legible to the next team. A storyteller frames it for the room above engineering.

Each asks different questions — and most of those questions arrive too late, in incident channels and retros.

Huddle exists to pull those questions to the front, in every voice the decision will eventually need to survive.

It runs repo-aware, multi-persona engineering discussions before the system is locked in. 21 named personas — Shaama (backend, rollback-first), Senthil (security, blast radius), Prabagar (PM, value metric), Suren (architect, DORA and boring tech), Deepak (tech writer, handoff-survivable docs), Kishore (storyteller, kills bullet points), and more — disagree with each other, grounded in your repo, then stop and wait for your call.

Install:

npx skills add muthuishere-agent-skills/huddle

Better systems aren’t built by faster answers. They’re built by better questions, asked at the right time — and answers that survive every room they enter.

Huddle helps you ask the right questions — and makes sure the answers survive every room they enter.

https://github.com/muthuishere-agent-skills/huddle/

Status: Open Source · github.com/muthuishere/offline-llm-knowledge-system

The Problem

You’re deploying software to a factory floor. An aircraft carrier. A hospital network. A customer site behind a firewall.

Your team needs an AI assistant that knows your product. Your runbooks. Your debug commands. But there’s no internet.

You can’t call OpenAI. You can’t hit Anthropic. You can’t even reach HuggingFace.

You have a browser. That’s it.

This isn’t niche. Every enterprise, every OT network, every air-gapped system hits this wall.

What I Built

Two web applications. Both run in the browser. Zero backend.

App 1: Export (needs internet once)

• Drop your documents
• Choose your LLM (Phi-3-mini, Gemma-2B, or TinyLlama)
• Model downloads from HuggingFace
• Documents get chunked and embedded
• Click Export → one zip file (500MB–2.4GB)

App 2: Import (never needs internet)

• Drop the zip
• Chat immediately
• All answers from your docs

No Python. No Node. No Docker. Just modern browsers.

One zip. One import. Chat works — forever offline.

The Use Cases

Why Browsers?

Because a browser is already there. Always No installer. No admin rights. No dependency hell. No pip install. No PATH drama.

A tiny model runs entirely on CPU in WebAssembly. No GPU needed.

The entire stack runs inside one browser tab.

What’s in the zip

knowledge-base.zip
├── manifest.json           ← model config, chunk settings, source list
├── chunks.json             ← all document chunks with pre-computed vectors
├── orama-index.json.gz     ← BM25 full-text search index (offline)
├── instructions.md         ← your system prompt
├── sources/                ← original documents
├── model/                  ← LLM weights (Phi-3-mini, Gemma-2B, or TinyLlama)
└── embed-cache/            ← embedding model (bge-small-en-v1.5, pre-bundled)

Everything the browser needs. Nothing more.

The search pipeline

When you ask a question:

1. Your query → 384-dimensional vector (ONNX worker, off main thread)
2. Hybrid search: BM25 text + cosine similarity → top 3 chunks
3. Chunks injected into the user message (not system) → model grounds on them
4. LLM generates answer in a Web Worker → UI stays responsive
5. Answer streams back token by token

Runs entirely on CPU. Works anywhere.

Why context goes in the user turn, not system

Small models are trained on instruction-following. They attend much better to content in the user turn than buried in the system prompt.

Classic RAG mistake: stuff retrieved chunks into the system message. Models treat them as background noise, not grounding facts. Answers get vague. Hallucinations creep in.

Gemma-2B gives the best grounding accuracy among the three models I tested (Phi-3-mini, Gemma-2B, TinyLlama).

Correct shape for small models:

System: (tiny, stable, <200 chars)
  "You are a grounded assistant. Answer only from context.
   If unsure, say so. Cite sources. Complete sentences."

User:
  Context:
  [DOC 1 — maintenance-guide.md]
  ... chunk text ...

  [DOC 2 — error-codes.md]
  ... chunk text ...

  Question: Why does error E-4417 appear after restart?

This shape gets 3–5× better grounding on smaller models than standard RAG prompts.

Why tiny chunks

Standard RAG uses 512-token chunks. For a 4K-context model running 3-chunk retrieval, that’s 1,536 tokens before you even add the question. Small models have no room left for output.

I use 100-token chunks (~400 chars).

Three chunks = 400 tokens. Plenty of room. More focused retrieval. Better answers.

The Stack

The embed model bundling trick

At export time, after embedding runs, I drain the browser’s transformers-cache Cache API — where transformers.js stores model files from HuggingFace.

I capture only the model files (not CDN runtime), bundle them as embed-cache/N.bin with a URL index.

At import time, the unzip worker restores them into cache with original URLs as keys.

When the embed worker starts pipeline(), transformers.js finds files in cache. Zero network requests.

Fully offline after one export.

What This Unlocks

• No-internet deployments: factory floors, ships, aircraft, hospitals, government networks
• Data sovereignty: your docs never leave your machine. Ever.
• Zero infrastructure: no vector DB, no GPU server, no API keys
• Portability: one zip, any browser, any OS
• Customizable: swap the LLM, change the prompt, add docs — just re-export
• Debuggable: kb.ask("question"), kb.search("question"), kb.prompt("question") in the console

The Console API

This tool is for technical environments. So there’s a full console API for testing without the UI:

// Ask a question (embed + search + LLM)
await kb.ask("Why does error E-4417 appear after restart?")

// Just show what documents were retrieved
await kb.search("network timeout configuration")

// Show the full prompt being sent to the LLM (debug)
await kb.prompt("How do I restart the monitoring service?")

// Show conversation history
kb.history()

// Batch test multiple questions
await kb.batch([
  "What is the default retry limit?",
  "How do I clear the event log?",
  "What ports need to be open?"
])

Drop this in your debug session. Keep it in your ops runbook. Test your knowledge base before shipping the zip to the field.

Why Now

The models got small enough.

TinyLlama: 550MB. Gemma-2B: 1.4GB. Phi-3-mini: 1.3GB.

The tooling matured: WebLLM, transformers.js, Orama.

WebAssembly made CPU inference practical.

Gemma-2B delivers the best accuracy while staying under 1.6GB.

The gap between “LLM works on your laptop” and “LLM works with your data in your browser” closed.

It’s now 15 lines of config and a zip file.

I tried existing Excalidraw MCP/Skill tools, but they didn’t work for me. Too rigid, too polished, no sketch feel, no animation, and they don’t fit into my coding flow. I don’t want a “perfect diagram generator” — I just want a fast starting point.

So I built a small skill: I describe what I want, and it generates a hand-drawn diagram(+ an animated version), opens in my hosted Excalidraw editor, and even gives me an animation view. From there, I just drag, tweak, and refine like I normally would.

GitHub - muthuishere/hand-drawn-diagrams: Hand-drawn diagram skill for Claude Code and Codex. Generates monochrome architecture, workflow, and UX blueprint diagrams as PNGs.

My VPS is running production services. Docker containers. A reverse proxy. SSH access.

This is a very common setup.

For a long time, I believed it was secure enough. SSH uses keys. Only required ports are open. Containers isolate services.

That belief was comfortable.

Then I looked at the logs.

cat  /var/log/auth.log | grep "invalid user"

What the Logs Tell You

They showed repeated failed login attempts. Not random. Automated. Continuous.

None of them succeeded.

That is not the interesting part.

The interesting part is this: they were happening all the time, and I was not watching.

Now HTTP logs

grep -E '(/\.env\b|\.env\b|/wp-admin\b|/phpMyAdmin\b|/\.git/config\b|aws(_)?credentials\b|/\.aws/credentials\b)' \
$(docker inspect kamal-proxy --format '{{.LogPath}}')

The reverse proxy was receiving requests for:

• /.env
• /.aws/credentials
• /phpMyAdmin
• .git/config
• cloud credential paths

Most returned 404. Some returned 403.

Again, nothing broke.

But this raises an important question.

What If You Forgot Something?

Security is not about what you know. It is about what you do not know.

You might have:

• a debug endpoint
• a framework default
• a misconfigured path
• a route exposed only with a certain query string

You may never notice it.

Attackers will try to find it.

This is the problem CrowdSec is designed to address.

What CrowdSec Actually Does

CrowdSec does not block requests directly.

It watches behavior.

The process is simple.

1. Read logs
2. Parse them into fields
3. Observe patterns over time
4. Take action ( Obviously ban it)

It does not react to a single request. It reacts to repeated behavior.

This distinction matters.

Why Bans Are Temporary

A common question is: “Is the ban permanent?”

No.

You can see this directly:

cscli decisions list

Every decision has an expiration time.

This is intentional.

Permanent bans cause problems:

• IPs are often shared
• false positives exist
• firewall rules accumulate silently

CrowdSec assumes behavior can change.

If the behavior repeats, the ban repeats. If it stops, the ban expires.

This keeps the system safe and reversible.

SSH Protection

SSH is the easiest case.

CrowdSec already understands SSH logs.

It detects:

• repeated authentication failures
• brute-force patterns
• user enumeration attempts

Once a threshold is crossed, the firewall blocks the IP.

Nothing custom is required.

In practice, SSH attempts stop quickly after detection.

HTTP Protection

HTTP is different.

Attackers do not repeat one path. They probe many paths.

This is why behavior matters.

My reverse proxy runs inside Docker. CrowdSec reads container logs directly.

That part is straightforward.

Parsers and Log Formats

CrowdSec ships with parsers for common software:

• Nginx
• Apache
• SSH

If you use these, nothing more is needed.

My proxy logs are JSON and slightly different.

So I added a small parser.

It extracts:

• source IP
• request path
• method
• status code

That is enough.

CrowdSec scenarios already know how to use these fields.

If you run something like Spring Boot with custom logs, the same rule applies. You write a small parser(just few config files). You do not rewrite detection logic.

Docker, Parsers, and Enrichers

Think of it in layers.

1. Raw logs from Docker
2. Parser extracts fields
3. Enricher adds context

This structure keeps things simple.

No magic. Just structured information.

Detecting HTTP Attacks

CrowdSec includes community-maintained HTTP scenarios.

They look for:

• multiple sensitive paths
• short time windows
• systematic probing

One request is noise. Repeated patterns are intent.

That is when a ban is issued.

Verifying Behavior

You do not guess.

You check.

Active decisions:

cscli decisions list

Available HTTP scenarios:

cscli hub list | grep http

Inspect why something triggered:

cscli scenarios inspect <scenario-name>

Everything is observable.

Notifications

Blocking silently is not enough.

I send notifications to Telegram.

Each ban tells me:

• which IP
• why it was banned
• how long it lasts

This provides visibility without noise

Automation

You can install CrowdSec manually and configure everything by hand — Telegram notifications, Kamal proxy logs, parsers, and so on.

But honestly, that feels odd.

Most security software makes installation messy. Many steps. Many files. Easy to forget something.

CrowdSec itself is very good, but I didn’t want a setup that depends on manual work.

So I created a small repo with one task.

That one task:

• installs CrowdSec
• installs required scenarios
• adds custom parsers
• configures notifications

That’s it.

With this, the setup becomes repeatable. You can apply it to any VPS and get the same result every time.

https://github.com/muthuishere/automated-crowdsec-kamal

This setup will not stop advanced, targeted attacks.

But it does stop the common stuff:

• automated SSH brute-force attempts
• scanning for common HTTP targets like .env and AWS credential files
• repeated 404-style probing across many paths

And honestly, this is most of what hits a public VPS.

Once this noise is blocked early, the system becomes much quieter and easier to manage.

Final Thought

Securing a production VPS is not about perfection. It is about paying attention.

If your server is public, it is being scanned. SSH attempts. HTTP probing. Repeated 404s.

The question is not whether this happens. The question is whether you are watching it.

Before adding new tools, look at your logs.

Check your SSH logs for repeated failures. Check your container or proxy logs for probing patterns and suspicious 404s.

If you are not doing this, you are flying blind.

Tools like CrowdSec help because they automate what an operator should already be doing: watch behavior, react early, and avoid permanent mistakes.

Temporary bans. Clear signals. Simple automation.

That is often enough to secure a production VPS — and to sleep better while running it.

https://youtu.be/hSiMfbJ4c0Q

For years, I pushed pair programming. Not because it is a ritual, but because it is a way of thinking together.

Most people avoided it. Not because pairing doesn’t work, but because it exposes things. Habits are visible. Gaps are visible. Over time, those were treated as wrong things and quietly tied to something else.

Once that happens, pairing becomes unsafe. People hesitate unless it is senior–junior. Otherwise, they avoid it.

Something important was lost there: asking questions.

When pairing was healthy, people didn’t trust the first answer. They checked multiple sources. That would have been questioned in pairing or in PRs. There were uncomfortable conversations, and those uncomfortable conversations helped the product.

Psychological safety mattered. People felt safe to say, “I don’t know,” or “let us check one more source.” Once safety is gone, questions disappear.

David J. Malan, the Harvard professor behind CS50, coded live and made mistakes. He didn’t hide them. That made it safe for others to ask questions.

Now we are in the Copilot era.

Everyone trusts the one-shot answer from LLM. Sometimes it is right. Sometimes it is good. But people don’t ask more questions.

Is this safe? Will this scale? What happens when we need to change or delete this?

Asking questions is a soft skill. That skill was built when people practiced pairing. Because we haven’t been there, this quality is missing across teams.

I don’t know whether pairing is needed or not. But the soft skill we built there is needed.

Put people into pairs for some time (no AI agents). Make friction for some time. Then let them work alone (with an AI agent).

This subtle habit may increase asking the right questions.

Treat Copilot as a pair, not as an authority. Ask two more questions.

Most system failures don’t come from bad code. They come from moments where no one asked, “Are we sure?”

Then one day(today), I ran just two Node.js apps and a Python app with docker compose, and my system completely froze. Even the mouse wouldn’t move.

Look at those numbers: Docker at 308%, Virtual Machine Service at 208%. Sure, the VM usage is tied to Docker’s VM config — but why on earth should Docker itself hit 300%?

That’s when I decided: I can’t keep doing this.

Looking for Alternatives

I knew about Podman and Colima. I had tried Podman before, but my skill issues and conservative mindset. I always went crawling back to Docker Desktop.

I thought should try colima — without uninstalling Docker Desktop.

Installing Colima

brew install colima
colima start --cpu 2 --memory 4

Colima is light, quiet, and barely touches CPU.

And the best part? My same docker CLI just works. I only had to switch the context:

docker context use colima

That’s it. No extra setup.

Running My Biggest Compose Setup

I ran the same heavy setup again: two Node.js apps and a Python app.

Virtual Machine Service still shows 97% (that’s macOS VM overhead), but Colima itself sits at just 0.2%.

From a system-hanging 308% → to basically nothing. That’s a massive win.

Adding Back a UI

The one thing I missed was Docker Desktop’s UI — the ability to see containers, images, consoles, networks.

Then I came across Portainer.

docker run -d \
  --name portainer \
  -p 9000:9000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  portainer/portainer-ce:latest

Open http://localhost:9000 and you get a clean, web-based Docker UI.

Now I have everything Docker Desktop gave me , containers , images , consoles — but without the CPU pain.

Handy Zsh Aliases

# Switch to Colima
alias switch-colima='colima start --cpu 2 --memory 4 && docker context use colima && export DOCKER_CONTEXT=colima && echo "🐳 Colima now"'
# Switch back to Docker Desktop
alias switch-docker='colima stop && docker context use desktop-linux && unset DOCKER_CONTEXT && echo "🐳 Docker Desktop now"'
# Quick status
alias docker-which='echo "Context: $(docker context show)"'

Final

You don’t have to stick with something just because you’re used to it.
There are always better options — you just need to look.