<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ionut Banu on Medium]]></title>
        <description><![CDATA[Stories by Ionut Banu on Medium]]></description>
        <link>https://medium.com/@ionutbanu?source=rss-e7f1871e4b4e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Kn-ougcnJ-Ir5-_gKBepwg.jpeg</url>
            <title>Stories by Ionut Banu on Medium</title>
            <link>https://medium.com/@ionutbanu?source=rss-e7f1871e4b4e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:39:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ionutbanu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Three RAG Best Practices That Didn’t Earn Their Keep]]></title>
            <link>https://ionutbanu.medium.com/three-rag-best-practices-that-didnt-earn-their-keep-31d04af22511?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/31d04af22511</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 09:44:17 GMT</pubDate>
            <atom:updated>2026-08-29T09:44:17.005Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What 750 test questions said about query rewriting, agentic retrieval, and bigger candidate pools.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ruaSiylWqB2bwDx-wmIvQQ.png" /></figure><p>I asked my retrieval system a question the way a person would actually ask it:</p><blockquote>is coffee bad for me at night?</blockquote><p>Before that question reached the index, an LLM rewrote it into something more searchable:</p><blockquote>effects of nocturnal caffeine consumption on health</blockquote><p>Cleaner. More clinical. Exactly what every RAG tutorial tells you to do — meet the corpus in its own vocabulary.</p><p>It also made retrieval measurably worse. Across 250 test questions, query rewriting cost me <strong>0.074 MRR</strong>, about 17% of the score.</p><p>That’s one of three techniques I added to my RAG pipeline because they were “best practice,” measured properly, and then switched off or left at their smallest setting. One of them actively hurt. The other two cost real money or complexity and bought nothing I could detect. This is what the numbers said, including where they refuse to say as much as I’d like.</p><h3>The setup, briefly</h3><p>I built a RAG assistant over health and performance podcasts — Huberman Lab and Andy Galpin’s <em>Perform</em> — as the capstone for <a href="https://github.com/DataTalksClub/llm-zoomcamp">DataTalks.Club’s LLM Zoomcamp</a>. You ask a question, you get an answer grounded in transcripts, with citations that jump to the exact timestamp in the episode.</p><p>The corpus: <strong>337 episodes → 35,035 chunks</strong> of roughly 350 tokens each, sub-chunked inside episode chapters so a chunk is one coherent moment rather than a fifteen-minute segment.</p><p>The evaluation set: <strong>750 question → chunk pairs</strong>. I took 150 chunks stratified by source, had gpt-4o-mini write five realistic user questions for each, and explicitly instructed it not to quote or closely paraphrase the transcript — otherwise keyword search wins on lexical overlap alone and the comparison is rigged before it starts.</p><p>One methodological decision shapes every number below, so it’s worth stating plainly:</p><p><strong>Matching is exact on the chunk ID.</strong> Each question targets one specific ~350-token passage. Retrieving the passage immediately before or after it — same chapter, same topic, likely a perfectly good answer — counts as a <strong>miss</strong>.</p><p>That’s deliberately harsh, and it’s why the absolute numbers look low. It measures whether the system finds <em>the answer</em>, not whether it finds the right neighbourhood. The end-to-end number that reflects what a user experiences is answer quality, and that comes in at 80% relevant. Both numbers are true; they measure different things.</p><p>If you take one thing from this article, take that: <strong>the metric you choose determines which techniques look good.</strong></p><h3>1. Query rewriting: the one that actively hurt</h3><p>The theory behind query rewriting is sound. Users write conversationally; corpora — especially transcripts of scientists talking — use technical vocabulary. Rewriting bridges that gap.</p><p>Here’s the prompt I used, which is about as standard as it gets:</p><pre>REWRITE_PROMPT = &quot;&quot;&quot;Rewrite the user&#39;s question as a search query for a database of health, \<br>fitness, and nutrition podcast transcripts.</pre><pre>- Keep the original meaning and all specifics; do not answer the question.<br>- Prefer the technical/physiological terms an expert would say out loud.<br>- Drop conversational filler (&quot;can you tell me&quot;, &quot;I was wondering&quot;).<br>- Return ONLY the rewritten query, no quotes or preamble.&quot;&quot;&quot;</pre><p>Note the second bullet: <em>keep all specifics</em>. I asked for exactly the thing that turned out to matter, and the model quietly didn’t.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04b89d95b694c79f5acd0acf11782d83/href">https://medium.com/media/04b89d95b694c79f5acd0acf11782d83/href</a></iframe><p>Look again at the coffee example. “At night” became “nocturnal.” “Bad for me” became “effects on health.” Every rewrite is a defensible paraphrase, and the aggregate effect is a <strong>more generic query</strong>.</p><p>Generic is fatal here. With 35,035 chunks, a hundred passages discuss caffeine and health. The specifics — <em>at night</em>, <em>bad for me</em> — are what narrow that field to one. Rewriting traded precision for fluency, and precision was the whole job.</p><p>This is the only one of the three where the effect clears statistical significance: a two-sample test on the MRR difference gives <strong>p ≈ 0.04</strong>, and that’s the conservative unpaired version — both arms saw identical questions, so a paired test would be stronger.</p><p><strong>The caveat that matters most, and it’s a real one.</strong> My ground truth was generated <em>from the target passages</em> by an LLM. Even instructed not to quote them, the questions inherit their source passage’s framing and vocabulary. Query rewriting deliberately pushes a query <em>away</em> from the user’s original phrasing — which means, in this eval, away from exactly those traces. The test is structurally tilted against rewriting.</p><p>So the honest claim is narrower than “rewriting is bad.” It’s: <strong>rewriting hurt on well-formed questions derived from the target passage.</strong> On genuinely vague input — “what about that sleep thing?” — or multi-turn conversation where the query needs context from earlier messages, I’d expect it to earn its keep. My eval set contains neither.</p><p>It ships implemented and disabled. rewrite=True turns it on.</p><h3>2. Agentic retrieval: paid more, got nothing measurable</h3><p>Agentic RAG — letting the model decide when to search, reformulate, and search again — is the direction everything is moving. I built it, exposed it behind a flag, and put it up against plain retrieve-then-answer with an LLM judge scoring 60 answers against the reference passage.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26db8333154ff52cd5af893392d2b413/href">https://medium.com/media/26db8333154ff52cd5af893392d2b413/href</a></iframe><p>Basic RAG scored higher: <strong>0.867 vs 0.800</strong>, 80% of answers rated relevant against 70%.</p><p>Here’s where I have to be careful, because this is the result I most wanted to be a headline. <strong>It isn’t significant.</strong> Sixty questions, a six-label margin — a two-proportion test gives p ≈ 0.29. I cannot claim agentic RAG is worse. What the data supports is narrower and still useful: agentic RAG was <strong>not detectably better</strong>, and it costs several times more, since it makes multiple API calls per answer.</p><p>For a default, “no measurable gain at several times the price” settles it. For a headline, it doesn’t. Those are different bars and it’s worth keeping them apart.</p><p>My hypothesis for <em>why</em>, offered as a hypothesis: once retrieval is strong enough that one well-targeted query surfaces the right passage, the agentic loop’s extra reformulations mostly add <em>marginal</em> chunks. Those chunks aren’t wrong — they’re plausibly related, which is worse. They dilute the context window with near-misses.</p><p>That prediction is testable in a specific way: agentic should improve <em>relative</em> to basic RAG as base retrieval gets <em>worse</em>, because then there’s something for the extra queries to rescue. On a weaker retriever I’d expect the ordering to flip. I haven’t run it.</p><h3>3. A bigger candidate pool: nothing, at two sample sizes</h3><p>My cross-encoder re-ranks the top 20 candidates from the first retrieval stage. Twenty is a number I picked, and re-ranking can only reorder what retrieval already found — so a deeper pool should mean more chances to surface the right passage. It’s the most obvious free win in the pipeline.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb2592f65b3ba03c71382d8fe4a2d181/href">https://medium.com/media/bb2592f65b3ba03c71382d8fe4a2d181/href</a></iframe><p>It isn’t one. At 750 pairs, going from 20 candidates to 50 moved MRR by 0.009 in favour of the <em>smaller</em> pool — p ≈ 0.67. That’s not a small effect I failed to detect; it’s a well-powered null.</p><p>The reason is the metric again. A deeper pool improves <em>recall</em> — the right passage is more often somewhere in the candidates. But my metric only rewards getting it to the top, and the cross-encoder was already finding it when it was there. The extra thirty candidates were passages the re-ranker correctly ranked below the ones it already had.</p><p>There’s a second lesson buried in that experiment, and it’s about me rather than the pipeline. I ran it first at <strong>n=100</strong>, saw 20 beat 30 by 0.016 MRR, and wrote up a confident explanation about how extra candidates “dilute” the re-ranker. That gap is noise (p ≈ 0.78). I’d invented a mechanism to explain a difference I hadn’t established — which is exactly the failure mode this whole article is supposed to be about. Re-running at 750 pairs is what turned a story into a result.</p><h3>The non-result that turned out to matter</h3><p>Three techniques didn’t earn their keep. A fourth measurement looked like a disappointment and was actually the most useful thing I learned.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e26b5ca18412066d2a20f67e532b3cfa/href">https://medium.com/media/e26b5ca18412066d2a20f67e532b3cfa/href</a></iframe><p>Keyword search: <strong>0.2608 MRR</strong>. Vector search: <strong>0.2626 MRR</strong>.</p><p>A local MiniLM embedding model, 384 dimensions, HNSW index — beat 1970s TF-IDF by 0.002 (p ≈ 0.93). That is not a difference.</p><p>Earlier in the project, under a looser chapter-level matching rule, vector search looked clearly better. Tightening to exact-passage matching erased its advantage entirely. Same system, weeks apart; only the ruler changed.</p><p>The explanation, I think, is that embeddings and TF-IDF are good at different halves of the same job. Semantic similarity reliably lands you in the right <em>topical neighbourhood</em> — every passage about sleep and caffeine clusters together. But inside that neighbourhood, twenty passages are all about sleep and caffeine, and a single 384-dimensional vector has no particular reason to rank the right one first. Keyword search has the mirror-image problem: when a rare term matches, it nails the passage; when the user’s phrasing misses the vocabulary, it finds nothing at all.</p><p>And that near-tie is not a disappointing result. It’s the <strong>precondition</strong> for the technique that gave me my biggest gain.</p><h3>What actually worked</h3><p>Reciprocal Rank Fusion combines two ranked lists by scoring each document 1/(k + rank) in every list it appears in:</p><pre>RRF_K = 60</pre><pre>def reciprocal_rank_fusion(result_lists, num_results):<br>    scores, docs = {}, {}<br>    for results in result_lists:<br>        for rank, doc in enumerate(results, 1):<br>            key = doc[&quot;id&quot;]<br>            scores[key] = scores.get(key, 0.0) + 1.0 / (RRF_K + rank)<br>            docs.setdefault(key, doc)<br>    ranked = sorted(scores, key=scores.get, reverse=True)<br>    return [docs[k] for k in ranked[:num_results]]</pre><p>Fusing by <em>rank</em> rather than score sidesteps a real problem: TF-IDF scores and cosine similarities live on incomparable scales, and normalising them into agreement is a fiddly, dataset-specific mess. Rank is scale-free.</p><p>Fusion pays off when your retrievers are <strong>comparably strong but fail differently</strong>. Two retrievers of similar quality that fail on the <em>same</em> questions give you nothing. Two that fail on <em>different</em> questions cover each other. Keyword and vector at 0.2608 and 0.2626 are the textbook case — which is why the boring tie above was good news. Hybrid landed at <strong>0.331 MRR</strong>, 26% better than either alone.</p><p>Then the single biggest lever in the project: <strong>cross-encoder re-ranking</strong>, worth another 30%.</p><pre>def rerank(query, chunks, top_k=5):<br>    scores = get_reranker().predict([(query, c[&quot;text&quot;]) for c in chunks])<br>    ordered = sorted(zip(scores, range(len(chunks))), key=lambda p: p[0], reverse=True)<br>    return [chunks[i] for _, i in ordered[:top_k]]</pre><p>The retrievers score your query against a <em>pre-computed</em> representation of each chunk — TF-IDF weights, or one 384-dimensional vector. Fast, because the work happened at index time. Lossy, for the same reason: the chunk was compressed before anyone knew what you’d ask.</p><p>A cross-encoder reads the query and the chunk <strong>together</strong> and scores that specific pair. Far more accurate, far too slow across 35,035 chunks. So it runs as a second pass over the candidates the first stage returns.</p><p>End to end, keyword baseline to shipped default: <strong>MRR 0.261 → 0.429</strong>, HR@1 nearly doubled from 0.172 to 0.331. Both stages run locally, on CPU, for free. The biggest quality gain in the whole project came from an 80MB model on CPU, applying an idea — cross-encoder re-ranking — that predates the current wave of LLMs entirely.</p><h3>The part I’d tell you to steal: distrust your own numbers</h3><p>The techniques above are portable. This section is more so.</p><p>Twice during evaluation I produced numbers that looked completely plausible and were completely wrong. Both times the thing that caught it was structural, not clever.</p><p><strong>The defaults that moved underneath me.</strong> My evaluation defined each approach as a config dict. Some entries omitted flags and inherited them from retrieve()&#39;s defaults — which track whichever configuration currently wins. So when I promoted re-ranking to the production default, an approach labelled &quot;hybrid&quot; silently <em>became</em> hybrid-plus-re-ranking. The table still rendered. The numbers were still numbers. They just no longer measured what their labels claimed.</p><p>The fix is unglamorous: every approach pins every flag explicitly, even when redundant.</p><pre>ALL_APPROACHES = {<br>    &quot;keyword&quot;:       {&quot;method&quot;: &quot;keyword&quot;, &quot;rerank&quot;: False, &quot;rewrite&quot;: False},<br>    &quot;vector&quot;:        {&quot;method&quot;: &quot;vector&quot;,  &quot;rerank&quot;: False, &quot;rewrite&quot;: False},<br>    &quot;hybrid&quot;:        {&quot;method&quot;: &quot;hybrid&quot;,  &quot;rerank&quot;: False, &quot;rewrite&quot;: False},<br>    &quot;hybrid+rerank&quot;: {&quot;method&quot;: &quot;hybrid&quot;,  &quot;rerank&quot;: True,  &quot;rewrite&quot;: False},<br>}</pre><p>Redundant configuration in an experiment isn’t noise — it’s what stops your experiment drifting when production changes under it.</p><p><strong>The control row.</strong> When I migrated the vector store from a local SQLite index to Postgres with pgvector, I re-ran everything. Keyword search improved.</p><p>That’s impossible. Keyword search is an in-memory TF-IDF index that never touches the vector store. Its score <em>must</em> be identical across that migration. Because it wasn’t, I knew the harness was broken rather than the backend being better — and I’d never have known from the vector rows, which had moved by a plausible amount in a flattering direction.</p><p>Now every run includes keyword search as a control, and it reproduces to four decimal places. <strong>Always measure something that cannot possibly change.</strong> When it changes, your harness is lying, and the interesting rows are lying in the same direction.</p><p>The third failure mode is the candidate-pool story above, and it’s the one no tooling catches. Guardrails catch broken harnesses. Nothing catches motivated reasoning except noticing you’re doing it — ideally before you’ve written the explanation down.</p><h3>What I’d take to the next project</h3><p><strong>Measure before you adopt.</strong> Query rewriting and agentic retrieval are both good ideas with real use cases. Neither earned its place <em>here</em>, and I’d have shipped both on reputation alone.</p><p><strong>Know which bar you’re clearing.</strong> “No measurable gain at several times the cost” is enough to pick a default. It is not enough to claim a technique is worse. I nearly wrote this article with the stronger claim in the title.</p><p><strong>Your metric is a design decision.</strong> Strict exact-passage matching made vector search look mediocre, made re-ranking look essential, and made a deeper candidate pool look pointless. A recall-oriented metric would have told me a different story about all three. Neither ruler is wrong — but pick it deliberately, and re-read every conclusion you drew under the old one when you change it.</p><p><strong>Two mediocre retrievers that fail differently beat one good retriever.</strong> The near-tie was the asset.</p><p><strong>Cheap local models did the heavy lifting.</strong> The biggest gain came from an 80MB cross-encoder on CPU. The entire evaluation — ground truth, LLM judging, every arm — cost about a dollar.</p><p><strong>Build the control before you need it.</strong> The keyword row cost nothing and caught a bug I’d otherwise have shipped, believed, and written about.</p><p>The project is on GitHub with the full evaluation write-up, the ground-truth set, and the raw result files, so you can check the numbers rather than take my word for them:</p><p><a href="https://github.com/ionut-banu/health-performance-rag"><strong>github.com/ionut-banu/health-performance-rag</strong></a></p><p>Built as the capstone for <a href="https://github.com/DataTalksClub/llm-zoomcamp">LLM Zoomcamp</a> by DataTalks.Club — a genuinely good free course, and the reason this project has an evaluation section at all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31d04af22511" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracing Java Structured Concurrency Without Touching a Line of Code]]></title>
            <link>https://ionutbanu.medium.com/tracing-java-structured-concurrency-without-touching-a-line-of-code-4002ba6ef704?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4002ba6ef704</guid>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[virtual-threads]]></category>
            <category><![CDATA[jfr]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sun, 24 May 2026 21:15:09 GMT</pubDate>
            <atom:updated>2026-05-24T21:15:09.344Z</atom:updated>
            <content:encoded><![CDATA[<h4>How a 200-line ByteBuddy agent makes JDK 26 StructuredTaskScope observable — and writes its own HTML report when JFR stops.</h4><p><strong>TL;DR</strong> — I built scope-tracer, an open-source tool that visualises Java structured-concurrency task trees. The library half is the boring part: wrap StructuredTaskScope, get JFR events. The interesting half is the <strong>Java agent</strong>, which traces <em>any</em> JDK 26+ application — including code you don&#39;t own — with zero source changes, and then writes a self-contained HTML report the instant a JFR recording stops. This article is mostly about how that agent works, and the JVM-level traps I hit building it.</p><h3>The observability gap</h3><p>JDK 21 gave us virtual threads. JDK 26 brings java.util.concurrent.StructuredTaskScope to maturity. Together they make the &quot;fan out, await all, fail fast&quot; pattern the <em>default</em> way to write concurrent Java:</p><pre>try (var scope = StructuredTaskScope.open(<br>        Joiner.awaitAllSuccessfulOrThrow(),<br>        c -&gt; c.withName(&quot;checkout-flow&quot;))) {<br>    var price    = scope.fork(() -&gt; pricingService.quote(cart));<br>    var shipping = scope.fork(() -&gt; shippingService.estimate(cart));<br>    scope.join();<br>    return new Checkout(price.get(), shipping.get());<br>}</pre><p>This is beautiful to write and miserable to <em>observe</em>. When checkout-flow takes 900 ms, which fork was on the critical path? When one subtask throws and the joiner cancels its siblings, which sibling got interrupted, and how far did it get? A thread dump won&#39;t tell you — the virtual threads are gone by the time you look, and most of them never had names anyway.</p><p>Structured concurrency gives the <em>runtime</em> a precise tree of who-forked-whom. That tree just isn’t exposed to you. scope-tracer exposes it.</p><figure><img alt="scope-tracer HTML report for the OrderProcessingDemo: an outer fulfillment scope with three forks, a nested payment-pipeline scope indented underneath, the critical-path task highlighted in amber, and one red failed bar." src="https://cdn-images-1.medium.com/max/1024/1*A9Ib6RoJAHJsqyLX-L0-VQ.png" /><figcaption><em>The same six JFR events drive every report. Outer scope, nested inner scopes, amber critical path, red failure — all from a single `.jfr` dump.</em></figcaption></figure><h3>Two roads to the same six events</h3><p>scope-tracer emits six JFR events at every lifecycle moment of a scope:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25b417ced0f2879554f442272b34c50d/href">https://medium.com/media/25b417ced0f2879554f442272b34c50d/href</a></iframe><p>An analyzer reads the .jfr file and renders a self-contained HTML report: a Gantt timeline per scope, colour-coded task bars, nested scopes indented under the task that opened them, and the critical path highlighted in amber.</p><p>There are two ways to produce those events:</p><ol><li><strong>The library</strong> (scope-tracer-core). Replace StructuredTaskScope with TracedScope. Same API — fork(), join(), try-with-resources. It emits the events from inside its own method bodies. Clean, explicit, and it requires editing every call site.</li><li><strong>The agent</strong> (scope-tracer-agent). Change nothing. Attach -javaagent:scope-tracer-agent.jar, and the JDK&#39;s own StructuredTaskScope class is rewritten at load time to emit the exact same events.</li></ol><p>The library is a few hundred lines of unremarkable wrapper code. The agent is where it gets fun, because it has to instrument a <strong>preview JDK class loaded by the bootstrap classloader</strong>. That sentence contains three separate landmines. The rest of this article steps on all of them.</p><p>If you own the code, the library is fine. But:</p><ul><li><strong>You often don’t own the code.</strong> The slow scope might be inside a framework, a driver, or a library you can’t patch.</li><li><strong>Production doesn’t want a redeploy.</strong> “Add a tracing wrapper and ship a new build” is not an incident-response plan. Attaching an agent and starting JFR with jcmd <em>is</em>.</li><li><strong>It’s all-or-nothing in the best way.</strong> One -javaagent flag traces every StructuredTaskScope in the entire JVM, including ones you forgot existed.</li></ul><p>So the agent’s job: intercept the JDK’s structured-concurrency machinery itself.</p><h3>Landmine 1: instrumenting a preview class</h3><p>StructuredTaskScope is a <strong>preview API</strong> in JDK 26. Its bytecode carries the preview-class flag, and the public type is really a thin facade over a private java.util.concurrent.StructuredTaskScopeImpl. The lifecycle moments I care about live in different places:</p><ul><li>the static open(...) factory — on StructuredTaskScope</li><li>the 3-arg constructor that receives the configured name — on StructuredTaskScopeImpl</li><li>fork(Callable) and close() — on both</li></ul><p>scope-tracer uses <strong>ByteBuddy</strong> for the rewriting. ByteBuddy 1.x refuses to parse preview-flagged class files by default, so the very first line of premain is a workaround:</p><pre>public static void premain(String args, Instrumentation inst) {<br>    // ByteBuddy won&#39;t touch Java 26 preview class files without this.<br>    System.setProperty(&quot;net.bytebuddy.experimental&quot;, &quot;true&quot;);<br><br>    AgentConfig config = AgentConfig.parse(args);<br>    // ... install transformers ...<br>}</pre><p>That single property is the difference between “agent works” and “ByteBuddy throws IllegalArgumentException: Unrecognized class file version&quot;. It&#39;s a deliberate, commented hack with a removal condition: drop it once ByteBuddy ships official JDK 26 support.</p><p>The transformer installation matches four advice sites. In ByteBuddy’s @Advice model you write <em>templates</em> — static methods whose bodies are inlined into the target method&#39;s prologue (@OnMethodEnter) or epilogue (@OnMethodExit). Crucially, the advice code must be reachable from wherever the target class lives — which brings us to landmine 2. First, the advice itself.</p><h3>The four advice sites</h3><h3>open() — birth of a scope</h3><p>ScopeOpenAdvice runs <em>after</em> the static factory returns. It assigns a global scope id, records per-scope state in a WeakHashMap keyed by the scope instance, and commits ScopeOpenedEvent:</p><pre>@Advice.OnMethodExit(suppress = Throwable.class)<br>public static void onOpen(@Advice.Return Object scope) {<br>    if (scope == null) return;<br><br>    String pending = AgentState.PENDING_SCOPE_NAME.get();<br>    AgentState.PENDING_SCOPE_NAME.remove();<br>    String name = (pending != null &amp;&amp; !pending.isBlank())<br>            ? pending<br>            : ScopeNameDeriver.derive();   // call-site fallback<br><br>    long scopeId = AgentState.SCOPE_ID_COUNTER.incrementAndGet();<br>    AgentState.SCOPE_STATES.put(scope, new ScopeState(name, scopeId, new AtomicLong()));<br><br>    ScopeOpenedEvent event = new ScopeOpenedEvent();<br>    event.scopeId    = scopeId;<br>    event.scopeName  = name;<br>    event.taskId     = 0L;<br>    event.threadName = Thread.currentThread().getName();<br>    event.commit();<br>}</pre><p>suppress = Throwable.class is non-negotiable for an agent: if the advice throws, it must <em>never</em> propagate into the host application. A tracer that can crash StructuredTaskScope.open() is worse than no tracer.</p><p>Note the WeakHashMap. The scope instance is the key; when the application drops its reference, the entry is collectible. An agent that pins every scope it ever saw is a memory leak with extra steps.</p><h3>The constructor — capturing the name</h3><p>Here’s a subtlety. The user configures the scope name via c -&gt; c.withName(&quot;checkout-flow&quot;). By the time open() <em>returns</em>, that name is buried inside the impl object — awkward to read reflectively across a preview API. But the StructuredTaskScopeImpl <strong>constructor</strong> receives the resolved name as its third argument. So ScopeConstructorAdvice grabs it on the way in and parks it in a ThreadLocal:</p><pre>@Advice.OnMethodEnter(suppress = Throwable.class)<br>public static void onConstruct(@Advice.Argument(2) String name) {<br>    AgentState.PENDING_SCOPE_NAME.set(name);<br>}</pre><p>The constructor runs on the same thread as, and just before, open() returns — so ScopeOpenAdvice reads the ThreadLocal back out and clears it. It&#39;s a deliberately tiny handoff channel between two advice sites that can&#39;t otherwise see each other&#39;s locals. If no name was configured, the ThreadLocal is empty and we fall back to deriving one from the call site (more on that below).</p><h3>fork(Callable) — and the overload trap</h3><p>fork is matched <strong>explicitly by parameter type</strong>. StructuredTaskScopeImpl has both fork(Callable) and fork(Runnable); advising the wrong one (or both, ambiguously) corrupts task accounting. ForkAdvice assigns a per-scope task id, emits TaskForkedEvent on the <em>caller</em> thread (so the fork shows up at the right point on the timeline), and then <strong>swaps the user&#39;s callable for a wrapper</strong>:</p><pre>@Advice.OnMethodEnter(suppress = Throwable.class)<br>public static void onFork(<br>        @Advice.This Object scope,<br>        @Advice.Argument(value = 0, readOnly = false) Callable task) {<br><br>    ScopeState state = AgentState.SCOPE_STATES.get(scope);<br>    if (state == null) return;<br><br>    long taskId = state.nextTaskId();<br><br>    TaskForkedEvent event = new TaskForkedEvent();<br>    event.scopeId    = state.scopeId();<br>    event.scopeName  = state.name();<br>    event.taskId     = taskId;<br>    event.taskName   = TaskNameDeriver.derive(task);<br>    event.threadName = Thread.currentThread().getName();<br>    event.commit();<br><br>    // Rebind the argument the JDK is about to run.<br>    task = new TracingCallable&lt;&gt;(task, state.name(), state.scopeId(), taskId);<br>}</pre><p>@Advice.Argument(value = 0, readOnly = false) is the magic. ByteBuddy lets the advice <em>reassign</em> the method parameter. The JDK&#39;s fork body then runs <em>our</em> TracingCallable instead of the user&#39;s lambda — without the user, or the JDK, knowing.</p><h3>close() — and the cross-thread ordering reality</h3><p>ScopeCloseAdvice runs after close() returns, emits ScopeClosedEvent, and removes the scope from the WeakHashMap. Worth calling out a property the analyzer has to tolerate: TaskSucceeded is committed on the <em>task</em> thread, while ScopeClosed is committed on the <em>caller</em> thread. JFR does not guarantee cross-thread flush ordering, so in the dumped file a task-success event can appear <em>after</em> the scope-close event even though it logically precedes it. The analyzer keys everything off the globally-unique scopeId and reconstructs ordering from event timestamps within a thread, never from file order. Build a JFR tool on file order and it will betray you under load.</p><h3>The task wrapper: one of three outcomes</h3><p>TracingCallable is the only piece that runs <em>on the subtask&#39;s virtual thread</em>. Its entire job is to emit exactly one terminal event:</p><pre>@Override<br>public T call() throws Exception {<br>    try {<br>        T result = delegate.call();<br>        emit(new TaskSucceededEvent());<br>        return result;<br>    } catch (InterruptedException e) {<br>        emit(new TaskCancelledEvent());      // observed scope shutdown<br>        Thread.currentThread().interrupt();  // restore the flag — critical<br>        throw e;<br>    } catch (Exception e) {<br>        TaskFailedEvent ev = new TaskFailedEvent();<br>        ev.exceptionType       = e.getClass().getName();<br>        ev.exceptionMessage    = e.getMessage();<br>        ev.exceptionStackTrace = TaskFailedEvent.formatStackTrace(e);<br>        emit(ev);<br>        throw e;<br>    }<br>}</pre><p>Three rules a tracing wrapper must never break:</p><ol><li><strong>Re-interrupt on </strong><strong>InterruptedException.</strong> Swallowing the interrupt flag would break the JDK&#39;s own fail-fast cancellation. The tracer observes; it does not interfere.</li><li><strong>Always rethrow.</strong> The wrapper changes <em>observability</em>, never <em>behaviour</em>. The joiner must see precisely what it would have seen without the agent.</li><li><strong>InterruptedException is cancellation, not failure.</strong> Treating a cancelled sibling as &quot;failed&quot; turns every fail-fast scope into a wall of false red. The distinction is the whole point of the visualisation.</li></ol><figure><img alt="FailFastDemo report: a short red bar for the failed inventory check and a shorter orange bar for the cancelled sibling that never got to finish." src="https://cdn-images-1.medium.com/max/1024/1*PCludtEpjOkzVjNocWaA4g.png" /><figcaption><em>`FailFastDemo` in one image: red is a real failure, orange is a sibling cancelled by the joiner. Conflating the two would hide what actually went wrong.</em></figcaption></figure><h3>Landmine 2: the bootstrap classloader trap</h3><p>Here’s the one that cost me an afternoon. StructuredTaskScope is a JDK class. JDK classes are loaded by the <strong>bootstrap classloader</strong>. When ByteBuddy inlines ScopeOpenAdvice into StructuredTaskScope.open(), the advice&#39;s <em>referenced</em> classes — AgentState, ScopeState, ScopeOpenedEvent — must be resolvable <strong>from the bootstrap classloader</strong>. But the agent jar is on the <em>system</em> classpath. Bootstrap can&#39;t see system. Result: NoClassDefFoundError thrown from deep inside the JDK, at scope-open time, in everyone&#39;s application.</p><p>The fix is the agent jar listing <strong>itself</strong> on the bootstrap class path, via its own manifest. The Maven shade plugin builds a fat-jar (classifier agent) and writes:</p><pre>&lt;transformer implementation=&quot;...ManifestResourceTransformer&quot;&gt;<br>  &lt;manifestEntries&gt;<br>    &lt;Premain-Class&gt;com.ionutbanu.scopetracer.agent.ScopeTracerAgent&lt;/Premain-Class&gt;<br>    &lt;Can-Redefine-Classes&gt;true&lt;/Can-Redefine-Classes&gt;<br>    &lt;Can-Retransform-Classes&gt;true&lt;/Can-Retransform-Classes&gt;<br>    &lt;Boot-Class-Path&gt;scope-tracer-agent-${project.version}-agent.jar&lt;/Boot-Class-Path&gt;<br>  &lt;/manifestEntries&gt;<br>&lt;/transformer&gt;</pre><p>Boot-Class-Path naming <em>itself</em> means: when the JVM loads this agent, also append this same jar to the bootstrap search path. Now advice code and JFR event classes are visible from inside java.util.concurrent. No extra JVM flags for the user — the manifest carries the bootstrap wiring. This is the single most important line in the entire build, and it&#39;s invisible until it&#39;s missing.</p><h3>Landmine 3: you can’t bring SLF4J to the bootstrap loader</h3><p>Because agent classes live on the bootstrap path, bundling an SLF4J binding there would collide with — or hijack — the <em>host application’s</em> logging. An observability tool that breaks the host’s logging stack has failed at its one job.</p><p>So the agent uses no logging framework at all. It has a 20-line AgentLog that writes [scope-tracer]-prefixed lines to System.err:</p><pre>final class AgentLog {<br>    private static final String PREFIX = &quot;[scope-tracer] &quot;;<br>    private static final boolean DEBUG = Boolean.getBoolean(&quot;scopetracer.agent.verbose&quot;);<br><br>    static void info (String m) { System.err.println(PREFIX + m); }<br>    static void warn (String m) { System.err.println(PREFIX + &quot;WARN &quot; + m); }<br>    static void debug(String m) { if (DEBUG) System.err.println(PREFIX + &quot;DEBUG &quot; + m); }<br>}</pre><p>The shade config also explicitly <em>excludes</em> Logback from the fat-jar. “Don’t ship a logging binding in your agent” is a rule you only learn by shipping one and watching someone’s structured JSON logs turn into garbage.</p><h3>Naming the unnamed</h3><p>Half the value of the report is readable scope names. Named scopes are easy — the constructor handoff above. But plenty of scopes have no name. For those, ScopeNameDeriver walks the stack and picks the first frame that is <em>user</em> code:</p><pre>public static String derive() {<br>    return StackWalker.getInstance()<br>        .walk(frames -&gt; frames<br>            .filter(f -&gt; StackFilter.isUserFrame(f.getClassName(), SELF))<br>            .findFirst()<br>            .map(ScopeNameDeriver::format)   // -&gt; &quot;CheckoutService#placeOrder&quot;<br>            .orElse(&quot;unknown-scope&quot;));<br>}</pre><p>The filter excludes only java.util.concurrent.StructuredTaskScope*, net.bytebuddy.*, and the deriver itself — <em>not</em> the whole agent package, because test subjects and user code can legitimately live near it. format also unwraps synthetic lambda$... method names back to the enclosing method, so an anonymous scope opened inside a lambda still reads as CheckoutService#placeOrder rather than CheckoutService#lambda$placeOrder$3. Small touch, large readability payoff in the rendered report.</p><h3>The payoff: the agent writes its own report</h3><p>Everything so far gets you JFR events. The feature that makes the agent genuinely <em>operational</em> is this: <strong>you never have to run a separate analyzer step.</strong> The agent registers a FlightRecorderListener. The moment any JFR recording with a destination file transitions to STOPPED, the agent parses the .jfr and writes an HTML report next to it:</p><pre>FlightRecorder.addListener(new FlightRecorderListener() {<br>    @Override public void recordingStateChanged(Recording recording) {<br>        if (recording.getState() != RecordingState.STOPPED) return;<br>        Path dest = recording.getDestination();<br>        if (dest == null) return;<br>        new Thread(() -&gt; maybeWriteHtml(dest, config), &quot;scope-tracer-html-gen&quot;).start();<br>    }<br>});</pre><p>maybeWriteHtml parses the dump, skips generation if the recording has fewer than min.scopes scopes (so idle windows don&#39;t spew empty reports), creates the output directory if needed, and writes the HTML. The work happens on its own thread so it never stalls whatever stopped the recording.</p><p>What this buys an operator is the entire workflow collapsing to:</p><pre># Service already running with -javaagent:scope-tracer-agent.jar<br>jcmd &lt;pid&gt; JFR.start  name=trace filename=/tmp/scope.jfr<br>#   ... let some traffic flow through the structured-concurrency paths ...<br>jcmd &lt;pid&gt; JFR.stop   name=trace<br>#   /tmp/scope.html now exists. Open it.</pre><p>No analyzer invocation. No classpath assembly. No redeploy, ever. Start a window, stop it, open the HTML. Repeat as many times as you want against the same live process. The agent reads its knobs from the -javaagent argument string:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a677fdad9e2356b79aaee8f2e3059a6/href">https://medium.com/media/0a677fdad9e2356b79aaee8f2e3059a6/href</a></iframe><p>Parsing is a deliberately forgiving switch over comma-split key=value tokens: unknown keys are ignored, a malformed min.scopes warns and falls back to the default. An agent that refuses to start because someone fat-fingered an argument is an agent that gets removed from the launch script.</p><h3>Try it</h3><p>I’ve published a standalone demo repo so you don’t have to build anything from source — it depends on the released com.ionutbanu:scope-tracer-* artifacts from Maven Central:</p><pre>git clone https://github.com/ionut-banu/scope-tracer-demo<br>cd scope-tracer-demo<br><br># Library demos: TracedScope, single command each<br>mvn -Pparallel-fetch  package exec:exec<br>mvn -Pfail-fast        package exec:exec<br>mvn -Porder-processing package exec:exec<br><br># The agent demo: AgentDemo contains NO scope-tracer code at all<br>mvn -Pagent package exec:exec<br><br># Production-style: agent + JFR, the agent writes the HTML itself<br>./scripts/run-agent-production.sh</pre><p>Open AgentDemo.java and note what isn&#39;t there: no import of scope-tracer, no TracedScope, no event code. Plain JDK 26 StructuredTaskScope. Every bar in the resulting timeline exists purely because the agent rewrote the JDK&#39;s bytecode as it loaded.</p><figure><img alt="AgentDemo report: a named “checkout-flow” scope and an anonymous scope named “AgentDemo#main” by the call-site deriver, both fully timelined despite the source containing zero scope-tracer code" src="https://cdn-images-1.medium.com/max/1024/1*DYYWix5pXXGzcrtTa_nFpA.png" /><figcaption><em>Produced from a source file that does not import scope-tracer. The “checkout-flow” name comes from `Config.withName(…)`; the second scope’s name is derived by `StackWalker` from the call site.</em></figcaption></figure><h3>Lessons worth keeping</h3><ul><li><strong>Agents are bootstrap-classloader citizens.</strong> Everything an advice touches must be visible from bootstrap. Boot-Class-Path self-listing is the cleanest fix and it lives in the build, not the code — which makes it exactly the kind of thing that&#39;s invisible until it breaks.</li><li><strong>suppress = Throwable.class on every advice.</strong> Observability code that can crash the observed code is a downgrade, not a feature.</li><li><strong>Observe, never alter.</strong> Re-interrupt, always rethrow, never change a result. The joiner must see exactly what it would have without you.</li><li><strong>Don’t trust JFR file order across threads.</strong> Key off a unique id, order by timestamp within a thread.</li><li><strong>Close the loop for the operator.</strong> Events in a .jfr are potential energy. Writing the HTML the moment the recording stops is what turns a library into a tool someone reaches for during an incident.</li></ul><p>scope-tracer is Apache-2.0 and on Maven Central. The agent is roughly 200 lines of advice plus a build file that knows about bootstrap classloaders. The hard-won knowledge is almost entirely in that last clause.</p><p><em>Source: </em><a href="https://github.com/ionut-banu/scope-tracer"><em>https://github.com/ionut-banu/scope-tracer</em></a></p><p><em>Demo repo: </em><a href="https://github.com/ionut-banu/scope-tracer-demo"><em>https://github.com/ionut-banu/scope-tracer-demo</em></a></p><p><em>JFR event schema: see </em><em>docs/jfr-events.md in the main repository.</em></p><p><em>Built and tested on JDK 26 (</em><em>--enable-preview). </em><em>StructuredTaskScope API shapes reflect the JDK 26 preview and may shift as the API finalises.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4002ba6ef704" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an End-to-End Flight Delay Pipeline with Kestra, dbt, and BigQuery — What I Learned]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ionutbanu.medium.com/building-an-end-to-end-flight-delay-pipeline-with-kestra-dbt-and-bigquery-what-i-learned-5eba34e6d171?source=rss-e7f1871e4b4e------2"><img src="https://cdn-images-1.medium.com/max/1376/1*7db7XL3AQLMV8FpTLQ-Fmg.png" width="1376"></a></p><p class="medium-feed-snippet">I&#x2019;ve spent most of my engineering career working with Java, Spring, and distributed systems &#x2014; including real-time stream processing with&#x2026;</p><p class="medium-feed-link"><a href="https://ionutbanu.medium.com/building-an-end-to-end-flight-delay-pipeline-with-kestra-dbt-and-bigquery-what-i-learned-5eba34e6d171?source=rss-e7f1871e4b4e------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://ionutbanu.medium.com/building-an-end-to-end-flight-delay-pipeline-with-kestra-dbt-and-bigquery-what-i-learned-5eba34e6d171?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5eba34e6d171</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 17:29:35 GMT</pubDate>
            <atom:updated>2026-04-04T17:29:35.552Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes on Raspberry Pi 5 — Part 7 Setting Up cert-manager with a ClusterIssuer Using…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-7-setting-up-cert-manager-with-a-clusterissuer-using-9cd70f08be9b?source=rss-e7f1871e4b4e------2"><img src="https://cdn-images-1.medium.com/max/1400/0*IQXwgy-zO37usmFd.jpeg" width="1400"></a></p><p class="medium-feed-snippet">I have been using a Raspberry Pi 4 as a host for a Kubernetes for about two years. Recently I purchased the new Raspberry Pi 5. I plan to&#x2026;</p><p class="medium-feed-link"><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-7-setting-up-cert-manager-with-a-clusterissuer-using-9cd70f08be9b?source=rss-e7f1871e4b4e------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-7-setting-up-cert-manager-with-a-clusterissuer-using-9cd70f08be9b?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cd70f08be9b</guid>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[cert-manager]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[https]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sun, 30 Nov 2025 13:45:28 GMT</pubDate>
            <atom:updated>2025-11-30T13:45:28.013Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes on Raspberry Pi 5 — Part 6 Install NGINX ingress controller]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-6-install-nginx-ingress-controller-9a26ccebfa9c?source=rss-e7f1871e4b4e------2"><img src="https://cdn-images-1.medium.com/max/1400/0*74zM5BYm2YWh06Ox.jpeg" width="1400"></a></p><p class="medium-feed-snippet">I have been using a Raspberry Pi 4 as a host for a Kubernetes for about two years. Recently I purchased the new Raspberry Pi 5. I plan to&#x2026;</p><p class="medium-feed-link"><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-6-install-nginx-ingress-controller-9a26ccebfa9c?source=rss-e7f1871e4b4e------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-6-install-nginx-ingress-controller-9a26ccebfa9c?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a26ccebfa9c</guid>
            <category><![CDATA[nginx-ingress-controller]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[raspberry-pi-5]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sat, 01 Mar 2025 19:55:01 GMT</pubDate>
            <atom:updated>2025-03-01T19:55:53.191Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes on Raspberry Pi 5 — Part 5 Install network load balancer (MetalLB)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-5-install-network-load-balancer-metallb-de1cd436d275?source=rss-e7f1871e4b4e------2"><img src="https://cdn-images-1.medium.com/max/1400/0*wBLy4khBukUpHYyd.jpeg" width="1400"></a></p><p class="medium-feed-snippet">I have been using a Raspberry Pi 4 as a host for a Kubernetes for about two years. Recently I purchased the new Raspberry Pi 5. I plan to&#x2026;</p><p class="medium-feed-link"><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-5-install-network-load-balancer-metallb-de1cd436d275?source=rss-e7f1871e4b4e------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-5-install-network-load-balancer-metallb-de1cd436d275?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/de1cd436d275</guid>
            <category><![CDATA[raspberry-pi-5]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[metallb]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[load-balancing]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sun, 17 Nov 2024 21:13:05 GMT</pubDate>
            <atom:updated>2025-02-28T15:41:36.731Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes on Raspberry Pi 5 — Part 4 Install K3S on agent nodes]]></title>
            <link>https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-4-install-k3s-on-agent-nodes-67b46ceb7944?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/67b46ceb7944</guid>
            <category><![CDATA[raspberry-pi-5]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[k3]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 18:58:45 GMT</pubDate>
            <atom:updated>2025-02-28T15:42:23.634Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes on Raspberry Pi 5 — Part 4 Install K3S on agent nodes</h3><p>I have been using a Raspberry Pi 4 as a host for a Kubernetes for about two years. Recently I purchased the new Raspberry Pi 5. I plan to also setup Kubernetes on it so I can have a cluster with two nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QSRUzNRgfE1yjC9nGmW0IQ.jpeg" /></figure><h3>K3s on Raspberry Pi 5</h3><p><a href="https://ionutbanu.medium.com/k3s-on-raspberry-pi-5-part-1-install-raspberry-pi-os-lite-64-bit-29cdf41c7f5b">Part 1 Install Raspberry Pi OS Lite 64-bit</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-2-mount-external-storage-31ed1d0fc4bd">Part 2 Mount external storage</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-3-install-k3s-on-master-node-f95ea35a8b1c">Part 3 Install K3S on master node</a></p><p><a href="https://medium.com/@ionutbanu/kubernetes-on-raspberry-pi-5-part-4-install-k3s-on-agent-nodes-67b46ceb7944">Part 4 Install K3S on agent nodes</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-5-install-network-load-balancer-metallb-de1cd436d275">Part 5 Install network load balancer (MetalLB)</a></p><p>So far I have went through the process of setting up K3S on a Raspberry Pi 5 board. This board will be the master node. Now I will add a second node or agent node. The steps are the same for adding more agent nodes. I will add my old Raspberry Pi 4 board as an agent node.</p><h4>Retrieve node token</h4><p>The first thing we need to do is to retrieve the access-token from the master node. You can do this by executing the following command on the master node:</p><pre>sudo cat /var/lib/rancher/k3s/server/node-token</pre><h4>Setup agent node</h4><p>Now move to the worker board and make sure you apply the instructions from step 1 (Update kernel configuration file) in this article <a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-3-install-k3s-on-master-node-f95ea35a8b1c">Part 3 Install K3S on master node</a>.</p><p>Use the token obtained above to replace the &lt;ACCESS_TOKEN&gt; placeholder below. Also use the IP of the master node to replace the &lt;MASTER_NODE_IP&gt; placeholder.</p><pre>export K3S_NODE_NAME=&quot;piworker1&quot;<br>export K3S_KUBECONFIG_MODE=&quot;600&quot;<br>export K3S_URL=&quot;https://&lt;MASTER_NODE_IP&gt;:6443&quot;<br>export K3S_TOKEN=&quot;&lt;ACCESS_TOKEN&gt;&quot;<br>export INSTALL_K3S_EXEC=&quot;agent --server $K3S_URL --token $K3S_TOKEN&quot;</pre><p>After setting these variables you will have to run the following command:</p><pre>curl -sfL https://get.k3s.io | sh -</pre><p>To check if the k3s service is running run this command:</p><pre>sudo systemctl status k3s-agent</pre><p>The output will start with something similar to the text block below:</p><pre>● k3s-agent.service - Lightweight Kubernetes<br>     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; preset: enabled)<br>     Active: active (running) since Fri 2024-07-12 15:29:46 EEST; 6h ago</pre><p>To check if the agent node has been added you can go back to the master node or a machine that has remote access to execute kubectl commands and execute this command:</p><pre>kubectl get node</pre><p>The output should look similar to this:</p><pre>NAME        STATUS   ROLES                       AGE   VERSION<br>pimaster    Ready    control-plane,etcd,master   30d   v1.29.5+k3s1<br>piworker1   Ready    &lt;none&gt;                      28d   v1.29.5+k3s1</pre><h3>Done!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67b46ceb7944" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes on Raspberry Pi 5 — Part 3 Install K3S on master node]]></title>
            <link>https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-3-install-k3s-on-master-node-f95ea35a8b1c?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f95ea35a8b1c</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[raspberry-pi-5]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[kubectl]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sat, 15 Jun 2024 12:24:06 GMT</pubDate>
            <atom:updated>2024-11-17T21:17:00.094Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes on Raspberry Pi 5 — Part 3 Install K3S on master node</h3><p>I have been using a Raspberry Pi 4 as a host for a Kubernetes for about two years. Recently I purchased the new Raspberry Pi 5. I plan to also setup Kubernetes on it so I can have a cluster with two nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QSRUzNRgfE1yjC9nGmW0IQ.jpeg" /></figure><h3>K3s on Raspberry Pi 5</h3><p><a href="https://ionutbanu.medium.com/k3s-on-raspberry-pi-5-part-1-install-raspberry-pi-os-lite-64-bit-29cdf41c7f5b">Part 1 Install Raspberry Pi OS Lite 64-bit</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-2-mount-external-storage-31ed1d0fc4bd">Part 2 Mount external storage</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-3-install-k3s-on-master-node-f95ea35a8b1c">Part 3 Install K3S on master node</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-4-install-k3s-on-agent-nodes-67b46ceb7944">Part 4 Install K3S on agent nodes</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-5-install-network-load-balancer-metallb-de1cd436d275">Part 5 Install network load balancer (MetalLB)</a></p><ol><li>Update kernel configuration file</li></ol><p>The <em>cmdline.txt</em> file contains command-line parameters that are passed to the kernel during the boot process. We need to add a couple parameters in this file. Open it using this command:</p><pre>sudo nano /boot/firmware/cmdline.txt</pre><p>It contains a set of parameters on one line. At the end of the line add the following parameters:</p><pre>cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory</pre><p>Save the file and reboot:</p><pre>sudo reboot</pre><p>2. Setup master node</p><p>Connect to the device that will be your k3s master node. Before installing k3s we first add a couple of environment variables.</p><p>These are the ones that I am using, you will have to replace the &lt;MASTER_NODE_IP&gt; placeholder.</p><ul><li>node-ip: the IP address of the node (the machine) on which the k3s agent or server is running. This IP address is used by the Kubernetes control plane to identify the node and facilitate communication within the cluster.</li><li>bind-address: the IP address on which the Kubernetes API server listens within a k3s cluster. This is particularly useful in scenarios where you have multiple network interfaces and want to ensure that the API server is accessible on a particular network.</li><li>advertise-address: the IP address that the Kubernetes API server advertises to other members of the cluster. This address is used by other nodes and components to connect to the API server.</li><li>flannel-backend: the backend for the Flannel network overlay, which is responsible for providing networking between the pods in a Kubernetes cluster. Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes, and it supports various backends that determine how the network traffic is managed and routed.</li><li>disable=servicelb: disable the embedded klipper-lb service load balancer. This is useful when you want to use an external load balancer, deploy a custom load balancer solution, or optimize resource usage by disabling unnecessary components. I will be using MeatlLB instead.</li><li>disable=traefik: disable the default Traefik ingress controller that is bundled with k3s. This option is useful if you want to use a different ingress controller, do not need ingress functionality, or require a custom ingress setup. I will be using nginx.</li><li>disable-cloud-controller: disable the cloud controller manager, which is responsible for integrating Kubernetes with cloud provider services. This option is particularly useful for on-premises deployments, edge computing, IoT scenarios, or any situation where cloud provider integration is not needed.</li><li>disable-helm-controller: prevent k3s from deploying and using the Helm controller. This is useful if you prefer to manage Helm charts manually or if you do not use Helm for managing your Kubernetes applications.</li><li>cluster-init: used to initialize a new Kubernetes cluster by setting up the first server node. This node acts as the initial control plane node and prepares the etcd datastore for the cluster. This option is essential for creating new clusters, particularly in high availability setups or when recovering a cluster. After initializing the cluster with --cluster-init, additional server and agent nodes can join the cluster to form a fully functional Kubernetes environment.</li></ul><pre>export K3S_NODE_NAME=&quot;pi-master&quot;<br>export K3S_KUBECONFIG_MODE=&quot;600&quot;<br>export INSTALL_K3S_EXEC=&quot;server --node-ip=&lt;MASTER_NODE_IP&gt; \<br>                                --bind-address=&lt;MASTER_NODE_IP&gt; \<br>                                --advertise-address=&lt;MASTER_NODE_IP&gt; \<br>                                --flannel-backend=host-gw \<br>                                --disable=servicelb \<br>                                --disable=traefik \<br>                                --disable-cloud-controller \<br>                                --disable-helm-controller \<br>                                --cluster-init&quot;</pre><p>I am using the setting K3S_KUBECONFIG_MODE=&quot;600&quot;, By setting this environment variable, you can control the permissions of the generated kubeconfig file. This is useful for managing access control and ensuring that the kubeconfig file has appropriate security settings. 600 means the owner user can read and write the file or directory and other users have no access to it.</p><p>After setting these variables you will have to run the following command:</p><pre>curl -sfL https://get.k3s.io | sh -</pre><p>To check if the k3s service is running run this command:</p><pre>sudo systemctl status k3s</pre><p>The output will start with something similar to the text block below:</p><pre>● k3s.service - Lightweight Kubernetes<br>     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)<br>     Active: active (running) since Sat 2024-06-08 12:55:15 EEST; 20h ago</pre><p>Because we used K3S_KUBECONFIG_MODE=&quot;600 we won’t be able to use kubectl yet. We need to make a copy of the config file and set it as an environment variable.</p><pre>export KUBECONFIG=~/.kube/config<br>mkdir ~/.kube 2&gt; /dev/null<br>sudo k3s kubectl config view --raw &gt; &quot;$KUBECONFIG&quot;<br>chmod 600 &quot;$KUBECONFIG&quot;</pre><p>In order for the environment variable to be persisted after restart you will have to add this line export KUBECONFIG=~/.kube/config at the end of the ~/.bashrc file. Then you can either restart or run this command source ~/.bashrc .</p><p>Now you are able to use kubectl :</p><pre>kubectl get pods -A</pre><p>You should see an output similar to this:</p><pre>NAME                                      READY   STATUS    RESTARTS      AGE<br>coredns-6799fbcd5-246w2                   1/1     Running   3 (31m ago)   44h<br>local-path-provisioner-6c86858495-kmn7s   1/1     Running   5 (31m ago)   44h<br>metrics-server-54fd9b65b-kv4s4            1/1     Running   4 (31m ago)   44h</pre><p>3. Setup kubectl remotely</p><p>In order to connect to the K3S cluster from your machine you will need the kube config file.</p><p>From you machine terminal execute the following command to copy the kube config file from the master node to your machine (don’t forget to replace the &lt;MASTER_NODE_IP&gt; placeholder):</p><pre>scp pi@&lt;MASTER_NODE_IP&gt;:~/.kube/config ~/.kube/config</pre><p>Now you should be able to execute kubectl commands remotely.</p><h3>Done!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f95ea35a8b1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes on Raspberry Pi 5— Part 2 Mount external storage]]></title>
            <link>https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-2-mount-external-storage-31ed1d0fc4bd?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/31ed1d0fc4bd</guid>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[external-hard-drive]]></category>
            <category><![CDATA[raspberry-pi-5]]></category>
            <category><![CDATA[fn]]></category>
            <category><![CDATA[fstab]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Sat, 04 May 2024 13:26:12 GMT</pubDate>
            <atom:updated>2024-11-17T21:16:23.948Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been using a Raspberry Pi 4 as a host for a Kubernetes for about two years. Recently I purchased the new Raspberry Pi 5. I plan to also setup Kubernetes on it so I can have a cluster with two nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QSRUzNRgfE1yjC9nGmW0IQ.jpeg" /></figure><h4>K3s on Raspberry Pi 5</h4><p><a href="https://ionutbanu.medium.com/k3s-on-raspberry-pi-5-part-1-install-raspberry-pi-os-lite-64-bit-29cdf41c7f5b">Part 1 Install Raspberry Pi OS Lite 64-bit</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-2-mount-external-storage-31ed1d0fc4bd">Part 2 Mount external storage</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-3-install-k3s-on-master-node-f95ea35a8b1c">Part 3 Install K3S on master node</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-4-install-k3s-on-agent-nodes-67b46ceb7944">Part 4 Install K3S on agent nodes</a></p><p><a href="https://ionutbanu.medium.com/kubernetes-on-raspberry-pi-5-part-5-install-network-load-balancer-metallb-de1cd436d275">Part 5 Install network load balancer (MetalLB)</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8O1hrG3MWTFySM_AfYfQA.jpeg" /></figure><p>I will be hosting a k3s deployment for my cluster. I have already installed Raspberry Pi OS Lite 64-bit, so the next step is to prepare the storage for my k3s cluster.</p><p>The drive that I am using is a Western Digital 5TB Elements external hard disk connected using a USB cable to one of the USB 3 ports on the board.</p><ol><li>[Optional] Formatting the drive</li></ol><p>If your drive is not formatted yet you will have to run the following commands after connecting the drive to your board. Running the command below will allow finding the partition of the drive.</p><pre>sudo fdisk -l</pre><p>In the output of the command your hard drive partition should be at the bottom. For example mine is <strong>/dev/sda</strong>.</p><pre>Disk /dev/sda: 4.55 TiB, 5000947302400 bytes, 9767475200 sectors<br>Disk model: Elements 2620<br>Units: sectors of 1 * 512 = 512 bytes<br>Sector size (logical/physical): 512 bytes / 4096 bytes<br>I/O size (minimum/optimal): 4096 bytes / 4096 bytes</pre><p>Now using the command below the drive will be formatted using the ext4 file system.</p><pre>mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sda</pre><p>The two options that I am using (lazy_itable_init=0,lazy_journal_init=0) initialise the inodes and the journal immediately during creation. This will slow down the formatting process, the kernel will not have to finish initializing the file system in the background when the file system is first mounted.</p><p>2. Create mount point</p><p>Now we create a directory where the external drive will be mounted (<strong>/mnt/hdd</strong>).</p><pre>mkdir /mnt/hdd</pre><p>3. Setup automatic mount</p><p>Because we need to have the drive to be mounted when the board starts we must setup automatic mounting using the<strong> /etc/fstab </strong>file. First we will run the command bellow to find the UUID of the external drive:</p><pre>blkid</pre><p>You will see a map where the id is the drive partition and the value is a list of properties including the <strong>UUID.</strong></p><pre>/dev/sda: UUID=&quot;&lt;UUID value&gt;&quot; BLOCK_SIZE=&quot;4096&quot; TYPE=&quot;ext4&quot;</pre><p>Open the <strong>/etc/fstab</strong> file using your preferred text editor. I am using nano.</p><pre>sudo nano /etc/fstab</pre><p>At the end of this file you will have to add a new line similar to the one below, but replacing <strong>&lt;UUID value&gt; </strong>with the one corresponding to your partition (the UUID from the <strong>blkid</strong> command output).</p><pre>UUID=&lt;UUID value&gt; /mnt/hdd ext4 defaults,nofail,async,noatime,nodiratime 0 0</pre><p>If you want to understand the meaning of the mount options that I am using (<strong>defaults,nofail,async,noatime,nodiratime</strong>) you can use the command <strong>man mount </strong>inside the terminal. You can also chose to use different options if you find that these do not suit you needs.</p><p>Now the drive is ready to be mounted. You can either run this command:</p><pre>sudo mount -a</pre><p>Or you can restart the board:</p><pre>sudo reboot</pre><p>And the drive will be mounted at boot time.</p><p>4. Sharing the drive</p><p>I order for the drive to be accessed by other boards in the cluster I am sharing it using NFS. In order to do this I need to first install <strong>nfs-kerner-server</strong>.</p><pre>sudo apt-get install nfs-kernel-server</pre><p>Next I go into the <strong>/etc/exports </strong>file:</p><pre>sudo nano /etc/exports</pre><p>And add the following line:</p><pre>/mnt/hdd 192.168.0.0/24(rw,no_root_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)</pre><p>This will allow only clients in the “192.168.0.xxx” subnet to connect to the drive.</p><p>Lastly I am restarting the NFS server in order to make it pickup the new setup.</p><pre>sudo systemctl restart nfs-kernel-server</pre><p>5. Mounting the shared drive</p><p>On the client boards I will install an NFS client:</p><pre>sudo apt-get install nfs-common</pre><p>Next I will create a mount point. I will use the same directory as on the server board, <strong>/mnt/hdd</strong>.</p><pre>sudo mkdir</pre><p>Like I did on the server board, I will open the <strong>/etc/fstab</strong> file:</p><pre>sudo nano /etc/fstab</pre><p>And add the following line where you will have to replace &lt;SERVER IP&gt; with the IP of your server board:</p><pre>&lt;SERVER IP&gt;:/mnt/hdd   /mnt/hdd   nfs    auto,bg,nofail,rw,async,retry=10,x-systemd.automount,x-systemd.idle-timeout=60s  0  0</pre><p>After rebooting the client board the shared drive should be mounted.</p><h3>Done!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31ed1d0fc4bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up key-pair ssh on Raspberry Pi]]></title>
            <link>https://ionutbanu.medium.com/setting-up-key-pair-ssh-on-raspberry-pi-9822b20037a0?source=rss-e7f1871e4b4e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9822b20037a0</guid>
            <category><![CDATA[ssh-keygen]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[ssh-copy-id]]></category>
            <category><![CDATA[ssh-keys]]></category>
            <category><![CDATA[ssh]]></category>
            <dc:creator><![CDATA[Ionut Banu]]></dc:creator>
            <pubDate>Tue, 20 Feb 2024 20:49:00 GMT</pubDate>
            <atom:updated>2024-02-20T20:49:00.066Z</atom:updated>
            <content:encoded><![CDATA[<p>If you plan to use a Raspberry Pi board as a home server you will most likely install an OS that does not come with an UI. In this case you will need to use ssh to remotely access and manage the server. In this article I am presenting how to setup using ssh with a key pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ixKBWOF8QkZsXTQlMrtowQ.jpeg" /><figcaption>Generated on <a href="https://dezgo.com/">dezgo.com</a></figcaption></figure><h4><strong>1. Generate private/public key pair on the client</strong></h4><p>There are multiple tools and libraries that facilitate key-pair generation. A common one is OpenSSL. In this tutorial I will be using a tool integrated in the macOS terminal, <em>ssh-keygen</em>.</p><p>Run the following command:</p><pre>ssh-keygen -t rsa</pre><p>Provide the path for the key pair to be stored at, for example:</p><pre>~/.ssh/rpi_rsa</pre><p>It is recommended to provide a passphrase when asked, but you will be asked for it each time you ssh into RPi.</p><p>At this point you should have two files at the location you provided, in my case the files are <em>rpi_rsa</em> and <em>rpi_rsa.pub</em>. The private key is kept secret and should never be shared. The public key can be freely distributed.</p><h4>2. Copy the generated public key to the RPi</h4><pre>ssh-copy-id -i ~/.ssh/rpi_rsa.pub &lt;RPi_USER&gt;@&lt;RPi_IP&gt;</pre><p>When a user attempts to connect to an SSH server, the server requests authentication. The user then presents their public key to the server.</p><p>The server verifies the authenticity of the public key by checking if it matches any of the authorized public keys stored on the server.</p><h4>3. Secure SSH into the server</h4><p>Now you should be able to execute the following command and ssh into the server without being asked for a password.</p><pre>ssh &lt;RPi_USER&gt;@&lt;RPi_IP&gt;</pre><p>When the public key is recognized, the server generates a challenge that can only be decrypted with the corresponding private key. The user’s SSH client uses the private key to decrypt the challenge and proves its identity to the server.</p><p>The decrypted challenge matches the expected response and access is granted to the user.</p><h4>[DEBUG]</h4><p>If you are still asked for the password you can check if the <em>ssh-copy-id </em>command worked.</p><p>SSH into the RPi and run the following command.</p><pre>nano ~/.ssh/authorized_keys</pre><p>You should find the content of the public key (the <em>rpi_rsa.pub </em>file) in here. If it is not here it means that the <em>ssh-copy-id </em>command failed.</p><h3>Done!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9822b20037a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>