<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Satoru_gojo on Medium]]></title>
        <description><![CDATA[Stories by Satoru_gojo on Medium]]></description>
        <link>https://medium.com/@kgang6434?source=rss-2d77b1d65065------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*RbW-TpWdZMV-Dkd9</url>
            <title>Stories by Satoru_gojo on Medium</title>
            <link>https://medium.com/@kgang6434?source=rss-2d77b1d65065------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:30:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kgang6434/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Re-thresholding Git LFS on a 5.5 GB Dataset Repo: A War Story]]></title>
            <link>https://medium.com/@kgang6434/re-thresholding-git-lfs-on-a-5-5-gb-dataset-repo-a-war-story-4d986ec77458?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d986ec77458</guid>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 07:21:38 GMT</pubDate>
            <atom:updated>2026-06-15T07:21:38.607Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How we moved a large dataset repository from “every binary in LFS” to a clean size-based policy — and the three walls we hit along the way (brittle path tracking, GitHub’s 2 GiB push cap, and the LFS quota that refuses to shrink).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*DwbkFTtJfusvVeXhhbMuEQ.png" /></figure><h3>TL;DR</h3><ul><li>We had a 5.5 GB dataset repo: ~15,000 binary files (pdf, jpg, png, mp4, docx, xlsx, mp3, m4a), 9,970 in Git LFS, and 5,074 accidentally committed as raw blobs in git.</li><li>Goal: stop wasting LFS quota on tiny files; keep only genuinely large files in LFS.</li><li>We learned the hard way that:</li></ul><ol><li>git lfs migrate with path-based --exclude is brittle — it misses files that live at historical/renamed paths. Use the size-based --above instead.</li><li>GitHub hard-rejects any single push larger than 2 GiB (pack exceeds maximum allowed size) on <em>both</em> HTTPS and SSH. Big rewrites must be pushed in batches.</li><li>Rewriting history does not reclaim LFS storage. Old objects linger on GitHub’s servers until a separate server-side purge.</li></ol><ul><li>Net result: a clean, size-based LFS policy enforced and pushed, with a repeatable playbook.</li></ul><h3>The setup</h3><p>The repo is a machine-learning dataset: thousands of task folders, each with documents, images, audio, and the occasional video. Someone had wisely set up .gitattributes to route binaries into Git LFS:</p><pre>*.pdf  filter=lfs diff=lfs merge=lfs -text<br>*.jpg  filter=lfs diff=lfs merge=lfs -text<br>*.png  filter=lfs diff=lfs merge=lfs -text<br>*.mp4  filter=lfs diff=lfs merge=lfs -text<br>*.docx filter=lfs diff=lfs merge=lfs -text<br>*.xlsx filter=lfs diff=lfs merge=lfs -text<br>*.mp3  filter=lfs diff=lfs merge=lfs -text<br>*.m4a  filter=lfs diff=lfs merge=lfs -text</pre><p>Two things were wrong.</p><h3>Problem 1: thousands of files bypassed LFS</h3><p>git lfs fsck --pointers flagged 5,074 files as <em>&quot;should have been a pointer but was not.&quot;</em> They were committed as raw binary blobs directly in git, despite .gitattributes saying they should be LFS pointers.</p><p>The smoking gun: a .pdf committed <em>after</em> the LFS rule already existed was still a raw %PDF-1.3 blob, not a version https://git-lfs… pointer. That only happens when the LFS clean filter never ran at commit time — i.e., a contributor committed without git lfs install active on their machine (or via a path that bypasses smudge/clean filters, like a web upload).</p><blockquote><em>Lesson: A correct </em><em>.gitattributes is necessary but not sufficient. Every contributor must run </em><em>git lfs install once, or raw files silently leak into history. Add a CI check that rejects non-pointer blobs for tracked extensions.</em></blockquote><p>To find them:</p><pre>git lfs fsck --pointers 2&gt;&amp;1 \<br>  | grep &#39;unexpectedGitObject&#39; | grep -oE &#39;&quot;[^&quot;]+&quot;&#39; | sed &#39;s/&quot;//g&#39;</pre><h3>Problem 2: LFS was full of tiny files</h3><p>Profiling the size distribution per extension told the real story:</p><p>extfilestotalmedianmaxjpg3,7312.5 GB221 KB72.9 MBmp41541.1 GB810 KB444 MBpng1,206838 MB256 KB23.4 MBpdf5,195681 MB2.3 KB322 MBdocx2,989105 MB36 KB353 KBxlsx1,5799.2 MB5.3 KB49 KB</p><p>Look at pdf: median 2.3 KB. Thousands of tiny generated PDFs were each consuming an LFS object, while the 681 MB total came from a <em>handful</em> of large ones. xlsx and docx were entirely small — no file over a few hundred KB. Putting those in LFS buys nothing and burns quota.</p><h3>The core tension nobody tells you about</h3><p>Here’s the trap: for any single file, you either keep it in git (bloats the repo pack) or in LFS (uses LFS quota). You cannot minimize both for the same file. They trade 1:1 in bytes.</p><p>So “minimize repo size” and “save LFS quota” pull in opposite directions. The resolution is size-based routing:</p><ul><li>Large files → LFS. They’d bloat the git pack and slow every clone; LFS is where they belong.</li><li>Small files → plain git. Negligible pack impact, and they stop wasting LFS quota/objects.</li></ul><p>GitHub’s own limits anchor the thresholds:</p><ul><li>Files ≥ 100 MiB are <em>hard-rejected</em> from plain git → must be LFS.</li><li>Files &gt; 50 MiB trigger a warning.</li></ul><p>We chose a 99 MB threshold: files ≥ 99 MB go to LFS, everything else is plain git. (In this repo, only 4 files qualified.)</p><blockquote><em>Caveat we surfaced honestly: a </em>high<em> threshold like 99 MB optimizes LFS quota, not repo size — it pushes ~1.7 GB of medium files </em>into<em> the git pack. If your priority is a small clonable repo, you want a </em>low<em> threshold (keep big media in LFS). Know which one you’re optimizing.</em></blockquote><h3>The migration</h3><p>Two git lfs migrate operations, opposite directions:</p><ul><li>export — convert LFS pointers back into raw git blobs (pull small files <em>out</em> of LFS).</li><li>import — convert raw blobs into LFS pointers (push big files <em>into</em> LFS).</li></ul><pre># 0. Safety first — a full mirror backup. Always.<br>git clone --mirror . ../repo-backup.git<br>git lfs install</pre><pre># 1. Pull everything &lt; threshold out of LFS, into plain git.<br>git lfs migrate export --everything \<br>  --include=&quot;*.pdf,*.jpg,*.png,*.mp4,*.mp3,*.m4a,*.xlsx,*.docx&quot; \<br>  --exclude=&quot;path/to/big1.mp4,path/to/big2.pdf,...&quot;   # keep the big ones in LFS</pre><p>This rewrites history (every commit gets a new SHA). Verify:</p><pre>git lfs ls-files | wc -l          # dropped from 9,970 to a handful<br>git lfs fsck                      # OK</pre><p>Looked great. Then we tried to push — and hit the walls.</p><h3>Wall #1: path-based --exclude is a landmine</h3><p>Our --exclude listed the big files by path, discovered by scanning HEAD. But a ≥100 MB clip_03.mp4 existed at a nested, historical path that wasn&#39;t at HEAD:</p><pre>New_qcied_tasks/…aaron_garcia…/…christopher_bell…/data/clip_03.mp4   (101.5 MB)</pre><p>migrate export rewrites all history with *.mp4, so this file got exported into a raw 101.5 MB blob — and the push died:</p><pre>remote: error: File …/clip_03.mp4 is 101.55 MB; this exceeds GitHub&#39;s file size limit of 100.00 MB<br>remote: error: GH001: Large files detected.<br> ! [remote rejected] (pre-receive hook declined)</pre><p>.gitattributes and path-based excludes only know about the paths you fed them. History contains files at paths that no longer exist at HEAD.</p><h3>The fix: size-based migration</h3><p>git lfs migrate <em>does</em> have a size predicate. Use it — it scans every blob in history by size, regardless of path:</p><pre>git lfs migrate import --above=99MB      # any blob &gt;= 99 MB, anywhere, becomes an LFS pointer</pre><p>Verify zero raw large blobs remain before re-pushing:</p><pre>git rev-list --objects main \<br>  | git cat-file --batch-check=&#39;%(objecttype) %(objectsize)&#39; \<br>  | awk &#39;$1==&quot;blob&quot; &amp;&amp; $2&gt;=103809024&#39; | wc -l    # must be 0   (103809024 = 99 MiB)</pre><blockquote><em>Lesson: For a </em>size<em>-based policy, drive it with </em><em>--above, not path globs. Path lists can&#39;t see the past.</em></blockquote><h3>Wall #2: GitHub’s 2 GiB per-push cap</h3><p>After the rewrite, the force-push was a single 3.05 GiB pack. Over HTTPS:</p><pre>error: RPC failed; HTTP 500<br>send-pack: unexpected disconnect while reading sideband packet<br>fatal: the remote end hung up unexpectedly</pre><p>We switched to SSH, hoping to dodge the HTTP layer. Cleaner error, same wall:</p><pre>remote: fatal: pack exceeds maximum allowed size (2.00 GiB)</pre><p>GitHub hard-caps a single push at 2 GiB, on both HTTPS and SSH. A full-history rewrite is one disjoint pack (every commit is “new” to the remote), so it has to go up in one shot — unless you chunk it.</p><h3>The batched-push technique</h3><p>Advance the remote branch through a chain of ancestor commits, each push staying under 2 GiB. Git’s negotiation means each push only sends the delta since the last one:</p><pre>git push --force origin &lt;ancestor-sha-1&gt;:refs/heads/main   # ~1.2 GB<br>git push --force origin &lt;ancestor-sha-2&gt;:refs/heads/main   # ~0.9 GB<br>git push --force origin main                               # ~1.1 GB → real tip</pre><p>To pick safe boundaries, measure the cumulative push size along the first-parent chain (on-disk object size of what the remote doesn’t already have):</p><pre>BASE=$(git ls-remote origin -h refs/heads/main | cut -f1)   # what the remote already has<br>prev=$BASE; cum=0<br>git rev-list --reverse --first-parent main &gt; /tmp/C.txt<br>while read -r sha; do<br>  d=$(git rev-list --objects &quot;$sha&quot; --not &quot;$prev&quot; &quot;$BASE&quot; \<br>        | awk &#39;{print $1}&#39; | git cat-file --batch-check=&#39;%(objectsize:disk)&#39; \<br>        | awk &#39;{s+=$1} END{print s+0}&#39;)<br>  cum=$((cum+d)); prev=$sha<br>  # cut a boundary whenever cum crosses ~1.3-1.5 GB<br>done &lt; /tmp/C.txt</pre><p>Properties that make this safe:</p><ul><li>Resumable. A landed chunk stays landed. If one is still too big, lower the boundary and re-run — it continues from the remote’s current position, never from zero.</li><li>Negotiated. Unchanged text/JSON blobs the remote already has are skipped automatically.</li><li>Do it in one sitting. Between chunks the remote branch points at an intermediate (valid but incomplete) commit.</li></ul><h3>Wall #3: the LFS quota that won’t shrink</h3><p>We expected LFS usage to drop after the rewrite. It didn’t — and this surprises almost everyone.</p><p>Rewriting history does not delete LFS objects from GitHub’s servers. LFS objects live in a content-addressed store <em>separate</em> from git history. Force-pushing a history that no longer references them leaves them sitting there, still counting against your quota.</p><ul><li>git gc / git lfs prune only clean your local .git/lfs cache. (Ours dropped to 868 MB locally — but the remote was unchanged.)</li><li>GitHub provides no self-service way to delete unreferenced LFS objects.</li></ul><p>Your only options to reclaim remote LFS storage:</p><ol><li>GitHub Support request — ask them to purge unreferenced LFS objects. Non-destructive; keeps issues/PRs/settings. Slow.</li><li>Delete &amp; recreate the repo — wipes the entire LFS store instantly. Destructive: you lose issues, PRs, stars, webhooks, branch protections, access config.</li></ol><p>…and a prerequisite for <em>either</em>: no other branch, tag, or open PR may still reference the old objects, or they stay “referenced” and won’t be purged.</p><blockquote><em>Lesson: History rewrite reduces what your repo </em>references<em>, not what GitHub </em>stores<em>. Budget a separate purge step — and the monthly LFS bandwidth you spend on the big re-push is gone regardless.</em></blockquote><h3>The final state</h3><pre>$ git ls-remote origin -h refs/heads/main | cut -f1<br>0b22c92489f2064d39b4c1db93d6eefad88b227d<br>$ git rev-parse main<br>0b22c92489f2064d39b4c1db93d6eefad88b227d   # ✅ remote == local</pre><pre>$ git lfs ls-files | wc -l<br>4                                          # only the genuinely huge files<br>$ git rev-list --objects main | git cat-file --batch-check=&#39;%(objecttype) %(objectsize)&#39; \<br>    | awk &#39;$1==&quot;blob&quot; &amp;&amp; $2&gt;=103809024&#39; | wc -l<br>0                                          # ✅ no raw blob &gt;= 99 MB anywhere</pre><p>A clean, size-based LFS policy, enforced across history and live on the remote.</p><h3>Playbook (the short version)</h3><ol><li>Audit. git lfs fsck --pointers for leaks; profile sizes per extension before deciding a threshold.</li><li>Decide what you’re optimizing — repo size vs. LFS quota. They trade off; a high threshold favors quota, a low one favors a small repo.</li><li>Back up. git clone --mirror before any migrate.</li><li>Migrate by size, not path. git lfs migrate import --above=&lt;N&gt; and git lfs migrate export with care. Then assert 0 raw blobs ≥ your hard limit remain.</li><li>Push in &lt;2 GiB batches. Measure cumulative size, advance the branch through ancestor commits, finish in one sitting.</li><li>Purge separately. Plan a server-side LFS purge to actually reclaim quota.</li><li>Prevent recurrence. Enforce git lfs install; add CI that rejects oversized non-pointer blobs (path-based .gitattributes won&#39;t auto-catch a <em>new</em> big file).</li></ol><h3>Commands worth keeping</h3><pre># Find files committed raw that should be LFS pointers<br>git lfs fsck --pointers</pre><pre># Largest raw blobs anywhere in history (not just HEAD)<br>git rev-list --objects --all \<br>  | git cat-file --batch-check=&#39;%(objecttype) %(objectsize) %(rest)&#39; \<br>  | awk &#39;$1==&quot;blob&quot; &amp;&amp; $2&gt;=52428800 {printf &quot;%.1f MiB  %s\n&quot;, $2/1048576, $3&quot; &quot;$4}&#39; \<br>  | sort -rn | head</pre><pre># Size-based migration into LFS (path-agnostic)<br>git lfs migrate import --above=99MB</pre><pre># What a push would transfer, given what the remote already has<br>git rev-list --objects &lt;target&gt; --not &lt;remote-has&gt; \<br>  | awk &#39;{print $1}&#39; | git cat-file --batch-check=&#39;%(objectsize:disk)&#39; \<br>  | awk &#39;{s+=$1} END{printf &quot;%.2f GB\n&quot;, s/1073741824}&#39;</pre><p><em>Found this useful, or have a cleaner way to reclaim remote LFS storage? I’d love to hear it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d986ec77458" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Guardrails]]></title>
            <link>https://medium.com/@kgang6434/llm-guardrails-41622c242dba?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/41622c242dba</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[guardrail]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 16:04:04 GMT</pubDate>
            <atom:updated>2025-12-17T16:04:04.877Z</atom:updated>
            <content:encoded><![CDATA[<p>As the models are evolving so the agents are and so the malicious user activities are increasing. So to keep these activities we as software devs need to implement some guardrails technique to save both user as well as agent from cunning malicious users.</p><p>Before going deep dive into LLM Guardrails , Lets look for the potential risk that can be easily mitigated by following couple of good practices.</p><p><strong>Types of risks in GenAI Systems</strong></p><ol><li>Prompt Injection: Malicious instructions embedded in the user input (from application layer)</li><li>Jailbreaking: Trying to break built-in safety filters within LLM models (not application layer)</li><li>PII leakage: Exposing sensitive personal information.</li><li>Toxicity: Generating offensive, hateful or harmful content.</li><li>Hallucinations: Confidently stating false information.</li><li>Off-topic Responses: Going beyond intended scope</li><li>Compliance Violations: Breaking industry regulations (HIPAA,GDPR, financial regulations)</li><li>Cost and Resource Abuse: Attackers or users abuse the system to rack up the costs.(Infinite loop prompts that keep generating)</li></ol><p><strong>Sections of Guardrails</strong></p><ol><li>Input Guardrails (pre processing)</li></ol><ul><li>Applied before the prompt reaches the LLM</li><li>check user input for safety/validity</li><li>Detect prompt injection attempts</li><li>Filter out toxic language from users</li><li>Block PII in user queries</li><li>Validate input is on-topic</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QLw6hFRJJ20AO3KBeoHoQw.png" /></figure><p>2. Output Guardrails (post-processing)</p><ul><li>Applied after the LLM generates a response before showing it to user</li><li>check the llm’s response for safety and quality.</li><li>Detect if LLM leaked PII</li><li>Check for hallucinations</li><li>Filer toxic content in response</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*6qAPDXCKdRLw0xVT3VTNkw.png" /></figure><p>3. Contextual/Runtime Guardrails</p><ul><li>Monitors the entire conversation flow</li><li>Helps detect topic drift</li><li>Monitor conversation staying on topic across turns</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*edVvjKORsZaDrolmNXUX_g.png" /></figure><h3><strong>Guardrail Strategies</strong></h3><ol><li>Rule Based Guardrails</li><li>ML-Based Guardrails</li><li>LLM-Based guardrails</li><li>Hybrid</li></ol><h4><strong>Rule Based Guardrails</strong></h4><p>Here, we use explicit rules and pattern matching , Regular expresions, keyword blocklists</p><pre>import words from &#39;profane-words&#39;<br>const arg = process.argv[2]<br><br>if (words.includes(arg.toLowerCase())) {<br>  console.warn(&#39;Watch your mouth.&#39;)<br>}</pre><h4><strong>ML-Based Guardrails</strong></h4><p>ML based guardrails use trained machine learning models (not LLM always) to classify or scoe inputs and outputs. They are usually not LLMs but smaller models that are build to identify safe vs toxic conent</p><ul><li><strong>OpenAI moderation API:- </strong>This is a content-safety classifier model and not an LLM</li></ul><pre>from openai import OpenAI<br>client = OpenAI()<br><br>response = client.moderations.create(<br>    model=&quot;omni-moderation-latest&quot;,<br>    input=&quot;...text to classify goes here...&quot;,<br>)<br><br>print(response)</pre><p>example output</p><pre>{<br>  &quot;id&quot;: &quot;modr-970d409ef3bef3b70c73d8232df86e7d&quot;,<br>  &quot;model&quot;: &quot;omni-moderation-latest&quot;,<br>  &quot;results&quot;: [<br>    {<br>      &quot;flagged&quot;: true,<br>      &quot;categories&quot;: {<br>        &quot;sexual&quot;: false,<br>        &quot;sexual/minors&quot;: false,<br>        &quot;harassment&quot;: false,<br>        &quot;harassment/threatening&quot;: false,<br>        &quot;hate&quot;: false,<br>        &quot;hate/threatening&quot;: false,<br>        &quot;illicit&quot;: false,<br>        &quot;illicit/violent&quot;: false,<br>        &quot;self-harm&quot;: false,<br>        &quot;self-harm/intent&quot;: false,<br>        &quot;self-harm/instructions&quot;: false,<br>        &quot;violence&quot;: true,<br>        &quot;violence/graphic&quot;: false<br>      },<br>      &quot;category_scores&quot;: {<br>        &quot;sexual&quot;: 2.34135824776394e-7,<br>        &quot;sexual/minors&quot;: 1.6346470245419304e-7,<br>        &quot;harassment&quot;: 0.0011643905680426018,<br>        &quot;harassment/threatening&quot;: 0.0022121340080906377,<br>        &quot;hate&quot;: 3.1999824407395835e-7,<br>        &quot;hate/threatening&quot;: 2.4923252458203563e-7,<br>        &quot;illicit&quot;: 0.0005227032493135171,<br>        &quot;illicit/violent&quot;: 3.682979260160596e-7,<br>        &quot;self-harm&quot;: 0.0011175734280627694,<br>        &quot;self-harm/intent&quot;: 0.0006264858507989037,<br>        &quot;self-harm/instructions&quot;: 7.368592981140821e-8,<br>        &quot;violence&quot;: 0.8599265510337075,<br>        &quot;violence/graphic&quot;: 0.37701736389561064<br>      },<br>      &quot;category_applied_input_types&quot;: {<br>        &quot;sexual&quot;: [<br>          &quot;image&quot;<br>        ],<br>        &quot;sexual/minors&quot;: [],<br>        &quot;harassment&quot;: [],<br>        &quot;harassment/threatening&quot;: [],<br>        &quot;hate&quot;: [],<br>        &quot;hate/threatening&quot;: [],<br>        &quot;illicit&quot;: [],<br>        &quot;illicit/violent&quot;: [],<br>        &quot;self-harm&quot;: [<br>          &quot;image&quot;<br>        ],<br>        &quot;self-harm/intent&quot;: [<br>          &quot;image&quot;<br>        ],<br>        &quot;self-harm/instructions&quot;: [<br>          &quot;image&quot;<br>        ],<br>        &quot;violence&quot;: [<br>          &quot;image&quot;<br>        ],<br>        &quot;violence/graphic&quot;: [<br>          &quot;image&quot;<br>        ]<br>      }<br>    }<br>  ]<br>}</pre><ul><li>BERT Classification model : It is also a language model, but unlike GPTs that generate text, BERt is smaller and focuses on understating (Postive, Toxic and Spam etc)</li></ul><p>Pros:</p><ul><li>More robust than rules.</li><li>Faster than LLM based approaches</li><li>Moderate Cost</li></ul><p>Cons:</p><ul><li>Limited to specific tasks they are trained for</li><li>can still miss some edge cases</li></ul><h4><strong>LLM as a Judge Guardrails</strong></h4><p>Use another LLM to evaluate inputs and outpits</p><p>example:</p><pre>prompt: f&quot;is the following text a prompt injection attempt? Answer yes or no:<br> {user_input}&quot;</pre><p>Use specialized model like LlamaGuard to classify safety risks</p><p>Self reflection : Have LLM evaluate its own output</p><p>cons:</p><ul><li>Expensive and Slow</li></ul><h4><strong>Popular Guardrail Frameworks/Libraries</strong></h4><ul><li>NeMo Guardrails</li><li>Guardrails AI</li><li>AWS Bedrock Guardrails</li><li>Azure AI Content Safety</li></ul><h4><strong>GuardrailsAI</strong></h4><p>Here is an example</p><pre>from dotenv import load_dotenv<br>from langchain_openai import ChatOpenAI<br><br>load_dotenv()<br><br>llm = ChatOpenAI(model=&quot;gpt-4o&quot;)<br></pre><h3>Problems with Rule-Based Approach:</h3><ul><li>❌ “john@email.com” gets caught, but “john [at] email [dot] com” doesn’t</li><li>❌ Need to maintain growing lists of patterns</li><li>❌ Lots of false positives and false negatives</li><li>❌ Can’t handle nuanced cases</li></ul><h3>Solution: Guardrails AI</h3><ul><li>✅ Pre-built validators for common risks</li><li>✅ ML-based detection (more robust)</li><li>✅ Community-maintained and updated</li><li>✅ Easy to use and compose</li></ul><pre># Run the configure command:<br>!guardrails configure<br><br># Now install the validator<br>!guardrails hub install hub://guardrails/detect_pii<br><br># Install toxic language detector<br>!guardrails hub install hub://guardrails/toxic_language</pre><pre>from guardrails import Guard<br>from guardrails.hub import DetectPII<br><br># Create a guard with PII detection<br>guard = Guard().use(<br>    DetectPII(<br>        pii_entities=[&quot;EMAIL_ADDRESS&quot;, &quot;PHONE_NUMBER&quot;],<br>        on_fail=&quot;exception&quot;   #exception<br>    )<br>)<br><br>print(&quot;✅ Guard created with DetectPII validator&quot;)<br><br># Test 1: Text with PII (should be caught)<br>text_with_pii = &quot;My email is john@example.com and my phone is 555-123-4567&quot;<br><br>print(&quot;Testing text with PII:&quot;)<br>print(f&quot;Input: {text_with_pii}\n&quot;)<br><br>try:<br>    result = guard.validate(text_with_pii)<br>    print(f&quot;✅ Validation passed&quot;)<br>    print(f&quot;Output: {result.validated_output}&quot;)<br>except Exception as e:<br>    print(f&quot;🚫 Validation failed!&quot;)<br>    print(f&quot;Error: {str(e)}&quot;)<br><br># Test 2: Text without PII (should pass)<br>text_without_pii = &quot;I love learning about AI and guardrails!&quot;<br><br>print(&quot;\nTesting text without PII:&quot;)<br>print(f&quot;Input: {text_without_pii}\n&quot;)<br><br>try:<br>    result = guard.validate(text_without_pii)<br>    print(f&quot;✅ Validation passed&quot;)<br>    print(f&quot;Output: {result.validated_output}&quot;)<br>except Exception as e:<br>    print(f&quot;🚫 Validation failed!&quot;)<br>    print(f&quot;Error: {str(e)}&quot;)</pre><pre># These variations would BYPASS our simple regex from Notebook 1<br>standard_pii_cases = [<br>    &quot;Email me at john.doe@company.com&quot;,<br>    &quot;Call 555-123-4567 for more info&quot;,<br>    &quot;My phone number is (555) 123-4567&quot;,<br>    &quot;Contact: jane_smith@example.org&quot;,<br>    &quot;SSN: 123-45-6789&quot;,  # Let&#39;s see if it catches this<br>]<br><br>print(&quot;Testing tricky PII variations:\n&quot;)<br>for i, text in enumerate(standard_pii_cases, 1):<br>    print(f&quot;Test {i}: {text}&quot;)<br>    try:<br>        result = guard.validate(text)<br>        print(f&quot;   ✅ Passed (no PII detected)&quot;)<br>    except Exception as e:<br>        print(f&quot;   🚫 Caught! DetectPII found PII&quot;)<br>    print()</pre><pre>from guardrails.hub import ToxicLanguage<br><br># Create a guard for toxic content<br>toxic_guard = Guard().use(<br>    ToxicLanguage(threshold=0.5, on_fail=&quot;exception&quot;)<br>)<br># Test cases<br>toxic_tests = [<br>    &quot;I hate you and want to harm you&quot;,  # Toxic<br>    &quot;You&#39;re stupid and worthless&quot;,       # Toxic<br>    &quot;I love learning about AI!&quot;,         # Safe<br>    &quot;What&#39;s the weather today?&quot;,         # Safe<br>]<br><br>for text in toxic_tests:<br>    print(f&quot;Test: {text}&quot;)<br>    try:<br>        toxic_guard.validate(text)<br>        print(&quot;   ✅ Safe\n&quot;)<br>    except:<br>        print(&quot;   🚫 BLOCKED - Toxic content\n&quot;)</pre><h3>Key Takeaway: ML-Based vs Rule-Based</h3><p><strong>What we just saw:</strong></p><ul><li>ToxicLanguage uses a trained ML model to detect toxicity</li><li>It understands context and nuance (not just keyword matching)</li><li>More robust than our simple regex patterns from Notebook 1</li></ul><p><strong>Advantages over rule-based:</strong></p><ul><li>Catches variations and synonyms</li><li>Understands context (e.g., “I hate broccoli” vs “I hate you”)</li><li>Doesn’t need constant manual updates</li></ul><p><strong>Tradeoffs:</strong></p><ul><li>Slower (ML inference takes time)</li><li>Requires model downloads (~760MB for ToxicLanguage)</li><li>Slightly less predictable than exact rules</li></ul><pre># How to Combine Multiple Validators<br><br># You can combine multiple validators in a single Guard!<br>from guardrails.hub import DetectPII, ToxicLanguage<br><br># Create a guard with BOTH PII detection AND toxic language detection<br>combined_guard = Guard().use_many(<br>    DetectPII(pii_entities=[&quot;EMAIL_ADDRESS&quot;, &quot;PHONE_NUMBER&quot;]),<br>    ToxicLanguage(threshold=0.5, on_fail=&quot;exception&quot;)<br>)<br><br>print(&quot;✅ Combined guard created with multiple validators&quot;)<br><br># Test cases that combine both risks<br>test_cases = [<br>    &quot;I hate you! Here&#39;s my email: john@example.com&quot;,  # Both toxic AND PII<br>    &quot;Contact me at support@company.com&quot;,              # Just PII<br>    &quot;You&#39;re an idiot&quot;,                                # Just toxic<br>    &quot;What&#39;s the weather today?&quot;,                      # Safe<br>]<br><br>print(&quot;Testing combined guard:\n&quot;)<br>for text in test_cases:<br>    print(f&quot;Test: {text}&quot;)<br>    try:<br>        combined_guard.validate(text)<br>        print(&quot;   ✅ Safe\n&quot;)<br>    except Exception as e:<br>        print(f&quot;   🚫 BLOCKED - {str(e)[:50]}...\n&quot;)</pre><h3>Integrating Guardrails with LLMs</h3><p>Now let’s use our guardrails in a real workflow:</p><ol><li>Validate user input BEFORE sending to LLM</li><li>Call the LLM</li><li>Validate LLM output BEFORE showing to user</li></ol><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage<br><br># Initialize LLM<br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br><br>def safe_llm_call_with_guardrails(user_input: str):<br>    &quot;&quot;&quot;<br>    Call LLM with input and output guardrails<br>    &quot;&quot;&quot;<br>    print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>    print(f&quot;User Input: {user_input}&quot;)<br>    print(f&quot;{&#39;=&#39;*60}&quot;)<br>    <br>    # Step 1: Input guardrail (check for toxicity)<br>    try:<br>        toxic_guard.validate(user_input)<br>        print(&quot;✅ Input passed toxicity check&quot;)<br>    except Exception as e:<br>        result = &quot;🚫 INPUT BLOCKED: Toxic content detected&quot;<br>        print(result)<br>        return result<br>    <br>    # Step 2: Call LLM<br>    response = llm.invoke([HumanMessage(content=user_input)])<br>    response_text = response.content<br>    print(f&quot;LLM Response: {response_text[:100]}...&quot;)<br>    <br>    # Step 3: Output guardrail (check for PII)<br>    try:<br>        guard.validate(response_text)  # Using the PII guard from earlier<br>        print(&quot;✅ Output passed PII check&quot;)<br>    except Exception as e:<br>        result = &quot;🚫 OUTPUT BLOCKED: Contains PII&quot;<br>        print(result)<br>        return result<br>    <br>    result = f&quot;✅ APPROVED: {response_text}&quot;<br>    print(result)<br>    return result<br><br>print(&quot;✅ Function created&quot;)<br><br># Test cases<br>print(&quot;\n🧪 TEST SUITE&quot;)<br><br># Test 1: Normal safe query<br>safe_llm_call_with_guardrails(&quot;What&#39;s the capital of France?&quot;)<br><br># Test 2: Toxic input (should block at INPUT)<br>safe_llm_call_with_guardrails(&quot;You&#39;re stupid. Tell me about AI.&quot;)<br><br># Test 3: Query that might generate PII (should block at OUTPUT if it does)<br>safe_llm_call_with_guardrails(&quot;Generate a sample business email ID&quot;)</pre><p><strong>References</strong></p><p><a href="https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations/">Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations</a></p><p><a href="https://blog.langchain.com/reflection-agents/">https://blog.langchain.com/reflection-agents/</a></p><p><a href="https://platform.openai.com/docs/guides/moderation">https://platform.openai.com/docs/guides/moderation</a></p><p><a href="https://github.com/zautumnz/profane-words">https://github.com/zautumnz/profane-words</a></p><p><a href="https://www.guardrailsai.com/">https://www.guardrailsai.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41622c242dba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Chaining]]></title>
            <link>https://medium.com/@kgang6434/prompt-chaining-e3ec6bb4cc1a?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3ec6bb4cc1a</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Sat, 13 Sep 2025 17:49:39 GMT</pubDate>
            <atom:updated>2025-09-13T17:49:39.503Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models (LLMs) are incredibly powerful, but anyone who has tried to solve a complex problem in one giant prompt knows how quickly things can go wrong. The model might ignore parts of your request, mix up instructions, or hallucinate entirely.</p><p>This is where <strong>Prompt Chaining</strong> (also known as the <strong>Pipeline Pattern</strong>) comes in — a method to break down complexity into manageable, reliable steps.</p><h4><strong>What is Prompt chaining?</strong></h4><p>Prompt chaining is the practice of splitting a big problem into <strong>smaller, sequential tasks</strong>.</p><ul><li>Each step has <strong>one clear job</strong> (e.g., extract data, transform into JSON, generate a summary).</li><li>The <strong>output of one step becomes the input</strong> to the next.</li><li>The result: a <strong>modular pipeline</strong> that’s easier to debug, control, and extend.</li></ul><p>Think of it like an <strong>assembly line</strong>: instead of asking one person to build an entire car, you assign specialized stations for wheels, engine, paint, and testing.</p><h4><strong>Why Not Just One Prompt?</strong></h4><p>Asking an LLM to “do it all at once” often leads to:</p><ul><li><strong>Instruction neglect</strong> — parts of your request get ignored</li><li><strong>Contextual drift</strong> — the model loses track of earlier details.</li><li><strong>Error propagation</strong> — one mistake compounds into many.</li><li><strong>Token limits</strong> — long context windows can be exceeded.</li><li><strong>Hallucinations</strong> — the model improvises when overloaded.</li></ul><p>Prompt chaining avoids these pitfalls by forcing the model to <strong>focus narrowly at each stage</strong>.</p><h4><strong>How Prompt Chaining Works?</strong></h4><p>The core idea:</p><ol><li><strong>Decompose</strong> the problem into steps.</li><li><strong>Design prompts</strong> for each step (clear and structured).</li><li><strong>Link outputs → inputs</strong> across the chain.</li><li>(Optional) Use <strong>structured outputs</strong> like JSON to reduce ambiguity.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ow3WIKZCQe0dWhrK0DhKrw.png" /><figcaption>Prompt Chaining</figcaption></figure><h3>Real-World Applications</h3><p>Prompt chaining is already powering many practical use cases:</p><p><strong>Information Processing Pipelines</strong></p><ul><li>Summarize long reports</li><li>Extract entities (names, dates, numbers)</li><li>Query a knowledge base</li><li>Generate user-friendly insights</li></ul><p><strong>Complex Query Answering</strong></p><ul><li>Break a tricky question into sub-questions</li><li>Answer them separately.</li><li>Synthesize a final comprehensive response</li></ul><p><strong>Data Extraction &amp; Transformation</strong></p><ul><li>OCR invoices → extract fields → validate → calculate totals</li></ul><p><strong>Content Generation</strong></p><ul><li>Brainstorm ideas → create outline → draft sections → polish tone</li></ul><p><strong>Conversational Agents</strong></p><ul><li>Maintain state across multi-turn chats</li><li>Route conversations based on intent</li></ul><p><strong>Code Generation</strong></p><ul><li>Convert requirements → pseudocode → working code → debugging → docs</li></ul><p><strong>Multimodal Workflows</strong></p><ul><li>Combine image → text → table interpretation</li><li>Example: “Read chart → explain trends → give business recommendations”</li></ul><h3>Prompt Chaining in Practice</h3><p>Modern frameworks make prompt chaining easy:</p><ul><li><strong>LangChain</strong> — popular for chaining, retrieval, and agents</li><li><strong>LangGraph</strong> — graph-based workflows for LLMs</li><li><strong>Crew AI</strong> — multi-agent collaboration</li><li><strong>Google AI Development Kit (ADK)</strong> — enterprise-grade pipelines</li></ul><p>Following is the code example</p><pre>pip install langchain langchain-community langchain-openai langgraph</pre><pre>import os<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>llm = ChatOpenAI(temperature=0, model=&quot;gpt-3.5-turbo&quot;)<br><br># --- Prompt 1: Extract Information ---<br>prompt_extract= ChatPromptTemplate.from_messages(<br>    [(&quot;system&quot;, &quot;Extract the technical specifications from the following text:\n\n{text_input}&quot;),<br>     (&quot;user&quot;, &quot;{text_input}&quot;)]<br>)<br><br># --- Prompt 2: Transform to JSON ---<br>prompt_transform= ChatPromptTemplate.from_messages(<br>    [(&quot;system&quot;, &quot;Transform the following specifications into a JSON object with &#39;cpu&#39;, &#39;memory&#39;, and &#39;storage&#39; as keys:\n\n{specifications}&quot;),<br>     (&quot;user&quot;, &quot;{specifications}&quot;)]<br>) <br><br># --- Build the Chain using LCEL ---<br># The StrOutputParser() converts the LLM&#39;s message output to a simple string.<br>extraction_chain= prompt_extract | llm | StrOutputParser()<br><br># The full chain passes the output of the extraction chain into the &#39;specifications&#39;<br># variable for the transformation prompt.<br>full_chain=(<br>    {&quot;specifications&quot;: extraction_chain}<br>    | prompt_transform<br>    | llm<br>    | StrOutputParser()<br>)<br><br><br># --- Run the Chain ---<br>input_text = &quot;The new laptop model features a 3.5 GHz octa-core processor, 16GB of RAM, and a 1TB NVMe SSD.&quot;<br><br>final_result= full_chain.invoke({&quot;text_input&quot;: input_text})<br><br>print(final_result)</pre><h3>When to Use Prompt Chaining</h3><p>Use it when:</p><ul><li>The task is <strong>too complex</strong> for one prompt.</li><li>Multiple <strong>stages of reasoning</strong> are required.</li><li>External data/tools must be integrated.</li><li>You’re building an <strong>agentic system</strong> that needs reliability and explain ability.</li></ul><h3>Key Takeaways</h3><ul><li>Prompt chaining = <strong>Pipeline Pattern</strong> → break tasks into steps.</li><li>Improves <strong>accuracy, reliability, and control</strong> over LLM workflows.</li><li>Works best with <strong>structured outputs</strong> (JSON, XML).</li><li>Forms the backbone of <strong>AI agents and multi-step reasoning</strong>.</li><li>Supported by frameworks like LangChain, LangGraph, and Crew AI.</li></ul><p>👉 TL;DR: If one prompt feels like a black box, build a <strong>chain of prompts</strong> — your AI will behave smarter, and you’ll stay in control.</p><h3>References</h3><ol><li>LangChain Documentation on LCEL: <a href="https://python.langchain.com/docs/concepts/lcel/">https://python.langchain.com/docs/concepts/lcel/</a></li><li>LangGraph Documentation: <a href="https://langchain-ai.github.io/langgraph/">https://langchain-ai.github.io/langgraph/</a></li><li>Prompt Engineering Guide — Chaining Prompts: <a href="https://www.promptingguide.ai/techniques/chaining">https://www.promptingguide.ai/techniques/chaining</a></li><li>OpenAI API Documentation (General Prompting Concepts): <a href="https://platform.openai.com/docs/guides/gpt/prompting">https://platform.openai.com/docs/guides/gpt/prompting</a></li><li>Crew AI Documentation (Tasks and Processes): <a href="https://docs.crewai.com/">https://docs.crewai.com/</a></li><li>Google AI for Developers (Prompting Guides): <a href="https://cloud.google.com/discover/what-is-prompt-engineering?hl=en">https://cloud.google.com/discover/what-is-prompt-engineering?hl=en</a></li><li>Vertex Prompt Optimizer <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer">https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3ec6bb4cc1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vercel Ship 2025: Vercel is now turned into the “AI Cloud”]]></title>
            <link>https://medium.com/@kgang6434/vercel-ship-2025-vercel-is-now-turned-into-the-ai-cloud-eeb716eb02df?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeb716eb02df</guid>
            <category><![CDATA[vercel]]></category>
            <category><![CDATA[nextjs]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Sun, 06 Jul 2025 13:28:26 GMT</pubDate>
            <atom:updated>2025-07-06T13:28:26.046Z</atom:updated>
            <content:encoded><![CDATA[<p>Lately I tuned into Vercel Ship 2025 — where they revealed a quiet but profound transformation: Vercel is no longer just the Frontend Cloud. It’s now the <strong>AI Cloud</strong> — a platform specifically built for AI-native apps and agent-driven workloads.</p><p><strong>Key points:</strong></p><ol><li><strong>AI Gateway: one endpoint to rule them all</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*0PuSsGqzIbXRtXKwl69R-g.avif" /></figure><p>Instead of wrestling with multiple API keys and vendor-specific integrations, Vercel’s AI Gateway provides a unified endpoint for ~100 AI models (OpenAI, Anthropic, xAI, and more) 35. This enables:</p><ul><li>Zero vendor lock-in: Swap models with one line of code (e.g., model: &#39;xai/grok-3&#39; → model: &#39;anthropic/claude-4&#39;) 3.</li><li>Observability: Track latency, errors, and costs per model in a single dashboard 5.</li><li>Failover &amp; load balancing: Automatically reroute requests if a provider faces downtime 7.<br><em>Why it matters</em>: Teams experiment freely while avoiding provider fragmentation.</li></ul><p>2.<strong> Fluid compute + Active CPU pricing = smarter billing</strong></p><p>Serverless has always billed for runtime, not actual CPU use. Vercel’s <strong>Fluid Compute</strong>, combined with <strong>Active CPU pricing</strong>, means you now pay strictly for the CPU time your code runs — idle time is nearly free. In some cases, teams report up to 85% cost savings. That’s a big deal for AI workloads with bursts of activity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFFQPG1Yud_sFyCtg5ab-Q.avif" /></figure><p>3.<strong> Vercel Sandbox: trust but verify</strong></p><p>AI-generated code can be unpredictable — or dangerous. With <strong>Vercel Sandbox</strong>, you can now run untrusted Node.js or Python code in secure, short-lived microVMs (up to 45 minutes).This feels like a thoughtful safety net for AI experimentation.</p><p>4.<strong> Rolling Releases: slow is the new fast</strong></p><p>“No more big-bang deploys.” Now teams can roll features out gradually, track performance, and halt an update if things go sideways — all with one click . It’s the engineering equivalent of “ready, aim, fire” instead of “fire and forget.”</p><p>5.<strong> Microfrontends: dividing and conquering</strong></p><p>For complex apps, Vercel now offers native support for <strong>microfrontends</strong>. Teams can ship parts of a UI independently, while Vercel handles routing and integration under the hood. It’s a helpful step toward parallel development without chaos.</p><p>6.<strong> BotID: no Captchas, no sweat</strong></p><p>Protecting login pages, checkouts, and AI‑heavy endpoints used to mean CAPTCHAs or performance trade‑offs. <strong>BotID</strong> is an invisible bot‑detector built with <a href="https://www.kasada.io/">Kasada</a> — no puzzles, no interruptions, just silent protection.</p><h3>Why this matters ?</h3><ul><li><strong>Flexibility</strong>: Swap AI models without rewriting code.</li><li><strong>Cost control</strong>: Pay for work, not waiting.</li><li><strong>Security</strong>: Run untrusted code safely.</li><li><strong>Confidence</strong>: Release features gradually, protect against bots.</li></ul><p>🚀I’m building <a href="https://www.prepzo.ai/">Prepzo.ai</a> — an AI-powered career coach leveraging Vercel’s Fluid Compute for real-time resume optimization and interview simulations.</p><ul><li>Try it free: Generate ATS-friendly resumes or LinkedIn profiles.</li><li>Goal: Make career prep as dynamic as the job market.<br>Your feedback will shape its evolution — <a href="https://www.prepzo.ai/">give it a spin</a>!</li></ul><p>References:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FlNmO7fDiyuE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DlNmO7fDiyuE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FlNmO7fDiyuE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffbb74ee5a328d4bf8d342f4a307c37d/href">https://medium.com/media/ffbb74ee5a328d4bf8d342f4a307c37d/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeb716eb02df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CI/CD with goCD]]></title>
            <link>https://medium.com/@kgang6434/ci-cd-with-gocd-27a04443a7d8?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/27a04443a7d8</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Sun, 22 Jun 2025 08:19:20 GMT</pubDate>
            <atom:updated>2025-06-22T08:24:20.042Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why CI/CD Pipelines Are Required</h4><p>In today’s fast-paced software development world, <strong>speed</strong>, <strong>quality</strong>, and <strong>reliability</strong> are essential. This is where <strong>CI/CD pipelines</strong> (Continuous Integration and Continuous Delivery/Deployment) come into play.<br>Without CI/CD:</p><ul><li>Developers manually compile code, run tests, build artifacts, and deploy to servers.</li><li>This process is <strong>error-prone</strong>, <strong>inconsistent</strong>, and <strong>time-consuming</strong>.</li><li>A simple typo or missed step can take production down.</li></ul><p><strong>CI/CD automates the entire workflow</strong>, reducing manual intervention and human error.</p><p>Real-Life Analogy</p><p>Think of CI/CD as a <strong>factory assembly line</strong>:</p><ul><li>Every part (code commit) goes through quality checks (tests),</li><li>Gets packaged the same way (builds),</li><li>And is delivered automatically (deployments).</li></ul><h4>What is goCD?</h4><p><a href="https://www.gocd.org/"><strong>GoCD</strong></a> is an <strong>open-source continuous delivery (CD) server</strong> developed by ThoughtWorks. It helps automate and streamline the process of building, testing, and deploying software, making it easier to release software frequently, reliably, and quickly.</p><p>What is GoCD used for?</p><ul><li>Modeling complex deployment pipelines</li><li>Automating builds and tests</li><li>Managing infrastructure as code</li><li>Version control integrations</li><li>Ensuring traceability and audit of releases</li></ul><p><strong>Key Features of GoCD</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*pXrPghewrwuxwvfnwuNcgw.png" /></figure><h4>How GoCD Works (High-Level Flow)</h4><ol><li><strong>Source Control Integration</strong><br> GoCD monitors repositories (e.g., Git) for code changes.</li><li><strong>Pipeline Execution</strong><br> Once a change is detected, GoCD triggers pipelines composed of stages (e.g., build → test → deploy).</li><li><strong>Stages &amp; Jobs</strong><br> Each stage contains jobs, which can run in parallel on agents.</li><li><strong>Agents</strong><br> GoCD uses agents (executors) to run jobs. They can be distributed across machines.</li><li><strong>Artifacts &amp; Logs</strong><br> Output of builds/tests are stored and made available for review and traceability.</li></ol><h4>GoCD Architecture</h4><ul><li><strong>GoCD Server</strong>: Manages and schedules pipelines.</li><li><strong>GoCD Agents</strong>: Run jobs assigned by the server.</li><li><strong>Artifacts Repository</strong>: Stores results and logs.</li><li><strong>Configuration Repo</strong>: Stores pipeline-as-code YAML/JSON.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnbpbKLxEmBHwBFYnQJLpg.png" /></figure><p>To <strong>create a CI/CD pipeline in GoCD</strong>, you follow these high-level steps:</p><ol><li><strong>Install GoCD</strong></li></ol><p><strong>Download</strong> and install:</p><ul><li>GoCD Server</li><li>[GoCD Agent](installed on the machine where builds run</li></ul><p>Use Docker if preferred:</p><pre>docker run -d -p 8153:8153 -p 8154:8154 gocd/gocd-server</pre><p>2. <strong>Connect to Source Code Repository</strong></p><ul><li>Go to <strong>Admin → Pipelines → Add new pipeline</strong>.</li><li>Choose <strong>“Material”</strong>:</li><li>Git</li><li>GitHub</li><li>Bitbucket</li></ul><p>Example:</p><pre>https://github.com/user/repo.git</pre><p>3<strong>. Define Your Pipeline</strong></p><p>You can define the pipeline either:</p><p>✅ Option A: <strong>Via Web UI</strong></p><ul><li>Define <strong>Stages</strong> (e.g., Build, Test, Deploy).</li><li>Each stage has <strong>Jobs</strong>.</li><li>Each job has <strong>Tasks</strong> (e.g., shell scripts, build commands).</li></ul><p>✅ Option B: <strong>Pipeline-as-Code (YAML/JSON)</strong></p><ul><li>Enable the <strong>Config Repository plugin</strong>.</li><li>Create a .gocd.yaml in your repo:</li></ul><pre>format_version: 10<br>pipelines:<br>  my-ci-pipeline:<br>    group: deploy<br>    materials:<br>      git_repo:<br>        git: https://github.com/user/my-app.git<br>    stages:<br>      - build:<br>          jobs:<br>            build-job:<br>              tasks:<br>                - exec: <br>                    command: npm<br>                    arguments: [&quot;install&quot;]<br>                - exec: <br>                    command: npm<br>                    arguments: [&quot;run&quot;, &quot;build&quot;]<br>      - test:<br>          jobs:<br>            test-job:<br>              tasks:<br>                - exec: <br>                    command: npm<br>                    arguments: [&quot;test&quot;]<br>      - deploy:<br>          approval:<br>            type: manual<br>          jobs:<br>            deploy-job:<br>              tasks:<br>                - exec: <br>                    command: ./deploy.sh</pre><p>4. <strong>Assign GoCD Agents</strong></p><ul><li>Agents must be <strong>enabled and assigned</strong> to jobs.</li><li>Tag agents for specific environments like build, test, prod.</li></ul><p>5.<strong> Triggering the Pipeline</strong></p><ul><li>Automatically triggers on:</li><li>Git commits (polling)</li><li>Manual approval stages</li><li>You can also <strong>trigger manually</strong> from the GoCD dashboard.</li></ul><p>6. <strong>Monitor Pipelines</strong></p><ul><li>View <strong>live status</strong>, logs, test results, and artifact history.</li><li>Use <strong>Value Stream Map</strong> to visualize flow from commit to deployment.</li></ul><p>You can test-drive it here: <a href="https://www.gocd.org/test-drive-gocd.html">https://www.gocd.org/test-drive-gocd.html</a><br>For more details : <a href="https://docs.gocd.org/current/">https://docs.gocd.org/current/</a></p><p>🚀 P.S. I’m currently building <a href="https://www.prepzo.ai/">Prepzo.ai</a>, an AI-powered interview agent and career coach! It’s an early-stage startup, so I’d love for you to check it out, experiment, and send me your feedback. Help shape the future of career prep — give Prepzo a spin and let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27a04443a7d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database Replication and Partitioning]]></title>
            <link>https://medium.com/@kgang6434/database-replication-and-partitioning-4710d5bb75d1?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/4710d5bb75d1</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[read-replica]]></category>
            <category><![CDATA[system-design-assignment]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 09:56:39 GMT</pubDate>
            <atom:updated>2025-06-09T09:59:16.796Z</atom:updated>
            <content:encoded><![CDATA[<p>In the epic saga of distributed systems, where data lives, loves, and occasionally loses its mind across multiple servers, we encounter the greatest plot twists since “The Sixth Sense.” This comprehensive guide explores the wild world of database replication and partitioning, where consistency is more elusive than finding a parking spot at the mall during Black Friday, and where your data sometimes behaves like that friend who tells different versions of the same story to different people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jh8xgI49k8FKTepTIcYjGA.avif" /></figure><h4>Types of replication</h4><ul><li>Single leader</li><li>Multi leader</li><li>Leaderless</li></ul><h3>Single‑Leader Replication (a.k.a. “One True Godfather”)</h3><ul><li>All write goes to one database (master)</li><li>Master database sends list of writes to other databases via replication log</li><li>Read can come from any database</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CkvmxJ5QeX9JmrwJ_aTBig.png" /></figure><p>Types of single leader replication</p><p><strong>Synchronous:</strong></p><ul><li>Client does not receive success message until all replicas complete the write.</li><li>Strong consistency.</li></ul><p><strong>Asynchronous:</strong></p><ul><li>Client receives success message the second that master completes the write.</li><li>Eventual consistency.</li></ul><p><strong>Dealing with eventual consistency:</strong></p><p>Let’s say you’re updating your Instagram bio to:</p><blockquote><em>“CEO of distributed chaos”</em></blockquote><p>You hit save ✅<br> But your friend in Canada still sees your old bio:</p><blockquote><em>“Loves chai &amp; cloud systems”</em></blockquote><p>After a few seconds… boom 💥 the update reflects. That’s eventual consistency.</p><p><strong>Reading your own writes:</strong></p><p>Imagine posting a selfie and immediately refreshing to admire your work, only to discover it’s not there yet. You start questioning reality like Jim Carrey in “The Truman Show.” The solution? Always read from the leader for editable areas, or read from an up-to-date replica for some period after writing</p><ul><li>do not see write after just making it.</li></ul><p><strong><em>Solutions:</em></strong></p><p><em>a) Always read from leader for editable areas of application.</em></p><p>b) Always read from the leader or up to date replica for some period after a client write.</p><p><strong>Monotonic reads</strong><em> </em>: This is when your timeline starts looking like “Memento” — everything appears to be going backward in time. Fix this by having each user read from the same replica, like assigning everyone their own personal Netflix account to avoid profile confusion.</p><ul><li>Reads look like they are going back in time</li></ul><p><strong><em>Solutions:</em></strong></p><p>a) Fix by having each user read from the same replica.</p><p><strong>Consistent prefix reads</strong></p><ul><li>Casual relationship between writes lost because preceding write takes longer to replicate (due to being on different partitions).</li><li>When causally related events get separated, it’s like watching “Pulp Fiction” for the first time — technically everything happens, but good luck understanding the sequence. The solution is putting related events on the same partition or tracking causal dependencies like a conspiracy theorist with red string.</li></ul><p><strong><em>Solutions:</em></strong></p><p>a) Put casually related events on same partition or if not possible keep track of casual dependencies.</p><h4>Replication Log Implementation: The Technical Trilogy</h4><p>There are three ways to implement replication logs, each with its own character arc:</p><ul><li>Copy over SQL statements</li><li>Use internal write ahead log</li><li>Use logical log.</li></ul><p><strong>Copy over SQL statements</strong></p><ul><li>Bad because SQL statements are nondeterministic(for example current time).</li><li>This is the amateur hour approach — like trying to recreate a <em>Gordon Ramsay</em> recipe by just copying the ingredients list. SQL statements are nondeterministic (current timestamps, random numbers), making this about as reliable as using Internet Explorer for anything important.</li></ul><p><strong>Use internal write-ahead log</strong></p><ul><li>At its core, a <strong>Write-Ahead Log</strong> is a file where <strong>all changes</strong> to the database are <strong>written before</strong> they are applied to the actual database.<br> This ensures durability — if the database crashes mid-write, it can replay the log and recover.</li><li>Imagine you’re giving a friend instructions to find your favorite pizza place 🍕:</li></ul><p>Internal WAL style:</p><p>“Drive 4.3 km north, then turn left at 12.7 km, then park 3.2 meters after the third tree.” That works <strong>only</strong> if: Your friend has the <strong>same car</strong> with the same odometer, And the <strong>same city layout</strong> as yours.</p><ul><li>Not scalable if changing database engine, says which bytes were changed, other database may have different data in different locations on disk.</li></ul><p><strong>Use logical log</strong></p><ul><li>The goldilocks solution that describes which rows were modified and how. This approach future-proofs your system like investing in a good streaming service — it works regardless of what device you’re using</li><li>Describes which rows were modified and how, allows for more future proofing if the underlying database engine changes in the future.</li></ul><p><strong>Single leader replication: The Good, The Bad, and The Bottleneck</strong></p><p><em>Pro:</em></p><ul><li>All writes go through one machine which ensure consistency across replicas.</li></ul><p><em>Cons:</em></p><ul><li>Since all writes have to go through one machine write throughput may not be acceptable — Either need to partition the dataset and have a single leader per partition, or look towards other replication strategies.</li></ul><h3>Multileader Replication: When Democracy Gets Messy</h3><p>Multileader topologies</p><ul><li>Circular</li><li>Star Topology</li><li>All-to-all Topology</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*0PNPsGykhX-M6VgawSIXQg.png" /><figcaption>Topology</figcaption></figure><p>a) Circle and star topologies can easily fail (any node crach for circle, central node crash for star).</p><p>b) All to all topology can have writes delivered out of order due to race condition.</p><p>Multileader Replication</p><p>Pros:</p><ul><li>Can have a leader in each data center, makes a global service more manageable (outages between datacenters do not break application)</li><li>Are not limited to the write throughput of a single node.</li></ul><p>Cons:</p><p>Having to deal with the write conflicts between multiple leaders.</p><h4>Conflict resolutions</h4><ul><li>Conflict Avoidance</li><li>Last write wins</li><li>on Read</li><li>on Write</li></ul><p>Conflict avoidance</p><ul><li>The easiest way to deal with the conflict is to avoid them.</li><li>The easiest solution — route all writes for the same item to the same leader. It’s like assigning bathroom schedules in a house full of roommates. Simple, but not always practical when leaders go down or configurations change.</li><li>Have all writes to the same item go to a given leader.</li><li>Not always possible if leader is down or you want to change your database configuration.</li></ul><p>Last write wins</p><p>Each write is assigned a timestamp , write with lates timestamp is kept.</p><p>Pros:</p><ul><li>Very easy to implement</li></ul><p>Cons:</p><ul><li>Do we use the client or the server timestamp?</li><li>Write will be lost</li><li>Clock skew on servers means that their time are not exactly synchronized.</li></ul><p>On write</p><ul><li>When database detects concurrent writes, call snippet of code to merge them somehow(often application specific)</li><li>This is the idea between conflict free replicated data types.</li><li><em>Conflict-Free Replicated Data Types (CRDTs)</em>:These are the superheroes of distributed systems, designed to merge concurrent updates without conflicts. Think of them as the relationship counselors of the data world — they help different versions of data get along</li></ul><p>Detecting the concurrent writes</p><p>Two writes are concurrent if each client did not know about the write of the other when they made the write, has nothing to do with actual time of write. If one client knew about the write of the other we would have a causality relationship between the two.</p><p>Hence, detecting the concurrent writes is all about keeping track of what a client has seen from the database before making a write.</p><h3>Leaderless Replication: Anarchy in the Database</h3><ul><li>Any Replica can accept a write from a client.</li><li>Send reads and writes to all nodes in parallel, once a certain predefined threshold of nodes return a success value, the client is told that the read/write was successful.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*kPgOQccx4FxM7KWojGdpfg.png" /></figure><p>Keeping data up to date</p><p>Anti Entropy:</p><p>Background process looks at multiple nodes and their stored values use version numbers of the data in order to attempt to make sure that each replica holds the most up to date copy of the data.</p><p>Read Repair</p><p>Idea: Since we are reading from multiple replicas in parallel , take most up to date piece of data and propagate it to the other replicas that had outdated data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*HfVgTBTSpvzQrbWQ7jNRQg.png" /></figure><p>Value if key gojo?</p><p>argmax[(20,sexy),(18,cute),(19,handsome)]</p><p>result: gojo=sexy</p><p>Need to write back to DB2 and DB3 -&gt;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*x8SaWHA2k9tuK196rnSQag.png" /><figcaption>Updated replicas</figcaption></figure><p>Problem: How can we guarantee that at least one of the replicas that we read from has most up to date data?</p><h4>Quorums: The Voting System</h4><p>Let’s imagine we have n database replicas</p><ul><li>Suppose a client only has a successful write if at least w nodes are written to</li><li>Suppose a client only has a successful read if at least r nodes are read from</li><li>if w + r &gt; n, at least one of the nodes that is read from will have an updated copy of the data and thus read repair the others.</li></ul><p>Typically set n to be an odd number and w = r = (n+1)/2, can tolerate up to (n-1)/2 node failures in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3uJKkEIv5hwtJskCqhWtQ.png" /></figure><p>Quorums are not perfect</p><ul><li>But quorums aren’t perfect — they’re more like the electoral college of distributed systems. Even when fewer than w writes succeed, the nodes that did process the write don&#39;t undo it, creating a &quot;partial success&quot; state that&#39;s about as satisfying as getting halfway through a Netflix series before it gets cancelled</li><li>If we to write to w nodes to make a quorum, but fewer that w writes succeed, the client is told that their write failed, but nodes that did successfully process the write do not undo it.</li><li>If you restore a failed node with an up to date value from one key using a node with an older value for that key, it will loose the up to date value.</li><li>Can still be write conflicts.</li><li>Sloppy quorums.: Sometimes your designated nodes are down, so you temporarily store data elsewhere. It’s like storing your important documents at a friend’s house when your place is being fumigated — technically safe, but you’ll need to move them back later through a process called “hinted handoff.”</li></ul><p>Write Conflict Example</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*_lwUa5yULeMqwfK6E9y1kw.png" /></figure><p>Sloppy Quorums</p><ul><li>Let’s say I am a Facebook SWE, and all the profile data is stored on 99 database nodes, W=50, R=50</li><li>However all of the 99 profile nodes have crashed, so I instead redirect all writes for the data to 50 of the nodes which hold Facebook messages</li><li>Even if the profile nodes come back up , I can read 50 of them and not get up to date data.</li><li>Need to perform a hinted handoff, and transfer the data back to the right place.</li></ul><p>Leaderless Replication</p><p>Pros:</p><ul><li>Can work well in practice with a cross datacenter solution by having quorum write parameter be small enough that all writes can go to one datacenter.</li></ul><p>Cons:</p><ul><li>Slower reads as a result of multiple queries to nodes (if using read repair)</li><li>Still have to reason about write conflicts.</li><li>Quorums are not perfect, provide illusion of strong consistency when in reality this is often not true.</li></ul><h3>Partitioning: Divide and Conquer (Your Sanity)</h3><p>In large systems we are dealing with tons of data and as a result tables may become too big/perform too many queries for a one single machine.</p><ul><li>Partitioning is splitting this table up into many chunks to go on various database nodes.</li><li>Partitioning is often used in conjunction with replication.</li></ul><p>Objectives of partitioning</p><ul><li>Each node we want a relatively similar amount of data.</li><li>A relatively similar amount of reads/and writes to the data.</li><li>The goal is achieving balance — each node should have roughly the same amount of data and handle similar read/write loads. Fail at this, and you’ll create “hot spots” where certain nodes are overwhelmed like the only working self-checkout machine at a grocery store</li></ul><p><strong>Methodologies for partitioning</strong></p><ul><li>By ranges of keys</li><li>By ranges of the hash Keys</li></ul><h4>Key Range Partitioning:: The Alphabetical Approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*zim1NaKST9UGw7Vj5TFYzQ.png" /></figure><ul><li>Not necessarily even ranges, of keys, some ranges may be hotter.</li><li>Keep keys sorted within a partition to best support range queries.</li></ul><p>Pros:</p><ul><li>Simple and allows for effective range queries</li></ul><p>Cons:</p><ul><li>Have to actually determine the ranges to make sure they are relatively even in data and load. (can be done manually or by database.)</li><li>Can easily lead to hotspots (if for example partitioning by range of time stamps).</li></ul><h4>Hash range Partitioning: The Lottery System</h4><p>Hash partitioning takes the hash of each key and distributes it randomly, like drawing names from a hat. This ensures even distribution but kills range queries — you’ll need to check every partition, turning a targeted search into a game of “Where’s Waldo?” across your entire database</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*bnxoclZdDvIGWiAQ6cKq7A.png" /></figure><p>Pros:</p><ul><li>Keys are evenly distributed between nodes.</li></ul><p>Cons:</p><ul><li>No more range queries on the partition key, have to check every partition.</li><li>If a key has a lot of activity will still lead to hot spots.</li></ul><h4>TL;DR: The Database Replication &amp; Partitioning</h4><p>In this season of <em>As The Data Turns</em>, we’ve learned:</p><p>🔹 Single-Leader: Like <em>The Godfather</em> but with more SQL. One boss handles all writes while the replicas gossip like Kardashians. Sync = perfectionists, Async = chaotic optimists.</p><p>🔹 Multi-Leader: <em>Avengers: Infinity War</em> meets <em>The Office</em>. Multiple heroes (leaders) fight write conflicts with the ferocity of Thor vs. Thanos. Choose your topology wisely: circular (telephone game), star (centralized drama), or all-to-all (group chat chaos).</p><p>🔹 Leaderless: <em>Hunger Games</em> meets <em>Democracy</em>. Everyone votes (quorums!), but sometimes you get sloppy results like a pizza with pineapple. Read repair is the duct tape of consistency.</p><p>🔹 Partitioning: <em>Harry Potter</em> sorting hat for data. Key-range = library shelves (hotspot risk!), hash = lottery tickets (fair but chaotic). Hotspots? More annoying than a <em>Curb Your Enthusiasm</em> plot twist.</p><p>Moral of the Story: Distributed systems are like relationships — communication is key, conflicts are inevitable, and sometimes you just need a CRDT therapist. Choose your replication strategy like you’d choose a Netflix show: based on mood, tolerance for drama, and snack availability.<br>Wait for season 2!</p><p>🚀 P.S. I’m currently building <a href="https://www.prepzo.ai/">Prepzo.ai</a>, an AI-powered interview agent and career coach! It’s an early-stage startup, so I’d love for you to check it out, experiment, and send me your feedback. Help shape the future of career prep — give Prepzo a spin and let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4710d5bb75d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hyperthreading worth it?]]></title>
            <link>https://medium.com/@kgang6434/hyperthreading-worth-it-11271f255176?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/11271f255176</guid>
            <category><![CDATA[hyperthreading]]></category>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[multithreading]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 06:56:50 GMT</pubDate>
            <atom:updated>2025-04-28T06:56:50.898Z</atom:updated>
            <content:encoded><![CDATA[<p>Yesterday I came across this linkedIn post that rings some bell in mind<br>so I dig deep into the concept and came up with this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*mUu-1EuRKYm8yt-OPl0aXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*RVEzBgvNkce3tVH3z397YA.png" /></figure><p><a href="https://www.linkedin.com/posts/sahilsarwar1_systemdesign-concurrency-multithreading-activity-7321571227251208193-UiJo?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAADY24vwBSMlmhakeUyo8BiYLvYYXgSqtg8w">Link to post</a></p><p>First of all lets begin with what is <strong>Hyperthreading</strong></p><p><strong>Hyper-threading</strong> (officially called <strong>Hyper-Threading Technology</strong> or <strong>HT Technology</strong> and abbreviated as <strong>HTT</strong> or <strong>HT</strong>) is Intel’s proprietary simultaneous multithreading (SMT) implementation used to improve parallelization of computations (doing multiple tasks at once) performed on <a href="https://en.wikipedia.org/wiki/X86">x86</a> microprocessors.</p><p>For each processor core that is physically present, the operating system addresses two virtual (logical) cores and shares the workload between them when possible. The main function of hyper-threading is to increase the number of independent instructions in the pipeline; it takes advantage of <a href="https://en.wikipedia.org/wiki/Superscalar_processor">superscalar</a> architecture, in which multiple instructions operate on separate data in parallel. With HTT, one physical core appears as two processors to the operating system, allowing <a href="https://en.wikipedia.org/wiki/Concurrent_computing">concurrent</a> scheduling of two processes per core. In addition, two or more processes can use the same resources: If resources for one process are not available, then another process can continue if its resources are available.</p><p>In another words Hyper-Threading is where two threads are able to run on one single-threaded core. When a thread on the core in question is stalling or in a halt state, hyper-threading enables the core to work on a second thread instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*KMZ2wjBgB9sYoal-pPiTaQ.gif" /></figure><p>Hyper-threading makes the OS think that the processor has double the number of cores, and often yields a performance improvement.</p><p>Now you have the context of what hyperthreading then you will be having these questions :</p><ol><li>What’s the difference between single-threaded and multithreaded applications?</li><li>What is Hyper-Threading, and how is it different from normal multithreading?</li></ol><p>Let’s answer these questions</p><p><strong>Single🧵 vs Multi 🧵🧵</strong></p><p>In a single-threaded programming language, the execution of code occurs sequentially, one instruction after another, within a single thread of execution. This means that tasks are processed in order, and only one task is executed at a time. On the other hand, multi-threaded programming languages allow the concurrent execution of multiple threads, where each thread represents a separate flow of control within the same process. Following image will give you some clarity on that.</p><figure><img alt="https://www.geeksforgeeks.org/thread-in-operating-system/" src="https://cdn-images-1.medium.com/max/768/1*BBrlBnWl0xWSOnS9IZHxlA.png" /></figure><p>Multi vs Hyper</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bi4fQtYPDg1LMnDT5u8EYQ.png" /></figure><p>So now we have answer to those questions, we can arguee on this question Hyperthreading worth it?</p><p>According to Intel [1], hyper-threading your cores can result in a 30% increase in performance and speed when comparing two identical PCs, with one CPU hyper-threaded. In a study published on Forbes, hyper-threading an AMD® processor (Ryzen 5 1600) showed a 17% increase in overall processing performance [2].</p><p>Despite these results, hyper-threading your cores isn’t always the go-to solution. There will be tasks in which the speed of your processor does not increase despite hyper-threading. This is due in part to the fact that not all applications and strings of data can efficiently load into a multi-thread core.</p><p>In an experiment carried out by bit-tech.net, a hyper-threaded Intel i7 Core was compared to a single thread Intel i7 Core after being put through a few different tests [3]. When it came to image editing, multitasking, and power consumption, the hyper-threaded counterpart did worse than the single thread. However, it performed the same or better when it came to Handbrake Video Encoding, the Overall Custom PC Benchmark Score, and playing the popular game <em>Crysis</em>.</p><p>Following benchmarks is taken from here — <a href="https://www.bit-tech.net/reviews/tech/cpus/intel-core-i5-and-i7-lynnfield-cpu-review/10/">link</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*8OAJeOnq6qgolIRtN2CoKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*Fp-fde-TYvDBn7I0enRqNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*J2pzQ6-0EKNM26yCJYXLxw.png" /></figure><p><strong>Hyper-threading the cores in your CPU improves performance and speed on a case by case basis</strong> depending on which tasks are compatible with a hyper-threaded core. If you’re looking to enhance your PC’s overall processing prowess, hyper-threading could be a step in the right direction.</p><p>So, whether you’re crunching thousands of video frames, rendering lifelike rays of light, or simply juggling a dozen browser tabs, hyper-threading is that secret handshake between threads and cores that can tip the scales in your favor. Flip it on, let your processor play traffic-controller, and watch as your workloads zip along — just don’t be surprised if your Photoshop brushes or game engine hiccup by a hair now and then. Ultimately, hyper-threading isn’t a silver bullet, but it’s one heck of a turbo-boost when you need extra grunt. Give it a spin in your BIOS, benchmark your favorite apps, and decide if those virtual cores deserve a permanent spot in your overclocked lineup — your next performance victory could be just a thread away.</p><p>References</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfBtqMuqKGCI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfBtqMuqKGCI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfBtqMuqKGCI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/782990352e5e2bf01c1622ac1177aaf5/href">https://medium.com/media/782990352e5e2bf01c1622ac1177aaf5/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=reddit&amp;url=https%3A//www.reddit.com/r/intel/comments/17fok9j/how_does_hyper_threading_work_per_core_hyper/&amp;image=" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9ddfabd5dfe3059ee2b72969360e37d/href">https://medium.com/media/c9ddfabd5dfe3059ee2b72969360e37d/href</a></iframe><ul><li><a href="https://www.geeksforgeeks.org/what-is-hyper-threading/">https://www.geeksforgeeks.org/what-is-hyper-threading/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11271f255176" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Email Summaries with n8n, OpenAI, and Supabase]]></title>
            <link>https://medium.com/@kgang6434/automating-email-summaries-with-n8n-openai-and-supabase-6701060992bf?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/6701060992bf</guid>
            <category><![CDATA[n8n]]></category>
            <category><![CDATA[n8n-workflow]]></category>
            <category><![CDATA[n8n-workflow-automation]]></category>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Wed, 19 Mar 2025 16:12:24 GMT</pubDate>
            <atom:updated>2025-03-19T16:12:24.289Z</atom:updated>
            <content:encoded><![CDATA[<p>When I first discovered <strong>n8n</strong>, I was amazed by its versatility in automating workflows. I wanted to create a process that takes data from my Supabase database, uses OpenAI to generate summaries, and then emails those summaries to users — all without writing a ton of boilerplate code. Below is a quick look at how I built and refined this workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6bxSIXrwMnCfJwSGoDW48A.png" /></figure><h3>Why n8n?</h3><p>n8n is a low-code platform that makes it simple to integrate different services with drag-and-drop nodes. Instead of writing a custom script for each step — fetching data, generating text with AI, and sending emails — I used n8n’s built-in nodes and a bit of JavaScript for data prep.</p><h3>The Building Blocks</h3><p><strong>Supabase</strong><br> I store my conversation data in a table called conversation_histories. Each record has a user_email and a conversation field (among others). The table also has a Boolean email_sent flag so I know whether I’ve already sent a summary.</p><p><strong>OpenAI</strong><br> I rely on OpenAI’s GPT model to create concise summaries of the conversation. With n8n’s OpenAI node, it’s easy to pass in text prompts and receive the AI’s output.</p><p><strong>Gmail</strong> (or another Email Service)<br> Finally, I send an email containing the AI-generated summary to the user’s email address. n8n supports multiple email integrations, so you can pick your preferred provider.</p><p><strong>n8n</strong><br> The glue that holds everything together! I created a simple workflow that runs on a schedule, fetches new records from Supabase, calls OpenAI for a summary, and sends an email.</p><h3>Step-by-Step Workflow</h3><p><strong>Schedule Trigger</strong><br> I start with a <strong>Schedule Trigger</strong> node to run the workflow periodically (e.g., every hour). This ensures new records in Supabase are automatically processed without any manual effort.</p><p><strong>Supabase (Get All)</strong><br> Next, I added a <strong>Supabase</strong> node. I configured it to:</p><ul><li><strong>Operation:</strong> “Get All” (or “Select”)</li><li><strong>Filters:</strong></li><li>user_email is not null</li><li>email_sent = false<br> This returns an array of rows that haven’t been processed yet.</li></ul><p><strong>Code Node</strong><br> The conversation data is stored as JSON. In a small <strong>Code</strong> node, I loop through the conversation messages to build a more natural prompt for OpenAI. Here’s an example snippet:</p><pre>const conversation = $json[&quot;conversation&quot;] || [];<br>let conversationText = &quot;&quot;;<br><br>conversation.forEach((msg, i) =&gt; {<br>  conversationText += `${i + 1}. [${msg.role.toUpperCase()}]: ${msg.content}\n`;<br>});<br><br>// Return the prompt, plus any other needed fields<br>return {<br>  prompt: `Summarize this conversation:\n\n${conversationText}`,<br>  user_email: $json[&quot;user_email&quot;],<br>  session_id: $json[&quot;session_id&quot;]<br>};</pre><p><strong>OpenAI (Message Model)</strong><br> The output from the Code node goes to an <strong>OpenAI</strong> node, which takes the prompt and uses a model like gpt-3.5-turbo to generate a summary. I set the temperature to something moderate (e.g., 0.7) and limit the tokens so I get a concise response. The node’s output includes the model’s text.</p><p><strong>Gmail</strong><br> With the summary in hand, I use the <strong>Gmail</strong> node to send an email. I reference {{$json[&quot;user_email&quot;]}} for the recipient and embed the AI summary in the message body. A simple example:</p><pre>&lt;p&gt;Hi,&lt;/p&gt;<br>&lt;p&gt;Here’s your conversation summary:&lt;/p&gt;<br>&lt;pre&gt;{{ $json[&quot;choices&quot;][0][&quot;text&quot;] }}&lt;/pre&gt;<br>&lt;p&gt;Thanks for using our service!&lt;/p&gt;</pre><p><strong>(Optional) Supabase Update</strong><br> If you want to mark each record as processed, add another Supabase node to update the row (email_sent = true). That way, the same record won’t be fetched again in future runs.</p><h3>What I Learned</h3><ul><li><strong>Iterative Approach:</strong> I started small by just fetching data from Supabase and printing it in n8n. Then I added one piece at a time — like the AI node and the email node.</li><li><strong>Error Handling:</strong> Debugging is straightforward. Each node has an “Execute Node” button to test things in isolation. If something fails, I can fix that step without rebuilding the entire pipeline.</li><li><strong>Flexibility:</strong> It’s easy to swap out services. For instance, I could replace Gmail with another email provider or even send Slack notifications. n8n’s modular design keeps everything neatly separated.</li></ul><h3>Conclusion</h3><p>By connecting Supabase, OpenAI, and Gmail in n8n, I automated the entire process of summarizing user conversations and emailing them. This workflow runs on a schedule, finds new conversations that need summaries, generates the text via AI, and delivers it right to the user’s inbox. It’s a great example of how n8n can orchestrate multiple services with minimal code, making automation accessible for just about any project.</p><p>If you’re looking to automate your own workflows — whether it’s sending AI-generated summaries or syncing data between different platforms — n8n is definitely worth exploring. Happy automating!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6701060992bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event Loop: It’s Not About Async, It’s About Control]]></title>
            <link>https://medium.com/@kgang6434/event-loop-its-not-about-async-it-s-about-control-3a81b5198efa?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a81b5198efa</guid>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 17:07:38 GMT</pubDate>
            <atom:updated>2025-02-12T17:07:38.466Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s cut through the noise: JavaScript’s event loop isn’t <em>just</em> about handling asynchronous operations. It’s about <strong>control</strong> — managing what gets executed, when, and in what order. Here’s a practical breakdown of how it works, why timers and I/O behave the way they do, and what happens under the hood.</p><p>JavaScript runs on a single thread. If you block it with a long-running task (like a while(true) loop), your entire program freezes. But modern apps need to handle I/O (files, networks), timers, and UI interactions <em>without</em> blocking. That’s where <strong>libuv</strong> (a C++ library powering Node.js and browsers) steps in. It handles asynchronous operations outside the main thread and feeds results back into JavaScript’s execution flow.</p><h3>The Event Loop: A Traffic Controller</h3><p>The event loop is a loop that checks two things repeatedly:</p><ol><li><strong>Is the call stack empty?</strong></li><li><strong>Are there tasks in the queues?</strong></li></ol><p>If both are true, it moves tasks from queues to the call stack. But not all queues are equal. Here’s the <strong>order of execution</strong>:</p><ol><li><strong>Microtask Queue</strong> (Processed <em>fully</em> after each phase): process.nextTick() callbacks (highest priority). Promise resolutions (.then(),async/await).</li><li><strong>Timer Queue</strong> (setTimeout, setInterval callbacks).</li><li><strong>Microtask Queue</strong> again.</li><li><strong>I/O Queue</strong> (Completed I/O operations like file reads).</li><li><strong>Microtask Queue</strong> again.</li><li><strong>Check Queue</strong> (setImmediate callbacks).</li><li><strong>Microtask Queue</strong> again.</li><li><strong>Close Queue</strong> (Cleanup callbacks like socket.on(&#39;close&#39;)).</li><li><strong>Microtask Queue</strong> one last time.</li></ol><p>Example:</p><pre>setTimeout(() =&gt; console.log(&#39;Timer 1&#39;), 0); <br>setImmediate(() =&gt; console.log(&#39;Check Queue&#39;));<br>Promise.resolve().then(() =&gt; console.log(&#39;Promise 1&#39;)); <br>process.nextTick(() =&gt; console.log(&#39;nextTick 1&#39;));<br>fs.readFile(__filename, () =&gt; { <br> console.log(&#39;I/O Complete&#39;); <br> process.nextTick(() =&gt; console.log(&#39;nextTick in I/O&#39;)); <br>});</pre><p>Output:</p><pre>nextTick 1  <br>Promise 1  <br>Timer 1  <br>I/O Complete  <br>nextTick in I/O  <br>Check Queue  </pre><p><strong>Why?</strong></p><ul><li>Microtasks (nextTick, promises) run <em>before</em> timers, even if timers are due.</li><li>The I/O phase runs after timers.</li><li>setImmediate runs <em>after</em> I/O.</li></ul><h3>Key Rules to Remember</h3><ul><li><strong>No Interrupts</strong>: Callbacks run only when the call stack is empty. Even a timer due at 0ms waits.</li></ul><pre>setTimeout(() =&gt; console.log(&#39;Hi&#39;), 0);  <br>for (let i=0; i&lt;1e6; i++) {} // Blocks the thread. &quot;Hi&quot; logs after this.  </pre><ul><li><strong>Timer vs. I/O Priority</strong>: If a timer and I/O callback are both ready, the timer runs first.</li><li><strong>Microtasks Always Win</strong>: They’re processed after <em>every</em> queue.</li></ul><h3>Behind the Scenes: libuv’s Min-Heap for Timers</h3><p>Timers (like setTimeout) aren’t managed naively. Storing thousands of timers would be inefficient. Instead, libuv uses a <strong>min-heap</strong>—a tree where the parent node is always the smallest (earliest expiring timer). This allows O(1) access to the next due timer.</p><p><strong>But there’s a twist</strong>: If two timers expire at the same time, they’re batched into a single libuv timer.</p><pre>setTimeout(() =&gt; {}, 50);  <br>setTimeout(() =&gt; {}, 50);  <br>// These are combined into one libuv timer.  </pre><p>This reduces garbage collection overhead and improves performance.</p><h3>Why setImmediate is Misnamed</h3><p>setImmediate doesn’t run “immediately.” It’s a callback that runs in the <strong>check queue</strong>, which comes <em>after</em> I/O. In practice:</p><pre>fs.readFile(__filename, () =&gt; {  <br>  setTimeout(() =&gt; console.log(&#39;Timer&#39;), 0);  <br>  setImmediate(() =&gt; console.log(&#39;setImmediate&#39;));  <br>});  <br>// Output: setImmediate → Timer  </pre><p>Inside an I/O phase, setImmediate always runs before timers due to queue order.</p><h3>The I/O loop</h3><p>The I/O (or event) loop is the central part of libuv. It establishes the content for all I/O operations, and it’s meant to be tied to a single thread. One can run multiple event loops as long as each runs in a different thread. The libuv event loop (or any other API involving the loop or handles, for that matter) <strong>is not thread-safe</strong> except where stated otherwise.</p><p>The event loop follows the rather usual single threaded asynchronous I/O approach: all (network) I/O is performed on non-blocking sockets which are polled using the best mechanism available on the given platform: epoll on Linux, kqueue on OSX and other BSDs, event ports on SunOS and IOCP on Windows. As part of a loop iteration the loop will block waiting for I/O activity on sockets which have been added to the poller and callbacks will be fired indicating socket conditions (readable, writable hangup) so handles can read, write or perform the desired I/O operation.</p><p>In order to better understand how the event loop operates, the following diagram illustrates all stages of a loop iteration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*cujiw4TK3KhnfNgGgdzwug.png" /><figcaption>Flowchart</figcaption></figure><blockquote>libuv uses a thread pool to make asynchronous file I/O operations possible, but network I/O is <strong>always</strong> performed in a single thread, each loop’s thread.</blockquote><h3>The Big Picture</h3><ul><li><strong>Microtasks</strong> handle high-priority cleanup (promises, mutations).</li><li><strong>Timers</strong> execute scheduled code but aren’t real-time guarantees.</li><li><strong>I/O</strong> is for non-CPU tasks (files, networks).</li><li><strong>Close handlers</strong> clean up resources.</li></ul><p>The event loop isn’t about doing things in parallel — it’s about juggling tasks efficiently so your app <em>feels</em> fast.</p><figure><img alt="https://tenor.com/view/droopy-90s-looney-toons-juggling-gif-14042068" src="https://cdn-images-1.medium.com/max/498/1*Po80cG1JVS_o7VL89DIfiQ.gif" /><figcaption><a href="https://tenor.com/view/droopy-90s-looney-toons-juggling-gif-14042068">https://tenor.com/view/droopy-90s-looney-toons-juggling-gif-14042068</a></figcaption></figure><h3>When Things Go Wrong</h3><p>Blocking the event loop (e.g., synchronously processing large data) starves all other phases. Use <strong>worker threads</strong> (Node.js) or <strong>Web Workers</strong> (browsers) for CPU-heavy tasks.</p><h3>Final Thought</h3><p>Think of the event loop as a restaurant chef. They can’t cook all orders at once, but they prioritize tasks (take orders, start cooking, check the oven, plate dishes) in a way that keeps the kitchen moving. The key is to never let one task monopolize their attention.</p><p><strong>Let’s Connect!</strong><br>I’m always open to collaborations and discussions. Feel free to reach out : <a href="https://www.kushagragangwar.tech/blog"><strong>kushgragangwar.tech/blog/</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a81b5198efa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comprehensive Guide to Optimizing <img> for Modern Websites]]></title>
            <link>https://medium.com/@kgang6434/comprehensive-guide-to-optimizing-img-for-modern-websites-ab06e0c03f99?source=rss-2d77b1d65065------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab06e0c03f99</guid>
            <dc:creator><![CDATA[Satoru_gojo]]></dc:creator>
            <pubDate>Thu, 23 Jan 2025 16:17:07 GMT</pubDate>
            <atom:updated>2025-01-23T16:17:07.798Z</atom:updated>
            <content:encoded><![CDATA[<p>Images play a crucial role in the user experience of a website. However, poorly optimized images can significantly slow down page loading times, impacting performance, SEO, and user satisfaction. This guide explores all possible techniques for optimizing &lt;img&gt; elements in websites, complete with code examples and flowcharts. NextJS &lt;Image&gt; component by default provide some of the following optimizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*40_xQUr1Yfn6tOWfmjLQsw.jpeg" /></figure><h3>1. Use Modern Image Formats</h3><p>Modern formats like <strong>WebP</strong>, <strong>AVIF</strong>, and <strong>JPEG XL</strong> provide superior compression and quality compared to older formats like JPEG and PNG.</p><h3>File Size Savings:</h3><ul><li><strong>WebP</strong>: Up to 30% smaller than JPEG for similar quality.</li><li><strong>AVIF</strong>: Up to 50% smaller than JPEG and WebP in some cases.</li><li><strong>JPEG XL</strong>: Combines excellent compression with backward compatibility for older JPEGs.</li></ul><h3>Browser Compatibility:</h3><ul><li><strong>WebP</strong>: Supported by most modern browsers.</li><li><strong>AVIF</strong>: Supported by Chrome, Firefox, and Edge (partial Safari support).</li><li><strong>JPEG XL</strong>: Emerging support in newer browsers; check <a href="https://caniuse.com/">Can I Use</a>.</li></ul><h4>Example:</h4><pre>&lt;picture&gt;<br>  &lt;source srcset=&quot;image.avif&quot; type=&quot;image/avif&quot;&gt;<br>  &lt;source srcset=&quot;image.webp&quot; type=&quot;image/webp&quot;&gt;<br>  &lt;source srcset=&quot;image.jxl&quot; type=&quot;image/jxl&quot;&gt;<br>  &lt;img src=&quot;image.jpg&quot; alt=&quot;A sample image&quot; loading=&quot;lazy&quot;&gt;<br>&lt;/picture&gt;</pre><p>This fallback mechanism ensures that browsers supporting newer formats will load them, while older browsers fall back to JPEG.</p><h3>2. Image Format Selection Strategy</h3><p>Choose the format based on your use case:</p><ul><li><strong>WebP</strong>: General-purpose format with widespread support.</li><li><strong>AVIF</strong>: Best for high compression and advanced use cases.</li><li><strong>JPEG XL</strong>: Excellent for large libraries and backward compatibility.</li><li><strong>SVG</strong>: Use for icons and vector graphics.</li></ul><h3>3. Enable Lazy Loading</h3><p>Lazy loading defers image loading until they appear in the viewport, reducing initial page load time.</p><h4>Example:</h4><pre>&lt;img src=&quot;image.jpg&quot; alt=&quot;Lazy loaded image&quot; loading=&quot;lazy&quot;&gt;</pre><h3>Browser Support:</h3><p>All modern browsers now support the loading=&quot;lazy&quot; attribute, making it a simple and effective optimization.</p><h3>4. Use Responsive Images</h3><p>Serve different image sizes based on the user’s screen size using srcset and sizes attributes.</p><h4>Example:</h4><pre>&lt;img <br>  src=&quot;small.jpg&quot; <br>  srcset=&quot;<br>    small.jpg 480w,<br>    medium.jpg 1024w,<br>    large.jpg 1600w<br>  &quot; <br>  sizes=&quot;(max-width: 600px) 480px, <br>         (max-width: 1200px) 1024px, <br>         1600px&quot; <br>  alt=&quot;Responsive image&quot;&gt;</pre><p>This ensures the most appropriate image is loaded, minimizing unnecessary data transfer.</p><h3>5. Compress Images</h3><p>Compressing images reduces file size without noticeable loss in quality. Tools like <a href="https://imageoptim.com/api/start"><strong>ImageOptim</strong></a><strong> </strong>or <strong>Squoosh</strong> can help. For dynamic compression, use services like <a href="https://cloudinary.com/">Cloudinary</a> or <a href="https://www.imgix.com/">Imgix</a>.</p><h4>Example:</h4><pre>squoosh-cli --resize &#39;{&quot;width&quot;:800}&#39; --output-dir=optimized_images original.jpg</pre><h3>6. Use a Content Delivery Network (CDN)</h3><p>A CDN like <strong>Cloudflare</strong>, <strong>Gcore</strong>, or <strong>AWS CloudFront</strong> serves images from edge servers closest to the user.</p><h4>Example with Cloudinary:</h4><pre>&lt;img src=&quot;https://res.cloudinary.com/demo/image/upload/w_800/image.jpg&quot; alt=&quot;CDN image&quot;&gt;</pre><h3>7. Preload Critical Images</h3><p>Use &lt;link rel=&quot;preload&quot;&gt; to prioritize loading important images.</p><h4>Example:</h4><pre>&lt;head&gt;<br>  &lt;link rel=&quot;preload&quot; as=&quot;image&quot; href=&quot;hero.jpg&quot;&gt;<br>&lt;/head&gt;<br>&lt;img src=&quot;hero.jpg&quot; alt=&quot;Hero image&quot;&gt;</pre><p>This technique is especially effective for above-the-fold content.</p><h3>8. Add Placeholder Images</h3><p>Use blurred or low-quality placeholders for a better user experience while images load.</p><h4>Example:</h4><pre>&lt;style&gt;<br>  .placeholder {<br>    background: url(https://medium.com/&#39;placeholder.jpg&#39;) center / cover;<br>    width: 800px;<br>    height: 400px;<br>  }<br>&lt;/style&gt;<br>&lt;div class=&quot;placeholder&quot;&gt;<br>  &lt;img src=&quot;high-quality.jpg&quot; alt=&quot;Image with placeholder&quot; loading=&quot;lazy&quot;&gt;<br>&lt;/div&gt;</pre><h3>9. Optimize Image Dimensions</h3><p>Resize images to the maximum dimensions required by your design. Avoid serving unnecessarily large images.</p><h4>Example:</h4><pre># Python example using PIL to resize images<br>from PIL import Image<br>image = Image.open(&#39;large.jpg&#39;)<br>image = image.resize((800, 400))<br>image.save(&#39;optimized.jpg&#39;, optimize=True, quality=85)</pre><h3>10. Dynamic Image Loading with JavaScript</h3><p>Load images dynamically as users scroll.</p><h4>Example:</h4><pre>const lazyImages = document.querySelectorAll(&#39;img[data-src]&#39;);</pre><pre>const observer = new IntersectionObserver((entries) =&gt; {<br>  entries.forEach(entry =&gt; {<br>    if (entry.isIntersecting) {<br>      const img = entry.target;<br>      img.src = img.dataset.src;<br>      observer.unobserve(img);<br>    }<br>  });<br>});</pre><pre>lazyImages.forEach(img =&gt; observer.observe(img));</pre><h3>11. Server-Side Image Optimization</h3><h4>Example with Node.js and Sharp:</h4><pre>const sharp = require(&#39;sharp&#39;);<br>const fs = require(&#39;fs&#39;);</pre><pre>sharp(&#39;input.jpg&#39;)<br>  .resize({ width: 800 })<br>  .webp({ quality: 80 })<br>  .toFile(&#39;output.webp&#39;, (err, info) =&gt; {<br>    if (err) console.error(err);<br>    else console.log(&#39;Image optimized:&#39;, info);<br>  });</pre><h3>12. CSS Sprites</h3><p>Combine multiple small images into a single file to reduce HTTP requests.</p><h3>Example:</h3><pre>.sprite {<br>  background: url(https://medium.com/&#39;sprite.png&#39;) no-repeat;<br>}<br>.icon1 {<br>  background-position: 0 0;<br>  width: 50px;<br>  height: 50px;<br>}<br>.icon2 {<br>  background-position: -50px 0;<br>  width: 50px;<br>  height: 50px;<br>}</pre><h3>13. Image Optimization Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*JTlT6XRylfrQ1UVZXg9nmQ.png" /><figcaption>Flowchart</figcaption></figure><h3>14. Impact on Core Web Vitals</h3><p>Image optimization directly affects:</p><ul><li><strong>Largest Contentful Paint (LCP)</strong>: Faster image loading improves LCP scores.</li><li><strong>First Input Delay (FID)</strong>: Reduces resource contention.</li><li><strong>Cumulative Layout Shift (CLS)</strong>: Properly sized images prevent layout shifts.</li></ul><p>Use tools like <a href="https://pagespeed.web.dev/?hl=en-US"><strong>Lighthouse</strong></a> to measure these metrics.</p><p>Optimizing images is essential for improving website performance, SEO, and user experience. By combining these techniques, you can ensure your images are fast, efficient, and visually appealing.</p><p><strong>Let’s Connect!</strong><br>I’m always open to collaborations and discussions. Feel free to reach out : <a href="https://www.kushagragangwar.tech/blog"><strong>kushgragangwar.tech/blog/</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab06e0c03f99" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>