<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Terris Linenbach on Medium]]></title>
        <description><![CDATA[Stories by Terris Linenbach on Medium]]></description>
        <link>https://medium.com/@terrislinenbach?source=rss-9f0a082d601b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9npsa2Lkgw1UClbgJqAAAQ.jpeg</url>
            <title>Stories by Terris Linenbach on Medium</title>
            <link>https://medium.com/@terrislinenbach?source=rss-9f0a082d601b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:07:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@terrislinenbach/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Teaching Codex to Run My Development Workflow]]></title>
            <link>https://terrislinenbach.medium.com/teaching-codex-to-run-my-development-workflow-48bf26412963?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/48bf26412963</guid>
            <category><![CDATA[ai-assisted-development]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 16:01:17 GMT</pubDate>
            <atom:updated>2026-11-18T06:57:46.000Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Or: Codex Using Codex Instead of You<br>Or: Writing Skills Instead of Bash Scripts<br>Or: Putting Yourself Out of Another Job, Again<br>Or: I Always Feel Like Somebody’s Watching Me<br>Or: I Hope You Like Management! You’re In It</strong></p><p><em>What a local development dashboard taught me about agent orchestration, verification, and learning from human feedback.</em></p><p>Like many developers today, I asked Codex to implement a ticket.</p><p>But in my case, it opened Chrome and ran Codex via an app named Minons that I’ll cover in more detail below. Codex running in the browser (or so it seems) then ran a skill which pulled main, created a branch, and started reading the ticket.</p><p>Visualize the following diagram:</p><p>ChatGPT Desktop -&gt; Chrome -&gt; Codex CLI.</p><p>Codex CLI, running in the browser, then asked for guidance. Everything ground to a halt, waiting for a human (me).</p><p>I told the “supervising agent” (Sol 6.1 running in ChatGPT Desktop) to respond. It did.</p><p><strong>Why am I burning tokens for Codex to control Codex via browser app? Isn’t that what agents and subagents are for? Is the browser app necessary?</strong></p><p>First of all, Minions is for me. I am the designer, “developer,” and sole user. I need to keep track everything I’m currently working on, how that work is (dis)organized on my laptop, and what state it is in. Herdr and Warp don’t solve all of my problems.</p><p>I have a hunch that a lot of my job can be automated. I was cooking dinner. My laptop was next to the stove. I decided to shoot for the moon: if I can use the app, so can Codex or Claude.</p><p><strong>Can I take myself out of the loop?</strong></p><p>More prompts followed: client generation; integration tests; production builds; PR creation. The supervisor kept approving commands and reporting that the implementation worker was still running.</p><p>Eventually, I asked what value it was adding. I could have enabled YOLO mode and avoided most of those prompts.</p><p>The conversation reached its most useful point when I said: <strong>“You weren’t looking at the code.”</strong></p><p>That exposed a gap in the workflow I was trying to build. The supervisor could operate the tools, but it had spent most of its time managing permission prompts. I wanted it to take responsibility for moving a ticket through development, testing, and review.</p><h4>The environment: a dashboard around tickets, local branches, and PRs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aoFS_ZE0bwbIyci--QHQhg.png" /></figure><p>Minions is a laptop-local development dashboard built with Phoenix, Ash, Oban, Postgres, React, and PostgreSQL. It organizes tickets, local branches, terminal sessions, and pull requests. I developed the base functionality in literally a few days, which is testimony to the power of Ash, Oban, and Elixir. However, improvement is continuous when you’re eating your own dog food.</p><p>The basic unit of work is a ticket associated with a physical checkout. Reserving a directory records ownership of that workspace. Git branch preparation happens separately.</p><p>From a reserved work item, I can launch an implementation session inside that checkout. The session runs a ticket implementation skill that retrieves requirements, inspects the repository, coordinates implementation, and runs verification.</p><p>Minions also exposes persistent terminal sessions in the browser. That gives a supervising agent a practical control surface: it can open an existing session, read output, answer questions, and switch to a shell in the same directory.</p><p>This setup makes orchestration possible. It also makes a subtle mistake easy: treating visible activity as evidence of completed work.</p><h4>Process state needs evidence behind it</h4><p>A session labeled <strong>Running</strong> can be waiting for a user decision. A terminal saying <strong>Working</strong> can be displaying an old snapshot. A worker can acknowledge a requested change while the source still contains the previous implementation.</p><p>We encountered all three.</p><p>The embedded terminal had a selectable text view with an explicit refresh button. Reading that snapshot repeatedly produced confident updates about stale state. At another point, the terminal was in scrollback while newer output existed below it.</p><p>Those are small interface details with architectural consequences. A supervisor needs to distinguish several kinds of state:</p><p>Process health<br>The session exists and its process is running</p><p>Agent progress<br>The agent is executing, waiting for input, or awaiting a worker</p><p>Implementation state<br>The requested changes exist in the current source</p><p>Verification state<br>Relevant checks passed against that source</p><p>Workflow state<br>The work is ready for testing, review, or a human decision</p><p>Each requires its own observation. A timer supports very few conclusions about the others.</p><h4>The useful work began in the running application</h4><p>My normal development loop includes opening another terminal and starting the application against a development or beta database. Once it is running, I inspect the actual behavior and give Codex concrete feedback.</p><p>In this session, a new conflict report became much clearer through that interaction:</p><ul><li>Put the two reports in tabs on the existing page.</li><li>Add a Search button.</li><li>Remove an unnecessary workspace column.</li><li>Make a row open the existing triage-note editor.</li><li>Remove the separate editor-action column once row opening handles that interaction.</li><li>Reuse the existing status-chip component.</li></ul><p>These decisions concern how an operator will use the feature. They are difficult to settle by watching test commands scroll past.</p><p>The report initially had no rows, so we needed realistic data to exercise the editor flow. I supplied a development patient and a local credentials file. The agent inspected the schema and discovered that each workspace could have only one operational-status note.</p><p>It created the synthetic conflict by changing a small set of metadata on the existing record. It preserved the note content and status, saved the original metadata, and prepared cleanup that would restore it only if the fixture remained unchanged.</p><p>That enabled a concrete check: the report displayed the row, and opening it brought up the existing conflict editor. We left the resolution actions untouched.</p><p>This was useful supervision: establish the conditions needed to test the feature, exercise it, and state precisely what was verified.</p><h4>My corrections are a specification source</h4><p>After several rounds, I realized the supervisor could learn a great deal by watching what I routinely tell Codex.</p><p>My corrections reveal recurring preferences: test in the running app, reuse existing editors and components, remove redundant navigation, provide explicit search controls, and inspect changes with representative data.</p><p>There are two layers of knowledge here:</p><p><strong>Ticket decisions</strong> belong to the current feature. A particular report needs two tabs. A particular column should disappear.</p><p><strong>Workflow preferences</strong> can guide future tickets. Start the application early enough to test it. Verify that an acknowledged change actually landed. Keep test fixtures reversible. Report failures accurately.</p><p>We started recording these observations in a repository Markdown guide. Each entry separates user instructions, observed behavior, and unresolved assumptions.</p><p>That is an explicit memory mechanism: capture a verified lesson, retrieve it on a later task, and apply it within the appropriate scope. It does not imply that the model silently retrains itself from watching a terminal.</p><h4>The supervisor’s job needs a completion contract</h4><p>The workflow I ultimately described was broader than answering prompts:</p><ol><li>Read the ticket and identify its repository.</li><li>Find and reserve a clean, available checkout in that repository.</li><li>Coordinate implementation and inspect the changed code.</li><li>Start the appropriate servers against the intended environment.</li><li>Exercise the solution and coordinate fixes as the code changes.</li><li>Run the local harsh-review and fix loop.</li><li>Create the pull request and initiate code review.</li><li>Update the Minions work-item status at meaningful transitions.</li></ol><p>Those transitions need evidence. “Awaiting testing” means there is an implementation ready to exercise. “Awaiting review” means a reviewable PR exists and review has been requested.</p><p>Verification also needs a relationship to the version being delivered. Our session reported passing database, backend, and UI suites, followed by additional UI refinements. The earlier results remained useful evidence for the earlier tree. The later edits needed appropriate follow-up checks.</p><p>The supervisor should preserve that distinction in its final report, alongside baseline failures and anything it could not verify.</p><h4>What I would optimize next</h4><p>The early version of this process consumed attention through repetitive updates and permission handling. Its most valuable moments involved inspecting source, using the application, constructing a reversible fixture, and translating my feedback into implementation requirements.</p><p>That gives me a concrete direction for improving it: make the supervisor responsible for gathering evidence, choosing the next useful action, and recognizing when a decision needs my input.</p><p>I want to hand it a ticket and get back a tested change, a reviewable PR, an accurate workflow status, and a concise account of the remaining risks.</p><p>Watching how I work can help it get there — provided those observations become explicit, scoped instructions that it actually checks against the next result.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48bf26412963" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Jev with Proprietary Processes, Data, and Requirements]]></title>
            <link>https://terrislinenbach.medium.com/using-jev-with-proprietary-processes-data-and-requirements-227904a6e771?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/227904a6e771</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mls]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 15:27:06 GMT</pubDate>
            <atom:updated>2026-10-25T15:52:22.083Z</atom:updated>
            <content:encoded><![CDATA[<p>Jev can work with your proprietary processes, data, and requirements <strong>without being trained on them</strong>. However, there’s an important distinction between understanding your business and making decisions using information you supply.</p><p>The approach is to <strong>put your business knowledge into the decision request</strong>, rather than trying to teach the neural network everything about your organization.</p><h4>1. Give Jev Your Proprietary Rules at Decision Time</h4><p>Suppose your internal medical-record triage process says:</p><ul><li>A referral without a subsequent specialist visit may indicate missing records.</li><li>A multiyear gap following a diagnosis requires investigation.</li><li>A previous unsuccessful provider request should not automatically prevent another request.</li><li>A duplicate request must have a documented rationale.</li></ul><p>You can supply those rules along with the case data.</p><h4>Decision Flow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xmuV5OzdQCUiWSWaAWWQXA.png" /></figure><p>Source for the above diagram:</p><pre>flowchart TD<br>    A[&quot;Proprietary Processes: Rules, policies, exceptions&quot;]<br>    B[&quot;Case Data: Records, dates, history&quot;]<br>    C[&quot;Jev Decision Model&quot;]<br>    D[&quot;Decision&quot;]<br>    E[&quot;Probability Scores&quot;]<br> <br>    A --&gt; C<br>    B --&gt; C<br>    C --&gt; D<br>    C --&gt; E</pre><p>The model doesn’t need to have seen your exact business process during training. It needs to understand the instructions and apply them to the supplied case.</p><p>However, that doesn’t guarantee it will consistently enforce every rule, especially when rules are complex or contradictory.</p><h4>2. Make Proprietary Knowledge Available Dynamically</h4><p>You don’t have to send every company policy with every request.</p><p>A better architecture is:</p><ol><li>Store your policies, requirements, and examples in your own database.</li><li>Retrieve the relevant rules for the particular decision.</li><li>Assemble the context and questions.</li><li>Send those to Jev.</li><li>Validate its decision against any mandatory business rules.</li></ol><p>This is similar to retrieval-augmented generation (RAG), except you’re retrieving knowledge for a decision model rather than a text-generation model.</p><p>Jev’s API supports a state containing text or structured data, plus typed questions with instructions and criteria.</p><h4>3. Understand the Limits</h4><p>There are three important types of proprietary knowledge:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*X0jOe3UpF8CyPIf4biJQwQ@2x.png" /></figure><p>The third category is particularly interesting.</p><p>Suppose your experienced triagers consistently recognize missing-record patterns that are difficult to express as written rules.</p><p><strong>Jev will not automatically learn those patterns from your organization’s history.</strong></p><p>Based on available documentation, Jev doesn’t currently offer customer-specific fine-tuning. Its customization is performed through the request rather than updating model weights.</p><p>You could instead retrieve similar historical cases and include their outcomes in the decision context, but you would need to test whether that actually improves accuracy.</p><h4>4. Evaluate Jev Against Historical Decisions</h4><p>Start with 200–500 previously resolved cases, including difficult edge cases.</p><p>Compare Jev’s decisions with the decisions made by experienced triagers.</p><p>Measure:</p><ul><li><strong>False positives:</strong> Unnecessary additional record requests.</li><li><strong>False negatives:</strong> Missing records that should have been requested.</li><li><strong>Decision accuracy:</strong> Agreement with validated expert decisions.</li><li><strong>Probability calibration:</strong> Whether Jev’s confidence corresponds to actual correctness.</li><li><strong>Rule compliance:</strong> Whether mandatory requirements are consistently followed.</li></ul><p>If it performs well, use it for recommendations and low-risk automation.</p><p>Keep deterministic rules and human review for consequential decisions.</p><h4>Data Privacy</h4><p>Your proprietary data is still transmitted to the hosted API.</p><p>TypeSafe’s published materials describe protections for customer data, including claims that requests aren’t used for training and an enterprise zero-retention option.</p><p>These protections should be verified contractually before sending proprietary records or patient information.</p><h4>5. The Bigger Architectural Question</h4><p>There are two fundamentally different goals.</p><h4>A. Make Jev Follow Your Organization’s Rules</h4><p>This can be achieved through:</p><ul><li>Instructions</li><li>Retrieved context</li><li>Structured inputs</li><li>Deterministic validation</li></ul><p>The neural network itself doesn’t need to change.</p><h4>B. Make Jev Acquire Your Organization’s Specialized Judgment</h4><p>This is a learning problem.</p><p>Supplying instructions may not be enough. You may need to train or fine-tune a separate model on your historical decisions.</p><h4>Conclusion</h4><p><strong>The critical distinction is between giving Jev knowledge at inference time and teaching a model new behavior through training.</strong></p><p>For explicit proprietary requirements, Jev may work well without additional training.</p><p>For specialized judgment developed through years of organizational experience, you need to establish whether contextual instructions and historical examples are sufficient or whether a custom-trained model is necessary.</p><p>The central design question is:</p><p><strong>Do you want Jev to follow explicit proprietary requirements, or do you want it to learn how your experienced employees make decisions from historical examples?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=227904a6e771" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why FOR UPDATE SKIP LOCKED Isn't Enough: Using pg_advisory_xact_lock to Build a Correct PostgreSQL…]]></title>
            <link>https://terrislinenbach.medium.com/why-for-update-skip-locked-isnt-enough-using-pg-advisory-xact-lock-to-build-a-correct-postgresql-d3eb9db46473?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3eb9db46473</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 16:23:56 GMT</pubDate>
            <atom:updated>2026-07-12T16:40:06.227Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why FOR UPDATE SKIP LOCKED Isn&#39;t Enough: Using pg_advisory_xact_lock to Build a Correct PostgreSQL Job Queue</h3><p><em>This article was written with </em><a href="https://m2m.tied-inc.com/"><em>Markdown to Medium</em></a><em> which is great but Medium’s editor added gratuitous blank lines and I don’t have the time to clean them up. We can’t have everything when it comes to Medium.</em></p><p>One of the most common patterns for implementing a distributed job queue in PostgreSQL is to use:</p><p>• UPDATE … RETURNING</p><p>• FOR UPDATE SKIP LOCKED</p><p>This pattern is excellent for ensuring that multiple workers don’t process the same job.</p><p>However, there’s a subtle race condition that SKIP LOCKED <strong>doesn&#39;t</strong> solve.</p><p>If your queue also needs to enforce <strong>global constraints</strong> — such as “never run more than N jobs concurrently” or “never insert duplicate active jobs” — you need another synchronization primitive.</p><p>That’s where PostgreSQL advisory locks shine.</p><h3>The Typical Claim Query</h3><p>Many queue processors use something similar to this:</p><pre>UPDATE job_queue<br>SET<br>    status = &#39;running&#39;,<br>    started_at = now(),<br>    updated_at = now()<br>WHERE id = (<br>    SELECT id<br>    FROM job_queue<br>    WHERE status = &#39;queued&#39;<br>    ORDER BY created_at<br>    FOR UPDATE SKIP LOCKED<br>    LIMIT 1<br>)<br>AND (<br>    SELECT count(*)<br>    FROM job_queue<br>    WHERE status = &#39;running&#39;<br>) &lt; $1<br>RETURNING *;</pre><p>This query does two jobs simultaneously:</p><p>1. Finds the oldest queued job.</p><p>2. Ensures the number of running jobs is below maxConcurrentJobs.</p><p>At first glance it looks completely safe.</p><p>It isn’t.</p><h3>What FOR UPDATE SKIP LOCKED Actually Protects</h3><p>FOR UPDATE SKIP LOCKED only protects <strong>rows being selected</strong>.</p><p>When two workers execute the query simultaneously:</p><p>• Worker A locks one queued row.</p><p>• Worker B skips that row and locks another queued row.</p><p>So far, everything is perfect.</p><p>No duplicate processing occurs.</p><p>The problem is the <strong>running job count</strong>.</p><h3>The Hidden Race Condition</h3><p>Suppose:</p><pre>maxConcurrentJobs = 3<br>currently running = 2</pre><p>Now two pods execute the claim query at exactly the same time.</p><p>Both transactions execute:</p><pre>SELECT count(*)<br>FROM job_queue<br>WHERE status = &#39;running&#39;;</pre><p>Both see:</p><pre>2</pre><p>Both conclude:</p><pre>2 &lt; 3</pre><p>Both update different queued jobs to running.</p><p>After both commit:</p><pre>running = 4</pre><p>The concurrency limit has been violated.</p><p>Nothing is wrong with SKIP LOCKED.</p><p>It simply was never designed to serialize reads of unrelated rows.</p><p>Under PostgreSQL’s default <strong>READ COMMITTED</strong> isolation level, each transaction performs its own independent visibility check.</p><h3>Serializing the Global Decision</h3><p>Instead of trying to make the entire database serializable, it’s much cheaper to serialize just the critical section.</p><p>That’s exactly what an advisory lock does.</p><pre>SELECT pg_advisory_xact_lock(hashtext(&#39;job_queue_claim&#39;));</pre><p>After acquiring the lock, execute the claim query.</p><p>Now only one worker in the entire fleet can perform the “count then update” operation at a time.</p><p>The second worker waits.</p><p>When the first transaction commits, the second transaction re-runs its count(*).</p><p>Instead of seeing:</p><pre>2</pre><p>it now sees:</p><pre>3</pre><p>and correctly decides not to claim another job.</p><p>The concurrency limit is never exceeded.</p><h3>Why Advisory Locks Are Perfect Here</h3><p>Unlike row locks, advisory locks aren’t attached to any table.</p><p>They’re simply application-defined mutexes maintained by PostgreSQL.</p><p>The lock key is an integer, so it’s common to generate one from a readable string:</p><pre>SELECT pg_advisory_xact_lock(hashtext(&#39;job_queue_claim&#39;));</pre><p>Here:</p><p>• hashtext() converts the string into a stable 32-bit integer.</p><p>• The string simply serves as a human-readable lock name.</p><p>Every worker that hashes the same string receives the same lock.</p><h3>Why Use pg_advisory_xact_lock()</h3><p>The _xact variant is transaction scoped.</p><p>That has several advantages.</p><h3>Automatic cleanup</h3><p>The lock is released automatically when the transaction commits or rolls back.</p><p>There is no corresponding unlock function to remember.</p><h3>Crash safety</h3><p>If a worker crashes:</p><p>• its database connection closes,</p><p>• PostgreSQL aborts the transaction,</p><p>• the advisory lock disappears automatically.</p><p>No stale distributed mutexes remain.</p><h3>Fleet-wide coordination</h3><p>Because advisory locks live inside PostgreSQL itself, they work across:</p><p>• multiple processes</p><p>• multiple containers</p><p>• multiple Kubernetes pods</p><p>• multiple application servers</p><p>No Redis lock service or ZooKeeper cluster is required.</p><h3>The Complete Pattern</h3><p>A typical claim transaction becomes:</p><pre>BEGIN;<br>SELECT pg_advisory_xact_lock(hashtext(&#39;job_queue_claim&#39;));<br>UPDATE job_queue<br>SET<br>    status = &#39;running&#39;,<br>    started_at = now(),<br>    updated_at = now()<br>WHERE id = (<br>    SELECT id<br>    FROM job_queue<br>    WHERE status = &#39;queued&#39;<br>    ORDER BY created_at<br>    FOR UPDATE SKIP LOCKED<br>    LIMIT 1<br>)<br>AND (<br>    SELECT count(*)<br>    FROM job_queue<br>    WHERE status = &#39;running&#39;<br>) &lt; $1<br>RETURNING *;<br>COMMIT;</pre><p>The advisory lock protects the global concurrency decision.</p><p>FOR UPDATE SKIP LOCKED protects the individual job row.</p><p>Together they make the queue both correct and highly concurrent.</p><h3>The Same Pattern Prevents Duplicate Work</h3><p>The same race condition appears when creating jobs.</p><p>Imagine code that does:</p><p>1. Check whether an equivalent queued or running job already exists.</p><p>2. Insert a new job if one doesn’t.</p><p>Without serialization:</p><pre>Worker A: no duplicate found<br>Worker B: no duplicate found</pre><pre>Worker A inserts<br>Worker B inserts</pre><p>Now there are duplicate active jobs.</p><p>Using another advisory lock solves this as well:</p><pre>SELECT pg_advisory_xact_lock(hashtext(&#39;job_queue_create&#39;));</pre><p>Now the existence check and insert execute as one serialized operation.</p><p>This pattern is especially valuable for handling:</p><p>• client retries</p><p>• double-clicks</p><p>• duplicate webhook deliveries</p><p>• idempotent APIs</p><h3>Performance Characteristics</h3><p>One concern is that advisory locks block.</p><p>That’s true — but in this case it’s exactly what you want.</p><p>Every worker attempting to claim work briefly queues behind the lock while another worker completes a very fast transaction.</p><p>Since the critical section is just one UPDATE, the lock is held for only a few milliseconds under normal conditions.</p><p>If workers poll every few seconds, the waiting time is typically negligible.</p><p>Interestingly, this serialization point also naturally bounds throughput. If claim attempts become excessively frequent, workers accumulate behind the advisory lock rather than all racing through the database simultaneously. This prevents a thundering herd of concurrent claim operations while still allowing job execution itself to proceed in parallel.</p><h3>pg_try_advisory_xact_lock()</h3><p>If you’d rather idle workers skip instead of waiting, PostgreSQL provides:</p><pre>SELECT pg_try_advisory_xact_lock(hashtext(&#39;job_queue_claim&#39;));</pre><p>Instead of blocking, this returns:</p><p>• true if the lock was acquired</p><p>• false if another worker already holds it</p><p>This is useful for periodic polling loops where there’s little value in having multiple idle workers queue behind the same lock. If one worker is already draining the queue, another worker can simply skip this polling cycle and try again later.</p><h3>One Small Footgun</h3><p>Advisory locks are keyed by integers, and the key space is shared across the entire database.</p><p>Using</p><pre>hashtext(&#39;job_queue_claim&#39;)</pre><p>is convenient because it lets you refer to locks by name rather than hard-coded integers.</p><p>The only theoretical risk is a hash collision: two unrelated lock names producing the same 32-bit hash. If that happened, the two subsystems would unnecessarily serialize each other.</p><p>In practice, such collisions are exceedingly rare, and for most applications the readability of named locks far outweighs this tiny risk.</p><h3>Conclusion</h3><p>FOR UPDATE SKIP LOCKED is excellent at preventing multiple workers from claiming the same job.</p><p>It is <strong>not</strong> sufficient when the operation also depends on shared global state, such as:</p><p>• maximum concurrent workers</p><p>• uniqueness constraints implemented in application logic</p><p>• rate limits</p><p>• quotas</p><p>• or any other check-then-act decision</p><p>In those cases, a transaction-scoped advisory lock provides an elegant solution. It serializes just the critical section, works across an entire fleet of application instances, cleans itself up automatically on commit or crash, and requires no external coordination service.</p><p>The combination of:</p><p>• pg_advisory_xact_lock()</p><p>• UPDATE … RETURNING</p><p>• FOR UPDATE SKIP LOCKED</p><p>forms a robust pattern for building PostgreSQL-backed job queues that are both scalable and correct.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3eb9db46473" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Better Concurrency in JavaScript with Asyncify]]></title>
            <link>https://terrislinenbach.medium.com/better-concurrency-in-javascript-with-asyncify-a64d6580b469?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a64d6580b469</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[concurrency]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 15:38:53 GMT</pubDate>
            <atom:updated>2025-07-20T17:04:21.346Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managing Concurrent Tasks in NodeJS API Requests</h3><p>When handling API requests in a shared NodeJS environment, or when processing a large number of fetch/etc. requests in a browser, it’s crucial to implement proper concurrency control to prevent resource exhaustion (mainly memory).</p><p>Without proper concurrency limits, it’s possible to spawn numerous Promise-based tasks, leading to:</p><ul><li>Memory overflow</li><li>CPU saturation</li><li>Degraded response times for other users</li><li>Potential server crashes under heavy load</li></ul><h3>Solution: Using Asyncify</h3><p>Asyncify provides a clean approach to managing concurrent tasks through:</p><pre>import { Asyncify } from &#39;asyncify-wasm&#39;;</pre><pre>const concurrencyLimit = 5; // Adjust based on server capacity<br>const asyncify = new Asyncify({<br>  maxConcurrent: concurrencyLimit<br>});</pre><pre>async function handleAPIRequest(req, res) {<br>  try {<br>    const tasks = generateTasks(); // Your task generation logic<br>    <br>    // Process tasks with concurrency limit<br>    const results = await asyncify.all(tasks);<br>    <br>    res.json({ success: true, results });<br>  } catch (error) {<br>    res.status(500).json({ error: error.message });<br>  }<br>}</pre><p>Here’s another example using mapLimit with asyncify:</p><pre>import { mapLimit } from &#39;async&#39;;<br>import { asyncify } from &#39;async-utility&#39;;</pre><pre>async function processPatients(patientIds: string[]) {<br>  return await mapLimit(<br>    patientIds,<br>    5, // Concurrency limit of 5 parallel operations<br>    asyncify(async (patientId: string) =&gt; {<br>      // Process each patient<br>      const result = await performPatientOperation(patientId);<br>      return result;<br>    })<br>  );<br>}</pre><p>This example demonstrates using mapLimit to process patient records with controlled concurrency, preventing server overload while handling multiple patient operations simultaneously.</p><h3>Key Implementation Points</h3><ul><li>Set appropriate concurrency limits based on server resources</li><li>Handle task queue overflow scenarios</li><li>Implement proper error handling and timeout mechanisms</li><li>Monitor task execution metrics</li></ul><h3>Best Practices</h3><p>When implementing concurrency control:</p><ul><li>Start with conservative limits and adjust based on monitoring</li><li>Include timeout mechanisms for long-running tasks</li><li>Implement proper error handling and recovery</li><li>Add monitoring to track concurrent task metrics</li></ul><p>By implementing these controls, you can ensure your NodeJS server remains stable and responsive even under heavy load conditions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a64d6580b469" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[View Upcoming Google Calendar Events via the Terminal]]></title>
            <link>https://terrislinenbach.medium.com/view-upcoming-google-calendar-events-via-your-terminal-e60c9ab368e2?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e60c9ab368e2</guid>
            <category><![CDATA[google-calendar]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 19:47:12 GMT</pubDate>
            <atom:updated>2025-03-14T17:22:56.413Z</atom:updated>
            <content:encoded><![CDATA[<p>Keep it simple and don’t miss anything important!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*YlJum-YE9qbb_NgmIXBKLA@2x.png" /><figcaption>Nerd Mode</figcaption></figure><p>Here’s how to list upcoming Google Calendar events in your terminal:</p><ol><li>Install Python</li><li>Install gcalcli via “pip install gcalcli” or consult the <a href="https://github.com/insanum/gcalcli">README</a></li><li>Run “gcalcli” and <a href="https://github.com/insanum/gcalcli?tab=readme-ov-file#initial-setup">set up your access token</a></li><li>Run the following script. The list of events will update itself every five minutes.</li></ol><pre>#!/bin/bash<br><br># 1. Install Python<br># 2. pip install gcalcli<br># 3. get an access token<br><br># Exit on any error<br>set -e<br><br># Handle SIGINT (Ctrl+C) and SIGTERM<br>cleanup() {<br>    echo<br>    echo &quot;Calendar check script terminated.&quot;<br>    exit 0<br>}<br><br>trap cleanup SIGINT SIGTERM<br><br># Check if gcalcli is installed<br>if ! command -v gcalcli &amp;&gt; /dev/null; then<br>    echo &quot;Error: gcalcli is not installed. Please install it first.&quot;<br>    echo &quot;You can install it using: pip install gcalcli&quot;<br>    exit 1<br>fi<br><br>echo &quot;Starting calendar check script (checking every 5 minutes)&quot;<br>echo &quot;Press Ctrl+C to exit&quot;<br>echo<br><br>while true; do<br>    clear<br>    echo &quot;=== Calendar Check: $(date) ===&quot;<br>    echo &quot;Fetching upcoming events...&quot;<br>    echo<br><br>    if ! gcalcli agenda --nostarted --nodeclined; then<br>        echo &quot;Error: Failed to fetch calendar events&quot;<br>        sleep 300  # Still wait 5 minutes before retrying<br>        continue<br>    fi<br><br>    echo<br>    sleep 300  # Wait for 5 minutes<br>done</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e60c9ab368e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Same as it ever was]]></title>
            <link>https://terrislinenbach.medium.com/same-as-it-ever-was-8b7e42a861a9?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b7e42a861a9</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Sat, 04 Jan 2025 16:36:39 GMT</pubDate>
            <atom:updated>2025-11-14T20:14:56.414Z</atom:updated>
            <content:encoded><![CDATA[<p>All the “little things” take their toll on the project schedule.</p><p>Productivity, even for small for teams, is a complex function involving resources and time. As teams learn and other concerns remain mostly constant, velocity should increase. The little things are very important but every decision is like a paper cut.</p><p>If you want your team to build fast, make sure they don’t get lost in the minutae. As we all know, naming things is a productivity killer. Anticipate the inevitable questions that arise from tools for which “there is more than one correct way to do it.” Document the decisions and move on. Try to not revisit them.</p><p>For example:</p><ol><li>Should “is” be used consistently as a prefix for boolean values, functions that return them, and so on?</li><li>Should a concept be modeled as a datetime (with timeZone) instead of a boolean?</li><li>Can a data element be null? What does null mean in context?</li><li>For object-oriented languages, should data be accessed as a property or a function? foo or getFoo()? See also 1 above.</li></ol><p>These decisions are hard to change and can be a significant cause of aesthetic refactoring, so watch out for them!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b7e42a861a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A bash script to zip uncommitted files]]></title>
            <link>https://terrislinenbach.medium.com/a-bash-script-to-zip-uncommitted-files-1115e4a6a5cd?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1115e4a6a5cd</guid>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Sat, 14 Dec 2024 22:07:28 GMT</pubDate>
            <atom:updated>2026-07-12T16:05:01.065Z</atom:updated>
            <content:encoded><![CDATA[<h3>A bash script to archive new and uncommitted files in a local git repository</h3><p>Want to back up your work without pushing it to the remote repository due to the precommit hooks?</p><p>Run this script within a local git repository. All modified and new files in the repository are zipped to ~/.git-backups/&lt;repo-name&gt;-&lt;branch-name&gt;/&lt;repo-name&gt;-&lt;branch-name&gt;-&lt;timestamp&gt;.zip</p><p><strong>Warning:</strong> This script ignores files that have been committed regardless of whether they’ve been pushed to the remote repository. If it’s not output by git status , it won’t be included in the archive.</p><p>This script runs fine on MacOS (which always requires extra work!) and it *should* run on Linux but I haven’t tried it.</p><h3>Preparation</h3><ul><li>MacOS: brew install zip</li><li>Copy the script below to (for example) ~/gitzip</li><li>chmod +x ~/gitzip</li></ul><pre>#!/bin/bash<br><br># Initialize debug flag (default: disabled)<br># Enable via: --debug<br>debug=0<br><br># Process command line arguments<br>while [[ &quot;$#&quot; -gt 0 ]]; do<br>    case $1 in<br>        --debug) debug=1; set -x ;;<br>        *) error &quot;Unknown parameter: $1&quot; ;;<br>    esac<br>    shift<br>done<br><br># Debug output function<br>debug_echo() {<br>    if [ &quot;$debug&quot; -eq 1 ]; then<br>        echo &quot;DEBUG: $1&quot;<br>    fi<br>}<br><br># Run this script within a local git repository.<br># All modified and new files in the repository are zipped to<br># ~/.git-backups/&lt;repository&gt;-&lt;branch&gt;/&lt;repository&gt;-&lt;branch&gt;-&lt;timestamp&gt;.zip<br><br># Function to print error messages to stderr and exit<br>error() {<br>    echo &quot;Error: $1&quot; &gt;&amp;2<br>    exit 1<br>}<br><br># Function to generate a timestamp in the format YYYYMMDD-HHMMSS<br>generate_timestamp() {<br>    date +&quot;%Y%m%d-%H%M%S&quot;<br>}<br><br># Function to sanitize strings for use in file and directory names<br>sanitize() {<br>    echo &quot;$1&quot; | tr &#39; /\\:*?&quot;&lt;&gt;|&#39; &#39;----------&#39;<br>}<br><br># Change to the root of the Git repository<br>git_root=$(git rev-parse --show-toplevel 2&gt;/dev/null) || error &quot;Not inside a Git repository.&quot;<br>cd &quot;$git_root&quot; || error &quot;Failed to change directory to Git root.&quot;<br><br># Retrieve the repository name using the basename of the Git root directory<br>repo_name=$(basename &quot;$git_root&quot;)<br>repo_name_sanitized=$(sanitize &quot;$repo_name&quot;)<br><br># Get the current Git branch name, fallback to commit hash if in detached HEAD<br>branch_name=$(git symbolic-ref --short HEAD 2&gt;/dev/null || git rev-parse --short HEAD 2&gt;/dev/null || echo &quot;no-branch&quot;) || error &quot;Failed to identify current Git state.&quot;<br>branch_name_sanitized=$(sanitize &quot;$branch_name&quot;)<br><br># Get the list of changed files (staged and unstaged)<br>if git rev-parse --verify HEAD &gt;/dev/null 2&gt;&amp;1; then<br>    changed_files=$(git diff --name-only HEAD | grep -v &quot;^\.specstory/&quot;)<br>else<br>    changed_files=&quot;&quot;<br>fi<br><br>debug_echo &quot;Changed files found:&quot;<br>debug_echo &quot;$changed_files&quot;<br><br># Get the list of untracked files<br>untracked_files=$(git ls-files --others --exclude-standard | grep -v &quot;^\.specstory/&quot;)<br>debug_echo &quot;Untracked files found:&quot;<br>debug_echo &quot;$untracked_files&quot;<br><br># Exit on error - do after the above because grep returns error when there are no matches<br>set -e<br><br># Before creating temp file<br>debug_echo &quot;About to create temporary file&quot;<br>if ! command -v mktemp &gt;/dev/null 2&gt;&amp;1; then<br>    error &quot;mktemp command not found&quot;<br>fi<br><br># Create temporary file with error handling<br>if ! file_list=$(mktemp); then<br>    error &quot;Failed to create temporary file&quot;<br>fi<br>debug_echo &quot;Created temporary file: $file_list&quot;<br><br># Verify the file exists and is writable<br>if [ ! -f &quot;$file_list&quot; ]; then<br>    error &quot;Temporary file was not created&quot;<br>fi<br>if [ ! -w &quot;$file_list&quot; ]; then<br>    error &quot;Temporary file is not writable&quot;<br>fi<br><br># Add trap right after successful creation<br>trap &#39;rm -f &quot;$file_list&quot; 2&gt;/dev/null; rm -rf &quot;$temp_dir&quot; 2&gt;/dev/null&#39; EXIT<br>debug_echo &quot;Trap set for cleanup&quot;<br><br># Store changed files<br>if [ -n &quot;$changed_files&quot; ]; then<br>    debug_echo &quot;Writing changed files to temporary file&quot;<br>    printf &#39;%s\n&#39; &quot;$changed_files&quot; &gt; &quot;$file_list&quot; || error &quot;Failed to write changed files&quot;<br>fi<br><br># Append untracked files<br>if [ -n &quot;$untracked_files&quot; ]; then<br>    debug_echo &quot;Writing untracked files to temporary file&quot;<br>    printf &#39;%s\n&#39; &quot;$untracked_files&quot; &gt;&gt; &quot;$file_list&quot; || error &quot;Failed to write untracked files&quot;<br>fi<br><br>debug_echo &quot;Reading combined files&quot;<br>combined_files=$(sort -u &quot;$file_list&quot;) || error &quot;Failed to combine files&quot;<br><br>debug_echo &quot;Files to be processed:&quot;<br>debug_echo &quot;$combined_files&quot;<br><br># Check if there are any files to back up<br>if [ -z &quot;$combined_files&quot; ]; then<br>    echo &quot;No changed or untracked files found.&quot;<br>    exit 0<br>fi<br><br># Create a temporary directory for copying changed and untracked files<br>temp_dir=$(mktemp -d) || error &quot;Failed to create temporary directory.&quot;<br>debug_echo &quot;Created temporary directory: $temp_dir&quot;<br># Ensure the temporary directory is removed on script exit<br>trap &#39;rm -f &quot;$file_list&quot;; rm -rf &quot;$temp_dir&quot;&#39; EXIT<br><br># Copy changed and untracked files to the temporary directory, preserving directory structure<br>while IFS= read -r file; do<br>    debug_echo &quot;Processing file: $file&quot;<br>    # Ensure the file exists (it might have been deleted)<br>    if [ -e &quot;$file&quot; ]; then<br>        # Create the target directory inside temp_dir<br>        target_dir=&quot;$temp_dir/$(dirname &quot;$file&quot;)&quot;<br>        mkdir -p &quot;$target_dir&quot; || error &quot;Failed to create directory structure for $file.&quot;<br><br>        # Copy the file to the target directory<br>        cp &quot;$file&quot; &quot;$target_dir/&quot; || error &quot;Failed to copy file: $file&quot;<br>    else<br>        debug_echo &quot;Skipping deleted file: $file&quot;<br>    fi<br><br>done &lt;&lt;&lt; &quot;$combined_files&quot;<br><br># Create the zip file with the repository and branch name<br>zip_file=&quot;${repo_name_sanitized}-${branch_name_sanitized}-$(generate_timestamp).zip&quot;<br>(<br>    cd &quot;$temp_dir&quot; || error &quot;Failed to change directory to temporary directory.&quot;<br>    zip -r &quot;$zip_file&quot; ./* &gt;/dev/null || error &quot;Failed to create zip archive.&quot;<br>)<br><br># Define the backup directory path<br>backup_dir=&quot;$HOME/.git-backups/${repo_name_sanitized}-${branch_name_sanitized}&quot;<br><br># Create the backup directory<br>mkdir -p &quot;$backup_dir&quot; || error &quot;Failed to create backup directory: $backup_dir&quot;<br><br># Move the zip file to the backup directory<br>mv &quot;$temp_dir/$zip_file&quot; &quot;$backup_dir/&quot; || error &quot;Failed to move zip file.&quot;<br><br># Inform the user of the backup location<br>unzip -v &quot;$backup_dir/$zip_file&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1115e4a6a5cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tagging Data in LLM Prompts for Repeatable Output]]></title>
            <link>https://terrislinenbach.medium.com/tagging-data-in-llm-prompts-445cd2756b13?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/445cd2756b13</guid>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Fri, 27 Sep 2024 05:37:44 GMT</pubDate>
            <atom:updated>2024-12-08T15:42:28.514Z</atom:updated>
            <content:encoded><![CDATA[<p>What’s the best way for a prompt to operate on runtime data? Should we use string concatenation like we did to inject data into SQL statements before we knew better?</p><p>Langchain, for example, provides templates that support many types of models. If you use native APIs instead of an abstraction layer like Langchain, you have to learn the <strong>Best Way</strong> for every model. But even if you use Langchain, you still need to get everyone on the team to use it consistently since LLMs are so adept at handling uncertainty.</p><p>Wishy washy instructions result in more randomness even with identical inputs, temperatures, p values, and random seeds.</p><h4>Let’s ask ChatGPT</h4><p>The attached conversation with GPT confirms my belief that variables and references to them should be explicit even when testing shows that it’s unnecessary. As with code, readability and fewer surprises usually trump everything else.</p><p>Let’s say a prompt says: “List the top cars in order of price and sales {cars} {sales}” and I provide a dictionary containing cars and sales keys, does the LLM know the difference between cars and sales automatically or should I add that information to the prompt explicitly?</p><p>In LangChain, when you create a prompt template and provide variables, the template will replace placeholders like {cars} and {sales} with the corresponding values from the dictionary you provide. However, <strong><em>the LLM does not inherently know the names of the variables</em></strong> unless they are explicitly mentioned in the prompt itself. The LLM only sees the final, rendered text after the placeholders have been filled in with their corresponding values.</p><p>Here’s a step-by-step breakdown of what happens:</p><ol><li>Template Creation: You define a prompt template, e.g., &quot;List the top cars in order of price and sales {cars} {sales}&quot;.</li><li>Providing Variables: You provide a dictionary with the variables, e.g., {&quot;cars&quot;: [&quot;Car A&quot;, &quot;Car B&quot;], &quot;sales&quot;: [100, 200]}.</li><li>Rendering the Prompt: LangChain replaces {cars} and {sales} in the template with the values from the dictionary. For example, the rendered prompt might look like:</li></ol><pre>List the top cars in order of price and sales Car A, Car B 100, 200</pre><p>4. LLM Execution: The LLM receives the rendered prompt as text. It does not know that &quot;Car A, Car B&quot; came from a variable named cars and &quot;100, 200&quot; came from a variable named sales. The LLM just sees the plain text and processes it accordingly.</p><h3>How to Ensure Clarity in the Prompt</h3><p>If you need the LLM to distinguish between cars and sales, you should make sure the prompt is clear and unambiguous in the rendered form. For example, you could rewrite the template like this:</p><pre>List the top cars in order of price and sales.<br>Cars: {cars}<br>Sales: {sales}</pre><p>When filled, the prompt might look like:</p><pre>List the top cars in <br>order of price and sales.<br>Cars: {cars}<br>Sales: {sales}</pre><p>This way, the LLM can clearly distinguish between the car names and sales numbers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=445cd2756b13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Coaxing GenAI to Reliably Create Grid Tables in Pandoc Markdown]]></title>
            <link>https://terrislinenbach.medium.com/using-genai-to-create-grid-tables-as-pandoc-markdown-documents-b7423ef1d91d?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7423ef1d91d</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[markdown]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Sat, 21 Sep 2024 05:26:51 GMT</pubDate>
            <atom:updated>2025-08-14T21:10:45.380Z</atom:updated>
            <content:encoded><![CDATA[<h3>Coaxing LLMs to Create Grid Tables in Pandoc Markdown</h3><p>Large language models can generate rich grid tables using HTML and <a href="https://pandoc.org/chunkedhtml-demo/8.9-tables.html">Pandoc’s Markdown extensions</a>, to name a few formats. This article focuses on the latter because Markdown is easier for humans to read and include in prompts.</p><p>Unfortunately, today’s LLMs struggle with Pandoc’s strict Markdown grid table format, necessitating retries. But how do we know when the generated Markdown is valid?</p><h4>Grid Tables</h4><p>Grid tables are more advanced than standard Markdown tables. Like HTML tables, they support spanning rows as well as columns.</p><p>The following example grid table contains no data and can be used as a template:</p><pre>+---------------------+:---------------------:+<br>| Location            | Temperature 1961-1990 |<br>|                     | in degree Celsius     |<br>|                     +-------+-------+-------+<br>|                     | min   | mean  | max   |<br>+=====================+=======+=======+=======+<br>|                     |       |       |       |<br>+---------------------+-------+-------+-------+</pre><p>Here’s the same table containing two rows of data:</p><pre>+---------------------+:---------------------:+<br>| Location            | Temperature 1961-1990 |<br>|                     | in degree Celsius     |<br>|                     +-------+-------+-------+<br>|                     | min   | mean  | max   |<br>+=====================+=======+=======+=======+<br>| Antarctica          | -89.2 | N/A   | 19.8  |<br>+---------------------+-------+-------+-------+<br>| Earth               | -89.2 | 14    | 56.7  |<br>+---------------------+-------+-------+-------+</pre><p>You can test this template via the <a href="https://tinyurl.com/5rh8srfu">Try Pandoc!</a> web page. Add some values to the empty cells, click the download link, and open the downloaded HTML file in your browser. Also try generating a Docx file.</p><p><em>Docx output doesn’t center the Temperature 1961–1990 header. This seems to be a bug.</em></p><h4>Pandoc’s Grid Table Markdown Format</h4><p>The layout of a Pandoc grid table is defined using just four characters:</p><pre>+ - = : |</pre><p>Colons are used in column borders to define whether values in that column are left-justified (the default?), right-justified (one colon on the right side, before the +), or centered (two colons).</p><p>In the above example, the “Temperature 1961–1990 …” heading is specified to be centered:</p><figure><img alt="A column header for three columns beneath it. The header text should be centered." src="https://cdn-images-1.medium.com/max/185/1*9NdrSFWCF1rfxiefZGL8qA@2x.png" /><figcaption>A centered multi-column header</figcaption></figure><h4>Pandoc Grid Table Markdown Rules</h4><p>Pandoc converts Markdown to other formats such as HTML and Docx.</p><p>The basic requirements for grid table Markdown are:</p><ol><li>Each line must have the same length</li><li>The first and last lines must not contain |</li><li>Each line must begin and end with either + or |</li><li>Each | must have either a + or | above and below it</li></ol><p>These rules are not comprehensive — good luck finding them! However, they are sufficient for validating the output of large language models, particularly GPT-4o.</p><h4>Prompting GenAI to Create Pandoc Grid Table Markdown Documents</h4><p>Grid table Markdown documents can be generated by LLM APIs in three steps:</p><ol><li>Create a prompt containing some instructions, a grid table template, and data to add to the table:</li></ol><pre>Use DATA to create a Pandoc Markdown grid table.<br><br>Only output the table.<br><br>Create a grid table using Pandoc Markdown using the following template. Rules:<br>- Each line must have the same length<br>- Vertically align | and + characters<br>- Output \ characters as-is<br><br>+---------------------+-----------------------+<br>| Location            | Temperature \         |<br>|                     | 1961-1990             |<br>|                     | in degree Celsius     |<br>|                     +-------+-------+-------+<br>|                     | min   | mean  | max   |<br>+=====================+=======+=======+=======+<br>|                     |       |       |       |<br>+---------------------+-------+-------+-------+<br><br>DATA:<br>&lt;add input data here (in JSON, for example)&gt;</pre><ul><li>The instruction “<em>Only output the table</em>” prevents (most of the time, anyway) the model from writing an introductory sentence about the table, which it only does intermittently. You could instead instruct it to add an introduction if desired.</li><li>The instruction “<em>Output \ characters as-is</em>” is needed because LLMs intermittently fail to output backslashes. As a demonstration, I added a backslash after ‘Temperature’ in the top-right header. Pandoc converts backslashes in grid table Markdown into line breaks.</li></ul><p>2. Validate the output</p><p>3. If the returned grid table Markdown is invalid, go to 1</p><h4><em>Why use an LLM for this?</em></h4><p>It’s not because I like using gold-plated bulldozers to swat flies!</p><p>Grid tables are more sophisticated (or <em>troublesome</em>, if you have to write the code) than meets the eye. Consider the case in which a cell’s contents are too big to fit in a single line. Long lines should be word-wrapped, which requires adding additional “rows” without adding grid lines for them. For example, let’s replace “Antarctica” with “Antarctica is a cold place.” Should we make the Location column wider or maintain the columns’ proportions specified in the template? Here’s what GPT-4o outputs:</p><pre>+---------------------+:---------------------:+<br>| Location            | Temperature 1961-1990 |<br>|                     | in degree Celsius     |<br>|                     +-------+-------+-------+<br>|                     | min   | mean  | max   |<br>+=====================+=======+=======+=======+<br>| Antarctica is a     | -89.2 | N/A   | 19.8  |<br>| cold place          |       |       |       |<br>+---------------------+-------+-------+-------+<br>| Earth               | -89.2 | 14    | 56.7  |<br>+---------------------+-------+-------+-------+</pre><p>This is exactly what we want, rather than:</p><pre>+----------------------------+:---------------------:+<br>| Location                   | Temperature 1961-1990 |<br>|                            | in degree Celsius     |<br>|                            +-------+-------+-------+<br>|                            | min   | mean  | max   |<br>+============================+=======+=======+=======+<br>| Antarctica is a cold place | -89.2 | N/A   | 19.8  |<br>+----------------------------+-------+-------+-------+<br>| Earth                      | -89.2 | 14    | 56.7  |<br>+----------------------------+-------+-------+-------+</pre><p><em>I’ll wait while you code this behavior. Be sure to support every possible grid table template.</em></p><p>Perhaps a better strategy is to have an LLM write the code. Or search PyPi etc. for a suitable library.</p><p>Using LLMs is overkill for most use cases. It’s slow and random and can cost quite a bit of money when using cloud-hosted models. But LLMs can save you a ton of time that you can spend on something else. Read on!</p><p><em>I can’t say why I’ve been using GPT-4o to create grid tables, but it’s been fun and productive. I’m happy to share what I have learned along the way.</em></p><h4>Troubleshooting Consistent Data-Induced Failures</h4><p>Models have been trained to maintain the column proportions specified in the provided grid table template. For example, in the above table, the Location column is considerably wider than the min, mean, and max columns. But sometimes LLM are unable to fill table cells with data while maintaining these proportions, resulting in undesired word wrapping or invalid grid tables. The solution to this problem is to change the template and make some if not all of the columns wider.</p><h4>Close, But No Grid Table</h4><p>GenAI’s inherent randomness inevitably produces grid tables that fail validation. For example, the following output is from GPT-4o (August 2024 version) in mid-September 2024. The second line contains an extra space.</p><pre>+---------------------+-----------------------------+<br>| Location            | Temperature 1961-1990 in °C  |<br>|                     +---------+---------+---------+<br>|                     |   min   |  mean   |   max   |<br>+=====================+=========+=========+=========+<br>|                     |         |         |         |<br>+---------------------+---------+---------+---------+</pre><p>This type of failure will occur occasionally regardless of the provided temperature and p-value. As of September 2024, it’s impossible to prompt LLMs to create perfect grid tables 100% of the time. However, common failures like these can be fixed automatically, thus reducing the need for retries considerably.</p><h4>Working Around GenAI’s Foibles</h4><p>The function validate_grid_table_markup() adds or removes -, =, and spaces at the end of each line in a grid table that has the wrong length. For example, the table:</p><pre>+:------------:+:--------------------:+<br>| Heading 1    | Heading 2         |<br>+---+----+-----+----+------+--------------:+<br>| A | B  | C   | D  |  E   | ABC |<br>+---+----+-----+----+------+------------+</pre><p>can be transformed to:</p><pre>+:------------:+:-------------------:+<br>| Heading 1    | Heading 2           |<br>+---+----+-----+----+------+--------:+<br>| A | B  | C   | D  |  E   | ABC     |<br>+---+----+-----+----+------+---------+</pre><p>However, the following table can not be repaired because the Heading 2 column is too narrow to fit the value in the bottom right cell:</p><pre>+:------------:+:--------------------:+<br>| Heading 1    | Heading 2            |<br>+---+----+-----+----+------+---------:+<br>| A | B  | C   | D  |  E   | FFFFFFFFFFFFFFFF |<br>+---+----+-----+----+------+----------+</pre><p>A grid table’s maximum line length is determined via the first line of the table, specifically:</p><pre>+: — — — — — — :+: — — — — — — — — — — :+</pre><h4>Necessary Complexity</h4><p>The last grid table presented in the previous section can be repaired by changing each line, if necessary, to conform to the length of the table’s longest line. However, <strong>d</strong>rastic changes to column widths can produce undesired results and therefore should be avoided.</p><p>That said, experience with GPT-4o has shown that the rightmost columns in generated grid table Markdown often need to be widened by a single character. GPT-4o intermittently fills these columns with one too many characters.</p><p>It is possible to work around this quirk in the today’s public models. In other words, the table:</p><pre>+-----+----------------------:+<br>| Red | Little Red Riding Hood|<br>+-----+-------------------------------+</pre><p>can be fixed by widening the rightmost column by one character:</p><pre>+-----+----------------------:+<br>| Red | Little Red Riding Hood|<br>+-----+-----------------------+</pre><p>This fix this should be performed only when at least one line in the grid table is longer than the first. However, validate_grid_table_markup() doesn’t check this condition because adding only one space could negatively impact the layout of extra wide tables, and it’s better to find out immediately instead of eventually. Therefore, this feature is optional.</p><h4>Pandoc Grid Table Markdown Validator in Python</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1647eaf322360fb25fa6a2690fe36899/href">https://medium.com/media/1647eaf322360fb25fa6a2690fe36899/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7423ef1d91d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CloudFormation and VSCode]]></title>
            <link>https://terrislinenbach.medium.com/cloudformation-and-vscode-308a3b8d87b5?source=rss-9f0a082d601b------2</link>
            <guid isPermaLink="false">https://medium.com/p/308a3b8d87b5</guid>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Terris Linenbach]]></dc:creator>
            <pubDate>Tue, 04 Jun 2024 19:38:15 GMT</pubDate>
            <atom:updated>2024-06-19T15:38:56.920Z</atom:updated>
            <content:encoded><![CDATA[<h3>Editing CloudFormation Templates with VSCodeCloudFormation templates. The linter makes useful recommendations and is sorely missing from the AWS console.</h3><ol><li>Install the <a href="https://marketplace.visualstudio.com/items?itemName=aws-scripting-guy.cform">CloudFormation extension</a></li><li>Install the <a href="https://marketplace.visualstudio.com/items?itemName=kddejong.vscode-cfn-lint">CloudFormation Linter extension</a></li><li>Complete the installation of the previous extension:</li></ol><p>- Terminal/New Terminal<br>- Run: pip install cfn-lint<br>- Restart VSCode</p><p>4. Press Control-Shift-P (macOS: Command-Shift-P)<br>5. Select Preferences: Open user settings (JSON)<br>6. Append the following and save the file</p><pre>// Custom tags for the parser to use<br> &quot;yaml.customTags&quot;: [<br> &quot;!And&quot;,<br> &quot;!If&quot;,<br> &quot;!Not&quot;,<br> &quot;!Equals&quot;,<br> &quot;!Or&quot;,<br> &quot;!FindInMap sequence&quot;,<br> &quot;!Base64&quot;,<br> &quot;!Cidr&quot;,<br> &quot;!Ref&quot;,<br> &quot;!Sub&quot;,<br> &quot;!GetAtt&quot;,<br> &quot;!GetAZs&quot;,<br> &quot;!ImportValue&quot;,<br> &quot;!Select&quot;,<br> &quot;!Select sequence&quot;,<br> &quot;!Split&quot;,<br> &quot;!Join sequence&quot;<br> ],<br> // Enable/disable default YAML formatter (requires restart)<br> &quot;yaml.format.enable&quot;: true</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=308a3b8d87b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>