<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Boris Smus on Medium]]></title>
        <description><![CDATA[Stories by Boris Smus on Medium]]></description>
        <link>https://medium.com/@borismus?source=rss-fbe27680c9fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Agd5pcitLOEUuZJtV8fOJQ.png</url>
            <title>Stories by Boris Smus on Medium</title>
            <link>https://medium.com/@borismus?source=rss-fbe27680c9fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:39:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@borismus/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[supernote-cli: pen, paper, and a pipe]]></title>
            <link>https://borismus.medium.com/supernote-cli-pen-paper-and-a-pipe-44def6aa1f5a?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/44def6aa1f5a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[command-line]]></category>
            <category><![CDATA[supernote]]></category>
            <category><![CDATA[minimalism]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 02:59:02 GMT</pubDate>
            <atom:updated>2026-10-04T02:59:02.708Z</atom:updated>
            <content:encoded><![CDATA[<p>A recent <a href="https://www.nytimes.com/2026/03/27/opinion/technology-mental-fitness-cognitive.html">NYT piece</a> argued we need a mental fitness revolution to combat the cognitive decay caused by algorithmic feeds and generative AI. It’s an efficient one-two punch. If you’re not brainrotting on short form video content, you’re outsourcing all of your thinking to an LLM. The result is a kind of cognitive strip-mining. What’s left requires active defense.</p><p>For me, one way of defending that capacity for deep work is with a pen on e-ink. Whether it’s annotating a paper or starting a sketch from scratch, I’m intentionally making room for focused thought. My army of clawed Claudes and Codexes will just have to wait.</p><p>As I’ve written <a href="https://smus.com/notes/2025/the-pursuit-of-frictionless-capture/">before</a>, my e-ink writer of choice is a Supernote Nomad. The Nomad does one thing: it removes the exits. No feed, no notifications, no reflex to ask the nearest model. Just the question you’re sitting with.</p><p>But there’s still a gap: extracting digests and handwritten notes relies on the awkward Supernote Partner desktop app, which doesn’t easily support bulk exports. I’m no luddite and use GenAI for a bunch of work, including critique of ideas and iteration on writing. If I can’t automatically pipe my focused thinking into Obsidian or my AI workflows, my <a href="https://smus.com/file-systems-for-thought/">system for thought</a> breaks. It doesn’t help that the <a href="https://pypi.org/project/sncloud/">Python sync libraries</a> that used to work no longer do.</p><p>So I extracted supernote-cli from my note management scripts to fix the plumbing and introduce a few conveniences. Here are some examples of usage:</p><p><strong>Extract handwritten notes</strong> and show their on-device transcript:</p><pre>$ supernote notebook ls --limit 3<br> 1251704792368021505  2026-04-22  A2A ideation for Agent book<br> 1254057731111780353  2026-04-24  San Francisco Note, April 20<br> 1254579462477971456  2026-04-27  20260424_081053<br><br>$ supernote nb 1254057731111780353<br>## Page 1<br>Excited to do that again it&#39;s been over a decade since I<br>went to that space and &lt;redacted&gt; is kind of a hero for me.<br>...</pre><p><strong>List annotated documents</strong> and extract handwritten highlights and notes, and transcribe them with a <a href="https://smus.com/notes/2025/local-e-ink-handwriting-recognition-with-on-device-vlms/">local VLM</a>.</p><pre>$ supernote annotation ls --limit 4<br>Breath_The_New_Science_of_a_Lost_Art_James_Nestor.pdf<br> 833859954824708096  Apr 18 17:43  Any gum chewing can strengthen the jaw a (A)<br> 833859955948781568  Apr 18 17:45  TUMMO There are two forms of Tummo—one t (A)<br> 833859956464680960  Apr 18 17:49  Breathhold Walking Anders Olsson uses th<br> 833859957852995584  Apr 18 18:00  Close the mouth and inhale quietly throu (A)<br><br>$ supernote an 833859955948781568<br>&gt; Any gum chewing can strengthen the jaw and stimulate stem cell growth, but<br>harder textured varieties offer a more vigorous workout. <br><br>seems ridiculous — so do night guards do the same?</pre><p><strong>Extract inbox highlights</strong> from notes using a custom VLM prompt:</p><pre>$ supernote nb 1254579462477971456 --prompt &quot;Whenever you see a line beginning with → or ☐ or ☑, transcribe the rest of the line (including any continuation onto subsequent lines) and include the leading → or ☐ or ☑ at the start of the output line. Emit nothing for any other lines.&quot;<br>☐ Finish supernote - cli blog<br>☐ Efoil: block hole w/ Epoxy (and fix up wiring)<br>☐ Canada passport renew &amp; send.<br>☐ Summarize Tillich</pre><p>Check it out <a href="https://github.com/borismus/supernote-cli">on GitHub</a>.</p><p>The scarce input in the AI era isn’t prompts or models; it’s focused thinking. Synthesis gets cheaper each year, while deep work gets harder. The pen-and-paper thing isn’t nostalgia but a line of defense.</p><p><em>Originally published at </em><a href="https://smus.com/notes/2026/supernote-cli-pen-paper-and-a-pipe/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44def6aa1f5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alloy: a local-first AI workbench]]></title>
            <link>https://borismus.medium.com/alloy-a-local-first-ai-workbench-8cd9fb5fe5ae?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cd9fb5fe5ae</guid>
            <category><![CDATA[local-first-software]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Sat, 03 Oct 2026 23:47:26 GMT</pubDate>
            <atom:updated>2026-10-03T23:47:26.571Z</atom:updated>
            <content:encoded><![CDATA[<p>AI model vendors usually ship their own desktop apps, and aside from coding agents, these tend to be the ways in which I interface with AIs the most. But these apps are sticky by design: each vendor wants you to stay with them and continue to use them. As a result, I tend to have very lumpy usage patterns, focusing on one model at a time to the detriment of the others. This leads to an incomplete understanding of the ever-shifting AI landscape, with models constantly leapfrogging one another on the leaderboards.</p><p>To remedy this, I built my own web-based desktop app which is model agnostic. Beyond easy access to Gemini, Grok, Claude, and GPT, I had a few additional goals in mind:</p><ol><li>Local-first: all conversations are stored as plaintext</li><li>Build an orchestrator to gain intuition around agents, tools, skills, etc,</li><li>Play around with new interaction patterns beyond the usual chat-and-wait</li></ol><p>The result of this is Alloy, which I began in December and have used daily since then. With it, I’ve been able to more easily see how good each model is at using tools, agent skills, etc. Alloy also serves as a personal playground for experimenting with new interaction patterns: freewriting with a quiet AI collaborator, delegating to sub-agents mid-conversation, and setting up triggers that monitor the world on your behalf.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gtt_Vj2AjUp2R_VQ.png" /></figure><p>When you start a chat, Alloy picks one of your favorite models from a list you configure. As you talk with a model, the exchange is appended to the file that backs it. Files have the benefit of being local, permanent, and searchable with any tool you already use. You can always switch models even mid-conversation.</p><p>The orchestrator has access to built-in tools and any custom skills you’ve added to your vault. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9lwJDt86k1muW3LO.png" /></figure><p>Here I asked Alloy to “critique this writing using 3 top-tier AI models from different providers.” It loaded the writing-critique skill, spawned Anthropic Opus, OpenAI o3, and Google Gemini 2.5 Pro as parallel sub-agents, each critiquing an early draft of this post independently. The orchestrating agent then synthesized their responses into a single structured critique.</p><p><strong>File over app</strong>: conversations, notes, triggers, and memory are all stored locally as plain text files. Everything is inspectable, searchable, and portable. The app still feels like ChatGPT or Claude Desktop though. Sync these files however you want: iCloud Drive, Google Drive, or whatever. Since I’m a heavy Obsidian user, I personally keep my Alloy directory as a subdirectory of my Obsidian vault and rely on Obsidian Sync.</p><p><strong>Transparent memory</strong>: Alloy eschews hidden memory vectors for a simple, explicit approach: memory.md. This markdown file is a small, curated, always-on prompt piece injected into the system prompt. There you might include relevant bits about your identity and preferences. You have full and transparent control over what information is sent, not a mercurial UI buried in settings that the lab&#39;s apps provide.</p><pre># Preferences <br>- Offer multiple viewpoints or solutions. <br>- Leave sycophancy at the door. <br><br># Personal <br>- Use Seattle for local context. <br>- Personal website/blog: https://smus.com</pre><p><strong>Deliberately constrained</strong>: Alloy can only read from and write to its own vault. It cannot run arbitrary code or scripts. Destructive writes require your confirmation. This is a design choice, not a limitation — the goal is a tool where it’s safe to let AI manage your notes, memory, and triggers without worrying about the blast radius (looking at you, <a href="https://openclaw.ai/">OpenClaw</a>).</p><p><strong>Still mostly online</strong>: Ultimately Alloy is about reclaiming control over AI conversations to the extent possible. It does support calling local LLMs via ollama, but so far they are too slow on my M4 Mac Mini and too limited in tool use to be practical. So despite all conversations being stored locally, the bulk of the actual AI interactions continues to be through APIs to the cloud hosted models.</p><p>Beyond standard chatting, Alloy has become my test bed for some practical AI experiments. Here are a few directions I’ve been excited about.</p><h3>Living memory via a note corpus</h3><p>I’ve noticed in my own usage of ChatGPT style assistants that there can be many threads about the same project. Manually grouping these conversations has always felt unsatisfying to me.</p><p>Rather than ask users to group conversations manually or even re-group them with the help of AI, I am trying something new. Alloy also includes an AI note corpus. The idea here is that the agent can read and write this corpus as you have conversations. Each note links back to the conversations that shaped it, so you always have provenance. This is still early, but I’m excited about the feedback loop that could result:</p><ol><li><strong>Extraction</strong>: Talk about a project or topic in any thread</li><li><strong>Consolidation</strong>: AI builds up a robust note on that topic</li><li><strong>Recall</strong>: Subsequent conversations about related projects become better</li></ol><h3>The home thread</h3><p>Most AI chat apps create a new conversation for every topic. Alloy has that too, but it also has a single persistent thread that’s always open — think of it as a command line for your AI.</p><p>Standard chat forces you to wait. While the response streams, you’re blocked from sending new messages. In the home thread, you can always send a new message. Responses are brief, direct, and conversational. But when something substantial needs to get done, the orchestrating agent delegates to a sub-agent. This interaction feels right for quick back-and-forth or rapid braindumping when thoughts aren’t quite fully formed yet.</p><h3>Riff mode for freewriting and dictation</h3><p>Sometimes you might not want to chat at all, but just capture semi-formed thoughts. Riff mode is more like thinking with a quiet collaborator sitting next to you, organizing your whiteboard while you keep talking.</p><p>You open a blank riff and just start typing or speaking — whatever comes to mind. As you pause, the AI quietly crystallizes your raw input into an organized draft. It weaves your scattered thoughts into something coherent, fills in gaps when you trail off (“something like…” or “I can’t remember the name of…”), and answers questions you pose mid-thought. It marks its contributions in <em>italics</em> so you always know what’s yours.</p><h3>Conditional triggers for monitoring topics of interest</h3><p>Most AI interactions require you to initiate, but some tasks require vigilance, not just intelligence. Conditional triggers turn the AI into a proactive sentry that only speaks when it has something to say.</p><p>You set up a “trigger” with a natural language condition like “Check prediction markets daily and notify me if the odds of regime collapse in Iran change by more than 5%.” The system runs silently, monitoring the probabilities without your input. It saves you from constantly refreshing charts or doom-scrolling, acting instead as a dedicated analyst who only interrupts you when the signal becomes significant.</p><p>I’ve been using Alloy daily for a couple of months. It’s early and rough, but I think it’s ready for you to kick the tires. It’s a “bring your own API keys” tool — if you’re comfortable with that and care about owning your AI conversations, <a href="https://github.com/borismus/alloy/releases">give it a try</a>. The <a href="http://github.com/borismus/alloy">source is on GitHub</a>, and I’d love to <a href="/service/mailto:boris@smus.com">hear what you think</a>.</p><p><em>Originally published at </em><a href="https://smus.com/alloy-local-first-ai-workbench/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cd9fb5fe5ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local e-ink handwriting recognition with on-device VLMs]]></title>
            <link>https://borismus.medium.com/local-e-ink-handwriting-recognition-with-on-device-vlms-025cfcb52d28?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/025cfcb52d28</guid>
            <category><![CDATA[e-ink]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[handwriting-recognition]]></category>
            <category><![CDATA[supernote]]></category>
            <category><![CDATA[ocr]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Tue, 23 Dec 2025 18:22:14 GMT</pubDate>
            <atom:updated>2025-12-23T18:22:14.144Z</atom:updated>
            <content:encoded><![CDATA[<p>For decades, I’ve carried a Field Notes notebook and a pen. I mainly used them to capture ideas on the go, but it would also be great to sketch out a diagram, or to journal a little bit, especially while traveling.</p><p>Fast note-taking via the iPhone’s action button has alleviated a lot of my need for quickly capturing ideas. But nothing can replace pen and paper for long form stream-of-consciousness writing or diagramming.</p><p>I wanted to give my writing a digital life alongside the rest of my notes. So a year ago, I bought an A5 e-ink writer called Supernote. I really like it so far: it’s a good size, input latency is reasonable, and the overall writing experience is fine. The device provides real-time text recognition on-device and a modest cloud syncing service. I’ve been using their unofficial API to sync notes and bring them into my Obsidian inbox. But then something happened…</p><ol><li>I wrote quite a bit on the Supernote during my trip to Patagonia, and came back to realize that the on-device recognizer has a pretty high error rate.</li><li>Supernote Cloud enabled mandatory two-factor authentication. This broke my programmatic access to their cloud. So much for <a href="https://smus.com/notes/2025/the-pursuit-of-frictionless-capture/#:~:text=Supernote%20has%20a%20cloud%20sync%20service%20with%20unofficial%20but%20stable%20APIs.%20Notes%20are%20saved%20in%20a%20proprietary%20but%20reverse%2Dengineered%20.note%20file%2C%20see%20unofficial%20python%20library.">“unofficial but stable”</a>.</li><li>The quality of handwriting recognition using Vision Language Models (VLMs) has skyrocketed, including for models capable of running locally on a modern Mac.</li></ol><p>Just three weeks ago (November 2025), someone released <a href="https://www.ocrarena.ai/">OCR Arena</a>, a knockoff of <a href="https://lmarena.ai/">LM Arena</a> but for comparing VLMs in their OCR abilities. AI is moving so quickly, as soon as you have a cool idea, someone’s just done it!</p><p>OCR Arena compares the latest crop of VLMs but obviously does not include Supernote’s on-device handwriting recognizer, nor does it include previous generation OCR libraries. I wanted a more broad-based multi-way comparison specific to my scenario. How well do the various approaches work on my handwriting on this device? I set out to compare a few different approaches to this problem:</p><ul><li>A state-of-the-art cloud model (e.g. Anthropic’s claude-opus-4-5-20251101)</li><li>A state-of-the-art open weights VLM capable of running on my stock Mac Mini M4 (e.g. qwen3-vl:8b)</li><li>The real-time handwriting recognition engine which runs on the Supernote e-ink writer device.</li><li>Traditional, pre-LLM OCR systems. (I realize tesseract is not the best for this, but still an interesting baseline.)</li></ul><p>I put together a tiny evaluation set which included five handwritten pages from my Patagonian journal, and manually transcribed them for ground truth. Claude and I built a small evaluation script for transcribing the examples in the dataset and comparing Word Error Rates (WER) across them.</p><p>To make this concrete, here’s a page I wrote from my travel log and the ground truth I manually transcribed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yp5IV4Humq6W3CpW.png" /></figure><p>And here’s a side-by-side comparison of the transcript produced by qwen3-vl:8b, Supernote’s on-device model, and tesseract:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mUgNAbYKYsCSAoqK.png" /></figure><p>As you can see, tesseract produces terrible output. This is not surprising, since it’s not trained for handwritten text but printed text. To do this right, you’d need to retrain it with a corpus of handwritten documents, but I still wanted to include it as a baseline.</p><p>Supernote’s real-time on-device handwriting recognizer does reasonably well sometimes, but other times goes off the rails, as you can see in the example above. For the curious, code for these benchmark scripts <a href="https://github.com/borismus/supernote-ocr-bench">is available</a>.</p><p>Calculating word error rates and aggregating all the pages’ WER scores resulted in the following table:</p><p>Surprisingly, the prompt given to qwen3-vl:8b matters quite a bit. My initial prompt was:</p><pre>prompt = &quot;Extract all text from this handwritten note. Return only the transcribed text in markdown format, without any additional commentary or formatting.&quot;</pre><p>With this basic prompt, qwen3-vl:8b would get stuck in a thinking loop and return an empty transcript at all about a quarter of the time. This failure mode would take hundreds of seconds and eventually end up repeating the same word dozens of times and then timing out.</p><p>After some experimentation with prompt and other parameters, I found empirically that updating the prompt seemed to reduce this rate to almost zero:</p><pre>prompt = &quot;&quot;&quot;You are an OCR engine, not a writing assistant. Task: - Read the handwritten note in the image. - Output the exact transcription of the text as plain markdown. Critical constraints: - Do NOT explain what you are doing. - Do NOT think step-by-step. - Do NOT describe, analyze, or comment on the note. - Do NOT use phrases like &quot;let&#39;s&quot;, &quot;wait&quot;, &quot;first line&quot;, &quot;next line&quot;, &quot;line X&quot;, &quot;Got it&quot;, or &quot;step by step&quot;. - Do NOT mention spellings or say how words are written. - Do NOT repeat any single word more than twice in a row. - If you notice yourself repeating a word or phrase, immediately stop and output your best single transcription of the whole note. - Your entire response must be ONLY the final transcription text, nothing else.&quot;&quot;&quot;</pre><p>Now that I had some metrics for word error rates across various approaches, I was able to make an informed decision driven by these considerations:</p><ul><li><strong>Privacy &gt; everything</strong>. Privacy preservation is extremely important to me as I’m journalling about potentially very personal topics. I don’t want my journal in anyone’s training set.</li><li><strong>Accuracy &gt; latency</strong>. Supernote’s on-device transcription is handy, but I’m mostly not in a hurry. I’d happily trade speed for higher quality transcription.</li></ul><p>I went with qwen3-vl:8b. It takes about a minute to process a page on my $450 Mac Mini. The word error rate is about 5% which is not far from the best possible results of 3% via Claude&#39;s high-end model.</p><p>Handwriting transcription is now implemented in my note-vault-utilities, a suite of scripts that <a href="https://smus.com/file-systems-for-thought/">garden my plaintext note corpus</a>. Cloud sync to desktop is achieved using the <a href="https://support.supernote.com/en_US/Tools-Features/supernote-partner-app-for-desktop">Supernote Mac app</a> and my daily note cron job picks up and transcribes any new handwritten notes into the Obsidian note corpus.</p><p><em>Originally published at </em><a href="https://smus.com/notes/2025/local-e-ink-handwriting-recognition-with-on-device-vlms/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=025cfcb52d28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The pursuit of frictionless capture]]></title>
            <link>https://borismus.medium.com/the-pursuit-of-frictionless-capture-fee544801669?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/fee544801669</guid>
            <category><![CDATA[notetaking]]></category>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[pkm]]></category>
            <category><![CDATA[tools-for-thought]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 16:42:26 GMT</pubDate>
            <atom:updated>2025-09-29T16:43:04.992Z</atom:updated>
            <content:encoded><![CDATA[<p>The most fruitful moments for contemplation are often the least conducive to capture. I’ve been working on reducing friction in capturing thoughts, feelings, and ideas wherever I may be and whatever I may be doing. At the same time, I want to stay in the moment and not get distracted by the act of capturing itself.</p><p>In addition, I want to retain control over all of this data. It should be stored in human-friendly plaintext format. It should touch as few servers as possible and come to rest on a surface that I control.</p><p>I am excited about all three of the capture methods I’m about to share with you. Three questions to whet your appetite:</p><ol><li>What if you could freewrite with pen on paper and have the salient bits magically show up in my digital note corpus?</li><li>What if you could use your locked smartphone to type a note without ever being distracted by its contents?</li><li>What if you could dictate notes while walking, running, or riding a bike <em>without even requiring a smartphone</em>?</li></ol><p>To motivate this, watch this quick summary of the scenarios above, and read on for some tech details for how it all works.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7B-42UeuIps%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7B-42UeuIps&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7B-42UeuIps%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7054dd8da035ddef73096e2ec7f68ef6/href">https://medium.com/media/7054dd8da035ddef73096e2ec7f68ef6/href</a></iframe><h3>Capture fragments from e-ink notebook</h3><p><strong>Vision: Freewrite with pen on paper and have the salient bits magically show up in my digital note corpus.</strong></p><p>I’ve been using an e-ink writer called the Supernote Nomad for the last half year. It’s light, portable, and almost feels like a reasonable replacement for paper. Additionally, the <a href="https://www.staedtler.com/us/en/discover/noris-digital/">Staedtler Digital pencil</a> feels nice in the hand and has a soft digital eraser which feels great. Most of my use is in meetings or at appointments where I want to be fully present.</p><p>Supernote has a cloud sync service with <a href="https://github.com/bwhitman/supernote-cloud-python">unofficial but stable APIs</a>. Notes are saved in a proprietary but reverse-engineered .note file, see <a href="https://github.com/jya-dev/supernote-tool">unofficial python library</a>. Supernote&#39;s built-in on-device transcription does a good job transforming sketches of symbols into unicode characters. For example, a hand drawn rightward arrow is reliably transcribed into &quot;→&quot; and a square box is reliably transcribed as &quot;☐&quot;. Relying on this, my conversion script scours the transcript for lines that begin with special characters. &quot;→&quot; is an indication that the following line should be captured in a note. &quot;☐&quot; indicates that what follows should be captured as a TODO item. These items are then saved in the daily inbox YYYY-MM-DD.md file along with a link to the note itself, which can be rendered by an <a href="https://github.com/philips/supernote-obsidian-plugin">Obsidian plugin</a>.</p><h3>Capture from phone</h3><p><strong>Vision: Press a locked iPhone’s action button and type a note without ever being distracted by your phone’s contents.</strong></p><p>My Apple Shortcut <a href="https://www.icloud.com/shortcuts/a0046291c36e4ff9b1e9e508a50af1be">Type Note</a> appends typed text to a YYYY-MM-DD.md file saved to a directory stored on iCloud Drive. This directory is then symlinked into my Obsidian vault. This works quite well in practice, with Obsidian Sync seeming to have no issues integrating this into the rest of the vault. Another Shortcut <a href="https://www.icloud.com/shortcuts/590b74fe358b42eaa919755a9783af34">Dictate Note</a>, does the same thing but with voice input. A third shortcut I call &quot;Magic Mode&quot; runs when the iPhone&#39;s action button is pressed, and decides which of the two to call depending on context.</p><p>This works super well. I’m pleased that Apple has fixed longstanding issues with the “Ask for text” action in Shortcuts, which, when activated from a lock screen, used to dismiss far too quickly despite the user actively typing on the onscreen keyboard.</p><h3>Dictation from watch</h3><p><strong>Vision: Dictate notes while walking, running, or riding a bike <em>without requiring a smartphone</em>. You are <em>never cut off</em> and the result is a high quality, timestamped transcript.</strong></p><p>My current solution is to use my custom <a href="https://www.icloud.com/shortcuts/6668b84b335949cd8ad82c5f86cbea79">Record Note</a> shortcut which records audio and saves it to an iCloud drive directory. An hourly script running on my trusty MacMini looks for new recordings and transcribes them with <a href="https://github.com/openai/whisper">whisper</a>. This approach is more reliable than other alternatives I have experimented with. Here are some failure modes I am now avoiding:</p><ul><li><strong>Dictation failures</strong>: sometimes using <a href="https://www.icloud.com/shortcuts/590b74fe358b42eaa919755a9783af34">Dictate Note</a> shortcut, dictation inexplicably fails, simply stating “dictation failed” with no ability to recover. This never happens with a recording.</li><li><strong>Minute limit</strong>: <a href="https://www.icloud.com/shortcuts/590b74fe358b42eaa919755a9783af34">Dictate Note</a> only works if the recording is under a minute. Beyond that, all transcription stops, and the result of dictation is an empty transcript.</li><li><strong>Location failures</strong>: <a href="https://www.icloud.com/shortcuts/590b74fe358b42eaa919755a9783af34">Dictate Note</a> does a location lookup. This often fails on the watch and is not critical for capturing the note itself. Record Note does not request location to increase reliability.</li><li><strong>Built-in mic</strong>: <a href="https://www.openplanetsoftware.com/just-press-record/">Just Press Record</a> is a well known Apple Watch compatible app that provides a dead simple recorder UI. It even works fully offline! But it has one major flaw, which is that it uses the default microphone attached to the watch, with no ability to override. This is a major problem while running or biking, where wind noise in the AirPods is overwhelmingly loud and ruins the quality of the recording. Meanwhile, the Record Note shortcut always uses the built-in microphone whether or not the AirPods are attached, producing far cleaner transcripts.</li></ul><p>Dictation failure is a catastrophic scenario. It’s also awkward as hell since as it turns out, you were just a crazy person talking to yourself all along. Rather than dictating in real-time, it’s totally fine to sacrifice latency in favor of reliability.</p><p>As you have seen, I currently have three note inboxes which originate from various devices. Obsidian itself also has an inbox which I tend to use if I am note taking while at the computer. The result is at least four inboxes all of which need to be consolidated. I’ll focus on this in the next post, as we’ll dive into my Friday afternoon note detangling ritual. This has become something I really look forward to as the week winds down.</p><p>This post serves as a snapshot-in-time of my current daily note taking rituals as of September 2025. My current setup is highly customized to my specific needs, but in the spirit of working “with the garage door up”, I thought it would be fun to share with you. Code available on request, email me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*PQwWlCex-weyZl2aFOr0zw.jpeg" /></figure><p><em>Originally published at </em><a href="https://smus.com/notes/2025/the-pursuit-of-frictionless-capture/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fee544801669" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unintended Consequences of False Equity]]></title>
            <link>https://borismus.medium.com/unintended-consequences-of-false-equity-aaeb5c666e57?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/aaeb5c666e57</guid>
            <category><![CDATA[equity]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[diversity-and-inclusion]]></category>
            <category><![CDATA[schools]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Tue, 29 Oct 2024 02:48:46 GMT</pubDate>
            <atom:updated>2024-10-29T02:48:46.766Z</atom:updated>
            <content:encoded><![CDATA[<p>What if our pursuit of educational equity is perpetuating inequality? Just as a rising tide lifts all boats, true educational equity must lift all students, regardless of background or ability.</p><p>In a school setting, equity refers not to the equality of outcomes, but to the equality of opportunities for all students to succeed. In recent years, Seattle Public Schools (SPS) has prioritized supporting students furthest from educational justice (FFEJ), a laudable goal that aims to address historical inequities. However, this narrow focus has unintended consequences. While healthy adults may not need medicine, children — even the brightest — require external stimulation to reach their full academic potential. Here are three negative secondary effects of an overemphasis on FFEJ students.</p><p>First is <strong>reduced enrollment and funding to the district</strong>. As high-end academics stagnate, displeased parents of capable children pull their children out of public school. This exodus is prominently displayed in Seattle, where a quarter of school-age children have now withdrawn from public school (up from one-sixth in 2022). Since Washington schools are funded per capita, less enrollment means less money funneling into the school district. From this purely self-interested financial perspective, neglecting capable students is like sawing off the branch you are sitting on. Furthermore, prioritizing FFEJ students also leads to <strong>less equity for highly capable students from poorer families</strong>. Private school is not cheap, and homeschooling is a luxury that only some families can afford, with at least one parent no longer able to work full-time. Thus, many families dissatisfied with public academics will not have the luxury of leaving the system. With the richest opting out of the public system, there are also less donations flowing to the Parent Teacher Student Associations (PTSA). With fewer capable students in the system and fewer exemplars of excellence for other students to follow, we will see a stronger correlation between socioeconomic status and academics. As a result, highly capable students forced to remain in the public system have reduced upward mobility.</p><p>A related side effect of this policy is an <strong>academic inversion in affluent neighborhoods</strong>, whose public schools become worse than those in poorer neighborhoods. When an upper-class neighborhood predominantly enrolls its children in private schools, the public school in the area will stagnate and cater to a smaller audience on the outer rim of the catchment area. In Seattle, this effect is on display in Laurelhurst.</p><p>Quoting <a href="https://mediadirectory.economist.com/people/charlotte-howard/">Charlotte Howard</a>, The Economist’s executive editor:</p><blockquote><em>There’s a real opportunity for leaders to talk not about coddling specific subgroups, but passing broad policies to help Americans writ large, regardless of their particular group to advance.</em></blockquote><p>What needs to happen is pretty simple: give each student what they need to thrive; actual equity. In SPS, this would mean bringing back programs for highly capable students. This does not need to be done in a separate school as in the recently terminated Highly Capable Cohort (HCC) program. But it should probably not be done in the same classrooms as the other students, with no additional resources for already overloaded teachers. Enriched and remedial classes within neighborhood schools are a tested and true solution to this problem. These new programs need measurable goals so that the district can begin <a href="https://smus.com/notes/2024/seattle-public-schools-accountability-problem/">to be accountable</a> in their pursuits of high-end excellence.</p><p>In conclusion, our current educational system’s emphasis on minimal standards directly impacts highly capable students and ultimately has an indirect negative impact on all students. It’s time for Seattle and other progressive school districts to reclaim “equity”, unlock the full potential of our children, and empower them to shape a brighter future for humanity.</p><p><em>Originally published at </em><a href="https://smus.com/notes/2024/unintended-consequences-of-false-equity/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aaeb5c666e57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Invention Cards enhanced by AI]]></title>
            <link>https://borismus.medium.com/invention-cards-enhanced-by-ai-be9032769367?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/be9032769367</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[history-of-technology]]></category>
            <category><![CDATA[history]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Thu, 30 May 2024 18:18:48 GMT</pubDate>
            <atom:updated>2024-05-30T18:18:48.606Z</atom:updated>
            <content:encoded><![CDATA[<p>I used GPT4’s multimodal features to accelerate and improve my recently rebooted <a href="https://smus.com/visual-chronology-science-discovery-v2/">Visual Chronology of Science &amp; Discovery</a> project now hosted at <a href="https://invention.cards/">https://invention.cards</a>. First I used image-to-text models to extract and structure content from pages of Asimov’s encyclopedia for <strong>accelerated data entry</strong>. Then I used text-to-image models to generate <strong>visually consistent imagery</strong> for each invention and discovery card. This post describes both workflows and my findings. Relevant source code is available in my <a href="https://github.com/borismus/asimov-gpt">asimov-gpt repository</a>.</p><h3>Accelerated data entry</h3><p>Invention Cards is based on Asimov’s encyclopedia, which is laid out like a newspaper: multiple sections of multi-column text:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zr8WbM_9_xlpl-v3.jpg" /></figure><p>The “In Addition” sections are interesting but do not directly map to a card in my chronology so should be skipped. The bi-fold above has four relevant sections which I’ve numbered in the image:</p><p>My initial idea was to completely transcribe the text using OCR, and then run an NLP extraction task on the extracted text. As I <a href="https://smus.com/visual-chronology-science-discovery/">wrote before</a>, off-the-shelf OCR solutions don’t work well for digitizing this sort of layout. Inevitably the flow of text gets mangled, and manual intervention is required.</p><p>Experimenting with the current crop of multimodal AI models, I was pleasantly surprised that I could skip the explicit OCR step and jump straight to extraction. gpt-4o happily follows detailed instructions given images of pages in the encyclopedia, extracting structured JSON for each heading on the page. Here&#39;s how my system works.</p><p>First I extract a page of interest from the 755-page PDF using the pypdf library. I send the page image to the OpenAI API along with the following prompt. I ironed out some kinks through trial and error:</p><blockquote><em>Summarize the inventions or discoveries on the following page. Please ignore sections entitled “In Addition”.</em></blockquote><blockquote><em>Provide the following information for each invention or discovery. Output should be described as an array of JSON objects with the following keys:</em></blockquote><blockquote><em>title: one to three words describing the invention or discovery.<br>year: the year the invention or discovery was made.<br>description: full description of the invention or discovery as close to the text as possible.<br>summary: two sentence summary without mentioning people or dates, focusing on what the invention or discovery is and does, and its impact.<br>description: full description of the invention or discovery.<br>inventor: full name of the inventor or discoverer.<br>location: in which country was the invention or discovery made? In the case of Great Britain or the United Kingdom, use “England”.<br>field: (one of Math, Science, Culture, War, General, Design, Geography, Space). Sub-fields can be indicated with a colon (e.g. “Science: Physics or Science: Biology”)<br>related: one or more related previous invention or discovery, separated by commas. If there are no related inventions or discoveries, use “”.</em></blockquote><blockquote><em>Remember to escape quotes in JSON strings.</em></blockquote><p>I found empirically that using JSON mode yielded worse results, but that not using JSON mode, I was able to easily extract JSON from the response. Sometimes it came in markdown-style code blocks, which are easy enough to strip. I then validate the resulting JSON object with <a href="https://pydantic.dev/">pydantic</a>, ensuring the types make sense and then sanity check it so that values are within ranges. If things do not validate, I automatically retry. Finally the script serializes each invention on the page in a Tab Separated Value (TSV) format, so that it can be copied from the terminal and pasted into my <a href="https://docs.google.com/spreadsheets/d/1hDNXas7DzwglB95HV2_2u1utWAwBZR2hQHlMPz-fj5A/edit#gid=1533876043">master spreadsheet</a>. Once entered into the spreadsheet, I can iterate on the content manually.</p><p>Here are my observations from extracting ~300 cards from Asimov’s encyclopedia using this AI-infused workflow:</p><ul><li><strong>Title</strong>: works quite well if the section is identified. About 2% of the sections are ignored.</li><li><strong>Year</strong>: works great with almost no errors.</li><li><strong>Summary</strong>: often wrong or incomplete, missing key details, especially when multiple inventions are mentioned under the same heading.</li><li><strong>Inventor</strong>: name works reasonably well when only one person is mentioned. A challenge arises when Asimov refers to previously mentioned people by just their last name. I haven’t yet built a system for resolving full names from last names yet.</li><li><strong>Location</strong>: works well too, but there are tricky cases like United Kingdom / Great Britain / England. Also expect failures when multiple places are mentioned in Asimov’s blurb.</li><li><strong>Field</strong>: sometimes wrong, often defaulting to “Science”. Sub-fields were also sometimes muddled or overly specific. One common confusion is the preference of “Science:Astronomy” over “Space”. To be honest, which field an invention or discovery belongs to is often fuzzy even for a human. The greatest discoveries span multiple fields or create new fields.</li><li><strong>Related</strong>: inventions are often missing, especially if none are explicitly mentioned in the originating text.</li></ul><h4>Aside on page boundaries</h4><p>Many of Asimov’s invention &amp; discovery descriptions span page boundaries. In the example above, Glider begins on 367 and wraps to the next page; Cholera begins on 368 and continues further.</p><p>I began working on an improved pipeline for handling page wrapped text, but realized that in practice, I needed to make revisions to many summarized card descriptions to make them more terse and informative than the AI generated summary. When I did this for inventions and discoveries that extended onto a second page, I ended up also manually incorporating content that was only available below the fold. This is not so bad because I’m curious about Asimov’s invention &amp; discovery stories, so end up at least skimming Asimov’s original text anyway.</p><p>To handle the page boundary robustly in the future, I’m imagining a two pass solution: 1) Extract the headings from the page. 2) Send a bi-fold image of the current and next page, and in the prompt, ask to provide summaries for the headings extracted from the first pass. How well gpt-4o or the next crop of models will handle multi-page images remains to be seen.</p><h4>Data workflow</h4><p>Here’s my overall data extraction workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1dRZO093I9YHbDia.png" /></figure><p>I run the script for a range of pages:</p><p>&gt; python3 src/extract_cards.py --from=388 --to=400</p><p>This outputs TSV for all entries on the page (terminal, window 2). I copy the TSV into the master spreadsheet (sheets, window 3), and then cross-reference the fields with the original text (preview, window 1).</p><p>Once ready, the spreadsheet is exported as TSV. I then load this TSV in the <a href="https://invention.cards/">https://invention.cards</a> site, and also use it as the source of truth for further processing. The inventions and discoveries on page 367–368 will be transformed into cards like this: <a href="https://invention.cards/glider">Glider</a>, <a href="https://invention.cards/cholera">Cholera</a>, <a href="https://invention.cards/kerosene">Kerosene</a>, and <a href="https://invention.cards/age-of-the-sun">Age of the sun</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ypIrsQ1deqh5fd_C.jpg" /></figure><p>At this point you might be wondering where the snazzy images came from. Read on to find out!</p><h3>Visually consistent imagery</h3><p>After I have a TSV file of inventions and discoveries from the AI-assisted process I described above, we can begin generating consistent imagery for these inventions. I use OpenAI’s dall-e-3 model to generate an image using the metadata provided. The f-string prompt looks something like this:</p><blockquote><em>Title: {invention.title}<br>Description: {invention.summary}<br>Category: {invention.field}<br>Year: {invention.year}<br>Person: {invention.inventor}</em></blockquote><blockquote><em>Generate vibrant art nouveau for the invention/discovery described above. The image should be a single object or scene that represents the invention/discovery.</em></blockquote><p>The results are stylistically consistent and visually pleasing. Sometimes things are a bit off though. Here are a few failure modes I’ve observed:</p><p>A. The image does not correspond to the factual details of the card (e.g. <a href="https://invention.cards/aerodynamics">Aerodynamics (1809)</a>, <a href="https://invention.cards/binary-system">Binary System (1700)</a> pictured):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-t2ZKTlD7jbus3Tv.jpg" /></figure><p>B. The card title has multiple meanings, and the image generator latches on to the wrong one (e.g. <a href="https://invention.cards/percussion">Percussion (1761)</a> and <a href="https://invention.cards/black-bodies">Black Bodies (1860)</a> pictured):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DRLQtl7ubtWOomAh.jpg" /></figure><p>C. The card is obscure and the image generator doesn’t really know what to do (e.g. <a href="https://invention.cards/discovery-of-lanthanum">Lanthanum (1839)</a>, <a href="https://invention.cards/molybdenum">Molybdenum (1778)</a>, and <a href="https://invention.cards/thallium">Thallium (1861)</a> pictured):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZV29vT902_k90O4f.jpg" /></figure><p>D. The card is abstract and the image generator doesn’t really know what to do (e.g. <a href="https://invention.cards/celsius">Celsius (1742)</a>, <a href="https://invention.cards/celluloid">Celluloid (1869)</a>, and <a href="https://invention.cards/positive-negative-charge">Positive negative charge (1733)</a> pictured):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7tgVej8Z1tUmkZym.jpg" /></figure><p>E. The card includes an object that the image generator struggles to generate, so creates a crazy version of the object (e.g. <a href="https://invention.cards/safety-pin">Safety Pin (1849)</a>, <a href="https://invention.cards/telescope">Telescope (1668)</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i6LVZIcNL0ahkgdM.jpg" /></figure><p>Two common and undesirable occurrences in these generated card images are people and text. The image often focuses on the visage of the inventor to the detriment of his actual invention. Sometimes it’s not the inventor or the discoverer, but a beautiful woman depicted in art nouveau style. The other failure mode is that the image contains unnecessary text. My favorite is when it is prominently misspelled, as in this set of attempts for <a href="https://invention.cards/carbon-dioxide">Carbon Dioxide (1754)</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u9_1z34qUILwLM9F.jpg" /></figure><p>First I tried threats to discourage these misbehaviors. Unfortunately threatening to kill dall-e-3&#39;s firstborn child triggers their AI safety subsystem. Then I tried cajoling. In the end I ended up pleading with very clear instructions. The prompt ends with:</p><blockquote><em>Please do not include any typography or text. Please do not draw any people.</em></blockquote><p>This helps but does not fully solve the problem. Roughly ~10% of the images still suffer from unnecessary people and text.</p><h3>Image workflow</h3><p>As a result of various problems in image generation, I can’t fully trust the system to do a good job. So I’ve become the human in this AI loop. Let me describe my workflow.</p><p>My script generates images for cards spanning a range of years, saving the corresponding images in a directory with the same filename as the card’s identifier</p><p>&gt; python3 src/gen_card_image.py --from_year=-100000000 --to_year=2000</p><p>After these images are generated, I review them. I leave the ones I like untouched and rename the duds, adding a suffix (e.g. carbon-dioxide.jpg becomes carbon-dioxide-1.jpg). After this pass, I run the script again which will ignore existing images unless forced. Now I have a smaller set of images to review and rename. Rinse and repeat. At the end of this iterative process, I have a list of canonical imagery for each invention.</p><p>I’d say that more than half of the card images generated on the first pass are good enough for my bar. Some images end up requiring multiple generations and eventually produce something compelling. Other cards fail completely and require workarounds. In some cases, like <a href="https://invention.cards/morphine">Morphine (1805)</a>, dall-e-3 simply refused to generate any image because it violated their policies. Winners don&#39;t use drugs!</p><p>In other cases, dall-e-3 gets confused by Asimov&#39;s terms and generates consistently wonky images. In these rare cases, I took liberties and retitled the card. Though a separture from the original text, the new names are less confusing to both the image generator and to the general public. I renamed <a href="https://invention.cards/black-bodies">Black Bodies (1860)</a> to &quot;Blackbody Radiation&quot; to be able to generate images without a black human body. Similarly, I renamed <a href="https://invention.cards/percussion">Percussion (1761)</a>, to &quot;Medical Percussion&quot; to avoid spurious musical instruments.</p><p><em>Originally published at </em><a href="https://smus.com/invention-cards-ai/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be9032769367" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visual Chronology of Science & Discovery 2.0]]></title>
            <link>https://borismus.medium.com/visual-chronology-of-science-discovery-2-0-30183f99b309?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/30183f99b309</guid>
            <category><![CDATA[history-of-technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[science-education]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Thu, 16 May 2024 15:47:32 GMT</pubDate>
            <atom:updated>2024-05-16T15:47:32.014Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently resumed my Visual Chronology of Science &amp; Discovery <a href="https://smus.com/visual-chronology-science-discovery/">project</a>, a Civilization-inspired tech tree but for the real life history of science. The content is grounded in Asimov’s <a href="https://smus.com/books/asimovs-chronology-of-science-and-discovery/">book</a> with some flourishes added, courtesy of yours truly. Four years since starting the project I’m sharing a significant update. Here’s a quick overview of the new viz:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTHE4xSGTQzc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTHE4xSGTQzc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTHE4xSGTQzc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/180b5ff4ad0e109e23baeee263138856/href">https://medium.com/media/180b5ff4ad0e109e23baeee263138856/href</a></iframe><p>And some major improvements vis-à-vis the original version:</p><ul><li>Revised the visual design of the cards, each of which now features an illustration.</li><li>Improved the UI of the overall system including search and field filtering and a separate timeline component.</li><li>Addressed table-stakes like panning, zooming, and mobile web support.</li><li>Extended the timeline from 1700 to 1850, more than doubling the number of cards to 597.</li></ul><p>In the remainder of this short post, I describe challenges I encountered along the way, and their solutions.</p><h3>Challenge 1: chronological graphs</h3><p>I <a href="https://smus.com/visual-chronology-science-discovery/">previously</a> described the challenge of combining a chronological timeline with a dependency graph. I had originally incorporated both in a single visualization:</p><blockquote><em>Technologies that enabled the focused card are shown to the left and technologies that the focused card enabled are shown to the right, along the x-axis. Chronologically previous and next entries are shown along the y-axis. f Overwhelmingly, people found this very confusing. Rather than attempt to wedge both dependencies and chronology into to the same view, I’ve split out the temporal bits into its own timeline component, which also allows search and filtering by field. This is much clearer!</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*Nw6DpWN5WYUSyHSy.png" /></figure><h3>Challenge 2: location, location, location?</h3><p>Asimov’s <a href="https://smus.com/books/asimovs-chronology-of-science-and-discovery/">book</a> is not particularly consistent with location names. England is often used interchangeably with Britain, Great Britain, the United Kingdom, etc. As I create a more structured version of his encyclopedia, I feel the need to rectify these inconsistencies.</p><p>One option is to provide the temporally correct name of the originating place. For example, if an invention was created on the territory of modern day Crimea in the 7th century, perhaps it should be called a Greek invention. In the 13th century it ought to be labeled Mongolian. In the 18th century it ought to be labeled Russian. In 2000, it ought to be Ukrainian. Another option is to use the modern name for the territory, regardless of its historically accurate name.</p><p>I think I’ve arrived at a compromise with an arbitrary dividing line between the ancient world and the modern world.</p><p>For ancient inventions and discoveries, I use the name of the empire or civilization broadly speaking, following Asimov’s lead. For example, <a href="https://borismus.github.io/asimov/#library">Libraries</a> are attributed to Assyria in 640 BCE, and <a href="https://borismus.github.io/asimov/#arch">Architectural arches</a> to the Etruscans in 750 BCE. Assyria and Etruria are no longer part of the political landscape, but I still think this makes a lot of sense.</p><p>For inventions in the last half-millennium is to use the modern country’s name for the place where the discovery or invention occurred. Attempting to provide a historically accurate place name is difficult and potentially confusing. Also, historically accurate place names would often change depending on political events irrelevant to the arc of invention and discovery, making it hard to trace golden ages to a meaningful place.</p><p>Conveniently this approach allows for a distinction between English and Scottish inventors, so subsequent analysis should be able to pinpoint the Scottish Enlightenment, for example.</p><h3>Challenge 3: improved design</h3><p>I was originally inspired by a set of physical card games and technology trees from computer games (see <a href="https://www.are.na/boris-smus/tech-tree-cards">are.na</a>), and finally decided to bite the bullet and clone the original Magic the Gathering card design. I also managed to solve the problem of consistent imagery across whole collection.</p><h3>Challenge 4: consistent imagery</h3><p>One of the biggest limitations of the first version of this project was to create a consistent visual design for this set of cards.</p><blockquote><em>While you can find reams of royalty free images, […] finding a set that is visually consistent was tough.</em></blockquote><p>Four years ago I speculated:</p><blockquote><em>Theoretically a style transfer model might be able to convert them all to a consistent look.</em></blockquote><p>Modern AI models helped me solve this problem by generating a set of imagery using a textual prompt to guide the content and ensure a consistent style. I’m pleased with the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1S1-nmX5f5oeqshw.jpg" /></figure><p>Compare to where I began several years ago:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fXJZXEnKU-jb6mMe.jpg" /></figure><h3>Challenge 5: data entry</h3><p>It was difficult to fully digitize the original book because of its multi-columnar and sectional layout. As I wrote <a href="https://smus.com/visual-chronology-science-discovery/">before</a>:</p><blockquote><em>optical character recognition (OCR) is not easy if your content is laid out like a newspaper […] Cloud Vision and tesseract don’t do well on complex text flows, often failing to recognize section boundaries.</em></blockquote><p>With modern AI, I’ve been able to extract content from the pages of the book into a structured output. This has been a boon to my productivity, and although I still verify the output of the model, my speed of data entry has greatly accelerated.</p><p>This project continues to be backed by a <a href="https://docs.google.com/spreadsheets/d/1hDNXas7DzwglB95HV2_2u1utWAwBZR2hQHlMPz-fj5A/edit#gid=0">spreadsheet</a> which has now grown to nearly 600 entries.</p><h3>Open challenge: dependencies</h3><p>One of my goals with the visual chronology of science &amp; discovery was to explore how science builds on itself and to articulate the way in which one invention sometimes serendipitously leads to the next.</p><p>Asimov sometimes cites previous inventions in his encyclopedic entries, but truly connecting the dots is left as an exercise to the reader. To capture this relationship, each entry has a list of dependencies that led to its invention or discovery. This list of dependencies is quite subjective, and produced mostly by my judgement.</p><p>There are a number of ways of interpreting this parent-child relationship between cards:</p><ol><li>Parent card loosely or directly inspired the person(s) to come up with their invention or discovery. Example: <a href="https://borismus.github.io/asimov/#paved-road">Paved roads</a> were improved upon with <a href="https://borismus.github.io/asimov/#macadamized-roads">Macadamized roads</a>.</li><li>Parent card was necessary to enable the child invention or discovery. Example: <a href="https://borismus.github.io/asimov/#improved-steam-engine">Improved steam engine</a> was necessary to build the <a href="https://borismus.github.io/asimov/#steamboat">Steamboat</a>. <a href="https://borismus.github.io/asimov/#screw-propeller">Screw propeller</a> was necessary to build <a href="https://borismus.github.io/asimov/#transatlantic-liner">Transatlantic liner</a>.</li><li>Parent represents a resource that became scarce, causing a child invention to be invented. Example: <a href="https://borismus.github.io/asimov/#bronze">Bronze</a> required tin, which was rare, causing Phoenicians to venture into the Atlantic and discover <a href="https://borismus.github.io/asimov/#ocean-navigation">Ocean navigation</a>.</li></ol><p>I think this is the shakiest part of the data that powers the visualization. If you are an expert in the history of science, let’s talk. I’d love to discuss where to draw the line. Pun intended.</p><h3>What’s next for this project?</h3><ol><li>I’ve only digitized half of the book! Another 600 cards spanning from 1850 to 1993 have yet to be added. I’ll continue chipping away at this.</li><li>Much of my recent progress on this project is due to hot new AI models. I’ll have more details to share about that in a future post.</li><li>In parallel, I want to explore a printed version of the same cards. Perhaps 1200 cards is an unwieldy number to hold in hand, but I can easily see the appeal of a deck containing a subset of the cards. It’s easy to imagine a deck of space-related discoveries, or one centered around chemistry, mechanical engineering, or any sufficiently deep scientific field. Is there a game to be played in there?</li></ol><p>Please <a href="https://borismus.github.io/asimov">try it out</a> for yourself and <a href="/service/mailto:boris@smus.com">send me</a> your feedback. I’m especially interested in hearing from you if you are a print designer (or an aspiring print designer) and want to collaborate on a physical version of this project! I’d also love to chat about the challenge of determining the dependency structure. On that note, if you find a mistake in the content, field, deps, or visual presentation, you can also file a <a href="https://github.com/borismus/asimov/issues/new">GitHub issue</a>, which is also linked to from the top right icon of any card. See you in the captivating world of invention &amp; discovery!</p><p><em>Originally published at </em><a href="https://smus.com/visual-chronology-science-discovery-v2/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30183f99b309" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tools for Thinking in Systems]]></title>
            <link>https://borismus.medium.com/tools-for-thinking-in-systems-cf68a4175c5e?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf68a4175c5e</guid>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[notes]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 18:19:39 GMT</pubDate>
            <atom:updated>2023-10-18T18:19:39.255Z</atom:updated>
            <content:encoded><![CDATA[<p>With this post I aim to synthesize some ideas from the Tools for Thought movement (e.g. Roam) with Systems Thinking (e.g. feedback loops). The result, as advertised in the title, is a tool for helping people think in systems. Let me first explain what I’m talking about, then walk you through some design considerations, and finally show you a prototype which takes a description of a system and converts it into causal loop diagram. Imagine if every news article included a little visual explainer to help you understand the story better.</p><p>If you’re impatient (who can blame you?), here’s a quick demo:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQLok-ubBJl4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQLok-ubBJl4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQLok-ubBJl4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a3970e0f0e73bac85fa5db09659a716/href">https://medium.com/media/5a3970e0f0e73bac85fa5db09659a716/href</a></iframe><h3>Systems and causal loop diagrams</h3><p>A couple of years ago, I read and was inspired by <a href="https://smus.com/books/thinking-in-systems-a-primer/">Thinking in Systems: A Primer</a> by Donella Meadows. One of the tools presented in Meadows’ book are causal loop diagrams. Here’s a simple riff on a classic causal loop diagram to help us grapple with chicken populations. More chickens mean more eggs, and more eggs mean more chickens (a reinforcing loop). As we know, chickens have a certain propensity for crossing the road, and the more chickens, the more chaotic the road crossings with unfortunate consequences for their overall population (a balancing loop).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dVxI3TP4ilSH3E3H.png" /></figure><p>Causal loop diagrams are a way to visually represent a complex system. They can be a good visual summary, giving us a sense of factors and feedback loops that relate to a topic.</p><p>Over the last couple of years I got a bit nerdy about the topic and “modeled” some of my own systems based on some of my reading. For example, based on a series of lectures about the Middle Ages, I sketched out causal loop diagrams that illustrated some secular trends. These can be found in the <a href="https://z3.ca/Medieval_feedback_loops">public version of my note corpus</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tPeAqxPrmkhqeC2F.png" /></figure><p>I found the process insightful as a way to process a complex topic, and the results to be interesting to share and generalize from. However, the process is time consuming and challenging. Could an AI help here?</p><p>There is no canonical mapping between a system and a corresponding causal loop diagram. In other words, there can be many diagrams which correspond to the same system, each emphasizing different aspects of it. Also, while they are well suited to illustrating fuzzy problems, causal loop diagrams are not well suited for rigorous system analysis.</p><h3>Tools for thought… for what?</h3><p>Popular Tools for Thought like Roam and Muse are general purpose organizational tools that help you to think and capture all kinds of thoughts. The new generation of these generic note-taking apps is now being imbued with AI. Some of these AI-powered features include:</p><ul><li>Transcribing audio and providing robust transcriptions (e.g. <a href="https://otter.ai/">Otter</a>)</li><li>Summarizing tracts of text into something more terse (e.g. <a href="https://reflect.app/">Reflect</a> and <a href="https://mem.ai/">Mem</a>)</li><li>Offering grammar correction or stylistic suggestions (e.g. <a href="https://languagetool.org/">LanguageTool</a>)</li></ul><p>What if you had a tool that helped you think or create in a specific domain? Here are a few examples I found compelling:</p><ul><li><a href="https://textfx.withgoogle.com/">TextFX</a> is a suite of “AI-powered tools for rappers, writers and wordsmiths”. These tools for thinking like a poet help people come up with similes, make a scene more unexpected, find alliterations, and fuse two concepts together.</li><li><a href="https://elicit.com/">Elicit</a> is a tool for thinking like a researcher, helping you quickly find prior art on a topic you might have no idea about, well outside your area of expertise.</li></ul><p>So now for the synthesis. What could a tool for thinking in systems look like?</p><h3>A tool for thinking in systems</h3><p>Given a description of a system, can an AI generate a causal loop diagram representation for it? Imagine if every news article included a little diagram explaining the systemic background for the news story.</p><p>The current state of AI is not quite ready to tackle this problem without human intervention. So I strove to create a tool that would help people co-create with the help of an LLM. The system’s vibe should not be <em>thanks for the text, here is the corresponding CLD</em>, but a metaphorical dialog: <strong>is this the CLD you are imagining?</strong> Or <strong>here’s a crazy take on this, WDYT?</strong>, with the person using the tool making the appropriate changes.</p><p>This dialectic approach has the potential to be useful, even if you are talking to yourself or an inanimate object. Studies have shown that conversations with yourself, as in <a href="https://www.youtube.com/watch?v=EAhtbxMV1lA">distanced self-talk</a> where we give ourselves advice by pretending like we’re advising a friend with the same issue, seem to help us get unstuck. In software engineering, <a href="https://rubberduckdebugging.com/">rubber duck debugging</a> is a method of debugging code by describing your approach to a lifeless rubber duck sitting on your desk.</p><h3>Principle 1: muses &gt; mentats</h3><p>I mentioned earlier that there is no canonical causal loop diagram for a given complex system. My goal was not to have a sequence of LLM invocations result in some perfect output, but to make headway on the cold start problem I found when attempting to model. You have a blank canvas; where should you begin? At what level of granularity should you be thinking about? What entities are at play? What relationships matter?</p><p>Thus, the first principle I’ve adopted is <strong>muses over mentats</strong> (from <a href="https://www.geoffreylitt.com/2023/02/26/llm-as-muse-not-oracle.html">ChatGPT as muse, not oracle</a>). Mentats in Dune are computer-like humans (long story), able to quickly compute and predict and produce factually correct results to help nobles rule. In contrast, Muses were inspirational Greek goddesses of literature, science and the arts, colloquially serving as someone’s source of artistic inspiration. In other words, correctness is optional; it is sufficient to be inspiring.</p><p>This principle jives well with the strengths and weaknesses of generative AI. LLMs can simulate reasoning and create a large volume of content quickly, but tend to hallucinate. Grounding LLMs in facts remains an open research problem.</p><h3>Principle 2: bikes &gt; genies</h3><p>How can the user feel like she is co-creating with a helpful partner, rather than putting her trust in a lifeless machine?</p><p>Bikes amplify our innate human abilities to get to where we’re going, while genies are simply teleport us to our destination. How do they do it? Nobody really knows! Hopefully you don’t run out of wishes, and hopefully the Genie doesn’t make any mistakes. And given that AIs make many mistakes, we need to give people a lot of control!</p><p>What’s needed here is not a genie to get you <em>the</em> answer. Instead, people need a tool to support getting to <em>their own</em> answer.</p><p>That these LLMs fib might not strictly be a bad thing. For example, if a generated diagram is blatantly incorrect, a user may be tempted to jump in and fix it, leading to new insights! After all, we all know what happens when <a href="https://xkcd.com/386/">someone is wrong on the internet</a>. The principle of bikes over genies ensures that people have the ability to make corrections where needed.</p><h3>Some examples</h3><p>Before diving into implementation details of my little tool, here are two examples of causal loop diagrams this system is capable of producing.</p><p><a href="https://borismus.github.io/llm4cld/?ents=Smoking%3BLungs%3BDiseases%3BDeath&amp;desc=Smoking+can+cause%C2%A0heart+attacks%2C+if+continued+for+years+of+using+tobacco+or+any+other+powders+used+for+smoking%2C+including+marijuana.+Smoking+can+cause+mental%2C+as+well+as+physical+health+problems%2C+such+as%C2%A0schizophrenia%2C%C2%A0depression%2C+mood+swings+and+many+others.+If+a+smoker+continues+smoking+for+years%2C+or+even+decades%2C+their+health+will+worsen.+Smoking+can+cause+coughing%2C+or%C2%A0inflammation%C2%A0%28swelling%29+of+the+throat+or+lungs.+The%C2%A0alveoli%C2%A0are+air+sacs+via+which+breathing+happens%2C+but+they+will+become+badly+damaged.+Once+breathed+in%2C+the+tobacco+smoke+travels+down+the%C2%A0trachea%C2%A0and+enters+the+lungs.+When+entering+the+body%2C+the+smoke+would+damage+the%C2%A0cilia%2C+which+are+millions+of+small+hairs+that+push+away%C2%A0mucus%C2%A0when+it+enters+the+body.+If+the+lungs+become+even+more+damaged%2C+smokers+may+not+be+able+to+breathe+on+their+own%2C+and+they+will+be+put+on+a+device+to+help+them+breathe+such+as+a%C2%A0ventilator.+In+some+cases%2C+smoking+causes+the+lining+of+the+lungs+to+become+inflamed%2C+making+the+smoker+unable+to+breathe%2C+which+leads+to+death.&amp;cgml=Smoking+o-%3E+Lungs+%2F%2F+Smoking+damages+alveoli+%5Cnand+cilia%2C+impairing+%5Cnlung+function.%0A%0ASmoking+--%3E+Diseases+%2F%2F+Smoking+damages+vital+%5Cnorgans%2C+leading+to+various+%5Cndiseases.%0A%0ASmoking+--%3E+Death+%2F%2F+Smoking+damages+lungs%2C+%5Cnimpairs+breathing%2C+and+%5Cnleads+to+death.%0A%0ADiseases+o-%3E+Lungs+%2F%2F+Smoking-induced+diseases+%5Cndamage+and+destroy+lung+%5Cntissue.%0A%0ADiseases+--%3E+Death+%2F%2F+Diseases+damage+body+%5Cnsystems%2C+leading+to+diminished+%5Cnfunction+and+death.%0A%0A">This diagram</a> is generated from a paragraph on the Simple Wikipedia page <a href="https://simple.wikipedia.org/wiki/Tobacco_smoking">“Tobacco Smoking”</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lswR88iIYAoQ-v_e.png" /></figure><p><a href="https://borismus.github.io/llm4cld/?ents=Knights%3BHeavier+Armor%3BStronger+Horses%3BFarmer+Prosperity&amp;desc=%C2%A0+%C2%A0+-+During+the+middle+ages%2C+knights+emerged+as+a+new+superweapon.%0A%C2%A0+%C2%A0+-+More+knights+meant+more+demand+for+horses.%0A%C2%A0+%C2%A0+-+And+they+needed+to+be+strong+to+carry+the+weight+of+the+knights+and+their+armor.%0A%C2%A0+%C2%A0+-+As+horses+got+stronger%2C+knight+armor+got+heavier+too%21%0A%C2%A0+%C2%A0+-+These+stronger+war+horses+interbred+with+regular+farm+horses%2C+and+also+more+horses+were+available+for+farmers%0A%C2%A0+%C2%A0+-+Improved+horse+collars+made+horses+pull+with+5x+more+power%2C+and+ultimately+displaced+oxen.%0A%C2%A0+%C2%A0+-+Now+farmers+had+horses+instead+of+oxen+and+could+also+ride+them+to+the+markets+to+sell+their+wares+on+their+own.&amp;cgml=Knights+--%3E+Heavier+Armor+%2F%2F+Knights+required+stronger+%5Cnhorses%2C+enabling+heavier+%5Cnarmor.%0A%0AKnights+--%3E+Stronger+Horses+%2F%2F+Knights%27+demand+for+stronger+%5Cnhorses+drove+selective+%5Cnbreeding.%0A%0AKnights+--%3E+Farmer+Prosperity+%2F%2F+More+knights+increased+%5Cndemand+for+stronger+horses%2C+%5Cnbenefiting+farmers.%0A%0AHeavier+Armor+--%3E+Stronger+Horses+%2F%2F+Heavier+armor+necessitates+%5Cnstronger+horses+for+effective+%5Cntransportation.%0A%0AStronger+Horses+--%3E+Heavier+Armor+%2F%2F+Stronger+horses+could+%5Cncarry+heavier+armored+%5Cnknights.%0A%0AStronger+Horses+--%3E+Farmer+Prosperity+%2F%2F+Stronger+horses+increased+%5Cnfarming+efficiency+and+%5Cnmarket+accessibility.%0A%0A">This diagram</a> is generated from some of my observations about medieval history:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WYFhaHOBuJ9J2CHC.png" /></figure><p>A few words about these diagrams:</p><ul><li>Boxes are entities involved in the system.</li><li>Solid edges labeled with +&#39;s are direct relations (more Source causes† more Target).</li><li>Dashed arrows labeled with -&#39;s are inverse relations (more Source causes† less Target).</li><li>Edge labels provide an explanation for why the relationship exists.</li><li>Bolded “R” and “B” in labels indicate a Reinforcing and Balancing feedback loop, respectively.</li></ul><p>I go over both in the <a href="https://www.youtube.com/watch?v=QLok-ubBJl4">demo video</a>.</p><h3>Prototype implementation details</h3><p>At a high level, the prototype works as follows:</p><ol><li>Extract entities based on description</li><li>Check if entities are directly related (e.g. mo money ➡️ mo problems)</li><li>Check if entities are inversely related (e.g. more children ➡️ less money)</li><li>For every relation, come up with a terse explanation</li><li>Generate the corresponding causal loop diagram</li><li>Label any reinforcing or balancing feedback loops</li></ol><h3>Entity extraction (Step 1)</h3><p>Entity extraction is done through an LLM, not because that is a good idea, but because it was expedient from a prototyping perspective. This is relatively uninteresting, and I used a straightforward prompt template like this:</p><pre>Text: ${groundingText} The following ${entityCount} entities appear in the text above: -</pre><p>Here, groundingText refers to the system description and entityCount is a configurable number.</p><h3>Causal entity relationships (Steps 2–3)</h3><p>I took a brute force approach to <strong>entity-to-entity causal relationships</strong>. Once all entities are available, we check each ordered entity pair to see if and how they are related. For each pair, I use the same LLM to check if there is a direct relation with the following prompt template:</p><pre>Text: ${groundingText} The text above suggests that more ${entity1} causes more ${entity2}. Answer one of &quot;true&quot; or &quot;false&quot;.</pre><p>To check for an inverse relation, I replaced “more” in the prompt above with “less”.</p><p>Now whether the model obeyed the request and produced a single word response “true” or “false” is another question. The state-of-the-art OpenAI LLM I used produced a variety of results with varying frequencies:</p><p>But this is still good enough. In practice, 80% of the time, parsing the result was trivial. The rest of the time, we treat the result as false.</p><p>In some rare cases (I’d estimate 5% of the time), the model would produce both a direct and inverse relationships. This is a sort of logical contradiction which I treated as a failure mode. In these cases, I ignored both relationships.</p><p>My approach here is pretty simple. Another interesting approach was taken by Long and her collaborators was to evaluate the strength of causality, by looking at the number of times the LLM responds positively or negatively. The same paper also notes that the prompt used matters quite a bit. They experimented with three variations on a prompt similar to the one I used:</p><ol><li>Introduced an authority by prepending something like According to Big Pharma...</li><li>Altered the linking, for example replacing “causes more” with something like “increases the likelihood”</li><li>Rephrased entity names to be more specific.</li></ol><p>These prompt engineering tweaks all made a difference and there wasn’t a clear winner.</p><h3>Generating explanations (Step 4)</h3><p>Terse explanation generation is also implemented using an LLM with a similar prompt:</p><pre>Text: ${groundingText} The text above suggests that more ${entity1} causes ${adverb} ${entity2}. Explain why in fewer than ten words.</pre><p>One design question for me was whether to show explanations at all and if so, how to best show them. The diagrams I generated initially did not have explanation, but I found it very mysterious why the system produced the connections that it did. When I added explanations, I initially added them as comments in the causal loop diagram’s markup language. This was step in the right direction, but resulted in sub-par UX. Because explanations were not inline in the graph itself, it took a lot of effort to try to find them.</p><p>Ultimately I decided that the best thing would be to show explanations as edge labels. This is a bit of a departure from the usual causal loop diagram conventions, but it helps make the diagram standalone. To generate these, I needed to compress the results significantly, and I found that “fewer than N words” was a really effective way to tune the results. Then, with a bit of additional wizardry to ensure that the explanations wrapped every ~30 chars, we were off to the races.</p><h3>Rendering causal loop diagrams (Steps 5–6)</h3><p>I used <a href="https://mermaid.live/">mermaid.js</a> to render the causal loop diagram because of its convenient markup. This lends itself well to being easily generated, and also edited without any need for WYSIWYG tools.</p><p>Instead of generating mermaid markup directly, I generated custom markup that I parsed with another, yet unreleased project of mine, which finds feedback loops in causal graphs, and labels them either reinforcing (R) or balancing (B). It was a nice excuse to brush up on some computer science! I cribbed from an implementation of <a href="https://github.com/Vacilando/js-tarjan/blob/main/tarjan.js">Tarjan’s Algorithm</a> for this purpose.</p><p>Still with me? If so, <a href="https://borismus.github.io/llm4cld">try the tool</a> on your own complex system descriptions! <a href="https://www.youtube.com/watch?v=FdFAqf_zfhM&amp;t=192s">All you need</a> is an OpenAI Key to start the engine.</p><p>Once you’ve taken it for a spin, please tell me where it works well, and where it fails for you. Lastly, please share your creations with me — it should be easy with this tool. The third button in the UI copies the URL to your clipboard, so that you can send it my way. Thanks for reading, and please <a href="https://smus.com/about">don’t be a stranger</a>.</p><p><em>Originally published at </em><a href="https://smus.com/tools-for-thinking-in-systems/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf68a4175c5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Similarity for Note Taking]]></title>
            <link>https://borismus.medium.com/semantic-similarity-for-note-taking-568e3fcc0799?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/568e3fcc0799</guid>
            <category><![CDATA[obsidian]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[notes]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 18:16:58 GMT</pubDate>
            <atom:updated>2023-10-18T18:16:58.260Z</atom:updated>
            <content:encoded><![CDATA[<p>Days after capturing a “new” insight, it can be humbling to realize that you are repeating yourself. This might not be a bad thing, as you mull over a complex idea in its various forms over the course of many weeks. But what if your note taking app could act as a co-pilot? It could surface similar notes that are relevant to your current writing, and if you use such a system for long enough, help you synthesize across your own thinking over many years. You might want to link to the semantically related note, or to merge with it entirely. Building on <a href="https://smus.com/ai-note-garden-linker">a previous technique</a>, I implemented this idea as an Obsidian plugin:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkZkDCjr8ZqU&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkZkDCjr8ZqU&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FkZkDCjr8ZqU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9940eed6d00d2c7192ccaea0b2d6029f/href">https://medium.com/media/9940eed6d00d2c7192ccaea0b2d6029f/href</a></iframe><p>I’m pleased with the result. It’s nice to feel like my note taking software is actively trying to help me reconnect with past selves. Whether this ultimately proves to be useful remains to be seen, but it works well enough that I plan to run this as a self-experiment over the next few months.</p><p>My <a href="https://smus.com/ai-note-garden-linker">first stab</a> at this problem generated a list of related notes across the whole corpus using semantic similarity. But my implementation had two fundamental limitations:</p><ol><li><strong>Time</strong>: My script ran at night, so that there was long delay between repeating myself and learning about this fact.</li><li><strong>Space</strong>: It generated a list of similar notes in a separate text file which I would rarely consult.</li></ol><p>By building this as an Obsidian plugin, I addressed both fundamental problems in the time and the space continuum!</p><ol><li>The current paragraph is continuously fed into a semantic similarity model, so that movement of the cursor or textual edits trigger updates in real-time.</li><li>Similar excerpts are presented in a dedicated sidebar, allowing you to see related notes and excerpts right away.</li></ol><p>The current implementation has two parts:</p><ol><li><strong>Indexer</strong> is a script that runs nightly, generating per-paragraph embeddings for each note in the corpus. These embeddings and other metadata are then saved as JSON.</li><li><strong>Obsidian plugin</strong> generates embeddings for any text within Obsidian, and then compares the result to the pre-generated embeddings.</li></ol><p>Both plugins use exactly the same embedding model to guarantee that the embedding mappings are identical. The one running inside Obsidian must use JavaScript, and I’ve already been using <a href="https://github.com/tensorflow/tfjs-models/tree/master/universal-sentence-encoder">Universal Sentence Encoder lite</a> for <a href="https://github.com/google-research/usnea">other projects</a>. For simplicity and to guarantee identical outputs, I built the indexer using the same exact model, running in node.js using tfjs-node. My previous python implementation of this used a <a href="https://tfhub.dev/google/universal-sentence-encoder-large/">slightly larger model of USE</a>.</p><p>The plugin works as follows:</p><ol><li>Find the currently active text: if there’s an active editor, find the paragraph at the current cursor position. If there’s a selection, use the selected text.</li><li>Extract its semantic similarity embedding, which in the case of USE is a 512-dimensional vector.</li><li>Compare its embedding to the rest of the embeddings for every other excerpt in the corpus, which is saved as an Nx512 matrix. Comparison here is a matrix multiplication, the result of which is a vector of scores corresponding to the similarity of the source excerpt with each of the N excerpt.</li><li>Find the top K scores and their corresponding excerpt.</li><li>Display the resulting notes and excerpts.</li></ol><p>Doing the above steps takes time. On my laptop it takes a second to generate embeddings, and another half second to do the multiplication and ranking. This will of course vary depending on your hardware and note corpus size. But even in my relatively favorable conditions, the resulting latency of a couple seconds is far too long to be doing this kind of work on the UI thread, which is the default behavior of tf.js in the Electron environment of the Obsidian plugin. I sought and found a workaround, which is to run <a href="https://erdem.pl/2020/02/making-tensorflow-js-work-faster-with-web-workers">tf.js in a web worker</a>, in CPU mode. This slows execution by about 50%, but is totally worth it to make the plugin usable in real life.</p><p>I’d love for the plugin to run standalone without the need for a separate indexer. This would allow others to use it far more easily without requiring them to setup a whole indexing <a href="https://smus.com/file-systems-for-thought/">system</a>.</p><p>Implementing indexing within the plugin, I ran into memory issues computing embeddings for my whole corpus. I haven’t yet found the time to dig in to why this was not a problem for the tfjs-node implementation of the indexer.</p><p>Before I seek workarounds, I want to live with the experience first. Is the plugin useful for navigating my note corpus? Do the excerpts it surfaces make sense as I’m writing a new note? Is the constantly changing semantic sidebar too distracting? Let’s find out; ƒor science!</p><p><em>Originally published at </em><a href="https://smus.com/semantic-similarity-note-taking/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=568e3fcc0799" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI note garden: link suggestions]]></title>
            <link>https://borismus.medium.com/ai-note-garden-link-suggestions-2e6a8fb03d0e?source=rss-fbe27680c9fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e6a8fb03d0e</guid>
            <category><![CDATA[tools-for-thought]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Boris Smus]]></dc:creator>
            <pubDate>Fri, 14 Jan 2022 20:12:25 GMT</pubDate>
            <atom:updated>2022-01-14T20:12:25.102Z</atom:updated>
            <content:encoded><![CDATA[<p>My latest <a href="https://smus.com/ai-note-garden-summarizer">AI gardener apprentice</a> finds pairs of notes that aren’t explicitly linked, but maybe ought to be. This project was inspired by the human subconscious, which creates, cements, and removes neural connections during sleep.</p><p>Like a sleeping brain, my python script runs nightly, scouring my note garden for related notes. The end result is a list of the most similar notes pairs in the garden, based on semantic similarity. I found the results to be illuminating, and a real-time version with solid UX would be a central feature of my <a href="https://smus.com/file-systems-for-thought">system for thought</a>.</p><p>Ideas are naturally interlinked. They build on one another, like stepping stones in the <a href="https://z3.ca/Adjacent_Possible">Adjacent Possible</a>. Concepts build on one another forming a dependency tree, and you can only learn something if you have first prepared a place to attach the new information. In other words, it must be in your <a href="https://z3.ca/Zone_of_Proximate_Development">Zone of Proximate Development</a>. Fortune favours the prepared mind.</p><p>Our brains reflect this interlinked nature of ideas, with billions of neurons and trillions of connections between them. This complicated machinery does its best to create a representation of the world around us.</p><p>When we sleep, automatic processes in the brain tend to these connections, creating new links, removing the obsolete ones, and paving the cowpaths. <a href="https://z3.ca/Sleep_has_a_profound_effect_on_memory_and_learning">Sleep has a profound effect on memory and learning</a>.</p><p>An ideal collection of personal notes mirrors your own thoughts. Obsidian emphasizes interlinking, and places a lot of weight on the note graph, a mind map-like view of the whole note garden resembling a connectome. Connections are emphasized in Roam’s marketing copy as a “note taking tool for <strong>networked</strong> thought”.</p><p>While today’s crop of note taking tools makes links and backlinks easier to manage than before, it’s still a pain to interlink notes. One major challenge is to recall what related notes you might have written in the past.</p><p>What is the note garden equivalent of sleep, during which the brain’s subconscious processes triage new information gleaned during the day? What might be useful automation for us note gardeners?</p><p>I take a fair amount of notes and sometimes have a feeling of deja-vu when I realize that I’ve been parroting something I already wrote many moons ago. If like me, you have a large note garden, a stable set of interests, and a crappy memory, you might know what I’m talking about.</p><p>The simplest remedy to this problem might be to scour the note garden, looking for redundancies: ideas that appear in multiple places. You might prune the redundant note, reword it, link it to something relevant, or refactor multiple notes to make more sense together.</p><p>Of course, redundancies aren’t all bad! Notes need enough context to stand on their own and topics that are interconnected can’t help but overlap somewhat. The goal here is not terseness, but cohesion and legibility. If a note is related to other notes, let’s cement that relationship with an explicit link.</p><h3>Sidebar: semantic similarity</h3><p>Now, how might we determine if two notes are related? One could simply consider two notes to be related if they have significant overlap in words. But we are better than that. The NLP keyword for this problem is semantic similarity.</p><p>For words, the situation is relatively simple. <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a> takes a word and represents it as an array of numbers. Words that are semantically similar end up closer together in the resulting vector space, and those that are semantically different end up far away. The same can be done with sentences, using some of the same machinery that powers large language models. We have the technology to take a whole sentence of variable length and create an embedding for it. Once we have the ability to generate embeddings from sentences, we can compare two sentences by evaluating how close or far their embeddings are.</p><p>We can do the same thing at a larger scale. Given N sentences, we calculate an NxN matrix of similarity values which might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*W7UDB44luy0cxcYu.png" /></figure><p>Quick sanity check:</p><ul><li>The diagonal answers the question “how similar is this sentence to itself” and has similarity 1 across the board</li><li>The matrix is symmetric (eg. A is as similar to B as B is to A).</li><li>In the above, notes named “Seattle culture” and “Seattle nearby child-friendly hikes” have high semantic similarity.</li></ul><p>I wrote a python script to find related notes in my note garden. Here’s how it works:</p><ol><li>Load all notes and their content</li><li>Split each note into paragraphs</li><li>Calculate embeddings for all paragraphs using <a href="https://tfhub.dev/google/universal-sentence-encoder-large/5">Universal Sentence Encoder (USE)</a>, keeping track of the source note.</li><li>Calculate the correlation matrix for all paragraphs using cosine similarity</li><li>Calculate a similarity score for each note pair using the above matrix</li><li>Sort note pairs by most similar, providing paragraph pairs as evidence</li></ol><p>Now we have a list of the most similar notes in the garden, their associated similarity scores.</p><h3>Sidebar: ignoring existing links</h3><p>Notes that are already interlinked should not show up in this list of similar notes. Their similarity is already legible to the system, and there are really a lot of such notes. For instance, most of my Medieval history notes are highly semantically similar, because they are derived from similar books and cover very similar themes.</p><p>We can easily filter out these note pairs by considering the graph structure of the note garden. An adjacency matrix gives O(1) lookup. In practice, going on step further and ignoring note pairs that are a minimum distance of 2 away has worked better on my note corpus. To help with this, I hired the <a href="https://en.wikipedia.org/wiki/Floyd%E2%80%93Warshall_algorithm">Floyd-Warshall algorithm</a> and its friendly <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csgraph.floyd_warshall.html">scipy implementation</a>. Once you have the matrix of minimum distances, filtering out the note pairs is easy.</p><p>I run the semantic similarity AI nightly. This generates an updated note called Similar notes.md which lists the most similar notes in my note garden. The resulting note is a collection of entries that look like this:</p><pre>Similarity 0.61: </pre><pre>Note 1: Inventing on what principle c. 2019 </pre><pre>&gt; Paragraph excerpt: How can you build software that doesn&#39;t need backends and SREs to run... </pre><pre>Note 2: Evogami without firebase </pre><pre>&gt; Paragraph excerpt: How might we create multiplayer apps with state, without relying on a backend at all...</pre><p>There is some mathematical beauty to a script which takes a collection of notes as input, and outputs another note. Having notes update overnight without my intervention makes the note garden feel like it’s alive and growing. That said, having an AI make arbitrary modifications to my hand written notes sounds like a real nightmare. To mitigate this, I have by convention decided to keep all AI-produced notes in the Nightly/ directory.</p><p>My note linker has been up and running for the last month, so I have a sense for how useful the resulting output is. One observation is that my note garden grows slowly, and most notes are relatively static. The first time I ran the similarity script, I found a lot of surprising and actionable output, but unless I have an especially prolific note taking session, running over the whole corpus nightly is probably overkill. More on this later.</p><p>Going through a fruitful generated list of similar notes, I tend to take one of the following actions:</p><ul><li>One of the notes supercedes the other ➡️ merge any useful bits from the deprecated note and remove it</li><li>Both notes point to the same concept ➡️ refactor the concept into a separate note and link to it from both</li><li>Both notes are highly related but not explicitly linked ➡️ create a link from one note to the other</li></ul><p>Any of these actions will cause the entry to disappear from Similar notes on a subsequent run. A virtuous cycle.</p><h3>Repetition in my book reviews</h3><p>I found some surprising redundancies in my book reviews. In 2017, I overused the word “vivid” (similarity 0.71):</p><blockquote><strong><em>Gatsby</em></strong><em>: Incredibly vividly written. Fascinating to see the self made struggles of people that are so fantastically well off. Very unsympathetic characters throughout, except for the narrator.</em></blockquote><blockquote><strong><em>Nothing is True and Everything is Possible</em></strong><em>: Super well written, incredibly engaging throughout. Incredibly so for non-fiction about a depressing topic. Much more entertaining than Gessen’s barrage of pessimism.</em></blockquote><p>I struggled with provenance in 2018 (similarity 0.63):</p><blockquote><strong><em>Bird by Bird</em></strong><em>: I’ve forgotten how this book appeared on my radar, but I think it was on someone’s list of most influential books ever. I am more ambivalent.</em></blockquote><blockquote><strong><em>The Fourth Turning</em></strong><em>: I don’t remember why I decided to read this book. The political theory it describes is loved by both Gore and Bannon. This surprisingly heterogeneous recommendation reignited my interest when I saw it on my reading list.</em></blockquote><p>In 2019, I did not mince words (similarity 0.64):</p><blockquote><strong><em>Building a Bridge to the 18th century</em></strong><em>: I found the book mostly useless. It’s a shallow summary of some stuff I already knew, and set forth an exaggerated technological pessimism that I, to some extent, share.</em></blockquote><blockquote><strong><em>You Belong To The Universe</em></strong><em>: The book itself is interesting but not groundbreaking. It involves overly flowery descriptions of things that I consider obvious: Google, Nest, Sim City, self driving cars, etc.</em></blockquote><h3>How to handle quotes?</h3><p>I have a special note where I track quotes that I really enjoyed from all over the place. Since I like them, these quotes end up in other notes too. Since the quotes are inserted verbatim, their presence is a very strong signal for note similarity.</p><p>My current solution is to ignore my Quotes I like.md note for this analysis, but this is a hack, since there are still many instances of the same quote across different notes used perhaps in different contexts. A shared quote is very weak evidence that the two notes are related, often insufficient to warrant a direct link.</p><p>One approach I haven’t yet tried might be to discard identical matches as meaningless. This would naturally discard quotes, since they are generally inserted verbatim.</p><p>The USE semantic similarity model I currently use is somewhat limited. It doesn’t perform well on non-latin character sets. According to it, anything with cyrillic characters is extremely semantically similar. Relatedly, it doesn’t perform well on non-words. Snippets of code, guitar tabs, and ASCII art all look same. Lastly, USE is already 4+ years old. AI/NLP is progressing so quickly, I should be using a more modern model to generate embeddings. I recently got access to OpenAI’s beta embedding feature, which might be a logical next step. But running OpenAI embeddings on tens of thousands of paragraphs sounds like an expensive proposition. I’d like to continue to be able to run semantic similarity in the comfort of my own computer.</p><p>Let me be the first to admit that what I have hacked together is a crude prototype, and a very rough approximation of the ultimate user experience I want my note taking app to provide. I want to be able to request suggestions for relevant notes to reference as I write, in a way that is not distracting.</p><p>If I had time, I’d build an Obsidian plugin version of what I’ve built as a shell script:</p><ol><li>Keep a cache of embeddings for notes in the note garden</li><li>As you write, periodically calculate this note’s embeddings and compare them to the rest of the corpus</li><li>Surface relevant but as yet unlinked notes and their similarity scores in a sidebar</li></ol><p>Wanna build it together? I promise to cheer you on from the sidelines! Drop me a line.</p><p><em>Originally published at </em><a href="https://smus.com/ai-note-garden-linker/"><em>https://smus.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e6a8fb03d0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>