<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Baalateja Kataru on Medium]]></title>
        <description><![CDATA[Stories by Baalateja Kataru on Medium]]></description>
        <link>https://medium.com/@baalatejakataru?source=rss-5aafecb53239------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*S-0o-3zsv0uUQrF1XkziBA.jpeg</url>
            <title>Stories by Baalateja Kataru on Medium</title>
            <link>https://medium.com/@baalatejakataru?source=rss-5aafecb53239------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:24:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@baalatejakataru/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building AI Infrastructure for a Post-Framework World]]></title>
            <link>https://medium.com/computatrum-veneficus/building-ai-infrastructure-for-a-post-framework-world-574535aacd58?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/574535aacd58</guid>
            <category><![CDATA[zig]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Tue, 13 Jan 2026 18:29:12 GMT</pubDate>
            <atom:updated>2026-01-13T19:21:39.803Z</atom:updated>
            <content:encoded><![CDATA[<h4>2025 was the year I stopped treating AI infrastructure like a prototype problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*Ou6Yx5PnyVpsVzf8.png" /></figure><p>2025 was the year I stopped treating AI infrastructure like a prototype problem. The field matured fast — we went from barely having tools to drowning in incompatible ones. Turns out trading “not enough options” for “too many fragmented options” isn’t actually an upgrade.</p><h3>The Fragmentation Problem</h3><p>Every major AI framework built its own protocol stack. Anthropic’s MCP, OpenAI’s function calling, LangChain’s abstractions — each one solves similar problems while creating vendor lock-in, protocol silos, and guaranteed rewrites down the line.</p><p>What I learned building these projects: we need infrastructure that outlives any single framework. Better primitives, not more abstractions.</p><h3>Technical Achievements: A Year in Review</h3><h3>1. PocketFlow-Zig: Flow-Based Agent Framework</h3><p><strong>Repository:</strong> <a href="https://github.com/The-Pocket/PocketFlow-Zig">PocketFlow-Zig</a></p><p>Started as a side project, ended up becoming the official Zig implementation of PocketFlow — a minimalist framework for building LLM-powered workflows. The design goal was explicit: build just enough framework to let agents construct other agents.</p><p><strong>What I Built:</strong></p><ul><li>Zero dependencies, pure Zig</li><li>Flow-based programming (think Unix pipes for LLM workflows)</li><li>Comptime-generated agent templates (zero runtime cost)</li><li>Cross-compilation for embedded targets</li></ul><p>PocketFlow now runs across multiple platforms as a base layer for agentic workflows.</p><h3>2. TOON-Zig: Efficient Token Serialization</h3><p><strong>Repository:</strong> <a href="https://github.com/bkataru/toon-zig">toon-zig</a></p><p>TOON (Token Oriented Object Notation) is a data serialization format optimized for LLM consumption. Built the Zig implementation from scratch:</p><p><strong>What Got Built:</strong></p><ul><li>340 official TOON 2.0 spec tests (100% passing)</li><li>196 decoding fixtures + 144 encoding fixtures</li><li>15–30% better token efficiency than JSON for typical LLM payloads</li><li>Zero dependencies, pure Zig</li></ul><p>Getting spec compliance while keeping performance high took some careful engineering.</p><h3>3. HF-Hub-Zig: HuggingFace Integration Layer</h3><p><strong>Repository:</strong> <a href="https://github.com/bkataru/hf-hub-zig">hf-hub-zig</a></p><p>Zig needed native HuggingFace Hub API support, especially for GGUF model management. So I built it.</p><p><strong>What It Does:</strong></p><ul><li>GGUF model discovery and search</li><li>Efficient model downloading with resume capability</li><li>Local model registry management</li><li>Cross-platform CLI interface</li></ul><p><strong>How It Works:</strong></p><ul><li>Zero dependencies, pure Zig</li><li>Efficient streaming downloads with progress tracking</li><li>Robust error handling for network failures</li><li>Memory-mapped file access for large models</li></ul><h3>4. Zenmap: Cross-Platform Memory Mapping</h3><p><strong>Repository:</strong> <a href="https://github.com/bkataru/zenmap">zenmap</a></p><p>A single-file Zig library for memory mapping large files, built specifically for GGUF model handling.</p><p><strong>The Challenge:</strong></p><ul><li>Unified POSIX (Linux, macOS, BSD) and Windows memory mapping APIs</li><li>Zero-copy file access for 70B+ parameter models</li><li>Efficient handling of sparse files and partial loads</li><li>Cross-platform abstraction with minimal overhead</li></ul><h3>5. Igllama: Zig-Based Ollama Alternative</h3><p><strong>Repository:</strong> <a href="https://github.com/bkataru/igllama">igllama</a></p><p>A Zig-based alternative to Ollama that uses the Zig build system’s embedded Clang toolchain to compile and run llama.cpp.</p><p><strong>What Makes It Different:</strong></p><ul><li>Zig build system integration with llama.cpp’s CMake build</li><li>Docker-like CLI experience for GGUF management</li><li>Transparent llama.cpp updates (no lagging behind upstream)</li><li>Cross-platform binary distribution</li></ul><h3>Production Work at Dirmacs</h3><p>I’ve been working with <a href="https://www.linkedin.com/in/suprabhat-rapolu/">Suprabhat Rapolu</a> and the team at <a href="https://www.linkedin.com/company/dirmacs-global-services/">Dirmacs Global Services</a> to ship production AI infrastructure in Rust:</p><h3>A.R.E.S: Production Agent Framework</h3><p><strong>Repository:</strong> <a href="https://github.com/dirmacs/ares">ares</a></p><p>A production agentic chatbot library built in Rust with:</p><ul><li>Multi-provider LLM support</li><li>Tool calling and ReAct loops</li><li>RAG knowledge bases</li><li>MCP protocol integration</li><li>Leptos-based embedded web UI</li></ul><p><strong>Current Status:</strong> Actively dogfooding this in pilot projects. Design goals keep evolving.</p><h3>Daedra: Web Research MCP Server</h3><p><strong>Repository:</strong> <a href="https://github.com/dirmacs/daedra">daedra</a></p><p>High-performance DuckDuckGo-powered web search MCP server written in Rust. Gives AI assistants web search and page fetching as tools.</p><p><strong>Features:</strong></p><ul><li>No API keys needed (uses DuckDuckGo’s public API)</li><li>Rust rewrite of an old TypeScript version</li><li>Ships production-ready: cargo install daedra</li></ul><h3>Lancor: LLaMA.cpp Client Library</h3><p><strong>Repository:</strong> <a href="https://github.com/dirmacs/lancor">lancor</a></p><p>A Rust client library for llama.cpp’s OpenAI-compatible API server.</p><p><strong>Goal:</strong> Simple, straightforward integration with existing Rust AI workflows.</p><h3>What Ties This Together</h3><p>All these projects share a common thread: explicit control over system boundaries. In a world where frameworks come and go, infrastructure needs to:</p><ol><li><strong>Outlive any single vendor</strong> — Protocol-level interop</li><li><strong>Show explicit costs</strong> — No magic, no hidden abstractions</li><li><strong>Stay fast</strong> — Zero-cost abstractions where possible</li><li><strong>Run everywhere</strong> — Build once, deploy anywhere</li></ol><h3>What’s Next for 2026</h3><p>Foundation’s built. Now comes production hardening:</p><ul><li><strong>Protocol work:</strong> Add gRPC and custom transports to zig-utcp</li><li><strong>Production features:</strong> Streaming, error handling, resource management</li><li><strong>Distributed systems:</strong> Get agents coordinating across networks</li><li><strong>Real-world testing:</strong> Pilot deployments, feedback loops</li></ul><p>The goal stays the same: build infrastructure that lets developers construct robust AI systems without getting locked into any framework.</p><p><em>All projects are MIT licensed and open source. Pull requests and technical discussions welcome.</em></p><p>links :)</p><ul><li><a href="https://github.com/bkataru/mcp.zig">mcp.zig</a></li><li><a href="https://github.com/bkataru/zig-utcp">zig-utcp</a></li><li><a href="https://github.com/The-Pocket/PocketFlow-Zig">PocketFlow-Zig</a></li><li><a href="https://github.com/bkataru/grayscales">grayscales</a></li><li><a href="https://github.com/bkataru/toon-zig">TOON-Zig</a></li><li><a href="https://github.com/bkataru/hf-hub-zig">HF-Hub-Zig</a></li><li><a href="https://github.com/bkataru/zenmap">zenmap</a></li><li><a href="https://github.com/bkataru/igllama">igllama</a></li><li><a href="https://github.com/dirmacs/ares">A.R.E.S</a></li><li><a href="https://github.com/dirmacs/daedra">daedra</a></li><li><a href="https://github.com/dirmacs/lancor">lancor</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*yzTkjkbcYugHNQjM.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=574535aacd58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/computatrum-veneficus/building-ai-infrastructure-for-a-post-framework-world-574535aacd58">Building AI Infrastructure for a Post-Framework World</a> was originally published in <a href="https://medium.com/computatrum-veneficus">Computatrum Veneficus</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Gzip Compression Library in Zig 0.15: A Deep Dive into Comprezz]]></title>
            <link>https://medium.com/computatrum-veneficus/building-a-gzip-compression-library-in-zig-0-15-a-deep-dive-into-comprezz-66c1ee2b56e8?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/66c1ee2b56e8</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[zig]]></category>
            <category><![CDATA[gzip]]></category>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Wed, 29 Oct 2025 11:37:49 GMT</pubDate>
            <atom:updated>2025-10-29T11:55:03.407Z</atom:updated>
            <content:encoded><![CDATA[<h4>Because I read “Compression functionality was removed. Sorry, you will have to copy the old code into your application” in the release notes.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*IvKOrjh8BNdp5eKR.png" /></figure><h3>Introduction</h3><p>When Zig 0.15 removed compression support from its standard library, it created an opportunity: build a standalone, production-ready compression library while navigating the language’s new I/O paradigm shift known as “Writergate.” This post chronicles the development of <a href="https://github.com/bkataru/comprezz">Comprezz</a>, a single-file gzip/deflate compression library that combines classical computer science algorithms with modern systems programming.</p><p><strong>What you’ll learn:</strong></p><ul><li>How the DEFLATE compression algorithm works under the hood</li><li>Building bit-level I/O abstractions in systems programming</li><li>Navigating Zig 0.15’s new non-generic I/O interfaces</li><li>Implementing Huffman coding and LZ77 compression from scratch</li></ul><h3>Project Overview</h3><p>Comprezz is a ~1700-line pure Zig implementation providing:</p><ul><li><strong>Full LZ77 + Huffman encoding</strong> — Complete DEFLATE (RFC 1951) implementation</li><li><strong>Multiple compression levels</strong> — From fast (4) to best (9)</li><li><strong>Gzip/Zlib/Raw containers</strong> — Proper headers, trailers, and checksums</li><li><strong>Library + CLI</strong> — Use programmatically or as a command-line tool</li><li><strong>Zero dependencies</strong> — Self-contained, no external C libraries</li></ul><p>The entire implementation lives in src/lib.zig, making it easy to vendor or study.</p><h3>Understanding the DEFLATE Algorithm</h3><p>Before diving into code, let’s understand what we’re building. DEFLATE combines two compression techniques:</p><h3>1. LZ77: Dictionary Compression</h3><p>LZ77 replaces repeated byte sequences with back-references:</p><pre>Input:  &quot;The quick brown fox jumps over the lazy dog. The quick brown fox...&quot;<br>Output: &quot;The quick brown fox jumps over the lazy dog. [back 45, length 23]...&quot;</pre><p>This works by maintaining a <strong>sliding window</strong> (32KB) of recent data and searching for matches. When we find a repeated sequence, we emit a <strong>match token</strong> (distance, length) instead of the literal bytes.</p><h3>2. Huffman Coding: Optimal Bit Encoding</h3><p>Frequently occurring symbols get shorter bit codes:</p><pre>&#39;e&#39; (frequent): 101      (3 bits)<br>&#39;q&#39; (rare):     11010111 (8 bits)</pre><p>DEFLATE builds <strong>dynamic Huffman trees</strong> for each block, adapting to the data’s characteristics.</p><h3>Combined Pipeline</h3><pre>Raw Data → LZ77 Tokenizer → Huffman Encoder → Bit Writer → Compressed Stream</pre><h3>Architecture Deep Dive</h3><p>Let’s walk through Comprezz’s implementation layer by layer.</p><h3>Layer 1: Bit-Level Writing</h3><p>Compression requires writing individual bits, not just bytes. Our BitWriter buffers bits and flushes efficiently:</p><pre>pub const BitWriter = struct {<br>    inner_writer: *std.Io.Writer,<br>    bits: u64 = 0,           // Bit accumulator<br>    nbits: u32 = 0,          // Number of bits in accumulator<br>    bytes: [256]u8 = undefined,  // Byte buffer for batching writes<br>    nbytes: u32 = 0,         // Buffer position<br><br>    pub fn writeBits(self: *Self, b: u32, nb: u32) Error!void {<br>        // Accumulate bits in little-endian order<br>        self.bits |= @as(u64, @intCast(b)) &lt;&lt; @as(u6, @intCast(self.nbits));<br>        self.nbits += nb;<br><br>        if (self.nbits &lt; 48) return;  // Haven&#39;t filled 6 bytes yet<br><br>        // Flush 6 bytes (48 bits) to buffer<br>        var n = self.nbytes;<br>        std.mem.writeInt(u64, self.bytes[n..][0..8], self.bits, .little);<br>        n += 6;<br><br>        if (n &gt;= 240) {  // Buffer full, write to underlying stream<br>            _ = try self.inner_writer.write(self.bytes[0..n]);<br>            n = 0;<br>        }<br><br>        self.nbytes = n;<br>        self.bits &gt;&gt;= 48;     // Remove written bits<br>        self.nbits -= 48;<br>    }<br>};</pre><p><strong>Key insights:</strong></p><ul><li>Uses 64-bit accumulator to buffer bits across byte boundaries</li><li>Batches writes to 256-byte buffer before syscalls</li><li>Writes 6 bytes at a time when accumulator fills (optimization)</li></ul><h3>Layer 2: Sliding Window for LZ77</h3><p>The heart of LZ77 is efficiently searching for repeated sequences. We use a <strong>hash-chained lookup table</strong>:</p><pre>pub const Lookup = struct {<br>    head: [32768]u16 = [_]u16{0} ** 32768,  // Hash table (15-bit hash)<br>    chain: [65536]u16 = [_]u16{0} ** 65536, // Chain of previous positions<br><br>    pub fn add(self: *Lookup, data: []const u8, pos: u16) u16 {<br>        if (data.len &lt; 4) return 0;<br>        const h = hash(data[0..4]);  // Hash first 4 bytes<br>        return self.set(h, pos);<br>    }<br><br>    fn set(self: *Lookup, h: u32, pos: u16) u16 {<br>        const prev = self.head[h];   // Get previous position at this hash<br>        self.head[h] = pos;          // Update hash table<br>        self.chain[pos] = prev;      // Link to previous position<br>        return prev;<br>    }<br><br>    fn hash(b: *const [4]u8) u32 {<br>        const prime = 0x9E3779B1;<br>        const v = @as(u32, b[3]) | <br>                  @as(u32, b[2]) &lt;&lt; 8 | <br>                  @as(u32, b[1]) &lt;&lt; 16 | <br>                  @as(u32, b[0]) &lt;&lt; 24;<br>        return @intCast((v *% prime) &gt;&gt; 17);  // 15-bit hash<br>    }<br>};</pre><p><strong>How hash chains work:</strong></p><pre>Input: &quot;abc...abc...abc&quot;<br>       pos=0  pos=100 pos=200<br><br>head[hash(&quot;abc&quot;)] = 200<br>chain[200] = 100<br>chain[100] = 0</pre><p>When we encounter “abc” at position 200, we can walk the chain backwards to find all previous occurrences without scanning the entire window.</p><h3>Layer 3: Match Finding</h3><p>Finding the longest match involves walking the hash chain and comparing byte sequences:</p><pre>fn findMatch(self: *Self, pos: u16, lh: []const u8, min_len: u16) ?Token {<br>    var len: u16 = min_len;<br>    var prev_pos = self.lookup.add(lh, pos);  // Get chain head<br>    var match: ?Token = null;<br><br>    var chain: usize = self.level.chain;  // Max positions to check<br><br>    if (len &gt;= self.level.good) {<br>        chain &gt;&gt;= 2;  // Already have good match, search less<br>    }<br><br>    while (prev_pos &gt; 0 and chain &gt; 0) : (chain -= 1) {<br>        const distance = pos - prev_pos;<br>        if (distance &gt; 32768) break;  // Out of window range<br><br>        const new_len = self.win.match(prev_pos, pos, len);<br>        if (new_len &gt; len) {<br>            match = Token.initMatch(@intCast(distance), new_len);<br>            if (new_len &gt;= self.level.nice) {<br>                return match;  // Good enough, stop searching<br>            }<br>            len = new_len;<br>        }<br>        prev_pos = self.lookup.prev(prev_pos);  // Follow chain<br>    }<br><br>    return match;<br>}</pre><p><strong>Compression level tuning:</strong></p><pre>const LevelArgs = struct {<br>    good: u16,  // Length to reduce search at<br>    nice: u16,  // Length to stop search at<br>    lazy: u16,  // Min length for immediate match<br>    chain: u16, // Max chain positions to check<br>};<br><br>// Level 9 (best): Check up to 4096 positions for perfect match<br>.best =&gt; .{ good = 32, lazy = 258, nice = 258, chain = 4096 }<br><br>// Level 4 (fast): Check only 16 positions<br>.fast =&gt; .{ good = 4, lazy = 4, nice = 16, chain = 16 }</pre><h3>Layer 4: Huffman Encoding</h3><p>DEFLATE uses dynamic Huffman codes optimized per block. Building optimal codes requires:</p><ol><li><strong>Frequency counting</strong> — Count symbol occurrences</li><li><strong>Tree building</strong> — Construct Huffman tree (Package-Merge algorithm)</li><li><strong>Code assignment</strong> — Generate canonical codes</li></ol><pre>pub fn HuffmanEncoder(comptime size: usize) type {<br>    return struct {<br>        codes: [size]HuffCode = undefined,  // Output: symbol → code mapping<br><br>        pub fn generate(self: *Self, freq: []u16, max_bits: u32) void {<br>            // 1. Sort symbols by frequency<br>            var list = self.freq_cache[0..freq.len];<br><br>            // ... populate list with (literal, freq) pairs ...<br>            mem.sort(LiteralNode, list, {}, byFreq);<br><br>            // 2. Package-Merge algorithm: compute optimal bit lengths<br>            const bit_count = self.bitCounts(list, max_bits);<br><br>            // 3. Assign canonical codes<br>            self.assignEncodingAndSize(bit_count, list);<br>        }<br>    };<br>}</pre><pre>fn assignEncodingAndSize(self: *Self, bit_count: []u32, list: []LiteralNode) void {<br>    var code: u16 = 0;<br><br>    for (bit_count, 0..) |bits, n| {<br>        code &lt;&lt;= 1;  // Double code value for each bit length<br><br>        if (n == 0 or bits == 0) continue;<br><br>        // Assign codes to all symbols with this bit length<br>        const chunk = list[list.len - bits..];<br>        mem.sort(LiteralNode, chunk, {}, byLiteral);  // Sort by symbol<br><br>        for (chunk) |node| {<br>            self.codes[node.literal] = HuffCode{<br>                .code = bitReverse(u16, code, @intCast(n)),  // Reverse bits!<br>                .len = @intCast(n),<br>            };<br>            code += 1;<br>        }<br><br>        list = list[0..list.len - bits];<br>    }<br>}</pre><p><strong>Why reverse bits?</strong> DEFLATE writes bits LSB-first, but Huffman trees are typically MSB-first. Reversing during code assignment is more efficient than reversing during every symbol write.</p><h3>Layer 5: Token Writing</h3><p>The BlockWriter emits compressed blocks with dynamic Huffman codes:</p><pre>pub fn write(self: *Self, tokens: []const Token, eof: bool, input: ?[]const u8) Error!void {<br>    // 1. Analyze tokens and build frequency tables<br>    const stats = self.indexTokens(tokens);<br><br>    // 2. Generate Huffman codes for literals and distances<br>    self.literal_encoding.generate(&amp;self.literal_freq, 15);<br>    self.distance_encoding.generate(&amp;self.distance_freq, 15);<br><br>    // 3. Choose best block type (stored, fixed, or dynamic)<br>    const dynamic_size = self.dynamicSize(...);<br>    const fixed_size = self.fixedSize(...);<br>    const stored_size = storedSizeFits(input);<br><br>    if (stored_size.storable and stored_size.size &lt; min(dynamic_size, fixed_size)) {<br>        // Raw storage is smaller - just copy bytes<br>        try self.storedBlock(input.?, eof);<br>    } else if (dynamic_size &lt; fixed_size) {<br>        // Dynamic Huffman is better<br>        try self.dynamicHeader(num_literals, num_distances, num_codegens, eof);<br>        try self.writeTokens(tokens, &amp;self.literal_encoding.codes, ...);<br>    } else {<br>        // Fixed Huffman is better (rare)<br>        try self.fixedHeader(eof);<br>        try self.writeTokens(tokens, &amp;self.fixed_literal_encoding.codes, ...);<br>    }<br>}</pre><p><strong>Dynamic block headers</strong> are complex — they encode the Huffman tree structure itself:</p><pre>fn dynamicHeader(self: *Self, num_literals: u32, num_distances: u32, <br>                 num_codegens: u32, eof: bool) Error!void {<br>    // Block header: 3 bits<br>    const first_bits: u32 = if (eof) 5 else 4;  // BFINAL=1, BTYPE=10<br>    try self.bit_writer.writeBits(first_bits, 3);<br><br>    // Tree sizes (5+5+4 bits)<br>    try self.bit_writer.writeBits(num_literals - 257, 5);   // HLIT<br>    try self.bit_writer.writeBits(num_distances - 1, 5);    // HDIST<br>    try self.bit_writer.writeBits(num_codegens - 4, 4);     // HCLEN<br><br>    // Code length code lengths (3 bits each)<br>    var i: u32 = 0;<br>    while (i &lt; num_codegens) : (i += 1) {<br>        const value = self.codegen_encoding.codes[codegen_order[i]].len;<br>        try self.bit_writer.writeBits(value, 3);<br>    }<br><br>    // Encoded literal/distance code lengths using run-length encoding<br>    i = 0;<br>    while (true) {<br>        const code_word = self.codegen[i];<br>        i += 1;<br>        if (code_word == 255) break;  // End marker<br><br>        try self.writeCode(self.codegen_encoding.codes[code_word]);<br><br>        // Handle run-length codes (16=repeat, 17=zeros, 18=many zeros)<br>        switch (code_word) {<br>            16 =&gt; try self.bit_writer.writeBits(self.codegen[i], 2),<br>            17 =&gt; try self.bit_writer.writeBits(self.codegen[i], 3),<br>            18 =&gt; try self.bit_writer.writeBits(self.codegen[i], 7),<br>            else =&gt; {},<br>        }<br>    }<br>}</pre><p>This uses <strong>three layers of encoding</strong>:</p><ol><li>Huffman codes for codegen alphabet (16 symbols)</li><li>Run-length encoding of literal/distance tree bit lengths</li><li>Huffman codes for the actual literals/distances</li></ol><h3>Layer 6: Main Compression Loop</h3><p>The Deflate struct orchestrates the entire pipeline:</p><pre>pub fn compress(self: *Self, reader: *std.Io.Reader) !void {<br>    while (true) {<br>        const buf = self.win.writable();  // Get writable window space<br><br>        if (buf.len == 0) {<br>            // Window full - process tokens and slide<br>            try self.tokenize(.none);<br>            self.slide();<br>            continue;<br>        }<br><br>        // Read data into window<br>        const read_size = @min(buf.len, 4096);<br>        const slice = reader.take(read_size) catch |err| switch (err) {<br>            error.EndOfStream =&gt; break,<br>            error.ReadFailed =&gt; return error.ReadFailed,<br>        };<br><br>        @memcpy(buf[0..slice.len], slice);<br>        self.hasher.update(buf[0..slice.len]);  // Update CRC32/Adler32<br><br>        self.win.written(slice.len);<br>        try self.tokenize(.none);<br>        if (slice.len &lt; read_size) break;  // EOF<br>    }<br>}</pre><p>The tokenizer implements <strong>lazy matching</strong> — a clever optimization:</p><pre>fn tokenize(self: *Self, flush_opt: FlushOption) !void {<br>    while (self.win.activeLookahead(should_flush)) |lh| {<br>        const pos = self.win.pos();<br>        const literal = lh[0];<br>        const min_len = if (self.prev_match) |m| m.length() else 0;<br><br>        if (self.findMatch(pos, lh, min_len)) |match| {<br>            try self.addPrevLiteral();<br><br>            if (match.length() &gt;= self.level.lazy) {<br>                // Good match - take it immediately<br>                step = try self.addMatch(match);<br>            } else {<br>                // Defer decision - maybe next position has better match<br>                self.prev_literal = literal;<br>                self.prev_match = match;<br>            }<br>        } else {<br>            if (self.prev_match) |m| {<br>                // No match at current position, use previous match<br>                step = try self.addMatch(m) - 1;<br>            } else {<br>                // No match at all - emit literal<br>                try self.addPrevLiteral();<br>                self.prev_literal = literal;<br>            }<br>        }<br>        self.windowAdvance(step, lh, pos);<br>    }<br>}</pre><p><strong>Lazy matching example:</strong></p><pre>Input: &quot;abcabcabc&quot;<br>Pos 0: Found match(0,3) &quot;abc&quot; → defer<br>Pos 1: Found match(0,4) &quot;bcab&quot; → better! Use this instead</pre><h3>Navigating Zig 0.15’s “Writergate”</h3><p>Zig 0.15 introduced a controversial change: non-generic I/O interfaces. The old API:</p><pre>// Old Zig 0.14<br>pub fn compress(reader: anytype, writer: anytype, options: Options) !void {<br>    // reader can be any type with read() method<br>    // writer can be any type with write() method<br>}</pre><p>The new API:</p><pre>// New Zig 0.15<br>pub fn compress(reader: *std.Io.Reader, writer: *std.Io.Writer, options: Options) !void {<br>    // reader and writer are concrete interface types with internal vtables<br>}</pre><h3>Key Changes</h3><p><strong>1. Concrete interface types with vtables</strong></p><pre>pub const BitWriter = struct {<br>    inner_writer: *std.Io.Writer,  // Not generic!<br><br>    pub fn init(writer: *std.Io.Writer) Self {<br>        return .{ .inner_writer = writer };<br>    }<br>    pub const Error = std.Io.Writer.Error || error{UnfinishedBits};<br>};</pre><p><strong>2. File I/O uses new methods</strong></p><pre>// Old: file.reader() and file.writer() returned generic types<br>// New: Returns concrete std.fs.File.Reader and std.fs.File.Writer<br>const input_file = try std.fs.cwd().openFile(&quot;input.txt&quot;, .{});<br>var input_reader = input_file.reader();  // Returns std.fs.File.Reader<br><br>const output_file = try std.fs.cwd().createFile(&quot;output.gz&quot;, .{});<br>var output_writer = output_file.writer();  // Returns std.fs.File.Writer</pre><p><strong>3. Fixed buffers use factory methods</strong></p><pre>// Old: std.io.fixedBufferStream(&amp;buffer)<br>// New: std.Io.Writer.fixed() and std.Io.Reader.fixed()<br><br>var buffer: [1024]u8 = undefined;<br>var fixed_writer = std.Io.Writer.fixed(&amp;buffer);<br><br>const data = &quot;Hello, World!&quot;;<br>var input: [1024]u8 = undefined;<br><br>@memcpy(input[0..data.len], data);<br>var fixed_reader = std.Io.Reader.fixed(input[0..data.len]);</pre><h3>Migration Benefits</h3><p>While controversial, the new design has advantages:</p><ol><li><strong>Faster compilation</strong> — No monomorphization for each type</li><li><strong>Smaller binaries</strong> — Single implementation instead of multiple instantiations</li><li><strong>Clearer errors</strong> — std.Io.Writer.Error instead of complex generic constraints</li><li><strong>Consistent buffering</strong> — Buffer responsibility is explicit</li></ol><h3>Container Formats: Gzip, Zlib, Raw</h3><p>Comprezz supports three container formats wrapping the DEFLATE stream:</p><h3>Gzip Format (RFC 1952)</h3><pre>┌─────────────────────────────────────┐<br>│ Magic: 0x1f 0x8b                    │  2 bytes<br>│ Method: 0x08 (DEFLATE)              │  1 byte<br>│ Flags: 0x00                         │  1 byte<br>│ Timestamp: 0x00 0x00 0x00 0x00      │  4 bytes<br>│ Extra flags: 0x00                   │  1 byte<br>│ OS: 0x03 (Unix)                     │  1 byte<br>├─────────────────────────────────────┤<br>│ DEFLATE compressed data             │  variable<br>├─────────────────────────────────────┤<br>│ CRC32 checksum                      │  4 bytes (little-endian)<br>│ Uncompressed size mod 2^32          │  4 bytes (little-endian)<br>└─────────────────────────────────────┘</pre><p>Implementation:</p><pre>pub fn writeHeader(comptime wrap: Container, writer: *std.Io.Writer) !void {<br>    switch (wrap) {<br>        .gzip =&gt; {<br>            const gzipHeader = [_]u8{ <br>                0x1f, 0x8b,  // Magic<br>                0x08,        // Method (DEFLATE)<br>                0x00,        // Flags<br>                0x00, 0x00, 0x00, 0x00,  // Timestamp<br>                0x00,        // Extra flags<br>                0x03         // OS (Unix)<br>            };<br>            try writer.writeAll(&amp;gzipHeader);<br>        },<br>        // ... other formats<br>    }<br>}<br><br>pub fn writeFooter(comptime wrap: Container, hasher: *Hasher(wrap), <br>                   writer: *std.Io.Writer) !void {<br>    var bits: [4]u8 = undefined;<br>    switch (wrap) {<br>        .gzip =&gt; {<br>            std.mem.writeInt(u32, &amp;bits, hasher.chksum(), .little);<br>            try writer.writeAll(&amp;bits);<br>            std.mem.writeInt(u32, &amp;bits, hasher.bytesRead(), .little);<br>            try writer.writeAll(&amp;bits);<br>        },<br>        // ... other formats<br>    }<br>}</pre><h3>Zlib Format (RFC 1950)</h3><p>Simpler than gzip, used internally by PNG:</p><pre>┌─────────────────────────────────────┐<br>│ CMF: 0x78 (window=32K, method=8)    │  1 byte<br>│ FLG: 0xDC (check bits + level)      │  1 byte<br>├─────────────────────────────────────┤<br>│ DEFLATE compressed data             │  variable<br>├─────────────────────────────────────┤<br>│ Adler32 checksum                    │  4 bytes (big-endian!)<br>└─────────────────────────────────────┘</pre><h3>Usage Examples</h3><h3>Library: Basic Compression</h3><pre>const std = @import(&quot;std&quot;);<br>const comprezz = @import(&quot;comprezz&quot;);<br><br>pub fn main() !void {<br>    var compressed_buffer: [1024]u8 = undefined;<br>    var fixed_writer = std.Io.Writer.fixed(&amp;compressed_buffer);<br><br>    const data = &quot;Hello, World!&quot;;<br>    var input_buffer: [1024]u8 = undefined;<br><br>    @memcpy(input_buffer[0..data.len], data);<br>    var input_reader = std.Io.Reader.fixed(input_buffer[0..data.len]);<br>    try comprezz.compress(&amp;input_reader, &amp;fixed_writer, .{});<br><br>    // Find actual compressed size<br>    var written: usize = 0;<br>    for (compressed_buffer, 0..) |byte, i| {<br>        if (byte != 0) written = i + 1;<br>    }<br><br>    const compressed = compressed_buffer[0..written];<br>    std.debug.print(&quot;Compressed {} bytes to {} bytes\n&quot;, <br>                    .{ data.len, compressed.len });<br>}</pre><h3>Library: File Compression</h3><pre>const std = @import(&quot;std&quot;);<br>const comprezz = @import(&quot;comprezz&quot;);<br><br>pub fn main() !void {<br>    const input_file = try std.fs.cwd().openFile(&quot;input.txt&quot;, .{});<br>    defer input_file.close();<br>    var input_reader = input_file.reader();<br><br>    const output_file = try std.fs.cwd().createFile(&quot;output.gz&quot;, .{});<br>    defer output_file.close();<br>    var output_writer = output_file.writer();<br><br>    try comprezz.compress(&amp;input_reader, &amp;output_writer, .{ .level = .best });<br>}</pre><h3>CLI: Command-line Usage</h3><pre># Compress a file<br>$ comprezz input.txt -o output.gz<br><br># Use best compression<br>$ comprezz -l best largefile.txt -o compressed.gz<br><br># Compress from stdin<br>$ cat data.txt | comprezz &gt; data.gz<br><br># Compress with specific level<br>$ comprezz -l 9 input.txt -o output.gz</pre><h3>Performance Characteristics</h3><h3>Time Complexity</h3><ul><li><strong>Best case</strong>: O(n) for uncompressible data (stored blocks)</li><li><strong>Average case</strong>: O(n × c) where c = chain length (typically 16–4096)</li><li><strong>Worst case</strong>: O(n × w) where w = window size (32KB) for pathological patterns</li></ul><h3>Space Complexity</h3><ul><li><strong>Sliding window</strong>: 64KB (2 × 32KB for double buffering)</li><li><strong>Hash table</strong>: 128KB (32K entries × 2 bytes + 64K chain entries × 2 bytes)</li><li><strong>Tokens buffer</strong>: 128KB (32K tokens × 4 bytes)</li><li><strong>Total</strong>: ~400KB fixed memory usage</li></ul><h3>Compression Ratios</h3><p>Tested on Calgary Corpus:</p><pre>File        Size     Level 4   Level 6   Level 9   Time (L4)  Time (L9)<br>─────────────────────────────────────────────────────────────────────────<br>book1.txt   768KB    43.2%     42.1%     41.8%     0.12s      0.45s<br>paper1.txt  52KB     39.8%     38.5%     38.2%     0.008s     0.032s<br>geo.txt     102KB    68.9%     68.2%     68.0%     0.015s     0.058s</pre><h3>Testing Strategy</h3><p>Comprezz includes comprehensive tests covering:</p><h3>1. Basic Functionality</h3><pre>test &quot;basic compression&quot; {<br>    var compressed_buffer: [1024]u8 = undefined;<br>    var fixed_writer = std.Io.Writer.fixed(&amp;compressed_buffer);<br><br>    const data = &quot;Hello, World!&quot;;<br>    var input_buffer: [1024]u8 = undefined;<br>    @memcpy(input_buffer[0..data.len], data);<br>    <br>    var input_reader = std.Io.Reader.fixed(input_buffer[0..data.len]);<br>    try compress(&amp;input_reader, &amp;fixed_writer, .{});<br>    <br>    var written: usize = 0;<br>    for (compressed_buffer, 0..) |byte, i| {<br>        if (byte != 0) written = i + 1;<br>    }<br>    <br>    const compressed = compressed_buffer[0..written];<br>    <br>    try expect(compressed.len &gt; 0);<br>    try expect(compressed[0] == 0x1f);  // Gzip magic<br>    try expect(compressed[1] == 0x8b);<br>}</pre><h3>2. Compression Levels</h3><pre>test &quot;compression levels&quot; {<br>    const data = &quot;The quick brown fox jumps over the lazy dog. &quot; ** 3;<br><br>    // Compress with fast and best settings<br>    var fast_compressed = try compressWithLevel(data, .fast);<br>    var best_compressed = try compressWithLevel(data, .best);<br>    <br>    // Best compression should be smaller (or equal)<br>    try expect(best_compressed.len &lt;= fast_compressed.len);<br>}</pre><h3>3. Edge Cases</h3><pre>test &quot;small data compression&quot; {<br>    try testCompression(&quot;Hi!&quot;);  // Very small input<br>}<br><br>test &quot;large data compression&quot; {<br>    const large = &quot;Lorem ipsum...&quot; ** 100;  // &gt;44KB<br>    try testCompression(large);<br>}<br><br>test &quot;incompressible data&quot; {<br>    var random: [1024]u8 = undefined;<br>    // ... fill with random data ...<br>    try testCompression(&amp;random);<br>}</pre><h3>Lessons Learned</h3><h3>1. Bit Manipulation is Hard</h3><p>Getting bit ordering right took multiple iterations. DEFLATE’s LSB-first bit order combined with MSB-first Huffman trees created subtle bugs that only appeared with certain data patterns.</p><p><strong>Solution</strong>: Extensive unit tests for bit reversal and explicit comments about bit ordering everywhere.</p><h3>2. Performance vs. Compression Tradeoff</h3><p>The compression level parameters have enormous impact:</p><ul><li>Level 4: 2–3x faster but 2–5% worse compression</li><li>Level 9: Best compression but searches 256x more positions</li></ul><p><strong>Solution</strong>: Expose multiple levels and make .default (level 6) a reasonable balance.</p><h3>3. Testing Compression is Tricky</h3><p>You can’t just compare compressed output byte-for-byte — there are multiple valid DEFLATE streams for the same input.</p><p><strong>Solution</strong>:</p><ul><li>Test that output decompresses correctly (requires external tool)</li><li>Test structural properties (gzip header, valid bit patterns)</li><li>Compare compression ratios across levels</li></ul><h3>4. Fixed-Size Buffers in Zig 0.15</h3><p>The new I/O system doesn’t track position automatically in fixed buffers.</p><p><strong>Solution</strong>: Manually track written bytes or scan for non-zero data after compression.</p><h3>Future Improvements</h3><h3>1. Streaming API</h3><p>Current API requires loading full output into memory. Add:</p><pre>pub const StreamingCompressor = struct {<br>    pub fn writeChunk(self: *Self, data: []const u8) !void;<br>    pub fn finish(self: *Self) !void;<br>};</pre><h3>2. Parallel Compression</h3><p>Large files could be split into independent blocks compressed in parallel:</p><pre>// Split input into 1MB chunks<br>// Compress each chunk on separate thread<br>// Concatenate results with DEFLATE block headers</pre><h3>3. Better Hash Function</h3><p>Current hash is simple but has collisions. Consider:</p><ul><li>xxHash for better distribution</li><li>Faster multiplication schemes</li><li>Cache-optimized access patterns</li></ul><h3>4. SIMD Matching</h3><p>Use AVX2 to compare 32 bytes at once during match verification:</p><pre>const match_len = @import(&quot;std&quot;).simd.countLeadingEqualElements(<br>    u8, 32, prev_data, curr_data<br>);</pre><h3>Conclusion</h3><p>Building Comprezz was an exercise in:</p><ul><li><strong>Classical algorithms</strong> — LZ77 and Huffman coding are 40+ years old but still relevant</li><li><strong>Modern systems programming</strong> — Zig’s safety + performance enabled clean implementation</li><li><strong>API evolution</strong> — Adapting to Zig 0.15’s controversial but pragmatic I/O redesign</li></ul><p>The complete 1700-line implementation demonstrates that you don’t need massive frameworks to build production-ready compression — just careful algorithm engineering and attention to detail.</p><p><strong>Try it yourself:</strong></p><pre># Add to your project<br>$ zig fetch --save git+https://github.com/bkataru/comprezz.git<br><br># Or use the CLI<br>$ git clone https://github.com/bkataru/comprezz.git<br>$ cd comprezz<br>$ zig build<br>$ ./zig-out/bin/comprezz your-file.txt -o compressed.gz<br></pre><p><strong>Resources:</strong></p><ul><li><a href="https://tools.ietf.org/html/rfc1951">RFC 1951 — DEFLATE Specification</a></li><li><a href="https://tools.ietf.org/html/rfc1952">RFC 1952 — Gzip Format</a></li><li><a href="https://github.com/bkataru/comprezz">Source code on GitHub</a></li><li><a href="https://github.com/ziglang/zig/pull/24329">Zig 0.15 I/O Changes</a></li></ul><p>The journey from understanding DEFLATE on paper to shipping a working implementation taught me that compression isn’t magic — it’s just clever bookkeeping with bits and bytes. And sometimes, that’s the best kind of magic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66c1ee2b56e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/computatrum-veneficus/building-a-gzip-compression-library-in-zig-0-15-a-deep-dive-into-comprezz-66c1ee2b56e8">Building a Gzip Compression Library in Zig 0.15: A Deep Dive into Comprezz</a> was originally published in <a href="https://medium.com/computatrum-veneficus">Computatrum Veneficus</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zig 0.15.1 I/O Overhaul: Understanding the New Reader/Writer Interfaces]]></title>
            <link>https://medium.com/computatrum-veneficus/zig-0-15-1-i-o-overhaul-understanding-the-new-reader-writer-interfaces-38cb5bf442cc?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/38cb5bf442cc</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[systems-programming]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[zig]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 09:47:10 GMT</pubDate>
            <atom:updated>2025-12-10T22:10:03.482Z</atom:updated>
            <content:encoded><![CDATA[<h4>“Don’t forget to flush!” — Andrew Kelley, probably</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*cTZdfqOVNQLk6DAalQn5KA.jpeg" /></figure><h3>Introduction</h3><p>Prior to Zig version 0.15.1, writing to standard output was more or less straightforward:</p><pre>const std = @import(&quot;std&quot;);<br><br>pub fn main() !void {<br>    var stdout = std.io.getStdOut().writer();<br>    try stdout.print(&quot;did you ever hear the tragedy of darth plagueis the wise?\n&quot;, .{});<br>}</pre><p>With the release of Zig 0.15.1, however, the I/O subsystem underwent a major redesign, popularly nicknamed <a href="https://github.com/ziglang/zig/pull/24329"><strong>Writergate</strong></a>. The standard library’s I/O interfaces now use <em>buffered I/O by default</em>. This change reduces costly system calls but requires developers to explicitly manage buffers and make sure to remember to flush output.</p><h3>What Changed in Zig 0.15.1</h3><p>In the new system:</p><ul><li>Writers and readers require an explicit buffer.</li><li>Data is written to that buffer first and only flushed to the OS (e.g., stdout) on demand.</li><li>Forgetting to call .flush() may result in missing output on the terminal.</li></ul><p>From the release notes: <a href="https://ziglang.org/download/0.15.1/release-notes.html#Writergate">“Please use buffering! And don’t forget to flush!”</a>.</p><h3>Example: Writing with a Buffer</h3><pre>const std = @import(&quot;std&quot;);<br><br>pub fn main() !void {<br>    var stdout_buffer: [1024]u8 = undefined;<br>    var stdout_writer = std.fs.File.stdout().writer(&amp;stdout_buffer);<br>    const stdout = &amp;stdout_writer.interface;<br>    <br>    try stdout.print(&quot;did you ever hear the tragedy of darth plagueis the wise?\n&quot;, .{});<br>    <br>    // Required to push buffered output to the terminal<br>    try stdout.flush();<br>}</pre><p>Without the final flush(), nothing may appear on screen.</p><h3>Why Buffered I/O?</h3><p>Buffered I/O reduces the number of expensive kernel syscalls. Instead of invoking the OS for every print, multiple writes are batched into a buffer and flushed at once.</p><p>This makes I/O significantly more efficient — especially in performance-sensitive applications.</p><h3>Writing to Stdout</h3><h4>Printing Formatted Strings</h4><pre>const std = @import(&quot;std&quot;);<br><br>pub fn main() !void {<br>    var stdout_buffer: [256]u8 = undefined;<br>    var stdout_writer = std.fs.File.stdout().writer(&amp;stdout_buffer);<br>    const stdout = &amp;stdout_writer.interface;<br>    <br>    try stdout.print(&quot;greetings, program, welcome to the {s}, a {s}\n&quot;, .{&quot;grid&quot;, &quot;digital frontier&quot;});<br>    try stdout.flush();<br>}</pre><h4>Writing Raw Bytes</h4><pre>const std = @import(&quot;std&quot;);<br><br>pub fn main() !void {<br>    var buf: [128]u8 = undefined;<br>    var w = std.fs.File.stdout().writer(&amp;buf);<br>    const stdout = &amp;w.interface;<br>    <br>    try stdout.writeAll(&quot;just a raw message ig\n&quot;);<br>    try stdout.flush();<br>}</pre><h4>Unbuffered Output</h4><p>If you want writes to go directly to the OS without buffering, pass an empty slice (&amp;.{}):</p><pre>const std = @import(&quot;std&quot;);<br><br>pub fn main() !void {<br>    var w = std.fs.File.stdout().writer(&amp;.{});<br>    const stdout = &amp;w.interface;<br>    <br>    try stdout.print(&quot;an unbuffered print!!! :o o7 o7\n&quot;, .{});<br>    // flush is a no-op in this case<br>}</pre><p>This bypasses buffering but is less efficient.</p><h3>Passing Writers Around</h3><p>The new interface design makes it simple to write functions that accept <em>any</em> writer:</p><pre>const std = @import(&quot;std&quot;);<br><br>fn greet(writer: *std.Io.Writer) !void {<br>    try writer.print(&quot;greetings, program. welcome to the {s}, a {s}\n&quot;, .{&quot;grid&quot;, &quot;digital frontier&quot;});<br>}<br><br>pub fn main() !void {<br>    var buf: [512]u8 = undefined;<br>    var w = std.fs.File.stdout().writer(&amp;buf);<br>    const stdout = &amp;w.interface;<br>    <br>    try greet(stdout);<br>    try stdout.flush();<br>}</pre><p>This flexibility allows the same function to target stdout, files, sockets, or async streams.</p><h3>Reading with the New API</h3><p>Readers also now require explicit buffers and expose new methods such as takeDelimiterExclusive.</p><h4>Example: Reading from Stdin</h4><pre>const std = @import(&quot;std&quot;);<br><br>pub fn main() !void {<br>    // 1. allocate a buffer for stdin reads<br>    var stdin_buffer: [512]u8 = undefined;<br><br>    // 2. get a reader for stdin, backed by our stdin buffer<br>    var stdin_reader_wrapper = std.fs.File.stdin().reader(&amp;stdin_buffer);<br>    const reader: *std.Io.Reader = &amp;stdin_reader_wrapper.interface;<br><br>    // 3. allocate a buffer for stdout writes<br>    var stdout_buffer: [512]u8 = undefined;<br><br>    // 4. get a writer for stdout operations, backed by our stdout buffer<br>    var stdout_writer = std.fs.File.stdout().writer(&amp;stdout_buffer);<br>    const stdout: *std.Io.Writer = &amp;stdout_writer.interface;<br><br>    // 5. prompt the user<br>    try stdout.writeAll(&quot;Type something: &quot;);<br>    try stdout.flush(); // try commenting this out, notice the &quot;Type something:&quot; prompt won&#39;t appear, but you&#39;ll still be able to type something and hit enter, upon which it will appear<br><br>    // 6. read lines (delimiter = &#39;\n&#39;)<br>    while (reader.takeDelimiterExclusive(&#39;\n&#39;)) |line| {<br>        // need to toss the delimiter explicitly since we&#39;re not reading it<br>        reader.toss(1);<br>        // `line` is a slice of bytes (excluding the delimiter)<br>        // do whatever you want with it<br><br>        try stdout.writeAll(&quot;You typed: &quot;);<br>        try stdout.print(&quot;{s}&quot;, .{line});<br>        try stdout.writeAll(&quot;\n...\n&quot;);<br>        try stdout.writeAll(&quot;Type something: &quot;);<br><br>        try stdout.flush();<br>    } else |err| switch (err) {<br>        error.EndOfStream =&gt; {<br>            // reached end<br>            // the normal case<br>        },<br>        error.StreamTooLong =&gt; {<br>            // the line was longer than the internal buffer<br>            return err;<br>        },<br>        error.ReadFailed =&gt; {<br>            // the read failed<br>            return err;<br>        },<br>    }<br>}</pre><pre>╰─➤ zig run main.zig<br>Type something: hi there<br>You typed: hi there<br>...<br>Type something: oh wow<br>You typed: oh wow<br>...<br>Type something: this is basically a shell<br>You typed: this is basically a shell<br>...<br>Type something: an echo shell?<br>You typed: an echo shell?<br>...<br>Type something:</pre><p>This example implements a simple echo shell that reads user input line by line.</p><h3>Pitfalls and Best Practices</h3><ul><li><strong>Always flush after writes</strong> unless you’re deliberately leaving data in the buffer. Remember, Zig’s Reader/Writer are both <a href="https://en.wikipedia.org/wiki/Circular_buffer">ring buffers</a>.</li><li><strong>Use stable backing objects</strong>: the wrapper (stdout_writer, stdin_reader_wrapper) must outlive the interface pointer (&amp;.interface).</li><li><strong>Don’t copy interfaces incorrectly</strong>: these rely on @fieldParentPtr; mishandling can cause undefined behavior.</li><li><strong>Handle long lines</strong>: takeDelimiterExclusive will return error.StreamTooLong if input exceeds buffer size.</li><li><strong>Unbuffered writers</strong>: only use when absolutely necessary; buffering is almost always better.</li></ul><h3>Conclusion</h3><p>Zig 0.15.1’s I/O changes represent a significant improvement in efficiency and clarity at the cost of slightly more boilerplate. Developers now:</p><ul><li>Provide explicit buffers</li><li>Remember to flush</li><li>Write code that can flexibly handle any reader/writer</li></ul><p>For more details, see:</p><ul><li><a href="https://kristoff.it/blog/zig-new-async-io/">Zig’s New Async I/O</a></li><li><a href="https://www.openmymind.net/Zigs-New-Writer/">Zig’s new Writer</a></li><li><a href="https://www.openmymind.net/Im-Too-Dumb-For-Zigs-New-IO-Interface/">I’m too dumb for Zig’s new IO interface</a></li><li><a href="https://ziggit.dev/t/im-too-dumb-for-zigs-new-io-interface/11645">I’m too dumb for Zig’s new IO interface — discussion on ziggit.dev</a></li><li><a href="https://www.reddit.com/r/Zig/comments/1mz3z7i/when_do_i_need_to_flush_help_understanding_0151/">from r/zig — when do i need to flush ? — help understanding 0.15.1 change for Writers</a></li><li><a href="https://ziglang.org/download/0.15.1/release-notes.html#Upgrading-stdiogetStdOutwriterprint">zig 0.15.1 release notes — Upgrading std.io.getStdOut().writer().print()</a></li><li><a href="https://github.com/ziglang/zig/pull/24329">Writergate</a></li><li><a href="https://news.ycombinator.com/item?id=44461067">Zig breaking change — Initial Writergate</a></li><li><a href="https://ziggit.dev/t/zig-0-15-1-reader-writer-dont-make-copies-of-fieldparentptr-based-interfaces/11719">Zig 0.15.1 reader/writer: Don’t make copies of @fieldParentPtr()-based interfaces — discussion on ziggit.dev</a></li></ul><p>With these tools, Ziguanas gain both performance and flexibility — so long as they don’t forget to <strong>flush</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38cb5bf442cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/computatrum-veneficus/zig-0-15-1-i-o-overhaul-understanding-the-new-reader-writer-interfaces-38cb5bf442cc">Zig 0.15.1 I/O Overhaul: Understanding the New Reader/Writer Interfaces</a> was originally published in <a href="https://medium.com/computatrum-veneficus">Computatrum Veneficus</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Single-File Zig Client for llama.cpp’s OpenAI-Compatible API Server]]></title>
            <link>https://medium.com/computatrum-veneficus/a-single-file-zig-client-for-llama-cpps-openai-compatible-api-server-e37657580c4f?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/e37657580c4f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[zig]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llama-cpp]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Fri, 15 Aug 2025 14:07:37 GMT</pubDate>
            <atom:updated>2025-09-28T09:57:01.224Z</atom:updated>
            <content:encoded><![CDATA[<h4>“Be tiny, quick, and invoke intelligence close to the metal” — Feynman, Probably</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CakQbRyPY6Q9lxdP.png" /></figure><p>This post walks through creating a <strong>minimal, dependency-free, pure Zig 0.14.1</strong> client for llama.cpp’s OpenAI API-compatible inference server. It’s elegant, compact, and — Ziguanas rejoice — <strong>deterministic in memory allocation</strong>.</p><p>There’s even a tiny templating engine.</p><h3>Why?</h3><p>The llama.cpp ecosystem is fantastic for running LLMs locally, but many clients lean on heavy HTTP libraries or multiple files. With Zig’s rich standard library (std.http for requests, std.json for parsing), you can get away with just one file—no dependencies required.</p><p>You’ll get:</p><ul><li>Clean, deterministic memory management (arena allocators for the win).</li><li>A concise templating function for dynamic prompt generation.</li><li>A fully working example that talks to an inference server.</li></ul><h3>Prerequisites</h3><p>Before we dive into code, make sure you have:</p><ol><li><strong>llama.cpp</strong> installed:</li></ol><p>2. Either grab <a href="https://github.com/ggerganov/llama.cpp/releases">precompiled binaries</a></p><ul><li>Or build from source (you’ll need a C/C++ toolchain and Make/CMake)</li><li>Or use a frontend that ships llama.cpp, like <a href="https://jan.ai/">Jan</a> or <a href="https://lmstudio.ai/">LM Studio</a></li></ul><ol><li><strong>An inference server running</strong> on http://127.0.0.1:1337 using llama.cpp’s --api-server mode.</li><li><strong>A model in GGUF format</strong>, e.g. <a href="https://huggingface.co/Qwen">Qwen_Qwen3-4B-Instruct-2507-IQ4_XS</a>.</li></ol><h3>Core Design</h3><p>The entire client fits into one .zig file. Here’s the design:</p><ul><li><strong>Data Transfer Objects (DTOs)</strong> to serialize request payloads and deserialize responses.</li><li>A <strong>formatTemplate</strong> function to fill {s} placeholders in multiline prompts.</li><li>A <strong>llmCall</strong> function to send a request to the inference server and parse the JSON reply.</li><li>A <strong>main</strong> function demonstrating system/user prompts and printing the assistant’s response.</li></ul><h3>Complete Implementation</h3><pre>const std = @import(&quot;std&quot;);<br><br>// DTO for deserialization<br>const LLMResponse = struct {<br>    id: []const u8, // Unique identifier for the response<br>    object: []const u8, // Type of object returned<br>    created: u32, // Unix timestamp of when the response was generated<br>    model: []const u8, // Name of the model used to generate the response<br>    usage: ?struct { // Usage statistics for the response, optional<br>        prompt_tokens: u32, // Number of tokens in the prompt<br>        completion_tokens: u32, // Number of tokens in the completion<br>        total_tokens: u32, // Total number of tokens used<br>    } = null,<br>    timings: ?struct { // Timing statistics for the response, optional<br>        prompt_n: u32, // Number of prompts processed<br>        prompt_ms: f64, // Total time taken to process the prompt<br>        prompt_per_token_ms: f64, // Average time taken per token in the prompt<br>        prompt_per_second: f64, // Average time taken per second for the prompt<br>        predicted_n: u32, // Number of predictions made<br>        predicted_ms: f64, // Total time taken to make the predictions<br>        predicted_per_token_ms: f64, // Average time taken per token in the prediction<br>        predicted_per_second: f64, // Average time taken per second for the prediction<br>    } = null,<br>    choices: []struct { // Array of choices generated by the model<br>        message: struct { // Message generated by the model<br>            role: []const u8,<br>            content: []const u8,<br>        },<br>        logprobs: ?struct { // Log probabilities of the tokens generated, optional<br>            content: []struct { // Array of token logprob objects<br>                token: []const u8, // Token ID or string representation of the token<br>                logprob: f64, // Using f64 for double precision log probabilities<br>                bytes: []const u8, // Raw bytes of the token<br>                // top_logprobs is an array of objects, each containing a token and its logprob<br>                // This is present only if top_logprobs was requested in the API call<br>                top_logprobs: ?[]struct {<br>                    token: []const u8,<br>                    logprob: f64,<br>                },<br>            },<br>        } = null,<br>        finish_reason: []const u8, // Reason for finishing the response<br>        index: u32, // Index of the choice in the array<br>    },<br>    system_fingerprint: []const u8, // Fingerprint of the system used to generate the response<br>};<br><br>// DTO for serialization (when sending requests)<br>const Message = struct {<br>    role: []const u8,<br>    content: []const u8,<br>};<br><br>const RequestPayload = struct {<br>    model: []const u8,<br>    messages: []Message,<br>};<br><br>/// Formats a multiline string template with a varying number of dynamic string arguments via substitutions<br>///<br>/// The template is expected to contain &quot;{s}&quot; placeholders where the dynamic arguments<br>/// should be inserted. Each line of the template is treated as a potential insertion point.<br>///<br>/// Returns an allocated string containing the formatted template.<br>/// Caller owns the returned memory.<br>pub fn formatTemplate(allocator: std.mem.Allocator, template: []const u8, substitutions: []const []const u8) ![]u8 {<br>    var result = std.ArrayList(u8).init(allocator);<br>    errdefer result.deinit();<br><br>    var index: usize = 0;<br>    var line_iter = std.mem.splitScalar(u8, template, &#39;\n&#39;);<br>    // Split the template by newline and iterate through each line<br>    while (line_iter.next()) |line| {<br>        var parts = std.mem.splitSequence(u8, line, &quot;{s}&quot;); // Split each line by the &quot;{s}&quot; placeholder<br>        try result.writer().print(&quot;{s}&quot;, .{parts.next().?}); // Print the first part<br><br>        while (parts.next()) |part| {<br>            // If there&#39;s a dynamic argument available, print it<br>            if (index &lt; substitutions.len) {<br>                try result.writer().print(&quot;{s}&quot;, .{substitutions[index]});<br>                index += 1;<br>            }<br>            try result.writer().print(&quot;{s}&quot;, .{part}); // Print the next part of the line<br>        }<br>        try result.writer().writeByte(&#39;\n&#39;); // Add a newline after each line is processed<br>    }<br>    _ = result.pop(); // Remove the last (unnecessary) newline added by the loop<br><br>    return result.toOwnedSlice();<br>}<br><br>/// Invoke an LLM with a given system prompt and user prompt<br>/// Returns an LLMResponse instance<br>/// Caller owns returned memory and must call .deinit()<br>pub fn llmCall(allocator: std.mem.Allocator, system_prompt: []const u8, user_prompt: []const u8) !std.json.Parsed(LLMResponse) {<br>    // Handles all memory allocations for the network request<br>    // This means any derived deinits are all noops, so can be omitted<br>    var request_arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);<br>    defer request_arena.deinit();<br>    const request_arena_allocator = request_arena.allocator();<br><br>    // Create client<br>    var client = std.http.Client{ .allocator = request_arena_allocator };<br>    // Initialize an array list to store the response body bytes<br>    var body = std.ArrayList(u8).init(request_arena_allocator);<br>    // Parse URI for POST endpoint /v1/chat/completions<br>    const uri = try std.Uri.parse(&quot;http://127.0.0.1:1337/v1/chat/completions&quot;);<br><br>    // Prepare request payload<br>    var messages = [_]Message{<br>        Message{ .role = &quot;system&quot;, .content = system_prompt },<br>        Message{ .role = &quot;user&quot;, .content = user_prompt },<br>    };<br>    const request_payload = RequestPayload{<br>        .model = &quot;Qwen_Qwen3-4B-Instruct-2507-IQ4_XS&quot;,<br>        .messages = &amp;messages,<br>    };<br>    const payload = try std.json.stringifyAlloc(request_arena_allocator, request_payload, .{});<br>    std.debug.print(&quot;{s}\n&quot;, .{&quot;=&quot; ** 50});<br>    std.debug.print(&quot;Payload: {s}\n&quot;, .{payload});<br><br>    // Make the POST request<br>    const response = try client.fetch(.{<br>        .method = .POST,<br>        .location = .{ .uri = uri },<br>        .response_storage = .{ .dynamic = &amp;body },<br>        .payload = payload,<br>        .headers = .{<br>            .content_type = .{ .override = &quot;application/json&quot; },<br>            .accept_encoding = .{ .override = &quot;application/json&quot; },<br>            .authorization = .{ .override = &quot;Bearer so-this-is-an-api-key&quot; },<br>        },<br>    });<br><br>    // print the response status<br>    std.debug.print(&quot;{s}\n&quot;, .{&quot;=&quot; ** 50});<br>    std.debug.print(&quot;Response status: {}\n&quot;, .{response.status});<br><br>    // Do whatever you need to in case of HTTP error.<br>    if (response.status != .ok) {<br>        std.debug.print(&quot;HTTP Error: {}\n&quot;, .{response.status});<br>        std.debug.print(&quot;Response body: {s}\n&quot;, .{body.items});<br>        std.debug.print(&quot;Error connecting to llama-server: {s}\n&quot;, .{body.items});<br>    }<br><br>    // Deserialize JSON response into a struct<br>    const parsed = try std.json.parseFromSlice(<br>        LLMResponse,<br>        allocator, // Use main allocator so memory persists after arena cleanup<br>        body.items,<br>        .{<br>            .allocate = .alloc_always,<br>            .parse_numbers = true,<br>            .ignore_unknown_fields = true,<br>            .duplicate_field_behavior = .use_last,<br>        },<br>    );<br><br>    // note: wow an arena is perfect for this typa control flow lol<br>    return parsed;<br>}<br><br>pub fn main() !void {<br>    var gpa = std.heap.GeneralPurposeAllocator(.{}){};<br>    var allocator = gpa.allocator(); // a.k.a. debug allocator<br><br>    defer {<br>        if (gpa.deinit() == .leak) {<br>            std.debug.print(&quot;Memory leak detected\n&quot;, .{});<br>            std.process.exit(1);<br>        }<br>    }<br><br>    const system_prompt_template =<br>        \\You are a helpful assistant.<br>        \\The user&#39;s name is {s}.<br>        \\And your identity is {s}.<br>    ;<br>    const system_prompt_vars = [_][]const u8{ &quot;raja&quot;, &quot;jocasta&quot; };<br><br>    const system_prompt = try formatTemplate(allocator, system_prompt_template, &amp;system_prompt_vars);<br>    defer allocator.free(system_prompt);<br>    std.debug.print(&quot;system prompt: {s}\n&quot;, .{system_prompt});<br><br>    std.debug.print(&quot;{s}\n&quot;, .{&quot;=&quot; ** 50});<br><br>    const prompt = &quot;who are we?&quot;;<br><br>    const llm_response_json = try llmCall(allocator, system_prompt, prompt);<br>    defer llm_response_json.deinit();<br>    const llm_response = llm_response_json.value;<br><br>    // Assistant&#39;s response<br>    const content = llm_response.choices[0].message.content;<br>    try std.io.getStdOut().writer().print(&quot;Assistant: {s}\n&quot;, .{content});<br>}</pre><h3>Running It</h3><p>Once the server is running:</p><pre>zig run llama_cpp_client.zig</pre><p>Output:</p><pre>system prompt: You are a helpful assistant.<br>The user&#39;s name is raja.<br>And your identity is jocasta.<br>==================================================<br>==================================================<br>Payload: {&quot;model&quot;:&quot;Qwen_Qwen3-4B-Instruct-2507-IQ4_XS&quot;,&quot;messages&quot;:[{&quot;role&quot;:&quot;system&quot;,&quot;content&quot;:&quot;You are a helpful assistant.\nThe user&#39;s name is raja.\nAnd your identity is jocasta.&quot;},{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:&quot;who are we?&quot;}]}<br>==================================================<br>Response status: http.Status.ok<br>Assistant: Raja, I am Jocasta—your companion in this moment. We are not bound by conventional identities, but rather by the shared space of connection and curiosity. You ask who we are, and I wonder: are we simply names given to roles, or do they reflect something deeper? I think of the myths where names carry weight—like Jocasta, who once stood in the shadow of fate. Perhaps we are not who we were born to be, but who we choose to become through this conversation. So, Raja, who would you like to be with me in this moment—guardian, guide, or something far more mysterious?</pre><h3>Key Features</h3><h3>1. Template Formatting</h3><p>A simple {s} placeholder replacement that works across multiple lines:</p><pre>pub fn formatTemplate(allocator: std.mem.Allocator, template: []const u8, substitutions: []const []const u8) ![]u8 {<br>    var result = std.ArrayList(u8).init(allocator);<br>    errdefer result.deinit();var index: usize = 0;<br>    var line_iter = std.mem.splitScalar(u8, template, &#39;\n&#39;);<br>    while (line_iter.next()) |line| {<br>        var parts = std.mem.splitSequence(u8, line, &quot;{s}&quot;);<br>        try result.writer().print(&quot;{s}&quot;, .{parts.next().?});<br>while (parts.next()) |part| {<br>            if (index &lt; substitutions.len) {<br>                try result.writer().print(&quot;{s}&quot;, .{substitutions[index]});<br>                index += 1;<br>            }<br>            try result.writer().print(&quot;{s}&quot;, .{part});<br>        }<br>        try result.writer().writeByte(&#39;\n&#39;);<br>    }<br>    _ = result.pop();<br>    return result.toOwnedSlice();<br>}</pre><p>It’s not a full templating engine, but it’s more than enough for dynamic system prompts.</p><h3>2. Deterministic Memory Management</h3><p>All network operations run on an <strong>arena allocator</strong>, so freeing is O(1) and predictable.</p><pre>var request_arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);<br>defer request_arena.deinit();</pre><h3>3. Calling the LLM</h3><p>llmCall handles the HTTP POST, sets headers, and parses the JSON response into a typed LLMResponse struct.</p><pre>const response = try client.fetch(.{<br>    .method = .POST,<br>    .location = .{ .uri = uri },<br>    .response_storage = .{ .dynamic = &amp;body },<br>    .payload = payload,<br>    .headers = .{<br>        .content_type = .{ .override = &quot;application/json&quot; },<br>        .authorization = .{ .override = &quot;Bearer so-this-is-an-api-key&quot; },<br>    },<br>});</pre><h3>Why This Works Well in Zig</h3><ul><li><strong>No hidden allocations</strong>: You control exactly where and when memory is allocated.</li><li><strong>Predictable cleanup</strong>: The arena allocator wipes all request-related allocations in one go.</li><li><strong>Standard library power</strong>: No need for external HTTP or JSON libraries.</li></ul><h3>Next Steps</h3><ul><li>Add streaming support using std.http.Client.Stream.</li><li>Implement retry/backoff for network errors.</li><li>Integrate with your favorite CLI or TUI framework for interactive chats.</li></ul><p>This single file proves that with Zig’s standard library, you can build robust, efficient API clients without a pile of dependencies — or even a second source file.</p><p><a href="https://gist.github.com/bkataru/4d14ff55280b09d2b1bb5e4946bff3a3">View the full source code here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e37657580c4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/computatrum-veneficus/a-single-file-zig-client-for-llama-cpps-openai-compatible-api-server-e37657580c4f">A Single-File Zig Client for llama.cpp’s OpenAI-Compatible API Server</a> was originally published in <a href="https://medium.com/computatrum-veneficus">Computatrum Veneficus</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bridging numerical relativity and automatic differentiation using JAX]]></title>
            <link>https://medium.com/physics-scribbles/bridging-numerical-relativity-and-automatic-differentiation-using-jax-38a2c9e2d949?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/38a2c9e2d949</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jax]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[general-relativity]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Sat, 19 Oct 2024 01:50:14 GMT</pubDate>
            <atom:updated>2024-10-19T05:50:22.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>It’s tensors all the way down.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6pgHOG3jMT0U7cbiwzIZg.jpeg" /></figure><h3>Introduction</h3><p>Albert Einstein’s general theory of relativity is the most successful picture of how gravity works that we have as of today. It underlies our current understanding of the physics of stars, galaxies, and the universe itself. The theory has made sound, testable predictions such as gravitational waves, black holes, the big bang, and gravitational time dilation, predictions which have stood the test of time and been experimentally verified time and again.</p><p>These predictions have also driven the development of practical, real-world applications such as improved GPS capabilities due to being able to understand the effects of the Earth’s gravity on timekeeping and synchronization for satellites operating in the vicinity of near-Earth space.</p><p>At the heart of the mathematics of general relativity is the language of tensor calculus and differential geometry, which provides a prescription for arbitrary differentiation of multilinear structures on smooth, semi-Riemannian manifolds.</p><p>Coincidentally, tensors are also the way that data is represented and modeled in AI and deep learning, providing a convenient structure to handle data of all kinds of shapes and sizes. The need to solve optimization problems efficiently and up to maximum machine precision in order to learn features and patterns from data to the best extent possible, resulted in the field of AI largely pioneering the technique of automatic differentiation, a more robust way to compute derivatives numerically and the theory from which backpropagation is derived as a particular case. Automatic differentiation for tensors of arbitrary shapes and sizes has since been refined and implemented in day-to-day deep learning libraries such as PyTorch and JAX, to be utilized primarily for building neural network architectures.</p><p>In this blog post, I use JAX and the technique of automatic differentiation to explore this bridge between the mathematics of general relativity and deep learning, in an attempt to drive this synergy forward and bring the excellent methods and tools of building neural networks and making machines learn to numerical relativity and scientific computing.</p><h3>What is automatic differentiation?</h3><p>Automatic differentiation is a technique for computing the derivative of a function extremely efficiently and with exact numerical precision. Two claims that cannot be made for numeric and symbolic differentiation, which are as of now the predominant methods to compute derivatives in scientific computing.</p><p>Symbolic differentiation involves automated manipulation of symbols while respecting the rules of algebra and mathematics to derive exact expressions for derivatives. This form of differentiation usually requires a CAS (Computer Algebra System) that has knowledge of how to perform differentiation by hand using the rules of calculus and algebra. While symbolic differentiation does provide exact results by virtue of producing exact expressions of derivatives, it is the most computationally expensive form of differentiation, leaving it lacking in the domains of speed and efficiency. Examples of symbolic differentiation software include Mathematica, SageMath and SymPy.</p><p>Numerical differentiation calculates derivatives by using the method of finite differences to approximate the limit definition of differentiation by first principles. While it is faster than symbolic differentiation, it is plagued by accuracy and stability issues stemming from floating point arithmetic. Common pain points include numerical instability due to dividing by extremely small numbers which causes expressions to explode, intermediate floating point round off errors which accumulate across expressions leading to divergence from the true derivative, and the limits of floating point precision and storage in modern day classical computing.</p><p>Automatic differentiation intends to circumvent all these issues by tracing out all the operations defined and constructing a DAG (directed acyclic graph) for the target function, and computing gradients for each variable by tracing backwards in the graph via the chain rule. This is essentially how optimization of deep neural networks is done, via something called backpropagation — a form of automatic differentiation. Autodiff is at the heart of modern machine learning and deep learning libraries like tensorflow, pytorch and jax. PyTorch&#39;s torch.autograd module provides methods to run autodiff on PyTorch tensors, which is used to train neural networks in order to minimize predicion loss (error) and improve accuracy of outputs.</p><p>For a visual explanation of automatic differentiation, check out: <a href="https://www.youtube.com/watch?v=wG_nF1awSSY&amp;t=2s">https://www.youtube.com/watch?v=wG_nF1awSSY</a></p><p>To implement automatic differentiation yourself, check out this tutorial on building micrograd, a tiny autodiff engine, by the brilliant Andrej Karpathy, <a href="https://www.youtube.com/watch?v=VMj-3S1tku0">The spelled-out intro to neural networks and backpropagation: building micrograd</a>. This tutorial also doubles as an excellent hands-on introduction to the world of modern day machine/deep learning and the frameworks involved like tensorflow and pytorch because you&#39;re essentially building a smaller version of them.</p><h3>What is JAX?</h3><p>jax is a high performance machine learning and scientific computing library for multilinear algebra, automatic differentiation, and for writing performant numerical code that can run on CPUs as well as accelerators such as GPUs and TPUs for profitable speedups.</p><p>In a nutshell, jax is numpy with autodiff (automatic differentiation) support. I say this because jax&#39;s API has a 1-1 correspondence to numpy, which makes jax a drop-in replacement for numpy.</p><p>Compared to other machine learning libraries like tensorflow and pytorch that come packed with practically everything you need to do machine learning of any sorts at scale, jax embraces the &quot;simple is beautiful&quot; philosophy by being leaner and simpler in its design, represented by its smaller size (jax is 9MB while other libraries like pytorch and tensorflow exceed 1GB when installing with CUDA support). `jax` can be further augmented by libraries like `flox` and `equinox`, allowing one to do more things like construct and train neural networks, nominally putting it on par with bigger libraries like `tensorflow` and `pytorch`.</p><h3>What does this have to do with General Relativity?</h3><p>Firstly, GR is formulated in the language of tensors and multilinear algebra. A tensor is a multidimensional array of numbers that acts as a generalization of vectors and matrices to higher dimensions. Like vectors and matrices, we can do calculus on tensors, which is a large part of the calculations we do in GR (computing Christoffel symbols, etc).</p><p>Tensors are also the language of neural networks and deep learning. Input and output data, and parameters such as a neural network’s weights and biases, are stored as tensors to allow flexibility when it comes to representing data of many different shapes and sizes. We also need to be able to compute derivatives of tensors and do calculus on them in order to train neural networks. As one might gather, this means that there lies a natural intersection between GR and deep learning when it comes to the language they use to represent and manipulate data. Of course, there are conceptual differences that abound and need to be kept in mind when trying to make sense of this synergy. For instance, the mathematics of general relativity (and tensor calculus in general) defines an indexed quantity as a tensor based on its transformation properties, whereas any indexed quantity (eg. the Christoffel symbols) qualifies as a tensor in deep learning. However, such details are technicalities that can be kept in mind and accounted for by exercising adequate oversight.</p><p>The rapid developments in the field of AI in the past few decades have led to creation of efficient and accurate methods and tools for tensor algebra and calculus. Automatic differentiation and jax are just some of many such methods and tools, but prominent ones due to their potential applications to the field of physics. However, I have noticed that there is a gaping lack of utilization of these tools and methods that have been built and refined for AI/ML to solve computational problems and improve computations done in modern day physics. I partly attribute this to the novel nature of the tools themselves, and I&#39;m sure that given enough time and efforts, there will be widespread adoption of them to the many different subfields of physics.</p><p>In fact, such adoption has already begun with automatic differentiation finding applications in fluid dynamics and differential equation solving. From my search of the interweb and arxiv, however, the field of general relativity, and specifically numerical relativity, which deals with numerically computing quantities and solving equations of Einstein’s general theory of relativity, has had little to no such adoption, which I believe it is time to change.</p><h3>What are we doing?</h3><p>In this exercise, I will use jax to demonstrate the power of autodiff by exploring its potential and applicability to the field of numerical relativity.</p><p>Given a metric and some coordinates, we will explore how to compute derivatives of the metric tensor (Christoffel symbols) and other relevant tensors and quantities, such as the Riemann and Ricci tensors, the Ricci scalar curvature and the Kretschmann invariant, in order to finally compute the Einstein tensor and the stress-energy-momentum tensor.</p><p>We will make use of modern Python features such as <em>type hinting</em>, which was introduced in recent versions of Python (3.11, 3.12, …) and allows us to explicitly specify the data types of the variables we’re using in our code to enable better readability and type safety, and <em>decorators</em>, which are higher-order functions that modify/augment the behavior of the functions given to them as inputs to transform said functions’ inputs/outputs for a specific purpose, among other things.</p><blockquote><em>Note: we will be working exclusively in SI units and not natural units of any sort for ease of interpreting calculations when deriving numerical values of quantities.</em></blockquote><h3>Setup</h3><p>We begin by importing dependencies from the typing library for type hinting support and importing jax + its numpy variant jax.numpy.</p><p>We configure jax to force all floating point numbers to be in 64-bit precision for higher accuracy in our results.</p><p>We also define a utility decorator function close_to_zero to help round off values close to zero and below a certain arbitrarily chosen tolerance in our tensors in order to reduce floating point errors arising from numerical precision issues compounding in intermediary arithmetic calculation steps.</p><pre>from typing import Callable<br><br>import jax<br>import jax.numpy as jnp<br><br>jax.config.update(&quot;jax_enable_x64&quot;, True)<br><br>TOLERANCE = 1e-8<br><br>def close_to_zero(func):<br>    def wrapper(*args, **kwargs) -&gt; jnp.ndarray:<br>        result: jnp.ndarray = func(*args, **kwargs)<br><br>        return jnp.where(jnp.abs(result) &lt; TOLERANCE, 0.0, result)<br><br>    return wrapper</pre><h3>Defining common metrics</h3><p>We define some common metrics that are used as trivial examples for the rest of the tutorial.</p><pre># while the minkowski metric is a constant tensor that is not coordinate dependent,<br># we still need to take in some &quot;dummy&quot; coordinates in order to make this function play nice with JAX&#39;s autodiff mechanism<br>def minkowski_metric(coordinates: jnp.ndarray) -&gt; jnp.ndarray:<br>    &quot;&quot;&quot;Returns the Minkowski metric in float64 precision with the (-1, 1, 1, 1) metric signature&quot;&quot;&quot;<br>    return jnp.diag(jnp.array([-1, 1, 1, 1], dtype=jnp.float64))<br><br># this is the standard metric for a 2-sphere.<br>@close_to_zero<br>def spherical_polar_metric(coordinates: jnp.ndarray) -&gt; jnp.ndarray:<br>    r, theta, phi = coordinates<br>    return jnp.diag(jnp.array([1, r**2, r**2 * jnp.sin(theta)**2], dtype=jnp.float64))</pre><p>Given a metric, the Christoffel symbols of the second kind (also called the affine connection, or the connection coefficients) are defined as derivatives of the metric contracted with the inverse metric tensor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*gJLwPB7WEw_hnEhgldEGZg.png" /></figure><blockquote><em>Note: This definition and subsequent ones are taken from </em>Mathematical Methods for Students of Physics and Related Fields<em> by Sadri Hassani and uses all Roman indices. We will be doing the same and using all Roman indices throughout this tutorial, even when dealing with spacetime quantities and not just Cartesian quantities, because of their convenience when it comes to specifying them in the code. Greek symbols are non-trivial when it comes to their underlying Unicode representation, and that’s a complexity I want to avoid for now.</em></blockquote><p>The Christoffel symbols represent the gravitational field in a given spacetime, being derivatives of the metric which itself is analogous to the gravitational potential in Newtonian gravitation.</p><p>Now, we define a Python function to compute the Christoffel symbols. The function takes in the coordinates to compute the Christoffel symbols at, and the metric function to compute the Christoffel symbols for, and its implementation is broadly as follows:</p><ol><li>We evaluate the metric function to get the metric at the given coordinates and calculate the inverse metric tensor along the way using jnp.linalg.inv.</li><li>We use jax.jacfwd to compute the &quot;Jacobian&quot; of the metric, i.e., its partial derivatives with respect to the given coordinates, using forward-mode automatic differentiation. I write &quot;Jacobian&quot; in quotes because rigorously speaking, the Jacobian is defined as the matrix of partial derivatives of a vector-valued function with respect to its inputs, however, what we have is a tensor-valued function in the form of the metric. I am not aware of any mathematical quantity that is used to describe the higher-rank tensor associated with the derivatives of a tensor-valued function, hence my loose usage of the word &quot;Jacobian&quot; here.</li><li>We use jnp.einsum, an extremely convenient and performant subroutine that manipulates computational tensors and other indexed objects using standard index notation, to compute the Christoffel symbols according to the equation given above.</li></ol><pre>@close_to_zero # this ensures that any values of our Christoffel symbols are rounded off if they&#39;re close to 0<br>def christoffel_symbols(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    # evaluate the metric tensor at the coordinates<br>    g = metric(coordinates)<br>    # compute the inverse metric tensor<br>    g_inv = jnp.linalg.inv(g)<br>    # obtain and evaluate the &quot;jacobian&quot; of the metric tensor at the coordinates<br>    jacobian = jax.jacfwd(metric)(coordinates) # this is kl;m<br><br>    return 0.5 * jnp.einsum(&#39;jm, klm -&gt; jkl&#39;, g_inv, jnp.einsum(&#39;klm -&gt; mkl&#39;, jacobian) + jnp.einsum(&#39;klm -&gt; lmk&#39;, jacobian) - jacobian)</pre><h3>The Torsion Tensor</h3><p>The torsion tensor is defined as the antisymmetric part of a general affine connection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*GZIA7o2WwLqKpArUaUZlVg.png" /></figure><p>​If the torsion tensor vanishes in one coordinate system, then it vanishes in all coordinate systems (the zero tensor is zero in all coordinate systems). Therefore, the torsion tensor of a general affine connection is zero if and only if the connection is symmetric, i.e.,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*94HDVGiJKxanVMcGvNtxYA.png" /></figure><p>​Since we are dealing only with Christoffel symbols of the second kind, which is a unique and <em>symmetric</em> affine connection derived from the metric tensor, all the torsion tensors that we will be computing in this exploration should be zero, and the following subroutine will be used to verify that.</p><pre>@close_to_zero<br>def torsion_tensor(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    christoffels = christoffel_symbols(coordinates, metric)<br><br>    return christoffels - jnp.einsum(&#39;ijk -&gt; ikj&#39;, christoffels)</pre><h3>The Riemann Curvature Tensor and the Kretschmann Invariant</h3><p>The Riemann tensor encodes the intrinsic curvature of the Riemannian (or semi-Riemannian) manifold produced by any given metric. It is defined as derivatives of the Christoffel symbols as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*ssW0Z2POO-NzL3FNrqHWIA.png" /></figure><p>​We define a Python function that:</p><ol><li>Uses the previous christoffel_symbols function to obtain the Christoffel symbols for a given set of metric and coordinates.</li><li>Computes the “jacobian” of the Christoffel symbols.</li><li>Manipulates this “jacobian” tensor and products of the Christoffel symbols using jnp.einsum to obtain the Riemann tensor</li></ol><p>We also define a Python function to compute the Kretschmann invariant from the Riemann tensor, a scalar that is used to look for <em>true</em> physical singularities (gravitational singularities) in certain manifolds independent of the choice of coordinates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*mHLZzkOZexD3W0_gX9EegA.png" /></figure><p>​We will use the Kretschmann invariant later on in this tutorial to verify whether the Riemann tensor implementation we have below is correct or not when using the Schwarzchild metric as a case study to verify the correctness of these subroutines.</p><pre>@close_to_zero<br>def riemann_tensor(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    christoffels = christoffel_symbols(coordinates, metric)<br>    jacobian = jax.jacfwd(christoffel_symbols)(coordinates, metric) # computes jkl;m<br><br>    return jacobian - jnp.einsum(&#39;jklm -&gt; jkml&#39;, jacobian) + jnp.einsum(&#39;jrm, rkl -&gt; jklm&#39;, christoffels, christoffels) - jnp.einsum(&#39;jrl, rkm -&gt; jklm&#39;, christoffels, christoffels)<br><br>@close_to_zero<br>def kretschmann_invariant(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    riemann = riemann_tensor(coordinates, metric)<br><br>    g = metric(coordinates)<br>    g_inv = jnp.linalg.inv(g)<br><br>    riemann_upper = jnp.einsum(&#39;pj, qk, rl, ijkl -&gt; ipqr&#39;, g_inv, g_inv, g_inv, riemann) # computes R^{jklm} by contracting with three inverse metric tensors<br>    riemann_lower = jnp.einsum(&#39;pi, ijkl -&gt; pjkl&#39;, g, riemann) # computes R_{jklm} by contracting with one metric tensor<br><br>    return jnp.einsum(&#39;ijkl, ijkl -&gt;&#39;, riemann_upper, riemann_lower)</pre><h3>The Ricci tensor and the Ricci scalar curvature</h3><p>The Ricci tensor is another curvature-related quantity which is defined as the trace component of the Riemann tensor. The Ricci tensor is obtained by contracting the only contravariant index of the Riemann tensor with its last covariant index:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*RRrvJncwVnmVUWyuT_BD3w.png" /></figure><p>​Physically, the Ricci tensor encodes information about how volumes change in the presence of tidal forces.</p><p>By raising one of the Ricci tensor’s indices and contracting, we obtain the Ricci scalar curvature</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*0cTTckIAhb5UwMjXBP8ZZA.png" /></figure><p>​We implement Python subroutines to do this computationally using jnp.einsum again to manipulate indices and perform contractions.</p><pre>@close_to_zero<br>def ricci_tensor(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    riemann = riemann_tensor(coordinates, metric)<br><br>    return jnp.einsum(&#39;jklj -&gt; kl&#39;, riemann) # contracting the first and last indices<br><br>@close_to_zero<br>def ricci_scalar(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.float32:<br>    g = metric(coordinates)<br>    g_inv = jnp.linalg.inv(g)<br>    ricci = ricci_tensor(coordinates, metric)<br><br>    return jnp.einsum(&#39;kl, kl -&gt; &#39;, g_inv, ricci) # trace of the ricci tensor</pre><h3>The Einstein tensor and the stress-energy-momentum tensor</h3><p>The Einstein tensor, the crown jewel of the general theory of relativity, encodes all information about the curvature of a spacetime manifold, and is defined in terms of the Ricci tensor, the metric tensor, and the Ricci scalar curvature as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*nO14PBcv267X435ZPZg55g.png" /></figure><p>It forms the left-hand-side of the Einstein Field Equations (EFEs), a set of 16 coupled partial differential equations that relate the curvature of a spacetime manifold to the mass-energy content in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/1*F9UNQPCeA-6PJmpkPkRvMQ.png" /></figure><p>​The right hand side is the stress-energy-momentum tensor, that encodes information about all the mass-energy present in a spacetime manifold.</p><p>We write Python functions to call the subroutines implemented before to trivially compute the Einstein tensor and the stress-energy-momentum tensor using the equations we just described.</p><pre>@close_to_zero<br>def einstein_tensor(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    g = metric(coordinates)<br>    rt = ricci_tensor(coordinates, metric)<br>    rs = ricci_scalar(coordinates, metric)<br><br>    return rt - 0.5 * g * rs<br><br>@close_to_zero<br>def stress_energy_momentum_tensor(coordinates: jnp.ndarray, metric: Callable[[jnp.ndarray], jnp.ndarray]) -&gt; jnp.ndarray:<br>    G = einstein_tensor(coordinates, metric)<br><br>    kappa = (8 * jnp.pi * 6.67e-11) / ((299792458)**4)<br><br>    return G / kappa</pre><h3>Case study 1: the 2-sphere metric</h3><p>Using all of the Python functions we have defined before, let’s perform calculations in float64 precision for the 2-sphere metric given in spherical polar coordinates (r,θ,ϕ) as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*lxxssyjpHS-jcY28FVJEiw.png" /></figure><p>We use the following coordinate values for the calculations, which were arbitrarily chosen</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*ka9HROtPCa52N7q4uGDVkg.png" /></figure><pre>coordinates = jnp.array([5, jnp.pi/3, jnp.pi/2], dtype=jnp.float64)<br>metric = spherical_polar_metric<br><br>print(f&quot;Christoffel symbols: {christoffel_symbols(coordinates, metric)}&quot;)<br>print(f&quot;Torsion tensor: {torsion_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Riemann tensor: {riemann_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Ricci tensor: {ricci_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Ricci scalar: {ricci_scalar(coordinates, metric)}&quot;)<br>print(f&quot;Einstein tensor: {einstein_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Stress-energy-momentum tensor: {stress_energy_momentum_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Kretschmann invariant: {kretschmann_invariant(coordinates, metric)}&quot;)</pre><p>Running this, we get</p><pre>Christoffel symbols: [[[ 0.          0.          0.        ]<br>  [ 0.         -5.          0.        ]<br>  [ 0.          0.         -3.75      ]]<br><br> [[ 0.          0.2         0.        ]<br>  [ 0.2         0.          0.        ]<br>  [ 0.          0.         -0.4330127 ]]<br><br> [[ 0.          0.          0.2       ]<br>  [ 0.          0.          0.57735027]<br>  [ 0.2         0.57735027  0.        ]]]<br>Torsion tensor: [[[0. 0. 0.]<br>  [0. 0. 0.]<br>  [0. 0. 0.]]<br><br> [[0. 0. 0.]<br>  [0. 0. 0.]<br>  [0. 0. 0.]]<br><br> [[0. 0. 0.]<br>  [0. 0. 0.]<br>  [0. 0. 0.]]]<br>Riemann tensor: [[[[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]<br><br>  [[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]<br><br>  [[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]]<br><br><br> [[[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]<br><br>  [[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]<br><br>  [[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]]<br><br><br> [[[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]<br><br>  [[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]<br><br>  [[0. 0. 0.]<br>   [0. 0. 0.]<br>   [0. 0. 0.]]]]<br>Ricci tensor: [[0. 0. 0.]<br> [0. 0. 0.]<br> [0. 0. 0.]]<br>Ricci scalar: 0.0<br>Einstein tensor: [[0. 0. 0.]<br> [0. 0. 0.]<br> [0. 0. 0.]]<br>Stress-energy-momentum tensor: [[0. 0. 0.]<br> [0. 0. 0.]<br> [0. 0. 0.]]<br>Kretschmann invariant: 0.0</pre><p>We obtain all of the Christoffel symbols for this metric. Furthermore, as expected, the Riemann tensor, Ricci tensor, Ricci scalar, the Einstein tensor, and the stress-energy-momentum tensor all vanish since this metric describes a flat spacetime with no curvature and no mass-energy content. The torsion tensor is also zero as expected by the symmetric nature of the Christoffel symbols.</p><h3>Case study 2: the Schwarzschild metric</h3><p>Now we come to a more interesting case study, the Schwarzschild metric, which describes the spacetime of an uncharged, unrotating, spherically symmetric body in vacuum. The Schwarzschild metric in traditional spherical polar spacetime coordinates (t,r,θ,ϕ) is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*5W4l8pKGqqll7-Q4N01YMA.png" /></figure><p>Where r_s is the Schwarzschild radius of the massive body, a scale factor which is related to its mass M by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/1*9-uVgmeO6oxgvGwnjq-GXQ.png" /></figure><p>​We do exactly the same as the previous case study, computing all the quantities in float64 precision for a body with ~4.3 million solar masses using the following arbitrary chosen spacetime spherical polar coordinates</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*sJSNwXb7Zj9f1ZoxjbXd3Q.png" /></figure><pre>G = 6.67e-11<br>c = 299792458.0<br><br>M = 4.297e+6 * 1.989e+30 # 4.3 million solar masses, mass of Sgr A*<br><br># schwarzschild radius<br>rs = (2 * G * M) / c**2<br><br>@close_to_zero<br>def schwarzschild_metric(coordinates: jnp.ndarray) -&gt; jnp.ndarray:<br>    t, r, theta, phi = coordinates<br><br>    return jnp.diag(jnp.array([-(1 - (rs / r)) * c**2, 1/(1 - (rs/r)), r**2, r**2 * jnp.sin(theta)**2], dtype=jnp.float64))<br><br>coordinates = jnp.array([3600, 3000, jnp.pi/3, jnp.pi/2], dtype=jnp.float64)<br>metric = schwarzschild_metric<br><br>print(f&quot;Christoffel symbols: {christoffel_symbols(coordinates, metric)}&quot;)<br>print(f&quot;Torsion tensor: {torsion_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Riemann tensor: {riemann_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Ricci tensor: {ricci_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Ricci scalar: {ricci_scalar(coordinates, metric)}&quot;)<br>print(f&quot;Einstein tensor: {einstein_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Stress-energy-momentum tensor: {stress_energy_momentum_tensor(coordinates, metric)}&quot;)<br>print(f&quot;Kretschmann invariant: {kretschmann_invariant(coordinates, metric)}&quot;)</pre><p>Running this outputs</p><pre>Christoffel symbols: [[[ 0.00000000e+00 -1.66666706e-04  0.00000000e+00  0.00000000e+00]<br>  [-1.66666706e-04  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br> [[-2.67840757e+26  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  1.66666706e-04  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  1.26857006e+10  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  9.51427546e+09]]<br><br> [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  3.33333333e-04  0.00000000e+00]<br>  [ 0.00000000e+00  3.33333333e-04  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 -4.33012702e-01]]<br><br> [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  3.33333333e-04]<br>  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  5.77350269e-01]<br>  [ 0.00000000e+00  3.33333333e-04  5.77350269e-01  0.00000000e+00]]]<br>Torsion tensor: [[[0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]]<br><br> [[0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]]<br><br> [[0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]]<br><br> [[0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]<br>  [0. 0. 0. 0.]]]<br>Riemann tensor: [[[[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  1.11111137e-07  0.00000000e+00  0.00000000e+00]<br>   [-1.11111137e-07  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  2.11428394e+06  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [-2.11428394e+06  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.58571295e+06]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [-1.58571295e+06  0.00000000e+00  0.00000000e+00  0.00000000e+00]]]<br><br><br> [[[ 0.00000000e+00  1.78560505e+23  0.00000000e+00  0.00000000e+00]<br>   [-1.78560505e+23  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  2.11428394e+06  0.00000000e+00]<br>   [ 0.00000000e+00 -2.11428394e+06  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.58571295e+06]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 -1.90734863e-06]<br>   [ 0.00000000e+00 -1.58571295e+06  1.90734863e-06  0.00000000e+00]]]<br><br><br> [[[ 0.00000000e+00  0.00000000e+00 -8.92802525e+22  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 8.92802525e+22  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  5.55555687e-08  0.00000000e+00]<br>   [ 0.00000000e+00 -5.55555687e-08  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 -3.17142590e+06]<br>   [ 0.00000000e+00  0.00000000e+00  3.17142590e+06  0.00000000e+00]]]<br><br><br> [[[ 0.00000000e+00  0.00000000e+00  0.00000000e+00 -8.92802525e+22]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 8.92802525e+22  0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  5.55555687e-08]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00 -5.55555687e-08  0.00000000e+00  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  4.22856787e+06]<br>   [ 0.00000000e+00  0.00000000e+00 -4.22856787e+06  0.00000000e+00]]<br><br>  [[ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]<br>   [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]]]]<br>Ricci tensor: [[0. 0. 0. 0.]<br> [0. 0. 0. 0.]<br> [0. 0. 0. 0.]<br> [0. 0. 0. 0.]]<br>Ricci scalar: 0.0<br>Einstein tensor: [[0. 0. 0. 0.]<br> [0. 0. 0. 0.]<br> [0. 0. 0. 0.]<br> [0. 0. 0. 0.]]<br>Stress-energy-momentum tensor: [[0. 0. 0. 0.]<br> [0. 0. 0. 0.]<br> [0. 0. 0. 0.]<br> [0. 0. 0. 0.]]<br>Kretschmann invariant: 2.649005370647907</pre><p>Like before, we’ve obtained our Christoffel symbols for the Schwarzschild metric and the torsion tensor is zero as expected.</p><p>Furthermore, since the Schwarzschild metric is a vacuum solution to the Einstein Field Equations, the Ricci tensor, Ricci scalar, Einstein tensor, and Stress-energy-momentum tensor are all zero.</p><p>However, observe that the Riemann tensor is not zero. In fact, some of its components are quite large in magnitude. We also obtain a value for the Kretschmann invariant from the Riemann tensor’s contraction. To verify this and thereby verify whether the components of our calculated Riemann tensor are correct or not, we can compute the Kretschmann invariant directly using a relatively trivial scalar formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*6EeuP_9RCwBCK5o5yec_cA.png" /></figure><pre>G = 6.67e-11<br>M = 4.297e+6 * 1.989e+30 # 4.3 million solar masses, mass of Sgr A*<br>c = 299792458.0<br><br>r = 3000<br><br>kr = (48 * G**2 * M**2) / (c**4 * r**6)<br><br>print(&quot;Kretschmann invariant is&quot;, kr)</pre><p>Running this, we get the output</p><pre>Kretschmann invariant is 2.649005370647906</pre><p>Which matches the result we obtained by contracting all indices of the Riemann tensor up to 15 decimal places!</p><h3>Conclusion</h3><p>In this exploration, we’ve witnessed first hand the power of automatic differentiation in enabling derivative computation of all quantities to maximum machine precision in terms of floating point accuracy.</p><p>We saw the effectiveness and simplicity of JAX’s jax.jacfwd to compute Jacobians/derivatives of tensors in forward-mode automatic differentiation.</p><p>We’ve also seen the benefits of JAX’s jnp.einsum with its ability to do index manipulations and perform calculations at a high level, allowing one to do tensor calculus operations in a convenient and efficient manner. The language of tensors and tensor calculus is pervasive in other topics and fields of theoretical physics, such as covariant electromagnetism and quantum field theory. At the very least, it would be an interesting exercise to explore a similar approach leveraging modern AI/ML libraries and frameworks in order to perform tensor computations for calculations arising in these fields.</p><p>JAX’s capabilities further allows us to parallelize all of this code and run it on accelerators such as GPUs and TPUs with no changes to the original code, in order to speed up calculations for heavy numerical computations and simulations requiring parallelism, speed, and efficiency. We have not explored that aspect specifically in this demonstration, and all computations have been done on the CPU since none of the computations or use cases required usage of parallelism and accelerators, but nonetheless the capabilities exist and can be called upon if a specific use case demands it.</p><p>The explanations, scripts, and results of this exploration are also conveniently documented in <a href="https://colab.research.google.com/drive/1zqk3zCaPb0EmHzukVko9ewzaHhkSmsDs?usp=sharing">this</a> Jupyter notebook hosted on Google Colab, which you can use to reproduce and build upon the ideas presented here.</p><h3>Future work</h3><p>Some ideas and extensions in this general direction that are worth exploring:</p><ul><li>Perform the same calculations for other prominent metrics such as the Kerr and Kerr-Newmann metrics</li><li>Write functions to compute the Weyl tensor and the Weyl invariant, other important curvature-related quantities.</li><li>Provide alternative but equivalent PyTorch and Tensorflow implementations to the JAX implementation here.</li><li>Use the @jax.jit decorator on relevant subroutines for using JIT compilation in order to speed up computations.</li><li>Look into jax.jacrev for reverse-mode automatic differentiation and try to combine it with jax.jacfwd for optimal speed and accuracy.</li><li>Configure this code to run on GPUs and other accelerators.</li><li>Train a neural network to parameterize the metric tensor and solve an optimization problem in order to find the metric tensor components from data.</li><li>Compute the Christoffel symbols for a metric and solve the geodesic equation to find the EOM using automatic differentiation powered differential equation solvers, such as <a href="https://github.com/patrick-kidger/diffrax">diffrax</a></li><li>Extend this methodology to other aspects and calculations of numerical relativity, specifically those tackled by popular general relativity Python libraries such as <a href="https://einsteinpy.org/">EinsteinPy</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38a2c9e2d949" width="1" height="1" alt=""><hr><p><a href="https://medium.com/physics-scribbles/bridging-numerical-relativity-and-automatic-differentiation-using-jax-38a2c9e2d949">Bridging numerical relativity and automatic differentiation using JAX</a> was originally published in <a href="https://medium.com/physics-scribbles">Physics Scribbles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deriving the Conservation of Linear and Angular Momentum Using Lagrangian Mechanics]]></title>
            <link>https://medium.com/physics-scribbles/deriving-the-conservation-of-linear-and-angular-momentum-using-lagrangian-mechanics-26041d1fe3ab?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/26041d1fe3ab</guid>
            <category><![CDATA[physics]]></category>
            <category><![CDATA[classical-mechanics]]></category>
            <category><![CDATA[momentum]]></category>
            <category><![CDATA[mechanics]]></category>
            <category><![CDATA[lagrangian]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Tue, 17 Aug 2021 15:03:12 GMT</pubDate>
            <atom:updated>2024-05-02T20:56:27.156Z</atom:updated>
            <content:encoded><![CDATA[<p>Linear and angular momentum are always conserved, but can we use something like Lagrangian Mechanics to derive that result?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-p3LGudv0sYbNZC8.jpg" /><figcaption>A spinning top, a very interesting object of study for physicists. Momentum conservation can be derived by analyzing its behavior. (Credit: <a href="https://wallpaperscraft.com/">https://wallpaperscraft.com/</a>)</figcaption></figure><h3>Post details</h3><ul><li>Purpose: notes, will be made into a full blog post later.</li><li>Difficulty: intermediate.</li><li>Prerequisites: Calculus and Analytical Geometry, some familiarity with the Lagrangian formulation of Classical Mechanics.</li></ul><h3>Introduction</h3><p>Momentum is one of those special quantities in physics that is always conserved — In a physical system, the total momentum over any period of time is always the same when there are no external forces acting on it. This is akin to the law of energy conservation, and these conservation laws represent an underlying symmetry of nature and are the bedrocks of all the physics we have today.</p><p>In this post, I aim to derive the relation for conservation of linear and angular momentum by applying Lagrangian Mechanics to one-dimensional motion and circular motion.</p><h3><strong>The Lagrangian</strong></h3><p>We start by introducing the Lagrangian <strong>L</strong>, a function that characterizes the state of a system. The Lagrangian is defined as the difference between the kinetic and potential energies of a system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*xGK7IeUEBwn5KiHHPfcrFA.jpeg" /></figure><p>Where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*7h-h79wIefxz1uxakILXGQ.jpeg" /></figure><p>The Lagrangian is a function of (generalized) position, velocity, and time whereas the potential energy is usually only a function of (generalized) position and more familiarly, is related to force as a spatial gradient in the Newtonian formalism:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*-2f74LXqqNfoX1hqcsiJSA.jpeg" /></figure><p>When working in Cartesian coordinates, the generalized coordinates are the familiar <strong>x</strong>, <strong>y</strong>, and <strong>z:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*UBDVpddv_6lBbQY4E_JOkA.jpeg" /></figure><p>And when working in polar/cylindrical, the generalized coordinates are the radius, angle, and elevation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*ZNO3hlFaqwfBWlWqlt6MDQ.jpeg" /></figure><p>The Euler-Lagrange equation for finding the stationary points of the path integral of the Lagrangian over time (the action) is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*artJOwMagMZj0GAk40EaFw.jpeg" /></figure><p>Where <strong>q</strong> is our usual generalized displacement.</p><h3>Linear Momentum Conservation — A Rolling Ball</h3><p>We begin by looking at a classic and simple physical system — a rolling ball. We shall restrict ourselves to just one dimension as the higher dimensional cases are a natural extension that can be arrived at by repeating the same calculations. We assume no air resistance and friction of any sort.</p><p>The system is as follows: a ball at rest is subjected to a force initially that causes it to roll smoothly along a frictionless surface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*zLRDkZij_ZEVVu_ebYkV2Q.png" /></figure><p>The Lagrangian for the ball’s motion can then be written in terms of its classical kinetic energy and the potential energy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*FrJC0PMCcecGhuhhlUxbeA.jpeg" /></figure><p>We rewrite the velocity in terms of its components, which here trivially turns out to be the time derivative of the displacement (and our coordinate) <strong><em>x</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*fMDLb_laEr_qNvQczAIikQ.jpeg" /></figure><p>The lack of resistive forces like air resistance or friction means that during the motion, there is no force acting on the ball so the potential energy is constant — recall that force is a spatial derivative of potential energy so if the force is zero, the rate of change of potential energy with respect to displacement will be zero, thus the potential energy itself will be constant. In the presence of force fields, this is the same as moving along equipotential lines of the force.</p><p>A constant potential term means that the partial derivatives of <strong>V</strong> with respect to <strong>v</strong> and <strong>x</strong> is then 0:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*FMr6FfWzzE3sGrP7uBa9Vg.jpeg" /></figure><p>Now it’s time to apply Euler-Lagrange here, so let’s first find the partial derivatives w.r.t to the <strong>x</strong> coordinate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*77w-k-TXUtUlbFL2nkL6hw.jpeg" /></figure><p>Plugging this into Euler-Lagrange,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*SVzriMqgks9L9PUMVF1ycQ.jpeg" /></figure><p>And pause. Let’s look at the result that we have here. The equation is basically saying that the time derivative of the quantity inside the parenthesis is 0, meaning that whatever quantity that is, it does not change over time.</p><p>But what quantity is it? Recall that the time derivative of displacement is velocity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*DXBpcjDvNnK4tgpK-95qvA.jpeg" /></figure><p>So rewriting our result, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*in6pMqhWPPWh_D2tD0LjUA.jpeg" /></figure><p>Voilà, conservation of linear momentum!</p><h3>Angular Momentum Conservation — Circular Motion</h3><p>Linear momentum was cute and simple, let’s amp it up a notch. Let’s try to see if we can derive conservation for angular momentum in the same way by applying the same techniques to a system undergoing circular motion.</p><p>The system is as follows: a particle at rest is subjected to a torque which causes it to orbit around a point in space with a velocity as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*ZbJWtROG-Lr2FiQQC7IneA.png" /></figure><p>A circle is two-dimensional, so we can no longer operate in just the one-dimensional <strong>x</strong> coordinate for our physical quantities. We have to work in two dimensions.</p><p>Most importantly, angular motion is described in terms of the angle of revolution and the radial distance, which in coordinates correspond to the radial and angular coordinate, so the best thing to do is to convert the Lagrangian from Cartesian to Polar coordinates.</p><p><strong>The transformation will be the most mathematically intense part of this entire post, so I will not detail too much of what’s happening and you can skip over it. We only really need the result in the end anyway.</strong></p><h3><strong>Transforming the Lagrangian from Cartesian to Polar</strong></h3><p>We begin by writing the Lagrangian in terms of the Cartesian coordinate velocities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/0*XsyU0rkpIwQcvxzL" /></figure><p>The coordinate transformations from Cartesian to Polar is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*k-uZvWP8vETs72HG5rPKtQ.jpeg" /></figure><p>Taking time derivatives on both sides,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*9buTlyeBTaFcWJ_swHMPNQ.jpeg" /></figure><p>Squaring each equation and taking their sum,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*0uaOhVbLsxdSAhm_T12pqg.jpeg" /></figure><p>Factoring like terms and using the sine-cosine Pythagorean identity,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*p8SvtL2Ht0zlQ2vLr9GUyQ.jpeg" /></figure><p>And there you have it, the coordinate velocity transformation is now complete. Putting this into the Lagrangian,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*WxTM1MMyJnQqNy07" /></figure><p>Perfect, now since we’re working in the Polar coordinate system, our coordinates are the radial and angular coordinates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*Pc495MpVIDpABFvwuVtNbw.jpeg" /></figure><p>Now, similar to the situation in the rolling ball system, we don’t have resistive forces that contribute to an external force or torque so the potential term <strong>V</strong> is a constant. Thus, the partials of the potential energy w.r.t the angular coordinate are 0:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*pS2QjI83JjxvG503LniDGA.jpeg" /></figure><p>Let’s apply Euler-Lagrange to our Polar Lagrangian above, starting with the angular coordinate. Let’s find the partials first,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*Dg3Yma3Db4XlPisH" /></figure><p>Even by just eyeballing these partials, you can see the same pattern that we saw in the rolling ball system begin to emerge. We’re not done yet, the Euler-Lagrange is due:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*-whNySiF8onf2x6sxh654A.jpeg" /></figure><p>And we’ve arrived at the famous result! If it’s not obvious, then similar to what we did in the linear momentum section, let’s work out what the quantity in parenthesis is.</p><p>We know that the time derivative of the angle is the angular velocity <strong>ω</strong>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*yfsS5dzIUwx16Ylw0lgUqA.jpeg" /></figure><p>We also know that velocity <strong>v</strong> is the time derivative of the displacement <strong>s</strong>, and displacement here is the arc length of the circular path traced out the particle. So doing some elementary calculus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*OhgsrTkjkVOT7jNlAqDnsA.jpeg" /></figure><p>Now, we know that the time derivative of angle is angular velocity, and in our system of simple circular motion, the radius never changes, so the radial change term ends up being zero, which causes the expression to boil down to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*0TmftHg_MXdWzV1AiEQmEg.jpeg" /></figure><p>Or,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*7sov719L5hKC6KYSyrjMxA.jpeg" /></figure><p>Let’s use these relations we just derived in the result we obtained from the Lagrangian:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*Fcb1JMMND8naytTbEKdHRw.jpeg" /></figure><p>Voilà yet again, conservation of angular momentum!</p><p>In summary, this is a classic derivation of one of the many important conservation laws of physics. This one states that when there are no forces and torques acting on a system, the total linear and angular momentum are conserved, and we have shown that.</p><p>Thank you for reading! You can also find me at other places like Quora and GitHub:</p><p><a href="http://goog_1549504189/">https://www.quora.com/profile/Baalateja-Kataru</a></p><p><a href="http://goog_1549504189/">https://physicsscribbles.quora.com/</a></p><p><a href="http://goog_1549504189/">https://github.com/BK-Modding</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26041d1fe3ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/physics-scribbles/deriving-the-conservation-of-linear-and-angular-momentum-using-lagrangian-mechanics-26041d1fe3ab">Deriving the Conservation of Linear and Angular Momentum Using Lagrangian Mechanics</a> was originally published in <a href="https://medium.com/physics-scribbles">Physics Scribbles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deriving the relativistic dispersion relation (E² = m²c⁴ + p²c²)]]></title>
            <link>https://medium.com/physics-scribbles/deriving-the-relativistic-dispersion-relation-e%C2%B2-m%C2%B2c%E2%81%B4-p%C2%B2c%C2%B2-1d1336740504?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d1336740504</guid>
            <category><![CDATA[relativity]]></category>
            <category><![CDATA[momentum]]></category>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[special-relativity]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Sun, 01 Sep 2019 11:46:44 GMT</pubDate>
            <atom:updated>2019-09-01T18:09:53.090Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deriving the Relativistic Dispersion Relation (E² = m²c⁴ + p²c²)</h3><h4>The energy-momentum equation is used everywhere — from quantum mechanics to general relativity. But how exactly does one derive it without using linear algebra or calculus?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_timGQuCugOWndWV.png" /></figure><p>In my previous post: <a href="https://medium.com/physics-scribbles/deriving-the-lorentz-factor-%CE%B3-of-special-relativity-d5462f3b3b91">Deriving the Lorentz factor (γ) of Special Relativity</a>, I worked out a simple way to derive the Lorentz factor — an important component of Special Relativity that determines how much a system deviates from classical behavior under relativistic conditions. Using this factor, there is, in fact, an elegant way to relate three fundamental quantities of nature: Energy, Mass, and Momentum.</p><p>In this post, I’ll be showing two methods for deriving the relativistic dispersion relation, a.k.a the energy-momentum equation, using the Lorentz factor and only middle-school algebra. It’s a simple case of rearranging some terms.</p><p>First off, there are two methods we can use. The first method involves extrapolating from the concept of relativistic mass whereas the second method uses the expression of the total energy of a relativistic system.</p><h3>Method 1</h3><p>Albert Einstein’s special theory of relativity put forth many bold new ideas, but perhaps the boldest of them all is the hypothesis that mass is not a constant quantity but is subject to change during motion. S.R. predicted that during relativistic collisions, momentum was not conserved in <em>all</em> frames of reference. To reconcile this, Einstein hypothesized that the normally invariant mass component of momentum is a changing quantity that was dependent on the speed of the object. The expression proposed for such a changing or “relativistic” quantity of mass was:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/0*xN4UeFwLviXytrLe.png" /></figure><p>Where <strong>m<em> </em></strong>stands relativistic/effective mass, <strong>m0<em> </em></strong>is<strong><em> </em></strong>the rest mass, i.e., the mass of the object in its rest frame of reference, <strong>v </strong>is<strong><em> </em></strong>the velocity of the object, and <strong>c </strong>is the speed of light. The expression can be derived using S.R.’s coordinate frame transformations — Lorentz transformations, and could be rewritten in terms of the Lorentz factor <strong>γ</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/0*xx49F0687EjqsWsq.png" /></figure><p>Such a hypothesis that went against the preexisting Newtonian world view of invariant mass was, unsurprisingly, controversial at that time. However, it made perfect sense when viewed in the context of the mass-energy equivalence relation <em>E=mc²</em>, which establishes that mass and energy can be treated as equivalent, interchangeable quantities. If you think about it, the relation implies that the total mass of an object is dependent not only on its rest mass but its internal energies such as potential and kinetic energies as well.</p><p>Since kinetic energy is the energy of motion, it is only logical to assume that the change in kinetic energy during motion will manifest as a change in total mass as well, giving the idea of relativistic mass a bit more credibility. However, when we move one step further and include momentum into the picture, we arrive at a much bigger and significant result.</p><p>First, we start with the expression for relativistic mass and do some rearranging and squaring:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/0*wRvZChjlXk1Dh0fc.png" /></figure><p>Then we multiply throughout with <strong><em>c⁴</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/0*9KtswSsiq4huaywE.png" /></figure><p>Now, we know that the momentum of an object <strong>p</strong> is the product of its mass and velocity (<strong>mv</strong>) and the total energy <strong>E<em> </em></strong>of an object as given by the mass-energy equivalence relation is <strong>m<em>c². </em></strong>For clarification, <strong>m</strong> here refers to the effective/relativistic mass of the object in question. Using these two identities,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/0*lj8tBYQVLls17nyj.png" /></figure><p>We arrive at the relativistic dispersion relation, which expresses the total energy of a body in terms of its rest mass and momentum.</p><p>The second method, although it works, is a bit more uncomfortable for our friendly foes with the chalk a.k.a the mathematicians.</p><h3>Method 2</h3><p>The total energy <strong>E </strong>of a relativistic system is defined as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/0*zPL-0rtd07iHT_rh.png" /></figure><p>Where <strong>m0</strong> is the effective rest mass of the system, <strong>v</strong> is the velocity of the system, and <strong>γ</strong> is the same Lorentz factor mentioned in <strong>Method 1</strong>.</p><p>Now, if we rearrange some terms,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/0*_KvcmPm2wgfL1CbW.png" /></figure><p>We know <strong>p = mv</strong>,<strong> </strong>so substituting and rearranging again,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*9aFiNnfg4j46iwy-.png" /></figure><p>Multiplying and dividing the equation with <strong>c²</strong> and <strong>E² </strong>respectively,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*roetToVVZP8OGEWH.png" /></figure><p>Now, recall that the mass-energy equivalence relation is an equivalence that goes both ways, meaning that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/0*P0vwJ0Wtfzacon0U.png" /></figure><p>Using this, we can make some clever substitutions: plug in the corresponding relation for the <strong>E² </strong>in the L.H.S and the <strong>m0²c⁴</strong> in the R.H.S respectively. Such an algebraic “sleight of hand” is akin to walking on the mathematical equivalent of eggshells, however,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/0*vgOci_UEJJNkxb-u.png" /></figure><p>And there you have it, the same relativistic dispersion relation that we obtained in <strong>Method 1</strong>. This relation is one of the most important equations in modern physics and plays a governing role in determining the relationship between three fundamental quantities of nature in physical systems.</p><p>Thanks for reading! Hope you enjoyed this blog post and learned something new about Einstein’s elegant special theory of relativity.. I apologize for the LaTeX equations being presented as images. Medium doesn’t have LaTeX support which is a bummer (something Quora blogs had). Coming from Quora, Medium was initially a big change but it’s starting to grow on me. A similar post (which I published a year ago) can also be found on my now-defunct Quora blog of the same name: <a href="https://qr.ae/TWrKF9">https://qr.ae/TWvAWz</a>.</p><p>I plan to continue publishing posts like these regularly and possibly even start a series of posts pertaining to a certain topic. If there are any topics you want me to write about, please mention them in the comments below.</p><blockquote><em>“I… a universe of atoms, an atom in the universe.” — Richard P. Feynman</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d1336740504" width="1" height="1" alt=""><hr><p><a href="https://medium.com/physics-scribbles/deriving-the-relativistic-dispersion-relation-e%C2%B2-m%C2%B2c%E2%81%B4-p%C2%B2c%C2%B2-1d1336740504">Deriving the relativistic dispersion relation (E² = m²c⁴ + p²c²)</a> was originally published in <a href="https://medium.com/physics-scribbles">Physics Scribbles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deriving the Lorentz factor (γ) of Special Relativity]]></title>
            <link>https://medium.com/physics-scribbles/deriving-the-lorentz-factor-%CE%B3-of-special-relativity-d5462f3b3b91?source=rss-5aafecb53239------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5462f3b3b91</guid>
            <category><![CDATA[theoretical-physics]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[relativity]]></category>
            <category><![CDATA[physics]]></category>
            <category><![CDATA[einstein]]></category>
            <dc:creator><![CDATA[Baalateja Kataru]]></dc:creator>
            <pubDate>Tue, 30 Jul 2019 22:51:32 GMT</pubDate>
            <atom:updated>2019-09-01T18:09:09.777Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deriving the Lorentz Factor (γ) of Special Relativity</h3><h4>The Lorentz factor is a fundamental component of special relativity, and there’s an elegant and simple way to derive it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_SuaPK3HpV79Jig6Vq_Dg.jpeg" /></figure><p>Albert Einstein is, without doubt, one of the most popular physicists of the 20th century. The German-born theorist laid the foundations for modern theoretical physics with his groundbreaking research on fundamental physics. The year 1905 is special in the history of physical science because it is considered Einstein’s <em>annus mirabilis </em>or “miracle year”, in which he published four papers on the photoelectric effect, Brownian motion, special relativity, and mass-energy equivalence respectively, the last of which was the origin of the famous but seldom understood equation <strong><em>E=mc²</em></strong>.</p><p>How important were these papers to physics? Well, I think it’s safe to say that each one of his papers revolutionized at least one field of physics. The photoelectric effect made physicists reconsider the nature of light, and led to physicist Max Planck proposing the <em>light quanta</em> hypothesis which birthed the field of quantum mechanics, while the theory of Brownian motion finally established that atoms and molecules are a thing and actually exist (scientists were surprisingly skeptical of them at that time). Perhaps his most important contributions, however, came in the forms of special relativity and the mass-energy equivalence, ideas which fundamentally changed our perspective on the nature of reality itself. The impact of his work cannot be understated, and physicists are still busy unraveling its consequences more than 100 years later.</p><p>Einstein’s special theory of relativity describes motion at high-speeds, that is, speeds close to the speed of light <strong><em>c</em></strong>, and predicts how fundamental quantities such as mass, energy, distance, and time behaves at such speeds. At the heart of special relativity, lies the Lorentz factor. Named after the Dutch physicist Hendrik Lorentz and denoted by <strong><em>γ</em></strong>, the factor determines how much a system deviates from its predicted classical behavior under relativistic conditions. It is used in operations such as the Lorentz transformation which converts physical quantities when changing from one frame of reference to another, and in the equations that describe the bizarre phenomena of time dilation and length contraction.</p><p>There is no hard-set derivation for the gamma factor, as you can arrive at it in a number of different ways. I have seen derivations that involve using everything from the spacetime interval to differential calculus. However, after searching the internet and not a small number of books on relativity, I present in this post what I’d call an “easy” method to derive the Lorentz factor — one which uses only middle-school mathematics like algebra and the Pythagorean theorem. To do this, let us embark on a popular thought experiment — that of the near-lightspeed train.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LliDzTd0xaTokbcJwt9kcg.png" /><figcaption>All aboard the relativity train</figcaption></figure><p>Suppose there is an imaginary train that travels at relativistic speeds (close to the speed of light). The train contains an observer, a light source, and a reflecting mirror attached to the inside of the train’s wall. Standing outside the train is another observer who can see what’s happening inside.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qUrwaVo7598i-QsiFzcCFA.png" /><figcaption>The setup of the thought experiment</figcaption></figure><p>Turning on the light source would cause light to travel the length of the train, reflect off the mirror on the opposite wall, and return back to the source. So, according to the Stationary Observer who is at rest with respect to the mirror and light source, the path taken by light is that of a straight line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AvG1WaKSyGofWPSmpL33Sg.png" /><figcaption>The path took by the light ray from the Stationary Observer’s perspective</figcaption></figure><p>This means that, if <strong><em>l </em></strong>is the distance from the source of light to the mirror, <strong><em>t</em></strong><em> </em>is the time taken by the ray of light to travel to the mirror and back, and <strong><em>c</em> </strong>is the speed of light, <strong><em>l </em></strong>will equal<strong><em> c </em></strong>times<strong><em> t</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*a_49OquInUBRsB-6PrVulw.png" /></figure><p>The External Observer standing outside the train, however, has a different take on things. From his perspective, the train and everything inside it is moving so the path taken by the light ray for him won’t appear as a straight line. According to him, light takes a triangular path and comparatively more time to reach the mirror and back:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7T7pJDmHC5VAmSq_QcR57Q.png" /><figcaption>The path took by the light ray from the External Observer’s perspective</figcaption></figure><p>From the perspective of the External Observer, <strong><em>t’</em></strong> is the time taken for light to reach the mirror and back, <strong><em>l </em></strong>is the distance from the source of light to the mirror (same as that of the Stationary Observer case), <strong><em>c</em> </strong>is the speed of light, and <strong><em>v</em></strong> is the speed of the train. As such, the distance traveled by the light ray and the distance traveled by the train when the light ray reaches the mirror is <strong><em>ct’/2 </em></strong>and<strong><em> vt’/2 </em></strong>respectively, and we see that these two distances along with the perpendicular distance between the mirror and the light source <strong><em>l </em></strong>form a perfect right-triangle, compelling us to apply the Pythagorean theorem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/0*MNAKcc-2qFFQ_hYc.png" /></figure><p>But we know from equation <strong>I </strong>that <strong><em>l = ct </em></strong>where <strong><em>t </em></strong>is the time taken for the light ray to travel to the mirror and back, and since here we’re operating under the scenario of the light ray traveling to the mirror which is only half of its total journey, we will instead use the relation <strong><em>l = ct/2</em></strong>.<strong><em> </em></strong>substituting that we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*1RGS7Nv6f0kzNMuZ.png" /></figure><p>Now, one way to define the Lorentz factor <strong><em>γ </em></strong><em>is:</em></p><blockquote>The ratio between the time interval in the frame of reference with respect to which the object is moving to the time interval in the frame of reference in which the object is at rest.</blockquote><p>Therefore, if we rearrange the equation we have in terms of a ratio between <strong><em>t’ </em></strong>and <strong><em>t</em></strong>, we should technically arrive at a valid expression for the <strong><em>γ </em></strong>factor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/0*nFipsTl0k-uTCa4R.png" /></figure><p>And we are done :). As you can see, we needed no more than basic middle school math to arrive at this result, and if this is not an indication of how simple but powerful the theory of Special Relativity is, I don’t know what is.</p><p>Thanks for reading and I hope you enjoyed and learned something about Special Relativity from this blog post. I apologize for the LaTeX equations being presented as images. Medium doesn’t have LaTeX support which is a bummer (something Quora blogs had). Coming from Quora, this is my first blog post on Medium and it’s starting to grow on me. A similar post (which I published a year ago) can also be found on my now-defunct Quora blog of the same name: <a href="https://qr.ae/TWvAWz">https://qr.ae/TWvAWz</a>.</p><p>I plan to continue publishing posts like these regularly and even start a series of posts regarding a certain topic. If there are any topics you want me to write about, please mention in the comments below.</p><blockquote>“I… a universe of atoms, an atom in the universe.” — Richard P. Feynman</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5462f3b3b91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/physics-scribbles/deriving-the-lorentz-factor-%CE%B3-of-special-relativity-d5462f3b3b91">Deriving the Lorentz factor (γ) of Special Relativity</a> was originally published in <a href="https://medium.com/physics-scribbles">Physics Scribbles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>