<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Davide Marro on Medium]]></title>
        <description><![CDATA[Stories by Davide Marro on Medium]]></description>
        <link>https://medium.com/@davideraffaele.marro?source=rss-3ca6d97d03d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lxr9Nu-U4_L8cfR7</url>
            <title>Stories by Davide Marro on Medium</title>
            <link>https://medium.com/@davideraffaele.marro?source=rss-3ca6d97d03d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:12:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@davideraffaele.marro/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Create an Operating System in Go — Part 2: The shell]]></title>
            <link>https://levelup.gitconnected.com/create-an-operating-system-in-go-part-2-the-shell-69891e3e0ca1?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/69891e3e0ca1</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Sat, 21 Mar 2026 16:39:04 GMT</pubDate>
            <atom:updated>2026-03-23T15:21:11.815Z</atom:updated>
            <content:encoded><![CDATA[<p>Well, if you don’t know my history I’m a simple software engineer trying to built an operating system in Go… that’s crazy!</p><p>It seems to be crazy but, starting from a simple <a href="https://www.reddit.com/r/golang/comments/1q5deef/new_os_kernel_in_go_hobby_project/">Reddit post</a>, I created a sort of a little contributors community and now we’re trying to go ahead with <a href="https://github.com/dmarro89/go-dav-os">Go-Dav-Os</a>.</p><p>Anyway that’s not the point of the story, I’m writing a series of tutorials on how to create an operating system using Go (and a bit of assembly of course).</p><p>In the <a href="https://medium.com/gitconnected/create-an-operating-system-in-go-part-1-af287bfc402a">first part</a>, we booted our kernel, written mostly in assembly, switched to 64-bit long mode and printed some text on screen. Honestly, that already felt magic to me. It could be enough to say: “<em>Ok, my first OS is done, let’s go through the next project</em>” — but it wasn’t the case.</p><p>After the excitement of the first boot, another thought came to my mind: <em>ok, nice… but now how can I improve it?</em></p><p>An operating system that only writes a string on the screen is still very far from being something alive. The next natural step was clear: I wanted to <strong>type something</strong> and see the kernel react.</p><p>Something like that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CYH8D200ye4188kMXkgdAQ.png" /><figcaption>New Generation Shell lol</figcaption></figure><p>So in this second part, we are going to do exactly this: read characters from the keyboard and build a very small shell on top of it, just the first real interaction between the user and the OS.</p><p>And believe me, when you type a key and your own kernel reacts, it feels much better than a simple “Hello world”.</p><h3>Polling — The best first approach</h3><p>Of course, in a more advanced kernel, the keyboard should eventually be handled through <em>interrupts</em>. But to make progress without opening too many doors at the same time, I decided to begin with a more direct approach: keep asking the PS/2 keyboard controller if a key is available, and only continue once it is.</p><p>The idea is simple:</p><ul><li>ask the keyboard controller if data is available</li><li>if not, keep waiting</li><li>if yes, read the scancode</li><li>translate it into a readable character</li><li>return it to the shell</li></ul><p>This approach is obviously not the final destination. The CPU is actively waiting, so it is not as elegant or efficient as an interrupt-driven design.</p><p>But for a first interactive shell, it is actually a great fit.</p><h3>Handling the keyboard typing</h3><p>The keyboard does not send Go strings, characters or anything that nice.</p><p>We have to talk directly to the hardware.</p><p>With the classic PS/2 controller, the important ports are:</p><ul><li>0x64 for the status register</li><li>0x60 for the data register</li></ul><p>The flow is straightforward.</p><p>First, we read the status register and check whether the output buffer contains data. If it does, then we can safely read the incoming byte from port 0x60.</p><p>So, first I created a new <a href="https://github.com/dmarro89/go-os-tutorial/tree/Part-2-Creating-a-Shell/keyboard"><em>keyboard</em></a> folder with a simple <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/keyboard/keyboard.go">keyboard.go</a> file in my <a href="https://github.com/dmarro89/go-os-tutorial">tutorial project</a> branch and started to implement the polling — this time there will be just few lines of assembly.</p><pre>package keyboard<br><br>func inb(port uint16) byte<br>func outb(port uint16, value byte)<br><br>const (<br> portData   uint16 = 0x60<br> portStatus uint16 = 0x64<br> statusOutputBuffer = 1 // bit 0 =&gt; output buffer full<br>)<br><br>func readScancode() byte {<br> for {<br>  status := inb(portStatus)<br>  if (status &amp; statusOutputBuffer) != 0 {<br>   return inb(portData)<br>  }<br> }<br>}</pre><p>But what is the <strong>inb</strong> function?<br>You got it! It’s an assembly function. Let’s see its <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/boot/stubs_amd64.s">implementation</a> but don’t panic it’s very easy to understand.</p><pre>.global github_0com_1dmarro89_1go_x2dos_x2dtutorial_1keyboard.inb<br>.type   github_0com_1dmarro89_1go_x2dos_x2dtutorial_1keyboard.inb, @function<br><br>github_0com_1dmarro89_1go_x2dos_x2dtutorial_1keyboard.inb:<br> movw %di, %dx<br> xor %eax, %eax<br> inb %dx, %al<br> ret<br>.size github_0com_1dmarro89_1go_x2dos_x2dtutorial_1keyboard.inb, . - github_0com_1dmarro89_1go_x2dos_x2dtutorial_1keyboard.inb</pre><p>The function put the parameter into the <em>DX</em> registry then delete the content of EAX and, at the end, calls the <strong>inb</strong> function with the parameter set in <em>DX</em>, storing the resulting value in <em>AL</em>.</p><p>In this way, we created a bridge between the external devices and our Go Code through the assembly code. We can use this function, in our Go program, to get info about what the user is typing on the keyboard.</p><p>Just a note, the byte returned into the <em>EAX</em> registry (yes, <em>AL</em> is part of <em>EAX</em>) is not yet a normal character like a, b or 1. It is a <strong>scancode</strong>, meaning a small numeric code that identifies the pressed key.</p><p>So the next step is translating that value into something readable.</p><h3>From scancode to character</h3><p>Once I had the raw scancode, I needed a way to turn it into something useful for the shell.</p><p>No advanced layout handling, no modifiers, no Shift state management, no Alt combinations. Just a simple lookup table mapping the scancodes I cared about to actual characters.</p><p>So I created a first <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/keyboard/layout.go">layout.go</a> file with the Italian layout, yes I’m Italian — so sorry for the other layouts — in order to get the right values from my Italian keyboard.</p><pre>var LayoutIT = Layout{<br>	0x02: &#39;1&#39;,<br>        0x03: &#39;2&#39;,<br>        0x04: &#39;3&#39;,<br>        0x05: &#39;4&#39;,<br>	// ...<br>	0x10: &#39;q&#39;,<br>        0x11: &#39;w&#39;,<br>        0x12: &#39;e&#39;,<br>        0x13: &#39;r&#39;,<br>        0x14: &#39;t&#39;,<br>        0x15: &#39;y&#39;,<br>	// ...<br>	0x1E: &#39;a&#39;,<br>        0x1F: &#39;s&#39;,<br>        0x20: &#39;d&#39;,<br>        0x21: &#39;f&#39;,<br>	// ...<br>	0x39: &#39; &#39;,<br>        0x1C: &#39;\n&#39;,<br>        0x0E: &#39;\b&#39;,<br>}</pre><p>And then the <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/keyboard/keyboard.go">keyboard reader</a> becomes:</p><pre>func ReadKey() rune {<br> for {<br>  sc := readScancode()<br><br>  if sc&amp;0x80 != 0 {<br>   continue<br>  }<br><br>  if sc &lt; byte(len(LayoutIT)) {<br>   r := LayoutIT[sc]<br>   if r != 0 {<br>    return r<br>   }<br>  }<br> }<br>}</pre><p>There are two important things happening here.</p><p>First, I ignore <strong>key release</strong> events. On a PC keyboard, key release scancodes have the high bit set, and for a tiny shell I simply do not need them yet.</p><p>Second, I return only valid characters. If a key is not mapped in my table, I ignore it and keep waiting.</p><h3>Reading a line</h3><p>Now that the kernel could read one character at a time, the next step was building something a bit more user-friendly on top of it.</p><p>Not a full command line. Just the simplest possible line reader.</p><p>The idea is:</p><ul><li>keep reading keys one by one</li><li>append normal characters to a buffer</li><li>handle backspace</li><li>stop when Enter is pressed</li><li>return the final line to the shell</li></ul><p>That gives us a blocking readLine() , a function I added to the existing <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/shell/shell.go">shell.go</a> file.</p><pre>func readLine(buf []rune) int {<br> pos := 0<br><br> for {<br>  r := keyboard.ReadKey()<br>  switch r {<br>  case &#39;\b&#39;:<br>   if pos &gt; 0 {<br>    pos--<br>    putRune(r)<br>   }<br>  case &#39;\n&#39;:<br>   putRune(r)<br>   return pos<br>  default:<br>   if pos &lt; len(buf) {<br>    buf[pos] = r<br>    pos++<br>    putRune(r)<br>   }<br>  }<br> }<br>}</pre><p>I actually like this phase a lot because it is the first time the OS starts behaving like something you can really use.</p><h3>Building a tiny shell</h3><p>Once I had a function able to read a full input line, the shell became much easier.</p><p>At that point, the shell is basically just a loop:</p><ul><li>print a prompt</li><li>read a line from the keyboard</li><li>parse the command</li><li>execute it</li><li>repeat</li></ul><p>That’s what the <strong>RunShell</strong> method inside the <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/shell/shell.go">shell.go</a> file is doing.</p><pre>func RunShell() {<br> var line [80]rune<br> for {<br>  Print(&quot;&gt; &quot;)<br>  n := readLine(line[:])<br>  execute(line[:n], n)<br> }<br>}</pre><p>And execute() can stay extremely simple at the beginning.</p><p>Take the input string, split it into tokens, look at the first word, and match it against a couple of built-in commands.</p><p>Something like:</p><pre>func execute(buf []rune, n int) {<br> if n == 0 {<br>  return<br> }<br><br> if n == 4 &amp;&amp;<br>  buf[0] == &#39;h&#39; &amp;&amp;<br>  buf[1] == &#39;e&#39; &amp;&amp;<br>  buf[2] == &#39;l&#39; &amp;&amp;<br>  buf[3] == &#39;p&#39; {<br>  Print(&quot;Available commands:\n&quot;)<br>  Print(&quot;  help   - show this help\n&quot;)<br>  Print(&quot;  clear  - clear the screen\n&quot;)<br>  Print(&quot;  about  - info about DavideOS\n&quot;)<br>  return<br> }<br><br> if n == 5 &amp;&amp;<br>  buf[0] == &#39;c&#39; &amp;&amp;<br>  buf[1] == &#39;l&#39; &amp;&amp;<br>  buf[2] == &#39;e&#39; &amp;&amp;<br>  buf[3] == &#39;a&#39; &amp;&amp;<br>  buf[4] == &#39;r&#39; {<br>  Clear()<br>  return<br> }<br><br> if n == 5 &amp;&amp;<br>  buf[0] == &#39;a&#39; &amp;&amp;<br>  buf[1] == &#39;b&#39; &amp;&amp;<br>  buf[2] == &#39;o&#39; &amp;&amp;<br>  buf[3] == &#39;u&#39; &amp;&amp;<br>  buf[4] == &#39;t&#39; {<br>  Print(&quot;Dav-OS-Go tutorial kernel\n&quot;)<br>  return<br> }<br><br> Print(&quot;Unknown command: &quot;)<br> for i := 0; i &lt; n; i++ {<br>  putRune(buf[i])<br> }<br> Print(&quot;\nType &#39;help&#39; for a list of commands.\n&quot;)<br>}</pre><p>As you can see how tiny shell supports just 3 commands: help, clear and about. Nothing of special but now we’ve the template to unlock every possibility and implement every command in our shell.</p><h3>Build and Test</h3><p>The last thing to do is to update the <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/Makefile">Makefile</a> in order to have all the new files compiled and linked in the exact way.</p><p>First, we need to define the import and sources for the new folder “keyboard” then define the names of the built file of that new folder.</p><pre>...<br>MODPATH      := github.com/dmarro89/go-os-tutorial<br>KEYBOARD_IMPORT := $(MODPATH)/keyboard<br>...<br>KEYBOARD_SRCS := $(filter-out %_test.go, $(wildcard keyboard/*.go))<br>...<br>KEYBOARD_OBJ := $(BUILD_DIR)/keyboard.o<br>KEYBOARD_GOX := $(BUILD_DIR)/github.com/dmarro89/go-os-tutorial/keyboard.gox</pre><p>Then we define how to generate these objects.</p><pre>$(KEYBOARD_OBJ): $(KEYBOARD_SRCS) | $(BUILD_DIR)<br> $(GCCGO) $(GCCGOFLAGS) -static -Werror -nostdlib -nostartfiles -nodefaultlibs \<br>  -I $(BUILD_DIR) \<br>  -fgo-pkgpath=$(KEYBOARD_IMPORT) \<br>  -c $(KEYBOARD_SRCS) -o $(KEYBOARD_OBJ)<br><br>$(KEYBOARD_GOX): $(KEYBOARD_OBJ) | $(BUILD_DIR)<br> mkdir -p $(dir $(KEYBOARD_GOX))<br> $(OBJCOPY) -j .go_export $(KEYBOARD_OBJ) $(KEYBOARD_GOX)</pre><p>And, at the end, we can link them together into the kernel elf.</p><pre>$(KERNEL_ELF): $(BOOT_OBJ) $(KEYBOARD_OBJ) $(SHELL_OBJ) $(KERNEL_OBJ) $(LINKER_SCRIPT)<br> $(GCC) -T $(LINKER_SCRIPT) -o $(KERNEL_ELF) \<br>  -ffreestanding -O2 -nostdlib \<br>  $(BOOT_OBJ) $(KEYBOARD_OBJ) $(SHELL_OBJ) $(KERNEL_OBJ) -lgcc</pre><p>Now we can built the project and test the shell.</p><pre>make docker-build-only</pre><p>And the run the iso qith qemu.</p><pre>qemu-system-x86_64 -cdrom build/go-os-tutorial.iso</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*psPiyukfSdPtaSsnfrEj3Q.png" /></figure><h3>What we unlocked</h3><p>This step may look small from the outside, but for me it was a major shift in the feeling of the project.</p><p>With the first part, I had a kernel that could boot and print text.</p><p>With this second part, I now have a kernel that can actually <strong>listen</strong>.</p><p>It can receive keyboard input, buffer it, process it and execute commands through a tiny shell. In other words, the OS stopped being a static demo and started becoming an interactive system.</p><p>This is a naive implementation, easy to understand and to implement.</p><p>In the next story we will improve this implementation using a real interrupts system and the things become to be though.</p><h3>References</h3><p>Guys, a couple of useful references that can make your life easier if you want to implement something.</p><p>This is the <a href="https://github.com/dmarro89/go-os-tutorial">Github project</a> of the tutorial and all the content of this story is included in this <a href="https://github.com/dmarro89/go-os-tutorial/blob/Part-2-Creating-a-Shell/">branch</a>.</p><p>This is the original <a href="https://github.com/dmarro89/go-dav-os">Github project</a> where the development is going ahead and we’re trying to implement a real OS — maybe putting inside some AI stuff (let’s see).</p><p>The project has a lot of TODO tasks that you can see <a href="https://github.com/dmarro89/go-dav-os/issues">here</a>, in the Github Issues section. If you want to contribute, please start from the ones labelled with “first good issue”: these are the easier tasks that you can implement also without knowing too much about OS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69891e3e0ca1" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/create-an-operating-system-in-go-part-2-the-shell-69891e3e0ca1">Create an Operating System in Go — Part 2: The shell</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create an Operating System in Go — Part 1]]></title>
            <link>https://levelup.gitconnected.com/create-an-operating-system-in-go-part-1-af287bfc402a?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/af287bfc402a</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 09:09:37 GMT</pubDate>
            <atom:updated>2026-02-18T17:08:26.841Z</atom:updated>
            <content:encoded><![CDATA[<p>This is how a fresh operating system <strong>kernel</strong> looks when built from scratch with Go and a bit of assembly (oh, we need it!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-X2me1qV450Js1CQZWJQcQ.png" /></figure><p>Yes, you can’t believe it: I started a hobby project to create an Operating System from scratch!</p><p>The last time I read the world <em>Operating System</em> was at university, something like 15 years ago, and it was one of my favourite exams.</p><p>After all these years of software development, recently a question came back to my mind : <strong>“Why I can’t build my own OS?”</strong></p><p>So I started reading this interesting <a href="https://wiki.osdev.org/Expanded_Main_Page">guide</a> and decided to follow the first tutorial to build a kernel in Go. Well, you <em>can</em> start from <a href="https://wiki.osdev.org/Go_Bare_Bones">there</a> to understand how to create a kernel with a 32 bit setup but — if you try to understand the state of the art — this kind of architecture is not useful anymore, so I tried to implement <strong>my kernel</strong> using a <strong>64-bit architecture</strong>.</p><p>And this is the result : <a href="https://github.com/dmarro89/go-dav-os">https://github.com/dmarro89/go-dav-os</a>. Many people are helping me with this hobby project and we implemented a lot of interesting features — v0.3.0 is the current version. Star the project and take a look at the <a href="https://github.com/dmarro89/go-dav-os/discussions">discussions</a> and the <a href="https://github.com/dmarro89/go-dav-os/issues">issues</a> to know more about it or contribute.</p><p>Bu now let’s back to business. I created a new project <a href="https://github.com/dmarro89/go-os-tutorial">https://github.com/dmarro89/go-os-tutorial</a> to show, step by step, how to create from scratch an OS and this is the reference for this story. In particular, look at the v0.1.0 — all the files explained here are related to that <a href="https://github.com/dmarro89/go-os-tutorial/releases/tag/v0.1.0">tag</a>.</p><p>Wow, <strong><em>kernel</em></strong>, yes! I forgot to say something important: I’m not implementing all the components of an Operating System (let’s use OS from now), we will use <strong>Grub</strong> to boot our <strong><em>kernel</em></strong> — so yes, actually we’re going to implement a <strong><em>kernel</em></strong> not a full OS — a <em>bootloader</em> with a specification, <strong>Multiboot2</strong>, to follow in order to give to it the capacity of understanding how to boot the <strong><em>kernel</em></strong> and run it.</p><p>But how do we launch our <strong>OS</strong>? In this case, I decided to use <strong>QEMU</strong> (if you uses <em>Colima</em>, you already know it) because of its simplicity : pass an ISO file path as parameter and you’re done.</p><p>Ok now, let’s start from scratch : the <strong>Bootloader</strong>.</p><h3>Grub (Multiboot2 header)</h3><p>We’re using <strong>GRUB</strong> with <strong>Multiboot2</strong> to load the kernel so, first of all, we need to understand how to store the data in the final binary so <strong>GRUB</strong> can find it.</p><p><strong>GRUB scans the first 32 KiB of the kernel image</strong> looking for a valid <strong>Multiboot2</strong> header. If the header isn’t there (or it’s invalid), <strong>GRUB</strong> won’t boot it.</p><p>The first thing to do is to create an assembly file (.s) where we control sections, alignment and constants. It’s how we say to Grub : “Here is my Multiboot2 header, go ahead”.</p><p>This is the complete <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/boot/boot.s">file</a> in the <a href="https://github.com/dmarro89/go-os-tutorial">repo</a>.</p><pre>.code32<br><br>.set MULTIBOOT_MAGIC, 0xE85250D6<br>.set MULTIBOOT_ARCH,  0</pre><p>First we need to say that the code should be interpreted in <strong>32-bit</strong> with the directive <em>.code32 </em>— this because <strong>Multiboot2</strong> <strong>hands control to the kernel in 32-bit protected mode</strong> (even if you want a 64-bit kernel). Then, with a <em>“jump”</em>, we switch to 64-bit long mode ourselves.</p><p><em>MULTIBOOT_MAGIC </em>is a standard value that Grub wants to know that you’re talking its same language and then the <em>MULTIBOOT_ARCH</em> set to 0 means the kind of architecture used (i386 protected mode). These two constants will be defined later in the <strong>Multiboot2</strong> header.</p><p><strong>Grub, </strong>when it start, looks for the <strong>Multiboot2</strong> header in the first <em>32KiB</em> of the binary file and it want to read these two constants otherwise it will fail.</p><p>After reading the variables, Grub validates them summing the first 4 words: if the results is zero than it goes ahead. For this reason we’re going to create the first 4 words of the <strong>Multiboot2</strong> header in a way that its sum is zero.</p><pre>.section .multiboot2<br>.align 8<br><br>multiboot_header_start:<br> .long MULTIBOOT_MAGIC<br> .long MULTIBOOT_ARCH<br> .long multiboot_header_end - multiboot_header_start<br> .long -(MULTIBOOT_MAGIC + MULTIBOOT_ARCH + (multiboot_header_end - multiboot_header_start))<br><br>....</pre><p>Good: .align 8 matters because Multiboot2 tags are 8-byte aligned.</p><p>After that we set 2 tags for the moment.</p><pre>...<br># Info request tag: memory map (6), ELF sections (9)<br> .align 8<br> .word 1<br> .word 0<br> .long 16<br> .long 6<br> .long 9<br><br># Entry address tag<br> .align 8<br> .word 3<br> .word 0<br> .long 16<br> .long _start<br> .long 0<br><br>/* End tag */<br> .align 8<br> .word 0<br> .word 0<br> .long 8<br><br>multiboot_header_end:</pre><p>The first tag is an <strong>Information Request</strong> one (type 1) and says “Grub, give me the <strong>memory map (6)</strong><em> </em>and the <strong><em>ELF</em> sections (9)</strong> in the <em>Multiboot information structure</em> you’re returning to me” (its address is stored in the <strong>EBX</strong> registry).</p><p>The second tag is <strong>Entry Address</strong> tag (type 3) and it’s saying where the entry point of the kernel is located — and it’s at <em>_start</em> (we’ll look at it later).</p><p>The last tag (type 0) indicates that the list of tags is completed.</p><p>So we can take a look to <em>_start</em> now.</p><pre> .section .text<br> .global  _start<br> .type    _start, @function<br><br>_start:<br> cli # disable interrupts (no IDT/PIC set yet)<br><br># initialize ESP to the top of our 16 KB stack in .bss<br> mov  $stack_top, %esp<br><br># Multiboot2: EBX contains the address of the multiboot info structure.<br> movl %ebx, multiboot_info_ptr<br><br> call setup_long_mode<br><br> cli<br><br>.Lhang:<br> jmp .Lhang<br>.size _start, . - _start</pre><p>First, we disable the interrupts —we don’t have any <em>IDT</em> set up.</p><p>Then we move the <em>$stack_top</em> to the <strong>ESP</strong> register (that’s our temporary 32-bit stack) and we save the Multiboot info pointer passed by <strong>Grub</strong> (now in the <strong>EBX</strong> register) in a section of the memory called <strong>multiboot_info_ptr</strong>.</p><p>After that, we call <strong>setup_long_mode</strong> which performs the “<em>jump</em>” from 32-bit protected mode to 64-bit long mode.</p><p>Here’s the <strong>bootstrap_stack</strong> and the <strong>multiboot_info_ptr</strong> definition.</p><p>This section contains:</p><ul><li>a simple <strong>bootstrap stack</strong></li><li>paging structures (PML4 → PDPT → PD0–PD3)</li><li>multiboot_info_ptr, where we stored <strong>EBX</strong></li></ul><pre>// --- Stack ---<br>.section .bootstrap_stack, &quot;aw&quot;, @nobits<br><br>.align 16<br>stack_bottom:<br> .skip 16384              # 16 KB di stack<br>stack_top:<br><br># Long mode paging structures (4 KiB aligned, zero-initialized).<br>.align 4096<br>pml4:<br> .skip 4096<br>.align 4096<br>pdpt:<br> .skip 4096<br>.align 4096<br>pd0:<br> .skip 4096<br>.align 4096<br>pd1:<br> .skip 4096<br>.align 4096<br>pd2:<br> .skip 4096<br>.align 4096<br>pd3:<br> .skip 4096<br>.global __bootstrap_end<br>__bootstrap_end:<br><br>.align 4<br>multiboot_info_ptr:<br> .long 0</pre><p>The setup_long_mode function handles the transition from 32 bit to 64 bit architecture. It performs these main steps:</p><ul><li><strong>Building paging structures:</strong> We create a hierarchy of page tables (PML4 → PDPT → PD0/PD1/PD2/PD3) that will translate virtual addresses to physical ones.</li></ul><pre>setup_long_mode:<br># Build minimal identity-mapped paging (4 GiB via 2 MiB pages).<br> lea pml4, %edi<br> movl $pdpt, %eax<br> orl $0x03, %eax<br> movl %eax, (%edi)<br> movl $0, 4(%edi)<br><br> lea pdpt, %edi<br> movl $pd0, %eax<br> orl $0x03, %eax<br> movl %eax, (%edi)<br> movl $0, 4(%edi)<br> movl $pd1, %eax<br> orl $0x03, %eax<br> movl %eax, 8(%edi)<br> movl $0, 12(%edi)<br> movl $pd2, %eax<br> orl $0x03, %eax<br> movl %eax, 16(%edi)<br> movl $0, 20(%edi)<br> movl $pd3, %eax<br> orl $0x03, %eax<br> movl %eax, 24(%edi)<br> movl $0, 28(%edi)</pre><ul><li><strong>Identity mapping:</strong> We map the first 4 GB of memory using 2 MB pages — creating 512 entries in each page directory. This way, virtual address = physical address.</li></ul><pre>movl $0x83, %edx           # present|rw|ps<br> lea pd0, %edi<br> xorl %ecx, %ecx<br> movl $0x00000000, %ebx</pre><p>Then we repeat this for PD0, PD1, PD2, and PD3 to cover the entire 4 GB address space.</p><pre>.Lmap_2m_pd0:<br> movl %ecx, %eax<br> shll $21, %eax             # ecx * 2 MiB<br> addl %ebx, %eax<br> orl %edx, %eax<br> movl %eax, (%edi)<br> movl $0, 4(%edi)<br> addl $8, %edi<br> incl %ecx<br> cmpl $512, %ecx<br> jne .Lmap_2m_pd0<br><br> lea pd1, %edi<br> xorl %ecx, %ecx<br> movl $0x40000000, %ebx<br><br>.Lmap_2m_pd1:<br> movl %ecx, %eax<br> shll $21, %eax<br> addl %ebx, %eax<br> orl %edx, %eax<br> movl %eax, (%edi)<br> movl $0, 4(%edi)<br> addl $8, %edi<br> incl %ecx<br> cmpl $512, %ecx<br> jne .Lmap_2m_pd1<br><br> lea pd2, %edi<br> xorl %ecx, %ecx<br> movl $0x80000000, %ebx<br><br>.Lmap_2m_pd2:<br> movl %ecx, %eax<br> shll $21, %eax<br> addl %ebx, %eax<br> orl %edx, %eax<br> movl %eax, (%edi)<br> movl $0, 4(%edi)<br> addl $8, %edi<br> incl %ecx<br> cmpl $512, %ecx<br> jne .Lmap_2m_pd2<br><br> lea pd3, %edi<br> xorl %ecx, %ecx<br> movl $0xC0000000, %ebx<br><br>.Lmap_2m_pd3:<br> movl %ecx, %eax<br> shll $21, %eax<br> addl %ebx, %eax<br> orl %edx, %eax<br> movl %eax, (%edi)<br> movl $0, 4(%edi)<br> addl $8, %edi<br> incl %ecx<br> cmpl $512, %ecx<br> jne .Lmap_2m_pd3</pre><ul><li><strong>Enabling paging infrastructure:</strong> We load the PML4 address into <strong>CR3</strong>, enable <strong>PAE</strong> (Physical Address Extension) in <strong>CR4</strong>, and enable long mode in the <strong>EFER</strong> <strong>MSR</strong> register.</li></ul><pre># Load PML4 and enable PAE.<br> movl $pml4, %eax<br> movl %eax, %cr3<br><br> movl %cr4, %eax<br> orl  $0x20, %eax<br> movl %eax, %cr4<br><br># Enable long mode in EFER.<br> movl $0xC0000080, %ecx<br> rdmsr<br> orl  $0x100, %eax<br> wrmsr</pre><ul><li><strong>Loading GDT and paging:</strong> We load our simple 64 bit GDT and finally enable paging by setting the PG bit in CR0.</li></ul><pre># Load GDT and enable paging.<br> lgdt gdt64_desc<br> movl %cr0, %eax<br> orl  $0x80000000, %eax<br> movl %eax, %cr0<br><br># Far jump to 64-bit code segment.<br> ljmp $0x08, $long_mode_entry</pre><ul><li><strong>Far jump to 64 bit:</strong> We perform a far jump with ljmp $0x08, $long_mode_entry to switch the CPU to 64 bit mode.</li><li><strong>64 bit setup:</strong> We update segment registers, set up the 64 bit stack, clear the BSS section, and finally call go_0kernel.Main with the multiboot info pointer.</li></ul><pre>.code64<br>long_mode_entry:<br> movw $0x10, %ax<br> movw %ax, %ds<br> movw %ax, %es<br> movw %ax, %ss<br> movw %ax, %fs<br> movw %ax, %gs<br><br> movq $stack_top, %rsp<br> andq $-16, %rsp<br> subq $8, %rsp<br><br># Clear BSS.<br> movq $__bss_start, %rdi<br> movq $__bss_end, %rcx<br> subq %rdi, %rcx<br> xor %eax, %eax<br> rep stosb<br><br> movl multiboot_info_ptr(%rip), %edi<br> call go_0kernel.Main<br><br>.Lhang64:<br> hlt<br> jmp .Lhang64</pre><h3>The Kernel</h3><p>Quick recap. We’ve set all the registers and addresses so Grub what to load, we’ve jumped to move from a 32-bit to a 64-bit and now we can implement the main code in <strong>Go</strong>.</p><p>We’re actually going to do something very simple, just an <em>hello world,</em> but it will be very interesting because we need to print the text on the screen and it’s not an easy job when you have literally nothing.</p><p>Let’s create first our <strong>shell.go</strong> file, a simple file rendering the text on the screen using the <strong>VGA text mode</strong>.</p><p><strong>VGA text mode</strong> is basically a memory region at <strong>0xB8000</strong> where each cell is 2 bytes:</p><ul><li>byte 0: ASCII character</li><li>byte 1: color attribute (foreground/background)</li></ul><p>So first of all we need the VGA height and width desired, as the pointer to the memory address where the text will be set.</p><p>This is the <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/shell/shell.go">file</a> in the repo.</p><pre>const (<br> VGAWidth  = 80<br> VGAHeight = 25<br><br> vgaCursorIndexPort uint16 = 0x3D4<br> vgaCursorDataPort  uint16 = 0x3D5<br>)<br><br>const videoMemoryAddr uintptr = 0xB8000<br><br>func getVidMem() *[VGAHeight][VGAWidth][2]byte {<br> return (*[VGAHeight][VGAWidth][2]byte)(unsafe.Pointer(videoMemoryAddr))<br>}</pre><p>Now we create in Go the definition of a function that we will link with an assembly implementation in order to put the character we’re typing inside the VGA view.</p><pre>func outb(port uint16, value byte)</pre><p>Let’s create now a new assembly file (<a href="https://github.com/dmarro89/go-os-tutorial/blob/main/boot/stubs_amd64.s">stubs_amd64.s</a>) where we put the implementation of the <em>outb</em> function.</p><pre># github.com/dmarro89/go-os-tutorial/shell.outb(port uint16, value byte)<br>.global github_0com_1dmarro89_1go_x2dos_x2dtutorial_1shell.outb<br>.type   github_0com_1dmarro89_1go_x2dos_x2dtutorial_1shell.outb, @function<br>github_0com_1dmarro89_1go_x2dos_x2dtutorial_1shell.outb:<br> movw %di, %dx<br> movb %sil, %al<br> outb %al, %dx<br> ret<br>.size github_0com_1dmarro89_1go_x2dos_x2dtutorial_1shell.outb, . - github_0com_1dmarro89_1go_x2dos_x2dtutorial_1shell.outb</pre><p>The outb function takes two parameters, moves them into <strong>DX</strong> (the I/O port) and <strong>AL</strong> (the byte value), and then writes them with the outb instruction.</p><p>In this file we add also other stubs to implement some functions needed by the Go runtime — you can take a look <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/boot/stubs_amd64.s">here</a> to see its implementation.</p><p>Now we can complete our shell with a Print() method able to print the rune (one each) on the terminal.</p><pre>package kernel<br><br>func Print(s string) {<br> for i := 0; i &lt; len(s); i++ {<br>  putRune(rune(s[i]))<br> }<br>}<br><br><br>func putRuneAt(col, currRow int, ch rune) {<br> vidMem[currRow][col][0] = byte(ch)<br> vidMem[currRow][col][1] = color<br>}</pre><p>Now, the last thing to do in Go is the creation of the <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/kernel/kernel.go">main</a> — we can call the file kernel.go that starts and prints a simple hello world on the screen.</p><pre>package kernel<br><br>func Main(multibootInfoAddr uint64) {<br>  Print(&quot;Hello world&quot;)<br>}</pre><p>I believe you’re disappointed because we’ve written so few lines in Go but this is just an <em>hello world</em> and creating more features will take you into more go code, so don’t worry.</p><h3>Linking</h3><p>But how all the magic works?</p><p>We missed an important connection point between the assembler, the go files and the memory layout at runtime: the linker.</p><p>The <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/boot/linker.ld"><strong>linker.ld</strong></a> file decides where the sections are loaded in memory.</p><pre>OUTPUT_FORMAT(&quot;elf64-x86-64&quot;)<br>OUTPUT_ARCH(i386:x86-64)<br><br>ENTRY(_start)<br><br>SECTIONS<br>{<br>  /* Physical load address for the kernel image (common for Multiboot kernels) */<br>  . = 1M;<br><br>  /* Multiboot header must be near the start so GRUB can locate it */<br>  /* Executable code */<br><br>  .text BLOCK(4K) : ALIGN(4K)<br>    {<br>        *(.multiboot2)<br>        *(.text)<br>    }<br><br>  /* Read-only data (const tables, strings, etc.) */<br>  .rodata BLOCK(4K) : ALIGN(4K)<br>  {<br>    *(.rodata*)<br>  }<br><br>  /* Writable data */<br>  .data BLOCK(4K) : ALIGN(4K)<br>    {<br>        *(.data)<br>    }<br><br>  /* Zero-initialized data and common symbols; stays zeroed at load time */<br>    .bss BLOCK(4K) : ALIGN(4K)<br>    {<br>        __bss_start = .;<br>        *(COMMON)<br>        *(.bss)<br>        __bss_end = .;<br>    }<br><br>  /* Keep bootstrap stack/tables out of BSS clear */<br>  .bootstrap_stack BLOCK(4K) : ALIGN(4K)<br>    {<br>        *(.bootstrap_stack)<br>    }<br><br>  __kernel_end = .;<br>}</pre><p>Outside of the <strong>format</strong> and <strong>arch</strong> declared, the first thing to see is that the entry is the <em>_start</em> instruction.</p><p>Then it defines a new section <em>text</em> where it puts all the stuff needed for <strong>Grub</strong>: if you put it at the start of the memory, <strong>Grub</strong> will find it easily.</p><p>Then we have in order a space for the <em>readonly data</em>, one for the <em>not initialised data</em> and one for the <em>stack</em>. <strong>kernel.end</strong> marks the end of the kernel.</p><h3>Oh no, the compilation!</h3><p>And now that we have everything in place we should compile all the files, generates the binaries and put them together in a single .elf file.</p><p>So we need a <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/Makefile"><strong>Makefile</strong></a>.</p><p>Your Makefile is doing a very specific thing: building <strong>freestanding Go</strong> code using <strong>gccgo</strong>, without libc and without the normal runtime startup.</p><pre>CROSS    ?= x86_64-elf<br><br>AS       := $(CROSS)-as<br>GCC      := $(CROSS)-gcc<br>GCCGO    := $(CROSS)-gccgo<br>OBJCOPY  := $(CROSS)-objcopy<br>GCCGOFLAGS := -m64<br>GRUB_CFG      := iso/grub/grub.cfg<br><br>QEMU          := qemu-system-x86_64<br><br>DOCKER_PLATFORM  := linux/amd64<br>DOCKER_IMAGE     := go-os-tutorial-toolchain<br>DOCKER_RUN_FLAGS := -it<br><br>BUILD_DIR := build<br>ISO_DIR   := $(BUILD_DIR)/isodir<br><br>KERNEL_ELF := $(BUILD_DIR)/kernel.elf<br>ISO_IMAGE  := $(BUILD_DIR)/go-os-tutorial.iso<br><br>BOOT_SRCS := $(wildcard boot/*.s)<br>LINKER_SCRIPT := boot/linker.ld<br><br>MODPATH      := github.com/dmarro89/go-os-tutorial<br><br>SHELL_IMPORT     := $(MODPATH)/shell<br>SHELL_SRCS := $(filter-out %_test.go, $(wildcard shell/*.go))<br>KERNEL_SRCS := $(filter-out %_test.go, $(wildcard kernel/*.go))<br><br>BOOT_OBJ   := $(BUILD_DIR)/boot.o<br>KERNEL_OBJ := $(BUILD_DIR)/kernel.o<br><br>SHELL_OBJ   := $(BUILD_DIR)/shell.o<br>SHELL_GOX   := $(BUILD_DIR)/github.com/dmarro89/go-dav-os/shell.gox</pre><p>In our <strong>Makefile</strong>, we first define some constants and then the real commands.</p><pre># -----------------------<br># Build directory<br># -----------------------<br>$(BUILD_DIR):<br> mkdir -p $(BUILD_DIR)<br><br><br># -----------------------<br># Assembly: boot.s -&gt; boot.o<br># -----------------------<br>$(BOOT_OBJ): $(BOOT_SRCS) | $(BUILD_DIR)<br> $(AS) $(BOOT_SRCS) -o $(BOOT_OBJ)<br><br># --- 1. Compile shell.go (package shell) with gccgo ---<br>$(SHELL_OBJ): $(SHELL_SRCS) | $(BUILD_DIR)<br> $(GCCGO) $(GCCGOFLAGS) -static -Werror -nostdlib -nostartfiles -nodefaultlibs \<br>  -I $(BUILD_DIR) \<br>  -fgo-pkgpath=$(SHELL_IMPORT) \<br>  -c $(SHELL_SRCS) -o $(SHELL_OBJ)<br><br># --- 2. Extract .go_export into shell.gox ---<br>$(SHELL_GOX): $(SHELL_OBJ) | $(BUILD_DIR)<br> mkdir -p $(dir $(SHELL_GOX))<br> $(OBJCOPY) -j .go_export $(SHELL_OBJ) $(SHELL_GOX)<br><br># --- 3. Compile kernel.go (package kernel) ---<br>$(KERNEL_OBJ): $(KERNEL_SRCS) $(SHELL_GOX) | $(BUILD_DIR)<br> $(GCCGO) $(GCCGOFLAGS) -static -Werror -nostdlib -nostartfiles -nodefaultlibs \<br>  -I $(BUILD_DIR) \<br>  -c $(KERNEL_SRCS) -o $(KERNEL_OBJ)<br><br># -----------------------<br># Link: boot.o + kernel.o -&gt; kernel.elf<br># -----------------------<br>$(KERNEL_ELF): $(BOOT_OBJ) $(SHELL_OBJ) $(KERNEL_OBJ) $(LINKER_SCRIPT)<br> $(GCC) -T $(LINKER_SCRIPT) -o $(KERNEL_ELF) \<br>  -ffreestanding -O2 -nostdlib \<br>  $(BOOT_OBJ) $(SHELL_OBJ) $(KERNEL_OBJ) -lgcc</pre><p>So what we’ve done here ?</p><p>We have defined some commands to compile our boot.s, kernel.go and shell.go file and get the .o corresponding files.</p><p>Then for the go files we export them into a gox files, the last step to link them into a single .elf.</p><pre># -----------------------<br># Docker helpers<br># -----------------------<br>docker-image:<br> docker build --platform=$(DOCKER_PLATFORM) -t $(DOCKER_IMAGE) .<br><br>docker-run: docker-image<br> docker run ${DOCKER_RUN_FLAGS} --rm --platform=$(DOCKER_PLATFORM) \<br>   -v &quot;$(CURDIR)&quot;:/work -w /work $(DOCKER_IMAGE) \<br>   make run<br><br>docker-build-only: docker-image<br> docker run --rm --platform=$(DOCKER_PLATFORM) \<br>   -v &quot;$(CURDIR)&quot;:/work -w /work $(DOCKER_IMAGE) \<br>   make<br><br>docker-shell: docker-image<br> docker run -it --rm --platform=$(DOCKER_PLATFORM) \<br>   -v &quot;$(CURDIR)&quot;:/work -w /work $(DOCKER_IMAGE) bash</pre><p>After that, we have created some helpers for docker — because we’re going to test them on docker. They calls our directive reported below.</p><pre>.PHONY: all kernel iso run clean docker-build docker-shell docker-run<br><br>all: $(ISO_IMAGE)<br><br>kernel: $(KERNEL_ELF)<br><br>iso: $(ISO_IMAGE)<br><br>run: $(ISO_IMAGE) disk.img<br> $(QEMU) -cdrom $(ISO_IMAGE) -drive file=disk.img,format=raw<br><br>disk.img:<br> dd if=/dev/zero of=disk.img bs=1M count=20<br><br>clean:<br> rm -rf $(BUILD_DIR) disk.img</pre><h3>Let’s Test it!</h3><p>To avoid installing dependencies on our machine, we have prepared a <a href="https://github.com/dmarro89/go-os-tutorial/blob/main/Dockerfile">Dockerfile</a> for the build phase (it takes long time for the first build).</p><p>So we can run a simple</p><pre>make docker-build-only</pre><p>This generates into the build folder of the project the .iso file.</p><p>Then you can install qemu on your local machine and run it with a simple command</p><pre>qemu-system-x86_64 -cdrom build/go-os-tutorial.iso</pre><p>Otherwise you can run it from docker in interactive mode.</p><p>Et voilà!</p><p>You have your personal OS running!</p><h3>Conclusion</h3><p>So a quick recap of the resources needed to create your personal OS from scratch</p><p>This is the <a href="https://github.com/dmarro89/go-os-tutorial">Github Project</a> of the tutorial where you can find all the resources.</p><p>Take a look at the <a href="https://github.com/dmarro89/go-os-tutorial/releases/tag/v0.1.0">v0.1.0</a> tag, it contains all the code created in this article.</p><p>Then you can take a look at the original project where a real OS is in phase of creation -&gt; <a href="https://github.com/dmarro89/go-dav-os">go-dav-os</a>.<br>Please star the project and join the discussions and issues sections if you want to contribute or learn more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af287bfc402a" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/create-an-operating-system-in-go-part-1-af287bfc402a">Create an Operating System in Go — Part 1</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Redis HNSW — Part 1]]></title>
            <link>https://levelup.gitconnected.com/understanding-redis-hnsw-part-1-42cb8fd0d354?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/42cb8fd0d354</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 16:31:07 GMT</pubDate>
            <atom:updated>2025-04-23T16:31:07.390Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Redis HNSW — Part 1</h3><p>As you know, Redis has <a href="https://redis.io/blog/announcing-vector-sets-a-new-redis-data-type-for-vector-similarity/">recently</a> released a new data type known as <strong>Vector Set</strong>, allowing vector similarity search. The vector sets are based on a vector index known as HNSW (Hierarchical Navigable Small World).</p><p>There are many papers on HNSW, so I’m not going into the theoretical aspects but, in this story, I will focus on how <a href="https://medium.com/u/e6bea48be688">Salvatore Sanfilippo</a> has implemented it in Redis. This will be just the first of several articles since the implementation iscomplex and requires substantial explanation.</p><h3>HNSWNode</h3><p>First thing first, let’s examine the Node struct.</p><p>In the <a href="https://github.com/redis/redis/blob/unstable/modules/vector-sets/hnsw.h">hnsw.h</a> the first struct to be defined is the <strong>Node</strong>. Here’s how it is composed.</p><pre> /* Node structure for HNSW graph */<br> typedef struct hnswNode {<br>     uint32_t level;         /* Node&#39;s maximum level */<br>     uint64_t id;            /* Unique identifier, may be useful in order to<br>                              * have a bitmap of visited notes to use as<br>                              * alternative to epoch / visited_epoch.<br>                              * Also used in serialization in order to retain<br>                              * links specifying IDs. */<br>     void *vector;           /* The vector, quantized or not. */<br>     float quants_range;     /* Quantization range for this vector:<br>                              * min/max values will be in the range<br>                              * -quants_range, +quants_range */<br>     float l2;               /* L2 before normalization. */<br> <br>     /* Last time (epoch) this node was visited. We need one per thread.<br>      * This avoids having a different data structure where we track<br>      * visited nodes, but costs memory per node. */<br>     uint64_t visited_epoch[HNSW_MAX_THREADS];<br> <br>     void *value;                    /* Associated value */<br>     struct hnswNode *prev, *next;   /* Prev/Next node in the list starting at<br>                                      * HNSW-&gt;head. */<br> <br>     /* Links (and links info) per each layer. Note that this is part<br>      * of the node allocation to be more cache friendly: reliable 3% speedup<br>      * on Apple silicon, and does not make anything more complex. */<br>     hnswNodeLayer layers[];<br> } hnswNode;</pre><p>Each Node contains some simple information like the <strong>level</strong>, the <strong>id</strong> and the generics pointer to the <strong>vector</strong> and to the vector <strong>value.</strong></p><p>It has also other more complex information like:</p><ul><li><strong>quants_range</strong>: a quantization range, used because vectors can be quantized to speed up the operations. This defines the range of values after quantization.</li><li><strong>l2: </strong>the L2 norm (Euclidean length) of the vector before normalization. This helps with vector normalization for distance calculations.</li><li><strong>visited_epoch</strong>: an array tracking the last time the node was visited by each thread, which avoids maintaining a separate visited nodes structure.</li><li><strong>*prev, *next</strong> : pointers to the previous and next nodes in the list. All the nodes are connected, by level, in a linked list for easier traversal.</li><li><strong>layers</strong>[]: each node stores also a collection of information for each layer. This is where all the graph structure is maintained.</li></ul><p>The structure is pretty easy, now let’s see how it’s initialized inside the method <a href="https://github.com/redis/redis/blob/41ecf7323ee96b50effcc8d0ae7d6f8a631a5f25/modules/vector-sets/hnsw.c#L494"><strong>hnsw_node_new</strong></a><strong> </strong>of the <a href="https://github.com/redis/redis/blob/unstable/modules/vector-sets/hnsw.c">hnsw.c</a> file.</p><pre>/* Create new node. Returns NULL on out of memory.<br> * It is possible to pass the vector as floats or, in case this index<br> * was already stored on disk and is being loaded, or serialized and<br> * transmitted in any form, the already quantized version in<br> * &#39;qvector&#39;.<br> *<br> * Only vector or qvector should be non-NULL. The reason why passing<br> * a quantized vector is useful, is that because re-normalizing and<br> * re-quantizing several times the same vector may accumulate rounding<br> * errors. So if you work with quantized indexes, you should save<br> * the quantized indexes.<br> *<br> * Note that, together with qvector, the quantization range is needed,<br> * since this library uses per-vector quantization. In case of quantized<br> * vectors the l2 is considered to be &#39;1&#39;, so if you want to restore<br> * the right l2 (to use the API that returns an approximation of the<br> * original vector) make sure to save the l2 on disk and set it back<br> * after the node creation (see later for the serialization API that<br> * handles this and more). */<br>hnswNode *hnsw_node_new(HNSW *index, uint64_t id, const float *vector, const int8_t *qvector, float qrange, uint32_t level, int normalize) {<br>...<br>}</pre><p>The function first allocates the memory needed to store the node structure itself (size of hnswNode) plus the size of all the layers (sizeof(hnswNodeLayer)*(level+1)). If memory allocation fails (out of memory), the function returns NULL as noted in the comments.</p><pre>hnswNode *node = hmalloc(sizeof(hnswNode)+(sizeof(hnswNodeLayer)*(level+1)));<br>    if (!node) return NULL;</pre><p>Next, it initializes all the basic properties of the struct setting them to default values.</p><pre>if (id == 0) id = ++index-&gt;last_id;<br>    node-&gt;level = level;<br>    node-&gt;id = id;<br>    node-&gt;next = NULL;<br>    node-&gt;vector = NULL;<br>    node-&gt;l2 = 1;   // Default in case of already quantized vectors. It is<br>                    // up to the caller to fill this later, if needed.<br><br>    /* Initialize visited epoch array. */<br>    for (int i = 0; i &lt; HNSW_MAX_THREADS; i++)<br>        node-&gt;visited_epoch[i] = 0;</pre><p>Then, the function handles to separate two scenarios.</p><p>First, non-quantized vectors (<strong>qvector</strong> == NULL).</p><p>In this case, the vector passed as parameter (<strong>*vector</strong>) is stored in the node after allocating space for it, normalized if the <strong>normalize</strong> parameter is true and then quantized if configured.</p><pre>if (qvector == NULL) {<br>        /* Copy input vector. */<br>        node-&gt;vector = hmalloc(sizeof(float) * index-&gt;vector_dim);<br>        if (!node-&gt;vector) {<br>            hfree(node);<br>            return NULL;<br>        }<br>        memcpy(node-&gt;vector, vector, sizeof(float) * index-&gt;vector_dim);<br>        if (normalize)<br>            hnsw_normalize_vector(node-&gt;vector,&amp;node-&gt;l2,index-&gt;vector_dim);<br><br>        /* Handle quantization. */<br>        if (index-&gt;quant_type != HNSW_QUANT_NONE) {<br>            void *quants = hmalloc(hnsw_quants_bytes(index));<br>            if (quants == NULL) {<br>                hfree(node-&gt;vector);<br>                hfree(node);<br>                return NULL;<br>            }<br><br>            // Quantize.<br>            switch(index-&gt;quant_type) {<br>            case HNSW_QUANT_Q8:<br>                quantize_to_q8(node-&gt;vector,quants,index-&gt;vector_dim,&amp;node-&gt;quants_range);<br>                break;<br>            case HNSW_QUANT_BIN:<br>                quantize_to_bin(node-&gt;vector,quants,index-&gt;vector_dim);<br>                break;<br>            default:<br>                assert(0 &amp;&amp; &quot;Quantization type not handled.&quot;);<br>                break;<br>            }<br><br>            // Discard the full precision vector.<br>            hfree(node-&gt;vector);<br>            node-&gt;vector = quants;<br>        }<br>    }</pre><p>In the second scenario, pre-quantized vector (common when loading from the disk) the flow is much easier: the vector is directly store inside the node.</p><pre>else {<br>        // We got the already quantized vector. Just copy it.<br>        assert(index-&gt;quant_type != HNSW_QUANT_NONE);<br>        uint32_t vector_bytes = hnsw_quants_bytes(index);<br>        node-&gt;vector = hmalloc(vector_bytes);<br>        node-&gt;quants_range = qrange;<br>        if (node-&gt;vector == NULL) {<br>            hfree(node);<br>            return NULL;<br>        }<br>        memcpy(node-&gt;vector,qvector,vector_bytes);<br>    }</pre><p>After that, the last step consists of initializing each node of the layers.</p><pre>/* Initialize each layer. */<br>    for (uint32_t i = 0; i &lt;= level; i++) {<br>        uint32_t max_links = (i == 0) ? index-&gt;M*2 : index-&gt;M;<br>        node-&gt;layers[i].max_links = max_links;<br>        node-&gt;layers[i].num_links = 0;<br>        node-&gt;layers[i].worst_distance = 0;<br>        node-&gt;layers[i].worst_idx = 0;<br>        node-&gt;layers[i].links = hmalloc(sizeof(hnswNode*) * max_links);<br>        if (!node-&gt;layers[i].links) {<br>            for (uint32_t j = 0; j &lt; i; j++) hfree(node-&gt;layers[j].links);<br>            hfree(node-&gt;vector);<br>            hfree(node);<br>            return NULL;<br>        }<br>    }<br><br>    return node;</pre><h3>HNSWNodeLayer</h3><p>Now let’s see how is structured an <strong>hnswNodeLayer</strong>, that is a struct contained in each node. It should keep track of every other node of the HNSW graph for each level.</p><pre><br>/* Layer structure for HNSW nodes. Each node will have from one to a few<br> * of this depending on its level. */<br>typedef struct {<br>    struct hnswNode **links;  /* Array of neighbors for this layer */<br>    uint32_t num_links;       /* Number of used links */<br>    uint32_t max_links;       /* Maximum links for this layer. We may<br>                               * reallocate the node in very particular<br>                               * conditions in order to allow linking of<br>                               * new inserted nodes, so this may change<br>                               * dynamically and be &gt; M*2 for a small set of<br>                               * nodes. */<br>    float worst_distance;     /* Distance to the worst neighbor */<br>    uint32_t worst_idx;       /* Index of the worst neighbor */<br>} hnswNodeLayer;</pre><p>So, the <strong>hnswNodeLayer</strong> represents the connections at a specific level in the HNSW.</p><p>For each layer, a node can have N neighbors and this is memorized as a pointer to an arrya of pointers in the <strong>**links</strong> variable.</p><p>After that, there are some info about the layer like the num of the nodes existing in the layer (<strong>num_links</strong>), the maximum allowed links for the layer (<strong>max_links</strong>) but also the distance (<strong>worst_distance</strong>) and the index (<strong>worst_idx</strong>) of the farthest neighbor.</p><h3>HNSW</h3><p>Now, it’s the time for the big HNSW structure containing all the nodes.</p><p>There’s how it’s structured.</p><pre>/* Main HNSW index structure */<br>typedef struct HNSW {<br>    hnswNode *enter_point;   /* Entry point for the graph */<br>    uint32_t M;               /* M as in the paper: layer 0 has M*2 max<br>                                 neighbors (M populated at insertion time)<br>                                 while all the other layers have M neighbors. */<br>    uint32_t max_level;      /* Current maximum level in the graph */<br>    uint32_t vector_dim;     /* Dimensionality of stored vectors */<br>    uint64_t node_count;     /* Total number of nodes */<br>    _Atomic uint64_t last_id; /* Last node ID used */<br>    uint64_t current_epoch[HNSW_MAX_THREADS];  /* Current epoch for visit tracking */<br>    hnswNode *head;             /* Linked list of nodes. Last first */<br><br>    /* We have two locks here:<br>     * 1. A global_lock that is used to perform write operations blocking all<br>     * the readers.<br>     * 2. One mutex per epoch slot, in order for read operations to acquire<br>     * a lock on a specific slot to use epochs tracking of visited nodes. */<br>    pthread_rwlock_t global_lock;  /* Global read-write lock */<br>    pthread_mutex_t slot_locks[HNSW_MAX_THREADS];  /* Per-slot locks */<br><br>    _Atomic uint32_t next_slot; /* Next thread slot to try */<br>    _Atomic uint64_t version;   /* Version for optimistic concurrency, this is<br>                                 * incremented on deletions and entry point<br>                                 * updates. */<br>    uint32_t quant_type;        /* Quantization used. HNSW_QUANT_... */<br>    hnswCursor *cursors;<br>} HNSW;</pre><p>The structure doesn’t directly store all the nodes, because each Node has already all the information needed, but has just the entry point for the graph (<strong>*enter_point</strong>) for graph traversal and the head (<strong>*head</strong>) for the linked list iteration.</p><p>It has also other configuration parameters like the number of the maximum level allowed (<strong>max_level</strong>), the dimension of the vectors stored (<strong>vector_dim</strong>), the number of nodes (<strong>node_count</strong>), the current epoch for tracking the visit of the HNSW for each thread (<strong>current_epoch</strong>), the locks for the thread safety (<strong>global_lock</strong> and <strong>slot_locks</strong>), information about the type of quantization used (<strong>quant_type</strong>) and an array of cursors (<strong>*cursors</strong>) to easily iterate through the HNSW nodes.</p><p>That’s how it’s initialized.</p><pre>/* Create new HNSW index, quantized or not. */<br>HNSW *hnsw_new(uint32_t vector_dim, uint32_t quant_type, uint32_t m) {<br>...<br>}</pre><p>Like for the nodes structures, if there’s no memory to allocate the HNSW the method returns null.</p><pre>HNSW *index = hmalloc(sizeof(HNSW));<br>    if (!index) return NULL;</pre><p>After that, the simple values are allocated with the parameters passed in the method or with default values if not.</p><pre>/* M parameter sanity check. */<br>    if (m == 0) m = HNSW_DEFAULT_M;<br>    else if (m &gt; HNSW_MAX_M) m = HNSW_MAX_M;<br><br>    index-&gt;M = m;<br>    index-&gt;quant_type = quant_type;<br>    index-&gt;enter_point = NULL;<br>    index-&gt;max_level = 0;<br>    index-&gt;vector_dim = vector_dim;<br>    index-&gt;node_count = 0;<br>    index-&gt;last_id = 0;<br>    index-&gt;head = NULL;<br>    index-&gt;cursors = NULL;</pre><p>At the end, each epochs array and lock is initialized (or cleaned if yet initialized).</p><pre>/* Initialize epochs array. */<br>    for (int i = 0; i &lt; HNSW_MAX_THREADS; i++)<br>        index-&gt;current_epoch[i] = 0;<br><br>    /* Initialize locks. */<br>    if (pthread_rwlock_init(&amp;index-&gt;global_lock, NULL) != 0) {<br>        hfree(index);<br>        return NULL;<br>    }<br><br>    for (int i = 0; i &lt; HNSW_MAX_THREADS; i++) {<br>        if (pthread_mutex_init(&amp;index-&gt;slot_locks[i], NULL) != 0) {<br>            /* Clean up previously initialized mutexes. */<br>            for (int j = 0; j &lt; i; j++)<br>                pthread_mutex_destroy(&amp;index-&gt;slot_locks[j]);<br>            pthread_rwlock_destroy(&amp;index-&gt;global_lock);<br>            hfree(index);<br>            return NULL;<br>        }<br>    }<br><br>    /* Initialize atomic variables. */<br>    index-&gt;next_slot = 0;<br>    index-&gt;version = 0;<br>    return index;</pre><h3>Priority Queues</h3><p>Maybe it seems that this structure is totally unlinked from the previous ones but the <strong>priority queues</strong> performances have a crucial role for the HNSW because are used a lot during the insertion and search processes.</p><p>The <strong>pqueue</strong> is structured in a very simple way.</p><pre>typedef struct {<br>    hnswNode *node;<br>    float distance;<br>} pqitem;<br><br>typedef struct {<br>    pqitem *items;         /* Array of items. */<br>    uint32_t count;        /* Current number of items. */<br>    uint32_t cap;          /* Maximum capacity. */<br>} pqueue;</pre><p>Each item of the queue (<strong>pqitem</strong>) is composed by a node (<strong>*node</strong>) and a float representing the <strong>distance</strong> from a specific node (usually the node searched or inserted). The <strong>pqueue</strong> has an array of items (<strong>*items</strong>), the count of the numbers (<strong>count</strong>) and the maximum capacity allowed (<strong>cap</strong>).</p><pre>/* The HNSW algorithms access the pqueue conceptually from nearest (index 0)<br> * to farthest (larger indexes) node, so the following macros are used to<br> * access the pqueue in this fashion, even if the internal order is<br> * actually reversed. */<br>#define pq_get_node(q,i) ((q)-&gt;items[(q)-&gt;count-(i+1)].node)<br>#define pq_get_distance(q,i) ((q)-&gt;items[(q)-&gt;count-(i+1)].distance)<br><br>/* Create a new priority queue with given capacity. Adding to the<br> * pqueue only retains &#39;capacity&#39; elements with the shortest distance. */<br>pqueue *pq_new(uint32_t capacity) {<br>    pqueue *pq = hmalloc(sizeof(*pq));<br>    if (!pq) return NULL;<br><br>    pq-&gt;items = hmalloc(sizeof(pqitem) * capacity);<br>    if (!pq-&gt;items) {<br>        hfree(pq);<br>        return NULL;<br>    }<br><br>    pq-&gt;count = 0;<br>    pq-&gt;cap = capacity;<br>    return pq;<br>}<br><br>/* Free a priority queue. */<br>void pq_free(pqueue *pq) {<br>    if (!pq) return;<br>    hfree(pq-&gt;items);<br>    hfree(pq);<br>}</pre><p>The creation and destruction of a priority queue are pretty simple operations: in these two methods are just allocated or de-allocated slots of memory for the items needed.</p><pre>/* Insert maintaining distance order (higher distances first). */<br>void pq_push(pqueue *pq, hnswNode *node, float distance) {<br>    if (pq-&gt;count &lt; pq-&gt;cap) {<br>        /* Queue not full: shift right from high distances to make room. */<br>        uint32_t i = pq-&gt;count;<br>        while (i &gt; 0 &amp;&amp; pq-&gt;items[i-1].distance &lt; distance) {<br>            pq-&gt;items[i] = pq-&gt;items[i-1];<br>            i--;<br>        }<br>        pq-&gt;items[i].node = node;<br>        pq-&gt;items[i].distance = distance;<br>        pq-&gt;count++;<br>    } else {<br>        /* Queue full: if new item is worse than worst, ignore it. */<br>        if (distance &gt;= pq-&gt;items[0].distance) return;<br><br>        /* Otherwise shift left from low distances to drop worst. */<br>        uint32_t i = 0;<br>        while (i &lt; pq-&gt;cap-1 &amp;&amp; pq-&gt;items[i+1].distance &gt; distance) {<br>            pq-&gt;items[i] = pq-&gt;items[i+1];<br>            i++;<br>        }<br>        pq-&gt;items[i].node = node;<br>        pq-&gt;items[i].distance = distance;<br>    }<br>}<br><br>/* Remove and return the top (closest) element, which is at count-1<br> * since we store elements with higher distances first.<br> * Runs in constant time. */<br>hnswNode *pq_pop(pqueue *pq, float *distance) {<br>    if (pq-&gt;count == 0) return NULL;<br>    pq-&gt;count--;<br>    *distance = pq-&gt;items[pq-&gt;count].distance;<br>    return pq-&gt;items[pq-&gt;count].node;<br>}<br><br>/* Get distance of the furthest element.<br> * An empty priority queue has infinite distance as its furthest element,<br> * note that this behavior is needed by the algorithms below. */<br>float pq_max_distance(pqueue *pq) {<br>    if (pq-&gt;count == 0) return INFINITY;<br>    return pq-&gt;items[0].distance;<br>}</pre><p>The push operation, instead, act as follow:</p><ul><li>If the queue isn’t full, it inserts the new item in the proper position to maintain order (higher distances first).</li><li>If the queue is full, it only adds the new item if it’s better (closer) than the worst item, then shifts elements to maintain order.</li></ul><p>The pop operation efficiently returns the closest element (the one with the smallest distance) in constant time.</p><h3><strong>Conclusion</strong></h3><p>Now that we’ve examined the basic structures of Redis’ HNSW implementation, we can move on to the more complex operations of insert and search in subsequent articles.</p><p>For each of tthem, there will be two different chapters — given the complexity of the operations.</p><p>If you like this kind of contents and you want to follow the next chapters of this series, please follow me on <a href="https://medium.com/@davideraffaele.marro">Medium</a>.</p><p>If you want, you can also follow me on <a href="https://github.com/dmarro89">Github</a> where I’m trying to recreate an HNSW in Golang within this <a href="https://github.com/dmarro89/hnsw-go">project</a> — but this is another story.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42cb8fd0d354" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/understanding-redis-hnsw-part-1-42cb8fd0d354">Understanding Redis HNSW — Part 1</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a Typescript Compiler in Go]]></title>
            <link>https://levelup.gitconnected.com/create-a-typescript-compiler-in-go-f3503abdca44?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3503abdca44</guid>
            <category><![CDATA[software-engineer]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Sat, 22 Mar 2025 17:20:57 GMT</pubDate>
            <atom:updated>2025-03-24T13:46:11.344Z</atom:updated>
            <content:encoded><![CDATA[<p>As some of you may have read recently, Microsoft has decided to rewrite the TypeScript compiler in Go because of the significant performance improvements (a 10x reduction in execution time).</p><p>While an initial implementation is available <a href="https://github.com/microsoft/typescript-go">here</a>, let’s see how simple it can be to build a compiler in Go from scratch.</p><p>In this first small project, we’ll only support few basic constructs such as function definitions, variable declarations and the console output. The goal of the example is simple: generate Javascript code starting from Typescript.</p><h3><a href="https://github.com/dmarro89/ts-go-compiler/blob/main/lexer/lexer.go">Lexer</a></h3><p>The first component required in a compiler is the <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/lexer/lexer.go"><strong>Lexer</strong></a> or “<em>Lexical Analyzer</em>”, which reads the source code and identifies each <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/token/token.go"><strong>Token</strong></a><strong> </strong>composing it. Each token is assigned a specific type so that we can ensure they are correctly positioned and followed (or preceded) by the appropriate tokens.</p><p>First, we define our <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/token/token.go"><strong>TokenType</strong></a> and all the types of <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/token/token.go"><strong>Tokens</strong></a> needed at the moment.</p><pre>type TokenType int<br><br>const (<br> ILLEGAL TokenType = iota<br> EOF<br> // Identifiers + literals<br> IDENT  // variable names, functions, etc.<br> INT    // integer numbers<br> STRING // strings<br> <br> // Delimiters<br> COMMA     // ,<br> SEMICOLON // ;<br> COLON     // :<br><br> LPAREN // (<br> RPAREN // )<br> LBRACE // {<br> RBRACE // }<br> <br> // Keywords<br> FUNCTION<br> LET<br> CONST<br> VAR<br> RETURN<br> IF<br> ELSE<br> CONSOLE<br> LOG<br> DOT<br>}</pre><p>Now, it’s possible to define our <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/lexer/lexer.go"><strong>Lexer</strong></a> struct along with the essential <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/lexer/lexer.go#L52C42-L53C1"><strong>NextToken</strong></a><strong> </strong>method that reads the next “word” from the source code and converts it in our <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/token/token.go#L3"><strong>TokenType</strong></a>.</p><pre><br>// Lexer is responsible for scanning the source code<br>type Lexer struct {<br> input        string<br> position     int  // current position in input (points to current character)<br> readPosition int  // current reading position in input (after current character)<br> ch           byte // current character under examination<br> line         int  // current line<br> column       int  // current column<br>}<br><br>// NextToken returns the next token from the input<br>func (l *Lexer) NextToken() token.Token {<br> var tok token.Token<br><br> // Skip whitespace<br> l.skipWhitespace()<br><br> // Save current position for the token<br> tok.Line = l.line<br> tok.Column = l.column<br><br> switch l.ch {<br> ...<br> case &#39;,&#39;:<br>  tok = token.Token{Type: token.COMMA, Literal: string(l.ch)}<br> case &#39;;&#39;:<br>  tok = token.Token{Type: token.SEMICOLON, Literal: string(l.ch)}<br> case &#39;:&#39;:<br>  tok = token.Token{Type: token.COLON, Literal: string(l.ch)}<br> case &#39;(&#39;:<br>  tok = token.Token{Type: token.LPAREN, Literal: string(l.ch)}<br> case &#39;)&#39;:<br>  tok = token.Token{Type: token.RPAREN, Literal: string(l.ch)}<br> case &#39;{&#39;:<br>  tok = token.Token{Type: token.LBRACE, Literal: string(l.ch)}<br> case &#39;}&#39;:<br>  tok = token.Token{Type: token.RBRACE, Literal: string(l.ch)}<br> case &#39;.&#39;:<br>  tok = token.Token{Type: token.DOT, Literal: string(l.ch)}<br> case &#39;&quot;&#39;:<br>  tok.Type = token.STRING<br>  tok.Literal = l.readString(&#39;&quot;&#39;)<br> case 0:<br>  tok.Literal = &quot;&quot;<br>  tok.Type = token.EOF<br> default:<br>  if isLetter(l.ch) {<br>   tok.Literal = l.readIdentifier()<br>   tok.Type = lookupIdent(tok.Literal)<br>   return tok<br>  } else if isDigit(l.ch) {<br>   tok.Type = token.INT<br>   tok.Literal = l.readNumber()<br>   return tok<br>  } else {<br>   tok = token.Token{Type: token.ILLEGAL, Literal: string(l.ch)}<br>  }<br> }<br><br> l.readChar()<br> return tok<br>}</pre><p>As you can see, the <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/lexer/lexer.go#L52C42-L53C1"><strong>NextToken</strong></a> method processes the next segment of the source (skipping any whitespace) and, based on the token’s composition, determines if it is an operator, an identifier, a delimiter, or a keyword.</p><h3><a href="https://github.com/dmarro89/ts-go-compiler/blob/main/parser/parser.go">Parser</a></h3><p>Next, we need a <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/parser/parser.go"><strong>Parser</strong></a> — a tool that analyzes the stream of tokens generated by the <strong>Lexer</strong> and builds an <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/ast/ast.go"><strong>AST</strong></a> (<strong>Abstract Syntax Tree</strong>), a structured and hierarchical representation of the code that mirrors its syntax. In practical terms, the parser verifies that the syntax used is correct.</p><p>As usual, we start by defining our <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/parser/parser.go"><strong>Parser</strong></a> and <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/ast/ast.go"><strong>AST</strong></a> structs.</p><pre>// Parser represents the parser<br>type Parser struct {<br> l              *lexer.Lexer<br> curToken       token.Token<br> peekToken      token.Token<br> errors         []string<br> prefixParseFns map[token.TokenType]prefixParseFn<br> infixParseFns  map[token.TokenType]infixParseFn<br>}<br><br>// Node represents a node in the AST<br>type Node interface {<br> TokenLiteral() string<br> String() string<br>}<br><br>// Statement represents a statement<br>type Statement interface {<br> Node<br> statementNode()<br>}<br><br>// Expression represents an expression<br>type Expression interface {<br> Node<br> expressionNode()<br>}<br><br>// Program represents a complete program<br>type Program struct {<br> Statements []Statement<br>}</pre><p>The<em> prexifParseFns</em> and <em>infixParseFns</em> in the <strong>Parser</strong> hold the functions registered to handle prefix and infix parsing — each starting from a token — to verify whether an expression is syntactically correct.</p><p>The <strong>AST</strong>, instead, is composed of a series of statements, where each statement is a node.</p><pre>// ParseProgram parses the program<br>func (p *Parser) ParseProgram() *ast.Program {<br> program := &amp;ast.Program{}<br> program.Statements = []ast.Statement{}<br><br> for !p.curTokenIs(token.EOF) {<br>  stmt := p.parseStatement()<br>  if stmt != nil {<br>   program.Statements = append(program.Statements, stmt)<br>  }<br>  p.nextToken()<br> }<br><br> return program<br>}<br><br>func (p *Parser) parseStatement() ast.Statement {<br> switch p.curToken.Type {<br> case token.LET:<br>  return p.parseLetStatement()<br> case token.RETURN:<br>  return p.parseReturnStatement()<br> default:<br>  return p.parseExpressionStatement()<br> }<br>}</pre><p>The <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/parser/parser.go#L96"><strong>ParseProgram</strong></a> method loops through all tokens until the end of the file, parsing each statement and adding it to the <strong>AST</strong>.</p><p>The <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/parser/parser.go#L111"><strong>parseStatement</strong></a><strong> </strong>method decides which parsing function to call based on the current token’s type. In this simplified implementation, we only consider three scenarios: variable declaration with <em>let</em>, returning a value, and parsing a generic expression.</p><p>Let’s examine how a <em>let</em> statement is parsed in the <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/parser/parser.go#L122"><strong>parseLetStatement</strong></a> method.</p><pre>func (p *Parser) parseLetStatement() *ast.LetStatement {<br> stmt := &amp;ast.LetStatement{Token: p.curToken}<br><br> if !p.expectPeek(token.IDENT) {<br>  return nil<br> }<br><br> stmt.Name = &amp;ast.Identifier{Token: p.curToken, Value: p.curToken.Literal}<br><br> if !p.expectPeek(token.ASSIGN) {<br>  return nil<br> }<br><br> p.nextToken()<br><br> stmt.Value = p.parseExpression(LOWEST)<br><br> if p.peekTokenIs(token.SEMICOLON) {<br>  p.nextToken()<br> }<br><br> return stmt<br>}</pre><p>Here, the <em>let</em> statement first expexcts an identifier (the name of the variable), then an assign operator (the <strong>=</strong> symbol) and finally calls the <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/parser/parser.go#L172"><strong>parseExpression</strong></a> method to parse the value assigned to the variable.</p><p>For example, in this line:</p><pre>let a = 10</pre><ul><li><strong>let</strong> is recognized as the variable declaration</li><li><strong>a</strong> is the identifier</li><li><strong>=</strong> is the assignment operator</li><li><strong>10</strong> is the parsed value</li></ul><p>The <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/parser/parser.go#L172"><strong>parseExpression</strong></a> method makes use of the registered prefix and infix parse functions.</p><pre>func (p *Parser) parseExpression(precedence int) ast.Expression {<br> prefix := p.prefixParseFns[p.curToken.Type]<br> if prefix == nil {<br>  p.noPrefixParseFnError(p.curToken.Type)<br>  return nil<br> }<br> leftExp := prefix()<br><br> for !p.peekTokenIs(token.SEMICOLON) &amp;&amp; precedence &lt; p.peekPrecedence() {<br>  infix := p.infixParseFns[p.peekToken.Type]<br>  if infix == nil {<br>   return leftExp<br>  }<br><br>  p.nextToken()<br><br>  leftExp = infix(leftExp)<br> }<br><br> return leftExp<br>}</pre><p>If a token is not recognized as valid, an error is returned indicating that the syntax is not correct.</p><h3><a href="https://github.com/dmarro89/ts-go-compiler/blob/main/typecheck/typecheck.go"><strong>Type Check</strong></a></h3><p>The next step is to add a <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/typecheck/typecheck.go"><strong>Type Checker</strong></a> to ensure that the Typescript’s type system is strictly enforced. We define a <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/typecheck/typecheck.go#L24"><strong>TypeChecker</strong></a> which has the responsibility to analyze the <strong>AST</strong> and be sure that the types used are correct.</p><pre>// TypeChecker performs type checking on the AST<br>type TypeChecker struct {<br> errors []string<br> env    *TypeEnvironment<br>}<br><br>// TypeEnvironment stores variable types<br>type TypeEnvironment struct {<br> store map[string]Type<br> outer *TypeEnvironment<br>}</pre><p>The <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/typecheck/typecheck.go#L24"><strong>TypeChecker</strong></a> has a method called <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/typecheck/typecheck.go#L50"><strong>Check</strong></a> doing this.</p><pre>// Check performs type checking on a program<br>func (tc *TypeChecker) Check(program *ast.Program) []string {<br> for _, stmt := range program.Statements {<br>  tc.checkStatement(stmt)<br> }<br> return tc.errors<br>}<br><br>func (tc *TypeChecker) checkStatement(stmt ast.Statement) Type {<br> switch s := stmt.(type) {<br> case *ast.LetStatement:<br>  return tc.checkLetStatement(s)<br> case *ast.ReturnStatement:<br>  return tc.checkReturnStatement(s)<br> case *ast.ExpressionStatement:<br>  return tc.checkExpression(s.Expression)<br> case *ast.BlockStatement:<br>  return tc.checkBlockStatement(s)<br> default:<br>  return &amp;BasicType{Name: &quot;void&quot;}<br> }<br>}</pre><p>The <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/typecheck/typecheck.go#L50"><strong>Check</strong></a> method inspects each statement and calls the corresponding function based on the statement type. For example, the <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/typecheck/typecheck.go#L99"><strong>checkExpression</strong></a> function handles expression type checking.</p><pre>func (tc *TypeChecker) checkExpression(expr ast.Expression) Type {<br> switch e := expr.(type) {<br> case *ast.IntegerLiteral:<br>  return &amp;BasicType{Name: &quot;number&quot;}<br> case *ast.StringLiteral:<br>  return &amp;BasicType{Name: &quot;string&quot;}<br> case *ast.Identifier:<br>  return tc.checkIdentifier(e)<br> case *ast.FunctionLiteral:<br>  return &amp;BasicType{Name: &quot;function&quot;}<br> default:<br>  tc.addError(fmt.Sprintf(&quot;unknown expression type: %T&quot;, expr))<br>  return &amp;BasicType{Name: &quot;unknown&quot;}<br> }<br>}</pre><p>At this stage, only numbers, strings, and functions are recognized. If any other expression type is encountered, a type error is raised.</p><h3><a href="https://github.com/dmarro89/ts-go-compiler/blob/main/codegen/codegen.go">Code Generation</a></h3><p>Now that we have our <strong>AST</strong> type-checked, we can convert each node into Javascript using a <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/codegen/codegen.go"><strong>code generator</strong></a>.</p><p>In a very simplified manner, our <strong>code generator</strong> converts each token of the AST into Javascript.</p><pre>// generateJavaScript generates JavaScript code<br>func (g *Generator) GenerateJavaScript(program *ast.Program) string {<br> var out bytes.Buffer<br><br> for _, stmt := range program.Statements {<br>  out.WriteString(g.generateJSStatement(stmt))<br>  out.WriteString(&quot;\n&quot;)<br> }<br><br> return out.String()<br>}<br><br>func (g *Generator) generateJSStatement(stmt ast.Statement) string {<br> switch s := stmt.(type) {<br> case *ast.LetStatement:<br>  return fmt.Sprintf(&quot;let %s = %s;&quot;, s.Name.Value, g.generateJSExpression(s.Value))<br> case *ast.ReturnStatement:<br>  return fmt.Sprintf(&quot;return %s;&quot;, g.generateJSExpression(s.ReturnValue))<br> case *ast.ExpressionStatement:<br>  return g.generateJSExpression(s.Expression) + &quot;;&quot;<br> default:<br>  return &quot;&quot;<br> }<br>}<br><br>func (g *Generator) generateJSExpression(expr ast.Expression) string {<br> if expr == nil {<br>  return &quot;&quot;<br> }<br><br> switch e := expr.(type) {<br> case *ast.IntegerLiteral:<br>  return e.Token.Literal<br> case *ast.StringLiteral:<br>  return fmt.Sprintf(&quot;\&quot;%s\&quot;&quot;, e.Value)<br> case *ast.Identifier:<br>  return e.Value<br> default:<br>  return &quot;&quot;<br> }<br>}</pre><h3><a href="https://github.com/dmarro89/ts-go-compiler/blob/main/compiler/compiler.go">The Compiler</a></h3><p>Finally, here’s how our complete <a href="https://github.com/dmarro89/ts-go-compiler/blob/main/compiler/compiler.go"><strong>compiler</strong></a> comes together.</p><pre>func (c *Compiler) Compile(input string) (string, error) {<br> // Initialize lexer<br> l := lexer.New(input)<br><br> // Parse input<br> p := parser.New(l)<br> program := p.ParseProgram()<br><br> if len(p.Errors()) &gt; 0 {<br>  return &quot;&quot;, errors.New(&quot;parse errors: &quot; + strings.Join(p.Errors(), &quot;, &quot;))<br> }<br><br> // Type check<br> tc := typecheck.New()<br> errs := tc.Check(program)<br> if len(errs) &gt; 0 {<br>  return &quot;&quot;, errors.New(&quot;type error: &quot; + errs[0])<br> }<br><br> // Generate code<br> generator := codegen.New()<br> output := generator.GenerateJavaScript(program)<br><br> return output, nil<br>}</pre><p>To recap, the <a href="https://github.com/dmarro89/ts-go-compiler/blob/0b45a57922f7be6b0a7f9da5c6bd9764b5e699eb/compiler/compiler.go#L42"><strong>Compile</strong></a> method follows these steps:</p><ol><li><strong>Lexer:</strong> Initialize a new lexer to scan the source code.</li><li><strong>Parser:</strong> Analyze the tokens to build the AST.</li><li><strong>Type Checking:</strong> Verify that type rules are adhered to.</li><li><strong>Code Generation:</strong> Convert each AST node into JavaScript code according to the defined rules.</li></ol><h3>Conclusion</h3><p>Here’s the link of the <a href="https://github.com/dmarro89/ts-go-compiler/tree/main">Github Project </a>— <a href="https://github.com/dmarro89/ts-go-compiler/tree/main"><strong>ts-go-compiler</strong></a> — where all this stuff (and other yet) is located (add a <strong>Star</strong> if you want to stay updated!) .</p><p>Of course this is just a simple example of how a compiler can be and how simple and also impacting can be when doing it in <strong>Go</strong>.</p><p>If you find the project interesting, please feel free to follow the <a href="https://github.com/dmarro89/ts-go-compiler/discussions">Discussions</a> and contribute your <a href="https://github.com/dmarro89/ts-go-compiler/discussions/2">ideas</a> for improvements. I look forward to collaborating on how to evolve this project further.</p><p>f you’re passionate about <strong>Go</strong> and enjoy this kind of content, follow me on <a href="https://medium.com/@davideraffaele.marro">Medium</a>, <a href="https://github.com/dmarro89">GitHub</a>, and <a href="https://www.linkedin.com/feed/">LinkedIn</a> to stay updated with the latest developments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3503abdca44" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/create-a-typescript-compiler-in-go-f3503abdca44">Create a Typescript Compiler in Go</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning in Go — Part 1]]></title>
            <link>https://levelup.gitconnected.com/machine-learning-in-go-part-1-f8ef8f489c91?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8ef8f489c91</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Mon, 23 Dec 2024 16:55:35 GMT</pubDate>
            <atom:updated>2024-12-27T01:59:30.171Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning in Go — Part 1</h3><p>When we think of machine learning, <strong>Python</strong> dominates the conversation. Its extensive libraries and frameworks, such as TensorFlow, PyTorch, and scikit-learn, make it the go-to language for most ML practitioners. But what if you’re like me, someone who loves Go for its speed and simplicity, and wants to use it for machine learning? Is it even possible?</p><p>Spoiler alert: yes, it is. In this article, I’ll show you how to implement a basic linear regression model using Go, inspired by <a href="https://colab.research.google.com/github/google/eng-edu/blob/main/ml/cc/exercises/linear_regression_taxi.ipynb">Google’s Linear Regression Taxi Example</a>. We’ll use the following Go libraries:</p><ul><li><a href="https://github.com/gomlx/gomlx"><strong>GoMLX</strong></a>: A machine learning framework in Go, built on top of XLA. Think of it as TensorFlow for Go but lighter and Go-native.</li><li><a href="https://github.com/go-gota/gota"><strong>gota</strong></a>: For loading and manipulating data from CSV files.</li><li><a href="https://github.com/gonum/plot"><strong>gonum/plot</strong></a>: For visualizing our results.</li></ul><h3>Why Machine Learning in Go?</h3><p>Before diving into the code, you might wonder, <em>Why use Go for machine learning when Python already does it so well?</em> A few reasons:</p><ol><li><strong>Performance</strong>: Go’s compiled nature and support for concurrency make it blazing fast compared to Python, especially when working with custom workflows.</li><li><strong>Ecosystem</strong>: While Go’s ML ecosystem is nascent, tools like GoMLX are bridging the gap.</li><li><strong>Curiosity</strong>: If you’re like me, the challenge of doing something unconventional in Go is just too tempting to resist.</li></ol><h3>The problem: predicting Taxi Fares</h3><p>The problem we’ll tackle is predicting <strong>taxi fares</strong> based on features like <strong>trip distance</strong> and <strong>trip duration</strong>. Our goal is to train a linear regression model that predicts fares based on these inputs.</p><p>What we’ll do, is just to replicate step by step all the operations performed in the <a href="https://colab.research.google.com/github/google/eng-edu/blob/main/ml/cc/exercises/linear_regression_taxi.ipynb">Google example</a> to understand what’s the difference between the current Python ecosystem and the rising ML libraries in Go.</p><h3>Step 1: Preparing the Data</h3><p>We’ll start by loading our data. For this example, I’m using a dataset named chicago_taxi_train.csv (you can find it in <a href="https://colab.research.google.com/github/google/eng-edu/blob/main/ml/cc/exercises/linear_regression_taxi.ipynb#scrollTo=sgR4YRjj5T-b">Google’s notebook</a>).</p><p>I’ve created some utilities structs in Go providing methods to extract data from a CSV and load them inside a <em>DataFrame</em> — a struct provided by the <strong>gota</strong> library.</p><p>Here’s the <a href="https://github.com/dmarro89/go-ml/blob/main/dataset/dataset.go">implementation</a> of the utilities struct, while here is reported the first step of the example to understand how it’s used.</p><pre>package linear<br><br>import (<br> &quot;flag&quot;<br> &quot;github.com/dmarro89/go-ml/dataset&quot;<br>)<br><br>var (<br> flagCSVFile      = flag.String(&quot;csv_file&quot;, &quot;https://download.mlcc.google.com/mledu-datasets/chicago_taxi_train.csv&quot;, &quot;Path to the CSV file containing training data&quot;)<br>)<br><br>func OneFeature() {<br> flag.Parse()<br><br> // Load the dataset and print the summary statistics<br> dataSet := dataset.NewDataFrameDataset()<br> df := dataSet.GetDataFromURL(*flagCSVFile)<br><br> columns := []string{&quot;TRIP_MILES&quot;, &quot;TRIP_SECONDS&quot;, &quot;FARE&quot;, &quot;TIP_RATE&quot;}<br> df = dataSet.ExtractColumns(df, columns)<br> dataSet.DescribeData(df)<br>...<br>}</pre><p>In this snippet, we’ve just read the csv, uploaded to a <em>DataFrame</em> and print the main information to get a description of the dataset.</p><p>Here’s what <strong>gota</strong> tells about it:</p><pre>[TRIP_MILES TRIP_SECONDS FARE TIP_RATE]<br>[float int float float]<br>[8x5] DataFrame<br><br>    column   TRIP_MILES TRIP_SECONDS FARE       TIP_RATE<br> 0: mean     8.289463   1319.796397  23.905210  12.965785<br> 1: median   5.920000   1081.000000  18.750000  12.200000<br> 2: stddev   7.265672   928.932873   16.970022  15.517765<br> 3: min      0.500000   60.000000    3.250000   0.000000<br> 4: 25%      1.720000   548.000000   9.000000   0.000000<br> 5: 50%      5.920000   1081.000000  18.750000  12.200000<br> 6: 75%      14.500000  1888.000000  38.750000  20.800000<br> 7: max      68.120000  7140.000000  159.250000 648.600000<br>    &lt;string&gt; &lt;float&gt;    &lt;float&gt;      &lt;float&gt;    &lt;float&gt;</pre><p>The computation is straight easy, the library can perform complex operations and calculate statistics without any kind of problem. The <em>DescribeData</em> method, prints out some basic info about the dataset (such as column names, types and dimensions of it) and some statistics like mean, median etc.</p><h3>Step 2: Exploratory Data Analysis</h3><p>The mandatory step to understand the data is to visualize them and better understand the relationships between features and the target variable. However, Go doesn’t provide a high-level library like Python’s ones for this purpose, so I decided to create my own utility methods for:</p><ol><li><strong>Generating a Correlation Matrix</strong>: A numeric summary of how strongly variables are related to each other.</li><li><strong>Creating a Pair Plot</strong>: A visual grid showing scatter plots for every combination of selected variables.</li></ol><p>By doing this, I could better explore the data and ensure the relationships align with assumptions of linear regression.</p><h4>Correlation Matrix</h4><p><a href="https://github.com/dmarro89/go-ml/blob/main/dataset/dataset.go">Here’s the code</a> I wrote to compute a correlation matrix using raw slices from the dataset:</p><pre>func (d *DataFrameDataset) GetCorrelationMatrix(df dataframe.DataFrame) CorrelationMatrix {<br> columns := df.Names()<br> corrMatrix := make([][]float64, len(columns))<br> for i := 0; i &lt; len(columns); i++ {<br>  corrMatrix[i] = make([]float64, len(columns))<br>  for j := i; j &lt; len(columns); j++ {<br>   if i == j {<br>    corrMatrix[i][j] = 1.0<br>    continue<br>   }<br>   col1 := df.Col(columns[i]).Float()<br>   col2 := df.Col(columns[j]).Float()<br>   corr := stat.Correlation(col1, col2, nil)<br>   corrMatrix[i][j] = corr<br>  }<br> }<br> return corrMatrix<br>}<br><br>type CorrelationMatrix [][]float64<br><br>func (corrMat *CorrelationMatrix) Print(headers []string) {<br> table := tablewriter.NewWriter(os.Stdout)<br> tableHeader := append([]string{&quot;&quot;}, headers...)<br> table.SetHeader(tableHeader)<br> for index, row := range *corrMat {<br>  stringRow := make([]string, len(row)+1)<br>  for i, val := range row {<br>   if i == 0 {<br>    stringRow[i] = headers[index]<br>   }<br>   stringRow[i+1] = fmt.Sprintf(&quot;%0.2f&quot;, val)<br>  }<br>  table.Append(stringRow)<br> }<br> table.Render()<br>}</pre><p>Here’s the usage.</p><pre>// Get and print the correlation matrix<br> corrMat := dataSet.GetCorrelationMatrix(df)<br> corrMat.Print(columns)</pre><p>The snippet prints it out the correlation matrix.</p><pre>+--------------+------------+--------------+------+----------+<br>|              | TRIP MILES | TRIP SECONDS | FARE | TIP RATE |<br>+--------------+------------+--------------+------+----------+<br>| TRIP_MILES   |       1.00 |         0.80 | 0.98 |    -0.05 |<br>| TRIP_SECONDS |       0.00 |         1.00 | 0.83 |    -0.08 |<br>| FARE         |       0.00 |         0.00 | 1.00 |    -0.07 |<br>| TIP_RATE     |       0.00 |         0.00 | 0.00 |     1.00 |<br>+--------------+------------+--------------+------+----------+</pre><p>At this point, a pair plot can make this data more readable.</p><h4>Pair Plot</h4><p>To visualize relationships, <a href="https://github.com/dmarro89/go-ml/blob/main/plotter/plotter.go">I built another utility</a> for generating pair plots using <strong>gonum/plot</strong>. It creates scatter plots for all pairs of variables (histograms when comparing the same variable) and organizes them in a grid:</p><pre>func (p *FloatPlotter) PairPlot(data [][]float64, labels []string) {<br> gridSize := len(labels)<br> tileSize := 4 * vg.Inch<br> imgWidth := tileSize * vg.Length(gridSize)<br> imgHeight := tileSize * vg.Length(gridSize)<br> img := vgimg.New(imgWidth, imgHeight)<br> dc := draw.New(img)<br><br> for row, yCol := range data {<br>  for col, xCol := range data {<br><br>   var plot *plot.Plot<br>   if row == col {<br>    data := make([]float64, len(xCol))<br>    copy(data, xCol)<br>    plot = p.PlotHistogram(data, labels[col])<br>   } else {<br>    xData := make([]float64, len(xCol))<br>    copy(xData, xCol)<br>    yData := make([]float64, len(yCol))<br>    copy(yData, yCol)<br>    plot = p.PlotScatter(xData, yData, labels[col], labels[row])<br>   }<br><br>   xStart := vg.Length(col) * tileSize<br>   yStart := vg.Length(gridSize-row-1) * tileSize<br>   canvas := draw.Canvas{<br>    Canvas: dc,<br>    Rectangle: vg.Rectangle{<br>     Min: vg.Point{X: xStart, Y: yStart},<br>     Max: vg.Point{X: xStart + tileSize, Y: yStart + tileSize},<br>    },<br>   }<br>   plot.Draw(canvas)<br>  }<br> }<br><br> err := os.Mkdir(&quot;plot&quot;, os.ModePerm)<br> if err != nil {<br>  fmt.Printf(&quot;error creating directory: %v&quot;, err)<br> }<br><br> w, err := os.Create(fmt.Sprintf(&quot;./plot/data_%s.png&quot;, strings.Join(labels, &quot;_&quot;)))<br> if err != nil {<br>  log.Panic(err)<br> }<br> defer w.Close()<br><br> _, err = vgimg.PngCanvas{Canvas: img}.WriteTo(w)<br><br> if err != nil {<br>  log.Panic(err)<br> }<br>}</pre><p>The results is reported in the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2rbCt9PuNxMKlyZcsS40hA.png" /></figure><p>As we can see, the <strong>FARE</strong> is highly related to the <strong>TRIP MILES</strong> so we can start to try to predict a model with a simple linear regression with one feature.</p><p>The linear regression model will try to predict the <strong>FARE</strong> just based on the <strong>TRIP MILES</strong> information.</p><h3>Step 3: Building and Training the Model with GoMLX</h3><h4>Model Definition: The modelGraph Function</h4><p>The <strong>modelGraph</strong> function defines the architecture of our model. Since we’re implementing a simple linear regression, the model consists of a single <strong>Dense Layer</strong> with one output node (representing the predicted fare).</p><p><a href="https://github.com/dmarro89/go-ml/blob/main/linearregression/onefeature/train.go">Here’s the code</a> for defining the model:</p><pre>func modelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node {<br>	_ = spec<br>	logits := layers.Dense(ctx, inputs[0], true, 1) // A single output node<br>	return []*Node{logits}<br>}</pre><ul><li><strong>ctx</strong>: This context object manages the weights (coefficients) and biases that the model learns during training.</li><li><strong>inputs[0]</strong>: Represents the input features (<em>TRIP_MILES</em> in this case).</li><li><strong>layers.Dense</strong>: A dense layer performs the operation y=Wx+by = Wx + by=Wx+b, where WWW (weights) and bbb (bias) are learned during training.</li></ul><h4>Preparing the Dataset</h4><p>To train the model, we need to convert the raw CSV data into tensors that GoMLX can process. The <a href="https://github.com/dmarro89/go-ml/blob/main/linearregression/onefeature/train.go"><strong>dataframeToSingleTensors</strong></a> function handles this:</p><pre>func dataframeToSingleTensors(df dataframe.DataFrame) (inputs, labels *tensors.Tensor) {<br> numRows := df.Nrow()<br><br> inputData := make([]float64, numRows)<br> labelData := make([]float64, numRows)<br><br> for i := 0; i &lt; numRows; i++ {<br>  row := df.Subset(i)<br>  labelValue := row.Col(FARE_COLUMN).Float()[0]<br>  labelData[i] = labelValue<br><br>  featureIdx := 0<br>  value := row.Col(TRIP_MILES_COLUMN).Float()[0]<br>  inputData[i+featureIdx] = value<br>  featureIdx++<br> }<br><br> inputs = tensors.FromFlatDataAndDimensions(inputData, numRows, 1)<br> labels = tensors.FromFlatDataAndDimensions(labelData, numRows, 1)<br> return inputs, labels<br>}</pre><p>This function extracts the <em>TRIP_MILES</em> (input feature) and <em>FARE</em> (target label) and transforms them into tensors.</p><h4>Training the Model</h4><p>Now that the data is ready, let’s train the model using <strong>GoMLX’s training loop</strong>. <a href="https://github.com/dmarro89/go-ml/blob/main/linearregression/onefeature/train.go">Here’s the core function that orchestrates the training process</a>:</p><pre>var (<br> flagCSVFile      = flag.String(&quot;csv_file&quot;, &quot;https://download.mlcc.google.com/mledu-datasets/chicago_taxi_train.csv&quot;, &quot;Path to the CSV file containing training data&quot;)<br> flagNumEpochs    = flag.Int(&quot;epochs&quot;, 20, &quot;Number of epochs&quot;)<br> flagLearningRate = flag.Float64(&quot;learning_rate&quot;, 0.001, &quot;Initial learning rate.&quot;)<br> flagBatchSize    = flag.Int(&quot;batch_size&quot;, 50, &quot;Batch size for training&quot;)<br>)<br><br>func OneFeature() {<br>...<br> // Create tensors from the dataframe just for the columns TRIP_MILES and FARE<br> inputs, labels := dataframeToSingleTensors(df)<br> fmt.Printf(&quot;Training data (inputs, labels): (%s, %s)\n\n&quot;, inputs.Shape(), labels.Shape())<br><br> // Create a new backend<br> backend := backends.New()<br> // Create an in-memory dataset from the tensors.<br> dataset, err := data.InMemoryFromData(backend, &quot;Dataset&quot;, []any{inputs}, []any{labels})<br> if err != nil {<br>  panic(fmt.Errorf(&quot;error creating dataset: %v&quot;, err))<br> }<br><br> dataset = dataset.Shuffle().BatchSize(*flagBatchSize, false)<br><br> // Create context with learning rate<br> ctx := context.New()<br> ctx.SetParam(optimizers.ParamLearningRate, *flagLearningRate)<br><br> // Create a new trainer<br> trainer := train.NewTrainer(backend, ctx, modelGraph,<br>  losses.MeanSquaredError,<br>  optimizers.StochasticGradientDescent(),<br>  nil, nil)<br><br> loop := train.NewLoop(trainer)<br> // Collect some information about the loops<br> attachEpochMetricsCollector(loop)<br> commandline.AttachProgressBar(loop)<br><br> // Train the model<br> _, err = loop.RunEpochs(dataset, *flagNumEpochs)<br> if err != nil {<br>  panic(fmt.Errorf(&quot;error training model: %v&quot;, err))<br> }<br><br> // Get the learned coefficients and bias<br> coefVar, biasVar := ctx.GetVariableByScopeAndName(&quot;/dense&quot;, &quot;weights&quot;), ctx.GetVariableByScopeAndName(&quot;/dense&quot;, &quot;biases&quot;)<br> learnedCoef, learnedBias := coefVar.Value(), biasVar.Value()<br> fmt.Printf(&quot;Learned coefficients: %0.5f\n&quot;, learnedCoef.Value())<br> fmt.Printf(&quot;Learned bias: %0.5f\n&quot;, learnedBias.Value())<br><br>...<br>}</pre><p><strong>Key Highlights:</strong></p><ol><li><strong>Shuffling and Batching</strong>: The dataset is shuffled and split into smaller batches to improve training stability.</li><li><strong>Loss Function</strong>: We use <strong>Mean Squared Error</strong> (MSE), the standard for regression tasks.</li><li><strong>Optimizer</strong>: Stochastic Gradient Descent (SGD) adjusts the weights and bias to minimize the loss function.</li><li><strong>Training Loop</strong>: The <em>train.NewLoop</em> automates the training process, iterating over the dataset for the specified number of epochs.</li><li><strong>Extracting Results</strong>: After training, the learned coefficients and bias are retrieved from the model’s internal context:<br>- <strong>Coefficients (learnedCoef)</strong>: Represent the relationship between the input feature and the target output. A higher coefficient indicates a stronger influence of the input feature on the output.<br>- <strong>Bias (learnedBias)</strong>: Represents the baseline prediction when all input features are zero. It essentially shifts the regression line vertically.</li><li><strong>Output Interpretation</strong>:<br>- The coefficients and bias together define the linear model: <br><em>FARE</em> = <strong>learnedCoef</strong> * <em>TRIP_MILES</em> + <strong>learnedBias</strong></li></ol><h4>Visualizing the results</h4><p>After the training, it’s essential to evaluate how well the model aligns with the actual data. To do this, I created two visualizations:</p><p><strong>Model Predictions vs Actual Data</strong>:</p><p><a href="https://github.com/dmarro89/go-ml/blob/main/plotter/plotter.go">I implemented a <em>PlotModel</em> function</a> that visualizes the model&#39;s predictions alongside the actual data. This function overlays the predicted regression line onto a scatter plot of the input feature (<em>TRIP_MILES</em>) versus the target variable (<em>FARE</em>). Here&#39;s what the function does:</p><ul><li><strong>Scatter Plot</strong>: Each point represents an observed value in the dataset, showing how the actual fare varies with the trip miles.</li><li><strong>Regression Line</strong>: This line represents the model’s prediction based on the learned coefficients and bias. It shows the trend the model identified in the data.</li></ul><pre>func (p *FloatPlotter) PlotModel(xData []float64, yData []float64, weights []float64, bias float64, feature, label string) {<br> if len(xData) != len(yData) {<br>  log.Fatalf(&quot;Mismatch between feature and label lengths&quot;)<br> }<br><br> linePoints := make(plotter.XYs, len(xData))<br><br> points := make(plotter.XYs, len(xData))<br> for i := range xData {<br>  points[i].X = xData[i]<br>  points[i].Y = yData[i]<br><br>  linePoints[i].X = xData[i]<br>  linePoints[i].Y = weights[0]*xData[i] + bias<br> }<br><br> p := plot.New()<br> p.Title.Text = &quot;Model Plot&quot;<br> p.X.Label.Text = feature<br> p.Y.Label.Text = label<br><br> scatter, err := plotter.NewScatter(points)<br> if err != nil {<br>  log.Fatalf(&quot;Failed to create scatter plot: %v&quot;, err)<br> }<br> scatter.GlyphStyle.Radius = vg.Points(3)<br><br> line, err := plotter.NewLine(linePoints)<br> if err != nil {<br>  log.Fatalf(&quot;Failed to create line plot: %v&quot;, err)<br> }<br> line.LineStyle.Width = vg.Points(2)<br> line.LineStyle.Color = plotutil.Color(1)<br><br> p.Add(scatter, line)<br><br> err = os.Mkdir(&quot;plot&quot;, os.ModePerm)<br> if err != nil {<br>  fmt.Printf(&quot;error creating directory: %v&quot;, err)<br> }<br><br> if err := p.Save(8*vg.Inch, 4*vg.Inch, &quot;./plot/model_plot.png&quot;); err != nil {<br>  log.Fatalf(&quot;Failed to save plot: %v&quot;, err)<br> }<br>}</pre><p><strong>Insights from the Plot:</strong> The plot below demonstrates that the model has captured the linear relationship between <em>TRIP_MILES</em> and <em>FARE</em> effectively. There are some deviations, but overall, the model provides a reasonable approximation for this feature-target pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l0PTrc9mQ7rRRD5nwJD1UQ.png" /></figure><p><strong>Training Loss Curve</strong>:</p><p>To ensure that the model is learning effectively, I monitored the training loss at the end of each epoch. A consistent decrease in the loss function over epochs indicates that the model is optimizing and improving its predictions.</p><p><strong>How it works:</strong></p><ul><li>The attachEpochMetricsCollector function collects the training loss metrics at regular intervals.</li><li>These metrics are then plotted to visualize the model’s learning process.</li></ul><pre>func attachEpochMetricsCollector(loop *train.Loop) {<br> currentEpoch := 0<br><br> loop.OnStep(&quot;EpochMetrics&quot;, 100, func(loop *train.Loop, metrics []*tensors.Tensor) error {<br>  if loop.LoopStep &gt; 0 &amp;&amp; loop.LoopStep%*flagBatchSize == 0 {<br>   currentEpoch++<br>   fmt.Printf(&quot;Epoch %d completed. Metrics:\n&quot;, currentEpoch)<br><br>   trainMetrics := loop.Trainer.TrainMetrics()<br>   for i, metric := range metrics {<br>    metricsMap[trainMetrics[i].Name()] = append(metricsMap[trainMetrics[i].Name()], must.M1(strconv.ParseFloat(trainMetrics[i].PrettyPrint(metric), 64)))<br>   }<br>  }<br>  return nil<br> })<br>}</pre><p><strong>Insights from the Loss Curve:</strong></p><ul><li>A smooth, downward-sloping curve indicates that the model is learning and converging effectively.</li><li>If the curve plateaus too early or fluctuates significantly, it could signal underfitting, overfitting, or the need for tuning hyperparameters like learning rate or batch size.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dN7rnzLFxC-3UnCUvNmatA.png" /></figure><p>The loss curve plot reveals that the model’s loss decreases steadily during training. However, it may stabilize around a certain point, suggesting that the model has reached its optimal performance with the current setup.</p><h3>Conclusions</h3><p>In the end, building this example in Go turned out to be quite complex. This is mainly because the <a href="https://github.com/gomlx/gomlx"><strong>GoMLX</strong></a><strong> library is still in its early stages</strong>, and it took some time to piece everything together. There were moments of trial and error, but the journey was worth it for the insights gained.</p><p>A special thanks to <a href="https://medium.com/u/e3d40a18f693">Jan Pfeifer</a>, the creator of <a href="https://github.com/gomlx/gomlx">GoMLX</a>, who was incredibly supportive and helped make this example possible. Jan’s availability and guidance were instrumental in resolving some tricky issues, and his commitment to making the library more accessible is evident.</p><h4>A Word of Caution</h4><p>That said, there are still hurdles when working with this ecosystem, especially if you’re using <strong>Mac OS</strong>:</p><ol><li><strong>PJRT is still a nascent technology</strong>: While it promises a lot, it’s not yet mature. Building the PJRT plugin for the CPU in XLA was a painstaking process — one I can only describe as a recurring nightmare. <a href="https://medium.com/u/e3d40a18f693">Jan Pfeifer</a>, fortunately, has fixed this stuff for all the Mac users.</li><li><strong>XLA’s limited support</strong>: Despite opening tickets and seeking help, XLA’s support system left much to be desired. Progress often felt like a solo endeavor.</li><li><strong>Running on Mac hardware</strong>: At the moment, I was only able to run this model on the CPU of my Mac, and even that took significant effort. Apple has developed <strong>JAX-Metal</strong> to enable GPU acceleration on Mac, but it’s far from being stable. Bugs and compatibility issues abound, and using it reliably feels like a distant dream.</li></ol><p>Despite these challenges, I’m optimistic. I hope that over time, both <strong>PJRT</strong> and <strong>JAX-Metal</strong> will mature into robust technologies, making ML workflows smoother and more accessible for everyone.</p><h3>What’s Next?</h3><p>This is just the beginning.</p><p>The next step, Part 2, is to train a model with two features instead of one and then validate and reuse the model itself.</p><p>It should be an easier example compared to this one but I think that the split is necessary because of the complexity of the argument.</p><p>More examples are coming soon, not just limited to a simple linear regression, so stay connected:</p><ul><li>The <a href="https://github.com/dmarro89/go-ml/tree/main"><strong>Github</strong> <strong>repository</strong></a> with the implementation of the example</li><li><a href="https://www.linkedin.com/in/davideraffaelemarro/"><strong>Follow me here on LinkedIn</strong></a> for updates and new projects.</li><li><a href="https://github.com/dmarro89"><strong>Check out my GitHub profile</strong></a> for code and experiments.</li><li>Or feel free to <strong>connect with me directly on LinkedIn</strong> to discuss machine learning in Go, GoMLX, or anything else you’re curious about!</li></ul><p>Let’s explore the future of <strong>machine learning in Go</strong> together. 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8ef8f489c91" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/machine-learning-in-go-part-1-f8ef8f489c91">Machine Learning in Go — Part 1</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to handle PJRT on macOS]]></title>
            <link>https://levelup.gitconnected.com/how-to-handle-pjrt-on-a-mac-mx-arm-26d16a0c9bd9?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/26d16a0c9bd9</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Sat, 23 Nov 2024 08:59:13 GMT</pubDate>
            <atom:updated>2024-11-24T22:27:00.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3xuG2r1OqZ312HqCNteQw.png" /></figure><p>If you’re like me, you enjoy diving into the guts of new technologies — not just using them, but understanding how they work under the hood. Recently, I decided to experiment with <a href="https://opensource.googleblog.com/2023/05/pjrt-simplifying-ml-hardware-and-framework-integration.html"><strong>PJRT</strong></a> on my Mac M1. Spoiler alert: it wasn’t straightforward, but I managed to make it work, and I think it’s worth sharing the process with you.</p><p>This guide isn’t about showing off a perfect setup; it’s about sharing the bumps along the way and helping you avoid them. If you’re curious about <strong>PJRT</strong>, how it connects with <strong>XLA</strong>, and how to make it run on macOS, you’re in the right place.</p><h3>What’s PJRT Anyway?</h3><p>For those of you unfamiliar, <strong>PJRT</strong> stands for <strong>Portable JAX Runtime</strong>. It’s essentially a runtime that lets you execute <strong>JAX</strong> programs using <strong>XLA</strong> (Accelerated Linear Algebra). What’s cool about <strong>PJRT</strong> is its flexibility: you can run your programs on CPUs, GPUs, or even TPUs without having to rewrite your code.</p><p>I’ve already written a story here on Medium to explain better what’s <strong>PJRT</strong>, <strong>XLA</strong> and <strong>JAX</strong> … <a href="https://medium.com/@davideraffaele.marro/xla-and-pjrt-powering-up-your-machine-learning-a08f47455059">take a look at it</a>!</p><p>On macOS, <strong>PJRT</strong> supports Metal-backed GPUs and CPUs. This is great news for anyone using Apple Silicon, as it unlocks more performance compared to using plain CPU-based execution. But getting it to work on a Mac isn’t as simple as downloading a binary — it requires rolling up your sleeves.</p><h3>Step 1: Compiling the PJRT Plugin on macOS ARM</h3><p>Here’s where the fun begins. To run PJRT on macOS, you need to compile the plugin for your architecture. I’m using a Mac M1, but the process should work for M2 as well.</p><h4>Prerequisites</h4><p>Install <strong>Bazelisk</strong>:</p><pre>brew install bazelisk</pre><p>Clone the <strong>XLA</strong> repo:</p><pre>git clone https://github.com/openxla/xla</pre><p>Set up your Python environment (optional but saves headaches later):</p><pre>python3 -m venv pjrt-env <br>source pjrt-env/bin/activate <br>pip install -U pip</pre><h4>Building the Plugin</h4><p>Here’s the part where I hit a few roadblocks, but here’s what eventually worked for me.</p><p>On macOS there’s a need to install <strong>clang</strong> — I have been able to make the build work only with <strong>clang</strong> <strong>version</strong> <strong>15</strong>.</p><p>Then, the XLA plugin build is based on Bazel. Fortunately, XLA have provided in its repository a python script (<a href="https://github.com/openxla/xla/blob/main/configure.py">configure.py</a>) that make the bazel configuration for you. It needs some options but it’s very simple to use.</p><pre>./configure.py --backend=cpu --host_compiler=clang --os=darwin --python_bin_path=$(which python3)</pre><p>We’re configuring the build of the <strong>PJRT</strong> plugin just to use the <em>CPU</em> for os darwin (so <strong>macOS</strong>), passing <strong>clang</strong> as host compiler (I’ve already installed the version 15) and setting the python path to the current installed on my Mac (version 13).</p><p>Now it’s time to build the plugin. The <a href="https://openxla.org/xla/build_from_source">XLA guide</a> recommend to build it in a Docker image but I prefer to build it on my machine.</p><p>It’s located under the <strong>xla/pjrt/c</strong> folder and this is the command to run:</p><pre>bazelisk build --test_output=all --spawn_strategy=sandboxed //xla/pjrt/c:pjrt_c_api_cpu_plugin.so</pre><p>If the build completes successfully (and it might take a while), you’ll find the plugin here:</p><pre>bazel-bin/pjrt/c/pjrt_c_api_cpu_plugin.so</pre><p>If you run into errors, don’t panic. Double-check your dependencies and the architecture flags — I had to recompile twice before I got it right.</p><p>In any case and for any kind of blocking issue please leave a comment, I’ll be happy to help you.</p><h3>Step 2: Integrating PJRT with JAX</h3><p>With the plugin compiled, you need to hook it up to JAX. I’ve followed the <a href="https://openxla.org/xla/pjrt/pjrt_integration">XLA PJRT Integration Tutorial</a>.</p><p>Here’s what worked for me:</p><p>Install JAX and JAX-lib in your venv:</p><pre>pip install --pre -U jaxlib -f https://storage.googleapis.com/jax-releases/jaxlib_nightly_releases.html<br>pip install git+https://github.com/google/jax</pre><p>Get your venv <em>site-packages</em> folder:</p><pre>python -c &quot;import site; print(site.getsitepackages()[0])&quot;</pre><p>Supposing that the site-packages path is &lt;site-packages-path&gt;, create your folder for the plugin:</p><pre>mkdir -p &lt;site-packages-path&gt;/jax_plugins/my_plugin</pre><p>Move the plugin from bazel-bin/xla/pjrt/c/pjrt_c_api_cpu_plugin.so to the folder just created.</p><pre>cp ./bazel-bin/xla/pjrt/c/pjrt_c_api_cpu_plugin.so &lt;site-packages-path&gt;/jax_plugins/my_plugin</pre><p>Now, create a file __init__.py in the <strong>my_plugin</strong> folder just created. It must contains an initialize method used to start the plugin.</p><p>Here’s the file content:</p><pre># &lt;site-packages-path&gt;/jax_plugins/my_plugin/__init__.py<br>import os<br>import jax._src.xla_bridge as xb<br><br># Flag to avoid multiple duplication of the plugin<br>plugin_initialized = False<br><br>def initialize():<br>    global plugin_initialized<br>    if plugin_initialized:<br>        return  # Avoid to register the plugin more than once<br><br>    path = os.path.join(os.path.dirname(__file__), &#39;pjrt_c_api_cpu_plugin.so&#39;)<br>    xb.register_plugin(&#39;pjrt_c_api_cpu_plugin&#39;, priority=500, library_path=path, options=None)<br>    plugin_initialized = True</pre><h3>Step 3 : Use the plugin</h3><p>Let’s start writing our python example using the PJRT plugin.</p><p>First of all, let’s create a file example.py and initialize the plugin.</p><pre>import jax<br>from jax_plugins import my_plugin<br>import jax.numpy as jnp<br><br>my_plugin.initialize()<br>jax.config.update(&quot;jax_platforms&quot;, &quot;pjrt_c_api_cpu_plugin&quot;)</pre><p>How we can check that JAX is really using the PJRT plugin?<br>Well, we can add a simple line to our program to print the devices.</p><pre>devices = jax.devices()<br>print(&quot;Devices available:&quot;)<br>for device in devices:<br>    print(f&quot;Device ID: {device.id}&quot;)<br>    print(f&quot;Device Kind: {device.device_kind}&quot;)<br>    print(f&quot;Platform: {device.platform}&quot;)<br>    print(f&quot;Host ID: {device.host_id}&quot;)<br>    print(f&quot;Process Index: {device.process_index}&quot;)<br>    print(&quot;-&quot; * 50)</pre><p>Here’s the output:</p><pre>WARNING:2024-11-23 09:37:04,762:jax._src.xla_bridge:1000: Platform &#39;pjrt_c_api_cpu_plugin&#39; is experimental and not all JAX functionality may be correctly supported!<br>Devices available:<br>Device ID: 0<br>Device Kind: cpu<br>Platform: cpu<br>Host ID: 0<br>Process Index: 0<br>--------------------------------------------------<br>Device ID: 1<br>Device Kind: cpu<br>Platform: cpu<br>Host ID: 0<br>Process Index: 0<br>--------------------------------------------------<br>Device ID: 2<br>Device Kind: cpu<br>Platform: cpu<br>Host ID: 0<br>Process Index: 0<br>--------------------------------------------------<br>Device ID: 3<br>Device Kind: cpu<br>Platform: cpu<br>Host ID: 0<br>Process Index: 0<br>--------------------------------------------------</pre><p>The output shows that there are 4 CPUs all belonging to the local node (index 0).</p><p>Now, we can add a simple example like this to the file:</p><pre># Define the function f(x) = x^2 + 1<br>def f(x):<br>    return x**2 + 1<br><br># Test values<br>inputs = jnp.array([0.1, 1, 3, 4, 5], dtype=jnp.float32)<br>expected_outputs = jnp.array([1.01, 2, 10, 17, 26], dtype=jnp.float32)<br><br># Execute the function on each input and compare it with the expected results<br>print(&quot;Testing f(x) = x^2 + 1 with PJRT plugin:&quot;)<br>for i, x in enumerate(inputs):<br>    result = f(x)<br>    expected = expected_outputs[i]<br>    print(f&quot;f({x}) = {result} (expected: {expected})&quot;)<br>    assert jnp.isclose(result, expected, atol=0.001), f&quot;Test failed for input {x}: got {result}, expected {expected}&quot;<br><br>print(&quot;All tests passed successfully.&quot;)</pre><p>Here’s the output:</p><pre>Testing f(x) = x^2 + 1 with PJRT plugin:<br>f(0.10000000149011612) = 1.0099999904632568 (expected: 1.0099999904632568)<br>f(1.0) = 2.0 (expected: 2.0)<br>f(3.0) = 10.0 (expected: 10.0)<br>f(4.0) = 17.0 (expected: 17.0)<br>f(5.0) = 26.0 (expected: 26.0)<br>All tests passed successfully.</pre><p>And with a simple add, we can test the execution of the example on each CPU found by PJRT.</p><pre>print(&quot;\nRunning computations on different devices:&quot;)<br>for device in devices:<br>    print(f&quot;\nExecuting on Device ID {device.id} ({device.device_kind})&quot;)<br><br>    # Move data on the cpu<br>    x_on_device = jax.device_put(inputs, device)<br><br>    # Execute the function on the CPU<br>    result = f(x_on_device)<br><br>    # Get the results and print them out<br>    result_host = jax.device_get(result)<br><br>    print(f&quot;Inputs: {inputs}&quot;)<br>    print(f&quot;Outputs: {result_host}&quot;)</pre><p>Here’s the complete example on <a href="https://github.com/dmarro89/jax-xla-pjrt/blob/main/pjrt_example.py">Github</a>.</p><p>If you found this article helpful, please consider following me and checking out the code:</p><ul><li><a href="https://github.com/dmarro89/jax-xla-pjrt"><strong>Code</strong></a></li><li><a href="https://github.com/dmarro89"><strong>GitHub</strong></a></li><li><a href="https://www.linkedin.com/in/davideraffaelemarro/"><strong>LinkedIn</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26d16a0c9bd9" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/how-to-handle-pjrt-on-a-mac-mx-arm-26d16a0c9bd9">How to handle PJRT on macOS</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[XLA and PJRT: powering up your Machine Learning]]></title>
            <link>https://levelup.gitconnected.com/xla-and-pjrt-powering-up-your-machine-learning-a08f47455059?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a08f47455059</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Wed, 13 Nov 2024 15:06:02 GMT</pubDate>
            <atom:updated>2024-11-24T22:15:54.905Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8JN6qnMPLCSXtwOKd1XLw.png" /></figure><p>As machine learning and deep learning models grow in complexity, so too does the need for efficiency and high performance across a variety of hardware platforms. Today, models need to run seamlessly on CPUs, GPUs, and even specialized accelerators like TPUs. Google’s <a href="https://github.com/openxla/xla"><strong>XLA</strong></a> (Accelerated Linear Algebra) and <a href="https://opensource.googleblog.com/2024/03/pjrt-plugin-to-accelerate-machine-learning.html"><strong>PJRT</strong></a> (Portable JIT Runtime) have emerged as key technologies in making this optimization not only possible but also practical and portable.</p><h3>What is XLA?</h3><p><strong>XLA</strong>, or Accelerated Linear Algebra, is a specialized compiler designed to make Machine Learning models faster and more efficient. Developed by <em>Google</em>, <strong>XLA</strong> transforms high-level operations, common in machine learning libraries like <em>TensorFlow</em> and <em>JAX</em>, into highly optimized code that can be executed more efficiently on a range of hardware.</p><p>Without <strong>XLA</strong>, these libraries perform mathematical operations exactly as written, often leading to redundant computations or inefficient use of memory. <strong>XLA</strong>, however, takes a holistic approach, analyzing entire groups of operations and reorganizes them for better performance. It identifies opportunities to streamline computations by reducing redundant steps, fusing multiple operations into single, hardware-efficient instructions, and using optimized data paths and memory layouts. These transformations produce faster-running models that consume less energy — a critical advantage in machine learning workflows.</p><h3>How XLA optimizes operations</h3><p>The optimization process in XLA involves multiple stages that transform the high-level code into efficient instructions tailored for specific hardware.</p><h4>1. Graph Analysis and Simplification</h4><p><strong>XLA</strong> starts by analyzing the computational graph generated by the high-level framework, such as <em>TensorFlow</em> or <em>JAX</em>. This graph represents the sequence and dependencies of operations in the model. The first step is to simplify this graph with constant precomputing, dead code elimination and common subexpression elimination.</p><h4>2. Fusion of Operations</h4><p>In the optimized graph, <strong>XLA</strong> identifies “fusion opportunities” where multiple operations can be combined or “fused” into a single kernel. This fusion process is critical because it reduces memory overhead and allows for efficient parallel processing.</p><h4>3. Data Layout Transformation</h4><p><strong>XLA</strong> also modifies how data is laid out in memory to optimize for specific hardware requirements. Different devices — CPUs, GPUs, TPUs — have their own preferences for data alignment and access patterns.</p><h4>4. Hardware-Specific Optimization Passes</h4><p><strong>XLA</strong> performs several hardware-specific optimizations, tailoring the instructions to the characteristics of the target device-</p><h4>5. Just-in-Time (JIT) Compilation</h4><p>Once the optimizations are applied, <strong>XLA</strong> generates machine code in a process called just-in-time (JIT) compilation. Here, the optimized code is translated directly into instructions that the device can execute.</p><h3>The Relationship Between XLA and JAX</h3><p><strong>JAX</strong> is a high-performance python library for numerical computing, particularly popular for machine learning and scientific research due to its ease of use and powerful transformation functions. At the heart of <strong>JAX</strong>’s efficiency is its deep integration with <strong>XLA</strong>, which it relies on for just-in-time (JIT) compilation and hardware-specific optimizations. This partnership between <strong>JAX</strong> and <strong>XLA</strong> allows users to write high-level Python code and then seamlessly compile it into low-level, hardware-optimized code.</p><p><strong>JAX</strong>’s core strength lies in its transformation functions, such as jit (just-in-time compilation), grad (automatic differentiation), and vmap (vectorization). When <strong>JAX</strong> code is decorated with jit, it triggers <strong>XLA</strong> to compile and optimize the function.</p><p>In essence, <strong>XLA</strong> acts as a powerful backend for <strong>JAX</strong>, enhancing its capabilities by making computations faster and automatically adapting to available hardware. This integration lets <strong>JAX</strong> users focus on high-level model development, while <strong>XLA</strong> handles the technical details of compiling and optimizing for speed and efficiency.</p><h3>Example</h3><p>To see <strong>XLA</strong>’s impact in action, let’s explore an example that performs a series of element-wise operations and a reduction on a large matrix. By comparing execution times with and without <strong>JIT</strong> compilation, we can observe the effects of <strong>XLA</strong>’s optimizations.</p><p><a href="https://github.com/dmarro89/jax-xla-pjrt/blob/main/jax-xla.py">Here the full example</a>.</p><h4>1. Define a complex function</h4><p>This function, intensive_computation, performs multiple mathematical operations within a loop. Each iteration applies sin, cos, tan, and log1p operations on the input matrix, followed by squaring the result. Finally, it performs a jnp.sum reduction operation to aggregate all elements in the matrix. The loop increases the computational load, allowing <strong>XLA</strong> more opportunities to optimize.</p><pre>import jax.numpy as jnp<br><br>def intensive_computation(x):<br>    for _ in range(10):  # Loop to increase the computational load<br>        x = jnp.sin(x) + jnp.cos(x) + jnp.tan(x)<br>        x = jnp.log1p(x ** 2)<br>    return jnp.sum(x)</pre><h4>2. Initialize a large matrix of random values</h4><p>To make the example computationally demanding, we use a large 10,000 x 10,000 matrix filled with random values. This size provides enough data for <strong>XLA</strong> to demonstrate substantial optimization benefits.</p><pre>key = jax.random.PRNGKey(0)<br>x = jax.random.normal(key, (10000, 10000), dtype=jnp.float32)</pre><h4>3. Measure execution time without JIT (no XLA optimization)</h4><p>Now, we run the function without JIT compilation to get a baseline measurement. In this version, each operation runs individually, leading to repeated memory access and less efficient computation.</p><pre># Measure execution time without JIT (no XLA)<br>start_time = time.time()<br>result_no_jit = intensive_computation(x).block_until_ready()<br>print(&quot;Execution time without XLA (no JIT):&quot;, time.time() - start_time, &quot;seconds&quot;)</pre><p>Using .block_until_ready() ensures that the computation completes before measuring the time, providing an accurate execution time.</p><h4>4. Compile the function with JIT to enable XLA Optimization</h4><p>By applying jax.jit to the function, we trigger <strong>XLA</strong> optimizations. <strong>XLA</strong> will fuse operations within the loop, reduce memory accesses, and create a single optimized computation graph that can be executed more efficiently.</p><pre># JIT-compile the function to enable XLA optimization<br>compiled_intensive_computation = jax.jit(intensive_computation)</pre><h4>5. Measure execution time with JIT (Using XLA)</h4><p>Finally, we execute the JIT-compiled function and measure the time. The JIT-compiled version should demonstrate faster execution due to <strong>XLA’</strong>s optimizations.</p><pre># Measure execution time with JIT (using XLA)<br>start_time = time.time()<br>result_jit = compiled_intensive_computation(x).block_until_ready()<br>print(&quot;Execution time with XLA (JIT):&quot;, time.time() - start_time, &quot;seconds&quot;)</pre><h4>6. Results</h4><p>In the result, we can clearly see how <strong>XLA</strong>, through jax.jit, provides substantial performance benefits for complex numerical computations in <strong>JAX</strong>.</p><pre>Execution time without XLA (no JIT): 5.552654981613159 seconds<br>Execution time with XLA (JIT): 1.3866047859191895 seconds</pre><h3>What is PJRT and how is connected to XLA and JAX?</h3><p><strong>PJRT</strong>, or Portable JIT Runtime, is a runtime interface designed to make it easier to run machine learning models on a variety of hardware platforms. While <strong>XLA</strong> focuses on optimizing individual operations within a computational graph and <strong>JAX</strong> provides high-level abstractions for defining and transforming those operations, <strong>PJRT</strong> plays a unique role by enabling these optimized computations to be executed seamlessly across different devices, such as CPUs, GPUs, and TPUs.</p><p>Here’s how the integration flows:</p><ul><li><strong>From JAX to XLA</strong>: When you write a function in <strong>JAX</strong> and decorate it with jax.jit, <strong>JAX</strong> transforms the function into a computational graph and sends it to <strong>XLA</strong> for optimization. Once optimized, <strong>PJRT</strong> takes over, deciding where and how to execute the compiled function.</li><li><strong>From XLA to Hardware via PJRT</strong>: Once <strong>XLA</strong> has compiled the code, <strong>PJRT</strong> manages the actual execution on the hardware, providing an efficient and hardware-agnostic runtime environment. <strong>PJRT</strong> interacts directly with the device’s APIs, such as CUDA for GPUs, cuDNN for NVIDIA’s deep learning libraries, and TPU libraries, to execute the code in a way that leverages the full capabilities of the hardware.</li><li><strong>Transition Between Devices</strong>: With <strong>PJRT</strong>, developers can write and test code on one device (e.g., a local CPU) and then deploy it on another (e.g., a TPU in the cloud) without changing the codebase. <strong>PJRT</strong> ensures that <strong>XLA</strong>-compiled <strong>JAX</strong> functions run optimally on any supported hardware by automatically selecting the best execution method for each device.</li></ul><h3>Running JAX with PJRT on macOS: CPU and GPU Examples</h3><p>Below, we’ll walk through how to set up and run the same function on both CPU and GPU, leveraging <strong>PJRT</strong> to handle the backend differences seamlessly.</p><h4>Step 1: Run the Example on CPU</h4><p>First, install JAX for CPU support only (without GPU):</p><pre>pip install jax jaxlib</pre><p>Then, run the following code to execute the function on the CPU.</p><p><a href="https://github.com/dmarro89/jax-xla-pjrt/blob/main/jax-pjrt-cpu.py">Here the full example</a>.</p><pre>import jax<br>import jax.numpy as jnp<br>import time<br>from computations import intensive_computation<br><br># Initialize a large random array<br>key = jax.random.PRNGKey(0)<br>x = jax.random.normal(key, (10000, 10000), dtype=jnp.float32)<br><br># JIT-compile the function for XLA optimization<br>compiled_fn = jax.jit(intensive_computation)<br>print(&quot;Devices available:&quot;, jax.devices()) <br># Measure execution time on CPU<br>cpu_device = jax.devices(&quot;cpu&quot;)[0]<br>with jax.default_device(cpu_device):<br>    start_time = time.time()<br>    result_cpu = compiled_fn(x).block_until_ready()  # Ensure synchronous execution<br>    print(&quot;Execution time on CPU:&quot;, time.time() - start_time, &quot;seconds&quot;)</pre><h4>Step 2: Run the Example on GPU with JAX-Metal</h4><p>To run the same function on a Metal-compatible GPU, first install <a href="https://developer.apple.com/metal/jax/"><strong>JAX-Metal</strong></a>, which includes <strong>PJRT</strong> support for the Metal backend:</p><pre>pip install jax-metal</pre><p>Now, you can run the following code to execute the function on the GPU.</p><p><a href="https://github.com/dmarro89/jax-xla-pjrt/blob/main/jax-pjrt-gpu-metal.py">Here the full example</a>.</p><pre>import jax<br>import jax.numpy as jnp<br>import time<br>from computations import intensive_computation<br><br># Initialize a large random array<br>key = jax.random.PRNGKey(0)<br>x = jax.random.normal(key, (10000, 10000), dtype=jnp.float32)<br><br># JIT-compile the function for XLA optimization<br>compiled_fn = jax.jit(intensive_computation)<br>print(&quot;Devices available:&quot;, jax.devices()) <br># Measure execution time on GPU<br>gpu_device = jax.devices(&quot;METAL&quot;)[0]<br>with jax.default_device(gpu_device):<br>    start_time = time.time()<br>    result_gpu = compiled_fn(x).block_until_ready()<br>    print(&quot;Execution time on GPU:&quot;, time.time() - start_time, &quot;seconds&quot;)</pre><h4>3. Results</h4><p>Here the results, first with the CPU and then with the GPU.</p><pre>Devices available: [CpuDevice(id=0)]<br>Execution time on CPU: 5.421982049942017 seconds</pre><pre>Devices available: [METAL(id=0)]<br>Execution time on GPU: 0.8059799671173096 seconds</pre><h4>What’s happening in the background with PJRT?</h4><p>PJRT manages device-specific optimizations and execution for both the CPU and GPU:</p><ul><li><strong>Device Discovery</strong>: When you call jax.devices(), <strong>PJRT</strong> detects all available devices (CPU, GPU, or TPU). In the first example, jax.devices() lists the CPU; in the second example, after installing JAX-Metal, <strong>PJRT</strong> detects the Metal-compatible GPU as &quot;METAL&quot;.</li><li><strong>Device Allocation</strong>: Using jax.default_device(cpu_device) and jax.default_device(gpu_device), <strong>PJRT</strong> controls where the computation is executed. For CPU, <strong>PJRT</strong> sends the function to run on the CPU core(s). For GPU, <strong>PJRT</strong> routes the computation to the Metal backend on the GPU.</li><li><strong>XLA Compilation</strong>: In both examples, jax.jit triggers <strong>XLA</strong> to compile intensive_computation. For the CPU, <strong>XLA</strong> optimizes the code for single-threaded or multi-threaded execution based on the CPU’s architecture. For the GPU, <strong>XLA</strong> and <strong>PJRT</strong> handle compilation specific to the Metal API, optimizing for parallel GPU execution and memory usage.</li></ul><h3>Conclusion</h3><p>It’s important to note that <strong>JAX</strong>’s support for Metal on macOS is still experimental. While <strong>JAX-Metal</strong> allows users on macOS to leverage the GPU, not all data types or operations are fully supported, and certain functionalities may not perform as expected. This can lead to slower performance for some operations, unexpected behavior, or missing features compared to more established backends like CUDA on NVIDIA GPUs.</p><p>The <strong>JAX</strong> team, along with Apple, is actively working to improve Metal support, which is likely to become more robust in future releases. As the <strong>JAX-Metal</strong> integration matures, it has the potential to become a powerful tool for high-performance computing on Apple hardware.</p><p>If you found this article helpful, please consider following me and checking out the code:</p><ul><li><a href="https://github.com/dmarro89/jax-xla-pjrt"><strong>Code</strong></a></li><li><a href="https://github.com/dmarro89"><strong>GitHub</strong></a></li><li><a href="https://www.linkedin.com/in/davideraffaelemarro/"><strong>LinkedIn</strong></a></li></ul><p>If you any doubt, please leave a comment! I’ll be happy to get your feedback and eventually answer to your questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a08f47455059" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/xla-and-pjrt-powering-up-your-machine-learning-a08f47455059">XLA and PJRT: powering up your Machine Learning</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Machine Learning in Go Struggles — and Why I’m Not Giving Up]]></title>
            <link>https://levelup.gitconnected.com/why-machine-learning-in-go-struggles-and-why-im-not-giving-up-46158452d0d0?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/46158452d0d0</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Fri, 01 Nov 2024 07:05:30 GMT</pubDate>
            <atom:updated>2024-11-24T22:15:52.252Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Machine Learning in Go Struggles — and Why I’m Not Giving Up</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-1XfDCmppeUR_DL1Z7mxw.png" /></figure><p>I recently started exploring machine learning (ML) in Go, curious about whether it could be a viable option for training and testing neural networks. However, after researching existing ML frameworks in Go, I found that many are outdated or have been abandoned. This led me to post a question on Reddit: <em>“</em><a href="https://www.reddit.com/r/golang/comments/1gfi902/are_golang_ml_frameworks_all_dead/"><em>Are Go ML frameworks all dead?</em></a><em>”.</em></p><p>The feedback from the Go community was insightful, highlighting several reasons why Go is rarely used for ML. In this article, I’ll share my findings and explain why I’m still interested in exploring Go’s potential in ML, with a particular focus on the promising GoMLX library.</p><p><strong>The Landscape </strong><br>My initial exploration led me to frameworks like <strong>Gorgonia</strong>, which, despite having potential, appears to be abandoned from months. This raised immediate concerns, as even libraries with initial traction seemed to lack ongoing support. Many responses on <a href="https://www.reddit.com/r/golang/comments/1bvy6at/is_gorgonia_dead/">Reddit</a> echoed my experience, with users noting that it’s difficult to find Go frameworks that are well-documented or regularly maintained. Given the pace of ML advancements, it’s clear that Go’s ML ecosystem is lagging compared to Python’s robust libraries.</p><p><strong>Why Doesn’t Go Thrive in ML?</strong><br>The responses to my question on Reddit shed light on the challenges of using Go for ML. Here are the main reasons discussed by the community:</p><ol><li><strong>Go’s Strengths in Tooling and Backend Services</strong><br>One of the most common points raised was that Go has found its niche in backend development, microservices, and tooling, rather than data science or ML. The language was designed with simplicity, concurrency, and reliability in mind, which makes it ideal for building APIs and handling backend logic efficiently. However, ML requires a different set of capabilities that Go’s standard library and core language features don’t necessarily support out of the box.</li><li><strong>Python’s Ecosystem and Network Effect</strong><br>Python has become the go-to language for ML, largely because of its extensive ecosystem. Many of the foundational ML libraries, like TensorFlow, PyTorch, and scikit-learn, are built on top of highly optimized C/C++ code. Python serves as a convenient wrapper, giving developers access to powerful libraries without requiring low-level programming skills. Additionally, tools like Jupyter Notebook facilitate an interactive, iterative approach to ML that Go doesn’t currently offer. The Reddit discussion emphasized that recreating Python’s ML ecosystem in Go would require duplicating a vast array of tools and resources — a daunting task that may not be worth the effort.</li><li><strong>Challenges with GPU Acceleration and CGO</strong><br>ML computations are resource-intensive and often rely on hardware acceleration through GPUs. Python can easily access GPU resources through libraries like CUDA, which are written in C/C++ and have well-established bindings. In contrast, Go’s garbage collection (GC) and limitations in calling C/C++ code through CGO introduce significant overhead. One user on Reddit pointed out that Go’s GC isn’t optimized for ML workloads, leading to inefficiencies when interfacing with low-level libraries, especially for tasks requiring intensive data processing or parallelism.</li><li><strong>Limitations in Expressiveness for ML</strong><br>Another drawback for Go in the ML space is its lack of support for certain language features that are essential in scientific computing, such as <strong>operator overloading</strong> and <strong>generics</strong> (only recently introduced). These features simplify matrix manipulations and tensor operations, which are fundamental in ML. Without them, Go developers must rely on verbose syntax and workarounds, making the language less expressive for the linear algebra operations that are central to ML algorithms.</li><li><strong>The Complexity of Developing Go-Native ML Libraries</strong><br>Some commenters suggested that writing ML libraries natively in Go could be theoretically possible but pointed out that it would be an enormous effort. The alternative — building wrappers around existing C/C++ ML libraries — presents additional challenges, as CGO introduces complexity and performance issues. Go was not designed for the type of low-level, memory-intensive calculations required in ML, and this creates friction when trying to build or adapt libraries to handle these tasks effectively.</li></ol><p><strong>Is There Hope for ML in Go?</strong><br>Despite these challenges, I’m not giving up on the idea of exploring ML in Go. In the Reddit thread, a few users mentioned <a href="https://github.com/gomlx/gomlx"><strong>GoMLX</strong></a> as a promising library that could fill some of these gaps.</p><p>GoMLX aims to provide a more modern and extensible ML framework in Go, potentially enabling the loading and execution of ONNX models and support for custom training. The developers behind GoMLX are working to integrate it with <strong>OpenXLA</strong> for acceleration, which could make it a competitive option if it continues to evolve.</p><p><strong>Conclusion</strong><br>While Go’s ML ecosystem is limited compared to Python, exploring it has given me a better understanding of the trade-offs involved. Go’s strengths in simplicity, concurrency, and backend infrastructure make it a fantastic language for building scalable systems, but those same design choices limit its utility in the ML field. Although it may never replace Python as the dominant ML language, I believe that Go could still have a niche role in specialized ML workflows, particularly in production environments where model inference and deployment matter more than training.</p><p>I’m excited to see where GoMLX goes and am committed to experimenting with it. I hope this article sheds light on the current state of ML in Go and encourages others to explore its potential, even if the path is less traveled.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46158452d0d0" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/why-machine-learning-in-go-struggles-and-why-im-not-giving-up-46158452d0d0">Why Machine Learning in Go Struggles — and Why I’m Not Giving Up</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authorization in Go with Casbin]]></title>
            <link>https://levelup.gitconnected.com/authorization-in-go-with-casbin-79c07fffce61?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/79c07fffce61</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Fri, 04 Oct 2024 13:56:46 GMT</pubDate>
            <atom:updated>2024-11-24T22:15:45.853Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VMtcMBrQ0Cfs8nb7Lp4Fpg.png" /></figure><h3>Introduction</h3><p>Authorization is a fundamental security mechanism in any application that controls what users can do once they are authenticated. While <a href="https://medium.com/@davideraffaele.marro/jwt-authentication-in-go-0a7e3eafe27c"><strong>authentication</strong></a> verifies the identity of a user, <strong>authorization</strong> determines their permissions and access to different resources. This article will explore the principles of authorization and show how <strong>Casbin</strong>, a flexible library for role-based access control (RBAC), can be implemented in Go, using examples from the <a href="https://github.com/dmarro89/dare-db"><em>dare-db</em></a> project.</p><h3>What is Casbin?</h3><p><a href="https://casbin.org/"><strong>Casbin</strong></a> is a library that uses a policy-based approach to manage both RBAC and ABAC. With Casbin, you define authorization logic using policies written in a simple configuration file. The core of Casbin’s authorization system revolves around matching <strong>requests</strong> to <strong>policies</strong>.</p><p>Casbin’s core components:</p><ol><li><strong>Subjects (sub)</strong>: The user or role making the request.</li><li><strong>Objects (obj)</strong>: The resource the subject is trying to access.</li><li><strong>Actions (act)</strong>: The operation the subject wants to perform (e.g., GET, POST, DELETE).</li></ol><p>Policies are stored in a configuration file and Casbin <strong>enforces</strong> them by matching requests (sub, obj, act) to the defined policies. If a request matches a policy, the action is allowed.</p><h3>Casbin Model and Policy File Structure</h3><p>In Casbin, there are two primary configuration files you need to set up:</p><ol><li><strong>Model file (</strong><a href="https://github.com/dmarro89/dare-db/blob/main/auth/rbac_model.conf"><strong>rbac_model.conf</strong></a><strong>)</strong>: Defines the structure of how requests, policies, and role relationships are handled.</li><li><strong>Policy file (</strong><strong>rbac_policy.csv)</strong>: Defines the actual permissions that map roles to specific actions on resources.</li></ol><h4>Model File: <a href="https://github.com/dmarro89/dare-db/blob/main/auth/rbac_model.conf">rbac_model.conf</a></h4><p>The <strong>model file</strong> defines the core concepts of Casbin’s rule enforcement. It specifies:</p><ul><li><strong>Requests</strong>: What the incoming request looks like (who is trying to access what, and how).</li><li><strong>Policies</strong>: The structure that defines how access control is applied.</li><li><strong>Roles</strong>: How roles are defined and linked to users.</li><li><strong>Matchers</strong>: How to match an incoming request with a policy to determine if access is allowed.</li></ul><p>Here’s a breakdown of the model used:</p><pre>[request_definition]<br>r = sub, obj, act</pre><ul><li><strong>sub (subject)</strong>: The entity making the request (e.g., a user or a role).</li><li><strong>obj (object)</strong>: The resource being accessed (e.g., a file, a webpage).</li><li><strong>act (action)</strong>: The operation being requested (e.g., GET, POST, etc.).</li></ul><pre>[policy_definition]<br>p = sub, obj, act</pre><ul><li>This defines how the policy is structured, specifying which subject can perform which actions on which objects.</li></ul><pre>[role_definition]<br>g = _, _</pre><ul><li>This defines role inheritance. If a user is assigned a role, they inherit the permissions of that role. For example, the role admin might have broader permissions than user.</li></ul><pre>[policy_effect]<br>e = some(where (p.eft == allow))</pre><ul><li>Specifies that if at least one matching policy allows the action, the request is granted.</li></ul><pre>[matchers]<br>m = g(r.sub, p.sub) &amp;&amp; (p.obj == &quot;*&quot; || keyMatch(r.obj, p.obj)) &amp;&amp; regexMatch(r.act, p.act)</pre><ul><li><strong>g(r.sub, p.sub)</strong>: Checks if the subject (user) belongs to the role defined in the policy.</li><li><strong>keyMatch(r.obj, p.obj)</strong>: Allows the object in the request to match policies using wildcards (*).</li><li><strong>regexMatch(r.act, p.act)</strong>: Allows action matching using regular expressions, which is useful for complex matching scenarios like HTTP methods.</li></ul><h4>Policy File: <a href="https://github.com/dmarro89/dare-db/blob/main/auth/rbac_policy.csv">rbac_policy.csv</a></h4><p>The <strong>policy file</strong> is where you define the specific rules for which roles or users are allowed to access which resources and perform what actions. Each line in this file represents a policy, formatted as:</p><p>p, subject, object, action</p><p>Here’s an example from <em>dare-db</em> <a href="https://github.com/dmarro89/dare-db/blob/main/auth/rbac_policy.csv">policy file</a>:</p><pre>p, admin, *, GET<br>p, admin, *, POST<br>p, admin, *, PUT<br>p, admin, *, DELETE</pre><ul><li><strong>admin</strong>: The role that’s granted access.</li><li><strong>*</strong>: Wildcard for any resource.</li><li><strong>GET, POST, PUT, DELETE</strong>: The HTTP methods that the admin role is allowed to use.</li></ul><p>In this case, the admin role is allowed to access all resources (*) and perform all standard HTTP actions (GET, POST, PUT, DELETE).</p><h3>How the Casbin Authorization Code Works</h3><p>Now that the model and policy structures are set, let’s explain the <a href="https://github.com/dmarro89/dare-db/blob/main/auth/authorizer.go">Go code</a> used to implement the authorization process.</p><p>The code has been written for the <a href="https://github.com/dmarro89/dare-db"><em>dare-db</em></a> project in the <a href="https://github.com/dmarro89/dare-db/blob/main/auth/authorizer.go">authorizer.go</a> file.</p><pre>type CasbinAuth struct {<br>	users    Users<br>	enforcer *casbin.Enforcer<br>	logger   logger.Logger<br>}</pre><ul><li><strong>CasbinAuth</strong>: This struct encapsulates all the key elements needed for authorization: the users, Casbin’s enforcer (which processes the authorization logic), and a logger for tracking permission checks.</li></ul><h4>NewCasbinAuth</h4><pre>func NewCasbinAuth(modelPath, policyPath string, users Users) *CasbinAuth {<br>	if users == nil {<br>		users = Users{GUEST_USER: {Roles: []string{GUEST_ROLE}}}<br>	}<br>	enforcer, err := casbin.NewEnforcerSafe(modelPath, policyPath)<br>	if err != nil {<br>		panic(fmt.Sprintf(&quot;Failed to create Casbin enforcer: %v&quot;, err))<br>	}<br>	return &amp;CasbinAuth{<br>		users:    users,<br>		enforcer: enforcer,<br>		logger:   logger.NewDareLogger(),<br>	}<br>}</pre><ul><li><strong>NewCasbinAuth</strong>: This function initializes the Casbin enforcer using the model and policy files, creating the core of the authorization system. If no users are passed in, a default guest user with limited roles is created.</li><li><strong>Enforcer</strong>: The Casbin enforcer is responsible for evaluating incoming requests (from users) against the authorization policies. It loads both the model and policy files to manage this.</li></ul><h4>HasPermission</h4><pre>func (a *CasbinAuth) HasPermission(userID, action, asset string) bool {<br> user, ok := a.users[userID]<br> if !ok {<br>  a.logger.Error(&quot;Unknown user:&quot;, userID)<br>  return false<br> }<br><br> for _, role := range user.Roles {<br>  if a.enforcer.Enforce(role, asset, action) {<br>   a.logger.Info(fmt.Sprintf(&quot;User &#39;%s&#39; is allowed to &#39;%s&#39; resource &#39;%s&#39;&quot;, userID, action, asset))<br>   return true<br>  }<br> }<br><br> a.logger.Info(fmt.Sprintf(&quot;User %s is not allowed to %s resource %s&quot;, userID, action, asset))<br> return false<br>}</pre><p><strong>HasPermission</strong> is the core method that checks if a given user has the permission to perform a specific action on a resource.</p><ul><li>First, the method fetches the user’s details using the userID. If the user doesn’t exist, it logs an error and returns false.</li><li>If the user exists, it checks all the roles the user has and evaluates whether the role allows the requested action on the resource using the Casbin enforcer.</li><li>If a matching role allows the action, it returns true, otherwise, it denies access (false).</li></ul><p>This method effectively enforces the role-based policies defined in your policy file, ensuring that only users with the appropriate roles can perform certain actions.</p><h4>Integrating Casbin Authorization into Middleware</h4><p>In a real-world scenario, the Casbin authorization logic can be integrated into HTTP middleware to automatically authorize incoming requests based on roles and policies.</p><p>Here’s an example of how you can integrate the Casbin authorization into the HTTP middleware of <a href="https://github.com/dmarro89/dare-db"><em>dare-db</em></a>:</p><pre>func (middleware *DareMiddleware) HandleFunc(next http.HandlerFunc) http.HandlerFunc {<br> return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {<br>  tokenStr := r.Header.Get(&quot;Authorization&quot;)<br>  if tokenStr == &quot;&quot; {<br>   middleware.logger.Info(&quot;Missing authorization token&quot;)<br>   http.Error(w, &quot;Unauthorized: missing authorization token&quot;, http.StatusUnauthorized)<br>   return<br>  }<br><br>  username, err := middleware.authenticator.VerifyToken(tokenStr)<br>  if err != nil {<br>   middleware.logger.Error(fmt.Sprintf(&quot;Invalid authentication token: %v&quot;, err))<br>   http.Error(w, &quot;Unauthorized: invalid authentication token&quot;, http.StatusUnauthorized)<br>  }<br><br>  asset := middleware.extractAssetFromPath(r.URL.Path)<br><br>  middleware.logger.Info(fmt.Sprintf(&quot;User %s is requesting %s resource %s&quot;, username, r.Method, asset))<br>  if !middleware.authorizer.HasPermission(username, r.Method, asset) {<br>   middleware.logger.Info(fmt.Sprintf(&quot;User %s is not allowed to %s resource %s&quot;, username, r.Method, asset))<br>   http.Error(w, &quot;Forbidden: you do not have permission to access this resource&quot;, http.StatusForbidden)<br>   return<br>  }<br><br>  next(w, r)<br> })<br>}</pre><ul><li><strong>Extract User Info</strong>: It attempts to extract the user’s ID from the request headers (for example, from an authentication token like a JWT).</li><li><strong>Check Permissions</strong>: The middleware checks if the user has the necessary permission to perform the requested action on the resource.</li><li><strong>Allow or Deny Access</strong>: If the user is authorized, the request is passed to the next handler. If not, a “403 Forbidden” response is returned.</li></ul><h4>Example Usage in HTTP Server</h4><pre>import (<br> &quot;log&quot;<br> &quot;net/http&quot;<br> &quot;github.com/dmarro89/dare-db/auth&quot;<br> &quot;github.com/dmarro89/dare-db/middleware&quot;<br>)<br><br>func main() {<br> // Initialize Casbin authorization<br> auth := auth.NewCasbinAuth(&quot;/path/to/model&quot;, &quot;path/to/policy&quot;, []{&quot;admin&quot;: {Roles: []{&quot;admin&quot;}})<br> // Create a sample handler<br> sampleHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {<br>  w.Write([]byte(&quot;Authorized request!&quot;))<br> })<br> // Wrap the sample handler with the AuthorizationMiddleware<br> http.Handle(&quot;/some/resource&quot;, middleware.AuthorizationMiddleware(auth)(sampleHandler))<br> // Start the server<br> log.Fatal(http.ListenAndServe(&quot;:8080&quot;, nil))<br>}</pre><p>In this example the middleware checks each request before passing it to the handler, ensuring that only users with the correct permissions can access the /some/resource endpoint.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79c07fffce61" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/authorization-in-go-with-casbin-79c07fffce61">Authorization in Go with Casbin</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JWT Authentication in Go]]></title>
            <link>https://levelup.gitconnected.com/jwt-authentication-in-go-0a7e3eafe27c?source=rss-3ca6d97d03d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a7e3eafe27c</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Davide Marro]]></dc:creator>
            <pubDate>Fri, 20 Sep 2024 07:09:47 GMT</pubDate>
            <atom:updated>2024-11-24T22:15:42.043Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JeRC4_ORebsib02bnLvzIQ.png" /></figure><p>In today’s world of web development, protecting your APIs with proper authentication is crucial. One common approach is using JWT (JSON Web Tokens), which provides a compact and stateless way to secure your endpoints. In this article, I’ll walk you through implementing JWT-based authorization using a simple Go HTTP server. This example is based on my personal project, <a href="https://github.com/dmarro89/dare-db">dare-db</a>, where I created a custom authentication system.</p><blockquote><strong>dare-db</strong> is an in-memory database written in Go, designed to be lightweight and efficient. It features a <a href="https://medium.com/@davideraffaele.marro/redis-hash-tables-implementing-efficient-hash-tables-in-go-feff678dcb50">custom implementation of an hash table</a>, making it optimized for high-performance operations. In this project, I have also integrated authentication and authorization mechanisms, including JWT, to secure access to the data stored in the database.</blockquote><h3>What is JWT and how it is used?</h3><p>JWT is a compact and URL-safe token that is used to securely transmit information between two parties. It is often used for authentication purposes. The main advantage of JWT is that it’s <strong>stateless</strong>: once a token is generated, the server doesn’t need to store session information. Instead, all the necessary information is encoded within the token itself. A JWT consists of three parts:</p><ol><li><strong>Header</strong>: Contains metadata, such as the type of token (JWT) and the hashing algorithm.</li><li><strong>Payload (Claims)</strong>: Contains the claims, which are statements about the user or other data, like the user’s ID or role in the system.</li><li><strong>Signature</strong>: A cryptographic signature that ensures the token hasn’t been tampered with.</li></ol><p>Here’s a simple example of how JWT works:</p><ol><li>When a user logs in with their credentials, the server generates a JWT token.</li><li>This token is sent back to the client and stored (usually in local storage or as a cookie).</li><li>On each subsequent request, the client sends this token to the server in the Authorization header.</li><li>The server then verifies the token to ensure it is valid and allows or denies access to the requested resource.</li></ol><h3>Using JWT in dare-db</h3><p>In <a href="https://github.com/dmarro89/dare-db"><strong>dare-db</strong></a>, the authentication logic is centered around JWT. Let’s take a look at how you can implement a similar solution.</p><blockquote>In this implementation, we use the <strong>github.com/golang-jwt/jwt/v5</strong> library to handle the creation and management of JWT tokens. This library is a widely adopted Go package for working with JWTs and offers robust functionality for both generating and validating tokens.</blockquote><p>The <a href="https://github.com/dmarro89/dare-db/blob/main/auth/authenticator.go">authenticator.go</a> file of the <a href="https://github.com/dmarro89/dare-db"><strong>dare-db</strong></a> project, generates and validates JWT tokens. Here’s how it works:</p><h4>Generating JWT Tokens</h4><p>To generate a JWT token, we create a function that accepts a username and returns a signed token string. Here’s the code snippet from <a href="https://github.com/dmarro89/dare-db"><strong>dare-db</strong></a>:</p><pre>package auth<br><br>import (<br> &quot;time&quot;<br><br> &quot;github.com/golang-jwt/jwt/v5&quot;<br>)<br><br>type Authenticator interface {<br> GenerateToken(string) (string, error)<br> VerifyToken(token string) (string, error)<br>}<br><br>type JWTAutenticator struct {<br> usersStore *UserStore<br> jwtKey     []byte<br>}<br><br>type Claims struct {<br> Username string `json:&quot;username&quot;`<br> jwt.RegisteredClaims<br>}<br><br>func (jwtAuthenticator *JWTAutenticator) GenerateToken(username string) (string, error) {<br> expirationTime := time.Now().Add(5 * time.Minute)<br> claims := &amp;Claims{<br>  Username: username,<br>  RegisteredClaims: jwt.RegisteredClaims{<br>   ExpiresAt: jwt.NewNumericDate(expirationTime),<br>  },<br> }<br><br> token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)<br> return token.SignedString(jwtAuthenticator.jwtKey)<br>}</pre><p>The GenerateTokenfunction generates a JWT token for a given username, with an expiration time of 5 minutes. It creates a Claims object containing the Username and expiration time, signs the token using the HS256 algorithm, and returns the signed token string.</p><h4>Validating JWT Tokens</h4><p>Once the token is generated and sent to the client, we need a way to validate it on subsequent requests. That’s what VerifyToken does.</p><pre>func (jwtAuthenticator *JWTAutenticator) VerifyToken(tokenString string) (string, error) {<br> claims := &amp;Claims{}<br><br> token, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {<br>  return jwtAuthenticator.jwtKey, nil<br> })<br><br> if err != nil {<br>  return &quot;&quot;, fmt.Errorf(&quot;failed to parse token: %w&quot;, err)<br> }<br><br> if !token.Valid {<br>  return &quot;&quot;, fmt.Errorf(&quot;invalid token&quot;)<br> }<br><br> userStore := jwtAuthenticator.usersStore<br><br> if userStore.tokens[claims.Username] != tokenString {<br>  return &quot;&quot;, fmt.Errorf(&quot;invalid token&quot;)<br> }<br><br> return claims.Username, nil<br>}</pre><p>This function performs the following steps to validate the token:</p><ol><li><strong>Parsing the Token</strong>: The jwt.ParseWithClaims function is used to parse the JWT string and populate the claims (Username and others).</li><li><strong>Validating the Signature</strong>: The token’s signature is verified using the secret key (jwtKey). If the signature is invalid, an error is returned.</li><li><strong>Checking the Token in the User Store</strong>: In addition to verifying the token’s signature, we check if the token stored for the user in the UserStore matches the one provided in the request. This ensures that the token hasn&#39;t been replaced or manipulated elsewhere.</li><li><strong>Returning the Username</strong>: If the token is valid, the function returns the Username claim. Otherwise, an error is returned.</li></ol><h4>How to Integrate into an HTTP Server</h4><p>Now that we have the JWT generation and validation functions, let’s look at how we can integrate JWT authentication into an HTTP server. In <a href="https://github.com/dmarro89/dare-db"><strong>dare-db</strong></a>, this logic is implemented in the HandlerLogin function, which is responsible for handling user login and issuing a JWT token upon successful authentication.</p><p>Here is the relevant code from <a href="https://github.com/dmarro89/dare-db/blob/main/server/dare-server.go">dare-server.go</a>:</p><pre>func (srv *DareServer) HandlerLogin(w http.ResponseWriter, r *http.Request) {<br> if r.Method != http.MethodPost {<br>  http.Error(w, &quot;Method not allowed&quot;, http.StatusMethodNotAllowed)<br>  return<br> }<br><br> username, password, ok := r.BasicAuth()<br> if !ok || !srv.userStore.ValidateCredentials(username, password) {<br>  http.Error(w, &quot;Unauthorized: missing or invalid credentials&quot;, http.StatusUnauthorized)<br>  return<br> }<br><br> authenticator := auth.NewJWTAutenticatorWithUsers(srv.userStore)<br> token, err := authenticator.GenerateToken(username)<br> if err != nil {<br>  http.Error(w, &quot;Failed to generate token&quot;, http.StatusInternalServerError)<br>  return<br> }<br> srv.userStore.SaveToken(username, token)<br><br> w.WriteHeader(http.StatusOK)<br> w.Header().Set(&quot;Content-Type&quot;, &quot;application/json&quot;)<br> err = json.NewEncoder(w).Encode(map[string]string{<br>  &quot;token&quot;: token,<br> })<br> if err != nil {<br>  http.Error(w, &quot;Failed to encode JSON&quot;, http.StatusInternalServerError)<br> }<br>}</pre><p>This function performs the following steps:</p><ul><li><strong>HTTP Method Check</strong>: The function only accepts POST requests. If any other HTTP method is used, it returns a 405 Method Not Allowed error.</li><li><strong>Basic Authentication</strong>: The handler extracts the username and password from the request’s basic authentication header and verifies the credentials against the stored values. If the credentials are missing or invalid, it responds with a 401 Unauthorized status.</li><li><strong>Token Generation</strong>: If the credentials are valid, the handler uses the JWTAuthenticator to generate a JWT token for the authenticated user.</li><li><strong>Saving the Token</strong>: The generated token is then saved in the userStore, associating it with the corresponding user.</li><li><strong>Returning the Token</strong>: Finally, the handler responds with the token in JSON format, setting the appropriate Content-Type header.</li></ul><h4>Securing Routes with JWT</h4><p>Now that we have the login mechanism, we can use the JWT token to protect other routes. In <a href="https://github.com/dmarro89/dare-db"><strong>dare-db</strong></a>, I’ve created a middleware that validates the JWT token before allowing access to a protected route:</p><pre>func (middleware *DareMiddleware) HandleFunc(next http.HandlerFunc) http.HandlerFunc {<br> return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {<br>  tokenStr := r.Header.Get(&quot;Authorization&quot;)<br>  if tokenStr == &quot;&quot; {<br>   middleware.logger.Info(&quot;Missing authorization token&quot;)<br>   http.Error(w, &quot;Unauthorized: missing authorization token&quot;, http.StatusUnauthorized)<br>   return<br>  }<br><br>  username, err := middleware.authenticator.VerifyToken(tokenStr)<br>  if err != nil {<br>   middleware.logger.Error(fmt.Sprintf(&quot;Invalid authorization token: %v&quot;, err))<br>   http.Error(w, &quot;Unauthorized: invalid authorization token&quot;, http.StatusUnauthorized)<br>  }<br><br>  next(w, r)<br> })<br>}</pre><ul><li><strong>Token Retrieval</strong>: The middleware retrieves the JWT token from the Authorization header of the HTTP request. If the token is missing, it returns a 401 Unauthorized error.</li><li><strong>Token Verification</strong>: The token is then verified using the VerifyToken function from the JWTAuthenticator. If the token is invalid, an error is logged, and a 401 Unauthorized response is returned.</li><li><strong>Proceeding with the Request</strong>: If the token is valid and the user has permission, the request is passed to the next handler in the chain.</li></ul><h3>Testing JWT</h3><p>Now that we’ve implemented JWT-based authentication in our HTTP server, let’s test the functionality using curl to simulate the client-server interaction. We’ll perform the following steps:</p><ol><li><strong>Login to obtain a JWT Token</strong></li><li><strong>Use the JWT Token to access protected routes</strong></li></ol><h4>1. Login to obtain a JWT Token</h4><p>First, we will log in by sending a POST request with the username and password using HTTP Basic Authentication. The server will respond with a JWT token.</p><pre>curl -X POST http://localhost:2605/login \<br>    --user username:password</pre><p>If the credentials are correct, you should receive a response similar to the following:</p><pre>{<br>    &quot;token&quot;: &quot;eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...&quot;<br>}</pre><p>This token will be used in the next step to authenticate further requests.</p><h4>2. Access a protected route using the JWT Token</h4><p>Once you have the token, you can use it to access protected routes. The token is included in the Authorization header.</p><p>For example, to access a protected resource like the get of a db key called “mykey”:</p><pre>curl -X GET http://localhost:2605/get/myKey \<br>    -H &quot;Authorization: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...&quot;</pre><p>If the token is valid, the server will allow access to the key, returning it to you.</p><pre>{<br>    &quot;myKey&quot;: &quot;myValue&quot;<br>}</pre><p>If the token is invalid or missing, you’ll receive an error message, such as:</p><pre>{<br>    &quot;error&quot;: &quot;Unauthorized: missing/invalid authorization token&quot;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a7e3eafe27c" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/jwt-authentication-in-go-0a7e3eafe27c">JWT Authentication in Go</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>