<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Minoca - Medium]]></title>
        <description><![CDATA[Official Minoca Blog - Medium]]></description>
        <link>https://medium.com/minoca?source=rss----5bb4d764b0ee---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Minoca - Medium</title>
            <link>https://medium.com/minoca?source=rss----5bb4d764b0ee---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:11:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/minoca" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Minoca OS 0.4: We love the eighties!]]></title>
            <link>https://medium.com/minoca/minoca-os-0-4-we-love-the-eighties-170a93112db1?source=rss----5bb4d764b0ee---4</link>
            <guid isPermaLink="false">https://medium.com/p/170a93112db1</guid>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Evan Green]]></dc:creator>
            <pubDate>Thu, 29 Jun 2017 17:10:38 GMT</pubDate>
            <atom:updated>2017-06-29T17:10:37.715Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HLgBPRcbTLIsC-0S3jzhRQ.png" /><figcaption>fceux running on the first Minoca OS X server</figcaption></figure><p>We’re pleased to announce the release of version 0.4 of Minoca OS. In the eight months since we released 0.3, our very first open source release, a lot has changed. Most notably Minoca is now sporting the <em>early</em> signs of a desktop. And of course no desktop would be complete without an application to run on it. So we’ve added <a href="http://www.fceux.com/web/home.html">fceux</a>, an NES emulator, which we used to test both graphics performance and sound. Hopefully you’ll be better at Super Mario Brothers than Chris, who I watched blatantly kill Mario probably upwards of 50 times in the name of testing sound. Here’s what, how, and why we did it.</p><p>For those who have never heard of us before, Minoca OS is a new operating system built completely from the ground up. It’s designed to be lean, portable, modular, and modern (NES emulator notwithstanding). We’re trying to do more with less, so devices with limited power and resource budgets can get full OS functionality with minimal overhead.</p><h3>What’s New</h3><p>Some new drivers in 0.4:</p><ul><li>AHCI (the controller usually behind your SATA port)</li><li>e1000 (Intel gigabit networking driver)</li><li>USB and PS/2 mouse</li><li>Intel HD Audio and Raspberry Pi PWM audio</li></ul><p>Some new third-party packages worth noting:</p><ul><li>Xorg-server</li><li>SDL</li><li>fceux (NES emulator)</li><li>miniDLNA</li><li>Mesa3D (software rendering)</li><li>GTK+</li></ul><p>You can grab the latest OS images on our <a href="https://www.minocacorp.com/download/">downloads</a> page. Bugs and feature requests can be filed on our <a href="https://github.com/minoca/os">GitHub</a> page. You can fire up X for yourself by running “opkg update; opkg install fceux xinit; startx”.</p><h3>Motivation</h3><p>First of all I’d like to say a huge thank you to everyone who tried out Minoca OS since we went open source back in November. An even bigger thank you goes to the intrepid folks who submitted bugs and pull requests on our GitHub page. Chris and I really appreciate that, please continue to do it!</p><p>One of the requests we seemed to get a lot was “where’s the GUI” or “can I run this graphical application XYZ?” When we launched 0.3 we had done no work towards a graphical environment, mostly citing the fact that accelerated graphics are hard. That’s still true, and we still don’t have accelerated graphics (remember these are the <em>beginnings</em> of a desktop), but we were curious to find out how far out we were from having some sort of basic graphical environment.</p><p>The goal here was not to put out 0.4 with the message of “behold, the glorious desktop GUI has arrived”, but rather to provide the elements and building blocks for people familiar with compiling software to potentially make progress on their own.</p><h3>All Roads Lead to X</h3><p>Our initial thought was actually to port over DirectFB, a Linux library for interacting directly with the frame buffer. In fact we went so far as to port DirectFB to Minoca. Things fell apart a bit when we went looking for applications built on top of DirectFB. There aren’t so many of those. Wayland/Weston looks super cool and new, but it also appeared to be pretty Linux-specific, so it didn’t seem like the ideal choice to start with. No, everything graphical seems to compile against X. The ancient terrifying medusa that is X. It was on the very bottom of my mental list of possible options, but it was becoming clear that it really was the <em>only</em> option.</p><p>X is not just one thing that you download and compile. It’s composed of over a hundred different packages. As developers of a new OS, we generally feel it’s a pretty good day if we’ve added one or two new packages. The idea of adding over 100 new packages even if they all ported perfectly seems daunting. And though we’ve worked hard to build a POSIX compatible C library, life has a way of not going that smoothly.</p><h3>Into the Abyss</h3><p>There’s really no way to check and see if you’re going to get stuck on a missing OS feature in package number 67 of 140, you just have to start building and hope the obstacles are surmountable. The chapter on X in the <a href="http://www.linuxfromscratch.org/blfs/view/svn/x/installing.html">Linux From Scratch</a> book is a precious map in an otherwise endless labyrinth. I have no idea how anyone built the X server before that book. Most of it isn’t super Linux-specific, with some light editing they could call it “Any OS From Scratch”.</p><p>It also helps that nearly everything is autoconfed. Not that I don’t struggle with autoconf and especially libtool, but I’d say autoconf and I have an “understanding” with each other. It’s the devil I know, at least.</p><p>The first batch of 25 or so packages contain mostly headers. A warmup round to get your fingers loose, I suppose. Then comes another batch of 30 or so library packages. To my surprise these seem to go by pretty smoothly, and even cross compile!</p><p>I hit my biggest obstacle with Mesa, the 3D graphics library responsible for libraries like libGL. Mesa was one of the few non-Xorg packages in the quest for the X server, and required some monkeying to get right. Supplying the right configure parameters for a non-Linux software-only pipeline that the X server was happy with took some trial and error. There were also a few OS-specific bits of code that needed the obligatory “and here’s Minoca, too” code snippets. The nastiest <a href="https://bugs.freedesktop.org/show_bug.cgi?id=35268">barb</a>, which I’d only discover later, was their forced use of the “initial-exec” thread-local storage model. I know this was done in the name of performance, but it leaves a bit of a mess for the OS designers, who either have to 1) accommodate it with a brittle “might work for Mesa but I hope no one else does this” kind of hack, or 2) force programs using libGL to load it on initial program load, which the X server wouldn’t have ordinarily done. We went for option 2, which leaves you feeling only slightly less dirty.</p><p>The remaining packages leading up to running an X server went surprisingly smoothly. My compliments to the Xorg developers. Xterm, a non-Xorg package, required a bit of work. There are some code bases that have a “shove your OS in here with ifdefs” vibe to them, xterm is one of those. It’s usually pretty easy to do, but it never feels very good.</p><h3>Minoca I/O</h3><p>In theory we had a running X server, but given that X couldn’t display video or receive input, there wasn’t much proof. I added Minoca framebuffer support, and wrote an input driver to allow X to gather keyboard and mouse events from the kernel. This also required actually adding mouse support to Minoca, including a USB HID parser. After some false starts, I was presented with this beauty:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*joQWIyQYfYtUFN74wRhpVA.png" /></figure><h3>To Mario</h3><p>Fceux took a little more time because it required GTK+, a GUI toolkit. But hey, after porting over roughly 140 packages, what’s five or six more. Eventually, here’s what I got:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qwP4wn89pdsCkFFxOMTKQ.png" /></figure><p>What fun! Of course, it’s fairly disappointing without sound. So Chris whipped up an OSS-ish audio framework and wrote a driver for the Intel HD Audio device, common in PCs. The Raspberry Pi 3 is our favorite ARM platform, so he also wrote a PWM audio driver for that.</p><p>The performance on a Raspberry Pi 3 isn’t too bad, Mario is quite playable despite the lack of accelerated graphics. I ported over <a href="https://github.com/gperftools/gperftools">gperftools</a> and took a quick look around at the performance of fceux and the X server. I was looking to see if we were spending too much time in the C library or in a system call, indicating we had done something boneheaded in a library function. As expected, most of the CPU time went towards copying pixels around, so I think the performance is about right for unaccelerated graphics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ylCUghlEskFOCIPZ4GJaZg.png" /><figcaption>The obligatory glxgears shot</figcaption></figure><h3>What’s Next</h3><p>Finally, the pinnacle of OS development, being able to emulate a video game console from the late 80s, with a window manager from the same time period. We’re all done here, right?</p><p>We’re looking at this release as a jumping off point. Doing the legwork in getting X running opens up new possibilities for Minoca OS. Having a modern desktop, or even a browser running now feels like it might be within reach. I haven’t looked into it at all, but I wonder how much of Chrome would compile now. Tackling a kernel display framework might be doable now that there’s something in user mode to consume it. Or maybe it’s time for a new lightweight GUI to go with a lightweight OS. Or heck, maybe we should just get more of <a href="https://www.libretro.com/">libretro</a> up and running.</p><p>There are a lot of avenues that open up, and we’d love help in exploring them. If you feel comfortable compiling open source packages and are willing to be an OS pioneer, grab your favorite package and see if it builds. If it works, we’ll add it to our <a href="https://gitlab.com/minoca/third-party">packages</a> repository. If you’re interested in working on the OS itself, check out our <a href="https://github.com/minoca/os/blob/master/docs/WISHLIST.md">wishlist</a>.</p><p>You can download the latest 0.4 images on our <a href="https://www.minocacorp.com/download/">download</a> page. Source code for the OS is available <a href="https://github.com/minoca/os">here</a>. Feel free to reach out to minoca-dev@googlegroups.com with questions. Newbies are definitely welcome. Or you can email me directly at evan@minocacorp.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=170a93112db1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/minoca/minoca-os-0-4-we-love-the-eighties-170a93112db1">Minoca OS 0.4: We love the eighties!</a> was originally published in <a href="https://medium.com/minoca">Minoca</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cache COW — a debugging adventure on Minoca OS]]></title>
            <link>https://medium.com/minoca/cache-cow-a-debugging-adventure-on-minoca-os-573a7bd0d5c9?source=rss----5bb4d764b0ee---4</link>
            <guid isPermaLink="false">https://medium.com/p/573a7bd0d5c9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[operating-systems]]></category>
            <dc:creator><![CDATA[Chris Stevens]]></dc:creator>
            <pubDate>Tue, 15 Nov 2016 18:27:22 GMT</pubDate>
            <atom:updated>2016-11-15T18:26:27.094Z</atom:updated>
            <content:encoded><![CDATA[<h4>Sharing my own copy-on-write story in light of the Dirty COW bug.</h4><p>What follows is the story of how I tracked down a particularly obscure bug in the early days of Minoca OS. Here goes.</p><h4>Catching the Bug</h4><p>In January 2015, the Texas Instruments <a href="http://pandaboard.org/">PandaBoard</a> was the only ARM platform we supported. And it occasionally failed to boot.</p><p>Back then, Minoca’s nightly build process ran on an x86 PC we found on the side of the road and the ARMv7 based PandaBoard. When all went well, Minoca OS would compile itself, install the natively built image to a second partition, reboot into that partition and run tests on the new build. But on some rare mornings, the “street PC” was up and running tests while the PandaBoard was hung. Without a debugger attached, there wasn’t much more to do than reboot.</p><p>Given the hang’s rarity, simply attaching a debugger to the PandaBoard every night would take too long to reach a diagnosis. This bug begged for a better way to be reproduced, but there were only two solid facts. The first was that it never hit on x86. In most cases, that means it’s cache related as x86 is cache coherent and ARM is not. The second was that it only hung on reboot and not in the middle of compiling the OS. Without enough details to craft a targeted repro, attaching a debugger to the PandaBoard and sticking it in a reboot loop seemed like a good place to start.</p><p>While not a very creative solution, it was worth a shot at gaining more information about the bug. And voila, in about fifteen minutes, there was some action:</p><pre>Error Code: CRASH_PAGE_FAULT (0x1)<br>Parameter 1: 0x00000000<br>Parameter 2: 0xbf8d46a7<br>Parameter 3: 0xbfbbbe48<br>Parameter 4: 0x00000001</pre><p>It crashed! Maybe this was the culprit. The routine that prints this message, KeCrashSystemEx, actually <a href="https://github.com/minoca/os/blob/6df799ff6bbfdea333485f5952d47a81b4c6fbd9/kernel/ke/crash.c#L160">spins forever</a> without a debugger attached, so this would have manifested as a hang during the nightly builds.</p><h4>Dissecting the Crash</h4><p>Analysis of the crash parameters indicated that the USB hub driver faulted attempting to access NULL (Parameter 1) while executing the instruction saved in Parameter 2.</p><pre>1 : kd&gt;u 0xbf8d46a7<br>usbhub!UsbHubpRemoveDevice+0x7:<br><strong>bf8d46a7 6808      ldr r0, [r1]</strong>               <strong>&lt;-- The crash is here!</strong><br>bf8d46a9 f7ff ef0e blx usbhub+0x4c8 [0xbf8d44c8]<br>bf8d46ad 6860      ldr r0, [r4, #4]<br>bf8d46af f7ff eeee blx usbhub+0x48c [0xbf8d448c]<br>bf8d46b3 6820      ldr r0, [r4]</pre><p>It was a load from the address stored in r1.</p><p>While it was tempting to investigate why r1 was NULL, the conundrum staring me in the face was why UsbHubpRemoveDevice was executing during boot! The only hub attached to the PandaBoard was internal and could not be removed. A NULL dereference did not seem like the real problem. The next step was to figure out what was calling UsbHubpRemoveDevice.</p><p>The trap frame (Parameter 3), which stores the register context at the time of a crash, was the best place to find that answer. The saved link register records the return address for the executing function and it pointed to an address within UsbHubDispatchStateChange. By inspecting the <a href="https://github.com/minoca/os/blob/6df799ff6bbfdea333485f5952d47a81b4c6fbd9/drivers/usb/usbhub/usbhub.c#L386">C code</a> for that routine, nothing seemed amiss. Device removal is a valid state change operation. The assembly, however, told a different story.</p><pre>1 : kd&gt;u 0xbf8d476b-0x4<br>usbhub!UsbHubDispatchStateChange+0xa3:<br>bf8d4767 f7ff ff8b bl usbhub!UsbHubpEnumerateChildren+0x1<br><strong>bf8d476b</strong> <strong>bdb0      pop {r4, r5, r7, pc}</strong>   <strong>&lt;-- Saved Link Register</strong><br>bf8d476d 6ce3      ldr r3, [r4, #76]<br>bf8d476f 2b02      cmp r3, #2<br>bf8d4771 d103      bne usbhub!UsbHubDispatchStateChange+0xb7<br>bf8d4773 4620      mov r0, r4<br>bf8d4775 4629      mov r1, r5<br>bf8d4777 f7ff ff93 bl usbhub!UsbHubpRemoveDevice+0x1 [0xbf8d46a1]<br><strong>bf8d477b bdb0      pop {r4, r5, r7, pc}</strong>   <strong>&lt;-- Expected Link Register</strong><br>bf8d477d 0134      movs r4, r6</pre><p>The 16 byte discrepancy between the saved link register and the expected link register for a call to UsbHubpRemoveDevice was enough evidence that the crash was not a simple case of an uninitialized pointer. But I wanted more confirmation that the link register was correct and the attempt to remove a hub during boot was bogus. Luckily, the trap frame also saves the stack pointer at the time of a crash.</p><p>The Minoca Debugger’s dps command can be used to dump memory, pointer by pointer, resolving each address to its corresponding symbol. This was the stack, snipped for brevity:</p><pre>1 : kd&gt;dps 0xbfbbbe98<br>bfbbbe98 ff800000 0xff800000<br>bfbbbe9c 80717eb0 0x80717eb0  <br>bfbbbea0 fffffff8 0xfffffff8<br><strong>bfbbbea4 bf8d476b usbhub!UsbHubDispatchStateChange+0xa7<br>...<br></strong>bfbbbeb0 bfbbbee4 0xbfbbbee4<br><strong>bfbbbeb4 bfa24dc7 kernel!IopCallDriver+0xa7<br>...<br></strong>bfbbbee0 bfbbbef4 0xbfbbbef4<br><strong>bfbbbee4 bfa24e7b kernel!IopPumpIrpThroughStack+0x23<br>...<br></strong>bfbbbef0 bfbbbf2c 0xbfbbbf2c<br><strong>bfbbbef4 bfa24ff3 kernel!IoSendSynchronousIrp+0xf7<br>...<br></strong>bfbbbf28 bfbbbf54 0xbfbbbf54<br><strong>bfbbbf2c bfa250e7 kernel!IopSendStateChangeIrp+0x33<br>...<br></strong>bfbbbf50 bfbbbf74 0xbfbbbf74<br><strong>bfbbbf54 bfa1a73f kernel!IopQueryChildren+0x23        &lt;-- Look here!<br>...<br></strong>bfbbbf70 bfbbbf84 0xbfbbbf84<br><strong>bfbbbf74 bfa1a801 kernel!IopProcessWorkEntry+0x21<br>...<br></strong>bfbbbf80 bfbbbfb4 0xbfbbbfb4<br><strong>bfbbbf84 bfa1a939 kernel!IopDeviceWorker+0x109<br>...<br></strong>bfbbbfb0 bfbbbfe4 0xbfbbbfe4<br><strong>bfbbbfb4 bfa152ad kernel!KepWorkerThread+0xad<br></strong>...<br>bfbbbfe0 bfbbbffc 0xbfbbbffc<br><strong>bfbbbfe4 bfa0dae7 kernel!PspKernelThreadStart+0x13</strong></pre><p>The presence of IopQueryChildren further up the stack was a great indication that the link register was correct and that UsbHubpEnumerateChildren actually got called before the processor ended up in UsbHubpRemoveDevice.</p><p>A quick inspection of the enumerate children routine’s <a href="https://github.com/minoca/os/blob/6df799ff6bbfdea333485f5952d47a81b4c6fbd9/drivers/usb/usbhub/usbhub.c#L610">C code</a> showed it to be quite simple. It called two functions and returned. But how far did it get? To answer that, I had to go to the assembly again.</p><pre>kd&gt;u UsbHubpEnumerateChildren<br>usbhub!UsbHubpEnumerateChildren+0x1:<br>bf8d4681 b598      push {r3, r4, r7, lr}<br>bf8d4683 af03      add r7, sp, #12<br>bf8d4685 4604      mov r4, r0<br>bf8d4687 6849      ldr r1, [r1, #4]<br>bf8d4689 f7ff ef06 blx usbhub+0x498<br>bf8d468d 4602      mov r2, r0<br>bf8d468f 4b03      ldr r3, usbhub!UsbHubpEnumerateChildren+0x1c bf8d4691 447b      add r3, pc<br>bf8d4693 6818      ldr r0, [r3]<br>bf8d4695 4621      mov r1, r4<br>bf8d4697 f7ff eef4 blx usbhub+0x480<br>bf8d469b bd98      pop {r3, r4, r7, pc}</pre><p>The stack proved that the first instruction executed. The saved link register from the trap frame was pushed on the stack along with three more values, which can be seen on the stack above. The second instruction added 12 bytes to the stack pointer and stored it in r7. My calculation of what should have been stored in r7 matched the value saved in the trap frame for that register. Enough evidence to convince me the add instruction had executed.</p><p>Then the trail ran cold. A move, a load, and a branch followed, but nothing in the stack or trap frame confirmed their execution. The branch seemed most interesting, but it just jumped to the <em>procedure linkage table</em> (PLT) for the USB hub driver:</p><pre>bf8d4498 e28fc600 add ip, pc, #0  ; 0x0<br>bf8d449c e28cca01 add ip, ip, #4096  ; 0x1000<br>bf8d44a0 e5bcf438 ldr pc, [ip, #+1080]! ; 0x438</pre><p>The PLT looked fine though. Each PLT entry calculates the address of a corresponding <em>global</em> <em>offsets table </em>(GOT) entry. The kernel’s image loader fills each GOT entry with the address of a routine that is implemented in another binary. In this case, the GOT entry in question stored the address of UsbHubQueryChildren — the routine that the branch instruction was attempting to execute. The second branch instruction in UsbHubpEnumerateChildren jumped to a different PLT entry, but its corresponding GOT entry correctly pointed to IoCompleteIrp.</p><p>This really was rock bottom. Had either of those two routines executed, even a single instruction, the stack would have told a different story.</p><h4>Educated Guess and Check</h4><p>So, to take a step back, I was looking at an ARM-only crash in driver code where the program counter did not line up with the link register and the stack trace left off around a branch to the PLT. Wanting to sanity check this, I let the reboot loop run again. Similar crash: a page fault on a bogus address. Similar dead end: an execution stack that failed in the USB hub driver.</p><p>Recalling the number one rule about ARM-only crashes, the next step was to try to reproduce the bug with the caches disabled. Given that the program counter was going off the deep end, the PandaBoard’s 32KB L1 instruction cache seemed like a good place to start. I removed bit 12 from <a href="https://github.com/minoca/os/blob/6df799ff6bbfdea333485f5952d47a81b4c6fbd9/include/minoca/kernel/arm.inc#L120">MMU_CONTROL_DEFAULT_VALUE</a>, which gets set in the <em>system control register</em> (SCTLR), rebuilt the image and fired off the reboot loop. If the bug disappeared, then Minoca’s memory manager or image loader was probably missing an instruction cache invalidation when loading driver code. The bug hit. And it also hit with the branch predictor and L2 caches disabled.</p><p>Unfortunately, I could not try it with the L1 data cache disabled. Not only would it have been painfully slow, but Minoca OS depends on the ldrex and strex instructions, which require an exclusive monitor. The PandaBoard only implements that in its L1 data cache.</p><p>With caching tests out of the way, the second rule of thumb for tracing weird bugs is to test on a single core. Multi-core introduces <em>inter-processor interrupts</em> (IPIs) for shooting down <em>translation lookaside buffer </em>(TLB) entries. Perhaps a stale TLB entry was causing problems. Alas, rebuilding the OS with <a href="https://github.com/minoca/os/blob/6df799ff6bbfdea333485f5952d47a81b4c6fbd9/kernel/hl/ipi.c#L86">HlRunSingleProcessor</a> set to TRUE still reproduced the bug.</p><p>Other hypotheses began to take form. Could the interrupt handler have been incorrectly restoring the pre-interrupt context, clobbering the program counter? No. That would have hit more often and in other code paths. Was there something wrong with this USB hub driver’s PLT? The disassembly looked correct and there was nothing special about the USB hub driver. Whatever explanations I could come up with were never as convincing as that number one rule — it had to be cache related.</p><h4>One Last Theory</h4><p>Not being able to boot Minoca OS with the L1 data cache disabled still bugged me. But if I couldn’t disable it, maybe I could be more aggressive about cleaning it. I decided to make each cache clean and invalidate request operate on the entire cache, not just one cache line. After modifying <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/mm/armv7/flush.c#L448">MmpCleanCacheRegion</a> to call <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/armv7/archsup.S#L68">ArCleanEntireCache</a> and <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/mm/armv7/flush.c#L393">MmpInvalidateInstructionCacheRegion</a> to call <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/armv7/commsup.S#L233">ArInvalidateInstructionCache</a>, I put the PandaBoard into the reboot loop. 12 hours later, I returned. No crash. The reboot loop had only been taking 15 minutes to reproduce the bug and now it was still rebooting after 12 hours. Cleaning and invalidating the entire caches all the time was not a solution, but it was progress!</p><p>How did these cache changes relate to the USB hub driver that was executing into the abyss? Well, if the driver’s .text section (i.e. its executable code) was read from disk into the data cache and never flushed, the instruction execution engine would fetch and execute whatever random bytes were sitting in RAM.</p><p>There was a lot going on in that last sentence, so let me explain. The ARMv7 architecture implements separate data and instruction caches. Normal reads and writes operate out of the data cache while instruction fetches read from the instruction cache. All cache misses are filled from RAM — that is, the instruction cache does <em>not </em>read from the data cache. When executable code is modified, the changes are stored in the data cache. Those changes will not be observed by instruction fetches until the data cache is flushed to RAM and the instruction cache is invalidated to remove stale entries. (If that didn’t quite make sense, there is a diagram below.)</p><p>Could this explain the similar, yet inconsistent outcomes of each crash — RAM had different bytes each time — and its rarity — most of the time the data cache got naturally flushed before the driver executed? It seemed plausible, but the PandaBoard’s SD driver used DMA. The .text section would have never been in the data cache unless the image loader modified the .text section, yet there were no text relocations in the image’s .dynamic section:</p><pre>Z:/src/armv7dbg/bin$ arm-none-minoca-readelf.exe -d usbhub.drv</pre><pre>Dynamic section at offset 0x834 contains 13 entries:<br>  Tag        Type                       Name/Value<br> 0x00000001 (NEEDED)                   Shared library: [kernel]<br> 0x00000001 (NEEDED)                   Shared library: [usbcore.drv]<br> 0x0000000e (SONAME)                   Library soname: [usbhub.drv]<br> 0x00000004 (HASH)                     0x94<br> 0x00000005 (STRTAB)                   0x2c0<br> 0x00000006 (SYMTAB)                   0x140<br> 0x0000000a (STRSZ)                    316 (bytes)<br> 0x0000000b (SYMENT)                   16 (bytes)<br> 0x00000003 (PLTGOT)                   0x18c4<br> 0x00000002 (PLTRELSZ)                 112 (bytes)<br> 0x00000014 (PLTREL)                   REL<br> 0x00000017 (JMPREL)                   0x3fc<br> 0x00000000 (NULL)                     0x0</pre><p>Before losing all hope on this theory, I figured it wouldn’t hurt to make sure nothing was modifying the USB hub driver’s code. The driver’s .text section had been consistently loading at the same address, so I put a check in <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/mm/fault.c#L262">MmHandleFault</a> to see if the system ever write-faulted on that address. Sure enough, it hit! The faulting address was owned by a privately mapped image section that was backed by the first page of the USB hub driver. The trap frame indicated that PspImMapImageSegment was <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/ps/psimag.c#L1631">zeroing</a> the slack space between image sections.</p><h4>Caches and Copy-on-write</h4><p>Following the write fault further, I observed it call MmpIsolateImageSection, allocate a private page for itself, and <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/mm/imgsec.c#L2390">copy</a> the image section’s existing contents to the private page. It was performing a copy-on-write! For those unfamiliar with the necessity of copy-on-write behavior, let me explain. Applications will often directly map files into memory (using mmap) to eliminate the system call overhead for I/O operations. The kernel loves this because it can dole out the same physical pages to all applications, saving memory. This gets tricky when an application writes to a virtual address that directly maps a file. Should the write modify the physical page shared by all applications? In most cases, the answer is no. The kernel traps the write attempt, allocates a new private page and copies the contents of the shared page over. The writing application can go on its way and the other applications are none the wiser.</p><p>This copy-on-write behavior is even more important when it comes to modifying drivers. The shared physical pages mapped into the driver’s image section are actually borrowed from the page cache. If the writes were allowed to operate on those shared pages, the changes would end up on the hard disk. Remember, the system was zeroing slack space in the driver. A bunch of zeros hitting the disk would have corrupted the binary. The copy-on-write was necessary and expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*AER3jP47XLt-2a9-Z1oO4Q.png" /></figure><p>That said, it was wrong. <a href="https://github.com/minoca/os/blob/0cb08c96150e70bb8710943e51259fc927550351/kernel/mm/virtual.c#L2675">MmpCopyPage</a> was caching the contents of the original physical address (arrow 1 above) as it copied them to data cache entries for the newly allocated physical address (arrow 2), but it was <em>not </em>cleaning the cache (arrow 3). This simple act of zeroing the slack space was triggering a copy-on-write and dragging the entire .text section over to a new physical page. Without a data cache clean, the instruction cache was filling its cache lines (arrow 4) with whatever random bytes were in the new physical page. The processor was glad to execute them and do odd things, like trying to remove an internal USB hub during boot.</p><p>Things were adding up. After inserting the below code into MmpCopyPage, I put the PandaBoard into its reboot loop one last time. No crashes! Bug fixed!</p><pre>if ((Section-&gt;Flags &amp; IMAGE_SECTION_EXECUTABLE) != 0) {<br>    MmpSyncSwapPage(ProcessorBlock-&gt;SwapPage, PageSize);<br>}</pre><p>This Cache COW bug — as I’m now affectionately calling it — was particularly memorable because of how hard it was to reproduce. Typically, I aim to isolate a bug with a targeted test application, but caches are fickle. Sometimes you just have to trust your gut and remember the number one rule about ARM-only bugs. The cache is king.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=573a7bd0d5c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/minoca/cache-cow-a-debugging-adventure-on-minoca-os-573a7bd0d5c9">Cache COW — a debugging adventure on Minoca OS</a> was originally published in <a href="https://medium.com/minoca">Minoca</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Minoca OS: A new open source operating system]]></title>
            <link>https://medium.com/minoca/minoca-os-a-new-open-source-operating-system-4bb7998df3a7?source=rss----5bb4d764b0ee---4</link>
            <guid isPermaLink="false">https://medium.com/p/4bb7998df3a7</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[internet-of-things]]></category>
            <dc:creator><![CDATA[Evan Green]]></dc:creator>
            <pubDate>Mon, 31 Oct 2016 17:05:55 GMT</pubDate>
            <atom:updated>2016-10-31T17:08:49.847Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we’re thrilled to announce that Minoca OS has gone open source. We are releasing the entirety of the Minoca OS source code under the GNU GPLv3. We’re excited to build a community of users and developers around this new operating system, and we need help. You can check out the source at <a href="https://github.com/minoca/os">https://github.com/minoca/os</a>. You can also check out our repository of third party source packages <a href="https://gitlab.com/minoca/third-party">here</a>. If you’re just looking to download the latest stable binaries of Minoca OS, head to the <a href="https://www.minocacorp.com/download">download page</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zt26QZia2Ec9BgtXPsONZQ.jpeg" /></figure><h3>What is Minoca OS?</h3><p>Minoca OS is a general purpose operating system written completely from the ground up. It’s intended for devices looking to conserve power, memory, and storage. It aims to be lean, maintainable, modular, and compatible with existing software.</p><p>In other words, it’s built for little devices that want a full-featured OS.</p><p>On the app side, we’ve got a package manager (opkg), and a growing suite of packages like Python, Ruby, Git, Lua, and Node. Under the hood, Minoca contains a powerful driver model between device drivers and the kernel. The idea is that drivers can be written in a forward compatible manner, so kernel level components can be upgraded without requiring a recompilation of all device drivers.</p><h3>Why?</h3><p>Minoca OS was written by two developers, Evan and Chris, over the course of a few years. We took a look at the operating systems out there, and realized it had been over 25 years since the major operating systems had been written. 25 years is a long time to accumulate baggage, not to mention the leaps and bounds by which the hardware has evolved during that time.</p><p>We wanted to see if with 25 years of hindsight and a clean slate we could create something interesting and unique in the operating systems space. We felt that the freedom to take a fresh look at the kernel-driver interface could lead to powerful new capabilities. We’re looking at areas like power management, serviceability, and resource isolation that are critical today but weren’t even on the radar twenty years ago. Can we achieve parity with what what operating systems are used for in today’s world, but with less code, and with fewer pain points? Can we do better? We’d like to try.</p><h3>How can I help?</h3><p>Right now our biggest problem with Minoca OS is that it’s simply not in widespread circulation. We’d love to have folks spin up an instance and take a look around. Which packages are we missing? Did it crash? Did a device you need not work? What do we need to do to make this system more usable?</p><p>What might you use a new lightweight OS for? A server? An embedded device? A home router? We’ve got some ideas, but we need to hear from others.</p><p>Our second biggest problem is that there are only two of us. The basics are working, but there’s a mountain of stuff to do, an endlessly long tail. If you’re interested in writing a driver, adding a feature, porting a package, fixing a bug, root-causing an issue, or participating in design discussions, we’d love the help. If you’re interested in integrating Minoca OS into your own project, we’d love to help!</p><h3>Getting in touch</h3><p>We’ve got a couple options if you’d like to get in contact with us. For general inquiries, email info@minocacorp.com. For patches, technical discussions, or community questions, email minoca-dev@googlegroups.com. Security issues go to security@minocacorp.com. And finally, if you just want to shoot the breeze, I can be reached at evan@minocacorp.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bb7998df3a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/minoca/minoca-os-a-new-open-source-operating-system-4bb7998df3a7">Minoca OS: A new open source operating system</a> was originally published in <a href="https://medium.com/minoca">Minoca</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>