<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jon Masters on Medium]]></title>
        <description><![CDATA[Stories by Jon Masters on Medium]]></description>
        <link>https://medium.com/@jon-masters?source=rss-47465f5ecafd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bu6QZg6Pd7bl9CDdmYfTlA.jpeg</url>
            <title>Stories by Jon Masters on Medium</title>
            <link>https://medium.com/@jon-masters?source=rss-47465f5ecafd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:13:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jon-masters/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Leaving Red Hat]]></title>
            <link>https://jon-masters.medium.com/leaving-red-hat-d8786a024a5d?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8786a024a5d</guid>
            <category><![CDATA[red-hat]]></category>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[codesign]]></category>
            <category><![CDATA[software]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Fri, 03 Sep 2021 13:34:43 GMT</pubDate>
            <atom:updated>2021-09-04T13:24:19.772Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETKOOh8vbqaPKU9Fe_WVwg.jpeg" /><figcaption>Red Hatters (and alumni) wear their Fedoras with pride</figcaption></figure><p>Here we are (again). Today is my last day at Red Hat for a second time. I leave behind several amazing teams spanning the gamut from Arm Architecture to Automotive and many things in between. I know that all of you are well positioned for success and although I will miss all of you, I also know that you will be fine without me. Keep doing awesome things dear friends :)</p><p>Red Hat is a special kind of place. There are few places where you can be surrounded by so many people who are passionate about Open Source. Red Hatters are driven by doing the right things for the communities that they participate in, and doing the right thing for Open Source, even if it were to ever go against any commercial interests. I am very grateful to have been able to spend so many years in the company of such passionate people and to have grown up with the growth of the company.</p><p>Speaking of passion, I have in recent years discovered that I am extremely passionate about hardware and software co-design, building optimal solutions at the intersection of the two. I left Red Hat once before to pursue an opportunity that involved blending hardware and software. I am a strong believer in the future being built upon amazing technologies that seamlessly mix hardware and software through a choreographed interplay of the two.</p><p>I am not yet ready to share where I am going, or what I will be doing when I get there, but I plan to live my dream, while making a dent in the universe.</p><p>So long, farewell, and see you on the internet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8786a024a5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exchange traded cloud compute]]></title>
            <link>https://jon-masters.medium.com/exchange-traded-cloud-compute-39cfaf9241b4?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/39cfaf9241b4</guid>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Mon, 18 Jan 2021 18:33:11 GMT</pubDate>
            <atom:updated>2021-01-18T18:33:11.617Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>WARNING</strong>: mentions a patent that I was recently granted</p><p>At the end of 2020, I received notice that a patent application I filed almost a decade ago (back in 2011) had been issued, on “Automated cost assessment of cloud computing resources”. The idea I had back then was that the first player to commoditize a “unit of compute” and figure out how to do exchange trading (including futures trading) would have a hugely disruptive edge.</p><p>The idea that occurred to me more than a decade ago was that the public cloud is actually extremely early in its journey. Today, there are several key players providing services, and they have tremendous power, but in the future, such services could become commoditized and the value would be in arbitraging those services (playing the cloud vendors off against each other).</p><p>Thus the value in the cloud of the future isn’t in providing the services themselves, but in providing the NYSE of cloud compute. The first player to provide this service will have a tremendous first mover advantage. What is needed is a standardized mechanism for trading cloud compute. Then, you will be able to trade based upon what it will cost to get a certain amount of work done when you need it, subject to whatever constraints you impose.</p><p>The rise of confidential computing was also something I foresaw as a key enabler of this progression. Technologies such as AMD’s SEV were obvious to me back then. In the future it will be possible to run workloads in geographic locations that are otherwise hostile (politically, or for other reasons) but that have cost advantages. It will also be key to determine when to migrate these workloads based upon the various attributes (e.g. security) of the workload.</p><p>It only took ten years for the USPTO to review what I filed, which is fair enough because it’s pretty all encompassing. But I like the outcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39cfaf9241b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pat Gelsinger’s return to Intel…]]></title>
            <link>https://jon-masters.medium.com/pat-gelsingers-return-to-intel-8e51beb8ee55?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e51beb8ee55</guid>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Sat, 16 Jan 2021 18:19:43 GMT</pubDate>
            <atom:updated>2021-01-17T01:57:19.124Z</atom:updated>
            <content:encoded><![CDATA[<p>A few days ago came the (not unexpected) news that Intel will have yet another new CEO, the third in three years, and the first in many years to come from an engineering background. This move represents a surprisingly good move for Intel as Pat is well regarded, and has a history leading microprocessor designs during his previous 30 years at Intel (prior to his leadership of VMWare).</p><p>In the semiconductor (and broader computing) industry, technical leadership is key. There is simply no alternative. You can’t financially engineer your way out of not having a solid technical grasp on the fundamentals. Well, you can do that, but not for very long if you want to be successful in the longer term. For this reason, I like Pat returning to Intel, but he has his work cut out because the world is rapidly shifting away from where it has been — general purpose systems — and toward a verticalized future built around tech silos.</p><p>This is actually the biggest problem Pat faces — that the world is changing. The fabs are a huge issue, and an urgent one at that, but even if that problem were solved tomorrow it wouldn’t offset the fact that the market is going to be fundamentally extremely different in the decade ahead than the decade past.</p><p>In the server space in particular, it’s no longer one in which a small number of big CPU vendors duke it out for the hearts and minds of consumers via OEM channels. A jingle and a marketing campaign might sell a lot of chips when you’re selling to consumers (who buy brands, not technology), but the future is one in which there are only 6 or 7 customers driving demand, and they are both extremely sophisticated AND more than capable of taking matters into their own hands, in-house. For the first time, they’re also far larger than any of the component suppliers from which they source their compute needs today.</p><p>The cloud folks all want to drive growth of the cloud, which is continuing to grow at an accelerated pace. By taking matters into their own hands, they can not only accelerate that growth, but also drive down economics, and create offerings more heavily optimized for their own needs. This also serves to create stickiness. The “lock in” of the next decade won’t be literally “your software only runs on my cloud” but more that it “runs better on my cloud”. Because I coupled a “right sized” design with the right accelerators and made it “just work”. Logically it looks like a VM or cloud function — or whatever — but underneath it might not even be a single physical machine. With the growth in interconnects (such as CXL), we can compose whatever you need dynamically across a datacenter when your app requests a particular service.</p><p>Simultaneously, computer architectures matter less than they did in the past. Once upon a time, you couldn’t have a new entry into the market because the amount of legacy software would absolutely kill you. But many of today’s best developers building the next generation of applications don’t even know what an architecture is, and they don’t care. They use higher level languages, and a DevOps mentality that leverages modern techniques, like containers to deploy scripts and micro-service apps. They don’t care about what chip it runs on.</p><p>Thus, we are entering a decade in which the market is ripe for cloud players to dominate the technical direction (which we are already seeing), and I expect this to accelerate rapidly. Amazon have shown what is possible with Graviton and there is no putting that genie back in the bottle. It’s not just about compute either. It’s about the kind of solutions you can build when you have an extremely technically sophisticated customer in the driving seat, building a solution that is focused on being “better”, not on simply more of the same old. And btw, winning marketing SPECint scores are irrelevant to those customers.</p><p>I believe innovation will only accelerate to the point where the economics so overwhelmingly favor the cloud that it is finally actually totally impossible to compete at scale with your own datacenters. And worse, the talent available to understand all of these pieces will continue to shrink over time. Many of us grew up building computers, visiting datacenters to get our first “colo machine” into a rack, and all of this fun. The next generation look at this as quaint. Like having a manual transmission in your car, or of driving a buggy.</p><p>And on that note, enthusiasts and gamers still love building their own computers, and they love spec numbers, etc. but the average developer is more interested in a laptop with all day battery life that gives them a good experience, and doesn’t heat up their lap in the process. They love the look of the Apple M1 Macs because they see a well-polished solution. And again, it’s a marriage of hardware and software — and acceleration. The compute is of course fascinating to some of us, but that’s actually the boring part.</p><p>So what would I do if I were Pat? I would think carefully about all of the above. The next decade isn’t going to be “x86 vs Arm”, or “x86 vs Arm vs RISC-V”. It isn’t going to be some motherboard with a chip and a bunch of PCIe slots you plug whatever adapter widget into. Nobody cares any more. Compute is a boring commodity. It’s something I can license and build for myself if I’m large enough (and many are). The future is about carefully marrying all of the different possibilities together into a whole that includes both compute, as well as acceleration, and leveraging hw/sw co-design.</p><p>Anything that makes this easier is a winning strategy. I would look to ways that help to accelerate this as a successful path forward. Personally, I am very very bullish on novel packaging techniques as a path rather than simply a better widget through an improved fab process. The first player to create a viable ecosystem based around standardized “pic ’n mix” chiplets you can glue together into a package is going to have tremendous leverage by the way. I can very clearly see a future in which the “CPU vendors” sell the clouds and OEMs chiplets that they integrate with accelerators into novel purpose fit solutions.</p><p>Finally, I would move beyond being defined by one architecture. The future isn’t going to be about x86. It may not be about Arm (although I think it will be), but it definitely won’t be x86. An agnostic viewpoint would be a better strategy. Give those customers what they want, and focus on integrating the pieces of an overall “XPU” solution together better than the next player.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e51beb8ee55" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A new direction — rejoining Red Hat]]></title>
            <link>https://jon-masters.medium.com/a-new-direction-rejoining-red-hat-32d42a0e87ee?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/32d42a0e87ee</guid>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Fri, 04 Sep 2020 22:07:34 GMT</pubDate>
            <atom:updated>2020-09-04T22:07:34.148Z</atom:updated>
            <content:encoded><![CDATA[<h3>A new direction — rejoining Red Hat</h3><p>Today is my last day at NUVIA. It’s been a great experience working with Gerard, Manu, John, and the whole team of Nuvians who are well on the way toward disrupting the datacenter. I am particularly grateful to Gerard for giving me the opportunity to build and lead the NUVIA software organization in its early days. Although I won’t be there, we have built an incredible team of talented people who are going to continue to do amazing things long after I am gone. I am proud of the work that we did together and will look back on this time with great fondness.</p><p>In a week or so from now, I will be rejoining Red Hat. A company that had been my home for almost 14 years, since even before moved to the US. When I left last year, I told my colleagues at the time that deciding to leave in order to pursue a passion in learning about silicon design was one of the toughest decisions I have ever made. And I really meant that. Red Hat isn’t just a company, it’s a family, and the love that I still have for that place permeates my soul. I know I am not alone in being someone who has left Red Hat and felt a profound sense of loss. The feeling quite literally filled my dreams.</p><p>In the time I was last at Red Hat, I taught myself everything I know about modern computer architecture while having so many fun adventures, whether in helping lay the groundwork for Arm server standards and Operating Systems to run on them, or in helping to mitigate gnarly industry-wide security vulnerabilities like Spectre and Meltdown. Through this journey, I became obsessed with what was really going on inside the microprocessors that we all rely upon. I used to joke that if I won the lottery I would pay Arm’s design team to let me intern in their group for just 6 months as a fly on the wall to learn all the things that I couldn’t learn from the literature.</p><p>Then along came NUVIA with a dream. A new way of thinking and a chance to learn about silicon from some of the best minds in the field. I had often said that I would have given almost anything to be at Transmeta, PA, DEC, or one of the other greats, even for a short time. And so I decided to enter a new and different world of silicon startups. Which felt like diving in the deep end, let me tell you that. While I had tinkered with FPGAs and written weekend Verilog in the past, I soon discovered so many awesome, yet foreign concepts to grasp quickly. I bought more books than usual, and did a lot of reading, all the while hopping regularly from coast to coast.</p><p>I brought with me my philosophy that there should never be “hardware people” and “software people”, only “people”, and we should all work together. When you design from the point of view of co-design you unlock so much more than you can ever get if the hardware and software are built in isolation. I used to say “the only barriers are those defined by the Arm architecture”, which is a nerdy way of saying that we should never have artificial walls that divide us. It is so fundamentally important to me that I plan to continue to espouse this way of thinking, forever.</p><p>Fast forward a year later and I’ve had many great adventures. I’ve learned more than I ever thought I would know about how microprocessors are really built. I’ve made some great friends at NUVIA and I’m looking forward to watching their journey unfold from afar. I wish them only the best, and I am convinced that with a team as good as they have, they will achieve amazing things in the years ahead. At the same time, I’ve come to realize how much I missed Red Hat. You never quite realize how deep that feeling goes until you leave. It’s time for me to go home.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32d42a0e87ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The importance of standards, on making Arm servers “boring”]]></title>
            <link>https://medium.com/silicon-reimagined/the-importance-of-standards-on-making-arm-servers-boring-d23c580f1ecc?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d23c580f1ecc</guid>
            <category><![CDATA[data-center]]></category>
            <category><![CDATA[servers]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[standardization]]></category>
            <category><![CDATA[arm]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 15:26:59 GMT</pubDate>
            <atom:updated>2020-06-05T15:26:59.310Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Importance of Standards, on Making Arm Servers “Boring”</h3><p>Author: Jon Masters, VP of Software, NUVIA</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*F-uFf-96jVD9ql6ADMCPrQ.jpeg" /></figure><p>Standards are a key part of our everyday lives. From waking up in the morning, until we go to bed at night, we continually benefit from the interoperability that comes from being able to take certain things for granted. We couldn’t live our modern lives without the benefits that arise from being able to plug things together and have them “just work”. Whether that’s literally plugging in an AC cord and knowing the power will work, or connecting computers together across a global network and knowing they will be able to communicate, standardization underpins it all.</p><p>The best standards are “boring”. They may (hopefully!) be quite rigorous, documented in giant tomes formed from dozens, hundreds, or even thousands of pages, often with references to many other specifications that are incorporated by reference, but successful standards result in that “just works” experience. Some of the best modern technology feels like magic because of the work that went into developing standards and specifications to ensure that all the components would fit together and get out of the way. And when technology gets out of the way, users benefit.</p><p>Take for example a morning cup of coffee (with apologies to tea drinkers). First, you add water to the machine. You do this using the faucet, a part of your household plumbing, itself a network formed from standardized piping defined by national codes of correct operation. After adding the water, you add coffee grounds (perhaps after grinding them yourself) to a standard-sized filter placed into the machine. While several sizes of filter exist, you have confidence that any of the desired size will fit into your coffee machine because they are all built to standard specifications.</p><p>Once the coffee is added to the machine, you press a button, and a few minutes later — presto! — a fresh pot of java fills the room with a fine aroma. During those few minutes, your coffee machine used electricity to heat an element under the pot and a bubble pump mechanism to cause the water to rise and drip onto the coffee grounds (these machines are pretty great because they do this <a href="https://www.youtube.com/watch?v=4j4Q_YBRJEI">without any moving parts</a>). That electricity — it was delivered to your home across a national or regional AC grid in which the phase is globally synced such that every one of the generators at each of the power stations was moving in time with every other one.</p><p>A morning cup of coffee requires the successful adoption and deployment of countless standards and specifications, all working together in hidden harmony, so that the end result is so fabulously boring. That “just works” experience of hitting a button and getting a cup of coffee at the end. This is what the user wants. They may be curious about how it works, and for that there is countless bedtime reading material available, but in the moment they want the technology to get out of the way and deliver an experience without wondering whether it’s going to work.</p><p>In the early days of Arm servers (nearly ten years ago), enthusiasts used technology designed for embedded use serving a single purpose to build machines that showcased what could be possible. Those machines were interesting trailblazing platforms that were very useful, but they were not standardized. They did not “just work” with all of the available software. They ran non-standard versions of Operating Systems that needed to be modified continuously with vendor-specific hacks. Users did not have that delightful “just works” coffee experience.</p><p>And then, we made it boring. Over the last ten years, we have come together as an industry to aggressively standardize what it means to be an “Arm server”. There are specifications galore, they have interesting sounding acronyms (SBSA, SBBR…), and there is now an industry-wide Server Ready certification program run by Arm to ensure compliance. With the reason behind this effort being to make things “just work”. Today, you can buy a compatible, certified, Arm server and have confidence that it will run the software that you are used to, with no hidden surprises.</p><p>We have come so far since those early days. Once you have a coffee machine that “just works”, you can focus on the taste and quality of the coffee, which is the part you as a customer really care about. There are many kinds of coffee on the market, and they will all brew a cup of java, but they are not all the same. And tastes differ between the many coffee drinkers on the planet, necessitating different roasts and customizations specific to a given market or geography.</p><p>It’s like this with servers. We build standards so that we can all run the same software and deliver a baseline experience to customers. We do this so that we can go above and beyond that minimum. We can deliver higher performance than the competition, we can strive to offer enhanced robustness and quality. We can open source the firmware so that you have confidence about what’s really running inside the machine. We can do all this, thanks to the power of standardization that underpins it all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d23c580f1ecc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/silicon-reimagined/the-importance-of-standards-on-making-arm-servers-boring-d23c580f1ecc">The importance of standards, on making Arm servers “boring”</a> was originally published in <a href="https://medium.com/silicon-reimagined">Silicon Reimagined</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debugging a 32-bit Fedora Arm builder issue]]></title>
            <link>https://jon-masters.medium.com/debugging-a-32-bit-fedora-arm-builder-issue-73295d7d673d?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/73295d7d673d</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[arm]]></category>
            <category><![CDATA[fedora]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Wed, 03 Jul 2019 23:37:15 GMT</pubDate>
            <atom:updated>2019-07-03T23:37:15.870Z</atom:updated>
            <content:encoded><![CDATA[<p>Fedora supports many different architectures, among them both the 64-bit, as well as the 32-bit Arm architectures. In Fedora, we call these by the names “aarch64” and “armv7hl” respectively. This post serves as a mini-writeup of a hard to debug problem with 32-bit builders, used to build packages, which in the case of Fedora run as guest virtual machines on 64-bit Armv8 hosts.</p><p>The Fedora Arm project began many years ago with efforts to support 32-bit generations of Arm hardware. In the early days, there were builds targeting the v5 32-bit architecture, then later this was replaced by the current minimum of Armv7 (technically the variant requiring the optional VFPv3 floating point hardware, and the updated associated ABI, collectively known as “armv7hl”). Armv8 introduces the nuanced notion of an execution state, of which it defines both “AArch32”, as well as “AArch64”. For the purposes of this discussion, you can think of AArch32 as being a continuation of Armv7.</p><p>For quite some time, 32-bit was the official target for Arm (and even before that, it was a “secondary” architecture thanks to the hard work of countless others who did the lifting). Indeed, even long after 64-bit hardware became widespread, 32-bit was the officially supported target (64-bit is now also long since officially supported), and for many reasons it remains an important and necessary target for a variety of users. Not the least of which is Fedora itself.</p><p>Builds in Fedora are always performed “on target”, meaning that Arm builds happen on Arm hardware. In the case of 32-bit, this used to be only early generation 32-bit servers, but later moved to 64-bit hosts running 32-bit guest virtual machines. One of the benefits of moving to 64-bit Arm server hosts was that one could have much more capable server-grade hardware, with large numbers of cores and memory available to 32-bit virtual machine builders, which can then be managed almost identically to those of other architectures (including “aarch64”, which also leverages VMs for builders).</p><p>In the case of the current generation of builders, the guest environment is running in AArch32 (backward compatible with Armv7) execution state, using the LPAE (Large Physical Address Extensions) added in Armv7 in order to support builders with greater than 32-bits of physical memory. As with the similar “PAE” approach used on 32-bit x86_64 systems to address greater than 4GB of physical memory, individual programs are still capped to using at most 4GB, but you can run many such programs on an LPAE system. This is useful when running builders that want to maximize available memory for builds.</p><p>Fast forward to May 2018 and the opening of this bug impacting the 32-bit Fedora Arm builders, which happened to be running a different host kernel:</p><p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1576593">1576593 - Fedorn 29 armv7 guests &quot;pause&quot; from time to time on rhel-alt 7.6 kernel-4.14.0-115.2.2.el7a.aarch64 kernel</a></p><p>The problem seemed to be divided into two different kinds of error:</p><ol><li>Guest processes would sometimes crash with an obscure error message (“Unhandled prefetch abort: implementation fault (lockdown abort) (0x234)”). The guest would continue running, aside from this.</li><li>The host hypervisor would occasionally “pause” a guest with a log in the host kernel (“load/store instruction decoding not implemented”).</li></ol><p>In both cases, builds were impacted because the processes running them would be stopped and have to be restarted. Over the past year, we’ve had various discussions on and off about this problem, but thanks to the hard work of various others (who have carefully managed what is running where, including precise kernel versions), it’s never really come to a head. But it was going to ultimately become a problem that needed to be addressed. Recently, I decided I really wanted to understand what was causing this to happen.</p><p>The first thing you want to do when debugging such an obscure bug is to find a way to reproduce it as reliably as possible. Eventually, thanks to the work of others, it was possible to determine that a simple loop running part of a kernel build would be enough to eventually trigger whatever was happening:</p><blockquote>$ while true; do make clean; make ARCH=arm KCONFIG_CONFIG=../../kernel-armv7hl-debug.config olddefconfig; done</blockquote><p>The more interesting situation appeared (to me) to be the case that the hypervisor (KVM) ended up “pause”ing the guest as a result of a “load/store instruction decoding not implemented” error. When this happens, we are fortunate that the guest VM is left in a state that we can attach various debug and diagnostic tooling to understand what it was trying to do at the time.</p><p>Running “virsh list” on the host hypervisor, one would see output similar to:</p><blockquote>1 fedora30-arm-1 paused</blockquote><p>This means that “fedora30-arm-1” is no longer executing but hasn’t quite been killed either. We can thus run a dump of its memory, again on the host:</p><blockquote>virsh dump — memory-only — format elf fedora30-arm-1 fedora30-arm-1.core</blockquote><p>The resulting dump file will contain <strong>both</strong> the full “physical” memory of the guest, as well as its active register file (although not nearly enough of that to be actually useful for debugging what comes next). Now, the host is a 64-bit environment, and to make life easier, I copied this dump over to another 32-bit guest for analysis. This way, I could run various 32-bit native tooling. Thus, we now have two guests and one host. I’ll call the guests “builder” (the one that was experiencing problems) and “debugger” (the one I run tools on).</p><p>On the debugger guest, I first dumped out the state of the machine registers used by the guest at the time that it was paused. This can be done with gdb:</p><p># gdb fedora30-arm-1.core</p><p>(gdb) info registers <br>r0 0xb6f2f640 3069376064<br>r1 0xb69b4138 3063628088<br>r2 0xb6f31000 3069382656<br>r3 0xb6a21958 3064076632<br>r4 0xb6f2f640 3069376064<br>r5 0x6ffffdff 1879047679<br>r6 0x6ffffeff 1879047935<br>r7 0xb69b4000 3063627776<br>r8 0xb6a22040 3064078400<br>r9 0xb6f2f79c 3069376412<br>r10 0xbec20574 3200386420<br>r11 0xbec204f4 3200386292<br>r12 0x6fffffff 1879048191<br>sp 0xbec203e0 0xbec203e0<br>lr 0xb6f0670c 3069208332<br>pc 0xb6f0b6f4 0xb6f0b6f4<br>cpsr 0xa0000010 2684354576</p><p>Here, we can see that the guest was running in userspace (“pc” is lower than 3GB KERNEL_OFFSET) at an address of 0xb6f0b6f4. Note that this address is not the same as would be reported in task_struct for the process running at the time because the guest kernel never saw a fault, it would have gone straight to the hypervisor (but more on that detail later on).</p><p>Next up, I installed the exact matching “kernel-debuginfo” package for the kernel that was running on the builder. I did this by manually downloading the associated RPMs, but you can also use yum-downloader or just a “dnf” command if it’s a recent kernel matching what you already have on board.</p><p>Either way, I installed the debuginfo, and its “common” dependency:</p><blockquote>rpm -ivh kernel-lpae-debuginfo-5.1.9–300.fc30.armv7hl.rpm</blockquote><p>Really all I wanted was the “vmlinux”. With that available, I could use Dave Anderson’s <strong>excellent</strong> “<a href="https://people.redhat.com/anderson/">crash</a>” utility to extract useful information:</p><blockquote># crash /usr/lib/debug/lib/modules/5.1.9–300.fc30.armv7hl+lpae/vmlinux fedora30-arm-1.core</blockquote><p>Once crash starts up, it drops you at a prompt from which you can use one of the various useful built-ins to display information about what the guest was doing when it rolled over. In this case, we’re going to use “ps” to see what the running set of tasks was. The output included:</p><blockquote>&gt; 725 716 0 da876900 RU 0.1 5968 972 make</blockquote><p>Now I can tell crash to switch to “make” (set -x 725) as the current task and dump some useful information about what it was doing. Among those are all of the current VM maps that were in place for the running binary:</p><blockquote>crash&gt; vm</blockquote><p>Included in the output is the mapping for the dynamic linker:</p><blockquote>da814000 b6f00000 b6f21000 875 /usr/lib/ld-2.29.so</blockquote><p>The first address is the “VMA” (Virtual Memory Area) for the memory defined by the region between the second and third entries (0xb6f00000–0xb6f21000). This means that we know the guest was running code located within the ld-2.29.so binary, and from subtracting the VMA from the pc, we can know that specifically it was at offset 0xb6f4 of the ld-2.29.so binary. Now we can of course load this library under a variety of tools, including gdb.</p><p>My favorite is to use objdump (I’m weird):</p><p># objdump -x -D ld-2.29.so | less</p><p>We see that 0xb6f4 is located within the _dl_setup_hash function:</p><p>0000b6e0 &lt;_dl_setup_hash&gt;:<br> b6e0: e5903150 ldr r3, [r0, #336] ; 0x150<br> b6e4: e3530000 cmp r3, #0<br> b6e8: 0a000013 beq b73c &lt;_dl_setup_hash+0x5c&gt;<br> b6ec: e5931004 ldr r1, [r3, #4]<br> b6f0: e92d4010 push {r4, lr}<br> b6f4: e5912000 ldr r2, [r1]</p><p>Perhaps more interesting, we can look at the preceding instructions to see what happened immediately before the offending load (ldr). We previously loaded a value into r1 that was 4 bytes offset from the value in r3. We can see that r3 contains 0xb6a21958, so the previously successful load performed was from virtual memory location 0xb6a2195c. Subsequently, we tried to load from virtual memory location 0xb69b4138 into r2 and fell over.</p><p>Now, we remember earlier that the reason the guest had “pause”d was that the host (hypervisor) was complaining about not being able to decode a load or store operation. There had been some speculation (prior to investigating the actual dump) that some kind of complex load/store instruction sequence was starting to appear in certain 32-bit binaries, and that this was causing the problem. Without getting too off into the weeds, suffice it to say that you can have load/store instructions in 32-bit Arm that are pretty fancy and nearly impossible to emulate when needed in hypervisors such as KVM.</p><p>But looking at the above load, it’s a very simple one. In fact, it’s even simpler than the succeeding load above it. And looking at the kernel code which outputs the “load/store instruction decoding not implemented” message, we see that it comes from virt/kvm/arm/mmio.c. Specifically from io_mem_abort, which is called by kvm_handle_guest_abort (in virt/kvm/arm/mmu.c) pretty much as a fall-through. We’ll come back to this later, but foreshadowing a little, this fall-through is unfortunate and results in a useless error message based upon assumptions that turn out to be incorrect.</p><p>Ok. So it’s not a weird load/store emulation is hard issue. But somehow we weren’t able to perform a load from the location nonetheless. Let’s try to read the value from within crash. Just trying to do the load with “rd” won’t work:</p><p>&gt; rd 0xb69b4138<br>rd: seek error: user virtual address: b69b4138 type: “32-bit UVADDR”</p><p>Probably because of the presence of LPAE, crash struggles to translate the guest virtual address into the right physical address, and to read that from the crash dump file. Never fear! We can do this manually!</p><p>Crash provides us with a useful accessor macro, “vtop”, which will perform a page table walk through a process’ virtual memory. Using vtop, we can try to translate 0xb69b4138 into the guest physical memory address contained in the dump file, which we can then directly read using the “rd” command. “vtop” makes an unfortunate mess of the LPAE translation, but it does dump out the correct page table entry (PTE), from which we can read the physical address for the page that contains the data that we want:</p><p>4200004387c1fdf 4387c1000 (PRESENT|DIRTY|YOUNG)</p><p>The guest physical address is thus 0x4387c1138 (adding in the lower 12 bits by taking them from the virtual address above). We know (from reading the guest virtual machine’s memory map) that it starts its view of physical addresses at 0x40000000. We also know that the crash dump file begins its memory dump at offset 0x234. We can find that out by running objdump:</p><p># objdump -x fedora30-arm-1.core</p><p>So for every guest virtual address, we can calculate the offset in the dump file using the following calculation (in a python terminal window):</p><p>&gt;&gt;&gt; print hex(guest_physical_address–0x40000000+0x234)</p><p>Thus the value we wanted to read in the original load was:</p><p>crash&gt; rd -f 0x3f87c136c<br>3f87c136c: 00000209</p><p>The guest should have loaded 0x209 into r2 but instead it triggered an abort into the hypervisor instead. At this point I started look at the page tables in use by the guest, to see if there might be some kind of protection issue that would mean I can read the memory dump, but the guest could not. Perhaps it could be as simple as a strange access protection issue, or a bad sequence of invalidation instructions in the guest had failed to properly invalidate a page table update, such that the guest page walker had an incoherent view.</p><p>I performed various “vtop” translations for succeeding and failing loads close to the one that aborted and looked at the pages. Nothing seemed too out of whack. My trusty copy of the 7500 page <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.subset.architecture.reference/index.html">Arm Architecture Reference Manual</a> came in handy more than once, especially section G5.5 “The VMSAv8–32 Long-descriptor translation table format”, which describes the paging tables used in 32-bit AArch32 execution state when LPAE addressing is in effect.</p><p>Then I began to wonder whether it was the hypervisor’s own memory translations that were at fault. In modern virtualization environments, guest virtual machines have their own “physical” view of memory, but it is quite disjoint from the host view of its real underlying physical memory. Instead, the guest translates its virtual addresses into what it thinks are “physical” addresses, but are in fact IPAs (Intermediate Physical Addresses). These are translated a second time by the hypervisor using its own page tables.</p><p>Think of it this way: the guest’s “physical” memory is simply a mapped region in the virtual memory of a host (qemu-kvm) process that is backing the KVM virtual machine instance. With this in mind, we can start to investigate the host hypervisor’s view of memory and its secondary translation tables.</p><p>As I mentioned before, whenever a KVM guest is running, it is backed by a “qemu-kvm” process running on the host. By taking a look at the running processes on the host (“ps fax”), we can see which one is backing our “pause”d virtual machine, using the ps command and a simple grep:</p><p>$ ps faxwww | grep fedora30-arm-1</p><p>6458 ? Sl 180:45 /usr/libexec/qemu-kvm -name guest=fedora30-arm-1,…</p><p>The output is certainly not as brief in reality. Next up, we can look at the mapped virtual memory for this qemu-kvm process. KVM virtual machine “RAM” is implemented in the form of one or more “ramblock” regions of virtual memory allocated by qemu using a regular “malloc” allocation. We can see what regions are mapped by using the /proc/PID/maps interface:</p><p>$ cat /proc/6458/maps</p><p>Scanning through the output, we see a large mapping at 0xfffba7e00000 equal in size to 16GB (or the size of the virtual machine’s physical memory):</p><p>fffba7e00000-ffffa7e00000 rw-p 00000000 00:00 0</p><p>Thus we can also read guest physical memory by accessing appropriate offsets within the qemu-kvm process’ ramblock region. For example, the previous guest physical address of 0x4387c1138 can be translated into the virtual memory mapping used by the hypervisor prior to its stage2 translation:</p><p>&gt;&gt;&gt; print hex(0x4387c1138–0x40000000+0xfffba7e00000)</p><p>We can run a second (“live”) “crash” process on the host hypervisor. If the appropriate debuginfo RPMs are installed, this is as simple as:</p><p># crash</p><p>Setting the current task to the “pause”d qemu-kvm process (with “set 6458”) we can begin to examine the state of the process from the point of view of the host. Sure, we could do the same thing with gdb but we’ll want some of the more advanced features in a few moments. An example of these is the ability to read (with “rd”) the guest memory from the hypervisor viewpoint:</p><p>crash&gt; rd 0xffffa05c1138</p><p>We can use the same “vtop” command to leverage the host’s page tables in order to translate this host virtual address into the physical address on the host side. This will show us the page table entries the host is using to perform the second address translation into the real physical address on the host.</p><p>At this point, it would be useful to understand more about KVM’s view of the world. The qemu-kvm process does provide a “monitor”, and you can talk to this using “virsh” on the command line, but the interface is pretty primitive and doesn’t provide any of the information that I wanted. It is, however, a useful thing to have such a monitor, and it is growing in capability all the time. It’s also possible that I missed the way one is “supposed to do this” :)</p><p>Anyway. We can gather still yet more information by walking through KVM’s internal hypervisor state. To do this, we must first remind ourselves that KVM uses a relatively simple control interface in order to communicate between qemu-kvm (the process that runs on the host) and the “kvm” kernel module that manages the actual virtual machine state. The API is defined in the kernel documentation, which says that we perform ioctl’s on <a href="https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt">/dev/kvm</a>.</p><p>Thus, in order to manage a running KVM virtual machine, the qemu-kvm process must have an active file handle upon which it is making ioctls. We can find this by using the “files” command in crash:</p><p>crash&gt; files</p><p>The output includes all files that are currently open by the qemu-kvm host process. Among them, is this file that it is using to make requests into the KVM kernel module to start hardware virtualization, and other purposes:</p><p>21 ffffeb0db06abf00 ffffeb0de549b6d0 ffffeb0de076b880 UNKN kvm-vcpu:0</p><p>We can use the “struct” command in crash to walk further into this open file pointer being using by qemu-kvm to talk to its hypervisor:</p><p>crash&gt; struct file ffffeb0db06abf00</p><p>&lt;lots of other output here&gt;</p><p>private_data = 0xffffeb0db06ef160,</p><p>Reading the KVM source code, specifically kvm_arch_vcpu_ioctl (virt/kvm/arm/arm.c) tells us that KVM stashes a pointer to a “struct kvm_vcpu” containing VCPU state in the private_data pointer of this open file. We can find the private_data pointer in the above output and dereference it:</p><p>struct kvm_vcpu -x 0xffffeb0db06ef160</p><p>We can use this to reconstruct and obtain any state we want from the KVM instance by reading the structure, paying particular attention to the “arch” sub-structure, and such sub-sub-structures as “fault”. It contains a copy of various registers used by the hypervisor to identify the reason for a fault. Of these registers, a few of them are particularly interesting:</p><p>esr_el2 = 0x92000086,<br>far_el2 = 0xb69b4138,<br>hpfar_el2 = 0x3deb90,</p><p>The first is the Exception Syndrome Register (ESR_EL2). It tells us information about why we aborted into the hypervisor (more on that later on). The second Faulting Address Register (FAR_EL2) tells us what guest address lead to the trap. Unsurprisingly, this is the offending address we looked at previously. Lastly, we have the Hypervisor (IPA) Faulting Address Register (HPFAR_EL2). This one contains the address in guest physical memory that caused the trap.</p><p>I spent many more hours than even might be inferred from the above manually walking through various parts of the guest and host memory. One of the things I did was to walk through the guest’s page tables for the currently running process within the guest at the time of the fault. I did a lot of this manually, by taking the page table base register for guest userspace, known as the Translation Table Base Register (TTBR0_EL1) and performing the dozen or so operations needed to walk down to the individual PTEs (Page Table Entries) that describe the guest physical address for a given virtual one.</p><p>Eventually, I realized that when we received these traps into the hypervisor, the HPFAR_EL2 register always contains an address that is close to that of the guest physical address of the last level PTE leaf needed to perform the translation from the guest FAR (faulting virtual address) into a guest IPA (guest physical address). The only difference was that the value within the HPFAR_EL2 register was truncated, missing the upper bits of the guest faulting IPA. 0x43deb9000 became 0x3deb9000, dropping the “4”.</p><p>Finally, I hypothesized that what was happening was that guests performing a stage 1 page table walk were hitting a faulting condition (e.g. host page backing the PTE needed an access bit update that is software managed in Armv8.0) during the walk which required a trap into the hypervisor in order to manage the page table update (e.g. access bit update). In that case only, there were occasions in which the value contained within HPFAR_EL2 might have been truncated, missing the top address bits added by the Arm LPAE.</p><p>My solution? In the case that we know we received a hypervisor abort due to a stage 1 translation (which is helpfully communicated to us in the ESR_EL2), we can reconstruct the value that should have been in the HPFAR_EL2 by manually walking the guest page tables and comparing the leaf nodes. If the leaf matches the faulting address minus the top bits, inject the correct address instead and allow the guest to continue its page walk, successfully this time.</p><p>We haven’t fully run this to ground yet, so it’s not clear whether there is a hardware errata, or some software issue we haven’t identified in Linux somewhere that is causing this odd truncation in certain unusual cases, and even if there were a hardware issue, there is hopefully a fairly straightforward workaround without needing to reconstruct the faulting address as above.</p><p>The purpose of this blog isn’t to point blame in any particular direction. Especially since we don’t know the exact circumstances of any bug or exactly how it is triggered. But I did want to write this up while it was still fresh, to aid those who may follow since it was a particularly interesting one in my mind.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73295d7d673d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Happy birthday, Red Hat Arm Team!]]></title>
            <link>https://jon-masters.medium.com/happy-birthday-red-hat-arm-team-4669fd2d17bb?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4669fd2d17bb</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[arm-server]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Fri, 01 Mar 2019 17:52:00 GMT</pubDate>
            <atom:updated>2019-03-01T17:52:00.834Z</atom:updated>
            <content:encoded><![CDATA[<p>8 years ago today, we started a little project to “ultimately support RHEL on ARM”. It started out as a skunkworks effort leveraging the great work that had already been started in the Fedora community around 32-bit Arm devices. 8 years later, we have a great story in both the Fedora and RHEL communities (and many other besides), and the future is strong. With the announcement of Amazon AWS A1 (Graviton) instances last fall, we enter a new phase in which the cloud adoption curve begins and Arm becomes mainstream.</p><p>Our goal was always to get to 64-bit as quickly as possible, and to focus exclusively on server use cases, not desktop, tablet, or mobile. We had a number of challenges to resolve along the way, all of which I articulated on day one, in the very first introduction I sent to the fledgling team.</p><p>These included:</p><ul><li>A single kernel image that can run on all 64-bit Arm servers</li><li>The need for a BIOS and suitable platform abstraction, including runtime services for installing the boot loader and kernel onto a system</li><li>Building natively on Arm servers as a first class citizen</li><li>Standardization and certification requirements</li></ul><p>My point is that we always knew exactly what it would take to make Arm servers a mainstream option, but it takes (many) years to shape the conversation and development in a manner suitable to the required enterprise and cloud use cases. It was always a ten year journey. At 8 years, we’re past all of the initial hurdles that we anticipated and headed for the home stretch.</p><p>It’s been the absolute most fun project I’ve been fortunate enough to help lead, and I’m looking forward to many more years of fun ahead.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4669fd2d17bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon AWS Graviton Processor in newly launched Amazon AWS A1 instances]]></title>
            <link>https://jon-masters.medium.com/amazon-aws-graviton-processor-in-newly-launched-amazon-aws-a1-instances-4e2414e27cb6?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e2414e27cb6</guid>
            <category><![CDATA[arm]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Tue, 27 Nov 2018 04:16:30 GMT</pubDate>
            <atom:updated>2018-11-27T04:16:30.295Z</atom:updated>
            <content:encoded><![CDATA[<p>I guess I can now admit to being at re:Invent in Las Vegas, and share a little bit about what I’ve been up to over the past little while. This is the first in a mini series, so stay tuned for more…</p><p>Today, Amazon announced their first Arm server offering. The new Amazon EC2 A1 instances provide a means for everyone to now share in the tremendous fun that is using and developing with Arm servers, which a few of us have been involved with for the past 8 years. I’ve been serving as technical lead on the Red Hat team that has enabled support for these new instances in RHEL7.6, creating various test AMIs and helping the regular teams that work on EC2 to get up to speed with Arm. It’s been fun, and the team has been absolutely amazing, completing the work in record time, with a quality product at the end of it all. This was made possible thanks to already having a RHEL offering for the Arm architecture, a labor of love over many years.</p><p>For the past 8 years or so, I’ve been Red Hat’s Chief Arm Architect. I co-created the Red Hat Arm server team back in 2011, alongside my trusty partner in crime Brendan. We had many fun adventures in the early days, and we were under no illusions that the journey would be long and arduous to get a new architecture into the mainstream. We always longed for a day like today, but we knew it would take determination and perseverance on the part of many involved for us to build the world as it would need to be in order for Arm to truly enter the mainstream. Today, Arm is ready to be a part of that mainstream of Cloud, and ready to help fuel the next wave of innovation that will continue to drive exciting solutions for end users and developers alike.</p><p>I remember with great fondness a rainy day toward the end of 2010. Standing in one of my exec’s offices, I held up a BeagleBoard and said that this would be a server within a few years. There were a few fits and starts, and some lessons along the way, but ultimately we built a new world of high performance 64-bit native Arm servers that should feel familiar to anyone using a mainstream server today. This is in part thanks to a heavy dose of industry standardization that was one of my personal contributions to the overall cause. I felt early on that Arm servers could only succeed if they had a common foundation in terms of functionality, similar to that seen on other architectures. And that is why Arm servers use ACPI, UEFI, and other existing standards — there was no benefit in re:inventing the wheel, and a lot to gain from adopting what worked well.</p><p>Standardization is why operating systems like RHEL, Fedora, CentOS, and others can “just work” on Arm server platforms such as those announced by Amazon today. In fact, if you didn’t look very closely, you might not at first even notice that you were using an Arm server. That was by design. And standardization will be key to the next wave during which many others building cloud offerings will (finally) realize the benefits that come from being able to build and deploy their own Arm server solutions. When you do, remember to give me a call early on for input. I’ve been there at the birth of almost every Arm server chip to date, and I am always willing to make myself available to help those entering this space to navigate the players and the landscape.</p><p>I’ve been playing with Amazon’s A1 instances for a while and I’m blown away by the quality of the engineering and the capabilities of the team, as I always am with Amazon. Stay tuned to hear more about my adventures with A1, and come find me at re:Invent if you’d like to talk about Arm servers, or running RHEL on Arm. It’s my first time here, but I’m hoping that it becomes a regular thing. I’m looking forward to seeing a few of you as the week progresses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e2414e27cb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speculative data load mitigation through register tagging and data provenance]]></title>
            <link>https://jon-masters.medium.com/speculative-data-load-mitigation-through-register-tagging-and-data-provenance-52abe24b719?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/52abe24b719</guid>
            <category><![CDATA[arm]]></category>
            <category><![CDATA[computer-architecture]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[meltdown]]></category>
            <category><![CDATA[respect]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Tue, 06 Mar 2018 13:44:32 GMT</pubDate>
            <atom:updated>2018-03-06T16:13:43.242Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern microarchitectural attacks often depend upon data value speculation using values passed between contexts through general purpose registers (GPRs). A value is passed from one privilege level (e.g. userspace, EL0) into a kernel (EL1), or from one sub-context of the same privilege level (e.g. within a JIT runtime). In either case, the registers can be automatically tagged to identify the provenance of the value and track it through the microarchitecture implementation, and in particular the speculative backend.</p><p>Modern machines are implemented to provide a certain number of generally usable registers (GPRs). These are used for interim calculations, often as part of a “load store architecture” in which values are loaded from memory, data is transformed, and results stored back to memory. Sophisticated contemporary computer architectures typically provide 31–32 GPRs and set few constraints upon their use, while less sophisticated legacy architectures usually provide fewer registers, and impose more constraints, but their function is similar.</p><p>These GPRs are not traditionally banked by exception level. That is to say that you get one flat file of registers (at the architecture level) that is shared by the different exception levels of privilege. An implementation may have a greater number of physical registers in its physical register file and use Out-of-Order computing techniques to map these onto the architectural register file, but there still remains no distinction between the differing exception levels.</p><p>As an example of contemporary use, suppose I have an ABI between userspace and kernel that implements a system call interface. Those values supplied by userspace are written into registers and then a system call instruction is performed to transition into kernel context. The registers are read by the kernel. No distinction is made between any of the registers (other than by software) as to who set them, and whether those values are trusted.</p><p>The problem comes when speculative computing backends are introduced. An implementation might aggressively speculate data loads based upon the values of GPRs, such as a load of a data value prior to performing a bounds check on another value. Since the backend has no way to know whether this speculation is safe, barrier instructions must be added to inform it not to perform subsequent loads speculatively prior to previous bounds checks.</p><p>Enter register tagging. Tagging the provenance of the registers allows for automatic restriction of speculation based upon those values.</p><p>This concept can be extended to same privilege level trivially by adding instructions or control registers to an architecture that allow the architecture to tag a subset of registers as being used by less privileged code (even within the same exception level), such as within a Java JIT or other runtime.</p><p>There are some problems with the idea as it stands. Chiefly, that of register spill and save/restore not conveying the tag information. There are a few potential ways to address this that I am currently working on. Thus, this is a work in progress, but I wanted to share this now to get people thinking. I will be writing this up in more detail and aim to turn it into a paper. If any of the architecture companies are interested in working with me on it, ping away.</p><p>Footnote: This idea occurred to me last year. Originally, it was going to be part of an (unrelated to security) patent filing, the purpose of which (as with my other microarchitectural filings) would have been solely to keep it out of the hands of certain others who might misuse it. But rather than file this one, I want to explicitly release it into the public domain as a useful idea. It needs work, there are holes (such as the register spill/restore), but we can hopefully refine this into something actually usable that lets us keep speculating away.</p><p>Addendum: When you write to memory (spill registers), store the tags in the D$. Then, when you reload, pull them from there. If you don’t have tags for registers to identify their provenance, default to disabling speculation. You can get creative by using e.g. a bloom filter to efficiently store the tag info.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52abe24b719" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On addressing #Meltdown and #Spectre in future silicon…]]></title>
            <link>https://jon-masters.medium.com/on-addressing-meltdown-and-spectre-in-future-silicon-bc9b15a7a52c?source=rss-47465f5ecafd------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc9b15a7a52c</guid>
            <category><![CDATA[meltdown]]></category>
            <category><![CDATA[respect]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[speculative-execution]]></category>
            <category><![CDATA[red-hat]]></category>
            <dc:creator><![CDATA[Jon Masters]]></dc:creator>
            <pubDate>Mon, 08 Jan 2018 00:36:29 GMT</pubDate>
            <atom:updated>2018-01-08T00:36:29.831Z</atom:updated>
            <content:encoded><![CDATA[<p>[ This is going to be part of a much longer blog post, with diagrams, soon, consider this a draft that might be useful but is going to be pages long later, and I am possibly going to edit as I re-read and improve the language ]</p><p>Last week, the world learned more broadly about a class of hardware security vulnerability known as a “side channel attack”. Three main variants (with some sub-variants, and other ongoing research) were disclosed by Google Project Zero, each relying fundamentally on the shared cache hierarchy within modern microprocessors, as well as a range of common performance optimizations.</p><p>One of those optimizations, called “speculative execution” has to do with extending “Out-of-Order” execution (invented by Robert Tomasulo in 1967, for which he ultimately won awards) to attempt to predict what an algorithm will do prior to having all of its data dependencies fully resolved. The key innovation relies upon turning in-order sequential machines as seen by programmers into “dataflow model” machines, as implemented by major silicon manufacturers. None of the modern high performance CPUs out there execute your program in any way resembling the order you actually wrote it.</p><p>Take branch prediction. When we hit an “if” statement in our program, we’re going to go in one of two possible different directions. One is “fall through”, the other is a branch to a different part of the program. We often don’t know ahead of time which way a branch is going to go because it hasn’t been fully “resolved” yet. Resolution of a branch involves evaluating both its condition, as well as loading any dependent data to perform that resolution.</p><p>Waiting for this resolution takes some time, especially if the branch condition variables aren’t in the fast CPU caches and need to be pulled in from memory (as is the case when I intentionally exploit this by performing a clflush against that condition variable in a “Meltdown” attack to force the machine to have to go perform this slow load from memory). This is time we don’t want to be sitting around twiddling our thumbs. So we don’t. We guess which way the branch is going to go and we begin to execute in that direction. If we’re right, great, we saved some time. But if we’re wrong, we need to do fixing — undo whatever it is that we performed “speculatively” in predicting the branch.</p><p>The good news for implementations is that Out-of-Order machines give us the mechanisms we need for such Speculative Execution. We have re-order buffers (and RATs, and other constructs depending upon your heritage) that separate the “architectural state” from the microarchitectural state of the machine. Until instructions complete (retire), and become part of the architectural state, they are recorded in internal structures. We can do some fancy optimizations involving swapping of internal structures that allow us to quickly commit results into architectural state, or undo speculated ops.</p><p>Thus we will store the speculative state of the machine without the conventional side effects that expose the speculation to the outside world. For example, all writes (stores) to memory are buffered and not actually written out to memory during speculation (which complicates load logic a little). This is something every high performance machine does. It works great, provided that the speculation remains a black box that is unobservable to the outside world, and provided that we can completely undo speculation.</p><p>Let’s see how this leads to “Meltdown”…</p><p>The problem with “Meltdown” is that this speculation black box is violated. It becomes violated because of an additional performance optimization that some companies added to their speculation process. In the case of those affected processors, loads being performed within a speculation “window” (a period of up to 192–224 instructions or even more) will have the usual permission checks to ensure that such access is allowed. But that permission check will happen in parallel with an attempt to load that data from the level 1 data cache (L1D). If the permission check later fails, this is recorded in the re-order buffer such that at any later attempt to retire the instruction, an exception is thrown. But the process of the data load is no longer fully atomic.</p><p>If we can disambiguate the load process and sneak in during the speculation window, performing some other activity based upon it that is observable to the outside, then we can break the black box. This is what “Meltdown” does. It exploits the fact that side channels allow us to infer whether an address is contained within the L1D cache of a processor simply by timing an access to that location. The longer the access, the further away in the cache hiearchy the address. And if we intentionally evict that address (clflush, carefully constructed array access, etc.) prior to reading it, we can guarantee it’s not in the cache. So, in Meltdown, we perform cache accesses based upon the value that we read speculatively out of the L1D, and then use those accesses to infer what the sensitive data was. Importantly, we don’t read that data directly.</p><p>Meltdown works because we do the privilege check in parallel, and also because the L1D cache is VIPT (Virtually Indexed, Physically Tagged) and thus can become populated with virtual addresses for which we have active live translations in our TLBs (Translation Lookaside Buffers) based upon the currently active page tables. We know, for example, that “prefetch” on x86 class cores will cause the TLB population (this is something Linus has ranted about before, when discussing NULL pointer prefetch) and on some cores it will also cause the load of the privileged data into the L1D cache as well.</p><p>To address “Meltdown” on affected cores is relatively trivial. Do the missing permission check. And don’t wait for instruction retirement. It’s the only way. And it’s the way that’s been in the academic literature for many years. The only reason not to do it this way is if, in the interests of “performance” you didn’t consider a cache side channel attack likely or plausible during that small (but ever increasing with CPU generations) speculation window.</p><p>The software mitigation is to ensure that addresses we don’t want userspace to read (such as kernel virtual addresses) can never be in the L1D at the same time that userspace is running, or that we don’t have live TLB entries for those addresses. Both parts are needed in order for the speculated load to occur because otherwise the core has to do a much slower load. It is possible on some cores that they will try to perform loads even for data not in the L1D, but few designers have yet gotten that crazy because of the amount of time required to wait for that load to come in. It could happen though if we ever get reorder windows that are large enough to keep that much state around.</p><p>Now, let’s think about “Spectre” like attacks against branch predictors…I’m focused here on “variant 2” since the speculated load attack is most easily addressed using software (at low impact to performance) for the moment…</p><p>Over the years, there have been attempts to teach compilers to re-order programs such that the branches contained within them are setup according to some understanding of how the CPU will favor them. For example, very simple branch predictors might do things like “BTFN” (Backward Taken, Forward Not) — modern CPUs have dedicated loop predictors that only do that kind of thing when really seeing actual loops. Other attempts at hacking up compilers to help out have sought to add “likely” and “unlikely” macros to programs (including the Linux kernel) that encode hints into the instructions emitted by the compiler. All of these are mostly nonsense and should be removed since they are ignored by modern microprocessors. Indeed, Intel explicitly ignore all of the hints such as “likely” and “unlikely” in modern cores.</p><p>Instead, modern microprocessors include very sophisticated branch predictors. For example, there might be three (or more) different predictors in a particular processor, and a voting algorithm arbitrating between them. The CPU will see an upcoming branch as early as instruction decode time (AMD processors do a pre-decode into the instruction cache, annotating branches in the instruction cache and tracking them there, Intel do some similar things), while other parts of the machine will track what those branches did in history. Various global and regional history of branches is built up and tracked using such things as a PHT, GHT, BTB, and other fancy data structures that ultimately are intended to help guess in the future what a branch will do.</p><p>And there are different kinds of branch. Direct branches are your classical “if” statement condition in which we go one way or another. Indirect branches are “far calls” indirecting through a function pointer to some other place in the program. When we hit an indirect branch we may literally have no idea where that function pointer is actually going to lead us, but often the same branch is going to be to the same pointer each time we hit the same branch. If we can predict where this branch is going to lead us, we can absorb some of the cost of pointer chasing to find the ultimate resolution of a given branch point.</p><p>For example, it might be that every time we hit a branch at a particular code instruction address in our program, we store that address in a cache/buffer internal to the microprocessor. When we hit that branch next time around, we see that we have an entry for it in our predictor tables, and we know which way it went the last time around, and where that branch took us, so we guess that it’s going to do the same thing this time around. For “performance” tradeoff (to avoid having huge content addressable tables) we index this table of (virtual — it’s indexing based off VAs from the icache) addresses using just a few low order bits of the code address of the branch.</p><p>Now it’s worth noting that modern CPU pipelines are 15+ stages long (many of them have variable sizes, while a few outliers over the years have gone totally nuts with the length in an ill fated attempt to get to 5GHz and beyond). When we get our prediction wrong, we need to flush this pipeline (possibly not entirely, there are some cute hacks) and proceed down the right path. So we generally like to get our predictions right. This means that a /lot/ of work has gone into making branch predictors both incredibly fast (this is critical path during instruction fetch and decode) as well as highly accurate.</p><p>Tradeoffs are necessary, including the size of the CAM (Content Addressable Memory) that is needed to lookup virtually indexed branches in our branch target buffers. Some processors will limit this to the low order 31 bits (just an example), while the high order bits of the address can merrily be from any application, or from the kernel, and we’ll think of them as homonyms (same branch entry). To resolve this is “trivial” in that we need to factor in the full virtual address of the branch, or the address space that contains it.</p><p>Future silicon is trivially fixed to address these problems by adding the PCID (x86), ASID, or other address space identifier to the branch target. Then we don’t allow the branch target to become poisoned by attacker code.</p><p>Temporary mitigation is simply to disable indirect branch prediction within the kernel (or hypervisor) on entry. Sure, it’s possible we could train a regular direct predictor, but the risk is low there (and if even possible in some implementations would only let us speculate very locally), while if we can train the indirect predictor to jump to gadget code in the kernel over which we have influence, we can then cause speculated loads that give us data we want to read. The expensive part of this mitigation comes from the cost of disabling the predictors on entry. Few designs do this directly without also disabling something else, and then there is the cost of context serializing MSR (x86) or other register writes during that entry code.</p><p>I’m still working on this and will turn it into a much longer post with some nice graphics. If there’s interest, I’ll try to get it on our corporate blog with some nicely produced animations of how cache attacks work.</p><p>Jon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc9b15a7a52c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>