<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Bolin on Medium]]></title>
        <description><![CDATA[Stories by Michael Bolin on Medium]]></description>
        <link>https://medium.com/@mbolin?source=rss-588a10a26d5a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*SOeLg85_Q-vu1u80.jpg</url>
            <title>Stories by Michael Bolin on Medium</title>
            <link>https://medium.com/@mbolin?source=rss-588a10a26d5a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:23:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mbolin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing My Daughter to The Secret of Monkey Island Was the Best Parenting Decision I Made…]]></title>
            <link>https://mbolin.medium.com/introducing-my-daughter-to-the-secret-of-monkey-island-was-the-best-parenting-decision-i-made-ff20bfd79c88?source=rss-588a10a26d5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff20bfd79c88</guid>
            <category><![CDATA[monkey-island]]></category>
            <category><![CDATA[pandemic-reflections]]></category>
            <category><![CDATA[video-games-for-kids]]></category>
            <category><![CDATA[lucasarts]]></category>
            <category><![CDATA[adventure-games]]></category>
            <dc:creator><![CDATA[Michael Bolin]]></dc:creator>
            <pubDate>Sat, 13 Mar 2021 14:52:40 GMT</pubDate>
            <atom:updated>2021-09-14T16:28:23.926Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing My Daughter to <em>The Secret of Monkey Island</em> Was the Best Parenting Decision I Made During the Pandemic</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*N_wuNREoOFrnOByV_IySVA.png" /><figcaption>Title screen for The Secret of Monkey Island (1990)</figcaption></figure><p>Like many parents across the US, on March 13, 2020, I became a kindergarten teacher. That is, due to COVID-19, my daughter’s school shut down and my wife and I suddenly had to figure out how to maintain our full-time jobs as well as our sanity while taking care of two small children without any outside help.</p><p>As we quickly learned, 6-year-olds don’t have a ton of patience for video chat with groups of 20 students, so our school provided only 90 minutes of live “instruction” each week. That left a lot of dead air for mom and dad to fill. Most days, we didn’t have anything left in the tank after spending six hours persuading our kid to do three hours of assigned work in Google Classroom.</p><p>Even before the pandemic, as parents, it was tough to figure out how to impart when to “follow the rules” versus “challenge authority.” Certainly while we were in the midst of shelter-in-place, this tension was pushed to its limits between “Can you please just do what we ask without asking questions so mom and dad can get some work done?” and “All of our kids’ social interactions were abruptly digitized overnight so maybe can we cut them some more slack when they start acting out?” In a world where my daughter had suddenly lost control over everything, I thought that playing a game where she got to make all of the decisions might be a healthy escape.</p><p>In my mind, there was only one choice — an <strong>adventure game</strong> I loved dearly from my own childhood: <em>The Secret of Monkey Island</em>.</p><p>In this article, I hope to explain:</p><ul><li><a href="#e9ee">What an adventure game is</a>, how it differs from other types of games, and what makes it ideal for young kids.</li><li><a href="#81e9">Why <em>The Secret of Monkey Island</em> is the perfect adventure game to start with</a> (and <a href="#64b7">how to get it</a>).</li><li><a href="#7099">How to play an adventure game with your child.</a></li></ul><p>Let’s dive in!</p><h3>What is an “Adventure Game?”</h3><p>To start, it is important to understand what qualifies as an “adventure game” and why it is a great fit for young kids. The <a href="https://en.wikipedia.org/wiki/Adventure_game">Wikipedia definition</a> captures it well:</p><blockquote>An adventure game is a video game in which the player assumes the role of a protagonist in an interactive story driven by exploration and puzzle-solving.</blockquote><p>Equally important to the <strong>interactive story</strong> and <strong>puzzle-solving</strong> design elements that define an adventure game are two that are forbidden: <strong>quick reflexes</strong> and <strong>irrecoverable actions</strong> (such as “death”). Note that not all “adventure” games satisfy this definition exactly, but the best ones generally do. As compared to other genres:</p><ul><li>An <a href="https://en.wikipedia.org/wiki/Action_game">action game</a> often emphasizes quick reflexes more than anything else. This includes <a href="https://en.wikipedia.org/wiki/Platform_game">“platformers”</a> such as <a href="https://en.wikipedia.org/wiki/Super_Mario_Bros.">Super Mario Bros.</a> or <a href="https://en.wikipedia.org/wiki/First-person_shooter">first-person shooters</a> such as <a href="https://en.wikipedia.org/wiki/Quake_(video_game)">Quake</a>.</li><li>An <a href="https://en.wikipedia.org/wiki/Action-adventure_game">action-adventure game</a> is a hybrid of the action and adventure genres, though the balance between the two varies. The game may have a strong focus on one or the other.</li><li>A <a href="https://en.wikipedia.org/wiki/Role-playing_video_game">role-playing game</a> (RPG) is arguably the most similar to an adventure game, though it often has some element of “combat” and/or “[health] metrics” that are not found in a pure adventure game. If it feels like an adventure game, but you are required to “level up” as part of the quest, it’s arguably an RPG.</li><li>A <a href="https://en.wikipedia.org/wiki/Puzzle_video_game">puzzle game</a> is, expectedly, about puzzle-solving, but the puzzles often have the same general structure and increase in difficulty over the course of the game (e.g., Angry Birds, <a href="https://store.steampowered.com/app/400/Portal/">Portal</a>). Although puzzle games are also friendly to those who want to exercise their brains rather than their reflexes, the puzzles are far less varied and there is no significant story component, as compared to adventure games.</li></ul><p>Based on this categorization, trying to picture an “adventure game” might evoke the image of this [fictional!] computer game that Tom Hanks plays in the movie <a href="https://amzn.to/2P9MME2"><em>Big</em> (1988)</a>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FysvNjGyqsxY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DysvNjGyqsxY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FysvNjGyqsxY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/432995673246debb8f0c873694351793/href">https://medium.com/media/432995673246debb8f0c873694351793/href</a></iframe><p>and…this is fairly representative of the genre at the time. Though one thing that happens in <em>Big</em> is that young Josh Baskin is too slow to hit Enter after typing THROW THERMAL POD and his character succumbs to a fatal blow from an ice scepter. Although typing and timing are elements of <em>some</em> adventure games, this is not always the case. Generally speaking, if there is a puzzle in an adventure game that requires quick reflexes, you get unlimited attempts to try again rather than a single failure resulting in death.</p><p>Indeed, the creator of <em>The Secret of Monkey Island</em> (which we’ll get to!), Ron Gilbert, published an essay, <a href="https://grumpygamer.com/why_adventure_games_suck">“Why Adventure Games Suck”</a> in 1989, enumerating a set of design guidelines to help maintain the “suspension of disbelief” that makes these games so great. In many ways, the brilliance of the adventure games that came out of LucasArts during the ‘90s was a reaction to the frustrations of <a href="https://en.wikipedia.org/wiki/Sierra_Entertainment">Sierra’s</a> adventure games (which, despite their design flaws, frequently outsold LucasArts, at least in the US, but more on that later).</p><p>A key word in this definition is <strong>story</strong>, and this design element, more than any other, is arguably what makes adventure games resonate with children. Think about how many times one of your kids has asked, “Will you please tell me a story?” As explained in <a href="https://amzn.to/3blE1zj"><em>Made to Stick</em></a> (a book that I have personally found tremendously helpful in improving my professional communications), <strong>S</strong>imple <strong>U</strong>nexpected <strong>C</strong>oncrete <strong>C</strong>redible <strong>E</strong>motional <strong>S</strong>tories are powerful tools to draw in your audience and land a message that sticks with them after they leave the room.</p><figure><a href="https://amzn.to/3blE1zj"><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Nbcd2W3LmYusHI32g_1Ygg.jpeg" /></a><figcaption>“When children say ‘Tell me a story,’ they’re begging for entertainment, not instruction.”</figcaption></figure><p>Certainly in my own experiences playing with my daughter, the desire to find out “what happens next” in the story is what motivated her persevere when she was stuck during a challenging part, which brings us to the other term I call out in the definition: <strong>puzzle-solving</strong>!</p><p>Generally speaking, the puzzles in an adventure game require some amount of logic and creativity, but are not “academic challenges” and therefore are generally accessible to all ages. (Though one notable exception is the use of <em>puns</em>, since those often require some minimum level of sophistication with language.) For example, consider this classic puzzle that can be solved by a kindergartner, yet will stump many adults:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*dlJVWPANBscK-nxRioDTSA.png" /><figcaption>Can you draw lines connecting A to A, B to B, and C to C such that none of the lines cross each other, nor do they go outside the surrounding rectangle?</figcaption></figure><p>As part of my personal journey in rediscovering adventure games, it has also been interesting figuring out what <em>doesn’t</em> work for kids (or at least my kids). As noted, action games are at the opposite end of the spectrum. If you’re curious to learn more and want a real page-turner, I strongly recommend the book <a href="https://amzn.to/3aYikoQ"><em>Masters of Doom</em></a>, which details the genesis of <a href="https://en.wikipedia.org/wiki/Id_Software">id Software</a>. For the uninitiated (as I was), id is generally credited with inventing the first-person shooter genre, as it was responsible for Wolfenstein 3D, Doom, and Quake, among others. In the book, co-founder of id, <a href="https://en.wikipedia.org/wiki/John_Carmack">John Carmack</a> (whose influence on the history of computer graphics cannot be overstated) is quoted during a brainstorming session for Doom:</p><blockquote>“Story in a game is like a story in a porn movie; it’s expected to be there, but it’s not that important.”</blockquote><p>As the story of id Software unfolds, co-founders John Carmack and John Romero ultimately go their separate ways. Romero, believing that his talents as a game designer can make up for the loss of his wunderkind technical co-founder, ends up falling miserably short in his next video game venture.</p><figure><a href="https://amzn.to/3aYikoQ"><img alt="" src="https://cdn-images-1.medium.com/max/600/1*oFg93pKDMoijUryIEZXSSg.jpeg" /></a></figure><p>Not being well-versed in video game history, when I first read <em>Masters of Doom</em> in 2016, my main takeaway was: “Wow, id just kept winning based on having better and better graphics — I guess that must be the secret to video games today!” And then I realized that in 2020, I found myself engrossed by games where the character’s legs were indistinguishable from the wall:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6iE-EOCezcFHOFYQB4NCA.png" /></figure><p>At first, this was a little hard to reconcile, but this only serves to underscore how different the adventure and action genres are. As discussed later, the simpler graphics of some of these adventure games is part of what made them so fun, as they invited more imagination on the part of the player. Further, without dazzling graphics to fall back on, the stories and puzzles had to be top-notch to keep people engaged.</p><h3>Why Should I Have My Kid Play The Secret of Monkey Island?</h3><p>When it comes to famous first lines, “Call me Ishmael” ain’t got nothin’ on “Hi! My name’s Guybrush Threepwood, and I want to be a pirate!” Whereas the former is reminiscent of sleepy afternoons in high school English class, the latter sparks an electric mix of intrigue, excitement, and adventure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frgSOdc2DPEzewFSd_PSSA.png" /><figcaption>Opening scene to The Secret of Monkey Island</figcaption></figure><p><a href="https://www.mocagh.org/lucasfilm/mi-manual.pdf">The game’s manual</a> provides the following introduction:</p><blockquote>In The Secret of Monkey Island, you play the role of Guybrush Threepwood, a young man who has just hit the shores of Mêlée Island (somewhere in the Caribbean). Our naive hero’s travels have led him to Mêlée Island in a quest to fulfill his life’s ambition…to become a fierce, swashbuckling, bloodthirsty <strong>Pirate</strong>. Unbeknownst to Guybrush, however, there have been some strange happenings in the area surrounding Mêlée Island and the more mysterious Monkey Island. As he walks into this maelstrom of mystery, Guybrush will soon be very aware that things are not what they appear to be, that even bloodthirsty pirates can be scared, and that there’s more to being a pirate than swinging a sword and drinking grog!</blockquote><p>Now who wants to be a pirate???<br><br>There are a lot of adventure games out there. Compared to <em>The Secret of Monkey Island</em> (hereinafter referred to as SoMI), most adventure games have fancy graphics, voice actors, and can be downloaded easily onto your mobile phone or tablet. <em>SoMI has none of these things! </em>(OK, the 2009 remake of SoMI has two of these three things, but we’ll get to why I discourage starting with that one later.) Given all that, why am I pushing SoMI so hard?</p><h4>[Optional] A Not-so-Brief History Lesson</h4><p><em>This section explains where SoMI fits in adventure game history and why it was such a breath of fresh air compared to those that came before it. If video game history isn’t your thing, feel free to skip this section and go on to the next one that explains </em><a href="#ba98"><em>why SoMI is a great game to play with kids</em></a><em>.</em><br><br>The “adventure game” genre is so-named for <a href="https://en.wikipedia.org/wiki/Colossal_Cave_Adventure"><em>Colossal Cave Adventure</em></a> (also known as, simply, <em>Adventure</em>), the first text adventure game, released in 1976. There were no graphics: it printed text to the screen describing your situation and you had to reply with the action you wished to take. The original <em>Adventure</em> had only 193 vocabulary words, which meant that it was common that what you typed in might easily be understood by a human, yet was incomprehensible to <em>Adventure</em>, triggering a frustrating cycle of trial-and-error. (For example, enter building to enter the building at the start of the game does not work, but go building does.) Making things worse, it was fairly easy for your character to die, forcing you to start over. Although this introduced the world to <a href="https://en.wikipedia.org/wiki/Interactive_fiction">interactive fiction</a>, the gameplay needed refinement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*lbynMrf07lyIqC4n4oDH6A.png" /></figure><p>One of the more well-known text adventure games inspired by <em>Adventure</em> was <a href="https://en.wikipedia.org/wiki/Zork">Zork</a>. Early versions of the source code <a href="https://github.com/MITDDC/zork">dating back to 1977</a> have been recovered, though the first commercial release (Zork I) was in 1980, followed by Zork II in 1981 and Zork III in 1982. Like <em>Adventure</em>, Zork did not offer any graphics, but its text parser was markedly more sophisticated. For example, it was no longer limited to simple noun-verb commands (hit troll), but could tolerate more natural language (hit the troll with the Elvish sword).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*uloEVGJXcPs8WUkFEVDpyQ.jpeg" /></figure><p>In 1984, <a href="https://en.wikipedia.org/wiki/Sierra_Entertainment">Sierra On-Line</a> (who would become the industry leader in adventure games in its time) released <a href="https://en.wikipedia.org/wiki/King%27s_Quest_I"><em>King’s Quest</em></a>, which was a massive step forward in the genre, being the first adventure game to introduce (gasp!) graphical animation into the game world so you could actually, you know, <em>have your character walk around</em>. To its credit, the game world also simulated depth perception, so you could walk behind objects and whatnot, so some consider it the first “3D-animated” adventure game. (Although Wikipedia may be correct in that <em>King’s Quest </em>was considered “3D” in 1984, it would most certainly be classified as 2D today.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*nx-J8-Q5c-0HhUUim-hKkA.gif" /><figcaption>Typical out of the box experience when playing <em>King’s Quest</em>…</figcaption></figure><p>Despite these graphical advances, <em>King’s Quest</em> (and Sierra adventure games, in general) maintained the use of textual commands to control your character’s behavior. Although Zork verified that it was possible to implement a more tolerant parser with the technology at the time, users still often found themselves playing guessing games with what the computer could understand. Further, Sierra games were also known for having a multitude of ways for your character to die. Sometimes the story line was clever and the resulting death scene was humorous, but more often than not, it was for something seemingly innocuous or petty. For example, Sierra’s <a href="https://en.wikipedia.org/wiki/Police_Quest:_In_Pursuit_of_the_Death_Angel"><em>Police Quest</em></a> was often criticized [or possibly praised, if the audience was a member of law enforcement] for being more focused on punishing the player for failing to follow proper police procedure than encouraging creative puzzle-solving.<br><br>Fortunately, rival game studio <a href="https://en.wikipedia.org/wiki/LucasArts">Lucasfilm Games/LucasArts</a> (which was indeed an initiative spawned by <a href="https://en.wikipedia.org/wiki/George_Lucas">George Lucas</a>) made a hard break from the Sierra mold with their introduction of <a href="https://en.wikipedia.org/wiki/Maniac_Mansion"><em>Maniac Mansion</em></a> in 1987. It was a “point-and-click” adventure game, as your character could be controlled exclusively by the mouse. As an alternative to text parsers of yore, it offered a fixed set of 15 on-screen verbs that you could select with the mouse, followed by clicking on either an object in your inventory or an object on the screen to act on, thereby constructing a command for your character to execute. To find out which pixels on the screen corresponded to actionable objects, you had to select the <strong>What is</strong> verb and move the mouse around until you stumbled upon something you could interact with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-MyAaXYDhk_mfhlhNUQmg.png" /></figure><p><em>Maniac Mansion</em> revolutionized adventure games in many respects, but one notable design element that remained unchallenged was character death/dead ends. That is, characters in <em>Maniac Mansion</em> were indeed susceptible to death (though not nearly as easily as compared to contemporary Sierra games); but more frustrating, you could easily make choices early in the game that would make the game unbeatable, which you would not discover until the very end, if at all. Co-designer/programmer of <em>Maniac Mansion</em>, <a href="https://grumpygamer.com/">Ron Gilbert</a> [arguably one the most renowned game designers and programmers of point-and-click adventure games today], would address this in his subsequent work.<br><br>At LucasArts, Gilbert went on to work on <a href="https://www.gog.com/game/zak_mckracken_and_the_alien_mindbenders"><em>Zak McKracken and the Alien Mindbenders</em></a> (1988), <a href="https://www.gog.com/game/indiana_jones_and_the_last_crusade"><em>Indiana Jones and the Last Crusade: The Graphic Adventure</em></a> (1989), and finally, <a href="https://www.gog.com/game/the_secret_of_monkey_island_special_edition"><em>The Secret of Monkey Island</em></a> (1990) followed up by <a href="https://www.gog.com/game/monkey_island_2_special_edition_lechucks_revenge"><em>Monkey Island 2: LeChuck’s Revenge</em></a><em> </em>(1991) a year later. In terms of gameplay, both <em>Zak McKracken</em> and <em>Last Crusade</em> were similar to <em>Maniac Mansion</em>, but SoMI offered some material improvements:</p><ul><li>Early releases of SoMI had 12 verbs, but eventually got down to 9, as <strong>Turn on</strong>, <strong>Turn off</strong>, and <strong>Walk To</strong> were dropped.</li><li>The reduction to 9 verbs also saw the textual inventory list replaced with a graphical one!</li><li>Now if you moused over an onscreen object you could interact with, its name would appear automatically. You no longer had to select <strong>What is</strong> all the time!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*7mrxJWUfXf0sc6ogg_p1wQ.png" /><figcaption>Here is the 12-verb version with the textual inventory list.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*8-Vnu3Xu5l-wX0zIHdUn7g.png" /><figcaption>Here is the 9-verb version with the graphical inventory list</figcaption></figure><p>But perhaps most importantly, SoMI consummated the “no dead-ends” philosophy(*). That is, your character could not die (with one minor exception, explained below), and you could never get your character into an unrecoverable situation. No longer did you have to nervously save your game at every turn! The ease with which you could explore the world and command your character truly was a breath of fresh air to those who had played other adventure games up to that point (yours truly included).</p><p>Although Sierra is not mentioned by name, this section from <a href="https://www.mocagh.org/lucasfilm/mi-manual.pdf">the SoMI game manual</a> unequivocally calls them out:</p><blockquote>Our Game Design Philosophy</blockquote><blockquote>We believe that you buy games to be entertained, not to be whacked over the head every time you make a mistake. So we don’t bring the game to a screeching halt when you poke your nose into a place you haven’t visited before. We make it clear, however, when you are in a dangerous situation.</blockquote><blockquote>We think you’d prefer to solve the game’s mysteries by exploring and discovering, not by dying a thousand deaths. We also think you like to spend your time involved in the <strong>story</strong>, not typing in synonyms until you stumble upon the computer’s word for a certain object.</blockquote><blockquote>Unlike conventional computer adventures, you won’t find yourself accidentally stepping off a path, or dying because you’ve picked up a sharp object. There are a few dangerous situations where Guybrush can die, but to anticipate them takes just a little common sense, not excessive paranoia. Save the game when you think you may be entering a dangerous area, but don’t assume that every wrong step will result in death. Usually you’ll get another chance.</blockquote><p>As best I could find, the first LucasArts game to feature this “Our Game Design Philosophy” passage in its manual was <a href="https://www.mocagh.org/lucasfilm/zak-manual.pdf"><em>Zak McKracken and the Alien Mindbenders</em></a>, though the guarantees around not dying are slightly weaker, as <em>Zak McKracken</em> has a number of ways to die whereas SoMI has exactly one. (Fear not, as the only way to die in SoMI is more of an <a href="https://en.wikipedia.org/wiki/Easter_egg_(media)">Easter egg</a> that you have to go out of your way to fall into rather than a devious trap. <a href="https://www.google.com/search?q=how+can+you+die+in+the+secret+of+monkey+island&amp;oq=how+can+you+die+in+the+secret+of+monkey+island">Google it if you’re curious</a>, as I don’t want to spoil it for you!)</p><p>Upon further inspection, the version of “Our Game Design Philosophy” in SoMI appears to be <em>exactly</em> the same as that in Gilbert’s previous game, <em>Indiana Jones and the Last Crusade</em>, with “Indy” replaced with “Guybrush.” As <em>Last Crusade</em> is more of an action-adventure game (light on action; heavy on adventure), the warnings about death were apt, so arguably the language could have been softened for SoMI.</p><p>(*) Although <em>Loom</em> came out several months before SoMI and was the first in the LucasArts adventure series to realize the “no dead-ends” philosophy, I argue it was really SoMI that crystallized the full set of design principles that all subsequent LucasArts adventure games would follow.</p><h4>That’s enough history — let’s get down to brass tacks!</h4><p>Again, when I introduced SoMI to my daughter, she was in kindergarten and only six years old. I had previously attempted platformers like Super Mario Bros. on the NES with her in the past, but those were met with resounding rejection because her reflexes were not up to the task. Although some kids’ initial response to the NES is: “I died immediately, but I want to get better at this,” it was clear that the introduction of Super Mario Bros. was not going to be the defining growth mindset moment in my kid’s life.</p><p>Though arguably, that demoralizing experience with action games helped me drive home the contrast with adventure games, which helped her be more open to persevering through the puzzles in SoMI. To that end, please indulge me as I enumerate the attributes of SoMI that make it a great fit for playing with a young child:</p><p><strong>“You cannot die.”</strong> I repeated this until I was blue in the face. “Why are you afraid to go into that dark room? You can’t die.” “Why are you afraid to talk to that creepy person? You can’t die.” Because of her experiences with, well, basically any other video game she had played herself or had watched me play, my daughter took a lot of convincing that SoMI was actually different and no matter what she did, Guybrush Threepwood would not die. I believe once she internalized this, her attitude towards these games was forever changed for the better.</p><p><strong>There are no dead-ends.</strong> A “dead end” in an adventure game would be an unrecoverable mistake, such as failing to pick up an item from a place that is unreachable later in the game when you actually need to use the item. SoMI has no dead-ends: even though there may be times when you think you have gotten yourself stuck in an impossible situation, you haven’t, and you never need to start over. It is important to get this through to your kid because often when something “bad” happens to your character, it is not because you made a mistake, but because “the game made the bad thing happen to your character” to advance the story. Helping avoid this sense of failure is helpful when encouraging one to try again.</p><p><strong>Encourages experimentation.</strong> This is a corollary to “there are no dead-ends.” That is, if there are no dead-ends, one should have no inhibitions about trying things! Pick everything up; talk to everyone; explore every path through a dialog tree, etc. Again, it took quite a bit of convincing that SoMI was “a safe space” where it never hurt to try. (I have played a lot of these games, and I even catch myself getting stuck every now and then because I wasn’t frivolous <em>enough</em>!) As a parent, it was exciting to see how, over time, it took less prompting on my part to explore things as it became more self-initiated.</p><p><strong>Not a test of reflexes.</strong> Again, the contrast to Super Mario Bros. was important here when introducing SoMI. I go into more detail about motor skills in the next section, but with respect to building confidence, it was important that I convinced my daughter that there would never be a puzzle that she would be unable to solve because she wasn’t fast enough with the keyboard or trackpad. Now, in reality, there <em>is</em> a handful of cases in SoMI where some amount of reflexes are required, but (1) in such cases, you always have unlimited chances to try again, and (2) the tolerance for reaction times is very generous (though maybe not generous enough for a six-year-old). To manage this, I made it clear that if we did encounter one of these edge cases, I would take over the controls for her, so long as she gave me the instructions for what she wanted me to do. (Somewhat related, Monkey Island is a 2D game, which makes it fairly easy for kids to move Guybrush around, compared to 3D games, such as Minecraft, which has a steeper learning curve.)</p><p><strong>You have to read!</strong> In the original version of SoMI (as opposed to the <em>Special Edition</em>, discussed later), there were no voice actors. Everything “spoken” by a character is displayed as text on the screen, so if you want to know what someone is saying, you have to read it. As a parent trying to motivate their kindergartener to learn to read, this technological “limitation” was an asset! Further, SoMI has solid design features to make this more approachable to developing readers: (1) the speed of the dialog is configurable, so you can slow it down, and (2) the game can always be paused with the space bar, which happens to be the largest key on the keyboard. In practice, this meant that I set the dialog to play back at the slowest setting, and my daughter quickly learned to hit the space bar if she sensed she was going to struggle with the text on the screen, thereby giving her as much time as she/we needed to work through it before unpausing.</p><p><strong>Technological limitations invite imagination.</strong> I found an interview with Ron Gilbert from 2017 where <a href="https://youtu.be/KgzWorfLjeE?t=2530">someone asks whether the technological limitations (i.e., 8-bit graphics) in older games are responsible for much of the charm</a>. Specifically, because things could not be drawn to look realistic, you as the player had to inject your own imagination to “fill in the gaps.” Ron’s short answer was “yes” (though I encourage you to watch the full clip), and he noted that in his most recent game, <em>Thimbleweed Park</em> (2017), they deliberately avoid any close-up shots of the characters, as a more detailed rendering might break with how the player had envisioned the character up to that point. Personally, I know that when I played <em>The Curse of Monkey Island</em>, which was the first installment in the series to introduce voice acting, and I heard Guybrush and Stan for the first time, I was actually a bit sad because I preferred the existing voices I had built up for them in my imagination from SoMI and I could no longer hear them.</p><p><strong>You will laugh out loud.</strong> The comic writing for SoMI is untouchable. For sure, many of the jokes will go over your kid’s head, but you personally will definitely be laughing out loud (it’s also fascinating to see what jokes your kid does and doesn’t get). Perhaps the most memorable puzzle of SoMI is the “insult sword fighting” where the game designers brilliantly managed to square the circle of: “How do you create a video game about pirates that involves sword fighting, yet doesn’t test the player’s reflexes?” Incidentally, <a href="https://en.wikipedia.org/wiki/Orson_Scott_Card">Orson Scott Card</a> (of <em>Ender’s Game</em> fame) is credited with some of the writing here, though based on a “30th anniversary documentary” I recently watched on YouTube, <a href="https://youtu.be/xgqEneDNQto?t=3414">Card notes that he got more credit than he deserved on this front!</a></p><p><strong>It’s a “clean” game.</strong> Technically, SoMI <a href="https://www.esrb.org/ratings/26914/The+Secret+of+Monkey+Island+Special+Edition/">has an ESRB rating of <strong>Everyone 10+</strong></a> for “alcohol and tobacco reference, comic mischief, mild cartoon violence, and suggestive themes,” but you really have nothing to worry about. (The comic mischief is one of the reasons I <em>wanted</em> my daughter to play the game in the first place!) Given that most games seem to be developed primarily for adolescent-to-adult males, it is refreshing to find a serious adventure game that is also appropriate for young girls. Though the fact that there were exactly two female characters in the game (both of which are strong women, at least!) did not go unnoticed by my daughter…</p><p><strong>The music rocks.</strong> This may seem like a small thing, but when you play an adventure game and are grinding it out when you’re stuck, revisiting the same places over-and-over to see what you have missed, you had better believe that having a killer soundtrack makes the repetition easier to swallow. As a kid, I happened to acquire SoMI on CD-ROM, which was also playable as an audio CD that I continue to enjoy to this day.</p><p><strong>Teaches “real computer” usage (i.e., not a tablet).</strong> A good friend of mine had his son using a laptop rather than a tablet from an early age. His reason: “Tablets are for consumers and laptops are for producers.” Although there are admittedly folks doing “professional” work on iPad Pros these days, I tend to agree and I was happy that this experience exposed my daughter to the keyboard and trackpad. Although it is possible to get SoMI running on an Android device, the game was designed to be played with a mouse (hunting around the screen via the pointer is a fundamental part of the experience), which does not translate seamlessly to touchscreens.</p><p><strong>It’s a real game.</strong> SoMI is a legitimately challenging computer game. If your child plays it all the way through, I would be sure to emphasize that “this is a game for older kids” and they should feel a real sense of accomplishment, regardless of how much help they got from mom or dad along the way.</p><h3>OK, fine, just tell me how to get this game already!</h3><p>The easiest way to get a legitimate copy of SoMI is to purchase it from <a href="https://www.gog.com/game/the_secret_of_monkey_island_special_edition">GOG</a> or <a href="https://store.steampowered.com/app/32360/The_Secret_of_Monkey_Island_Special_Edition/">Steam</a> for $10 and play it on your PC.</p><p><em>If you have a Windows PC, stop reading NOW and go spend the $10. Do it. I’ll wait…The rest of this section is for folks who do not have a Windows machine or enjoy the unnecessary level of detail I cannot seem to resist when discussing this topic. Everyone else can feel free to skip ahead to </em><a href="#7099"><em>How to Play an Adventure Game with Kids</em></a><em>.</em></p><p>It appears that <a href="https://marketplace.xbox.com/en-US/Product/Monkey-Island-SE/66acd000-77fe-1000-9115-d80258410995">SoMI can still be purchased for Xbox</a>, but <a href="https://www.pocketgamer.com/articles/064928/look-behind-you-monkey-island-1-and-2-have-disappeared-from-the-app-store/">the versions for iOS were pulled from the App Store years ago</a>. Though note that all of these vendors are selling <em>The Secret of Monkey Island: Special Edition</em>, but not the original SoMI. What makes it <em>Special</em> is the upgraded graphics, use of voice actors, and introduction of a built-in hint system. Rest assured that everything else about the game is true to the original. Indeed, the assets for the original game are bundled with <em>Special Edition</em>, so you can hit F10 at any time to switch between <em>Special</em> and SoMI without losing your place in the game! The one kicker is that the <em>Special Edition</em> is available for Windows, but not macOS or Linux. However…</p><p>…the original SoMI bundled with <em>Special Edition </em>can be extracted so it is playable on macOS or Linux via the magic of <a href="https://www.scummvm.org/">ScummVM</a> (which, in turn, <a href="https://wiki.scummvm.org/index.php/Platforms">makes it playable on a ton of platforms</a> that did not even exist when SoMI came out!). Unfortunately, wiring this up takes quite a bit of work. You see, ScummVM is a freely available piece of software that can run old games, but [for legal reasons] <em>it does not include the games with it</em>. That means you have to come up with your own copy of SoMI for ScummVM to run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WclUATsKQmdg-ki0ttq0Lw.jpeg" /><figcaption>As a kid, <em>The Secret of Monkey Island</em> fell into my lap by means of one of those CD-ROM/SoundBlaster big-box upgrades that had a random collection of games bundled with them. I don’t believe I ever had a proper box for my copy, just this CD-ROM in a sleeve. I have not recovered the manual that came with it, but I remember thinking at the time that “it couldn’t possibly be the real manual” because it was so short and it must have been some sort of “abridged version” that they made for these SoundBlaster bundles. (I think I was also convinced that the “real manual” would have had more clues or something.) Clearly the elegance of the simplicity of the instructions was lost upon me at the time.</figcaption></figure><p>As best as I can tell, here are your options for playing SoMI on Mac or Linux:</p><ul><li>Get an original CD-ROM of the game and copy of the necessary bits for ScummVM (though originals can be pricey on eBay). I have my childhood copy of SoMI and I verified that copying the MONKEY.000 and MONKEY.001 files from the CD-ROM to a folder on local disk named monkey was sufficient for ScummVM to be able to run it (though some extra work is required to get the audio just right). Note <a href="https://wiki.scummvm.org/index.php?title=Datafiles#Secret_of_Monkey_Island.2C_The">the ScummVM wiki</a> details which files you need to copy for various games.</li><li>Purchase the game on <a href="https://www.gog.com/game/the_secret_of_monkey_island_special_edition">GOG</a> or <a href="https://store.steampowered.com/app/32360/The_Secret_of_Monkey_Island_Special_Edition/">Steam</a> and extract the bits you need for ScummVM. <a href="https://forums.scummvm.org/viewtopic.php?t=7671">This thread</a> explains how to do it, but it is admittedly pretty complicated. I have not attempted to do this myself, and it almost seems like you have to start from the most recent updates to the thread and work backwards because many of the links from the start of the thread no longer work.</li><li>Instead of using ScummVM to run the original SoMI on your operating system, run Windows on your operating system and run <em>Special Edition</em> on it! Although this sounds complicated, it is likely easier than trying to extract the bits you need from the <em>Special Edition</em> for ScummVM. Personally, I run an unactivated version of Windows 7 under VMWare Player on my Linux machine, all of which is free. (You could also set up a <a href="https://en.wikipedia.org/wiki/Multi-booting">dual-boot</a> or use <a href="https://www.parallels.com/">Parallels</a>, but it’s easy to screw up a dual-boot and Parallels isn’t free. <a href="https://chromium.googlesource.com/chromiumos/docs/+/HEAD/containers_and_vms.md">Crostini</a> seems promising for Chrome OS, but it is only available on newer Chromebooks.) Although I prefer to play SoMI via ScummVM, I ended up having to set up Windows 7 on VMWare so I could play <a href="https://www.gog.com/game/tales_of_monkey_island">Tales of Monkey Island</a> on my Linux box. This makes it possible for me to play a number of other classic adventure games on my Linux box, such as <a href="https://www.gog.com/game/myst_masterpiece_edition">Myst Masterpiece Edition</a>.</li><li>What would Guybrush do? Hint: remember he’s a pirate. I do not condone this approach, but if you do go this route, the least you can do is buy a legitimate copy of <a href="https://www.gog.com/game/thimbleweed_park">Thimbleweed Park</a>, <a href="https://www.gog.com/game/day_of_the_tentacle_remastered">Day of the Tentacle</a>, or <a href="https://www.gog.com/game/broken_age">Broken Age</a> (all of which are fantastic adventure games, available on all platforms) or something to put money in the original designers’ pockets.</li></ul><p>I hope you can find a solution that works for you!</p><p>As a final heads-up, if you end up playing the <em>Special Edition</em>, it differs from the original in some key ways as compared to playing in ScummVM.</p><p>The good:</p><ul><li><strong>There is a built-in hint system!</strong> If you are stuck, all you need to do is press H to get a hint that is specific to your current progress in the game. Although I find <a href="https://www.mocagh.org/lucasfilm/miuk-hintbook.pdf">the print version of the original hint manual quite charming</a>, it is too easy to accidentally “read too far ahead” and give yourself more hints than you intended. Incidentally, <a href="https://mocagh.org/lucasfilm/mi2-hintbook.pdf">the print version of the hint manual for Monkey Island 2</a> improved upon this by requiring one of those “red cellophane decoders.” This was a clever improvement, though even reading the table of contents might reveal something that you meant to work out on your own.</li><li>The updated art for the background scenes is beautiful! That said, I agree with a number of critics who feel this new rendering of Guybrush looks weird and prefer the original, more pixelated version.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*8nS5ASMji-bSz24G7SiOQA.png" /><figcaption>When going through old things at my parents’ house, I found this email <em>that *I had printed out*</em> from September 1995. It was from a camp friend who I must have been discussing SoMI with that summer. If you were stuck in a video game in 1995, you couldn’t “Google it.” Though emailing a friend seems pretty technologically advanced compared to <a href="https://www.quora.com/Have-you-ever-played-any-of-the-old-LucasArts-games-Which-were-your-favorites">this Aussie whose mom had to put in ad in the local paper for help when their family got stuck in <em>Zak McKracken and the Alien Mindbenders</em></a>!</figcaption></figure><p>The not-so-good (which may make you reach for F10):</p><ul><li>It has voice actors. If you were relying on SoMI to force your kids to read, you might want to disable this. Fortunately, it supports turning the voice volume down to zero while leaving the music and sound effects at full blast.</li><li>Pausing the game cuts to the game management screen, so you can’t use this as a way to pause with the captions on screen to give early readers a chance to catch up. Subtitle speed in classic mode is still configurable (and pause works as it always did there).</li><li>It allows only 10 “save game” slots whereas ScummVM allows 100. I find it’s nice to be liberal with saving games so that you can go back and rewatch cut scenes or replay your favorite parts without much effort. (Indeed, later LucasArts games made rewatching cut scenes a built-in feature of their adventure games.)</li><li>The verbs and inventory no longer appear on-screen by default: ctrl will overlay the table of verbs and alt will overlay your inventory. The benefit is that this leaves more pixels for the scene, though personally I find having all of my options displayed in front of me at all times helps spark solutions to the puzzles.</li></ul><p>My recommendation is to play whichever version is more engaging to you, but avail yourself of the hint system in the <em>Special Edition</em> if you find yourself getting frustrated because you have been stuck on the same puzzle for too long.</p><h3>How to Play an Adventure Game with Kids</h3><p>Hopefully I have finally convinced you to give this whole “adventure game” thing a shot! As I spent much of the past year playing these games with my 6-year-old, I like to think I have picked up a few tips along the way that will improve the overall experience for both you and your kiddo:</p><h4>Play the Game Yourself First, By Yourself</h4><p>If you aren’t into adventure games yourself, this may be a tall order. For example, <a href="https://howlongtobeat.com/game?id=10162">according to howlongtobeat.com</a>, the average time to beat <em>The Secret of Monkey Island</em> is 7 hours. Bear in mind that is based on self-reported data from people who read web sites like <a href="http://howlongtobeat.com/">howlongtobeat.com</a> [and who actually finished it!], so if you’re new to these type of games, expect to take even longer. If you don’t have the patience or interest in playing through the game, <a href="https://google.com/search?q=monkey+island+walkthrough">you can always search for a <em>walkthrough</em></a>, which is the term for a self-reported write-up (or often these days, video) of the steps you need to take to make your way through an adventure game.</p><p>By playing the game yourself first (or reading through the walkthrough), you can give “just the right amount of help” when your kid gets stuck. Every child has their own capacity for <em>grit</em>. I am assuming you know what your kid’s is. For example, if you know that it took you an hour to solve a puzzle, either due to sheer difficulty or because you were wandering around in the wrong place for too long, you might want to provide some “gentle redirection” to prevent your young player from spending too much time pursuing a fruitless path. Alternatively, maybe you want them to have the experience of making that discovery themselves and you think they can handle it. Again, this depends on a combination of perseverance, interest, and experience of the player, which you are in a unique position to evaluate.</p><p>In particular, note that some puzzles might effectively be <em>impossible</em> for your child, so you certainly want to intervene when they get to one of those to avoid a total meltdown. For example, one of the things that comes into your possession during SoMI is gunpowder, so if you child lacks a rudimentary understanding of how firearms work, they may have no reasonable chance of figuring out what to do with it on their own. Similarly, there is heavy use of puns and/or double meanings in LucasArts adventure games, so your kid might benefit from some help with those, too.</p><p>You may receive a bit of pushback from your kin on the “playing it by yourself” part. I know I did. Daughter always wanted to know what Daddy was playing, and at first I let her, but then I decided to stop. The number one reason: spoilers. For example, I assumed that if she watched me play <a href="https://www.gog.com/game/loom"><em>Loom</em></a>, she would forget everything by the time I let her play it six months later. Wrong. Remember that adventure games are <em>stories</em>, and that <em>kids love stories</em>, so they hang on to details you wouldn’t expect. As a result, there was more “reciting” than genuine puzzle-solving than I intended when it was finally her turn to play.</p><p>Finally, playing the game by yourself <em>should be fun</em> (it is a <em>game</em>, after all)! One thing that made playing materially less fun for me is when I had an audience who was, shall we say, less patient than myself. For example, when I enter a new room in an adventure game, I want to take my time to explore room to ensure I haven’t missed anything. Other times, I might just want to grind it out and retrace many of my steps because I’m stuck. Trying to do that with, “Daaaaad, this is boring!” in the background certainly doesn’t help. By allowing yourself the authentic adventure game experience, with all its tribulations in addition to its triumphs, you can help tailor the experience to your child to ensure it is a good one.</p><h4>Read the Manual</h4><p>An adventure game manual, when well-done, is a short read. Most of what you need to know is “revealed” to you in the course of the game, so you only need to put in a little bit of time up front before getting started. There are often some key lines in there that are worth your attention, such as the actions that can bring your game to an end. I have found it to be valuable when setting expectations with a child whether it is possible to “die” or not. Here some samples from various LucasArts game manuals where each game is <em>slightly</em> different with regards to death:</p><ul><li><em>“Unlike conventional computer adventures, you won’t find yourself accidentally stepping off a path, or dying because you’ve picked up a sharp object. There are a few dangerous situations where Guybrush can die, but to anticipate them takes just a little common sense, not excessive paranoia. Save the game when you think you may be entering a dangerous area, but don’t assume that every wrong step will result in death. Usually you’ll get another chance.”</em> — The Secret of Monkey Island</li><li><em>“Since many adventure game players prefer solving puzzles to testing their reflexes, we’ve provided opportunities for you to steer Indy around any and all fighting — if you’re clever enough.”</em> — Indiana Jones and the Last Crusade</li><li><em>“Don’t be afraid to experiment. Nothing in the game can ‘kill’ you.”</em> — Loom</li><li><em>“Anything potentially disastrous that happens to the crew of the Attila mission is </em>supposed<em> to happen to them.”</em> — The Dig</li></ul><p>Specifically, within <a href="https://en.wikipedia.org/wiki/LucasArts_adventure_games">the canon of LucasArts point-and-click adventure games</a>, I believe only <em>Maniac Mansion</em> and <em>Zak McKracken and the Alien Mindbenders</em> have true “dead ends.” That is, a <em>dead end</em> is an action you can take that prevent you from winning the game, but the game will not tell you and therefore you could potentially wander aimlessly with a sense of false hope. (At least in the games where it is possible to “die,” it is transparent that you have made a misstep.) Incidentally, <em>Mansion</em> and <em>Zak</em> were the first two such games in the history of LucasArts: this “design flaw” was eliminated in all subsequent games.</p><p>Most manuals provide a small walkthrough to explain how to get navigate the first puzzle or two in the game. Experienced adventure games will likely go out of their way to avoid this section, but novice gamers might enjoy this as a “confidence boost” when getting started. Some go a bit further (such as <a href="https://www.mocagh.org/lucasfilm/zak-alt-manual.pdf">the <em>Zak McKracken</em> manual</a>) and provide a more detailed list of hints that may be useful as you progress through the game. Whether you take this “help” is up to you, but it is good to know whether it is there.</p><p>Finally, among the LucasArts games, a handful of them came with an additional text to supplement the manual. Having access to these is often material to making your way through the game:</p><ul><li><em>Maniac Mansion</em> came with a <a href="https://www.mocagh.org/lucasfilm/maniac-poster.jpg">poster</a>.</li><li><em>Zak McKracken and the Alien Mindbenders</em> came with a <a href="https://www.mocagh.org/lucasfilm/zak-paper.pdf">newspaper</a>.</li><li><em>Indiana Jones and the Last Crusade</em> came with a “<a href="https://www.mocagh.org/lucasfilm/indycrusade-diary.pdf">Grail Diary</a>.”</li><li><em>Loom</em> came with the <a href="https://www.mocagh.org/lucasfilm/loom-book.pdf"><em>Book of Patterns</em></a><em>.</em></li></ul><p>As you can see, digital versions are freely available online, though physical versions can go for a small fortune on eBay and the like.</p><h4>Learn the Controls</h4><p>Although a point-and-click adventure game can be played exclusively with the mouse, I would not recommend it. Consider how the mouse-only flow plays out:</p><ul><li>mouse around until you find the region of the screen that corresponds to an object you want to act on</li><li>move the mouse to the part of the screen where the “verbs” are, and left-click on the appropriate action, such as <strong>Pick up</strong></li><li>move the mouse back to the original region of the screen and left-click</li></ul><p>Note this requires moving the mouse quite a bit, remembering <em>exactly</em> where something is, and a lot of precision throughout. For small hands with developing motor skills, this can be challenging if not outright exasperating, particularly when mistakes are made and one has to start all over.</p><p>By comparison, consider how the keyboard-and-mouse flow plays out for the equivalent action:</p><ul><li>mouse around until you find the region of the screen that corresponds to an object you want to act on</li><li>let go of the mouse (as to avoid accidentally moving the pointer) and press <strong>p</strong> on the keyboard</li><li>left-click on the mouse</li></ul><p>With less mouse movement, this results in fewer errors and will give your child more confidence in their command of the game. (Anyone who has played <em>Maniac Mansion</em> and has struggled to find the pixel on a completely black screen that corresponds to a light switch only to have to start all over again once they moused away to select <strong>Turn on</strong> knows this pain well.)</p><p>Again, this is where spending a little time with the manual pays off. For example, in <em>The Secret of Monkey Island</em>, although <strong>p</strong> is an intuitive mnemonic for the verb <strong>[P]ick up</strong>, there are two other verbs that start with p: Push and Pull. It turns out that their keyboard shortcuts are <strong>s</strong> (pu<strong>[s]</strong>h?) and <strong>y</strong> (<strong>[y]</strong>ank?), respectively, which you would only know if you bothered to read the manual. Similarly, when choosing what to say during an interactive dialog, you can use the number keys to identify the desired response to the list rather than clicking on it (which I learned by reading the manual). Again, I only have a sample size of one, but there were far fewer errors when using the keyboard, and nothing was more frustrating for me than a cry of, “That’s not what I wanted them to say!” because <em>some</em>one clicked on the wrong line.</p><p>Note that even within the LucasArts canon, keyboard shortcuts and interaction models changed quite a bit over time, so take a minute to familiarize yourself with the controls for your current game. You will be much happier if you can devote your mental energy to solving the puzzles if you don’t have to get distracted by the fundamentals of moving your character around the screen.</p><p>One other thing to look out for is how the game communicates whether you are in a cut scene. Specifically, a <em>cut scene</em> is when the game takes complete control of the action and you can only observe, as if you were watching a movie. (Often this communicated in a subtle way, like the mouse pointer no longer becoming visible.) It is important to understand this game element and communicate it to your child or else you might find yourselves hopelessly trying to change an outcome over which you have no control. Moreover, sometimes the transition out of a cut scene may be subtle and is part of the puzzle itself!</p><h4>Don’t Expect to be Off-Duty</h4><p>I will admit that, selfishly, I initially thought that maybe if I could get my daughter hooked on <em>The Secret of Monkey Island</em>, I could use this as an alternative to Netflix-as-a-babysitter to get a little more of my own work done. In my case, it was quite the opposite, though I expect every kid is different in this regard. (Though as we have played more of these games, she has become much more independent.)</p><p>Although my daughter seemed to genuinely enjoy playing, she would only play <em>with me</em>. Even if I just needed to take a break to go to the bathroom, she was unwilling to make a single move until I came back. Although this was antithetical to the sense of independence I had hoped to instill, I got over it, and honestly, it was a lot of fun. As I mentioned previously, some puzzles would have been impossible for her, so it was for the best that I was there to redirect and support her, helping build her confidence. Best of all, I got to celebrate with her every time she had a “Eureka!” moment, which is reliably my favorite part of whatever game we are playing.</p><h3>Conclusion</h3><p>If you made it all the way through this article, you deserve a break. Maybe you should go play a game?</p><p>More seriously, I hope I have given you an appreciation for adventure games as well as the information you need to decide if this is something you want to introduce to you and/or your kids. As noted, age-appropriate adventure games that are engaging and challenging are hard to come by, which is one of the many reasons <em>The Secret of Monkey Island</em> stands out among the rest.</p><p>I hope to write about more games that fit the bill in the future, though if you can’t wait, I would also look into the sequels, <a href="https://www.gog.com/game/monkey_island_2_special_edition_lechucks_revenge"><em>Monkey Island 2: LeChuck’s Revenge</em></a> as well as <a href="https://www.gog.com/game/the_curse_of_monkey_island"><em>The Curse of Monkey Island</em></a>. I would say that both are more difficult than SoMI, but fortunately, both also include an “easy mode” that preserves the full story while omitting some of the more challenging puzzles, which may be the right option for your budding gamer. If you can’t be bothered to mess around with installing old games on desktops, <a href="https://www.gog.com/game/broken_age"><em>Broken Age</em></a> is a wonderful game from LucasArts alum Tim Schafer that I have also been enjoying with my daughter. As it is a “modern” game (released in 2014), it is also available on tablets and game consoles, so we have been enjoying it on the iPad, which feels a bit more casual.</p><h3>PostScript: Why Did I Write This?</h3><p>Putting in the time to research and compose an 8,000+ word article that tries to convince parents to go through a ton of effort to set up a 30-year-old video game for their small children to play [and to play it themselves] is arguably a strange way to choose to spend one’s nights and weekends. It seems unlikely that many people will make it all the way through this piece, let alone install <em>The Secret of Monkey Island</em>, so why do it?</p><p>More than anything, I deeply enjoyed playing SoMI with my daughter, and I want others to have that experience with their own children. If you are someone who isn’t particularly comfortable with computers or video games, then investing the time to set up SoMI is likely a tall order. I get it. Nevertheless, I did my best to add context, enumerate the benefits, and provide detailed instructions to hopefully inject enough activation energy for you to give SoMI a shot and increase the chances that the overall experience is a positive one for you and your child.</p><p>Further, there are so many people out there shilling products that “teach <a href="https://en.wikipedia.org/wiki/Science,_technology,_engineering,_and_mathematics">STEM</a>.” Living in Silicon Valley, we hear about [and give into] a lot of them (<a href="https://amzn.to/2OEhJjm">we own our share of Osmo products</a>, for example), but instead of “teaching kids to code,” how about we give them a reason to code in the first place? Many, many people the software industry started programming because they played games and then wanted to build one of their own. My hypothesis is that adventure games, at least age-appropriate ones, will have a much wider appeal to young audiences than action games. If we can leverage that to spark a genuine interest in technology, we win.</p><p>Finally, I wish there were more games like SoMI, particularly for kids. In writing this, I hope to hear from folks with recommendations. I have discovered a lot of great adventure games through this journey, but most were designed with adolescent males in mind, so I have had trouble figuring out what to play next with my daughter. Ideally, there would be an “adventure game renaissance” of sorts and we would see more games like this, though I’m skeptical the economics work out. Though <a href="https://thenextweb.com/gaming/2018/05/15/monkey-island-fans-are-begging-disney-to-sell-the-rights-back-to-its-creator/">if we could just get Disney to sell the rights to Monkey Island back to Ron Gilbert</a>, then perhaps that’s all we really need.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff20bfd79c88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I ended up writing opensnoop in pure C using eBPF]]></title>
            <link>https://mbolin.medium.com/how-i-ended-up-writing-opensnoop-in-pure-c-using-ebpf-dedb81c25e6e?source=rss-588a10a26d5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/dedb81c25e6e</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[performance-engineering]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Michael Bolin]]></dc:creator>
            <pubDate>Sat, 29 Sep 2018 18:55:38 GMT</pubDate>
            <atom:updated>2018-09-29T18:55:38.253Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>tl;dr</strong> <em>I rewrote </em><a href="http://www.brendangregg.com/blog/2014-07-25/opensnoop-for-linux.html"><em>opensnoop</em></a><em> in </em><a href="https://github.com/bolinfest/opensnoop-native/blob/master/opensnoop.c"><em>C using eBPF</em></a><em>. This is the story of how I broke through layers of abstraction to figure out how to do it.</em></p><p>Recently, I was trying to debug a performance issue in <a href="https://github.com/facebookexperimental/">Eden</a> with a coworker. He started spitting out things to try: “run vmstat 1,” “check iostat,” “ok, now mpstat,” etc. If you know what all of those are and how to interpret their output, then you are way ahead of me. At the time, I unquestioningly typed in the commands and stared blankly at the output, quietly nodding, waiting for my coworker to interpret the numbers and tell me what to run next.</p><p>Shortly thereafter, I asked myself: “Was I out playing Frisbee in the quad the day they went over Linux performance tools in school? How am I supposed to learn this stuff? Is there one book I can read to learn what I need to know? Why are there so many commands I need to learn?” I asked around and a friend told me to look at <a href="http://brendangregg.com/">http://brendangregg.com/</a> to learn about performance engineering.</p><p>When you first visit Brendan’s site, it’s pretty overwhelming. With his long list of articles, talks, books, and of course, software, it’s clear that Brendan is incredibly prolific and conscientious about sharing what he knows with everyone. When it comes to learning about computing performance, I haven’t found anything that is a close second to Brendan or his web site.</p><p>So where to begin? On his <a href="http://brendangregg.com/linuxperf.html">Linux Performance</a> page, the section on talks is organized “[i]n rough order of recommended viewing or difficulty, intro to more advanced,” so I figured I would start at the top and work my way through the list. This led me down an interesting path from which I learned about a number of new topics and was challenged to find answers to questions that couldn’t be easily answered by Google or Stack Overflow. In writing this up, I hope it is helpful in teaching about eBPF, problem solving, or both.</p><h3>Part 1: Learning Begins</h3><p>I started watching Brendan’s videos with the goal of learning how to debug performance issues (primarily on Linux). Historically, I have not been as deep in the stack as some of my colleagues, so I know I have some things to work on there and I am often eager to read quality material to help me improve on that front. I also knew enough to know that there are a lot of tools out there, and I only had a passing familiarity with a small handful of them.</p><p>Similar to when I was first overwhelmed when learning Git, one of my first thoughts was: “OK, what are the four or five commands I need to know so I can get through the day?” Suffice to say, I was a bit overwhelmed when I first encountered <a href="http://www.brendangregg.com/Perf/linux_perf_tools_full.svg">Brendan’s diagram of performance tools to audit the performance of various parts of the system</a>: it was clear there weren’t going to be any shortcuts here.</p><p>As I watched the videos, I was introduced to <a href="http://www.brendangregg.com/usemethod.html">the Utilitization Saturation and Errors (USE) Method</a>, which is a methodology for analyzing the performance of a system. The general idea is to create a map of your system, dividing it into functional components (or “resources”) and examine each one individually for utilization, saturation, and errors. The idea is that by examining the components in isolation, you can find the true bottleneck. Because each resource often has its own tool for checking its vital signs, you end up having to learn a lot of tools.</p><p>Fortunately, in addition to Brendan’s comprehensive checklists for various platforms (<a href="http://www.brendangregg.com/USEmethod/use-linux.html">Linux</a>, <a href="http://www.brendangregg.com/USEmethod/use-solaris.html">Solaris</a>, <a href="http://www.brendangregg.com/USEmethod/use-macosx.html">macOS</a>, <a href="http://www.brendangregg.com/USEmethod/use-freebsd.html">FreeBSD</a>), he has mercifully put together some simpler lists for those of us with shorter attention spans, such as <a href="http://www.brendangregg.com/blog/2016-05-04/srecon2016-perf-checklists-for-sres.html">Linux Perf Analysis in 60s</a>, which is a “Top 10” list of commands to run from the command line. This seemed like a good place to start to build up some basic competency before branching out further.</p><p>Unsurprisingly, Brendan has created and released a number of performance analysis tools. One that is incredibly easy to explain and demos well (so it is frequently featured in his talks) is <a href="http://www.brendangregg.com/blog/2014-07-25/opensnoop-for-linux.html">opensnoop</a>, which is a tool that prints out every open(2) call on the system as it happens. (I believe it is roughly equal to doing strace -p 1 -f -e openat, though opensnoop is much, much more efficient. Also, please don’t run that strace -p 1 command because I’m told it may degrade your sytem irreparably!) Also, in case you are afraid open(2) had “most favored syscall status,” Brendan has also created <a href="http://www.brendangregg.com/blog/2014-07-28/execsnoop-for-linux.html">execsnoop</a> and <a href="http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html">iosnoop</a>, while others have contributed <a href="https://github.com/iovisor/bcc/tree/master/tools">some “snoops” of their own</a>.</p><p>Another topic that comes up in a number of Brendan’s talks is eBPF, which is <a href="https://lwn.net/Articles/740157/">a sorta new/sorta old technology in Linux that seems a little crazy to me</a> in that it lets you upload code from user space to the kernel and then runs your program every time a designated code path in the kernel is taken. You can then communicate with your program from user space via a file descriptor.</p><p>When I first read how this worked, it seemed crazy to me because in my mind, the kernel is this sacred bastion that needs to be incredibly secure and performant and yet we’re going to let end users send in whatever they want and run it in there? As you might expect, you can’t send “arbitrary code,” but a program written in eBPF bytecode, which is a restricted instruction set. There are also limits on the size of the program and other constraints to ensure it is statically analyzable. The kernel verifies that there are no loops and that your code will terminate before it attaches the program to the specified codepath. In this way, it imposes an upper bound on how much “drag” your eBPF program can add on the system.</p><p>eBPF is what enables Brendan to <a href="http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html">build performant tracing tools for Linux</a>. In the case of opensnoop, he registers an eBPF program that is “attached” to the open(2) syscall and logs each one to an <a href="https://prototype-kernel.readthedocs.io/en/latest/bpf/ebpf_maps.html">“eBPF map.”</a> The userland code is able to read (or write!) the contents of the eBPF map via a file descriptor. In the case of opensnoop, it reads the data from the map and formats it so it can be displayed to the user on the console.</p><p>Although I meant to learn about debugging performance issues, this eBPF stuff seemed pretty wild to me, so I decided to learn more about how it worked. I started reading other articles (<a href="https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/">this page appears to aspire to be the clearinghouse on BPF reading material</a>, though to save you some time, <a href="https://cilium.readthedocs.io/en/v1.2/bpf/">this guide</a> was one of the most information-rich pages I found on eBPF).</p><p>If you end up doing Google searches to learn more about things like eBPF and opensnoop (and of course, <a href="https://jvns.ca/categories/strace/!">strace</a>), another name you’re likely to see is <a href="https://jvns.ca/">Julia Evans</a>. Like Brendan, she has produced a tremendous amount of great content. In particular, her <a href="https://jvns.ca/zines/">zines</a> are quick reads and are incredibly approachable, though perhaps my favorite thing was her <a href="https://youtu.be/0IQlpFWTFbM">“You can be a kernel hacker!”</a> talk where she showed how to create a kernel module that intercepts all of open(2) calls on the system, checks if the specified path ends in .mp3, and if so, opens Rick Astley - Never Gonna Give You Up.mp3 instead. Brilliant!</p><p>At this point, I decided I had done enough high-level background research: it was time to get into the mechanics of how this stuff worked!</p><h3>Part 2: Initial Foray into eBPF</h3><p>After spending a bit of time with Brendan and Julia’s material, what seemed like an obvious question to me was: “Why not build a version of strace in Rust that uses eBPF?” From what I had read, it sounded like eBPF could help with the performance issues of strace (which uses ptrace(2) under the hood) while also satisfying my personal (yet admittedly misguided) desire to replace C code with Rust whenever I encounter it. My gut feeling was that it should be possible to precompute the eBPF bytecode such that this new tracer written in pure Rust could start up very fast. This seemed like a worthwhile project that would give me an excuse to get my hands dirty with eBPF. I could call it <strong>rstrace</strong>!</p><p>Because I suspected this would not be an original idea, instead of just trying to <em>do</em> it, I decided to see if someone else had already done it. I found a handful of projects/articles that were related:</p><p>I thought it was strange that both vltrace and the opensnoop example in the bcc crate chose to compile the eBPF bytecode from C source on the fly. As a number of folks had noted in various articles I read, this required linking in a chunk of the LLVM toolchain and exercising it at runtime, which seemed inelegant/heavyweight. For example, from Geoffroy’s blog post:</p><blockquote><em>“Unfortunately, those tools make a tradeoff that’s slightly annoying for me: they require installing BCC, which requires Python, LLVM and the complete Linux sources, on the target machines. It might be possible to precompile the programs though, but it does not look like it’s a common use case with BCC.”</em></blockquote><p>I also thought it was strange that various attributes in Geoffroy’s example were used to ensure code was written into specific sections of the ELF binary. My gut feeling was that this was <em>accidental</em> rather than <em>essential</em> complexity, but I wanted to be sure.</p><p>Because Geoffroy’s article was inline with what I wanted to do and contained a lot of useful exposition, I decided to try to understand it in detail before I wrote any code of my own. Although his post contained a bunch of sample code [which initially turned out to contain numerous typos due to HTML escaping errors], it did not have an associated Git repo to clone to reproduce the results. I decided that my first order of business would be to reproduce the results of the blog post locally. (Ultimately, I would make my code available at <a href="https://github.com/bolinfest/rust-ebpf-demo/">https://github.com/bolinfest/rust-ebpf-demo/</a>, then I <a href="https://twitter.com/bolinfest/status/1039298420240863232">notified Geoffroy</a> about my work and the typos, and <a href="https://twitter.com/gcouprie/status/1039424926615330817">he fixed everything</a>.)</p><p>I believe I used different versions of rustc and LLVM than the Geoffroy because (1) <a href="https://twitter.com/bolinfest/status/1039550675850416128">I couldn’t compile his Rust code</a>, and (2) my version of the eBPF program was only 14 instructions whereas his was 36. I assumed that I was using a newer version of the LLVM toolchain with improved optimizations for eBPF codegen, but I wanted to be sure they were functionally equivalent. To that end, I decided to go through the assembly line-by-line to ensure I understood everything.</p><p>I spent a lot of time reading the <a href="https://github.com/iovisor/bpf-docs/blob/b5ac15bfefc25fb13b4178a3fed2932fc2a795f1/eBPF.md">“Unofficial eBPF spec”</a> on GitHub to learn about the different opcodes and their arguments. I <a href="https://github.com/bolinfest/rust-ebpf-demo/#annotated-bytecode">updated my README with a line-by-line annotation of the eBPF program</a>. As someone who has never bothered to learn x86[_64] assembly, this was quite tedious, but also enlightening. (This knowledge would also prove helpful down the road.) Ultimately, I discovered what the program is doing is pretty simple:</p><ul><li>First, it writes the C string &quot;hello from rust\n&quot; to the stack, which took two “wide” writes of 8 bytes each (for &quot;om rust\n&quot; and &quot;hello fr&quot;, respectively), plus a write of 1 byte for the trailing &#39;\0&#39; because it is a C string.</li><li>The address of the C string is written to r1 and its length (17, which includes the NUL byte) is written to r2.</li><li>With the appropriate arguments for bpf_trace_printk() written to r1 and r2 (<a href="https://cilium.readthedocs.io/en/v1.2/bpf/#instruction-set">as per the BPF calling convention</a>), the function (which corresponds to the number 6 because of <a href="https://elixir.bootlin.com/linux/v4.7/source/include/uapi/linux/bpf.h#L153">its position in the bpf_func_id enum</a>) is called.</li><li>Finally, the exit code (0) is written to r0 and the program exits.</li></ul><p>Reading through the bytecode from Geoffroy’s blog post, it appears that it writes &quot;hello from rust\n\0&quot; to the stack one byte at a time, which explains why it takes so many more instructions.</p><p>Although I had now explained to myself how the eBPF program worked, understanding how to run it was another matter. Geoffroy’s blog post mentioned <a href="https://kinvolk.io/blog/2017/09/an-update-on-gobpf---elf-loading-uprobes-more-program-types/">a blog post on gobpf</a> that had code that could be used to run the hello.o ELF binary. Indeed, I found the code and <a href="https://github.com/bolinfest/rust-ebpf-demo/blob/9ce6106649770ab7c5097513e744977715deaad6/go/main.go">slightly modified it for myself</a> to run hello.o successfully. I spent a bit of time looking at <a href="https://github.com/iovisor/gobpf/blob/2e314be67b1854ad226f012f08a984e0e89b6da9/elf/elf.go">elf.go from the gobpf project</a> and figured out how it loaded data based on ELF section names to make calls into functions like bpf_prog_load(), but I still wasn’t convinced that packaging an eBPF program this way was strictly <em>necessary</em>. I would try to write my own standalone eBPF program to find out for sure.</p><h3>Part 3: Standalone eBPF program without libbpf</h3><p>My first thought was to write a standalone C program that would load and run the bpf_trace_printk() eBPF program from Geoffroy’s blog post. I figured this would help me answer the question of whether I needed the labeled ELF sections mentioned in the blog post and whether I would have to port a bunch of the Go code in <a href="https://github.com/iovisor/gobpf/tree/master/elf">https://github.com/iovisor/gobpf/tree/master/elf</a> to Rust to build rstrace.</p><p>After reading bpf(2), I thought this program was going to be simple to implement. I assumed all I would have to do is:</p><ul><li>Define the eBPF program as an array of BPF bytecodes.</li><li>Call bpf() with cmd=BPF_PROG_LOAD and attr.type=BPF_PROG_TYPE_KPROBE to load the eBPF program and attach it as a kprobe. (kprobe is short for “kernel probe,” which is a predefined location in the kernel where an eBPF program can be attached, such as the entry point to a system call.)</li></ul><p>These assumptions were not quite correct.</p><h3>Loading the Program</h3><p>My first surprise was that despite doing #include &lt;linux/bpf.h&gt;, there was no bpf() function for me to call. Instead, I had to use syscall(2) to call bpf(2) even though I expected there to be a bpf() function available since everything else I ever looked up in <a href="https://en.wikipedia.org/wiki/Man_page#Manual_sections">section 2 of the man pages</a> had a function of same name:</p><pre>syscall(__NR_bpf, BPF_PROG_LOAD, &amp;attr, sizeof(attr));</pre><p>I had never used syscall(2) before, but I figured out it was what I needed by looking at the source for <a href="https://github.com/iovisor/bcc/blob/master/src/cc/libbpf.c">libbpf.c</a>. As I would discover, this would not be my only use of syscall(2) for this program. (My hypothesis is that there is no C wrapper function for bpf() because it came along later and could risk colliding with a user-defined function named bpf().)</p><p>My next step was to generate the eBPF program. I didn’t have to produce an array of binary bytecodes as I expected, but an array of struct bpf_insn objects. Honestly, this was easier than producing binary values because I didn’t have to worry about the <a href="https://cilium.readthedocs.io/en/v1.2/bpf/#llvm">endianness issues</a> I read about. Using BCC, <a href="https://github.com/bolinfest/rust-ebpf-demo/commit/914b24a8c81189d8e278e2e1ce1df53c583f1a5d#diff-3552c49fbb4741b6095394bf1b4086a7">I wrote a Python program to dump the eBPF bytecode from the C version of my program</a> and print it out as an array of struct bpf_insn objects:</p><pre><strong>import</strong> struct<br><strong>from</strong> bcc <strong>import</strong> ArgString, BPF<br><br><em># define BPF program</em><br>bpf_text <strong>=</strong> &quot;&quot;&quot;<br>#include &lt;uapi/linux/ptrace.h&gt;<br>#include &lt;uapi/linux/limits.h&gt;<br>#include &lt;linux/sched.h&gt;<br><br>int trace_entry(struct pt_regs *ctx, int dfd, const char __user *filename)<br>{<br>    bpf_trace_printk(&quot;hello from rust\\n&quot;);<br>    return 0;<br>};<br>&quot;&quot;&quot;<br><br><strong>def</strong> <strong>parse_instruction</strong>(instruction):<br>    <em># u8</em><br>    opcode <strong>=</strong> struct<strong>.</strong>unpack(&quot;B&quot;, instruction[0])[0]<br><br>    src_and_dst <strong>=</strong> struct<strong>.</strong>unpack(&quot;B&quot;, instruction[1])[0]<br>    <em># The low-order nibble is dst.</em><br>    dst_reg <strong>=</strong> src_and_dst <strong>&amp;</strong> 0x0F<br>    <em># The high-order nibble is src.</em><br>    src_reg <strong>=</strong> (src_and_dst <strong>&amp;</strong> 0xF0) <strong>&gt;&gt;</strong> 4<br><br>    <em># s16</em><br>    offset <strong>=</strong> struct<strong>.</strong>unpack(&quot;h&quot;, instruction[2:4])[0]<br>    <em># s32</em><br>    imm <strong>=</strong> struct<strong>.</strong>unpack(&quot;i&quot;, instruction[4:8])[0]<br>    <strong>return</strong> (opcode, dst_reg, src_reg, offset, imm)<br><br><br>bpf_insn_template <strong>=</strong> &quot;&quot;&quot;\<br>  ((struct bpf_insn) {<br>      .code    = 0x%x,<br>      .dst_reg = BPF_REG_%d,<br>      .src_reg = BPF_REG_%d,<br>      .off     = %d,<br>      .imm     = %d,<br>  }),&quot;&quot;&quot;<br><br><br><strong>def</strong> <strong>print_bpf_insn_struct</strong>(bytecode):<br>    &quot;&quot;&quot;Prints the eBPF bytecode as the equivalent bpf_insn[] in C.&quot;&quot;&quot;<br>    <strong>print</strong>(&quot;struct bpf_insn prog[] = {&quot;)<br>    <strong>for</strong> b_offset <strong>in</strong> xrange(0, len(bytecode), 8):<br>        instruction <strong>=</strong> bytecode[b_offset : b_offset <strong>+</strong> 8]<br>        opcode, dst_reg, src_reg, offset, imm <strong>=</strong> parse_instruction(instruction)<br>        <strong>print</strong>(bpf_insn_template <strong>%</strong> (opcode, dst_reg, src_reg, offset, imm))<br>    <strong>print</strong>(&quot;};&quot;)<br><br><br>bpf <strong>=</strong> BPF(text<strong>=</strong>bpf_text)<br>bytecode <strong>=</strong> bpf<strong>.</strong>dump_func(&quot;trace_entry&quot;)<br>print_bpf_insn_struct(bytecode)</pre><p>Because this was a one-time conversion, I copied the generated C code into my program.</p><p>Incidentally, bpf_insn is the first struct I had ever seen that declared fields as nibbles. (For the uninitiated, “nibble” is a real term that is means “4 bits,” so named because it is “half a byte,” get it?) Here is the declaration of the struct:</p><pre><strong>struct</strong> bpf_insn {<br>  __u8 code;       <em>// opcode<br></em>  __u8 dst_reg<strong>:</strong>4;  <em>// dest register<br></em>  __u8 src_reg<strong>:</strong>4;  <em>// source register<br></em>  __s16 off;       <em>// signed offset<br></em>  __s32 imm;       <em>// signed immediate constant<br></em>};</pre><p>Note how there is a :4 after the second two fields, indicating that each field contributes 4 bits. Because the maximum value of 4 (unsigned) bits is 1111 in binary (15 in decimal), the compiler will complain if you try to assign a value greater than 15 to either dst_reg or src_reg. To verify the bit representation of the struct, I wrote a small program:</p><pre><strong>#include &lt;linux/bpf.h&gt;<br>#include &lt;stdio.h&gt;<br></strong><br><strong>int</strong> <strong>main</strong>(<strong>int</strong> argc, <strong>char</strong> <strong>**</strong>argv) {<br>  <strong>struct</strong> bpf_insn instruction <strong>=</strong> (<strong>struct</strong> bpf_insn){<br>      .code <strong>=</strong> 0x18,<br>      .dst_reg <strong>=</strong> 0x0e,<br>      .src_reg <strong>=</strong> 0x07,<br>      .off <strong>=</strong> 0xcafe,<br>      .imm <strong>=</strong> 0x72206d6f,<br>  };<br>  fwrite(<strong>&amp;</strong>instruction, <strong>sizeof</strong>(instruction), 1, stdout);<br>}</pre><p>When I compile and run the program on my system, piping the output to xxd, I get:</p><pre>00000000: 187e feca 6f6d 2072                      ....om r</pre><p>As you can see, the values in the struct are written as little-endian based on the way the imm field is serialized. What is perhaps less obvious is that dst_reg is the low-order nibble and src_reg is the high-order nibble in the second byte written to the output.</p><p>Incidentally, you are not limited to nibble boundaries when declaring structs (or even requiring all sub-byte fields to be the same type or signed-ness) as I discovered I could also do:</p><pre><strong>struct</strong> example {<br>  <strong>char</strong> f1 <strong>:</strong> 3;<br>  <strong>char</strong> f2 <strong>:</strong> 3;<br>  __u8 f3 <strong>:</strong> 2;<br>};<br><strong>struct</strong> example ex <strong>=</strong> (<strong>struct</strong> example){<br>    .f1 <strong>=</strong> <strong>-</strong>1,<br>    .f2 <strong>=</strong> 2,<br>    .f3 <strong>=</strong> 3,<br>};<br>fwrite(<strong>&amp;</strong>ex, <strong>sizeof</strong>(ex), 1, stdout);</pre><p>Anyway, back to the problem at hand…</p><p>Things got interesting/frustrating here because my call to bpf(2) was failing with EINVAL. I wasn’t sure what I could be doing wrong as the only relevant mention of EINVAL in the man page was:</p><blockquote><em>For </em><em>BPF_PROG_LOAD, indicates an attempt to load an invalid program. eBPF programs can be deemed invalid due to unrecognized instructions, the use of reserved fields, jumps out of range, infinite loops or calls of unknown functions.</em></blockquote><p>I tried running my program under gdb to make sure all of the values I was passing to bpf(2) were what I expected. I discovered that the way I calculated the instruction count was incorrect, so I fixed that. Again, I looked at libbpf.c to see how they implemented bpf_prog_load() and I noticed that they <a href="https://github.com/iovisor/bcc/blob/89aefcc9a89216da3558b891e0d6cb235b0e07a0/src/cc/libbpf.c">zeroed the bpf_attr before writing to it</a>, so I added code to do that, too. These were issues that had to be fixed, but I was still getting EINVAL.</p><p>I tried looking in various places for a more detailed error message. I ran dmesg -T | tail, but there was nothing there. Looking at libbpf.c further, <a href="https://github.com/iovisor/bcc/blob/89aefcc9a89216da3558b891e0d6cb235b0e07a0/src/cc/libbpf.c#L599-L608">I discovered I should also check the log_buf I passed into bpf_prog_load()</a>, as an error message could be printed there, but that also came up empty.</p><p>I was pretty frustrated at this point because my implementation of bpf_prog_load() was basically <a href="https://github.com/mkerrisk/man-pages/blob/man-pages-4.15/man2/bpf.2#L724-L742">taken straight from the man page</a>:</p><pre><strong>char</strong> bpf_log_buf[LOG_BUF_SIZE];<br><br><strong>int</strong> <strong>bpf_prog_load</strong>(<strong>enum</strong> bpf_prog_type type,<br>                  <strong>const</strong> <strong>struct</strong> bpf_insn <strong>*</strong>insns, <strong>int</strong> insn_cnt,<br>                  <strong>const</strong> <strong>char</strong> <strong>*</strong>license)<br>{<br>  <strong>union</strong> bpf_attr attr <strong>=</strong> {<br>    .prog_type <strong>=</strong> type,<br>    .insns     <strong>=</strong> ptr_to_u64(insns),<br>    .insn_cnt  <strong>=</strong> insn_cnt,<br>    .license   <strong>=</strong> ptr_to_u64(license),<br>    .log_buf   <strong>=</strong> ptr_to_u64(bpf_log_buf),<br>    .log_size  <strong>=</strong> LOG_BUF_SIZE,<br>    .log_level <strong>=</strong> 1,<br>  };<br><br>  <strong>return</strong> bpf(BPF_PROG_LOAD, <strong>&amp;</strong>attr, <strong>sizeof</strong>(attr));<br>}</pre><p>Even though I was using gdb, <a href="https://stackoverflow.com/questions/5999205/cannot-step-into-system-call-source-code">I couldn’t step into a system call</a>, so I had only one other option to find the source of the EINVAL: <strong>to read the Linux source code</strong>.</p><p>For whatever reason, I have never looked at the Linux source code before (mostly GPL paranoia, if I’m honest about it). However, <a href="https://github.com/torvalds/linux/blob/v4.15/kernel/bpf/syscall.c#L1140-L1142">my answer was right there in the code</a>, plain as day:</p><pre><strong>if</strong> (type <strong>==</strong> BPF_PROG_TYPE_KPROBE <strong>&amp;&amp;</strong><br>        attr<strong>-&gt;</strong>kern_version <strong>!=</strong> LINUX_VERSION_CODE)<br>        <strong>return</strong> <strong>-</strong>EINVAL;</pre><p>As you can see, the sample implementation of bpf_prog_load() from the man page makes no mention of kern_version, though clearly the Linux kernel will reject your eBPF program with EINVAL if prog_type=BPF_PROG_TYPE_KPROBE and kern_version != LINUX_VERSION_CODE. I then searched the man page for kern_version, and there is indeed a note buried in there:</p><pre><strong>union</strong> bpf_attr {<br>  <strong>struct</strong> {    <em>/* Used by BPF_MAP_CREATE */</em><br>    __u32   map_type;<br>    __u32   key_size;    <em>/* size of key in bytes */</em><br>    __u32   value_size;  <em>/* size of value in bytes */</em><br>    __u32   max_entries; <em>/* maximum number of entries<br>                            in a map */</em><br>  };<br><br>  <strong>struct</strong> {    <em>/* Used by BPF_MAP_*_ELEM and BPF_MAP_GET_NEXT_KEY<br>                 commands */</em><br>    __u32   map_fd;<br>    __aligned_u64 key;<br>    <strong>union</strong> {<br>      __aligned_u64 value;<br>      __aligned_u64 next_key;<br>    };<br>    __u64   flags;<br>  };<br><br>  <strong>struct</strong> {    <em>/* Used by BPF_PROG_LOAD */</em><br>    __u32   prog_type;<br>    __u32   insn_cnt;<br>    __aligned_u64 insns;      <em>/* &#39;const struct bpf_insn *&#39; */</em><br>    __aligned_u64 license;    <em>/* &#39;const char *&#39; */</em><br>    __u32   log_level;        <em>/* verbosity level of verifier */</em><br>    __u32   log_size;         <em>/* size of user buffer */</em><br>    __aligned_u64 log_buf;    <em>/* user supplied &#39;char *&#39; buffer */</em><br>    __u32   kern_version;<br>                        <em>/* checked when prog_type=kprobe<br>                           (since Linux 4.1) */</em><br>  };<br>} __attribute__((aligned(8)));</pre><p>I added #include &lt;linux/version.h&gt; to the top of my file and set kern_version to LINUX_VERSION_CODE on my bpf_attr and now my bpf_prog_load() function succeeded!</p><p>That said, I had misgivings about hardcoding LINUX_VERSION_CODE in my source because that meant I could not give the binary version of my program to someone on a slightly different version of the kernel because the version number from my machine would be hardcoded into the binary, causing the version check in a machine with a different kernel to fail. I thought it would be better to calculate the version number dynamically using uname(2); however, I tried running the following on my machine:</p><pre><strong>#include &lt;linux/version.h&gt;<br>#include &lt;stdio.h&gt;<br>#include &lt;stdlib.h&gt;<br>#include &lt;string.h&gt;<br>#include &lt;sys/utsname.h&gt;<br></strong><br><strong>int</strong> <strong>main</strong>(<strong>int</strong> argc, <strong>char</strong> <strong>**</strong>argv) {<br>  <strong>struct</strong> utsname buf;<br>  <strong>if</strong> (uname(<strong>&amp;</strong>buf) <strong>&lt;</strong> 0) {<br>    perror(&quot;Error calling uname()&quot;);<br>    <strong>return</strong> 1;<br>  }<br><br>  printf(&quot;utsname.release: %s\n&quot;, buf.release);<br><br>  <strong>int</strong> version <strong>=</strong> atoi(buf.release);<br>  <strong>char*</strong> firstDot <strong>=</strong> strstr(buf.release, &quot;.&quot;);<br>  <strong>int</strong> patchlevel <strong>=</strong> atoi(firstDot <strong>+</strong> 1);<br>  <strong>char*</strong> secondDot <strong>=</strong> strstr(firstDot <strong>+</strong> 1, &quot;.&quot;);<br>  <strong>int</strong> sublevel <strong>=</strong> atoi(secondDot <strong>+</strong> 1);<br><br>  printf(&quot;LINUX_VERSION_CODE: %d\n&quot;, LINUX_VERSION_CODE);<br>  printf(&quot;KERNEL_VERSION(%d, %d, %d): %d\n&quot;, version, patchlevel, sublevel,<br>         KERNEL_VERSION(version, patchlevel, sublevel));<br><br>  <strong>return</strong> 0;<br>}</pre><p>When I ran it on my machine (which reports 4.15.0-33-generic when I run uname -r), I got:</p><pre>utsname.release: 4.15.0-33-generic<br>LINUX_VERSION_CODE: 266002<br>KERNEL_VERSION(4, 15, 0): 265984</pre><p>In other words, calculating the Linux version based on uname(2) yields a number that does not match LINUX_VERSION_CODE! It seemed I would have to hardcode it. (Incidentally, BCC appears to support a <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/docs/reference_guide.md#2-kernel-version-overriding">BCC_LINUX_VERSION_CODE environment variable</a> to help with this, but I don’t believe it would help with the uname/LINUX_VERSION_CODE mismatch that I was seeing.)</p><h3>Attaching the kprobe</h3><p>Unfortunately, my program did not actually do anything because I needed to attach a kprobe. Again, I took a look at <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/cc/libbpf.c#L820-L872">the implementation of bpf_attach_kprobe() in libbpf.c</a> to see what was going on: it was not a simple call to bpf(2) as I thought. Because I wanted to avoid linking in libbpf because I (erroneously! see next section) thought it would pull in LLVM, <a href="https://github.com/bolinfest/rust-ebpf-demo/commit/55e880d04329c25d1271a44e9c9d147c1c04ba2d#diff-3552c49fbb4741b6095394bf1b4086a7">I decided to copy the minimal subset of libbpf.c that I needed directly into my program</a>. (I also had to lift some code from <a href="https://github.com/iovisor/bcc/blob/27e7aeab5d7b9f0f4259fb0f996274af0521243f/src/python/bcc/__init__.py#L575-L598">the attach_kprobe() function in the Python wrapper</a> to figure out what arguments to pass to bpf_attach_kprobe() in C.)</p><p>I made a few typos in the process, so I finally had the bright idea to modify the Python program I used to generate the eBPF bytecode to attach the kprobe:</p><pre><em># bytecode = bpf.dump_func(&quot;trace_entry&quot;)</em><br><em># print_bpf_insn_struct(bytecode)</em><br>bpf<strong>.</strong>attach_kprobe(event<strong>=</strong>&quot;do_sys_open&quot;, fn_name<strong>=</strong>&quot;trace_entry&quot;)</pre><p>and then run it under strace -s 256. Most of the output was BCC doing its thing, but once I found my bpf(2) call, I could attribute everything thereafter to the bpf.attach_kprobe() call in the Python code:</p><pre>bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=14, insns=0x7f4d928737d0, license=&quot;GPL&quot;, log_level=0, log_size=0, log_buf=0, kern_version=266002, prog_flags=0, ...}, 72) = 3<br>openat(AT_FDCWD, &quot;/sys/bus/event_source/devices/kprobe/type&quot;, O_RDONLY) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, &quot;/sys/bus/event_source/devices/kprobe/format/retprobe&quot;, O_RDONLY) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, &quot;/sys/kernel/debug/tracing/kprobe_events&quot;, O_WRONLY|O_APPEND) = 4<br>getpid()                                = 21014<br>write(4, &quot;p:kprobes/p_do_sys_open_bcc_21014 do_sys_open&quot;, 45) = 45<br>close(4)                                = 0<br>openat(AT_FDCWD, &quot;/sys/kernel/debug/tracing/events/kprobes/p_do_sys_open_bcc_21014/id&quot;, O_RDONLY) = 4<br>read(4, &quot;3987\n&quot;, 4096)                 = 5<br>close(4)                                = 0<br>perf_event_open({type=PERF_TYPE_TRACEPOINT, size=0 /* PERF_ATTR_SIZE_??? */, config=3987, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 4<br>ioctl(4, PERF_EVENT_IOC_SET_BPF, 0x3)   = 0<br>ioctl(4, PERF_EVENT_IOC_ENABLE, 0)      = 0<br>ioctl(4, PERF_EVENT_IOC_DISABLE, 0)     = 0<br>close(4)                                = 0<br>openat(AT_FDCWD, &quot;/sys/kernel/debug/tracing/kprobe_events&quot;, O_RDONLY) = 4<br>getpid()                                = 21014<br>fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0<br>read(4, &quot;r:kprobes/r_do_sys_open_bcc_529 do_sys_open\nr:kprobes/r_do_sys_open_bcc_536 do_sys_open\nr:kprobes/r_do_sys_open_bcc_694 do_sys_open\nr:kprobes/r_do_sys_open_bcc_726 do_sys_open\nr:kprobes/r_do_sys_open_bcc_828 do_sys_open\nr:kprobes/r_do_sys_open_bcc_26648 do&quot;..., 4096) = 4058<br>read(4, &quot;p:kprobes/r_do_sys_open_bcc_14005 do_sys_open\np:kprobes/p_do_sys_open_bcc_14008 do_sys_open\np:kprobes/r_do_sys_open_bcc_14008 do_sys_open\np:kprobes/p_do_sys_open_bcc_14128 do_sys_open\np:kprobes/r_do_sys_open_bcc_14128 do_sys_open\np:kprobes/p_do_sys_open_bc&quot;..., 4096) = 4086<br>read(4, &quot;p:kprobes/p_do_sys_open_bcc_19790 do_sys_open\nr:kprobes/r_do_sys_open_bcc_19790 do_sys_open\np:kprobes/p_do_sys_open_bcc_20086 do_sys_open\nr:kprobes/r_do_sys_open_bcc_20086 do_sys_open\np:kprobes/p_do_sys_open_bcc_21443 do_sys_open\nr:kprobes/r_do_sys_open_bc&quot;..., 4096) = 4092<br>read(4, &quot;p:kprobes/p_do_sys_open_bcc_30643 do_sys_open\nr:kprobes/r_do_sys_open_bcc_30643 do_sys_open\np:kprobes/p_do_sys_open_bcc_30861 do_sys_open\nr:kprobes/r_do_sys_open_bcc_30861 do_sys_open\np:kprobes/p_do_sys_open_bcc_30866 do_sys_open\nr:kprobes/r_do_sys_open_bc&quot;..., 4096) = 2840<br>close(4)                                = 0<br>openat(AT_FDCWD, &quot;/sys/kernel/debug/tracing/kprobe_events&quot;, O_WRONLY|O_APPEND) = 4<br>getpid()                                = 21014<br>write(4, &quot;-:kprobes/p_do_sys_open_bcc_21014&quot;, 33) = 33<br>close(4)                                = 0</pre><p>(I felt a little sheepish that I didn’t think of trying this earlier because if you scroll far to the right on the first line, you can see that kern_version is set on the bpf_attr passed to bpf(2), which might have saved me a bunch of the time I spent debugging bpf_prog_load().)</p><p>Once I could use strace to see what the Python+BCC version of my program was doing, it was easy to run the C version of my program under strace and verify that it was making the same syscalls. Moreover, one notable advantage of the pure C version of the program was that the strace output was 42 lines compared to the 10000+ lines from the Python+BCC version, making it much easier to read.</p><p>At long last, I had <a href="https://github.com/bolinfest/rust-ebpf-demo/blob/master/c/load-bpf.c">a single C file</a> that ran the equivalent of Geoffroy’s trace_printk() demo! Now it was time to try something more advanced.</p><h3>Part 4: Standalone opensnoop using eBPF in C</h3><p>At this point, I believed rstrace was possible, but if I was going to be successful, I thought I still needed to learn a bit more about the code I would be trying to generate from Rust. To that end, I decided I to try to create something a little more complex than the trace_printk() example. In one of his talks, I believe Brendan mentioned that opensnoop is one of the first performance tools he writes for a new platform, so I thought I would try to do the same. Specifically, I would implement an opensnoop.c with the same design goals as my previous program: a standalone executable that loads and runs eBPF without using the BCC/LLVM toolchain at runtime. It would be functionally equivalent to <a href="https://github.com/iovisor/bcc/blob/f138fea5a9ab279b7347fa6acfd2f53777068b27/tools/opensnoop.py">the version of opensnoop in the BCC repo that leveraged the Python bindings</a>. (I would spend a lot of time with the original opensnoop.py in the course of this project.)</p><p>There were two new challenges in creating opensnoop that proved to be important opportunities for learning this time around:</p><ul><li>Parameterizing the eBPF program based on process id or thread id, as appropriate.</li><li>Using BPF maps.</li></ul><p>I also felt that copypasta’ing code from libbpf.c as I had done in my previous program was a bit gross and that I should #include &lt;bcc/libbpf.h&gt; instead. However, I wanted to be sure that I avoided pulling the LLVM toolchain into the final binary. It turned out that I had a number of incorrect assumptions on this point that are worth explaining.</p><h3>What is libbpf?</h3><p>The main way I played with BCC/BPF was via the Python bindings. In particular, when reading <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/python/bcc/__init__.py">the main Python file</a> (which I spent a lot of time with), I assumed that every call to a method in lib was a call to a method in libbpf. For example, because I saw a call to lib.bpf_module_create_c_from_string() <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/python/bcc/__init__.py#L315">in the Python code</a> and <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/cc/bpf_common.h#L29">a method of same name in bpf_common.h</a>, I assumed that libbpf included all of bpf_common.h, which meant it depended on LLVM. Adding to my confusion was <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/cc/compat/linux/bpf_common.h">another file in the BCC repo named bpf_common.h</a>, but with a completely different API (the latter is a header-only file that defines a bunch of constants). I was very confused.</p><p>When I first started on opensnoop, instead of taking bits of libbpf.c piecemeal, I decided to take it in its entirety and copy over its transitive dependencies and see how big it was. I was surprised to discover that the list of files I ultimately needed to copy was quite small:</p><pre>src/cc/compat/linux/bpf.h<br>src/cc/compat/linux/bpf_common.h<br>src/cc/libbpf.h<br>src/cc/libbpf.c<br>src/cc/perf_reader.h<br>src/cc/perf_reader.c<br>src/cc/setns.h</pre><p>This seemed weird. The src/cc folder in the BCC repo was giant, but libbpf was only a tiny piece of it! I ran nm -g /usr/lib/x86_64-linux-gnu/libbpf.so to confirm that functions like bpf_module_create_c_from_string() were not part of libbpf.</p><p>After quite a bit of digging, I had a new mental model of how the code was organized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/0*z9M4GZX6hUKdup69.png" /></figure><p>This was an important breakthrough because it clarified the difference between libbcc (which is currently <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/cc/api/BPF.cc">src/cc/api/BPF.cc</a> and <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/cc/api/BPFTable.cc">src/cc/api/BPFTable.cc</a> and their transitive dependencies) and libbpf for me. (Personally, I would love to see the repo reorganized such that all of the code for libbpf is in its own directory.) In particular, I was now convinced that I could #include &lt;bcc/libbpf.h&gt; in my standalone opensnoop without pulling in LLVM.</p><h3>Generating the Bytecode</h3><p>In one of Brendan’s talks, he mentioned that one of the nice things the Python bindings for BCC allowed you to do was to dynamically populate a template of the C code for your BPF program at runtime. When I first watched the talk, I had no idea how important it was, but now that I was looking at his opensnoop.py code, I could see the appeal.</p><p>Specifically, opensnoop supports a -p option to filter events by process ID and a -t option to filter events by thread ID. Although this filtering could be done in user space, it is much more efficient to check it in kernel space (i.e., in the BPF program) to avoid writing unnecessary entries to a BPF map. Because the PID/TID filter values are not known to opensnoop until runtime, the BPF program cannot be precompiled. opensnoop works around this by <a href="https://github.com/iovisor/bcc/blob/f138fea5a9ab279b7347fa6acfd2f53777068b27/tools/opensnoop.py#L121-L128">dynamically generating the C code for BCC to compile</a> in Python based on a template that is parameterized by -p and -t. At first glance, this seemed like it could potentially undermine everything I was trying to do: one of my prized design goals was to avoid using BCC at runtime.</p><p>My first potential workaround was to create a static BPF program that could read “parameters” at runtime via a BPF map. I <a href="https://github.com/bolinfest/bcc/commit/b60e496af5f76f3ea43a4dc2df4ed08d2b911285">created a working prototype of this approach</a> and even <a href="https://twitter.com/bolinfest/status/1039620317822345216">tweeted it to Brendan to get his thoughts</a>. He main concern was the extra array lookup each time the kprobe was called. I also felt it was a bit inelegant, so I decided to try to do better.</p><p>My next idea was to try to create a version of the program without any filtering logic and store the array of instructions in my C program. If the user specified -p or -t, I could have some code that inserted extra bytecodes into the array to accommodate the specified filter. Although I think I could have made this work, I didn’t think this was a good approach in the general case because inserting extra instructions at arbitrary positions could throw off existing jump instructions.</p><p>(I also noticed that perf_event_open(2) has a pid_t pid parameter. I thought I could use this to take care of filtering by PID, so I tried hardcoding the value to a specific process on my machine, but it did not have the intended effect. I noticed that attach_kprobe() in the BCC Python bindings hardcodes pid=-1 when it calls into libbpf, whereas attach_uprobe() has a pid parameter that it passes through to libbpf, so perhaps it is ignored for kprobes.)</p><p>Because there were only three versions of the filter, I decided to use BCC to create three versions of the kprobe program: one with no filtering, one that included only events with PID=654321, and one that included only events with TID=123456. I chose canned filter values that were unlikely to appear as imm values in the generated bytecode.</p><p>Next I wrote some <a href="https://github.com/bolinfest/opensnoop-native/blob/5295cb008954bfeb7b169a3494d69e12d75b568c/debug.py#L202-L229">Python code to post-process the bytecode</a>, generating a C function that would write the appropriate array of bpf_insn values to the struct bpf_insn[] passed into the function. The key trick was that the Python code to generate the C code would take the original bytecode as well as a “placeholder” argument. For example, when generating the C function to filter by PID, <a href="https://github.com/bolinfest/opensnoop-native/blob/5295cb008954bfeb7b169a3494d69e12d75b568c/opensnoop.py#L94">the “placeholder” argument to the Python function</a> is:</p><pre>{&quot;param_type&quot;: &quot;int&quot;, &quot;param_name&quot;: &quot;pid&quot;, &quot;imm&quot;: 654321}</pre><p>This means that when the Python code is iterating over the bytecode, if it finds an instruction with an imm equal to 654321, instead of hardcoding the value 654321 for imm in the corresponding struct bpf_insn, it should specify it as the variable pid instead. Also, the function prototype for the generated C function should include int pid as a parameter.</p><p>The function prototypes for the C functions that generate the corresponding BPF programs are as follows (the int fd3 param will be explained in the next section):</p><pre><strong>void</strong> generate_trace_entry(<strong>struct</strong> bpf_insn instructions[], <strong>int</strong> fd3);<br><strong>void</strong> generate_trace_entry_tid(<strong>struct</strong> bpf_insn instructions[],<br>                              <strong>int</strong> tid, <strong>int</strong> fd3);<br><strong>void</strong> generate_trace_entry_pid(<strong>struct</strong> bpf_insn instructions[],<br>                              <strong>int</strong> pid, <strong>int</strong> fd3);</pre><p>The <a href="https://github.com/bolinfest/opensnoop-native/blob/5295cb008954bfeb7b169a3494d69e12d75b568c/opensnoop.py#L105-L118">Python code also inserts some constants into the generated C code</a>:</p><pre>#define MAX_NUM_TRACE_ENTRY_INSTRUCTIONS 35<br>#define NUM_TRACE_ENTRY_INSTRUCTIONS 28<br>#define NUM_TRACE_ENTRY_TID_INSTRUCTIONS 32<br>#define NUM_TRACE_ENTRY_PID_INSTRUCTIONS 35<br>#define NUM_TRACE_RETURN_INSTRUCTIONS 82</pre><p>Each of the generated functions writes a BPF program into instructions (<a href="https://github.com/bolinfest/opensnoop-native/blob/5295cb008954bfeb7b169a3494d69e12d75b568c/opensnoop.c#L333-L344">the generated constants make it possible for the caller to create the instructions[] array with the appropriate size at compile time</a>). The versions of the program that support filtering take tid and pid as parameters, as appropriate, and use them to set imm in the necessary place(s) in the generated BPF program. Although we are still generating the BPF program from a template dynamically at runtime, we are not using BCC at runtime, as it is only needed at build time when generating the C code.</p><p>One thing that I thought was interesting about compiling different versions of the program is that you get the “best” version in each scenario. That is, I was surprised to see the PID and TID versions of the program have different lengths:</p><ul><li>generate_trace_entry() generates a program with 28 instructions</li><li>generate_trace_entry_tid() generates a program with 32 instructions</li><li>generate_trace_entry_pid() generates a program with 35 instructions</li></ul><p>It turns out that it is slightly more work to calculate the PID than the TID because calculating the PID requires a bit shift.</p><p>With the parameterization problem solved, I was ready to solve the mystery of how to wire up BPF maps!</p><h3>Working with BPF Maps</h3><p>Perhaps my biggest challenge in this whole process was figuring out how BCC transformed logic in C code for BPF maps into eBPF bytecode. Specifically, figuring out how to associate the declaration of a BPF map with its use:</p><pre>BPF_PERF_OUTPUT(events);<br><br><strong>int</strong> <strong>trace_return</strong>(<strong>struct</strong> pt_regs <strong>*</strong>ctx) {<br>  <strong>struct</strong> data_t data <strong>=</strong> {};<br>  <em>// ...<br></em>  events.perf_submit(ctx, <strong>&amp;</strong>data, <strong>sizeof</strong>(data));<br>}</pre><p>When looking at this code, my first question was: “What is the BPF_PERF_OUTPUT macro doing such that it creates an object named events with a perf_submit() method that can be called in trace_return()?” If you look at <a href="https://github.com/iovisor/bcc/blob/f138fea5a9ab279b7347fa6acfd2f53777068b27/src/cc/export/helpers.h#L80-L91">the definition for the macro</a>, it appears to expand to:</p><pre><strong>struct</strong> events_table_t {<br>  <strong>int</strong> key;<br>  u32 leaf;<br>  <em>/* map.perf_submit(ctx, data, data_size) */</em><br>  <strong>int</strong> (<strong>*</strong>perf_submit) (<strong>void</strong> <strong>*</strong>, <strong>void</strong> <strong>*</strong>, u32);<br>  <strong>int</strong> (<strong>*</strong>perf_submit_skb) (<strong>void</strong> <strong>*</strong>, u32, <strong>void</strong> <strong>*</strong>, u32);<br>  u32 max_entries;<br>};<br>__attribute__((section(&quot;maps/perf_output&quot;)))<br><strong>struct</strong> events_table_t events <strong>=</strong> { .max_entries <strong>=</strong> 0 }</pre><p>OK, so BPF_PERF_OUTPUT(events) has the side effect of declaring a new struct type (events_table_t) and creating an instance of it named events. There is also that __attribute__ thing to ensure events gets a special annotation in the resulting binary, though again, I was still trying to understand whether these annotations were strictly necessary. In particular, I didn’t see anything here that would result in the bpf(BPF_MAP_CREATE {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, ...) syscall I could see when I ran opensnoopy.py under strace. Because there wasn’t much else to look at, I decided to try to figure out what that __attribute__ stuff was all about.</p><p>In the BCC repo, I did a search for maps/perf_output, which matched <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/src/cc/frontends/clang/b_frontend_action.cc#L1169">a promising line of code in b_frontend_action.cc</a>. From here, I discovered a visitor that would examine section attributes and perform actions based on their contents. In this case, <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/src/cc/frontends/clang/b_frontend_action.cc#L1225-L1227">it would call bpf_create_map() and record the resulting file descriptor as a field of a TableDesc</a>. (I discovered that BPF maps are generally referred to as “tables” in the BCC codebase.) It was also in this file where I found the logic that explained where the perf_submit() method came from, as <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/src/cc/frontends/clang/b_frontend_action.cc#L809-L815">there was a visitor that rewrote the original call</a> as:</p><pre>bpf_perf_event_output(<br>  ctx,<br>  bpf_pseudo_fd(1, 4),<br>  CUR_CPU_IDENTIFIER,<br>  <strong>&amp;</strong>data,<br>  <strong>sizeof</strong>(data));</pre><p>Now the perf_submit() method had become a pure function! Though this led to new questions, such as, “What is bpf_pseudo_fd(1, 4)?” Fortunately, somewhere around here I discovered the various ways to dump debugging info using the Python bindings. For example, if I changed <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/tools/opensnoop.py#L135">this line in opensnoop.py</a> to the following:</p><pre><strong>from</strong> bcc <strong>import</strong> DEBUG_PREPROCESSOR<br>b <strong>=</strong> BPF(text<strong>=</strong>bpf_text, debug<strong>=</strong>DEBUG_PREPROCESSOR)<br><strong>import</strong> sys; sys<strong>.</strong>exit(0)</pre><p>Running sudo ./tools/opensnoop.py, I got:</p><pre>Running from kernel directory at<strong>:</strong> <strong>/</strong>lib<strong>/</strong>modules<strong>/</strong>4.15.0<strong>-</strong>33<strong>-</strong>generic<strong>/</strong>build<br>clang <strong>-</strong>cc1 <strong>-</strong>triple x86_64<strong>-</strong>unknown<strong>-</strong>linux<strong>-</strong>gnu <strong>-</strong>emit<strong>-</strong>llvm<strong>-</strong>bc <strong>-</strong>emit<strong>-</strong>llvm<strong>-</strong>uselists <strong>-</strong>disable<strong>-</strong>free <strong>-</strong>disable<strong>-</strong>llvm<strong>-</strong>verifier <strong>-</strong>discard<strong>-</strong>value<strong>-</strong>names <strong>-</strong>main<strong>-</strong>file<strong>-</strong>name main.c <strong>-</strong>mrelocation<strong>-</strong>model <strong>static</strong> <strong>-</strong>mthread<strong>-</strong>model posix <strong>-</strong>fmath<strong>-</strong>errno <strong>-</strong>masm<strong>-</strong>verbose <strong>-</strong>mconstructor<strong>-</strong>aliases <strong>-</strong>fuse<strong>-</strong>init<strong>-</strong>array <strong>-</strong>target<strong>-</strong>cpu x86<strong>-</strong>64 <strong>-</strong>dwarf<strong>-</strong>column<strong>-</strong>info <strong>-</strong>debugger<strong>-</strong>tuning<strong>=</strong>gdb <strong>-</strong>momit<strong>-</strong>leaf<strong>-</strong>frame<strong>-</strong>pointer <strong>-</strong>coverage<strong>-</strong>notes<strong>-</strong>file <strong>/</strong>usr<strong>/</strong>src<strong>/</strong>linux<strong>-</strong>headers<strong>-</strong>4.15.0<strong>-</strong>33<strong>-</strong>generic<strong>/</strong>main.gcno <strong>-</strong>nostdsysteminc <strong>-</strong>nobuiltininc <strong>-</strong>resource<strong>-</strong>dir lib<strong>/</strong>clang<strong>/</strong>7.0.0 <strong>-</strong>isystem <strong>/virtual/</strong>lib<strong>/</strong>clang<strong>/</strong>include <strong>-</strong>include .<strong>/</strong>include<strong>/</strong>linux<strong>/</strong>kconfig.h <strong>-</strong>include <strong>/virtual/</strong>include<strong>/</strong>bcc<strong>/</strong>bpf.h <strong>-</strong>include <strong>/virtual/</strong>include<strong>/</strong>bcc<strong>/</strong>helpers.h <strong>-</strong>isystem <strong>/virtual/</strong>include <strong>-</strong>I <strong>/</strong>home<strong>/</strong>mbolin<strong>/</strong>code<strong>/</strong>bcc <strong>-</strong>D __BPF_TRACING__ <strong>-</strong>I .<strong>/</strong>arch<strong>/</strong>x86<strong>/</strong>include <strong>-</strong>I arch<strong>/</strong>x86<strong>/</strong>include<strong>/</strong>generated<strong>/</strong>uapi <strong>-</strong>I arch<strong>/</strong>x86<strong>/</strong>include<strong>/</strong>generated <strong>-</strong>I include <strong>-</strong>I .<strong>/</strong>arch<strong>/</strong>x86<strong>/</strong>include<strong>/</strong>uapi <strong>-</strong>I arch<strong>/</strong>x86<strong>/</strong>include<strong>/</strong>generated<strong>/</strong>uapi <strong>-</strong>I .<strong>/</strong>include<strong>/</strong>uapi <strong>-</strong>I include<strong>/</strong>generated<strong>/</strong>uapi <strong>-</strong>D __KERNEL__ <strong>-</strong>D __HAVE_BUILTIN_BSWAP16__ <strong>-</strong>D __HAVE_BUILTIN_BSWAP32__ <strong>-</strong>D __HAVE_BUILTIN_BSWAP64__ <strong>-</strong>O2 <strong>-</strong>Wno<strong>-</strong>deprecated<strong>-</strong>declarations <strong>-</strong>Wno<strong>-</strong>gnu<strong>-</strong>variable<strong>-</strong>sized<strong>-</strong>type<strong>-</strong>not<strong>-</strong>at<strong>-</strong>end <strong>-</strong>Wno<strong>-</strong>pragma<strong>-</strong>once<strong>-</strong>outside<strong>-</strong>header <strong>-</strong>Wno<strong>-</strong>address<strong>-</strong>of<strong>-</strong>packed<strong>-</strong>member <strong>-</strong>Wno<strong>-</strong>unknown<strong>-</strong>warning<strong>-</strong>option <strong>-</strong>Wno<strong>-</strong>unused<strong>-</strong>value <strong>-</strong>Wno<strong>-</strong>pointer<strong>-</strong>sign <strong>-</strong>fdebug<strong>-</strong>compilation<strong>-</strong>dir <strong>/</strong>usr<strong>/</strong>src<strong>/</strong>linux<strong>-</strong>headers<strong>-</strong>4.15.0<strong>-</strong>33<strong>-</strong>generic <strong>-</strong>ferror<strong>-</strong>limit 19 <strong>-</strong>fmessage<strong>-</strong>length 109 <strong>-</strong>fobjc<strong>-</strong>runtime<strong>=</strong>gcc <strong>-</strong>fdiagnostics<strong>-</strong>show<strong>-</strong>option <strong>-</strong>vectorize<strong>-</strong>loops <strong>-</strong>vectorize<strong>-</strong>slp <strong>-</strong>o main.bc <strong>-</strong>x c <strong>/virtual/</strong>main.c <strong>-</strong>faddrsig<br><strong>#if defined(BPF_LICENSE)<br>#error BPF_LICENSE cannot be specified through cflags<br>#endif<br>#if !defined(CONFIG_CC_STACKPROTECTOR)<br>#if defined(CONFIG_CC_STACKPROTECTOR_AUTO) \<br>    || defined(CONFIG_CC_STACKPROTECTOR_REGULAR) \<br>    || defined(CONFIG_CC_STACKPROTECTOR_STRONG)<br>#define CONFIG_CC_STACKPROTECTOR<br>#endif<br>#endif<br></strong><br><strong>#include &lt;uapi/linux/ptrace.h&gt;<br>#include &lt;uapi/linux/limits.h&gt;<br>#include &lt;linux/sched.h&gt;<br></strong><br><strong>struct</strong> val_t {<br>    u64 id;<br>    <strong>char</strong> comm[TASK_COMM_LEN];<br>    <strong>const</strong> <strong>char</strong> <strong>*</strong>fname;<br>};<br><br><strong>struct</strong> data_t {<br>    u64 id;<br>    u64 ts;<br>    <strong>int</strong> ret;<br>    <strong>char</strong> comm[TASK_COMM_LEN];<br>    <strong>char</strong> fname[NAME_MAX];<br>};<br><br>BPF_HASH(infotmp, u64, <strong>struct</strong> val_t);<br>BPF_PERF_OUTPUT(events);<br><br>__attribute__((section(&quot;.bpf.fn.trace_entry&quot;)))<br><strong>int</strong> trace_entry(<strong>struct</strong> pt_regs <strong>*</strong>ctx)<br>{<br> <strong>int</strong> dfd <strong>=</strong> ctx<strong>-&gt;</strong>di; <strong>const</strong> <strong>char</strong> __user <strong>*</strong>filename <strong>=</strong> ctx<strong>-&gt;</strong>si;<br>    <strong>struct</strong> val_t val <strong>=</strong> {};<br>    u64 id <strong>=</strong> bpf_get_current_pid_tgid();<br>    u32 pid <strong>=</strong> id <strong>&gt;&gt;</strong> 32; <em>// PID is higher part<br></em>    u32 tid <strong>=</strong> id;       <em>// Cast and get the lower part<br></em><br>    <br>    <strong>if</strong> (bpf_get_current_comm(<strong>&amp;</strong>val.comm, <strong>sizeof</strong>(val.comm)) <strong>==</strong> 0) {<br>        val.id <strong>=</strong> id;<br>        val.fname <strong>=</strong> filename;<br>        bpf_map_update_elem((<strong>void</strong> <strong>*</strong>)bpf_pseudo_fd(1, 3), <strong>&amp;</strong>id, <strong>&amp;</strong>val, BPF_ANY);<br>    }<br><br>    <strong>return</strong> 0;<br>};<br><br>__attribute__((section(&quot;.bpf.fn.trace_return&quot;)))<br><strong>int</strong> trace_return(<strong>struct</strong> pt_regs <strong>*</strong>ctx)<br>{<br><br>    u64 id <strong>=</strong> bpf_get_current_pid_tgid();<br>    <strong>struct</strong> val_t <strong>*</strong>valp;<br>    <strong>struct</strong> data_t data <strong>=</strong> {};<br><br>    u64 tsp <strong>=</strong> bpf_ktime_get_ns();<br><br>    valp <strong>=</strong> bpf_map_lookup_elem((<strong>void</strong> <strong>*</strong>)bpf_pseudo_fd(1, 3), <strong>&amp;</strong>id);<br>    <strong>if</strong> (valp <strong>==</strong> 0) {<br>        <em>// missed entry<br></em>        <strong>return</strong> 0;<br>    }<br>    bpf_probe_read(<strong>&amp;</strong>data.comm, <strong>sizeof</strong>(data.comm), valp<strong>-&gt;</strong>comm);<br>    bpf_probe_read(<strong>&amp;</strong>data.fname, <strong>sizeof</strong>(data.fname), (<strong>void</strong> <strong>*</strong>)valp<strong>-&gt;</strong>fname);<br>    data.id <strong>=</strong> valp<strong>-&gt;</strong>id;<br>    data.ts <strong>=</strong> tsp <strong>/</strong> 1000;<br>    data.ret <strong>=</strong> PT_REGS_RC(ctx);<br><br>    bpf_perf_event_output(ctx, bpf_pseudo_fd(1, 4), CUR_CPU_IDENTIFIER, <strong>&amp;</strong>data, <strong>sizeof</strong>(data));<br>    bpf_map_delete_elem((<strong>void</strong> <strong>*</strong>)bpf_pseudo_fd(1, 3), <strong>&amp;</strong>id);<br><br>    <strong>return</strong> 0;<br>}<br><br><strong>#include &lt;bcc/footer.h&gt;</strong></pre><p>What we appear to have on the first line is a [long!] clang command to run followed by the C code that was passed to the BPF constructor rewritten by the preprocessor. Note how the clang invocation contains arguments such as -include /virtual/include/bcc/bpf.h -include /virtual/include/bcc/helpers.h -isystem /virtual/include. I was a bit confused by this because these paths starting with /virtual do not exist on my machine, so it was not as if I could copy/paste the clang command and run it on the command line. However, I found this <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/src/cc/exported_files.cc">exported_files.cc</a> file in the BCC repo, which appears to provide an in-memory mapping for these non-existent files that LLVM uses when it does the compilation.</p><p>Looking at the preprocessed source code itself, we can see that operations on the maps have been replaced with calls to bpf_map_update_elem(), bpf_map_lookup_elem(), bpf_perf_event_output(), and bpf_map_delete_elem(). Understanding this translation helped remove some of the abstractions between the high-level BCC toolkit and the low-level understanding I sought.</p><p>Trying other debug options with BPF() unlocked more information. Specifically, if tried using DEBUG_SOURCE instead of DEBUG_PREPROCESSOR and ran opensnoop.py again, I got:</p><pre>Disassembly of section .bpf.fn.trace_entry:<br>trace_entry:<br>; { // Line  35<br>   0:    79 17 68 00 00 00 00 00     r7 = *(u64 *)(r1 + 104)<br>; int dfd = ctx-&gt;di; const char __user *filename = ctx-&gt;si; // Line  36<br>   1:    b7 01 00 00 00 00 00 00     r1 = 0<br>; struct val_t val = {}; // Line  37<br>   2:    7b 1a f8 ff 00 00 00 00     *(u64 *)(r10 - 8) = r1<br>   3:    7b 1a f0 ff 00 00 00 00     *(u64 *)(r10 - 16) = r1<br>   4:    7b 1a e8 ff 00 00 00 00     *(u64 *)(r10 - 24) = r1<br>   5:    7b 1a e0 ff 00 00 00 00     *(u64 *)(r10 - 32) = r1<br>; u64 id = bpf_get_current_pid_tgid(); // Line  38<br>   6:    85 00 00 00 0e 00 00 00     call 14<br>   7:    bf 06 00 00 00 00 00 00     r6 = r0<br>   8:    7b 6a d8 ff 00 00 00 00     *(u64 *)(r10 - 40) = r6<br>; struct val_t val = {}; // Line  37<br>   9:    bf a1 00 00 00 00 00 00     r1 = r10<br>  10:    07 01 00 00 e8 ff ff ff     r1 += -24<br>; if (bpf_get_current_comm(&amp;val.comm, sizeof(val.comm)) == 0) { // Line  43<br>  11:    b7 02 00 00 10 00 00 00     r2 = 16<br>  12:    85 00 00 00 10 00 00 00     call 16<br>  13:    67 00 00 00 20 00 00 00     r0 &lt;&lt;= 32<br>  14:    77 00 00 00 20 00 00 00     r0 &gt;&gt;= 32<br>  15:    55 00 0a 00 00 00 00 00     if r0 != 0 goto +10<br>; val.fname = filename; // Line  45<br>  16:    7b 7a f8 ff 00 00 00 00     *(u64 *)(r10 - 8) = r7<br>; val.id = id; // Line  44<br>  17:    7b 6a e0 ff 00 00 00 00     *(u64 *)(r10 - 32) = r6<br>; bpf_map_update_elem((void *)bpf_pseudo_fd(1, 3), &amp;id, &amp;val, BPF_ANY); // Line  46<br>  18:    18 11 00 00 03 00 00 00 00 00 00 00 00 00 00 00     ld_pseudo    r1, 1, 3<br>  20:    bf a2 00 00 00 00 00 00     r2 = r10<br>; val.fname = filename; // Line  45<br>  21:    07 02 00 00 d8 ff ff ff     r2 += -40<br>  22:    bf a3 00 00 00 00 00 00     r3 = r10<br>  23:    07 03 00 00 e0 ff ff ff     r3 += -32<br>; bpf_map_update_elem((void *)bpf_pseudo_fd(1, 3), &amp;id, &amp;val, BPF_ANY); // Line  46<br>  24:    b7 04 00 00 00 00 00 00     r4 = 0<br>  25:    85 00 00 00 02 00 00 00     call 2<br>; return 0; // Line  49<br>  26:    b7 00 00 00 00 00 00 00     r0 = 0<br>  27:    95 00 00 00 00 00 00 00     exit<br><br>Disassembly of section .bpf.fn.trace_return:<br>trace_return:<br>; { // Line  54<br>   0:    bf 16 00 00 00 00 00 00     r6 = r1<br>; u64 id = bpf_get_current_pid_tgid(); // Line  56<br>   1:    85 00 00 00 0e 00 00 00     call 14<br>   2:    7b 0a f8 ff 00 00 00 00     *(u64 *)(r10 - 8) = r0<br>   3:    b7 01 00 00 00 00 00 00     r1 = 0<br>; struct data_t data = {}; // Line  58<br>   4:    7b 1a f0 ff 00 00 00 00     *(u64 *)(r10 - 16) = r1<br>   5:    7b 1a e8 ff 00 00 00 00     *(u64 *)(r10 - 24) = r1<br>   6:    7b 1a e0 ff 00 00 00 00     *(u64 *)(r10 - 32) = r1<br>   7:    7b 1a d8 ff 00 00 00 00     *(u64 *)(r10 - 40) = r1<br>   8:    7b 1a d0 ff 00 00 00 00     *(u64 *)(r10 - 48) = r1<br>   9:    7b 1a c8 ff 00 00 00 00     *(u64 *)(r10 - 56) = r1<br>  10:    7b 1a c0 ff 00 00 00 00     *(u64 *)(r10 - 64) = r1<br>  11:    7b 1a b8 ff 00 00 00 00     *(u64 *)(r10 - 72) = r1<br>  12:    7b 1a b0 ff 00 00 00 00     *(u64 *)(r10 - 80) = r1<br>  13:    7b 1a a8 ff 00 00 00 00     *(u64 *)(r10 - 88) = r1<br>  14:    7b 1a a0 ff 00 00 00 00     *(u64 *)(r10 - 96) = r1<br>  15:    7b 1a 98 ff 00 00 00 00     *(u64 *)(r10 - 104) = r1<br>  16:    7b 1a 90 ff 00 00 00 00     *(u64 *)(r10 - 112) = r1<br>  17:    7b 1a 88 ff 00 00 00 00     *(u64 *)(r10 - 120) = r1<br>  18:    7b 1a 80 ff 00 00 00 00     *(u64 *)(r10 - 128) = r1<br>  19:    7b 1a 78 ff 00 00 00 00     *(u64 *)(r10 - 136) = r1<br>  20:    7b 1a 70 ff 00 00 00 00     *(u64 *)(r10 - 144) = r1<br>  21:    7b 1a 68 ff 00 00 00 00     *(u64 *)(r10 - 152) = r1<br>  22:    7b 1a 60 ff 00 00 00 00     *(u64 *)(r10 - 160) = r1<br>  23:    7b 1a 58 ff 00 00 00 00     *(u64 *)(r10 - 168) = r1<br>  24:    7b 1a 50 ff 00 00 00 00     *(u64 *)(r10 - 176) = r1<br>  25:    7b 1a 48 ff 00 00 00 00     *(u64 *)(r10 - 184) = r1<br>  26:    7b 1a 40 ff 00 00 00 00     *(u64 *)(r10 - 192) = r1<br>  27:    7b 1a 38 ff 00 00 00 00     *(u64 *)(r10 - 200) = r1<br>  28:    7b 1a 30 ff 00 00 00 00     *(u64 *)(r10 - 208) = r1<br>  29:    7b 1a 28 ff 00 00 00 00     *(u64 *)(r10 - 216) = r1<br>  30:    7b 1a 20 ff 00 00 00 00     *(u64 *)(r10 - 224) = r1<br>  31:    7b 1a 18 ff 00 00 00 00     *(u64 *)(r10 - 232) = r1<br>  32:    7b 1a 10 ff 00 00 00 00     *(u64 *)(r10 - 240) = r1<br>  33:    7b 1a 08 ff 00 00 00 00     *(u64 *)(r10 - 248) = r1<br>  34:    7b 1a 00 ff 00 00 00 00     *(u64 *)(r10 - 256) = r1<br>  35:    7b 1a f8 fe 00 00 00 00     *(u64 *)(r10 - 264) = r1<br>  36:    7b 1a f0 fe 00 00 00 00     *(u64 *)(r10 - 272) = r1<br>  37:    7b 1a e8 fe 00 00 00 00     *(u64 *)(r10 - 280) = r1<br>  38:    7b 1a e0 fe 00 00 00 00     *(u64 *)(r10 - 288) = r1<br>  39:    7b 1a d8 fe 00 00 00 00     *(u64 *)(r10 - 296) = r1<br>  40:    7b 1a d0 fe 00 00 00 00     *(u64 *)(r10 - 304) = r1<br>; u64 tsp = bpf_ktime_get_ns(); // Line  60<br>  41:    85 00 00 00 05 00 00 00     call 5<br>  42:    bf 07 00 00 00 00 00 00     r7 = r0<br>; valp = bpf_map_lookup_elem((void *)bpf_pseudo_fd(1, 3), &amp;id); // Line  62<br>  43:    18 11 00 00 03 00 00 00 00 00 00 00 00 00 00 00     ld_pseudo    r1, 1, 3<br>  45:    bf a2 00 00 00 00 00 00     r2 = r10<br>; int trace_return(struct pt_regs *ctx) // Line  53<br>  46:    07 02 00 00 f8 ff ff ff     r2 += -8<br>; valp = bpf_map_lookup_elem((void *)bpf_pseudo_fd(1, 3), &amp;id); // Line  62<br>  47:    85 00 00 00 01 00 00 00     call 1<br>  48:    bf 08 00 00 00 00 00 00     r8 = r0<br>; if (valp == 0) { // Line  63<br>  49:    15 08 1f 00 00 00 00 00     if r8 == 0 goto +31<br>; bpf_probe_read(&amp;data.comm, sizeof(data.comm), valp-&gt;comm); // Line  67<br>  50:    bf 83 00 00 00 00 00 00     r3 = r8<br>  51:    07 03 00 00 08 00 00 00     r3 += 8<br>  52:    bf a1 00 00 00 00 00 00     r1 = r10<br>  53:    07 01 00 00 e4 fe ff ff     r1 += -284<br>  54:    b7 02 00 00 10 00 00 00     r2 = 16<br>  55:    85 00 00 00 04 00 00 00     call 4<br>; bpf_probe_read(&amp;data.fname, sizeof(data.fname), (void *)valp-&gt;fname); // Line  68<br>  56:    79 83 18 00 00 00 00 00     r3 = *(u64 *)(r8 + 24)<br>  57:    bf a1 00 00 00 00 00 00     r1 = r10<br>  58:    07 01 00 00 f4 fe ff ff     r1 += -268<br>  59:    b7 02 00 00 ff 00 00 00     r2 = 255<br>  60:    85 00 00 00 04 00 00 00     call 4<br>; data.id = valp-&gt;id; // Line  69<br>  61:    79 81 00 00 00 00 00 00     r1 = *(u64 *)(r8 + 0)<br>; data.ts = tsp / 1000; // Line  70<br>  62:    37 07 00 00 e8 03 00 00     r7 /= 1000<br>  63:    7b 7a d8 fe 00 00 00 00     *(u64 *)(r10 - 296) = r7<br>; data.id = valp-&gt;id; // Line  69<br>  64:    7b 1a d0 fe 00 00 00 00     *(u64 *)(r10 - 304) = r1<br>; data.ret = PT_REGS_RC(ctx); // Line  71<br>  65:    79 61 50 00 00 00 00 00     r1 = *(u64 *)(r6 + 80)<br>  66:    63 1a e0 fe 00 00 00 00     *(u32 *)(r10 - 288) = r1<br>; bpf_perf_event_output(ctx, bpf_pseudo_fd(1, 4), CUR_CPU_IDENTIFIER, &amp;data, sizeof(data)); // Line  73<br>  67:    18 12 00 00 04 00 00 00 00 00 00 00 00 00 00 00     ld_pseudo    r2, 1, 4<br>  69:    bf a4 00 00 00 00 00 00     r4 = r10<br>; bpf_probe_read(&amp;data.comm, sizeof(data.comm), valp-&gt;comm); // Line  67<br>  70:    07 04 00 00 d0 fe ff ff     r4 += -304<br>; bpf_perf_event_output(ctx, bpf_pseudo_fd(1, 4), CUR_CPU_IDENTIFIER, &amp;data, sizeof(data)); // Line  73<br>  71:    bf 61 00 00 00 00 00 00     r1 = r6<br>  72:    18 03 00 00 ff ff ff ff 00 00 00 00 00 00 00 00     r3 = 4294967295 ll<br>  74:    b7 05 00 00 28 01 00 00     r5 = 296<br>  75:    85 00 00 00 19 00 00 00     call 25<br>; bpf_map_delete_elem((void *)bpf_pseudo_fd(1, 3), &amp;id); // Line  74<br>  76:    18 11 00 00 03 00 00 00 00 00 00 00 00 00 00 00     ld_pseudo    r1, 1, 3<br>  78:    bf a2 00 00 00 00 00 00     r2 = r10<br>; bpf_probe_read(&amp;data.comm, sizeof(data.comm), valp-&gt;comm); // Line  67<br>  79:    07 02 00 00 f8 ff ff ff     r2 += -8<br>; bpf_map_delete_elem((void *)bpf_pseudo_fd(1, 3), &amp;id); // Line  74<br>  80:    85 00 00 00 03 00 00 00     call 3<br>; } // Line  77<br>  81:    b7 00 00 00 00 00 00 00     r0 = 0<br>  82:    95 00 00 00 00 00 00 00     exit</pre><p>This was great! This provided output analogous to what llvm-objdump provided in Geoffrey’s blog post. (Incidentally, <a href="https://github.com/iovisor/bcc/issues/1964">I built BCC using LLVM 7</a>. At other points, I had built it with LLVM 5, but I’m not sure the DEBUG_SOURCE output was as rich with the earlier version of LLVM, so that could explain why you get different results if you try to reproduce this.) Again, there were references in this output to bpf_pseudo_fd() and ld_pseudo that I didn’t understand. Grepping for these terms in the BCC codebase revealed very little.</p><p>Looking at <a href="https://github.com/iovisor/bcc/blob/master/src/cc/libbpf.h#L37-L39">the bpf_*_elem() functions</a> declared in libbpf.h, the first parameter for each was int fd, so based on the output of debug=DEBUG_PREPROCESSOR, it occurred to me that (void *)bpf_pseudo_fd(1, 3) would have to return the file descriptor for the infotmp map. Somewhere around here, it dawned on me that because stdin/stdout/stderr are fds 0/1/2, respectively, then there must be some logic to infer/ensure that the fds for infotmp/events must be 3/4, respectively. I tried swapping the order of the BPF map declarations in opensnoop.py and verified that the file descriptors were swapped accordingly!</p><p>Now I was getting somewhere. I realized that I should use bpf_create_map() in libbpf.h to create my maps and get their file descriptors. To figure out what values to pass, <a href="https://github.com/iovisor/bcc/blob/master/src/cc/export/helpers.h#L135-L136">I looked at the code in helpers.h</a> and determined that 10240 was the appropriate default value for the max_entries of a BPF_HASH. (I confirmed this is the value that was actually used by running strace on opensnoop.py.) Though for events, max_entries was 8 according to strace. I looked at the code in b_frontend_action.cc and discovered <a href="https://github.com/iovisor/bcc/blob/master/src/cc/frontends/clang/b_frontend_action.cc#L1170-L1174">the value for max_entries was determined by the number of “possible” CPUs</a> at runtime. This is a good reminder to look at the source as well as strace to avoid simply hardcoding the observed values!</p><p>I thought it was a bit weird that the instructions that were annotated with ld_pseudo in the debug output corresponded to things like:</p><pre>instructions[18] <strong>=</strong> (<strong>struct</strong> bpf_insn) {<br>  .code    <strong>=</strong> 0x18,<br>  .dst_reg <strong>=</strong> BPF_REG_1,<br>  .src_reg <strong>=</strong> BPF_REG_1,<br>  .off     <strong>=</strong> 0,<br>  .imm     <strong>=</strong> 3,<br>};</pre><p>That is, why was the src_reg specified as BPF_REG_1 when the value didn’t come from r1, but from imm? This seemed superfluous, so I tried changing src_reg to 0 instead. However, when I tried loading the program, I got the following error:</p><pre>R1 type=inv expected=map_ptr</pre><p>I had no idea what that meant, so I Googled it and found <a href="https://github.com/iovisor/bcc/issues/209">this issue</a> filed against BCC by Brendan himself (at least now I felt I was in good company!). What I eventually deduced was that when loading a 64-bit value (opcode=0x18), src_reg must be set to 1 so that the value being loaded is tagged as a “map_ptr” (i.e., a file descriptor for a BPF map) rather than an arbitrary value, which is necessary to satisfy the BPF program verifier.</p><p>With this newfound knowledge, I realized that every bytecode with opcode=0x18 and src_reg=1 would have to be rewritten to take imm as a parameter to the generated C function. I updated my Python logic that was responsible for generating generate_trace_entry() and friends so that it would put all of the imm values used with ld_pseudo into a set that would then be used to determine the additional parameters to add to the function prototype. I now had my code to create the BPF maps and load the BPF program with the appropriate file descriptors for the maps.</p><h3>Putting it all together</h3><p>I thought that I was just about done at this point, but it turned out I was wrong. <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/tools/opensnoop.py#L189-L193">This small snippet of Python code at the end of opensnoop.py</a> would turn out to be a lot of work to port to C:</p><pre><em># loop with callback to print_event</em><br>b[&quot;events&quot;]<strong>.</strong>open_perf_buffer(print_event, page_cnt<strong>=</strong>64)<br>start_time <strong>=</strong> datetime<strong>.</strong>now()<br><strong>while</strong> <strong>not</strong> args<strong>.</strong>duration <strong>or</strong> datetime<strong>.</strong>now() <strong>-</strong> start_time <strong>&lt;</strong> args<strong>.</strong>duration:<br>    b<strong>.</strong>perf_buffer_poll()</pre><p>It was at this point that I would begin to fully appreciate the additional abstraction that the BCC Python bindings provided. Initially, I expected the BCC Python bindings to be a thin wrapper around libbcc/libbpf, but it turns out they do quite a bit more than that. (Moreover, they also painted a picture of what sort of functionality a Rust wrapper around BPF should look like.) For example, <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/src/python/bcc/table.py#L553-L597">that little open_perf_buffer() method</a> turned out to have quite a bit of logic behind it. It also contained an innocuous helper function, get_online_cpus(), which took many lines of C code to implement (see <a href="https://bolinfest.github.io/opensnoop-native/#footnote-struggles-with-c">the next section on “struggles with C.”</a>)</p><p>Nevertheless, once I got the basics working end-to-end, I started to polish things up. For starters, I introduced getopt_long(3) for arg parsing so I could support the exact same options as opensnoop.py. I also discovered an unused variable in the BPF program, <a href="https://github.com/iovisor/bcc/pull/1981">which I fixed upstream</a>. I also discovered that the BPF program was doing an extra division on a u64 that I believe was necessary to ensure the value would still fit in a Python int when converted, <a href="https://github.com/bolinfest/rust-ebpf-demo/commit/afa0e91e17939be80def67b224e761f5288d2d82">so I deleted that step</a> since I didn’t have that concern in my program. Finally, <a href="https://github.com/bolinfest/rust-ebpf-demo/commit/f528a8ad9b7305f482623b1b94448df035a58e18">I moved some definitions into a common header file</a> that could be shared with opensnoop.c and the codegen step, which felt cleaner than trying to keep the Python and C versions in sync <a href="https://github.com/iovisor/bcc/blob/54e377d29b1e09968b8278c039fc3e146da9d962/tools/opensnoop.py#L142-L149">as opensnoop.py does</a>.</p><p>It was finally time to declare my program complete!</p><p>Truth be told, this entire project was my first time writing straight C (though I did read quite a bit of it as part of reading <a href="https://amzn.to/2NGuSqQ">Advanced Programming in the UNIX Environment, 3rd edition</a> cover-to-cover, but that’s a story for another blog post). Before I started on <a href="https://github.com/facebookexperimental/eden">Eden</a> a little over two years ago, I had never written any C or C++ before. Literally zero. Although there were points where it would have been easier to implement opensnoop in C++, I decided to stick it out to get some firsthand experience writing a non-trivial C program.</p><p>The thing I missed the most from C++ was <a href="https://en.cppreference.com/w/cpp/language/raii">RAII</a>. That is, I missed being able to leverage the compiler to ensure resources were released reliably. For example, I am used to wrapping a file descriptor with <a href="https://github.com/facebook/folly/blob/c09be5f410f6f567a9ec60ab1d898e4998d9cd9d/folly/File.h">folly::File</a> to ensure the file descriptor is closed when the folly::File goes out of scope, avoiding the need to call close(2) explicitly. Instead, I took a cue from libbpf.c and used a goto for the first time of my life, filling me with trepidation as I could only think of <a href="https://xkcd.com/292/">XKCD 292</a> the entire time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*0o-CuyI8BIwSWZgN.png" /></figure><p>The thing I missed second most from C++ was the standard library, notably std::string and std::vector. For example, here is the definition of <a href="https://github.com/iovisor/bcc/blob/3d2211632247ad0d1ee9d1ecc1162764322eb974/src/python/bcc/utils.py#L21-L36">a simple utility function from the BCC Python bindings</a>, get_online_cpus():</p><pre><strong>def</strong> <strong>_read_cpu_range</strong>(path):<br>    cpus <strong>=</strong> []<br>    <strong>with</strong> open(path, &#39;r&#39;) <strong>as</strong> f:<br>        cpus_range_str <strong>=</strong> f<strong>.</strong>read()<br>        <strong>for</strong> cpu_range <strong>in</strong> cpus_range_str<strong>.</strong>split(&#39;,&#39;):<br>            rangeop <strong>=</strong> cpu_range<strong>.</strong>find(&#39;-&#39;)<br>            <strong>if</strong> rangeop <strong>==</strong> <strong>-</strong>1:<br>                cpus<strong>.</strong>append(int(cpu_range))<br>            <strong>else</strong>:<br>                start <strong>=</strong> int(cpu_range[:rangeop])<br>                end <strong>=</strong> int(cpu_range[rangeop<strong>+</strong>1:])<br>                cpus<strong>.</strong>extend(range(start, end<strong>+</strong>1))<br>    <strong>return</strong> cpus<br><br><strong>def</strong> <strong>get_online_cpus</strong>():<br>    <strong>return</strong> _read_cpu_range(&#39;/sys/devices/system/cpu/online&#39;)</pre><p>Compare this with the equivalent version I wrote in straight C:</p><pre><strong>int</strong> <strong>getOnlineCpus</strong>(<strong>int</strong> <strong>**</strong>cpus, <strong>size_t</strong> <strong>*</strong>numCpu) {<br>  <strong>int</strong> fd <strong>=</strong> open(&quot;/sys/devices/system/cpu/online&quot;, O_RDONLY <strong>|</strong> O_CLOEXEC);<br>  <strong>if</strong> (fd <strong>&lt;</strong> 0) {<br>    <strong>return</strong> <strong>-</strong>1;<br>  }<br><br>  <strong>const</strong> <strong>int</strong> bufSize <strong>=</strong> 256;<br>  <strong>char</strong> buf[bufSize];<br>  <strong>int</strong> numRead <strong>=</strong> read(fd, buf, bufSize);<br>  <strong>if</strong> (numRead <strong>==</strong> bufSize <strong>||</strong> numRead <strong>==</strong> 0) {<br>    <em>// We are not prepared for the output to be this big (or empty)!<br></em>    errno <strong>=</strong> EINVAL;<br>    <strong>return</strong> <strong>-</strong>1;<br>  }<br>  <strong>if</strong> (close(fd) <strong>&lt;</strong> 0) {<br>    <strong>return</strong> <strong>-</strong>1;<br>  }<br><br>  <em>// Ensure the contents of buf are NUL-terminated so that strtol() does not<br></em>  <em>// read unintended values.<br></em>  buf[numRead] <strong>=</strong> &#39;\0&#39;;<br><br>  <strong>size_t</strong> capacity <strong>=</strong> 16;<br>  <strong>*</strong>cpus <strong>=</strong> malloc(capacity <strong>*</strong> <strong>sizeof</strong>(<strong>int</strong>));<br>  <strong>if</strong> (<strong>*</strong>cpus <strong>==</strong> NULL) {<br>    <strong>return</strong> <strong>-</strong>1;<br>  }<br><br>  <strong>int</strong> lastEndIndex <strong>=</strong> <strong>-</strong>1;<br>  <strong>int</strong> lastHyphenIndex <strong>=</strong> <strong>-</strong>1;<br>  <strong>size_t</strong> numElements <strong>=</strong> 0;<br>  <strong>for</strong> (<strong>size_t</strong> i <strong>=</strong> 0; i <strong>&lt;=</strong> numRead; i<strong>++</strong>) {<br>    <strong>if</strong> (i <strong>==</strong> numRead <strong>||</strong> buf[i] <strong>==</strong> &#39;,&#39;) {<br>      errno <strong>=</strong> 0;<br>      <strong>int</strong> rangeStart <strong>=</strong><br>          strtol(buf <strong>+</strong> lastEndIndex <strong>+</strong> 1, <em>/* endptr */</em> NULL, <em>/* base */</em> 10);<br>      <strong>if</strong> (errno <strong>!=</strong> 0) {<br>        <strong>return</strong> <strong>-</strong>1;<br>      }<br><br>      <strong>int</strong> rangeEnd;<br>      <strong>if</strong> (lastHyphenIndex <strong>!=</strong> <strong>-</strong>1) {<br>        errno <strong>=</strong> 0;<br>        rangeEnd <strong>=</strong><br>            strtol(buf <strong>+</strong> lastHyphenIndex <strong>+</strong> 1, <em>/* endptr */</em> NULL, <em>/* base */</em> 10);<br>        <strong>if</strong> (errno <strong>!=</strong> 0) {<br>          <strong>return</strong> <strong>-</strong>1;<br>        }<br>      } <strong>else</strong> {<br>        rangeEnd <strong>=</strong> rangeStart;<br>      }<br><br>      <strong>int</strong> numCpusToAdd <strong>=</strong> rangeEnd <strong>-</strong> rangeStart <strong>+</strong> 1;<br>      <strong>int</strong> extraSpace <strong>=</strong> capacity <strong>-</strong> numElements <strong>-</strong> numCpusToAdd;<br>      <strong>if</strong> (extraSpace <strong>&lt;</strong> 0) {<br>        <strong>size_t</strong> newSize <strong>=</strong> capacity <strong>-</strong> extraSpace;<br>        <strong>*</strong>cpus <strong>=</strong> realloc(cpus, newSize <strong>*</strong> <strong>sizeof</strong>(<strong>int</strong>));<br>        <strong>if</strong> (<strong>*</strong>cpus <strong>==</strong> NULL) {<br>          <strong>return</strong> <strong>-</strong>1;<br>        }<br>        capacity <strong>=</strong> newSize;<br>      }<br><br>      <strong>for</strong> (<strong>int</strong> j <strong>=</strong> 0; j <strong>&lt;</strong> numCpusToAdd; j<strong>++</strong>) {<br>        <strong>*</strong>(<strong>*</strong>cpus <strong>+</strong> numElements<strong>++</strong>) <strong>=</strong> rangeStart <strong>+</strong> j;<br>      }<br><br>      lastEndIndex <strong>=</strong> i;<br>      lastHyphenIndex <strong>=</strong> <strong>-</strong>1;<br>    } <strong>else</strong> <strong>if</strong> (buf[i] <strong>==</strong> &#39;-&#39;) {<br>      lastHyphenIndex <strong>=</strong> i;<br>    }<br>  }<br><br>  <strong>*</strong>numCpu <strong>=</strong> numElements;<br>  <strong>return</strong> 0;<br>}</pre><p>The most obvious difference is the size of the code, but what is responsible for all those extra lines?</p><p>The first 16 lines or so are responsible for reading the contents of /sys/devices/system/cpu/online into a char[], checking all of the intermediate return values for errors, and ensuring the file descriptor gets closed. Life is certainly a lot easier when you can use an existing library function that just throws if anything goes wrong.</p><p>The loop logic is a bit verbose for a couple of reasons. First, the API for strtol(3) is a bit wonky, so we have to reset errno before calling it and then check it afterwards. Secondly, because we are returning an array whose size is not known at compile time, we have to have logic to realloc(3), as necessary. I’m grateful that I don’t have to write code this way all the time, as it seems exhausting. (Though if I had my own large C project, I’m sure I would build/use libraries to help with some of these tasks whereas I tried to keep opensnoop to a single C file.) Ultimately, this only furthered my desire to replace C code with Rust wherever possible.</p><h3>Part 5: Conclusions and Next Steps</h3><p>This journey was a challenging exercise as I had to learn a lot of new things in a space that does not have a ton of documentation. Incidentally, I was on paternity leave from work while I did this, so I didn’t have access to my coworkers to tap for suggestions when I got stuck. Nevertheless, coming out the other side, I feel much more confident in venturing into unfamiliar domains than I did before. I am more proficient with gdb and strace, and I will be sure to always keep a copy of the Linux source locally so I can grep through it in the future.</p><p>Although this project started as an attempt to teach myself how to debug performance issues, it ended up focusing on building tools using eBPF. Although I didn’t get that deep into the former, I have some initial questions that I still want to explore, as well as further work on the latter.</p><h3>Performance Engineering</h3><p>My biggest surprise is that Brendan has a number of checklists of tools to run rather than one tool that runs everything in the checklist and prints out a summary. It’s possible that it’s too hard to create a “one size fits all” tool for this sort of thing because the USE method requires that you start from a functional diagram of your system, which a general tool cannot be expected to know (at least not without any configuration). Despite the challenges, I think there could be a big opportunity here.</p><p>From my cursory understanding of the space, it feels like there are a lot of <em>CLIs</em> out there that you can cobble together to make an application-specific performance tool, but not a lot of <em>libraries</em>. Perhaps this is the right place to start. I know I would be more inclined to build tools on top of libraries that I can access programmatically rather than having to write a bunch of logic to parse stdout in various formats.</p><p>Because I was at home on paternity leave, it’s not like I had a bunch of massive distributed systems lying around the house whose performance needed tuning. In other words, it seems difficult to improve one’s skills in the abstract: this feels like an area where you have to build knowledge through experience. At this stage, there’s no way I could remember how <a href="http://www.brendangregg.com/Perf/linux_perf_tools_full.svg">all of these tools</a> work offhand, but at least I know Brendan’s diagram exists and that I can use it as a starting point.</p><p>That said, both Brendan and Julia had examples in their talks where they would run a program with some sort of bad behavior, treating it as a black box, and then walking through the way you would use different tools to poke at the system to figure out what was going on (and then revealing the source code of the program at the end). It would be <em>amazing</em> if someone put together some sort of online learning course in this way to teach performance engineering: I’d certainly pay for that!</p><h3>eBPF</h3><p>I thoroughly enjoyed learning about eBPF. I mainly focused on using it with kprobes, but I realize there are other integration points that I haven’t played with yet (uprobes, tracepoints, XDP). The general idea of being able to dynamically (and safely and efficiently!) add functionality in the kernel from user space is pretty awesome. I’m excited to see if/how eBPF is extended to other domains. For example, Ph.D. student Ashish Bijlani recently presented a prototype that used <a href="https://events.linuxfoundation.org/wp-content/uploads/2017/11/When-eBPF-Meets-FUSE-Improving-Performance-of-User-File-Systems-Ashish-Bijlani-Georgia-Tech.pdf">eBPF to improve FUSE performance</a>. Also, remember that Android devices are also running Linux, so <a href="http://www.joelfernandes.org/linuxinternals/2018/01/08/bpfd-bcc.html">with a bit of effort</a>, eBPF-based tools can be used to diagnose issues on phones, as well. (Though when trying to use your eBPF tool with a specific device, be sure to cross-check the <a href="https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md">version of the Linux kernel running on your Android device</a> with <a href="https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md">the set of BPF features supported for that kernel</a>.)</p><p>In terms of the C version of opensnoop, I don’t know that anyone really needs it, so I don’t have any plans to try to popularize it. <a href="https://github.com/bolinfest/opensnoop-native">I put the final version of the code in its own repo</a> so that others can easily look at it and link to it. In general, I don’t like checking in generated code, but I made an exception in this case because I think it’s valuable for people to be able to study it without installing/building BCC in order to generate it themselves.</p><p>Finally, what about rstrace? I’ll <a href="https://github.com/bolinfest/rstrace">set up a repo</a> and probably mess around a little bit, but my paternity leave is about to come to an end, so I don’t expect to have much time for it in the near future. One thing I plan to do is include the source for libbpf verbatim in the rstrace repo (I’ll sync it from upstream periodically) so that people can build rstrace without building BCC. As I discovered <a href="https://github.com/rust-bpf/rust-bcc/pull/29">when submitting a small patch to rust-bcc</a>, that project is trying to figure out how to support multiple versions of BCC at once, so I’d rather avoid that headache altogether. (Again, it’s easier for me to include the source directly because I only need libbpf, not libbcc.) Hopefully I can find some time for rstrace because I never actually got to write any Rust as part of my explorations thus far!</p><p><em>Originally published at </em><a href="https://bolinfest.github.io/opensnoop-native/"><em>bolinfest.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dedb81c25e6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Airing of Grievances]]></title>
            <link>https://mbolin.medium.com/python-airing-of-grievances-db4d09edb75b?source=rss-588a10a26d5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/db4d09edb75b</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Michael Bolin]]></dc:creator>
            <pubDate>Mon, 20 Mar 2017 09:06:24 GMT</pubDate>
            <atom:updated>2023-05-22T04:13:51.397Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Although this list focuses on the negative aspects of Python, it was originally written as a footnote for my article </em><a href="https://medium.com/@mbolin/javascript-vs-python-in-2017-d31efbb641b4"><em>“</em>JavaScript vs. Python in 2017<em>,”</em></a><em> which is about my </em>positive<em> experiences with Python 3.6.</em></p><p>I am excited by many of the recent improvements in Python, but here are some of my outstanding issues with Python (particularly when compared to JavaScript):</p><h3>PEP-8</h3><p>What do you get when you combine 79-column lines, four space indents, and snake case? Incessant line-wrapping, that’s what! I prefer 100 cols and 2-space indents for my personal Python projects, but every time a personal project becomes a group project and true Pythonista joins the team, they inevitably turn on PEP-8 linting with its defaults and reformat everything.</p><h3>Having to declare self as the first argument of a method</h3><p>As someone who is not a maintainer of Python, there is not much I could do to fix this, but I did the next best thing: I complained about it on Facebook. In this case, it worked! That is, I provoked <a href="https://github.com/ambv">Łukasz Langa</a> to add a check for this (it now exists as <a href="https://github.com/PyCQA/flake8-bugbear/commit/ff6afda2bffc376a0a0b33e6181324367483598c">B902</a> in <a href="https://github.com/PyCQA/flake8-bugbear">flake8-bugbear</a>), so at least when I inevitably forget to declare self, <a href="https://nuclide.io/docs/languages/python/#code-diagnostics">Nuclide</a> warns me inline as I&#39;m authoring the code.</p><h3>.pyc files</h3><p>These things are turds. I get why they are important for production, but I’m tired of seeing them appear in my filetree, adding *.pyc to .gitignore every time I create a new project, etc.</p><h3>Lack of a standard docblock</h3><p>Historically, Python has not had a formal type system, so I was eager to document my parameter and return types consistently. If you look at the top-voted answer to “What is the standard Python docstring format?” on StackOverflow, <a href="http://stackoverflow.com/a/24385103/396304">the lack of consensus extremely dissatisfying</a>. Although Javadoc has its flaws, <a href="http://www.oracle.com/technetwork/articles/java/index-137868.html">Java’s documentation story</a> is orders of magnitude better than that of Python’s.</p><h3>Lack of a Good, free graphical debugger</h3><p>Under duress, I use <a href="https://docs.python.org/2/library/pdb.html">pdb</a>. I frequently edit Python on a remote machine, so most of the free tools do not work for me. I have not had the energy to try to set up <a href="https://www.jetbrains.com/help/pycharm/2016.3/remote-debugging.html">remote debugging in PyCharm</a>, though admittedly that appears to be the best commercial solution. Instead, <a href="https://github.com/facebook/nuclide/commit/694d3c504a32f788ca28f4c044f8d65d6cdcbf67">I tried to lay the groundwork for a Python debugger in Nuclide</a>, which would support remote development by default. I need to find someone who wants to continue that effort.</p><h3>Whitespace-delimited</h3><p>I’m still not 100% sold on this. In practice, this begets all sorts of subtle annoyances. Code snippets that you copy/paste from the Web or an email have to be reformatted before you can run them. Tools that generate Python code incur additional complexity because they have to keep track of the indent level. Editors frequently guess the wrong place to put your cursor when you introduce a new line whereas typing } gives it the signal it needs to un-indent without further interaction. Expressions that span a line frequently have to be wrapped in parentheses in order to parse. The list goes on and on.</p><h3>Bungled rollout of Python 3</h3><p>Honestly, this is one of the primary reasons I didn’t look at Python seriously for awhile. When Python 3 came out, the rhetoric I remember was: “Nothing substantially new here. Python 2.7 works great for me and the <a href="https://docs.python.org/2/library/2to3.html">2to3 migration tool</a> is reportedly buggy. No thanks.” It’s rare to see such a large community move forward with such a poorly executed backwards-compatibility story. (Presumably this has contributed to the lack of a default Python 3 installation on OS X, which is another reason that some software shops stick with Python 2.) <a href="http://www.i-programmer.info/news/167-javascript/7932-angularjs-20-is-radically-different-.html">Angular 2</a> is the only other similar rage-inducing upgrade in recent history that comes to mind.</p><h3>I don’t understand how imports are resolved</h3><p>Arguably, this is a personal failure rather than a failure of Python. That said, compared to Java or Node, whatever Python is doing seems incredibly complicated. Do I need an __init__.py file? Do I need to put something in it? If I do have to put something in it, does it have to define __all__? For a language whose mantra is “There&#39;s only one way to do it,” it is frustrating that the answer to all of these questions is “it depends.”</p><h3>Busted Scoping Rules</h3><p>For a language with built-in map() and filter() functions that underscore the support for first-order functions, I am perplexed with how bad the support for closures is in Python. Apparently your options are (1) lambda (which is extremely limited), or (2) a bastardized closure. Borrowing an example from <a href="https://www.smallsurething.com/a-quick-guide-to-nonlocal-in-python-3/">an article that explains the new </a><a href="https://www.smallsurething.com/a-quick-guide-to-nonlocal-in-python-3/">nonlocal keyword in Python 3</a>, consider the following code in JavaScript:</p><pre>function outside() {<br>  let msg = &#39;Outside!&#39;;<br>  console.log(msg);<br>  let inside = () =&gt; {<br>    msg = &#39;Inside!&#39;;<br>    console.log(msg);<br>  };<br>  inside();<br>  console.log(msg);<br>}</pre><p>If you ran outside(), you would get the following output (which you would expect from a lexically scoped language like JavaScript):</p><pre>Outside!<br>Inside!<br>Inside!</pre><p>Whereas if you wrote this Python code:</p><pre>def outside():<br>    msg = &#39;Outside!&#39;<br>    print(msg)<br>    def inside():<br>        msg = &#39;Inside!&#39;<br>        print(msg)<br>    inside()<br>    print(msg)</pre><p>and ran outside(), you would get the following:</p><pre>Outside!<br>Inside!<br>Outside!</pre><p>In Python 2, <a href="http://stackoverflow.com/questions/141642/what-limitations-have-closures-in-python-compared-to-language-x-closures">the only way to emulate the JavaScript behavior is to use a mutable container type</a> (which is gross):</p><pre>def outside():<br>    msg_holder = [&#39;Outside!&#39;]<br>    print(msg_holder[0])<br>    def inside():<br>        msg_holder[0] = &#39;Inside!&#39;<br>        print(msg_holder[0])<br>    inside()<br>    print(msg_holder[0])</pre><p>Though apparently Python 3 has a new nonlocal keyword that makes this less distasteful:</p><pre>def outside():<br>    msg = &#39;Outside!&#39;<br>    print(msg)<br>    def inside():<br>        nonlocal msg<br>        msg = &#39;Inside!&#39;<br>        print(msg)<br>    inside()<br>    print(msg)</pre><p>I realize that JavaScript generally forces you to type more with its var, let and const qualifiers, but I find this to be a small price to pay in exchange for unambiguous variable scopes.</p><p>This post originally appeared on <a href="http://blog.bolinfest.com/2017/03/python-airing-of-grievances.html">bolinfest.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db4d09edb75b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JavaScript vs. Python in 2017]]></title>
            <link>https://medium.com/hackernoon/javascript-vs-python-in-2017-d31efbb641b4?source=rss-588a10a26d5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d31efbb641b4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Michael Bolin]]></dc:creator>
            <pubDate>Mon, 20 Mar 2017 08:44:29 GMT</pubDate>
            <atom:updated>2019-05-17T05:32:07.119Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I may be one of the last people you would expect to write an essay criticizing JavaScript, but here we are.</em><br> <br>Two of my primary areas of professional interest are <a href="https://www.youtube.com/watch?v=M3uWx-fhjUc">JavaScript and “programming in the large.”</a> I gave a <a href="https://www.youtube.com/watch?v=PCL3dXQZ_Wk">presentation back in 2013 at mloc.js</a> where I argued that static typing is an essential feature when picking a language for a large software project. For this reason, <a href="http://blog.bolinfest.com/2017/03/python-airing-of-grievances.html">among others</a>, I historically limited my use of Python to small projects with no more than a handful of files.<br> <br>Recently, I needed to build a command-line tool for work that could speak Thrift. I have been enjoying the <a href="https://nuclide.io/">Nuclide</a>/<a href="https://flowtype.org/">Flow</a>/<a href="https://babeljs.io/">Babel</a>/<a href="http://eslint.org/">ESLint</a> toolchain a lot recently, so my first instinct was to use <a href="https://hackernoon.com/tagged/javascript">JavaScript</a> for this new project, as well. However, it quickly became clear that if I went that route, <a href="https://github.com/apache/thrift/pull/1175">I would have to spend a lot of time up front on getting the Thrift bindings to work properly</a>. I couldn’t convince myself that my personal preference for JavaScript would justify such an investment, so I decided to take a look at Python.<br> <br>I was vaguely aware that there was an effort to add support for static typing in Python, so I Googled to find out what the state of the art was. It turns out that it is a tool named <a href="http://mypy.readthedocs.io/en/latest/">Mypy</a>, and it provides <a href="https://en.wikipedia.org/wiki/Gradual_typing">gradual typing</a>, much like <a href="https://flowtype.org/">Flow</a> does for JavaScript or <a href="http://hacklang.org/">Hack</a> does for PHP. Fortunately, Mypy is more like Hack+HHVM than it is like Flow in that a Python 3 runtime accepts the type annotations natively whereas Flow type annotations must be stripped by a separate tool before passing the code to a JavaScript runtime. (To use Mypy in Python 2, you have to put your type annotations in comments, <a href="https://github.com/google/closure-compiler/wiki/Annotating-JavaScript-for-the-Closure-Compiler">operating in the style of the Google Closure toolchain</a>.) Although Mypy does not appear to be as mature as Flow (support for incremental type checking is still in the works, for example), simply being able to succinctly document type information was enough to renew my interest in Python.<br> <br>In researching how to use Thrift from Python, a <a href="https://hackernoon.com/tagged/google">Google</a> search turned up some sample <a href="https://github.com/facebook/fbthrift/tree/master/thrift/tutorial/py.asyncio">Python code that spoke to Thrift using asynchronous abstractions</a>. After gradual typing, async/await is the other feature in JavaScript that I cannot live without, so this code sample caught my attention! As we recently added support for building projects in Python 3.6 at work, it was trivial for me to get up and running with the latest and greatest features in Python. (Incidentally, I also learned that you really want Python 3.6, not 3.5, as 3.6 has some important <a href="http://mypy.readthedocs.io/en/latest/python36.html">improvements to Mypy</a>, <a href="https://bugs.python.org/issue28613">fixes to the </a><a href="https://bugs.python.org/issue28613">asyncio API</a>, <a href="https://www.python.org/dev/peps/pep-0498/">literal string interpolation like you have in ES6</a>, <a href="https://docs.python.org/3/whatsnew/3.6.html">and more</a>!)<br> <br>Coming from <a href="https://hackernoon.com/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f">the era of “modern” JavaScript</a>, one thing that was particularly refreshing was rediscovering how Python is an edit/refresh language out of the box whereas JavaScript used to be that way, but is no more. Let me explain what I mean by that:</p><ul><li>In Python 3.6, I can create a new example.py file in my text editor, write Python code that uses async/await and type annotations, switch to my terminal, and run python example.py to execute my code.</li><li>In JavaScript, I can create a new example.js file in my text editor, write JavaScript code that uses async/await and type annotations, switch to my terminal, run node example.js, see it fail because it does not understand my type annotations, run npm install -g babel-node, run babel-node example.js, see it fail again because I don&#39;t have a .babelrc that declares the babel-plugin-transform-flow-strip-types plugin, rummage around on my machine and find a .babelrc I used on another project, copy that .babelrc to my current directory, run babel-node example.js again, watch it fail because it doesn&#39;t know where to find babel-plugin-transform-flow-strip-types, go back to the directory from which I took the .babelrc file and now copy its package.json file as well, remove the junk from package.json that example.js doesn&#39;t need, run npm install, get impatient, kill npm install, run yarn install, and run babel-node example.js to execute my code. For bonus points, babel-node example.js runs considerably slower than node example.js (with the type annotations stripped) because it re-transpiles example.js every time I run it.</li></ul><p>Indeed, the JavaScript ecosystem offers all sorts of tools to speed up this process using daemons and caching, but you or someone on your team has to invest quite a bit of time researching, assembling, and maintaining a JavaScript toolchain for your project before anyone can write a line of “modern” JavaScript. Although you may ultimately achieve a nice edit/refresh workflow, you certainly will not have one out of the box as you would in Python.</p><blockquote>“<em>JavaScript is no longer an edit/refresh language.</em>”</blockquote><p>Another refreshing difference between JavaScript and Python is the “batteries included” nature of Python. If you look at the standard library that comes with JavaScript, it is fairly minimal. The Node environment does a modest job of augmenting what is provided by the standard library, but the majority of the functionality you need inevitably has to be fetched from npm. Specifically, consider the following functionality that is included in Python’s standard library, but must be fetched from npm for a Node project:</p><ul><li><a href="https://www.npmjs.com/package/rimraf">deleting a directory</a></li><li>argument parsing (<a href="https://www.npmjs.com/package/commander">I</a> <a href="https://www.npmjs.com/package/yargs">can’t</a> <a href="https://www.npmjs.com/package/argparse">even…</a>)</li><li><a href="https://www.npmjs.com/package/temp">creating a temp file</a></li><li>Logging (there are too many to options for me to decide which ones to link to)</li><li>unit testing (again, <a href="https://www.npmjs.com/package/mocha">too</a> <a href="https://www.npmjs.com/package/chai">many</a> <a href="https://www.npmjs.com/package/jasmine">to</a> <a href="https://www.npmjs.com/package/jest">list</a>)</li><li><a href="https://www.npmjs.com/package/sprintf-js">printf format strings</a></li><li><a href="https://www.npmjs.com/package/left-pad">left-justify a string</a> (<a href="https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/">Node developers are still having nightmares over this one</a>)</li></ul><p>As you can see, for many of these features, there are multiple third-party libraries that provide overlapping functionality. (For example, if you were looking for a JSON parser, would you choose parse-json, safe-json-parse, fast-json-parse, jsonparse, or json-parser?) To make matters worse, npm module names are doled out on a first-come, first-serve basis. Much like domain names, this means that great names often go to undeserving projects. (For example, judging from its download count, <a href="https://www.npmjs.com/package/logging">the npm module named </a><a href="https://www.npmjs.com/package/logging">logging</a> makes it one of the least popular logging packages for JavaScript.) This makes the comparison of third-party modules all the more time-consuming since the quality of the name is not a useful signal for the quality of the library.<br> <br>It might be possible that Python&#39;s third-party ecosystem is just as bad as npm&#39;s. What is impressive is that <em>I have no idea whether that is the case because it is so rare that I have to look to a third-party Python package to get the functionality that I need</em>. I am aware that data scientists rely heavily on third-party packages like NumPy, but unlike the Node ecosystem, there is <em>one</em> NumPy package that everyone uses rather than a litany of competitors named numpy-fast, numpy-plus, simple-numpy, etc.</p><blockquote><em>“We should stop holding up npm as a testament to the diversity of the JavaScript ecosystem, but instead cite it as a failure of JavaScript’s standard library.”</em></blockquote><p>For me, one of the great ironies in all this is that, arguably, the presence of a strong standard library in JavaScript would be the most highly leveraged when compared to other programming languages. Why? Because today, every non-trivial web site requires you to download underscore.js or whatever its authors have chosen to use to compensate for JavaScript’s weak core. When you consider the aggregate adverse impact this has on network traffic and page load times, the numbers are frightening.</p><h3>So…Are You Saying JavaScript is Dead?</h3><p>No, no I am not. If you are building UI using web technologies (which is a lot of developers), then I still think that JavaScript is your best bet. Modulo the emergence of <a href="http://webassembly.org/">Web Assembly</a> (which is worth paying attention to), JavaScript is still the only language that runs natively in the browser. There have been many attempts to take an existing programming language and compile it to JavaScript to avoid “having to” write JavaScript. There are cases where the results were good, but they never seemed to be great. Maybe some transpilation toolchain will ultimately succeed in unseating JavaScript as the language to write in for the web, but I suspect we’ll still have the majority of web developers writing JavaScript for quite some time.<br> <br>Additionally, the browser is not the only place where developers are building UI using web technologies. Two other prominent examples are <a href="https://electron.atom.io/">Electron</a> and <a href="https://facebook.github.io/react-native/">React Native</a>. Electron is attractive because it lets you write once for Windows, Mac, and Linux while React Native is attractive because it lets you write once for Android and iOS. Both are also empowering because the edit/refresh cycles using those tools is much faster than their native equivalents. From a hiring perspective, it seems like developers who know web technologies (1) are available in greater numbers than native developers, and (2) can support more platforms with smaller teams compared to native developers.<br> <br>Indeed, I could envision ways in which these platforms could be modified to support Python as the scripting language instead of JavaScript, which could change the calculus. However, one thing that all of the crazy tooling that exists in the JavaScript community has given way to is transpiler toolchains like Babel, which make it easier for ordinary developers (who do not have to be compiler hackers) to experiment with new JavaScript syntax. In particular, this tooling has paved the way for things like <a href="https://facebook.github.io/react/docs/jsx-in-depth.html">JSX</a>, which I contend is one of the key features that makes React such an enjoyable technology for building UI. (Note that you can use React without JSX, but it is tedious.)<br> <br>To the best of my knowledge, the Python community does not have an equivalent, popular mechanism for experimenting with DSLs within Python. So although it might be straightforward to add Python bindings in these existing environments, I do not think that would be sufficient to get product developers to switch to Python unless changes were also made to the language that made it as easy to express UI code in Python as it is in JavaScript+JSX today.</p><h3>Key Takeaways</h3><p>Python 3.6 has built-in support for gradual typing and async/await. Unlike JavaScript, this means that you can write Python code that uses these language features and run that file directly without any additional tooling. Further, its rich standard library means that you have to spend little time fishing around and evaluating third-party libraries to fill in missing gaps. It is very much a “get stuff done” server-side scripting language, requiring far less scaffolding than JavaScript to get a project off the ground. Although Mypy may not be as featureful or performant as Flow or TypeScript is today, the velocity of that project is certainly something that I am going to start paying attention to.<br> <br>By comparison, I expect that JavaScript will remain strong among product developers, but those who use Node today for server-side code or command-line tools would probably be better off using Python. If the Node community wants to resist this change, then I think they would benefit from (1) expanding the Node API to make it more comprehensive, and (2) reducing the startup time for Node. It would be even better if they could modify their runtime to recognize things like type annotations and JSX natively, though that would require changes to V8 (or Chakra, on Windows), which I expect would be difficult to maintain and/or upstream. Getting TC39 to standardize either of those language features (which would force Google/Microsoft’s hand to add native support in their JS runtimes) also seems like a tough sell.<br> <br>Overall, I am excited to see how things play out in both of these communities. You never know when someone will release a new technology that obsoletes your entire toolchain overnight. For all I know, we might wake up tomorrow and all decide that <a href="http://facebook.github.io/reason/">we should be writing OCaml</a>. Better set your alarm clock.</p><p><em>This post originally appeared on </em><a href="http://blog.bolinfest.com/2017/03/javascript-vs-python-in-2017.html"><em>bolinfest.com</em></a><em>.</em></p><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a>family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>To learn more, <a href="https://goo.gl/4ofytp">read our about page</a>, <a href="http://bit.ly/HackernoonFB">like/message us on Facebook</a>, or simply, <a href="https://goo.gl/k7XYbx">tweet/DM @HackerNoon.</a></blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d31efbb641b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/javascript-vs-python-in-2017-d31efbb641b4">JavaScript vs. Python in 2017</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>