AWS Outposts is a rack-scale computer that runs on premises. The most recent re:Invent had a bunch of sessions about changes to Outposts. One change that happened without much fanfare is a new lower price (note: LOW-ER, not LOW). I looked at Outposts pricing last year shortly after it was released. As with that analysis there are some more stories and oddities hiding in the numbers: customer and sales feedback, trial balloons popped, months-long miscommunications, and — as with any pandemic-era story — a reversion to tedium.

AWS Outposts is an extension of the cloud that lives in your own facilities (either your DC or a colo). It’s a 42U rack that comes with different quantities of EC2 instance types (and therefore different quantities of CPU threads (vCPUs), DRAM, GPUs, and SSD capacity). It’s not a private cloud in that it is tightly bound to the AWS control plane (and a particular AWS region and availability zone). Rather, it’s a little chunk of AWS physically close to your other systems.

All the analysis below is grounded on calculations you can find in this spreadsheet.

AWS Outpost comparison 2021Q1

Margin Call

In last year’s analysis we saw that the cover charge for an Outpost rack was about $200k before you start adding useful components like CPUs and DRAM. That’s for the rack, the power, the switch, fans.. other stuff? Wheels? In case that doesn’t sound crazy: it is. Deeply. The margin on those components must have been astronomical. Maybe some customers noticed and rebelled.

Outposts come in several flavors that support different EC2 instance types. The price of each model is a simple linear combination of the component instances whose prices we can derive. The new price of the rack is exactly $80k — slightly more reasonable than it was last year.

The price reduction of the rack by $121,651.20 across all models accounts for all price changes except for 3: one reasonable, one kind of annoying, and one goofy.

No dev unit stimulus

A year ago the cheapest dev unit (actually two models, distinguished only by their external network type) had a special discount. Each AWS Output model is advertised as supporting various EC2 instance types. Using some algebra (or Excel Solver) I calculated the price of each instance type in Outposts. Most are priced similarly to their (3 year reserved instance) cloud-operated EC2 analogs (±8%). Last year, however, the m5.12xlarge came at an 80% discount, but that instance type only came in the cheapest dev unit, making it — ostensibly — an attractive model to start with and kick the tires.

No more! In 2021, the m5.12xlarge is no longer an aberration (see above). The dev unit is still cheaper than it was a year ago, but only by $56k (not $121k) as it was — effectively — already discounted by $65k. The Jassy giveth and the Jassy taketh away. This also means it is no longer the cheapest model — and it never should have been since other dev models with fewer vCPUs and less DRAM were previously more expensive.

The smallest dev unit comes with 96 threads and 192GB of DRAM. Even with the new low (low!) price of $118k, that seems … exploitative. A comparable server (singular) from Dell is about 1/5th of that price.

Note that of the instance types that appear in Outposts, the m5.12xlarge was the only one to see a significant price reduction when comparing the 3 year reserved instance price. I was pretty surprised by this. AWS epitomizes the intersection of relentless capitalism and steady technical advancement. Weren’t we promised cheaper and cheaper computing in the cloud? Where’s my Moore’s Law dividend? As with so many incremental benefits, they’re going disproportionately to the relentless capitalist. More on this later, but here’s exhibit A on why the future isn’t going to be one where Jeff Bezos owns and operates every computer on the planet.

Price Correction

The OR-LR4DI6P is (with an asterisk) the most significantly discounted Outpost model. It supports a smattering of instance types tallying to 1152 vCPUs and 5TB of DRAM. The good news is that its price dropped $40k more than any other; the bad news is that it was previously overpriced by $40k. Last year I called this guy out as an anomaly. Was it overpriced to attempt to reduce the number of models? That seems weird for — what was a year ago — a new product line. Was it a divining rod for finding wells of dumb customers? Either away the experiment seems to be over.

The other short-lived experiment was the OR-M6GJ8FE which has been EOLed. With 2 x g4dn.12xlarge and 4 x m5.24xlarge, this was the only dev unit to include a GPU-enabled instance. It was also — by far — the most overpriced model, 21% / $125k more expensive than it should have been based on the per-instance type unit prices. It was hard to imagine a scenario where it should not have upgraded to the OR-H7KSNNO — $90k more for 2 extra g4dn.12xlarge and 3 extra m5.24xlarge. It’s been replaced, more or less, with the OR-LMF5LJ1 which just has 2 x g4dn.12xlarge for $161k.

Comedy of errors

The largest nominal price reduction was for the OR-HUZEI16 with 8 x m5.24xlarge and 4 x c5.24xlarge (1152 vCPUs and 3840GB of DRAM). This guy dropped by $200k from $791,586.16 to $590,216.36.

The weird thing is that nothing changed in the unit math. A year ago it was priced exactly equal to the sum of its parts; same story this year. What changed, it turns out, was its parts. A year ago, those 4 x c5.24xlarge instances were 4 x r5.24xlarge; c5 means “half the memory of the m5” and r5 means “double the memory of the m5”. Last year that higher price applied to a unit with an extra 2304GB of DRAM.

Looking back at my data from a year ago I now see that difference sources were inconsistent. The website said r5 while the console (and API) said c5.

There must have been a transcription error somewhere; clearly the quants in the pricing team were looking at the website (r5) and not the console (c5). Someone figured that out in the past year. I like to think it was a customer who went looking for their missing 2TB of DRAM.

Amazon Financial Services

A quick note on the financing option. Outposts have three ways to buy: cash upfront, monthly installments, or monthly installments with 50% down. This is typical of other long term commitments in AWS. Check out the financing options in EC2:

The terms are all within epsilon of 8.5% with half the money down, and 9.2% with $0 down.

In contrast, the 2019 rates for Outposts were as wild as you get in the office of the CFO:

Clearly someone put an end to that fun in 2020. The new rates are precisely 8.496% and 9.247%. No exceptions.

(Smaller) Component Pricing

Last year I just looked at the prices of each EC2 instance types that make up the Outpost models. I’ve spent the past year a lot closer to computer components so broke it down further, looking at the per vCPU (thread), GB of DRAM, GPU (an NVIDIA T4), and the SSDs. How? Excel’s Solver function. It’s seriously life-changing. I set up the inputs (the component prices) and then wrote formulas multiplying the price of each component by the number in each model to get the expected price per model. Solver will maximize or minimize a single value so I wrote a formula that sums the absolute values of the differences between actual and computed cost for each Outpost model. A few iterations of Solver later and I had this:

AWS hasn’t announced the pricing for their upcoming 2U Outpost model, but from this we can take a pretty good guess (the 1U will use a Graviton processor so we don’t have data to extrapolate from for that model). 128 vCPUs, 512GB of DRAM, 8TB of SSD, plus some extra for a slimmed down chassis, I’d expect it to be something like $75k. Which is nuts.

Maybe AWS will surprise us, but I’m not holding my breath. Outposts are not just expensive but significantly overpriced — despite the big drop and by quite a bit.

Take this next bit with a big grain of salt, but I picked some parts off the Dell website that seemed like fair comparisons. There might be more expensive parts in Outposts and I’m not accounting for all the other stuff that goes into a server, but CPU/DRAM/GPU/SSD are going to dominate everything else. Outposts pricing remains at least mildly rapacious.

Conclusion

I understand why customers want Outposts. Despite the grousing in this post, I’m an AWS fan and have tremendous respect for their rate of innovation. Customers want API-driven infrastructure, the flexibility and speed of development that it provides. Traditional on-premises vendors, most concisely represented by the EMC/Dell/VMware conglomerate, have failed to evolve their way into this experience which is why the hardware and software from the hyperscalers (including Outposts) looks quite different. Customers are faced with a literal dilemma: on one hand they can continue buying the moderately priced, expensive to operate, inefficient to use, legacy gear from Dell/HPE/etc or they can pay exorbitantly for something like AWS Outposts. The former is throwing good money after bad, investing in an ecosystem that has continued to underdeliver; the latter cedes more and more control to AWS and locks them into an unaffordable future.

Shortly after writing my first blog post on this topic a year ago, I joined Oxide where we’re building a computer that combines the hardware and software lessons from the cloud with sensible economics. It’s exciting building this computer the world needs.

This was kind of a silly project. If I had known it would end with a bunch of wholesome Christmas decorations dancing along to Lizzo I probably would have done it a long time ago. Here’s the story of how I took some beloved kitsch, a logic analyzer, sed, awk, Google Sheets, Python, an FT232H, some wires, data from a VR game, and a few evenings to make it happen. (Just want the punchline? Skip to the end.)

Happy Tapper eBay addiction

My family has a fair bit of music-producing stuff from Hallmark: books, stuffed animals, and, of course, holiday decorations. Chalk it up to an enthusiastic grandparent. The Happy Tappers are a favorite. We started with two — a reindeer and a snowman — and I was tasked with filling out the full set of four. We now have 13, so mission accomplished and then some.

Each Tapper plays two songs and dances; you can plug in other Tappers via ports on either side and they become backup dancers. Deeply alluring, clearly. I got a good deal on Santa, then a better deal on a different Santa. Then there was a great deal on a set of 4, so had to get it. Then — after months of waiting — I pounced on a set of 5, including the very rare “painter” elf. So, in short: I have a problem. This solstice holiday season my plan was to figure out how they work and get them to do some out-of-spec dancing.

Vivisection

My first step was to pull one apart. There’s a microcontroller (covered with scratched up epoxy) that controls the speaker and electromagnet actuators while sending and receiving signals over the bus formed by connecting multiple Tappers (henceforth, the Dance Bus).

Examination with a multimeter showed that pin 3 is ground, pin 2 is for dance signals, and pin 1 is high when one of the Tappers on the Dance Bus starts playing. I tried sending some pulses down pin 2, but that didn’t produce any tapping (happy or otherwise). So I hooked one up to my brand new logic analyzer.

Reverse Choreography

You can spend a lot for a great logic analyzer which is what I didn’t do. I had never used a logic analyzer of any kind until a few weeks ago when I took a hardware hacking class. Basically as a party favor for the class, we were each given a BitMagic logic analyzer ($30) (worse, by the way, than the fancy Saleae we used during the class). I used PulseView, the open source front-end, to gather and examine signal data. The docs were great; it was an testament to the power of open.

I hooked up the leads to the target Tapper, pushed the button, and… nothing. Its button didn’t work. Whether an artifact of the cheap logic analyzer or due to some pilot error, the logic analyzer was actually sending a signal — 3V measured between pins 1 and 3 — which means on the Dance Bus that a Tapper is active and others should expect to receive signals. I worked around this by pushing the Tapper’s button and then quickly hooking up the leads.

Here’s 15 seconds worth of Dance Bus activity in PulseView:

Each of those little blips corresponds to a message on the Dance Bus. Zooming in we can see a couple of the messages:

The cursor tool lets you measure the time between features:

I measured a few by hand, but it was going to be tedious to look at each signal that way. I needed the data in a form I could examine programmatically. PulseView has a bunch of output formats; the one I landed on (by trying them all) is “Value Change Data” whose output looks like this:

$date Thu Dec 19 23:17:09 2019 $end
$version libsigrok 0.6.0-git-ef62ab6 $end
$comment
Acquisition with 1/8 channels at 1 MHz
$end
$timescale 1 us $end
$scope module libsigrok $end
$var wire 1 ! D0 $end
$upscope $end
$enddefinitions $end
#0 1!
#1291266 0!
#1309128 1!
#1311126 0!
#1313143 1!
#1315111 0!
#1319103 1!
#1321095 0!
#1323096 1!
#1325115 0!
...

Microsecond timestamps (noted in the header; perhaps due to the 1 MHz sampling frequency I used) and values at the time of changes (Value Change Data — in retrospect: duh). I stripped out the header and line prefixes with sed, then computed deltas between timestamps with awk:

$ cat xx.vcd | sed -n 's/^#\(.*\)/\1/p' | awk 'NR>1{ print $1-last } { last=$1 }'
1291266
17862
1998
2017
1968
3992
1992
2001
2019
2002
289808
17531
1989
2010
1962
3978
1984
1995
2003
3958

The longest signals I care about appear to be no more than 18ms so I used some more awk to convert >20ms durations to newlines and everything else to CSV:

$ cat xx.vcd | sed -n 's/^#\(.*\)/\1/p' | awk 'NR>1{ print $1-last } {last=$1 }' | awk '{ if ($1 < 20000) printf "%s,", $1; else print ""; }'
17862,1998,2017,1968,3992,1992,2001,2019,2002,
17531,1989,2010,1962,3978,1984,1995,2003,3958,
17811,1995,1999,1997,4020,1964,1999,1993,2005,
17631,1996,2020,1967,3995,1991,2003,2012,3973,
17676,1986,1995,2004,3953,1983,1990,1987,2013,
17765,2008,1993,1956,3978,1989,2011,1960,3979,
...
17788,1995,2001,2016,3970,1993,1999,1997,2022,
17959,1966,1998,1992,4018,1968,1999,1994,4003,
17525,1989,2009,1961,3977,1985,1994,2005,1997,
17612,1965,1997,1991,3998,2012,2001,1963,3993,
17879,1963,1990,1983,3981,2005,1994,1956,1992,
17782,1993,1998,1996,4015,1963,2000,1990,4018,
17968,1970,1997,1992,3998,2013,2001,1964,2002,81,2,

Scanning vertically, the numbers looked close enough that felt like I might be onto something, so I threw them into a spreadsheet to verify (I ❤️ spreadsheets):

The signals looked pretty consistent until the last segment, 9 pulses of the following durations: 17.7ms, 2ms, 2ms, 2ms, 4ms, 2ms, 2ms, 2ms, ???. I split the last pulse into subsets above and below the average to get a pretty clear bimodal distribution, one right at 2ms and the other at 4ms:

Two signals for the two feet of each Tapper on the Dance Bus. I suppose I could have guessed that, but based on the preprogrammed dances I expected a single signal that just cycled from foot to foot. Now that I knew how the Tappers communicated to each other, I had to generate those signals myself.

Tapping Out Signals

Back when I was a kid (in the days before our serial buses were universal) PCs had a parallel port. You could hook up printers and other peripherals (like Zip Drives and slow-ass network devices), but with the tiniest bit of ingenuity you could write code to turn certain pins on and off. We used them to do really cool stuff like control LEDs… and control other LEDs. The possibilities were endless.

Another party favor from that hardware hacking class was an Adafruit FT232H Breakout ($15) which Limor Fried (founder of Adafruit) called the “modern parallel port”. It’s a USB bridge that you can use to speak useful, well-known protocols like SPI and I2C, or — in our case — less-useful, dance protocols via general purpose input/output (GPIO) pins. This thing is super cool: extremely approachable, easy to use, no IDE. It’s a great starting point for anyone who wants to write software that can control simple electronics.

The Tappers are ~4.5V (3 x AAA batteries) and the FT232H outputs 3V signals, but it’s what I had so I went with it. Adafruit operates mostly in Python so I went with that too. Because I was paranoid about the resolution of time.sleep() I decided to spin for the allotted time which resulted in this code:

And I tacked on a little interactive test program:

Full of anticipation, I plugged in a Tapper expecting to j-k-j-k a little two-step and… nothing. So I hooked up the logic analyzer again, this time to my FT232H to see if it was producing the signals I expected. Here’s what I got:

The first low pulse that was supposed to be 17.7ms was only around 2ms. I thought a bit about the many sources of latency that could cause this, but then opted for urgency over curiosity, and added another 15.7 to that first value. I measured again and things looked good, so I hooked up the Tapper. Lo and behold, he started tapping:

I could move their feet now I needed to make them dance.

Tap Tap Revolution

I knew I couldn’t outtap the rhythmic geniuses from Hallmark, but I could tap into one of the great breakthroughs of our era: rhythm games. The greatest of these — in my completely unqualified opinions — is indisputably Beat Saber. It’s the killer app for VR. I’m not going to explain; just check it out:

Beyond just the songs arranged by the developer, there’s an awesome community of content creators that make new songs available for download on BeatSaver (and other places I infer). The file format is pretty simple. You can check out the code if you want the details, but eventually you’ll poke deep enough in a JSON file to find an array of objects that look like this:

{
  "_time": 51.875,
  "_lineIndex": 1,
  "_lineLayer": 0,
  "_type": 0,
  "_cutDirection": 7
}

Time is in units of beats (we get BPM elsewhere); index and layer indicate the location of the block to hit; cut direction is pretty self-explanatory; type indicates which hand you need to hit the block with. I ignored location and direction, and just used time and type. With some simple Python, I unpacked the file format, played the music, and fed the tap data into my Tappers. Here’s the result:

You can plug in more Tappers to the Dance Bus so without further ado, Truth Hurts like you’ve never seen it before:

There were a bunch of moving parts; I was stoked (and surprised) to see it all come together. It turns out not to be that hard, and the parts weren’t that expensive. Give it a shot; just don’t outbid me on eBay.

Parts List:

• Happy Tappers (available on eBay)
• BitMagic Basic Logic Analyzer from 1 Bit Squared
• PulseView
• Adafruit FT232H Breakout — General Purpose USB to GPIO+SPI+I2C
• You’ll need some sort of soldering iron for the FT232H; I used my one from RadioShack in 1995
• Etekcity MSR-R500 Digital Multimeter (no endorsement other than it being cheap and available on Prime)
• HiLetgo 3pcs 400 Ties Mini Solderless Breadboard (not strictly necessary)
• EDGELEC 120pcs Breadboard Jumper Wires
• Song data from BeatSaver

Optional:

• Oculus Quest / HTC Vive + a gaming PC
• Beat Saber
• Lots of time

AWS announced Outpost a year ago. It’s a rack-scale computer that lets customers run a limited set of AWS services on-premises. A couple of weeks ago, AWS announced that Outpost was out of beta and generally available.

Now that the pricing is public, I ripped apart the data. These are big, expensive boxes; what do you get for the money? How does it compare to running instances in the cloud?

What follows is the analysis I wanted, but couldn’t find. If you’ve been thinking “I wish I had a spreadsheet to get a deep understanding of Outpost pricing”, you’ve come to the right place. If you think spreadsheets are tedious, or perhaps a bizarre thing for a person to do in his idle time, spoiler: I’m going to get really excited about an Excel optimizer, conditionally color some cells, and use RATE() for the first time. So, yeah, it’s going to be wild / I won’t be offended if you leave after watching the “is it on-premise or on-premises?” supercut.

Here’s the google sheet with all raw data and analysis:

AWS Outpost comparison 2020Q1

Data Gathering

We can see all the data: price, instance types, etc. But which data? On the morning of December 10th the data looked like this:

That afternoon, it looked like this:

Every instance type had its count incremented by one e.g. the first one, OR-R2XVM9Q, had 1 m5.24xlarge, c5.24xlarge, and r5.24xlarge and the later grab shows 2 of each.

I decomposed the configuration strings into a count of instances types for each Outpost configuration, and then plugged them into Excel’s Solver to figure out the Outpost pricing model. The numbers didn’t converge to a rational answer with the old data, but it did with the new data. So I’m sticking with the new data… even though the data in the AWS console (and the JSON data) still match the old data:

Not an ARM

No surprise: these suckers ain’t cheap. A quarter million for “Development & testing units” up to $900k for one chock-a-block full of memory. As with other term commitments from AWS, you can choose to pay all up front, pay in monthly installments, or pay a 50% downpayment with smaller monthly installments. Effectively Amazon lets you take out a mortgage on your outpost at around 7-8% — 0.7% less if you put 50% down.

The were notable outliers: the dev/test models with either a single c5.24xlarge or m5.24xlarge both had an interest rate closer to 4%. It makes sense that Amazon might want to make it a bit less painful to get that first evaluation Outpost in your lab, but it’s weird that that discount only applies to 2 of the 5 dev test models.

Outpost vs. Inpost

How do Outposts compare to the cost of running workloads in AWS? For this, I used reserve instance pricing with a 3 year term. I figure if you’re ready to drop some serious racks on this gear, a 3 year commitment probably doesn’t scare you. For this comparison I made the nominally gross simplification of ignoring storage (since that seems to have a consumption-based monthly component), power (a large and growing cost center), as well as other service charges (more on that in a bit).

For each Outpost configuration, I decomposed it into a count of the included instance types. I figured each Outpost probably had a fixed cost component (the rack itself, ToR, etc.) and then you’re just paying a fixed cost for each instance type. I mentioned Excel Solver earlier — it’s a brilliant tool to have at the ready, go learn it if you haven’t already— on a spreadsheet with formulas, it modifies values (the Outpost cost for each instance type) and optimizes for a goal value (minimizing the delta between the computed and known cost for Outpost configurations). With the first set of data I pulled off the Outpost pricing page, the numbers weren’t converging at all, but with the new data they converged right away. In fact, it was pretty straightforward to do by hand with simple operations like this:

m5.24xlarge
= 12 m5.24xlarge ($738,3141) - m5.24xlarge ($246,373) / 11
= $44,722

Now that we have the pricing model for Outpost, we can compare it to the prices for the corresponding cloud-hosted instance.

Most of them are close-ish. The obvious outlier is the m5.12xlarge instance type. This only appears in the two least expensive (dev/test) configs. So — as before — it might make sense that Amazon is trying to create financial incentives for that first development box.

The pricing model with values for the chassis and each instance type produces an expected price for each config. Only two aren’t spot on: a colossal $800k / 5TB memory config which is about 5% more expensive than we’d expect, and the GPU-enabled dev/test config which is a whopping 20% overpriced.

The 5% lift on the “Mixed capacity large unit” is hard to figure. Maybe they wanted to push people to the 6TB model which is only $70 (dollars — not kilo-dollars) more (aka two weeks of running an on-demand m5.24xlarge instance).

The dev/test “Graphics medium unit” is the only dev/test unit with a GPU (8 actually). It’s also the only dev/test that doesn’t have some sort of price incentive (on the contrary). On a relative basis it’s by far the most expensive Outpost config, and it’s hard to imagine why someone would opt for it over the “Graphics large unit”. For $90k more (15%) you go from 2 to 4 GPU-enabled instances and from 4 to 7 m5.24large. In short: weird price; it’s either an oversight or a story.

Off Premise

Now that you’ve paid for it, wheeled it into your data center, and powered it, you’re good to go right? Sort of. You pay for the Outpost and then you keep paying by the hour. EBS is $0.15/GB-mo and up. Apparently the meter’s also running ECS and EKS, but the Outpost pricing page sends you to the ECS/EKS pricing pages which point back to the Outpost pricing page, so TBD on that. The managed database service (RDS) is available in Outposts, but it’s pretty expensive. For example, if you want to run MySQL on one of those m5.24xlarge instances in an Outpost that’s an $3.60/hr or $94,608 on top of the however many hundred thousand for the gear. Compare that to the 3 year, upfront cost, all-in cost to run that same database in AWS of $86,671 (no additional charge from EC2).

Cloud Migration vs. Mitigation

An Outpost isn’t a hyperscaler-type computer you own and operate. It’s aptly named for what it is: an extension of AWS (indeed, linked to a particular region) with a limited set of services that sits on-premise in close proximity to other systems you might care about. Outposts meet a real need; I expect there’s an even larger need to build true private clouds with the efficiency of the hyperscalers.

The top tier Outpost comes with 576 cores (1152 vCPUs) and 9.2TB of DRAM for $900k. There’s margin in there for a system that meets the private cloud demand without falling farther down the AWS gravity well.

The first choice you make when creating a new workflow is how it will be triggered. Presumably there will be more options in the future; right now it’s just three: when a user picks it from a menu of workflows (like a macro), when a new user joins a specific channel, or when a user reacts with an emoji. Curiously, you can’t react with a message action (the … on each message) which seemed like the preferred trigger over emojis — future work I suppose.

Once you’ve picked the event source, you specify a sequence of steps. There you have even fewer choices: send a message or post a form.

What’s pretty cool is that both the initial event and subsequent steps produce output values that subsequent steps can reference. For example, you can create a step that posts a message that @mentions the person who triggered the workflow, or you could send a message to a different user that includes a value from a form someone filled out. It’s pretty slick and despite the limited range of actions you can take today, it’s easy to imagine how Slack might extend this in the future with more action types and more outputs.

And that’s really it. It’s like a poor man’s Zapier or IFTTT. Destitute maybe. But you can see the outline of how this could expand: more event sources, more steps, and maybe more complex, branching workflows.

Teardown

The basic hypothesis of workflows is that non-technical users need a visual builder. Fortunately, it’s all represented as a json object that we can download. The overall structure looks like this:

{
  "source_id": "<looks like some unique timer-based id>",
  "version": "1",
  "workflow": {
    "name": "name of the workflow
    "blueprint": {
      "version": "1",
      "trigger": {
        "type":
          "reaction_added | channel_action | member_joined_channel",
        "id": "<uuid>",
        "config": {
           ... trigger-specific data ...
        }
      },
      "steps": [
        {
          "type": "dialog | message",
          "id": "<uuid>",
          "config": {
            ... step-specific data ...
          }
        },
        ...
      ]
    }
  }
}

Triggers and steps have an id and type; the type defines the contents of the config member. The id in the trigger and each step be referenced in later steps to pull out certain info. For example, a message step looks like this:

"type": "message",
  "id": "<uuid>",
  "config": {
    "user": {
      "ref": "<uuid of trigger>==user"
    },
    "has_button": false,
    "message_text": "howdy {{<uuid of trigger>==user}}!"
  }
}

This sends a message to the person who triggered the workflow. Note that the user object can either have a ref member that refers to a value from a previous step or a “value” member that’s a literal value. The message can also make reference to the output from the trigger and previous steps using that mustache-ish syntax.

Let’s break it

You can craft your own json and upload new workflows. It’s pretty easy to sneak stuff past the upload step that violates the rules.

If the various steps don’t have actually unique ids, the editor gets sad without being overly explicit about what’s making it sad:

You can make poorly constructed forms with tons and tons of fields, non-unique ids, etc. The editor doesn’t complain, but when you trigger the action, the form just isn’t shown — there’s probably an entry in a lot somewhere, but there’s nothing visible to the user or the workflow author.

Finally — my personal favorite — the visual builder enforces that the number of steps in a workflow be bounded by propriety and good taste. Not so when importing workflows. I wrote a little python script that generates a workflow that sends 500 messages. This is particularly cruel because 1. it forever (about 20 minutes; I don’t know if workflows just run slow or are rate limited) and 2. it appears to be unkillable even if you unpublish the workflow (deleting it does seem to work). It would be easy to write something even more noxious… fortunately workflows do identify the author.

Starting a workflow instantiates a state machine that walks through each step. It locks in the version of the workflow at the time it was initiated: changes to the published workflow don’t effect instances already in flight. This is a sensible approach and curtailed my plans for mischief. I can’t tell what happens to workflows that are never completed — do they hang around in some database forever or does Slack clean them up after awhile? I called the (undocumented) apps.actions.run API, but didn’t want to DoS the new feature. But speaking of APIs…

Missions Impossible

There isn’t a documented API (that I could find) for Workflow Builder. Running Slack from the browser let me sniff out some of the calls:

Missions! That acquisition Slack made in July of 2018. And while it’s been rebranded to Workflow Builder, the name lives on in the API.

Slack acquires Robots & Pencils' Missions to make it easy for non-tech teams to streamline work - Several People Are Typing

The Missions website doesn’t have much info left on it but I was able to dig up some details from the Internet Archive. Here what Missions supported:

So… a LOT more functionality in Missions than in what we see in the initial launch of Workflow Builder. It probably gives a good sense of the kinds of stuff to come.

Where Workflow Happens

Zapier already supports vastly more functionality than Workflow Builder, but I’m sure being built into Slack is compelling for a bunch of users. Even with the extremely limited functionality, you can see how triggers and steps are built to be generalized. It’s also clear that workflows are an important piece of Slack’s broader platform ambitions. There are going to be more triggers, more steps, and deep, broad integration with the zillions of cloud applications that Slack wants you working with through their collaboration platform.

Also Teams doesn’t have it; that seems pretty important.

Previous eras of software consumption were defined by their bundles. Our current era is defined by its diverse application ecosystem, its unbundling. Even recently, corporate consumers wore their fealty to a vendor with pride: we’re a Microsoft shop, we’re an Oracle shop, we’re an IBM shop. While devoted, the relationship wasn’t monogamous. The right tool for the job, but guided by relationship as the north star. Bundling had the benefit (at least perceived) of interoperability — many tools, intermeshed, helping us work well as a team.

SaaS changed both the economics of software, and the adhesion of the bundle. Even if you wanted to be a mono-shop, it’s harder to see how to do that (or why you’d want to). Teams are picking the tools they want. Applications are able to be deeper and narrower than ever before. Consequently everyone has many more applications from a much more distributed collection of vendors.

Established and new companies are repositioning for this distributed, diverse unbundleverse. We started Transposit to help build the apps composed of the APIs from unbundled software. We’re betting on this trend. But I heard a new vision for this modern era of software a couple of weeks ago at Slack’s Frontiers conference: that there can be hubs even amid unbundling, centers in an era of decentralization. Rather than grouping like-applications, these hubs group like-activities. In an unbundled world, the monopolies will be horizontal, aggregating similar kinds of interactions across applications.

Slack on Slack

At the conference, Slack laid out a significant and compelling vision about becoming a major epicenter amid this sectoral unbundling. While the conference felt bold and enlightening, Slack’s summary blog post was more pedestrian: mostly new features and non-specific enthusiasm. It listed shared channels, Workflow Builder, and Gmail/Outlook gateways as some big pieces. Under the heading “Building the tools for modern work”, the video featuring CPO, Tamar Yehoshua, touches on a common theme of the conference:

“But communication at work doesn’t just happen between people. We need access to data and other workflows. A typical enterprise uses more than 1000 cloud applications. These software tools fragment attention and make it harder for information to flow across an organization. So we want to do more and more to connect Slack to the critical tools behind every business.”

1071

To anyone at Frontiers, the number 1071 will feel at least familiarly significant. I counted it in no fewer than 6 different talks by various members of the Slack ensemble. It’s the average number of cloud applications in an enterprise. (Apparently from this Netskope report.) This was the strong theme of the conference: companies have tons of SaaS applications and Slack is the hub for collaboration, approvals, workflows, etc.

As an aside, Slack’s famously humble, empathetic culture was also on display. The second most common theme was speakers thanking the audience for laughing — just take your yuks and move on folks! (And we have the perfect hype man!)

Apps form silos. Silos impede cross-team communication and collaboration. If a task needs approvals from a bunch of folks in a bunch of tools, it’s going to be faster if it happens in Slack. If people use tools infrequently, offer common tasks through Slack. And rather than gathering data from multiple apps, bring it all together in the tool everyone’s already got open, Slack. It’s the aspiring epicenter of work.

Hub at the center of the unbundleverse

SaaS has fractured the traditional bundles. The new monopoly opportunity is horizontal. Rather than taking over collections of applications, these hubs will own common tasks across the landscape of apps. Slack wants to own collaboration and lightweight interactions across apps. It’s a monopoly that embraces a sprawling ecosystem rather than trying to own it. These catalyze expansion and heterogeneity where bundles sought to contain it. Every major software vendor is looking for their hub opportunity.

How will Slack become this hub? How will anyone? In their S-1 (published the day after Frontiers), Slack cited several components of their growth strategy. The first 6 are different aspects of sales and marketing. The 7th addresses their platform vision (and could be copied almost verbatim from other aspiring hubs):

Grow our application platform and developer ecosystem
We will continue investing to expand the number of developers building applications that integrate with Slack and to make Slack work with an increasing number of third-party and internally developed custom applications.

(The 8th and final growth strategy is “Leverage artificial intelligence, machine learning and advanced search”, a shibboleth of our times I suppose.)

For any hub, there’s the core platform, the surround of apps and APIs, and a yawning gap in between. Platforms and users alike are dependent on developers to fill that divide, to write the software that wires together hub and edge. From Zapier, to Airtable, and Transposit, that’s why we’re here. As much as the hubs will define our future so will the developers, tools, and platforms that bind it all together.

—

Building Slack bots or other connections between hub and edge? Try Transposit.

Enter Google Hire.

As anticipated, Google Hire wasn’t everything what we wanted. What it was though was cheap. It’s not terrible; it just doesn’t do anything quite the way we’d like. Filter by recruiter, handoff candidate ownership, find folks we haven’t pinged in awhile — Hire didn’t do any of those things.

No problem, right? That’s what APIs are for. Your service might not suit my needs, but an API lets me work around the parts I hate and build the extensions I need. Even if the product improved to meet our needs, there would always be a next need. But — amazingly for a Google product[1] — there was no API.

All product need APIs

No product is perfect for all users. APIs are the escape hatch. They’re the panacea for any complaint. They’re the answer for the long tail of features you’ll never ever get to.

With no API for Google Hire, we just suffered. And in the meantime we hired our team and built our product that — with some irony — made it easy to work with APIs. By a happy coincidence, Google Hire released their beta API right around the time Transposit opened our beta. Transposit puts a SQL and JS interface in front of APIs, manages auth, and makes it easy to build API-driven APIs. Here are the first couple of queries I ran against the Google Hire API. (You can try everything out live once Google gives you API access.)

I can list all the candidates we’re tracking:

SELECT * FROM google_hire.list_candidates
  WHERE tenant=‘my_tenant’

I can see the past employers of all the folks we’re talking to:

SELECT employmentInfo.employer FROM google_hire.list_candidates
  WHERE tenant=’my_tenant’ EXPAND BY employmentInfo

(Amusingly in the context of this post, the top hit is Google)

Solving problems

After being held at arm’s length for far too long, I had a lot of fun exploring the newly public(-ish[2]) API. The first real problem I wanted to solve was to show the candidates whom we haven’t pinged recently. Why is this not built in? I have no idea. It was easy enough to get to with Transposit. We treat each API like a view or table within a database; getting the data I wanted amounted to a JOIN between the candidates and applications tables:

[
  {
    "name": "tenants/abc/candidates/def",
    "job": "tenants/acb/jobs/ghi",
    "candidate": "tenants/abc/candidates/jkl",
    "createTime": "2019-04-02T00:31:48.630Z",
    "status": {
      "state": "ACTIVE",
      "processStage": {
        "stage": "Initial Screen",
        "reportingCategory": "SCREEN"
      },
      "updateTime": "2019-04-03T05:27:29.321Z"
    },
    "personName": {
      "givenName": "David",
      "familyName": "Lightman"
    },
...

I can focus that in on a particular job by adding another SQL filter:

… and then get the data I want by plucking out particular fields and adding an ORDER BY:

[
  {
    "updateTime": "2019-03-12T17:25:33.176Z",
    "givenName": "Michael",
    "familyName": "Jennings"
  },
  {
    "updateTime": "2019-03-21T20:33:08.454Z",
    "givenName": "Kate",
    "familyName": "Libby"
  },
...

Finally! The data I’ve wanted since we first started using Hire!

All Hail Our Benevolent Slack Overlords

Slack is rapidly becoming the central interface across so many tools we use, so I wanted the data from Google Hire connected up with Slack. Building a Slack slash command is a snap (check out the Transposit Quickstart). You just need to point Slack at a webhook that returns the data you want:

Now when I worry if we’ve dropped the ball on a candidate and they haven’t heard from us in a while, I can just type my Slack slash command:

Next

As excited as I was for the Google Hire API, the endpoints in the first beta are pretty basic. It will be extremely powerful when they open up the ability to do things like create new candidates, transition candidates between stages of the hiring process, and add comments (the mechanism we use to hand off candidates between members of our team).

An API is a critical feature of any tool we use; I’ll never again make the mistake of choosing a tool that doesn’t give me access to my data via an API. Transposit let me explore the Google Hire API interactively, check out the data, and poke around at different endpoints. You can try it out with the Hire API or any of the other APIs it connects to. And if this sounds like something you’d like to get involved in, we’re hiring (and we’re good at follow up).

SELECT title FROM google_hire.list_jobs
 WHERE tenant=’my_tenant’
 AND filter=’state=OPEN’

[
  {
    "title": "Developer Advocate"
  },
  {
    "title": "Growth Marketing Manager"
  },
  {
    "title": "Product Marketing Manager"
  },
  {
    "title": "Senior Site Reliability Engineer"
  },
  {
    "title": "Software Engineer"
  }
]

[1] I call it a Google Product — and I knew this beforehand — but Hire came from the Bebop acquisition that brought Diane Greene into Google. I suspect acquiring an amazing leader at the head of GCP may have been the primary goal.
[2] You need to sign up and get approved for the Google Hire API. I wrote a pleading, but firm mail to our sales rep to nudge things along.

About a year ago, my co-founder and I started work on Transposit. It’s not that weird to start a company, but one way in which we were an outlier is that my co-founder is a woman. Starting a company comes with many many lessons; some of the most surprising have come from sitting next to Tina: a front-row seat of what it’s like to be a woman in tech. I see these moments of profound obliviousness and lack of empathy and squirm. I apologize to the women reading this: I know everything that follows will be obvious (but hopefully not redundant). I hope for the men reading this that it will help them be more aware of the subtle (and glaring) ways tech is unwelcoming / discriminatory / hostile to women.

Disclaimers

This is not meant to be the breathless allybrag in vogue on Medium / Twitter / HN / LinkedIn. Greater awareness has, I think, made me better, but I have been flawed and unaware and (despite my best efforts) will be again at times. I’ve definitely fucked up, not realized, and not apologized. My hope is that others can learn from lessons that would have been instructive to me.

Others have and will write more authoritatively and more eloquently on the subject. Go read about the need for recognition, repentance, respect, and reparation from my friend, danah. Read this distressing story about two folks in a service role swapping genders and how massively their experiences improved/declined; I lamented how oblivious I must have been when running a technical support organization. And if you’d just like to cringe yourself into a migraine, either of these will do the trick: Yes, Uber board member David Bonderman said women talk too much at an all-hands meeting about sexism at Uber, Antisocial Coding: My Year At GitHub.

When you assume…

Tina and I attend a lot of meetings together. We meet prospective investors, advisors, partners, customers, colleagues (engineers!). We’re small and lean; we do our own scheduling. Often the folks we’re meeting with are big (and fancy); when it comes time to set up a meeting they have people for that. I can’t even count the number of times that scheduling emails drop me and go to Tina. Or I get invited to the scheduled meeting, but Tina doesn’t. The perpetrators of this are just as often women as men. I won’t claim I don’t know why they make this assumption, but it’s vexing that they continue to act on it.

When I was a kid I used to read Dave Barry’s column in a PRINT NEWSPAPER (millennials, if you made it this far, I’ll wait while you show yourselves the door); I think of this quote when people carelessly create these awkward moments:

You should never say anything to a woman that even remotely suggests you think she’s pregnant unless you can see an actual baby emerging from her at that moment.

I recently asked someone if they’d extend an invitation to my Chief Technology Officer, Tina; fine, he responded, as long as she’s technical. Each individual time a woman is presumed to be less technical or have a particular role might seem like not a big deal, forgivable, but think about what an asshole you look like when you’re the 1,000th person to do it. Think about Dave Barry’s advice.

Knock knock. Who’s there?

This was definitely a surprise: to some people Tina is, apparently, invisible. If I hadn’t witnessed it personally I might have found it hard to believe. We’ve sat through so many meetings where the person we were talking to never looked at her. This includes meetings with prospective customers (whose money we want) and prospective advisors (who, presumably, want our money). Not only is it demoralizing, but it makes it hard to participate — forget about inhaling or clearing your throat, the subtle social cues that we’d like a turn to speak. They wouldn’t see Tina if she were waving semaphores. It leads to these am-I-crazy moments where I ask, did that guy not look at you for the full 90 minutes?

Standing around at a conference I’ve seen men do this weird thing. Rather than facing inward to the group to create a multi-way conversation, they orient themselves to face me, physically closing Tina out of the conversation. My strategy is awkward overcompensation. I’ll try shifting my body to face Tina, directing questions to her, or excusing myself from the conversation briefly. It never works.

Take my wife… please!

I was at a tech meeting, and while you might assume without my stating it, the attendees were mostly men. In fact Tina was the only woman who wasn’t on the event staff. The presenter was discussing a use case with Pinterest as a motivating example. You might not know much about Pinterest, he explained, but ask your wife or girlfriend, ho ho ho. In moments like these I feel my brain waving a hasty adieu to my body: I know you need to stay, but I’ve got to get out of here. Why not use an example that you think will resonate with your audience? Or why not just explain the aspects of Pinterest that are relevant to the example? Less obviously misogynistic uses of the wife/girlfriend prop still serve to exclude and divide. I’ve been guilty of looking for that cheap laugh: me, ostensibly, hapless and my wife domineeringly keeping me in line. We feel like exclusion can create this greater notion of cohesion or rapport; it should be obvious, but we forget that it’s at the expense of the people we’re excluding (even if they aren’t in the room).

The devil you know

Casual sexism from strangers is a constant grind for women in tech. It’s even more toxic when it comes from close collaborators, people who shouldn’t just assume better, but should know better. Men, if your female peer gets credit that you rightly deserve — either in full or by proportion — take some solace that it’s the statistical equivalent of tripping over a SHA-1 collision. Examples of women being excluded or diminished happen every day in subtle and glaring ways; I continue to try to be more aware.

Second-Hand Misogyny was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building Modern Applications

A little backstory…

As CTO at Delphix, I spent a lot of time working with customers who wanted to build applications more effectively. A key precept of DevOps is to have production-like environments early in the development process: it’s critical for rapid iteration. Delphix brings this to applications built on large relational databases. Over time I started to see more and more applications where Oracle or MS SQL was just one data connection among many. As datasets become specialized and fragmented, applications need more connections to do the same job. Richer datasets create the desire for richer applications, leading to still more connections. Data, connections, and features all add to the cost and complexity of building applications. Test data is important, but, as developers, there is a growing inefficiency in how we build applications.

My co-founder, Tina Huang, has been building applications for her whole career. From application frameworks at Apple, to Google News and Twitter, she’s been building applications that drive those modern, scale businesses. In her spare time she’d build applications for friends: connecting billing to Google calendar for a dog walker, workflow tools for local businesses, and, most recently, writing the initial backend for acclaimed startup Winnie (like Yelp for parents). I met Tina at Sutter Hill Ventures where she educated me about some of the ways that writing these applications are a huge pain in the ass. From the basics like connection, authentication, and error handling to later issues like caching, pagination, and scale, there’s a lot more effort expended dealing with data than there is innovating, expressing the core idea in code.

A Common Goal

As EIRs at Sutter Hill, Tina and I started working with CIOs to understand their urgent problems. For what needs in the market could we develop new technical solutions? Naturally each CIO had a different perspective, but we started hearing some common themes:

“Enterprise data is a mess.”

“Our applications are constantly delayed.”

“When we upgrade SAP, applications eventually break in subtle ways.”

As the ideas for Transposit were starting to converge, one CIO just nailed it for us:

“Agile and DevOps are great, but the software lifecycle slows way down once you involve data.”

This so precisely encapsulated what we had seen in our previous roles and heard from these other CIOs. Tina lived that development lifecycle firsthand, seeing the complexities of building data-fueled applications. I saw it with customers as they improved a critical piece of the lifecycle, but discovered new and growing obstacles. Both Tina and I love building for developers, and we love solving critical problems. Shortly after that last CIO conversation we started building Transposit.

Transposit

Our mission is to make applications simpler to build. Interesting applications require data. More data from more sources — spanning old/new, on-prem/cloud, etc — involves more overhead. Transposit seeks to eliminate the drudgery of working with data, to let developers focus on their core intent. Developers hate writing one-off solutions to the litany of issues that surround these data interactions. Solving these in a framework, building durable components is what fires us up. The full software lifecycle involves testing, deployment, monitoring, debugging, upgrade, and migration. Our goal is to create better facilities for all of these than would be economical for most development teams to build themselves.

Interested in joining us on this journey? The product is in limited beta, and we’re signing up development teams who are looking for a better way to build their applications. We’re a team of five so far, and hiring more developers. If you’re a potential early adopter, team-member, or just someone who wants to find out more, we’d love to hear from you: info@transposit.com

A ZFS developer's analysis of the good and bad in Apple's new APFS file system

In that piece I left one big thread dangling. Apple shipped APFS as a technology preview, but they left out access to one of the biggest new features: snapshots. Digging around I noticed that there was a curiously named new system call, “fs_snapshot”, but explicitly didn’t investigate: the post was already too long (I thought), I had spent enough time on it, and someone else (surely! surely?) would want to pull on that thread.

Slow News Day

Every so often I’d poke around for APFS news, but there was very little new. Last month folks discovered that APFS was coming to iOS sooner rather than later. But there wasn’t anything new to play with or any revelations on how APFS would work.

I would search for “APFS snapshots”, “fs_snapshot”, anything I could think of to see if anyone had figured out how to make snapshots work on APFS. Nothing.

A few weekends ago, I decided to yank on that thread myself.

Prometheus

I started from the system call, wandered through Apple’s open source kernel, leaned heavily on DTrace, and eventually figured it out. Apple had shipped snapshots in APFS, they just hadn’t made it easy to get there. The folks at Ars were excited for a follow-up, and my investigation turned into this: “Testing out snapshots in Apple’s next-generation APFS file system.”

Snapshots were there; the APIs were laid bare; I was going to bring fire to all the Mac fans; John Siracusa and Andy Ihnatko would carry me on their shoulders down the streets of the Internet.

Sisyphus

On the eve that this new piece was about to run I was nervously scrolling through Twitter as I took the bus home from work. Now that I had invested the time to research and write-up APFS snapshots I didn’t want someone else beating me to the punch.

Then I found this and my heart dropped:

Rich Trouton on Twitter

Slides from the "Storing our digital lives: Mac filesystems from MFS to APFS" session at MacTech Conference 2016: https://t.co/uJJuqLL8n9

Skim past the craziness of MFS and the hairball of HFS, and start digging through the APFS section. Slide 49, “APFS Snapshots” and there it is “apfs_snapshot” — not a tool that anyone laboriously reverse engineered, deciphering system calls and semi-published APIs — a tool shipped from Apple and included in macOS by default. F — .

Apple had secreted this utility away (along with some others) in /System/Library/Filesystems/apfs.fs/Contents/Resources/

What To Do?

The article that was initially about a glorious act of discovery had become an article about the reinvention of the wheel. Conversely, vanishingly few people would recognize this as rediscovery since the apfs_snapshot tool was so obscure (9 hits on Google!).

We toned down the already modest chest-thumping and published the article this morning to a pretty nice response so far. I might have happier as a FAKE NEWS Prometheus, blissfully unaware of the pre-existence of fire, but I would have been mortified when the inevitable commenter, one of the few who had used apfs_snapshot, crushed me with my own boulder.

Testing out snapshots in Apple's next-generation APFS file system

I’m not the first person to hit this. The problem seems to have existed since CS6 was released in 2012. None of the solutions was working for me, and — inspired by Sara Mauskopf’s excellent post — I was rapidly running out of the time bounds for the project. Enough; I’d just DTrace it.

A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can’t imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I’m forever grateful to the Apple team that ported it to Mac OS X).

First I wanted to make sure I had the name of the Illustrator process right:

$ sudo dtrace -n ‘syscall:::entry{ @[execname] = count(); }’
dtrace: description ‘syscall:::entry’ matched 500 probes
^C
pboard 1
watchdogd 2
awdd 3
...
com.apple.WebKit 7065
Google Chrome He 7128
Google Chrome 8099
Adobe Illustrato 36674

Glad I checked: “Adobe Illustrato”. Now we can be pretty sure that Illustrator is failing on setrlimit(2) and blowing up as result. Let’s confirm that it is in fact returning -1:

$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }'

dtrace: description 'syscall::setrlimit:return' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0    532                 setrlimit:return -1 1

There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let’s confirm that it is in fact setting the files limit and check the value to which it’s being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process’s address space:

$ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this->r.rlim_cur, this->r.rlim_max);  }'

dtrace: description 'syscall::setrlimit:entry' matched 1 probe
CPU     ID                FUNCTION:NAME
  0    531              setrlimit:entry 1008 2800 7fffffffffffffff

Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMIT_NOFILE | _RLIMIT_POSIX_FLAG). Illustrator is trying to set that value to 0x7fffffffffffffff or 2⁶³-1. Apparently too big; I filed any latent curiosity for another day.

The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy:

$ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->i = (rlim_t *)alloca(sizeof (rlim_t)); *this->i = 10000; copyout(this->i, arg1 + sizeof (rlim_t), sizeof (rlim_t)); }'

dtrace: description 'syscall::setrlimit:entry' matched 1 probe
dtrace: could not enable tracing: Permission denied

Oh right. Thank you SIP. This is a new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it’s new I haven’t yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It’s easy enough to disable, but I’m reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb.

First I used DTrace to find the code that was calling setrlimit(2): using some knowledge of the x86 ISA/ABI:

$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato" && arg1 == -1/{ printf("%x", *(uintptr_t *)copyin(uregs[R_RSP], sizeof (uintptr_t)) - 5) }'

dtrace: description 'syscall::setrlimit:return' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0    532                 setrlimit:return 1006e5b72
  0    532                 setrlimit:return 1006e5b72

I ran it a few times to confirm the address of the call instruction and to make sure the location wasn’t being randomized. If I wasn’t in a rush I might have patched the binary, but Apple’s Mach-O Object format always confuses me. Instead I used lldb to replace the call with a store of 0 to %eax (to evince a successful return value) and some nops as padding (hex values I remember due to personal deficiencies):

(lldb) break set -n _init
Breakpoint 1: 47 locations.
(lldb) run
...
(lldb) di -s 0x1006e5b72 -c 1
0x1006e5b72: callq  0x1011628e0     ; symbol stub for: setrlimit
(lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90
(lldb) di -s 0x1006e5b72 -c 4
0x1006e5b72: xorl   %eax, %eax
0x1006e5b74: nop
0x1006e5b75: nop
0x1006e5b76: nop

Next I just did a process detach and got on with making that holiday card…

DTrace Every Day

DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn’t write a program doesn’t mean you can’t fix it.