I have a confession that might make some people uncomfortable: I am running my personal life and household on AI agents.

Not in the way most people mean when they say that. I don’t have a chatbot that tells me the weather or summarize my unread emails. What I have is Claude Cowork sitting on top of my emails, my texts, and my calendars — reading, triaging, and acting on them — while I sleep. Every morning by 6:30 AM, my inbox has been scanned and anything important filed. My wife’s shared calendar and mine have been cross-referenced, conflicts flagged. The weekly meal plan is written and pushed to the family workspace. The grocery shopping list is auto-generated on Todoist based on the meal plan — what we’re cooking, what’s already in the pantry, what needs restocking.

You don’t need another app or subscription to do this, this is just Claude Cowork. You can make it write all your email drafts, even teach it when you want it to draft one. You can make it create tasks based on your emails (and close them automatically when a new email comes in and resolves it). Your wife says you are low on olive oil, Todoist gets a new task (oh, and dedupe please). The sky’s the limit. Most of this kinda works already out-of-the-box if you have a Claude subscription.

There are 154 tools involved. Six MCP servers. It updates every 15 minutes. My wife doesn’t know how any of it works, but she knows the meal plan shows up in Craft every Saturday morning and the grocery list is on Todoist by Sunday, and that’s the part that matters.

Here’s why this matters beyond my household: the moment your AI can see your emails and your texts and your calendar and your to-do list at the same time, it stops being a chatbot and starts being something closer to a chief of staff. My wife texts me that she canceled the restaurant booking because her mom can’t make it — Claude sees the WhatsApp message, finds the matching calendar event, removes it, and flags the open evening in my daily briefing. No one told it to do that. It just knows, because it has context across all three channels. That’s the unlock — not any single integration, but the composition of all of them running in the same reasoning loop.

At this point, you can just go to https://endara.ai if you just want to download and run the gadget that I built (instead of reading why I needed to build something in the first place).

It started with emails. I have three email accounts — personal, work, and one for a construction project I’m managing. I wanted Claude to scan all three, flag what matters, and file everything into a shared workspace. MCP made this possible — Gmail MCP servers exist, they work, they’re great. But Claude can only access one Gmail account at a time.

Then texts. I live in Asia, where WhatsApp is the communication layer. My son’s school, our nanny, my parents, my wife’s parents — everything runs through WhatsApp. I wanted Claude to read my messages and know what’s going on without me having to relay context manually. But there was a problem: if Claude reads my personal WhatsApp and also sends messages from it, people would get texts from me that… aren’t from me. Unsettling, to say the least.

So first side quest, I built a WhatsApp MCP server. A full Electron app — Baileys for the WhatsApp Web protocol, SQLite for message sync, multi-account support so I could connect two numbers: my personal account (read-only, Claude can see but never send) and a separate number that acts as my PA (read and send, with a different identity). I open-sourced it: github.com/panghy/whatsapp-mcp-server.

Then calendars. Multiple Google Calendar accounts, same pattern — personal, work, shared family calendar. Claude needs to see all of them to reason about scheduling.

Then Todoist for grocery lists and household tasks. Then Craft for family documents and meal planning. Then GitHub — and this is a small thing, but it’s the kind of thing that adds up: Claude’s built-in GitHub integration uses OAuth, which somehow doesn’t see across organizations. I just run the GitHub MCP server as a STDIO process with my own personal access token (PAT), and it sees everything. You could configure this in Claude’s config file directly, but then you’re back to maintaining per-client configs, and Cursor doesn’t see it, and the token lives in a JSON file on disk. Through Endara it’s configured once, available everywhere.

Each server worked. Each server was its own little miracle of open-source engineering. And each server was also its own configuration, its own process lifecycle, and its own failure mode.

Here’s what my Monday mornings looked like around February:

6:30 AM. Wake up. Open Claude. Todoist tools are gone. Something about the token — Claude just stops seeing the server. Re-add the config, restart, wait 30 seconds, try again. This happened constantly.

6:35 AM. WhatsApp tools are there, but only one account. I need two: one read-only connection to my personal WhatsApp, one write-enabled connection to the PA number. Claude doesn’t support connecting two instances of the same MCP server with different configurations. So I’m running two copies of the server on different ports and pretending they’re different servers.

6:40 AM. Gmail and Calendar are working. But now I open Cursor for some morning coding and — none of these tools are available there. The config is Claude-specific. Want the same tools in Cursor? Duplicate the entire configuration. Update one, manually update the other. Inevitably forget.

6:45 AM. Everything is finally running. Claude has 154 tools available. Now, both Claude and Cursor have gotten smarter about this — they do tool searching and lazy-loading so they’re not dumping every definition into context at once. But 154 tools is still 154 tools. The token budget isn’t free, and when the model needs to chain three tools across three servers in a single reasoning step, those definitions add up fast.

This is the duct tape problem. Each piece of tape works. The thing it’s holding together works. But you spend your mornings re-applying tape instead of doing the thing the tape was supposed to help you do.

The engineer in me — the one who spent three years building a financial exchange from scratch, and then seven years building an observability platform that got acquired, and then a few more years watching what it means to operate infrastructure at planet scale — that engineer looked at this duct tape and saw a missing layer.

MCP had won. It’s the protocol. Anthropic, OpenAI, Google, Microsoft, Amazon — everyone adopted it. There are tens of millions of monthly downloads of the MCP SDK. There are thousands of servers. But there’s no infrastructure. No aggregation layer. No way to say: “Here are my seven servers. Give me one endpoint. Handle the auth. Handle the reconnection. Handle the tool explosion. Let me point any client at it and just have it work.”

So I built one. I called it Endara.

Endara is two things: a relay and a desktop app.

The relay is a Rust binary that aggregates MCP servers behind a single endpoint. You add servers — STDIO, SSE, HTTP, OAuth — and it proxies them all through localhost:9400/mcp. One URL. One config. Every client that speaks MCP can connect to it. Claude, Cursor, Windsurf, VS Code, ChatGPT — they all point at the same endpoint and see the same tools.

A few things that turn out to matter a lot:

Tool prefixing. When you have 154 tools from 6 servers, name collisions are inevitable. The relay prefixes each tool with its endpoint name — todoist__find-tasks, craft_household__documents_search, clement_whatsapp__get_chat_history — so there's no ambiguity.

OAuth built in. Sign in to services through your browser. Endara handles token refresh, reconnection, and health monitoring. My tool connections haven’t dropped since I switched.

JavaScript execution mode. Claude and Cursor already do tool searching to keep context manageable — but each tool call is still a round-trip through the model. The relay takes it further: it exposes three meta-tools (search_tools, list_tools, execute_tools), and the AI writes a JavaScript snippet that chains multiple tool calls in a single execution. The relay runs the script in a sandboxed runtime. One model turn, three tool calls, no round-trips. 154 tools collapse to 3 definitions in context, and multi-step workflows that used to cost five model turns now cost one.

Hot-reload. Edit the config, the relay picks it up. No restart, no reconnect.

The desktop app wraps the relay in a native macOS app (Tauri + Svelte). Menu bar icon, endpoint dashboard, OAuth login flows, a tool browser, real-time logs. You can add a server from a curated marketplace with one click or paste in a custom server URL.

Here’s what my Monday mornings look like now:

6:30 AM. Wake up. Claude already ran six scheduled tasks overnight: scanned my emails across all three accounts and filed anything important, planned the week’s meals and pushed them to Craft, generated the grocery list on Todoist based on what we’re cooking this week, groomed my own task list, and sent a WhatsApp summary of today’s schedule to the family group chat.

I open my phone. Everything is there. I didn’t configure anything, restart anything, or re-authenticate anything.

That’s it. That’s the product.

I want to be precise about what I think is interesting here, and what isn’t.

What isn’t interesting is that I automated my morning routine. People have been doing that with Zapier and IFTTT for a decade. What’s interesting is the shape of the infrastructure that made it possible. MCP servers are the new integration layer — they’re richer than REST APIs, because they carry tool schemas and structured output that AI models can reason about natively. But they were designed to be consumed one-at-a-time, by one client, on one device. The moment you want to compose them — the moment you want an AI that can read your texts across two numbers, check your calendars across three accounts, search your docs, and manage your tasks in a single reasoning loop — you hit the duct tape wall.

Endara is the layer that makes composition work. One endpoint, every tool, any client, any device.

It’s open source. You can get started at endara.ai. The WhatsApp MCP server is also open source: github.com/panghy/whatsapp-mcp-server.

I’ve been dogfooding this daily since early April. It’s early — v0.1.5, rough around some edges, thoroughly tested in others (the relay has over a thousand unit tests, because that’s who I am and I’m not going to pretend otherwise). If you’re the kind of person who runs multiple MCP servers and has felt the duct tape problem, I’d love to hear how you’re solving it today.

There’s a photo on my phone of my desk at the GLMX office in downtown Palo Alto. A laptop, two Mac monitors, a lamp, a jacket thrown over the chair. It’s grainy in the way phone photos from 2011 are always grainy. I no longer remember which evening I took it — there were a lot of evenings that looked like that — but what I do remember, very clearly, is the hotel room a couple of years later, during go-live week. My CEO had told me to stay close to the office for seven days because trading was scheduled to start around 5 AM, and somebody needed to be on call when it did.

GLMX was the exchange we’d spent the previous two years building. Big banks — Bank of America, JPMorgan, Chase, all the usual suspects — had signed letters of intent saying they’d trade on our platform. They were all provisioned: logins created, centers set up, currencies loaded, credit lines entered. Our front end was in Flash, running inside Internet Explorer (I told you this was a while ago), and I remember the feeling of refreshing the admin dashboard the morning of day one, in a hotel bathrobe, and seeing it: one trade.

For the whole first week, we had one trade.

For a product designed to replace voice brokers, one trade a week was worse than not having the product. The voice brokers were cheaper and, for a single transaction, just as fast. We’d built the thing right. It was correct, it was scalable, it cleared. We had built the wrong thing.

There’s a book by Alberto Savoia called The Right It that I keep returning to. Alberto was a VP of Engineering at Google for a long time before he wrote it, which matters, because the argument he makes is one that only somebody with real engineer’s scar tissue can make without sounding glib. The argument, stripped down, is this: engineers tend to build it right, but not build the right it. The package layout, the test coverage, the algorithmic elegance, the checkstyle-green commit — all of that is the easy problem. The hard problem, the one we don’t have a compiler for, is whether anyone actually wants the thing once it’s built.

My favorite story in the book is the one about IBM’s speech recognition work in the early 70s — the research that eventually became ViaVoice. Before they built it, they wanted to understand whether anyone would pay for speech-to-text, and how people would actually use it. So they set up a room with a computer in it, invited people in, and told them: “This machine will transcribe your voice as you speak.” People sat down. They spoke. Text appeared on the screen. People were amazed.

Of course there was no speech recognition. There was a typist in the next room with a keyboard. But within a couple of weeks IBM had learned the single most important thing they could have learned, which is that nobody was going to dictate their emails — because then the entire office would hear what you were writing to your boss. No amount of algorithmic sophistication would have surfaced that. No market study, no focus group, no McKinsey slide. Just a typist in the next room, and the cheapest possible simulation of the finished product.

The reason the story lodged in my head is that I built the wrong exchange with three years of extremely careful engineering, and the market didn’t bother telling me the truth until the first week of live trading. A typist in the next room would have told me in a month.

That was fourteen years ago. Three startups, one acquisition, and approximately a thousand refactors later, I’m an Entrepreneur in Residence at Sutter Hill Ventures — which is a polite way of saying I get to spend my days trying to figure out what to build next. And because of that job, I am writing a lot of exploratory code right now. Week-long spikes. Throwaway prototypes. Proofs-of-concept I don’t expect to survive past Friday.

Here’s the uncomfortable thing I’ve noticed about my own process lately: I’ve stopped running checkstyle on those spikes. I let Claude write the unit tests. I let the package layout be whatever it wants to be. If I see a function I’d normally extract into its own file for readability, I leave it. If there’s a helper I’d normally promote to a module, I don’t. I just keep going. I want to put the thing in front of somebody — in front of myself, most of the time — and see if it does the thing I’d hoped it would do.

And I have to tell you, this feels wrong to me. In a very specific, bodily way. I am the guy who skipped his next-door neighbor’s party freshman year at Rice because I was typing and refused to look away from the monitor. I am the guy whose former colleagues used to joke about a “Clement run-over-by-a-bus” problem at every company I’ve been at, because I’d internalized so much of the infrastructure that nobody else could touch it. I am, let me be clear, the guy who would spend an entire afternoon refactoring a working piece of code into a slightly nicer shape because the existing shape gave me a little twitch every time I looked at it.

What’s actually happening to me isn’t that I’ve learned to let go. It’s that the cost of building the wrong thing has finally collapsed enough that I can no longer justify the twitch.

That, I think, is the one real thing AI has changed in my working life. It isn’t the agents running in parallel. It isn’t the junior devs on tap. It isn’t getting 80% of a CRUD app out of a single prompt. All of those are features of build it right, faster — they’re the same movie I was watching in 2013, just at 2x speed. They make me more efficient at the thing I was already too good at.

The actual change — the one that matters — is that AI finally removed the last excuse I had to keep being the kind of engineer I already was. For fifteen years I could always tell myself that the reason I was going deep on the implementation before going deep on the customer was that the implementation had to be this careful. These are financial systems. These are planet-scale observability systems. We are not allowed to get this wrong. And some of that was even true. But underneath the truth was a preference, which was that I liked the craft more than I liked the question, and the craft was something I was already good at, and the question was something I was bad at and a little bit afraid of.

I still have that office desk photo in my phone. Every so often I’ll be on a plane and it’ll surface in Memories and I’ll look at it and think: I was so careful. I was so careful about all the wrong things.

I should be clear about one thing, though: I haven’t stopped caring about the craft. The code I ship to production still needs to scale, still needs to be resilient, still needs tests I’d be willing to defend in front of an on-call pager at 3 AM. What AI has actually changed for me isn’t whether any of that matters — it’s the order. I used to build it right, and then find out whether it was the right it. Now I find the right it first, in the ugliest and most unrefactored and most embarrassing form I can stomach, and only then do I come back and give it the engineering it deserves.

The guy sitting at that office desk in 2011 would still be horrified by the spike branch I’m running this week. But I think if I could sit him down and explain it, he’d eventually get there: the goal was never to stop building it right. The goal was to stop building it right first.

After almost a two-year hiatus (and as a result of me finally having some time to pause and reflect on the larger story of Wavefront), we return to the fundamental question of Why Wavefront? (shamelessly borrowed from our ex-colleague Matthew Zeier)

Why Wavefront?

Our ex-Director of Operations, Matthew Zeier (whom we all refer to as mrz) explains the platform through a series of answers to the question “Why Wavefront?”. To engineers who have operated any software at scale, the concept of measuring something and observing it overtime shouldn’t be a new concept. The question one should ask is why could one build a company and a business around it.

If one is familiar with the plethora of “monitoring” solutions (the gamut spans from open source tools like graphite to commercial ones such as New Relic), another question that one could ask is whether there is any white space for building something and having customers pay “additional” money for it (let’s face it, most shops already have some sort of monitoring).

When we started ideating about Wavefront, logs (free-form, unstructured text, mostly indexed) were the primary means of observing a production system (generated by logging frameworks or System.out.println()). There were some nascent efforts in the open-source world to build time-series databases (Graphite and OpenTSDB come to mind). Still, they were very much limited to infrastructure metrics (CPU, memory, disk, and network usage). The idea of using time-series to observe code is just starting to percolate in the developer world with things like Yammer metrics. This trend was somewhat driven by the log-less culture in Google (where logs refer to structured, binary blobs, while stdout logs are discarded rapidly after collection). As engineers moved around companies, they started to influence their new companies on how to operate their respective software stacks. It’s the dawn of a metrics “decade,” and we are mostly still in this industry-wide journey of recognizing what should be collected, named, visualized, alerted, and retained in the world of time-series data for operations.

Wavefront was a journey that four of us undertook in 2013 without knowing full-well whether we could in-fact build something fundamentally better than the open-source offerings or whether there would be a market large enough to sustain a business. We, however, knew that the multi-billion dollar IT monitoring market existed even back then, and hence it was primarily a question of “can we build it?” and not so much “if we can build it, will they come?” (meaning if we could build something substantially better, we believe we can displace existing players at the very least).

Why Metrics?

The word “metric” is heavily overloaded as you’ll discover even in the Wavefront domain. Metrics in the general business sense are numerical (or quantitative) measurements that one could rely on to make decisions. They are different from qualitative (often subjective) opinions that somewhat rely on someone having the right “hunch” or a unique “eye” for things. As the movie Moneyball suggests, we are no longer living in a world where data-driven decisions are either impossible due to the sheer scale of the data volume or because of the lack of collection mechanisms (having the relevant probes). We are also able to crunch data at planet-scale and derive insights that could drive decisions as to, for instance, whether a movie script would get the green-light to production or should be shelved.

Similar quantitative measurements also meant that in the computer software and service industry, operators of complex computer systems no longer have to guess at problems or look at second-order or third-order effects to judge whether a bug exists or pinpoint a performance issue. Even with voluminous logging, one would be surprised at how bad humans are at computing rates of occurrences or even simple ratios between two sets of events by staring at log lines flying by. Worse yet, certain attempts to pre-compute rates, ratios or aggregations (and then logged as text) meant that analyzing systems at scales is impossible at best and misleading at worst (the classic example of pre-computing local p95 and averaging or computing the max across a population of systems come to mind).

In a nutshell, metrics are simply timestamped measurements that we can name. They can be logged as textual content for sure, but, as anybody will discover, storing them in their “native” format (hence with restrictions on what a single “point” ought to look like) can lend themselves to blistering fast retrieval, vastly reduced storage requirements, and post hoc analysis that shoving a bunch of textual content around could not do.

Metrics are not restricted, of course, to just the bare observations that a probe might collect. In the examples above of local pre-computations, the derived data themselves are also timestamped (examples include the rate of occurrences of a particular event, for instance, as well as the p95 of sampled events that have numerical properties, say, the latency of a request). The trade-off here is that the “closer” to the source of a metric, the more likely it lends itself to meaningful analysis post hoc.

An example of this is that while it is possible to measure CPU usage per second and send the information to a time-series database. It is, at best, a second-order approximation to potential performance issues on the machine. A much better way to isolate the problem is to measure, on a per-request level, the number of CPU seconds expended by all threads to service the request, the actual time spent in critical components, as well as the number of times these components were called. Errors can also be similarly observed accurately by components themselves emitting error metrics (in the form of counters typically) instead of measuring HTTP error rates after the whole request chain has failed, and the error has percolated upstream.

Why Metrics #1: Metrics allow surgical measurements of event occurrences in a system.

Such measurements are possible because of the proliferation of metrics libraries for different languages (e.g., Dropwizard Metrics, Spring Micrometer for Java, and go-metrics for Go, for instance). Such measurements are typically collected in three flavors: gauges, counters, and histograms.

Gauges

Perhaps the easiest to understand, a gauge is like the speedometer of your car. We can, in fact, sample the speed of your car every second, or even every millisecond, and there would be a valid, timestamped value. Hence, metric systems have to decide how often we need to collect and report the values with the idea that the more frequently that we do, the more insight we may be able to glean from the data (for instance, sub-minute microbursts of traffic will require per-second level metrics in order to detect).

Examples of gauges include things like CPU utilization (sometimes in aggregate by usage type or further by cores), number of elements in a cache, or the number of active users in a system (as determined by unexpired sessions for instance). One property of gauges is that their values can be integral or floating-point and that their values can go up or down.

Counters

Counters are perhaps a special form of gauges that never goes down. Before we explore this particular metric type, it is helpful to understand that time-series analysis often involves computing rates. For instance, the rate of requests against a system or the rate of errors of a particular method call against a component in Java.

Traditionally, systems have computed rates by intercepting occurrences of an event and emitting a rate (/s) over different time-windows (such as the rate of occurrence of an event per second over the last minute, the last 5 minutes or the last 15 minutes). Such computations usually involve a timer of sorts on the process itself so that a rolling-window calculation can be done with the relevant bins discarded over time. As systems matured and the need to compute rates of occurrences beyond fixed time windows, the concept of cumulative counters came into purview. Instead of locally computing rates, occurrences of events are simply added together and the aggregate value emitted (for instance, the total-lifetime calls to a component, or the total life-time CPU-seconds spent in executing a portion of a request tree).

An added benefit of storing monotonically increasing counters is the ability to detect “resets” and infer that a process has died, and the data (for instance, when it takes part in an aggregation) is incorrect until another collection interval has elapsed after the reset is detected. It also provides an easy way to know the life-time occurrences of events in a single process (the integral of all the rates emitted if you will).

To compute rates, a time-series database will have to be combined with an analytics engine that can take pairwise values and divide that by time elapsed. The simplest analogy could involve counters and timestamps as two columns on a spreadsheet in Microsoft Excel and rates as a new column with the formula to combine two cells, dividing against time to produce a rate.

Why Metrics #2: Metrics can be analyzed as a stream of numbers whereby local pre-computations can be eliminated in favor of post hoc analysis

At this point, I should mention that there is a special atom within Wavefront known as the Delta Counter (emitted with the metric name prefixed with the delta symbol: Δ). This was introduced for large scale aggregations of “deltas”, whereby the aforementioned cumulative counters might not make sense.

For instance, if we were to compute the rate of requests across all mobile phones on a platform with millions of users, while we can model them each as singular monotonically increasing series (with much of them being ephemeral as the app might become active only minutes every day), it would be better for each session to send timestamped “deltas” (e.g., +5 at 12:30:00PM; another +2 at 12:30:05PM for instance) and allow the time-series database to “bin” them appropriately and produce global rates accurately and quickly. Often times, this is used to reduce the “individuality” of a time-series. However, it is indeed possible to send both and retain the ability to look at individual user session’s request patterns, for instance, over time while having the ability to access pre-computed delta counters to look at rates over large populations (for instance, for a particular version and platform).

Why Metrics #3: Metrics can be pre-aggregated to produce lightning-fast aggregations at scale as long as the data type and operation are monoids

In the statement above, we introduce the concept of monoids, a fancy way of saying that the axiom of associativity (a • b) • c = a • (b • c) holds and the identity element e • a = a • e = a exists. In practice for metrics, we find that the implementation of delta counters, histograms, and HLL (HyperLogLog) benefit from monoidic operations because large-scale aggregations are possible without having to deal with the ordering of data, without needing any state information along the way, and operations can be replicated across the wire without any concept of existing values.

Histograms

With the concept of monoids out of the way, we introduce the concept of histograms in the context of metrics. Histograms are ways to analyze distributions. They are oftentimes visualized with bins and counts (think the population of a state in the US broken down by age groups). To compute a histogram, one would need a numerical property that exists for all members of a group (for the population, that would be their age; for a computer system, that could be the latency of a request, the size of a payload, the number of database calls needed to service a request, etc.).

Collecting this data turns out to be rather easy (oftentimes just “I saw something with a value of X”). However, the reporting of such data isn’t as straightforward. Revisiting the topic of local aggregations, metrics libraries will often produce statistical measurements of the distribution over-time. The statistical summaries include mean, median, min, max, p25, p90, p95, p99, p99, etc., as well as rates in all its pre-computed flavors (after all we are observing events that happened to have a property we would like to compute a distribution of).

One classic problem that is perhaps immediately obvious to the reader is that the distribution itself can change rapidly over time, and such summaries might vary wildly depending on how we draw the “boundaries” of what we need to store in memory before we compute the results. As such, it is common practice to use decaying samples (i.e., older samples contribute less to the statistical summaries) or simply outright discard the data and emit these statistical summaries at some predefined intervals (every minute for instance).

Collection and reporting of these statistical summaries aside, local computations suffer from similar issues as counters (albeit more pronounced) since the only summaries that are easily combinable across distributions are probably the {sum, count}, which does, in fact, allow one to compute the global mean, total counts, total sum, and rates. Any other statistical summaries, however, cannot be combined across processes without us first storing all observed measurements. For example, to compute the p99 latency of a system that consists of 2 machines, one does need to collect the latencies of all requests, sort them by duration, and pick the p99 element. The maximum or average of the two local p99s are not at all the true p99.

Because of this limitation, Wavefront (as well as some competitors, although this is not yet a common feature of observability platforms), allows customers to ingest histograms in the form of centroids and counts. Simply put, this is a data structure that can, in some cases, store every observation (say you have only 30 distinct values for latencies as measured in milliseconds; you can represent that distribution in perfect accuracy by sending us those 30 centroids and counts over the course of a minute, an hour or even a day). Those time bins are predefined by us, but because of the mergeability of the data-type, we can compute 5-minute histograms in real-time. And, because we have the raw observations, we can also compute the true p99 in those 5 minutes. This also means we can merge independently collected histograms across machines and produce the true p99 of a population of systems, all observing the same events passing through them.

However, as the astute reader would note, the number of centroids might be much larger than 30 (the default accuracy is 32 in the system today). This is where TDigests comes in. In his paper on TDigests, Dunning describes the need for a histogram compression algorithm that maintains tighter errors bounds towards the extreme percentiles of distributions (with e = q (1-q), hence the error can be up to 25% for the median). By prioritizing the extreme percentiles, the argument is that the ability to compute p99, p999 and even p9999 is more valuable than the median (a fact that’s largely true as analysis of running systems often involves looking at those extremes and trying to understand what’s going on, rather than understanding the behavior of the system between p50 and p60).

As the histogram data-structure and the merging of them are monoidic, one can emit histograms from a large variety of sources and any metrics platform, like Wavefront, can aggregate them upon ingestion (this is hidden from the user), producing combined distributions that, while continuing to sacrifice accuracies in the middle of the distribution, capturing accurate percentile measurements.

Why Metrics #4: Metrics can capture high-fidelity distribution statistics that are time-series in nature but allows for accurate post hoc population analysis and statistical summaries

The statement above is important because where-as metrics are by definition “detached” from individual events (they are summaries of occurrences), their strength is that we can very quickly look at trends and understand the distribution that contributes to those trends, alert upon them, visualize them, and because of that, allows the operator of complex systems a way to look at what’s going on underneath the hood without having to peer into every event itself. It gives you the forest so you can spot the trees, so to speak.

Why Metrics #5: Metrics are summarizations of events that allow an operator to quickly isolate a problem or make informed inferences about what’s going on

An example of the above is the “there exists” question. It is often important to know whether something definitively happened in the system or it didn’t. Many bugs and code push happen because developers think that the code is not being executed and/or that an error condition has occurred somewhere. Traditional logging or even distributed tracing would require full retention of all requests before one can definitively prove that. With metrics, it is relatively easy, as long as the code in question emits a single metric, to prove that a code path has never been executed or that there was not a change in the way that it is called, compared to the rest of the population, say, before and after a code push.

Distributed Tracing

Having just perhaps pointed out the inadequacies of having distributed tracing, there are perhaps obvious reasons why one would need distributed tracing: the need to look into individual requests (including every downstream call) and isolate what’s going on between components in a complex system. It is an offering that was added to the platform shortly after histograms were introduced (hence the 3rd atom of the platform, so to speak), and it represents yet another angle to observing a running system.

Distributed tracing has been around for more than a decade when we decided to embrace it (fun story, we originally thought we could build Google’s Dapper as a SaaS service only to realize that without frameworks that came later, such as OpenTelemetry, we could only not infer causal relationships and can only compute probabilities across spans, something that we thought wasn’t useful enough to warrant building a platform for in 2013). There were libraries and tooling such as Zipkin and Jaeger that allowed users to annotation their code to emit spans, store them for future analysis, and visualize them in ways that allow one to look at flame graphs.

The reason for us entering into the foray was two-fold: a) we did not want to fight the logging battle yet (something that we eventually did in 2020) and b) having built a planet-scale ingestion engine, a query language for metrics, enterprise features around policies, security, HA, DR, and backup, etc., we thought we would have a shot at claiming territory by taking the core engine and allowing the ingestion of spans (which, if one were to look at them as metrics, are similar to values being durations and timestamps being the starting time of operations).

Another strength of the platform for us venturing into distributed tracing is the fact that we also happen to have first-class support for metrics, counters (introduced in late 2019), and histograms. Anecdotal evidence suggests that while teams think they need tracing, the utilization of these platforms is abysmally low. One possible issue is that alerting on traces largely involves derived metrics (if one only has traces) and a platform with only traces is unlikely to see other data points to determine whether there is a real issue (you’d also need to build all the analytical features of Wavefront in order to support various use cases). This meant that derived metrics are just duplicated in a customer’s metric system (visualized, alerted, and analyzed no doubt), while traces are consulted only when necessary.

For Wavefront, as we have long-term retention of data (contractually at 18 months), and our charging model gives away span for free while charging only on the derived metrics (controllable by the user as to the dimensions that we collect): the alerting problem is solved (one would alert on the metrics), the analytics problem is solved (we can convert spans back to metrics and one can slice-and-dice the data to your heart’s content; the derived metrics are also stored for 18 months, so you have a “gentler” cliff when the data is purged), the ROI problem is solved (spans are free, and one would need to pay for the metrics anyways if running anything of substance), and the usage problem is solved (we can link the data from the metrics you’re looking at to traces that might be of interest, increasing the chances that during an outage, an engineer might be prompted to “jump” to the relevant traces that shows the root-cause).

Intermission

As I begin my fourth year at VMware, having recently moved back to Hong Kong temporarily, I hope to be writing more, mentoring more, and yes, building more. 2020 is a seminal year for many, I consider myself very blessed that I still get to do the things I love, with people I love.

How do you define “up”?

This is a tough question since with modern microservice architectures, you can almost always serve a page (that’s different from serving all requests properly of course). For Wavefront, we have a mix of flight.js and React.js code (transitioning to the later) so even if AWS is down, a browser with everything loaded will continue to “work” until you refresh the page.

Arguably, you can look at our SLA documentation on how we define “up” (something that our customers tell us isn’t that easy to get from our competitors which is odd). It defines an outage (at 99.95% availability monthly, you have basically 20 minutes) as having no points ingested for 3 minutes or more, no queries executed for 3 minutes or more, no alerts checked for 3 minutes or more (and things of that sort). Having an SLA matters mainly because then there is a system by which you can credit customers (Algolia, which is a Wavefront customer and us their customer, offers 100x rebates; we offer 10% of the bill for each incident, note but I am not a lawyer, please really read the fine print =p). But in the final analysis, the fine print doesn’t really define what we mean by the system being up.

Why?

The reason is that if the system is slightly degraded, many of our customers immediately go into code/deployment freeze. It might not even be us that’s having the issue even (our MillionEyes project has probes around the world to make sure we aren’t having an upstream issue). It doesn’t help anyways that we can credit customers when the system is “out” for 5 minutes and queries just took longer to load, we will be getting a call and it’s all hands on deck to figure out what the issue is. Engineers don’t care about credits.

For that reason, operating Wavefront meant tons of engineering hours trying to hit our SLIs and SLOs.

SLIs and SLOs

SLIs are indicators (metrics) that we measure that indicate the general happiness of the platform. Since we offer high-resolution histograms (basically TDigests as a service, but that’s really downplaying what it is to be honest), tracking medians, p95 and p99 of all kinds of operations and computing the true measurement of those across a cluster is quite easy. Query latency is obviously a critical metric, ingestion push backs (which are quite rare these days but it could be that a rate limiter tripped) and then finally the alerting period (alerts are ran every minute, it doesn’t matter if you have tens of thousands of alerts or just a handful, our system will automatically scale up to handle them but it’s always possible that you added a thousand more and we need to scale up).

Since we operate in HA configurations (more about this later), 1) the time that we are truly in HA (meaning we truly have two share-nothing, in-sync mirrors and can switch serving in an instant with automated logic), 2) the time that we are in HA but one mirror is behind, and 3) the time that both mirrors are behind (possibly because the customer lost connectivity to us and is now sending a barrage of points to backfill, also more about this later) are also key SLIs that we track.

SLOs are monthly aggregates of these SLIs so that we can score and track our progress towards perfection. There are good months and there may be a month when AWS knocks out an AZ because of a nasty EBS bug (just 2 months ago), but having the numbers help keep us honest.

So what’s the deal?

So the question is whether SLAs are important. I suppose for a commercial Observability Platform, they matter in the sense that we don’t like buying things with no guarantees. At the same time, it doesn’t matter much because it doesn’t mean that you’ll get a service that works 100% all the time, without fail. Slowdowns are a fact of life, and even Chrome releases have caused issues with rendering. Only a fanatical focus on reliability and quality can separate the best from the rest.

Does it matter?

We have heard some say that reliability of your observability platform doesn’t matter since, well, “it’s just monitoring”. Since we use Wavefront to monitor Wavefront, we understand the feeling when you’re flying blind — a very uneasy and panicky feeling (imagine if you’re flying a plane at night and the instrument panel just went dark). As I said above, our customers literally go into code freeze if we are out and we go into release freeze for them if they are in a sensitive moment (note this is only if you go with the dedicated cluster option). The greatest feeling for any Wavefront engineer is perhaps to walk into a customer NOC and see that the software is what’s driving all the monitors. It’s literally how they know the business is humming along. It’s the instrument panel of the car.

So I firmly believe that it matters whether your monitoring setup is up, and not just the “hitting the SLA” kind of up but the “cruising at 65 mph with much horsepower to spare” kind.

High Availability comes Standard

With Wavefront, even if you’re in a trial cluster, your data is kept in HA configurations. That means that we keep a total of 4 copies of your data on 2 availability zones. It also means we route traffic through 3 separate witness nodes that allow us to steer traffic in a moment’s notice if there is anything odd with an AZ. We had to run without ELBs because we felt it was just too opaque as to what’s happening. We run fleets of nginx machines for traffic routing with custom code strapped on them to rewrite routing rules on the fly. For dedicated customers, we also offer a direct VPN drop or IP whitelisting.

Data replication is designed to tolerate complete AZ failures and queue data up for weeks. Within an AZ, we can tolerate complete storage failure of EC2 instances (happens when status check fails), EBS failures (fact of life but not as often as machine failures from our experience) and network partitions (FoundationDB has its limits when it comes to partial partitioning, however).

DR options are also available which takes HA to a whole other level by having triple redundancy for operational data (dashboards, alerts, users, etc.), a witness region to make decisions with a remaining region if one region is out and global locality-aware DNS traffic routing.

The Wavefront Proxy

I guess I did say previously that we are going with Telegraf as the agent of choice for us but I omitted one piece of (open-source) software that we do ask customers to run and that’s the Wavefront Proxy (or Secure Gateway I guess, because that’s what other companies call things like that to make them sound sexier =p).

It’s a piece of Java code that relays data from customer datacenter to our service in the cloud. Now, it’s possible to just send data directly to us via https but with proxies, you can configure per proxy limits, scrub sensitive data, do all kinds of transformations you can dream of and most importantly, survive a connectivity outage.

The last point brings me to perhaps the most interesting challenge any time-series analytics engine faces: traffic doesn’t just go away and come back, it comes back with a vengeance.

Any outage of our end or on the customers' end will inevitably lead to a reservoir of points on the proxies and you have to include that capacity in traffic sizing in order to keep up.

This capability also means that our customers never lose data because the Internet is out or in the unlikely event that Wavefront is down. This is something that’s often overlooked with homegrown solutions since you can always tip your hat, say sorry, and move on with the data after the outage. With Wavefront, we really take care of your every data point.

Intermission

The more I think about it, the more dimensions of a time-series analytics engine come to mind which are more need-to-have than nice-to-have. We’ll see what happens when I start thinking more about the second category I guess.

Thoughts from the front-line as a founder, engineer, and operator of one of the largest SaaS observability platform on the planet. Wavefront started in 2013 and was acquired by VMware in 2017. Opinions are mine.

At the heart of a modern observability stack is inevitably a time-series analytics engine. Of course, it’s not the only thing you’d have, many DevOps shops have logging stacks (ELK, Splunk or Sumologic), distributed tracing stores (Zipkin, Jaeger, etc.), inventory systems (might very well be API calls to AWS/GCP/Azure and managed with things like Terraform/Infra-as-Code) and event/incident management systems (Bigpanda, Moogsoft, etc.).

The focus of this story is how we (Wavefront), operating Wavefront, built, what believe, the best-of-breed time-series analytics engine, operated it at scale and how it still puts a smile to our face (and our customers) when we use it every day to serve our customers' 70 trillion data points (and counting) they have entrusted to us.

Criteria for Success

There are many time-series database out there: open-source, commercial, commercial open-source, etc. and it’s a crowded space to be sure so what would a modern DevOps team consider to be a good time-series analytics engine?

Flexible

You need something that can take data points from your applications, your infrastructure, your cloud fabric, even your user interactions. You want all of that because when you have an issue or wanted to explore correlations, you don’t want to plop them into Google Sheets or fling timestamps around just so that you can see how two things interact.

Agents are only part of the story here, we thought long-and-hard, and concluded that there are probably too many agents out there already (it’s not surprising to see a handful these days on production machines, we use Scalyr for logging and Lacework for security for instance). Nowadays, we recommend using Telegraf (from the TICK stack). We are also active contributors to the project itself.

Most of our metrics, however, come from language SDKs, Kubernetes control planes, cloud APIs, etc. Someone, somewhere, will be sending you data from curl or from a program written in an obscure language — as long as it can talk to a socket.

Scale

You want something that scales up and down, and fast. Scale is an overloaded term — many benchmarks of time-series databases do bulk loads which are inherently not how real data comes in. When you spin up a thousand containers you want your system to work just like you’d expect it to. Not a minute later, not 15 minutes later, not when you found resources to throw at it. Scale is also the speed at which you can pull data back out of the system and perform computations on top of it.

We observe that data is queried at 10x the ingestion rate typically but unlike most datastores, the scans are concentrated on 2% of the ingested data (hence the datastore needs to be able to scale for lobsided reads, Cassandra, for instance, without any awareness of read/write bandwidth and shard rebalancing would just have to be inefficiently sized for the workload). We use FoundationDB (have been since the early days) and full-disclosure, I am a member of the Program Committee for the upcoming FDB Summit on December 10, which runs concurrently with Kubecon and CloudNativeCon in Seattle. I might be a little biased as to what Key-Value store is the best out there. :)

Analytics

You can probably get scale and flexibility even if you picked a traditional RDBMS (although running one at 1M writes per second sustained in HA/DR configuration is probably not a walk in the park). That, and you will probably quickly run into the need to perform sliding window moving averages, aggregating streams that are not aligned, wants predictable autocomplete with trigram searching, etc., and SQL is probably not the language you want to use. It’s just not designed for time-series analytics where you want composable functions, first-class concept of time, and data as streams and not rows.

At Wavefront, we built composable functions with common sense math operators that operate on “key-ed streams” (simply put, think of it like passing variables of the type Map<Key, Stream> but not having to deal with types at all). Our users seem to love it and, again, full disclosure, it is heavily inspired by Viz @ Twitter and Borgmon @ Google (although even though I had Borgmon “readability” back in the day, it was considered an oxymoron, i.e. “to have readability in Borgmon… it wasn’t readable, to begin with…”; that story is an important one on how to design a language, but I digress).

Real-time

All systems exhibit some latency (speed of light problems) but the reason why we don’t just dump all the data into an OLAP database with all the materialized views of how we want to dice the data is because you can’t collect a day’s worth of metrics and bulk insert them (OLAP systems also preclude data mutability, something that our customers actually want, as we found out). We find that average insertion latency needs to be <2s for the system to “feel” real-time and that queries themselves need to be faster than 100ms on average in order for users to feel like they are “conversing” with metrics (or as we call it, surfing metrics). This is largely because systems with high query latencies “penalizes” users for not finding the right thing or for not being able to incant the exact expressions for their intent (we do this all the time with query refining in Google searches BTW, if Google searches take 2 seconds to complete, we’d think a lot harder before we hit “enter”).

We happen to have a streaming architecture (with patents on composing un-aligned streams, continuous vs. non-continuous data, and synthetic time series, etc. to name a couple) that drives a single point from disk through stream operators all the way to the browser (through Server-Sent Events).

“Everything should really be in the form of iterators. I like the idea of an iterator next() call reading data off the disk and streaming it straight to a browser as fast as possible.”
http://www.xaprb.com/blog/2014/06/08/time-series-database-requirements/

We sometimes see people refer to Apache Spark or any streaming platform as the “proper way” to do time-series analytics but at the end of the day, every system needs to have a persistent storage story, an event bus can only give you “from now on” semantics, Gmail wouldn’t be very useful if you need to land on a browser and it just shows you your emails from that point onwards. Arguably, new data can only be as voluminous as what is the ingestion rate of the system; analytics are usually done over much larger data-sets.

Alerting

The ability to ingest, store, visualize, and analyze time-series with data would be complete if you just have humans looking at charts 24/7. Many open-source offerings separate alerting with the engine which means you need to stand-up yet another system that may or may not understand the specifics of what it means to be monitoring a plurality of time-series, with differing thresholds triggering different remediations/notifications, and the nuances of maintenance windows (how they play with your tagging strategy for instance). Just a simple alert of “please email me if any system’s CPU goes over 80% and page if it goes over 90% for 5 minutes” would require an alerting system to know how to properly remember where each time-series was in the last cycle (not to mention series can disappear altogether, escalations can happen for a single series, which means a single alert is effectively N alerts where N is the number of series it monitors multiplied to M notification strategies based on severity). Our largest customers monitor millions of time-series with tens of thousands of alerts (many requiring historical data from weeks or even months of data).

Mutability

Again, we have heard that if one stood up AWS Kinesis streams it would solve everything but alas I don’t want to sound like a broken record since a) historical data, and b) backfills are a fact of life (ts_now > ts_before is highly undesirable since we don’t have accurate GPS clocks on every machine and you don’t want to sort time-series before you ingest them =p). At Wavefront, our customers expect to be able to ingest data from a month ago, 6 months ago, and overwriting them again and again (scripts are mostly what’s doing that since some observability data are aggregated from other systems for various reasons). We also have the ability to ingest derived metrics from the result of queries (think “INSERT INTO … SELECT …”, a feature we borrowed from Kapacitor in the TICK stack) and it basically mandates that the system allow overwriting of existing data points.

Intermission

We have also collected a whole set of like-to-haves for a time-series analytics engine and observability platform in general. Some might think that they are need-to-haves for sure but the six points above are table stakes in my opinion.

Hope to share more in the next story.

Wavefront is a hit with SaaS companies large and small dealing with the complexities of monitoring modern public/private cloud applications giving users full-stack visibility into infrastructure, cloud, application, packaged applications (e.g. Redis, etcd, MongoDB) and even end-user behavior. While the primary interaction with the system is over the browser for most users, I decided to go nostalgic and see what we can do with Wavefront and a bit of coding (no servers required).

I happened to stumble upon a company based in France called Smiirl.com from Facebook and they sell these airport flap displays (I suppose they are technically called Split-flap display: https://en.m.wikipedia.org/wiki/Split-flap_display) which allows you to not only monitor your Facebook or Instagram likes but basically anything that exposes a publicly accessible endpoint (you can secure it somewhat with a pre-shared secret but given how AWS API gateway works, it’s probably relatively safe to assume that it would be pretty hard to guess the URL).

They are a bit pricey but they let you customize the graphic on the side of the flaps and it’s quite a marvel to see when it’s delivered. You can hang it on the wall and it just needs wifi and power (Ethernet port is available and for many work places that needs wifi signin, that helps a bit).

Here’s the goal: display the results of a Wavefront query continuously on the Smiirl counter without hosting a single server, updating once an hour. This is not just a cost issue but rather you probably don’t want it to update all the time and end up having the room sound like an airport =p

Things you can conceivably show on the counter:

• Number of users on the site in the last X hours
• AWS cloud costs
• Storage/Database utilization metrics
• Business metrics
• Metrics from your Tesla! (We have an integration for that)

Enter AWS lambda, a service on AWS that anybody can sign up for and it manages all the mechanics of spinning up a server, running your code, and spinning it down. It has a free tier of 1M requests a month (as of this writing) and with AWS API gateway (which, as of this writing, charged me 2 cents for the whole month), it allows you to host a code snippet that executes a Wavefront analytics query (which, btw, can be very complex, or simple, whatever you want it to be) and update the counter. You get to choose how often Smiirl polls lambda for the results and the code snippet is really as straightforward as it can be:

You just need to replace ts(my.metric) with the Wavefront query you want (you might need to url encode it). You also need a Wavefront API token (reach out to support@wavefront.com for a role account token so that it’s not tied to a user). You also need to specify the cluster (e.g. longboard.wavefront.com). The rest of the code is pretty self-explanatory. Hooking up AWS API Gateway nets you the URL:

Finally, you plop in the URL into Smiirl.com and select how frequently you want the flaps to, well, “flap”. We chose one hour at the office since it’s pretty loud (you can fix that also by changing the query so that it reports aligned hourly summarized for instance too).

I did all of this while standing in the kitchen computer after opening the box from Smiirl.com so it’s safe to say it’s relatively easy to pull this off (given that I already have an account with AWS that is, and even that is easy since you can use your Amazon.com sign-in and payment credentials to get started).

Finally, Wavefront offers a first-class integration SDK with AWS Lambda which lets you monitor beyond what CloudWatch gives you (also allows you to emit custom metrics). For Node.js you can see the instructions here. With that, you also get metrics from your AWS Lambda functions in Wavefront.

Computing has gone a long way in recent years with cloud, containers and now server-less. While server-less or FaaS is still a nascent offering (and I doubt we’ll be wholesale writing everything with it anytime soon), there are use cases that works very well — I need a URL that does 10 lines of logic every hour probably fits the bill perfectly. Not having to run, maintain, patch and pay for the server for one.

Being a founder of a start-up can be a pretty lonely experience. There’s the constant feeling that you have to lead by example, motivate people (but not appear as though you’re doing so because your employees are working “for” you), fight fires, diffuse tensions and bad news, believe in yourself and your team (but knowing when to let people and ideas go) and ultimately, when to let your baby go after it’s all grown-up and ready to fly away.

I told myself that I would find the time someday to write down all the learnings I’ve had but I realized that some of the thoughts will slowly drift away (perhaps because of age) and if I don’t start doing this piecemeal, I might never do it. So for someone who doesn’t really write anything apart from code, design docs, JIRA tickets, post-mortems and the odd seminary paper here and there, I hope I can manage to make some sense out of my own core dumps (going in reverse chronology but that’s probably because the last stretch is still the freshest in my mind).

Thoughts are mine and mine alone. I am blessed to have two other amazing founders: Dev and Durren and I always said Dev should write a book some day (it’s the real “Silicon Valley”). It would be a much better read. :)

Maybe start off with some of the emotional pieces:

• Not having been able to converse with anybody about selling the company, due diligence, tax planning, financial planning, charity intentions, etc. is a pretty tough thing. As an engineer, I have taught myself that Google is the omniscient website with all the answers and I was quite surprised to find that many of those questions I had (or wished I knew what the questions I should have) have no easy answers (if at all). Here’s where having co-founders, a handful of trusted advisors (who are sworn professionally to secrecy but obviously one would still use discretion) and divine intervention helped.
• Nobody told me about the process until I finally asked our CFO and discovered a lot more about what I don’t know (What’s an LOI? What does it mean? What does it mean to shop? Exclusivity? Merger Agreements? Offer Letters? Key Employees? Critical Employees? 280G? Disqualifying Dispositions? NSOs? ISOs?).
• As a Hong Kong-er who grew up in Asia (until college), an exit seems so foreign since everybody else had stable jobs, didn’t switch around too much, had a salary, perhaps the lucky ones had a bonus here and there (or regularly I suppose). The concept of selling your company (with its people, customers, vision, down to the last pen and paper clip), engaging in the delicate dance of “We are going to take the product and team to the next level” and “Well, here’s my payout, see-ya-later!”, and the idea that you can retire and ponder about “What’s next?” is not really what 18 years of primary and secondary education (and even a degree in Computer Science and Electrical Engineering) have taught me.
• David (a friend whom I have grown to know a bit better thanks to Winnie through a ski trip and just valley talk over coffee) was the first to confess to me that the worst day of his life is the day that the company he worked for got acquired. It was a day where half of the company would be fired (a common outcome since only the IP and the engineers were “needed”) and the worst yet is that you can’t even rely on HR to do the firings since HR is also let go (and no-one would consent to do the dirty work for you). I remember looking through Glassdoor and that’s when I tweeted:

Clement Pang on Twitter

Humbled by the awesome people that makes up @WavefrontHQ Come join us! https://t.co/7t8dkueicf #saas #bigdata #monitoring #analytics

• I was sad to the core that a successful exit meant we had to let so many talented, driven and loyal folks go even if they did everything right. I told HR at VMware that if I had to get down on one knee I would so that they would take everybody from Wavefront as a unit. Perhaps it was because VMware had good experiences with absorbing full companies (down to finance folks which they thought they’d need since we are a SaaS business and they need to know how to continue to bill our customers!), they decided to do that to Wavefront.
• I wasn’t involved in any discussion of the price (although we all shared our price and are aware of what is a price that we could “move forward”). As an outsider to that part of the process, I thought to myself that all of it seemed quite arbitrary (as in, for an engineer such as myself, I could see no formula or rationale behind the price going ever higher except that it was like an art form; either that or it could be viewed as the primal dance of two mating animals). Once the price was nailed down, or as I would now say nonchalantly (as if I knew the three-letter word as a seasoned entrepreneur): the LOI was finalized, we suddenly found ourselves in a conundrum: nobody in the company could know that number (and still nobody does today apart from some news wires suggesting that it would be revealed in a future regulatory filing) and nobody would know that a potential deal is in the works and now we need all the resources that we can muster from the founders, CEO/CFO to go through the due diligence process.
• I remember 10 hour days of non-stop talking (which ended often with me losing my voice) at VMware’s expansive campus was the norm for a couple of days when the technical DD started (another one of those acronyms that I’ve only heard once or twice from my lawyer friends). That’s usually coupled with production issues, bug fixes, managing agile sprints, answering questions, deciding on who to hire, responding to customer emails, triaging tickets back at the company (thank goodness that VMware and Wavefront are in the same city). The story was that we are exploring a partnership with VMware but certainly some of the silicon valley veterans in the company knew something more is going on (but also probably knew that they can’t really ask for the details).
• Technical DD by the company transitioned to DD by external parties. Those are the firms that can see your code (I wasn’t even aware that I can’t show any code until I whipped out my laptop at a meeting one time and the Corp Dev guy was like, “What are you doing? Under no circumstances can you be projecting code”). It’s another 20 hours from scratch to get people onboard every time (I probably refined my delivery of our architecture significantly after giving it to at least 10 different audiences throughout the journey, a shame since I never offered many of our new hires the same level of detail I gave to those outside parties). Firms made sure they all got their dedicated time since they have their own “perspectives” that they need to address (we apparently had the same firm that did the DD of VMware when they got acquired by EMC not too long ago). Sitting down and iterating through all your third-party dependencies and tools (hoping you didn’t find something that had a “problematic” license) and chasing people to tell you how something is used is what I would categorically state as “not fun”.
• I remember drinking a lot of Coke Zero from their kitchen (our snack bar at Wavefront is a lot better) and having lunch for a change (at least I didn’t have to pay). The receptionist at VMware probably could figure out something is going on and I remember also starting to get frustrated about the lack of parking after going there so many times. The coordination person got access to my personal and work calendar and started booking meetings directly without asking (sometimes with only a 30-minute heads-up). It was at that time that I also learned about Google Calendar’s feature to automatically block off time slots in your calendar as long as you gave it goals (2 hours every day in the afternoon). Video conferencing is broken everywhere (but I was told they are upgrading to S4B — Skype for Business soon).
• There were always something that is going to derail the deal. Powwows in the CFO’s room happened all the time and I started camping in a small conference room for a while with all the things I needed (including my personal laptop) so that I can scoot in for the odd call or view something sensitive. Note to future self: don’t use personal emails for anything work related (for example, signing up for anything), it might not be changeable and could derail a deal.
• Then came the negotiation of our offers. It is probably one of those negotiations that you only get to do once for one’s tenure with a company (and only through an acquisition perhaps). They have to make it look like it’s really worthwhile to stay for a couple of years (since the front-end is already quite significant). I certainly haven’t had any offer letter in my life with so many numbers designed to imply an incentive and also a restraint. I remember I was at the Google Next conference at the city (for which I booked a room at the Four Seasons last minute since there really was so many things going on and driving back home every day just seemed impossible) and there were many running back and forth between Moscone Center and the hotel room to take DD calls. On the last day after the closing of the conference, I sat in the car in the basement of the hotel, talking to yet another Corp Dev person about compensation. It’s a surreal conversation since he’s not a recruiter or a hiring manager but a professional negotiator trained to read the inflection in your voice and figure out what’s working and what’s not. On the other hand, we needed to figure out what is movable and what isn’t. We deferred a lot to our lawyers (yeah, our CFO told us that at some point, the corporate counsel stops looking out for us so we gotta find our own, and so we had a brief time interviewing lawyers, fun times), accountants (all of which happened during the weeks leading up to the deadline for the tax season so they must love us) and financial advisors to model the meaning of many of those numbers and clauses.

• There were thoughts about moving to Nevada or Washington (and then realizing for myself that I really live here, I have all my friends, church, activities here in the Bay Area). In the end, I’d say the offer didn’t change too much (I am redacting a lot of the movements for obvious reasons) and we signed it and they got some stuff wrong and had to fix it after signing the merger agreement. No biggie. I guess everybody is stressed and it’s just a miracle that things like this actually happen correctly without everybody suing everybody (right?). Our CFO actually told us that the M&A agreement (as well as the 280G) is the pinnacle of an M&A firm’s work and he literally told us to read it since we paid so much for it. :)

Alright, I gotta stop and call this episode 1. During the time I wrote down these points, I cherry-picked 3 commits from master into our release branch, made our 8.10 release and deployed to our staging cluster to address some customer issues with the rollout of Wavefront 8.x… still looking forward to taking some time off soon and transitioning.

Until then.