It’s never too early to start planning for next re:Invent!

The AWS Lambda, Amazon API Gateway, AWS Step Functions, SAR and SAM teams, and many others launched an incredible array of features this year that expand the “addressable market” for AWS’s Serverless offerings. (Looking for a field guide to those launches? Check out my 2019 summary.) Collectively, those 2019 releases make AWS Serverless an ideal solution for low latency applications, users of Amazon RDS databases, bursty use of VPC resources, high frequency APIs, and more. As impressive as that list is, though, enterprise and startup developers still turn to “serverful” solutions for many of their applications, and even diehard serverless fans will find themselves wanting some things to be easier. So, with a huge bow to last year’s features, I want to look forward to what the 2020 re:Invent has in store. My wishlist falls into three broad thematic buckets:

• Eliminate the need to run containers due to missing functionality. The 2019 re:Invent launches took a huge bite out of this category, but unfortunately there are still some holes (like gRPC support) where a rational person will still pick containers as the easier solution.
• Tackle net-new markets. Lambda could be a supercomputer…but not without better networking, “reducer” capability, high-speed code customization, and other functionality. Decentralized data is going to be huge, and Lambda is so close to being an ideal platform for building it…but isn’t fully there yet.
• Delight developers. Developer satisfaction never gets old, and some basic capabilities (convenience libraries, copy commands, recursion support, routing, polling, etc.) are still missing. Looking at things like Logic Apps in Azure is inspiring to imagine how AWS Serverless offerings could be both simpler and more convenient.

Ok, enough with the preamble. Let’s start wishing!

How much these matter depends critically on the type of architecture you’re creating, so I haven’t placed them in any special order. But since AWS is fundamentally driven by business needs, I do include thoughts on pricing where relevant.

• 1ms duration granularity.

Why: Five years ago when we launched Lambda, 100ms seemed infinitely small compared to paying a minimum of an hour for an EC2 instance. But today, with Go functions able to do useful work in single digit milliseconds, it feels like AWS is cheating by rounding up to 100ms on every call. Meanwhile, this limits Lambda’s addressable market, because smart devs rule it out right up front for short-lived calls. It’s time to get with the program here and “price cut” by way of tightly enveloping. Don’t wait for Azure or Google to beat you to the punch, AWS!

Price: Unchanged, just more fine-grained.

• EFS Integration in Lambda (and /tmp that scales along with memory)

Why: An infinite disk drive, mounted to an infinite computer? What could be better? Sadly, we still don’t have it, and Lambda’s restricted /tmp still doesn’t scale up like memory and CPU do when you pay more. As nice (and serverless) as S3 is as a blob store, its latency is high and it isn’t a real unix filesystem…making it impossible to use with a lot of existing library code.

Price (/tmp): Should scale without additional charges — users are already paying more for increased CPU and memory; the drive size should match that for no additional cost. (Save the ability to get a lot more storage for a lot more money for the EFS option.)

• Typechecking — An optional JSON Schema or protobuf definition for Lambda function arguments and results. (Bonus points for full gRPC support.)

Why: The most obvious reason is to remove boilerplate code from Lambda functions and shift it to the service, which is in a position to implement high speed, multi-tenanted JSON validation or gRPC* support for the masses better than any individual customer. It improves security (e.g., you could write IAM policies that prevent invocation of a Lambda function that doesn’t possess strong typing). It also offers a pleasing symmetry to the same feature in API Gateway (which obviously isn’t available for async invocation or when a function isn’t fronted by an API). Static typing offers all the usual benefits you’d get at the language level, such as knowing whether a function expects a number as a JSON integer versus a quoted string without reading the source code or being able to process the result of a Lambda function call without having to guard against malformed JSON (especially when you’re not the function’s owner).

I also can’t stress enough how much the absence of gRPC and protobuf support is costing AWS’s Serverless adoption. AWS teams don’t seem to realize this is the #1 checklist item when a Bay Area developer (and lots of others) make a technology choice. Every time someone fires up k8s, a Lambda PM should interpret that as hearing from a potential customer that gRPC support was missing in Lambda.

Pricing: Ideally calls that fail validation would result in an invoke charge but no duration charge. This also incents developers to add validation: it’s cheaper than paying to do it yourself.

*Yes, there’s a ‘g’ in it, but don’t worry AWS — it’s not really a Google thing.

• Integrate type checking with Event Bridge (and other upstream and downstream event sources and sinks).

Why: Once type checking is present, it affords nice opportunities to integrate with event schema discovery and validation. Example: When another Lambda function is the target of a Lambda Destinations setting, the output of the first and the input of the second can be used to statically verify that the pair will succeed (at least syntactically) even before an event is passed, which is especially useful for infrequent (“cold”) error paths. When wiring up an event directly to S3 or another event source, it helps avoid silly errors, like choosing the wrong function. The schema discovery feature of Event Bridge is a good start, but it doesn’t go far enough.

Pricing: Ideally free :).

• Support the “multiple entry points” design pattern directly through routing and multiple handlers.

Why: Another cool thing you can do once you have some built-in comprehension about the “type” of a function is support patterns like sending multiple events/APIs to a single piece of code. Religious debates aside, deployment is sometimes best done DRYly, which means you may have a single, more complex function to which multiple APIs, routes within an API, or events get routed. In this model it would be great to have systematic support for using a portion of the argument to get to the right handler, instead of having to write, test, and maintain a DIY dispatch table.

Pricing: If you run dispatch code as this part of the function’s execution, then the overhead just rolls into the normal duration charge.

Bonus points for: Pre-invocation Lambda hooks (also see “BYOA” below) and post-invocation hooks (aka Destinations) for sync calls.

• Polled event sources beyond SQS, Kinesis, and DynamoDB Streams.

Why: These event sources are part of Lambda’s super power (though see below for what I consider to be the crippling problem with Kinesis and how to address it). But no matter how much we might all turn up our noses at it, polling is still a thing in the real world. APIs, databases, queues that aren’t SQS, streams that aren’t Kinesis…there are a LOT of data sources that have to get polled. Lambda’s timer events, ability to scale, concurrency controls, etc. help a lot here…but polling is still super manual. Why not expose more of that polling super power as its own (extensible and customizable) service for us? See Microsoft’s LogicApps for a good idea of what this could look like (and how to make common tasks feel simple).

As a customer, I’m not hung up on *where* this happens…Lambda’s built-in pollers (like Kinesis and SQS), EventBridge, a new poller service, etc.

Pricing: Polling gets done mostly for free today, despite having very real costs. I’d expect to pay something for the privilege of polling a 3rd party event source.

• Idempotency protection / exactly once semantics / results retrieval / verification support for Lambda functions / memoization option.

Why: A common need in application code is to perform a non-idempotent action at most once. One of the most annoying characteristics of Lambda for developers is that it’s an at-least-once execution model, although most of the time it works in an exactly once fashion. This is a classic “garden path” problem…it’s e-x-t-r-e-m-e-l-y tempting to treat Lambda as if it were an exactly once service. One of the reasons people don’t work harder to “do the right thing” with Lambda is that it’s super annoying: Either you have to create (and pay both literally and in the sense of increased latency for*) a one-action Step Function to do this for you or you have to craft a DynamoDB table with auto-expiry and use it to record a moniker for each application invocation to protect against duplicate executions. Who wants to go through that hassle? This should be as simple as providing an optional JSON path for each Lambda function that indicates the route to a unique id along with an optional “de-duping” window length (which could default to 5 minutes). Bam! Idempotency protection for the masses, with no mistakes, and with way less cost than the DIY approach requires…after all, what’s the point of paying to persist this info to disk when it doesn’t need to live beyond a few minutes?

This is also another place where typed function (#1 above) helps, because you can then describe the UID to uniquify on as a JSON path to a known (numeric or string-valued) datatype, or as protobuf metadata.

Closely related to idempotency, and super useful in building decentralized applications, is verification. This is essentially a generalization of duplicate invocation protection: Instead of just storing the ID of an invocation (and thus implicitly the Boolean value that indicates it has been executed previously), store the result of the function. This makes it easier to use Lambda to implement concepts like a smart contract, which needs to execute exactly once but which might need to have its results verified multiple times, using the cloud vendor (AWS) as a trusted authority to retrieve the outcome of the contract execution. It also makes it possible for Lambda to offer memoization (result caching) as a feature, albeit with more complex (storage-based) pricing implications.

Pricing: Exactly once semantics and verification both require storing additional information in a durable (though not persistent) form for a limited window of time, which likely requires a higher invocation charge per call (and one that gets larger the longer the information must be retained).

*Note that the more recently released Express Workflows don’t help here, since they also have at-least-once semantics.

• Shutdown / Batching Hook

Why: Startup initialization is cleanly integrated with language mechanisms in Lambda, making it trivial to express “do this just once per function, before the first invocation gets to proceed”. Provisioned capacity increases the time between initialization and first run, but stays true to the concept.

Unfortunately, there’s no “shutting down” companion to the “starting up” hook. And even though it’s 2020 as I write this, fat client libs, stateful protocols, and resource locking are still very, very real in enterprise code. It’s not just old code, either — even “serverless” design patterns are impacted by the lack of a shutdown hook, because any time you want to batch downstream calls *across* Lambda invocations you’re out of luck — there is simply no reliable way to do it that doesn’t require using S3, Dynamo, or some other aggregator that often becomes self defeating in its cost and complexity.

There are a bunch of ways this could be retrofitted; an optional hook, set by a context method, is one. (Sending a special event would require everyone to start writing a new, top-level condition, which feels too intrusive.)

There are some corner cases (e.g., a function that crashed will probably not be in a position to execute its shutdown hook) but now that Destinations exists, there’s even a natural way to deal with that.

Pricing: Call the delete hook an invocation and charge for the duration it runs, just as if it were “normal” event invocation.

• Ephemeral Functions

Why: Let’s say you have a Lambda function that wraps ImageMagick or some other image processing library. You have two directories in S3, “small” and “large”, and you want to thumbnail source images to different resolutions depending on the directory name. Easy breezy: With just two directories it’s easy enough to either code a conditional into a single Lambda function or just create two different Lambda functions (optionally by sharing a Layer for the common image transformation library). Problem solved.

But what if you had a *lot* of directories? Such as one per user for a large b2c app with millions of users, each one of whom gets to customize settings differently for their images? You still really only need one function, but you need a way to associate its arguments with the user. We could give the Lambda function access to a database of users and their configs, but that’s a lot of exposure that doesn’t seem truly required. Couldn’t there be a way to get those arguments passed in, leaving the image transform function stateless and avoiding having it read a database it doesn’t really need to see?

The real problem here is the hard split between Lambda’s control and data planes. In this case, we have a single, “master” function that we want to create once (the control plane part) and then partially evaluate by setting some but not all of its arguments many times over. These partially evaluated functions have to be created on the data plane because there could be many of them. The cardinality has to be massive: E.g., it should be possible for every object I store in S3 to have its own ephemeral Lambda function as an event handler without ever “running out of space” and needing to get one’s Lambda limits increased.

How could this be done? One way would be to expand Lambda aliases to include partial evaluation monikers, stored in DynamoDB. When I call foo:bar(), Lambda looks up bar, sees that it wants the first argument to be 5 and the second argument to be “yellow”, and then proceeds to call foo(5, “yellow”, <the rest of the args>).

Pricing: Since a little extra work is required, it would be reasonable to charge a premium on top of the normal invocation fee to use an ephemeral (partially evaluated) function. Then again, it saves on the increasing expensive “real” control plane operations, so perhaps it could be a wash?

• Affinity / Poller “State”

Why: To motivate this request, let’s imagine a really simple (if unfortunately hypothetical) model for processing a sharded event source, such as Amazon Kinesis. Suppose we’re processing some US domestic data, such as ecommerce sales, and we have 50 shards, one per state. We want to create a dynamic business dashboard that shows sales in the last hour on a state-by-state basis, say by illustrating it on a color-coded map of the US for the product specialists. We’d also like the code to be easy…ideally, it’s as simple as “hourly_sum += record.sale_amount” with an end-of-time-window hook that writes hourly_sum to the database powering our dashboard. Oh, if only it were that easy…

Lambda’s existing Kinesis poller integration handles a lot of the annoying work of dealing with streaming data, but it leaves two key problems unaddressed. With no shutdown hooks, there’s no way to tell a function that it’s done aggregating data. With no affinity, there’s no way to aggregate (batch) anything locally, because the next batch of records from the same shard might get processed by a different instance of the same Lambda function. So today, building this “simple” running counter means turning every Lambda call into a DynamoDB read & write. It feels downright silly to have to use one persistence mechanism to aggregate another.

With just shutdown hooks added, you could hack up a solution: Use the built-in Lambda Kinesis integration as the “frontend” while sending records to shard-affinitized Lambdas on the “backend”, each with a concurrency setting of 1. This gets very complicated once you have to start dealing with shard splitting and combining, limits on the number of Lambda functions (and the overhead of creating new ones), and so forth. To really work well, it needs to be a first-class, fully built in capability.

Pricing: This is an extra constraint for Lambda’s scheduler, but doesn’t require doing anything more while the function is running. So hopefully no upcharge at all, and worst case perhaps a slightly increased invocation charge for that additional tracking.

BTW, we talked about this in the form of Kinesis, but there’s nothing special about polling event sources; this technique would also be super useful to create “reducers” of any type where a subset of events need to be routed to a subset of instances in order to execute some kind of aggregation function unique to the shard. It’s way better than exposing addressable instances outside the service, while still providing a lot of the semantic richness that people believe can only be achieved with ‘stateful’ services and containers (or servers) today.

• Lifecycle events on the control plane.

Why: It’s a bit ironic that the ultimate event handling service doesn’t itself generate useful events. Function creation, state changes, etc. should all emit events. This would make it possible to do things like hook in compliance checks on code updates, security verifications for code changes, etc. (See above for lifecycle events on invokes, aka pre- and post-hooks.)

Pricing: Should be free (of course the functions that handle the events will be charged as normal).

• Bring Your Own Agent (BYOA)

Why: There are a lot of uses for “sidecars” that run alongside code, including logging, performance monitoring, security checks, and more. BYOA would also make it easier for serverless monitoring and security companies to offer products that are both easier to use and that perform better. Language and lifecycle considerations aside, the big needs for such agents are:

• Clean initialization (easy — already exists)
• Clean shutdown (could be a minor extension to shutdown hooks; see above)
• Heartbeat / checkpoint events (relatively easy, but net-new functionality)

Pricing: Assuming shutdown and heartbeats charge as if they were normal invocations, this could look just like normal pricing.

• From “Destinations” to “Continuations”

Why: Ever hear someone say, “If you run out of time on a Lambda function, just have it call itself recursively”? Except…that’s problematic — how do you represent how much progress has been made? What happens if a function dies for some reason…how does the system recover, especially if it represents a lot of work/time? What if a transient outage causes a single invocation to fail in the chain?

It’s interesting that Lambda already has a model for handling this: Kinesis and DynamoDB streams present a function with a batch (array) of events, and can re-present subsets of those arrays to handle partial work (see “Bisect on Failure” in the Lambda docs).

Why not generalize this? Allow *any* invocation to pass in and/or out an array of events, along with an optional continuation (a function to call if the current function doesn’t finish, just like a Lambda Destination). With static typing and ephemeral functions, this should be pretty straightforward. As with many of these suggestions, static typing is an effective prerequisite to getting the service to help out, since it needs to understand the array type and how to send it to the continuation.

Pricing: Should be free; this is basically just passing the result of one function to the next in a more controlled manner that the Lambda infrastructure can help track.

• Lambda Spot.

Why: I’m pretty sure that Lambda has some spare capacity at 3am in any given region. Why not offer a price break to incent usage during off-peak hours with dynamically computed discounts?

Pricing: Hopefully lower!

• Serverless Networking.

Why: I’ve talked about this extensively elsewhere. Streaming data, using Lambda to build peer-to-peer networks, high-speed data processing, gRPC between Lambdas (especially to serve up computed or cached data), and so on.

Bonus points for: More polite (i.e., not forcibly symmetric) managed NAT inside VPCs.

Pricing: Free :)

• Intra- and x-region COPY command

Copying Lambdas is annoying. Surely this could be more convenient.

Pricing: Free — it’s a control plane thing.

• “Global Lambdas” (aka x-region autocopying)

Why: Sometimes you really want to copy a function and then make a little tweak, and it’s a PITA to do that today, especially if you’re been using the UI. Similarly, keeping functions consistently deployed across regions is annoyingly hard, requiring CodePipeline and CloudFormation and then straggler control on top of all that…can’t this just be a setting? The capability already exists (see SAR public apps and Lambda@Edge) but is cleverly hidden from customers today :).

Pricing: Free.

• Encoding payloads in transit.

Why: Because it’s the enterprise and it’s 2020 folks. Service-to-service payload encryption should be “1-click” (or its programmatic equivalent). IAM should be expanded to make it possible to set policies that prohibit calling/being called by functions/event sources/other services that aren’t on-the-wire encryption configured. Frankly I think all newly created functions should get this turned on by default.

Pricing: Free (because mostly duration charges, and where applicable, data transfer charges, will address it).

• More Layers containing popular language libraries.

Why: Take the top 50 libraries in every language and bundle them into Lambda convenience layers. Yes, yes, yes — I know, it’s a terrible idea because versions. Horrible. Inconceivable, almost. So, can we have it please, and without all the damn whining? Thanks, love ya. Seriously — if Amazon Linux can exist, then this can exist. AWS, stop fussing and make Lambda easier to use.

Pricing: Free of course.

• Additional support for building decentralized services and applications.

Why: Data and code that has to be shared among / across companies and organizations is increasingly common…and super hard to build on AWS unless one account can own everything, which often isn’t viable. Lambda is in better shape than most services with its immutable versions (and verification support would help even more; see Idempotency above).

But just like peer-to-peer networking requires capabilities not needed in client/server connections (like NAT punching), decentralized software requires capabilities that only the cloud vendors are in a position to offer. Some additional features that would make Lambda even easier and more powerful in building decentralized applications and services are:

• Fix cross-account resource policies — Inside an account, Lambda resource policies are nicely fine grained. For example, as the owner of a function you can specific specific workflows, APIs, and other functions that can call it. But across accounts, where you think AWS would care about granularity and security even more, it’s a disaster — to enable cross-account use of a Lambda function, you have to open it to *ANYTHING* in the calling account. Hello, enterprise security anyone?
• WORM functions — make it possible to mark a function so that it can’t be deleted, just as immutable data in S3 is handled. (To be useful, this fact, and its expiry, must also be easily and programmatically shareable across accounts as a critical piece of metadata in addition to the actual data.)
• Equivalent functions — create a namespace where a function — if it exists in an account — must be identical in its code & config to every other AWS account. This makes hash checks to convince yourself that you (or someone else) is running the “right” code a thing of the past. (To put it another way: This is like a provable COPY command that can cross accounts, but without the operational hassles of actually doing the copy; you just have to say you want to allow a subdirectory of the namespace to be “mounted” into your Lambda account.) Ideally this doesn’t count against your storage limit! Also, super cool feature for SAR to consider…
• Permission monotonicity and immutability — a bit you can set when creating a function that allows its permissions (both resource and role) to contract but never expand. (This is really a broad ask for all IAM, but I’ll list it here for completeness, as having it even just for Lambda would be super beneficial in building decentralized databases.) Note that this is very different than permission boundaries: There is no explicit document required, and the owner of the function / account cannot subvert the setting; all they can do is delete the function (unless it’s been WORMed as described above). Similarly, a set-on-create bit that makes the role and resource policy of a function completely immutable, even by its owner. (The function can still be deleted unless it’s WORMed.) These make it possible to give another company/organization durable, predictable access to a function with the confidence that not only won’t its code change, but neither will its capabilities. It’s a critical set of guarantees that makes AWS a platform for decentralized systems and application coding.

Pricing: Free. This is all security related or control plane magic.

• Serverless redis

Why: Low latency storage that can seamlessly scale in an impedance-matched fashion with Lambda’s compute would be a game changer. EFS and a “serverless” redis would make it possible to convert existing enterprise applications to Lambda that just simply can’t be moved out of containerland today.

Pricing: Great question, but let’s focus on having it exist first. Hopefully just by making it 100% utilized by definition (like S3), it will be super cost effective compared to the inevitably overscaled “infrastructure redis” of today.

Bonus Round: Everything else (outside of Lambda) that would make Lambda better

API Gateway

• Built-in support for gRPC/protobuf (similar to how websockets work).
• Put humpty-dumpty together again: The v1/v2 API split is a mess.
• Add resource policies to API Gateway websockets.
• Make it possible to get the AWS account number when IAM auth is in use.

Event Bridge / CloudWatch Events / SNS

• Ordered events (the SQS FIFO model would be fine). C’mon already…not everything is arbitrarily reorderable and at-least-once-able.

SAR

• “Auto-region” copying for private as well as public apps.
• Lifecycle events.

CloudFront/Wavelength

• Lambda@Edge done right aka CloudFlare Workers aka “Serverless Wavelength”

IAM

• Immutable permissions (can be set only on create; irreversible even by owner)
• Monotonically contracting permissions (set any time, but irreversible once set)

Cloud9

• A serverless backend
• Seamless interoperability with VS Code
• A shell mode that uses Lambda to run an Amazon Linux bash shell (so I can stop having to own and maintain an EC2 instance just for the privilege of running AWS commands)

Other AWS Services That Lambda Connects To:

• X-Ray support
• Resource policies, especially fine-grained x-account ones
• In-transit / on-the-wire encryption support
• Built-in chaos testing

Another re:Invent is over, and an army of weary AWS employees have flown back home, their launch tasks complete. The AWS Serverless teams, including AWS Lambda, Amazon API Gateway, AWS Step Functions, Amazon EventBridge and others were especially prolific in in the run up to, and during, re:Invent. But what does it all mean? With so many new releases, it can be hard to find a forest and not just a bunch of trees. That’s especially true this year, since much of what was launched wasn’t coolness for the already-drank-the-Kool-Aid crowd: This time, AWS is trying hard to win container users over to Serverless. It’s the dawn of a new “hybrid” era.

We’re already seen this phenomenon in reverse: AWS Fargate and Google Cloud Run take containers and try to “serverlessize” them by removing details of the underlying infrastructure management, while preserving the things developers love about containers (like Docker images as a deployment and runtime abstraction). This year, the most pure-play Serverless compute offering out there, AWS Lambda, got in the hybrid game by offering more “server-like” features. Tim’s Prediction: We’re in for more bridge building from both sides from each of the Big 3 cloud vendors!

If you’re a developer thinking about building your next in-house service, public API, or mobile app, what’s different now versus a couple of months ago? If you’re already a heavy user of Serverless, what changed that you critically need to know about after this year’s re:Invent? I’ll decode these releases for both demographics to give you an easy cheat sheet for what to care about. (And for a look at what *isn’t* different, see my upcoming companion article on what AWS teams still need to do to help enterprise developers adopt Serverless architectures.)

First, to re:Iterate the most important thing you need to know: 2019’s AWS Serverless releases were mostly about getting k8s and other container/server developers to come to the Serverless party. Yes, these AWS teams love their existing users, but without this context, you might be forgiven for wondering why features that seem to blunt some of the magic of serverless (like provisioned capacity) made the re:Invent cut. It’s part of the long game AWS is playing: They’re systematically looking at every objection that non-serverless developers raise and going after them, with an eye towards eventually enabling all applications to be serverless.

To that end, the AWS team focused on the top objection that container and server devs level at Lambda: cold starts. AWS decided enough was enough, and went for a full-on mic drop here with a trifecta of solutions that represented their biggest and baddest releases in 2019:

• Zero latency (aka “ENI-less”) VPC access. Don’t know what an ENI is? Don’t worry, it doesn’t matter. What does matter is that scaling up a Lambda function that needed to use resources behind a VPC used to take up to 30 seconds on each initial call to get that network config established. Now that’s more like 30 microseconds on the first call. So if you’ve heard someone say, “Lambda doesn’t work when you use VPCs”; that’s now outdated info.

If you’re not an existing user: PAY ATTENTION TO THIS! You probably had a buddy steer you away from using Lambda because of this issue (or perhaps you did the steering). It’s time to forget those old tapes, because Lambda is now just as fast whether you use VPCs are not. (BTW, there really isn’t any such thing as “non-VPC” Lambdas; it’s just a question of whose VPC it runs in — the Lambda service’s multi-tenanted VPC or your own.)

If you’re an existing user: If you use the VPC feature and your Lambdas are synchronous and they’re latency sensitive, then rejoice. Otherwise, you can ignore this launch except to feel good that if you hit that use case down the road you’re covered.

• RDS Proxies. Heard of Aurora Serverless? It’s essentially built-in autoscaling + an HTTP frontend on an AWS-flavored SQL database. RDS Proxies are basically the latter applied to Amazon RDS dbs. Or in simple terms: If you want to use Lambda with an RDS database, it used to suck (because connection pools and Lambda didn’t play nicely; also VPC) and now it doesn’t suck. Caveat: Still in preview, only MySQL today, etc. but expect this to rapidly get to production grade.

If you’re not an existing user: PAY ATTENTION TO THIS! You probably heard “Lambda doesn’t work well with SQL databases” at some point, so if you didn’t have the time/energy/will to migrate to Amazon DynamoDB you might have ruled Serverless out as an option early on. That’s no longer the case. However, note that SQL databases still won’t “impedance match” for overall throughput as well as a Lambda + DynamoDB combo play will. RDS Proxies will keep Lambda from blowing up your relational database; it doesn’t send magical unicorns over to RDS to auto-scale capacity, so Aurora Serverless and DynamoDB still have more built-in“serverless scalability” if you have the flexibility to choose one of the latter two.

If you’re an existing user: Chance are you’re already using DynamoDB or perhaps Aurora Serverless because they were the preferred solutions. If you are using RDS, you probably have some messy homegrown connection pooling and/or concurrency control mess that you should now throw away in favor of letting AWS do this for you.

• Provisioned Capacity. The ability to tell Lambda that you need N instances of a function, warmed and ready to use, for a given amount of time. Or to put it another way: An SLA from AWS to make up to N simultaneous calls to a given function with a guarantee that they will be 1) available (you won’t hit a provisioning limit) and 2) low latency (you won’t have to wait for Lambda to get them ready). Note that you can still make more than N calls in parallel; you just won’t get the provisioned capacity guarantees for any “excess” calls.

If you’re not an existing user: PAY ATTENTION TO THIS! If there were a classic “I use k8s/containers/servers because Lambda can’t …” lament, this was it. If you needed predictable low latency or predictable burst capacity with Lambda, getting that required scaffolding out your own “prewarmer”. It was a messy job that existing customers vehemently disliked and an obvious gap that caused potential serverless adopters to turn away and head straight back to containerland. Now, getting the combo play of low latency, guaranteed capacity just like you can with servers is as easy as a config setting on Lambda. This is the biggest game changer of re:Invent for folks who aren’t already using Serverless.

If you’re an existing user: Use Lambda for async workloads (S3, SNS) or polled event sources (SQS, Kinesis, DynamoDB streams)? Then you almost certainly shouldn’t care about (and shouldn’t use) this feature. But if you use Lambda for something like a flash sale where you need predictably low latency and/or guaranteed instantaneous high volume capacity, throw away your nasty DIY prewarmers today.

(By the way, lumigo has done a fantastic deep dive writeup on this feature.)

The fine print. #1 and #2 are uniform goodness — everyone who uses them will benefit. But #3 is different — You shouldn’t use it if you don’t truly need it, because it destroys one of the key economic advantages of serverless compute: 100% guaranteed utilization. There’s no way to say this nicely: Paying for unused capacity makes Lambda less serverless. This is probably why the team formally named it “provisioned concurrency”, but don’t be confused: You’re paying for what will probably be a lot of unused capacity, just as you do with servers and containers.

Will provisioned capacity get abused? Absolutely — especially by developers and operators who are used to provisioning servers and really wanted the equivalent functionality in the Serverless realm. Premature optimization, developer hubris, and confirmation bias will undoubtedly result in way more provisioning that is truly necessary. Will this cause developers to defer important architectural changes, such as slimming down dependencies and startup times, where Lambda previously provided useful “design pressure”? Yes, also unfortunately true. But it will also result in hordes of new developers adopting Serverless who otherwise would have spent more years using containers and servers, so it’s a reasonable tradeoff. Bottom line: Think about Lambda provisioned capacity the same way you think about Google Cloud Run, AWS Fargate, and premium Azure Functions: These are all hybrid solutions that aren’t “pure play” Serverless, but they provide an important bridge for meeting a lot of developers where they are today. If it helps you operate one less server, don’t feel guilty about getting started there…just know you’re likely not at the end of your journey yet.

Ok, so much for the cold start trifecta. What else got launched?

• HTTP APIs. It’s basically an API Gateway cost reduction for the common case.

If you’re not an existing user: PAY ATTENTION TO THIS! You probably looked at Amazon API Gateway once upon a time and decided that $3.50 per million API calls was too rich for your blood, and went back to running your own frontend. At $1 per million calls, you might feel differently, and let me just say: Not running your own frontend fleet is a joy that keeps giving.

If you’re an existing user: Any AWS price cut is nice; getting 70% off what might be one of the more expensive parts of your bill is pretty cool. But make sure to look at the fine print: The tradeoff here is that a bunch of the advanced API Gateway features aren’t available in this new “economy seating” variant. If you’re using Velocity templates, Lambda authorizers, or other advanced features, you’re going to be gritting your teeth once you realize this lower price doesn’t apply to you.

• Express Workflows. Run a simple workflow (choreography) at scale and on the cheap.

If you’re not an existing user: You can probably ignore this. It won’t make you change your mind about using (or not using) serverless. It’s aimed at price- and scale-conscious existing users. If you do need/use workflows on AWS, just know there are now two flavors to choose between, not including the (really old) SWF service, which technically makes three.

If you’re an existing user: This might feel like a price cut or “limit lifting”, but in fact the semantics are so different that it’s better to think of this as another service entirely. What is it? Imagine you had a bunch of activities to choreograph, but you were happy to have the whole thingamabob succeed or fail as a unit. You could just code it up as a Lambda function…but there’s this cute workflow language, ASM, and a nice JSON abstraction that Step Functions defined…wouldn’t it be cool if AWS let you run the Step Functions ASM interpreter as a built-in Lambda language? That’s basically what Step Functions Express Workflows are. Now that you understand what it is, though, should you use it?

AWS says that Express Workflows are for programmatic workflows. What does that mean? Let’s say that you have an S3 bucket with a Lambda function set up as an event handler. When a new file arrives, you have a bunch of things to do: Call an API, update a database, put some stuff in a queue and wait for it to drain, etc. But you might get a lot of files (and thus events) firing at once, potentially exceeding either the ability of a “normal” Step Function to launch (rate limiting) or your willingness to pay for it (cost concerns). The new Express Workflows let you launch much faster and for a lot less money. The tradeoff? They don’t offer the reliability of existing Step Functions — they have at-least-once semantics, versus exactly once, they don’t do any internal checkpointing, and they have a 5 minute limit on total running time versus a year with normal Step Functions.

So the big question: Are these limitations worth it, versus just writing conditionals and loops in your favorite programming language? This is somewhat akin to the “CloudFormation versus CDK” debate raging right now: If you love declarative semantics and purpose-built DSLs, you’re going to love Express Workflows. If you think the CDK was a breath of fresh air and you value Intellisense in VS Code above all else, you’re going to find Express Workflows crazy. If you’re seeking practical advice, I’d say the breakeven is when you have more than 3 “things to do” in a function, especially if they involve other AWS services: Less than that, and just writing the code and avoiding the use of another service is probably best. More than that, and you’re getting to the point where your function is dominated by a workflow, and separating the choreography from the computational logic starts to sound like a good 12-factorish separation of concerns.

• Lambda Destinations (and retry controls and improved Kinesis controls, …). Lambda added a whole bunch of controls to make working with event chains and polled event sources easier, safer, more reliable, and more debuggable.

If you’re not an existing user: You can ignore all this; it probably won’t make you change your mind about using (or not using) serverless. Just know that AWS continues to use their operational heritage (and first mover advantage in serverless) to continue breaking new ground in event handling.

If you’re an existing user: PAY ATTENTION TO THIS. You almost certainly have existing solutions and designs that absolutely need to get reconfigured as a result of at least one of the following improvements:

• Customizable retry count (0, 1, or 2 retries; it used to be a fixed 2 retries, for 3 total attempts)
• Retry TTL, to avoid out-of-date processing in the event of an outage or delay
• Kinesis and DynamoDB stream worker fanout (beyond the previously built-in 1:1 ratio)
• Poison pill recovery for Kinesis and DynamoDB streams (the somewhat professorially named, “Bisect on Failure”)
• SNS DLQs (technically an SNS feature, but one which effectively makes the use of SNS for Lambda fanout reliable, since you can hook that queue up to a Lambda function easily)
• Built in FIFO SQS for Lambda
• Event chaining and generalized DLQ enablement, via Lambda Destinations. This is a big one, so let’s drill into some of the cool things you can do with it:

Generalized error handling: Wish you could do something other than just dropping an event into as SQS queue when an asynchronously invoked Lambda has a problem? Now you can (reliably) execute another Lambda function to handle the error instead.

Simple continuations: Not quite done with processing that Lambda function? Call another one when it finishes to carry on! (Sorry, it has to be a different one…no direct tail recursion allowed ;-) Think of this option as the complement of error handling for async invokes: It’s a success handler.

Callbacks: Wish you could get called back when async functions complete without the overhead of wrapping it in a Step Function or having to hack it into your source code? Now you can set up a Lambda Destination to do just that.

Enhanced logging or tracing hooks: Want to record some information when an async function runs without having to change its source code? Turn on Lambda Destinations and hook away.

These are some pretty cool patterns (especially those of the “post invoke hook” flavor) that will take developers awhile to understand and adopt, but they greatly expand the range and capability of event-driven chains using Lambda…for existing users, this is definitely a “sleeper hit” release that will keep on giving!

• CloudWatch Synthetics. What AWS calls “canaries” — customer-proxy monitors that are managed for you.

If you’re not an existing user: It won’t change your mind about serverless one way or the other, though you might find it independently useful for your other microservices.

If you’re an existing user: No need to rush, but you should consider trying this for “outside-in” uptime and latency testing for your key user flows. I’d expect AWS to keep investing here; canaries are a core part of how AWS teams think internally about operational hygiene (and a H-U-G-E part of Amazon’s DevOps culture), so getting in on the ground floor on this feature is probably worth it.

What they are: Like Express Workflows, Synthetics are basically a custom Lambda runtime plus a cron setting plus a little built-in wire-up to CloudWatch metrics plus a dashboard to make it easy to consume. There are a few templates for different kinds of testing. Today, you could do this almost as easily yourself; the managed service just makes it a little easier to get started and monitor in an ongoing fashion. Over time, though, I’d expect AWS to put more muscle behind this, with automation for testing from multiple regions, scaling controls (think artillery), etc. that add value (and which would be increasingly expensive to DIY).

• Event schema discovery and registry. Users of EventBridge can now use it not just as an event bus but also as a service registry for schemas and as a way to dynamically detect schema updates.

If you’re not an existing user: You almost certainly aren’t using EventBridge anyway, so you can ignore what’s going on here. (But see previous comment about AWS being the thought leader on async design patterns for the day when you decide you want some of that.)

If you’re an existing user: If you were wondering how EventBridge intends to differentiate itself from SNS, this release telegraphs the answer: typed messages with strong governance and auditing features at a “semantic” level of abstraction. But forward-looking reveal aside, it’s still early days here, and with neither type checking for Lambda nor a completed open standard for cross-cloud events, the schema discovery and registry features are of somewhat limited utility. Summary: Give this one some more bake time, but keep an eye on it.

• Pro forma stuff. Language updates, SAR author badges, percentiles for duration metrics, X-Ray tracing improvements, improved CloudWatch dashboarding, increased limits for classic Step Functions, and more.

If you’re not an existing user: You can ignore all this.

If you’re an existing user: Take a look at your leisure. The language updates are probably the most important to prioritize.

Other re:Invent 2019 Takeaways

The 2019 launches that weren’t (or weren’t what we might have hoped for) for current and aspiring Serverless developers.

The biggest missing launch: EFS/Lambda integration

You know that moment on the Thursday of re:Invent, around 10:30am or so, when you finally accept the fact that the launch you were hoping for isn’t going to arrive this year? It’s not like seeing the shape of a special present you were hoping for under the Christmas tree.

EFS in Lambda should be a match made in heaven: An effectively infinite disk drive mounted to the world’s largest pool of CPU silicon? Wow, what could be better? Apparently a lot, since EFS integration once again didn’t launch this year. Sad pandas.

The biggest launching miss: AWS Wavelength

I’d love the hassle of deploying, managing, and monitoring thousands of tiny EC2 servers mounted on carrier equipment with the need to rapidly re-route traffic for mobile users…said no one, ever. With the Verizon partnership, AWS finally had the hardware and networking reach to truly do global “edge” functions…but instead they reverted to their most cumbersome, hardest to manage, and least fault- and location-tolerant compute abstraction, EC2. Oh well, there’s always re:Invent 2020.

Not Using Serverless Yet? Why You Need to Care about re:Invent 2019’s Serverless Launches was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

What if every developer had access to the world’s largest supercomputer?

The Serverless Journey So Far

When AWS Lambda launched on November 13, 2014, it created a new way of writing applications in the cloud. Today, developers around the world in every conceivable type and size of organization enjoy the lower costs and faster time to market of Serverless apps.

The journey from that initial release to today isn’t just about growth in adoption and scale: It’s also the story of an ever-increasing “target space” for Serverless apps.

What began as a way to react to objects stored in Amazon S3 has grown to encompass event handling across the portfolios of every major cloud vendor. Over the years new capabilities were added that enabled Serverless architectures to be used for building mobile and web backends, process streaming data, replace cron jobs, and much more.

Cloud vendors have found ways to make more and more of their capabilities serverless: not just the obvious ones like object stores and pub/sub services, but also SQL databases, data analytics queries, IoT, and more.

This innovation is far from over— recent advances such as Amazon API Gateway’s ability to auto-frame websocket traffic and turn it into Lambda function calls, also known as mullet architectures, enables a whole new class of capabilities that formerly required heavyweight servers and deployment methodologies.

These innovations come in many forms: new patterns and frameworks, new services (like AWS’s AppSync and Google’s Cloud Run), and the ever-growing set of features among the existing cloud vendor Serverless offerings.

The simplicity of Serverless coupled with its rapid pace of innovation has helped it become wildly successful — the benefits of not needing to own, scale, manage, or secure infrastructure lets developers focus on their business instead of the care and feeding of servers or containers.

These benefits aren’t just technical — companies that adopt Serverless architectures also typically lower their cloud compute bills by up to 90% or more thanks to the fact that Serverless functions achieve 100% utilization, versus typical corporate server utilizations of 10–15%.

But as great as these advances have been, even enabling some companies to go entirely Serverless, they’re not enough. Large scale data processing, parallel algorithms, streaming content, distributed job management and plenty of other tasks are still primarily handled using the same multi-tier server architectures we were deploying in the 1990's.

Is it possible to apply Serverless techniques to big data, ML modeling, generic algorithms, video transcoding, and other compute-intensive domains and gain the same kinds of cost and time-to-market improvements that it offers today for web, mobile, and event-based applications?

Can we get the same benefits for High Performance Computing (HPC) applications that Serverless brought to event handling and web and mobile backends?

The Rise of the Serverless Supercomputer

Complexity is never a virtue. The promise at the heart of Serverless approaches is that mapping code to servers (or containers), and keeping those servers running securely at scale, should be the responsibility of cloud vendors.

Figuring out where to run code and how to keep the infrastructure healthy represents an unnecessary distraction for every application developer — regardless of what their application might be.

This has never been more true than in the world of big data and big compute, where the problems of partitioning, deploying, and managing servers can be an overwhelming challenge to the developers. It’s even more true when you consider that the “developer” of such applications might be a biology researcher, statistician, astronomer, or other professional with a need to crunch numbers but not necessarily the desire or skill to scale and babysit servers.

Meanwhile, Serverless functions now possess massive fleets, with aggregate computing power that’s larger than the entire computing capability of the public cloud just a few years ago. With that scale of processing power, Serverless functions in the cloud now rival purpose-built supercomputers in the total amount of silicon they employ.

The question is no longer whether they will scale as a service, but whether we can find ways to harness all that power in actual HPC applications.

There are some tantalizing suggestions that this is possible. The world’s fastest video transcoding algorithm known today, ExCamera, doesn’t run on GPUs or other specialized hardware — it was built using AWS Lambda.

Serverless algorithms for highly parallel linear algebra are already within 33% of conventional approaches while exhibiting 240% better compute efficiency.

Fannie Mae has ported massive Monte Carlo simulations on loan portfolios from mainframes to AWS Lambda, resulting in faster completion times, lower costs, and more flexible scaling. Other researchers have found ways to take jobs normally run on laptops and desktops and move them into the cloud, shaving hours off distributed build and compute times.

Solving existing problems at this scale is exciting, but it doesn’t stop there. It’s also highly likely that Jevon’s Paradox will apply. For example, if we can find ways to parallelize video transcoding such that editing can happen in real time, more people will edit more videos.

This suggests that, far from being the rarefied province of researchers and a few hardcore number crunchers, “Serverless Supercomputing” will actually unlock new applications and enable powerful new features in the apps we use today.

Bottom line? The easier and cheaper it becomes to apply massive computing power to find answers quickly to hard problems, the more likely we are to ask such questions.

Serverless supercomputers are still missing some parts

The examples above are exciting — they illustrate what’s possible with a Serverless approach and that it has the potential to revolutionize these applications — just as it changed how we write event handlers and API-driven backends.

But here’s the bad news: Each of the teams building one of the applications listed above had to manually create a lot of scaffolding in order to get their job done. None of these were simple “out of the box” applications, and creating that underlying framework for distributed algorithms on top of vendor-provided Serverless offerings was nontrivial in every case.

The good news? It was also fairly similar — and we can read the list of what was required as a form of product spec for building the Serverless Supercomputing platform of the future.

Before we tackle the specifics, it’s worth asking the broader question: Why would Serverless algorithms and architectures differ from what’s come before? The answer is simple: For most of history we’ve thought about the theory and practice of distributed systems as unlimited time on limited space.

The problem statement in pretty much every distributed systems research paper or highly parallel algorithm has been to harness a limited set of machines (servers) that live forever to perform a computation that can’t fit inside just one of them.

Serverless turns that fundamental assumption on its head: Its computation model is limited time on unlimited space. That shift causes us to rethink everything about our approach to distributed algorithms — and creates the disruptive potential for entirely new innovations that can bring previously unavailable computing power to the masses!

Serverless changes our model of distributed computation.

What’s needed to enable a Serverless Supercomputer?

Creating that capability requires a combination of closing gaps to conventional (“serverful”) compute, like networking, as well as crafting new services that address the impedance mismatch between compute (cloud functions like AWS Lambda) and server-based storage technologies like redis.

It requires fundamentally new algorithms for sorting and searching that can exploit Serverless’s ability to tightly envelope costs while playing nicely within its temporal limitations.

And it requires new frameworks, including the ability to rapidly distribute and choreograph work across tens of thousands of functions running in parallel.

Lets take a look at some specific “missing parts” and the roles they would play in a Serverless Supercomputer architecture:

#1 Distributed networking
All distributed algorithms depend on the ability to connect computing nodes to one another, and in many cases on the ability to dynamically establish and modify the topology of those communication paths.

Today’s cloud functions don’t support this capability directly, and in fact most disable inbound TCP/IP. To make Serverless Supercomputing possible, we need to address this gap in the fundamentals.

#2 Fine-grained, low-latency choreography
While existing managed workflow services do a nice job at modeling business processes, they are generally orders of magnitude too slow (and too costly) to manage the lifecycle of tens of thousands of function lifetimes, let alone to coordinate intra-function invocation transitions.

A Serverless Supercomputer will also need custom workflow functionality, such as straggler management, that reflects its unique semantics and challenges.

#3 High-speed key-value stores
Probably the most obvious missing piece in the managed portfolios of today’s cloud vendors is a high-speed (DRAM-based) key-value store. Cloud object stores such as Amazon S3 offer low-cost bandwidth and high throughput, but at the expense of high latency.

NoSQL databases, such as Amazon DynamoDB, offer faster performance but at costs that are too high to effectively serve as a shared memory system for transient state. This isn’t surprising; DynamoDB is designed to make writes persistent and durable…neither of which are typically necessary for storing the in-flight state of an algorithm that could simply be restarted.

Server-based infrastructure deployments like redis (or their cloud vendor equivalents) suffer from a grave impedance mismatch with Serverless functions, taking us back to the world of individual server management and scaling. Fortunately, recent work in academia shows promise in better matching storage lifetimes to Serverless algorithms and in designing cost effective scalable storage.

#4 Immutable inputs and outputs associated with function invocations
Systems that have succeeded at utilizing the power of Serverless at massive computational scale have all needed to find a way to create input and output bindings in order to execute large-scale dataflows.

Unlike the event-based bindings that exist today in some Serverless architectures, these objects are usually ephemeral outside of the algorithm’s execution and occur at massive scale.

Fortunately, immutable object management frameworks have proven tractable to layer on top of existing cloud vendor services; they just need to be harvested out of academic research and made available to developers at large.

#5 Ability to customize code and data at scale
Serverless functions like AWS Lambda have “control planes” (APIs for creating, updating, and deleting functions) that were designed for human interaction scale. But these control planes are ill-suited for a situation where 10,000 parallel function invocations each need slightly different code to perform a coordinated algorithm and then disappear forever.

Similarly, cloud function invocation is designed to transmit arguments (such as event payloads) only once, when the function is invoked, and then return a single result when it completes. Distributed algorithms may require large amounts of data to be delivered throughout a function’s lifetime, with partial results being transmitted before the function exits.

#6 Execution dataflows
With support for the preceding items, it’s possible to create and execute dataflow graphs that represent complex, multi-node distributed algorithms at massive scale. These dataflows will represent a new layer of abstraction for Serverless computing, because they describe the entire computation, not just how to run or invoke an individual function.

#7 General purpose and domain-specific algorithms customized to the “size and shape” of Serverless functions
Finally, developers need support for both general purpose (sorting, searching, grouping, etc.) and domain-specific (video transcoding, genetics) libraries that are tuned to the Serverless platform. With these in place, they can focus on the specific needs of their algorithms and avoid reinventing the wheel to get things done.

The Journey Ahead

As the saying goes, the future’s already here — it’s just not evenly distributed yet. Building any new technology stack takes time and work, so expect to see the Serverless Supercomputer emerge incrementally over the next several years.

University researchers are already hard at work and companies like ServerlessTech are already tackling the problem of distributed networking for cloud functions.

Fortunately, it’s not an all-or-nothing proposition: Every incremental step will open up new application domains to take advantage of Serverless approaches.

Just adding support for networking, for example, makes prewarming capacity, peer-to-peer file sharing, real-time content distribution, and exchanging additional arguments and results with a cloud function possible.

Cloud vendors also have critical roles to play that only they can address: improvements to network bandwidth and jitter, additional memory and file system storage, higher (and easier to self manage) limits on concurrency, and a host of feature details from managed NAT customization to more contextual information inside of functions.

There’s a lot to do, but it’s never been a better time to be a developer or innovator in this space — the world’s most powerful and easiest to use supercomputer is being designed and built right now!

This article includes material from my ServerlessConf NYC ’19 presentation (slides here).

The Serverless Supercomputer was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

Back in the spring of 2016, A Cloud Guru hosted the first-ever Serverless conference in a Brooklyn warehouse. In many ways, that inaugural conference was the birth of the Serverless ecosystem.

Serverlessconf was the first time that this growing community came together to talk about the architectures, benefits, and approaches powering the Serverless movement.

Last week A Cloud Guru once again brought top cloud vendors, startups, and thought leaders in the Serverless space to New York City to exchange ideas, trends, and practices in this rapidly growing space.

In addition to the “hallway track”, which was a great way to meet and (re)connect with talented and passionate technology experts — there were multiple tracks of content.

Collectively, these conferences are a great way to take the pulse of the community — what’s getting better, what’s still hard, and where the bleeding edge of innovation sits.

With apologies to the vendors and talks I didn’t manage to get to, here’s my take on the State of Serverless after spending nearly a week with many of its best and brightest.

Enterprise users have shown up — with their stories
Back in 2016, much of the content (and nearly every talk’s opening slides) at Serverlessconf was some flavor of “Here’s how we define Serverless.”

Content focused on how to get started and lots of how-to talks. Notably absent back in 2016? Enterprise users talking about their experiences applying Serverless in real life with the sole exception of Capital One.

While the latest Serverlessconf retains its technology and practice focus, it was fantastic to see companies like the Gemological Institute of America, Expedia, T-mobile, Mutual of Enumclaw Insurance, and LEGO up on stage in 2019 talking about adopting and benefitting from Serverless architectures.

Growing ecosystem
The highly scientific metric of “square feet of floor space devoted to vendors” continues to grow year over year. But more importantly, those vendors have moved from early stage awareness and information gathering to offering products and services in the here and now.

System integrators and consulting firms specializing in Serverless practices are also showing up — more evidence of enterprise traction in the space.

Configuration & AWS CloudFormation are still creating friction
The buzz was around whether declarative or imperative “Infrastructure-as-Code” is the better approach, alternatives to CloudFormation, easier ways to construct and deploy Serverless architectures. Topics like these featured strongly in both actual content and hallway conversations in 2019 — just as they did in 2016.

Whatever your position on recent approaches like AWS’s cdk and the utility of declarative approaches like AWS SAM, it’s clear that CloudFormation and other vendor-provided options still aren’t nailing it.

Vendors like Stackery.io got a lot of foot traffic from attendees looking for easier ways to build and deploy Serverless apps, while talks from Brian LeRoux and Ben Kehoe explored both the problem, and potential solutions, to the difficulties of using CloudFormation today.

Google and Cloudflare are playing the role of category challengers
Google Cloud Run is taking an intriguing approach — offering customers a container-based specification with the scales-on-usage and pay-per-request model of AWS Lambda. It’s still too early to call GCR’s product market fit, but it’s exciting to see Google step back and reimagine what a Serverless product can be.

Meanwhile, Cloudflare workers exploit that company’s massive edge infrastructure investment to run chunks of computation that make Lambda functions look huge by comparison. It’s not necessarily a solution to general compute, but given expectations that the bulk of silicon will live on the edge, rather than in data centers, in the future, I’d keep my eye on this one.

Serverless innovation isn’t over
Johann Schleier-Smith talked about UC Berkeley’s position paper on Serverless and the growing attention that Serverless is getting from the research community.

Yours truly laid out a recipe for building the Serverless Supercomputer, starting with Serverless Networking that opens the door to building distributed algorithms serverlessly.

Chris Munns reviewed the pace of innovation for AWS Lambda since its launch in 2014 and hinted at more to come at next month’s AWS re:Invent in Las Vegas.

With their amusing name helping to grab attention, The Agile Monkeys presented a Serverless answer to Ruby on Rails with a high-level object model that compiles down to Lambda functions and other serverless componentry.

It’s still not easy enough
Serverless might sound like a technology stack, but it’s really a vision for software development. In contrast to the ever-growing complexity of servers and Kubernetes, attendees at a Serverless conference are looking for ways to do more with less — less infrastructure, less complexity, less overhead, and less waste.

But while a desire for simplicity and “getting the business of business” done unites the attendees at a conference like this, it’s still the case that too much non-essential complexity gets in the way.

Tools, IDEs, debuggers, security, config & deployment, CI/CD pipelines…a lot of energy from vendors to startups to consultants to enterprise adopters is flowing into getting Serverless projects across the finish line. It may be way easier than servers (and containers), but it’s clearly still not easy enough.

Conferences like this help, but more and better documentation, more sharing of best practices, and tools that can truly streamline the job of delivering business value on top of Serverless remain a work in progress…leaving a lot of untapped potential opportunity in the space still to explore!

Author disclosures: I presented at Serverlessconf NYC ’19 for which I received a registration fee waiver. I’m a former employee of both AWS and Microsoft and currently an independent board member of Stackery.io. I received no compensation from any of the companies or organizations cited above for writing or distributing this article and the opinions provided are my own.

The State of Serverless, circa 2019 was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

Serverless is one of the fastest growing movements in our industry, and it’s changing how developers and companies alike approach delivering business value using the public cloud.

Until today, serverless functions like AWS Lambda have been missing a critical ingredient needed to unlock their full potential — peer-to-peer networking.

Without peer-to-peer networking, many important use cases are either impossible to achieve or require so much heavy lifting as to place them out of the range of most application developers.

Some sample use cases include streaming content to or from a function, sharing files and in-memory objects among two or more functions, harvesting the results of an asynchronously invoked function, and pre-warming capacity.

Without these capabilities, many companies struggle to adapt AWS Lambda or forego the benefits of serverless architectures to solve compute-dense distributed algorithms or demanding communication topologies — such as genetic algorithms, video transcoding, Monte Carlo simulations, and more.

The launch of Serverless Networking fills this gap by making it easy to connect two Lambda functions to each or to connect a server-based implementation to a Lambda with two lines of code.

Serverless Networking isn’t just easy to connect — it’s easy to use with high-level Python 3 language bindings that make it easy to send and receive strings, byte buffers, memory buffers, files, and streams.

Pausing a function to create pre-warmed capacity, performing barrier synchronization, harvesting asynchronously invoked child function results, and other simple orchestrations are now as easy as sending and receiving messages.

Serverless Networking takes care of the heavy lifting by creating a reliable UDP-based transport using one of the fastest known approaches, udt. All the painful details are abstracted away — including sliding windows, retransmissions, ACK/NACK transmission, flow control, and more.

This allows developers to focus on the job to be done, rather than how to dependably ship bits over the wire. Serverless Networking complements Lambda’s existing TCP/IP and UDP outbound networking capabilities without requiring any changes to the code you already have.

To enable communication between two Lambda functions, or between a Lambda function and a server-based implementation, Serverless Networking also offers NAT Punching-as-a-Service — a scalable, multi-tenanted service that replaces the heavyweight STUN and TURN servers typically used to connect or relay UDP traffic among media servers.

The Serverless Networking NAT Punch service employs a Serverless mullet architecture that exploits the latest capabilities in AWS managed services to achieve scalability at low cost.

The NAT Punching service is free to use during the beta and has a perpetual free tier so that anyone can experiment with Serverless Networking without needing to sign up or deploy a server. Premium and enterprise subscribers can take advantage of low-cost pay-per-use or pre-paid volume discount pricing or deploy their own service.

The Serverless Networking framework is distributed as a public AWS Lambda layer, so you can start using it with just a few clicks. It’s also open source, with an enterprise-friendly BSD license that makes it easy to adopt, use, and enhance for any organization.

You can download a sample app from the AWS Serverless Application Repository to see a running example in your AWS account in just a few minutes.

The beta is open to everyone and is currently available for AWS Lambda functions in the us-east-1 region. To get started, check out the docs.

You can also review the code on git and learn more about the open source effort to build the world’s best serverless networking stack and how you can participate. To learn more about how reliability is created on top of UDP sockets, check out the udt project.

The slides from my Serverlessconf NYC ’19 talk also has a basic overview of NAT punching and discusses peer-to-peer networking’s role as a layer in the “Serverless Supercomputer” tech stack.

Tim Wagner on Twitter

Gonna talk about what it takes to enable the next round of serverless innovation and scale. And maybe do a little server damage 😉

Rafael Lopes on Twitter

THIS IS WHY I CAME. @timallenwagner #SERVERLESSCONF https://t.co/LUOm71LiKV

I can’t wait to see the amazing new innovations developers will create with the advent of Serverless Networking!

Serverless Networking is the next step in the evolution of serverless was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

The metrics point to crypto still being a toy until it can achieve real world business scale demonstrated by Amazon DynamoDB

14 transactions per second. No matter how passionate you may be about the aspirations and future of crypto, it’s the metric that points out that when it comes to actual utility, crypto is still mostly a toy.

After all, pretty much any real world problem, including payments, e-commerce, remote telemetry, business process workflows, supply chain and transport logistics, and others require many, many times this bandwidth to handle their current business data needs — let alone future ones.

Unfortunately the crypto world’s current solutions to this problem tend to either blunt the advantages of decentralization (hello, sidechains!) or look like clumsy bolt-ons that don’t close the necessary gaps.

Real World Business Scale

Just how big is this gap, and what would success look like for crypto scalability? We can see an actual example of both real-world transaction scale and what it would take to enable migrating actual business processes to a new database technology by taking a look at Amazon’s 2019 Prime Day stats.

The AWS web site breaks down Amazon retail’s adoption and usage of NoSQL (in the form of DynamoDB) nicely:

Amazon DynamoDB supports multiple high-traffic sites and systems including Alexa, the Amazon.com sites, and all 442 Amazon fulfillment centers. Across the 48 hours of Prime Day, these sources made 7.11 trillion calls to the DynamoDB API, peaking at 45.4 million requests per second.

45 million requests per second. That’s six zeros more than Bitcoin or Eth. Yikes. And this is just one company’s traffic, and only a subset at that. (After all, Amazon is a heavy user of SQL databases as well as DynamoDB), so the actual TPS DynamoDB is doing at peak is even higher than the number above.

Talk about having a gap to goal…and it doesn’t stop there. If you imagine using a blockchain (with or without crypto) for a real-world e-commerce application and expect it support multiple companies in a multi-tenanted fashion, want it to replace legacy database systems, and need a little headroom to grow — a sane target might look like 140 million transactions per second.

That’s seven orders of magnitude from where we are today.

The Myth of Centralization

Why are these results so different? Let’s examine this dichotomy a little closer. First, note that DynamoDB creates a fully ordered ledger, known as a stream, for each table. Each table is totally ordered and immutable; once emitted, it never changes.

DynamoDB is doing its job by using a whole lot of individual servers communicating over a network to form a distributed algorithm that has a consensus algorithm at its heart.

Cross-table updates are given ACID properties through a transactional API. DynamoDB’s servers don’t “just trust” the network (or other parts of itself), either — data in transit and at rest is encrypted with modern cryptographic protocols and other machines (or the services running on them) are required to sign and authenticate themselves when they converse.

Any of this sound familiar?

The classic, albeit defensive, retort to this observation is, “Well, sure, but that’s a centralized database, and decentralized data is so much harder that is just has to be slower.” This defense sounds sort of plausible on the surface, but it doesn’t survive closer inspection.

First, let’s talk about centralization. A database running in single tenant mode with no security or isolation can be very fast indeed — think Redis or a hashtable in RAM, either of which can achieve bandwidth numbers like the DynamoDB rates quoted above. But that’s not even remotely a valid model for how a retail giant like Amazon uses DynamoDB.

Different teams within Amazon (credit card processing, catalog management, search, website, etc.) do not get to read and write each others’ data directly — these teams essentially assume they are mutually untrustworthy as a defensive measure. In other words, they make a similar assumption that a cryptocurrency blockchain node makes about other nodes in its network!

On the other side, DynamoDB supports millions of customer accounts. It has to assume that any one of them can be an evildoer and that it has to protect itself from customers and customers from each other. Amazon retail usage gets exactly the same treatment any other customer would…no more or less privileged than any other DynamoDB user.

Again, this sounds pretty familiar if you’re trying to handle money movement on a blockchain: You can’t trust other clients or other nodes.

These business-level assumptions are too similar to explain a 7 order of magnitude difference in performance. We’ll need to look elsewhere for an explanation.

Is it under the hood?

Now let’s look at the technology…maybe the answer is there. “Consensus” often gets thrown up as the reason blockchain bandwidth is so low. While DynamoDB tables are independent outside of transaction boundaries, it’s pretty clear that there’s a lot of consensus, in the form of totally ordered updates, many of which represent financial transactions of some flavor in those Prime Day stats.

Both blockchains and highly distributed databases like DynamoDB need to worry about fault tolerance and data durability, so they both need a voting mechanism.

Here’s one place where blockchains do have it a little harder: Overcoming Byzantine attacks requires a larger majority (2/3 +1) than simply establishing a quorum (1/2 +1) on a data read or write operation. But the math doesn’t hold up: At best, that accounts for 1/6th of the difference in bandwidth between the two systems, not 7 orders of magnitude.

What about Proof of Work? Ethereum, Bitcoin and other PoW-based blockchains intentionally slow down transactions in order to be Sybil resistant. But if that were the only issue, PoS blockchains would be demonstrating results similar to DynamoDB’s performance…and so far, they’re still not in the ballpark. Chalk PoW-versus-PoS up to a couple orders of magnitude, though — it’s at least germane as a difference.

How about the network? One difference between two nodes that run on the open Internet and a constellation of servers in (e.g.) AWS EC2 is that the latter run on a proprietary network. Intra-region, and especially intra-Availability Zone (“AZ”) traffic can easily be an order of magnitude higher bandwidth and an order of magnitude lower latency than open Internet-routed traffic, even within a city-sized locale.

But given that most production blockchain nodes at companies like Coinbase are running in AWS data centers, this also can’t explain the differences in performance. At best, it’s an indication that routing in blockchains needs more work…and still leaves 3 more orders of magnitude unaccounted for.

What about the application itself? Since the Amazon retail results are for multiple teams using different tables, there’s essentially a bunch of implicit sharding going on at the application level: Two teams with unrelated applications can use two separate tables, and neither DynamoDB nor these two users will need to order their respective data writes. Is this a possible semantic difference?

For a company like Amazon retail, the teams using DynamoDB “know” when to couple their tables (through use of the transaction API) and when to keep them separate. If a cryptocurrency API requires the blockchain to determine on the fly whether (and how) to shard by looking at every single incoming transaction, then there’s obviously more central coordination required. (Oh, the irony.)

But given that we have a published proof point here that a large company obviously will perform application level sharding through its schema design and API usage, it seems clear that this is a spurious difference — at best, it indicates an impoverished API or data model on the part of crypto, not an a priori requirement that a blockchain has to be slow in practice.

In fact, we have an indication that this dichotomy is something crypto clients are happy to code to: smart contracts. They’re both 1) distinguished in the API from “normal” (simple transfer) transactions and 2) tend to denote their participants in some fashion.

It’s easy to see the similarity between smart contract calls in a decentralized blockchain and use of the DynamoDB transaction API between teams in a large “centralized” company like Amazon retail. Let’s assume this accounts for an order of magnitude; 2 more to go.

Managed Services and Cloud Optimization

One significant difference in the coding practices of a service like DynamoDB versus pretty much any cryptocurrency is that the former is highly optimized for running in the cloud.

In fact, you’d be hard pressed to locate a line of code in DynamoDB’s implementation that hasn’t been repeatedly scrutinized to see if there’s a way to wring more performance out of it by thinking hard about how and where it runs. Contrast this to crypto implementations, which practically make it a precept to assume the cloud doesn’t exist.

Instance selection, zonal placement, traffic routing, scaling and workload distribution…most of the practical knowledge, operational hygiene, and design methodology learned and practiced over the last decade goes unused in crypto. It’s not hard to imagine that accounts for the remaining gap.

Getting Schooled on Scalability

Are there design patterns we can glean from a successfully scaled distributed system like DynamoDB as we contemplate next-generation cryptocurrency blockchain architectures?

We can certainly “reverse engineer” some requirements by looking at how a commercially viable solution like Amazon’s Prime Day works today:

• Application layer (client-provided) sharding is a hard requirement. This might take a more contract-centric form in a blockchain than in a NoSQL database’s API, but it’s still critical to involve the application in deciding which transactions require total ordering versus partial ordering versus no ordering. Partial ordering via client-provided grouping of transactions in particular is virtually certain to be part of any feasible solution.
• Quorum voting may indeed be a bottleneck on performance, but Byzantine resistance per se is a red herring. Establishing a majority vote on data durability across mutually authenticated storage servers with full encoding on the wire isn’t much different from a Proof-of-Stake supermajority vote in a blockchain. So while it matters to “sweat the details” on getting this inner loop efficient, it can’t be the case that consensus per se fundamentally forces blockchains to be slow.
• Routing matters. Routing alone won’t speed up a blockchain by 7 orders of magnitude, but smarter routing might shave off a factor of 10.
• Infrastructure ignorance comes at a cost. Cryptocurrency developers largely ignore the fact that the cloud exists (certainly that managed services, the most modern incarnation of the cloud, exist). This is surprising, given that the vast majority of cryptocurrency nodes run in the cloud anyway, and it almost certainly accounts for at least some of the large differential in performance. In a system like DynamoDB you can count on the fact that every line of code has been optimized to run well in the cloud. Amazon retail is also a large user of serverless approaches in general, including DynamoDB, AWS Lambda, and other modern cloud services that wring performance and cost savings out of every transaction.

We’re not going to solve blockchain scaling in a single article 😀, but there’s a lot we can learn by taking a non-defensive look at the problem and comparing it to the best known distributed algorithms in use by commercial companies today.

Only by being willing to learn and adapt ideas from related areas and applications can blockchains and cryptocurrencies grow into the lofty expectations that have been set for them…and claim a meaningful place in scaling up to handle real-world business transactions.

Crypto can’t scale because of consensus … yet Amazon DynamoDB does over 45 Million TPS was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

I recently spent a very long 15+ hour day transcribing a working application that I had built through the AWS console UI and command line scripts into a declarative deployment using AWS CloudFormation.

Most of the examples in the official AWS docs are usually pretty “micro” and my application — while not large — is sufficiently complicated that it took a lot of searching and experimenting to get it all to hang together and to run inside of AWS CodeStar.

I benefitted a lot from others who had blogged solutions and tutorials, so in a desire to give back, here it is: An annotated tour through a non-trivial CloudFormation template that deploys an Amazon API Gateway websocket backed by AWS Lambda and an Amazon DynamoDB table. Think of it as “CRUD for Serverless Websockets”.

What follows is all details…if you’re looking to learn about websockets, hear why Serverless is cool, or want to read another religious treatise on the declarative versus imperative debate — these aren’t the droids you’re looking for (tm).

While this post focuses on the deployment spec per se, if you’d like to learn more about the design pattern it encodes, check out my companion piece on Serverless Mullet Architectures:

CloudFormation is Tricky

Okay, let’s get the bad news over with right away: AWS CloudFormation (aka CF) can be tricky. And verbose. And hard to debug. And while the tooling could be better (I tried and struck out at finding a VS Code extension that can validate CF references) the fact is, both the CloudFormation implementors and we users of it have some hard problems to solve.

This is true no matter what you’re deploying: In an infrastructure-based deployment, there are a lot of gritty details to deal with. But even in a Serverless deployment, we’re shifting big chunks of the application burden onto the cloud vendor, so there’s quite a bit of “configurating” to get them to do all that work for us, as we’ll see.

Including comments, this CF template is over 500 lines, longer than the code in the Lambda function it deploys. Brace yourself.

What’s Not In My CloudFormation Template

Let’s get started by looking at the “negative space” — what I didn’t put into this template and why:

One-time account-level setup
When I began, it seemed intuitive to me that I should try to “CloudFormation all the things”; in other words, create a single script that would take a fresh AWS account and turn it into one running my Serverless NAT Puncher.

Unfortunately, I quickly learned that one-time setup is tricky to include. CloudFormation has a strong bias towards building and managing deployments that can be torn down again.

That’s a critical distinction — it means that tasks like the one-time setup of an API Gateway logging service role are not well tolerated, because undoing them could render everything else in the same account unusable.

Lesson learned: Keep the content of the deployment limited to things you are comfortable deleting.

Hard to reverse (or re-run) activities
Same lesson as above, but learned trying to embed Amazon Certificate Manager (aka “ACM”) into a template. I was hoping I could reference a cert’s simple name and have CF use ACM to look up its ARN (as the simple name felt a little less like a magic string), but alas this failed.

Lesson learned: Accept that there will be places where you have to keep strings consistent manually.

Test client packaging
In my original “all in one” approach I tried packaging both tests and the service they tested into a single CF template. While YMMV, for me this was a mistake — I ended up needing to tear down, restructure, and recreate the stacks for the tests far more frequently, and quickly pulled them out of the service template.

Lesson learned: Template the tests (or other swim lanes of different speeds) separately.

CodeStar (and its brethren)
I also relearned another critical limitation very quickly: AWS CodeStar only makes projects; it won’t “adopt” existing projects.

If you’re excited to use the Code* suite to build, manage, deploy, and share your projects, you have to decide that RIGHT UP FRONT, generate a CodeStar sample for one of the Lambda-based projects, and then edit the CF template it generates.

[This was the case on 9/30/2019 when I wrote this, but AWS moves quickly, so it could have changed by the time you read this.]

In any case, building the pipeline with itself is too meta to work, so you have to stand that up first, then use it for your (real) application deployment.

Let’s Get to the Code

Ok, enough preliminary stuff. Let’s get to the code that *is* in there!

The Transform section is pretty straightforward; just remember that the only workable way to get that AWS::CodeStar in there is to let CodeStar generate your template for you. The CodeStar transformer is supposed to keep the permission boundary up to date…more on that below, but note that it doesn’t quite work as advertised.

I’ve included the Serverless transformer so that I can use the simplified SAM syntax for items such as the Lambda function, though it doesn’t do much to lower the overall complexity of the template, and you can easily forego it here and write in “pure” CF if you prefer.

The Parameters section represents the “arguments” to the CF template. I imagined that this would be tightly integrated into the Code* family — for example, I assumed that I’d get to configure a CodePipeline pipeline to be “development” (and another for “production”) and then it would in turn tell CodeBuild and CloudFormation the stage.

Alas, that doesn’t happen automatically…it’s all rather manually set, and in fact getting additional contextual information to flow between CodePipeline and CodeBuild appears to be pretty tough. Note that ProjectId and CodeDeployRole are inserted by CodeStar.

The Globals section is better thought of as the “defaults” section — it establishes defaults that will apply if you don’t override them on a per-object basis. These are inserted for us by CodeStar when it creates the project, and enable incremental Lambda deployments.

The “canary” in the name hints at what this means: AWS CodeDeploy will watch the function’s metric and back out the deployment if error rates go up. You can learn more about CodeDeploy’s support for traffic shifting of Lambda deployments here.

Operational aside: This setting means that, in addition to the overhead of CodeStar, github (or CodeCommit if you chose that), CodePipeline, CodeBuild, and CloudFormation, you will also have a minimum of 5 minutes for Lambda canary deployments on each successful deployment. You may want to turn this setting down (or off) during development and then reinstate it during production.

AWS Lambda Function Definition

Yay — our first actual definition!

There’s a lot of unpack here, so we’ll take it in steps:

SAM versus CloudFormation
I used the SAM type here, as you can see from the ::Serverless:: in the type. For simple Serverless projects, SAM can save a lot of typing. In this project, there’s so much verbosity that SAM doesn’t (currently) address that using it doesn’t make a ton of difference. Still, if you want to use the SAM variants, you’ll need the transformer (see above).

Manual Name Mangling
CloudFormation will name mangle (through some unpleasant hash suffixes) the outputs it creates, and CodeStar forces a little bit of project-specific name mangling into the template, but mostly you’re on your own to ensure that names don’t collide.

I prefaced the (many) items associated with the NAT Puncher with the project name so that I could add things to this template later on without worrying about collisions. Naming your function “function” or your API “api” might seem like an easy thing to avoid, but as the number of deployable objects creeps up, it gets easy to relax the mangling.

CodeUri
This is a little confusing — in this template it’s a local file or directory. The packaging step that happens at the end of the CodeBuild phase will create another copy of this template, and in that copy the CodeUri will have been rewritten to instead refer to an object in an S3 bucket (so the term “uri” sort of makes sense, eventually).

I only had one file to include, but if you have multiple files I’d suggest putting them into a src directory and making the value of this property “src/”. Note that important trailing slash.

Layers
I chose to package my Python prerequisites into a layer. Whether this is a good idea depends on your development practices. This project started life as a prototype in which I used the API Gateway console UI to construct a web socket and typed code directly into the AWS Lambda console.

In that phase of design, not having to re-upload the Python dependencies every time I made a little tweak was a huge win. It’s less of a win when you’re going through the overhead of a full CodeStar project build, particularly if the only consumer of your layer is your own project.

Still, I find it easier to have some segregation between the “real” function and the Python modules it uses. If you do decide to keep your Python prerequisites in layers, I strongly advise writing a custom script to ZIP them up…there’s a ton of cruft you can pick up if you’re not careful (like PIP) that will really bloat your Lambda function!

My suggestion is to go through every single object in the ZIP and make sure you truly need it at runtime, rather than just zipping up whatever landed in site-packages.

Role
More on this later when we define the actual role, but this is a good place to note the difference between “ref”ing and “getattr”ing an object: The former returns its simple name but you need the latter to get at its ARN.

Environment Variables
The Lambda code needs to know the name of the Amazon DynamoDB database so it can access the table at runtime. We could also have ref’ed the table here, but the function doesn’t otherwise depend on the existence of the table at deployment time.

Ref’s, sub’s, and getattr’s embed a dependency DAG into your CF template. CF runs a topological sort on that DAG to solve for a viable deployment order. Here I kept it simple by avoiding a spurious additional edge in that graph, at the cost of duplicating the name mangling formula for the DB. YMMV…pick the lesser evil :).

Coffee Break!

Tired yet? We’re just getting started, so this may be a good time for a coffee break! Now let’s get into roles and API Gateway websockets!

Lambda Layer — the Python Modules

This one is pretty simple, although the challenge of filtering the layer down to just what’s required isn’t captured here — it’s in a bash script that carefully edits what gets zipped up. Note that easy-to-miss trailing slash on the directory name.

Finally, to avoid running out of space as you develop (unless you’re a “hole in one” type), you’ll probably need to set the layer retention policy to Delete. I thought this would simply reuse layer version #1 over and over again, since that’s what happens when you delete and re-upload the same version name using the Lambda UI. But instead I found it incremented the version number while eliminating the previous version.

Lambda Function’s IAM Role

Ok, so here’s one of the longer sections: The Lambda function’s IAM role. In a pure SAM template, this can be mostly invisible, but the CodeStar-required complexity causes this to get displayed in all its verbose glory:

AssumeRole
This is the boilerplate that every Lambda function needs — permission to run as the role being defined here.

Managed Policies
These are the “meat” of the role — the things the Lambda function will be able to do. LambdaBasicExecutionRole enables the Lambda function to emit logs to AWS CloudWatch, the “minimum” role permissions for a normal function.

BTW, I don’t know why this one gets a “service-role” in its name, while other service-provided roles don’t. Anyone at AWS have insight into that naming convention?

The DynamoDB role is pretty self explanatory, though it’s overkill for what we need…and this is the double edged sword of managed policies: you end up using them because they’re simple, but they have to make the resource they control “*” since they’re generic, and they usually grant far more actions than are strictly required.

Both of these “far more than minimum privilege” sins are being committed here. On the plus side, the role is managed by AWS and worked on the first shot without having to write ASPEN (the IAM permission language) myself, saving me a bunch of time.

I left constructing a hand-authorized, tightly enveloped custom policy as a future TODO item for this project. API Gateway invocation permission is required to use the HTTPS endpoint for asynchronous callbacks; this is part of what makes websockets more powerful than just request/response to a Lambda function, but has to be reflected in the IAM role or it won’t be permitted.

Permission Boundary
Oh, CodeStar, you had to make things difficult, didn’t you? Permission boundaries are a security feature of AWS IAM that keep developers (or others) from “minting” permissions for themselves by creating a Lambda function, EC2 instance, or other role-asssuming deployment that escalates their privileges into the AWS equivalent of superuser status.

The CodeStar transformer is supposed to watch the Lambda function role and automatically manage the permission boundary to accommodate the use of additional resources, at least according to the CodeStar docs.

The idea is to enable flexible development while keeping teams safe from harm, but I found the mechanism to be poorly documented, difficult to bootstrap, and mostly a hindrance to getting things to work, because random parts of your deployment will just silently stop working if they fall outside the permission boundary.

You can see what’s permitted by going to the IAM console and viewing the permission boundary. This was a real time burner for me…I lost a few hours to debugging this particular rough spot, especially as the permission boundary for the function and CodeBuild can be different, leading to different experiences depending on how you’re testing.

AWS::Partition versus “aws”
Some AWS regions replace “aws” in the ARN with a segregated partitioning of the AWS cloud, such as “aws-cn” (China) or “aws-us-gov” (GovCloud). Using AWS::Partition in your template instead of “aws” makes it possible to reuse it transparently in these other regions…although in some cases, as is true for APIs, legal or regulatory restrictions prevent that for other reasons.

To make it more portable, I could (should) go through this template and fix the remaining ARNs to use the partition notation, but for now I don’t have plans to deploy it to any of those areas. Copy-pasting from the AWS docs and blogs will lead to inconsistent results, which is how I got into the state of mismatched partition handling.

DynamoDB Table Definition

This one is easy, and using the AWS::Serverless::SimpleTable cuts down on the clutter a bit. See note about about the name consistency versus ref.

By the way, I wanted my unit tests to be able to read & write actual data in the database, which created an interesting chicken & egg problem — I have to deploy this template to create the database, but normally deployment doesn’t happen unless the unit tests in the build stage succeed.

I just toggle off the unit tests in the (very rare) case where I need to adjust something in the table definition, but otherwise I’d have to make the unit tests run purely standalone and add another CodePipeline stage to do a post-deployment integration test.

Worth having eventually, but it was a level of complexity on top of the vanilla CodeStar pipeline that I got “for free”, and not a place where I initially wanted to spend a lot of my time.

Still with me? Ready for the hard part?

API Gateway Web Socket

API Gateway Web Socket, along with routes, integrations, responses, and models.

This one is a bear, and the gist above doesn’t even include the deployment and stage pieces (see below for that), just the functional elements.

Let’s first talk about the structure of an AWS serverless websocket as it appears in CF:

AWS::ApiGatewayV2::Api
This is the “core” definition; here, “Api” means websocket, though you’d only know that by looking at the ProtocolType property. Note that everything to do with the websocket has to use V2 of the Amazon API Gateway API; several times I slipped up and got mysterious errors until I realized one of the pieces didn’t specify V2. Nothing will warn you explicitly, sadly.

Routes
Routes are the “commands” that a websocket knows about. API Gateway plays the role of dispatcher, framing up the data and then figuring out which Lambda function to call. $connect, $disconnect, and $default are the built-in routes, while in our case pair and status are custom routes.

Integrations
Each custom route has a required integration, which is the Lambda function to call when that route is discovered in a data frame by API Gateway. For built-in routes, an integration is optional. If API Gateway can’t otherwise find a custom route to use, it uses $default; in our case, we just hook that up to an error response.

There’s a lot of religion swirling about whether to do one-Lambda-per-route or one-Lambda-for-all-routes; I’m of the opinion that the latter is easier to manage when your functions are closely related and sharing a role makes semantic sense.

Here, all my routes are part of the “CRUD” — they all read and/or write to the shared DynamoDB table, so breaking them up doesn’t accomplish anything; it’s no faster to develop and offers no additional security, while greatly complicating matters. So, one function it is.

Responses
Integrations in websockets come in two flavors: With or without a response. With a response does pretty much what you’d expect: The Lambda function that gets called can return a value; API Gateway waits for the function to complete and then places its result on the websocket, sending it back to the client — who is presumably blocking, waiting for it, although that’s outside the scope of our discussion here.

In our case, status is a good example of a route with a response: It’s basically a REST API disguised as a pair of websocket messages. Pair is an example of a route without a response; the “answer” to the pairing request is sent asynchronously, via a API Gateway-provided callback URL.

To turn on a response for a given route, you use the expression RouteResponseSelectionExpression: $default. The $default here is a bit unintuitive, as this is basically just a Boolean switch and no other values are permitted.

Models and Well Formedness Checking

The built-in routes are parameterless, but custom routes can optionally have models, as shown here for pair and status. Models allow you to associate a JSON schema with a route, allowing API Gateway to verify the syntax of the message for you.

It adds to the complexity of your deployment script, but avoids the need for you to write syntax checking code yourself, since you can count on getting only well formed messages. I couple this with using $default as an error handler.

That way, my code only has to process expected, well-formed requests and can ignore the corner cases unless it’s dealing with $default, and in that case it can just ignore the message and tell the caller that it’s malformed. (FWIW, when I wrote this was wasn’t able to get Refing of a model to work correctly, and so just used its name in the template. The route should really Ref the model.)

Websocket Miscellaneous Items

The rest of this is straightforward, if a bit finicky. By the way, the operation name seems to just be for display purposes and doesn’t carry any semantic meaning that I could find.

Given how much we had in the previous block for the websocket definition, it would seem like we’d be done with it…but in fact, there’s a lot of non-structural material needed to turn the previous objects into a functioning websocket deployment.

Before we dive into the details, though, let’s talk about The Many Notions of Deployment in AWS Serverless Services:

1. CloudFormation deployment. What we’re mostly talking about in this article — a stack (set of related resources) built by CloudFormation following the recipe in a template.
2. API Gateway deployment. API Gateway APIs (including websockets) exist in two or more states: A development state, and one or more deployed stages. API Gateway deployments can be executed as incremental (canary) deployments, though we’re not doing that here.
3. Lambda aliases. Strictly speaking, Lambda aliases don’t have any deployment semantics, but their typical use is to perform a controlled deployment or rollback.
4. CodeBuild/CodePipeline. These tools manage building and releasing a deployment, typically (though not necessarily) by integrating with CloudFormation to do the actual resource construction. They can optionally be wrapped in CodeStar, as is the case in our project.
5. CodeDeploy. CodeDeploy has a lot of infrastructure deployment uses, but in our case it’s being used to do a controlled deployment of the Lambda function after CloudFormation is complete. It relies on (3) above to handle the low-level traffic shaping.

Ok, so in our case we have a CodeStar (4) team project consisting of CodeBuild (4), CloudFormation (1), and CodeDeploy (5) running inside a CodePipeline (4) that includes a Lambda canary deployment (5) using traffic shaping (3) and a production stage deployment (2) of the API Gateway websocket as deployed by CloudFormation (1). Whew.

Websocket Deployment
Back to our regularly scheduled program: The websocket deployment is straightforward except that it includes a bunch of DependsOn statements.

Now, I’m going to say that this feels wrong to me — the value of a declarative specification should be that it gets things correct without having to encode imperative notions, such as what depends on what.

In almost all cases, CF gets this right — the service object types in CF “know” their deployment semantics, and occurrences of Ref and GetAttr supply the required additional dependency information for CF to construct a valid total ordering on deployment sequencing.

But here, none of the usual mechanisms applies, and the construction order information can get lost, leading to race conditions. Hopefully this will get cleaned up in a future release and the DependsOns can be removed.

Logging

Here’s the requisite Medium blog post sentence imploring you to turn on logging: Please…turn…on…logging.

The syntax for access logging is finicky, but you can see a working example here, and it’s worth it, especially for websockets. When a request fails to get through, it’s really useful to be able to determine how it failed — authorization, model validation, integration, etc.

The downside to having so much done for you is that you really need the added transparency when it all goes to hell. Unlike Lambda, you won’t get logging for your websocket unless you ask for it.

You also need to make sure you’ve set up your account to enable API Gateway with a logging role — that only needs to be done once per account, versus in each application’s deployment template, but without it you’ll get nothing when you go to CloudWatch.

One other minor point: Items like the access logging format here really benefit from being able to split a literal string in your template over multiple lines. You can either copy this verbatim from a known working example or get comfortable with the nitty gritty of YAML lexical rules.

Custom Domain Handling

If you’re content with the API Gateway-provided moniker for your websocket (which is probably fine if it’s an internal service), skip this section. But if you expect customers or partners to deal with it, you probably want some kind of vanity URL that’s a little easier to remember than a randomly generated GUID.

Custom domains bring together six pieces of information: The domain and subdomain, the API (in our case, a websocket), the path, the cert, and the deployment stage.

Note that the domain (and optional subdomain) show up as a literal string when you create the AWS::ApiGatewayV2::DomainName, but AWS::ApiGatewayV2::ApiMapping refers to the object…even though both call this property “DomainName”.

Domain name certification and DNS testing is a joy all its own; suffice it to say you want that fully working and debugged before attempting to use a custom domain in a CF template.

Last, but not least, you need bidirectional permissions between the Lambda function serving as the integration and the websocket attempting to integrate with it.

There are two sides to this:

Side #1: API Gateway calling Lambda
This is the role (no pun intended) of theAWS::Lambda::Permission object; it adds a resource policy that allows the NATPunch websocket to call the Lambda function serving as the integration target.

To keep it from being more verbose, we use a “*” for the route name. That’s very different than using a “*” for the entire thing — we want calls restricted to this specific API, but we know all routes (current and future) will go to this Lambda function.

By the way, if you want to list these ARNs explicitly, not that you can’t GetAttr them (AWS folks, note this bug!) so you’ll have to construct them manually, and there is an actual, unavoidable asterisk even in the full ARN. Seems like a questionable idea for something that has to wind up in an ASPEN statement, but there it is.

Side #2: Lambda calling API Gateway
Synchronous websocket responses don’t require any additional permissions; the Lambda function’s response is automatically turned into a websocket message by API Gateway, which is authorized by (1).

But an asynchronous message sent back to a client requires a Lambda function to use the callback endpoint provided by the websocket for this purpose. Remember the !Sub ‘arn:${AWS::Partition}:iam::aws:policy/AmazonAPIGatewayInvokeFullAccess’ in the Lambda function’s role? That’s where this magic happens.

CloudFormation Outputs

The outputs section is super useful for two reasons: Diagnostics and downstream activities. You may need these if subsequent steps in a CodePipeline need to know the names of things that got created…especially the names that get mangled by CF, which you can’t otherwise capture in a build script since you won’t be able to predict them in advance.

Parting Advice

And that brings us to nearly the end of our story.

Two parting bits of advice:

1. First, use the CloudFormation object name to search for documentation (so, “AWS::ApiGatewayV2::Stage”, not “API Gateway stage”) to avoid searching random pages of unrelated docs.
2. Second, it can be easier to create a working application (or at least pieces of it) outside of CF, and then “transcribe” that app to CF. This lets you use the AWS console UI, AWS CLI, or any of the several AWS SDKs to see how AWS thinks your app is configured. Often, one or more of those will display enough to help you correctly guess how to configure your CF properties to recreate the same behavior.

If you made it all the way to the end, congrats, and happy CloudFormation construction!

15 Hours with AWS CloudFormation was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

In residential construction, a mullet architecture is a house with a traditional front but with a radically different — often much more modern — backside where it faces the private yard.

Like the mullet haircut after which the architecture is named, it’s conventional business in the front — but a creative party in the back.

I find the mullet architecture metaphor useful in describing software designs that have a similar dichotomy. Amazon API Gateway launched support for serverless web sockets at the end of 2018, and using them with AWS Lambda functions is a great example of a software mullet architecture.

In this case, the “front yard” is a classic websocket — a long-lived, duplex TCP/IP socket between two systems established via HTTP.

Classic uses for websockets include enabling mobile devices and web browsers to communicate with backend systems and services in real time, and to enable those services to notify clients proactively — without requiring the CPU and network overhead of repeated polling by the client.

In the classic approach, the “server side” of the websocket is indeed a conventional server, such as an EC2 instance in the AWS cloud.

The serverless version of this websockets looks and works the same on the front — to the mobile device or web browser, nothing changes. But the “party in the back” of the mullet is no longer a server — now it’s a Lambda function.

To make this work, API Gateway both hosts the websocket protocol (just as it hosts the HTTP protocol for a REST API) and performs the data framing and dispatch. In a REST API call, the relationship between the call to the API and API Gateway’s call to Lambda (or other backend services) is synchronous and one-to-one.

Both of these assumptions get relaxed in a web socket, which offers independent, asynchronous communication in both directions. API Gateway handles this “impedance mismatch” — providing the long-lived endpoint to the websocket for its client, while handling Lambda invocations (and response callbacks — more on those later) on the backend.

Here’s a conceptual diagram of the relationships with its communication patterns:

When is a serverless mullet a good idea?

When (and why) is a serverless mullet architecture helpful? One simple answer: Anywhere you use a websocket today, you can now consider replacing it with a serverless backend.

Amazon’s documentation uses a chat relay server between mobile and/or web clients to illustrate one possible case where a serverless approach can replace a scenario that historically could only be accomplished with servers.

However, there are also interesting “server-to-server” (if you’ll forgive the expression) applications of this architectural pattern beyond long-lived client connections. I recently found myself needing to build a NAT puncher rendezvous service — essentially a simplified version of a STUN server.

You can read more about NAT punching here, but for the purposes of our discussion here, what matters is that I had the following requirements:

1. I needed a small amount of configuration information from each of two different Lambda functions. Let’s call this info a “pairing key” — it can be represented by a short string. For discussion purposes, we’ll refer to the two callers as “left” and “right”. Note that the service is multi-tenanted, so there are potentially a lot of left/right pairs constantly coming and going, each using different pairing keys.
2. I also needed a small amount of metadata that I can get from API Gateway about the connection itself (basically the source IP as it appears to API Gateway, after any NATting has taken place).
3. I have to exchange the data from (2) between clients who provide the same pairing key in (1); that is, left gets right’s metadata and right gets left’s metadata. There’s a lightweight barrier synchronization here: (3) can’t happen until both left and right have shown up…but once they have shown up, the service has to perform (3) as quickly as possible.

The final requirement above is the reason a simple REST API backed by Lambda isn’t a great solution: It would require the first arriver to sit in a busy loop, continuously polling the database (Amazon DynamoDB in my case) waiting for the other side to show up.

Repeatedly querying DynamoDB would drive up costs and we’d be subject to maximum integration duration of an API call of 30 seconds. Using DynamoDB change streams doesn’t work here, either, as the Lambda they would invoke can’t “talk” to the Lambda instance created by invoking the API. It’s also tricky to use Step Functions — “left” and “right” are symmetric peers here, so neither one knows who should kick off a workflow.

Enter…The Mullet

So what can we do that’s better? Well, left and right aren’t mobile or web clients, they’re Lambdas — but they have a very “websockety” problem. They need to coordinate some data and event timing through an intermediary that can “see” both conversations and they benefit from a communication channel that can implicitly convey the state of the barrier synchronization required.

The protocol is simple and looks like this (shown with left as the first arrival):

Here we take full advantage of the mullet architecture:

• Clients arrive (and communicate) asynchronously with respect to one another, but we can also track the progression of the workflow and coordinate them from the “server” — here, a Lambda/Dynamo combo — that tracks the state of each pairing.
• API Gateway does most of the heavy lifting, including detecting the data frames in the websocket communication and turning them into Lambda invocations.
• API Gateway model validation verifies the syntax of incoming messages, so the Lambda code can assume they’re well formed, making the code even simpler.

The architecture is essentially the equivalent of a classic serverless “CRUD over API Gateway / Lambda / Dynamo” but with the added benefits of asynchronous, bidirectional communication and lightweight cross-call coordination.

One important piece of the puzzle is the async callback pathway. There’s an inherent communication asymmetry when we hook up a websocket to a Lambda.

Messages that flow from client to Lambda are easy to model — API Gateway turns them into the arguments to a Lambda invocation. If that Lambda wants to synchronously respond, that’s also easy — API Gateway turns its result into a websocket message and sends it back to the client after the Lambda completes.

But what about our barrier synchronization? In the sequence chart above, it has to happen asynchronously with respect to left’s conversation. To handle this, API Gateway creates a special HTTPS endpoint for each websocket. Calls to this URL get turned into websocket messages that are sent (asynchronously) back to the client.

In our example, the Lambda handling the conversation with right uses this special endpoint to unblock left when the pairing is complete. This represents more “expressive power” than normally exists when a client invokes a Lambda function.

Serverless Benefits

The serverless mullet architecture offers all the usual serverless advantages. In contrast to a serverful approach, such as running a (fleet of) STUN server(s), there are no EC2 instances to deploy, scale, log, manage, or monitor and fault tolerance and scalability comes built in.

Also unlike a server-based approach that would need a front end fleet to handle websocket communication, the code required to implement this approach is tiny — only a few hundred lines, most of which is boilerplate exception handling and error checking. Even the JSON syntax checking of the messages is handled automatically.

One caveat to this “all in on managed services” approach is that the configuration has a complexity of its own — unsurprisingly, as we’re asking services like API Gateway, Lambda, and Dynamo to do a lot of the heavy lifting for us.

For this project, my AWS CloudFormation template is over 500 lines (including comments), while the code, including all its error checking, is only 383 lines. Asingle data point, but illustrative of the fact that “configurating” the managed services to handle things like data frame syntax checking by exhibiting an embedded JSON Schema makes for some non trivial CloudFormation.

However, a little extra complexity in the config is well worth it to gain the operational operational benefits of letting AWS maintain and scale all that functionality!

Mullets all Around

Serverless continues to expand its “addressable market” as new capabilities and services join the party. Fully managed websockets backed by Lambda is a great step forward, but it’s far from the only example of mullet architectures.

Amazon AppSync, a managed GraphQL service, is another example. It offers a blend of synchronous and asynchronous JSON-based communication channels — and when backed by a Lambda instead of a SQL or NoSQL database, it offers another fantastic mullet architecture that makes it easy to build powerful capabilities with built-in query capabilities, all without the need for servers.

AWS and other cloud vendors continue to look for ways to make development easier, and hooking up serverless capabilities to conventional developer experiences continues to be a rich area for new innovation.

Business in the front, party in the back …

bring on the mullets!

Serverless Mullet Architectures was originally published in A Cloud Guru on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the interesting opportunities I’ve enjoyed in my career is getting a ringside seat to innovative technologies that have changed the world. Being at companies that have a huge impact on their industry, including Microsoft, Amazon, and now Coinbase, means I get to experience and work on disruptive technologies every day. One particular aspect of that is experiencing the Hype Cycle up close and personal multiple times. At Amazon I got to see the entire lifecycle of a service — from writing the initial business plan for AWS Lambda to launching it at re:Invent to ultimately building it into one of the largest enterprise-grade cloud businesses on the planet. As VP of Engineering at Coinbase, I’m struck by some of the parallels between Serverless computing and crypto as they go through their respective lifecycles.

How is innovation perceived over time? How do you gauge the success of ideas that have large potential for change before they’re fully adopted?

Let’s look back at Lambda and Serverless computing as a way to better understand where Crypto has been, where it’s headed, and how (and when) it’s being adopted.

Lambda’s Innovation Trigger and the Peak of Inflated Expectations

Every significant new technology almost inevitably suffers from being overhyped; in early days, nascent implementations, immature tooling, and emerging processes haven’t yet caught up to customer and developer needs. Let’s look back at Lambda’s initial launch in November 2014. At that time it only allowed a single minute of execution time, 1 GB of memory, operated in a single region, had no ability to connect to customer Virtual Private Clouds (VPCs), and didn’t even allow for synchronous (direct) invocations. It had two event sources, Amazon S3 and Amazon Kinesis, and a (very) rough “double beta” of Amazon DynamoDB change stream events.

Meanwhile, the hype was incredible — the press talked about how this new computing model could replace existing computation and applications. Customers’ excitement was unbounded as they imagined a world without servers, with the promise of infinite cloud capacity, and with perfect utilization. The initial launch presentation was standing room only, as was the repeat session added the following day. It seemed that cloud nirvana had arrived. Back in Seattle, the team toasted our launch, confident in a bright future filled with customer success and a new, growing business for Amazon.

Lambda’s Trough of Disillusionment

In the ensuing months, reality set in: Running time and memory limitations, the lack of access to SQL databases (virtually all of which sat behind VPCs), the limited number of event sources, and other limitations became more apparent. Articles, posts, and tweets stopped talking about hope and excitement and began lamenting the then-current problems and limitations. Initially limited capacity in new regions and throttles on calls due to limits angered some customers. “Cold starts” and other friction from those early days caused some early adopters to declare the entire approach a failure. By the time the term “serverless” entered the popular lexicon, it was already being described as DOA by some of the very people who had initially raved about it. On the business side, adoption was growing, but volumes were modest in the first few months, and on the team we began to doubt ourselves — had we really done something unique and exciting, or was it an expensive flop?

Hockey Stick Growth and the Slope of Enlightenment

Meanwhile, the Lambda team was working hard to remove limitations and deliver features: synchronous invocations, dozens of new event sources, larger memory, longer running times, new regions…the service quickly grew into the enterprise-grade offering that customers needed to really get value from it. And they did: In a little more than a year after launch, enterprise users became a significant customer segment. Customers who had dismissed the idea came back to reevaluate. Revenue, volume, and adoption were all growing rapidly, just like we’d been waiting for.

It wasn’t just the technology itself that needed to grow up — meeting customer expectations as a full-fledged solution required an ecosystem. For Lambda, that looked like AWS Solutions Architects who could train customers on best practices, new conferences being established, and lots of examples to help explain this new way of writing applications.

Growing into its shoes took time, in part because new technologies, like Firecracker, which enables multiple hypervisors on a single EC2 instance, had to be developed in order to break through some of the limitations. This “second wave” of R&D was actually more involved than what we originally launched, in part because along the way Lambda had grown so much, so every change had to keep the “production promise” for hundreds of thousands of customers.

Plateau of Enlightenment

Meanwhile, the ecosystem was growing up, too. Application modeling and deployment tools were created, Lambda achieved compliance with major assurance programs like SOC and PCI, and formal SLAs were introduced. Third party tools and services emerged for everything from security to monitoring to support. Today, Lambda performs trillions of executions per month and has hundreds of thousands of active customers. It’s found a place in the technology portfolios of many Fortune 500 companies, with 60% of IT departments already using it or planning to deploy, and is considered an essential part of best practice cloud architectures. Reality caught up with that exciting initial hype — it just took time!

Crypto’s Hype Cycle

If you’ve been watching the crypto space, this all sounds rather familiar. Lots of initial hype (‘fiat money will be replaced!’) followed by painful reality. Falling prices in 2018 for popular assets like Bitcoin were roundly equated with crypto’s failure as a concept; a moment commonly referred to as “crypto winter” sent a palpable chill through the broader ecosystem. Meanwhile, legitimate limitations of the early technical expressions of crypto and blockchains were (and still are) mistaken for their eventual outcomes. Comparisons of Bitcoin’s 4.6 transactions per second versus Visa’s average of 1,700 called into question the idea of using blockchain as a payment processing platform. Scalability, latency, and throughput limitations, proof-of-work’s high economic costs, and the challenges some companies experienced as they grappled unsuccessfully with the realities of handling customer funds all diminished the initial luster of the approach.

But as with Serverless, the underlying technologies (along with their ecosystems of tools and processes) are growing up. Just as the hype cycle might be ready to declare crypto dead, fundamental innovation is broader than ever, and growing rapidly on every front:

• Scalability and throughput are being addressed through classic optimizations, like sharding, as well as new, crypto-native techniques including batching, STARKS, and Layer 2 protocols. Proof-of-stake and related governance mechanisms provide a complementary model to proof-of-work, offering lower latency and aggregate power consumption.
• Smart contracts and built-in application platforms are raising the level of abstraction to create a “distributed cloud” that reimagines application architecture and economics.
• At the same time technology and performance fundamentals are being optimized, innovative founders are exploring expansions of the business model. Including how real estate, art, and other assets and activities can be expressed in a crypto-centric way.
• And true to crypto’s origins, conventional finance is being reimagined and reinvented. Both the technology, and the businesses bringing it to market, are growing into their (very big) shoes at last.

You Are Here

Crypto’s “Hype Cycle”

What determines the length of time it takes to move through the hype curve? The larger the impact of a change, the broader its market reach, and the more disruptive it is versus what came before, the longer it takes to reach the final stages, where the technology reaches its full maturity in both implementation and adoption. With Lambda, that took about three years from the initial launch. Lambda is now firmly in its “productivity plateau,” having been adopted by a third of all enterprises, with serverless now representing the fastest growing cloud computing technology.

Crypto as a technology, meanwhile, has the potential to change how everyone on the planet experiences financial services — its reach is broader than software development and IT efficiency, and its economic impact runs deeper than the cloud. As a result, there’s still more time to go before we reach the stage where crypto is as ubiquitous as fiat currencies and as easy to program as Serverless. It hasn’t reached its productivity plateau yet, but we’re close enough now to see what some of those improvements could look like across the entire cryptoeconomy. (Note, these are innovations we look to support both via Coinbase and our investment arm, Coinbase Ventures):

• Scaling to aggregate global payment transactions per second so that every individual and commercial transaction can be captured in a blockchain.
• Per-transaction latency (through L2 solutions and other mechanisms) in low-digit milliseconds, regardless of geography.
• Compute and storage costs that reflect the underlying costs of cloud infrastructure and which are independent of the distribution scale of a blockchain’s network.
• Tools, business processes, user interfaces, and vibrant third party ecosystems for both developers and finance workers that rival those of cloud and mobile solutions today.

Blockchains and cryptocurrencies today are where the public cloud was 10 years ago, but they’re on an amazingly fast ramp, with core R&D, customer-centric engineering, and ecosystem enablement all happening simultaneously. The innovative implementation work going on in the many open source communities and at companies like Coinbase to create new capabilities, remove restrictions, and bring compelling technologies to market is one of the most amazing things I’ve seen in my career. Hype curve, meet reality!

PS. We’re always looking for talented folks to share this amazing journey with us, so if this sounds like an adventure you want to be part of, check out our open roles to learn more!

All links to third-party websites are for convenience and informational purposes only (“Third-Party Sites”). Coinbase is not responsible for the content of any Third-Party Site, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase.

Unless otherwise noted, all images provided herein are by Coinbase.

What the technology hype cycle tells us about the future of crypto was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.