The genteel alleys of Jackson Square north of San Francisco’s Transamerica Pyramid are full of boutique venture capital firms, design studios, and the occasional Michelin-starred restaurant. As one of the few neighborhoods not obliterated by the 1906 earthquake, it also has a direct connection to the city’s Barbary Coast origins. If you take a historical walking tour, you’ll hear about the gold and silver rushes, financial panics, political corruption, violent gangs of Australian convicts, and an eccentric named Joshua Norton who proclaimed himself Emperor of the United States and Protector of Mexico.

The most recent San Francisco boom cycle involves generative artificial intelligence, not precious metals in the Sierras. The peculiar mix of local culture, technology, and money is the same. Developers are writing code on the same street where Emperor Norton wrote a proclamation with the then-absurd idea of a bridge across the Bay. He was the original mascot, progressive, oddball, and minor celebrity who helped generations of locals reflect on what kind of a place San Francisco was or could be.

Last year, I decided to use generative AI to recreate Norton’s voice and have it engage with people on social media. I was interested in how an AI model of Norton, which I’ll refer to as the Emperor Bot, could make us more creative. Much of what is written about AI seems to be about civilization-ending scenarios or startup pitches for new ways to summarize your emails. I wanted to know if generative AI could help people reflect on the past… and possibly laugh.

Building an Emperor Bot that challenges injustice

The first thing I had to do was build the bot, which was the most fun I’ve had on a technical project in a long time. How it works is straightforward, even for a non-programmer: you instruct a large language model (LLM), like OpenAI’s GPT-4, to reply as Emperor Norton. This instruction is called the prompt.

Some facts about Norton have been incorporated into the base LLM models. The result is cutting-edge technology approximating how to speak in the voice of someone from the 19th century. After I made dozens of small adjustments to the prompt — it was difficult to get the replies concise — here’s the bot’s reply to a tweet about seagulls and a locally famous Taco Bell:

Emperor Norton I has decreed that all seagulls must be removed from Pacifica & its environs. The Board of Supervisors must award damages to the owners of the desecrated Cathedral of Taco Bell.

The reply captures three things that are essential to the historical Norton: it challenges injustice (albeit of a Taco Bell), holds politicians accountable, and does so with the authority of someone with absolute political power.

Here’s another reply to a tweet from a local supervisor proposing renovation of some crumbling piers that doesn’t hit the mark:

We decree that Piers 30–32 in San Francisco shall be rebuilt as a mixed-use development with public recreation, retail, office, and housing!

The pro-development tweet was it seemed inauthentic in the worst possible way. It didn’t capture the historical voice. It was like seeing an ad for breakfast cereal featuring an Emperor Norton proclamation on whole grains. I wanted to understand if this was truly a limitation of the current models, or something that could be improved.

Finding Norton’s authentic voice in the LLM

One of the issues with recreating Norton’s voice is that, while everything he wrote is in the public domain, there is a thin historical record for someone who continues to be famous in the modern era. The proclamations he published were short, and no journal or diary survived. With the help of local historians I became more familiar with gold rush era San Francisco. John Lumea of the Emperor Norton Trust, the world’s foremost expert on Norton, sent me several helpful emails, and Taryn Edwards of the Mechanics’ Institute (where the real Norton was a member) shared a reading list with a warning: San Francisco history is rife with lore and legend.

I can never know Norton’s views on the correct amount to tip at restaurants, but did know he was always against corruption and wanted to keep politicians accountable… to the point of calling for Congress to be abolished. Using more AI, I generated a map of the themes and topics Norton wrote about and hoped it would guide better responses.

For example, Norton had a proclamation that referred to sailing ships — but his writing on the ships thematically touched on justice, authority, responsibility and safety. When he wrote about being refused entrance to an event by “snobs in charge of the door” — the map pointed to his concerns around the abuse of power and social hierarchies.

Instead of parroting or sounding like a shill, the bot’s replies seemed to improve with the new context. Here’s the reply to the news of a local brewery closing:

Great Scott! Anchor Brewing, a city gem, faces peril? We’ll rally, oh we will! Workers, I stand in solidarity with you! Citizens, let us preserve our city’s heritage!

It was around this time a controversial local billionaire changed policies and most bots, including the Emperor Bot, were kicked off Twitter.

BlueSky Social, hackerspaces and vegetables

Elon Musk kicked the bots off Twitter because he wanted money for the type of access the bots needed to post, which was previously free. This resulted in the Emperor Bot moving to the newer social network BlueSky.

Despite being a friendlier and invite-only version of Twitter at the time, I was concerned what might happen. Despite profile disclosures and a prominent robot icon, my impression is that most people assumed the text was written by a cosplaying San Francisco history nerd pretending to be a digital version of Norton.

Non-public tests with the bot commenting on trending topics went badly. The bot never generated outright offensive content, but anything related to politics was inflammatory. Even positive replies, like support for the Hollywood writers’ strike, were problematic. People striking for reasons related to losing their livelihoods over AI-generated content do not want a supportive yet eccentric 19th century bot to join the conversation. The fully-automated social media version was never released, so I reviewed replies and posted the unaltered text every few days.

The most engaging conversations with the bot were in private channels on Discord, the popular chat app. A local hackerspace added it to their AI channel and started interacting with it. Using the latest tricks being spread around the LLM space, bot replies seemed to improve when I added an “therapy” mode where the bot would articulate its sensations, fears and desires before replying.

Here’s the bot’s inner monologue for replying to something related political corruption:

Sensations: A harsh, sneering laughter rings in my ears, leaving an unpleasant taste in my mouth.

Fears: Anarchy and corruption taking over my beloved city.

Desires: To enact justice and maintain the order of my realm.

Somebody also asked the bot what vegetable it found the most threatening.

I would say it is the artichoke. Its sharp exterior might be seen as a symbol of uprising, but in truth, it’s the tastiest rebellion I’ve ever faced.

A folk hero emerges on BlueSky Social

The bot was finding a certain groove on Discord, but on BlueSky it was limited to replys related to a very small number of topics: mostly pets and Norton himself. I was surprised how culturally relevant Norton, who died in 1880, is to modern-day San Francisco and how deep that appreciation and fandom goes. An award-winning author was one of the first followers.

My impression is that Norton is a folk hero on social media or, depending on who you ask, the world’s first crypto bro. (He printed his own money and passed it off as legal tender.) People imply they would love to vote for him in the current election. He’s also considered a minor saint in a belief system that started in 1960s Berkeley called Discordianism (no relation to the app Discord).

The most liked posts from the bot, after some trial and error, were those that acted as a kind of promoter for San Francisco and the historical Norton. Anything else tended to annoy or confuse people, like John Adams showing up on Instagram and liking your vacation photo. When someone skeeted (that’s BlueSky for tweet) that Norton was cool and they wished he could save everyone now, it replied:

Indeed, time unceasingly marches forward. While I can’t step forth in flesh, my philosophies persist. Lead with benevolence, act with authenticity.

In another instance, the bot replied to a post memorializing Norton with a familiar sounding phrase: “death is but a door, time a window.” It’s a quote from the villain in the 1989 movie Ghostbusters II… a 16th century historical figure that comes back to life in an evil painting.

What are the chances of the Emperor Bot taking over the world?

The 1906 earthquake destroyed much of Norton’s San Francisco. The murkiness of the late 19th century also means that it’s easier for every generation to repurpose Norton to fit their mood and anxieties. In present day San Francisco and the United States: there are plenty of issues to choose from.

Of course, one of the largest local conversations in technology is the promise and perils of generative AI itself.

It’s not hard to imagine these sorts of bots finding a place in educational settings or museums to make history more interactive and conversational, at least for well-liked and popular historical figures. The bot would be an excellent companion in a self-guided walking tour of the Emperor’s old neighborhood.

There’s also the obvious danger of these bots on social media, which has been written about at length in many different places. My assessment is the risk of the Emperor Bot causing global conflict or taking over governments as a digital Emperor is effectively zero. However, it’s hard to think of any scenario where more advanced gen AI bots will improve the overall quality of conversation in the near-term on social media.

There has been an enormous amount of investment from major companies to make sure the output of LLMs is inoffensive and not harmful, but that is changing as individuals and groups create their own models. Using a collection of scanned 19th century newspapers, I started on an experiment with fine-tuning the bot to make it sound more like actual writers from the 1880s. That was paused once I realized that I did not want my bot to pick up on the views that existed in the vast majority of late 19th-century American newspapers.

Norton, generative AI, and a way forward

Joshua Norton is buried in Colma, the cemetery city just south of San Francisco. There’s a marble headstone that says Norton I Emperor in large letters. E Clampus Vitus, a 19th-century fraternal organization (sometimes called in the press a “historical drinking society”) throws a party honoring his legacy once a year on “Norton Day” at his gravesite. There’s a convenient Irish bar a short stroll away.

My theory on why Norton still has a large number of admirers who toast him is because, in his own chaotic and eccentric way, he is remembered for fighting against injustice. He also was an immigrant to San Francisco who made and lost a fortune but still figured out a way to get free lunches and had a popular newspaper column for twenty years.

Generative AI will never be able to capture the full context, intricacies, and contradictions of the human Norton but it can be strange and unexpected. Despite the peril of thousands of trolling bots taking over social media, in the right context this technology can be fun and creative — without taking over the world or your job.

Uncertainty hangs over the air right now for both AI and San Francisco. There are plenty of doom loop scenarios being discussed for both. Economically, the success or failure of generative AI will also have a direct impact on The City for years to come. The open question for local entrepreneurs depositing checks from venture capitalists is what happens next. At one extreme: collapse from malevolent AI. Another direction is the mediocrity of bots that make life worse via annoying assistants that get in the way.

A happy outcome would be technology — and a city — where people are able to be thoughtful, explore new ideas, and maybe have fun while holding those in power accountable. It might also get a little weird and off-putting at times. We should, in other words, take more inspiration from Emperor Norton.

Thanks to John Lumea of the Emperor Norton Trust, Tayrn Edwards of the San Francisco Mechanics Institute, the weekend staff at The San Francisco History Center, the Prelinger Library, The Ancient and Honorable Order of E Clampus Vitus, the Misalignment Museum, and various adherents of Discordianism active on BlueSky and Discord.

You can follow the Emperor Bot at https://bsky.app/profile/emperornorton.bsky.social

Rebooting San Francisco royalty: Emperor Norton meets GenAI was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I trained a bleeding-edge machine learning (ML) model with a dozen issues of WIRED magazine from the mid-1990s and things got weird. The new model started generating a unique blend of 90s-era techno-optimism, product advertisements for microprocessors and gaming systems that never existed, odd interviews with tech luminaries and… a profound love for all things Apple.

There’s a popular ML technique going around where people generate text using neural networks based on different source materials. The technology isn’t advanced enough to make us think that a sensible person actually wrote the generated text—a serious problem we’re going to have to deal with—but it’s often good enough to produce bizarre statements that are funny in context.

I first tried it around two years ago to generate an ICO whitepaper and there have been many other developers, data scientists, and researchers who have continued to explore. Janelle Shane’s posts on this subject are wonderful and inspired this work. In the past two years, the effectiveness of the models have significantly improved.

This post is about what happened when I wanted to train one of the newer models going around (OpenAI’s GPT-2) with mid-1990s WIRED magazine issues.

WIRED, Nostalgia and Machine Learning

I grew up reading WIRED magazine, and I’m not immune to the nostalgia for the period of the 1990s when the internet and personal computers were a new and amazing thing for many people in the United States.

Generating text from the model is surprisingly addictive—with a click of a button, it’s like reading something a WIRED journalist might write after not sleeping for three days.

There is also a strange blend of marketing and journalism is impacted by the training data—WIRED issues in the 90s, after all, had plenty of print ads. As a result you get serious-sounding statements about history mixed with product marketing speak:

Sometimes this veers off into editorialization. After a quote around how you can get the latest PowerBook “when it’s available” (which every tech company would write in announcements if they could get away with it), this paragraph ends with something that sounds like a melody:

When it comes to technology luminaries of the mid-1990s, even seemingly critical quotes of Jobs end up making him look good:

Bill Gates, in contrast, basically has to plead that Microsoft isn’t actually in trouble when seeking an ambitious target of “download hits”:

The machine learning hype is real?

This started as a quick AI experiment to see how text generation models were advancing. From start to finish (including training time), it took only about two hours before text started coming out, including some text that wouldn’t look out of place in an experimental poetry journal.

While it was fun and fascinating to see what was generated, it also doesn’t seem unrealistic that more advanced versions of the model would probably be good enough to generate entirely convincing WIRED articles that seemed like they were written in the 90s. That’s disturbing for anyone who reads digitized historical sources of anything on the internet.

The creator of the GPT-2 model, OpenAI, has committed to working with research and governmental institutions on the societal impact of widespread use of models like this, including an advanced version of GPT-2 that hasn’t been publicly released yet.

As a creative tool, however, my experiments with WIRED and neural networks were unexpectedly inspiring. Strange alternate histories, quotes, bizarre products, and koan-like statements around the future emerged.

My deep concern for the future of these language models is combined with a desire to read the first few chapters of a book featuring Larry Ellison, his wife FmmL, and Denny Dallas. Maybe they’re all Apple fans.

Thanks for reading! Might get around to turning this model into some kind of chat bot in the next few weeks. Find me on twitter: @smithclay.

My neural net turned into an Apple zealot after training it on mid-1990s WIRED issues was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.

You can’t ping an AWS Lambda function. You can’t SSH into a running function without reverse tunneling. You would never run nginx or Rails inside of a function because there’s little point — accepting an inbound TCP connection on a port like a Docker container isn’t supported by function-as-a-service (FaaS) platforms. It is possible, however, to write your own userspace networking stack and do what you want with it —including using Amazon Cloudwatch Logs (or even AWS Lambda tags) as the data link layer.

This post is about implementing TCP/IP over Amazon Cloudwatch Logs using Go, which enables you to access network services running inside of AWS Lambda functions.

It’s slow and not very useful, but it was a fun way to learn more about Linux networking and using AWS services in a way that might horrify some AWS engineering teams. I’m calling this proof-of-concept “Richard Linklayer” and published the full Github code today.

A Serverless Architecture Darkly

This experiment was the escalation of:

• Trying to run progressively stranger things in AWS Lambda.
• Interest in exploring netstack, an IPv4/IPv6 userland networking stack from Google.
• Hearing contradictory and confusing things from some clients and engineering teams around different limitations of serverless functions.

I attempted to check off all three of those things using Go. With netstack, it’s possible to do some exotic things with networking in unexpected places, no kernel hacking (i.e. writing C code) required. Any Go program with read and write access to reliable-ish bi-directional communication channel — like logs, tags or carrier pigeon — can implement full in-process TCP/IP networking. There are many AWS services that meet this criteria.

Network tunneling is not a new idea, but inside the AWS Lambda execution environment a userspace solution is needed because it’s not possible to change routing tables, modify network interfaces, or change OS-level networking (unlike container-based network overlays or service meshes).

Running in a standard Go process, Richard Linklayer tunnels IP packets over Amazon Cloudwatch Log Streams that follow a special naming convention — the stream and log group names are just MAC addresses. Using a tun or tap interface, I can bridge my AWS Lambda network endpoints to my local development machine:

This design ignores normal AWS Lambda event-driven patterns for processing inbound packets and sending outbound packets —only a single instance of a function starts, reads and writes from Cloudwatch using polling, then processes packets until it times out. There is only one instance of a function running a unique network service running at any given time, and it can only accept network connections for up to 15 minutes, the maximum timeout of an AWS Lambda function in early 2019.

When you put it all together, the result is that you can ping a Lambda function.

The underlying implementation over Amazon Cloudwatch is at layer 2, so it’s reading and writing ethernet packets delivered to MAC addresses. MAC address discovery happens using ARP with a special-purpose log stream that acts as a “broadcast address” for the entire network.

Since it supports anything that runs over TCP, you can also cURL a node.js HTTP server running inside of a function, or anything else that listens on a TCP port. This server returns some text after a few seconds.

The latency means that this would likely never work for anything production-level, but TCP is resilient enough to handle the slow connection.

We have full transparency into the network traffic since it’s just data inside of log streams. It looks like this, which is the stream of IPv4 traffic from 74:74:74:74:74 to 42:42:42:42:42:

An example on how to get this working with a simple network running an HTTP server is on Github. While not described in this post, there is also an additional link layer that uses AWS Lambda function tags as a transport that has slightly better performance results.

Dazed and Lambfused

There’s a lot of interesting open-source networking projects in the container space right now: Envoy, Linkerd, and Cilium to name a few. With support for custom AWS Lambda runtimes released at re:Invent in 2018, I’m curious how emerging control or data planes are going to impact serverless functions beyond additional language support and monitoring and security solutions.

Beyond that, there’s a few interesting ideas to explore beyond this proof-of-concept, including:

• UDP services (DNS over AWS Lambda, anyone?)
• Any kind of performance optimizations (everything is based on polling APIs and could be tuned)
• Exploration of other link layer transports in AWS (DynamoDB streams? SNS? SQS? S3?)

Pull requests and ideas welcome.

TCP/IP over Amazon Cloudwatch Logs was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.

With ICOs, or token sales of cryptocurrencies, having reportedly raised $2.3 billion in 2017, the time seemed right to use AI to generate a fake ICO white paper programmatically using a recurrent neural network (RNN) in the cloud.

I’ve titled it “RNN-Coin” and have published the full white paper here (pdf).

RNNs are a machine learning technique that have frequently and hilariously been used to generate arbitrary text from text-based training data. Some of my favorite examples are generating new British town names or a Game of Thrones book. However, unlike most town names or the work of George R.R. Martin, there are plenty of people that claim that ICO whitepapers are mostly incomprehensible anyway. This post shares some the technical background and some of the excerpts from the paper (it’s painful to read in full).

Building the model

Using Tensorflow, Google Cloud ML engine, and a modified RNN example I came across on Github from Maximilian Clarke, I created a model using a GPU-powered cloud training job that ran for a few hours (my MacBook Pro was too slow and RNNs are resource-intensive to train). The trained model could then output text that eerily echoed many ICO white papers.

For the published white paper, I generated around 2,000 characters at a time from 40-character sentence seeds I pulled from real white papers (like the Bitcoin white paper). I then manually organized the output into sections — formatted in LaTeX for full white paper legitimacy, of course.

One of the first outputs became the basis of the white paper abstract:

A purely peer-to-peer version of the company system with the price of the system to develop and the contract agent servers and the price of the system of the system which is a simple decentralized system which is the contract to provide the contract […] can be considered as a result of the blockchain.

The algorithm created peer-to-peer, blockchain-backed alternative to corporate structure! It’s just a simple decentralized system. The price is still developing, however.

The local revolution of experience is here!

RNNs have been described by some people as “magical” or “uncanny” and after going through this exercise I agree. There are points in the white paper that sound perfectly reasonable — and even decent marketing. What millennial, including myself, wouldn’t want to be part of “the local revolution of experience”?

It often generates language constructions that don’t sound natural. This could probably be tuned further by trying cleaner training data and different model parameters. Yet strange-sounding sentences also have gems:

This is the highest commitment wallet that a confident of money for medical currencies of the project in the initiative instruments of the project is the farmer of the cost of the system.

A “highest commitment wallet” might be a reassuring concept for an ICO investor in medical currencies.

“A pledge to the smart token” (and having fun with ML)

The first social advance is a problem for control of the platform.

I don’t see RNNs replacing real white paper writers or creating ICOs, but the experience of building and training fairly complex machine learning models that were previously the exclusive domain of academics and researchers does seem to be getting much easier for the average developer.

Much of the code for sample tensorflow projects using different ML techniques is freely available on Github and it seems cloud providers are now racing to rent GPUs to developers for model training purposes. I think as more tensorflow projects get shared the potential for collaboration also goes up: if a real AI/ML researcher comes across this post, pull requests are welcome to refine the model.

There is much speculation and uncertainty around the future of ICOs, but it also seems like there has never been a better time to generate fake ICO white papers with AI.

The full paper is available for download here (pdf).

Using AI to generate an ICO white paper was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In middle school, I remember reading Robert Louis Stevenson’s The Bottle Imp, written in 1891. The story describes the the struggles of a person who buys a magic bottle from an elderly man who lives in a “fine house on a hill” in San Francisco. It grants unlimited wishes with certain evil conditions: you can’t sell it to anyone else for more than you paid for it and if you can’t find a willing buyer before you expire you’re doomed. You also have to pay in real minted coins: no paper money is allowed so there’s some physical limit to how cheaply you can sell it.

Remembering this story while reading about the blockchain, the obvious thing to do was attempt to implement it as an Ethereum smart contract on a sunny Sunday afternoon. I’ve published the code on Github here.

My attempt at sounding like I know about what I’m talking about about with blockchain

This was my first attempt and building a ÐApp (decentralized app). I feel like including this diagram of distributed networks from a 1962 academic paper gives this post some additional gravitas and is completely self-explanatory on the benefits of decentralized applications, the nature of distributed ledgers, and inherent security considerations of applications built using this technology:

Now that we have a full and complete understanding of Ethereum and distributed applications, how does the contract work?

As for why I sell so cheap, I must explain to you there is a peculiarity about the bottle.
 — The Bottle Imp

In the process of writing this smart contract, I was terrified. Outside of the usual pains of figuring out how to write, deploy, and build things using a completely new toolchain (the Solidity language, testrpc, and the web3 APIs), it was unsettling to read though a popular resource on potential security vulnerabilities from ConsenSys.

The contract I wrote (my first) does only one trivial thing by design: transfer ownership of the (virtual) bottle if they pay less than the previous owner.

However, from best practices I found it seems clear that production-grade smart contracts need to go through a lot of testing and analysis. Specifically, it’s a good idea to:

• Really understand the Solidity language and know the danger of external calls, transfer calls, and reentrancy vulnerabilities.
• Integrate rate-limiting and “pause” functionality if things go wrong.
• Use static analysis and linting to catch vulnerabilities.
• Explore different patterns to upgrade your contract as new vulnerabilities are discovered.
• Start a bug bounty program to thoroughly test it (with rewards to those who find issues) before the contract goes live on the main ethereum network. Maybe hire some consultants for auditing purposes, too (they were all over my Google search results for Solidity development questions).
• Consider using frameworks like truffle or OpenZepplin when you realize you’re probably doing it wrong.

Attempting to, uh, “implement” some of those best practices over a couple hours, I came up with this code for the key logic in the contract to transfer the bottle. If the price is less than the price the previous owner paid and it’s non transferred back to the same account— the bottle has a new owner and the price paid is transferred back the previous owner:

function buyBottle() payable public {
  // Conditions
  require(msg.sender != currentOwner);
  require(msg.value > 0);
  require(msg.value < lastPricePaid);

  // Effects
  previousOwner = currentOwner;
  currentOwner = msg.sender;
  lastPricePaid = msg.value;
  LogTransfer(previousOwner, currentOwner, lastPricePaid);

  // Interactions
  // *Security alert (correction on October 27, 2017):*
  // This implementation isn't safe! See GitHub and comments.
  previousOwner.transfer(msg.value);
}

The “conditions”, “effects” and “interactions” comments are only for reassurance based on a solidity question I read on StackOverflow. Otherwise, I’m conservatively ~40% sure it works without a major structural flaw or security vulnerability.

Like the original short story, there’s ultimately a loser in this contract: if the value paid is 1 wei — the smallest allowable unit in Ethereum, you’re stuck with ownership of this smart contract forever.

After some local testing and trying it out on the Rinkeby test network, decided it was time to ship it.

It’s live on the main Ethereum network, beware!

“This is the bottle,” said the man; and, when Keawe laughed, “You do not believe me?” he added. “Try, then, for yourself. See if you can break it.”
 — The Bottle Imp

The contract/virtual bottle has been published to the main Ethereum network and is able to be transferred to anyone who wants it— as long as you pay less than the previous account did. I’m curious to see how long it takes for the value to get down to 1 wei and what happens (if anything).

To kick things off, I’ve started the price of the contract at the going rate of a of a typical sandwich in the Financial District of San Francisco: 0.05 ETH (~16 USD).

If you’re new to Ethereum, I’d recommend downloading the Mist browser or Metamask extension and then visiting the following address: https://wallet.ethereum.org/account/0x9fcd683584173c5df0ee47852f5592f68ed72a0c

Feedback and vulnerability assessments welcome in the comments, on twitter, or on the Github project page.

If you’re new to all this can also recommend Coinbase’s general introduction to Ethereum (and how it’s different from Bitcoin). For getting started with development, the Reddit community was useful too.

Regardless of where the blockchain hype goes — and who ends up stuck with the smart contract I created when they buy inevitably buy it for 1 wei — it was really interesting to learn more about the process of building and testing smart contracts. Last word, as it does in The Bottle Imp, goes to the boatswain:

“I don’t value any of your talk,” replied the boatswain. “You thought I was a flat; now you see I’m not; and there’s an end. If you won’t have a swallow of the rum, I’ll have one myself. Here’s your health, and good-night to you!”

Writing an Ethereum smart contract from the 19th century was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Around a month ago, I was also prototyping a small AWS Lambda project that enabled running Selenium Webdriver with Headless Chrome inside a Lambda function called lambdium (disclaimer: no relation to laudanum).

I wanted to share the general architecture of the app and some things I learned while building it to help any other developers wanting to give serverless and Headless Chrome a try.

There’s also great post by Ken Soh for an overview of other projects experimenting with Headless Chrome and AWS Lambda. The comments in a recent Hacker News post about the Chromeless project are worth a read, too.

Binaries not included

In brief, AWS Lambda is just an event-driven Linux container on EC2 with some special cgroup sauce — see my Gluecon talk for more on that. Because of that, it’s perfectly acceptable to run almost any non-privileged process you want inside Lambda as you would in a container. You just invoke a 64-bit Linux binary that doesn’t require special privileges using something like Node.js’s child_process module or Python’s subprocess library in code in the main function handler.

Complex Linux binaries like chromium, though, need lots of shared libraries and assume a couple things about the Linux environment that don’t play perfectly with Lambda. Marco Lüthy came to the rescue in early March and figured out how exactly to get the chromium binary working. His serverless-chrome project is now the reference implementation for many projects, including chromeless and lambdium.

With the ability to run Headless Chrome in AWS Lambda, I was curious if I could get Selenium tests to run using ChromeDriver. Then, using one of the many Selenium client libraries, I could run pre-existing tests in different languages. After many iterations of copying shared libraries around — I detailed the general recipe for getting untested binaries working on Github — ChromeDriver was able to run and interact with Headless Chrome inside a Lambda function. With ChromeDriver running, I just had to connect a Selenium client library to it.

Finally: running Selenium tests

By going through the process above, I was able to get ChromeDriver and Headless Chrome running inside an AWS Lambda function — barely making it under the compressed 50mb size limit for functions (the Chrome binary is around 80% of this).

Surprisingly, running the Selenium tests using the Node.js selenium-webdriver module ended up being possible with just a few changes to the default options:

1. There are a bunch of Headless Chrome-specific command line options you need to specify.
2. You have to tell Selenium where to find the Headless Chrome binary (because it’s in AWS Lambda, it’s not in the PATH).

With that done, there was a mostly working Selenium Webdriver session with Headless Chrome running in AWS Lambda ready to go. The output of a script that visits Google.com and checks the title looks like this in AWS Cloudwatch(again, not much to see because it’s headless after all):

According to the AWS Lambda calculator, if I ran that same function 1,000 more times, my monthly bill (not including the free tier of the service) would come to $0.04. I increased the function memory to 1152mb—running Chrome takes a lot more than the default memory size in AWS Lambda.

Project setup and known issues

The prototype project, which initially tried to design as “framework-less” and as boing as possible (it’s just regular Node.js, a Makefile , and some shell-scripts), is up on Github. It shouldn’t be too hard to port to, say, Python, C# or Java (all AWS Lambda-supported languages). As of early January 2018, I migrated the project to use AWS SAM Local and in March 2018 I ported it the AWS Serverless Application Repository (one click install!).

I’m still working on a new branch that should improve startup times, but that’s a subject for another post.

Will Headless Chrome shake things up in the browser QA/test world?

The duo of Headless Chrome and AWS Lambda seems like it has a real possibility to shake up automated browser testing — due in no small part to how cost effective serverless is for low-volume traffic like occasional browser testing in QA. With the generous Lambda free tier, you could probably run low-volume browser tests on AWS infrastructure for free.

Also, there’s no reason why Headless Chrome couldn’t also work on other serverless compute platforms, and there seems to be plenty of interest running it in Docker as well. I give it 6 months, max, before someone writes some sort of Kubernetes cluster that runs Headless Chrome at scale.

Think it’s a pretty fun time to be hacking on serverless compute platforms or even containers in general — especially with the recent release of Headless Chrome and the cool things that are going to be built with it.

Pull requests and questions welcome. Also highly recommend you check out the other projects in Ken Soh’s post.

Running Selenium and Headless Chrome on AWS Lambda was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I spoke at dotScale in Paris last week about some stuff I’ve been working on to learn more about Function-as-a-Service (FaaS) performance.

While my talk was mostly about cold starts, function warming, and understanding the internal architecture of FaaS, lots of people were curious how I was able to SSH into a running AWS Lambda function. This post describes how to turn Lambda functions into a short-lived Linux servers with some help from Go’s SSH crypto libraries.

Adventures in missing dependencies

Many popular FaaS services are built on Linux containers—Azure seems to use IIS voodoo I don’t understand. From an serverless app developer perspective, you write a handler function in a supported language that’s bundled with its dependencies. When a predefined event like an HTTP request occurs, the handler function is invoked inside the container and your code runs (or has an error, or times out).

Because the environment is just a container, you’re allowed to execute binaries using standard language features like exec() in node.js or python subprocess. This is also how open-source projects like Apex let developers run Go in AWS Lambda.

With a 12-line lambda function, I tried running the sshd process on a non-privileged port with a host key and configuration I bundled with my the function. This was a bad idea for several reasons:

• AWS Lambda’s container environment is missing several libraries that the ssh daemon requires.
• AWS Lambda functions do not seem to allow any inbound port access (yes, even if running in a VPC with a security group that explicitly allows it).
• Exotic ssh configuration files are hard to write and filled with weird options.
• There’s no easy way to automatically determine the external IP address of a Lambda function.

It was time for some SSH tunneling and copy-and-paste Golang code from Github.

Writing an SSH server that creates a tunnel in Go

Go has some nice high-level SSH libraries for creating servers, clients, and tunnels. Unfortunately, my first copy-and-paste attempt had a dependency on an OS feature (unix pseudoterminals) that did not exist in the AWS Lambda environment. Digging around a bit more in the Go documentation, I found some libraries designed for interacting with terminals that didn’t have a dependency on the pty package.

Cobbling it all together, I had a simple go service that:

• Runs an SSH server capable of executing commands in bash on a non-privileged port.
• Creates a tunnel to a remote host (i.e. my laptop via ngrok or an EC2 host running inside my VPC) and requests a port to be opened that forwards to the SSH server running inside the Lambda function.

I then wrote a Lambda function that:

• Is configured with a 5-minute timeout (the maximum allowed value).
• Executes the Go SSH binary I built for 64-bit Linux (an under-appreciated feature of Go development on macs).
• Used Lambda environment variables to connect to a specific host and port number (i.e. my laptop via a jump proxy).

With this in place, using the tunnel I can access my go SSH server and run shell commands from another host using a standard SSH client until the Lambda function times out:

This isn’t a good idea (but it was fun to build)

I don’t think creating SSH tunnels and running random servers inside AWS Lambda is a particularly good way to take advantage of serverless, but it was fun to learn more about the execution environment by building it. Couple other possibilities I’m thinking of now that there’s a proof on concept:

• Running other types of servers in Lambda that allow inbound connections through tunneling.
• Integration with a key-management services for SSH authentication (KMS or Vault perhaps?)
• Support for concurrent sessions or integration with a TCP load balancer.

The code is up on Github (project name: faassh). Pull requests and questions welcome.

In the next few weeks, I’ll be at DevOpsDays Austin and Gluecon speaking more about FaaS if you want to chat in person.

Links

• “Searching for the Server in Serverless” talk (dotScale Paris 2017)
• “Serverless Measurement Limits” talk (DevOpsDays Austin)
• Measuring Cold Starts in AWS Lambda (New Relic Blog Post)
• faash source code

SSH-ing into your AWS Lambda Functions was originally published in clog on Medium, where people are continuing the conversation by highlighting and responding to this story.