This is a mildly edited transcript of my JSConf Hawaiʻi talk. Use this link to share this post and bypass Medium’s paywall.

Hey everyone! My name is Malte, I’m a Principal Software Engineer at Google, and today I want to talk about Designing Even Larger Applications. This is a sequel to a talk I gave at JSConf Australia two years ago. And just like last time, I want to kind of ground this talk in our career progression as software engineers. I think many of you in the audience would call yourself senior engineers; or, if you’re not there yet, you aspire to be one.

The way I would describe what it means to be a senior engineer is that if someone comes to me and they say something like “Hey, Malte, do this project in this domain that you are already familiar with”, I would say “Yeah, that is actually something I know how to do. I don’t need to get anyone else’s help”.

In my last talk I looked at how to go beyond that level of seniority in software engineering. It stops being only about yourself and instead your craft starts impacting other engineers. You’d say “I can anticipate how others do things and design APIs accordingly.”

In this talk I want to go yet one more level beyond that. I want to reach the stage where I can say, “I can design software such that for large groups of engineers the probability increases that the software that they produce is good.”

There are three key words in this statement.

• First of all, large groups of engineers. If you work at a startup or a three-person company, then this talk may seem superfluous and boring. But I think many of you may work for insurance companies, banks, agencies, big tech, etc.–companies big enough that you have a bunch of folks and multiple teams and you need to coordinate stuff.
• Next I’m talking about probability. There is no certainty here. You can only try to set things up so that they are likely to work. But there are no guarantees. There is no silver bullet.
• The third key word is good. This is not a product management talk. This is not a talk that will not help you write the right program, but it will help you write that program well. What I mean is it being maintainable, high-performance, having low bug-density, being delivered on-time, etc.

I want to clarify one more thing: I’m going to say the words framework and software infrastructure a lot. I treat them kind of interchangeably. What I mean is software that helps us build better software. For this talk, I would like you all to kind of put your mind into you being the person in your company who is responsible for defining how people write software and, then, build the infrastructure that they’re using to write software. And when I say, you build a framework, I don’t mean that you necessarily make your own React or Angular. Instead I think whenever you have a set of teams you want to kind of standardize how they build applications. So, you put a literal framework around it.

Again, think about you being the person who is responsible for your team’s software infrastructure. This is a talk about how to make you successful at that job. And I’m going to talk about that in three chapters.

1. The first one I call understanding the degree of uncertainty
2. …and then we’re going to learn how to solve all known problems.
3. Finally, we’re going to learn about deploying change.

Understanding the degree of uncertainty

How certain are we about the type of problems that we will need to solve in the future? This question is absolutely key for software engineering and I’m going to talk about a technique to understand how well we understand things.

What we’ll do is try to answer a set of questions:

• What is it in the existing infrastructure that users struggle with and how are they struggling with it?
• What is the class of applications that folks will build using your work in broad terms?
• What are the trends that influence the industry and how do those trends influence the software that folks are probably going to write in the future?

All of these may sound a bit like reading the tea leaves but I think that with a bit of experience there are many scenarios where you can actually answer them quite well. The key in this exercise is really to observe how easy or hard it is to answer the questions.

If it is really easy and you write it down and everything is clear, then you have a low degree of uncertainty. If, however, you actually have no idea how to answer, then you have a high degree of uncertainty.

Even more powerful than knowing what it is that you need to do is knowing what you don’t need to do. Those are the non-goals.

What are non-goals? It is not a non-goal if you’re working for a software company to say “I don’t need to design surfboards”. That is not a non-goal, that is just ridiculous. Of course, you don’t want to do this. A non-goal is something that is actually super reasonable and even likely that you want to do it but you know you don’t need to do it (like “I need to support low-end devices”). If you can easily define your non-goals, you know that your degree of uncertainty is even lower.

So, why is it important for the degree of uncertainty to be low? When uncertainty is high you’re going to have to make tradeoffs and be flexible. You may think “I might need to support this special case so I better make sure that it would be possible”. You likely need to make a tradeoff to enable that special case. And the problem with tradeoffs is that they make stuff complex and less than ideal because you have to find the balance between conflicting concerns.

Things really get unpleasant when a high degree of uncertainty leads you to make an unnecessary tradeoff.

Unnecessary tradeoffs are the root of all evil.

Let’s look at an example: imagine you’re building infrastructure for internal mobile apps at your company and maybe the salespeople all have old phones with low memory. Based on this you design things such that they work on low memory devices. But then your company goes out and buys everyone fancy new phones. All your work to support low-end devices was unnecessary, and even worse it probably made the software more complex in the future as well. Interestingly this is a generalization of Donald Knuth’s “premature optimization is the root of all evil”. Premature optimization is a special case of an unnecessary tradeoff. Every time you design your software for something it didn’t really need to support that is likely an unnecessary tradeoff.

This talk is being held at a web development conference, so I wanted to have one section on web development. We as a community have been building web frameworks for a few years. Over 20 years now. We actually understand how to build web frameworks really well. The degree of uncertainty for what people want to do with web frameworks is really low.

Compare that to this scenario: imagine it is 2015 and your manager comes and says “I want you to build infrastructure for deep learning”. You’re like, well, I have never done this before. No one in my company has done it before. There are like literally three blog posts on the topic in total. The degree of uncertainty is really high. You won’t be able to build infrastructure that is quite as nice as a web framework in 2020.

Dealing with uncertainty

And so the question is: what do we do if the degree of uncertainty is very high? We tailor the degree of abstraction.

Let me get something out of the way: abstractions are awesome. You want to abstract everything as much as possible because abstractions make everything really expressive, more correct, reusable, and generally, more awesome. However, things rapidly fall apart if the abstractions don’t quite allow us to do exactly what we’d like to do. We now have to work around it and beat them into doing what we want. Everything is suddenly painful and terrible.

If uncertainty is high then reduce the degree of abstraction.

That is why we need to understand the problem space really well: It allows us to tailor the abstraction to the degree of uncertainty. And if uncertainty is high then we reduce the degree of abstraction.

Solving all known problems

Every software project is going to have some degree of uncertainty. And so it is useful to have a set of techniques in our back pocket that are always good–no matter how uncertain we are: we’ll solve all known problems of software engineering.

Iteration velocity

The first technique is optimizing iteration velocity. What I mean is: you write some code and hit save in your editor–how long does it take to go from that moment until you find out whether it was a good change? You might have super awesome hot code reloading and everything updates instantly like magic or you may need to manually kill your Java application server, recompile, and restart the server which takes like 20 minutes. I think many of us have probably been somewhere in between these two extremes.

Slow iteration cycles actually introduce tradeoffs into our software design. If it takes a long time to find out that you made a mistake then as a software designer, the professional thing to do is to then go and say “I have to design this API so that no one can ever get it wrong”. If on the other hand everything is super fast and folks can figure things out iteratively because the cost of failure is so low, then you as a designer of that API can say, maybe it is fine for people to do a little bit more exploring here.

Debuggability

Next is debuggability. As a framework author we actually have a strong influence on how hard it is to debug a system. Maybe you designed this awesome but complex black box state machine that no one can figure out. Maybe your stack traces are super long and confuse people. Or maybe you put in a sensible logging and tracing system and people understand what is going on. They make a mistake, they debug it, they fix it. No problem.

Testability

Similar to the debuggability you’re in control of how testable a system is. Maybe your framework makes it really hard to instantiate things to be in a given state that can be tested? Or you make that really easy. Folks will write more tests, have higher confidence, and be a happier customer of your framework.

Empathy

For the last technique to solve all known problems of software engineering we’ll move a little bit away from the pure software side of things. We’ll talk about empathy.

And like I mentioned in my last talk: As a software engineer having empathy with other software engineers is empathy in easy mode. Our intuition for how a software engineer feels about something is much more likely to be correct than with a random person where we know little about their background. Today I want to talk about a very special aspect of empathy: You as the designer of framework could build the perfect application with it. You know everything about it. You can get everything right. But every other person who uses your framework will know less about it.

Think about what it means to not know everything about your framework.

So, think about what it means to not know everything about your framework. And how you can make the framework be robust in the presence of such not-all-knowing users.

Deploying change

Now that we have learned to solve all known problems of software engineering, let’s go to the final chapter which is deploying change.

If it has no users then it has no impact.

This part is really, really key. Software infrastructure, no matter how great it is, if it has no users then it has no impact. However, in this field it is incredibly common that people build ivory towers with no real users in mind. They build something they’re excited about, and it was probably really fun to build. And then they come and say “Hey, I have a thing” and you’re like “But that thing doesn’t do what I need” and everyone is sad and they move on to build the next ivory tower.

If you want to professionalize building software infrastructure then this is obviously not the right way to do it. Getting your stuff adopted is everything. It is a big part of your job. So much so that my first piece of advice on how to get this right is actually totally in the realm of marketing.

Software engineers are human beings. They want to work on stuff that they think is cool and that they’ve heard about on Twitter (Think “Serverless”, “Machine Learning”, “Virtual DOM”, $HypedThingOfTheYear). I think it is fine to sprinkle some of that glitter into your framework. As long as it isn’t the worst unnecessary tradeoff in the world. Everyone will be a little happier and no harm is done. That was the marketing portion of this talk. Let’s go to a bit more serious topic.

Incremental adoption

Adopting software is easier if you can do it incrementally. What this means is that instead of going like “Hey, try this new framework, rewrite everything from scratch in two years and hopefully it is good”, you migrate smaller parts step by step–and thus you see positive impact long before that 2 year full rewrite. What I’ve seen is that even projects that start out as a full rewrite will often pivot to ship at least a few parts early as the pressure from essentially having to do double-work (maintaining the legacy system and building the new one) increases and management wants to see results.

The path to incremental adoption, however, can be difficult for the framework designer. The following techniques have been proven to work in such scenarios.

Legacy composition

The first technique is really simple and comes in two parts.

1. The first is composition of your new framework code into legacy code: Let’s say you haven’t gotten around to rewriting the app shell yet. The app is still built on the old architecture. But then the team comes and says, “I’m going to build that new feature based on the new framework and put it into that old code base”. That is composition of new code into legacy code.
2. And then there is the other side which is composition of legacy code into your new code base. Let’s say you have this super ultra-customized component, a jQuery date picker, and people want to keep it, right? And so you design your framework such that the jQuery thing, which probably violates all the assumptions you ever made, still works in your new code base. That is composition of legacy code into new code.

Both of these strategies put tradeoff pressure on your system because suddenly all these assumptions are no longer valid. But the tradeoff is probably worth it because it allows people to incrementally adopt your software.

Temporary imperfection

The second technique is what I call temporary imperfection. As a framework author, you probably have this really idealized view of the world how people build software using your framework. But then it comes crashing down in the process of incremental adoption because not everything is ideal at all, but rather an entangled mess of old and new code.

My advice would be to do the following: Write a literal lint rule using a tool like ESLint that identifies the old way of doing something. And then when there is code in the code base that is not written to the new way, you get an error message. Hopefully the error message also says something helpful like “This is no longer the right way of doing it. Here are pointers to documentation for how to do it right”.

Step two is to then go and allow all existing violations using an allow-list. With that you’re in a position where no code is fixed, but where all new code has to comply with the new way of doing things. And this is really powerful because there might be all these engineers in the organization who have not read the email where they were told they need to do something differently. Now they get an error message and will just turn around and fix it.

A ledger of technical debt

There is one more interesting insight. This allow-list which you check into your repository is, in fact, a ledger of technical debt. Technical debt can be this abstract concept that you know you have. It is probably out there somewhere but it is really difficult to nail down where it is. The ledger allows you to literally know that file X.js, line 15 has technical debt in it.

Knowing where the technical debt is, is the first step to paying it off. You can e.g. have a team fix-it where all you do is make the allow-list shorter. I think that is a powerful way to make tech debt very concrete just like debt from your bank account.

Automated migration

The next technique is to enable automated migration. It is not going to get you all the way but it is pretty powerful. Tools like codemod from Facebook can help with such a process. You declare how to go from A to B and the computer does the rest of the work. It is great.

As many things in this talk, designing for automated migration puts tradeoff pressure on your new API because now your new API must ideally only require information that is already available in the existing codebase. But it might be worth it because no one has to do that work manually and it is going to make people more happy than having to do everything by hand.

Customer Zero

Let’s talk a little bit more about humans. The one thing I would like everyone to take away from this talk is that when you’re in this role of being the person responsible for the software infrastructure, you always need a customer zero. Your customer zero is the antidote to the ivory tower. They are your first customer and you’ll be helping them to build a real product.

Your customer zero is the antidote to the ivory tower.

What you need to do is go and literally write their software with them–become part of their team. This is all happening before you declare your framework to be shippable because in this step you can actually validate your assumptions. You can find corner cases and you can iterate on your framework until it is actually good. This first customer should not be a migration. It should be something that you built from scratch. Because migrations are always going to be somewhat painful and long winding and you really don’t want to migrate to something that hasn’t been proven to be good.

One of the ironies of being an infrastructure engineer is that you rarely build anything with your infrastructure yourself. That makes it really difficult to actually write the documentation for how to get started with your infra. If you go and have a customer zero and you work with them, that is the best chance you get to write that onboarding doc for everyone who comes after. Don’t miss the chance!

Actually finding a customer zero can be challenging in large organizations. You might not find the team that is cool with working with an unstable framework that might be pretty bad. I’ve found a few arguments that work well to convince a team to be your customer zero:

• There is typically a team that likes to be on the cutting edge. One of the things that you can pitch is to say “Hey, you’ll eventually have to migrate to this because it is the future stack everyone will have to use. Why don’t you want to be on the new thing already and skip that migration?”. That could be very attractive for a team. And you can argue that because you’re actually going to work with the team, they are going to get a framework tailor made for them–as opposed to everyone else who will have to make do with what is there.
• If that pitch doesn’t work, you’ll have to move on to harsher tactics. In that case the organization has to be mature enough to accept that you’ll have to have a mandate. Someone will have to decide which team is going to be customer zero and they’ll have to accept it. Because the alternative is that you’d build an ivory tower. And that would be bad for everybody, not just that one team.

Migration Zero

Eventually we’ll have to do a first migration of an existing code base to our new infra. Once again it is actually your job to do this migration yourself. That is because your framework might not actually be a great migration target yet. Let’s assume your framework is at point B and some project is at point A. It might be super hard to go from A to B. Any given team would probably just go ahead and do it anyway. But you as the author of the framework might see chances to slightly change that point B such that it is easier to get there. And because you’re a lazy person, you’re going to make that migration really easy and that is going to help everyone that comes after you have a much better time.

Summary

These were the three things that I wanted to talk about. Just let me summarize really quickly:

1. The first thing that we need to do is to understand the degree of uncertainty and then tailor the degree of abstraction respectively.
2. Even if uncertainty is high, we want to solve all known problems: Increase iteration velocity, make things debuggable, testable and practice our empathy with a focus on what folks might not know.
3. And then finally, nothing here matters if you don’t focus on team adoption. Make your software likable, focus on incremental adoption, make a ledger of technical debt, and get that customer and migration zero done.

That is all I have today. Thank you very much.

Malte Ubl, 2020

Thanks to Esther, Ryan, and Jan for helping me copy-edit my verbal mess 🤣

PS: If you’d like to share this post and bypass Medium’s paywall, use this link.

JSConf EU 2009 had 125 attendees. When I read that the BIPoCiT space ended up having 125 attendees in 2019 it got me to reflect on the tremendous privilege that emanated from the opportunity to attend our first event, and it makes me incredibly optimistic on the impact that the BIPoCiT space may have on people’s life going forward.

But first look at us:

Clearly not everyone felt invited in 2009. Those denied opportunities cannot be righted by only creating a more inviting conference in 2019, because the access and connections created at the original intimate events no longer translate to the mega conference of today.

It may be hard to understand just how impactful the privilege to attend that first JSConf EU has been to the lucky few like this introverted programmer who had too much fun implementing Perl 6’s object model in JavaScript. I’ll go through a list of connections, not as a humblebrag, but to really clarify the extend of the privilege that access to the event granted.

• I met Nikolai Onken whose family will join mine for a summer vacation in Denmark this year. He is now an executive at Amazon.
• I met Joe McCann who would 2 years later DJ at my wedding and then go on to found NodeSource.
• I met Paul & Eamon who’d later invite me to speak at their exclusive conference that took a private train to a helicopter to a small island and get drunk with Amazon’s CTO.
• I met Brian LeRoux, creator of PhoneGap, who’d invite me to his wedding off the coast of Vancouver and be my neighbor in San Francisco.
• I met Nicole Sullivan, who I now have the privilege of working with at Google, and who’s son hangs out with my kids.
• I met Ryan Dahl, creator of Node.js, who’d show me around San Francisco when I moved there.
• Speaking of San Francisco: I made so many friends at my conference, when I moved to SF three years later I literally had more friends there than in the city I previously lived in for 10 years.

This list could go on and on, and I’m sorry that I left some of y’all out of it, but the point really is: Imagine you didn’t feel welcome. Imagine that the opportunity that attending the event would have presented to you was denied because some trait of your identity felt to be incompatible with the group that was celebrating the technology you loved.

This is why it was so important to create a dedicated BIPoCiT space. I sincerely hope that the new 125 created some of the tight-knit bounds that can only form among smaller groups of people who share an experience.

I had no part in organizing the space. All credit goes to the BIPoCiT team. But I will be watching their progress, make connections, write references, grant access, and maybe in 2029, if I do well, make a bullet point on their blog post about the next 125.

This is a mildly edited transcript of my JSConf Australia talk. Watch the whole talk on YouTube. A sequel to this post is available over here.

Hello, I used to build very large JavaScript applications. I don’t really do that anymore, so I thought it was a good time to give a bit of a retrospective and share what I learned. Yesterday I was having a beer at the conference party and I was asked: “Hey Malte, what actually gives you the right, the authority, to talk about the topic?” and I suppose answering this is actually on topic for this talk, although I usually find it a bit weird to talk about myself. So, I build this JavaScript framework at Google. It is used by Photos, Sites, Plus, Drive, Play, the search engine, all these sites. Some of them are pretty large, you might have used a few of them.

This Javascript framework is not open source. The reason it is not open source is that it kind of came out at the same time as React and I was like “Does the world really need another JS framework to choose from?”. Google already has a few of those–Angular and Polymer–and felt like another one would confuse people, so I just thought we’d just keep it to ourselves. But besides not being open source, I think there is a lot to learn from it and it is worth sharing the things we learned along the way.

So, let’s talk about very large applications and the things they have in common. Certainly that there might be a lot of developers. It might be a few dozens or even more–and these are humans with feelings and interpersonal problems and you may have to factor that in.

And even if your team is not as big, maybe you’ve been working on the thing for a while, and maybe you’re not even the first person maintaining it, you might not have all the context, there might be stuff that you don’t really understand, there might be other people in your team that don’t understand everything about the application. These are the things we have to think about when we build very large applications.

Another thing I wanted to do here is to give this a bit of context in terms of our careers. I think many of us would consider themselves senior engineers. Or we are not quite there yet, but we want to become one. What I think being senior means is that I’d be able to solve almost every problem that somebody might throw at me. I know my tools, I know my domain. And the other important part of that job is that I make the junior engineers eventually be senior engineers.

But what happens is that at some point we may wonder “what might be the next step?”. When we reached that seniority stage, what is the next thing we are going to do? For some of us the answer may be management, but I don’t think that should be the answer for everyone, because not everyone should be a manager, right? Some of us are really great engineers and why shouldn’t we get to do that for the rest of our lives?

I want to propose a way to level up above that senior level. The way I would talk about myself as a senior engineer is that I’d say “I know how I would solve the problem” and because I know how I would solve it I could also teach someone else to do it.

And my theory is that the next level is that I can say about myself “I know how others would solve the problem”.

Let’s make that a bit more concrete. You make that sentence: “I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem.” I think this is a powerful concept that allows me to reason about how the choices I’m making impact an application.

I would call this an application of empathy. You’re thinking with other software engineers and you’re thinking about how what you do and the APIs that you are giving them, how they impact how they write software.

Luckily this is empathy on easy mode. Empathy is generally hard, and this is still very hard. But at least the people that you are having empathy with, they are also other software engineers. And so while they might be very different from you, they at least have in common that they are building software. This type of empathy is really something you can get quite good at as you gain more experience.

Thinking about these topics there is one really important term that I want to talk about, which is the programming model–a word that I’m going to use a lot. It stands for “given a set of APIs, or of libraries, or of frameworks, or of tools–how do people write software in that context.” And my talk is really about, how subtle changes in APIs and so forth, how they impact the programming model.

I want to give a few examples of things that impact the programming model: Let’s say you have an Angular project and you say “I’m going to port this to React” that is obviously going to change how people write software, right? But then you’re like “Ah, 60KB for a bit of virtual DOM munging, let’s switch to Preact”–that is an API compatible library, it is not going to change how people write software, just because you make that choice. Maybe then you’re like “this is all really complex, I should have something orchestrating how my application works, I’m going to introduce Redux.”–that is going to change how people write software. You then get this requirement “we need a date picker” and you go to npm, there are 500 results, you pick one. Does it really matter which one you pick? It definitely won’t change how you write software. But having npm at your fingertips, this vast collection of modules, having that around absolutely changes how you write software. Of course, these are just a few examples of things that might or might impact how people write software.

Now I want to talk about one aspect that all large JavaScript applications have in common, when you deliver them to users: Which is that they eventually get so big that you don’t want to deliver them all at once. And for this we’ve all introduced this technique called code splitting. What code splitting means is that you define a set of bundles for your application. So, you’re saying “Some users only use this part of my app, some users use another part”, and so you put together bundles that only get downloaded when the part of an application that a user is actually dealing with is executed. This is something all of us can do. Like many things it was invented by the closure compiler–at least in the JavaScript world. But I think the most popular way of doing code splitting is with webpack. And if you are using RollupJS, which is super awesome, they just recently added support for it as well. Definitely something y’all should do, but there are some things to think about when you introduce this to an application, because it does have impact on the programming model.

You have things that used to be sync that now become async. Without code splitting your application is nice and simple. There is this one big thing. It starts up, and then it is stable, you can reason about it, you don’t have to wait for stuff. With code splitting, you might sometimes say “Oh, I need that bundle”, so you now need to go to the network, and you have to factor in that this can happen, and so the applications becomes more complex.

Also, we have humans entering the field, because code splitting requires you to define bundles, and it requires you to think about when to load them, so these humans, engineers on your team, they now have to make decisions what is going into which bundle and when to load that bundle. Every time you have a human involved, that clearly impacts the programming model, because they have to think about such things.

There is one very established way that solves this problem, that gets the human out of the mess when doing code splitting, which is called route based code splitting. If you’re not using code splitting yet, that is probably how you should do it as a first cut. Routes are the baseline URL structure of your application. You might, for example, have your product pages on `/product/` and you might have your category pages somewhere else. You just make each route one bundle, and your router in your application now understands there is code splitting. And whenever the user goes to a route, the router loads the associated bundle, and then within that route you can forget about code splitting existing. Now you are back to the programming model that is almost the same as having a big bundle for everything. It is a really nice way to do this, and definitely a good first step.

But the title of this talk is designing VERY large JavaScript applications, and they quickly become so big that a single bundle per route might not be feasible anymore, because the routes themselves become very big. I actually have a good example for an application that is big enough.

I was figuring out how to become a public speaker coming up to this talk, and I get this nice list of blue links. You could totally envision that this page fits well into a single route bundle.

But then I was wondering about the weather because California had a rough winter, and suddenly there was this completely different module. So, this seemingly simple route is more complicated than we thought.

And then I was invited to this conference, and was checking out how much 1 US dollar is in Australian dollars, and there is this complex currency converter. Obviously there is about 1000s more of these specialized modules, and it infeasible to put them all in one bundle, because that bundle would be a few megabytes in size, and users would become really unhappy.

So, we can’t just use route based code splitting, we have to come up with a different way of doing it. Route based code splitting was nice, because you split your app at the coarsest level, and everything further down could ignore it. Since I like simple things, how about doing super fine-grained instead of super coarse-grained splitting. Let’s think about what would happen if we lazy loaded every single component of our website. That seems really nice from an efficiency point of view when you only think about bandwidth. It might be super bad from other point of views like latency, but it is certainly worth a consideration.

But let’s imagine, for example, your application uses React. And in React components statically depend on their children. That means if you stop doing that because you are lazy loading your children, then it changes your programming model, and things stop being so nice.

Let’s say you have a currency converter component that you want to put on your search page, you import it, right? That is the normal way of doing it in ES6 modules.

But if you want to lazy load it, you get code like this where you use dynamic import, which is a new fancy thing to lazy load ES6 modules and you wrap it in a loadable component. There are certainly 500 million ways to do this, and I am not a React expert, but all of these will change how you write the application.

And things aren’t as nice anymore–something that was static, now becomes dynamic, which is another red flag for the programming model changing.

You have to suddenly wonder: “Who decides what to lazy load when” because that is going to impact the latency of your application.

The human is there again and they have to think about “there is static import, there is dynamic import, when do I use which?”. Getting this wrong is really bad because one static import, when it should have been dynamic suddenly may put stuff into the same bundle that shouldn’t be. These are the things that are going to go wrong when you have a lot of engineers over long periods of time.

Now I’m going to talk about how Google actually does this and what is one way to get a good programming model, while also achieving good performance. What we do is we take our components and we split them by rendering logic, and by application logic, like what happens when you press a button on that currency converter.

So, now we have two separate things, and we only ever load the application logic for a component when we previously rendered it. This turns out to be a very simple model, because you can simply server side render a page, and then whatever was actually rendered, triggers downloading the associated application bundles. This puts the human out of the system, as loading is triggered automatically by rendering.

This model may seem nice, but it does have some tradeoffs. If you know how server side rendering typically works in frameworks like React or Vue.js, what they do is a process called hydration. The way hydration works, is you server side render something, and then on the client you render it again, which means you have to load the code to render something that is already on the page, which is incredibly wasteful both in terms of loading the code and in terms of executing it. It is a bunch of wasted bandwidth, it is a bunch of wasted CPU–but it is really nice, because you get to ignore on the client side that you server side rendered something. The method we use at Google is not like that. So, if you design this very large application, you have think about: Do I take that super fast method that is more complicated, or do I go with hydration which is less efficient, but such a nice programming model? You will have to make this decision.

My next topic is my favorite problem in computer science–which is not naming things, although I probably gave this a bad name. It is the “2017 holiday special problem”. Who here has ever written some code, and now it is no longer needed but it is still in your codebase? … This happens, and I think CSS is particularly famous for it. You have this one big CSS file. There is this selector in there. Who really knows whether that still matches anything in your app? So, you end up just keeping it there. I think the CSS community is at the forefront of a revolution, because they realized this is a problem, and they created solutions like CSS-in-JS. With that you have a single file component, the 2017HolidaySpecialComponent, and you can say “it is not 2017 anymore” and you can delete the whole component and everything is gone in one swoop. That makes it very easy to delete code. I think this is a very big idea, and it should be applied to more than just CSS.

I want to give a few examples of this general idea that you want to avoid central configuration of your application at all cost, because central configuration, like having a central CSS file, makes it very hard to delete code.

I was talking before about routes in your application. Many applications would have a file like “routes.js” that has all your routes, and then those routes map themselves to some root component. That is an example of central configuration, something you do not want in a large application. Because with this some engineer says “Do I still need that root component? I need to update that other file, that is owned by some other team. Not sure I’m allowed to change it. Maybe I’ll do it tomorrow”. With that these files becomes addition-only.

Another example of this anti-pattern is the webpack.config.js file, where you have this one thing that is assumed to build your entire application. That might go fine for a while, but eventually needing to know about every aspect of what some other team did somewhere in the app just doesn’t scale. Once again, we need a pattern to emerge how to decentralize the configuration of our build process.

Here is a good example: package.json, which is used by npm. Every package says “I have these dependencies, this is how you run me, this is how you build me”. Obviously there can’t be one giant configuration file for all of npm. That just wouldn’t work with hundreds of thousands of files. It would definitely get you a lot of merge conflicts in git. Sure, npm is very big, but I’d argue that many of our applications get big enough that we have to worry about the same kind of problems and have to adopt the same kind of patterns. I don’t have all the solutions, but I think that the idea that CSS-in-JS brought to the table is going to come to other aspects of our applications.

More abstractly I would describe this idea that we take responsibility for how our application is designed in the abstract, how it is organized, as taking responsibility of shaping the dependency tree of our application. When I say “dependency” I mean that very abstractly. It could be module dependencies, it could be data dependencies, service dependencies, there are many different kinds.

Obviously, we all have super complicated applications, but I’m going to use a very simple example. It has only 4 components. It has a router that knows how to go from one route of your application to the next, and it has a few root components, A, B, and C.

As I mentioned before this has the central import problem.

Because the router now has to import all the root components, and if you want to delete one of them you have to go to the router, you have to delete the import, you have to delete the route, and eventually you have the holiday special 2017 problem.

We at Google have come up with a solution for this, that I want to introduce to you, which I don’t think we have ever talked about. We invented a new concept. It is called enhance. It is something you use instead of import.

In fact, it is the opposite of import. It is a reverse dependency. If you enhance a module, you make that module have a dependency on you.

Looking at the dependency graph, what happens it that there are still the same components, but the arrows point in the opposite direction. So, instead of the router importing the root component, the root components announce themselves using enhance to the router. This means I can get rid of a root component by just deleting the file. Because it is no longer enhancing the router, that is the only operation you have to do to delete the component.

That is really nice, if it wasn’t for the humans again. They now have to think about “Do I import something, or do I use enhance? Which one do I use under which circumstances?”.

This is particular bad case of this problem, because the power of enhancing a module, of being able to make everything else in the system have a dependency on you is very powerful and very dangerous if gotten wrong. It is easy to imagine that this might lead to really bad situations. So, at Google we decided it is a nice idea, but we make it illegal, nobody gets to use it–with one exception: generated code. It is a really good fit for generated code actually, and it solves some of the inherent problems of generated code. With generated code you sometimes have to import files you can’t even see, have to guess their names. If, however, the generated file is just there in the shadows and enhances whatever it needs, then you don’t have these problems. You never have to know about these files at all. They just magically enhance the central registry.

Let’s take a look at a concrete example. We have our single file component here. We run a code generator on it and we extract this little route definition file from it. And that route file just says “Hey Router, here I am, please import me”. And obviously you can use this pattern for all kinds of other things. Maybe you are using GraphQL and your router should know about your data dependency, then you can just use the same pattern.

Unfortunately this is not all we need to know. There is my second favorite problem in computer science which I call the “Base bundle pile of trash”. The base bundle in your graph of bundles in your application is the one bundle that will always get loaded–independent of how the user interacts with the application. So, it is particularly important, because if it is big, then everything further down will also be big. If it small, then dependent bundles at least have a chance of being small as well. A little anecdote: At some point I joined the Google Plus JavaScript infrastructure team, and I found out that their base bundle had 800KB of JavaScript. So, my warning to you is: If you want to be more successful than Google Plus, don’t have 800KB of JS in your base bundle. Unfortunately it is very easy to get to such a bad state.

Here is an example. Your base bundle needs to depend on the routes, because when you go from A to B, you need to already know the route for B, so it has to always be around. But what you really don’t want in the base bundle is any form of UI code, because depending on how a user enters your app, there might be different UI. So, for example the date picker should absolutely not be in your base bundle, and neither should the checkout flow. But how do we prevent that? Unfortunately imports are very fragile. You might innocently import that cool util package, because it has a function to make random numbers. And now somebody says “I need a utility for self driving cars” and suddenly you import the machine learning algorithms for self driving cars into your base bundle. Things like that can happen very easily since imports are transitive, and so things tend to pile up over time.

The solution we found for this are forbidden dependency tests. Forbidden dependency tests are a way to assert that for example your base bundle does not depend on any UI.

Let’s take a look at a concrete example. In React every component needs to inherit from React.Component. So , if your goal is that no UI could ever be in the base bundle just add this one test that asserts that React.Component is not a transitive dependency of your base bundle.

Looking at the previous example again, you just get a test failure when someone wants to add the date picker. And these test failures are typically very easy to fix right then, because usually that person didn’t really mean to add the dependency–it just crept in through some transitive path. Compare this to when this dependency would have been around for 2 years because you didn’t have a test. In those cases it is typically extremely hard to refactor your code to get rid of the dependency.

Ideally though, you find that most natural path.

You want to get to a state where whatever the engineers on your team do, the most straightforward way is also the right way–so that they don’t get off the path, so that they naturally do the right thing.

This might not always be possible. In that case just add a test. But this is not something that many people feel empowered to do. But please feel empowered to add tests to your application that ensure the major invariants of your infrastructure. Tests are not only for testing that your math functions do the right thing. They are also for infrastructure and for the major design features of your application.

Try to avoid human judgement whenever possible outside of the application domain. When working on an application we have to understand the business, but not every engineer in your organization can and will understand how code splitting works. And they don’t need to do that. Try to introduce these thing into your application in a way that is fine when not everybody understands them and keeps the complexity in their heads.

And really just make it easy to delete code. My talk is called “building very large JavaScript applications”. The best advice I can give: Don’t let your applications get very large. The best way to not get there is to delete stuff before it is too late.

I want to address just one more point, which is that people sometimes say that having no abstractions at all is better than having the wrong abstractions. What this really means is that the cost of the wrong abstraction is very high, so be careful. I think this is sometimes misinterpreted. It does not mean that you should have no abstractions. It just means you have to be very careful.

We have to become good at finding the right abstractions.

As I was saying at the start of the presentation: The way to get there is to use empathy and think with your engineers on your team about how they will use your APIs and how they will use your abstractions. Experience is how you flesh out that empathy over time. Put together, empathy and experience is what enables you to choose the right abstractions for your application

If you found this interesting, consider reading my article on design documents or the sequel to this post “Designing Even Larger Applications”.

Here is the thing: There might actually be a pipeline problem. It is just different from what you think. You have a pipeline problem, because the the candidates from underrepresented groups in tech don’t want to work for you.

Being the only one of some sub group in a larger group is always terrible. I used to work in agencies, which had lots of projects with widely varying team distributions and I learned: The same people behave very different depending on the environment. All-male teams have what might be called “locker room talk”. Nothing like that would occur in diverse teams. But the team that has 1 or 2 women: The locker room talk is still there. Would I want to be a woman in that otherwise all male team: Hell, no.

If your organization has many teams where each one has a small count of members of underrepresented groups you have effectively gerrymandered your organization.

If you neglected hiring a diverse team early in the process, it will be incredibly hard to fix it, because the first people who break your pattern will have a bad time. And it is not their job to suffer through that bad time just so you can eventually have a diverse team.

With that, here is my slightly provocative proposal for fixing your 15 people startup with 14 guys, because lets face it, you won’t fire 7 of them and hire 7 women:

Start a diversity bad bank. Put the 14 guys in a team, and make a new team that has great diversity. The 14 guys are your bad bank. Maybe initially only 3 or 4 people are on the new team. Start hiring into this team with diversity in mind. Put in the effort to reach out. Tell candidates that they get to work in a diverse team. As things grow, slowly move members of the 14 into your good business–or in the gerrymandering analogy: Try to get to 3 purple district as fast as you can. Maybe eventually you can close the bad bank.

Is this perfect? No, people from the new team will still have to talk to the bad bank. Does it even work? I don’t know. But it certainly seems better than putting the burden on your more diverse hires to put up with the 14.

A few days ago Jake Archibald ruffled some feathers with this observation:

Jake Archibald on Twitter

It seems the web community has decided that developer experience dwarfs user experience, and I'm not sure what to do about that...

This strongly reminded me of the first and most important item in AMP’s design principles:

User Experience > Developer Experience > Ease of Implementation

So, given that my project is setting very similar guidelines compared to what Jake implied in his tweet, I thought it’d be useful to explain the thinking behind it.

When creating systems for user facing software, then user experience is more important than developer experience. We add a third item for AMP: “Ease of implementation” is less important then developer experience. The implementation here refers to the creator of the underlying system and framework (in our case AMP). The idea is that the framework creator should always strive to optimize developer experience, even if they have to work harder for it.

Of course, there is a lot of nuance to the statement that user experience is more important than developer experience. The most important nuance is that in many, many cases they do not stand in conflict at all. Furthermore, in a large number of cases improving developer experience is the best way to improve user experience, so rather than being in conflict these goals are often in congruence. My favorite example is to improve developer experience by speeding up the iteration cycle (e.g. by improving build time or providing hot reloading): This is great for developers and it allows them to spend more time improving UX in any given day, thus leading to better results.

Similarly, there are many cases where improved developer experience might just enable a piece of software being build in the first place. The UX might not be perfect, but at least the thing exists. That should typically be a net win :)

Having said all this, there are trade offs where improved developer experience, just leads to bad UX. My favorite example is

<img src=”./image.jpg”>

which has much better developer experience, than

<img src=”./image.jpg” width=400 height=300>

The first image tag is missing the dimensions. (Ignoring the complexities of responsive sizing that do not impact the point) This is great for the developer. They literally do not need to write the piece of software that determines the dimensions, which saves a little bit of time. Now what happens is that the browser will assume: “Ah, no dimensions, the image is probably 0 pixels big”. Then it will download the image and realize “Ah, it is actually 400x300, lets redraw the page”. This creates a lot of jank for the user every single time a page is looked at.

A system that favors user experience over developer experience would always make sure that the image dimensions are known when the page is downloaded. Now, of course, one could improve developer experience again, by somehow making it very easy (or automatic) to provide the image dimensions. Another case, where one can have the best of both worlds, but getting good UX does require going the extra mile during framework development.

Server side rendering, lazy loading of JavaScript bundles and splitting up CSS files are all optimizations of user experience that do incur a cost on developers. Again, a given framework can chose to make them as automatic as possible, but such abstractions tend to be at least a little bit leaky and add visible complexity.

My post about tradeoffs in server and client side rendering tries to dive into that topic in particular. The React style hydration of a server side render right upon initial load is a user experience and developer experience trade off: It is better than not doing server side rendering at all, but it is strictly worse than lazily initializing the JS on the client (because it does an initial eager diff and installs event handlers). User experience may not be ideal, but going the extra mile may not be worth it or delay shipment indefinitely for a large class of applications (maybe that class includes everything except google.com, twitter.com, and facebook.com).

Having said all this, the premise that user experience is more important than developer experience is still an important thing to keep in mind as a framework developer. We always need to ask ourselves:

• What is the right thing to do for user experience?
• How can I provide the best possible DX given my UX goals?
• What is the smallest possible UX tradeoff given my DX goals?
• How can I make tradeoffs that improve DX while sacrificing UX, but only in development mode?
• How can I document things for developers, so they can make informed tradeoffs between UX and DX?

• super simple code splitting.
• Currently exactly one configuration option.
• ES6 modules.
• CommonJS modules (with some caveats).
• extremely efficient packing of modules.
• dead code elimination (also sometimes called tree shaking).
• smaller code for real world projects than all other known module bundlers.
• JSX (and React)

Why this exists

There has been a lot of chatter in the JS community lately about code splitting being essential for shipping performant web apps on mobile.

The 1–0–1 on code splitting is: Every application will eventually be too large, for loading the entire app at once being feasible, but only a small part is needed for it to start up. Code splitting allows turning an application into multiple bundles that are loaded when needed (or when the app has nothing better to do).

Yet, no solution for code splitting was really satisfying:

• Webpack has a reputation of being relatively hard to use and it generates pretty verbose code.
• Closure Compiler equally has a reputation of being hard to configure. I’m personally not aware of anyone outside of Google using its code splitting feature (although there are certainly projects that do!) since it is essentially undocumented.
• Rollup generates awesome code and is easy to use, but doesn’t support code splitting, and has no direct support for CommonJS modules.

And so I decided to make a solution that is hopefully both easy to use and that generates close-to-ideal code.

Splittable uses Babel and Browserify for dependency resolution and then uses Closure Compiler’s little known code splitting feature to generate super tightly packed code. In particular:

• all module code is compiled away leading to zero per-module overhead (similar to Rollup).
• all code is subject to dead code elimination (tree shaking) including private functions, classes, methods, and other declarations.
• module code is split apart across bundles (so that functions that are not called in a bundle get moved to the bundle where they are called, or removed).

Usage

It is very early days for splittable. Give it a try! Feedback and pull requests welcome!

There has been some critique of this design decision so I wanted to take the time to explain the reasoning behind it and lay out how to do better in the future.

The most obvious consequence of centrally hosting the JavaScript is that every AMP page on the internet is always on the latest release. A pull request to AMP’s GitHub that makes AMP faster is typically deployed to all AMP pages on the internet within 2 weeks, making all AMP pages faster at once. No page is left behind.

Another big benefit of loading the very same JS URLs on every page is that this leads to much, much better utilization and hit rate of the browser cache. If you’ve been to an AMP page on publisher X, going to another AMP page on publisher Y will now be faster and cost less bandwidth. AMP is currently in the process of shipping a foreign-fetch ServiceWorker that will make this even more impactful.

Finally, having all AMP pages use explicitly shared JavaScript opens up the possibility of additional optimization by the AMP Caches. While there are a set of requirements for AMP Caches, they don’t actually need to use AMP’s own JavaScript URLs. Knowing what version (the latest) of AMP a page is compatible with, they can then go and perform additional optimizations using similar techniques to an optimizing compiler. This would be infeasible if they’d need to support what would effectively be all AMP releases ever.

There are, of course, some valid reasons why using centrally hosted JS might not be such a great idea. We do mitigate some of them:

• AMP is designed to gracefully deal with the JS failing to load.
• AMP is rolled out slowly across pages to catch and fix errors before the full release.
• Developers can opt into the “AMP Dev Channel” to test their site with a new release before it is pushed to users.
• The JS is loaded from a domain that doesn’t have cookies (avoiding the possibility of tracking through them).
• Caching (especially with the foreign-fetch ServiceWorker active) leads to the JS being downloaded rarely (further avoiding possibly correlation of IP addresses and referrers).
• AMP is 100% open source. Everybody can check that the JS being served from the CDN is exactly what is in the open source repo.

Decentralized hosting of JavaScript files

Having said this, the AMP Project is very interested in supporting decentralized hosting of its JavaScript files.

“Build a decentralized cross-internet 3P script repository with semver semantics.”

Our idea is to build a decentralized cross-internet 3P script repository with semver semantics (Think NPM, but with first class support in the web platform and with multiple providers). That could achieve many of the same goals that I outlined above (AMP would mandate wildcarding the patch level in the semver version). Many of us had hopes that subresource integrity could bring some of the cross site caching benefits, but unfortunately that comes with privacy problems (it allows finding out which sites a user has been to) and thus cannot be taken advantage of in web browsers. By limiting such a feature to well known repositories that can be controlled by the user, one could get the clear benefits in a privacy friendly way.

Here is how this would look like:

With this a page author can say:

• I’d like to load this library.
• Dear browser: Check if the user has a version of it from a set of hosts the client trusts.
• If that is not available, load the library from a given location (that may be local).

I think such a mechanism would be a great benefit to most web applications and would likely save users from downloading many petabytes of data every day.

Update 4/20/2019: Much of this is now rolled out on Google’s ad stack.

A significant part of my job working on the AMP Project has been to engage in discussions on Twitter and in our GitHub issues around what AMP is, what it should be and whether it is doing the right things. Often these discussions eventually come to a point where somebody says:

“Cool, you made the content faster, but what about the ads? Aren’t they the worst offenders for page load times?”

AMP’s original goal was to lift up the user experience across as much web content as possible. This meant we couldn’t just build some idealistic system–existing monetization methods would need to be supported in order to get the wide adoption that would lead to widespread user experience improvements. On today’s web that meant: AMP had to support advertising.

I learned a lot about internet advertising in the first months of working on AMP, and one of those things was that the web advertising industry doesn’t exactly change fast. In fact, looking at lots of ad related JavaScript on the web, one cannot avoid frequent 90s flashbacks.

That led to the design compromise in AMP: Support legacy web advertising, but do so in a way that would mitigate impact on the pages it is embedded into. To achieve this goal AMP uses the following techniques:

• Content first. AMP loads ads after the rest of the content, so ads would never slow down load time.
• Containment. AMP strictly manages the area of a page that ads can access and control to a well defined rectangle. Among other things this avoids pages “jumping around” as ads pop in.
• Mitigation. AMP mitigates against various JavaScript worst practices often found in ad tech such as `document.write`, by limiting their effect on the ad itself vs. the entire page.
• Intervention. AMP slows down timers such as those used for animations for ads that are not currently visible, so that ads use much less battery and CPU when they aren’t being seen. Firefox, Safari, Opera and Chrome have recently started doing this by default and we’ll be happy to delete the code when this rolls out to more users. Because in AMP all legacy ads run inside iframes (many ads outside of AMP have access to the host page), these new browser interventions work particularly well in AMP.

These measures, together with AMP’s aggressive optimizations for general web content, significantly improve both page load time and mitigate effects of the ads on the user experience. In particular, the fact that content always loads first guarantees that ads are never in the way of users doing what they came for on a given page.

There are, however, limits to the fidelity that can be achieved with the approach we have taken so far. This starts with the obvious: Delaying ad load until after content might be an OK compromise, but it is far from the best possible strategy one could employ with finer grained control over an ad’s lifecycle. But the biggest issue, which AMP so far has not addressed, is what we call the coordination problem.

The coordination problem

One of the greatest strengths of the web is that it has a super flexible composition model. This allows easy and ad-hoc assembly of things like YouTube videos, Instagrams, Tweets and ads on a single page. But while composition is simple, all these components share a single thread for JavaScript computation and most UI operations. This is where the coordination problem comes into play: While each component, in isolation, might work super well, things can easily deteriorate to bad UX when assembled together.

This is a frequent problem with ads. For instance, it is super easy to envision 3 ads that each use 6ms of CPU per frame for their animations. That is in itself great — easily staying within the frame budget to achieve 60 frames per second (or 16ms total frame time). But when those ads appear together on the same page, they suddenly need 18ms per frame and the browser can no longer deliver the smooth experience of 60 frames per second.

That is the coordination problem: Even perfectly engineered ads can, in aggregation, negatively impact user experience.

Finally, even ignoring the above, there is a lot that can be done to improve overall quality of ad creatives in terms of their impact on user experience. AMP was first created as a simple path to creating well performing documents on the web, and it seems that it would be great to make that same leap forward for advertising on the web.

Introducing AMP for ads

Earlier this year, a team at Google asked themselves the question:

“AMP worked super well for general content. Couldn’t we just use it for ads?”

…which is what they did. These ads based on AMP are normal, valid AMP documents in their own right, that just happen to be ad creatives. I’m super excited about this as it provides a path to solving all the technical challenges outlined above, bringing us to a more technically healthy advertising ecosystem on the internet.

It is early days for AMP for ads, or A4A, but after announcing the effort on GitHub a few weeks ago, the team has merged their pull request into AMP and is in the process of setting up a live experiment. AMP itself will continue to support non-AMP ad creatives to allow for a gradual transition of the wider ecosystem.

Separating ad requests from ad rendering

The first innovation in A4A seems very obvious in retrospect, but has a super large impact: As outlined above, AMP, by default, loads ads after the content and generally lazily. This is because rendering ads can be expensive in terms of CPU and RAM. However, the ad request itself can be slow, because a lot of work (think: the auction) has to be done on the server side — doing it late is not great for fast load times. From the client’s perspective making the request itself is super cheap, but its side effect (the rendering of the ad) is expensive. By separating the two, A4A achieves much faster ad rendering at no additional CPU and memory cost.

Going deeper: A restricted subset of AMP

One of the core elements of AMP is that is ships with a validator, that checks compliance of AMP documents with certain rules. The idea is: If the rules are met, then the document will load fast. The ruleset we designed for AMP has a broad set of document types and use cases in mind. Ad creatives, on the other hand, are a much more focused use case, and hence they do not actually require the full breadth of functionality that AMP exposes. For this reason, AMP for ads carefully picks out the aspects of AMP that are required to build ad creatives. One example is that AMP based ads will not be allowed to load iframes with arbitrary JavaScript.

Contrast this with the status quo in ad tech: ad creatives typically have full control over the browser and they can dynamically load more code at runtime, so it is impossible to ever really know what they will do. AMP based creatives on the other hand will have well defined, statically analyzable behavior while still having access to the vast majority of web platform functionality.

AMP being optimized for documents right now is certainly missing many types of components needed for engaging ads. The team is hard at work at closing that feature gap. One example is that AMP for ads will ship with a component for smooth and interactive timeline based animations.

Re-using AMP

Ads often come with their own set of measurement tools to collect important information such as whether an ad was actually seen by a user. This substantially increases ad payload size, initial compilation and execution time, battery usage and other aspects of runtime performance.

AMP, through its established `amp-analytics` mechanism, already ships with all the code to perform these measurements. It is vendor neutral and supports a wide range of metrics. This means ads can take advantage of the same “instrument once, report many times” feature that benefits AMP pages today, completely eliminating the bandwidth and runtime cost outlined above.

The coordination solution

AMP based ads will participate in page layout like all other AMP resources. That means they automatically take advantage of AMP’s features for minimizing resource impact on runtime performance.

In particular, AMP only animates things that are visible on the screen. Period. While browsers are working on achieving this at the platform level, they need to be conservative in not breaking existing use cases. AMP for ads being new and special purpose technology, can pinpoint when animations are needed and thus further reduce CPU usage and battery consumption.

AMP will act as a supervisor for ads and ensure that they do not negatively impact primary content on a page. A4A based animations will be throttled to lower-but-uniform frame rates when AMP detects that 60 frames per second cannot be achieved on the current device. Similarly, if AMP is unable to stabilize the frame rate it will turn off animations. This ensures that every device gets the best experience it can deliver and makes sure that ads cannot have a negative impact on important aspects of the user experience such as scrolling.

Cross platform

As part of the AMP Project, AMP for ads is a vendor neutral initiative. While we are still in the early experimentation and implementation phase, the technology is designed to support all ad networks. If you work on ad tech, say Hi on GitHub. We believe A4A is a big step forward for user experience and would love to see wide adoption across the internet advertising industry.

So, that is what we are doing with ads

It is early days, and we are just getting started to explore using AMP for ads.

In this future

• ads will be statically analyzable to be safe and behave well,
• they will be able to use a common set of functionality to significantly reduce bandwidth consumption,
• CPU usage will be limited to on-screen ads, maximizing battery life,
• and ads will be coordinated with the page making sure that primary content and functionality can always be buttery smooth at 60 frames per second.

We are trying to build a user-experience-first ecosystem for advertising on the web and, looking at the success of AMP in publishing, it might just work.

I’ve been wanting to write this post for a while based on previous experience. But then I joined Google and happily dove deep into their infrastructure software for 5 years with all those hard earned skills not needed anymore; until recently when I suddenly found myself talking to more profession-diverse audiences again.

In 1998, still a high school student, I started working in the then-new internet team of a large advertising agency as employee^Wstudent #2. I ended up staying for 10 years, turning the little team into a serious software consulting company. But through all that time we’d go to clients who needed a lot of convincing to trust their core business software to this tiny offspring of this other company that made their fliers and business cards. I thought there were the toughest sales meetings I’d ever go to. I’d later join a much larger competitor and thought “Now its all going to be super easy with all that reputation in the back” — only to realize that their prospective clients perceived them as too expensive and complicated. The ways to win clients in both scenarios remained the same.

The 2 strategies below are about winning people’s trust in business meetings. Once people trust you, everything else is easy.

Strategy #1 is a bit less ethical. If that isn’t your thing, jump to #2.

#1 Brutal honesty

One benefit of being considered the introvert, engineery-type-person in a meeting is that people put some inherent trust in you. They think you’re the scientist with a bias for theoretical proof and empirical research over emotion and intuition. By playing to these stereotypes one can get a significant advantage.

I call this strategy brutal honesty, because it starts by making something that nobody in the meeting expects: Casually admit to a weakness in the thing you are presenting. It is important that this weakness is not a killer-argument against whatever you are trying to sell, but it must be bad enough that normal business-common-sense would be to talk around it and try to avoid it being a topic. Game-theory common sense should say: You just made a mistake.

Making this mistake changes expectations that people have in you. They might have had a slight honest-bias already and now it is strongly reinforced. You’re the anti-sales person. It also helps doing other things a professional sales person would never do; like answering a questions with:

No.

Yes also works. The important part is that it is completely unqualified. Possibly a bit awkward, but very precise.

By playing to stereotypes and being more-honest-than-expected, people’s expectations change and their initial distrust, “they are just trying to sell me something”, has a chance of going away.

While this might all seem a bit disingenuous, the nice thing about brutal honesty is that is can be used in a charmingly recursive fashion: by explaining the very strategy to your prospective client over lunch.

#2 Empathic listening

When people judge you against your competitors, who might have equally qualified offerings, it often matters a lot who the client thinks understands their needs better. There is one-weird-trick how to increase the perception of the degree of your understanding in a very significant way. Unfortunately, people don’t do this much at all. They have their well studied sales pitches and can’t seem to actually adapt to expressed needs in realtime — and thus create the perception that they never could.

I call this empathic listening, but one can really do it in a completely formulaic way without any real empathy:

1. Get a piece of paper.
2. Listen to your client speak. Very closely. They are typically happy to describe their needs. Too often I see people not pay attention or not even giving them a chance to speak.
3. Try to filter out the words they find important. These might be words familiar to you; or some type of domain language not used beyond the walls of their office.
4. Write down these words.
5. Try as hard as possible to understand what they mean to them and why they are important. They might use some common word in a special way. Make a note of it.
6. And now for the rest of the meeting, and your business relationship, explain everything you do in terms of these words.
7. Non-awkwardly if possible. There is the risk of misusing a term which then results in the opposite effect.

If done well this transports the message “They really understand me and what I need” which can make all the difference in an otherwise little-differentiated market. But more importantly, it might actually help further the mutual understanding.

I’ll go back now writing a post about ES6 module compilation, but I hope this excursion into the world of business was a bit useful. The summary is that one can use people stereotypes to one’s own advantage and with just a little bit of true listening it it easy to optimize one’s language for a huge boost in mutual understanding.

1. Virtual DOM has given us a programming model on the web that works well with non-immediate access to native DOM APIs.
2. The UI thread has gotten too damn busy.

I’m writing this post now, before workers become mainstream, because concurrency is hard. While we have a promising programming model (as outlined above), we still need to experiment with this a lot.

If you need additional convincing that worker-based concurrency is coming, some recent examples that are blazing the trail are Henrik Joreteg’s super interesting FeatherApp and this amazing article on the software design of the Pokedex web app.

The rest of this post are the slightly edited notes from my talk at JSConf US 2015 Last Call going a little bit deeper into how and why I think we need to invest in concurrency on the web. It explains things in the context of my current project AMP, but I believe the message applies beyond that narrow scope.

• We need to talk about the web.
• Things have gotten into kind of a bad state and this is even more true on the mobile web.
• To me reading content on the mobile web really is a slow, clunky and frustrating experience — and this is even true for content that is supposedly optimized for mobile.
• My personal pet peeve: The page finally loaded and I start reading the article and then some ad or whatever at the top of the page starts loading and pushes everything down, so I lose my reading position. Not very awesome.

• What we’ve seen is average page weight to increase at over 15% year over year for the last two years
• With just some casual manual checks it is not hard to find mobile news sites that download over 10MB for a single page.
• This is why we’ve created the AMP Project.
• And by we I mean: Google, Twitter, Pinterest, LinkedIn and a whole lot of publishers
• We do 2 things at the core:

1. Optimize content for prerendering and thus achieve instant loading for the web
2. And even without prerendering achieve reliable, fast performance.

• HTML + CSS + Extra validator
• Makes things that might be slow invalid
• Focus on load time and consistent consumption experience.
• Still very webby: You can literally deploy it with FTP. Every common web browser renders AMP files without modifications
• Actual restrictions are very narrow. Basically all HTML (or equivalent custom elements) and all of CSS (with only limitation on animations)
• But there is a big BUT
• AMP doesn’t allow JavaScript programming for page authors. Hello JSConf.
• I’d like to spend some time on the why.

• The reason is what we call the coordination problem.
• A modern web page has many things going on.
• Lots of these things come from various third parties.
• And they are uncoordinated.
• On the other hand there are some hard limits to making a performant web page:
• The RAIL model: 16ms per frame and 50ms from idle to response.
• It is practically impossible to hit this when 20 random things on a page might take up CPU at every instant without any type of coordination.
• Initially AMP took the easy way out.
• No JS, no coordination problem.
• It still allows third party frames, though. So in some extent the problem was only removed from the critical load path, but it still exists at runtime.
• There are also some very legitimate use cases for JavaScript. As some of you might agree. Even beyond ads, modern journalism can include awesome interactives.
• 2 years ago Ray Daly, at this very conference presented on “JavaScript Journalism” — and we absolutely want to continue that being a thing.
• For the rest of this talk I want to talk about how we are going to bring back JavaScript Journalism into AMP — AND — fix the coordination problem.

Coordinating analytics

• Not as visually interesting, but I’d like to talk about analytics:
• Who has been through the following scenario:

• Business person to engineer: “Yo, we’d like to switch to this new analytics provider. They have much more awesome metrics.”
• Dev: “Cool, looks easy enough, I’ll put it in”.
• 3 days later, business person: “Uhm, on this one metric that is important to my manager we are going slightly down on this new analytics thing. Can you put the old one back in, just for this one thing?”
• Dev: “Grumble. Whatever.”
• And just like that your site ends up with literally all of them.
• For AMP I was like: “What if we instrument pages just once and allow configuring the collected data to be sent to N analytics providers?”
• So, we went and talked to many of them. And every single one was like: “That sounds like a great idea”.
• With this launching really soon in AMP, if you have 15 analytics providers your page will be exactly as heavy as if you only had one.
• With this we got analytics under control, but there is so much more to the page.

A general solution for the coordination problem

• I’d like to talk about my vision as to how to solve the coordination problem in a more general fashion:
• Do some of you still remember this:

• Until the late 90s (Before OSX), Macs ran with cooperative multitasking.
• The thing about cooperative multitasking is that it can be super fast.
• The currently active process gets all the CPU for as long as it wants.
• That can be awesome. E.g. for Games. No chance of some stupid background service slowing things down.
• But sometimes your program might rely on something else running. Or you just want to be nice to other programs.
• So, you yield.
• Now you won’t get back the CPU until something else yields. Which in the worst case is never.
• In practice this did not work very well.
• Which is why modern operating systems support preemptive multitasking where a scheduler controls who gets how much CPU.

The web works exactly like old MacOS and Windows 3.11.

• If an ad says: I’m gonna be all fancy and ray trace myself. It can do that. And stall your UI thread for seconds.
• This actually happens: You thought it was a cool hack someone wrote a H264 decoder in pure JavaScript? Ads use that to implement auto play video where it would be not allowed by the browser.
• So, we’ve solved this problem before. In a preemptive multitasking system a h264 decoder can totally run while an app remains responsive.

We need to bring the equivalence of preemptive multitasking to the web.

• There have been many talks at JSConf that talk about threading and how we need it to get AAA games into the browser.
• I don’t know about you, but I’d be pretty happy if I could just read a simple article without my browser falling over.

I believe that we’ve seen concurrency on the web in the wrong light so far.

It isn’t only needed for fancy games and graphics.

We need concurrency to make the web not suck.

• At this point I do not have all the answers, but I do think that we need to re-think the concurrency primitives that we do have on the web in the light of using them for, you know, every day web pages.
• We have web workers for shared nothing threading in JavaScript.
• They aren’t used much.
• And they are still lacking a model for things to work together.
• My plan is to use the AMP Project to bring a feasible concurrency model to the web and at the same time use this to bring interactivity to AMP.
• Want to animate the page? Cool, write code in this worker.
• If there is an ad on a page and wants to run JS? Cool, run it in a worker.
• AMP can start being a scheduler. Basically it would tell a worker: Hey we want to draw a frame and then the worker can do that.
• If it uses too much CPU the scheduler can just decide: I’m going to throttle you to 30 FPS. Or: I’m going to suspend you altogether.
• If an ad is misbehaving or there are too many ads on a page for a particular device: Again they can be throttled.
• But on a high end device everything runs at 60fps.
• We still have big question marks with respect to the programming model that would be used inside those web workers.
• We’ll be conservative about it.
• Most likely initially we’ll allow workers to send a set of mutations once per animation frame.
• And we’ll limit what things can actually be done to things that can be accelerated on the GPU.
• But eventually as we learn more about concurrency on the web we can open things up.
• Eventually I could totally see, that we’ll have AMP pages that get updated through React components living in web workers.
• So, here is my pitch:
• With AMP we brought the web to a known basic state that cannot be slow.
• We reduced the capabilities while doing it.
• Like, you know, take out JavaScript.
• Over the coming months (and I assume it’ll be more like 12 months) we’ll bring it back.
• But we’ll do so in a fashion that will not regress performance.

The solution is coordinated, scheduled, preemptive concurrency for web components.

• It’ll be awesome.