The best approach I know to turn the bad systems into good ones is called Technical Coherence and the full write-up is here.

Understanding ‘Technical Coherence’ was originally published in Dangerous Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

A preview of the first chapter is below

Your company might run out of money.

This is a foundational truth of business and a reason for your CEO’s restless nights. Your job, as CTO, is to help remove your CEO’s worry and to ensure your engineers take the least time to deliver the maximum value.

Some lucky startups do everything wrong and still have revenue. Trying to copy their internal processes is useful only if we’ve also got their luck. For most of us, at most companies, even doing everything right may end with us closing our doors. Therefore every executive needs to ensure their function of the company costs less than the value it delivers. If we can’t do this, we’re not yet operating at the executive level; we’re just a really senior manager.

Doing this is hard for executives of any function but it’s super hard for whoever leads Engineering. The work that Engineering does only results in revenue after it passes through many layers of abstraction and ultimately through Sales, Marketing and Customer Support. So even identifying the value that Engineering provides is a herculean task, to say nothing of trying to increase it.

And yet, that’s the job. To ensure Engineering is delivering for the company and to prove that to some degree. Enough for the CEO and your peer executives to not worry if you’ve got this handled.

You might be tempted to borrow the habits of large, rich, established companies here. If you have a monopoly on hardware or ad publishing or defense contracts then it’s best to use the practices entrenched at Apple, Google, and Microsoft, respectively. If not, you’ll need to be much more careful about understanding what you’re doing, what you’re not doing, how it connects to the business plan, and how it’s all going at any given time.

Continue reading

Book launch – Executive Engineering now available was originally published in Dangerous Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

But effective engineering is a collaborative and creative discipline, where the team is the performative unit rather than the individual, and the performance cannot be measured by the team’s actions but rather by the outcomes of the system that the team produces.

Like any heist film, a software engineering team comprises multiple individuals with relative strengths and weaknesses that complement each other. One engineer might love to make new features while another might be entirely consumed by improving the quality of the team’s existing software. Either one working alone, given enough time, would eventually become entirely useless to the company. Together, as a team, they can produce value while lowering cost indefinitely.

Attempting to reward or discipline either of these two individuals would throw the team itself off balance. Incentivize new feature addition and the quarter-over-quarter feature development speed will consistently slow. Incentivize improvements and no feature will ever again launch. But what if we reward them, individually, for their specific and different contributions? Yes, we can do that and all we’ll need is to create a metric for the engineer who likes to build features and another metric for the one who fixes existing features. Then measure each engineer by the appropriate metric.

But what are the units of that measurement? How do you compare 6 clumsy features launched in a month against 1 extremely hard to find bug that took all month to discover and required only a one-line change? What if an engineer did neither of these activities but mentored their colleagues?

It’s impossible to say one person performs better than another without a shared measurement. And, while the outputs of these two hypothetical (yet all too real) engineers cannot be compared, their daily activities can be. Number of changes to a code base. Number of lines added or removed. Time spent in the office. Number of messages sent to colleagues. Number of bugs noticed by a QA team.

I know of companies that have, at one point, used all of the above to measure the ‘performance’ of their software developers. And, in each case, the company got a lot of useless (or harmful) activity and enormous employee attrition.

So, if we can’t compare engineers against each other without incentivizing the wrong behavior, we must accept that their success or failure is inherently collaborative. Like a surgical operating team in a hospital whose success is tied to the patient’s. Or a unit of firefighters who enter a hot building together.

One would naturally, then, try to measure the performance of the team instead of the individual.

But software is a unique art: It’s the encoding of human decision-making into machines. The performance that matters isn’t that of the team that builds the software but of the computers themselves.

Imagine if the surgical operating team in a hospital were a robotic AI. Or the firefighters entering a 4-alarm blaze were fire-retardant autonomous robots. We would reward the engineers who created these tools for the outcomes of the surgeries and the lives saved from the fires. No one would ever care if the engineers worked late or took extra vacation — what matters is the outcomes.

All software engineering teams produce these same outcomes, and always indirectly. The way the engineers manage their software is much like how a sales executive manages their team. Throughput, latency, and quality can all be measured and charted over time. A sales leader can identify problem areas and intervene when an individual account executive underperforms. Engineers do the same thing but call it ‘debugging’.

This inherent indirection — that the engineers produce the system that produces value — is what makes software engineering a creative discipline.

Like a gardener thoughtfully removing a blight, any engineer at any time can increase the value of the company in unexpected ways. Or, like a gardener rewarded only for the sheer number of weeds removed, we can task them with goals that produce huge activity while the company gets outpaced by competitors with more creative teams.

The solution is to define what, precisely you want from your engineering teams. It’s never sheer activity. And it’s likely not blindly implementing whatever they’re asked to build because that’s how consultancies operate. And a consultancy only makes money because it doesn’t have to deal with the inevitable failure of most of the things they build for poorly-informed clients.

What you want from your engineering team is to produce the maximum number of market-impactful launches. It’s on your product leadership to effectively measure the market and maximize what that impact is. But it’s up to your engineering leadership to maximize the sheer number of launches that your product function believes could be useful.

Define those top-level metrics, install competent leaders with an unyielding devotion to coaching and developing their people, and watch as your engineers work together to make something impossibly beautiful.

Whatever you do, don’t create metric to measure the performance of your individual engineers.

You must not measure individual software engineer performance was originally published in Dangerous Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most software engineering literature assumes you need to process data efficiently. The algorithm-focused, Google-style CS questions that dominate software interviews put particular focus on this — we test candidates on code that works well when data size increases.

In my time leading engineering teams at Square and Gusto I’ve found that this big-data approach to software engineering is a poor fit for many product companies. Rather, product scalability problems are along a different axis: Sprawling domains and massive schemas implementing those domains. These companies face three very specific, very hard problems:

• Correctly modeling the domain
• Creating clear interfaces between parts of the system
• Finding ways to continually accelerate product development speed

Secure, small-data product engineering

This problem isn’t unique to Gusto or Square — there’s a huge set of companies that have small amounts of critical data in complex domains. It’s every fintech company, every product company with a heavy operations arm, and any company trying to model an existing regulatory system.

Here’s a highly inaccurate but illustrative plot of how companies fall on this tradeoff of concerns:

If we’re building Twitter or Instagram we’ll need to process hexabytes of simple information. If we’re building Flexport or Gusto or even Square¹ we’ll be hard-pressed to even find terabytes of information. The scalability challenges are still very real, they’re just along the axes of domain complexity and security.

Welcome to small-data product engineering

If you’re at one of these small-data companies you can’t mitigate the core scalability problems by optimizing source code. Your company probably has enough folks working at that low, focused level. They need people to zoom out and address the whole system.

The three levels of engineering sophistication

There are three levels of sophistication an engineer grows through in their career:

1. Working with code
2. Working with applications
3. Working with systems of applications

#1 Working with code

When we start our career we work with logic in a single method, function, or class. The skills at this level are extremely basic: Editing source code, expressing basic concepts in the given language, storing basic data, etc. This is where features are made.

#2 Working with applications

When we become senior we can work with a whole application at a time. We store lots of relational data and model complex business logic across many files and directories. This is where products are made.

#3 Working with systems of applications

And then we become grizzled veterans working with whole constellations of applications in multiple environments. At this level we perform major migrations, simplify dependencies within the distributed data model, and shepherd the whole multi-application system. This is where companies are made.

The way we work at each level is quite different. The phrase ‘refactoring’ usually applies to the level of source code and, less commonly, at the level of an application. The phrase ‘tech debt’ is more common for working with an application as we handle major framework upgrades and make the data model more expressive.

But at the level of systems both ‘refactoring’ and ‘tech debt’ are inappropriate concepts. At this level of the system there are only two activities you should permit yourself to do:

• Bring a system in line with its design
• Design a replacement system

Anything else is busywork.

Both of those activities assume that the thing we’re building is designed. For this reason, if you feel like a senior engineer your job is now primarily an engineering designer. You should design and help implement software systems that solve the Product team’s current and future needs. If you imagine your job is just the implementation you’ll miss the design step and the overall system won’t make sense.

Levels of PM sophistication

Speaking of Product, these same levels of sophistication apply to PMs. Early-career PMs can launch and steward features. This is valuable and helpful, though at this level the PM can introduce tension between product and engineering. Shipping a single feature quickly is just about never the best thing for the long term health of the interconnected product suite.

Mid-career PMs can launch and steward whole products. This is hugely valuable and here the tension between product and engineering is eased. The PM takes a long-term view of the whole feature-shipping cadence and can work with engineers to make calculated investments to increase shipping speed.

Late-career PMs can launch and steward entire product suites. At this level the PM takes responsibility for an entire ecosystem of engineering and product. They are natural allies to engineers and will often push for systemic simplifications and increased coherence between parts of the product suite — something that allows engineers to make dramatic simplifications to the underlying system.

Growing technology at a product company

If your product’s production dataset is smaller than a terabyte then you have no data. None. 100% of your data can fit on two iPhones. The solutions from Google, LinkedIn, and other big companies are useless here. Your peers are Square, Flexport, Gusto, Stripe, Coinbase, and any place with huge amounts of logic but small, critically important pieces of data.

Your problems give you four focus areas that are more important to you than to the average company:

• Prioritizing domain model correctness over computational efficiency
• Refactoring the data model early and often
• Ensuring data is encrypted by default everywhere except in your product’s memory space
• Separating data by sensitivity (PHI, PII, PCI) to empower your Security team

And straight-up disregard any blog post from a company that has any of the following:

• Lots of data
• A ridiculous amount of money
• Low security restrictions

Twitter can lose a tweet. LinkedIn can lose a ‘like’ on a post. Square can’t lose a payment. Gusto can’t lose an IRS filing.

So your problems are different. Instead of using something that works well when you process trillions of records per minute you need to use tools that are simple to operate and that support encryption powered by role-based access control (i.e. if service A produces data then service B can only decrypt A’s data if it’s allowed to).

Other small-data, security-conscious companies have solved these problems. But as of yet none of their in-house tools for securely handling data in prod have been open sourced². So you’ll have to just build something very simple and design it carefully.

How to select technologies

Which brings us to the rule for when to build technology in-house or use something that exists externally:

“If a problem is unique to your product offering then you may invent an appropriate solution. Otherwise go with an industry standard approach — and only use one approach at a time.”

Invent software that extends your competitive business advantage. And for everything else just use off-the-shelf tools. Your deployment system should look eerily similar to one of the top ten results for a generic search like “AWS deployment EC2”. Your cloud networking setup should look identical to the AWS official standardized architecture for PCI compliant systems. Your database tables should use the third normal form.

Performing Migrations

Note that last phrase in the above rule: “–and use only one approach at a time.” You’ll need a way to move from one version to another. This means you and your team need to get good at performing migrations.

Whether you’re making a tiny change to a database column or overhauling a major system all good migrations have five stages:

1. Design the v2 and the path to get there
2. Fix v1
3. Implement v2
4. Carefully migrate from v1 to v2 in atomic, safe steps
5. Delete v1

Most teams forget steps #2 and #5. If v1 is poorly factored you’ll inevitably have to make significant changes to v1 before it’s in a shape to be migrated. Since we need to do that work anyway we might as well stop pretending v2 will rid us of v1. Fix v1 in place before you start building v2.

And don’t accept any migration that doesn’t have, as its last step, an act like “drop these tables” or “delete this code”. If you can’t actually replace the current thing don’t bother with the migration. Do it right or don’t do it.

What teams should ‘own’

Some companies allow teams to carve out juicy projects for themselves and prevent other folks from working on them. This kind of technical bureaucracy cuts through engineering morale like a hot knife through butter. Teams shouldn’t claim projects, they should claim problems. And they should accept as much help from any direction when solving those problems.

Don’t say “We’re going to roll out a new frontend technology” when you can say “We’re going to solve that adding new storage classes in the frontend is hard”. Then you can have an open debate across all interested stakeholders in how that problem should be solved. Once there’s agreement you start work and you gratefully accept commits from everybody.

No architecture astronautics

As Alyssa Henry (founder of AWS S3, Head of Engineering at Square) sometimes says: “The ideal system in your head is never as good as the working system in front of you.” Beware of architecture astronauts. And be even more wary of becoming one.

You can identify architecture astronautics when someone (including yourself) starts to believe that there’s a technological solution to poor design.

If I create a React component that mixes a signup form, account approval, and uploading a user’s avatar all in one Javascript function that’s just poor design. I cannot fix this problem by porting it to a new Javascript framework.

If I create an HTTP endpoint that performs five separate operations and also calls out to a different service multiple times — that’s poor design. That cannot be solved by introducing new service boundary technology.

The work of building long-term value is really hard but really simple: Decide what it is you’re actually trying to build and then implement that in the simplest way possible.

How to prioritize systemic fixes

Imagine putting sand into a jar, then pebbles, then big rocks. The big rocks don’t fit.

We have to do the big rocks first. When the right solution to a problem takes 10 engineers for a year then put 10 engineers on it for a year and make sure it’s solved well. And if it takes longer let it take longer. Don’t put 3 engineers on it for a quarter. That just adds sand to the jar. Small wins are great and easy to prioritize but at some point you need more in the jar than just sand.

Building a product platform for your customers

See: The four interfaces of SaaS product suites

If you have any customers then you’ve already solved one of the hardest problems. Now you can create and sell other products to your existing customer base — massively increasing the value of your company and your competitive advantage against competitors.

Unfortunately, you may have built your entire engineering system to support just your first product.

Service classes are an encapsulation of logic that separates the ‘what’ of its purpose from the ‘how’ of its implementation. Building a platform that allows you to rapidly compose new products for your existing customers is merely the small matter of creating interfaces that perform valuable product operations. Imagine a hundred single-purpose APIs in your system that allow you to move money, sign up customers, list resources belonging to those customers, etc. all encapsulated such that you can use each one in isolation.

Creating those APIs will allow product engineers to build products quickly on top of them, while folks working on the underlying platform carefully move data around behind the APIs. The two tracks of work don’t need to be coordinated — the API gives us a decoupling.

You can move from this:

To this:

You’ll know you’re on the right track if your product engineering teams spend less time coordinating with whatever teams work on the internal platform. An interface replaces enormous amounts of tightly-coupled planning.

How to think about security

Now that you’re working at the level of a whole system, let’s address security. No matter what engineering roles you’ve had you’ll do great security engineering work if you keep this one rule in mind:

Simplicity, correctness, security, scalability, and maintainability are all neighbors. When we’re far from one of them we’re far from all of them.

Twitter had trouble scaling in the early days. Nick Kallen popularized the idea that you can’t make something scale by adding “magic scaling sprinkles.” The same is true for correctness, maintainability, and security. You can’t bolt on correctness to a busted system. You can’t just “do security” for a quarter to prevent security breaches.

Security is, more than anything else, a result of a simple design implemented correctly. Something is either simple and obviously secure, simple and obviously insecure, or it’s too complex to ever make secure. Your security team can only help you after you’ve refactored your system into a thing that’s so obvious any new hire can immediately make sense of it.

If a company is valuable with small amounts of data then that data is likely very important. That means a data breach is worse for you than for other companies. Leaking sensitive data functionally ends your company . That’s bad, but the damage to your customers is even larger. Your data is people’s home addresses and names and bank accounts and personal preferences. A breach in your system means that data is now in the hands of the highest bidder. If you have git access to a system that might expose sensitive customer data it’s your professional responsibility to fix it. Don’t wait for your manager or CTO or PM to ‘allow’ you to fix it . Just fix it. These are people’s lives.

How to prioritize product work versus maintenance

There are two investment models for engineering:

1. Creating new value
2. minimizing the cost of existing value

Product engineering uses #1, platform/foundation/infrastructure engineering uses #2. If there’s a single leadership team trying to prioritize projects from both groups they will always pick maximizing new value. They’ll do this until the cost of that value is so high that it wipes out the gains.

The way around this is to determine how fast you’d like to go now versus how fast you’d like to go in the future. And use that model to fund the percentage of people you put on increasing value versus minimizing cost. These need to be separate leadership teams, each focused on just one of those investment models.

For many companies it works out like this:

• When you’re starting out you want 100% of people maximizing value because there may not be a tomorrow, much less a next year.
• When you’ve reached product market fit it’s important to switch this up and see if you can lose a year of value creation to gain a year of cost reductions — earning yourself a step-function for future growth.
• When you have a shot at becoming ecologically dominant in your industry it’s important to put nearly 100% of people on cost reductions until the competitive moat around you gets massive.
• Then, if you’re ecologically dominant you can add huge new headcount to do value creation.

This is a slider you can play with at any time to match the risk model of the company. But being unaware of the tradeoffs means you’ll either reduce cost without producing value (see: 90% of infrastructure/tools startups) or increase value until you choke yourself and a leaner competitor comes through and steals all your victories (see: every company that Amazon has annihilated).

If you’re an engineer on the side of the org that maximizes value then one of your main jobs is deciding when some component is so important that its cost needs to be minimized. Then you have to productionize that component and hand it over to the other side of the org.

If you’re an engineer on the cost-reduction side you need to consolidate all the technologies and make the whole system hum along with zero maintenance. You also need to help product engineers upgrade their best stuff until it’s in a shape that’s maintainable.

How to extract services from a monolith

Service extraction is now an entire field of engineering. But the basic rule here is simple: You cannot run from your poor domain modeling. No matter what application you send your poorly-factored code to the pain will only increase.

The problems with “let’s extract this into a microservice” are (1) there’s no such thing as a microservice and (2) you need to fix the data dependencies first or you’re in for a world of hurt.

That said, there are some simple patterns to follow that will help you as you extract things from a monolith into new applications (or parts of the monolith that are more separate).

Extract verbs, not nouns

You cannot extract User to a service. What would that service do? Just give you the data from the users table? Sounds like you’ve just moved a database table super far away from you. Have fun with that.

But if you extract the concept of, say, approving a user to a service then you’re simplifying things. The monolith used to know all the steps of how a user was approved. Now that you’ve extracted all that logic and the intermediate data to perform the approval, the monolith can just have an approved_at timestamp. If anyone wants more data on the approval they can go to your new UserApprovals service to look up the details.

Learn in public and only do high-quality work.

I’d like to leave you with one last exhortation. The best engineers I’ve ever worked with (like these) have two qualities I’ve observed:

• They never, ever do work that’s less than their best
• They learn in public, never hiding their process or their mistakes.

I’ve noticed that each engineer I know has roughly a specific speed at which they work. No matter how many hours we clock in our career the speed doesn’t really increase. And when we work at a higher level of quality the speed takes a hit, but only briefly. Then you’re back to full speed but you’re better.

Multiple times in my career I’ve made the decision to adopt a higher bar for my own work. Each time it slowed me for a bit but then I sped right back up. This, as far as I can tell, is how great software engineers get that way. Their brains aren’t different than yours or mine, they just have higher standards for themselves.

To that end:

• Put as much effort into naming things as possible. Never cut corners on naming, it’s basically the whole job.
• Always write code comments that are as expressive and clear as possible, including ASCII art drawings if necessary. When you edit any code, update the comments. This is more work but it’s higher value. Do the highest value work.
• Don’t leave TODOs in code unless you’ve tracked a ticket and link to the ticket in the TODO.
• Never fix a bug without fixing the underlying design flaw that allowed it to happen. Even if the real fix takes two weeks, do it. Any place that doesn’t let you fundamentally fix the system that you’re in charge of is a place you don’t want to work.
• Make data migrations as freely as you make code changes. If someone looks at your database they should not be able to guess at what your code used to be.
• Before you merge a PR ask yourself “If I had more time, could I do this better?”. Take the time, do it better. Anyone who’s frustrated by that can be frustrated. They’ll be very, very happy in 9 months when the software works well and you’ve grown a lot.

Many thanks to my dear friends at Gusto for my time there and for the opportunity to better articulate my approach to technical leadership.

[1] During the time of the Square IPO all of the data for the dozen-plus products still fit into a single MySQL database under one giant Rails app.

[2] Please tell me I’m wrong.

A reference guide for fintech & small-data engineering was originally published in Dangerous Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

I get it. When I read the following words it felt like I was reading someone who sees things like I do:

I value diversity and inclusion, am not denying that sexism exists, and don’t endorse using stereotypes. […] If we can’t have an honest discussion about this, then we can never truly solve the problem.

This sounds sincere and, what’s more, it reminds me of what other guys have told me for years: That they have a slightly different perspective on a current political trend but are afraid that if they attempt to share it they’ll be shamed back into silence and possibly fired.

The fear is real. Shame, condescension, and mocking are your inheritance when you show up unpopularly in issues of social justice. So instead of talking to the people at work whose hearts are in the right place but don’t know about, say, Berkson’s Paradox (a statistical explanation for why minorities may not appear to be the highest performers at work) you just keep your head down. And then somebody writes a piece like the one linked above and it’s like you can breathe again. We’re finally talking about it!

Thoughtcrime versus scaring people at work

Here’s the problem: If you have a controversial opinion and you express it alone in your home or to trusted friends that’s obviously fine. If you express it in public but in a way where it causes no harm that is fine. But if you have a controversial opinion that, by its very nature, sparks a conversation that causes people harm, then it causes harm to express it.

That last sentence is a deliberate tautology. It’s up to our communities to decide what to do about the damage, but some discussions are harmful, and those who push for them bear responsibility (regardless of if the majority opinion where it’s taking place supports or opposes the main points).

“That’s absurd,” you say. “How can a rational discussion cause harm?” Again, I get it. That does sound absurd on its face. And it is absurd as long as you’re seeing it from the position of someone with power. If you lack power, then it becomes clear how mere conversations cause harm.

Imagine your name is Mike. You’re nine. Two bullies several years older have you on your belly in an alley and one holds a knife. They begin to calmly, rationally have a discussion about whether people named ‘Mike’ are too populous and whether the world even needs you. They, precociously, pull out bar charts of name distributions over time to show that your name is well outside the standard deviation of use. They discuss the Hebrew and Aramaic origins of your name and wonder if you, a WASP kid, really even came by it honestly. The knife twirls.

Maybe these two bullies have a low heart rate and low levels of cortisol (a stress hormone). They are relaxed. It’s a totally rational discussion! We’re just talking about facts! Why are you panicking, Mike?!

That’s effectively what we’re doing when we debate, say, whether women are less inclined to computer engineering. There are two separate things happening.

The man at Google was fired not because of the discussion but because of the knife-twirling. Twirling that he wasn’t even aware he was doing. I sympathize with those who are baffled and hurt by the firing — I really do. Because not being able to discuss what we think and why we think it is a giant problem. We absolutely need to have good science on what’s really at work here. But carefully. And not at all like this.

How is a discussion hurtful?

Just right out of the gate stereotype threat means having the discussion in public is going to make it harder for any minorities mentioned to summon the energy to do their job. Would you fire somebody if they went from desk to desk distracting hundreds of employees for personal reasons?

Then the document conjoins discussion of race, gender, political leaning, service oriented architecture, the role of data in decisions, and epistemological closure. All of that should be discussed but each discussion is very different and must be handled separately. To mix them is as if the writer had said “Black people might not like to code because of genetics and lunch should be moved back one hour” and now it’s impossible to discuss appropriate meal scheduling. Because even the fact that we’re accepting the invitation to discussion sends a signal to all readers that we might take the rest of the claims seriously.

So, no, we can’t talk about the document at all as it’s written. Furthermore, we can’t have men talk publicly about their opinions on gender like this. Is that super frustrating? Yes. But it’s because women face a very real presence of physical violence when they are visible in digital communities. We men face shame and firing if we say the wrong thing. Women face the same plus rape threats, death threats, and all kinds of sustained harassment. So women can’t speak up safely and therefore they would have to watch their male colleagues discuss how a woman’s brain determines her interests. How impossibly maddening that would be.

Also racism. Yes, racism. Really. Yes, that racism.

And I didn’t realize that the Googler’s manifesto had racial implications until it was pointed out to me. Give me a second to convince you because I think you’ll see what I see. Here’s the argument laid out at the beginning:

* The lack of discussion fosters the most extreme and authoritarian elements of this ideology.
* Extreme: all disparities in representation are due to oppression
* Authoritarian: we should discriminate to correct for this oppression
* Differences in distributions of traits between men and women may in part explain why we don’t have 50% representation of women in tech and leadership. Discrimination to reach equal representation is unfair, divisive, and bad for business.

[emphases mine]

Do you see what happened? This person started to talk about diversity initiatives and how maybe not all disparities in representation are due to oppression. And then focused on gender. But a huge amount of diversity works to bring people who’re members of other minorities into tech.

I’m going to give the author the benefit of the doubt and say that questioning the need for diversity because of gender essentialism and thereby implying there may also be racial essentialism was inartful, not ill-intended. It was still, however, harmful. It was, unknowingly, knife-twirling. The author doesn’t have to think about how careless words might cause harm because nobody holds a knife over him (well, maybe now he thinks about it). But to publicly ask whether the reason that Latinxs and African Americans are underrepresented is due to innate bias — and state that we majority-ethnic-group men would like to publicly discuss our opinions on the data — is a discussion that causes needless harm.

This carelessness makes it impossible to talk frankly about issues of race and gender because folks who feel safe at work don’t realize that what’s an interesting intellectual issue for them is sometimes quite literally life and death for women, people of color and other minorities.

(While we’re at it, let’s make sure we’re all on the same page about the Google hiring bar: Google doesn’t discriminate when making job offers. They discriminate in getting traditionally-underrepresented people to apply and into the start of the hiring funnel. It’s a way to fix systemic biases in who normally would consider applying to Google. This is an important distinction.)

But what if I really want to talk about it?

How we discuss this is important and can’t be decided unilaterally by a person who doesn’t understand the stakes. Broaching the topics of essential racial and gender preferences is going to cause a bunch of people to be so on their guard that they cannot do their jobs.

If I’m a cisgender straight white male who works at Google doing software engineering, the discussion I want to have is not possible as I imagine it. What I think is an abstract, rational discussion is so incredibly emotionally charged for others. When people seem to overreact I’ll say “They’re not thinking clearly”. I’ll link to a study showing that men’s brains are larger than women’s. The knife twirls.

How we can have this conversation

You have insight into how we should be doing diversity. Your unique background gives you that. Everybody has something to contribute. If we want to share our opinions we must first understand that we’re a person standing on a shore attempting to converse with someone drowning. The fact that women and people of color in tech have held down their jobs and educated us about diversity and endured social biases against them all at the same time is amazing. When we initiate discussions on diversity it’s like we’re trying to convince the drowning person that it’s best to wait for the low tide and they’re there, gulping water, carefully rebutting our proposal, hoping for a rope. It’s nothing less than supernatural what some of our bravest and most vulnerable colleagues have endured.

So can we talk about this stuff? Yes. But we have to talk to the people who’re involved, not just talk about them. The conversation has to start in person and only with people who already know they can trust you. Any of us who want to discuss the capabilities or preferences of minorities must first see the biases at work in our industry and then show that we see what people put up with. That’ll build trust. Then we’ll have a real discussion. It’ll be so good.

How we can never have this conversation

Publicly, with only majority-culture men participating. We’ll learn nothing and we’ll cause so much damage that we’ll never hear about because people won’t trust us enough to speak up about their pain. They’ll just leave.

Which is precisely why we have trouble retaining women and other minorities in this field. Not because they can’t do the work, they just don’t want to do it while putting up with the environment we’ve created.

If you want to debate the Googler’s Manifesto and you’re also a good person was originally published in Dangerous Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

There needs to be a handbook for migrating a monolithic application into a service-oriented SaaS product suite.

The same things keep going wrong at every company: Moving code without moving the data that code needs, language proliferation, conflating synchronous and asynchronous work, not anticipating the need for backfilling data, etc.

This post is a guide to help navigate just one of the problems your team will face: Picking the right technologies for your four different interfaces.

Four? Yeah, four. A SaaS product suite eventually implements four interfaces for four different kinds of clients:

1. The one your customers use
2. The ones your employees use to manage customer data
3. What you offer to third-parties to extend the value of your products
4. The internal server-to-server connections that power your SoA

But that’s not how it starts.

When you’re first building your system you probably have a single application. Customers connect directly to this and it’s… well it’s your whole product. This is the code that has access to all of your data so when you need to build an admin view of your system you build it more or less right there where the data is.

This works great for the v1 of your admin interface. You didn’t have to duplicate a bunch of work and you can operate your product.

When you need a 3rd-party API you might build it on that same application as the other two interfaces or maybe you’ll put it off to the side because you’re starting to decouple your growing monolith.

By then you’ll have something like this:

Which is… well, it’s starting to get complicated. Those circles in the middle represent different applications. Managing all of those is hard. But what’s really going to bite you is that the authentication and authorization logic for at least two very different customer types is coupled with your legacy code in the original application. How do you provide an admin interface to these other applications that are showing up?

Well, you could add an admin interface to each application.

But this means you need to implement that authentication system in multiple places. And, depending on how you’ve built the admin UI, you may even need to ship the admin views in multiple applications.

Service-oriented architecture is notoriously hard to get right. Not because the technology is difficult to implement (it’s now a solved problem). SoA is tricky because when you have many places to put something it’s costly to have anything in the wrong place.

In a monolith putting some code in ‘./lib’ or ‘./models’ is mostly an aesthetic difference. But if you put some code in, say, the customer lookup service when it actually needs to be in the highly-available report-generating service you might have destabilized the whole system. If you’re moving to multiple services you’re not simplifying your system, you’re making it more complex and hoping it’s worth the tradeoff.

There is some core functionality in your product. It does something meaningful. If you conflate the different interfaces together you may have to implement that meaningful core functionality up to four times. But if you can clearly separate the authentication you can implement your core product value just once and provide windows to it through your authentication layers.

Put another way: if you can separate “who is this and which features can they use?” from the feature implementation then you can put all your effort into making the features themselves really good.

Four technologies for four interfaces

A common mistake when rolling out one of these interfaces is to reuse the technology from one of the others. Just because, say, HTTP + JSON is how you communicate with your customers doesn’t mean it’s the right choice for your internal service-to-service calls. And no matter what the other interfaces use your admin interface pretty much has to be a plain web app because that’s the cheapest thing to maintain.

Let’s review the needs of the different interfaces to help us pick a good technology for each one.

1. The first-party customer interface

The first API is the one you already have. No matter what your product does there is a frontend to your system that your customers use. This could be an HTTP-based HTML or JSON frontend. If you offer an email service maybe it’s over SMTP. If you’re Dropbox this is your own proprietary sync protocol. Whatever the transport system, your customers use your product through some kind of front door.

This API is one you control both sides of. You change server endpoints and client/display behavior in lockstep, creating a coherent user experience.

If you find your team calling this ‘the api’ you might be in trouble. There are three more to go.

A good technology for this is anything that your customers can use. This is where you sweat the UX until the customer experience is straight-up joyful.

The authentication layer for this needs to identify a single user and enforce that any data passing out of the system is data that is specific to that user.

In terms of actual implementation you have two choices:

• You can have a highly-available frontend that performs the authentication and passes credentials along to your product code
• You can have a highly-available authentication service that each service uses when responding to user requests.

In either case the authentication for a customer of your product is located in one place and separate from the actual features of your product.

2. The Admin Interface

Most companies build a half-assed version of this at first to gain any visibility at all into the system. Over time the admin interface becomes generally useful: Engineers diagnose production data problems with it, operations teams build their core workflows into it, and it becomes the primary tool for customer support.

The authentication layer here is more robust and needs to integrate with your IT system. You need to ensure only people who are currently employed at your company can access this and you need to audit everything anyone does — because through this interface a user can access the data of any user in your system.

The most scalable, sustainable way to build an admin system like this is a web application separate from the product services by a routing layer. This routing layer both performs authentication and translates HTTP+JSON or whatever the new web hotness is into standard service-to-service calls.

3. Third-party REST API

Most products will, at some point, benefit from offering a third-party API. Your customers get more value from you when they can opt in to extensions of your product.

This API basically has to be RESTful HTTP. That’s the standard that most other developers understand and it’s the easiest to document. Tools like OpenAPI let you generate clients for it automatically in any language. This interface must have clear versioned releases (with old versions supported indefinitely) because you don’t have control over the clients.

The implementation will look an awful lot like the admin one but with a different permission model. The routing layer will need to check if the 3rd party making a call is allowed to access a specific user’s data. This requires a complex system of registering 3rd-party applications, generating tokens, allowing users to enable/disable individual apps for their account, and rate-limiting. All of which would really complicate your product if this kind of logic leaked into core features.

4. The internal SoA

The fourth interface is fully internal — it’s the inter-process communication between server-side systems. You can use any technology for this but some are much friendlier to your engineers than others.

RPC: Verbs over Nouns

REST is all about nouns and it forces all operations to be reasonably simple. When you don’t know who’s using your API that’s a good fit. But when the other users of your API are your colleagues you need something more sophisticated.

An RPC (‘remote procedure call’) system allows your team to express their full intent, iterating on the API between services at the same cadence that they iterate on method and class signatures.

Structured schema

At some point your system becomes too large to run all of the different applications on a laptop. Before you reach that point you’ll benefit from implementing structured, strongly typed APIs between your internal services. You want to be able to automatically generate running mocks of dependent services (maintaining them by hand will fail) and you need to rely on the API type system in your unit tests to catch bugs. Something like Thrift or Protocol Buffers or Cap’n Proto will give you this natively.

Forwards- and backwards-compatible

You’ll make gradual changes to this API forever. How do you ensure that deploying one part of your system doesn’t introduce an API incompatibility between the new and the old code? How do you ensure that you can always read data encoded in a version of the API from last year with today’s API?

Tools (like the 3 mentioned above) ensure you can always read new data even in old clients (and vice versa) from any language.

Mutual SSL and Access Control Lists

Authentication and authorization for your internal interfaces is very different than the other three. You still need to ask “who is performing this action?” but the ‘who’ here isn’t a person, it’s a service. So you need some way to identify services (authentication) and some way to permit them to do stuff (authorization).

Here there’s likely an ACL where specific services are allowed to use specific endpoints. Defining permissions at this level of granularity is tedious but it lets you easily verify that low-security parts of your system aren’t gaining access to higher-security parts and it provides the means to generate a dependency graph of your services.

Focusing on your product

When your company is big enough your interfaces become products in themselves. But even while you’re small keeping them isolated at the edges of your system means you can get them right and you can more easily prove that they’re secure.

It also frees your team up to focus on the actual product work. When the interfaces are defined and separated from the core features then everybody — your customers, your employees, your 3rd-party partners, and other products— benefit from every improvement you make.

The Four Interfaces of SaaS product suites was originally published in Dangerous Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.