So my dad is one of those legendary figures in modern science. His H-index is 70+. His research transformed what we think of as hospital medicine, from a series of narrowly scoped disciplines to a systems theory of the human body where the center was critical care of key systems. And along the way, he transformed medicine in at least two countries, Canada and Greece, with his tireless advocacy and work to improve the medical systems either directly or through the people and institutions he built.

Whenever I need humility, I talk to him.

I mean, I have done a lot of big things in my life, but no, I didn’t change the fundamental way we think about how we die (body first, brain last).

In oh-so-many meetings, I stand in front of a large number of engineers and product managers and engineering managers and say, “see this impossible hill; we will climb it.”

And there is a natural inclination, “how?” And “how fast?” And “we have so much insanely hard work to do!”

For most of my professional life, I view that as an affront. That the person asking the question was dismissing the solution because it was too hard. As if there was some easier path that I had deliberately chosen to ignore. I thought they were saying you came up with the wrong answer. And I would get angry, and pissed off, and frustrated.

Recently I took this EQi test. And a key element of that test was how much you used reality to make judgments. And, well, I scored poorly.

And that got me thinking, why? Because I tend to look at reality not as immutable but as mutable.

As someone who has seen how the future changed because of what individuals like my dad did, and more modestly I did, my opinion of the relentless forces of nature and history is that I will always be willing to use my living hand to challenge any invisible or dead hand.

So once again, I was in a meeting where someone said, “this is the moral equivalent of scaling the North Face in a snowstorm.”

And I was like — “Man, you and I are so different, in a world that is 70% water, I look at an empty glass and think how easy it is to fill it.”

And, of course, that went down poorly. Because that person felt I was dismissing their observation. And, to be honest, I was.

And so I told this to my dad. My dad goes, “blah. Filling a glass with water is easy. Go to a lake or the sea. Finding a glass is easy. Go to any store. But make the right kind of glass that fits in the right place. That’s what’s hard.”

And he then looked at me with that look, “Good you can do hard things, I am proud of you.” My dad was never one for recognition, etc. H was always about the next hill to climb. And then we spent the rest of the time talking about how filling that glass is so damned hard.

And it got me thinking a lot about how strategic software architecture is about finding that right glass. And that is hard. And when I find that glass, I am excited, and all I can see is how we could achieve miracles if we just filled it.

But the work has barely begun for everyone else.

And filling that glass with water is hard.

And being dismissive of the challenge and not recognizing that the work has barely begun is critical. And when people tell you, “dude, I am not climbing the North Face in a snowstorm unless you break this down a little bit more,” I shouldn’t get angry; I should be delighted. Any sane person would be like — “good luck.”

Next time, I’ll not stand waiting for the applause and get annoyed that everyone isn’t admiring my achievement of identifying the right glass.

So let me adjust my thinking.

When I find a glass, I know we can fill it with water. The effort to fill it with water in a timely fashion will make the finding of the glass feel like a trivial subtask.

And to ask people to be happy for me for finding the glass is kind of like the tenor asking for applause for clearing his voice. I’m asking them to trust me that I know where the safe path up the North Face is. Maybe, instead of asking them to applaud, I should start preparing for the climb and be grateful that they might follow me.

Originally published at https://wrongtool.kostadis.com on April 10, 2021.

Personal observations about Easter.

In 1994, I was taking a class in Computer Science. And the final project extended over Orthodox Easter.

I was in a quandary. The combination of workload, poor project planning had lead to the difficult decision of what to do. I needed more hours in the day to get it done. If it were just my grade, I would have been happy to take a hit. But it wasn’t; I was part of a group project. And I felt I was going to let my team down.

I went to the professor, and his reaction was illuminating — “Tough.”

At first, I was infuriated because I thought it was a statement about my Easter. Later on, I learned it was a statement of prioritizing my personal life over my work.

It was an essential and evil lesson. My personal life had to take a secondary place to my work life. If I cared about my career, frivolities like friendships and family could not intervene.

And so I missed Easter celebrations.

And I did that for the next 30+ years.

But not only Easter, all sorts of celebrations, because work mattered over everything.

Over the years, I saw this fetishization of deadlines. That was somehow hitting the date at all costs was the right thing to do.

And I have learned something from that experience. Some jobs require that kind of sacrifice, and they have a high burnout ratio. I had one of those jobs. Within two years, I was a burned-out husk, getting drunk by 6 pm to get through the day. Until I had my nerves dulled from booze, I couldn’t relate to my family.

As time has moved forward, I have concluded that jobs that impose that sacrifice have to be a matter of life and death because your life is a casualty of those jobs.

Any other employer who makes that kind of request all of the time is evil.

And anytime an employer makes that kind of demand, it reflects poor project planning.

As a strategic software architect, when I see a team in that kind of state, I view that as my failure. It’s my job to ensure that I never have to make that ask.

But this persistent evil behavior endures. And it got me wondering where it comes from. I realized where it comes from. It comes from the academic community. Academics have firm deadlines. The paper arrives by the date, or it might as well never arrive. The paper that gets published first gets all of the credit. And so high-end schools teach that the work deadline is all that matters.

And that academic community was primarily male that happily dumped every personal obligation to their family on their wives. My mother and I lived through that period with my dad, who decided to abandon his pregnant wife and son in a tiny apartment on nun’s island, so he could do research in Paris. Fortunately, his friend and colleague explained the foolishness of his ways, and he adjusted.

I saw some of that through the SIGGRAPH era.

After all, the 1990’s SIGGRAPH deadline meant if you were publishing there, your Christmas is about writing a paper. While your spouse takes care of the family and social obligations, you have time to work without interruption.

And I still remember my outrage during the Fukushima disaster. The SIGGRAPH committee sent out a note saying that the deadline for papers was extended for folks submitting from Japan IP addresses. It was an absurd comment and yet utterly consistent with that professor who said, “tough.” I could imagine the academics who wrote that note thinking — “notice how compassionate we are! We extended an immutable deadline!”

So here’s my Easter comment to everyone. Enjoy Easter. Enjoy your life. It’s a short one. Life is a marathon, and you can’t sprint all of the time. Pick the sprints, pick the times to relax, and find the teams that will support you. And leave the teams that won’t. And if you want to sacrifice your life for a greater cause, that’s your calling. Just make sure it’s really a matter of life and death.

And if you are blessed to have a family, find the time to be with them. They matter. I don’t regret many things in my life, but I regret every single Easter I didn’t spend with my mom.

Originally published at http://wrongtool.kostadis.com on April 4, 2021.

In the tech industry, there are as many ways to buy products as there are companies.

Today there are two big variants. The first variant is what we call “Packaged Software,” and the other variant is what we call “SaaS.”

And the corporate transition from selling one variant to the other can feel like the ocean hitting land. And the transition is made worse because organizations insist that the problem is culture and mindset and not how the software is built. No amount of mindset can overcome software architecture. And because software architecture is hard, and because the two teams are screaming at each other, the real problems never get addressed.

But, as I stared at the problem recently, I realized that we architects must provide meaningful guidance. Saying “but the architecture” is as useful as “but culture.”

So let me try.

Packaged Software is when you go to a store, buy the bits, and install them.

The other variant is SaaS.

Now the word SaaS translates to “Software as a Service,” but like any FLA (Four letter acronym), it’s so devoid of specificity to mean nothing.

I claimed that you were a SaaS service many years ago, not if you run in the cloud or on-prem but if you satisfied one criterion.

1. The provider of software decided when software gets upgraded

This definition was interesting, but big enough customers can demand special upgrade cycles even for SaaS.

So I started adding all sorts of other criteria, “easy to use, consumable, freemium, cloud delivery, DevOps,” and every single time, it floundered because I could trivially point to how packaged software offered the same features.

But SaaS was different. And then it hit me; SaaS is different because of another important criterion.

2. If someone other than the provider wants to extend the software what’s important is how to connect, not where the Software is.

The significance of this — is that the software consumer assumes the software is always running and that the consumer and integrator doesn’t have to care about how it’s deployed, how many resources the SaaS system requires, etc.

Integration doesn’t involve step 0; install the software.

Elements 1 and 2 starts to separate SaaS and non-SaaS.

But there is in my mind this third criterion:

3. The producer of the SaaS software chooses where it can run, not the consumer.

Now, this is a weird one. Almost every piece of software written has some minimal hardware and/or software requirements. So, at some level, why is this different from non-SaaS? The answer is that in non-SaaS, the actual location of where the software runs is opaque to the producer.

So now we have — in my mind — a minimal test for a team that wants to be SaaS.

1. Does the team owning the software decide when it gets upgraded and consequently what features the software has?
2. Does the team/customers that use the software just care about how to connect to the software?
3. Does the team that owns the software get to decide where it physically runs?

These are necessary conditions, although not sufficient.

But they are necessary for service ownership to become a thing.

Service Ownership?

In any SaaS company, service ownership means you have a pager (or someone/somewhere on your team) has a pager.

When a company first goes to SaaS, they typically install much software from a bunch of teams into some cloud and offer that as a cloud offering.

The team that installed the software then finds itself triaging bugs in all of the software installed.

The team that did the installation then screams at the other teams: “You need a culture change. You don’t care about customers!”

The teams that produced the software look at the cloud team and said: “How dare you! We respond to every bug!”

The cloud team says: “We are a service.”

The other team says: “we produce software.”

And the screaming continues.

The truth is the following. The cloud team built a service out of packaged software. They control the version; they control how you connect to the service; they control where the software runs.

From the other team’s perspective, the cloud team is just another unhappy customer, running the wrong version of the software, has done all sorts of weird things that don’t quite match other customer usage patterns, etc.

And like every other customer, they just want people. And the product team has mastered the fine art of saying no.

But why does it feel to the product team that they are just another customer? Because the product team doesn’t have any control over when a bug gets fixed and deployed, the cloud team does. Because the product team doesn’t control how you connect to the software, the cloud team does. And finally, because the product team doesn’t even have any say in where the software runs, the cloud team does.

And the SaaS standoff occurs.

The SaaS standoff ends when the cloud team demands a specific service, not support, not cultural shifts, but a service. And that service has to be well-architected and engineered to provide both teams with what they want.

And the minimal requirements of that service are the three things I enumerated.

1. Does the team owning the software decide when it gets upgraded and consequently what features the software has?

2. Does the team/customers that use the software just care about how to connect to the software?

3. Does the team that owns the software get to decide where it physically runs?

Originally published at http://wrongtool.kostadis.com on March 20, 2021.

bout 15 years ago, I met an Inktomi architect who said that the future of all computing was the cell phone. It was before the iPhone.

And I stared at him and his dorky phone, and I said to myself, “nope.”

For 15 years, that was THE worst bet of all time.

But COVID changed all of that.

Mobility ground to a halt. And more importantly, the emergence of a whole slew of technologies called into question the value of physical mobility.

And then got me thinking about the future of mobile computing.

Mobile computing intends to allow the person to go anywhere with their computing device.

That was the paradigm.

But what if the paradigm shifts a little bit, that mobile computing is about letting the person be present anywhere without physically being there.

I like to follow mobile apps as proof points of a thesis.

For example, before Covid, Doordash’s desktop features lagged their mobile devices. It has since changed.

The simplistic answer is that people are at home more and in front of a computer more.

But that’s not satisfying.

What if, instead, more people had more computers? And had created places to work at home that allowed them to use their computers more?

In that case, more people would stop carrying their phones with them and use them to do work stuff. Instead, they would go to their workspace and do their work stuff there.

So?

In that case, technologies that improve our ability to be present are more important than technologies that allow us to carry our computers with us.

And that mobile computing and battery life, the dominant trend and a perceived inevitable future, was just technology that accommodated the previous paradigm, well.

My point in this discussion wasn’t to focus on COVID. My point was to focus on how paradigm shifts can occur.

And that assumptions about the inevitability of anything are usually wrong.

Tying this back to strategic architecture, this falls into my general thesis that the role of a technologist who owns a portfolio of technologies is to recognize that things fall out of fashion. That when they do, it is tempting to call them dead. It is also tempting to ignore things that are in fashion and call them a fad. Instead, we need to keep our eye out for trends that change and then think through all of the implications.

And that if you are leading a broad enough portfolio of products, the strategy of how to navigate that shift is very important.

Originally published at http://wrongtool.kostadis.com on March 15, 2021.

Over the years, I have struggled with multi-tenancy. The definition of it feels fluid and imprecise.

And so, in a desperate attempt, I wrote some incoherent thoughts down. I’ll take another stab at being precise.

Multi-tenancy has two elements; the first is that there are a set of users who have access to a set of resources. And the users and their rights to those resources are a technical problem. What is the hierarchy of users? What is the hierarchy of permissions? What rights do they have on the resources?

There is a separate problem about the unit of isolation of resources. If two tenants have access to resources, the immediate follow-up question is how much isolation does the resource provide between tenants?

For example, a physical server is a unit of isolation. A virtual machine is a unit of isolation. And so on.

Let’s dig into the notion of isolation a bit. Isolation is used to provide security, performance, and availability isolation. A great example is a black site. A black site is completely isolated for reasons of security from the rest of the world. A dedicated server running a single application is used to isolate that application from any other application that may affect it.

At one extreme level of isolation, each computer system is isolated from another.

There is no multi-tenancy at all.

This model is costly.

And so consumers ask — can I trade off some security, performance, and availability for cost optimization?

In short, can I group a set of applications on a set of resources where the isolation isn’t as strong to get some sharing?

Time-sharing systems were, in many ways, the first such system when they replaced batch systems.

And that’s where things got confusing.

Because the only real isolation is hardware isolation, the minute the hardware is shared, two applications share something.

And although hardware vendors and software vendors have gone to heroic lengths to ensure isolation, the reality is if you are sharing something, you are not isolated.

Now let’s introduce a second person, the hardware resource provider. The hardware resource provider wants to offer resources of varying degrees of isolation. And wants to bill accordingly. The resource provider’s challenge is that the consumer may not be willing to pay for resources if they are too expensive. In other words, the consumer of the resources is willing to trade off some isolation for cost savings.

And now we introduce a third player, the manager of the resources that the consumer of resources uses.

The resource provider gives up some resources, the manager divides them up amongst consumers, and the consumers consume them.

The resource provider provides some resources that are isolated. The manager associates those resources into something that consumers can use, and the consumers consume them.

This simple model assumes that a resource that is isolated can be controlled and offered to a consumer and is at the right granularity.

But what if it isn’t?

And thus, we have the fourth player in our dance, a person who takes isolated resources from resource providers that they then offer to a manager of resources that offers them to consumers. Let’s refer to these resources as “virtual physical resources” (VRP).

And that’s fine, except in one very annoying use-case.

What if two different sets of virtual resources are sharing the same underlying physical resources?

If the VRP’s assume that is not the case, then the entire system falls apart.

Why? Because the guarantees of isolation assume that the underlying hardware has some degree of isolation (remember, the VRP is built on isolated resources).

Huh?

Imagine VRP-1 assumes that a socket is isolated and it controls the resource. VRP-1 can make assumptions about security and performance. If VRP-2 is also using that resource, VRP-1’s is being lied to. And that’s fine until VRP-1 and VRP-2 have conflicting objectives.

At that point, the thing that actually controls the socket has to arbitrate between VRP-1 and VRP-2.

And we’re still good. What if VRP-1 assumed it had control of the socket and gave out virtual sockets, and VRP-2 assumed that it had control of the socket and was giving out a full-time slice of the socket.

The problem is that the isolation unit was a socket, and there was no way to arbitrate between the two VRP’s.

So?

The real challenge in multi-tenant systems is the proliferation of VRP’s. Some VRP’s can not be layered on top of each other because they rely on isolation primitives at a lower level that loses critical information about how the VRP wants the resource to be used.

An obvious solution is to create a new VRP, called VRP-12, the union of VRP-1 and VRP-2 and can effectively share a system between the two users. I worked on such a system at SGI. It was a batch scheduler called Miser.

The problem with that approach is that as the VRP’s proliferate, there is a cost in pushing them down a layer.

What’s the cost?

There are three costs. The first is the system resources for the control plane itself. The second is the complexity of using the control plane. And the third is that as the control plane becomes more complex and supports more resource isolation, the control plane needs isolation.

At some point, it becomes simpler to have different VRP’s operating independently rather than on a single shared system.

But wait, there’s more.

Because a VRP only requires a layer that isolates hardware, and every VRP isolates hardware, VRP’s proliferate and get layered on top of each other. There is no final VRP layer.

So?

What it boils down to, in my mind, that multi-tenancy really means three things:

1. A set of physical resources that need to be shared
2. Some VRP system being used to share the physical resources (1) that uses mechanisms to isolate the hardware and provides isolated resources coupled with a mechanism to assign those resources to an individual group that can use them.
3. Some mechanism for grouping users into hierarchies’.

And what makes multi-tenancy even more complicated is that in any system, there are multiple VRPs.

And so the challenge of multi-tenancy is really about how you enable this ever-expanding hierarchy of VRPs such that they don’t require their own isolated hardware, and each consumer, resource provider, and resource manager can effectively do their job. And understand what guarantees they are getting and providing.

To make this a little bit more concrete, let’s consider k8s.

Some developer wants to deploy a k8s application and request a certain amount of CPU cycles.

To do that, there must be physical CPUs.

Those physical resources are partitioned by some hypervisor that hands out virtual CPUs to an OS.

The OS, in turn, hands out threads that run on the vCPUs.

The k8s system in turns groups threads into things called Pods.

The k8s system is deployed in such a way that each tenant has his own namespace.

Each layer, the hypervisor, the OS, and k8s is providing virtual resource pools. Each layer has a distinct manager.

A multi-tenant system is the combination of those virtual resource pools and the physical hardware resources.

Originally published at http://wrongtool.kostadis.com on March 10, 2021.

My father is Charalampos Roussos. His H-Index is 76. He was a key player in creating modern hospital medicine. He did that as a side effect of his research on lung behavior that among other things determined that a human being’s will to live ends after the body dies, not the other way around. While still at the top of his game, he went to Greece in 1986.

And if you read this thread, https://twitter.com/greekanalyst/status/1367818017581768704?s=21, you will begin to understands the challenges associated with his decision to return to Greece.

And my dad saw them all. And yet, he managed to create. Why? He believed that the only way to resist the forces trying to destroy was by choosing to do high-quality work at all times.

His point-of-view was summarized with a saying, the title of my blog, that translates in intent if not precisely “the resistance of high-quality work.”

He is writing his memoirs. And he wrote a longer version. And I want to share it because it summarizes my view on so many different things.

Όσο δύσκολη και να είναι οι καιροί,
όσο ελλιπείς και αν είναι οι υλικοί και ανθρώπινοι πόροι,
όσο σοβαρή και αν είναι η αρρώστια,
όσο και αν ενδημεί η ζήλεια, η συκοφαντία και ο φθόνος,
πάντα οι ελεύθεροι και οι δημιουργοί μπορούν:
να αναπτύσσουν το χώρο της ευθύνης τους,
να προβάλουν αντίσταση με την ποιότητα της εργασίας τους,
να πιστεύουν στην αριστεία,
μια αρετή που δεν χαρίζεται,
μια αρετή που μόνο κατακτάται.

no matter how scarce material and human resources are,
no matter how serious the illness,
as much as jealousy, slander and envy are endemic,
freelancers and creators can always:
develop their area of responsibility,
resist with the quality of their work,
believe in excellence,
a virtue that is not bestowed,
a virtue that is only conquered.

Originally published at http://wrongtool.kostadis.com on March 8, 2021.

When I started my career in software engineering, I didn’t know how to interview. But I did know that the basic paradigm was engineered to keep people like me out.

Let me explain. I have ADHD. And I can’t read quickly. And I was taught at a young age that speed measured fluency in a task, not the ability to perform it automatically. And I had marginal dyslexia, so my handwriting was poor.

Put me on a whiteboard, and ask me to write code, and it looked messy. I would forget semi-colons. Error conditions would be skipped over. Subtle details about the problem would be missed.

In short, the test would be an excellent evaluation of my ADHD/Dyslexia and a poor evaluation of my ability to be a software engineer.

SGI offered a program on how to interview, and so I decided to take it.

And the key insight the person who was doing the class had was that past success was a measure of future success.

And so I changed my strategy. I stopped asking stupid technical puzzles and instead asked them about what they did.

The problem with that strategy is that what you did is a good predictor of what you will do in a similar circumstance, but will it predict what you will do in a new environment on a new problem?

Also, it results in overemphasis on skills rather than aptitude. And the specific skills you may need change over time. And I saw this play out at a former employer where we switched to mobile, and the needs of backend software dropped. Engineers quit or were asked to leave because they lacked the skills and either lacked the interest or aptitude to acquire new ones.

At Zynga, we had this level that was just above MTS, called Principal Engineer. I discovered that a lot of those engineers who were hired into those roles flamed out. After I dug into it, I discovered that we had created a role for people who had mastered some technology. When we hired them at Zynga, they had to master a new thing, and not all of them succeeded at that. I will observe a larger part of it that we didn’t set them up for success, but it was interesting.

Especially because the level just above Principal Engineer was Architect, an architect demonstrated that they had mastered multiple technologies.

Architects almost always succeeded at Zynga.

This then leads to my next observation, that a good predictor of a senior engineer is someone who has mastered multiple different technologies and made an impact on them in multiple environments.

And then I went to VMware and realized that that wasn’t a perfect definition either.

Some companies need to have world-class experts in a specific technology. For example, VMware has world-class experts in compute virtualization. Those individuals are undoubtedly senior engineers who have mastery and depth that is astonishing.

And then, I learned about how some people don’t get an opportunity because of their gender, orientation, and skin color. And that reminded me of the story of Devil and the King. The King asked — “Who was the General that could have saved us?” and the Devil pointed to the King’s slave, “he was.” And the King said indignantly- “but he’s a slave!” And the Devil said, “yep, and you lost the war and your country because that is all you saw him as. Funny how that works.”

Finding a senior engineer is about finding a needle in a haystack. You have to look hard, and wide and deep and in places, you wouldn’t. You have to look at objective criteria. And you have to keep your ears and eyes open to see people you don’t normally see.

So? So, hiring a senior engineer is a multi-faceted process. It would help if you looked at a bunch of criteria, your team composition, the problem you are solving for in their hiring, and in the end, make a judgment call.

Anyone who says that they can evaluate a senior engineer by themselves in 45 minutes is — I hate to admit this about myself and people I have judged — lying to themselves.

Originally published at http://wrongtool.kostadis.com on March 5, 2021.

Rowing in the Swedish archipelago, the wind suddenly stopped, and the water became this oily, super smooth surface. We stopped moving and got fascinated by the water drops forming patterns on the surface. Even the most common thing can turn into something beautiful under the right circumstances.

Over the last year, I have been thinking a lot about the nature of software and software systems requirements. When I started my career, I was at the tail end of the era that only cared about performance. The fundamental thesis of that era was that CPU cycles were not plentiful and that writing efficient code was critical.

It was 1996 when I left Brown University. At that time, the forward-thinking software engineers advocated for higher-level programming languages to alleviate some of the tedium of writing assembly.

Getting a computer to work, though, was more important than it working for very long. Having a machine that did a computation was the hard problem. As we got better at building computers, a lot of energy was expended to make those computers more robust.

It’s important to realize, in 1996, writing efficient code on computer systems that were assumed to work was the only mental model and constraint you had when designing system software.

In 2004, when Steve Jobs introduced the iMac with colors, the industry was aghast. The design of the user experience was no longer an afterthought but the centerpiece of software. I remember doing my master’s degree at Stanford and wondering why human-computer interaction was a thing. After 2004, we all had to get religion very quickly.

In 2009, the next profoundly disruptive requirement was availability delivered through clustering. Before 2009, the mental model for availability was to buy bigger and bigger machines that were robust and reliable. But AWS happened. And although the debates about clustering had indeed begun in the late 90s, they took center stage when AWS released infrastructure where availability was very poor and required native clustering to deliver uptime.

At about the same time as AWS in the cloud happened, the dev-ops movement emerged. Rather than fixate on what dev-ops is and is not, let us look at what it was a response to. In the 1990s, when you wrote software, the expectation was that you would buy new hardware when the new version came out. Upgrades happened when there were critical bugs.

What AWS enabled was a completely novel form of software delivery. Instead of delivering software with hardware, it became possible to only deliver the software. In short, the software went about from being something that you bought once to a machine that continuously was being maintained. This need for continuous maintenance created a new set of requirements.to deliver software continuously

In the early 90s, the serviceability of the hardware was not an important consideration of hardware design. As customers bought more and more computers, and the time to repair each computer took more and more time, the ability to service the computer became more and more important. So much like hardware in the 90s suddenly started caring about how fast and how easy it was to replace the disk drive, software engineers had to start caring about how easy it was to operate the software. Why? Because it was expected that the software would always have to go through some regular maintenance.

Let me try this differently. In the early 90s, when hardware failed, the computer went down. Through a series of hardware innovations, it became possible to replace most hardware components when they failed if you are willing to pay the money. In 2009 we adopted software architectures that enabled a similar kind of upgrade of components and replacement components while the system was continuously running. Once that became possible, it became possible to add new value to the software continuously. In short, the software became just like the hardware, something whose serviceability became critical.

To give a flavor of this, in 1996, SGI released the Octane. The Octane was a fabulous system. But serviceability was never considered. You couldn’t rack it. It generated too much heat for an office. And so, it wasn’t that useful if you needed to buy 50 for a rendering farm.

In 2013 Edward Snowden happened. After he betrayed his country and fled to Russia, information security which had been a sideshow, became front and center. The difference in how computer system architects considered security before 2013 and after 2013 is night and day.

The next trend to emerge in software at large-scale companies will be the discipline of strategic system software. Most software companies have exactly one product. Large companies have many different products that must all work together and be independent. Software architecture has historically only considered how you architect a single software system with a single market in a single business. Strategic system software architecture considers how a collection of products, each with their own markets and business, fundamentally integrate to deliver more business value than the individual components by themselves without turning them into a single product.

Because of this discipline’s boutique nature: you have to work at a large company, be fairly senior, and the executive team has to see the value of that; there are very few practitioners of this art.

But as the arrow of time moves forward, I expect more of us to emerge.

Now those of you wondering what’s next, I believe it is the intersection of the law and of software. As more and more software gets embedded more and more deeply into people’s lives, the need to understand the law, the law’s intent, and technology will be a new architectural discipline that software systems will need to incorporate. To date, software systems and their architects view regulatory constraints as an inconvenience, not a necessity. Software system architects, who wish to define a new era, are advised to get a legal degree and understand the complex regulatory legal and its technical nexus that will drive the next important technical innovations.

When I started my career, system architecture was essentially about making the data path go fast. 20+ years later, it’s much, much, much, much more than that. And the experience of these last 20 years is that the set of constraints will only increase over time.

What does this mean? I don’t know.

Cross posted from http://wrongtool.kostadis.com

In 2009, I left NetApp in a huff. My attempts to redirect our investment strategy to either pursuing aggressive investments in replication or what later on became known as HCI went nowhere. Worse, for me, NetApp was more interested in what was known as convergence and later on became known as 8.0, and I wasn’t interested in that effort.

And so I left NetApp.

And I went to Zynga. Zynga was an incredible experience. The first startup, first pre-IPO, first IPO, cloud, on-prem, hybrid.

And for the first three years, I was flying on top of the world. I remember feeling that I had finally arrived!

Then Facebook changed its algorithms, did Libra 1.0 (aka Facebook credits), and Zynga fell, And we failed to deliver on mobile games. And I went from speaking to hundreds of analysts, building cool games, to being out. In less than 12 months, I went from the top of the world, to out.

So I looked for a job and took one at Juniper. And that was a professional experience that was surreal. In short, I joined to help transform the culture and got laid-off in 14 months.

And then, I ended up at VMware. By then, I had gone from a high-flying senior architect to joining VMware at a time when VMware was not that cool.

And the team I joined had just endured a lay-off, a re-org, and after I joined, the attrition was high.

I like to describe our situation in the following way. We were a team in a dark freezing room. Inside the room were a few pieces of dry lumber. And we had two matches. With the first match, we found the wood, and with the second, we lit it. We were great, and we were lucky.

The gap between success and failure of 6.5 was minimal. We had managed to assemble the right kind of team that could pull it off, but we didn’t know it at the time.

And then, after we worked to figure out our k8s strategy, and we got lucky again.

The Heptio team decided to join VMware. And thanks to their decisive contribution, what was an excellent k8s technology strategy, became a tremendous k8s strategy.

The wheel has turned, and I gave a keynote at k8s about the work my team did. And the first time I did my talk at Zynga, I thought that this is why it must be because I am that good. And now, nearly seven years late, I realize that I wasn’t that good. I was lucky.

[embed]https://youtu.be/BSF8HgkgxZE[/embed]

One of the particular challenges of being a software architect is that the average manager and vice-president of engineering assume you are incapable of adding value, right now.

One of the particular challenges of being a software architect is that the average manager and vice-president of engineering assume you are incapable of adding value, right now.

Engineering managers have releases they need to deliver to now. They have engineers who have problems now. The managers have budget squabbles now. The teams have debates with product management now.

An architect has visions of the future, and those visions are often years away from delivery, and worse, even more years away from solving any of the immediate problems engineering management has.

And so the VP of engineering begins to see the architect as a distraction. The architect distracts in two ways. The first is that they are always complaining that the team is not investing in the future. The second is that the things they want are unimplementable. And worse, the architect is typically unwilling or unable to go figure anything out. Powerpoint gets produced, confluence pages get edited, wiki pages get updated, and in some cases, small prototypes get written, but no useful code gets changed.

And thus, the VP tries to manage that distraction. There are two fundamental approaches. The first is to keep them away from anything that matters and fire them at some opportune moment. The second is to focus them on a small problem. A small problem keeps the architect from distracting other people and makes it easy to measure their delivery of value. In other words, force the powerpoint jockey to write code, and either they figure out how to write code, or they get fired.

And you know what, the VP of engineering is right.

The thing about being a software architect is that you always have to be right. What I mean is that you are guiding a team, and the direction has to be correct. If it’s not correct, then the team is heading towards a catastrophe. You can change course over time, but it always has to be the right course.

In addition to always being right, you also have to be sufficiently high-level, so everyone is doing the right thing. Pulling this off is another tricky thing to master. If the correct answer is not inclusive of the entire organization, then somebody is doing something wrong. Furthermore, if it’s too low-level, then there are no white-spaces for engineers to innovate.

Another customer for the long term architecture is the CEO/GM and Product Management team. They have to see enough value that they can talk about it to their customers.

In short, you need to be right, and high-level enough that you can’t be wrong, but if that’s where you end, you fail.

The software architect must also add value right now. What does that mean? It means if the head of product management chooses to fund X features, all X are stepping stones to your architectural vision. If the engineers have to design something, it should be evident from the high-level architecture what they should do. If the managers have to figure out how to trade off long term vs. short-term execution, they should make that decision in the context of the global vision.

But it’s more than that. Managers operate in terms of things that need to get done with some set of resources. And so it’s vital that the architecture, be broken down into discrete tasks.

And so my mantra,

• I must always be right.
• I must be sufficiently high-level to be always right.
• I must be right, right now.