See it here: The My Porsche Technology Radar Vol. 2

What is a Techradar?

Our Techradar is inspired by, follows and uses the tools from Thoughtworks. They have a very good explanation on their website.

How does a Technology Radar help us?

As you may know or remember from previous articles we at My Porsche align many decoupled teams. We are proud of our self-sufficient teams that work on one seamless user experience for “My Porsche”.

To understand the impact a Techradar has on our agile teams we have to look back. Not long ago a central board made the decisions about technologies to be used in all projects. There was no product-thinking for software. A book of standards defined which way to go and what to use. The teams had no choice except the company-wide tool set and methods. Regardless if this set supports a seamless user experience.

We shift responsibility into the teams.

Therefore we now allow the teams to use the tool set that fits the team best. They are free to choose their technology to solve customers’ demands. This is a huge responsibility since the teams have to:

• maintain their own software stack and keep it up to date
• setup a secure cloud infrastructure
• follow our rules to be part of My Porsche (like security is not negotiable, infrastrcuture as code etc.)

We setup teams with less effort.

At the same time we setup new teams in new regions. We can only provide the best user experience when not only our backends are close to the users, but also our teams.

The Technology Radar shows the “trodden path” that worked for existing teams and helped these teams to build their product. New teams participate from that knowledge and experience.

We preserve our knowledge to share it.

By discussing the blips for our Technology Radar we identified a most precious value: knowledge (about what works well and what attempts were in vain). The Technology Radar preserves our knowledge — but only works if we update it regularly. We also like to share this knowledge with the world and are open for feedback.

If you are interested in how we organize our teams, have a look at this article: Decoupled Alignment.

History of our Technology Radar

Creation of Technology Radar Vol. 2

We learned from our first approach (Vol. 1) and introduced the following rules:

• Reduce the number of participants and instead asked for input from all the teams via Email, Slack etc.
• Remove all blips from Vol. 1 that didn’t cross our way in the last 6 months
• Discuss what worked for our teams
• Be more strict what we really set to adopt or trial (based on the discussion above)

These might seem simple and self-evident. But we didn’t follow them in first place.

This time discussions were much better than in our first session. We really took the time to discuss the relevant blips from several team’s perspective. Only if a blip was successfully used in production or resulted in a productive feature in at least two teams, we set it to adopt.

If only one team uses a blip but it really had a huge positive impact during planning and implementing the feature we set it to trial.

Whenever we had the impression that we should use a certain technology to shorten the time-to-market or increase security we carefully put them into assess.

From our regular Tech Sync and Product Owner Sync meetings we identified the blips that are currently on hold.

We also used the Technology Radar to get rid of legacy software or gently push the teams into certain directions. A good example is our self-managed instance of Hashicorp Vault, which we abandoned in favour of a managed service. Managed services have better availability, are monitored permanently and require no administrative resources from our side.

The result is what we expected. The Technology Radar Vol. 2 is right to the point. It contains the right amount of blips to setup your team, your software stack and provide interfaces to build and maintain a product in our ecosystem.

Since we reduced the number of blips we managed to describe all of them.

We created a Technology Radar which is inspired by Thoughtworks and keep it stupid simple: use their template.

Where is Vol. 1?

Technology Radar Vol. 1 was never published. We started with a small group of Tech Leads and the IT Architects to collected all blips that we could think of. Then we discussed the blips, placed them in quadrants and rings — et voila: Vol. 1 was ready by March 2018.

But we missed an important part of the exercise: We did not filter the blips for their importance and also put too many of them into adopt. The result was way to big collection of all our technologies, methods, languages etc. We never managed to add even a short description for the blips.

Vol. 1 did not look like a Technology Radar (and therefore it was never published). We created an internal Confluence page but never invested time to advertise this one. I’m pretty sure, no one ever opened that page.

We decided to wait 6 months until we update the radar, filter and reduce the blips.

Highlights of this edition

Techniques — Public APIs as Default

We really want to decouple our teams but share our data. Therefore we introduced several public API topics in different quadrants — but all in assess. If the teams start to think “API first” it will reduce overhead to plan and develop interfaces for their data.

Techniques — Operations as initiatives

With great power comes great responsibility.

Teams gained freedom (=power) when moving into the cloud, but at the same time were overwhelmed by the amount of operations tasks still required. To make efforts visible to all stakeholders we would like to create initiatives for operations tasks vs. a fixed budget for these tasks.

Platform — Lambda functions (for Apps) & Serverless

We were really impressed how serveless accelerated the speed of developement for backend services — even if the teams had no dedicated backend developers. We are sure this will make a difference in time-to-market for MVPs.

AWS Lambda is currently our tool of choice to go for our Serverless approach.

Languages & Frameworks — Java

Oracle changes their licensing policy which will have an impact on our budget. We therefore would switch to OpenJDK wherever possible. Therefore we set the OracleJDK strictly on hold and set OpenJDK to adopt.

Tools — Træfik

Træfik made it into our Techradar in a very short time frame. It’s surprisingly easy to setup, configure and maintain for the teams. It therefore replaces NGINX in our cloud infrastructure setup. It is perfectly prepared to play with microservices (e.g. circuit breaker, routing, reload-less reconfiguration).

Conclusion

By using the tools from Thoughtworks it is fairly easy to visualize a Techradar. We had the task to collect, discuss and decide on the blips in our My Porsche ecosystem. We learnt from Vol. 1 and made Vol. 2 much more lean and to the point.

There are other tools and methods available to collect all technologies of our teams (e.g. see our software stack at stackshare.io).

But the most important learning from all: We entirely changed the way how we are dealing with techniques, platforms, languages & frameworks and tools. We first evaluate and give the teams the freedom to try it out. Then we sort it out within our regular Tech meetings and bring this up (or not) in the Techradar. We no longer stick to centrally pre-defined project methods, software stacks and limitations imposed by decisions form central boards or books of standards.

Technology Radar Vol. 2 was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Remember, that we at My Porsche decided to decouple our systems in a way that teams can develop their functionality independently from others. Our main driver for this is time-to-market — and thus — relevance for our customers.

On the other hand, it is important to ensure that the different services we offer in our digital eco-system are perceived as “one”. The user should not notice that he is switching between services developed by different teams (people who bought a product in a shiny store in seconds and faced hours in support hotlines once having problems with the product they purchased know what I am talking about).

The picture above shows a high-level diagram of our approach to tackle the challenge to balance alignment and team independence. On top you can see the customer — using our services. In the bottom you can see some of our teams offering their services (in grey).

In-between you can see that we develop common standards for cross-cutting concerns we believe are important for a seamless digital experience (in blue). Apart from having a single digital identity and a common UI design, we came up with additional rules & common standards for a seamless digital user experience.

In this article I want to share the different aspects we have identified so far. Adding technical details to the diagram, the picture looks like this (apologies if this is detailed, but for the sake of transparency, I decided to use the very same drawing we use internally):

You will notice that we have identified rules for scoping the product (exposed as services), the conceptual components (in the service and the UI layer) and the interfaces between our products.

Take our messaging product for example. It provides a secure inbox for delivering messages to our customers. When a system wants to deliver a message to the customer, it uses a REST-API in the service layer to submit the message.

The messaging service show the message to the customer via web application that is part of the messaging product in the UI layer. Other products — such as our landing page can use the REST-API to show the number of unread mails — and create a deep link to the UI showing the inbox, filtering etc. Other systems — such as a Backend for Frontend (BFF) for an app could subscribe to a Kafka topic that signals a new message — so that the BFF can send a push notification to a mobile app.

In addition, we have a number of common conventions / requirements to support a seamless user experience. I will list the major ones and state the cost of not having that rule:

Supported Languages and Countries: We have different offerings and texts (e.g. instructions, legal texts) that differ per country. Without agreeing what languages we support in what country, there is the danger that language changes inside a user journey.

Browser support: Without agreeing on the same set of browsers, we cannot ensure that the user does not end up in a dead-end when being forwarded to a system that does not support the browser he started with.

Number of system environments (e.g. development, integration, production): Without agreeing on how many stages/system environments we have on a system level, we will not be able to test user journeys end-to-end because comprehensive test data will be impossible to produce.

Logging & monitoring: Without agreeing on standards on logging and monitoring we will not be able to provide support.

Authentication: Without agreeing on standards for user-authentication, users will have to re-authenticate during their user journey.

UI design / UX: Without standards / samples for user interface design, pages will look different. We ensure this via a UI-Kit that contains ready-to use UI elements (actually this goes beyond UI elements and also includes a common dictionary, translation process etc. to ensure that we use the same terms for things).

Documentation: Without common standards for documentation of our products (think of this as a product catalog) we risk people solving the same customer-relevant aspects twice (e.g. two products offering services how addresses are formatted — leading to incompatibilities when re-using address data across systems).

Networking & Connectivity: Without standards for networking and connectivity (we chose the public internet), we risk situations where we cannot connect systems to each other, because products might have been created in legacy network zones that restrict internet connectivity.

There are other aspects where we believe we need to invest in (e.g. resilience and performance/timeouts), but we haven’t come up with good rules, yet. Which brings me to how we create such rules. The rules are not created by a single governance body. Instead, the product teams identify these rules as they come up. In our guilds (e.g. UX, tech, devops, …) we discuss them — and check if they seem relevant (e.g. the networking rule seems obvious, but we missed that at some point in time). When discussing, we first check if we risk negative user impact if we do not have such a rule. Next, we investigate the status quo, asking why people solved things differently — and finally trying to find consensus for a rule.

Behaviour over inner workings: You will notice that above rules focus on the BEHAVIOUR of a system (old term would be requirements). I am often asked by fellow architects why we do not standardize around the building blocks of the application. They — like myself have been trained to fight redundancy in IT systems.

I strongly believe that redundancy is not bad in general. Accepting redundancy can increase delivery speed. It becomes evil if it contradicts our requirements. Remember, our requirements are: short time-to-marked, highly integrated user experience.

Will a customer notice if we are using mongoDB in one service and dynamoDB in another? I doubt it.

Will we have better time-to-market if we use mongoDB compared to DynamoDB? I believe it depends on a number of factors… factors I believe the developers of a product can judge best.

But what about cost? I believe in the power of the market. If we have the feeling that a variety of databases is too expensive or time-consuming (because each team has to figure out how to solve a particular problem), I am convinced it is best to make an offer to the teams that they are not able to resist. This can come in forms of a trodden path (e.g. we already figured out how to do backups in DynamoDB, this is how we do it and here is the documentation necessary to prove that we comply with data protection laws, we already have set up a contract to order xyz) or by having one team creating a product to solve a specific problem.

If a company fails to make such an offer, but tries to standardize via strict rules and policies, it will remove the motivation for the team offering a service to remain competitive — and eventually will result in higher costs and frustration.

Identifying potential candidates for a trodden path or a product is a challenge on its own and a topic for another post.

So as a summary, in our setup, the “enterprise architecture” must focus on rules and policies around the behaviour of a service, but not for the building-blocks used to create it.

Decoupled alignment was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.

When looking at Enterprise Content Management Systems it often feels as they were designed for print, treating websites as paper. Many CMS came with complex inheritance hierarchies, workflows and vendor-specific customization. None of these features makes us fast or flexible.

Through the last years, Headless CMS products gained a rapidly growing market share. Headless CMS separates representation from the content. The content itself is available through APIs, its representation will be implemented from the respective client applications. But publishing a Content API is only one part of the story. At My Porsche we evaluated different Headless CMS products and introduced Contentful as a fundamental part of our Content as a Service platform one year ago.

Enable content creation

Obviously, content editing is a major part of any CMS product. While developers often do not think about content creation, it is a crucial success factor. Many Enterprise CMS are powerful, but at the same time complex to use. Editors often need a comprehensive training to learn how to maintain content. Usually content is organized in tree structures, following an inheritance schema. For instance there is a content tree for a certain language. If you want to reuse content of a page for other languages, it is often required to duplicate that page into the schema of the second language. Customizing requires in-depth knowledge of the product.

What makes a Headless CMS different and why do we call it Content as a Service? First of all we don’t edit entire pages only functional components. Editors don’t have to deal with the complexity of the entire frontend. Contentful is fully based on APIs and comes with an easy to use editing interface, which also relies on the same APIs the developers use. It provides full control over the content model allowing us to quickly implement our use cases, without having to consult agencies or tasking the CMS vendor to customize their product.

At My Porsche we started with local content maintained by our partners in the markets. The editor user interface is just a graphical representation of the content model. Ease of use is important, as our editors are not CMS experts but experts in marketing or sales of the respective market. There is no time for comprehensive trainings.

As we develop new content models, we can directly provide access to the future editors and improve with their feedback. This way we gained speed and reduced organizational overhead. We don’t need highly specialized CMS developers for that. Having knowledge in REST and a major programming language is enough to get used to the provided SDKs and Content Management APIs.

WYSIWYG?

In a world of devices thinking about break points and supported browsers is not enough anymore. The boundaries between HTML based frontends and native applications are fluid. One big advantage of many CMS products is the ability to see a preview of the future site. As content editors also have control over the layout.

When using a Headless CMS the API is the product and the representation is implemented at client applications. So how do we deal with the lack of previews?

We strongly believe the best preview is to look at the latest content changes in the target frontend. Contentful provides a Preview API having draft contents and production contents. Usually client applications integrate the Preview API in their non-productive stages.

If it comes to content for authenticated areas, such as the My Porsche Portal, our editors have to deal with a bunch of different test accounts having different roles and capabilities. As a problem solver we started to invent own demo pages using the same components as our frontends do. This of course does not replace a full review in the target frontend, but it allows to quickly set some dropdowns and flags and see what content pieces will be visible for the target user having this context. At the same time it is a good practice in development, it forces our team to think about the content API as a consumer prior to the actual integration with the target frontends.

Content modeling

Content modeling is really important as the Content Type defines the API endpoint as well as the editing experience. In the past we had to consult an agency forwarding our requirements to the CMS vendor if any change at the Content Model was required. Now we can define Content Models on our own in code and highly automated.

We usually start do discuss requirements with dependent teams and intiative owners as soon as possible on the whiteboard. If the basic concept is aligned, we start to develop the first content models within our development environments just by clicking the fields together in the web app of Contentful. This enables us to get early feedback from the future editors of that content and adapt. Typical discussions are mandatory or optional fields, localization and so on. As editing is really easy most editors can immediately create their first contents without reading manuals or receiving extensive trainings.

After we have a certain state mature enough to continue we start to develop the content models in code, using the vendors SDK. In the end we have the full model in code, including tests and we are ready to deploy. As the content model also designs the content API, we are always aiming at a balance of having an easy editorial frontend and an API that allows all sort of filtering and selection with a minimal set of queries for the client developers.

Having flat content models is quite easy and straight forward. The complexity grows when you have to link the content pieces with each other. The content is organized in a content graph, allowing to associate the content entries with each other.

References are very powerful but they also come with additional complexity. Editors have to be aware of publishing all references of a content piece as well. Especially cyclic dependencies must be avoided. For client developers it also gets more complex to fetch all content pieces with ideally a single query when using references.

As an easy example we do have a simple banner rotation, distinguishing which banners to display at My Porsche for a certain target audience. Using a reference to the banners allows to reuse a banner in multiple rotations while not having to duplicate the actual banner.

At the same time the Banner Rotation limits us when it comes to tagging with further attributes, that are not dependent on the banner rotation. What if we want to tag Banners with ‘Golf’ or ‘Electric Vehicle’? Think about the queries a client must request to fetch these information. This easy example illustrates pitfalls you have to be aware of at designing Content Models. We are constantly learning and evolving our content models.

Content API consumption

Across uses cases currently running on our platform we see both, clients that directly integrate content APIs in their frontends as well as applications having a server-side aggregation. While such an Aggregation Service allows to integrate further logic and apply specific rules on top of the content, it misses the advantage of the content API which computes queries at globally distributed Edge servers.

Workflows

Content creation must be easy and fast. This applies to publishing as well. If publishing is limited through headquarters only, it feels save but it does not scale pretty well. We believe our editors must be empowered to take over responsibility for their contents.

Workflows are not supported, basically Contentful is mainly distinguishing between content in draft state and published content for production. As the product is highly extensible, from alerting to custom UI components and complex workflows tend to feel as overhead, we see this fact more as an advantage. For instance we can define a webhook that is triggered on certain events and fires a request to an AWS Lambda function, having all event data and parameters inside the JSON body.

Providing a content platform for product teams

Another important aspect is the capability of developing and deploying content models. In the past we had to describe requirements which where provided to the vendor through an agency. Having a content as a service product we can now describe content models on our own. As all content management tasks are available through a fully documented API it is possible to describe the complete content models in code, using practices such as a CI/CD pipelines and automated tests. This makes us faster but even more important, we can enable other product teams to develop their own content models for specific use cases. Usually a small kick-off is enough to explain the basics and share best practices with the other teams.

We distinguish between ‘Content’ and ‘Platform’ as products. For use cases having ‘Content’ as a product we develop all content models, take care about APIs and editors. The editors maintain content and assets.

If other teams want to develop their own content models and manage their own space, we onboard their developers and share best practices. The product team is fully responsible over content modeling, editing and API client integration.

Conclusion

Looking at the past year the ability the create content models on our own really made us faster and more flexible. It also encourages our partners in the markets to directly get involved instead of just writing down requirements.

The Contentful web app is easy to use and the product is pretty popular among our editors. Markets are able to directly maintain content for their respective customers on their own.

As content team our major advantage is to have all the building blocks like content model, editing interface, web hooks and so on under full control to rapidly develop and deploy new uses cases. Contentful allows us to focus on the content and to limit organizational overhead.

Providing new content models fast is only one part of the story, we still need to improve our planning and dependency management with regards to content to decrease time 2 market.

Do we miss classic CMS features like Content Preview, Asset Management or Workflows? I think we have to answer this with the classic enigneering quote — “it depends”.

We strongly believe that the best preview is to have a look at the contents in the target client application. The Asset Management in Contentful is pretty basic. We also do not miss Workflows, but something like scheduled Publishing or assigning some unfinished content entry to somebody would be appreciated by our editors.

Definitely Contentful is highly extensible. Developing custom scripts and tools always comes with the cost of having to maintain and operate these tools. I would recommend to define a Code of Conduct of how you want to use your platform. Sometimes complex problems lead to a simple conclusion — business logic should not be implemented in the CMS. Implicit logic and dependencies between content fragments make editing and API consumption more complex and error prone.

Whats next?

Content as a Service based on Contentful gave us new capabilities. For the future it will be important to integrate further content sources. We’ll have to aggregate these contents based on certain rules and provide more dynamic APIs, aware of the user context. It will be a big challenge to provide new approaches while keeping the client APIs simple and the frontends generic enough to keep flexibility.

Introducing Content as a Service was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Before diving into the effects this change has onto Porsche’s IT system architecture, I want to give a little perspective on the key factor why we want to change.

The car/mobility becomes more and more integrated into the digital ecosystem to deliver the best experience to our customers. This requires a tight integration with all sorts of systems such as phones, calendars, preferred streaming services, social networks etc.

In contrast to a mainly closed system such as the traditional car (which also contains a lot of highly advanced software applications), this ecosystem is not controlled by a single organization — therefore timing and features are not controlled by a single instance. Every participating player needs to adapt to changes outside their control quickly and to seize new business opportunities.

This adaptability and the means to deliver new functionality quickly, fundamentally influenced our system architecture. We started with the systems that provide digital touchpoints with our customers (such as identity management, web portals).

This flexibility was in contrast to the requirements that mainly drove our IT in the past (the need for well-planned, effective, long lasting systems). And we came up with the following rules:

Decoupled — Rather than developing one large system, we have chosen to implement systems that concentrate on a small set of functions. But decoupling also meant that we accept creating similar components twice (something we tried to avoid in the past).

Highly aligned — While the decoupling gives the individual teams greater independence, it is imperative, that we ensure the interoperability between the systems. This goes far beyond purely technical aspects. It also covers non-functional requirements such as supported languages or common standards to enable service and support.

Web standards — Existing web standards are the technical representation of this alignment. The web is an impressive example of highly decoupled services that integrate well. We have chosen to follow this great example.

Evolutionary and demand driven — To prevent our systems from being over-engineered, we invest in flexibility rather than up-front guesswork of future requirements.

Community driven — We integrate with a great number of existing systems and work with a lot of different partners to implement our systems. They all bring specific domain knowledge and experience. We believe that a community of this diversity ensures a reasonable balance between the decoupling and the alignment mentioned above.

What worked well

The chosen approach brought us quite far and showed positive effects on our ability to deliver new functionality quickly.

Our rules work well, but they are not free of contradictions. E.g. our decision to use Kafka for inter-system communication is contradictory to our commitment to open web standards (see Heiko’s Post “Apache Kafka at Porsche”), but in full alignment with our evolutionary and demand driven approach.

The evolutionary and demand driven rule needs to be handled with care. There is the danger of creating solutions on a basis that this not fit for handling a requirement (alas, the German word “frickeln” — does not translate well into English — to tinker is probably close) — so the existing state must be challenged constantly.

We found that the approach to form teams that are responsible for their systems works well. We see that it ensures that everyone works on things he/she knows best. A side-effect is that problems often show-up in the same place where they can be fixed.

What is next

The rules and what we have done so far brought us far but they are only the beginning of our journey. This is what comes next.

The Cloud: When we started the implementation of our systems, we used the existing building blocks our IT-systems were relying upon. These building blocks are the same that can be found in many manufacturing companies: a set of software components (such as Tomcat, Oracle DBs etc.) operated on on-premise hardware. These building blocks cannot be controlled by software — but rather require manual work by those responsible for the services offered by these components. This greatly impediments our ability to refactor our existing services and encourage work-arounds. See Florians post about our cloud journey for our approach on that topic.

Self-service APIs: There will always be communication and thus dependency between our systems. To make sure we can manage them in the future, we need to make sure that every service we create can be consumed via self-service (APIs). Again, we have a great example to follow: Cloud services accessible as self-service made the dependency to infrastructure less problematic.

Fitness functions: The rules above only work in a smaller environment, where their meaning can be taught to new colleagues and evaluated in the community. As described in Building Evolutionary Architectures, the desired state of an architecture could also be made measurable by fitness functions. This (among other metrics) could move us forward in measuring our progress we all feel we have achieved so far.

One of my key learnings is that you as an organisation need to draw some conclusions yourself. There is a lot of great literature, excellent examples and great people to learn from, but there is no such thing as the single blue-print on how to get digital products right. So the things written here are a snapshot (hopefully you find interesting) and will change over time.

Gaining new capabilities was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Moving towards an agile software delivery model brings changes to the way teams work. They

• are empowered
• need to gather feedback quickly
• move fast
• use modern technologies

For this to succeed, they need infrastructure tools and processes that support this workflow. They need insights and control over their technology stack to iterate quickly and build the best solutions.

IT departments’ traditional ticket-driven processes and standard products are often not the right fit for this type of work. They need to evolve alongside software delivery teams.

This post describes the Porsche Connected Car programs’ journey towards a new, AWS based infrastructure.

Finding the hook

For business leaders, it can be hard to grasp the challenges, tradeoffs and benefits of technical tasks (be it a small refactoring or replacing the whole infrastructure). So if we want to get their support, we need to frame technical decisions in a way that makes sense in their world:

“This might make us better in developing software” might be true and obvious to any engineer, for outsiders it is vague and hard to follow.

“This will improve customer experience and increase revenue” is concrete, measurable and a straightforward argument for any business leader so finding it will go a long way to bring them to our side.

For us, this argument was the performance experience for our customers. As a car manufacturer that aims to provide a superior quality to customers all over the world, serving all of them from data centers in Germany does not seem to be the best approach. But how to prove it?

Measuring Performance

If we want to argue from a well founded base, we first have to find out if we even have a problem.

So first, we need to look at how fast our product is for users all over the world. For this, WebPagetest served us very well, allowing us to run simple scripted user sessions across a wide range of locations and gathering detailed performance-data. It also allows us to generate videos comparing user experience across regions that allow non-technical people to understand performance impact.

So, we established that we have a problem. Now it’s time to drill down to find the actual bottlenecks affecting performance.

Properly instrumented applications can provide a wealth of data to do this, logging performance metrics for incoming and outgoing requests that we can query and visualise in a log analysis tool (in our case Splunk).

Once we had a more detailed picture on the critical requests in our applications path, we used Gatling to build load tests that allow us to reproduce critical aspects in various scenarios.

In the end, we found a number of different aspects that affected performance and we were also able to prove our hypothesis that network latency between our customers, their cars and a central datacenter played a role end user performance, giving us a great starting point for discussions with various stakeholders.

A vision and new team

Having convinced various stakeholders that changes to our infrastructure stack would not only improve the ability to build and operate our software but also give our users tangible benefits, a small team formed around a core vision:

An agile, modern infrastructure stack owned by empowered delivery teams.

By design, this team would be temporary. They would prototype and clear the way, then coach delivery teams to take ownership of their infrastructure.

Experimentation and Prototyping

Before we could get started migrating our applications, we first needed to get a feel for the new platform. We needed a realistic testbed that would take real-world constraints into account.

To accomplish this, we decided to deploy a minimal subset of our existing application landscape, consisting of part of our edge layer (responsible for, among other things, managing authentication of requests) as well as one of the simpler applications with a minimum of dependencies to external systems.

This setup was simple enough to iterate quickly but also complete enough to run complete user journeys and to test a number of more complicated requirements like dependencies between systems, encryption in transit and at rest, secret management and monitoring.

Using this testbed, we were able to figure out which technologies and approaches worked for us, which solutions we want to carry over from the existing infrastructure and where we needed to find new solutions. We could experiment with migration strategies, find dependencies and pitfalls. We could extend it to new use-cases, allowing us to learn aspects we didn’t know before.

Scale it

After a while, our testbed evolved more and more into a production-like setup we were comfortable with. We felt we had covered most of the critical aspects and found solutions that worked for our case.

At this point, our testbed became the seed for our new infrastructure. Since it was built from the start with the same approaches we would use for a “real production” system (testing and monitoring, infrastructure as code, resilience), there was no need for re-engineering. We could use our existing testbed infrastructure and scale it.

In our case, this meant starting to hand over responsibilities. While application delivery teams were always in touch with the experiments of the cloud enablement team, now was the time to get more hands-on. To work together to create infrastructure they would own and evolve, for the applications the team owned.

This is where we are right now. Team by team, application by application, we are moving pieces onto a new kind of infrastructure, learning, adjusting and improving as we go.

Learnings

While this process is far from over, we already learned quite a lot. So here’s a collection of things that we found out, that worked well for us or learned the hard way.

Be prepared for surprises and plan accordingly

It is in the nature of transformation that not all aspects can be planned reliably. You are trying to establish something new, something your organisation doesn’t have experience with so necessarily there will be surprises, there will be things you didn’t expect. Things might take longer or won’t work in the way you want them to. Your stakeholders should be prepared for this, be on board with it and account for it in their planning.

Use realistic examples and build for production from the start

When starting something new, we might be tempted to start with something that is simple, to “do it properly after we are done experimenting”. This divorces your experiments from reality. You end up with situations where your results don’t work when tried “for real” or where a lot of rework is necessary.

Instead, try to find a small subset of what you are trying to do and do it as if it were a production system: Build testing and monitoring in from the start. Automate. Create delivery pipelines. Design for resiliency and scalability.

You don’t have to do all of this at once but until you took those aspects into account, you won’t discover the little edge-cases, hidden constraints and intricacies that will make or break your solution sooner or later.

Find out what you can reuse but don’t shy away from building new things

When moving from on-premise to cloud infrastructure, it will be tempting to reuse as much as possible. After all, you probably already have a lot of infrastructure code, custom deployment tools, technologies, architectures and processes that contain years of organisational knowledge. And some of those can be very valuable. Others don’t apply in the new world and there’s no sense in keeping them around: For lots of solutions, the cloud works fundamentally different from your on-premise IT.

As one example, on-premise datacenter often work with long-lived, mutable servers while cloud providers optimise for a model where individual servers are unreliable, short-lived and immutable. This has consequences for the way you build and maintain your infrastructure and for the tools you use:

• Deployment changes from deploying a new artefact to replacing the whole server
• You roll out new images instead of orchestrating changes from a puppet-master
• Scaling up and down based on demand replaces static upfront capacity planning

Make sure you start early working on organisational aspects

While most of this post focuses on technical aspects, we shouldn’t forget that in large organisations, non-technical or organisational aspects will dominate a lot of what you do.

Architecture Committees will want to know about your plans, IT Security might need to sign off. Accounting and Procurement need to figure out how to pay the bills and business units will want to know how this affects their existing features and your ability to build new ones.

Most of these things will take a while to sort out. They can derail a project if we don’t. So involve people early and work with them but don’t let them set your agenda: You are trying to build something new so by definition, it will be different in some sense so standardised approaches will not apply.

Build for your teams current capabilities but design for growth

As a team with the mission to enable others, one of the hardest tradeoffs to make is how much abstraction to design upfront vs. how much freedoms to allow individual delivery teams. Do we want delivery teams to have a single “deploy” button and that’s all they can do or do we want them to design their own networking structure, CIDR-ranges and all?

Too much flexibility means more things to learn and keep track of for the individual, more things to go wrong and more inconsistencies across teams. Too much abstractions means longer lead times (because abstractions need to be built), less flexibility, more dependencies and the very real danger to build the wrong abstractions.

The truth will almost certainly (unless all your employees already are expert cloud engineers) be somewhere in between and will depend on your organisation. But as a general rule of thumb, try to find a solution that works for your audience’s current capabilities (to allow them to learn and adapt quickly) but design it in a way that allows for them to grow out of them eventually, to drop the abstractions when necessary.

For example, for us that meant starting to prepare Terraform modules for some common infrastructure pattern (e.g. a web application with a load balancer). Teams can use them if the default works for them but can always choose to abandon them and work on a lower layer and with different tools if they need a more custom solution.

Prevent scope creep, aim for production as early as possible

Our approach started with a phase of experimentation, prototyping and preparation. It allowed us to gain confidence and find solutions that we could recommend to delivery teams.

It’s also very vulnerable to scope creep: There’s always another thing to explore, another thing that “just needs to be done before we are ready”, another thing that just “isn’t good enough yet”.

So keep in mind that in isolation, this phase doesn’t add value. Value is only added once delivery teams start building on top of it, once real software is running on real infrastructure for real users.

So aim for production early. Aim for production not when things are perfect (they never are) when things are just barely good enough. This will tell you what’s important and what’s just nice to have. It will give you feedback if you are on the right track and tell you all the things you missed.

Modern Infrastructure for Modern Software Delivery was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.

As part of our digital transformation at Porsche the mindset of the people plays the most important role. But agility, speed and organisational culture must also find its way to the product teams and sooner or later we talk about speed in development teams.

In this article I will describe how Apache Kafka helped us to decouple microservices and led to a more independent software development process by dissolving dependencies between both organisational units and technical components. The result is an even better velocity in the teams and an even more reliable event backbone.

Circumstances

You might think of delivering software in ONE development team is not a big deal? You are probably right when the following requirements are met:

• the team understands the requirements and acceptance criteria that are described in the epics and user stories
• the scrum roles are fully established
• the team members trust each other and have a good personal relationship
• the team is passionate
• the team uses the technology stack on which it performs best
• no dependencies to other teams

Basically, this is an ideal state and during my 12 years experience as Project Manager I have never seen such a perfection in regards to these circumstances.

Think of complex Connected Car projects: beginning from R&D, Electronic Control Units (ECUs) in a vehicle up to Vehicle Documentation Systems, various After Sales and Sales Systems (CRM System, Identity & Access Management Services, Shop Systems, License & Activation Management Systems for Connect Services, Picture Services, Messaging & Communication Services for customer notifications, etc.) to only mention a few of them. There are massive complexities and huge dependencies between various development teams.

In 1967, Melvin Conway was right when he said that the complexity of the IT-landscape strongly depends on the complexity of the company’s organisation and communication structure (Conway’s law). The more organisational units the more heterogenous IT systems.

The complexity mentioned by Conway posed a challenge to our development teams: the vast amount of dependencies between legacy systems, microservices and their organisational ownerships led to communication overhead, many meetings and too many bounded developers for harmonising both interface integrations and test(-data) management between these systems.

Long story short: the dependencies between teams and organisational units grew and team members were most of the time arguing in meetings than developing new features.

The result: the development teams’ velocity slowed down.

For sure you can rethink the size of your microservices and shift its organisational ownership to improve the communication channels and hence speed up the development of components. But you definitely want to avoid that technical components are tailored to particularly fit to organisational units that either might change in future and/or the component’s ownership will eventually change. Remember Conway’s words!

Welcome Apache Kafka

The point at Porsche came pretty early when we had to decide how to solve these dependencies on a technical layer.

At the same time we talked about „process-like“ microservices that encapsulate the overall business process and in turn orchestrate downstream microservices. But this kind of architecture collapsed already on the question of the product ownership of that „process-like“ microservice since processes usually span business domains.

Additionally, we doubted that introducing a new microservice orchestration layer will really speed up our development process since it increases the complexity, apparently.

So, we decided for a hybrid event-driven architecture. What does that mean?

Actions, triggered by the frontends (e.g. Portal- or Webshop-based-microservices) always result in synchronous transactions and synchronous communication with downstream components. This synchronous communication pattern always guarantees that the Frontend-Users get a meaningful response to their CRUD (Create, Read, Update, Delete)-operations and data consistency in frontend widgets is ensured during navigation.

The synchronous main transaction (e.g. establish/release car relationship between owner and vehicle) either succeeds or fails, i.e. there are no doubted transactions.

Microservices themselves might initiate several synchronous calls to downstream components and are hence responsible for the consistency of the data that are in turn provisioned back to the frontend user. But this is all done in a synchronous way.

The synchronous Create, Update and Delete main transaction usually results in a changed state from a business perspective, e.g.:

• a car relationship between owner and vehicle released successfully compared to the previous state when the owner-vehicle-relation existed
• a connect service was successfully activated in a vehicle compared to the previous state when the service was not available in this specific vehicle

In these cases the microservice that successfully finished the main transaction publishes a document-based “state-changed-event” which is then subscribed by various microservices. The microservice that initiated the main transaction is out of responsibility after having published the “state-changed-event”. Furthermore, the publishing microservice (event producer) does not even know if — and if yes, — which microservices subscribed to the “state-changed-event”.

Subscribing microservices (event consumers) are now in charge of consuming this event and of processing it in a new transaction. The event consumers act independently and are self-contained from both a technical and business point of view.

Between event producers and event consumers, Apache Kafka is established as publish/subscribe event backbone.

The big advantage: both the technical components and hence the development teams are decoupled in regards to the „state-changed-events”. This circumstance finally lead to faster test results, less integration efforts and hassle free integrations with help of the publish/subscribe pattern.

Why Apache Kafka?

Apache Kafka is an open-source distributed streaming platform that we evaluated last year in a three-months Proof-of-Concept (PoC) phase. We decided against other queuing platforms since Apache Kafka offers more than just realising the above described publish/subscribe pattern.

The main benefits for our use-cases were

• continuously running real-time query capabilities
• excellent topic/partition management
• very high throughput
• low administrative overhead
• many settings are decentralised, i.e. the event consumer decides what rollback, timeout, etc. behaviour fits best for its specific use-case
• both decentralised settings and administration capabilities fit best for a DevOps approach
• Open-Source Software (-license)

After the successful PoC we decided that Apache Kafka is the perfect match for our event backbone: it decouples our microservices, is easy to handle and — most important — helps to speed up the overall development process. Additionally, it is a future proof streaming platform for both vehicle events and event analysis.

Furthermore, we decided for Apache AVRO as data serialisation system and exchange format in Kafka messages and checked-in a blueprint consumer and producer code in our bitbucket for the sake of re-use in other development teams.

After six months running a clustered Apache Kafka in production more and more development teams within Porsche but also from other corporate group brands ask for experiences with the platform and I usually highly recommend this setup.

In some cases the use of traditional queuing systems makes definitely more sense and should be taken into consideration but in our case at Porsche Connect, Apache Kafka was the first choice.

What comes next?

We are just in the process of establishing domain-specific (e.g. After Sales, Sales, etc.) Apache Kafka hubs at Porsche and interconnect these hubs with help of Apache Kafka’s Mirror Maker. This will offer new scalable and flexible possibilities across business domains especially if you think of moving some domains into the cloud.

The journey has just begun. Come along with us on this journey.

Apache Kafka at Porsche — Writer meets car manufacturer was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Moving from waterfall to agile

At Porsche we had a standardized approach for IT projects: more or less a waterfall approach with lots of documents to write. The business side had to define the requirements, after approval of the requirements specification the IT guys started with technical specification and implementation, finally the software was delivered. For Porsche Connect with a very challenging time constraint and lots of uncertainties we had to change this approach fundamentally.

A small group of people, business and IT, started to push an agile approach. All people needed to develop the software products were organized in cross-functional teams. Writing documents was replaced by writing code and delivering working software as early as possible was the main goal. One of the most demanding tasks was to overcome internal concerns about the agile approach. We had to convince lots of people that it‘s worth to follow the agile path. Beside of training the Scrum methodology, the change of the people’s mindset was a very important enabler for our small agile outpost. Delegating responsibility to the teams, encouraging the teams to make decisions on their own in the best interest of the product and to trust each other was crucial for our success.

Agile at scale

Now we had this small group of people embracing the agile approach. But how to scale this out? We had to deliver a huge amount of features with a fixed deadline and reducing the scope was not an option. Thus we started to setup more cross-functional product teams with specific feature responsibilities. Each team should be capable of delivering features without or little dependency on other teams. Actually we tried to adapt the well known approach of Netflix and Spotify and we had some very inspiring talks with guys from Otto. The more we scaled out the more we struggled with dependencies between the teams and dependencies to the car. These dependencies made us slow and we had lots of trouble because we couldn’t deliver working user journeys as soon as more than one team was involved.

We hade to find a way to reduce these dependencies or to manage them. Getting rid of them was impossible in most cases. For example the car was one of these fixed constraints we couldn’t change. So we decided to manage the dependencies by making them visible for all teams. Furthermore we introduced an agile portfolio which allowed us to align all teams regarding scope and time. With the agile portfolio all teams should have the same goals and priorities in mind when implementing new features. This approach helped us a lot to deliver working user journeys continuously.

Another obstacle was lack of communication. Of course we had our regular meetings. But after the meeting each team went back to work and there was only little collaboration between the teams. A turning point was when we introduced HipChat to enable communication between all teams. This was the starting point for building up guilds. People with the same interests or roles started to communicate via chat. For example the UI/UX guys had a regular chat about the overall user experience. They aligned concepts and designs without any further push from outside.

Enabling communication and collaboration was definitely one key for successful product development. Another success factor is trust and to share the same mindset. We started to have events together with all teams. One very successful event is a product marketplace, where all teams are invited to show the current state of the product and everybody can discuss with each other and share information. Today we reach lots of people across the whole company with these events and it helps us to build up a community for our digital products.

Summary

One main thing we learned is that there is no blueprint out there to get an agile approach up and running within an organisation. We had to find our way, the Porsche way, and there were and are lots of challenges. The journey has just begun and there is more to come. Software is getting more and more important for the car manufacturers as customer expectations are changing rapidly. Digital transformation is one key to cope with the future challenges. But besides of technology and methodology it’s all about people, how they feel, how they act and how they work together. A lot of work has to be done in the area of mindset and cultural development and I’m happy to share our experiences in this blog.

Shifting from a car manufacturer to a software company was originally published in PorscheDev on Medium, where people are continuing the conversation by highlighting and responding to this story.