Working on Event-Driven Architectures is one of the most satisfying experiences for software engineers: it’s stunning to see how many simple actions collaborate to form emerging behaviors and make the magic happen. Saga pattern is an elegant way to design long-term, distributed transactions in a BASE fashion, fostering scalability and resilience in distributed, reactive systems that can afford huge workloads, still providing a good user experience

In this article, we’ll see how to implement the saga pattern without using specialized frameworks (like Axon, Eventuate Tram, etc, which are mainly used in the context of CQRS/ES/CDC implementations). Our goal here is to improve the understanding of the mechanism and see how it can be achieved even using a “vanilla” stack, for example with the popular Spring Boot/Spring Data. For the implementation, I’ve used Kotlin because it’s a nice chance to keep practicing with this modern and elegant JVM language 😊

NOTE: through this article, it will be assumed that the reader has a clue of what Saga Pattern is and how it works: there are a lot of articles and papers which explain it comprehensively. Despite that, if you’ll enjoy this story and you want to read from me an in-depth description of the pattern and the fundamentals of distributed systems, just leave a comment here!

TL;DR
here is the code : https://github.com/cingaldi/sagapattern

Analysis Phase

Requirements

Let me introduce our use case:

We are working on a trip management platform: our customers can create Trips where they can book flights along with hotels. We can handle both these bookings using codes that will be used by 3rd party services. In case the flight and the hotel room are successfully reserved, we notify the customer that the trip is confirmed. In case the trip confirmation fails, we notify the customer

First things to notice:

• The trip can be confirmed once flight and hotel reservations are. Both steps are assumed to happen asynchronously. Therefore, they can give us an outcome in seconds, hours, or never
• For the sake of system performance, all the interaction should be executed in a non-blocking fashion
• If for some reason, a flight or hotel cannot be reserved, we should revert the entire operation. So every booking should be canceled (look here at how emerges the concept of eventual consistency)

Modeling these requirements with a micro-Event Storming, this is what we are trying to achieve

Take a look at the purple sticky note there: this is called Policy and indicates that in our workflow there is a step that requires some decisions to be taken, potentially outside our system. In this case, we can summarize it summarized as

• whenever flight and hotel are booked then confirm the trip
• whenever a flight or hotel are canceled then abort the trip

Usually, a policy translates into a saga

Architecture

Now imagine we are working in architecture like this

We see that we are ready to handle the workflow asynchronouly and provide a consistent façade towards external services. The whole transaction will be coordinated by TripService which exposes a REST API to create a trip

Building Blocks of the Saga

The Saga Manager

This is the entry point of the Saga and acts as a glue code: In a nutshell, it implements a listener for each event involved with the Saga and decides what command to dispatch after which event. Spring framework offers an event bus out of the box, so we won’t struggle a lot on this detail.
Notice that the Saga Manager belongs to the application layer: it doesn’t implement real domain logic, it takes only care to choreograph the flow

So why couldn’t just add a bunch of event listeners to the TripService? The answer of course is SRP, and pointing out that an application service wraps an aggregate root while the Saga Manager has not just to deal with one aggregate root

Saga state and Saga Repository

The saga state (or, for the sake of simplicity, just saga) is what sits at the core and is a domain concept, so — in general — we can find it in our ubiquitous language with a name that represents a Status, a Progress, a Process or something similar. It represents a part of the application state, so we can fairly consider it as a Domain Model object, just like entities, aggregate roots, or value objects. It’s persisted so it will have an associated saga repository. What we expect from the Saga is to

• Make data associations Since saga reacts to events coming from disparate aggregates, we need to bind the saga with each of the aggregates that participate in the choreography. We can see in our code that hotelCode, flightCode, tripId uniquely identify the saga. For this reason, the saga repository defines query methods to fetch status associated with them
• Keep the actual status The saga can persist whatever you find relevant to make decisions on each step. Notice that we are defining methods that contain that state transition logic, it’s a good OOP habit to encapsulate object state and keep behavior close to data
• Decide what’s next every time the saga status changes, some commands can be dispatched, this is how the saga coordinates the involved components. Thus, every method decides what command to apply next

The Command Façade

Ok, this isn’t matter of life or death, but if you want a nice separation of concerns, an expressive code, and higher extendability, this pattern is just for you. In a nutshell, we are implementing the concept of command as a message that flows into a bus and matches the correct handler which, in turn, will apply that command over an aggregate. If this seems similar to what an ApplicationService/UseCase do, you’re not wrong: it is but in a more decoupled way

There are few ways to implement this pattern, the easiest is to provide a proper façade object which will wrap dependencies on needed services and exposes as many functions override as more command objects we have. The most complex way can use reflection and annotations to get the matching between the command and the correct handler. Let’s just stick with a halfway solution that provides enough flexibility and uses the sweet Kotlin’s syntactic sugar

Handling Deadlines

Don’t forget that a saga is meant to handle asynchronous processes. And if there is a thing that all asynchronous things do, is to not show up when you need them. The Hotel server can crash, the flight company can run out of business, so sometimes we need to wrap up somehow

For this reason, saga can come with deadlines. A deadline is a particular event listener that is triggered by a time-based event. It can be triggered after some time after the saga starts. In both cases, despite the solution looks trivial at the SagaManager level, its implementation depends on the available infrastructure tools

You can just use a cronjob or an external trigger. Furthermore, some message-oriented middlewares (like Amazon SQS or, with some hacks, RabbitMQ) provide delayed message features that let you dispatch a command at saga start, like “after 10 minutes, trigger a timeout event”. At this point, we will make our saga gracefully fail

In our case, we will just adopt a quick-and-dirty solution using Kotlin coroutines. Now it’s just a matter to extend the code to persist the scheduled tasks and serve them robustly. Moreover, like optimization, it would be nice to provide the saga manager with ways to locate the task and cancel it in case of successful completion

Testing the Saga

Usually, we can get how much code is good from how tests look like. It’s important to have a simple way to test the entire saga, defining all the possible paths the workflow can follow and asserting that the outcome is the expected. After working with Axon Framework, I’ve felt in love with their fluent, behavior-driven testing tools. Basically here we want to test (also) the SagaManager, and the test should be like:

given(SomePriorEvents)
when(CertainEventTriggered)
then(DispatchedThisCommand)

Of course, asserting against CommandDispatcher helps a lot. Moreover, to keep the state across the step invocations, we can use a fake SagaRepository which holds an actual instance of the saga. This can increase test confidence concerning a solution where we use stubbing to test individual steps

What’s missing?

• We have only implemented the happy path, but what happens when either hotel or flight is not confirmed?
• We are not really managing internal failures. How to overcome exceptions due to concurrent writes? What should happen if the message broker fails to distribute tasks to Hotel/Flight services?
• A flight (or a hotel as well) can be canceled even after the confirmation, and this can trigger refund policies for the customer. How can we extend this saga to fulfill this part of the process?

Conclusion

In this article, we have implemented a small application that choreographs a distributed transaction thanks to the Saga Pattern. The use case was fairly simple and the implementation quite far from being production-grade. Nonetheless, it was fun for me to develop it and also gave some little challenges. Hoping that it was the same for the reader, I’ll conclude the story with a hint: if you want to know something better, just code it from scratch!

Case Study: A DIY Saga Pattern Implementation was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

When I’ve entered in this mad industry, Scrum was a thing. Nowdays seems that the dream has come to an end for everyone; Disillusion started sneaking inside startups and enterprises. Oppression for developers, confusion for the business, the medicine that became even worse than the desease that used to heal, that’s how many Scrum teams started to crumble

In this article we’ll go through common criticisms and disbeliefs around Scrum, using a fictional back-and-forth in which I’ll try to address main issues I’ve found during my experiences and research. At the bottom of the article, the reader can find some references that inspired me and gave me some food for thoughts

DISCLAIMER #1: As of today, when we talk about “Scrum” we are actually talking about a blend of the original definition of Scrum methdology and eXtreme Programming: for example, it’s common belief that Scrum involves usage of User Stories and story points for estimation although these concepts are part of XP practices. This is just one of the implementations of Scrum, but for the sake of simplicity we will assume that this is the way we intended to apply Scrum

DISCLAIMER #2: It’s frequent to assume that Scrum IS Agile or that Agile is achievable through the usage of Scrum. Nothing more wrong than this: Agile is a mindset which drives a company towards its goals while Scrum is a project management framework (which usually fits with software projects) inspired by the same principles as Agile. turns out that Agility is rather a pre-requisite that helps creating successful Scrum teams.

Everything as an MVP

I really like the idea of creating incremental changes and continuous evaluation of our priorities, but that’s not how it works in my company: stakeholders expect MVPs within certain deadlines to make evaluations and respect business plans

Let’s recall that MVP stands for Minimum Viable Product. Although sometimes the term is not used according to the initial intended meaning for MVP, in practice we do expect that we are developing a “product” that is

• Viable: it’s possible to create a working version with given resources (time-people-money)
• Minimum: contains only characteristics that add enough business value to accomplish MVP goals

It may seem that the definition of an MVP goes against Scrum way of working, but it’s not: every Scrum sprint can end with an MVP, but in some contexts MVPs become so little “minimum” and “viable” to transform themselves into weird hybrid monsters halfway between Agile and waterfall

As an engineer, too much time I’ve had conversations that were like

<<Hey Carmine! So, you’ve studied requirements, you know the code and the people, how long do you think you guys will take to implement all these stuff?>>

<<Well, to be honest I’ve found a lot of corner cases, some of them are not addressed in the reqs, moreover we will need some refactoring and accurate regression tests. I think this project is quite large and it will take 6–10 weeks, if nobody gets sick>>

<<Oh, that’s a very bad news man! Directors wanted to see this by the end of the month because they want to unveil the new feature in the FooBarEvent, you know that all the world is gonna watch us there>>

The precise idea behind a Scrum working model is to take this conversation as a starting point, and apply it to the MVP in order to (re)define a scope. We cannot increase the resources and it’s very important for the company success (yeah, the company that pays team members and promote them) to not take more time than the expected. So the only thing left to do is to discuss on the scope

But how to do it? estimations are always guesses, there is always the risk of neglecting important cases or just end up writing code for the most critical feature just the day before the demo (one day I will tell about how I ended up implementing a feature in a car, driving on the Bologna-Milan highway, but that’s another story). Application of Scrum methodologies already overcomes these issues. The MoSCoW approach can be used to assess the impact of the planned activities over the final (initial, indeed) proposition of the product. This, though, needs to be carefully shaped with a PM/PO who has enough autonomy and buy-in to take informed decisions along with the team. Proxy Product Owners are an anti-pattern found in many organizations where the Scrum way is not fully understood or accepted

Deliverables are not Releasables

Let’s be honest: in an ideal world would be very nice to work on small user stories, let our customers see our platform improve day by day, but most of times the way we split features into user stories just doesn’t make sense! what’s the point of implementing a “checkout” button in the e-commerce section without the “clear cart” button?

There is a common misconception about user stories and sprints, and it starts from the idea that, at the end of the sprint, we have to release some value while user stories should represent a minimum increment of value. So you may wonder about “what it’s really valuable? do we only care about the marginal value we introduce with a story? or should there be some interaction between factors?”
Not the first nor the latter. We can simplify the problem just separating the concept of delivery from the concept of release. Let’s provide some definitions:

• Deliverable: An activity produces a deliverable when meets the expectations set up through a “definition of done”. Delivery of the feature is a tactical concern and tech team is responsible for ensuring it. For example, we could consider a story as the minimum deliverable feature
• Releasable: A feature is releasable when it provides enough increment to product value that the customer is ready to start using it. Thus, release of a feature is a strategic concern. For example, we could consider an epic as the minimum releasable activity

So it’s clear how, in the Product Backlog, we are creating a two-layered planning which has the goal to define a roadmap that encompasses both tactical and strategic standopoints. Turns out that we cannot correctly apply this mechanism without the right toolings that ensure an effective separation between delivery and release. Tech industry practies are fundamental in the waypath: Feature Toggling systems, DevOps culture, SRE methodologies and all related and alternatives approaches (as well as release strategies, experiments) are the tooling that you will use to take your recently completed item from the backlog to production hand-by-hand

Why this is Overwhelming?

Oh, I like Scrum, I understand the benefit it’s bringing to us, but we have the planning meeting, then the demo…and what about the infinite grooming sessions? without forgetting the endless daily standup, then the retro..How much time is left to work? why can’t I just assign a task on the fly?

Some people took Scrum as a religion, some people took whatever as a religion, perhaps we just need to believe in something to turn off our brain and be confident that our behaviour is the best and will take us to paradise.

But, if Scrum is not a religion and we won’t follow the Bible, what Scrum really is? How to make sure that your idea of Scrum does what Scrum supposes to do while addresses the problem stated in the quote?

First and foremost: the problem is not that you have too many meetings or that they take too long, the problem is that you are not improving your process. Scrum masters are made for this: they should not be project managers “in disguise”, but people whose goal is just to analyze problems in the day to day job and propose remediations, observing some sort of team level OKRs

The planning meeting takes two hours because people discuss too much about stories descriptions? extend the grooming session, set up 1:1 calls with POs to pre-refine backlog items, kill the dev who talks too much about code during the discussion, do whatever you need but keep in mind that the idea behind most of the Agile-related methodologies is that those methologies are not a recipe for the success but a forma mentis that needs to be trained and refined to meet the necessary goals…which are

• Eliminate unclarity: everyone knows exactly what it’s expected and they know so much about it that the solution is trivial
• Provide visibility: everyone knows everything about everything in the team and if a guy has a problem, another guy has the solution for it
• Work for a goal: The final result has to be tangible and it’s clear what are the motivations that drive the team towards it. This is true for business side with respect of technical activities as well
• Track the progress: Every day the team knows exactly what else is needed to reach the goal and is able to predict if the goal can be reached. Everyone should have a feeling about whether the overall performance of the team are improving or not
• Enable Feedbacks: every action should lead to an outcome, people have room and confidence to provide a transparent feedback and use it to improve the product and the process
• Self Organization: the team has all the tools needed and provided from the outside in order to reach the goal, they are empowered to take decisions without interference

As you see, Scrum is not about the meetings but the purpose of these. Shape it in the most preferred team’s way to reach these goals

Efficiency or Effectivness?

From when I’ve started working with Scrum, the only thing that matters is to get the job done. All the team is focused on delivering stories at the end of the sprint, while tech debt increases and we have never time to really do something innovative

You see here, Scrum is a methodology which should adhere to Agile principles. It’s not useful here to recall what Agile is and what principles it fosters: the ugly, old (but gold) Agile Manifesto website is enough for us. One of the principles that I like more, and I’ve found totally fitting with the Scrum process is

Individuals and interactions over processes and tools

It’s all about that: the success of a project is the result of an equilibrium between business needs, technical concerns and the collaboration that both ends have to define a roadmap. Only great professionals will create a great Scrum team, that’s the sad thruth and I suppose there is nothing else we can do to fix this*: it’s time to admit that every solution that promises to make a product look better than the people who have built it, is a false promise

So how we can leverage this epiphany? At te core there’s negotiation, trust and mutual understanding. Business needs are not more important than technical concerns, they have just a faster payback period. Technical excellence, in turn, is a consequence of the business needs, since solutions are good only when they solve someone’s problems.

Scrum teams should have an high velocity, not an high speed. They have to know when to sweep under the carpet and when to face the problem. Two-weeks sprints are not always the right size for having a feature done in the proper way. There should always be some room left for improvements, like solving tech debts, making experiments. All these activities have a part in the process and provide a value increment for stakeholders: because the team itself is a stakeholder, am I wrong? Summing up, the fastest path to a failure is to worship efficiency, fearing the effectiveness. Turning your process into “eating tickets” without a purpose-driven mindset, and no means to evaluate the future impact of today’s choices will kill every business regardless on what project management framework it will adopt

*except for training people or making agile skills assessment as part of the hiring process ;)

Conclusions

At the end, my personal take is that nobody really dislikes Scrum or criticizes against the purpose of Agile. There are so many deviations that ended up wrapping the old way of working into new fancy empty boxes, hoping to get problems solved.This made drop the trust on those things. What I’ve learned during the last years working in/for Scrum teams is not only that Scrum alone is not a magic formula, but the key of the success is to abandon the idea that a magic formula exists. Agile, at the end, was made to embrace the complexity of the new world we have built, not to hide it behind reassuring recipes

References

• Why Scrum sucks | TechBeacon
• The End of Agile
• Ron Jeffries Says Developers Should Abandon "Agile"

Why Don’t We Like Scrum Anymore? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

From its birth in 2003, Domain Driven Design doctrine has, in some way, changed the way we look at software architecture. This is even more true at the verge of the 20s, when microservices are a serious thing and we all have rediscovered the importance of Eric Evans theory and his “strategic patterns”. Most enterprise frameworks are based on DDD concepts, but only few of them consider what happens in the domain layer since, by definition, that is the territory of domain experts and very few patterns could be enforced (unless you are into CQRS/ES!)

Almost each of the DDD implementations start from a Service. But “Services” are a very broad concept in computer engineering, there are many kind of services as many roles a generic component could have in an app architecture. Most of times though, we build layered architectures where an Application Service segregates domain logic, which soon becomes a Transaction Script that implements behaviours for anaemic domain models. To prevent this, I’ve found that is important to implement an often overlooked object but originally discussed from Eric Evans: the Domain Service, which is able to separate domain logic from application logic when it’s not clear where some operation fits better

So what are Domain Services?

A service is an operation offered as an interface that stands alone in the model, without encapsulating state, as entities and value objects do. Services are a common pattern in technical frameworks, but they can also apply in the domain layer.

(Eric Evans 2003 , DDD Blue Book)

If you feel that things are not more clear after reading the official definition, I can tell you how I personally see them: Domain Services are domain objects that don’t encapsulate data, just expose behaviours. But let’s go deeper

Domain Logic vs. Application Logic

To have a clue of the subject we should try to address this thorny issue.

This how a DDD Layered (onion-ed, in this representation) Architecture looks like, at its core

Here you can recognize that two components play a key role: the Application Service that exposes API to execute entire/parts of business workflows (Uncle Bob would call them Use Cases) and Aggregate Root that acts as a façade towards the aggregate. Therefore, their partnership defines the Business Logic

So, which part of the logic belongs to which layer is always a debated subject, but two remarks can help us to make the right decision

• In layered/exagonal/onion/whateverNameItsAlmostTheSame architectures, the upper layers implement policies and the lower layers implement rules. In brief, we go from “What to do?” to “How to do it?”
• Domain Logic is, precisely, Domain. It’s distilled from domain knowledge. It’s what a domain expert usually does by hand before you implement a software to replace him/her. If the answer to the question “Should I have domain knowledge to understand this code?” is “Yes”, than you know where to put it

In case of Doubt…

Now it looks clear why Anaemic Domains harm our projects. They mix the “what” and the “how”, violate the most traditional concepts of decoupling and separation of responsibilities, so the software is difficult to understand, maintain and -worse- extend. A good indicator of how much rich a domain is, is to observe how application logic looks like. In my ideal world every “*Service” object method, if used to execute commands, should just

1. load an aggregate root
2. call a method on the aggregate root
3. save the aggregate root

Too bad, this always remains in my dreams. Sometimes I find out that some domain logic is not aggregate logic in sense that does not fit with aggregate responsibility to just check invariants (using its own data), mutate its internal state and, bonus, emit an event. The reason can be motivated by essential domain complexity or just that anaemic domains doesn’t permit to add aggregate behaviours easily.

Here domain services come into play. They follow the ubiquitous language, and resemble operations, so they are called “ClosestPoint”, “DocumentFormatting”, “BillCalculation”. As we said, they don’t hold state. This doesn’t mean that they have to be pure functions, being stateless doesn’t prevent to have dependencies from other objects, potentially for the sake of refactoring. Moreover, Domain Services could be just ports that have their implementations in the outer circles of the architecture…But no hurry! Let’s discuss about it with some code example

Where The Domain Service Fits?

Move Things With Aggregates

First rule of Aggregates is: You do not talk about Aggregates

Aggregates instances cannot collaborate with each other (including other instances of the same aggregate), and this will save us to think about concurrency problems and failure modes. In the real world, though, aggregates sometimes collaborate together. DDD addresses this aspect through domain events that can cause domain actions outside the triggering aggregate but, as we know, events imply async interaction patterns

There are situations where we want to avoid this: sometimes just for the sake of pragmatism that suggests us to not overengineer simple things, but sometimes instead, it’s matter to guarantee transactional behaviour to our business logic. Distributed* transactions in event driven applications are sagas and they’re an overkill if not needed to coreograph complex flows between remote systems through significant time spans.

Let’s look at an example where a banking application performs transfers between its own customer accounts in a transactional fashion**

So we choose just to load aggregates in a single use case executions and work with them…but where to put that “orchestration logic”? Here domain services come into play

Here the domain service hides business decisions and operations from the application logic, which is always good. Careful readers have already thought that some of the logic performed into domain service are about checking invariants, so they could belong to the aggregate logic. This should not be taken for granted as not every check is an aggregate responsibility, only those who are domain-relevant:

• Checking account balance against transfer amount would be a withdraw() resposibilty
• Checking that transfer amount is below the limit is not related to any account (maybe to a transfer VO? )

This will keep aggregates as small as possible and, the rest would belong to Domain Services

*don’t forget that even monolithic applications are distributed systems: they communicate with databases!
**Some technologies, such as Spring Framework, permit transactional behaviour even in interactions driven by local events. Since this is a technical detail, I’m not taking it into account for this example. Moreover, there are some pitfalls in that approach that should lead to a careful implementation

Create Strategies for Aggregates

As already mentioned, we separate domain logic from application logic using the following rule of thumb: What to do or How to do it? In some cases the line is more blurred than in the others, especially when there are multiple ways to do it and the decision on which strategy to choose is based on information that come from elsewhere (in the next paragraph we will go deeper into it). In e-commerce applications, for instance, when customers do checkout, discount campaigns would be applied on products if ever some rules were matched

Here we are doing odd things in application logic because we need to make a decision. This decision could be made directly from the aggregate (in checkout() method body) but this leads your user to know about the calendar and, worse, ties up you core domain logic to policies that change frequently and in unpredictable ways. The reasoning is correct, the implementation could get better:

In this implementation, application logic still decides what to do (check if some kind of discount can be applied and than complete the purchase, which is the same thing a casher would do in a physical shop!). The how to do it part now belongs completely to domain logic but we got both things achieving a good extensibility and clearly separating responsibilities.

Two remarks:

• in FixedRateDiscountStrategy I’m passing the rate as a property, It wasn’t really necessary but I’ve done it to create confusion! Now that the object holds data, is it still a Domain Service? The answer is no. That now is a Value Object. To be more precise, it’s what Eric Evans defines as a “Specification”. The difference here is subtle again, but the most important thing is that we can use Domain Services to define different behaviours, apply rules outside the aggregate boundary and then make it evolve to something else, if needed
• Making a Domain Service instance pass through aggregate boarders it’s not very good in general, since it creates coupling. But here the aggregate just knows an interface contract, so it’s ok even if we can see domain services as a thiny upper sublayer in the domain side

Observe External Status

Many scenarios cannot be executed just relying on information that can be retrieved from the aggregate. Interactions with other Bounded Contexts are unavoidable. Domain Services can act as abstractions of external services

Here we have slightly changed out checkout use case implementation, ’cause product management has decided to introduce promocodes to distribute discounts. I suppose that promocodes are a concept that belongs to marketing function, so they don’t have a meaning in the user purchase BC. Nevertheless, I still need it, so promocodes are entering, in some extent, into my domain, but still doesn’t belong to any aggregate, nor we want to hydrate Promocodes in our BC listening to events. So this is a Domain Service job.

As every services, they could follow the “port and adapters” pattern: they don’t have to hold a state, so Domain Services could be offered as an interface (port) in the domain layer and be implemented (adapter) at infrastructure level, to implement communication with external system (roughly speaking, the HTTP call to Promotions-service) . And what about monolithic apps? A simple approach here could be the following

package com.myecommerce.promotions_srv.promocode.application

public class PromocodeService extends PromocodeInfoProxy {
    
    @Override
    public PromocodeInfo getInfo(String promocode) {

    ....

Very ugly but still pragmatic. A step behind writing and inter-BC adapter that will be replaced in few minutes with an HTTP call, making microservices splitting easier

Refactoring an Anaemic Domain?

Almost everything in the examples that we have seen could be done in a different way, ofter this way is more DDD-friendly. Aggregates should not be mutated in the same Unit of Work, many calculations find place directly in entity methods (or in projections, if we switch to CQRS) and finally, if you need to combine data from multiple aggregates, even in different BCs, you should ask yourself first if you are doing your design correctly

Yes, true…but quoting Alberto Brandolini

Software development is a learning process, working code is a side effect

So, if it’s true we should start focusing on learning and evolving our code. Refactoring just means coding new things we have learnt from the last commit, and we do it step by step, gently

The advantage of using Domain Services is to start to factor out domain logic, readabilty and testability will increase and lets you figure out how to enrich aggregate behaviour. In some extent, they act as an anticorruption layer

Conclusions

Most of the mid/large applications are a perfect field to explore with DDD approach. Doing DDD, though, doesn’t just mean blindly applying patterns and concepts. It’s more about the way you gain knowledge and how you use this knowledge to make your architecture evolve. Patterns and concepts are the way we achieve this goal, and Domain Services can play a crucial role here

The Domain Driven Design’s Missing Pattern was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Even if user stories were officially introduced with the eXtreme Programming framework, they quickly became an essential part of the Scrum toolset. The reason is that a user story is a promise for a conversation about a feature that is meant to increase the product value. Thus, They fit with a feature-driven, Agile mindset. Many teams though, decide to not use them to define their backlog, preferring to conceive their tasks/tickets/cards as sticky notes where to describe requirements, activities or solution to problems. The goal of this story is not to criticize about this approach: every team can and should find its “magic formula” to create an healthy work environment. Nonethless, a part of them give up on writing user stories because they feel that they cannot get most out of them. Here I’ll try to collect some examples on how this happens, based on my (never enough) experience. This article is not the “golden thruth”: I will appreciate, in fact, if readers want to share criticism or experiences :)

Antipattern #1: Forgetting the “So that” part

As a user, I want to decide which kind of push notifications I will receive

At a first glace, this user story seems very clear: we know the actor and the interaction with system that the team will build. As a team member, though, a question comes to my mind:

Why we need to build this? Users are finding notifications bothering? So is this a gentle way to dimiss a failing feature? Or are we just planning to add a lot of different notification events and we want to provide means to not get users overwhelmed?

A standard user story is made of three sections

As a (who)

I want (what)

So that (why)

But POs just want to get work done, without explaining unnecessary details. The point here is: is the why an “unnecessary detail”? Working in a self-organizing, cross-functional team means that every member is empowered to make decisions instead to obey orders, and decisions are made within a context. Giving more context has two benefits:

• Helps the team to understand which technical solution fits with business goals
• Shows alternative approaches based on the technical knowledge that the team owns

Last but not the least, during the backlog construction/refinement, the PO should write that section and think carefully on what to write there: this helps her/him to think about “why do we need this feature”?

Antipattern #2: Use Cases, not Scenarios

As a user, I want to go to the page user->settings->notifications and find a checkbox list that shows me which kind of notifications I can subscribe/unsubscribe, so that I won’t be overwhelmed with notifications I’m not interested in.

This story seems very detailed. Too much detailed. To write a story like that, being a PO isn’t enough, you should also be a UX designer. And what about the fact that another story in the backlog could aim to remove the “settings” page? How mobile devs should handle this specifications since mobile apps have a different structure? How to handle empty lists, failures, etc.?

A user story is needed to point at a problem, the user need, while explaining the user interaction is already a piece of solution. So, since the backlog changes with same pace as the technical/business knowledge change, we should be 101% sure that the specification will never change until that story is completed

The reason behind that misconception could be the confusion about what is a use case (the problem) and the scenario (the solution) and the idea that a user story is its own description, while instead it should be trated like a “title” of a container that could add some additional details. Which details? good question. This depends on the team habit, but for sure every team know when a story has all details needed to be worked, and this can become the definition of ready of user stories: a checklist of requirements to complete before the story enters in a sprint. A totally dumb and unreliable example:

• The story is written, with sections “As a” , “I Want”, “so that”
• Mockups provided from UX team
• Architectural Analysis document ready
• Acceptance Criterias added

Huh, Have I told “acceptance criterias”? uhm…

Antipattern #3: Overlook at Acceptance Criterias

As a myAwesomeEcommerce user, I want to sign in in order to access my profile, so that I can make sure anyone else can access my personal data

Well, whoever will read that story, will have clear in mind that here it’s all about security. But what can prevent a lazy developer to write a /login API that hardcodes a 200 status without checking provided email and password? Yes, during the review, the smart PO will ask “wow, what a cool login page, but hey! Show me the error message when I put the wrong password!”

Silence in the room. Shame. Story rejected

Aside from the fact that no decent team could do things like that,the situation could be avoided just pointing out which behaviour the system should show to define the story releasable.

Given a user with email <u.email> and password <u.password>

When login with <u.email> and password “wrong”

Then the UI shows an error

This is one of the few forms in which you could encounter an acceptance criteria or “AC”. ACs are consraints that add more details to user stories, enforce the team to consider corner cases and POs to make sure that the story could really add business value and won’t harm users. You just need to define few of them, and moreover

• You could even transform them in automated end2end tests.
• In a ATDD approach, the tests could be written upfront and run in a test environment after each pull request.
• If the story doesn’t cover all ACs at the end of the sprint, the failing ACs could become a good way to split the story for the future sprint.
• ACs will give the time room to find some “story smells”, and undestand if the story needs to be splitted or definied in a different way.

How to write good acceptance criterias is an art that should be mastered with experience, sometimes made from team mistakes collected over time

Antipattern #4: The “Technical Story”

As a Frontend Engineer, I want an API to list all users with different criterias, so that I can build a user search page

No man. You are doin’ your scrum veeeery wrong. Many teams have specialists that know in advance what to do during the story development, so teams blow up their mind to understand how divide the work to meet their competence avaliability. But if you are using user stories, I bet you want to work in a feature-driven way. This means that, on Friday, You want to please users with a shiny new user search page, not an API that nobody cares about. The clear exception is when your product offers API…well in this case a so-described story would be fine! Thinkng in features means thinking in value, and every team member is committed to build that value. The value, at the end of iteration, is a binary value: There is or it lacks. This will make you feel like working for a goal and you will collaborate with your peers to succed, instead of hiding in a corner, adding Vue components and listening Swedish metal music. And if the feature is not ready or not accepted? no fear: you will have a feature toggle, one per story (or per epic, why not…) to disable in production environment

So the question is still here: how can i allocate the right work on the right skills? Well, stories are not work items. A story could be still divided in subtasks. A story might have a single “owner” that acts as a “story leader”. A guy who collects the right people and communicate with them to get the job done. The job, though, is divided in subtask, each one made by the right specialist. each subtask could involve a PR or a document approval. Once all subtasks are completed, we are closer to the completion of the whole story

Antipattern #5: Chores, Tasks, Whatever but not Features

Refactor UserCartCheckoutService to remove unnecessary dependencies

I’m sure the guy that added this card in the backlog knows what is needed. I won’t argue about that. But can we say that this refactoring will add value to the product?

No…well…yes…but in a very “indirect” way. So indirect that the risk to neglect that value is higher as time goes by. Too bad, not everything can be described as a feature that improves the people life, but many “chores” could be just addressed as part of other stories. This is a way to ensure that the the effort is really necessary: if someday we will need to implement something like a new checkout flow, well we will point out that we prefer to refactor the existing one before to dive deep into the new feature. We will take it into account during story estimation and everyone is happy: team and PO as well, because has more context to understand why devs are wasting time to rewrite code already written (even if most POs do understand it without help)

On the other side, there are many activities that are not feature but also complex enough to not fit in a subtask on or to be just a “related activity” to get done together with a story. They could be “spikes” (an experiment to understand a new technology or approach), or activities that will improve team work (eg. set up a CI server). Some team will describe that ones as “developer stories”

As a developer, I want to test and build a docker image everytime I push commits on git remote, so that I can save time and have more confidence about my code

The question that you should ask, is if it’s really necessary to formulate a need in this form. If you think that you still need to discuss about these activities, or to align everyone on its need and priority, well, spend some effort to write as a story. Otherwise you can use a different type of card, a “task” card (those ones that have a blue icon on Jira) .

So, at the end of time, you will have three kind of items in your backlog, each one with its subtasks: Stories (oh, I forgot epics…this will be one of the next topics ;) ) Tasks and Bugs. I think you agree that you’d want that most of them are stories!

Conclusion

Writing user stories is hard and it’s far from being a “silver bullet” approach. Teams do a lot of mistakes and a lot of improvements to achieve a good backlog quality through stories, but once the goal is reached, the team will benefit from them in many ways, especially from having clear goals and a reason to improve the collaboration, while the business will gain control over the product roadmap and how to please users

Microservice ecosystems, starting from some degree of complexity, are event driven microservices architectures or they are not microservice architectures. This is what I’ve learned from my experience in SCM Group. Distribuited
systems have to be made of independent components, each one with its own “local” state, while the overall state of the system is made up from the union of each individual component state along with the state of the communication channels. But, too bad, components state are too often overlapping. This is an intrinsic property of microservices architecture, where different applications manage different aspects of the same domain, so if those aspects (bounded contexts) were not overlapping it wouldn’t be a real microservice architecture but just a bunch of applications maintained by the same company

Event Driven Architectures are a neat way to manage this concern: there are components that have responsibility to manage a particular piece of the overall state, executing local transactions (as whole or as part of distribuited transactions) and triggering notifications to unknown external components interested in updating the overlapping part of their local states

But now there are the bad news: when you start refactoring microservices to EDA, you will find out that there are many nuances and issues to address, you will see that your system starts to work asynchronously and
this leads to loss of control of the real workflow. You will learn that communication is a serious thing, it’s not about contract anymore, because you cannot set a contract when you don’t know who is at the other side
and a component able to intermediate beetween the parts involved and keep the channel state is very useful, but also something to care about. In practice you will learn five important things:

#1 You have to Deal With Message Brokers

When we think about a generic web platform we think about a three tiers system: the interface, the backend and the database, the one and the only state holder (hopefully). So you already know that you will have to struggle to design the data organization, size the infrastucture in a proper way, manage the content through migrations/seeding, monitor the servers in order to perform operations, run a local instance during your integration tests and take care to clean it up to keep tests independent. Well, If you decide (and I’m pretty sure you do) to put a message broker in your architecture, you have to know that you’ll make the same things for it. A broker has not tables and columns but queues and topics, it can have many replication models so that you will decide which one to use according to requirements (more scalability? Kafka! many standard protocols? RabbitMQ! and so on) and have ops guys able to monitor infrastructure and respond quickly, since workload conditions are more variable with respect to DBs. If you manage a test automation pipeline, don’t forget to add steps to clean the queues and “seed” them (if needed) and design things in order to keep environments totally independent.
Obiviuosly, things are easier when you choose a managed solution, such as Kafka’s Confluent or CloudAMQP for RabbitMQ. Manage things whitin your organization makes sense only when business gets very big

#2 Message Design is the Key of Success

Messages are the blood of your system. Eventually, each microservice will produce them and each microservice will consume them. For which reason? This is the important question to ask. The most of the messages are event notifications, as you expect into an EDA. They should carry relevant status updates but developers don’t know yet what is relevant for which colleague that is gonna using that message. Then there are other folks that leverage message queues to send tasks, since need to trigger async jobs (eg. send an email) and leverage backpressure to keep equilibrium in system congestion conditions. Event messages themselves could be used for Event Sourcing systems, that are a subset of EDAs. Well, you have to address all these issues and make in the most elegant way, because only elegance can become the language that will be spread across your ecosystem. It would be awesome if a developer was able to hardcode a topic name in its microservice whitout even know if that topic exists, but just trusting that the needed event will be carried through it

So work on best practices, naming, formats. Declare which are the categories of services and separate them, use expressive names for topics/routing keys/queues, the granularity of event (should I issue a “UserInfoUpdated” event or separate it in “UserEmailUpdated” and “UserFirstNameUpdated”?) Decide if to carry out event notifications (a pure event with a reference to the state update) or a state transfer

Event Notification

eventNotification = brokerProxy.consume(“domainevents.users”)

log(“received event: user” + eventNotification.eventType)
user = httpClient.get(eventNotification.url)

userRepository.save(new User(user.id , new Cart()));

State Transfer

user = brokerProxy.consume(“domainevents.users”)

userRepository.save(new User(user.id , new Cart()));

There are pros and cons in every decision you will take, but keep in mind what are your goals: less functional coupling, less organizational coupling, keep a good balance beewteen consumed network bandwidth and consumed disk space

#3 Things Could Be Wrong Even When You Get a 200

Event Driven Architectures are also async and eventually consistent systems, everyone knows it but not everyone deals with the system paying attention on this subtle property. I’ll show you two examples to clarify what are the
challenges that you will face

In a e-commerce platform, there are two microservices: the User service, responsibile of CRUD operations on user entities, and the Cart service that retrieves and add products to each single user. Is clear that Cart service needs
to know when a user has been created or deleted to initialize or remove its cart, but the User service has some flaw that prevent it to send correct event messages. Since it’s an async operation, catching a potential message sending exception and make the user creation fail would be a very poor choice but, on the other hand, fault toleration will lead to de-synchronization beetween the user list and its “projection” held by cart service

Solution: log these errors, use an alerting system, fix the bug and be ready to run a sanitization script

Since you are a big devops adept and you believe in the shift-left paradigm, you are eager to catch all these issues with automated tests. You’ll write a test like this

 Given user <testUser>

 When perform GET /users/<testUser>/cart

 Then result.status == 200

You’ ll implement and execute this test but the test fails because API call returned 404. You start inspecting logs and troubleshooting but you cannot understand: everything seems to be fine!!! And it is actually. It turns out that
the “Given” step (that created the user) and the “When” step are executed subsequently but cart service has not yet completed the user creation event handling, so it appears that there is no <testUser>. This is eventually consistency, baby

Solution: use timeouts in test, timeouts are not just a quick fix but the expectation that a non-functional requirement has been met: the system must be conistent after XXX milliseconds

 Given user <testUser>

 And waiting of 500 ms

 When perform GET /users/<testUser>/cart

 Then result.status == 200

If this test fails, we should not increment the timeout but work on the performance, given that the timeout set it’s enough to declare “you cannot take this time under these system conditions to do that stuff”. This also implies
You have a test environment that manage a predictable workload in predictabile conditions

#4 Design Is Like to Play Chess

Chess players are phenomenal to act with recursion: for each possible move they build a tree of possible gameplays , finding which sequence of moves could be more effective and pushing adversaries to follow it. This mindset is very good to apply when you try to recanstruct the data flow in an event driven architecture. In some case, a user interaction can trigger few event, that then trigger other few events, which in turn trigger other events, until you don’t loose the way. Correlation Ids are a powerful tool to help with inspection on some event flow: each message carries a unique random id. This id is saved into thread context of receiving application. The application uses the code to mark all log entries and all outgoing messages, so searching into logs for a certain id lets you see the distributed sequence of actions triggered from the original user interaction throughout the system. But this is helpful in analysis, while in design phase it’s all up our chess player aptidude. Maybe the easier way to win the game is to not creare complicate message flux. An event should be consumed without generating other actionable events, or at maximum an event handler could issue an external task message

#5 Human Communication Is More Important Than Microservices Communication

I’ve aleady demonstrated how EDA impacts on team communication. Organizing things right through convention, building libraries to promote code and configuration reuse, maintain comprehensive documentations: these are the important steps where you go through, but building EDAs needs also to think about the way in which the team communicates with the external, how to collect requirements and how to make sense of business processes to model. Event Driven is an approach that is adopted from early stages of the software lifecycle. For this reason means like Event Storming are born and help teams to design domains and system behaviour through individuation of significant events

Conclusions

Event Driven Architectures introduce some additional complexity in microservices contexts. Anybody likes complexity, but I think there is something that is even worst of a complex thing: a complicated thing. EDAs could transform complications in complexity, so this is good ;)

Nella mia (breve per carità) esperienza ho imparato alcune lezioni ormai molto care, tra queste ce n’è una in particolare:

Non puoi migliorare qualcosa che non puoi misurare

Nel mio percorso di introduzione dei principi e metodi Agile all’interno della mia organizzazione, ho iniziato a chiedermi cosa potessi fare per capire dove migliorare le pratiche del mio team. Ho studiato, provato tanti strumenti, e alla fine mi sono innamorato di uno in particolare. Il Burndown Chart

Il Burndown Chart è il grafico che mette in relazione il tempo di lavorazione richiesto per raggiungere un obiettivo di progetto con il tempo effettivamente trascorso. Si possono creare dei burndown chart in qualsiasi ambito e per qualsiasi tipologia o framework di gestione del progetto, in un’ unica iterazione (laddove è ammesso questo concetto) o sull’intera storia del progetto. Può essere usato per valutare le performance di un team o di un singolo, ma in realtà il burndown chart può dire di più: può raccontare, ad esempio, quali dinamiche intercorrono tra il team di progetto e gli stakeholders, quanto buona è la definizione delle attività e dà indicazioni utili su come migliorare in ogni aspetto.

Per comprendere meglio le potenzialità del burndown chart, di seguito saranno mostrati alcuni casi esemplari da analizzare insieme: vi serviranno quando, dotati di un Tool di Project Management, leggerete i burnodown del vostro team e riconoscerete questi pattern

Note

1. Durante l’analisi parlerò di Scrum, ma è un dettaglio insignificante. Il burndown può essere applicato a qualsiasi tipo di progetto
2. I grafici riportano, sull’asse delle ordinate, una stima pensata essere in ore (o forse meglio ore/uomo) in realtà l’unità di misura è ininfluente sul metodo di analisi; è importante far notare però che il plot è basato su stime, dunque su un modello predittivo di gestione del progetto. Sarebbe in teoria possibile anche effettuare un’ analisi burndown sulle ore effettive lavorate, a consunivo, a patto che sia possibile tracciarle
3. Nel corso dell’articolo si assume che l’aggiornamento di stato delle attività pianificate sia sempre coerente con la situazione reale. In realtà, il burndown chart è uno strumento utilissimo anche per capire se il team si impegna ad aggiornare correttamente lo stato dei task presi in carico

Caso 0: L’Iterazione Perfetta

Se si potesse disegnare su un foglio di carta la gestione perfetta delle attività, ll disegno apparirebbe così. Nel burndown chart viene solitamente tracciata una linea retta che parte nel giorno zero con un’ordinata uguale alla stima cumulata di tutte le attività e finisce a 0 nell’ultimo giorno dell’iterazione. Questa retta ideale è la guida per la valutazione della retta “reale”, che può andare “sopra” o “sotto”, indicando che nella giornata sono state effettuate, rispettivamente, meno o più attività del previsto. Ovviamente far coincidere le due rette nel punto 0 è l’obiettivo di ogni team, ma arrivarci non facendo discostare eccessivamente le due linee indica che

1. Le attività sono correttamente stimate
2. Il leader del progetto non ammette ingerenze dall’esterno
3. Il carico di lavoro è sempre bilanciato
4. Le attività sono ben suddivise e definite
5. il team è in grado di andare avanti col lavoro con pochi blocchi
6. non ci sono evidenti colli di bottiglia tra le varie fasi di lavorazione

Insomma, è il team dei sogni! E nello sprint corrente del mio team, ne stiamo facendo uno così ❤

Caso 1: Scope Creep

Lo scope creep è quel fenomeno nel quale i manager dicono ai clienti/stakeholders “Hey, noi siamo Agile! Noi possiamo fare tutto quello che vuoi, in qualsiasi momento tu lo voglia. Lo faremo in maniera maledettamente veloce e sai perchè? Ma dai, è facile! Perchè siamo Agile”. Ovviamente tutto questo non ha nulla a che vedere con l’Agile, il quale serve proprio a non trasformare progetti “complessi” in progetti “caotici” (vedi Cynefin Framework)

Il burndown di un progetto affetto da scope creep è subito riconoscibile: la linea sale anzichè scendere. E se poi scende, magari risale. Questo accade perchè nuove attività vengono pianificate durante l’ iterazione: priorità cambiate, attività urgentissime da fare il giorno dopo, operazioni preliminari da pianificare improvvisamente prima di eseguire le attività definite, insomma nessuno è più in grado di dire quando consegneremo cosa, perchè ogni giorno si può stravolgere ogni cosa decisa ieri

Azioni Correttive

• Comunica con il business: costringili a riflettere di più prima di definire una priorità, imponi il rispetto della pianificazione, fai comprendere a tutti che se il progetto precipita nel caos, tutti ne pagheranno le conseguenze
• Effettua un triage di bug e segnalazioni: non tutto è urgente, non tutto è importante
• Se affronti progetti e soluzioni innovative o sconosciute, pianifica delle attività esplorative. Proof of concepts, Spikes, Workshop con il team. Capisci cosa devi fare prima di dire quanto ci metterai a farlo
• Considera di passare da un approccio “predittivo” ad uno più “just in time”, ad esempio passa da scrum a kanban

Caso 2: Microwaterfall

Sei lo Scrum Master di uno Scrum Team. Fissi ogni giorno la tua Board e vedi tutte le card in fase “Doing”. Sudi, sei disperato. <<E se lo scopre il PO? Diamine questo si arrabbia>> Parli con tutti i membri del team, sono tutti ingolfati, bloccati. Il QA sta seguendo i test, speriamo che non trovi bug. Diamine dopodomani c’è la review, mi sento un po’ di tosse, quasi quasi mi metto in malattia!

Ma all’improvviso, il miracolo. I dev spostano le attività in testing, poi in done. Tutto viene completato a poche ore dall’iterazione, lo sprint è salvo. Ma perchè è accaduto? forse le stime di effort sono troppo ottimistiche, oppure forse le attività e le loro fasi non sono definite con il giusto livello di indipendenza e di granularità, trasformando lo sprint in un micro progetto waterfall?

Azioni Correttive

• Misura il bias tra le stime delle attività e il tempo occorrente effettivo. Se le stime sono in story points, ricorda il numero di operazioni richieste per completare quegli story points. Usa l’esperienza delle stime più divergenti per migliorare la stima delle storie future. Usa sempre metodi che facilitino la collaborazione, come il planning poker
• Suddividi meglio le tue attività considerando che il caso d’uso consegnabile sia sempre uno e l’output prodotto sia verificabile attraverso una breve lista di semplici criteri di accettazione
• Controlla se esistono colli di bottiglia nel tuo processo, tracciando tutti gli stati di un’attività (QA testing, code review, accettazione ecc. ) e, attraverso strumenti come il Cumulative Flow Diagram Capire se il tempo-ciclo delle attività sia regolarmente distribuito tra gli stati

Caso 3: Chiuso per ferie

Ok, qui due sono le cose: O il tuo è un team di fenomeni, o tu sei stato troppo tenero. C’è anche la terza ipotesi, cioè quella dove tutto il team bluffa al planning poker, ma non voglio essere diffidente. In ogni caso, potevate mettere più attività nella pianificazione, perchè il burnodown dice che dopo 4 giorni il team aveva già fatto il grosso del lavoro e al nono era tutto finito

Azioni Correttive

• Inserisci nuove attività quando la retta “reale” resta definitivamente sotto quella “attuale”. E’ per questo che è importante avere più attività definite in backlog di quelle veramente necessarie
• Durante la pianificazione, assicurati di stimolare a parlare tutti per capire la giusta stima delle attività. Se usi il planning poker, farai si che il voto con la stima più bassa convinca sempre la stima più alta
• Se col tempo hai imparato che il team è in grado di gestire un carico X, pianifica ad ogni sprint un carico X*1.2, ovvero quella che Ferdinando Sodano chiamerebbe “regola della palestra”. Meglio finire lo sprint con qualche attività (non prioritaria) residua, piuttosto che essere costretti a cambiare lo scope pur di lavorare alla fine

Caso 4: Il metodo Faraone

Diciamoci la verità, esistono alcuni manager che, alle porte di questi stupendi anni ’20, credono ancora che il miglior modo di rendere produttive le persone sia quello di caricarle di lavoro, spaventarle e dire loro che deve essere tutto pronto per domani anche se il manager , in gran segreto, ha contrattato con il prodotto un rilascio beta fra un anno. Bene, questo ovviamente è molto, molto più vicino alla barbarie che al Project Management, soprattutto in ambito Agile. Pianificare non vuol dire solo far lavorare le persone, ma creare per loro l’ambiente migliore per esprimersi, un ambiente fatto di cose certe, definite e di tranquillità, un ambiente che premia le migliori performance e non punisce al primo errore

Azioni Correttive

• Aggiorna il CV, invialo a degli ottimi recruiter. Se sei arrivato a questo punto dell’articolo, vali di più della cultura che produce questo burndown

Everyone today loves Single Page Applications just as they love containers. The first ones let us build rich-client web applications that interact with users in very sophisticated ways, keeping high efficiency. Furthermore, many means are built by dev community to achieve a rapid development of complex features with very challenging constraints, eg. app footprint. Two main technologies here should catch your attention:

• SPA Frameworks, like Angular, React, Vue: they provide MV* architectural pattern implementation, the de facto standard to organize components of complex applications. Along with it, a lot of interesting features and tools
• Task automation and bundling tools: modern web applications are written with the good ol’ javascript, the same ecosystem born in 90s to achieve very simple goals. How to deal with complex goals of current production-grade webapps, without changing ecosystem? Simply transforming it at build time! So tools like webpack are real “compilers” that take plain html5/js/css code and produce compressed,obfuscated, translated bundles of resources, ready to be deployed and served

You could want to join all this benefits with the ones provided by containerization technologies, which let developers build artifacts with predictabile environments, easy to configure and deploy with very advanced tools, like Kubernetes. In my daily job, i use Docker containers to build and release backend services. I also like the possibility to configure them with environment variables, as the 4° of the 12 factor app suggests. Environment variables values reflect, in general, the execution environment: if the container is about to be deployed in a staging environment, it should be configured to ‘talk’ with other services in staging environment, this is crucial because anybody should NEVER build a different artifact for each environment: you must build it once and then promote it, from dev to production; this is the only way to verify and validate it properly and in a predictable and traceable fashion. The same discourse for webapps, where i want to configure the right API URLs, and some “feature flag” to switch off experimental stuff without manage different codebases. Though, there is a huge difference between frontends and backend apps: the latter are often deployed with application servers, they implement computing logic and can access to OS resources, so also to env variables. SPA webapps are static files from a server point-of-view,so they are not configurable with env vars. What i can do now?

Use a config file, it’s so simple!!!!

Too bad, is not simple: if i deployed my app in a plain server machine, i would copy the right config file onto filesystem. With Docker i have to mount it on container filesystem and then manage servers configuration, loosing the advantage of using Docker. Furthermore, is not very elegant to use files when you have Kubernetes on your cloud infrastrcture.

Use a config file the same!!!

Ok, i will use it. I mount config/config.json to my container and then i put reference to it in my source code.

• I create my very simple SPA with vue.js framework and vue-cli, the rapid cli interface to create vue apps

npm i -g @vue/cli
vue create spa-docker-demo

and then follow very simple steps

in my app.vue file

My app is very simple, has one button to send data to the backend services and two features that maybe i want to show, or maybe not

Yes, it’s not so gorgeous but, hey! I’m a backend guy!!!

my config.json looks like this

Yes, the flags are strings, don’t ask me why yet :-D

Now I build my app with the script in package.json, which uses an already configured webpack, both are provided from the vue-cli generated boilerplate

npm run build

It generates output files into dist/ directory, ready to be served by some HTTP(s) web server.

Hey but, where is my config.json? how can i copy another configuration or edit the current one ? Simply, webpack “ingests” all source files and “bundle” them in few, tiny, minified files, so I can’t just use a config.json file

Expose the config.json and let your app request HTTP request at start

Come on dudes, let’s be serious….

Environment variables

I want to use environment variables, but I’cant access them from the javascript code!!! I know that Webpack uses NodeJs to build the webapp, I could use some mechanism to get environment (eg process.env.SOMEVAR ) but this is achieved done at build time, it means that when i start the container, the first thing to do would be building the webapp. Very inefficient in terms of footprint and boot time!

The Script.

Before to introduce the solution, let me describe the “dockerization” step. I write my Dockerfile, is very straighforward and well documented how to do it even on the Vue.js official docs

This is a multi-stage build, it means that I’m building and intermediate image which is used to produce the dist/ directory, that I’m going to copy in the final image: this is based on Nginx, super simple, flexible and efficient even as web server. Notice three things:

• The order of Dockerfile commands: It is very important, because Docker works in following way: transforms each command in a “layer”, so that the image results in an amount of stacked layer. It is capable of caching layers, but if it notices that one changes, the following cached layers are invalidated. You have to leverage this knowledge to place the operations with most frequently changing output at the end
• Now I’ve created a config.json.template file and i replace it with the original json file, I’m about to explain why
• I’ve added a script execution, in this script i would boot the server as well as other things

Now, beacuse I’m lazy, I’ll use docker-compose to configure the container build+run process

Someone told me that exists a nice tool, called envsubst that does the job but it can do it in a clumsy way, due to the fact that replaces strings with env variables on a fixed placeholders syntax: it searchs for $VARIABLE or ${VARIABLE} in a file and afterwards replacesit with a env variable value, if any, or an empty string. I could apply envsubst to all javascript bundled files, but th placeholder syntax is also a valid syntax for javascript ES6 language:

So created the config.json.template file

It uses a custom placeholder syntax: replaces the original config file only before to build the image, while in a local dev environment i keep on messing with the original one. Then in the run.sh

Let’s try to explain it

• the first row sets the current directory with the js bundled code directory
• the for loop at row 5 looks for all environment variable names, thanks to the powerful awk program
• In the row 6 i assign to variable VAR the env variable name with ‘printenv $i’ , being careful to remove the apex added when a string is outputted as variable in bash, with the tr command
• In the row 7 i print the variable name and value, just to see what is happening!
• The inner loop applies the next instruction to each file in the working directory
• At row 10, sed does the magic: for each js file, search for the pattern %%VARIABLE_NAME%% and replaces it with the corresponding env variable’s value, it does the job “in place”, in the same file
• At the end, let nginx start!

Now i’m ready to run my container in staging envirnment, hiding feature 1

Spin the container up with ‘docker-compose up’

See what happens

It woooooooorks!!!!!!!

Additionally, i could use other default env variables if needed.

Notice: you’ve got to be careful when you declare a new configuration attribute, it must ALWAYS be paired with the corresponding environment variable. If you forget this rule, you could finde some %%TRASH%% value in production code. To be more safe, think to encapsulate configuration retrieval in a function:

config.get(“backendUrl” , “/”) 

will look for “backendUrl” attribute in config.json or returns the default value “/” when the attribute is not present or not replaced with env variable

Can the mechanism be improved? if so, leave a comment, if it’s useful as it is, leave a clap ;-)