The Most Special Retail Market Date

There is one big and very special date on the e-commerce and retail market and it's called Black Friday. For many, it means "discount prices", "sale!", "50% off", but for developers and IT people it is a challenging adventure.

To begin with, two weeks before, feature deployments are frozen and everything passes through load tests. On the week, machines are scaled up and, at the event's day, all eyes are on metrics and logs.

But there is a catch in this whole story: the feature development can't stop. This means features will begin to stack up soon after the feature deployment freeze and usually it continues a bit after the event, especially when your software is multi-tenant and each client has its own Black Friday agenda.

In the case which gave this article its title, the deploy was delayed until close Christmas and, as it was Christmas, the team preferred to not deploy until the next year.

January came by and, at some point, the delayed deploys would have to start. As four sprints have been passed since Black Friday, the usual process would be to deploy each sprint package atomically, monitoring its results in the production environment. But this, this is not an ideal world.

Product Development Life Is Full Of Surprises

When a product is just in its early days, the first clients are tech early adopters and the development team have to be close to them, as their inputs are valuable.

At some day in January, one of the major platform's clients informed they would launch more shops on the system. Besides the fact that, usually, some preparation is required, the biggest issue was that a specific feature was required. It had been developed at some point, but it was not well tested yet.

Even worse, this feature depended on others, implemented in late November and early December, which means "Goodbye atomic deploys": the team will have to prepare a deploy package with all four sprint features. Dangerous, isn't it? Remember the challenging adventure part?

While finishing and polishing the required feature, the QA team had to work deliberately to test all features again to be sure that everything would be pretty smooth. Even those which were already tested as, since November, some hotfixes were made on the production environment and who knows if they could have affected one of the already tested features.

With the feature finished, the QA and management teams got together to test it. After a period of tests, they confirmed it was working as expected, with no bugs or business rules flaws. Even though this surprise feature made us work insanely, the deploy looked like it would be a walk in the park.

D(eploy)-Day

It was 2 am when the whole deploy started. Low traffic and not that many orders flooding the system made it the perfect time to deploy the application. While deploying it, everything seemed fine, even though minor adjustments and migrations were needed. As soon as the clock hit 3h30 am, the team members were already wishing "Good Night" to each other.

For a package with a lot of features and one quite critical, it was quite smooth… and this is when things started to fall apart. At 4 pm, one of the clients started to complain about some orders not being registered due to HTTP 400 errors and some inventory not being updated for no reason. "WHAT THE F*CK IS GOING ON!" everybody said, in despair.

All developers stopped to investigate the problem and, of course, hidden in the middle of one of the sprint release packages, there were two bugs: one related to how the application dealt with the inventory queue producer and the other related to how addresses were validated. Ten lines of code worth of trouble.

After roughly one hour of stress, everything was calm again, but the results of this deploy changed the entire team.

THAT Post-Mortem And Its Results

Soon after, a post-mortem had been written. Everybody collaborated: developers, managers, and QAs. The information contained in this single document would allow the team to learn, grow and know better how to deal with problems like this in the future.

At the other day, a morning meeting was scheduled to discuss the points written in the post-mortem. Besides the issues from the deploy itself, many other problems arose in the heat of the discussion. All members agreed on the pointed issues and suggested some actions to tackle them.

Deploy age

One could point to the code and say it was a bug, but it was not only that. Deploys that are too many weeks old are a problem, as it has features developed way too long ago, where devs and testers probably don't remember all specifics for these. This can lead to database issues (if a migration was forgotten), schema validation problems, business rules flaws, and so on.

The best way to tackle this is to deploy the feature package not too long after the sprint has been finished. Everything will be fresh for all team members and no git gimmicks will be required for this (rebase hell mostly).

Rushed deploys

Never do rush deploys, especially big ones. The team will get stressed, communication probably will not be the best, documents (such as the release notes) can have errors or miss information. In this deploy, one of the features was not specified on the JIRA Release Board, but it was on the GitHub release. A developer with more time would have noticed that some stuff was missing.

These release notes usually will not only be used by the clients, but by the QAs as well, as it will have which features will be deployed. As these have impacts on everybody, the best thing to do is to have a consensus on what can be fully delivered (developer side) and well tested (QA side). The idea is to avoid features which can have bugs or not well tested because they were made in a rush.

Create Processes

Usually, people love to complain about "processes". Some people even say they love startups because of the lack of them. But they exist for a reason and, in this case, it would have helped if everything followed a well-defined process.

There was already a process, but it was not that specific. After the incident, the team got together and wrote a document describing each step of the product feature development, with a pipe of what should be done on each step of it: feature discussions and development, peer reviews metrics, deploy and release notes for tests environments, test processes and finally, the production deploy.

Is it more bureaucratic? For sure! But until now, there were no complains by the team members and, especially, the client.

More Automated Tests

Each sprint feature package is tested by QAs in a staging environment. The best scenario is to have a lot of different automated test scenarios, which would give 100% guarantee that nothing will break the APIs.

But, this isn't a perfect world. The usual is to have part of it automated and part of it is tested by humans. But, as this aging deploy was urgent and made in a rush, some features passed through without being well tested.

How to solve this problem? Well, one side of the problem is organization: a board tool would help with this but, as aging deploys such as this one have too many tasks, the best thing to do is organize THAT release note. The QA team will be able then to plan how to do its tests.

The other side, which is a bit more complicated, is to create more automated tests. Not only QAs but developers as well. Everybody knows someone who wants to skip the test development, but this can help to avoid countless problems (especially in the future). The other developers should not allow a feature to go on without automated tests for it. The same applies for QAs.

Error Metrics Monitoring and Alerts

Soon after the deploy, the team has checked some error metrics on Graylog and CloudWatch, but these error metrics were just for a part of the system, not its whole. If more metrics had been monitored, probably the problem would have been discovered way before and the impact would have been smaller.

The team should monitor at least some of these metrics manually, through dashboards or even the application log, but the best way to tackle this is through automated alerts. Nowadays, all major IaaS providers have a service for metrics and logs. With CloudWatch (AWS), the team can set alerts for when too many error HTTP responses are given by the applications on a certain determined period. It can send e-mails, SMS and even be integrated with mobile apps such as OpsGenie.

There are other tools besides these, such as NewRelic and Librato, but they are a bit more specific and the team should analyze if they are really required.

End Of Problems?

This will not be the last problem the team will have to struggle with. Stressful moments like these will appear but this is when the team will learn and grow, being able to rise and aim towards high stakes through time.

As the saying goes: never trust user input. People coming from PHP and Java have many validation libraries available. But what about JavaScript? There are some options, but none seems more interesting than Joi.

Joi is maintained by Hapi.js project. Even though hapi.js is a web framework by itself, Joi is independent and can be used in any type of node project. This is great for people using express or restify, for example.

What makes it even more interesting is that, besides user validation, it can be used to define database schemas as well. This post will introduce a quick start guide, with some usage cases and general tweaks.

How to use it?

First things first: it has to be installed on the project. Considering an already initiated node project, the command to install it is: npm install joi. After being installed, a schema is required to use it, such as the file rating-schema.js below:

This defines a model where an username, a rating (from 1 to 5) and an email (which is a required field) are expected. This is a simple example, but there are way more validation options, which can be found on the API reference. To use the defined schema, consider a file called rating-test.js:

More about Joi errors

Real project usage

Validation middleware

In a real project, Joi can be used on a middleware layer. On express, for example, this can be done using:

In this case, if the ratingsValidator returns an error due to validation, express will stop the request before even reaching the controller.

Database schema (MongoDB and DynamoDB)

Some tools can use it to define their database schema (keep in mind that Joi is not an ORM). For MongoDB, mongoose can be used together with joigoose, which easily converts a Joi schema. For example:

Through vogels, the same can be done in projects using DynamoDB. By default, it accepts Joi as its schema definition. The following example is based on vogels documentation:

Exchangeability for front-end validation

Some developers prefer to define the schema once and then use everywhere, including on browsers. Today Joi doesn't support browser usage, but a package called joi-browser does this job, enabling the usage of exactly the same schema on the front-end as well.

Useful tweaks

#1: Disable Joi number() convert

By default, Joi allows users to pass numbers formatted as strings, in cases of number() fields. This is specially bad if the validate() result is only used to check if there was an error, but not for it's value field. To solve this, there are two possible solutions

1. On schema.validate(value), an extra parameter can be passed to disable it, as: schema.validate(value, { convert: false })
2. Put a strict() at the end of the desired number field, as: Joi.number().strict()

Both of them are quite annoying, because either all fields or all function calls require a specific config. One way to solve this is by extending Joi…

#2: Extend Joi defaults

In cases such as the above, or for new field types, or even when the behavior of a default field has to be changed, Joi.extend() can be an option.

When using Joi through this file, all number fields will be on the strict mode — which solves the issue of applying tweak #1 everywhere — and a zipcode() validator is created, based on a pre-defined regex.

More rules can be added and, actually, the Joi API enables much more customisation on Joi.extend.

#3: Create default fields

In some projects, a lot of models will have some default fields, such as an extensionAttributes, where custom vendor data will be put into (as Magento already does) or updatedAt. In these cases, the Joi.object can be extended to include default fields:

#4: Show all validation errors

Usually Joi validation dies on the first error, returning on it's message which field failed. This is an issue when an input have many fields with problems. The user will have to do many requests to discover all problematic fields. To fix this issue, the following option can be added on the function call:

const result = schema.validate(value, { abortEarly: false })

With this config, if an error occurs, the result.error will return all validation failures.

#5: Built-in Joi conditional validations

If a field depends on values from another field, the conditional when() can be used.

In this case, the field a accepts the value x, y, z. But, to accept y as a value, the field b have to be equals 5, otherwise only x and z will be accepted.

Of course, this is a simple and fictitious conditional validation, but it is quite useful in cases where some field requires a specific validator when some other had a specific input.

Are you ready to use it?

I hope this post convinced you to use some validation tool such as Joi, instead of validating everything by hand (or not validating at all). But, if you are using it already: do you know any other cool trick or tweak? Let me know in the comments section below.

References

• Joi repo: https://github.com/hapijs/joi
• Joi API Reference: https://github.com/hapijs/joi/blob/v10.6.0/API.md
• Joigoose repo: https://github.com/yoitsro/joigoose
• Joi-browser repo: https://github.com/jeffbski/joi-browser
• Mongoose website: http://mongoosejs.com
• Vogels repo: https://github.com/ryanfitz/vogels

GitFlow is a branching model created by Vincent Driessen on 2010 (original article). Since it was published, many companies had tested and implemented it, which allows us to have many reviews about how well (or not) it works. After some discussions within our team, we decided to not go with GitFlow, but use a simpler model instead, together with a tightly defined workflow. Some of the discussed reasons of why not go with GitFlow are the same written on this blog post.

The Feature Branch Model

Compared to GitFlow, it is easier to implement and does not require any plugins to be properly used. The step-by-step of this model would be:

1. Create a branch from the master (feature-x), which is where the feature will be developed: git checkout -b feature-x
2. Push the branch to the remote: git push -u origin feature-x. With the branch in the remote repo, a pull request should be opened with it (How to open it in GitHub). A pull request is where all modifications are available to other members and they will be able to review it
3. Fix the reviewed code and wait for approval. If a new release on the master generates a conflict, a best practice would be to rebase it (instead of merging)
4. (optional) If a rebase is needed: checkout to master git checkout master, pull the changes git pull, go back to the feature branch git checkout feature-x, do the rebase git rebase master and then sync the rebased branch git push --force-with-lease . A good tutorial about merging x rebasing is available on this Atlassian article.
5. If there are no conflicts and it was approved ⇒ squash + merge

This Atlassian article have a more detailed view on the feature branch model

Why Squash + Merge instead of just Merge?

The squash and merge is made up of two processes: the squash, which compact all commits in one big commit/patch, and then the merge itself. After squashing + merging, you will have only one commit in the target branch (usually master) containing all your modifications. This enables two things:

1. It is easier to move this feature, as the whole patch/feature will be on one commit hash
2. The target branch will be cleaner, less messy and more readable — without those 67 commits you have made to finish the feature.

There are more information about about why devs prefers squash and merge, instead of only merging, on this article.

Managing release versions with git tags

In the feature branch model, a merge is considered a new version release. To track each release version, tags can be used. These will be used as reference to choose which version should be deployed at the servers.

To manage these tags/release, a good practice is the usage of semantic versioning:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes,

2. MINOR version when you add functionality in a backwards-compatible manner, and

3. PATCH version when you make backwards-compatible bug fixes.

The process to create the releases can be automated using grunt-release or gulp-release-tasks. But, following the steps bellow, it can be easily done by hand:

1. Checkout to the master branch: git checkout master
2. Pull changes from the remote git pull
3. Get the most recent tag using git describe --abbrev=0 (let's say it returns v0.1.0)
4. Create a tag using git tag -a <version>⇒git tag -a v0.2.0
5. Push the modifications and the tag: git push origin v0.2.0 --follow-tags
6. Done!

Deploying

In many PaaS, such as AWS Beanstalk or Heroku, a remote repository is set-up where, when changes are pushed (eg. git push heroku master), a deploy is triggered using the latest commits on master. In these cases, a simple push force using the release tag will deploy the desired version: git push -f <deploy/env-remote> v0.2.0^{}:master. Easy, eh?

NOTE: At Chaordic New Offers Team, a grunt script was developed where we publish which tag should be deployed: grunt deploy:<version>:<env>:all

What happens if a hot-fix is needed?

At some point, an issue will be raised and the production version will need a hot-fix ASAP. A feature branch can't just be opened to develop a fix, as the master will probably be ahead of the production version. In this case, the fix needs to be done directly on the production version:

1. Checkout to the production version tag git checkout v0.10.0
2. Create a new branch from this tag git checkout -b hotfix-v0.10.1-weirdbehavior
3. Create the fix and commit it
4. Create a tag for this new release git tag -a v0.10.1 (notice the SEMVER pattern)
5. Push the branch and tag to remote git push -u origin hotfix-v0.10.1-weirdbehavior --follow-tags
6. Deploy the tag v0.10.1 to the production environment
7. A push request should be opened, as the fix should be applied at the master afterwards

If more patches are needed, this process can be repeated on the same version, incrementing only the patch version.

What about applying it to other environments?

This patch probably should be applied to other environments as well, which can be done through git cherry-pick <commit-hash> . It basically applies the chosen commit to the actual HEAD.

1. Checkout to the environment version tag git checkout v0.13.0
2. Create a new branch for the patch git checkout -b hotfix-v0.13.1
3. Do a git cherry-pick v0.10.1 or a git cherry-pick <commit-hash> to apply the desired commit
4. git tag -a v0.13.1 and git push origin v0.13.1 (push just the tag)
5. Deploy it

What if I want to get a modification from master and sent to one of the environments?

It is very similar to the above one: a git cherry-pick should be done using a commit hash from the master as, after squash + merge a push request, a new commit is generated with all changes (big patch of commits condensed in one).

Just keep in mind…

The gap between the environments versions should be as short as possible. Otherwise, some issues may appear:

• If the production is on v0.1.10, the latest release is v0.10, but the version v0.3 will be deployed: the team members will have to check if some of the production patches are still required and then apply them, one by one.
• If some feature was only finished on v0.10.0, and it is required for the roll-out, but the v0.7.0 is still not well tested: the release should be hold until the v0.7.0 has been tested

Usually, these version gaps occur when the producing capacity is higher than the testing capacity (developers x testers ratio).

Conclusion

The model is still being tested but, until now, it has been working well. The only faced drawbacks were the ones pointed on the session above.

You are developing some feature together with your team. Most likely you have done the usual dance: checkout a branch from the master or development branch, done your commits, pushed this feature branch to the remote and opened a push request for peer review.

In ideal cases, you will be able to just squash and merge after some reviews and fixes. But, the world is not ideal (as you might know), and maybe one of your mates modified the same files and already merged it. So, surprise surprise, you got a merge conflict to resolve!

This usually means you will need to do a git rebase before merging. After doing it, this new rebased branch need to be pushed back to the remote. And here is one of the steps that people usually do wrong:

git push —-force

You might be thinking: what is the problem with this?

Well, push force ignore everything on the remote and override it with local changes. This is "kinda" what you want, but what if you did some mistake in the rebase but didn't notice? Or someone pushed more modifications while you were rebasing? Yes, you thought right: kaputt, gone. One safer way to do this is using:

git push --force-with-lease

This simple command will not push force your modifications until it checked if everything that is in the remote is also in your rebased version. It is the same as a push force, but safer.

If you want to know more about it, checkout this article. But the basic idea is: start to use force-with-lease instead of force.