In this post, I’ll share some things I learned about it through the years (I fail to apply some of them though). Some of them come from experience, others from my wandering past in creative writing, and others from reading things about this very topic. Please fasten your seatbelts as we are taking off 🛫.

Get to the point fast 🏃

We live in the information era. Millions of colours, texts, shapes and data are thrown to our eyes every second. Our little recently evolved brains try to keep up by taking shortcuts and plainly ignoring a great part of that information overload.

Do you want to be heard and understood? Get to your point fast, like sonic speed fast. This is a thing you can see applied to any published piece of text. The first chapter of a book for example (even the first sentence!) has to attract the attention of the reader or it will lose her forever.

Apply this to any communication you make. Be speed, communicate what is important lightning-fast ⚡.

Be specific and synthesize 🗒

In addition to being quick in getting to your point you should be concise, you should synthesize what you convey. And I will remark the size part of syntheSIZE. Use the number of words you need to express your ideas and no more.

I always return to the great words of Stephen King and his 10% rule. If that man knows how to do something reaaally good is writing. His book On Writing is an amazing piece of gold if you want to improve your writing (and actually start doing it).

His rule of the 10% consists of letting your words breathe for a while after finishing your writing. Then you come back with a machete (I’m paraphrasing here) and cut 10% of them 🔪. For a text of 500 words, you should be left with 450 and so on.

This is not only applied to novels like the ones he writes, it can be used for any text or spoken discourse. I apply this to much anything I write. I’ll apply it to this post when finished so you’ll never see all the words sublimated in the process, but their sacrifice will have improved the end product you are currently reading.

Know your audience 👥

Who you want to communicate with is really important. As an effectual communicator, your main job is to communicate in a way that people understand.

I’m going to obviate the topical things about using the right language with the right people and all that BS. We have common sense, we already know that. Obviously, I will not speak with my academic tutor the same way as with my friends.

With knowing your audience I focus more on what knowledge this person has, how deep their understanding of the field we are talking about is, and the opportunity cost they have in exchange for participating in the communication.

Elevate 🏋️‍♀️

You should elevate low-level stuff to a higher level if you are communicating with someone not versed in the topic (like upper management in the case of technical explanations). With this we have to have clear that they are not dumb, it is just not their field and they need a little push in order to understand the details.

I experienced this myself recently at Mercadona Tech. We have this high economic knowledge high effective business guy called Ignacio. He deeply knows his economics stuff and I know nothing about profitability and indirect costs. In a short hour, he was able to explain all the minutiae and details of the full costs models we have for our different preparation models. Like child's play. I left the room understanding each concept and being able to relate them to other contexts and projects. Amazing communication skills.

I myself often apply this elevation principle too, like when communicating high-impact incidents for example. I usually prepare two explanations of what happened. One more high level to publish in public channels where not technical people will go to check what happened and the business impact (usually short to maintain their attention span). And another more deeply technical one to relay the tricky details to other engineering members interested in why it happened.

It is like going to a doctor, understanding what happens to your body is really important for you. But you don’t know what alopecia is, so the doctor should exemplify and elevate the topic to terms your balding head can understand.

That meeting could have been an email ✉️

Communication emerges in many ways. Written and spoken, in person or remotely, on email, via slack, synchronously and not…

You can choose how and in what format to communicate and you should choose wisely to avoid time losses and misunderstandings.

Maybe that message doesn’t convey the mood in what sounds inside your head, maybe it can be misunderstood. Maybe that half-hour meeting with commute included could have been a two paragraphs email and have saved us all an unjustified long amount of time. Maybe if you chosed the correct way of communicating you wouldn’t have to redo half of your work due to a misunderstanding.

Choose wisely when communicating, your message is as important as the way your share it.

Understand the problem 🧠

Another important point in communicating is understanding what you are talking about. Albert Einstein said:

If you can’t explain it simply, you don’t understand it well enough.

Well, maybe he did not, but it’s a nice quote to affirm my narrative. As I mentioned in previous posts, the first and most crucial step in facing a problem (and thus communicating it) is understanding it.

LISTEN 👂

Communication works both ways, from you to your listener and back (when you are the listener). We usually hear but we rarely listen. Most of the time people tend to think about the next thing they are going to say and don’t actively listen to the other party. Any information transmitted in communication can change the course of it, so don’t miss anything and pay respect to your peers paying them the attention they deserve.

Over-communicate 🗣️

You have to over-communicate just to communicate (unknown reference 🙁). We usually overestimate how much listeners understand. You always have to make sure that your message is being received and that is being understood.

Don’t fear to over-communicate in order to be understood. Repeat if necessary to get your message acknowledged.

Ask 🤔

Ask them! Ask your peers when communicating. It will help you solve some of the problems the lack of understanding provokes. And more important, it will help engage people in the dialogue.

Ask them to involve them, ask them to know their opinion, ask them to check if everything is being understood, ask them to expand your point of view, and ask them to expand theirs. Believe me in this, just ask them.

And DO communicate 🙏

And finally and more important of all. Do communicate.

Communication does not start without two involved parties, and you are one of them. Engage in active communication when necessary (that happens to be almost on any occasion). Don’t let a lack of communication worsen your relationship with your team, don’t let that unspoken words guide the fate of your future ✨.

Final words 🎵

Hey! I hope some of these tips help you and your peers communicate better. Communication is a skill that has to be learned and practice is your best ally in this. Start communicating today and ask for feedback when necessary to improve it.

I’m going off now, if you liked this content you can find more of it here.

Also, you can follow me on LinkedIn or Twitter for more thoughts and derailed tales 😄

💐 Here lie the 10% of the original words of this post taken out to improve its conciseness.

It should match the business needs 💡

The first thing to take into account is a big one. The technology choice you are making should match what the business expects. You are making a technology decision to solve some problem, to give a solution to a concrete business gap. When choosing, you have to be sure you understand what we are making the decision for, what are the requirements, and what we expect from it.

An example I always give to illustrate this topic is the choice of database technology for data analytics. What should we use? A relational database? A NoSQL one? A graph database? Well, it depends on what type of data analysis we want to get from it! You should know and understand what the purpose is and what kind of conclusions we want to extract from that data.

If your analysis focus on aggregations then a columnar database will do a better job. If you need to access different fields of each object in big processing batches, a relational database should be the choice. If you want to make conclusions from the relationships between objects, then a graph database sounds like the right decision. The business needs should guide the technology choice, not the other way around.

Social learning, use what your friends use 👫

Another important point when taking tech stack decisions is who you will be able to learn from. This is usually proxied through the technology’s community but can be expressed in terms of its support around you.

A good rule of thumb that I like to give as advice is that you should use what your friends use. There is no better support. If you use what the people around you are using then any doubt, problem and block can easily be solved just by turning around and asking your friend.

Social learning is a powerful thing, we humans are collaborative by nature and it makes wonders in productivity if you know how to take advantage of it.

Hiring ⛑

Hiring is important, do not let anyone trick you to think the opposite. Any project, even any of your side-fun-pet projects can turn into something bigger in the future. And in that future, you will face a really big problem: finding people to help you grow it to something else.

That said, if you choose a technology or language or framework that nobody uses (or nobody uses at least yet) you must be very sure that the advantages you are getting for that choice are bigger than the headaches future hire searches can give you. Bet for the widespread, believe me, you will make the incredibly difficult work of recruiters much easier.

Inertia is a powerful force 🐉

This topic is closely related to the previous one. In this industry, inertia is a powerful force. Established ancient forces that reign over the kingdom and rule with an iron fist. Technologies that are so rooted in enterprises that they are really difficult to avoid using in one project or another.

For these cases my advice is simple: accept your fate and kneel before them… for now. But look closely at the trends, the hidden heroes, the ones that do not fear to take down the giants. Secretly give them your support and force. In the future, they will be the ones ruling all under the sun.

Vendors and platform 🤖

This is another headache. Yes, if you want to develop native iOS apps you have only one choice and a shot in the foot. I mean, you can choose between Swift and Objective-C and that’s it. For anything else, you will have to go hybrid.

The platform you pretend to develop for is not key when choosing technology but it is usually a set of restrictions that narrow the number of your choices.

Updates and maintanibility 🌀

A software being updated is happy software. Lack of evolution support for projects, languages and libraries is their doom. They can be heavily used in the industry, and they may be really useful, but if they are not being updated to the latest systems, language versions and needs, they are destined to fall.

The worst thing you can see in a code repository of an open-source project is something along the lines of last updated 5 years ago or not maintained anymore.

So be careful when choosing this kind of technology, they can break anytime and no one is going to fix them.

Conclusions 📑

These are some of the things I take into account when choosing technologies and when I try to prevent future headaches. I hope this inspires some of you and help you when facing these hard decisions.

There is one more thing that I didn't mention and it is that I like to break some of those principles for fun 😄. Trying new technologies is really amazing and sometimes you find real gold, so although it may contradict some of the reasons given in this article, my last advice is to be flexible and experience new things whenever you have the time. This is really important.

And finally, to conclude, here comes the moral of this fable (a TL;DR):

Choose what your business needs and try first what your friends are happily using. Take into account the difficult work of your recruiters. Follow established trends but often bet for the new. Also, be clear about the restrictions of your platform and seek the maintained and the updated. And finally, but not least important, have fun!

Final words 🎵

Hey! I hope that some of this tips help you and your peers choose better technologies.

I wish you a great day! If you liked this content you can find more of it here.

Also you can follow me on LinkedIn or Twitter for more thoughts and derailed tales 😄

The problem

In everyday life and working in a product or in software development you face uncountable problems that have to be addressed. You have to understand them, prioritize them and decide on solutions.

In this article, I describe some of my thoughts about it (all of them highly influenced by reading a lot about problem-solving and watching great talks like How to Become a Great Software Architect by Eberhard Wolff).

What is the problem?

The first and most crucial step in facing a problem is understanding it.

Quoting the philosopher John Dewey (and avoiding the typical problem-solving quote):

A problem well put is half solved.

When you want to solve an issue, the first thing to do is to understand it. In order to propose good solutions, you must know what we are discussing about. What is the problem we are trying to solve.

Quoting this time Malcolm Forbes:

It’s so much easier to suggest solutions when you don’t know too much about the problem.

And those solutions will probably be flawed without the right context.

The problem is not a solution

Talking about solutions: an important fact is that a solution is not a problem. Avoid per example sentences like the problem is the lack of… that is a solution, not the problem.

You must ask the correct questions to identify the needs and constraints of the problem in order to find proper solutions. So be careful not to fall into this kind of disguised solutions that can influence the outcome decision.

Specify, do not generalize

It’s important to be specific in the problem definition.

It’s not the same to say that our systems are crashing as our systems are going down due to an increase in memory usage.

Specifying data or details that support the definition of your problem will help you understand it better and make better decisions based on more available context.

Express the problem in the stakeholders’ terms

Most of the time the problem you are facing is not affecting only you but some of your stakeholders too.

It’s important to address the problems from the point of view of all the implicated parties and how it affects them.

An increase in resources consumption that is affecting some processes in engineering can also make load times 4 times slower for the customers or crash some dashboards in customer support.

Knowing all the affected stakeholders and what problems they are facing will help you decide and prioritize the best solution.

It is really a problem?

Now, having all the context and available information about your problem, you must decide if it’s really a problem we have to solve right now. It is worth solving? Should we prioritize it?

Not all problems are critical to your business, some of them can be assumed and handled later. It’s your decision what and when to tackle each of them and work on what is really important at the moment.

The decisions

Once the problem, the implications and the affected stakeholders are understood, it is the time to make decisions about it. At this point you already have half of the battle won. Now let’s make the decisions that will solve the problem.

Who has to make them?

In a modern structure, probably your team is self-organized and composed of multi-disciplinary members.

One person may be an expert in a field or part of the architecture. This person may overlap knowledge with other team members, but probably no one has context and knowledge of everything. That would be very unrealistic.

So, who must take the correct decisions to face the problem? The answer is the team. Problem-solving is a collaborative game, everybody will have different points of view and expertise. The points where they disagree is where the real implications of your solutions will arise.

And be aware of unanimous and not problematic decisions, if it looks too good maybe it’s lacking opinion diversity and a challenge may be needed. Someone should defend other opinions although they are not hers to strengthen the final decision.

Who owns the decision?

Although the decision making is a collaborative effort someone has to own the decision taken. Shared ownership often creates confusion and slows its implementation.

When everyone is responsible for a decision, no one is really responsible for it. You want the person with the most context to own the decision and the outcome of implementing it. We must then differentiate joint accountability from decision ownership.

When to make the decision?

We face uncountable problems every day, and, as stated above, not all of them have the same priority or impact. Prioritize what is important, what is impacting your business the most and leave lesser problems for later.

Don’t be afraid of postponing relevant decisions until they really become a problem.

What if we do nothing at all?

So maybe your problem can be solved in a trivial way, at least for now, and then you have more time to work on problems that would really help you grow and progress.

Do we need a system to reschedule orders right now? Can it be done manually for now so we can focus on other important problems?

You can patch that kind of problems for now, until they grow enough to be relevant. In the meanwhile, you can focus on other things.

What if it’s too late?

If you did prioritize your problems more or less right then the too late shouldn’t be too critical. Maybe some revenue is lost, a couple of clients may be upset, but you did use the opportunity to work in something more critical that could have had worst implications or helped you advance.

You can’t favour everyone and every part of your business, but if you prioritize well, everybody will be happy in the long term.

What if it’s too soon?

Often decisions are made too early.

If you rush making decisions you can end up working on unimportant stuff. You can also fall into the trap of making choices based on too little information. In that scenario, you could end up doing twice the job due to some misconception about what the problem really is.

But don’t confuse having too little information with having it all. You usually have to make decisions with unclear information because retrieving all of it could be too expensive or take too long.

Do not precipitate but have the courage to make decisions in unclear terms if you need it to advance.

Review your decisions

The available information is key in the decision progress. When working on something, new information will naturally arise. The basis of your choices could have changed.

You must always be reviewing your decisions taking into account the new information available and reevaluate them accordingly. Most of the time what appears to have been a bad decision is only a decision not reviewed.

Conclusions

So there you have it, understand your problem. It is very important to do a good job understanding what is really happening to be able to solve it.

Don’t fall into solutions disguised as problems. Ask yourself first what is really happening instead of how you would solve it.

Understand the involved parties and ask them how they are being affected. This will give you more context that will help you make better decisions.

Problem-solving is a collaborative game. Work as a team to find the best solution and its possible implications. But after the decision is made, someone has to own it, shared ownership is often not a good idea.

Don’t be afraid of postponing the solution and focus on the most impactful problems first.

And don’t be afraid of making decisions with unclear information, you will never be able to have it all under control.

And finally, always review your decisions, new information can change how the game is played and your decisions can be outdated.

I hope you found this article interesting. Problem-solving is a really important and powerful skill to work on. If you are interested in other topics like software development, product or data science I invite you to check other of my articles like Business logic in a Django project or my series about recommendation systems.

Have you ever wondered where in the hello world is your business logic meant to reside in a Django project? In the models, you can argue, maybe in the managers, who really knows? Let’s talk a little about it.

Active Record

Quoting Django’s design philosophy models should:

Include all relevant domain logic

Models should encapsulate every aspect of an “object,” following Martin Fowler’s Active Record design pattern

Mmmm, so Active Record uh, a popular design implementation used in a great variety of ORMs. Also claimed an antipattern by a relevant part of the community.

Quoting Robert Martin in Clean Code:

Active Records are special forms of DTOs [Data Transfer Objects]. They are data structures with public variables; but they typically have navigational methods like save and find. Typically these Active Records are direct translations from database tables, or other data sources.

Unfortunately we often find that developers try to treat these data structures as though they were objects by putting business rule methods in them. This is awkward because it creates a hybrid data structure and an object.

So yeah, we see some contradictions here. As Martin Fowler described them: Active Records are objects that encapsulate data access and add domain logic to that data. The problem is that this kind of structure flushes the sink any sign of SRP you could expect in it. And that could be a problem.

Fat models

So how this recommended structure is really implemented in practice? Following that model, we start building what the community calls fat models (and thin views). An extended practice recommended in a lot of learning material like django-best-practices or ultimatedjango.

Using this pattern, we implement all the business logic in the models and managers making them fat. That’s not really a good idea because that makes the testing really difficult and very coupled to the ORM (creating you a mocking mayhem and the need of a big number of fixtures and data to create different test scenarios). Following this practice makes your business logic nearly impossible to test without using a database.

Growing your models to infinity makes maintaining them really difficult too. With every new use case, your model will grow one or several new methods and it can quickly and dangerously become a god object.

So what can we do to avoid this kind of problems?

Quoting again Robert Martin’s Clean Code:

The solution, of course, is to treat the Active Record as a data structure and to create separate objects that contain the business rules and that hide their internal data (which are probably just instance of the Active Record).

The business layer

The answer is the almighty business layer where our business rules live. Sometimes divided into subsets like domain logic and application logic, it is the layer where we shall implement our business flows, actions and logic.

David Winterbottom in his great and funny article Why your Django models are fat teaches us some good lessons like:

As a rule-of-thumb, your application logic should live in modules that aren’t Django-specific modules (eg not in views.py, models.py or forms.py). If I had my way, Django would create an empty business_logic.py in each new app to encourage this.

So yeah, a business logic module sounds like a great idea. A place where you implement your commands, use cases, actions or flows (they have so many names) and where they remain loosely coupled to your data sources.

Just be careful of not taking too much away from your models or you will be left with anemic domain models. Only business logic is meant to leave, the domain logic like validations, calculations, etc are already at home.

The data layer

And how we can maintain this loose coupling between business logic and the dangerous and awesome Django ORM? Because it is soo easy and wonderful to use, it’s a pity that you can fall into it so easily.

The answer is treating it like a data repository. A really not used pattern to a big part of the Django developers I used to know (highly influenced by using the Django ORM everywhere).

In reality, this solution is the recommended way of doing things, although it is not advertised as a data repository at all. In every Django tutorial, you are suggested to write your queries inside custom managers to avoid repetition and writing the same query every time. But you are gaining a lot more than that, implementing the queries there decouples the ORM implementation from its use in the application, a win-win situation.

Following this principle, you can then use your Model.objects as it was your data repository in the business logic, reducing the coupling to the ORM and allowing you to do integration tests for the database interactions only. In the next example, we’ll see how to use it.

Real example

Now let’s see an example to understand the situation better.

The example is very simple: we have an Address model that has an is_default field that indicates if that address is the default one of its user.

And then we have an action that makes an address instance the default one, setting its is_default to True and setting it to False to the other user’s addresses. After setting the default address, we publish the changes to anyone interested.

Here first we have the example done using fat models (high coupling and low cohesion):

Some important things to notice are:

• The model indeed contains the business action set_default.
• It has dependencies with other models (in this case itself).
• There are side-effects in the action inside the model, breaking the SRP.
• To achieve unit tests for our business logic we need to mock up all the dependencies.

In the last point, we have something important to discuss. You can follow 2 approaches: do an integration test where you validate the business flow with all its dependencies or do a unit test mocking all the external calls.

The problem with the integration test is that you have to test the behaviour using a real database as all the dependencies will end using the ORM, slowing down your tests and not validating the logic flow independently of the dependencies. The problem with the unit test is that you lose the integration validation of database behaviours and you have to mock everything.

So let’s look to the same example but applying a business logic layer and abstracting the database interactions to a manager (low coupling and high cohesion):

As you can see we have easier tests and more maintainable code. In this case, we implement unit tests for our business logic that are fast and validate its flow. And then integration tests for the manager that validate the data operations and its dependencies with the database.

The key in this structure is that our business logic is not aware of data integrity or how it is stored, that is the job of the managers. This way we can unit test every flow and leave the integrity checks for the integration tests of the data layer.

Final words

Django is a great framework, but after all, it is only a framework. It’s your job to use it correctly and structure your application to your needs.

Fat models in Django can make sense for properties (like full_address) and object validations, but including business logic and side effects into your Active Record objects is not really a good idea, some kind of service (or business) layer is my recommended way to go.

Be careful not to abuse the Django ORM and rely on it for everything. It is one of the best ORMs out there but as a consequence, it makes it too easy to end up with a giant, coupled application if you don’t pay attention.

And to conclude just say that these are my always evolving ideas about how to structure logic into Django projects. There are a lot of ways of doing things and every situation is different so there is no correct answer for it.

I hope you found this interesting! If you want to read about other topics like product or data science you can check my e-commerce recommendations series. I hope to see you soon in future articles!

Recommendations IV — A tale about data roles

A car without wheels is a useless paperweight, a sailboat without the sail is only a boat and a boring one to be precise ⛵️. In data-projects it happens the same. You can have the best data science team and the best ideas, but without the data available to play with and the ability to take the results to your customers, you can’t do anything at all.

Data roles 📚

Here comes the superheroes of our story, the roles of machine learning engineer and data engineer. This positions are relatively new roles that appeared due to the latest bigdata hungry and the growing affordability of this kind of developments outside the big techs.

Firstly, the data engineer 🔧. When starting with a project involving data, the most important thing to do first is to have the data available. Provide the team with the data they need may sound trivial, but it’s a real challenge. A lot of pipelining and infrastructure work is involved and sometimes is difficult to connect data from different sources and make it available to work with it efficiently.

On the other hand, the data scientist 👩‍🔬. The professional that given a problem explores this available data and research and develops a solution. The result of its work can be, per example, a model or a statistical analysis.

Finally, the machine learning engineer 👷‍♀. The person that takes this model and then cleverly connects it to a production environment. Many of the times, the real value of the solution comes with this implementation.

Our story 📕

This role differentiation brings us to our problem. In previous articles of this series, we discussed some of our efforts in bringing our customers personalized content to fasten their purchases. We explained the fascinating parts and omitted the boring ones. Not all was a bed of roses.

In our case, we lacked both data engineers and machine learning engineers 😢. But don’t get me wrong. The problem was not the lack of a person dedicated to these developments, but the lack of developers with the experience to take those roles and deliver their results reliably.

With the suggestion of alternatives in the ghost products issue, there was no much problem. We only handle a catalogue of around 10K products, so the computation and memory usage was far from being an issue.

With the calculation of the related products, we started to experience some trouble. At first, we were processing this inside our monolith, using database aggregations that made the process not so complicated. When we finally decided to move all the recommender stuff to its service we hit the wall of “Oh! we don’t have the data here!” and had to work to pipeline it to it with some magic (that took more time than developing the solution itself).

Then we went into Helm’s Deep trying to create recommendations for hundreds of thousands of clients, and all went how it was expected to go, not very well.

It is not easy to move data around. A few hundreds of megabytes are comfortable to work with, but when you are handling tens of gigabytes, things start to fail. Timeouts, Out of memory errors (OOM Kills), unending Extract-Transform-Load (ETL) jobs… That kind of things began to be our day-to-day, and frustration and tedium started to pile up 😫.

Of course, we got all the pipelining and crafting done and working, but believe me, the journey was a real nightmare.

The moral of the story 🕊

Experience is important. Specialization is becoming more and more noticeable with the emergence of new paradigms and highly technical fields. Invest in those specialized roles and developments will become more robust and reliable.

That doesn’t mean that a professional team in the field can’t take those roles and perform well and bring value to your customers. But they are going to hit with problems that experience and knowledge would have handled them quicker.

Here ends this recommender series. Thanks to have joined us in such an amazing adventure. All our recommendation experiences had been a rollercoaster of emotions and fun and we are glad to have had you on board.

As a final note, if you like challenges and figuring out things yourself, you could be a great match for us. And this is wonderful because we are hiring! We are looking for great professionals for different roles, and you may fit in, who knows! If you want a challenge and want to try something different, please don’t hesitate to contact us (And yes, we still need that data engineer!).

Previous articles 🗒

Recommendations I — A scary tale about vanishing ghosts

Recommendations II — A twisted tale about twin products

Recommendations III — A flavoured tale about similar tastes

Recommendations IV — A tale about data roles was originally published in Mercadona Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recommendations III — A flavoured tale about similar tastes

When we talk about recommendations we usually refer to the magical suggestions that online platforms offer us based on our tastes and historical purchases.

There is a lot of black magic involved in modern recommendation systems. Platforms like Spotify or Netflix build their entire business model around them and have full teams dedicated to improving them. Some retail businesses even have reported increments up to 30% of their revenue by implementing custom recommendations to their customers.

As mentioned in previous articles in this series, in MercadonaTech we have the vision of making the life of our clients simpler. Helping them do their regular groceries faster and easier. Recommending them products they may be interested in is one of the ways we help them.

In this article, we will learn how we recommend new products to customers from their previous purchases and learn from other clients decisions.

The study 📚

All started with an idea, learn from our customers. As we mentioned in the last article, groceries e-commerces are not the usual online platform where you buy occasionally. They are made to make recurrent purchases of food and care products. This means that all the orders of one customer should be quite similar.

So we knew we had customers with similar preferences. And that we could improve the purchases of our clients recommending them products that other customers with similar product profiles bought but them not.

We started our journey checking the state of the art. In these types of things, a lot of people before you have already fought against the problem. I recently read a blog post from Bill Gates that starts with a quote that I totally agree with and that represents this point very well:

“I tackle every big new problem by asking two questions: Who has dealt with this problem well? And what can we learn from them?”

We learned a lot in a short time. We read some great references like this (towardsdatascience.com blog post), this (google own curse) and this (Mining of Massive Datasets book) among others.

We got a rough idea about the different existing solutions and how they work. In product recommendations there are basically two paths to follow: the called “traditional machine learning” (I quote it because I don’t quite like the differentiation 🙁) and some new deep learning approaches.

Looking for real use cases (aside from some big players and big techs) the overall implementations made use of the first path. So we decided to start small and get some recommendations to hit production quicky in a lean way!

Content-related recommendations 🍱

There are two long-standing proven “traditional machine learning” approaches to recommendation systems: content-related recommendations and collaborative filtering.

To suggest products to our customers, the first thing you can think of is to recommend them products similar to the ones they have already purchased.

To do that, the logical thing to do is to take into account the characteristics of the products they have bought (the description, category, size…) and recommend similar items based on this information.

This technic is called content-related recommendations. And it’s a great way of suggesting new products to customers.

It has a couple of downsides too tho. Like, recommend you toilet bowls for two weeks after replacing the one that you broke accidentally 🚽 or create content bubbles where the customers only see the same set of products forever until they buy something new.

Content-related systems are a relatively easy way to create recommendations. Sadly in our case 😔, due to logistical issues and packaging fluctuations, we do not have available the descriptions or content of our products. Although it’s an approach that we have to explore soon, we have not implemented it yet.

Collaborative filtering 👫

The case of collaborative filtering is more interesting. This approach allows you to recommend products without having the context that surrounds the product and without needing its description or characteristics.

To calculate the recommendations, it takes into account purchases from users similar to you (we explain how this is done later). Recommendations are then based on products that the other users have bought but you have not.

To achieve this we use a traditional approach that makes use of matrix factorization. It consists in converting a matrix into a smaller matrices product and recalculating it again to fill the voids. If this sounded too weird 🤔, now we will show you an example of how this works and how it manages to calculate recommendations.

Calculating recommendations 🧮

In the example, we see the purchase history of Alice, Bob and Carol. Alice has bought milk, Bob has bought milk and eggs and Carol has bought bread and toasts. The objective here would be to recommend eggs to Alice since Bob has also bought milk like her and has also bought eggs. As a result, they could be of interest to her as well. In the case of Carol, as her tastes do not coincide with Alice’s, it should not recommend her bread or toasts.

The goal of the process is then filling those zeros with some sortable value to recommend Alice products that she has never bought and may be interested in.

As mentioned above, matrix factorization is based on taking this purchase matrix and transforming it into a product of two smaller matrices called latent vectors. Once we have these two matrices, we multiply them and obtain an approximate matrix to the original one but with variations that will allow us to rank the products that had 0 as value.

We are not going to enter in details about how matrix factorization works as it exceeds the scope of this article, but you can visit this link where it is explained in detail.

In the resulting matrix above we can see that the bread and toasts score is very tiny in the Alice vector due to the low correlation between Alice and Carol. Eggs, however, have a significant score and would be recommended.

In the heatmaps below, we can see the original matrix and the recomposed one with the potential recommendations.

As mentioned earlier, eggs have a significant score versus bread and toasts that are close to zero.

By sorting the products by value and applying a threshold we get the final product recommendations.

Conclusions 📓

Customer oriented recommendations are a great and poweful tool in e-commerces. The client discovers new products that he might interested in taking in account the decisions of others with similar tastes to them.

Since the release of the recommendations in our home page, our customers have been adding around 4 products in average (products they never had bought before!). And the number of items added from this page have experienced a 15% increase 🚀.

In conclusion, our clients are finding new products they didn’t know about and are taking them home. Easier and faster purchases is all about!

More work is needed tho, we will continue to improve our recommendation systems to get our customers what they want, wherever and whenever they want (great goals).

This recommendations series ends here 😞 (more or less). The next and last article will bumble around the infrastructure hardships that building our recommendation systems implied. And the important figure of machine learning engineers in the new world of big data. Have fun and a good day 😄!

Previous articles 🗒

Recommendations I — A scary tale about vanishing ghosts

Recommendations II — A twisted tale about twin products

Next articles 🗒

Recommendations IV — A tale about data roles

Recommendations III — A flavored tale about similar tastes was originally published in Mercadona Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recommendations II — A twisted tale about twin products

Doing groceries is a really hard task! Usually our typical shopping adventure takes around 45 minutes and getting there has its cost too, reaching the supermarkets takes around 15 minutes more in average.

So we spend about an hour (more or less) around 1.5 times a week doing groceries (so about 53 hours a year 🙀). This can be a problem to some of our customers. And that’s one of the reasons why we decided to handle this e-commerce thingy from bottom-up and offer a full online Mercadona experience that can provide you what you need wherever and whenever you need it.

That’s why we have built our platform around this idea, quoting the previous part of this series:

In MercadonaTech we have the vision to build a platform that allows the customer to do their groceries as fast as possible and save their time to spend on something more interesting.

In this tale we will talk about another one of our systems that is in charge of recommending products that are related between them.

This time tho, this product adventure will be a little different, we will focus less in the trip itself and we will go deeper into the mathemagic 🧙 involved.

So be ready for some numerical bumps and enjoy!

Time is gold 💰

This may sound a little repetitive but this is really really important to us. You may suspect that some retailers want you to spent as much time as possible purchasing in their stores. E.g. changing the product placements from time to time to make you look more around or making you follow a path that tours through all their catalog inside a wooden maze (🇸🇪). All in order to trick you and lure you into purchasing things that you may not really need (we all have fallen into this sometime).

Our business model goes totally in the other direction. We work hard to help you make your weekly or monthly groceries purchase as easy and simple as it can get. We are talking here about a regular purchase of essential things like food or care products, not whims or souvenirs. You want to purchase what you need for you and your family, and we help you get it the easiest way possible. Our wish is that you get all you need and that you return next month to trust us once again.

Discoverability 🔍

For you to find all you need in the less time possible some things are important to take in account. A big deal in an e-commerce is the discoverability of products you don’t know yet but you may be interested in.

Many times the discoverability of this products is based on analyzing and recommending the different variants it has or comparing those that may be related to it.

For a more visual representation about what we are trying to accomplish, we have prepared an example of two purchases. These purchases contain both milk and eggs. In one of them a customer bought bread and in the other one some other customer bought toasts.

The aim in this example would be to relate the bread with the toasts as they are usually bought together with milk and eggs, so they are bought in a similar context.

By recommending products that are bought in similar purchases we help the user find products that belong to a similar context and that may be of interest to them. The objective is to group together items that have been purchased in orders of similar composition. In other words, products that have been bought together with the same set of products. This allows us to recommend similar products that may not have been purchased together.

These recommendations should not be confused with the cross-selling functionality used by other platforms. We are not looking to sell complementary products related to the first one, instead we strive to offer the users more variety and allow them to discover new products they didn’t know about (like different types of croquettes 😋).

The maths 🧮

Here is where things start to going mathy. If you don’t have a strong algebraic background do not worry, we’ll try to explain the magic behind it as easy as possible with trivial and self-explanatory examples. You only need to read it carefully and LinkedIn us if you have any trouble 😊.

Cooccurrence matrix

To sum up before we start, we seek to find products whose relationship to other items is similar. We are not trying to find products that are usually bought together but we are looking to recommend products that are purchased in similar contexts. Like if you are looking at a pizza we will want to recommend you products that are usually bought in similar purchases, like other pizzas or other kind of prepared food (let’s prepare that party 😎).

To do this, the first step is to group products by purchase. The easiest way is to count how many times every product has been purchased with each other to know in how many purchases they have coinicided. This is called the coocurrence matrix. It will tell us for each of the products how many times it has been purchased with the rest of the catalogue.

Starting from the sample purchases presented in the previous section, we can check the resulting coocurrence matrix in the image above. You can clearly see that the bread has been bought once with the eggs and milk and the same with the toasts.

In the example, we can also see that we have represented the matrix in two ways, one of them counting the coincidence of a product with itself and in the other one setting to 0 the diagonal of the matrix, making that a product cannot coincide with itself. In the next section we will see the importance that these values have in the calculation of the related products.

Cosine similarity

Once we have the vectors that represent the coincidences of a product with the rest of the catalogue, we will relate the products that have similar vectors to each other. The more similar the vectors are, the more similar the products will be.

For this we use the cosine similarity of the vectors which calculates the cosine of the angle formed by the two vectors. The closer the value is to 1, the more related they are. This is just a way of calculating the difference between each of the products.

In the heat maps above 🔥, we see the results of calculating the cosine similarity between the vectors that define each product. The lighter the color, the more related the products are.

We see that in the first image the result relates equally the bread to milk and eggs, and to a lesser extent to toast. This happens because they have been sold together and they both are well-selling products (they appear in both purchases).

However, in the second heat map we see how bread is strongly related to toasts instead of milk and eggs that are in second place. We achieve this by eliminating the coincidences of products with themselves (setting the diagonal to 0) and thus discriminating the most sold products that appear in a passive way in all purchases.

By sorting the products by similarity and applying a threshold we get related products with production ready results 🙌!

Results 🗒

Helping our customers to discover new products helps them to make their groceries in less time. Our average customer adds around 3–4 products from related products in every purchase. Items that are added throught organic navigation instead of losing time looking for them.

We also observed that customers that add items from related products tend to have bigger carts, indicating that they find new products that are interesting to them, helping them discover new tastes (cod croquettes are a thing).

Conclusions 📓

Time is important for our customers. Studies have shown that a better discoverability of products in a e-commerce is key. Bad user experience and difficulties looking for products make customers ditch the store before having the chance of finding what they are looking for 😔. That means lost time for the customer and a lost sale for the e-commerce.

The little details are what make an e-commerce great. We strive to provide the best purchasing experience possible and related products is one of the features that make this possible.

This tale has reached its end, but our efforts don’t end here. Our main recommendation system (the one that handles the heavy stuff in our home) lays hidden waiting for you all in the next article of this series. Stay tuned 📻!

Previous articles 🗒

Recommendations I — A scary tale about vanishing ghosts

Next articles 🗒

Recommendations III — A flavored tale about similar tastes

Recommendations IV — A tale about data roles

Recommendations II — A twisted tale about twin products was originally published in Mercadona Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recommendations I — A scary tale about vanishing ghosts

Much of the online shopping experience today is enhanced by opportunities to explore new products. At MercadonaTech we have a centralized system called recommender that calculates different types of recommendations oriented to a wide variety of areas to provide you the best purchasing experience.

We have the vision to build a platform that allows the customer to do their groceries as fast as possible and save their time to spend on something more interesting. Our goal is to be able to do the purchase for you, and so, the groceries would just show up magically at your home ✨. Recommender is one of the first steps in that direction.

In an effort to show some of our product adventures (let’s call them tales from now on) and how we deliver recommendations value to the customer in a lean way we wrote four articles talking about some interesting case studies that we pulled out (they are really interesting believe me).

In this first article we will talk about how a logistic problem that affected thousands of customers every month was solved with a simple implementation that brought quick value and a great purchasing experience improvement.

Be comfortable, grab some popcorn 🍿and enjoy the trip.

Ghost products 👻

Despite our best efforts, unfortunately we can not maintain the full catalog of products throughout the year. For example, there are seasonal products or changes in their format or characteristics. Or just sometimes we change the provider for a given class of product and so the old version you put in your cart is discontinued. Therefore, occasionally there are cancellations of products that may affect the active cart of some users 😢. Some examples are seasonal package changes like coca-cola does, product improvements that change the article’s reference or temporal products that are removed after an event ends (like Christmas products).

This implies that customers can have discontinued products in their active cart. Products that are cleaned out in the checkout process due to their current status. The client added them to their cart and wanted them but due to catalog changes they are not available anymore. This flux cause cart lines to disappear between the phase of building your cart (that can last from days to weeks because of the nature of our ecommerce) and the phase of creating the checkout. This vanishing products then are acting as a spirit disappearing the morning after Halloween 🎃. We call these entities ghost products (so spooky).

This is a double edged sword. On the one hand we are losing potential sells and the other hand (and most important) we are not delivering the customer all the products they want and they think they have bought. That is a big problem.

First steps 🐣

Our first step in our lean journey dealing with ghost products was simply to alert the customers about it. Alert them that some of the products in their active cart were discontinued and no longer available so they would not pass the checkout process.

This alert allowed the customers to be aware of the incidences in their carts and have an opportunity to manually substitute the discontinued products by similar ones from the current catalog.

This quickly contributed to reducing the number of ghost products that were being cleaned out in the checkout process, bringing great value to both the customers and us (yaay!).

With only this change we reduced the ghost products arriving to the checkout process on an amazing 60%!

The not so lean detour 🚵

Once we got the customers aware of the problem the next logical step was to offer them tools to handle it. Substitute the products manually searching similar ones in the catalog was not the best experience and could be improved greatly with a helping UI powered by some intelligence behind the scenes.

Our plan was to recommend customers products that were similar to the ones that were discontinued, sorted by similarity and with ease to be swapped with the troubled ones. Wasn’t that an amazing plan? 🤯

In a short break from our lean adventure, some developers (and a Product Manager 🤷🏻‍♂️) went to the BigThings conference in Madrid. That gave us some free time between talks and in our train journey back to make a short but intense hackathon where we tried to face the problem and experiment a little bit with reinforcement learning.

We developed and deployed a funny website where we were shown two products from the same category. Then we were asked to confirm if we would substitute the first with the second or not. The idea was that with enough data and a clever algorithm like the multi-armed bandit we would be able to score and suggest similar products to one given.

Spoiler alert, was funny but not so useful (yet 🤔). The result of the experiment was not to have the required amount of data to build an usable system. Having +9000 products in our catalog did not make it easy but it was fun to share with our colleagues at the office and sort some products together. As an incentive it shown a random meme from our internal office meme pool (yes, we have one) every time you interacted with it so at least we had some laughs.

Back to work 💪

Coming back to our lean journey we took the problem seriously and we tried to bring some quick value to the customer.

We had some clues about how to tackle it. We knew that we only had some basic info about the products (topic that we will talk about in a couple of tales) but it seemed enough for this task. We had available data like the name, the price and the category, so we went easy.

We implemented a trivial scoring system that made use of the levenshtein distance (that calculates the number of differences between two text strings). Two items would be similar if the levenshtein distance of their names was lower than a threshold. Sorting all similar candidates by ascending order got us acceptable results.

With some UI love from our wonderful designer and our beloved Product Manager we finally got a tool for the customers to substitute discontinued products with similar ones that would do the work.

But this was only an alpha. By the time this implementation reached production we already had outlined a stronger, more robust (and again really trivial 😅) algorithm to rule all future substitutions.

This new algorithm groups similar products tokenizing and comparing the name of the products. The tokenization process is really simple. It removes the brand and stopwords (irrelevant words like “the”, “and”, etc) then converts all words to lowercase and creates a set from them. Then calculates the size of the intersection between the sets of tokens that compose the different products inside the same category.

In the example above the products’ similarity score would be 2, as the intersection of tokens has a length of 2.

By ordering the products according to the number of tokens they have in common and applying a threshold, we manage to show products similar to the discontinued ones. So awesome and simple.

With this trivial process and some infrastructure magic that you will read about in the fourth part we have managed to deliver great value to our customers and to our dear ecommerce.

To give some results we currently have a 28% conversion rate. That means that nearly 1 every 3 products that are checked to be substituted are swapped by a similar one.

Next steps 👣

The similar products recommendation system currently works as expected and it works really well, we are very happy with it. But as a team with a strong data background we can not let it rest there. Remember the experiment we made in the hackathon? The problems with the amount of data and the “yet 🤔” inside brackets in the value contribution result? Well, it happens that we finally have enough data! The system has been working for a few months now and the people have been hungrily substituting products non stop. A multi-arm bandit reimplementation seems not a far reality in the short-middle term. We will keep you updated about it!

Conclusions 📓

As already mentioned, working in a product focused team is really fun. We face challenges every day where we innovate, experiment and have fun. We are able to follow a lean methodology that brings great value to our customers with small and iterative steps that never stop.

This tale about ghosts and forgotten products has taught us that we are able to bring real value to our customers with little changes that slowly accumulate to make our product what it is today.

Unfortunately this tale finishes here, but don’t be sad! This recommendations trip doesn’t end yet, we have three more wonderful tales to tell and you are more than welcome to keep going with us in this amazing journey.

Next articles 🗒

Recommendations II — A twisted tale about twin products

Recommendations III — A flavored tale about similar tastes

Recommendations IV — A tale about data roles

Recomendations I — A scary tale about vanishing ghosts was originally published in Mercadona Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.