Disclaimer: This is about databases, run while you still can!

Intro

In one of our recent projects we, at Product League, were asked to build a messaging system. This chat would have to include almost all the features that known applications such as WhatsApp offer nowadays, such as:

• Real time messaging;
• Support both mobile and web (with no mobile backup — in your face WhatsApp!);
• 1-on-1 and group/channel conversations;
• Sending and downloading attachments;
• Read receipts (the little tick marks that tell you if a message is sent, received or read).

While we were analyzing the requirements for this MVP, we identified the data model as a potential growth bottleneck. The reference architectures for this type of applications adopt NoSQL databases to support their data storage. But since we were in the MVP phase, where the time is of the essence and the level of adoption of the app was still unclear, it was strategically decided to focus on achieving other requirements and leave the NoSQL data model on the backlog for future iterations.

Since we’re using OutSystems, we decided in the meantime to leverage its power and build the chat on top of the included SQL Server database.

For the sake of simplicity, the data model looked like this:

• Conversation and ConversationUser tables, to store the high level information about a conversation and its related users. Data growth won’t be a problem since these tables will be mostly used for reads;
• ConversationMessage, where messages are stored. The amount of information grows linearly with the number of messages sent in a conversation;
• ConversationMessageUser, storing the status (sent, received, read, etc…) of each message and each user. In this case, the amount of entries is expected to grow exponentially since it depends on the amount of users in a conversation.

The challenge

We were quite happy with our result since the application was successfully rolled to production. The adoption was great and the client decided to invest further in the concept. Besides all new requirements and changes due to the close feedback loop between the Users and the Team, we knew that it was time to give priority to the NoSQL entry in the backlog.

The first thing we did was to analyze the implications of the new expected user base, which was predicted to grow over 22000 users.

We noticed that, even with a low usage of the chat (around 5 messages sent per day, per user), the ConversationMessageUser table, that stores the read receipts, would grow on average 1GB per day. Now imagine groups with over 200 users, in which a single message would generate the same amount of entries!

We knew that, sooner or later, performance would be affected by the data volume growth.

Now what?

We started to study possible NoSQL database engines that would support this data growth while keeping the desired performance.

Since we were already using Azure Cloud services in other features of the application, we took a glance at the CosmosDB service, which offers NoSQL solutions. Among the different options we chose the MongoDB API, since it’s a widely adopted database in most of the similar investigated scenarios. The growing community of Mongo also played a big role in this selection. The integration with it was straightforward since there was already a component for Mongo integration in the OutSystems Forge.

Our MongoDB knowledge was limited, therefore we knew that a “fail fast” approach to this data model change was the way to move forward.

First steps

Our initial approach was to have one MongoDB collection and store each conversation as a document. Everything related to a conversation — users, messages and respective read receipts — would be nested inside a single document. This meant that:

• Each action performed by a user (e.g. writing, receiving or reading a message) would update the conversation document;
• Each document would grow indefinitely as the number of messages also grew.

At the same time, we quickly became aware of the CosmosDB pricing model, the Request Units. We’ll call them RU from now on.

RU is a normalized measure that represents the cost of all database operations, abstracting systems resources such as CPU and memory. It defines the maximum throughput (every CRUD action uses an amount of RU) and, most importantly, the monthly cost.

For example, a basic limit of 400 RU/second represents a cost of 23$/month.

This throughput is set when the MongoDB collection is created and can be changed both in Azure and programatically. This way, a smaller value can be set initially, when the usage is lower, and increased over time when deemed necessary.

After some tests, we quickly concluded that this first iteration would become very expensive for our client. Each conversation document was growing quickly and soon we were hitting the 400 RU/s limit with all the updates performed over each document. This data model allowed us to conclude that:

• We were thinking too much in a relational way. After all, each document looked almost as its SQL equivalent;
• Too many updates on such a big document, with so many nested fields, was consuming a lot of RU.

For all this we knew that we had to improve our approach to NoSQL.

Climbing the ladder

The second iteration made us do a 180-degree turn over our initial approach. The idea was to keep the documents smaller, with less nested fields, and to try to stay away from the costly updates. The concept looked like this:

• Each document represented a message from a user’s perspective. For example, in a 1-on-1 conversation, each message was represented by 2 documents, one for each user;
• In any action over a message (e.g. user receiving or reading a message) we would fetch the latest message document, update the necessary fields in the back-end and insert the new document in the database;
• This meant that a message, over time, would be represented by lots of documents — snapshots of the message status in any moment of time. This isn’t a problem since, under the NoSQL paradigm, data replication is a common practice;
• Since the documents had a timestamp, it allowed us to easily identify the latest state of a message.

With the second iteration we also took advantage of partitions (a.k.a. shards), which is how CosmosDB handles the database scaling in order to meet good performance levels. A well defined partition key allows CosmosDB to efficiently spread the load over different servers. A couple of rules of thumb when using partitions are:

• It’s important to define a partition key that is evenly distributed across all the documents. For example, if we’re to use the conversation identifier as the partition key and almost all the messages exchanged in the app were sent in that particular conversation, the load would be focused on that partition. As we want to avoid that, a good partition key should ensure that the partitions are evenly used when inserting documents.
• When retrieving documents, it’s good practice to use the partition key as the first filter condition. This way, CosmosDB only has to access a single partition, thus using less RU and being way faster. Cross partition queries are possible, but are slower and costlier.

We ended up choosing the user identifier as the partition key, which seemed a reasonable approach since the same app usage is expected from most of our users.

On our second concept’s initial tests we saw a decrease on the RU usage and a better performance overall. But, in some cases, the data was not consistent due to concurrent receiving or reading of messages.

Since we were retrieving the latest snapshot, editing it and creating a new one, if two users receive or read a message at the same time, the following scenario happened:

• Users A and B retrieve latest snapshot (in each retrieved document no one has read the message);
• User A creates a snapshot where only he has read the message;
• User B creates a snapshot where only he has read the message, when in fact the created snapshot should say that both users had read it.

We evaluated the possibility of using a message queue, which could help us solve the concurrency issues, but in the end we felt that we needed to improve the data model further.

That’s how we reached the last iteration, in which we aimed to fix the concurrency problem and improve even more the RU consumption.

Final step

We ended up keeping most of the previous concept but decided to store the messages and the read receipts separately, using two collections:

• The messages collection — where we stored each message per user and only once — when the message was written;
• The read receipts collection — each time a user receives or reads a message, a new document is inserted with the latest status. This way, each user is responsible for creating documents that only concern his actions, avoiding the concurrency issues of the previous approach.

The same partition key was kept, meaning that each user would have its own representation of a message and read receipts.

We also tried to keep the documents as plain as possible (avoiding nested fields) in order to keep a low RU usage.

After running a lot of tests, this version was working well in all scenarios and was considered stable. It allowed simpler queries, which helped bring down the RU consumption overall.

What next?

We’re now eager to roll the solution to production, but taking in account the following considerations:

• The MongoDB feature will initially live in parallel with the existing SQL solution. It can be toggled on and off without affecting the user experience;
• A/B testing can be applied to just a few conversations, ensuring that we have a way to control and adjust the RU limit to a real scenario. It will allow us to roll out the change progressively to all the users;
• With the data migration from SQL to NoSQL we expect a higher RU consumption. Fortunately, CosmosDB allows you to adjust the RU/s limit when needed.

Final notes

This was an exhilarating learning journey, from which the key takeaways are:

• The MVP approach is the best way to promote adoption and evolution;
• Failing fast allowed us to improve our knowledge on non-relational databases and quickly iterate in order to gradually enhance our MongoDB implementation.

Wrapping up, we adopted NoSQL and reached a great solution! Nonetheless, we’ll continue to look for improvements and to learn from other’s experiences.

References

CosmosDB request units:

• https://docs.microsoft.com/en-us/azure/cosmos-db/request-units

CosmosDB partitions:

• https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview
• https://azure.microsoft.com/en-us/resources/videos/azure-documentdb-elastic-scale-partitioning/

A good IDE to explore MongoDB documents:

• https://studio3t.com/

Learning NoSQL the easy way was originally published in ProductLeague on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Firebase?

I’m sure everyone who builds a mobile app will want to know their users, what they do and how much time they spend on it.

Having this kind of information will allow you to make better decisions in order to expand your user base and to keep them happy with the app. Acquiring it, however, can be a hard task.

That’s when Firebase comes to the rescue. It was built with mobile apps in mind and comes packed with many products that provide us with ways to get to know our users.

Firebase features

Firebase comes with an Analytics tool which gathers information on the application usage and user engagement, through event reporting and user properties. It gives us insights that go from user acquisition to app interaction and user retention. Some cool features are:

• Some events are automatically captured — such as app installation, app start or app removal. It also allows us to define our own custom events, with optional event parameters, to measure what fits our business needs, and use them on button clicks, screen views, etc.
• User properties can be used to define audiences or user segments. These segments can be later on targeted with notifications or specific app configurations. We could, for example, pick only the users that are inactive for a period of time and send them a push notification to get them back in the app.

Firebase also comes with other useful products such as:

• Crashlytics, a crash reporting tool, which gives us a detailed report on the app crashes and how many users it affected.
• Performance monitoring, helping us to track down possible performance issues. It allows us to measure key interactions (such as login, data synchronization, etc) and to check how long they take on a variety of different conditions, such as app version, mobile phone brand, type of internet connection, operating system and so on.
• Cloud messaging, giving us the possibility to define and send push notifications to users.
• Remote configuration, which allows us to define configurations and apply them to a set of users. For example, we could target 10% of our user base and have them experience a new app theme. Joined with A/B testing, audiences and analytics, it proves to be a powerful tool to test the efficiency of new features on a subset of your user base.

This all sounds great, but what’s even better is that its base plan is free!

Now, an important question... How can I integrate my OutSystems mobile app with Firebase?

Our component

Well, we built the Firebase Mobile component and published it in the OutSystems Forge! It provides the integration with Firebase for both Android and iOS operating systems, while also including a server-side module responsible for sending push notifications through Firebase’s API.

We also added to the Forge a sample application showing some of our plugin’s features and how to use them. You can check it here.

Configuring Firebase may seem the hardest part, so that’s why I’m going to only focus on that for now. Feel free to follow this guide and configure the sample app with your own Firebase project in your OutSystems environment.

How to configure

So, first of all, go to the Firebase console and create a project if you don’t have one already.

Add a new Firebase app for each mobile operating system your app supports. Since OutSystems allows us to build native apps for Android and iOS, let’s add both. While adding each Firebase app, use the app identifier obtainable from Service Studio.

In the Firebase console, open the Android app wizard, register the app in the first step and download the configuration file in the second step. Keep the file — google-services.json — since we’ll be using it next. Skip the remaining steps.

Similarly, register the iOS app and download the configuration file — GoogleService-Info.plist. Also, safely skip the remaining steps.

Now that you have both configuration files, let’s prepare them so that we can add them to the OutSystems app.

Add both google-services.json and GoogleService-Info.plist to a zip folder. Ensure that the folder is called google-services.zip and that it only contains both config files.

Finally, open your app module in Service Studio and add the zip folder in the Resources tab. Make sure that:

• The Name is “google-services.zip”
• The Deploy action is “deploy to target directory”
• The Target directory is the concatenation of “firebase.” with the app identifier. So, if your app identifier looks something like “com.example.myapp”, the target directory should be “firebase.com.example.myapp”.

With the configuration done, you can now build the application and use Firebase Mobile’s actions in your code. When you install the app and open it, it’ll be successfully registered with Firebase.

Support to multiple environments

If you’re wondering why we have to configure the “target directory” of the configuration files to include the application identifier, that’s because we need to support having Firebase in a multiple environment scenario.

For example, if we have 3 environments (DEV, QA and PROD), and a different app identifier in each environment, we’ll also end up with a Firebase project for each environment.

In this case we’ll need to create 3 zip folders with the respective configuration files and add them in the Resources tab in the lowest environment (DEV).

Afterwards, we can safely deploy our application to the upper environments with Lifetime.

The plugin will pick the correct zip folder when the app is being built and apply the right configurations to the app.

Final notes

It’s important to note that Firebase is constantly being improved and receiving new features. Also, the mobile ecosystem is an ever-changing world, with new OS versions almost every year. Our plugin will also continue to adapt to all the external changes.

If the article aroused your interest in this amazing tool, please visit the Firebase Mobile plugin page, which also includes some useful documentation, and the Firebase Mobile Sample, where you can test some features and take a look at the code.

If you have any doubt, feel free to ask a question in the forums.

To wrap it up, I would like to thank my Truewind team, the github repo which served as a base to our plugin, and finally you for reading the article!

Firebase for your OutSystems mobile app was originally published in Truewind on Medium, where people are continuing the conversation by highlighting and responding to this story.