Why should Tokopedia have Bubbles?

In Tokopedia, we believe that communication between sellers and buyers are important to ensure the transaction will go smoothly. Like many e-commerce platforms out there, the communication in Tokopedia between sellers and buyers is done mainly using a chat room.

On a daily basis, Seller receives a hundred chats. While also they are not 100 percent dedicating their time to replying to chat as they are also doing other work instead. They could be opening the other apps instead of Tokopedia Seller, browsing, watching movies, or even playing games on their phone while a promising buyer was asking for their products in the chat.

While quick responses will make sellers more trustworthy. It will also increase the probability of the transaction being through as a longer response time can make buyers look for other sellers instead. This is a pain point for sellers since their performance also relies on their response time.

Goals

So to avoid any miscommunication between Sellers and their potential buyers, Sellers should be able to:

• Access the most urgent chat faster
• Respond faster, Sellers need to be able to access the chatroom faster. So that more incoming orders can be generated

Here, our team has decided to implement the Bubble Chat features in Tokopedia SellerApp. It enables chats to appear in a pop-up Bubble over other apps’ screens, making it easier for Sellers to see incoming chats and making it easier for them to access the chatroom; hence reply faster.

As Android Developers explored the opportunity to solve the problem, our research showed that Google has released an API that could help with those use cases. We finally decided to use Bubbles API.

What is Bubbles? And how is its behavior?

Before we explore the implementation, it’s crucial to understand the overall behavior of this Bubbles feature. This will help us identify the aspects that can be advantageous for us and align with our needs in addressing the initial issues (Sellers and Buyers). Additionally, it allows us to prepare for the potential user or technical challenges by implementing necessary mitigation strategies.

Bubbles API started in Android 11 or later OS versions, previously developed by the Android team in the developer preview for Android 10. Bubbles are integrated into the Notification system with several user behaviors, including:

• Bubbles can be expanded to reveal app functionality and information or collapsed when not in use.
• Bubbles float on top of other apps’ content, and clicking the shortcut icon in the push notification bar makes them appear.
• The most recent chat bubbles appear on the far left.
• Bubbles only display 5 shortcuts; additional ones are shown by clicking the plus icon, depending on the OS version.
• Android provides one floating Bubble for multiple applications that support Bubbles.

Bubbles Implementation Journey on Tokopedia Seller App

Alright, In this section we would like to explain in general how to implement notification Bubbles. The Bubbles are created into the Notification API, and according to Bubble Documentation, Bubble API has three components that have to be implemented including the MessagingStyle, SharingShortcut, and BubbleMetaData.

Prerequisites

• For the targetSdkVersion, you should set Android 11 (Android API 30) or later.
• The minimum AndroidX.core version required for Bubble support is 1.5.0. However, it is recommended to use version 1.7.0 or the latest version, according to the Androidx.core documentation.

Messaging Styles

According to the Notification documentation, Starting Android 7.0 to implement best practices within the Notification, it’s recommended to use NotificationCompat.MessagingStyle. This approach allows for more customization, including conversational titles, additional messages, and custom content views (e.g., avatars). There are many benefits to using MessagingStyle than the old approach using BigTextStyle or InboxStyle itself.

Sharing Shortcut

After implementing messaging styles, we continued to implement the Sharing Shortcut. Notifications must be associated with a reference to a long-lived shortcut, enabling straightforward content sharing in conversations directly from the published notification.

BubbleMetaData

BubbleMetaData is one of the most important to implement Bubbles, The BubbleMetaData class has the responsibility for launching the intent, setting the Bubble height layout, and displaying the avatar based on the icon parameters including the width and height of the avatar.

Person

Messaging Styles and Sharing Shortcuts should implement the Person class. Person class is used to associate a sender or recipient with messages within notifications. The Person class will help with the representation of messages by displaying the sender’s name and an adaptive Icon as a Bubble avatar.

Icon

The icon in the Bubbles is one of the most important to handle your Bubbles to ensure delivering bubbles features with a seamless experience. Since your Bubbles avatar icon comes from the Backend team as an image URL, you need to convert the image URL first to Bitmap in the IconCompat class. In the past, we learned something from this conversion process, if you’re not handling it properly the notification or Bubbles will crash or it will not display to the users.

How to prevent the crash or issue? When the process of converting the image URL to Bitmap, we need a few seconds to set the threshold time and make sure the notification is not delaying the process of converting the image URL to Bitmap. In case the conversion fails, you should set the default avatar to prevent a broken user’s experience.

Additionally, to prevent repeated requests for the same image, we utilize a caching mechanism to improve performance.

For full implementation of Bubbles can be read on the sample app.

How are Sellers aware of this Bubbles feature?

Before we get into explaining the Monitoring Bubbles, we had a challenge: how do we let sellers know we got this cool new feature that could really help them out?

Currently, the Bubbles entry point is built only at the right bottom of incoming notification and it’s highly and definitely skippable for some users. Even me and my team don’t know how to access Bubbles easily.

Our sellers have diverse profiles, and not all of them may be familiar with the Bubbles feature. To address this, we collaborated with the Product, Product Design, and Data teams to boost awareness. Our objective is to introduce sellers to Bubbles by implementing a ticker in the Seller chat list and the Seller Chat Settings. The objective is to introduce how to use the Bubbles to our sellers such as implementing the ticker in the Seller Chat list and also in the Seller Chat Setting.

Impacts: The feature’s awareness initiative has generated positive results, with an adoption rate in the thousands, showcasing its effectiveness.

Monitoring the Bubbles by implementing tracker

As Android developers, implementing a chat bubbles is not just about getting things done; it’s crucial to monitor the performance of this bubbles feature. Therefore, we need to implement a tracker within the chat bubbles so that we can improve performance based on user usage in the future.

To properly implement the tracker, we must understand the lifecycle of the chat bubbles and know how to distinguish between a chat bubbles screen and a regular screen. This ensures that the tracker works accurately, without any errors in its implementation within the chat bubbles.

Bubble Lifecycle Explanation

As you know, the Bubbles launched used the Activity class and it is similar to the normal page as usual. In this part, I would like to explain the Bubbles lifecycle that we experienced when implementing the Bubbles feature. Okay, let’s say sellers have many chats from buyers, and notifications are coming.

The first time a notification is received, Bubbles has not been created yet. When notifications arrive and a seller clicks on the Bubbles chat entry point in the notification tray or popup, it triggers onCreate() and continues until the activity reaches the onResume() lifecycle.

When the next notification comes and the Bubbles have been created, but the bubble position is collapsed, the Bubbles still stay in the onStop() lifecycle with the new flag thus indicating an incoming new message which is a red dot in the Bubble avatar.

How about the onStop lifecycle in the Bubble Activity?
If your page has a back button, you should override the onBackPressed() method as usual. When the user clicks the back button, the activity will enter the onStop() lifecycle.

How do I destroy the Bubble activity?
As you may know, when used programmatically, we simply invoke the finish() method within the Activity class, or you can use cancel(notificationId) it in the notification. Afterward, the Bubble will be destroyed or dismissed. Alternatively, the user can manually drag the Bubble to the close button at the bottom.

How do we differentiate the Bubble Screen and the Normal Screen?

As mentioned earlier, after understanding the bubble lifecycle, it’s also crucial to know how to distinguish between bubble screens and normal screens to properly implement trackers in a bubble.

So, the Android API library provided the flag to differentiate between Bubble Screen and Normal Screen. In the starting AndroidX Core 1.7.0 version, we’re able to use the isLaunchedFromBubble flag from ActivityCompat to differentiate between the Bubble Screen and Normal Screen like the code below.

val isLaunchedFromBubble = ActivityCompat.isLaunchedFromBubble(activity)

Mitigate Users’ Issues with Bubbles Settings

How do we allow Sellers to manage settings in the Bubbles Chat?

When rolling out a new feature, it’s important to think about ensuring a smooth experience for users who may not be familiar or not comfortable with it. We discovered the Bubbles Settings page in the Android Documentation, which helps in seamlessly introducing the Bubbles feature. This page can be a lifesaver, preventing a ton of complaints, as sellers can easily disable Bubbles directly from the Bubbles settings page.

Here is an intent name you could redirect to Bubbles settings based on your app package.

val bubbleSettingsIntent = Intent(Settings.ACTION_APP_NOTIFICATION_BUBBLE_SETTINGS)

On the Bubble Settings Page, there are three options you can choose to manage the Bubbles. These are:

• All Conversations can bubble: When new notifications come from the user (buyer), the notification will always appear as a Bubble. Even though the user doesn’t click on the Bubble notification entry point.
• Selected conversations can bubble: This preference will be selected if the user has clicked any notifications as the Bubble.
• Nothing can bubble: If you want all the notifications not appear as a Bubble even though the conversation has already been enabled.

Can we check the Bubble setting preference programmatically?

Absolutely!! Fortunately, the Android team provided the Bubbles preference method from the NotificationManager class, allowing us to easily check whether Bubble settings were enabled or not in the Seller App. Here’s a sample code you can try:

private val notificationManager: NotificationManager? = ContextCompat.getSystemService(context, NotificationManager::class.java)

val bubbleSettingPreference = notificationManager.bubblePreference

Also, The NotificationManager class has provided the three constants, which can be checked along the BubblePreference method

• BUBBLE_PREFERENCE_ALL
• BUBBLE_PREFERENCE_SELECTED
• BUBBLE_PREFERENCE_NONE

Recap

• Our journey was truly worth it as the Bubble Support for Sellers initiative, supported by ticker awareness, help pages accessible within Bubble Chat onboarding, mitigation documents, etc., proved to be genuinely beneficial for our end-users.
• Consider providing seamless experience by converting the image URL to a Bitmap in the IconCompat class for Bubble shortcuts.
• When implementing a new feature, as a developer we should think of a long-run plan, such as mitigating the risk, etc.

Collaborating with tech, product, and design teams was impactful but posed challenges, especially in implementation. But it pays off! The Seller Bubble feature increased sellers’ read speed by up to 90%, leading to faster reply speeds and a notable rise in daily orders. These outcomes highlight how the feature significantly improves communication between sellers and buyers, resulting in increased daily transactions.

Credits

A big shout-out to our Tech counterparts: Reivin Oktavianus, Kelvin Saputra, our Android Leaders, Product, Product Design, Data, and Test Engineer Teams for making this incredible collaboration a reality! Thank you for your efforts in bringing the Bubble Support for Sellers initiative to live!

How a Bubble Chat Increases Up to 90% of Seller’s Read Speed was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background

Imagine your company has many cloud resources and collaborates with multiple cloud service providers. Thousand of resources consist of multiple types of resources distributed unevenly across multiple cloud service providers and you, as the infrastructure engineer need to decentralize this management to make sure that every software engineer has enough visibility to do anything but not too much as many of these also contain confidential information. What should you do?

Introducing the Infrastructure Encyclopedia

Analytical Infrastructure Encyclopedia or Let’s just call it Infrastructure Encyclopedia will help to unify all of the information into one single place and will help achieve the ultimate decentralization of Infrastructure Information knowledge.

Sources and Integrated Data

Some of the examples of sources and data that are usually integrated into the encyclopedia:

1. Cloud Services Provider
   List of Infrastructures, Machine Specification, Storage Specification, Billing, tag, Infrastructure alert and event, etc
2. Observability Platform
   Infrastructure metrics such as CPU, Memory, and disk usage and any other custom metric as well if needed such as API total request, API latency, etc
3. Other Platform
   Another platform that manages Engineer hierarchy, ownership, and any other platform that might elevate the usage of the encyclopedia

Layer Explanation

The Encyclopedia is separated into 3 layers which have their functioning purposes:

1. Integration
   Which will be oriented toward integrating all of the data above across multiple sources. I think this is easily achievable by using an application service that connects with API through all of the sources. The data can be stored in the data warehouse, or maybe another analytical data store.
2. Process
   Integrated data will be processed internally to make sure that everything is already been processed before moving to the presentation layer.
3. Presentation
   Processed data then will be displayed into any kind of result, be it a sheet report, business insight tool such as Looker Studio, Alert, or used in another application as well

End Result

1. Infrastructure Resources Utilization Recommendation & Budgeting

As the encyclopedia already has these data:

• List of resources and billing from cloud services provider
• Utilization usage such as CPU, Memory, Disk, etc. in the observability platform
• Ownership and team hierarchy from another platform

Creating a recommendation system that can give visibility to all of the engineers based on the owned resources to accelerate optimization action is a doable task alongside the visibility of the expense of each resource.
But Don’t forget to implement Role Based Access Control (RBAC) as this information is quite confidential if not targetted correctly.

2. Infrastructure Resources at Risk Monitor

Some observability tools already have an agent that pushes all of the information including other installed agent versions, OS versions, etc. Combining with the similar data above, we can construct a dashboard that can show any kind of software that already reaching its End Of Life or is already at risk, making a decentralized patching much easier

3. Infrastructure Resources Summary

As we have many types of instances across cloud service providers, having visibility for all of the resources makes it easier to make any decision based on the data that we have.

And many more use cases that can be elevated through this encyclopedia

In Conclusion

If your company has multiple types of resources across multiple providers, Having a way to store all of the necessary information into one single platform will be beneficial as it will accelerate any kind of engineering decision-making and action. Infrastructure Encyclopedia works as a foundation for solving many complex problems that utilize infrastructure information.

In Tokopedia, we are implementing multiple cloud strategies to get the best from each of the providers to further enhance the reliability and capability of our infrastructure. We utilize the encyclopedia as our stepping stone to decentralize any massive act of engineering, from tech improvement initiatives, patching, Optimization, and many more aspects that we elevate.

Decentralize Infrastructure Knowledge through Analytical Infrastructure Encyclopedia was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tokopedia has a strong culture of experimentation, we do lots of experiments across our properties. For example, when you open Search Result Page in Tokopedia App, there’s a pretty good chance you’re part of many experiments running at the same time. With a very high stake, we always try to ensure that every component of our experiment tool is running reliably. In this chance, we will discuss how a good experiment tool does user distribution correctly.

What can possibly happen?

In a nutshell, the AB testing tool’s job is to distribute users between 2 buckets of control and treatment. However, we noticed that this seemingly simple task is not that simple. There are 3 possible issues that can happen if the distribution is not done well:

1. Biased Allocation
   Ideally, a group of users being allocated to the buckets can represent the diversity of the whole population, in terms of demography, buying power, or other criteria important to Tokopedia. Biased allocation happens when a bucket doesn’t represent this diversity, for example when users with high buying power are too dominant in certain buckets.
2. Selection Bias
   This can happen when a group of users with certain traits is selected, so the result of the experiment is compromised. When a group containing a dominant user with high buying power is selected, it is almost guaranteed that this bucket will perform better.
3. Memory Effect
   Once an experiment is finished, it can have lasting effects on users, good or bad. This lasting effect can affect future experiments if the group of people is distributed unevenly in the next experiment, or worse the group is reused.

Modulo operation is not random

It seems that those issues are hardly possible to happen, especially when the distribution is done randomly. However, we observed that the commonly used technique of using modulo operation with static or dynamic divisor can suffer from those problems.

There are common 2 ways of using modulo to distribute users to a bucket:

1. Static slot
   100% population will be divided into 50 slots using % 50, here 1 slot represents 2% of the user. Bucket distribution will be done by selecting slots until the desired population size is reached, e.g. For the 10% experiment, 5 slots will be used.
2. Dynamic slot
   100% population will be divided into an arbitrary number of slots according to the needs. Eg. for an experiment targeting 50% of users, a modulus of 100 is used and the first 50 slots are allocated. In this case, the divisor can be different between experiments.

As you might see now, the traits of simple modulus-based distribution are :
a. It’s not random enough, and the member of a slot never or rarely changes. The randomness of the allocation relies on the order of the user_id, which doesn’t have any randomness guarantee at all. This can lead to a biased allocation problem.
b. The slot’s member never changed with the static divisor, even on the dynamic divisor some users will always get allocated to the same slots. Not only memory effect can happen, but each slot can also develop its own set of behaviors/tendencies which can lead to selection bias.

Double Hashing: A better way to distribute user

So far we gathered 2 important requirements for our experiment tool, it has to be random and well-rotated for every experiment. In Tokopedia we develop a technique to do the allocation and randomization, on the allocation side we still rely on modulus operation, and for the randomization, a hashing function is used. The process is being done twice, that’s why we call this technique double hashing.

First Step

The population is divided into 100 buckets using modulo operation. However, instead of using user_id directly, the user_id will be hashed beforehand to get better randomness.

user bucket = hash(constant_prefix+user_id)%100

In this step buckets of users will be randomly allocated according to the experiment target size.

Second Step

We further divide allocation from the previous step into 10.000 buckets. Again the user_id will be hashed, this time using a prefix coming from the experiment.

user bucket = hash(experiment_name+user_id)%10000

From those 10.000 buckets will be divided further to form Control and Experiment (A/B) buckets.

By using 2 steps allocation and randomization the intention is to avoid the 3 problems mentioned above and minimize lasting effects between experiments even when their population is overlapped.

AA test to prove

At the end how good our tool or method to do things will need to be validated, better if the validation can be done regularly before every experiment is started. To prove that there is no Bias on the population in the experiment, AA usually is being used.

AA test refers to getting statistical significance from 2 different buckets of experiment with before any treatment being introduced. If we see no significance difference between buckets, meaning that our allocation is free from any bias.

Tool to “Roll” them all

Topic of online experimentation is always interesting to discuss, it combines product development, statistical and scalability challenges. Reliability, scalability and correctness of the system is very important, to make sure that data driven decision making can be done confidently.

In Tokopedia E2E journey of experimentation including allocation, AA test, AB Test, post analysis and gradual features rollout are being done on top of Rollence Experimentation Platform. An homegrown experimentation tool that has been reliably servicing our needs since 2019, as the successor of another homegrown tool we use earlier.

The importance of user distribution in online experiments (AB Testing) was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unit testing is an essential part of software development, especially for large-scale applications that serve millions of users. It’s crucial for the developers to ensure that every feature is tested thoroughly and that every new feature implemented does not create new bugs for existing features because it can cause significant loss for both developers and users.

While having unit tests is important, maintaining and running the tests can be time-consuming and resource-intensive, especially on a large-scale app that has tens of thousands or more test cases.

When one cycle of building and unit testing takes too long time, it would decrease the productivity of the developers themselves. This is especially true on a packed CI/CD system that only has few machines handling dozens of developers.

For our case, we have 9 CI/CD machines with ~65 developers developing daily. Usually each developer works on multiple branches, with each branch triggering both build and unit test job. Can you imagine if single unit test job could take 30 minutes?

In this article, we’ll go through our journey to optimize our daily unit testing process in Tokopedia iOS Team, which started with how we did our unit testing back in the days to where we are today.

The Pain Points
By optimizing our unit test performance, we addressed some pain points our team has:

“OMG! This fix needs to be merged ASAP, but there are 10 idle queues on the CI machine waiting to be executed :(”

“I just edited our GitHub code owner, the PR only has a single changed file with two new lines. What did you say? I need to run all the unit tests, really?”

“Ok, my PR should be ready, let’s trigger the CI to run all unit tests.”
*30 minutes later*
“What? How could my unit test fail? Oh, I forgot to commit my latest changes. 30 mins retrigger here we go, let’s do the housekeeping while waiting.”

The Physical Education teacher way: let’s run all unit tests!
In Tokopedia, we adopted modular app architecture using Bazel as our build system (read more about it here). Therefore, all explanations will be based on Bazel, not Xcode’s native build system.

Our first unit testing process is by querying all unit test targets using the bazel query, then run test on each target from query results. Please note that by adopting modular app architecture, our app consists of dozen of modules (targets) for each feature/test case.

The process is fairly simple but it takes a really long time to run tests on each target individually. This is because every time Bazel runs a build or test, it has its own build process. Namely the dependency analysis, validation, and linking process that could slow down overall testing time.

While this approach is not necessarily bad, it does not work well for our large-scale app. We have hundreds of test targets with tens of thousands of test cases, and running them individually takes a really long time.

On the other hand, if we have a simple app with a few test cases, the time difference is negligible, so improving the unit test process may not be worth the effort.

Remember that I mentioned the Bazel’s build process in the last paragraph? Yep, in the next section, we will talk about how we optimize this build process to make it faster. Let’s head into the next section.

The Marriage Way: where all unit tests unite into one

After using Physical Education (PE) teacher way for a while, we realized that Bazel’s build process takes a toll on our unit test pipeline performance. The dependency analysis and overhead time starting and ending XCTest runner make our unit test time vary from ~20 mins to ~35 mins because we have hundreds of test targets.

So we tried to unify all unit test targets into one single target. The idea is to simplify dependency analysis and overhead time caused by running each test one by one. A rough idea of the process is in the following pic:

The downside of this approach is that it won’t utilize Bazel’s caching to its fullest. Whereas the old approach will cache the test cases that didn’t have code changes.

After testing it for about ~2 months, we realized that this new approach will run faster compared to the old unit test way despite the downside that I stated before. It saves time on the Bazel’s build process (analyzing packages, dependency, etc) whereas the old approach will repeat it for all test targets.

The plot above illustrates that employing a single unit test enables us to achieve more stable unit testing times, with an average duration of approximately 450 seconds. This indicates that the single unit test approach can lead to more consistent and reliable testing results.

Conversely, using the traditional non-single unit test approach leads to a significant number of spikes in the unit testing time, with the highest value being 50% greater than the highest value obtained using the single unit test approach. This suggests that the traditional approach can result in less predictable and less reliable testing outcomes.

Lastly, single unit test also significantly decreased our unit test median to ~563s from ~1772s. This indicates that the single unit test approach can also lead to faster and more efficient testing processes, which can ultimately result in time and cost savings.

Our unit test journey doesn’t end here, we keep trying to find ways to improve our unit test performance to make our workflow more efficient.

The Buffet Way: where only selected unit test matter

Here comes our latest approach, let’s use an analogy to make it easier to understand.

“If you spilled something on your work table, should you wipe all your co-workers tables on our floor?”

“No, are you crazy? I will only wipe my own table and its surrounding which were hit by its splashes.”

“Yep, that’s how you should do unit tests too.”

Do you get it? Our last approach is the selective test where only related unit tests will be tested. So for example, let’s say we have the following structure for our unit test:

Let me give some context about what the arrows and circles mean. Each circle means a module and every arrow means that something is dependent on the other thing. So for example in the above diagram, we have modules A and E pointed to module C, which means that modules A and E are dependent on module C. In other words, A and E are the reverse dependencies of C.

Based on the previous analogy, if module C has some changes (getting milk spilled on), then directly affected modules will be module C itself, and indirectly affected modules will be A and E. So back to our unit test discussion, when we have changes in module C, we should test modules C (obviously), A, and E as illustrated below.

Why A and E? You may ask. Because modules A and E use module C as their dependency, so every change in module C will indirectly affect modules A and E. We won’t test the rest of the modules because of their transitive dependency nature. When A depends on B, and B depends on C, if C is correct, then A should be correct too.

With the long boring chit-chat, let’s jump into the flowchart of our selective test implementation!

With the selective test, we implement these three main steps:

Step 1 — File changes comparison
Usually, when a developer starts to create their features, they will create a new branch from the main branch where all changes will be combined. This step will compare the file changes from their branch to this main branch. The output from this step will be a list of paths for each updated file.

Step 2 — Target path extraction
The output from step 1 is only the file path, we need to find its target name before we proceed to the next step. In this step, we will recursively find unique Bazel’s BUILD file path for each directory.

For example, we have this path:

parent/childOne/childTwo/childThree/ChangedFile.swift

We will recursively find Bazel’s BUILD file from parent/childOne/childTwo/childThree. If Bazel’s BUILD file exists, we will save its path and break the recursion tree.

Step 3 — Reverse dependencies query
After we found the changed target, we will find its reverse dependency using Bazel’s Sky Query allrdeps command. Then we will combine each reverse dependency into one target just like approach number two above, and test it.

After selective test is live, we tested it for about ~3 months in a real development environment. We found significant improvement in unit test speed, especially when the file changes only affect small modules that not many modules use as a dependency.

Now let’s get back to comparison plot. As you can see, the main difference in the distribution between selective test ON and OFF is on the lower end of the data. Since selective test focuses on running the minimum amount of tests while staying true, it is expected that we saw increased amount of low (fast) unit testing time. The median of our unit testing duration also decreased to ~250s from ~500s.

The main advantage of selective test is we also could reach zero unit test time (yep, the test is directly marked as passed because there is nothing to be tested!). This is the case when our file changes does not affect any main code base, such as GitHub’s code owner file. The evidence of this advantage is the amount of data point that reach approximately zero second when selective test is activated in above plot.

In above graph, we could see that single selected test could halven the unit test speed. Thanks to the possibility to reach instant unit test experience when nothing to be tested. Do note that if the file changes are related to a lot of modules, the unit test time will be more or less the same as single unit test from second approach.

Recap

Unit testing is one of the most important step in our engineering process to make sure whether something is impacted or not, at least on the feature logic side. Therefore, its essential to improve unit testing time because unit testing could take a very long time, especially in a large-scale app.

Our unit testing improvement journey wasn’t as smooth as this article stated, it was full of ups, downs, and many experiments which led to where we are today. But as we always say in Tokopedia — Make it happen, make it better.

Credits

Last but not least, enormous thanks for Wendy Liga who started this odyssey with single unit test and involves me on our latest improvement — Selective Test which in total improved our unit testing duration 8X FASTER compared to first iteration.

Also thanks to our CI/CD machines for working with us, they are in happier state now 😆

Stop Wasting Time on Unit Testing: How Tokopedia Achieved 8X Faster Results was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cross platform native development has been the craze since the first launch of React Native back in 2015.

Tokopedia had its own history with React Native, having been used for several years on both iOS and Android apps, we soon found out that all the benefits came with the cost of poor performance which of course significantly impacted the user experience.

With the #FocusOnConsumer mindset, we eventually decided to sunset all React Native along with the hopes and dreams of having a single code base for both Android and iOS.

…Until Flutter was introduced.

The Project

For the Tokopedia START Summit 2022 event we wanted to enhance the participant experience from last year by creating an application on all platforms that will support participants by showing the agenda, sponsors, and ongoing sessions.

The mockup of the Application

The caveat? we only had approximately 3 weeks to develop, publish and announce to the participants, no doubt many questions arose from the team along the same topic of

“How do we deliver an application in less than a month on both Android and iOS in less than 3 weeks?”

Of course going back to a cross platform framework came into mind. Although learning our mistakes we had to ensure the technology and approach we used were all appropriate and scalable enough for future usage.

The Development Phase

The decision to go ahead with Flutter was because solved our problem of time and manpower, furthermore it was a very good investment. With dozens of new libraries appearing, growing community support, and continuously improved performance we foresaw that Flutter might have a better future. However we were soon faced with several challenges.

Challenge #1: Cross Team Collaboration

The team consisted of engineers from iOS, Android and Web Platform engineers. Each team had its own set of rules and standards and had to collaborate on the same codebase, this of course created some issues. For example; Some teams prefer to use squash & merge instead of merge commit when merging to the master branch or using rebasing instead of pulling when syncing branches.

We found out that it’s important to have an alignment in the beginning to set the same expectations amongst the engineers in the beginning and also over-communicate any issues they had.

Challenge #2: Project Scope & Collaboration with Product Managers

As mentioned before, Flutter is relatively new in Tokopedia. Because of this a feature that may only take 2 days to develop in normal conditions, might take an additional 2 days with a new technology. These 2 days act as a buffer because when working with a new technology engineers need time to research and of course debugging when stuff goes sideways.

It’s important to factor in those additional man days when creating your tech plan and ensure all stakeholders are aware so the team can adjust the scope & timeline accordingly.

Challenge #3: Deciding on a State Management

State management can be defined as a way to manage states, or various properties that are constantly changing inside of an application. Various platforms have different approaches when handling states, web engineers use state managers like React Context or Redux for React.js, Android Engineers use state holders and state hoisting for Jetpack, and iOS engineers use ObservedObjects for SwiftUI. With different paradigms, it was difficult to settle on solution.

First on foremost we all agreed that the approach had to be extendable, predictable and of course easy to learn. After several discussions we decided to use BLoC because it solved each of our concerns well:

Extendable

BLoC uses the concept of “business logics” that are abstract and can be used in multiple platforms and use cases. As a matter of fact, because of the abstraction and power of BLoC, we ended up using BLoC for most of our state management implementations.

Predictable

We always want to give the best experience, hence handling every edge case well. For example, what should the users see when there’s a query error? When there’s no internet connection? Or when the data is empty? The power of BLoC gives the powers to us engineers to do just that. BLoC leverages of “states” to control the UI. So we can set the UI to update when the state is “empty data”, “no internet connection”, and “query error”.

Easy to learn

BLoC uses the concept of events that are emitted to update the states of a UI. This approach feels familiar to both mobile and web engineers, making the learning curve faster for everyone.

Application Performance and Size

Performance is a main concern when using a cross platform framework based on previous experiences. Thankfully we did not see any significant performance degradations when using Flutter.

App Size

iOS → 29.5MB

Android → 7.9MB

Build Time

Average build time for our application took on average between 1m 30s until 2m to build

Conclusion

Overall our experience on working with Flutter was delightful, most of the challenges we faced was a results of everything except Flutter itself.

Of course this is all based on our conditions and environment so we recommend all engineers to research thoroughly before going fourth with a new technology like Flutter.

If you’d like to check out the application itself feel free to download it through these links:

Android Play Store
Apple App Store

References

https://blog.logrocket.com/complete-guide-flutter-architecture/

Flutter - Build apps for any screen

https://blog.logrocket.com/state-management-flutter-bloc-pattern/

https://bloclibrary.dev

https://flutter.dev/

Our Experience with Flutter in Tokopedia was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the first part of the modularization tutorial. Refer to the list below for the complete parts :
Part 1: Introduction
Part 2: Extracting Catalog
Part 3: Final Extraction and Micro App

App modularization is an issue everyone runs into as a software engineer, moreover, if your app is already grown to the scale that it takes too long to build the entire app, just like the Tokopedia app. In our case, we have to find a way to fasten our build time to meet our weekly release and this is where the modularization helps. By modularizing our app, we can choose to build what we need at the moment thus improving our build time and everyone is happy!
Tokopedia uses a custom modularization approach to enable fast development, but before going to custom modularize your app, you have to understand the basics of modularization and all the other related things you need to know, this article will cover all of them and guide you to have your first taste of modularization, hang tight!

There is a slew of solutions you can use to tackle modularization. From splitting codebase chunks to increase maintainability to creating good encapsulation. But what is the motivation behind modularization?

When should you split your codebase into modularized parts?

Why should you even do such a thing?

What is Modularization

Modularization is the practice of dividing features of an application into separated modules so that they are interchangeable, reusable, and easily maintained. If you prefer a less-technical explanation, think of a car stripped off its steer, dashboard, and wheels. All these smaller parts of the vehicle can be fully customized on their own and easily replaced by other parts of the same kind if needed.

Most teams fall back to the modularization philosophy to split monolith codebases that are too large to handle. It is because as time goes on, the build time grows linearly with the lines of code. Slow build time hurts developers’ productivity in the long run and will result in long waiting times before developers can fully test their apps.

By splitting the features, developers will only build the needed features without having to compile the entire app. Theoretically, this will decrease the compile time, improving developers' productivity.

When Should You Modularize Your App?

One question might arise when considering modularization. How do you know if your app needs to be modularized or not?

The answer to such a question is pretty subjective since it depends on your aims from the start and your current situation. For example, suppose you are building a startup that needs to have as many features delivered as possible within a tight schedule. In that case, modularizing your app from the very beginning will result in either :

1. Premature optimization.
2. A good foundation while the app is still in its infancy.

All of these concerns are valid and you must make decisions with this in mind.

How to Modularize An iOS App

Let’s say your team finally decide it is the best time to perform modularization, and the next question is how to start modularizing your app.
Rest assured, you will learn how you can modularize an iOS project step-by-step.

Consider an illustration below :

Let’s say you have an iOS app with three features: catalog, product detail, and shared. One way to get the app modularized is to split the app into four different modules, each with a library and a bundle. In iOS, a module is called a framework, and this article will use those two words interchangeably.

A library is a collection of resources and the code itself, compiled for one or more architectures, in shorter words, it is all the code you have written. Whereas a bundle in iOS is a directory with subdirectories usually used to store assets, files, and many other specific types of content. The framework itself is a group that contains all the libraries and the resources it needed.

Furthermore, there are two important frameworks that need to be explained :

1. Main

Like its name, the main is your main framework; think of it as your app’s entry point.

2. Shared

Think of this framework as the place of all components used within the entire app, containing everything like the code of UI components, (e.g accordions, bottom sheet, popup, etc. ). The shared framework is the best place to start splitting the app since it does not depend on anything externally.

Moreover, iOS has two types of frameworks, dynamic and static, respectively.
A dynamic framework is a bundle of code loaded into an executable at runtime instead of compile time. It behaves exactly like a dynamic library, except for the difference that a dynamic framework is a dynamic library embedded in a bundle along with optional assets, such as images. UIKit and the Foundation frameworks belong to this category.

On the other hand, a static framework is a bundle containing a static library file. They are statically linked, or copied into the executable binary, and not loaded at runtime.

For the sake of simplicity, this article will only cover extracting a monolith into dynamic frameworks.

You can read more about dynamic and static libraries here.

Static and Dynamic Libraries

Let’s Get Started

Clone the starter project here

GitHub - tokopedia/ios-modular-tutorial: Repository for tutorial code base for our medium blog post

Open the MyAwesomeApp.xcodeproj. Try to run the project to get a clue about the app you are trying to modularize.

As you can see, the app is pretty straightforward, you have a catalog page showing all the product cards and several rows of promotional and inspirational products. Another feature to add is navigating to the respective product detail page upon tapping one of the product cards.

1st Step: Convert App Into Xcworkspace

The first thing you have to do to modularize our app is to convert our app to an xcworkspace

Click file > new > workspace, and set the workspace name as MyAwesome. Then, save the .xcworkspace inside the modular folder.

Another window will appear and you can close the previous window. Click File, Add Files to MyAwesomeApp, and add your previous myawesome.xcodeproj to your workspace.

Your Xcode will then become something like this.

Until now, you have successfully added the xcodeproj inside an xcworkspace. Now, you can start to split the features into their own corresponding frameworks.

Remember that you must start extracting from the least dependable frameworks. By referring to the dependency diagram before, you can see that the least dependable framework is the Shared framework. Start by extracting this framework first.

2nd Step: Creating the Shared Framework

Create a new framework named Shared and add it to our workspace. Refer to the GIF below for the step-by-step.

One thing to note here, remember to choose MyAwesomeApp.xcworkspace the place to add the new framework. On top of that, also make sure the group refers to the MyAwesomeApp.xcworkspace.

Now your workspace will look something like this, with two xcodeproj inside an xcworkspace.

The next step is you have to move the all contents of the folder Shared in the MyAwesomeApp framework ( we will call this the main framework in future usages for simpler and better understanding ) to its own Shared framework’s folder. You can use the help of the finder app in order to do so.
Right-click on the Shared folder in the main framework, choose Show in Finder , we will call this Shared folder the source folder.
Do the same thing for the Shared folder in the Shared framework, and we will call this folder the destination folder.

Move all the folders inside the source folder to the destination folder, and remember to tick copy items if needed when prompted. You can refer to the GIF below for a better understanding.

Next, you have to adjust the module property of the moved asset files. Refer to the UI folder in the Shared framework. Click the custom class section on the right pane of the code editor, click the module text field and type Shared, the framework’s name. Doing so will cause the Inherit module from target checkbox will be automatically unchecked.
This step is to set the resource file’s module explicitly to the Shared framework since you have moved the file there, if you do not, the file will still think it belongs to the main framework, since it inherits from the target, causing unexpected behaviours.

Do the same for the InspirationItemCollectionViewCell.xib and InspirationCollectionViewCell.xib .

The next step is the most essential part of integrating your Shared framework into the main framework.

Navigate to the main framework, locate the Frameworks, Libraries, and Embedded Content section, and choose the Shared framework to be embedded. This step is to tell the main framework that it has the Shared framework as the dependency when running the app.

Try to run the app, but expect the build to fail because of these errors.

3rd Step: Resolving the errors

Let’s solve the errors sequentially, navigate to the CatalogPageNetworkProvider.swift

As you can see, most of the errors are caused by the file not having the definition of NetworkResult , this is because the NetworkResult is originally located in the Shared folder, and because you have moved all of the Shared’s contents to its own frameworks, the code from the main framework does not have access to them again.

The solution for this is you have just to import the Shared framework in the file.

But the error persists even if you have imported the framework, this is still happening because apparently the NetworkResult has an internal access control level, any definition that has internal access control level can only be accessed from its own module, and since you are trying to access the enum from outside the framework, you still cannot locate the definition, which explains the error.

Note that access control level plays a huge role in modularization, you have to determine which code that are exposed and which are not

Change the access control level to looser access controls, such as public or open. Navigate to the shared framework, find the NetworkResult.swift file and add an access control before the enum definition.

The errors in CatalogPageNetworkProvider.swift should be solved now.

Navigate to the ProductDetailPageViewController.swift . The errors are because the file cannot find the definition of the Product struct and the load function since those definitions are already moved into the shared framework, hence the solution is the same as the previous errors, which is
1. Adjust access control levels
2. Import Shared

Import the shared framework at the top of ProductDetailPageViewController and add a public control level in front of the load function definition in UIImageView+Extension.swift . You should have resolved the errors.

Next up, navigate to the CatalogPageViewModel.swift , the same errors are happening in this file, it cannot identify the HashDiffable protocol and the ProductResult .
You can use a similar solution, import the shared framework on the file and add a public access control to the HashDiffable definition and its isEqual function.

Lastly, the CatalogPageViewController , where it cannot identify the Product struct. Simply importing the shared framework in this file will solve the error.

Try building the app and it should have been able to run now.

4th Step: Handling Bundle Issues

One thing that you will notice upon the app launch is your app does not look like it used to be before modularization, you have only presented a white screen and your list of product cards is gone. Why is this happening?

Luckily, looking at the app console, you should see the error’s root. A message : URL NOT FOUND.

If you try to find the error string in the entire app, you will notice that it is only coded in the CatalogPageNetworkProvider.swift . Furthermore, If you look further into the code, the errors are because you can't locate the ProductData.json

This happens because the ProductData.json is already moved to the Shared framework, but how can you access the file using a bundle? 🤔

First, to access a bundle from a framework, you have to know its identifier.

Navigate to the Shared framework, click on the target and inspect the general tab. Turns out you already have a bundle identifier of the shared framework, and you can use this in order to access the files within the framework.

Navigate back to the CatalogPageNetworkProvider.swift , change the guard validation in both of the fetchProductand fetchInspiration

Instead of using Bundle.main , use your shared framework’s bundle by initializing it using the bundle ID. Remember to update the bundle ID to your framework’s bundle ID as your organization name most likely will be different.

Try to run again the app. This time, instead of getting the error message URL NOT FOUND, your app will crash.

Suppose you try to debug further from the stack trace, you will see that the crash is caused by CatalogPageViewController calling the cellForIndexPath function, which triggers the collectionView to load the ProductCollectionViewCell, but it was unable to do so, causing the crash.

On the brighter side, this crash proves you have successfully loaded the JSON files since it is happening on the CatalogPageViewController ‘s function that is triggered upon receiving data.

Your view controller causes the error since it cannot identify the ProductCollectionViewCell . The solution is to specify the bundle where this view controller loads the NIB.

Locate the viewDidLoad function inside the CatalogPageViewController , and change the code to become :

Try running the app, and you will get another crash caused by the same issue, the InspirationCollectionViewCell could not load its NIB. Locate its awakeFromNIB function and change it into this

Remember to adjust the identifier to your own bundle ID

After changing the code, try to rebuild the app and it should run successfully!

You have successfully extracted your shared module, leaving only two more features before you finish modularizing your app. We will continue extracting the catalog feature into its framework next.

You can access the code materials until this point here.

GitHub - tokopedia/ios-modular-tutorial at feature/extract_shared

Now let’s continue to the second part here

Getting Started with iOS App Modularization — An Introduction was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the second part of the modularization tutorial. Refer to the list below for the complete parts :
Part 1: Introduction
Part 2: Extracting Catalog
Part 3: Final Extraction and Micro App

In this article, you will continue to extract the catalog feature.

Refer to the link below for the starting project template, or you can continue with your existing project if you come from the first part of the article.

GitHub - tokopedia/ios-modular-tutorial at feature/extract_shared

1st Step : Creating The Catalog Framework

We will start this article by creating a new framework named Catalog. Choose File > New Project, choose Framework and remember to add it to your xcworkspace . Refer to the GIF below for a better understanding.

The next step is to move all the Catalog’s folder contents in the main framework into its framework now, refer to another GIF below for clarity.

The first thing to do upon framework creation is to add the framework’s dependency. In this case, you have to add the Shared framework as a dependency since Catalog relies on it heavily for the models and other assets. Click on the Catalog project, find the targets tab and scroll until the Framework, Libraries, and Embededded Content section. Click the plus icon and choose the Shared framework.

The next step is to set asset files in the new Catalog framework to point to its module, click the CatalogPageViewController.xib . Locate the File Owner option on the Placeholders section, then choose the custom class tab, and type Catalog at the Module text field, and the Inherit module from target will be automatically unchecked.

Try to run the app but expect the build to be failed because of the ACL and import-related issues.
One place that errored first is SceneDelegate.swift located in the main framework. Simply importing Catalog will resolve the issue, but before that, you have to add the Catalog framework as the main framework’s dependency.

Try to rebuild the app again, but the error will persist. This is caused by ACL since the CatalogPageViewController ‘s access control is internal. Add a public access control before the class definition and make all its exposed functions public, the resulting code will be like this.

Even after all these ACL modifications, our catalog feature still will not build because the file cannot identify the ProductDetailPageViewController .
One might argue that the solution to this is just to
1. Extract the product detail into its framework
2. Simply import the product detail module at our catalog framework

This approach will work but it will cause a dependency issue if later, the product detail page needs to use classes/models from the Catalog framework, you will end up having a circular dependency in your app.

2nd Step : Designing The App Router

A better solution to the previous one mentioned before is you can create an abstraction that acts as a route handler in the Shared framework, then when any features that want to navigate to another module can just use the abstraction without having to be tightly coupled to the destination framework.

Navigate to the Helper folder inside the Shared framework and add a new file named Router.swift

This will be a blueprint for our navigation capability. Now, head back to the CatalogPageViewController and replace the didSelectItemAt indexPath function extension to become like this

Try to run the app. Now, it will be successfully built but crashed as soon as it runs. The crash is caused by the CatalogPageViewController since it cannot find its XIB.
Navigate to CatalogPageViewController , locate its init function and change them.

Bundle(for: Self.self) is used to refer the bundle within the module itself, you can use this directly instead of Bundle(identifier:) since the CatalogPageViewController.xib lives inside its own framework.

Try to rebuild the app again, and this time the app will successfully run.

But one capability is missing from our app, if you try to click one of the product cards, it will do nothing instead of navigating to the respective product detail page, why is this happening?

Remember atCatalogPageViewController , you have changed the didSelectItemAt indexPath function definition to use Router instead of manually navigating. The navigation will not work yet because you have not created the definition for the route function, which explains the bug.

Navigate SceneDelegate.swift inside the main framework. Add the route definition at the very end of the scene function :

By doing this, the app will be able to perform navigation from any modules without ending up having a circular dependency or becoming tightly coupled with unnecessary modules.
Try to run the app, and the navigation will now work smoothly.

Up until this point, you have successfully extracted the Shared and Catalog frameworks, leaving only the Product Detail Page to be extracted.
In the next part, you will continue to extract the Product Detail Page into its framework and learn how to build one single framework instead of the entire app.

Now, let’s continue to the final part here

Getting Started with iOS App Modularization — Extracting Catalog was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the final part of the modularization tutorial. Refer to the list below for the complete parts :
Part 1: Introduction
Part 2: Extracting Catalog
Part 3: Final Extraction and Micro App

Refer to the link below for the starting project template, or you can continue with your existing project if you come from the second part of the article.

GitHub - tokopedia/ios-modular-tutorial at feature/extract_catalog

1st Step : Creating the Product Detail Framework

Let’s continue this series to extract the Product Detail feature into its own framework.
Create a framework and add it to the existing .xcworkspace

Next up, move all the contents inside the ProductDetailPage folder in the main framework to its own framework, then our project will look at this.

Always remember to add the framework’s dependency first before adjusting the framework’s code.

Add the Shared framework as the ProductDetail ‘s dependency.

Next, don’t forget to change the ProductDetailPageViewController.xib ‘s module name to ProductDetail .

After that, try to run the app but always expect it to be failed since you have not adjusted anything yet. If you try to inspect the error, it is caused by the file SceneDelegate.swift since it cannot find the definition for ProductDetailPageViewController .

The solution is to add ProductDetail as the main framework’s dependency, by doing this, the main framework will have access to the ProductDetail framework’s APIs. Refer to the previous GIF to understand how to add a dependency to a framework.

Next, try to import ProductDetail on the SceneDelegate.swift , the file will become like this.

The error still has not been resolved due to ACL issues, navigate to ProductDetailPageViewController and modify the code to the following :

Try building the app, it will run smoothly with all features working as expected!

What’s more, you have successfully modularized the monolith app into several modules! 🎊🎊

2nd Step: The Micro App Concept

Now you already know how to split an app into several smaller features and achieve modularization, but what if you want to achieve building only necessary features?
Let’s say you have to implement some enhancements on the product detail page, theoretically, you only have to build the respective framework without having to compile the entire app. How can you achieve this?

Here’s where micro apps jump in, what you have to do is just to create a new app target, inside the framework you are working with. This xcodeproj will act like your playground, and developers needed only to compile this feature on development, hence the name micro app.

Click on the Product Detail framework, find the Targets tab and click the + button underneath the tab. After the choosing window show up, search for app and name it ProductDetailSample.

If your newly generated project files end up being red and can’t be opened like the GIF, quit and reopening the XCode will do the trick.

Next, add the Product Detailframework as the Product Detail Sample ‘s dependency. This is needed since the sample will only act as a playground for our code in Product Detail , hence you need to import them before you’re able to use them.

Find the Frameworks, Libraries, and Embedded Content section of the sample’s general tab and add the Product Detail framework.

After that, try to run the ProductDetailSample target

You will then be presented with an empty screen, which comes from ViewController.swift in the ProductDetailSample folder.

Try changing the background colour and check whether the screen changes accordingly. :)

That’s it. You can use the ViewController to embed the ProductDetailPageViewController or do whatever you want to do with the feature.

In this tutorial, however, we will try to simply present the ProductDetailPageViewController instead of using the ViewController , to achieve the capability to build only what you need.

Remove the ViewController.swift and the Main.storyboard .
Navigate to ProductDetailSample ‘s general tab and remove the Main from the Main Interface field, and lastly, locate the Info.plist file, expand all the keys from the dictionary until you find a key titledStoryboard Name, and then delete the entry.

Finally, navigate to SceneDelegate.swift , change the code inside the scene function to become :

Try to run the app again, then it will show only the product detail page.

Now, if you want to enhance the product detail page, for example by adding an image carousel, product description, or anything else, you can code them and directly build this ProductDetailSample to speed up your development.

Congratulations! 🎉🎉
You have modularized the app and even provided a minor app in one of your features!

you can also go the extra mile to create a minor app for Catalog and Shared framework as well. :)

The final code of the project can be found at :

GitHub - tokopedia/ios-modular-tutorial at feature/micro_app

Where to Go From Here

In this article, you have learnt the definition and some motivations behind modularization, furthermore you have modularized an app yourself and even made a sample app for one of the frameworks, this is just the first step of the modularization journey, there are still so many use cases out there that require us to get help from libraries, or tweak our approach to modularizing. We also have this one story of adopting Bazel in our app to further improve our build time, you can check it here.

How Tokopedia Achieved 1000% Faster iOS Build Time

Thanks to Digital Khrisna, Bondan Eko Prasetyo, Ferico Samuel, and Andrey Yoshua for helping to review this series.

Getting Started with iOS App Modularization — Final Extraction and Micro App was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background

In February 2022, Tokopedia introduced an affiliate program to all Tokopedia users and the public. To become an affiliate, users need to register their social media accounts by providing the social media URLs. As each social media URL has a different domain and URL path, the application needs to validate the URL according to the social media.

We agree that regex is one of the easiest ways to solve many parsing problems However, this case is complex due to several rules for each social media and will be challenging to customize in the future. Some social media have more than one URL path for the user page. For example, Youtube has several paths as follows:

https://www.youtube.com/user/example
https://www.youtube.com/c/coldplay
https://www.youtube.com/channel/UC9zY...

One way to solve this is by creating a reusable validator that allows validating specific social media URLs in the simplest way. This article will use and introduce SuperValidator as our swift validator library to solve this problem.

SuperValidator

SuperValidator is an option-based validator and an open-source library managed by Tokopedia iOS Engineers. The validator provides options to customize the validation according to your needs. Let’s go through this article to know more about this!

URL Validator

If we look at the URL anatomy, we can create separate validation for each of the URL’s components.

SuperValidator has split up the components as parameters such as the following:

protocols: [String] = ["https", "http", "ftp"],
requireProtocol: Bool = false,
requireValidProtocol: Bool = true,
paths: [String] = [],
allowQueryComponents: Bool = true,
domainWhitelist: [String] = [],
domainBlacklist: [String] = [],
fqdn: FQDN = .init() // Fully Qualified Domain Name

protocols

An array of strings for valid protocols, such as http or https. If we store https only then it will only allow https.

requireProtocol

If set to true, then the URL must have a protocol.

requireValidProtocol

If set to true, then the URL protocol must be the same as one of the valid protocols listed in the protocols options.

paths

An array of strings for URL paths /shop/{shopID}.
There are two types, which include fixed and value.

/shop/{shopID}

The path will be separated by /,
shop is a fixed type, which means the URL path inputted must be exactly the same.
{shopID} is a value type, curly braces {} indicate the content inside is the value type.

allowQueryComponents

If set to false, the query component and anchor are not allowed.

domainWhitelist

An array of strings for the whitelist domain. It can also use regex.

domainBlacklist

An array of strings for the blacklist domain. It can also use regex.

FQDN

FQDN stands for Fully Qualified Domain Name. It will check if the domain is good to go.

Basic Usage

We can just call isURL function to return a boolean that indicates a valid URL.

If you want to customize the validator, you can set the options parameters covered in the previous section.

You can also use validateURL function to return success or error result. You can use the error type provided by SuperValidator to generate an error message.

Use Cases

Let’s break down the users’ social media URLs. They have different domains and paths, and some of them have multiple valid paths. Cases like this can also happen in other URLs. We can create an enum to store options for every social media.

Then you can use isURL or validateURL function from SuperValidator.

SuperValidator.shared.isURL(
   "https://instagram.com/myuser", 
   options: SocialMediaURL.instagram.options
)

Fortunately, you don’t need to create a new function for social media URL validation, because SuperValidator already covers this. You can call isURL or validateURL with socialMedia parameters.

You can use validateURL to get the error reasons and give it to the user.

For more test cases you can see this link.

Other Validators

Besides URL, there are several validators that SuperValidator already covers:

FQDN

Help validate fully qualified domain names. (Wikipedia)

Email

Help validate email with options such as length limit of the local part, whitelist, and blacklist domain.

Phone number

Help validate phone numbers with International, North American, and Extensible Provisioning Protocol formats, validate country code or the national prefix.

Future Plans

• More validators
• Bind with UITextField
• Bind with RxSwift

Contribute🚨

Calling contributors!! SuperValidator is open-source, and we’re excited to hear and learn from you. Your experiences will benefit others who use SuperValidator. Let’s make it even better!

https://github.com/tokopedia/ios-supervalidator

Credits

Thanks to Christopher Teddy for helping with the development. Kudos to Superman squad and all developers who helped make this happen.

As always, we have an opening at Tokopedia.
We are an Indonesian technology company with a mission to democratize commerce through technology, and our vision is to build an ecosystem where everyone can start and discover anything.
Find your Dream Job with us in Tokopedia!
https://www.tokopedia.com/careers/

Validating Social Media URLs in Swift was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Android 10 has included the Dark Theme to its devices a while ago, and its adopters are rising quickly over time. Dark Mode has many benefits, such as reduced battery usage, improved visibility, reduced eye strain, etc.

This article will share and explain Tokopedia team’s journey to convert our Android App to support Dark Mode. It will be bundled with some technical implementation as well.

Opening

We know that users have a massive demand for the Dark Mode feature. Well, the wait is over!!, Tokopedia has officially published its support for Dark Mode in the Beta phase. We’ve spent a lot of time and effort converting our Tokopedia App, which consists of hundreds of modules to accommodate dark mode support. There were lots of problems that we encountered in our quest. I will share all of them in this article and the solution.

Prerequisite

• Basic Knowledge of Dark Mode

It would be best to have basic knowledge about how to make an application that supports dark mode. Therefore, this article will only share major hurdles in converting our application. For the official documentation about dark themes, please refer to this link.

• Color Helper Library

We also have an internal library for storing and managing color. This library is only a module that contains colors.xml with values-night resources. We recommend using this approach. So the color in your application is centralized in one place, and less effort is needed to do any maintenance.

• Force App to Light Mode

While we were in the development phase, we applied the code that forced the app to light mode only. Force to light mode will prevent the app from using dark mode in production while developers are still in the development phase.

Imagine if we have ten pages and only two of them already supports the dark mode and the device theme of the users is in the dark theme, then the users will have a broken experience with dark and light mode mixed up in the app. If we don’t force the app to light mode, the app will follow the device system theme.

Use the below code to force your app to light mode (place it in your application class).

AppCompatDelegate.setDefaultNightMode(
AppCompatDelegate.MODE_NIGHT_NO);

Finally, after all of the pages support dark mode. Then we can start using the default system theme and begin to remove the code that we created to force to light mode.

How do we convert hundreds of modules to Dark Mode with as little effort as possible?

Since we have hundreds of modules owned by several teams, the manual approach is not feasible and requires too much effort. The manual approach means we have to change the colors one by one, create night res folders for colors, or even check and change the theme. Those could take forever to convert and the chance of making a mistake is high.

Our solution is to create an automation script to convert each particular module to dark mode.

Automation Script

The script is created in python. Basically, it will traverse to the selected module and scan all of the files that generally contain colors such as .kt, .java, .xml. The script will ignore files with types aside from the ones mentioned before.

If the script finds the type of file, for example, ProductFragment.kt, it will open the file and read lines one by one to search for any occurrence of color.

for line in fileinput.input({path_to_product_fragment}, inplace=1):
    # find if line contains color with regex

Regex is used to get each color in each particular step because each step has different color target to be replaced. This regex for example :

'com\.[\w_]+\.[\w_]+\.R\.color\.[\w_]+|R\.color\.\w+|\.color\.\w+|@color/\w+|android.R.color.\w+|@android:color/\w+

After a color is found in a line. The script will check whether the color is available on our dark mode library. If it’s available, then the color needs to be converted to the correct color. Then the script will get the certain color that matches and replace it. Example of library :

Old Color, Dark Mode Color

R.color.Blue_B100,R.color.Unify_B100
@color/Blue_B100,@color/Unify_B100
R.color.Blue_B200,R.color.Unify_B200
@color/Blue_B200,@color/Unify_B200
R.color.Blue_B300,R.color.Unify_B300
@color/Blue_B300,@color/Unify_B300
R.color.Blue_B400,R.color.Unify_B400

For instance, in this line of xml file, we get the color as @color/Blue_B100 and the script will change it based on the color library.

android:textColor="@color/Blue_B100" <- before converted

android:textColor="@color/Unify_B100" <- after converted

How it works?

The high level flow can be presented in this flow chart:

In Tokopedia we’re doing three steps to completely convert all colors to dark mode support :

1. Convert all existing colors from the library that are not supported as the dark mode.
   As we mentioned before, We have an internal library that centralized our color. Unfortunately, the previous colors that we used are not supported as a dark mode because those do not have night value. So we run the script as we describe above.
2. Convert all custom colors into our internal colors.
   In Tokopedia we have hundreds of modules, each module has its custom colors (usually located at res -> values -> colors.xml). It’s another problem if we want to convert it to dark mode support. So the script will do :
   - Get all of the custom colors in our source code then place it in one place.
   - Get the hex color of each and find the nearest similar color to our color library. The nearest color script will return the correct closest color result.
   - Change the color to the result and change the color in class.
3. Convert all hard coded colors into our internal colors.
   In this case, we will cater the hard coded hex color in our class, for example, android:background=”#123456". The approach is fairly similar to the first one :
   1. Loop all of the android classes
   2. Find hex color by using this regex r’#([0–9a-fA-F]{3}|[0–9a-fA-F]{6}|[0–9a-fA-F]{8})\b’
   3. Get the hex color and pass into nearest color picker script. The script will return our correct library color. Example : #123213 will turn into @color/Unify_Black
   4. Change the color in class to the result correct color.

Other Tool

• Nearest Color Script. You can create your own algorithm to find the closest color to color, by giving the list of colors. By using KNN Algorithm, python already has a library to find the nearest neighbor by using scipy, example click here.
  Turn your library color and your input color to RGB . Submit to the scipy and get the nearest RGB in the list.
• Theme Changer Script. This script would change our existing theme to theme that recommended by google to dark mode support (Theme.AppCompat.DayNight).
• Dark Mode Coverage Dashboard
  To maintain dark mode coverage in our app and ensure all of the new codes are ready for dark mode, We’ve created a dashboard to track whether the modules are ready or not. This tool is also linked to Pull Request Checker in our Jenkins.
  If there is new pull request that detected with less than 100% coverage, then the PR will be block and developer’s feature can’t be merge to our master branch.

Best Practice

• Prevent creating theme styles with the same name; the exact name can cause dark mode not to work.
• If using CardView, use app:cardBackgroundColor instead of android:background to apply dark mode color.
• Use a custom color that doesn’t have any night-valuesto the color you don’t want to change in dark mode.
• Beware of strings.xml; always check this class because the color resource is possible located in strings.xml file.
• Avoid color reference (@color/blue_night) in .xml that contains vector; just hardcoded the color with hex (#003c94) to prevent crash in lower os.
• Create a unique identifier to prevent the script to keep your intended custom color. So whenever the script is re-run, your color will not be converted. For example, I’m using _dms_ (dark mode support) as an identifier :
  <color name=”product_detail_dms_tradein_blue”>#32afff</color>
• If your app contains webview, we also need to support it to dark mode. First thing first, the web itself should support dark mode. Then to tell the web whether we need to use dark mode or not is, the app needs to send some additional flag to webview from the header. Google also has an approach for this check here.
• In case to keep the dark mode experience always good and on track, we need some prevention and automation to block the feature that not using dark mode color. Usually, we block the Github pull request auto merge and wait till developer fix with the correct color.

Result

These are some of the results of dark mode support in our Tokopedia customer app.

That’s pretty much how we convert our app to dark mode. Automation helps us a lot and reduces human errors. But we still have to continuously improve our dark mode experience because some of the cases need to be manually converted and are not being catered in the script.

As always, we have an opening at Tokopedia. We are an Indonesian technology company with a mission to democratize commerce through technology and help everyone achieve more. Find your Dream Job with us in Tokopedia!
https://www.tokopedia.com/careers/

Tokopedia Android App Journey to Support Dark Mode was originally published in Tokopedia Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.