Over the course of two years, our Reco AI has assisted our sellers in generating millions of additional transactions that would not be possible otherwise. That said, when talking about the positive impact for the AI, our proudest metric is that it has been helping hundreds of thousands of Indonesians in earning an additional 1 trillion IDR (US$70 million) of income. If there is one thing we love to do at Bukalapak, it is to improve the lives of Indonesians with our products.

The Origin Story

Every hero requires an origin story, and our AI is no exception. The origin story for our AI starts on the day I moved back to Indonesia. Having just dropped out of my Ph.D. in computer vision a couple of years before, I was eager to start a culture of AI research within Bukalapak and see what kind of AI we can build from our mountains of data.

Wait, did I say “mountains of data”?

It turns out, one of the core principle at Bukalapak since day one is “store everything, we’ll be able to build something useful with those data someday.” We have more than a petabyte of data on our data warehouse by the time I joined, ranging from transactions to product descriptions to user-generated traffic. The last one turns out to be our gold mine.

After talking with several engineers, data scientists, and product managers, we formulate a strategy to quickly experiment and validate the value of using our tons of traffic data to build an AI that can learn from the traffic and provide product recommendations to potential buyers.

We scoped in a small project to build a working prototype within a couple of months to one of the product squad, a small team of five engineers and two data scientists. The short cycle is intentional since we do not want to spend more efforts than that if the endeavor turns out to be fruitless.

Designing and Building the AI

What we built back then was quite rudimentary compared to the complex model we have right now. We started with one of the most common types of recommendation AIs: collaborative filtering.

The idea is deceptively simple: if a user viewed product X and then viewed product Y, then another user viewing product X might also be interested with product Y. We are effectively using the traffic data to form associations between our millions of products. Multiply this approach with billions of “viewed together” events, and what started as a simple thought exercise becomes a powerful training dataset for the AI that we want to build.

Moreover, this approach is highly parallel and highly scalable and thus becomes the perfect use case for MapReduce computations. We use Spark MapReduce running on a 32-core machine for our initial prototype, crunching 4 GB of compressed traffic snapshot in several hours.

One crucial context: prior to the AI, we use Elastic’s “more like this” feature to give our visitors a product-to-product recommendation. We took a couple of samples and visually compare the output from Elastic and our AI:

The samples show good quality recommendations. Instead of showing similar products, we present our visitors with various product alternatives for their shopping journey. So far so good, the next step is to validate the AI prototype in production.

Validating the AI

We decided to put the recommendation at the bottom slot of every product detail page, as shown below.

We use an approach called A/B testing to validate whether the AI-based recommendation that we put on that slot gives more impact than the Elastic-based similar products. The idea is to have both the old and new versions running at the same time, but we randomly choose which version to present when a user visits our site. We then track the user’s journey within our site, taking careful note of the relevant metrics. (e.g. whether the user made a purchase or not)

In our case, when a user visit a product page, the recommendations at the bottom of the page are being populated from either the reco AI or from Elastic. We then use our internally built A/B testing framework to track multiple metrics related with purchases to gather real-world objective data regarding the performance of the AI.

We ran the A/B test for two weeks before analyzing the outcome of the experiment. The result was astounding. We observed a significant increase in visitor purchase when they were being presented with recommendations coming from the new AI.

Convinced with the positive impact of the AI, we decided to spend another month building automated data pipelines and process to incrementally train the AI and build fresh recommendations every day for our visitors. At the end of the first month alone, we saw that the AI had generated an additional US$1 million of revenue for our sellers.

Now and the Future

Fast forward two years later, we have built a dedicated team that focus on increasing the quality of our recommendation AI. We have expanded the computing cluster dedicated to build our recommendations to a thousand-core cluster and we are now able to generate the reco-set in a matter of minutes instead of hours, enabling us to do more rapid experimentation with our models.

We have made several iterative improvements to our AI models, always validating the idea with further A/B tests, and continue to experiment with more ideas as we go on. Several of the “big wins” in the past include expanding our event models and adding personalization context to our recommendations. Thanks to these improvements, our reco AI is now bringing home 75 billion IDR (US$5 million) per month in additional income for sellers all over Indonesia.

For future improvements, we are currently experimenting with modeling the recommendations in a dense multi-million node knowledge graph, and see whether we can coax more insightful recommendations for our visitors based on the connections within the graph.

Expanding to More AI Products

Reco AI was the first AI product that we built, but since that time two years ago, having seen the kind of potential impact an AI could do to help our millions of users, we want to establish an AI division to expand our focus and gain expertise at other areas where we could use AIs in order to assist our visitors, sellers, and operational teams.

Building up an AI division is a lengthy subject that warrants a separate article, but a great deal of inspiration for the way we organize our AI efforts came from the following talk with Dani Yogatama, who shared with us about his experience being part of large research organizations at Google DeepMind and Carnegie Mellon University.

What we have established at Bukalapak so far consist of two streams of research efforts. The first stream is for building regression or neural-net based AI models from millions of rows of structured and labeled data. We use an AutoML platform to rapidly build basic models to test out the feasibility of using AI to approach the problem. This platform enables our data scientists to quickly iterate and validate their models, typically within a couple of weeks instead of months.

The second stream is for building more complex AI models. We establish several AI R&D teams to build and improve various AI services ranging from using computer vision to recognize objects/watermarks within product images, to using natural language processing to understand search queries better and assist our users in finding the right product for their shopping journey.

Recommendations for Building AIs for Your Industry

Sorry for the pun. We have been asked about this numerous times by others within the industry ever since word got around that we are building a high impact AI to help our visitors and sellers. Here are three recommendations from us if you want to build AIs for your particular industry/business:

1. Collect, store, and verify that you have the data. Starting the project with an already organized petabyte-scale data warehouse helps saves months within the project. Since otherwise, we would need to spend several months just to gather and clean up the dataset necessary to train our AI.
2. Initiate, validate, iterate. Don’t get stuck on trying to build the perfect AI or trying to strictly follow some research paper within the field. Sometimes starting with a simple model will be enough as long as you can quickly validate the model against real-world traffic or data. Then you can iterate and build progressively more complex and better quality models.
3. A/B test, A/B test, A/B test. Oh and did we say you should A/B test your AI? Running manual analysis using a small number of samples are fine for quick and early validations, but don’t forget to validate your model using real-world data and A/B tests, doing so will help you avoid possible selection bias and gather objective data to guide your AI development.

Obligatory “We Are Hiring!” Closing Remark

We have built one of the best tech workplaces in Indonesia, where we combine strong learning and research culture, mutual respect, work-life balance, and the capacity to improve and bring positive impact to millions of Indonesians.

We have more than 850 tech talents as of September 2018, working together to build exciting tech products to help improve the livelihood of Indonesians, and we are hiring many more to join our growing tech family. Sounds interesting? Check out our career site, we have tons of interesting roles!

How We Build a Multimillion Dollar AI for Indonesia was originally published in Inside Bukalapak on Medium, where people are continuing the conversation by highlighting and responding to this story.

About a year ago, we found an edge case issue in our API that caused a tiny percentage of our iOS users to experience disruptions to their checkout flow.

Since data-driven is part of our core culture, we ran a quick impact analysis based on the tens of billions of data points that we collected daily and calculated that the issue is causing us to lose $8000 worth of transactions every single day.

Sounds scary, right? To top it off, we found the issue on a Friday afternoon, right when the team is about to log off for their weekend.

So what did we do?

If you guessed that the team spent their weekend fixing the issue, you guessed wrong! After calculating the impact, we simply called it a day and asked the team to enjoy their weekend and fix the issue in Monday instead.

So the team did exactly that, came back Monday feeling fresh and reinvigorated, jumped to the issue, quickly found the root cause, and deployed a fix on the same afternoon.

But why would we choose to ignore the impact?

Well, it’s not really ignoring the impact. After all, we all can agree that $8000/day is not an insignificant amount of money. We can easily imagine scenarios where business leads screamed out of their lungs and pushed the team to fix the issue ASAP.

As with most other things in the world, we need to take a step back and obtain more perspective on the scale of things. Yes, the impact is not insignificant, but when we’re handling millions of dollars worth of e-commerce transactions per day at Bukalapak, in relative percentage $8000 is quite small.

Let’s take another perspective. Over here the majority of our 650+ tech talents are focusing on experiments and incremental improvements to our overall product. Teams often celebrate Big Wins, improvements that our A/B Test experiments have shown to deliver at least +1% growth to our global KPI. At our scale, those improvements could translate to tens of millions of dollars worth of additional annual transactions.

What this means within the context of having an issue with $8000/day impact is, why should we bother disrupting the team and pushing them to fix a 0.1% loss through weekend overtimes if it meant that they have less energy to finish a potential 1% growth improvement? Everyone, including our business leads, understands this very well and triaging production issues based on objective data becomes a powerful prioritization tool for us.

Taking yet another, perhaps more powerful perspective (and my personal favorite), this core value means that we respect the weekend of our teams more than $8000/day worth of transactions. How’s that for a work-life balance company? :)

P.S. We’re hiring! ;)

We have built one of the best tech workplaces in Indonesia, where we combine strong learning culture, mutual respect, and amazing positive impact on millions of Indonesians. This year alone we’re on track to hire more than 300 new tech talents to help accelerate our growth even further. Sounds interesting? Check out our career site, we have tons of interesting roles!

P.P.S. All the pics (except the kid’s facepalm) came from our Work at Bukalapak video, check it out if you want to see how it feels like joining our tech family!

How We Deal with Friday Afternoon Bugs was originally published in Inside Bukalapak on Medium, where people are continuing the conversation by highlighting and responding to this story.