The Product Manager role has become a popular career choice among fresh graduates and professionals in the digital industry. As startup companies rise in the past decade, the role of a digital product manager has gotten more attention than before.

The figure above implies that working on digital-based products often requires intense coordination between the Product, UX/UI, Tech, and Business/Marketing team.

An entry-level product manager (Associate/Senior Associate) tends to focus on execution and spends more time with the tech team. This interaction involves discussion around the product architecture, functionality, behavior, and presentation of the product/feature. Therefore, as a PM, you will hear and use some technical terms, whether you like it or not.

As a PM, I am personally privileged by my educational background and prior experience as an IT technical consultant. However, many people who do not possess this technical background want to pursue a role in Product Management. (You can read my journey into my PM role in my previous post here https://medium.com/@frederycolie/journey-of-a-co-incident-pm-tech-consultant-banking-end-up-at-e-commerce-giant-46bc47ea3d73.

The question that I often receive from peers, practitioners, and college students is: “Does coming from a non-technical background hurt a person’s chance to succeed in a product management role?” By saying non-technical background, I am referring to a background with no involvement with tech stack and coding.

A product manager is a tech-literate, not a tech fluent

Melissa Perry wrote in “Escaping the Build Trap”: “A Product Manager is a tech-literate, not a tech fluent.”

I find that articulation robust and helpful in answering our main question.

If I stop for a moment and imagine my day-to-day as a product manager, I can see where Perri is coming from.

As a product manager who works hand-in-hand with software engineers, technical discussions are inevitable. Remember, to succeed, a product manager has to effectively communicate the product vision to the team — and engineers are a part of this team. Someone with a tech background who possesses the ability to communicate fluently with software engineers would have an advantage. He can grasp the technical complexities of a product and foresee technical challenges and blockers.

Here are some of the perks of being a tech-literate product manager:

1. Asking relevant questions to the engineering team
   Sometimes you find out that your features/product is not working properly due to bugs or errors. Having decent knowledge about the stack and code enables the product manager to address a relevant question to the engineer. Sometimes you can even participate in brainstorming the technical solution when you are good at technical.
2. Writing a proper technical specification
   Every product manager has a set (usually short) timeline to create and get the product out the door. When you are familiar with technical stuff, you can optimize development time by writing a proper technical specification document. Having well-written documentation, enable the engineer to grasp your idea and craft it for you without consuming too many alignments session.
3. Having higher confidence
   A great product manager is a good storyteller. You are not only looking for a VP of product to buy in your vision but also necessary to get the engineer to buy-in. Displaying a high grasp of the technical elements of the product to your engineer can boost your confidence level while you are trying to convince them. Furthermore, you can provide the rough timeline estimation of the product/feature delivery or gauge technical capability for some business requests that come across during a meeting by the business team or other stakeholders, making you a helpful counterpart to the other stakeholders.

In a nutshell, having an understanding of technical things like UX wiring, design, software development, and data manipulation will help a product manager to communicate, problem-solve, and coordinate with the team. You don’t want to squander your software engineer’s prestigious time with a meaningless and going-nowhere of back and forth discussion just because you lack an understanding of the common technical stuff. Again, you don’t have to be a master of writing code or designing web, but you ought to be understanding enough to let your team grasp what exactly you want to build or solve.

What are the common technical things you need to know as a digital Product Manager?

Now, if you want to devote some of your time learning and understanding pertinent technical things, you can start by discussing with your engineer or engineer lead, subscribing to some online learning program or Bootcamp, or by reading some helpful books. To help you onboard on this learning journey here is a few suggested topics to start you off:

1. Product Architecture Learning about product architecture will help you profoundly understand each layer of your application and how each part integrates with each other. For instance, if you are a product manager of e-commerce apps, and you want to build an app where users can see and buy the product. Your application will comprise of the following layers:
   
   Front-End or the presentation layer for users to interact with the apps. It shows the product catalog, text, image, button, field, animation, etc. on the user screen.
   Back-End or the logic layer or the “input-process-output” layer. Here is where you put the logic, instruction, or behavior. For example, when users buy one book in your apps, the Back End layer will run the instruction to deduct the quantity of that book by one, and send the remaining amount in the inventory to the front-end layer to show it on the user screen.
   Data Layer: it is where you store any related data of your application — for instance, the user data, seller data, transaction data, inventory data, etc.
   
   There are some other approaches In learning product architecture in the market (e.g., client-server architecture), so go talk to your engineer lead for a proper start.
   
   Understanding your product architecture benefits you in designing your product functionality and writing your technical requirements. It also helps tremendously when figuring out bugs or errors.
2. System Architecture
   Moreover, you can also learn about System Architecture. It is the broader and high-level architecture level compared to product architecture. There are two popular system architectures that internet companies adopt nowadays: (1) Monolithic architecture and (2) Microservices architecture, Understanding both will help you in expanding and enhancing your product especially when you are in a more prominent company.
3. API
   API or Application Programming Interface, you could find plenty of resources that lecture about it (e.g. Understand API). My definition is that API is the library of the function and behavior of your product or feature. API is not merely being consumed by a single application, but also by other systems/apps internally or externally. I highly recommend you have a deep understanding of API and creating API contracts, as you are undoubtedly going to deal with API in the product development phase.
4. Performance data measurement and manipulation
   I bet every product manager needs to continuously keep an eye on their product metrics for further decision making. Providing metrics data in big companies is commonly served by data analysts however, in a small startup with a lean team, sometimes it takes product managers to do it themselves. Hence, data manipulation skills help product managers in providing data, as well as help to pick the right dataset to be gauged as the north star metrics. You can learn SQL Query language, a ubiquitous tool for manipulating raw data into a more readable and useful form. A great product manager won’t jump to a conclusion without lighting from data, so possessing data manipulation skills can avoid product managers to make groundless decisions towards the product.

On top of that, during your time exploring those skills above, you might be exposed to other technical things around, so don’t stop there, instead explore as many as you can, because the more you know, the more comfortable for you to communicate with your team.

See you in the next post!

Should PMs Possess Technical Skills? was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

On September 30, we hosted a Behind a Great Product event discussing Machine Learning (ML) and Artificial Intelligence (AI) in a Marketplace Ecosystem. Jose Curto, Professor at IE University Spain, opened our session by discussing the use case of Machine Learning in the e-commerce context. Leonardo Saerang, our PM Lead from Tokopedia, goes into more detail on the product development process at Tokopedia that we follow and explains how Tokopedia brings a ML product to live.

Algorithms are mechanisms for us to accelerate our biases to help users achieve their desired outcome faster. Machine learning is the study of computer algorithms that improve automatically through experience and is a subset of Artificial Intelligence.

Machine Learning is quite commonly used in products, so it’s great for us to learn what Machine Learning is, the applications of Machine Learning and how we develop machine learning models for our use cases.

Session 1: Machine Learning for a Marketplace Ecosystem by Professor Jose Curto

There are 4 common use cases of machine learning:
1. Assessing customer churn

2. Dynamic forecasting

3. Customer segmentation

4. Recommendation generation

Recommendations help companies to: (1) Initiate user engagement (2) Increase user engagement (3) Promote the discovery of new content. Jose focuses the discussion on recommendation generation, which allows companies to help customers discover better products faster.

Recommendation systems help companies to initiate user engagement, increase user engagement and promote the discovery of new content.

Building recommendation systems

We can first differentiate Recommendation Systems on whether they are personalized or not personalized. If information about the user is available, we can provide personalized recommendations where different users receive different suggestions. If information about the user is not available, we provide non-personalized, generic recommendations that are typically rule-based. Examples of non-personalized recommendations are popular products, trendings articles.

In practice, companies start by building non-personalized recommendations to initiate their recommendation system when user information is not available, while building the foundations for personalized recommendation systems.

There are 3 major types of personalized recommendation systems:

1. Content-based filtering

This is based on the item-content. Recommended items are described by keywords, while user mode is structured based on preferred keywords (or classifier). The pro here is that we do not need user data to build this. Another major pro is that this model has the ability to recommend to users with unique tastes, and recommend new and unpopular items, which are long-tail in nature.

Content-based filtering is also more “explainable” — you can understand the attributes behind the algorithm and make sure that the algorithm is aligned to the logic you want to promote and avoid information bubbles. The con of content-based RS is that the machine requires content that can be encoded as meaningful features. Second, feature extraction may be difficult and there is also the danger of overfitting our data. Finally, recommendations are not related to the user’s tastes.

2. Collaborative filtering

In collaborative filtering, recommendation is based on explicit or implicit opinions of users on items (e.g. star rating or reviews). The company has to define a metric to measure similarity between users, such as Jaccard Index, Intra-list Similarity and other similarity measures.

The pro is that this model produces good enough results in most cases. The con is that this model requires a large number of reliable user feedback data points (e.g. star rating, reviews) and products have to be standardized. Additionally, this model assumes that prior behavior determines current behavior.

3. Hybrid

A hybrid approach combines the best features of two or more recommendation systems. A typical combination is to combine collaborative filtering recommendation system with other recommendation systems.

A hybrid approach combines the best features of two or more recommendation systems.

Netflix, for example, combines different recommendation systems. On your Netflix homepage, you will typically see generic recommendations such as new releases, as well as personalized recommendations based on your taste in sections like “You might like this.”

Final tips from Jose

• Start with data governance and ensure high quality data before machine learning
• Collect data in the right way. Think about the attributes of products and capture the relevant user engagements and attributes to fuel your RS and ML in the future
• Give time for your company to generate enough data
• Take into account seasonality in your startup when analyzing data and designing your models

Session 2: Building Machine Learning in Tokopedia by Leonardo Saerang

Machine Learning models are commonly used in Tokopedia. We use ML for product recommendation, search, ads placement, fraud detection, chatbot and many more. We continually improve our models and try to shift away from rule-based models into the more scalable and automated ML models that can serve customers better.

Leonardo gives a sneak peak inside the collaboration that happens behind every new product or product iteration.

The product development cycle for ML products

The Search teams follows the typical product development cycle:

1. Problem statement: Defining a clear problem statement for the customer

2. Defining solution: Design the model or logic to solve the customer pain point

3. Developing the product: Bringing the product — in this case, a new model — to live, with data and development

4. Validating the product: Various testing to validate the product hypothesis and performance

5. Rollout: Introduce the product to users and experiences

A key part of the development team are Data Scientists, Data Analysts and Data Engineers. Data Analysts are experts in analyzing the data and helps the product team develop and validate the product hypothesis. Data Scientists, on the other hand, are the experts on the ML model and build the ML model. Finally, Data Engineers manages the data and creates the data flow to the products.

Final tips from Leonardo

• Write clear and sharp problem statement and expected output from the product
• Choose iterative model: Plan models that can be created in stages
• Manage development timeline: Build in buffers in timeline and proactively manage the timeline. Overcommunicate especially during the development stage
• Validate and learn: Continue to test and iterate the model with offline validation as well when necessary
• Ensure that data quality is high and continue to improve data quality. “Garbage in, garbage out” — your model is only going to be as good as the data input
• Learn more about data science from YouTube and online classes available on Coursera

Behind a Great Product: Machine Learning and AI for a Marketplace Ecosystem was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our take on ensuring that everything the users see and feel is safely reinforced by a reliable system performance.

As a Product Manager, you need to constantly question the state your product is in right now — is it performing? Then you dig deeper. How do I know that my product or service is performing well? Then of course, being a good PM that you are, you dive into deep analysis mode, come up with the ideas, work together with research and data teams to validate your theory, and then you write the requirements and draft up the roadmap for your improvement. Good job!

But then, reality hits. The technical aspects to support the initiative that you proposed are not there yet. The process is too complicated for the current architecture to cater, and you need to take some additional time to reassess all of your options. Let’s say your product has been a huge success on the segmented user testing, all ready for the big launch, and when you’re hype-building for it, suddenly your team pats you on the shoulder and says that it’s going to be too heavy on the system — 3 days before the launch — and you’re going to be left with limited options.

As the Product Manager for the transaction platform in Tokopedia, this is a problem that the team and I face on a daily basis. The whole process, experience, and performance that our services produce is critical in order to keep the users happy on one of the most important decisions in the whole user journey : can I buy these products that I need easily? Our team tries to answer this by a rigorous two-pronged continuous improvement, which are the experience improvements, as well as the performance improvements.

For a beautiful harmony between experience and performance to happen, there is a definite need for a well-crafted plan between the product and tech team to coexist. Here are the important points that our product and engineering team collaborate on, in order to ensure both the growth and performance of our product continuously evolve.

1. Overcommunicate your long term plan with the technical team to stay out of misalignment,

Sometimes, this is true. But every time, this is needed. Source: The Cooper Review

On a high level basis, it’s important to always consult and communicate upcoming proposals to engineers. It is vital that the initial user stories are translated into actionable tasks after the requirements are final. Let the engineers help with a detailed breakdown of tasks, so that your timelining can be more detailed (which will also help you to follow up tasks and reprioritize easily). In the end of the day, the product team needs to be able to be transparent to the engineering team, and vice versa. Everybody will go on undistracted if this can be achieved!

2. Own the service performance as one of the product metrics.

Many PMs devote their time to retrospect about the product’s past performance. But in many cases, the ownership of service uptime (the time during the service is running as expected) is overlooked by the product team. But if we look at it this way : if the users are unable to access your service, the user will take it as a bad experience anyway. No matter what the reason is, when your product is not working as intended, you disappoint users.

In order to understand this in a necessary depth, especially if your product is a core part of the basic end user experience, it is key to ensure in the development cycle that constant uptime performance is a basic requirement that needs to be met. Acknowledge the performance issues so that you have all the information needed to move forwards in development.

3. Ensure there are tech improvements and optimization in each release cycle.

If you’re a PM, I bet you spend a big chunk of your time managing your backlog, and maybe even spend half a work day’s time just to ensure the prioritization is all good (don’t worry, I’ve been there as well). But have you considered the tech team’s needs as well?

Allocating a portion of the backlog for the tech team helps you to better understand the actual effort needed in order to execute the tasks needed in order to ship that cool product that you’ve been obsessing about. Treat the work board as “our” work board instead of “my” work board.

4. Discuss user stories with engineers. They WILL surprise you.

My tech lead once said to me after I pivoted our upcoming product for the umpteenth time, “We are a logical bunch. If it doesn’t make sense at first glance, and it makes our life difficult, We will be the first ones to let you know.”

The statement above is 100% on point. Take some time to let them review your user stories and let them challenge you. You might discover a lot of blind spots, especially if it is a technical product. Another plus point, this routine will help build trust among the team, thus sharpening the product-engineering team’s bond. Win-win!

All in all, a responsible product manager will acknowledge every issue that their product might encounter, and there is no one more important than the engineering team to collaborate with in this setup. Never forget to let them know that they are the awesome people behind every successful product, and take their concerns into account. After all, it is our product.

Special shoutout to our awesome engineering team, David Simanjuntak, Rina Halim, Timothy Agustian, Suwarnold, and Riyanto Adhi Wibowo and the whole Purchase Platform engineering team!

Get inside our product mindset through the Tokopedia Product Team.

Product — Engineering Teamwork : Building The Right Product Whilst Building The Product Right. was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why does it matter for PM?

Many PMs equate innovation with creativity. Though both are related, innovation is not creativity. Creativity is having an idea of a solution, whereas innovation executes or converts the idea into a successful product. As PMs, it’s easy for us to be creative and have new product ideas but we often forget to do decent research. The next thing we know is we’re innovating things that nobody wants.

And how can we avoid this and make sure we are on the correct innovation path? This is where the design thinking framework is beneficial for you.

As Kaaren Hanson, formerly head of design innovation at Intuit explained: “Anytime you’re trying to change people’s behavior, you need to start them off with a lot of structure, so they don’t have to think. A lot of what we do is a habit, and it’s hard to change those habits, but having very clear guardrails can help us.”

Design thinking framework is a series of steps that product managers can use to validate an idea. Although the application can be varied, it’s particularly useful when given a large task such as to revamp an end to end experience of a product or create an entirely new product line.

What is Design Thinking?

Design thinking itself is not an approach where it only can be used by UX or a design-related field. It’s a method that you can practically use for any kind of field (Originally implemented by the architecture area the first time) and soon popularized by IDEO (Well known design company help to design first Apple’s mouse).

Design thinking focused heavily on understanding customers’ behavior before developing a product. With this understanding, it helps you to unravel the unknown problems or unmet needs of your product in the most human-centric way.

Design thinking has 5 phases and it’s not a rigid process. You do not have to follow any specific order, It could even occur in parallel depending on the needs that can be varied per project.

Phase 1 — EMPATHISE

The first step is to shower ourselves with lots and lots of data in order to arrive at insights. Sources of data can come from many angles such as interviews, competitive analysis, internal data analysis, and many more. At the end of the journey, we should be able to create empathetic maps in which it covers (says, thinks, does, and feels)

The objective of doing empathy map exercise is to understand what your customers are experiencing (pain and gains) by analyzing through all their senses. For instance, let’s try to build an empathy map for a high school student:

• Say — What users say during an interview? “Math is really hard and important for my futures”
• Thinks -What’s the background of thought that these users are having? — “I need to pass this exam so that i got a chance to attend top university”
• Does — What users actually do? “Users looks for the perfect tutors”
• Feels — How do the users feel? “Tired”, “Waste of time”

From here, we can summarize that the users experiencing finding tutors is hard and waste of time — pains, but it’s worth it if I can have a ticket to my university — gains

Phase 2 — DEFINE

Based on the previous exercise, now we do know what our users need and this enables us to extract problem statements and prioritize them accordingly. To make sure the extraction is easier and actionable, it’s advised to write:

POV / Problem Statement

[User . . . (descriptive)] needs [Need . . . (verb)] because [Insight . . . (compelling)]

When describing the user needs, remember to use VERB instead of NOUN. Describing it as a noun will help you to put the focus on the right solution to find clarity.

After defining the focused statement, we’ll need to create questions that propel towards solutions. Design thinking approach suggests creating a “How might we” (HMW) question that’s designed to spark ideas and the best way to open up a brainstorming session.

Here’s one example on how to implement the HMW questions:

Challenge

Redesign the ground experience at the local international airport

Problems

Harried mother of three, rushing through the airport only to wait hours at the gate, needs to entertain her playful children because her child already irritated fellow passengers.

How Might We … ?

Here’s a list of possibilities HMW Frames that can be used:

• Amp up the good: HMW use the kids’ energy to entertain fellow passengers?
• Remove the bad: HMW separate the kids from fellow passengers?
• Explore the opposite: HMW make the wait the most exciting part of the trip?
• Question assumption: HMW entirely remove the wait time at the airport?
• Go after adjectives: HMW we make the rush refreshing instead of harrying?
• ID unexpected resources: HMW leverages free time of fellow passengers to share the load?
• Create an analogy from need or context: HMW make the airport like a spa? Like a playground?
• Play POV against the challenge: HMW make the airport a place that kids want to go?
• Change a status quo: HMW make playful, loud kids less annoying?
• Break POV into pieces: HMW entertain kids? HMW slow a mom down? HMW mollify delayed passengers?

Phase 3 — IDEATE

Problem statement + HMW questions are our content to start the ideation. At this step, we try to come up with ideas from the HMW questions thrown out. Keep in mind in this process is we focus on quantity first and quality can come later. For this to flourish, it’s important for us to have no judgment on all of the ideas so they can be built on top of each other. Using a visual language helps to illustrate and bring clarity to everyone involved in the process. All ideas need to be documented then do an elimination round by voting before moving on to the next steps.

“The best way to get a good idea is to get a lot of ideas” — Linus Pauling, Nobel Prize winner

Phase 4 — PROTOTYPE

Prototyping is all about speed and in general, there’re 2 prototypes: Low-Fidelity (LoFi) and High-Fidelity (HiFi).

The strategy here is when first creating the prototype avoids the Hi-Fi, since the more a UX designer invests time in it the more emotionally attached it can get. It’s better to create a LoFi with a pen-paper, or just a simple box just to illustrate and get your message across is much preferable. HiFi, like digital prototypes, is useful when you’ll try to test it across with real users.

Prototyping is crucial in every product design project. Prototyping when done right, gives teams a shortcut to learning without building an actual product.

Phase 5 — TEST

The rule here is to just show and don’t tell. Especially a sales pitch is a big no-no. It’s better to run as many as small tests rather than do many elaborate tests upfront (based on research best results comes no more than 5 users).

When you get a chance to conduct a test to users it’s important to make them feel comfortable, conducting the testing in a casual place such as restaurants might help.. It’s always a good idea to give a testing scenario and share alternatives for comparison. Last but not least grab as much negative feedback as possible. Negative feedback gives you an opportunity to level up your product and open your eyes to difficulties that previously haven’t been thought about and possibly gives you more insight to be considered later.

It’s a wrap…

As a PM, understanding design thinking gives you the benefit to not jump into conclusions too early. Instead, it makes you self reflect, understand users by immersing yourself with data and behavioral evidence. This way it will help you to ensure your innovation is aligned with the user’s needs). Hopefully, this article brings an interest for you to dig deeper about design thinking, so that you’re able to integrate and modify it to suit your own product development styles.

Where to go from here:

• Do watch how IDEO create a shopping cart — https://www.youtube.com/watch?v=M66ZU2PCIcM
• Example of Design Thinking workshop — https://www.youtube.com/watch?v=Z4gAugRGpeY&t=1580s

REFERENCES

• https://www.slideshare.net/janschmiedgen/design-thinkingbootcamp
• https://hbr.org/2018/09/why-design-thinking-works
• https://www.interaction-design.org/literature/article/stage-2-in-the-design-thinking-process-define-the-problem-and-interpret-the-results
• http://crowdresearch.stanford.edu/w/img_auth.php/f/ff/How_might_we.pdf
• https://uxplanet.org/6-common-pitfalls-when-prototyping-6fefa4bd80ea
• https://www.businessinsider.com/difference-between-creativity-and-innovation-2013-4

What Product Manager needs to know about Design Thinking? was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

On August 26th, our team held another Behind a Great Product event. We continued our theme on addressing product management in a Pandemic by talking about pivot. In times like this, where market and customer behaviour changes quickly, it is important to be agile in our product development. Below is the summary of the presentations.

Topic 1 — Tokopedia Salam

By Iman Cinderamata — Head of Product for Tokopedia Salam

What is Tokopedia Salam? Tokopedia Salam is a sharia initiative that allows users to do good deeds and provide Islamic products and services through seamless digital solutions. This includes services such as Umrah, Donation, Zakat, Sharia Mutual Fund, and more.

In 2020, the main focus of Tokopedia Salam is to capture the Umrah market. Tokopedia Umrah service itself was launched in November 2019, but a lot of improvements needed to be made on pre-, during, and post-Umrah to cover the end to end Umrah experience.

However, Arab imposed their country lockdown in February due to the Covid-19 pandemic, which stopped the whole Umrah business. The team morale was low, it was a challenging time for Tokopedia Salam since they have been investing in the Umrah service for nearly a year. This is where Iman, as a product leader, steps up to strategize a pivot on his 2020 strategy and once more brings up the spirit of his team.

Iman mentioned three steps to conduct his pivot: First, is to identify the problems by analyzing market trends and hearing users; second is to validate by testing to understand whether you’re trying to solve the right problem; and lastly evolve to execute your alternative solutions quickly.

Tokopedia Salam quickly pivoted their focus from Umrah to three other services to capture the new market opportunity. In March, they focused on Donation, due to the increase of the Covid-19 donation trend throughout Indonesia. In April, the team chooses Zakat, to prepare for the Ramadan month where Muslims usually present their Zakat. In June, they improved Qurban to capture the Hari Raya momentum of people buying cows and goats.

Iman shared how he uses the three steps to pivot from Umrah to the three individual products and the results Tokopedia Salam able to achieve.

Case 1 — Donation

Identify

Early March, Iman noticed a growing trend of people raising donation funds to fight the Covid-19 pandemic in Indonesia.

Validate

On the current donation entry point in Tokopedia checkout page, the Salam team simply changed the donation wording into something related to the Covid-19 pandemic. Turns out, users are responding positively to the change, and find conversion increased compared to the previous wording. This shows a market appetite for donations related to Covid-19 pandemic.

Evolve

Tokopedia Salam created a full-blown webpage and worked with the marketing team to build campaigns around the Covid-19 donation, as well as increasing the donation ticket size from an average of Rp5,000 to around Rp100,000. The result was a staggering increase of 2.8x during March compared to the previous monthly average.

Case 2 — Zakat

Identify

There are predictions from the Muslim community that online Zakat will rise 30% this year because of the pandemic since more people are avoiding offline interactions including giving Zakat.

Validate

Analyzed the traffic of Zakat page compared to last year, and there were indications of healthy growing organic traffic of users trying to find online Zakat information.

Evolve

The team focused on creating more online Zakat education along with features such as Zakat calculator to simplify the user experience of giving Zakat online. The result is a 2.5x increase compared to last year.

Case 3 — Qurban

Identify

Indonesians were not able to conduct their Ibadah Haji due to the pandemic, so a lot of their budget are transitioned into other good deeds, the biggest trend was spending on Qurban.

Validate

Similar to Zakat, there is an increasing traffic of organic users searching for Qurban in Tokopedia.

Evolve

Tokopedia Salam builds an end to end Qurban purchase experience whereas previously they used the physical goods purchase process which is not ideal for doing a Qurban purchase. This decreases the steps for a user to purchase from 16 to 8. The result is a 1.5x increase compared to last year.

In conclusion, market dynamics are changing rapidly especially with the pandemic. As a product manager we need to keep our eyes and ears open to adapt quickly and seize new opportunities as they replace the old ones. Pivot quickly, accurately, and last but not least, make sure you keep your team morale high through these difficult times or else you wouldn’t have the capability to execute all of this.

Change is constant, adapt you must — Yoda

Topic 2 — Product Marketing

By Maisyalina Agustiana — Lead of Product Marketing

In Tokopedia, Product Marketing is responsible for developing effective marketing strategies and plans to communicate the features and benefits of the products to customers. Our Product Marketing team helps Product Managers launch, market, and sell products. Moreover to prepare things such as marketing communications, marketing channels, and marketing goals.

“Adapt, Evolve, and Embrace”. These are the mantras that Lina uses to navigate the Covid-19 pandemic situation. The pandemic brings a lot of sensitivity to the company communication that triggered changes in the planned campaigns the team needed to execute. As market dynamics changed in matters of hours, Lina and the Product Marketing team needed to make adjustments to their processes. She revamped the whole campaign submission process from a month-long to only 3 days workflow.

These are three great pivot examples from the product marketing team from a seller communication plan standpoint.

Case 1 — #JagaEkonomiIndonesia Campaign

March is supposed to be the month where businesses are busy preparing for a big festive season of Ramadan, instead many of them barely survive due to the lockdown imposed throughout Indonesia. The Product Marketing team viewed this as an opportunity to help offline sellers survive the pandemic by moving their sales online.

They deprioritized their initial plan that revolves around existing Tokopedia sellers and pivots to help offline sellers onboard online. They successfully reached 3 million offline businesses to raise awareness of online commerce.

Case 2 — Adjusted Seller Education

As the pandemic situation changed rapidly, there were more and more new regulations being introduced by the government that can be limiting to sellers. The Product Marketing team pivoted their educational content plan to be a quick and reliable source of information for the sellers to navigate the pandemic. Things such as new normal regulations, logistic limitations and also shifting market insights, all of these helped Tokopedia sellers to survive or even thrive.

Case 3 — Introduce Empathy to the Marketing Journey

The Product Marketing team also adjusted their marketing messaging and copy along the seller journey to be more empathetic to the situation. Starting from when a new seller opens a shop up until they fulfill their orders, the team weaved empathy and pandemic context to reflect the new normal situation.

There’s no such thing as a free lunch. Being alert 24/7 to new information, give extra effort to do quality checks on sensitive messaging, as well as intensive coordination with multiple stakeholders. These adaptations are great for the users but the team needed to work twice as hard to accommodate the changes. As a product leader, Lina reminded us of the importance to not only adapt externally but also internally as a team.

Behind Great Products: Pivoting Your Product Roadmap was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

Not many people in my circle have a strong understanding of what product management really is, since it’s new especially in Indonesia.

Product management is a relatively new field that has been getting more spotlight in the past few years with the growing popularity of startups in the late 2010s. It is one of the few professions where there is no corresponding degree or concentration in most universities.

When I first started my career in product management, I felt distant from this role, like a Lego piece that didn’t quite fit with anything else. My educational background had very little overlap with what I did on a daily basis as a product manager.

So how the heck did a graduate of macroeconomics, like myself, ended up with a role in product management?

The beginning

The first few months after I finished my university thesis, I was lost in thoughts on what kind of career I wanted to pursue. At that point, I was faced with three major options: pursue a master’s degree, start working at an established conventional company, or follow my curiosity into the tech startup world. I chose the latter because I wanted to satisfy my curiosity and understand the inner workings of tech unicorns in Indonesia.

I assumed that if I wish to jump to the tech startup scene, I need to build my knowledge in software development, right? So, I reached out to a software development studio in a co-working space I worked with before and managed to secure the role of a project manager there.

Kicking off with project management

During my time in this wonderful studio, I experienced firsthand how software was built — from ideas to storyboards to wireframes and prototypes, to fully functional web or mobile applications. I also got the chance to pick up some technical knowledge on system architecture, especially how clients (front-end), applications (back-end, logics) and databases were arranged and connected via APIs. I participated in daily stand up meetings, joined problem-solving discussions to achieve weekly sprint goals, and helped communicate progress to clients and internal team members.

This role helped paved the way for my product management career in Tokopedia.

I was overwhelmed by the jargon and terminology, but the one thing that I managed to do very well in understanding was the architecture and data flow of a typical software. By looking at how components were carefully built by front end developers, business logics were arranged by back end developers, and databases were broken down into simple concepts of tables and functions, I was able to see how engineers and products were interacting with each other.

As I entered my second year in this studio, I started to lose some level of ownership and creativity when executing project-based solutions. This feeling grew more day by day, and I started to become more curious about how I will approach bigger and longer-term product challenges.

Becoming a product manager in Tokopedia

To feed my curiosity, I started looking for startups that would accept people with my background into their product organization. Many startups came to my mind, but one in particular left a strong impression on me. The startup is Tokopedia, one who frequently put on ads on the cinemas on how it helps SMEs in Indonesia. Something inside me resonated with this vision. The more I read on Tokopedia, the more I became interested in being part of the company’s positive impact. So I applied.

My job interview did not go smoothly, as I was ill-prepared on some points, but that’s a story for a different time. Eventually, I managed to get hired in the end and was immediately trusted to manage two important modules for Tokopedia: Order Management and Seller Home.

In this role, my entry-level understanding of tech architecture turned out to be extremely handy for getting by in the beginning. Engineers became my go-to help desk when I tried learning about the scope and behavior of the modules that I handled.

First learning: PMs exist to solve real problems

Working under an ex-Amazon VP of Product heavily influences my view on product management: words and concepts don’t matter if they do not solve real user problems. And so I made it my mission to understand my users and reached out to my users. I talked and listened to their real problems and slowly learned about their daily struggles and how our products have and might solve their day-to-day pain points.

In parallel, our Data team would help process data and generate insights that would help guide my decision-making in product development. Once I have a good grasp of the pain point we are trying to solve, I would then work with our Product Design team to craft product solutions, while our engineers do their magic to realize the solutions that we have considered and planned. But this is not the only way a product manager in Tokopedia operates.

Having a good knowledge of tech and decent data intuition helped me kick off my product management journey.

New scope, new learning

As my team and I were progressing through our ambitious goals, we stumbled upon a much bigger and fundamental problem: how do we know that we are truly solving the user’s problem exhaustively, and not reactively? How do we really establish the democratization of commerce in our ecosystem?

Working with and on “features” is not the only way Product Managers operate, but experimentation is.

After we regrouped and restructured, I am now in charge of Seller Segmentation and Growth. My new mission is to ensure that we are solving our seller’s problems holistically and ensure that our product fundamentals lean toward the democratization of commerce. It is key that I understand the complexities of our seller landscape and help sellers with their needs at every stage of their journey in Tokopedia.

This new module is a whole new world with a very different pace. No longer do I talk to as many engineers and designers as before. By being constantly exposed to data and problems, I am challenged to be adaptive and critical, to have a scientific mind and experiment with many hypotheses. I am still in the process of grasping and developing the strategy, and hope to share with you my reflection on this journey in another occasion.

Two months into this role, I am certain that there is another modus operandi for a product manager, demanding the essence of product management which is to identify and solve problems: by turning curiosity into insights, insights into hypotheses, hypotheses into experiments, and experiments into solutions!

I am not a misfit — nobody is!

I am experiencing first-hand how a product manager’s role and skill set evolve and adapt with the scope and the nature of the problems he is dealing with. This made me realize that I am not a misfit, I just need to recreate myself with every new challenge and user problem. That’s what I’m doing now: applying some theories and learning from my university as a guideline for my job now!

In a nutshell…

After all this time, I still think that product management is a new and constantly evolving field. It really feels all new to me.

Once in a while, my relatives or acquaintances mistake my role of “product manager” as someone who deals with merchandising or curates catalogs for Tokopedia. No matter how many times I have explained this role to people, it seems like I will continue to have to explain and re-explain my role in product in my annual family gathering, since the role of a product manager continues to evolve.

But such is the nature of this field. There is no single correct way to do it, as every product manager has to adapt to the situations and challenges faced in front of the person. It is no wonder that adaptability and learnability are some of the major qualifications required by many startups and companies for the product management role.

One way to stay adaptive is to be critical and data-driven, talk to as many people as you can, and test things out to find what works and what does not.

My journey away from economics brings me back to economics.

My role in Tokopedia brings me back to my undergraduate major in Development Economics. I feel like I can actually help democratize commerce and economy via technology. This was my motivation when I first joined Tokopedia, and it still is today.

Though my path was one of an unpredictable and unprepared at the start, I can now call myself a proud Nakama because I’m #JustGettingStarted to build impact in Indonesia with over 7 million sellers.

Do you dare to take on a similar challenge and pursue a product management career in Tokopedia?

How a development economics graduate ended up in product management was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

Managing vast data points as a technology company can be complicated. Whether it’s making faster, more informed business decisions, improving flexibility to react to uncertainties, or increasing awareness of risks, having structured and connected data is crucial. A graph database excels in these areas by enabling organizations to create robust preventive measures, especially for fraud detection.

What is a graph database?

A graph database is a structure that manages relationships between data as equally important as the data itself. One common method is the property graph model, which consists of:

Nodes: Represent entities like users, products, or devices.
Relationships: Connections between nodes (e.g., “BOUGHT,” “REFERRED”).
Properties: Metadata describing nodes and relationships (e.g., timestamps, quantities).

Tokopedia’s Journey with Graph Databases
From Experimentation to Adoption

In December 2018, Tokopedia Risk Management began its graph database journey, initially using an experimental project known as The Graph. Unfortunately, it was riddled with bugs, making us the “test engineers” for an unstable tool. By May 2019, we decided to abandon The Graph.

By September 2019, we explored alternative graph database solutions, settling on a more popular and reliable option. As its usage grew, in 2020, we took a big leap of faith, dedicating a squad to expand graph database applications within Tokopedia Risk Management.

Here are some key takeaways over the course developing Graph Database in Tokopedia.

We Have to Create a Specific Relation Name

Initially, we used a generic relationship name, “HAS,” for all connections. This limited our ability to analyze specific relationships. Migrating to meaningful relationship names like “BOUGHT” or “REFERRED” unlocked deeper insights.

Deadlock Might Happen

Graph databases can face deadlocks when inserting large volumes of data simultaneously. Plan for this by managing transaction queues effectively.

Unstructured Data, Different Way to Query

Graph databases differ from SQL; they lack fixed schemas and avoid joins. Learning NoSQL querying concepts (e.g., Cypher for Neo4j) is essential.

Resource Intensive

Graph databases struggle with database-wide queries and heavy workloads, requiring additional server and RAM resources to maintain performance.

Community vs. Enterprise

While community editions are free, enterprise editions are more scalable and better suited for commercial needs. Evaluate based on your requirements.

Data Can Be Dangerous

With great data comes great responsibility. Log all activities involving your graph database to maintain transparency and accountability.

Limitless Use Case

Graph databases are not just for fraud detection. They’re used globally for real-time recommendation engines, identity management, social networks, artificial intelligence, and knowledge graphs. A famous example is the Panama Papers, where graph databases unveiled hidden relationships between offshore bank accounts, exposing assets of politicians, celebrities, and even royalty.

At Tokopedia, we leveraged graph databases primarily for fraud detection but have extended its use to detect nuanced behaviors like Cross Buying, Cluster Cross Buying, and Concentrated Purchases.

Detecting Fraud and Purchase Behaviors with Graph Databases

1. Cross Buying

Cross buying occurs when a user purchases across multiple product categories, such as electronics, groceries, and fashion. While often legitimate, abnormal patterns can signal coordinated abuse.

• How Graph Databases Help:
  Graph databases map user-product relationships, linking users to purchased categories. Queries can detect unusual diversity in purchases during short timeframes.

2. Cluster Cross Buying

Cluster cross buying happens when a group of users (a fraud ring) shows coordinated cross-category purchases to exploit rewards.

• How Graph Databases Help:
  By connecting users through shared attributes like IP addresses or devices, graph databases can detect dense clusters exhibiting similar buying behaviors.

3. Concentrated Purchases

Concentrated purchases involve bulk buying of specific products, often exploiting volume discounts or limited-time promotions.

• How Graph Databases Help:
  Graph databases link users to purchased products and quantify transaction volumes, flagging unusually high purchases.
• Cross-Cluster Insight:
  Overlay transaction patterns with shared attributes like devices or payment methods to uncover fraud rings coordinating bulk purchases.

Real-World Fraud Detection Scenarios

1. Coordinated Referral Abuse:
   Fraud rings create multiple accounts using shared devices or IPs, refer each other, and trigger rewards. Graph databases can expose referral loops and dense clusters.
2. Promotional Exploitation:
   By detecting shared attributes across accounts exploiting cross-category promotions, businesses can flag abuse early.
3. Reward Claim Patterns:
   Analyze claim nodes to identify users or clusters repeatedly claiming high-value rewards during short periods.

No matter your business stage or challenges, there’s no need to hesitate when exploring the potential of graph databases. Their unique ability to analyze relational data can empower your company to learn and scale more efficiently than ever before.

At Tokopedia, graph databases transformed our fraud detection approach, uncovering hidden relational patterns that distinguished legitimate users from fraudulent ones. They allowed us to identify sophisticated behaviors like Cross Buying and refine our fraud prevention strategies for greater impact.

Whether you’re combating fraud, developing recommendation systems, or managing complex data relationships, graph databases offer a powerful, scalable solution to address these challenges effectively.

I would love to hear about what experience and idea you have encountered while developing graph databases.
Feel free to contact me at fahrell.giovanny@tokopedia.com

Special shoutout to:
Alluka Team — Yuga, Rian, and Josua
‘Technical Architect’ — Zaki and ‘Principal Engineer’ — Tjandrayana

Click the 👏🏻 below so other people will see this here on Medium.

How Tokopedia Build Their First Graph Database was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

How Tokopedia mitigate and prevent fraudulent users from unsecured lending perspective

Fraud is a constant cat and mouse game between established institutions and criminals. Once a financial institution upgrades its fraud prevention measures, fraudsters are already looking for ways to outsmart the system. From global banking to emerging financial technology companies, all have suffered from fraudulent activity. The average amount of losses incurred could reach up to $5000 per scam (KPMG Global Banking Fraud Survey, 2019).

What we’re fighting: social engineering

Ever heard of the “Nigerian Prince Scheme”? This is called social engineering/phishing, an act of cybercrime in which victims are contacted by someone posing as a legitimate institution to lure individuals into providing sensitive data such as personally identifiable information (PII — e.g. Mother’s Maiden Name, ID card number, birth date), banking and credit card details, and passwords. In Tokopedia, we highly value user’s private information, as such that a One-Time Password (OTP) is worded as follows:

As straightforward as it is, many users still get tricked by criminals who usually pretend to be a Tokopedia representative (we also had a case where a scammer identified himself as a “Toko Sebelah” representative, following which the user willingly gave the OTP to them!).

The story of why fintech products are the target of malicious users

Let’s cut to the chase. What actually happens in Tokopedia? I am pleased to introduce a pioneer in merchant credit line services on an e-commerce platform, Modal Toko. Modal Toko provides merchants with a seamless lending facility by assessing merchant activity, sales, reputation, fraud history, and merchant credit risk. Merchants only have to upload their Know Your Customer (KYC) details and some personal information. Service level agreements for the whole process will require at maximum 2 working days.

As previously explained, these lending products are much more convenient than traditional loan products. However, convenience and security are not correlated. When you give customers more convenience, the measurements put in place become less secure. Hence, finding the balance is the main challenge.

Then, fraudsters often make a move to lure mature merchants with a good reputation, as their probability of being offered Modal Toko is much higher. Usually these merchants provide detailed information on their Product Details Page (PDP) or shop, or even give out their personal number which is also registered as user login details. They can easily attempt to login using only a phone number, trigger the OTP and take over the user account. In most of our cases, users reluctantly give the fraudster their OTP details despite all the warning signs.

When account takeover happens, fraudsters can easily apply for the feature as the merchant’s KYC has been uploaded beforehand (or uploading a stolen identity). Since the approval process usually needs less than 1 hour (which we initially considered was a cutting-edge improvement), fraudsters generally manage to finish their takeover just before the authentic user can grasp reality. Meanwhile, active Modal Toko users are already prone from the moment the feature is activated. Fraudsters can easily withdraw loans and disburse funds. The losses that a genuine user, Tokopedia, and partner must bear ranges from Rp 2 million to as much as 100 million per account.

Getting to the root of the cause: creating targeted friction

We identified that almost 90% of recent fraud cases have one root cause — Sensitive Data Changes (SDC). This means that actions need to be done without sacrificing the user’s experience. Hence, we created an API to identify a user’s last data change to inform us on the user’s data-changing activities. We then consume the data and create a validation with the API at the point of:

1. User when entering the application form
2. User when initiating loan disbursements

In some situations, we block actions outright. However, in this situation we give users the opportunity to satisfy an additional verification step called a friction. Using the validation mentioned, we created a friction targeted to those who are changing phone numbers or e-mail details within our time range criteria. Whenever users, genuine or fraudulent, attempt to click the Apply/Disburse button, we will immediately block their next activity.

To inform the genuine user, who might become the victim of such an activity, we will send a notification to their old and new phone numbers informing them that there has been an attempt to access the user’s Modal Toko feature, urging them to report to us if they had not initiated that action. User accounts will only be unblocked after the user verifies their authentic identity to Tokopedia through the Fraud Operations team.

Our Credit Risk team also creates a daily report of all Modal Toko applicants of the day who conducted a SDC within the last couple of days, for partner assessment in regard to loan applications. Moreover, the Modal Toko team assumes that these cases must be coming from syndicates as the fraud mechanisms are very similar to one another. Therefore, we are working closely with the Tokopedia Fraud team to handle occurring cases.

Our achievements and the preventive measurements taken

Since kicking off this activity in September, our ticket from social engineering cases has declined by nearly 90%. We managed to prevent losses of more than Rp 500 million from 15 different accounts. We also found several good leads in identifying the criminals, as numerous activities point to either the same devices, bank account name or bank account number.

Well after the SDC cases had been (somewhat) resolved, we still identified another pattern of fraud. Which shows us that the challenge does not end here due to the unsecured lending environment. Some improvements have been made after we launched, to minimize the number of false positives such as lengthening the tenor of blocker, providing access to users’ extensive account history to the Credit Analyst team, and shortening the approval process. Nevertheless, our current fraud prevention and risk management measures are still far from ideal, and many more steps must be taken. Experimentation and analytics orchestrated with more variables could be taken to compose machine learning models in order to target more identifiable patterns of fraudulent activity inside e-commerce.

Our final thoughts

It’s true that what we are doing will never be enough, as fraudsters will always find a way. It is also highly probable that they are currently reading this Medium article and thinking about their next move! So, among the major key takeaways, it is crucial that we act faster than those looking to take advantage of the Tokopedia community. Tokopedia is really about user trust and everyone should act based on this belief. Then, it is the Product Manager’s job to think faster, act faster, and not lose sight of the long-term possibility when carrying out a product roadmap while not sacrificing the user experience.

— — — — — — -

Special thanks to my mentor Tyonardo, Merchant Lending team, Profiles team, Fraud team, and Operations team for the solid collaboration in tackling the issue.

Do you have something in mind that you think can be improved from what I have elaborated? Let’s discuss and hit me up at vincentia.maudy@tokopedia.com!

Adapting to Cyber-crime Attack with Targeted Friction was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gold is a go-to investment for many Indonesians, given a history of steep fluctuations in the rupiah. While older generation preferred to kept gold in the forms of jewellery, millennials are preferred to kept gold in the form of bar (mostly made by Aneka Tambang, or Antam) which mostly sold in gold shops and pawn shops.

The two main reason why gold are very popular in Indonesia is because :

1. Most people still prefer something tangible and can be physically kept in their own houses.
2. Other popular choice for investment was property/real estate, but its high entry barrier made it difficult for the mass population to invest in.

Only few are aware of the more savvy investment products like stocks, bonds, and mutual funds. Knowledge and access to each product also contributed on the gold popularity. While gold may be popular today with a lower barrier to entry than real estate, many would still consider it expensive.

By the time this article was written, a certified 1gr gold bar is valued as much as IDR 760,000 (USD 54). Let’s consider the monthly minimum wage in Jakarta in 2019, where the number is equal to IDR 3,9 million (USD 277). This means that in order to obtain a 1gr gold bar, people need to spend 19,5% of their monthly wage. This is the reason why there are perception that, gold is a luxury item that is not affordable for most people. That being said, our vision was how to make gold accessible to most people — not only for price, but also access.

Together with Pegadaian, an Indonesian government-owned corporation which has a monopoly on fiduciary services, we agreed to build a platform where gold can be bought with a price as low as IDR 500 (USD 0,036). With the help of technology, our users can have gold saving account by starting in a small amount and have the option to redeem the gold bar only when they have enough amount to do so. Since the gold bar isn’t exactly needed, the barrier to purchase a minimum value can be excluded.

Designing an easy-to-use investment product

Our next challenge was, how to build an easy-to-use product that most of our users are familiar with. A lot of research has been done with the objective to find user pain points and expectations towards investment product experience, which resulted in unit of value being the most concerning element to users. Gold have two different units: The weight (X gr) and amount (Rp X). Gold also have different amount based on the buy-price and the sell-price, which may look more confusing to new users.

From the research, there are two things that we can learn, as also seen on the above illustrations.

• Balance in dashboard. Based on the research, we learn that most users prefer to have a display of the amount instead of the weight. This is because users believe it would be much easier to read the real-time exact value. Also, since gold price fluctuates, it helps them to quickly identify whether the price is increasing or decreasing. Thus, we decided to emphasize more on the gold amount instead of the weight by making it bigger in size and use bold typeface. Sell-price (including admin fee) is displayed by the amount, because all that users care is how much money they actually get by selling their golds. So, we need to make sure that the displayed amount is exactly what the users will get, not decreased with any additional amount or whatsoever in the back.
• Transactional Unit. Some users are more comfortable using the amount as value in transaction, while some other more are comfortable using the weight. We see this as an equal preference and thus we think it’s equally important. If users input Rp X in the amount, the weight field is automatically calculated using real-time gold price, making it more functioning as a gold-calculator. This will help users in their decision-making on the value of their transaction.

The result was amazing! Based on our Net Promoter Score (NPS) report, 65% of our correspondent claimed that Tokopedia Emas is easy to use, making it as the top reason to recommend the platform. Meanwhile, based on our Regression Analysis, user interface is the second most important factor that significantly impact user intention to use our platform. We achieved our goal to build a gold saving platform that is easy to use, even for those who are new to investments.

Hypergrowth with growth hack strategy using existing ecosystem as support

We’re done building the platform, but the real challenge is yet to come. Indonesians are fond of a lot of things. Unfortunately, saving isn’t one of them. Our challenge is how to shift the mindset of our users from consuming to saving. It may sound simple, and some companies may just run a huge ads or campaign in this kind of situation. But for us, we believe that what we need is a growth hack strategy that solves our users’ problems.

After doing a lot of data analysis, we found out that we have to shift the user behavior from consuming to saving. Based on our data, Tokopedia Emas user only shared a small portion out of all Tokopedia user. Hence, we need to raise the awareness of Tokopedia Emas and make user journey to investing as simple and seamless as possible.

We analyze our users behaviour and get into the conclusion that we may be able to do cross-selling with our marketplace transaction flow in the form of round-ups feature. As the name suggests, this feature will round-up every transaction value that users make and deposit the round-ups value into gold savings based on the real time gold price. For example, if users purchase a total transaction of IDR 124,100, they can voluntarily add IDR 900 as the gold purchase and the transaction amount will become IDR 125,000. This feature will help our users to save up during transaction. Since the offer will be on the transaction flow, it means we also have a huge exposure coming in. We ran the pilot and the result was crazy.

Our new users are increased by almost 300% in the first month we implement the feature (from month 2 to month 3). As seen on the cart above, the round-ups keep contributing to most of the new user acquisition with average monthly distribution of 83%. Users who try this round-ups feature are coming back to standalone gold platform and be retention user. The cycle keeps going and the best part was that this is 100% free from any cost.

The only thing that we need to keep on doing is to frequently analyze users behaviour on our pricing strategy. We do a lot of A/B testing on the round-ups value to fine tune which numbers drive what metrics the most. Right now, our goal is to acquire new users for Tokopedia Emas. Should there be any times when we aim for different metrics, we already have the data and we would be able to adjust faster to achieve specific goals.

We’ve made a huge step towards our mission to democratize gold savings through technology. Going forward, we will keep evolving ourselves to provide the best experience for our users, so that more and more people can access gold savings easily.

Democratize Gold Savings Through Technology was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

When it comes to defining the right objective of Product Development, various data points including quantitative and qualitative research is needed to ensure we are heading in the right direction. With the dynamic user response and behavior, measuring the effectivity of our products can be tricky. Then, how we as a Product Manager measure we ship the right product to majority of users?

That’s when A/B testing take a vital role.

On February 2018, Instagram was testing their feature to send notification when user take a screenshot of Instagram story similar to the one in Snapchat. But, the majority of Instagram users didn’t even know that this feature was once exist on Instagram, as we were not their sampling population for testing. If we cannot see the feature today, it is most probably because they decided to cancel to ship the feature to 100% users after seeing the bad results from the testing.

This testing method is called A/B testing. It is where we isolate control and experiment segment consisting of random population to test two or more variants, using statistical analysis to determine which variation performs better for a given success metric(s).

Imagine how big the loss and complain would be if this feature went to all Instagram users without testing. To this point, testing result can be the ultimate last layer for decision making whether to continue ship a feature to all users or not.

In Tokopedia, you might have once seen a feature that no longer exists or has been heavily improved by now. Those features might have a bad result in the A/B testing, and so we did continuous iteration to provide the best browsing experience for users, to fulfill our #1 DNA: Focus on Consumer.

The Importance of A/B Testing

On the left graph shown above, we might think that a feature delivers impactful result when we compare the before vs after deployment metrics, without considering external factor of Big Promo after deployment date that have significant contribution to the metrics growth.

With A/B Testing, we can isolate the sampling population on the same testing period to get the accurate impact of a feature. On the right graph, we can see both Control and Variant have increment pattern due to Big Promo, while the real impact of new feature are shown on control vs variant.

Executing A/B Testing

In Tokopedia, we do sets of continuous experiment in weekly basis (yes, even when we don’t have any new features to be released that week, we still do sets of optimization experiment), to ensure that all users will always have the best experience on our apps.

The continuous experiment need to be followed by a smooth execution to get accurate results. Here is a real-life example on how we execute A/B testing in Tokopedia:

What:

• To execute an A/B testing we need to define the right problem. Example: Among all Flash sale funnel, Product Detail Page (PDP) has the highest drop-off rate.
• After defining the right problem, we can generate hypotheses. Example: PDP is not compelling and relatable to Flash Sale (FS).
• Hypotheses need to be broken down into a set of user stories. Example: as user visit FS PDP, I should be able to feel a sense of urgency and benefits created from the campaign
• Set quantifiable metrics to set end goals for the experiment. Example: Conversion Rate to Add to Cart (ATC)

Who:

We pick random population of users and divide them into several groups. Then, we pick up which group are getting tested, but only the relevant users (e.g: FS users)

If you look at this illustration, green dot is the group we pick and red dot is the FS users. The group that we picked has to be tested against a control group of the same size. Hence, the two dots in the picture.

When:

Correct testing period also contribute in results accuracy. We usually use good rule of thumbs, where we do experiments for 2 weeks, to avoid cyclical effect.

How:

1. Result accuracy highly depends on how you monitor the execution.

We had some bad experience where we executed experiment in 2 weeks. Seeing the results after 2 weeks, only 3-days data were reliable due to tracking error on the rest of the week.

As a cost to the aforementioned lack of monitoring issue, we need to repeat the experiment again to get a more accurate data. Thus, health check monitoring are important. We can have several checkpoints such as:

• Ensure the right user population are receiving right feature (control or experiment)
• Ensure tracker are doing good in testing period
• Ensure feature that being tested are working perfectly
• Detailed analysis to understand context

2. To present the data better so that every stakeholders can learn in the same understanding, we can convey the data into story. For example:

• “CVR to ATC shows XX% improvement on Variant version”
• “As result of optimizing above the fold info, X more user spend less than 15 seconds in deciding to ATC”

Various Type of A/B Testing

A/B testing can be done to check new feature performance or to optimize existing features. Here are a few examples on low hanging fruit improvement we have done through A/B testing for existing feature in Tokopedia: Layout, Content (Text or Image), and Copy.

1. Layout

• User Problem: majority of landing page have rich content in order to provide thorough and inspiring information for user. Yet, due to low Click Through Rate (CTR) and Conversion Rate (CVR), we assume this rich content distract user from high quality products that we actually offer.
• Hypothesis: by reducing the number of content and displaying straight forward product, user can be more focused on campaign purpose.
• Result: reducing unnecessary content bring xx% increase CVR and xx% increase CTR

2. Content

• User Problem: CTR on promotional banner are low.
• Hypothesis: optimizing space on banner by reducing copy and Key Visual (KV) will be easier for user to digest, thus, improving the CTR.
• Result: Simplified content deliver XX% CTR.

Case A:

Case B:

3. Algorithm

• User Problem: User need to scroll far on product section to find relevant product for them.
• Hypothesis: Providing relevant product to each user (personalized) will reduce buying decision time thus improve CVR.
• Result: Providing personalized product instead of curated product improve XX% CVR.

Those are some successful case of A/B Testing in Tokopedia. But unlike Disney movie, we don’t always get a happy ending result on our testing. Some of them shows positive impact while the other tell us to work harder.

When we put our best effort to develop a new feature yet the result is below our expectations, bad result on A/B Testing could be seen as a downside. At the same time, it is a big relief for us to figure it all out and make room for improvements before we ship it 100% to all users.

A/B Testing method is a good exercise for Product Manager to get to know their existing and upcoming products better, to get continuous learning and to share it with other stakeholders. Specifically for us in Tokopedia, to be able to get to know our consumers better to continuously run our #1 DNA: Focus On Consumer.

— — — — —

Special thanks to Gabriella Amanda Kawilarang and Harvey Tjiupek for your contribution on this Blog!❤️

A/B Testing to Save Your (Company’s) Life was originally published in Tokopedia Product on Medium, where people are continuing the conversation by highlighting and responding to this story.