The Fastest Answer
Among the (many) things I learned during my time at Zynga, one topic that stands out was the obsession to lead with experimentation and rapid iteration.
At Zynga, in addition to typical A/B feature testing this meant relatively aggressive use of dry tests (aka “fake front door” or heat tests) to spot check customer sentiment prior to investing heavily in a new area¹.
Racing to the conclusion
Over time, I’ve distilled the concept in my own way as follows
What is the key product question we’re trying to answer, and how do we answer it as quickly and inexpensively as possible?
I feel this captures the spirit of the approach without being overly prescriptive regarding the solution. In that sense my toolbox typically includes
- Market surveys (outside the app, ask the customer)
- Dry tests
- Lightweight betas (MVP, maybe)
- Actual features (guided by A/B testing)
Each has its own strengths and weaknesses in terms of cost, time to execute, and the qualitative nature of the feedback.
The goal is always to accelerate learning, and get to the conclusion (whether good or bad) as quickly as possible. Go faster by executing a larger number of small increments, and front-ending key questions before building on top of potentially flawed assumptions.
(Ironically in my roles after Zynga as an engineering leader I have found myself cautioning product managers not to “over-engineer” solutions, to concentrate first on answering key questions☺)
Outcomes
But does it actually work?
We of course used this approach to good effect at Zynga — one situation that I experienced personally was boosting revenue growth on Café World (TL;DR it became one of the top company franchises but didn’t start that way).
At Tile, we ran this playbook with great success to roll out a recurring revenue model and close a key funding round (this is a pretty decent story worthy of a separate future write up — [edit: the Tile Premium Story can now be found here]).
Culture shift
Not every software company is deep into A/B testing, and even those who are may feel deeply uncomfortable with aggressive heat testing. Typical feedback includes
It’s a horrible UX to click on an offer and receive a “Thank you — coming soon!” response
or
These tests will destroy our brand. What if someone posts one of these fake offers or pricing variants on social media and it goes viral?
or
This MVP is insufficiently compelling. If we burn our users on this experience, they may never be willing to try it again even when it’s great.
This are valid concerns, however it turns out the collateral damage rarely manifests in practice. In the end, it’s a matter of balancing relative risks, limiting exposure while collecting meaningful data.
Get Steve Klinkner (aka Klink)’s stories in your inbox
Join Medium for free to get updates from this writer.
Of course the tests themselves should be constructed in the most tasteful way possible, and exposure should be limited to just enough customers to obtain meaningful feedback.
Caveats
A few caveats are worth mentioning
- Not all measures give you the right answer
- Data at scale requires data at scale
- Some MVPs are bigger than others
Not all measures give you the right answer
You need to take all test results with healthy skepticism, and make them one input among many. Sometimes the data runs counter to what you believe elsewhere and you need to take a leap of faith.
As I like to say if you queried consumers in 2009
Would you like to find a random stranger on the internet and have them take you for a ride in their car?
you’d expect the response to come back something like “no effing way”. However then perhaps Uber may never have been founded².
Data at scale requires data at scale
Collecting data quickly at scale requires your app to already be at scale, which can be a frustrating chicken-egg problem.
In this sense, apps at scale have an unfair advantage — they can get experiments to statistical significance quicker with smaller user sets.
Not being at scale doesn’t mean you shouldn’t test. It just means your tests may initially be riskier or take longer to collect data.
Some MVPs are bigger than others
I’m mindful that some features are sufficiently complex that a substantial investment may be required in order for them to be truly compelling.
An recent example is the airport landing notifications that we build at Life360.
While it’s not hard to heat test interest (“Do I want to know when a loved one lands safely? Yes please!!!!”), it’s a little harder to bring a meaningful MVP to life
- If we send “Steve landed at DTW” notifications to my family as a test and it’s literally not true, it will just cause confusion
- An actual MVP that sends timely alerts and avoids false alarms isn’t really amenable to a heat test
This is an example where we made an investment based on initial sentiment and intuition — it’s a great feature that in addition to being useful has fueled other partnership opportunities.
In closing
This article describes a methodology and toolbox for accelerated learning.
Part of the game is to get the team on board not just with A/B experimentation but willingness to expose customers to concepts that are half baked (or not baked at all).
That requires taking greater risks in small quantities, and not everyone is comfortable with that.
Nothing wrong with that, but if you ever build something expensive that flops, you’ll find yourself wondering whether you could have figured it out faster and cheaper³.
Footnotes
¹ In fact if you failed to test prior to building something, you could expect to be called out — a good example appears in this podcast with Vaibhav Sahgal describing the rollout of fast play in Words with Friends. [edit: Marc Pincus also refers to this example in chapter 3 (“The MVP Trap”) of his new book]
² Yes of course this could have been worded in a more appealing way (“solve the problem of can’t find a taxi”), I state it this way this for humorous effect.
³ Mark Pincus recently announced that he has a book forthcoming — am looking forward to it. Mark was certainly an inspiration for many of the ideas in this write-up. Thanks Mark!
