Getting started: How to adopt a marketing methodology that leads to predictable outcomes based on the scientific method for testing hypotheses.

Over the past 4 years, I have been fortunate enough to test nearly 250 ideas at Pioneer Square Labs. Almost every single one of the 25 spinouts went on to become venture-backed companies and the other 225 were killed. (You can read more about how we kill ideas here). That means that 90% of the ideas we tested were killed. You would think a 90% failure rate would be a bad thing, but it’s actually what I’m most proud of.

Killing bad ideas is critical for us as it means we can spend more time focused on building the good ones.

Thus, in PMP the primary goal is learning, not selling. In the early stages of an idea, we likely have very little real data, so we need to obtain as much data as possible quickly — even it is course — to accomplish learning as fast as possible.

This is very different than PMF. In PMF you have trailing indicators like LTV:CAC, NPS, sales velocity, reviews (among others) that allow you analyze and understand market fit. In the first few months, you don’t have any of this. There are scores of great blog posts on PMF, but we needed to create a system that could produce leading indicators that lead to predictability around PMF.

For this reason, we developed what we call Product Market Pull (PMP) in which you systematically test the market for resonance before you build. In PMP, you build test-driven evidence that support (or refute) your hypothesis on who wants your perceived product or service and why.

Below are fundamentals of PMP:

• In Product Market Pull (PMP) the primary goal is to learn, not sell.
• Thus, the foundation of PMP is building our learning system that can produce leading indicators towards PMF.
• We spend our time in the beginning, designing our test matrix of customer segments and core positions / value props.
• This allows us to test against specific hypotheses using discrete elements.
• Quality test design gives us clarity into specifics, even in black boxes like Facebook/Instagram.
• Consistent test design allows us to scale testing dramatically and find signal faster.

It’s simpler than it sounds. We want to be able to incrementally collect data in a consistent way that compounds. Meaning, as we run consecutive tests, we want the evidence to grow in a way that makes it easy to believe there is a company to build or kill it quickly.

Below is our (overly simplified) playbook for test-driven design at the start of a project:

1. Define your who, what, why statement— articulate what problem do you think your customers want you to solve.
2. Be Scientific — Create a testable hypothesis against the problem.
3. Build an Audience Matrix (of real people and companies) — Build audiences that acutely represent your vision.
4. Test For Resonance — actually go into the market and run experiments that prove or refute your hypothesis.
5. Measure and Iterate — evaluate your results and see where you were correct.

1. Define Your ‘Who, What, Why’.

We have all been burned on building things people really don’t want or need.

This is especially true when ideating in a rapidly evolving technological environments like ML and AI, where we tend to focus a lot of our energy at PSL.

Before we start building screens and writing code, we need to prove that there is a real customer, who believes our product or service is going to solve problems they have. This is our “who → what → why.”

At the broadest possible level, being test-driven through PMP can help produce answers to the following questions:

• Who? — which specific customer segments responded the strongest?
• What? — what problem did they respond to?
• Why? — which core benefits resonated the most with each customer segment?

For the audience (“who”), we focus on how to test real customer response to new ideas. While we always interview those we believe to be in our “ideal customer profile” (ICP), through voice of customer interviews (which we will discuss in detail in a later article) we go beyond interviews to accumulate significant amounts of customer demand data. This includes everything from surveys to email to demand-gen across platforms like Facebook/Instagram, Google, LinkedIn and Reddit. Above all, we want to ensure that we are precise in the audience we test against.

⚠️ Note: We will go into extensive detail on how we run tests across various digital channels in later installation of “Market Research at Scale”.

For the “what”, we focus on validating the solution (not just the product) we envision building. Typically, we make use of very helpful methodologies like Job-To-Be-Done, (JTBD) to articulate what the consumer’s real goal is and how our solution would help them achieve this.

💡 If you are not aware of JTBD, you don’t need it to execute the ideas discussed in this article, but it is a great way to think about.

Below is a very basic representation of disambiguating products from the job to be done. As you can see from the example, consumers don’t know they want an iPod, they are focused on an outcome they can achieve with an iPod.

As Tony Ulwick states: “Markets aren’t defined around products, they are defined as groups of people trying to get a job done.” Thus, we want to be sure that our what is solution-mapping to the consumers goal.

For the “why”, this is focused on the messaging to communicate the value of the product or service. We focus on segmenting messages into clear psychological motivator buckets (aka core benefits, or core motivators) so that we know which motivator resonated most strongly with a consumer. Common motivators include: FOMO, Saving Money, Saving Time, Exclusivity, and Convenience.

As we will see later when we run our tests, we use these core motivators as copy variants in our digital marketing to suss out what themes motivate customers to engage.

2. Be Scientific! Create a Testable Hypothesis.

Now that we have our who →what →why defined, we can start laying the foundation of our tests. We like to think of the “who, what, why” as our atomic units, which we can discreetly test against to see if they are true.

Here, we are normalizing our testing data so we can analyze results faster. It’s essentially, good data in == good data out.

In the rest of this article, we will use examples from real PSL tests, that demonstrate of how we translate our atomic units into a workable hypothesis we can acutely test.

Financial Controllers (who) want a ML-based accounting platform when reconciling their books (what) because it will [reduce costs],[improve quality],[improve quality] (why) for their monthly close.

3. Build an Audience Matrix (of real people and companies)

It’s very important to us at PSL that we get accurate data when running tests.

Our typical testing landscape includes:

• Consumer (B2C) — Facebook, Instagram, Reddit, Snap, TikTok, Google AdWords, Google Display, and SurveyMonkey.
• Prosumer — Reddit, Google AdWords, Google Display.
• Business (B2B) — LinkedIn, Quora, Reddit, Google AdWords, Google Display, SurveyMonkey.
• Developer — StackOverflow, LinkedIn.

In this step, we are doing a combination of “bottoms-up” market sizing, and finding the customer segment where our solution finds the most resonance (“identifying the ideal customer profile”). This ensures we have:

• Big (Enough) Market
• Distinct Customer Segments — If you ask any founder who has Product Market Fit who their ideal customer profile is, you will get a very specific answer. We want to get that level of specificity around customer segment earlier in the process.
• Validated Core Benefits / Positioning Statements — We need to show that our messaging tests resonate with our customer segments enough that they compel would-be customers to engage.

The chart below is a very basic representation of this concept. You have a market that is comprised of real customer segments (businesses or people). In order to identify if you have PMP, you test for resonance of your concept, broken out by positioning statements, within each test group in the market.

We now have a framework to test market response across discrete segments, and we can see response by positioning and segment. We can then leverage several different channels to test for response.

4. Test For Resonance (the fun part!)

This is one of my favorite things and have seem it perform magic many times.

Now that we have the elements that we want to validate, we get to test our concept with real consumers in the market and get real data.

For this section, I’ll give you an example of a real idea pitched to us called ChefJoy 👩‍🍳. The idea was that people want to be able to easily rent personal chefs on-demand when they travel & entertain. (Think of it like AirBnB for chefs.)

Hypothesis: Affluent individuals (who) want to easily book a local, private chef when traveling or entertaining (what) because it’s more exclusive than a restaurant (why).

So, how do we test this concept in a PMP world? We know we want to reach affluent people and see if they truly want to rent a chef when they travel. To test this hypothesis, used Instagram as our channel, and tested the concept with over 100,000 people.

Here is the process we used specifically:

• Came up with a name — We built a brand called ChefJoy, aimed at being “the easiest way to hire a personal chef when traveling or entertaining.”
• Created a website with landing pages for each customer segment and positioning statement — We built several different landing pages for each core benefit, with specific visuals for each audience segment and scenario (traveling or entertaining).

Since we are testing resonance with a consumer product in which the target customer is affluent, we have a great option with Instagram, (with an estimated 900 million users), and arguably the strongest consumer affinity social network.

With Ads Manager, Facebook’s platform which allows you to run ads on Instagram, Facebook, Messenger, etc… we can put in the qualifications of the audience we are trying to reach, then get an audience size. This is fantastic, because FB is giving us actual market data that we can use to build our models as we will see later.

• Run ads to several segments on Facebook and Instagram — We ran dozens of ads to 100,000 people, across our key segments, measuring CTR, CPC, CPM, and scale.

• Collect “Early Adopter” Signups — We drove thousands of people to landing pages and measured their willingness to use an app to find a personal chef.

⚠️ Note: We always make this very transparent for consumers, letting them know the “service is not yet available in their area”, so they are not misled. Many times these early engagers go on to become customers, so it’s important we retain their trust.

Traditionally, marketers would just think of this as a campaign, but the point here, as it relates to PMP, is that we are actually just running an experiment, to learn as fast as possible about the market pull on AirBnB for Chefs (what), against specific audiences (who), seeing if they resonate with specific benefits (why).

5. Measure and Iterate

After we run our tests for several days (usually about a week) and have sufficient data, we get to see the results! (To be honest, I peek several times a day, but never edit as that would be cheating.)

At the end of the day, we need to believe that the business we want to build can achieve venture scale. At PSL, that means that the business has the potential do generate hundreds of millions of dollars in revenue five years from now, so we can work backwards from that number and understand the types of price points and customer volume required to achieve that goal.

Below are the results of the first set of tests. (We typically have dozens of columns of data, but we will keep it simple at this point).

As you can see we now have clarity into both our market segment response, as well as the messaging response within each segment.

We could also express this in a different way, according to the framework we outlined previously to analyze our market pull matrix.

Based on the first hypothesis, we were not seeing performance indicating there is significant market pull. We’ll discuss the details in a moment, but when you begin to model out these economics, they just don’t work to build a business with profitable unit economics.

⚠️ Note: We sometime get asked about Stat-Sig, or the statistical significance of this data. At this stage, we are really focused on finding early indicators that we can double down on later. As a baseline, we try and test 5%–10% of the audience population during PMP. Meaning, if the audience size was 380,000 we would try and test between 19,000–38,000 people.

Since the point of PMP and test-driven design is to rapidly iterate and test we again, we then created a second hypothesis, again focusing on who → what → why as we would with PMP. This time, we focused on daily needs as opposed to travel.

Hypothesis 2: Affluent individuals (who) want to easily book a local, private chef when entertaining (what) because it’s more [exclusive],[convenient] than a restaurant (why).

The thinking was that individuals would more likely take advantage of on-demand services like a personal chef if it was easy to book. But, this also proved to be mostly-false, showing very low pull from the market. That said, we started to see signs of life for new parents, focused around the “convenience” of having an on-demand chef. These early pull insights often lead to key iterations.

Below are the results of the first set of tests, including the first and second hypothesis.

Now, from the data we collected, we can actually build a pretty basic but highly informative model from our tests, based on our atomic units from PMP.

📈 This is so important because, we can now squint at the real economics of the business and start to think critically. As you can see below, using a testing population of 45,000 people, we were not able to produce results that would support a great business, and our core hypotheses audiences actually performed the worst!

So, the question is, should we build ChefJoy?

Therefore, with PMP, and using the Scientific Method approach, we can answer the following questions that directly align with the foundational elements of PMF around our ChefJoy idea.

• Is the market big enough? Thanks to Facebook, we can see that we could reach audiences in the millions. So, the market size would be big enough if we could engage them and if it were cost-effective. 😃
• Which customer segments showed interest in the concept? In this case, we saw very little resonance, with the exception of new parents. None of our audiences (0/4) showed interest. 😐
• Which positioning statements or core benefits resonated? Again, there was very little resonance (0/4) with any of the core positions outside of new parents / convenience 😳

We learned that from the segments that we identified, we didn’t see audiences showing early excitement to solve this problem. This means that in the future, it would likely be expensive to acquire customers (and thus have poor unit economics), and we should expect high churn.

So, after testing against 2 hypotheses, and 4 audiences, we have signal from a real market on pull. It’s pretty obvious, the results don’t look appetizing 🤢. (My one and only joke, I swear)

To wrap things up

As we begin to work through this guide, the point is that

• In Product Market Pull (PMP) the primary goal is learning, not selling.
• Thus, the foundation of PMP is building our learning system that can produce leading indicators towards PMF.
• We spend our time in the beginning, designing our test matrix of customer segments and core positions / value props.
• This allows us to test against specific hypotheses using discrete elements.
• Quality test design gives us clarity into specifics, even in black boxes like Facebook/Instagram.
• Consistent test design allows us to scale testing dramatically and find signal faster.

Next Up:

Customer Segmentation — In the next section, we will be focused on how to build out market segments using data providers, in house tools and frameworks to think about how you segment your market.

5 Steps to Embrace Test-Driven Learning to find Product Market Pull (PMP). was originally published in Product Market Pull on Medium, where people are continuing the conversation by highlighting and responding to this story.

I am going to be releasing a10(ish)-part deep-dive on how we’ve learned to find PMP using customer interviews, messaging tests, digital marketing, data mining, growth hacking, python, chrome extensions, google apps script… all in that order more or less.

Over the past 4 years, I have been fortunate enough to test nearly 250 ideas at Pioneer Square Labs (BTW, how fun is that?). Almost every single one of the 25 spinouts went on to become venture-backed companies and the other 225 were killed. (You can read more about how we kill ideas here). That means that 90% of the ideas I worked on didn’t even make it to step 2 but honestly I couldn’t be more proud.

Now, you might be thinking that most of those ideas might have been terrible, but they really weren’t. Many of them were pretty good. Many I personally believed would be great companies and I was sure that people would want them. But the funny thing was people didn’t want them. Even when my ego told me they must want it, I was just selling it wrong. But no matter how many times we ran it though the process, it just came out the same…dead.

This is why I am so proud of a 90% failure rate. It was because we developed a pretty reliable system for getting the pulse of a market before we wrote a bunch of code, designed a ton of mocks, and tried to sell something people didn’t want.

So, What is Product Market Pull?

It’s important to state that what we are doing this early is not looking for signs of the oh-so-discussed Product Market Fit (PMF), but something earlier — a precursor… Product Market Pull.

Where PMF is more of a lagging indicator (CAC declines, retention holds, customer ❤️), PMP is more of a leading indicator.

PMP is looking for:

• Market confirmation a problem exists / your hypothesis is true.
• Customer signal there is demand/need for a solution.
• Clear articulation of who fits this need and why.

Think of PMP as a framework for incrementally learning which customers are most interested in your product and why.

The basic framework Product Market Pull helps you answer is:

• Does anyone immediately resonate with your startup or idea? (Broad market pull)
• If so, who? — which customer segments responded the highest?
• If so, why? — which value props resonated the highest with each customer segment?

By working within this framework, you can create a matrix that guides you towards the early adopters and understand what it is specifically they are interested in. As you can see in the example ‘Resonance Matrix’ below, one of the components of PMP, you get a heat map of resonance across customer segments and value props.

What is the outline for the PMP guide?

In each section we will do a deep dive into approach, tactics, and the technologies we use to accelerate learning through a reusable framework. Topics will include:

• Embrace Test-Driven Learning for PMP — how to build a testing matrix that leads to incremental insight.
• Customer Segmentation — How to build out distinct audience segments using a variety of tools and techniques.
• Rapidly Finding Early Adopters — How to leverage list building and segmentation to rapidly reach out to potential customers for interviews.
• Systematically Interviewing Customer Segments — how to do a proper “Voice of Customer” interview that helps you derive quantitative learnings from qualitative discussions.
• Market Research at Scale — how we use demand-gen to find learn quickly.
• Analyzing the Results — Learn how to slice and dice data in a way that leads to meaningful iterations.
• Finding the Wedge — Developing a heat map for resonance across segments and value props.
• Doubling Down on Signal — Embrace and iterative approach to early learning to help build your foundational marketing.
• Incrementalism Towards Fit — Once you have a framework for incremental learning, it can help you find fit.
• Scaling FTW — Once you have identified the discreet components of your marketing, it’s much easier to scale.

Who is the PMP guide for?

If you are working on new product ideas, either as a startup or as part of an organization, you can utilize PMP as a framework to help understand where their is market pull before committing valuable resources to build.

I have continued to see the significant ROI from having a framework that leads to incremental learning gains, that help show signs of Product Market Fit (PMF). Product Market Pull is a framework to learn faster with more precision to help reduce waste in the iteration cycles and build confidence in who you are building your product for and why.

If this sounds interesting, please follow or reach out as I would love to make this an interactive and collaborative guide to help everyone accelerate learning and find PMP.

The Complete Guide to ‘Product Market Pull’ (PMP) was originally published in Product Market Pull on Medium, where people are continuing the conversation by highlighting and responding to this story.