Keith Schacht’s Weblog

For the past seven years I’ve been building Mystery.org. Our mission with the company has been to make it possible for children to explore all the questions they have. We help children stay curious by creating better explanations.

Our main product was Mystery Science (and the newer Mystery Doug, which got folded into mysteryscience.com). This is what we’ve become known for. Children use this product in schools. It fits well with the classroom environment and helps teachers field all the questions that students ask them. It’s become the most widely used science resource in elementary schools across the U.S. It’s used in more than 50% of elementary schools, reaching more than 4 million children every month.

For the past seven years I’ve been building Mystery.org. Our mission with the company has been to make it possible for children to explore all the questions they have. We help children stay curious by creating better explanations.

Our main product was Mystery Science (and the newer Mystery Doug, which got folded into mysteryscience.com). This is what we’ve become known for. Children use this product in schools. It fits well with the classroom environment and helps teachers field all the questions that students ask them. It’s become the most widely used science resource in elementary schools across the U.S. It’s used in more than 50% of elementary schools, reaching more than 4 million children every month.

For the past year we’ve also been working on a new direct-to-consumer product, focused on the same mission. Outside of school, whenever a child has a question, we want them to be able to independently explore anything they’re curious about. Many people are surprised when I tell them that this is a problem. But children (10 years and younger) don’t use web browsers, they don’t google, and they don’t search. The internet as we know it is unusable for children.

In 2020 I began incubating a new startup inside a growth-stage company. We have 50+ people working on the school business and still have lots of opportunity ahead. We had just a couple people working on the new consumer product. Recently, I decided that the benefits of running two businesses inside the same corporate structure wasn’t worth the complexity.

Mystery Science has been acquired by Discovery Education (EdSurge, TechCrunch, Discovery Ed, and others reported on this). As part of the acquisition, I spun out the consumer product into a new company, The Explanation Company. Our website and first product are in the works.

I’ll still be involved in Mystery Science as an advisor to help the Mystery Science and Discovery Education team reach a lot more children in schools. My full-time focus is now with The Explanation Company. We’re creating video explanations for every question children have. We’re going to make it possible for them to independently explore anything they wonder about.

Short version:
We made a thing called Mystery Science. It’s starting to get big!

Short version:
We made a thing called Mystery Science. It’s starting to get big!

Longer version:
If you haven’t heard, over a million children experienced Mystery Science in the classroom last year. Every month, teachers in more than 10% of elementary schools in the U.S. use the program.

I remember four years ago flying down to the L.A. area to watch Doug in the classroom. He’d recently been voted one of the top teachers in southern California, and his science classes were consistently voted the students’ favorite class at school. He was doing something different and I wanted to see this for myself.

Over dinner that evening I said to Doug, “You’re helping these children learn how to solve problems for themselves — how to figure things out. This is a skill they now have for life. But every year, you’re reaching… what? A hundred kids? Every child in the country should be learning science this way. We have to bring this to millions of children.”

And so Mystery Science was born. It was 2014.

“Mystery Science is the best thing to come along in my 24 years of teaching. Whoever designed these lessons has made it so appealing to both teacher and student. I get just as excited as the kids every time I do a lesson.”

— Kathy Thibodeau, 5th grade teacher

Today marks the end of the 2016–17 school year and every day we are blown away by the outpouring of gratitude and excitement we receive from teachers and students. Nearly half a million children get their science lessons from Mystery Science each month. Business is strong, more than 4% of elementary schools are paying customers without us having a single sales person on our staff.

But this isn’t fast enough for us. The only thing holding back human progress is the number of people capable of solving problems. Fixing STEM education is the #1 way to create more problem solvers in the next generation.

The Mystery Science team has grown to 19 people. We’re now looking for experienced leaders in every area of the company as we scale.

To accelerate our growth, we’ve joined Y Combinator and are announcing that we’ve raised $2M in funding. As for where this is going to take us? Stay tuned…

If you walk into a casino and on a whim bet your entire life savings and you win. Was it a good decision? No. Even though many would congratulate you on a job well done, it was a bad decision. Or the opposite example of simple expected value. You make a decision that has a 90% chance you’ll lose $5, but a 10% chance you’ll receive $1000. Each time you make the same decision, the expected value of that decision is $95.50, even the times you lose!

Any individual decisions can be badly thought through, and yet be successful, or exceedingly well thought through, but be unsuccessful, because the recognized possibility of failure in fact occurs. But over time, more thoughtful decision-making will lead to better overall results, and more thoughtful decision-making can be encouraged by evaluating decisions on how well they were made rather than on outcome.

If you walk into a casino and on a whim bet your entire life savings and you win. Was it a good decision? No. Even though many would congratulate you on a job well done, it was a bad decision. Or the opposite example of simple expected value. You make a decision that has a 90% chance you’ll lose $5, but a 10% chance you’ll receive $1000. Each time you make the same decision, the expected value of that decision is $95.50, even the times you lose!

Any individual decisions can be badly thought through, and yet be successful, or exceedingly well thought through, but be unsuccessful, because the recognized possibility of failure in fact occurs. But over time, more thoughtful decision-making will lead to better overall results, and more thoughtful decision-making can be encouraged by evaluating decisions on how well they were made rather than on outcome.

The above quote is from this section of a book, Be The House, that a friend sent me. I’ve been thinking this way for awhile, although I would not have said it quite so eloquently.

The swing for the fences success in Silicon Valley get a disproportionate amount of attention, but over the years I’ve been surprised at the number of very successful entrepreneurs I’ve met who quietly make a million or more in profit a year or have had a string of small to medium business that they’ve sold. This doesn’t mean, don’t think big, just be diligent, focus on facts, and tip the odds in your favor—be the house.

In recent years there have been some great resources on metrics for startups, in particular Dave McClures metrics for pirates and Andrew Chens articles on user acquisition and KissMetrics article on conversion. Over the last few products I worked on I synthesized this into a core model that I found very helpful. Here are some questions I had difficulty understanding early on that led to my approach:

• You notice that your power users all have taken some action (e.g. filled out their profile) so you try to encourage all users to fill out their profile to get them more hooked on your product. Does this actually help?
• You have 24 hours of downtime, the next day you come back up your traffic is down. Will this have a long-term effect you need to worry about?
• You have 100K uniques per day and so does your competitor, but are these 100K people who come back every day or 700K people who each come once per week? Does it matter?
• You turn on a new advertising campaign and see your # of unique visitors per day start to increase, you assume that this will continue increasing so long as you keep the ads running, right?
• You start having email deliverability problems (or Facebook turns off notifications) so you can’t notify users of new activity on the site. The # of unique visitors decreases slightly but you’re not too worried, should you be?

In recent years there have been some great resources on metrics for startups, in particular Dave McClures metrics for pirates and Andrew Chens articles on user acquisition and KissMetrics article on conversion. Over the last few products I worked on I synthesized this into a core model that I found very helpful. Here are some questions I had difficulty understanding early on that led to my approach:

• You notice that your power users all have taken some action (e.g. filled out their profile) so you try to encourage all users to fill out their profile to get them more hooked on your product. Does this actually help?
• You have 24 hours of downtime, the next day you come back up your traffic is down. Will this have a long-term effect you need to worry about?
• You have 100K uniques per day and so does your competitor, but are these 100K people who come back every day or 700K people who each come once per week? Does it matter?
• You turn on a new advertising campaign and see your # of unique visitors per day start to increase, you assume that this will continue increasing so long as you keep the ads running, right?
• You start having email deliverability problems (or Facebook turns off notifications) so you can’t notify users of new activity on the site. The # of unique visitors decreases slightly but you’re not too worried, should you be?

What follows is a detailed how-to for analyzing your product’s customers.

The key insight is this: the most important thing is how many unique customers you have, and by implication, knowing what causes any change in this number. And by customers I don’t mean visitors or pageviews, I mean the subset of those people that will be long-term repeat users. Knowing how to measure those customers is the key concept that connects your pageviews, visitors, conversion funnel, retention %s, and all those other numbers you’re collecting and not quite sure how to make sense of. This is the heart of your products and business’s success.

Here is the simple model:

The first step is to figure out how many new customers you add each day and to do this you have to define what makes a customer.

People seeing your product for the first time go through a series of first-time-visitor steps and then start using your product (landing page, registration, inviting their friends, etc). People usually refer to this series of steps as their new user funnel and the end of the funnel is usually some step like uploading their first photo or completing the first level in your game—some core action for your product. But where should your funnel end? This is actually a critical question that a lot of people get wrong; they misdefine the point of transition from a new visitor to an actual customer.

Here is how to figure it out. First, hopefully you’re already measuring the conversion from each step in the funnel to the next step: 50% of people who see the landing page click the sign up link, 60% of people who start signing up complete the process, etc. Ask yourself, when someone has been using your product for awhile, what count as a meaningful return visit? Say you have a photo sharing website. If someone returns but only looks at one page and doesn’t click anything, that probably doesn’t count. But if they come and get sucked in and view 5+ unique photos, I’d say that counts. Or if they view one photo but they comment on it, that probably counts too. Once you’ve defined a meaningful visit then you need to work backwards and find the point where % chance of a repeat meaningful visit is 95+ %.

To switch examples, say you created an online game. After your user completes the sign-up process and takes their first few turns, there is still a good chance they will not return. However, those users who have come back on 100 unique days (where they took a turn on that day) should have a near 100% chance of returning the 101st time to take a turn. Find the point in between where the drop-off hits 5%, meaning 95% of people who reach this point will return. At this point you can call someone a customer. In a game I worked on once, after someone cast 24 spells and had visited the game on at least two unique days, they were hooked. We only lost 1% of these people each day. This point was the end of our new user funnel.

(Note: if you don’t have a very sticky point like this, you may not have created a high-value product yet)

The second step is to figure out when you’ve lost a customer. After you’ve converted them from a visitor to a customer, by definition, you know you have a high likelihood of them engaging with the product again, but even these customers may not visit every single day. At what point have you lost them?

You should have an intuition about the interval between meaningful visits that you care about; start there and look for an unexpectedly long gap. In the game I mentioned above most people played every day (it was designed with this intent). A month after launch I grabbed all my customers who had played today and had been around for at least three weeks. I looked at how long of a break these people took. Lots of people skipped a day of playing, many skipped two or three consecutive days of playing, but very few took people ever took a 4 day break and came back again so after 4 days I considered a customer to be dead. For a second example, consider a product like Shazam on the iPhone (it identifies a song you are listening to). I use this product once month. Assuming this is okay to Shazam as a company (meaning that they can still build a successful business on people like me) then their lost point will be much different than with my game. I bet some Shazam users skip a month or two, but if you’ve skimmed four months you’ve probably forgotten about Shazam and have a very low chance of returning.

Carrying Capacity

At this point, every day you know how many customers you gained, how many you lost, and how many total you currently have. With this data you can now calculate your carrying capacity.

Lets say you have 400 new visitors each day (i.e. from Adwords or Facebook Ads) and 50 of them turn into new customers; and of your current customers you lose lose 1% of them each day. Your product will continue to grow until you reach 5000 customers and then it will stop. Why? Because the # of customers you add each day is mostly independent of your total audience size, whereas the # of customers you lose each day will steadily increase until you’re losing the exact same number of customers you gain each day. At this point you plateau.

Once you have reached this equilibrium state, if you increase your daily ad spend so you’re acquiring 75 new customers each day you’ll start growing again until you reach 7,500 total customers then you’ll stop. And conversely, if you keep your # of new customers the same but figure out how to get your daily loss to 0.66% you will also grow to 7500 customers and then level off.

Carrying Capacity = # of new daily customers / % customers you lost each day

(The reality is a little more complex than this because the # of new customers you get each day is influenced to some extent by the total # of customers you have, but you can tease out those effects later.)

Enhancing the Model

The next thing to understand is where your new customers are coming from. There are 3 possibilities worth isolating:

1. This is a brand new person you’ve never seen before (usually from advertising, SEO, or referral)
2. This is a former customer you considered lost or dead that has been resurrected
3. This is someone who tried your product in the past but never became a customer; now they’ve come back and converted (I call these the skeptics)

Here is the original diagram expanded. In this picture you’ve added 50 new customers today but you isolated the three different sources (new, skeptics saved, resurrected). You also track the intermediate pools (skeptics and dead customers).

There is one more expansion I find helpful. Within the 5,000 total customers pictured above, some % of those returned today so they’re as good as new. Here is what my full customer dashboard with that addition:

Virality

Virality is the one concept which has the biggest potential to impact this model. Most people look at virality in a simplified way. You take all the new people who tried your product today and see how many friends they brought (either today or on any day in the near future). If 50 people came today and they inspired 25 of friends to come over the next few days then you’re K-factor is 0.5 (each new visitor will bring 0.5 friends). This doesn’t have any impact on the model I explained above. See the 400 new visitors? If your K-factor is 0.5 this just means you only had to buy 266 of them, the other 134 came for free. Correction: You only had to buy 200 of them, the other 200 came for free. I was only considering the second generation of free visitors, but these will bring a third generation, which will bring a fourth, etc. A K-factor of 0.5 means 1/(1–0.5)=2 times the number of users you bought will ultimately come. Any K-factor less than 1.0 is essentially just a discount on advertising spend (0.5 equals a 50% discount).

What most people overlook is that retention has a huge effect on your K-factor. Sure, most of the invitations people send occur in the first visit or first few days of using a product; this is why you initially spend so much time optimizing your new user funnel and trying to get people to refer their friends as part of this. However, after this first-visit kicker most users who come to love your product and stay around for a long time will continue to refer a steady stream of their friends. This can give you a nice bump to your lifetime K-factor and give you a greater chance of getting it over 1.0.

Note: In the case that your first-visit viral coefficient is greater than 1.0, things can be very confusing without having a clear model like the one above. You could be converting 0% of your visitors to customers (i.e. all the water going into your bucket is instantly flowing out), but you’re steadily turning up the water since each new visitor is bringing more than one friend. You will see a steady increase in all your naive measurements of success (i.e. unique pageviews and visitors per day) but in reality you may not be acquiring any customers.

Benefit

In closing, lets revisit the example questions I raised at the start. Here is how this model helps:

1. You notice that your power users all have taken some action (e.g. filled out their profile) so you try to encourage all users to fill out their profile to get them more hooked on your product. Does this actually help? Without this model its easy to mislead yourself into thinking it helps. You will probably be able to increase the metric but it may just weaken the correlation with power users. However, with this model you just watch your loss % each day and if it doesn’t change then you haven’t had any net impact on retention.
2. You have 24 hours of downtime, the next day you come back up your traffic is down. Will this have a long-term effect you need to worry about? Even if your traffic is lowered for a few days, all you have to do is measure your new visitors per day and lost customers per day and you’ll see if your carrying capacity has changed. If it has not changed then you don’t have to worry, you know your traffic will rise back up to your carrying capacity.
3. You have 100K uniques per day and so does your competitor, but are these 100K people who come back every day or 700K people who each come once per week? Does it matter? If you’re incorrectly caught up in # of unique visitors per day then this does seem like an important thing. And in fact, if you realize your visitors are not returning as often as your competitors you may even be tempted to spam them regularly because this will increase # of unique visitors each day and can give you the illusion of improving things. In reality a move like this would probably increase your % lost each day and hurt your carrying capacity but you wouldn’t notice this for awhile because the increased # of uniques each day would mask the harm. However, with the model your emphasis is on # of customers not # of visitors. You’ll quickly realize that you don’t care how frequently people visit as a primary factor; if it doesn’t impact # of new customers per day or % of lost per day then you haven’t actually helped your product.
4. You turn on a new advertising campaign and see your # of unique visitors per day start to increase, you assume that this will continue increasing so long as you keep the ads running, right? Nope, you’ll level off once you’ve reached your new carrying capacity.
5. You start having email deliverability problems (or Facebook turns off notifications) so you cant notify users of new activity on the site. The # of unique visitors per day decreases slightly but you’re not too worried, should you be? This is similar to question 3, it may or may not be important but you’ll quickly be able to tell by focusing on the core numbers. Increasing # of unique visitors per day does not necessarily lead to more total customers.

Its been a few years since I’ve been in start-up mode: dissecting great products to learn, generating lists of ideas, and meeting lots of new people. In undergoing this process I’m reminded of my fundamental approach to developing a new product and how unusual it seems to be among other entrepreneurs. (This is actually not a new insight, I’m reposting this from a blog post a few years ago.)

When I have a new idea that I’m intrigued by that I want to bring into the world, the two most important first features I stay focused on are: (1) simplicity, and (2) community.

Its been a few years since I’ve been in start-up mode: dissecting great products to learn, generating lists of ideas, and meeting lots of new people. In undergoing this process I’m reminded of my fundamental approach to developing a new product and how unusual it seems to be among other entrepreneurs. (This is actually not a new insight, I’m reposting this from a blog post a few years ago.)

When I have a new idea that I’m intrigued by that I want to bring into the world, the two most important first features I stay focused on are: (1) simplicity, and (2) community.

Let me explain. 1) Simplicity. Too many people create too many features way too soon. The most important feature in the first version of your product is that its simple. In one sense this is the absence of features, but in a very real sense simple is a feature in and of itself. Simple is achieved by distilling your concept down to its essence, by including only those features which are absolute core to your idea. Achieving simple is a very selective cutting of features: you don’t necessarily leave the easy features and cut the hard ones, you leave the essential/core features and cut the peripheral ones. Why? Because the most important goal at this stage of product development is that when a potential customer sees your product, they get what it does. You want them to almost instantly recognize the problem and understand how this product can potentially solve it. This first version of the product is a conversation starter.

When you’re trying to teach kids what a mammal is, you don’t start with the egg-laying platypus as the example. Likewise, you want your first product centered firmly around a quintessential use case. Lots of features you’ll add down the road will be added to address edge cases, real-world scenarios, but they’re a distraction at this first phase where clear understanding is the first goal. Its not so important that the first version of your product actually solve the problem, it doesn’t have to be ready for someone to dive in and start inputting real data and using it on a daily basis, the purpose of the first version of your product is that you can show it to actual customers and they get it and get excited about it.

This leads directly into feature #2, Community. There are lots of different forms that community can take at this stage: a forum, a regular conference call, group sessions; but the essential is that you create a regular on-going interaction between an early group of customers that you get to eavesdrop on and participate in. Note that this is not a bunch of people talking only to you, you want a small group of real customers talking about your product among themselves, brainstorming ideas, hearing what each other think and disagreeing.

Your goal at this stage is to build an early version of your product that is distilled down to its essence and then get a group of vocal passionate real customers using it, and then, only then, do you start really building the product with them. You start listening and responding and iterating. This distilled essentialized first version of your product is situated at crossroads. There are a lot of subtly different but significant ways you can take this product, but you don’t want to add the small features that push it down one of these paths. You want its essentialized form to stand on its own feet, and then you want this early group of customers to push it in a particular direction.

What happens if you dont achieve this? What happens if you build a first version of the product and no one really gets excited about it? Well, this means one of a few things. Either: 1) this is the wrong group of test customers, 2) this is the right group of customers but youre wrong about their being a real underlying problem, 3) theyre not understanding how this particular solution solves their problem.

Youre allowed to try again with a slightly tweaked and modified version of a distilled product, but dont make the mistake of thinking my customers need more real working features and start building, building, building. Youll know that youre making this mistake if, as you build more, your group of engaged passionate beta customers gets smaller and smaller. Youll know youre doing things correctly when youre group of engaged passionate beta customers starts getting bigger and bigger, both because some of the initial people who werent quite clear on it start to really get it and get excited, and because those who are excited want to start showing it to their friends.

Some tips to increase your chance of achieving this:

• Keep your product ugly, don’t get slick graphics involved at this stage. You’ll actually get better, more honest feedback if your product looks and feels a little rough. Implicitly this communicates that its a work in progress and they’re helping to shape it. If you show them a really slick product and they don’t quite like it, they feel worse about criticizing it so they wont, or you’ll start to hear feedback like, “I can see how some people might like this” which is code for “I don’t like it, but a lot of effort went into this so I hope someone out there does!”
• If essential features are hard to program, use lots of screen shots, mock-ups, and fake data. Real customers generally have bad imaginations, so even though you tell them, “XYZ will happen when you click here,” and the customer says “oh yea, I get it” they’re either being nice and have no clue what you mean, or they’re imagining something but its very different than what you’re imagining. You don’t want an illusion of understanding. Create a mock-up with fake or static data, put the most weight on feedback to stuff they’ve actually seen and clicked on, not on stuff you hypothesized about with them.

Pinterest is the best new social application I’ve seen in awhile. Exploring it has rekindled a lot of thinking about what it takes to get users to create online content. This is not an analysis of Pinterest in particular, but the mental framework I use to think about this design problem. (This is a re-post from my blog circa 2012).

The naive view of encouraging content production is to create really flexible creative controls so that your tool has a wide range of uses. The reality is that a more constrained container encourages production. Think about a coloring book versus a blank sheet of paper. Not only is the freedom of a blank piece of paper paralyzing, but the pre-drawn outline makes your final creative output better.

Pinterest is the best new social application I’ve seen in awhile. Exploring it has rekindled a lot of thinking about what it takes to get users to create online content. This is not an analysis of Pinterest in particular, but the mental framework I use to think about this design problem. (This is a re-post from my blog circa 2012).

1) The creative container

The naive view of encouraging content production is to create really flexible creative controls so that your tool has a wide range of uses. The reality is that a more constrained container encourages production. Think about a coloring book versus a blank sheet of paper. Not only is the freedom of a blank piece of paper paralyzing, but the pre-drawn outline makes your final creative output better.

Let me elaborate on those two separate principles, first on the constraints. Think GeoCities vs Blogger. GeoCities was an easy way to create websites. You clicked to insert a page and had a WYSIWYG editor that let your imagination run wild. Then blogging came along. What was so great about blogging was that you were given a template; this wasn’t for creating just any website, it assumed your site was a series of posts, each with a title and body, ordered by date down the page. This spurred an explosion of content creation because a post template is helpful creative suggestion, yet its still incredibly versatile in its uses. Then in 2007 Twitter came along. For many people, its even easier to fill 140 characters than it is to write a complete post. Constraints breed creativity.

The second principle of the container is that it serves as imagination scaffolding. The constraints of your container, if done well, make the final creation more awesome than if the person created it from scratch. Blogging software assumes all the sites will be a list of posts so they can design beautiful themes for you to choose from. Paste ten random 140 sentences into an email and your friends will think your drunk, but call them Tweets and put them on your Twitter profile page with a customized background and the pretty icons and they become pearls of wisdom. Character creators in video games are another great example of this. By allowing you to customize the hair, eyes, skin, and clothes of this character there are millions of unique combinations but everyones character ends up looking really professional. Give them scaffolding to help creations turn out better.

Its important to think a lot about the creative container youre offering to users. A friend who is an avid Twitter user once remarked, “That would make a great Tweet!” when someone said something funny. Twitter power users filtering the world for Tweetable moments. Your users are going to be spending a lot of time thinking about what they can fill your containers with.

2) Community feedback and validation

When presented with a new creative container there is a learning curve in figuring out how to produce great content within it. Its important that youve architected a good feedback loop so people can hear the opinions of others to improve, and to celebrate the great creations when one is crafted.

Blogging is an easy example to see with its comment thread beneath each post. Within video games where you have created a custom character, the feedback comes from your interactions with other players. When chatting with other people in the game or virtual world someones avatar is a conversation starter.

Wikipedia is a particularly interesting example. Their creative container is an encyclopedia article which turns out to be a great container for summarizing your knowledge on a subject youre an expert of. But what you may not realize is that the feedback loop is a critical part of encouraging Wikipedia editors to continue creating great content. On every article is a Talk tab at the top where you can see the editors getting feedback about what theyve written.

But its not enough to simply allow feedback, its also important to reduce friction in the feedback process and think about the constraints youre setting up within the feedback mechanism. Facebooks Like button lets you endorse someones post with a single click and Amazon reviews can be marked as helpful. Following someone on Twitter makes it clear youre interested in what this person has to say. All of these are easy and constrained. Even comments, which are relatively unconstrained: dont allow titles of the comment, usually have a max length to keep them short, links are often prohibited to keep the discussion focused on the original post. And most of the time comments are not multi-threaded—you arent supposed to reply to another commentor, youre there to give feedback to the original creator.

Your feedback process will drive the quality of your creations and the longevity of your creators. Figure out how to encourage it but to simultaneously keep it enjoyable and constructive.

I should write a full analysis of Pinterest, but they’ve done an amazing job with their creative containers and with their feedback loop.

It’s accepted wisdom in the startup space that you begin by testing your product on early adopters and over time you work to expand to the mainstream users in your market. But if you’re struggling to bridge this gap, maybe what you thought were early adopters are actually just mainstream users in a small niche. The people who are using your product first may not be the beginning of a larger opportunity.

How do you tell the difference? I was curious about this myself and last week I had a chance to talk with Steve Blank. I asked him this question and he shared some insights that clarified my thinking. This is what I took away from our conversation (these are not Steve’s words).

It’s accepted wisdom in the startup space that you begin by testing your product on early adopters and over time you work to expand to the mainstream users in your market. But if you’re struggling to bridge this gap, maybe what you thought were early adopters are actually just mainstream users in a small niche. The people who are using your product first may not be the beginning of a larger opportunity.

How do you tell the difference? I was curious about this myself and last week I had a chance to talk with Steve Blank. I asked him this question and he shared some insights that clarified my thinking. This is what I took away from our conversation (these are not Steve’s words).

You start by talking to a random group of potential customers in your target market. A small number of them are willing to try your product and the rest of them have some reason for saying “No.” You need to focus on the reason that some customers say “Yes” and some say “No.” What are the attributes which differentiate the adopters from the decliners?

Are these attributes temporary conditions which will cease to be relevant over time? Or are they attributes which will keep you relegated to a niche market.

For example, maybe your decliners like the value your product offers but want backwards compatibility, validation from other high profile customers in the space, SLAs, or time to get comfortable with the idea of change. Your adopters agree these extra details would be nice but they are willing to use your product without them. This is a classic case of early vs late adopters. The extra requirements of your late adopters will naturally be addressed as your product evolves while catering to your early adopters.

But if your decliners don’t feel like your product addresses a pain they have or are indifferent to the idea whereas your adopters are excited by it, don’t make the mistake of framing this as early vs late adopters. You don’t have reason to expect that your decliners are every going to come around and you may just have a product that caters to a small niche.

As an entrepreneur you spend a lot of time telling other people about your business. Early on you do this to learn—you solicit feedback from people smarter than you. Later your goal may be fundraising or recruiting—but you will still get a side of feedback and advice for free.

There is a common pattern you will hear from people outside your business which is important to identify or you risk drawing the wrong conclusion from it. The telltale sign are statements like this:

As an entrepreneur you spend a lot of time telling other people about your business. Early on you do this to learn—you solicit feedback from people smarter than you. Later your goal may be fundraising or recruiting—but you will still get a side of feedback and advice for free.

There is a common pattern you will hear from people outside your business which is important to identify or you risk drawing the wrong conclusion from it. The telltale sign are statements like this:

“Companies in your space need to …”
“Given the stage you’re at, you should …”
“No one with your background has …”

When you’re talking with wise people, these statements are generally true so it’s worth listening. But they are also admissions of ignorance and you must treat them as such. These are attempts to draw a conclusion about your business based on a general pattern you seem to fit. It’s a form of statistical thinking.

Statistics are generalizations about the world which are helpful when you are ignorant of the particulars. As soon as you have knowledge of the particulars, the generalization is mostly useless.

For example, if you find a lump in your neck and you go in for a biopsy, while waiting for the results your doctor may tell you there is a 90% chance it’s cancer based on its size and location. But as soon as the biopsy results come back and it’s negative—the lump is benign—then the statistic is irrelevant. It’s still true: 90% of lumps of this size and in this location do turn out to be cancerous. But the cause behind this pattern is unknown which is why it was possible for yours to be an exception.

In business, just like in medicine, it is helpful to understanding what the typical outcomes are for people who have already walked the path you’re heading down. But only when you are ignorant of the underlying cause that drives the range of outcomes. Want to predict your net margins? Look at competitors in your space. Want to budget for hiring a senior engineer? Talk to startups who have hired similar people and learn the typical salary. Want to evaluate a term sheet from an investor?

Entrepreneurs: seek to change the world. Set out to achieve something no one before you has. Bring about some change in the world you want to see. To accomplish this, you need to figure out key ways to achieve a different outcome than those who have ventured before you. Understanding typical outcomes is a starting point, not a constraint. Most people who give you advice about your business will treat them as laws of nature and think you are naive for thinking they don’t apply to you.

But you can be the exception. The key is to understand the cause behind the patterns you see. Once you understand the cause and can control it, then historical outcomes are irrelevant.

Find the competitor who has exceptional net margins and understand what they’re doing differently. Look to industries where net margins are in the range you’re targeting, learn how they’ve achieved them, and figure out which of these learnings you can apply. Want to hire a senior engineer but can’t afford the typical salaries? Understand what’s driving the price up. Is it because most senior people are older and have kids? Then reach out to the youngest in the field or find those without kids.

Work to understand what drives the typical outcomes if you wish to be atypical. Do this by learning to ask questions that reveal the cause behind outcomes. And seek out the exception cases as key examples to study and learn from.

You can be the exception. Now go forth and change the world.