- You notice that your power users all have taken some action (e.g. filled out their profile) so you try to encourage all users to fill out their profile to get them more hooked on your product. Does this actually help?
- You have 24 hours of downtime, the next day you come back up your traffic is down. Will this have a long-term effect you need to worry about?
- You have 100K uniques per day and so does your competitor, but are these 100K people who come back every day or 700K people who each come once per week? Does it matter?
- You turn on a new advertising campaign and see your # of unique visitors per day start to increase, you assume that this will continue increasing so long as you keep the ads running, right?
- You start having email deliverability problems (or Facebook turns off notifications) so you can’t notify users of new activity on the site. The # of unique visitors decreases slightly but you’re not too worried, should you be?
What follows is a detailed how-to for analyzing your product’s customers.
The key insight is this: the most important thing is how many unique customers you have, and by implication, knowing what caused any change in this number. And by customers I don’t mean visitors or pageviews, I mean the subset of those people that will be long-term repeat users. Knowing how to measure those customers is the key concept that connects your pageviews, visitors, conversion funnel, retention %s, and most all those other numbers you’re collecting and not quite sure how to make sense of. It’s the heart of your product’s and business’s success.
Here is the simple model:
People seeing your product for the first time go through a series of first-time-visitor steps and then start using your product (landing page, registration, inviting their friends, etc). People usually refer to this series of steps as their “new user funnel” and the end of the funnel is usually some step like uploading their first photo or completing the first level in your game (some core action for your product). But where should your funnel end? This actually the critical question that a lot of people get wrong; they misdefine the point of transition from a new visitor to an actual customer.
Here is how to figure it out. First, hopefully you’re already measuring the conversion from each step in the funnel to the next step: 50% of people who see the landing page click the “sign up” link, 60% of people who start signing up complete the process, etc. Ask yourself, when someone has been using your product for awhile, what count as a meaningful return visit? If you had a photo sharing website, for example. If someone returns but only looks at one page and doesn’t click anything, that probably doesn’t count. But if they come and get sucked in and view 5+ unique photos, I’d say that counts. Or if they view one photo but they comment on it, that probably counts too. Once you’ve defined a “meaningful visit” then you need to work backwards and find the point where % chance of a repeat meaningful visit is 95+ %.
To switch examples, say you created an online game. After your user completes the sign-up process and takes their first few turns, there is still a good chance they won’t return. However, those users who have come back on 100 unique days (where they took a turn on that day) should have a near 100% chance of returning the 101st time to take a turn. Find the point in between where the drop-off hits 5%, meaning 95% of people who reach this point will return. At this point you can call someone a customer. In the last game I worked on, after someone cast 24 spells and had visited the game on at least two unique days, they were hooked. We only lost 1% of these people each day. This point was the end of our “new user funnel.”
(Note: if you don’t have a very sticky point like this, you haven’t created a high value product yet)
The second step is to figure out when you’ve lost a customer. After you’ve converted them from a visitor to a customer, by definition, you know you have a high likelihood of them engaging with the product again, but even these customers may not visit every single day. At what point have you lost them?
You should have an intuition about the interval between meaningful visits that you care about, start there and look for an unexpectedly long gap. In my last game most people played every day (it was designed with this intent). A month after launch I grabbed all my customers who had played today and had been around for at least three weeks. I looked at how long of a break these people took. Lots of people skipped a day of playing, many skipped two or three consecutive days of playing, but very few took people ever took a 4 day break and came back again so after 4 days I considered a customer to be dead. For a second example, consider a product like Shazam on the iPhone (it identifies a song you are listening to). I use this product once every few weeks. Assuming this is okay to Shazam as a company (meaning that they can still build a successful business on people like me) then their “lost” point will be much different than with my game. I bet some Shazam users skip a month or two, but if you’ve skimmed four months you’ve probably forgotten about Shazam and have a very low chance of returning. They still need to pick the actual point that their data justifies but this is my initial guess.
At this point, every day you know how many customers you gained, how many you lost, and how many total you currently have. With this data you can now calculate your carrying capacity.
Let’s say you have 400 new visitors each day (i.e. from Adwords or Facebook Ads) and 50 of them turn into new customers; and of your current customers you lose lose 1% of them each day. Your product will continue to grow until you reach 5000 customers and then it will stop. Why? Because the # of customers you add each day is mostly independent of your total audience size, whereas the # of customers you lose each day will steadily increase until you’re losing the exact same number of customers you gain each day. At this point you plateau.
Carrying Capacity = # of new daily customers / % customers you lost each day
(The reality is a little more complex than this because the # of new customers you get each day is influenced to some extent by the total # of customers you have, but you can tease out those effects later.)
Enhancing the Model
The next thing to understand is where your new customers are coming from. There are 3 possibilities worth isolating:
- This is a brand new person you’ve never seen before (usually from advertising, SEO, or referral)
- This is a former customer you considered lost or dead that has been resurrected
- This is someone who tried your product in the past but never became a customer; now they’ve come back and converted (I call these the “skeptics”)
There is one more expansion I find helpful. Within the 5,000 total customers pictured above, some % of those returned today so they’re as good as new. Here is what my full customer dashboard with that addition:
Note: In the case that your first-visit viral coefficient is greater than 1.0, things can be very confusing without having a clear model like the one above. You could be converting 0% of your visitors to customers (i.e. all the water going into your bucket is instantly flowing out), but you’re steadily turning up the water since each new visitor is bringing more than one friend. You will see a steady increase in all your naive measurements of success (i.e. unique pageviews and visitors per day) but in reality you may not be acquiring any customers.
In closing, let’s revisit the example questions I raised at the start. Here is how this model helps:
- You notice that your power users all have taken some action (e.g. filled out their profile) so you try to encourage all users to fill out their profile to get them more hooked on your product. Does this actually help? Without this model it’s easy to mislead yourself into thinking it helps. You will probably be able to increase the metric but it may just weaken the correlation with power users. However, with this model you just watch your loss % each day and if it doesn’t change then you haven’t had any net impact on retention.
- You have 24 hours of downtime, the next day you come back up your traffic is down. Will this have a long-term effect you need to worry about? Even if your traffic is lowered for a few days, all you have to do is measure your new visitors per day & lost customers per day and you’ll see if your carrying capacity has changed. If it has not changed then you don’t have to worry, you know your traffic will rise back up to your carrying capacity.
- You have 100K uniques per day and so does your competitor, but are these 100K people who come back every day or 700K people who each come once per week? Does it matter? If you’re incorrectly caught up in # of unique visitors per day then this does seem like an important thing. And in fact, if you realize your visitors are not returning as often as your competitors you may even be tempted to spam them regularly because this will increase # of unique visitors each day and can give you the illusion of improving things. In reality a move like this would probably increase your % lost each day and hurt your carrying capacity but you wouldn’t notice this for awhile because the increased # of uniques each day would mask the harm. However, with the model your emphasis is on # of customers not # of visitors. You’ll quickly realize that you don’t care how frequently people visit as a primary factor; if it doesn’t impact # of new customers per day or % of lost per day then you haven’t actually helped your product.
- You turn on a new advertising campaign and see your # of unique visitors per day start to increase, you assume that this will continue increasing so long as you keep the ads running, right? Nope, you’ll level off once you’ve reached your new carrying capacity.
- You start having email deliverability problems (or Facebook turns off notifications) so you can’t notify users of new activity on the site. The # of unique visitors per day decreases slightly but you’re not too worried, should you be? This is similar to question 3, it may or may not be important but you’ll quickly be able to tell by focusing on the core numbers. Increasing # of unique visitors per day does not necessarily lead to more total customers.
14 Responses to “Web and Mobile Products: Understanding your customers”
Kevin Hillstrom Says:
November 16th, 2010 at 11:47 am
I’ve spent a lifetime analyzing these metrics, I’ve written 1,700 blog posts, largely on this topic. I doubt I’ve ever done as nice a job a you have in this article.
And you used the “carrying capacity” concept from ecology/biology … nicely done!!
Keith Schacht Says:
November 16th, 2010 at 11:50 am
Thanks Kevin. It was quite a bit more work than I expected to tie everything together and clearly explain it. It’s great to hear feedback on how it was received!
Jonathan Jaeger Says:
November 16th, 2010 at 7:18 pm
Your tweet this button didn’t work for me, not sure if anyone else had problems. Great article though, so I’ll manually tweet it, haha.
Keith Schacht Says:
November 16th, 2010 at 7:43 pm
Ah yes, the ‘&’ in the subject was breaking the URL. I just fixed that, thanks for the tip.
Mary Meeker Delivers Ad Boom Prez; AdBrite Announces Audience Data Product; Addressable TV For All In 5 to 8 Years Says:
November 16th, 2010 at 9:05 pm
[...] Product guy Keith Schacht provides a detailed plan for finding out about who your customers are and the way in which they use your product. He writes, "The key insight is this: the most important thing is how many unique customers you have, and by implication, knowing what caused any change in this number. And by customers I don’t mean visitors or pageviews, I mean the subset of those people that will be long-term repeat users." This is a great post. Read more. [...]
November 17th, 2010 at 1:51 pm
Enormously useful post, thank you.
One thing I’m debating with myself is – how important is the carrying capacity really?
Carrying capacity may or may not correlate with total revenue per customer.
Brian Barela Says:
November 17th, 2010 at 3:02 pm
great post. super insightful and perfect for getting a better handle on traffic, visitors, and customers.
Four short links: 17 November 2010 | mobilei.tk Says:
November 17th, 2010 at 8:07 pm
[...] Understanding Your Customers — I enjoyed Keith’s take on meaningful metrics. We talk a lot about being data-driven, but we interpret data with a model. The different take on meaningful metrics reflect the different underlying models that are lit up by data. It’s an important idea for the Strata conference, that gathering and processing data happens in the context of a world view, a data cosmology. (via Eric Ries) [...]
Keith Schacht Says:
November 17th, 2010 at 9:41 pm
Andrew, the revenue you make per customer is independent of the total number of customers you have. Those are two separate things you can optimize, both of which increase your total revenue. This model is intended to help you increase the latter. Carrying capacity is simply the total number of customers your product can have at one time.
Michael Lehman Says:
December 29th, 2010 at 12:59 pm
Do you know of any analytics companies that do a good job of tracking both mobile and web activity — especially funnel?
Keith Schacht Says:
December 29th, 2010 at 6:43 pm
@Michael: You can setup a good funnel in Google Analytics, but also checkout KissMetrics and Kontagent
4 liens rapides pour la semaine (2011-02) « La BI ça vous gagne! Says:
January 9th, 2011 at 7:01 am
[...] par Keith Schacht, une méthode pratique pour comptabiliser et suivre ses clients pour une entreprise web. Plein de bon [...]
Jonathan Andrew Wolter Says:
September 23rd, 2011 at 2:51 pm
I keep coming back to this article again and again. Thanks Keith, this is amazing. I’m referring it to another friend again today.
November 13th, 2011 at 4:09 pm
I am a little nervous about models that look sexy on paper but dont have a lot of research backing them up; how does this add more value or life to say rogers diffusion of innovation concept? end of the day, you need a structured approach to market planning and research! and unfortunatly, IT boffins dont know the basics.