(Readin’ time: 1m 47s)
List member Josh Earl said something really crucial in response to my SPoF email. Josh graciously gave me permission too share this with you:
Hey Philip, good email. I generally agree with you about doing many smaller validations along the way.
One caution that I wanted to add, based on my own experience, is that there’s a strong danger of selection bias here.
The top 1% of your email subscribers tend to love you so much, they will literally sign up for anything, do anything, and buy anything you create.
This can really skew your perception of demand for a product if you’re not aware of how strong the effect is.
Here’s a scenario that I’ve seen multiple times before:
Say you have an email list of 10,000 people.
You have an idea for a product that comes to you, proverbially naked in the proverbial shower.
You send out a couple of emails with a link to opt in to a pre-launch list. You get 100 signups.
Yay! People are interested.
You start building the product, and you do a beta offer to this tiny little list. You make 25 sales.
Yay! People are REALLY interested.
You can’t help yourself, and you start doing the math—”Wow, I got a 25% conversion rate on this, and I only had 100 people signed up. If even 5% of my main list buys this, I’d make 500 sales!”
You build the final product, and you do a launch to your entire list, and…
You end up making another 15 sales.
When this happens the reason is that almost everyone who had enough interest in this offer to consider buying had already signed up for the pre-launch list.
When launch time comes, there aren’t a lot of people left on your main list who would consider buying.
One way to account for this is to raise the bar in the early validation steps.
And assume that whatever response you get early on captures the majority of the interested people.
For example, you might set a threshold of getting 500 or more signups to a pre-launch list instead of 100.
You want to see a very strong response from your entire list early on, not just the hyper-responsives.
I think I’ll do my email about this today. 🙂
Josh is right. I missed this really important manifestation of bias that you need to watch out for in validating an idea.
Consider signing up for Josh’ email list: joshuaearl.com
It’s great stuff.