Sample size needed for original research?

List member Josh asked an important question about original research:


Quick question about original research. I know you don’t have a doctorate in statistics, but I’m curious to know what you’ve seen out in the wild.

If you were going to do some research, how large would your sample pool be? Your Trust Velocity sheet has 46 items. Is 50-ish a safe balance of large enough to be useful and small enough you don’t need a grant to complete it?

Great question, Josh!

Here’s how I think about it: the goal of original research is to help you and your client make better decisions. That’s *what* the research should accomplish.

Turning that into a question helps you figure out what area(s) you want to look into using research. What decisions do you help your client make on a project, and which of those need to be better decisions? Better = less risky, made more quickly, or would achieve better results if they were decided differently.

*How* research enables better decisions is by reducing uncertainty. Not eliminating it, but reducing it. Some notes from the excellent book “How to Measure Anything” are relevant here:

* Mathematically speaking, when you know almost nothing, almost anything will tell you something.
* The first few observations are usually the highest payback in uncertainty reduction for a given amount of effort.
* One way to underestimate the amount of available data is to assume that only direct answers to our questions are useful.
* The rule of five: There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.
* Single Sample Majority Rule (i.e., The Urn of Mystery Rule) Given maximum uncertainty about a population proportion— such that you believe the proportion could be anything between 0% and 100% with all values being equally likely— there is a 75% chance that a single randomly selected sample is from the majority of the population.

If any of that seems interesting or relevant to you, I’d recommend the whole book. It’s quite good.

The TL;DR of those notes above is this: large sample sizes aren’t always necessary. Even small sample sizes can be effective in reducing uncertainty.

Said differently… most of the time your client has literally zero data to enable making better decisions about important aspects of a software project. Sure, they may have *opinions* out the wazoo and *beliefs* based on non-data.

It may not take too much legwork on your part to assemble some data that helps them make significantly better decisions during a software project. That’s really all original research needs to be.

If you need low-cost, high-quality help with your positioning, check out the only complete manual on positioning for self-employed software developers: http://thepositioningmanual.com

Talk to you soon,
-P

Want to stop competing against every other developer?

"This is by far the most valuable email I have ever received from an email list." - Timothy Voice

x