As far as I can tell, multivariate experiments suck donkey balls. Unfortunately, I don’t think pure experiments exist in the world of business. They’re all multivariate.
From May 2019 to December 2019, I ran a paid email list experiment to test whether my thinking is good enough that some would pay for access to it. The experiment succeeded but hurt my lead flow and left things in disarray in my business. Here’s what I learned.
This experiment was inspired by the second principle of The Expertise Incubator (TEI):
- Anything you create in this program should be good enough to spread by word of mouth alone.
- Anything you create in this program — if you give it away for free — should be good enough that some would gladly pay real money for it.
- You should be willing to work daily for 2 to 3 years to make ideals 1 and 2 become true in your work. In other words, you should be OK with not living up to ideals #1 and #2 at first so that you can build up the skills you need to ultimately achieve those ideals
Around April/May of 2019, I thought: “what if I charged actual money for the daily emails I send?”
I was already a Stratechery.com subscriber, and Ben Thompson’s implementation of this idea seemed to be working well, so I bough a license for Restrict Content Pro, adapted a bunch of Ben Thompson’s language for the subscription infrastructure, and launched a paid segment to my email list. If you paid nothing, you got 1 email a week for free, and if you paid $10/mo or $100/year, you got an additional 3 emails/week.
I seeded my paid email list with some free trial invites.
A bit later in 2019, I migrated from Restrict Content Pro to Substack.
The Restrict Content Pro setup was a lot of work. It’s great general purpose software that, when used for a paid list, provides a terrible experience compared to a special-purpose paid email list solution like Substack. Having experienced this, I would never not use Substack or a Substack competitor in the future for a paid email list. Ben Thompson (again, Stratechery.com) uses Memberful, which seems fine and perhaps at his scale is a better option than Substack, et al. Memberful does offer something I consider a killer feature that Restrict Content Pro and Substack don’t have: a private RSS feed for paid subscribers.
Maintaining the publishing cadence was no problem for me. That “muscle” is already built, regularly maintained, and baked into my life and business design in a big way.
Navigating the existence of a free and a paid segment of the same email list was massively disorienting and frustrating. I constantly felt like I didn’t know what I was doing, and it raised questions I still feel like I don’t have great answers to:
- Which segment gets what kind of content?
- What adds actually value for those who are paying?
- Wouldn’t fewer more useful emails be a value add for some folks?
- What’s the natural model for this kind of experience? Is it like watching DVD extras, something for the superfans who can’t get enough? Is it like the Cliffs Notes, for those who are time-pressed and want to pay for more concision? Etc, etc.
The ramp from free to paid email list subscribers was a seldom-travelled ramp. Standing at the top of that ramp and encouraging free list subscribers to ascend it by paying was a vexing experience. Again, what exact value are they paying for? I mean, beyond 3 more emails per week!
The nagging feeling in my gut was that what I was actually offering was “pay to get more homework assigned to you”. Most people would prefer a reduction rather than an increase in homework!
My clients are different, and special, but even they don’t want more arbitrary homework. They are happy to take on risk and expend effort in carefully designed ways, but… want more busywork they do not.
This experiment was a medically-induced coma for my list. About 2 years into the daily publishing experiment — to be clear, before the paid list experiment — I measured my list responsiveness: 2.7 replies/day, on average. That’s how often a list subscriber would have something to say about what I wrote. Yes, some subscribers were more chatty than others, and yes not all of that chatter led to new business, but still, it engendered a feeling of “this list is paying attention”.
I haven’t measured it during the paid list experiment of 2019, but I can definitively say list responsiveness fell off a cliff.
I believe that in businesses, there are not clean, pure single variate experiments. There’s always a background noise of change and variation that makes pure single variable experiments impossible. So not only would it be impossible to say this paid email list experiment was a single variable experiment with clear results, but I also willingly added other variables.
A big one was a change in my approach to writing. That could be equally or more responsible for the reduction in a feeling of “liveliness” from my email list. I do have at least one really vivid data point that suggests a cause-effect relationship between publication frequency (which went from 7x/week to 1x a week for the free segment of my email list), writing approach, and list responsiveness:
Two responses to that first comment: 1) Ouch and 2) Thank you. I really mean the thank you part! Oh, and one more response: 3) You were right. See! I’m changing as a result.
I’ll temper this observation about list responsiveness by saying that there still has been conversation and response to my thinking, just on a smaller scale. And I thank all of you who have had something to say about my thinking as shared either on my paid or free email list segment. But the general depression in liveliness is a grave concern for me because of how that is a leading indicator for other important business metrics.
Lead flow: diminished! This is again where the multivariate demon enters the picture. Multiple other things changed roughly simultaneous with the paid list experiment:
- A feeling of completion in exploring the specialization/positioning question, leading to wanting to enlarge the context in which I operate.
- A guiding question for my work also entered the picture this year, and that is: how do implementors become advisors? Yeah, some folks are happy to remain implementors their whole self-employed career, but some folks have the ganas to change things and I’m really interested in how and why they do it, and learning what I can in service of those who want to make this transition but need support doing so. This guiding question pushed me headlong into exploring anything and everything that seemed relevant to it, which seemed to de-focus my writing. In a very real way, I was both expanding and becoming more focused. I became more focused on the self-made-expert segment of self-employed tech workers while expanding to a broader set of concerns than just specialization. Vertical contraction, horizontal expansion.
- This year I also really wanted to emphasize TEI as a service offering. The results from the program have greatly exceeded my expectations. This service offering is about expertise rather than about specialization per se, which is a significant topical change that trickled down into my email marketing.
- In 2019 I started thinking my way through how the differences between brand and direct response marketing effect how self-made experts approach their marketing.
- This led to me feeling strongly that direct response marketing poses an inherent tension with the promise of rare, valuable expertise, and a subsequent attempt to move my business more towards a brand marketing approach.
- Oh and also: I put CPP on haitus, and the “next act” of a new podcast called The Self-Made Expert took much longer to get up to speed than I thought, creating a gap in generating visibility for my work. I also did less podcast guesting in 2019.
So to recap, here are the multiple variables that changed throughout 2019:
- Change in the focus of my work
- Change to my primary visibility mechanism
- Change to my primary trust-building mechanism
- Change to the ethos of how I invite folks to take action
- Change to service offerings (not to mention de-emphasizing selling TPM which also reduced revenue and lead flow)
- Oh and double-also, my wife and I rather abruptly moved 1,272 miles from Sebastopol, CA to Taos, NM and started a house construction project.
Talk about a multivariate experiment!!
With this many simultaneous changes, it’s a miracle I’m still in business. Risk itself doesn’t frighten me nearly as much as my own cavalier approach to risk does. Revenue is actually up about 30% over 2018. Thank goodness for that!
Again, I learned that multivariate experiments suck donkey balls. Even if I’d only attempted one big intentional change (in the form of the paid email list experiment), there is always a background “noise” of small unintentional changes that cause even attempted single-variable experiments to actually be multivariate experiments. This is why so much business advice is bullshit: the advisor is reporting the results of faux clean single-variable experiment that was actually a noisy multivariate experiment. It’s well-intentioned bullshit, but bullshit nevertheless.
If I’d only attempted one intentional change, I might have somewhat better conclusions about this paid email thing. Given how outrageously multivariate the experiment actually was, all I can definitively say is that change is risky. 🙂
Here’s what I can less definitively (but hopefully still usefully) say.
Problems with a paid email list for self-made experts
Suboptimal monetization: I’ve speculated that even someone with a brand as strong as, let’s say Alan Weiss, would see revenue harm from implementing a paid email list.
Imagine that Alan had a test cohort of 1,000 people. That group can either all receive email marketing for free, or all can subscribe to a $10/mo paid email list ($10,000/mo in list revenue). I speculate that Alan would ultimately generate more than $10k/mo by letting that cohort of 1k get his emails for free and using those emails to inform them of paid services.
I don’t think this is wild speculation; here’s one quick attempt at modeling it using some simple Pareto assumptions:
My takeaway: paid email lists are a great fit for, as a beloved client of mine has said, journalists who are just happy that anybody would pay for their writing. For self-made experts in a niche topic, there are other ways of “monetizing attention” (gawd I hate typing those words but it is, at some level, an accurate way to describe email marketing that provides value while promoting paid products/services). The tried and tested way to monetize attention is to send remarkably good emails to a list and also use those emails to promote paid offerings. I’m now more confident this approach can pay off better than a paid email list, perhaps with less effort!
Awkward free -> paid ramp: The ramp from free to paid emails was always awkward, at least in my case. I’ve heard Ben Thompson speak about starting his newsletter business (the “Conversations With Tyler” interview is a good one) and my takeaway was probably not a very high-fidelity one, but it amounted to this: “I wrote good shit, added a paid option, and then it was like turning on a money spigot that is now a large pipeline of money”. That may be a takeaway with some fidelity to Ben’s actual situation, but importing that model from a mass market context to a niche context means there may be issues with the viability of the model in the new context. And in my case, there were!
In the mass market context, there’s always a background river of novelty. Microsoft just did this, Cisco just did that, etc. I think of Stratechery as one of those small, short kayaks you can do tricks in, and one of the tricks is you can surf static waves on a river, if the river is fast-moving enough.
You’ve got this river of constant novelty, of new news that has broad appeal and relevance, and then you’ve got Ben Thompson with his intriguing models and point of view and opinions doing tricks on the waves created by the river of news. It’s a lot of fun to watch! And it works as a business for Ben, and a few others in different areas.
In the niche market context, that same model is harder to pull off. If the river you are “kayaking” is more placid, then you can ride it downstream, but you will ride out of view of those on the shore. The slower-moving river deprives you of the ability to put on a performance for them in a fixed spot, so they don’t get the experience of waking up each day, feeling the rushing energy of change and uncertainty countered by the comforting presence of your evolving-but-locationally-static interpretation of the news. That tension is what creates value for Stratechery subscribers, and it’s hard to re-create that tension in many niche markets.
There are niche markets that have the same rushing energy of change albeit on a smaller scale than tech writ large, and I could see a paid list model working there. I imagine any US-based business with a supply chain that runs through China being happy to pay $10/mo, hell $100/mo!, for intel on that evolving situation. That’s not the nicheyies of niche markets, but it’s certainly more niche than tech writ large.
Are there other models aside from “surfing a river of broadly interesting news with your insight” that could make for a good paid newsletter business? I think so. Briefly:
- The aforementioned “critical business intel” model. Think of a much more niche version of Sinocism (Sinocism’s tagline is: “Get smarter about China”). Any business in a changing regulatory or political or tech context is a candidate for this model.
- A relatively infrequent, relatively large installment of educational content delivered via paid newsletter where the tension is between what people want to know and what they perceive they don’t know. Ben Settle’s paid monthly (print) newsletter on email marketing is one such example, though sold with HEAVY DUTY direct response marketing methods.
- A relatively infrequent (perhaps monthly) publication of analysis delivered via paid newsletter. This might lend itself to a more placid river of change, or simply working through a lengthy agenda of items to analyze where the items aren’t news items, but rather are aspects of how things are typically done or evergreen open questions in a niche market.
I see my daily publication practice as a form of cutting trail; building a path forward through a wilderness for those I serve. There’s still a lot of trail for me to cut, so I’m literally afraid to stop publishing daily. I don’t want to lose the momentum I have, and I don’t trust that I have the discipline needed to publish great stuff less frequently. So the models above do feel generall usable in my context, just not by me specifically.
Tactical annoyances: Substack hasn’t invested much into the technology of subscriber acquisition. At the time of writing, they basically offer an iframe subscriber form you can embed elsewhere and that’s it. No Zapier integration, no plain HTML form you can style as you like, etc. Zapier integration isn’t even on their near term development roadmap as of mid-2019 (this based on an email exchange with one of their founders).
This lack of investment in acquisition is understandable given their relative newness, but it’s very limiting when you’re coming from a world where the go-to products (ConvertKit, Mailchimp, etc.) put a ton of effort into optimizing for subscriber acquisition and offering their users tons of options (easily styled forms, plain HTML forms, Zapier, API, etc.).
It’s not that the world of paid subscription media doesn’t optimize for subscriber acquisition. Dear God, look at the herculean efforts the NY Times puts forth along this dimension! (And dear God, can’t my subscription revenue be enough for them? Apparently not. It’s almost as if they market to me more now that I’ve subscribed after being broken by their combination of paywall, generally sloppy reporting, and occasional brilliant reporting.)
Anyway, yes, the world of paid subscription media does take subscriber acquisition seriously — at least at the high end — but the world of direct response marketing and the tool ecosystem it’s nurtured also takes scrappy, lean, inexpensive, DIY subscriber acquisition seriously, and that results in email marketing software that makes it way easier to insert in-line opt in forms, and other methods of subscriber acquisition that make sense for businesses like ours. Substack ain’t there (yet?).
Good results from a paid email list for self-made experts
It’s not all bad.
Initial impulse validated: My initial impulse to try this experiment feels validated. Some people (it was never very many, but still a positive non-zero number) seemed to indicate that my thinking was good enough to pay for. Principle 2 of TEI validated! It’s critical to me that I live out every principle of this program, and nice to know that I objectively am.
But again, I’m confident there’s a better way for folks to pay for my thinking: buying products/services. I didn’t close down this ramp from email list to buying product/service, but I definitely made the entrance to the ramp less clear, and that hurt my revenue potential in 2019. That’s OK. Part of why I structure my business the way I do is to generate evidence like this from firsthand experience. And that evidence is another positive outcome of this experiment; I can speak more authoritatively about this idea of a paid email list. I still don’t have a ton of data on it, but I do have fewer un-tested assumptions clouding my view of the issue.
Context sidebar: I see 3 primary ways to accelerate self-made expertise: 1) work with more clients as quickly as you can 2) publishing-as-laboratory, or thinking out ahead of your clients and 3) importing from other domains.
The writing bootcamp: This experiment was a sort of writing and thinking bootcamp for me, with positive outcomes. As soon as that paywall went up, I immediately felt pressure to think more deeply and write differently.
This wasn’t a totally un-alloyed good because it disrupted the status quo of how I do email marketing. And that status quo was working well! I’m still recovering from that disruption, but it’s had some really positive effects mixed in with the disruption.
At first, my attempts at thinking more deeply felt like a kid wearing his dad’s comically-too-large suit. The intention to “grow up” in my thinking was genuine, but the view from outside might have provoked some giggles. I do care about status and how others perceive me, but not enough to avoid taking social risks like this. I would do — and have done! — almost anything in service of the future expertise and impact I desire.
Baby, there’s just one thing
One thing that does it, does it for me
Baby, we’ll find a way
We’ll go out, out for a day
And I want it so bad
It’s the first thing I see when I wake
It’s the first thing I see when I wake
“Just One Thing”, My Morning Jacket
I mean, I won’t comprise other core values of mine, but I will gladly look silly day in and day out in service of this vision of really deeply understanding the implementor -> advisor transition and being a world-class level of support for those undertaking it. I will gladly fall short of this goal over and over again in order to one day achieve and ultimately surpass it. That’s the third principle of TEI, and I live that fucker.
This “writing bootcamp” experience turned down the volume on the entertainment element of my emails. Way down! I’m still not clear on whether this has helped my cause or not. It seemed necessary to forsake being entertaining, for a time at least, in service of thinking more deeply, but I’m keenly aware that it limits the appeal of my emails, and makes them less tasty to consume. Very few people are going to have the patience to hang out in a laboratory and learn by osmosis. Email marketing does benefit from a bit more razzmatazz. 🙂
De-rutting: The flip side of finding something that works is that it can become its own rut. The “start with a fun/wacky story, transition to making a point, transition to a CTA” is a great email format. It’s almost its own genre, and it’s like an Oreo cookie or Reeses cup: the near-perfect combination of disparate elements, of salty and sweet, of entertaining and informative/inspiring.
But this format/genre isn’t great at everything. It has limits. This experiment got me out of this format/genre/rut, and into trying other things. Not all of which were successful!! But at least I tried things like long think-ey pieces, and recently, an Axios-inspired Q&A format. I think of it like cross-training for thought leadership.
I’ve toyed with the idea of keeping the paid email list as a sort of extension of my larger email list, but as a somehow “special” segment to which I send special/exclusive content. I do like some aspects of that, but dislike the additional complexity, so I won’t actually do that.
I’ve toyed with keeping Substack as a sort of Patreon-like “way to pay if you dig the free emails” thing, but not actually publishing anything there and instead publishing via ConvertKit (where my email courses live). I’m unsure if that jives with Substack’s ToS (why would they let me charge people via their service to receive emails I don’t actually send via their service?) and don’t really care to find out. And more importantly, I really don’t love the signal that
expertise + consulting services + Patreon sends. So no to this as well.
That leaves me with this option: shut it down. Go back to the tech stack described here: https://theexpertiseincubator.transistor.fm/3
That’s what I’m gonna do.
I can’t express enough gratitude to those of you that paid a few bucks to join me in the dark carnival of this experiment! Thank you! And if you feel you didn’t get enough for your subscription fee, email me and we’ll figure that out.
Those of you that watched with curiosity from afar: did you see this coming? Was this one big fat “I told you so, Philip!” for you? I’d love to hear from you. I can take it. 🙂
Those of you that find the idea of publishing as a discipline that rapidly cultivates expertise interesting: please check out TEI. It’s a great way to get support and kinship in that journey. A new cohort begins January 13, 2020.
Title inspiration: Of course, the Mike and The Mechanics song “The Living Years” is kinda cheesy, but kinda powerful and endearing even 31 years after it was published. “You can listen as well as you hear” seems to express something important about taking seriously the results of experiments, and more broadly: the importance of being a student of life.