Luca Dellanna has a great video on optimizing for reproducibility. Loved his point on working to make your chances of success higher (more reproducible).

But what if even the best in your field don’t have reproducible success? Just because Michael Bay has absolutely crushed it with some movies, doesn’t mean he’s guaranteed to have his next movie be a blockbuster. Just because people love the Dyson vacuum and (maybe) hairdryer, doesn’t mean James Dyson’s next idea will be a hit. Or on a small scale, not all of Kevin Hart’s jokes land. He tests routines for eight to nine months to get something that works. There have got to be a lot of misses in there.

The reality is, sometimes you can’t increase reproducibility much, even if you’re the best in the world.

Make sure you can afford to take enough shots to make success inevitable

If a baseball player got fired on his first strikeout, there would be zero returning players. (A .300 batting average will get you into the hall of fame - that means the greats strikeout seven out of ten times). A player is going to get 150 to 500 at-bats in a season. If they strikeout every time they’re up in a season, that’s a different story.

Let’s make this concrete:

If you want to start a business, how many “at-bats” do you need to give yourself?

Think about starting a consumer info business. Someplace that sells courses on cooking or language or something. Even experts in the field have low reproducibility (3-10% closer to 3%).

If you have $10,000 to start a business, what would I suggest you do? Well, take a wild guess as to how many swings you will need. In my experience if you want to sell a book with a funnel behind it, you need to be able to test 10-30 front ends to a similar audience to find a good one. You must run a good test on each front end offer or else this number is much higher.

A good test probably costs $2,000 to $4,000. A person experienced with this would take 10-30 to find one winner, so someone who doesn't have the skills for launching that business yet will need probably double, since the first few rounds are developing their skills. That means they'll launch 20-60 front end tests which will cost between $40k and $240k on the high side.

Even the best case scenario is significantly over the bankroll we've got to invest, so I would say look for a different game that has a lower cost to find a winner. Maybe print on demand which could be closer to $200 per good test? Or get your skills good enough that you can decrease your cost per test.

Rule of thumb: if you don't have 3x the worst case scenario cash and time bankroll for testing, I would suggest switching games.