When working with clients, part of my job is to keep them informed, motivated and focused. We may be running several ad tests, spending lots of money, and they always want to know which ad is the winning ad.
Without question, they will always choose the ad that got the most conversions, or if there were no sales, they will choose the ad with the highest Click Thru Rate (CTR). You probably do the same thing
Now that I’ve dusted off my Crystal Ball, I’ll show you how to properly forecast your marketing future.
A long time ago, Thomas Bayes came up with a theory. He could predict where a marble would land on a table based on where the previous marble landed. In essence, he could predict the future with a mathematical formula.
As Frank Kern says sarcastically, “Oh yeah, a math class.”
Bayes Theorem is used extensively by stock investors, but is little known to the marketing community. I have been quietly using Bayes Theorem to predict marketing efforts since 1979, using very little data.
Generally speaking, large tests are not needed because we are not analyzing the relationship of numbers to the audience as a whole, but rather, the relationship of numbers to each other.
Here’s an example of a fictional FaceBook ad test:
The Control ad had a Reach of 878 people, of which 171 click through to the opt-in page. The Variable ad set had a Reach of 1252 people with 224 clicks. Obviously, the Variable ad is the winner, because it has the most clicks. Is it really the winner? Most marketers would say, “YES!”
Using a Bayes calculator found on the Internet, we punch in the numbers to calculate Lift. Lift is how much better (or worse) an ad will perform over time, expressed as a percentage.
The Control has a CTR of 19.48%, while the Variable has a CTR of 17.89%. That’s a Lift of +8%% in favor of the Control. In essence, the Control ad is 8% better than the Variable, even though the Control had nearly 27% fewer clicks.
According to Bayes, the Control will yield about 144 more clicks to the opt-in page, over a 30-day period, than the Variable ad. Additionally, Bayes shows us, there is an 81% chance these numbers will hold true for the entire life of the ad.
In the real world, we would like to see the Probability at 90% or higher. In social statistics, 50% is a guess and 66% is the minimum allowable metric. So, at 81%, we are better than a guess, over the minimum and slightly less solid than statistical concrete.
Most would say, “David, you don’t have enough data to make a statistically relevant conclusion. You have less than 1,000 clicks.” To which I sternly respond, “Come on Kern, have some faith.”
To be relevant for an audience of 2 million people, we would only need 384 total clicks to have a 95% confidence rating with a margin of error of 5%. That is the exact metric used during elections to predict the winner. It is also the social scientist’s rule.
As social scientists, we are happy to have 95% confidence in our data, with a +-5% margin of error. This means if we have 100 clicks within 3 days, we can be 95% confident that we will get 95-105 clicks within another 3-day period. By the way, there is no 100% confidence in prediction. The highest is 99% confidence with a margin of error of one percent. In order to reach that level, we would have to have 16,451 clicks, or 44 times more data, time and money.
Are you beginning to appreciate the importance of Bayes?
As it stands, our respondents total 395, eleven more than the minimum. Even though math is not an exact science (another Kernism), using a sample size calculator, we discover our data has a margin of error of 4.45%, better than the 95/5 rule.
[Insert Victory Dance, here]
So, our data is solid gold. We can take these numbers and scale, with confidence. While our competition continues to test their lives away.
Get acquainted with Bayes. You don’t have to be an expert to appreciate the value of marketing prediction forecasting. There are dozens of Bayes calculator on the Internet. Plug and play. Easy.
Bayes has been my ‘Secret Sauce’ for many years and it’s the reason why ‘they’ call me Merlin. They also call me The Curmudgeon, but that’s another story. Now, back to my dark corner….