How to test your marketing ideas and check the results are valid

How to test your marketing ideas and check the results are valid

Testing is an important part of marketing. By testing version A against version B you will be constantly improving results. But as a marketing scientist, I would always recommend using a formula and calculate the validity of your tests.

Below are 3 Tables showing how many responses you need for your test results to be considered valid. They are based on 3 different confidence levels.

Table I – Valid test sample sizes; 90% confidence

Valid Sample Sizes

Confidence:90%
Error margin:1%3%5%8%
PopulationSample size
10099897452
50046630117688
1,00087243021496
10,0004,036700264105
15,0004,662716266105
30,0005,520734269106
100,0006,336746270106
500,0006,674751271106
1,000,0006,719751271106

Table II – Valid test sample sizes; 95% confidence

Valid Sample Sizes

Confidence:95%
Error margin:1%3%5%8%
PopulationSample size
10099928061
500475341217116
1,000906517278148
10,0004,899965370148
15,0005,856997375149
30,0007,2761,031380150
100,0008,7621,056383150
500,0009,4231,065384151
1,000,0009,5121,066384151

Table III – Valid test sample sizes; 99% confidence

Valid Sample Sizes

Confidence:99%
Error margin:1%3%5%8%
PopulationSample size
10099958773
500485394285171
1,000943649399206
10,0006,2391,557622253
15,0007,8781,642636255
30,00010,6821,737650257
100,00014,2271,810659259
500,00016,0551,837663260
1,000,00016,3171,840663260

Here’s a breakdown of the key elements of valid tests:

  • Sample size: The number of people you need to respond to your test for it to be valid.
  • Population: The number of people in the group your Sample represents. For example, we recently calculated there are 35,000 Microsoft freelance contractors in the UK – so our Population is 35,000.
  • Confidence level: This tells you how sure you can be the results are repeatable. You have a choice of 90%, 95% or 99% confidence. Most tests are based on 95%.
  • Error margin: The plus or minus figure. An error margin of 5 on a result of 75%, means that between 70% and 80% of the population would choose the same version of your test.

As an example, if we want 95% confidence in the results and are willing to accept that the percentage response for each test may vary by +/- 3% (error margin), then the sample size would need to be 1,031 people (assuming a total audience or ‘population’ of 30,000 individuals).

So let’s assume you conduct a test with two different headlines. Headline A gets a 60% response and headline B gets 40%. Because you accepted a 3% error margin, you can be 95% confident that if you repeated the test headline A would get a response between 57% and 63% and answer B would get between 37% and 43%.

You can see from the Tables that as the Population figure rises above 15,000 people, so the additional number of respondents needed for your test to be valid narrows. So if you are not sure of the Population size of your audience just choose 15,000.

You can download my Test Results Verifier to check whether your marketing tests are valid. Download the PDF and simply follow the instructions.



Get my latest blog posts, reports and videos delivered straight to your Inbox, just 8 times a year. It’s free but not cheap. Complete the form below to receive the Marketing Graham Bulletin; you can unsubscribe at any time.

Marketing Graham Bulletin


    Data protection laws stipulate I must get permission to store your data and send you emails. So please tick the box.

    Yes, please send me the Marketing Graham Bulletin no more than 8 times per year. I understand that I can unsubscribe at any time by clicking the link in the footer of your emails, and you will store my data but never sell it to third-parties.

    For information on this websites privacy practices, please visit my Privacy Notice
    I use MailChimp as my marketing platform. By subscribing you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.

    Leave a Reply