How to test your marketing ideas and check the results are valid

How to test your marketing ideas and check the results are valid

Testing is an important part of marketing. By testing version A against version B you will be constantly improving results. But as a marketing scientist, I would always recommend using a formula and calculate the validity of your tests.

Below are 3 Tables showing how many responses you need for your test results to be considered valid. They are based on 3 different confidence levels.

Table I - Valid test sample sizes; 90% confidence

Valid Sample Sizes

Confidence: 90%
Error margin: 1% 3% 5% 8%
Population Sample size
100 99 89 74 52
500 466 301 176 88
1,000 872 430 214 96
10,000 4,036 700 264 105
15,000 4,662 716 266 105
30,000 5,520 734 269 106
100,000 6,336 746 270 106
500,000 6,674 751 271 106
1,000,000 6,719 751 271 106

Table II - Valid test sample sizes; 95% confidence

Valid Sample Sizes

Confidence: 95%
Error margin: 1% 3% 5% 8%
Population Sample size
100 99 92 80 61
500 475 341 217 116
1,000 906 517 278 148
10,000 4,899 965 370 148
15,000 5,856 997 375 149
30,000 7,276 1,031 380 150
100,000 8,762 1,056 383 150
500,000 9,423 1,065 384 151
1,000,000 9,512 1,066 384 151

Table III - Valid test sample sizes; 99% confidence

Valid Sample Sizes

Confidence: 99%
Error margin: 1% 3% 5% 8%
Population Sample size
100 99 95 87 73
500 485 394 285 171
1,000 943 649 399 206
10,000 6,239 1,557 622 253
15,000 7,878 1,642 636 255
30,000 10,682 1,737 650 257
100,000 14,227 1,810 659 259
500,000 16,055 1,837 663 260
1,000,000 16,317 1,840 663 260

Here’s a breakdown of the key elements of valid tests:

  • Sample size: The number of people you need to respond to your test for it to be valid.
  • Population: The number of people in the group your Sample represents. For example, we recently calculated there are 35,000 Microsoft freelance contractors in the UK – so our Population is 35,000.
  • Confidence level: This tells you how sure you can be the results are repeatable. You have a choice of 90%, 95% or 99% confidence. Most tests are based on 95%.
  • Error margin: The plus or minus figure. An error margin of 5 on a result of 75%, means that between 70% and 80% of the population would choose the same version of your test.

As an example, if we want 95% confidence in the results and are willing to accept that the percentage response for each test may vary by +/- 3% (error margin), then the sample size would need to be 1,031 people (assuming a total audience or ‘population’ of 30,000 individuals).

So let’s assume you conduct a test with two different headlines. Headline A gets a 60% response and headline B gets 40%. Because you accepted a 3% error margin, you can be 95% confident that if you repeated the test headline A would get a response between 57% and 63% and answer B would get between 37% and 43%.

You can see from the Tables that as the Population figure rises above 15,000 people, so the additional number of respondents needed for your test to be valid narrows. So if you are not sure of the Population size of your audience just choose 15,000.

You can download my Test Results Verifier to check whether your marketing tests are valid. Download the PDF and simply follow the instructions.


Get my latest blog posts, reports and videos delivered straight to your Inbox, just 8 times a year. It’s free but not cheap. Complete the form below to receive the Marketing Graham Bulletin; you can unsubscribe at any time.

Marketing Graham Bulletin


Data protection laws stipulate I must get permission to store your data and send you emails. So please tick the box.

Yes, please send me the Marketing Graham Bulletin no more than 8 times per year. I understand that I can unsubscribe at any time by clicking the link in the footer of your emails, and you will store my data but never sell it to third-parties.


For information on this websites privacy practices, please visit my Privacy Notice
I use MailChimp as my marketing platform. By subscribing you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.

Leave a Reply