Testing is an important part of marketing. By testing version A against version B you will be constantly improving results. But as a marketing scientist, I would always recommend using a formula and calculate the validity of your tests.
Below are 3 Tables showing how many responses you need for your test results to be considered valid. They are based on 3 different confidence levels.
Table I – Valid test sample sizes; 90% confidence
Valid Sample Sizes | ||||
| Confidence: | 90% | |||
| Error margin: | 1% | 3% | 5% | 8% |
| Population | Sample size | |||
| 100 | 99 | 89 | 74 | 52 |
| 500 | 466 | 301 | 176 | 88 |
| 1,000 | 872 | 430 | 214 | 96 |
| 10,000 | 4,036 | 700 | 264 | 105 |
| 15,000 | 4,662 | 716 | 266 | 105 |
| 30,000 | 5,520 | 734 | 269 | 106 |
| 100,000 | 6,336 | 746 | 270 | 106 |
| 500,000 | 6,674 | 751 | 271 | 106 |
| 1,000,000 | 6,719 | 751 | 271 | 106 |
Table II – Valid test sample sizes; 95% confidence
Valid Sample Sizes | ||||
| Confidence: | 95% | |||
| Error margin: | 1% | 3% | 5% | 8% |
| Population | Sample size | |||
| 100 | 99 | 92 | 80 | 61 |
| 500 | 475 | 341 | 217 | 116 |
| 1,000 | 906 | 517 | 278 | 148 |
| 10,000 | 4,899 | 965 | 370 | 148 |
| 15,000 | 5,856 | 997 | 375 | 149 |
| 30,000 | 7,276 | 1,031 | 380 | 150 |
| 100,000 | 8,762 | 1,056 | 383 | 150 |
| 500,000 | 9,423 | 1,065 | 384 | 151 |
| 1,000,000 | 9,512 | 1,066 | 384 | 151 |
Table III – Valid test sample sizes; 99% confidence
Valid Sample Sizes | ||||
| Confidence: | 99% | |||
| Error margin: | 1% | 3% | 5% | 8% |
| Population | Sample size | |||
| 100 | 99 | 95 | 87 | 73 |
| 500 | 485 | 394 | 285 | 171 |
| 1,000 | 943 | 649 | 399 | 206 |
| 10,000 | 6,239 | 1,557 | 622 | 253 |
| 15,000 | 7,878 | 1,642 | 636 | 255 |
| 30,000 | 10,682 | 1,737 | 650 | 257 |
| 100,000 | 14,227 | 1,810 | 659 | 259 |
| 500,000 | 16,055 | 1,837 | 663 | 260 |
| 1,000,000 | 16,317 | 1,840 | 663 | 260 |
Here’s a breakdown of the key elements of valid tests:
- Sample size: The number of people you need to respond to your test for it to be valid.
- Population: The number of people in the group your Sample represents. For example, we recently calculated there are 35,000 Microsoft freelance contractors in the UK – so our Population is 35,000.
- Confidence level: This tells you how sure you can be the results are repeatable. You have a choice of 90%, 95% or 99% confidence. Most tests are based on 95%.
- Error margin: The plus or minus figure. An error margin of 5 on a result of 75%, means that between 70% and 80% of the population would choose the same version of your test.
As an example, if we want 95% confidence in the results and are willing to accept that the percentage response for each test may vary by +/- 3% (error margin), then the sample size would need to be 1,031 people (assuming a total audience or ‘population’ of 30,000 individuals).
So let’s assume you conduct a test with two different headlines. Headline A gets a 60% response and headline B gets 40%. Because you accepted a 3% error margin, you can be 95% confident that if you repeated the test headline A would get a response between 57% and 63% and answer B would get between 37% and 43%.
You can see from the Tables that as the Population figure rises above 15,000 people, so the additional number of respondents needed for your test to be valid narrows. So if you are not sure of the Population size of your audience just choose 15,000.
You can download my Test Results Verifier to check whether your marketing tests are valid. Download the PDF and simply follow the instructions.
Get my latest blog posts, reports and videos delivered straight to your Inbox, just 8 times a year. It’s free but not cheap. Complete the form below to receive the Marketing Graham Bulletin; you can unsubscribe at any time.
| Marketing Graham Bulletin |
|---|