You can run an A/B test with low traffic, but you usually cannot detect small improvements. A site with few conversions per week will take a very long time to separate a real difference from random variation, so only large changes are realistically testable. For most small sites, the better return comes from fixing obvious problems, learning from a handful of real users, and saving formal tests for bold changes.
Why low traffic is a problem
A conversion rate is built from small counts. If a page gets a handful of enquiries in a week, one extra enquiry by chance can make a variation look dramatically better or worse. Statistical tests need enough events in each variation to tell chance from effect, and the smaller the real difference, the more events you need. Lift in the order of a few percent is invisible at small scale.
Three practical consequences follow:
- Count conversions, not visitors. Ten thousand visitors with a few conversions is a small sample for the outcome that matters.
- Stopping early misleads. Checking daily and stopping when one variation looks ahead produces false winners.
- Seasonality interferes. A test that spans a holiday, a campaign burst or a quiet fortnight can reflect timing rather than design.
This article does not give a universal minimum sample size, because it depends on your baseline conversion rate and the difference you want to detect. Use a sample size calculator from a reputable statistics source, with your own numbers, before you start.
What Nielsen Norman Group warns about
Nielsen Norman Group notes that A/B testing can only pick the best of the options you tested, so a variation built on a wrong guess about the problem will not fix it however much it is tweaked. It also cautions that judging a concept by a single design is risky because a weak result may reflect poor execution. Its recommendation is to use user research to find likely causes before testing, such as surveys on intent and objections, usability testing and tree tests for findability (Nielsen Norman Group, A/B testing and UX research).
For a small site, that research-first approach is not a consolation prize. It is often the stronger method.
Alternatives that work at small scale
Qualitative testing with a few people
Nielsen Norman Group advises that a qualitative usability study of about five users per study finds nearly as many usability problems as much larger groups, with the best return from iterating: fix what you find, then run another round. It also notes that quantitative studies need far more participants, at least 20 users for statistically meaningful results (Nielsen Norman Group, How many test users in a usability study). Ask five people from your audience to complete a real task, watch where they hesitate, and fix those points before anything else.
Before-and-after comparisons
Make one considered change, then compare the following period with the previous one. This is weaker than a controlled test because other things change, so treat it as an indication, not proof. Record what else happened, such as campaigns, seasonality and pricing changes.
Bigger, bolder variations
If you do split traffic, test radically different pages or offers, not button colours. Large differences can show up with fewer visitors. Do not test several small changes at once.
Test where traffic is concentrated
Paid campaign landing pages, email links and ad copy often get more focused traffic than organic pages. Ad platforms report results at the ad level, which is why testing ad copy can be more practical than testing a website page.
Use a higher-volume proxy cautiously
Instead of final enquiries, you could compare an earlier step such as clicks on the main call to action. This detects effects faster, but a change that raises clicks may not raise enquiries, so confirm with the real outcome afterwards.
A sensible process
- Find the likely problem with analytics, recordings, surveys or user testing.
- Write a hypothesis: “We believe changing X will improve Y for audience Z, because of evidence W.”
- Estimate whether your traffic can detect the effect size you hope for. If not, skip the formal test and make the change on the evidence.
- Define the test period and success measure in advance, and do not stop early.
- Record results, including inconclusive ones. An inconclusive result is information, not failure.
When not to test at all
If a page has a clear fault, such as a broken form, an unreadable layout on mobile or missing contact details, fix it. Testing whether a broken form converts less than a working one wastes time. The conversion fundamentals for business websites and landing page guide cover the improvements most likely to matter first.
Where Radiance Impact fits
Radiance Impact can help review a website for the issues most likely to affect enquiries and plan changes in a way your traffic can support. See our Digital Technology services or contact us.
Sources
- A/B Testing and UX Research, Nielsen Norman Group
- How Many Test Users in a Usability Study?, Nielsen Norman Group
