A/B Testing

A/B testing is a controlled experiment in which visitors are randomly split between two versions of something, such as a page layout, an offer or a price, and the results are compared on one agreed metric. Version A is usually the current experience, called the control, and version B is the change being tested.

How a split test works

The core idea is random assignment. Each visitor is placed in group A or group B by chance, so the two groups are alike in every way except the change under test. Any meaningful gap in the outcome can then be credited to that change rather than to the weather, the day of the week or a new advertising campaign.

  1. Write a hypothesis, for example that a shorter checkout form will raise completed orders.
  2. Choose one primary metric before the test starts.
  3. Estimate how many visitors each group needs to detect the effect you care about.
  4. Run both versions at the same time until that sample is reached.
  5. Compare the results and decide whether to keep, reject or retest the change.

Choosing the right metric

Conversion rate is the most common metric, but it can mislead when price or basket size changes. A lower price may lift conversion while cutting revenue. For commercial tests, revenue per visitor or gross profit per visitor usually tells a truer story, because it combines how many people buy with how much each order is worth.

Keep one primary metric and a short list of guardrail metrics, such as refund rate, support contacts or page load time. Guardrails catch a winning variant that quietly damages something else.

Reading results without fooling yourself

Small samples produce noisy results. Suppose, as an example, that a home goods shop shows each version to 2,000 visitors, and version B converts 3.3 percent against 3.0 percent for A. That gap is only six orders, and chance alone can easily produce it. A significance test or a confidence interval shows whether the difference is likely to hold.

  • Decide the sample size in advance and avoid stopping the moment a variant looks ahead.
  • Run tests for whole weeks so weekday and weekend shoppers are both included.
  • Treat many simultaneous comparisons with care, since more comparisons mean more false winners.
  • Record losing tests too, because they prevent the same idea being tested again and again.

Price A/B tests in particular

A price test shows different prices for the same product to different visitors at the same time. It answers a question surveys cannot fully answer: what real customers do when they face a real price. For example, a shop selling a 40 dollar phone stand might test 40 dollars against 44 dollars and compare gross profit per visitor, not just orders.

Many teams prefer softer designs that avoid two shoppers seeing different prices for the identical item. Common options include testing a shipping threshold, a bundle, a discount code shown in one group, or a new product variant with its own price. Another approach is a time-based test, in which the whole store moves to a new price for a set period and is compared with a similar earlier period, accepting that this is less clean than a true split.

Fairness, trust and legal cautions

Customers who discover they paid more than someone else for the same item may feel cheated, even when the test was random. Screenshots spread quickly, and a short experiment can cost long-term trust. Some teams honour the lower price for anyone in the higher group who asks, or refund the difference after the test ends.

Rules on price differences, price display and consumer protection vary by country and sometimes by sector. Testing must never vary prices by protected characteristics, and some places require the price shown to be the price charged at checkout. Personal data used to assign groups may also fall under privacy law. Treat this as general information and ask a qualified legal adviser before running live price tests in a new market.

Common mistakes

  • Testing tiny cosmetic changes on low traffic and waiting months for an answer
  • Changing the variant halfway through the test
  • Letting returning visitors flip between versions, which mixes the groups
  • Ignoring competitor moves during the test, which can shift demand for both groups
  • Declaring a winner on conversion rate when the change also lowered average order value

External prices matter more than many teams expect. If a major competitor runs a promotion in the middle of a price test, both groups are affected, and the result may not repeat in a normal week. Keeping a record of competitor prices for the test period, for example with price history, makes the result easier to interpret.

Putting A/B testing to work

Start with changes that are likely to matter and are cheap to reverse: offer presentation, shipping thresholds, bundle structure and product page clarity. Move to direct price tests once the team has a reliable process for sample sizes, guardrails and customer communication.

Keep a simple test log with the hypothesis, dates, traffic, metric, result and decision. Over time the log becomes a map of what customers in your market respond to, which is often more valuable than any single winning variant.

Key takeaways

  • A/B testing randomly splits visitors between a control and a variant and compares one agreed metric.
  • For price tests, revenue or gross profit per visitor is usually a better metric than conversion rate.
  • Plan sample size in advance and run tests over whole weeks to avoid false winners.
  • Price tests carry trust and legal risks that vary by market, so design them carefully and seek legal advice.
  • Record competitor prices during a test so outside moves do not distort the result.

Frequently asked questions

How long should an A/B test run?

Long enough to reach the sample size you planned and to cover at least one full weekly cycle, often two. Stopping as soon as one version looks ahead tends to produce false winners.

Is it legal to show different prices to different customers?

It depends on the country, the sector and how groups are chosen. Random assignment is treated differently from targeting by personal traits, and display rules also apply. Ask a legal adviser for the markets you sell in.

What is the difference between A/B testing and multivariate testing?

An A/B test compares two versions of one change. A multivariate test changes several elements at once and studies their combinations, which needs much more traffic to give a reliable answer.

Can a small shop run A/B tests?

Yes, but it should test bold changes with large expected effects, since small effects need very large samples. Low-traffic shops often learn more from time-based price changes reviewed carefully.

Related terms

In BlackAha, Smart Price helps you get a recommended selling price for every product, based on competitor and market prices.

Back to the Pricing Dictionary