A/B Test Calculator Online

First of all, find out what kinds of things you want to compare. It can be employee A and employee B, or it can be design A and design B, or product A and product B, and so on. Anything. You also need an event that can be measured, that has a count. For example count of visits, count of emails sent, count of people in the experiment or something like these. Then specify what is your conversion, i.e. a special event or a positive case that is a specified subset of all cases. If you have these all, just fill in the four input fields and you will get the ratios, difference and the certainty of the result. The question, the names of events and cases can also be rewritten, despite they are not input fields, but avoid these A/B test common mistakes.

Whenever you have to make a decision, you will want to compare cases and choose the better one. You may do so by tossing a coin, intuition, common sense, or other unreliable ways but it is better to use your data. You can read more about "Ten Reasons Why A/B Split Testing Is Necessary"

Suppose you have two cases, case A has 20 conversions out of 100 events (20%), and case B has 30 conversions out of 100 events (30%). In this example, the simple difference: 30% - 20% = 10%. The proportional difference: (30% / 20%) * 100 - 100 = 50%. This means that case B is 50% better than case A, despite that simple substraction of conversions rates shows only 10%. Proportional difference is generally more useful because that will tell you the increase in the number of conversions.

It's a two-sample hypothesis testing that calculates the statistical significance with a very complex algorithm based on gauss distribution. But we don't want to bore you with math behind it, just use it. You don't need university-level math in order to use a tool that is based on that.

Because the difference between the rates of case A and case B may come from the work of coincidence. Your data comes from an experiment or observation that is not exactly repeatable, they are not accurate, there is a fluke in them. If you would measure again, you would get different values. This distribution causes that you can be sure about one case is better than the other, only if the numbers and differences are big. The greater the numbers and the difference level of the ratios, the greater the certainty. Think about it: If you have a red website with 100 visitors and 10 of them buy something, and you have a blue website too with 100 visitors likewise where 12 of them buy, it may be just the effect of randomness, so you cannot say it with complete certainty that the blue website performs better.

You can easily calculate whether case A or case B is better, but can you tell how sure you are? Well, sometimes yes, sometimes no. In case of small numbers and small differences in ratios (e.g. A:100->51, B:100->52) it is not sure that there is a real difference. In case of big numbers and big differences in ratios (e.g. A:1000->45, B:1000->756) it is sure that there is a real difference. But what if A:100->50 and B:100->60, how sure are you? The above certainty value shows you as a percentage how much you can be sure about the fact that one case is better than the other.

It means that from these numbers it cannot be known whether case A or case B is better. So it does not mean that there is no difference and the difference experienced is the work of coincidence, but it means you cannot be sure, whether it is the work of coincidence or a real difference. The certainty is said to be low below 95%, so you can't be sure about the result. Generally, 95% or greater sureness is required. If you reach 99% or better, then you can be sure already. (But there is 1% chance, that the difference happened because of a very rare coincidence.)

It means that one case is surely better than the other. Usually above 95% or 99% certainty level is considered to be high. It's important that despite of the certainty being high, it only means that one case is better than the other, but the difference between the two cases may be minimal. This is why you must also check the experienced difference of ratios.

You need more data. If you continue your experiment or observation with a larger number of events, you will get better certainty, even if the ratios don't change.