Statistical Significance Criteria

By default, Keak uses robust statistical calculations on each new visitor to evaluate significance.

Statistical significance is calculated continuously so that each test is exposed to the minimum number of visitors required to get a trustworthy result. This means Keak won’t unnecessarily keep a test running once it has enough evidence to pick a winner, saving you traffic and time.

Before Keak concludes a test, two conditions must be met:

Confidence level threshold is reached: The current confidence (statistical certainty) that one variation is better than the other must exceed 95%.
Minimum sample size is met: The test has accumulated at least the minimum number of impressions (visitors) you specified. This is to ensure the result isn’t called too early on very little data. For example, you might require at least 1000 visitors per variation before trusting the outcome, even if confidence shoots high with fewer views.

Only when both criteria are satisfied will Keak stop the test and declare a winner. If the confidence level is high but you haven’t hit the minimum sample, the test will continue (and vice versa). These safeguards maintain test integrity and give you reliable results.

Here is how the confidence level is calculated:

Conversion Rate Comparison We calculate each variation’s conversion rate by dividing conversions by total visitors. We then use a standard statistical test (a two-proportion z‑test) to see whether any difference in these rates is large enough to be more than random fluctuation.
Confidence & p‑Value The two-proportion z‑test produces a p‑value, which tells us how likely it is to see a difference as large as we observed if, in reality, there were no true difference. If the p‑value is below 0.05, we conclude that the difference is statistically significant.
Automated Decision-Making Our platform continuously monitors each test’s p‑value and automatically declares a winner when the difference meets or exceeds the specified confidence level (95% by default).

PreviousReal-Time Analytics and Dashboards NextKeak Video

Last updated 11 days ago

Statistical Significance Criteria

By default, Keak uses robust statistical calculations on each new visitor to evaluate significance.

Before Keak concludes a test, two conditions must be met:

Confidence level threshold is reached: The current confidence (statistical certainty) that one variation is better than the other must exceed 95%.
Minimum sample size is met: The test has accumulated at least the minimum number of impressions (visitors) you specified. This is to ensure the result isn’t called too early on very little data. For example, you might require at least 1000 visitors per variation before trusting the outcome, even if confidence shoots high with fewer views.

Here is how the confidence level is calculated:

Conversion Rate Comparison We calculate each variation’s conversion rate by dividing conversions by total visitors. We then use a standard statistical test (a two-proportion z‑test) to see whether any difference in these rates is large enough to be more than random fluctuation.
Confidence & p‑Value The two-proportion z‑test produces a p‑value, which tells us how likely it is to see a difference as large as we observed if, in reality, there were no true difference. If the p‑value is below 0.05, we conclude that the difference is statistically significant.
Automated Decision-Making Our platform continuously monitors each test’s p‑value and automatically declares a winner when the difference meets or exceeds the specified confidence level (95% by default).

PreviousReal-Time Analytics and Dashboards NextKeak Video

Last updated 11 days ago