A/B testing is everywhere: button colors, pricing, algorithms. But most implementations use frequentist statistics. Bayesian approach gives you probability of being better, not just “statistically significant”. Let’s implement it in Go.
Frequentist vs Bayesian
Frequentist: “Is the difference statistically significant?”
- p-value < 0.05 → reject null hypothesis
- Doesn’t tell you probability of A being better than B
Bayesian: “What’s the probability that B is better than A?”
- Direct answer: “B is better with 95% probability”
- Updates beliefs as data arrives
Basic A/B Test
type Variant struct {
Name string
Conversions int
Visitors int
}
func (v *Variant) Rate() float64 {
if v.Visitors == 0 {
return 0
}
return float64(v.Conversions) / float64(v.Visitors)
}
Beta Distribution
For conversion rates, we use Beta distribution:
Continue reading