Ilya Brin - Software Engineer

History is written by its contributors

Probability Theory: Bayesian Statistics for A/B Testing in Go

2024-01-18 5 min read Mathematics Ilya Brin

A/B testing is everywhere: button colors, pricing, algorithms. But most implementations use frequentist statistics. Bayesian approach gives you probability of being better, not just “statistically significant”. Let’s implement it in Go.

Frequentist vs Bayesian

Frequentist: “Is the difference statistically significant?”

  • p-value < 0.05 → reject null hypothesis
  • Doesn’t tell you probability of A being better than B

Bayesian: “What’s the probability that B is better than A?”

  • Direct answer: “B is better with 95% probability”
  • Updates beliefs as data arrives

Basic A/B Test

type Variant struct {
    Name        string
    Conversions int
    Visitors    int
}

func (v *Variant) Rate() float64 {
    if v.Visitors == 0 {
        return 0
    }
    return float64(v.Conversions) / float64(v.Visitors)
}

Beta Distribution

For conversion rates, we use Beta distribution:

Continue reading
Newer posts