Ilya Brin - Software Engineer

History is written by its contributors

Probability Theory: Bayesian Statistics for A/B Testing in Go

2024-01-18 5 min read Mathematics Ilya Brin

A/B testing is everywhere: button colors, pricing, algorithms. But most implementations use frequentist statistics. Bayesian approach gives you probability of being better, not just “statistically significant”. Let’s implement it in Go.

Frequentist vs Bayesian

Frequentist: “Is the difference statistically significant?”

  • p-value < 0.05 → reject null hypothesis
  • Doesn’t tell you probability of A being better than B

Bayesian: “What’s the probability that B is better than A?”

  • Direct answer: “B is better with 95% probability”
  • Updates beliefs as data arrives

Basic A/B Test

type Variant struct {
    Name        string
    Conversions int
    Visitors    int
}

func (v *Variant) Rate() float64 {
    if v.Visitors == 0 {
        return 0
    }
    return float64(v.Conversions) / float64(v.Visitors)
}

Beta Distribution

For conversion rates, we use Beta distribution:

Continue reading

Arguing that you don’t care about the right to privacy because you have nothing to hide is no different than saying you don’t care about free speech because you have nothing to say.

Edward Snowden
Newer posts