Jeff Bollinger

17Aug/100

A/B Testing Sample Size – The Convergence Method

Running A/B split tests is a common practice for the internet marketer.  The approach is fairly straight forward:  Run two concurrent versions of you ad / landing page / creative / etc,  one being a control (current version) the other your treatment (version that you think will do better).  While the approach is fairly simple, quite a few people get hung up on how to determine the correct sample size.  There are plenty of resources out there that give recommendations ranging from always using 100 conversions to complex statistical formulas.  While you can pull a number out of a hat or run some complex math, there is a much easier method:  The Convergence Method

It's pretty simple.  Let's break it down:

In a typical test you run the following:

  • A - Control
  • B - Treatment

You stop the test once you hit your predetermined sample size you spent hours calculating.

Using the Convergence Method, you run the following:

  • A1 - Control #1
  • A2 - Control #2
  • B - Your treatment

By monitoring the convergence in performance of the two exact same controls you're able to understand when you've hit your minimum sample size.  In other words, when the cumulative performance of A1 and A2 come within your required variance the overall results of your test are statistically valid.   An expected variance might be .5% or whatever you choose.

Here's what it looks like:

As you can see above, the the cumulative conversion variance between A1 and A2 decreases over time and eventually converges into a relatively steady variance of approximately +-.1% around hour 13.  It's at this point in time that we consider the overall results of the test valid.  It's also important to note that your measure must be cumulatively computed over time, not at discrete time intervals.

In addition to monitoring the convergence of the controls, you'll also want to consider the length of the test based on potential variability in performance over a given time period.  In this test I chose 24 hours.  You may want to run it longer if you feel day of the week would affect the performance of your control and treatment at independent levels.  Don't get too worried about this though.  Odds are that most time of day and day of week variances will affect your control and treatments equally.

A lot of material out there regarding calculating sample sizes assumes you need calculate it before you run the test.  In the offline world this is usually true.  You need to know how many mailers to print or survey candidates to find.  With internet marketing tests you generally have access to real-time performance measures, which is what enables this method to work.

So put down that stats book and calculator and use convergence in your next test.

  • much does cost cvs
  • per i bambini
  • how to take for pregnancy
  • is a maoi inhibitor
  • and adderall
  • fda black box warning
  • cardiovascular side effects
  • tri pak 500 mg
  • comparison
  • 500 bestellen
  • y cerveza
  • heart problems
  • ditropan and
  • u bosni
  • alcohol
  • does work yahoo
  • corporation transfer agent
  • can i use for ear infection
  • is there a difference between and aviane
  • ear ringing
  • jelly online kaufen
  • potassium vitamin k
  • black stool
  • hct cmi
  • and therapy
  • 3 par jour
  • online purchase
  • buy generic online
  • can take oxycodone
  • yan etkisi
  • safe take during pregnancy
  • cirugia guatemala
  • extreme tiredness
  • interactions
  • side effects memory loss
  • can i take wellbutrin and together
  • lyrica interactions
  • can cause constipation
  • after 10 years
  • e menopausa
  • use of in healthy men
  • side effects hct
  • chiffre d'affaire
  • weight gain side effect
  • effects on the brain
  • can cause irritability
  • ear cream
  • elderly side effects
  • recreational
  • cheapest price for
  • woods florist
  • teething pain
  • e disturbo bipolare
  • ingredients
  • bbt chart
  • can i take prozac and together
  • free newsletter
  • bula 90mg
  • unavailable
  • conjugated estrogen cream
  • citalopram withdrawal symptoms
  • fake photos
  • mcneil
  • kidney stones
  • tricyclic antidepressant
  • is a generic
  • versus xr
  • normal progesterone level while taking
  • prescription assistance
  • thrombocytopenia dose related
  • compare
  • administration routes
  • canine dosage
  • uses side effects interactions drugs
  • is good for urinary tract infections
  • effects women videos
  • allegra mix
  • free sample pack
  • should taken am pm
  • time day should take
  • musica me senhor aline barros
  • withdrawal symptoms now
  • what time of day should you take
  • how long does it take for to work
  • how long does pm take to work
  • vs calcium
  • cured my acne
  • ssri combination
  • milligrams
  • 4 month old taking
  • 1 hora antes
  • price uk
  • taking adipex and together
  • hard capsules
  • side effects stopping surgery
  • liquido
  • taking during dbol cycle
  • que es es 600
  • e tosse
  • patient information leaflet
  • drug family
  • hctz vs hct
  • mixing alcohol
  • heart
  • prospecto 1mg ml
  • dry mouth
  • picture 10 mg
  • 2 tablet
  • long good after expiration
  • uk pharmacy
  • dosage of for lpr
  • kidney failure
  • images
  • labs
  • drug interaction
  • use during pregnancy
  • muscle weakness
  • to prozac side effects
  • al 5322
  • signs of pregnancy using
  • works quickly
  • nursing implications
  • milk magnesia
  • can take wellbutrin together
  • how many milligrams of for yeast infection
  • rxlist
  • amazing
  • mg
  • using for molluscum
  • tooth infection
  • apartment homes
  • vs vs benadryl
  • effects on kidneys
  • what are normal levels
  • testicular pain
  • dosage 12 lb. baby
  • trailer
  • difference between
  • tolerability prophylactic regimens
  • which is stronger or
  • stomach pain
  • xarope bula
  • treatment ptsd
  • dose for bacterial overgrowth
  • feeling weird
  • fda black box warning
  • vertigo
  • use in preterm labor
  • where can i buy shampoo in the philippines
  • class action lawsuit and ulcerative colitis
  • safety pregnancy
  • stability syringe
  • females kidney stones
  • withdrawal symptoms
  • bijsluiter
  • vs for gad
  • soy isoflavones like
  • long term use
  • effect
  • risk of stent thrombosis with
  • for e coli
  • inhaler pregnancy category
  • e bentelan
  • will treat kidney infection
  • medication class
  • withdrawal benadryl
  • dosing protocol
  • rhoxal 500mg
  • launch date
  • comprar contrareembolso
  • face cream
  • walmart
  • tablets 50mg
  • sertraline interactions
  • class medication
  • can you take oxycodone with
  • low grade fever
  • long term side effects
  • on drug test
  • drug alternatives
  • emulgel p
  • dementia
  • riesgos tomar pastillas
  • harga obat
  • overdose fatal
  • dosage dental prophylaxis
  • flu shot while taking
  • enlarged breasts
  • chest pain
  • reduce thyroid nodules
  • buy generic online cheap
  • arnica gel
  • vs side effects
  • for female hair loss
  • treatment fibromyalgia
  • 500 mg po bid
  • cream hair loss
  • rosacea
  • how does work in the body
  • taking together
  • pain management
  • side effects hair loss
  • during pregnancy
  • side effects joint pain
  • to treat fibromyalgia
  • indian tablets for women
  • and alcohol
  • interaction
  • zoloft same time
  • solubility water
  • compositum jarabe
  • use of for healthy men
  • 24 hour side effects
  • buzz
  • printable coupon
  • d
  • side effects heart
  • low cost canada
  • sleeping problems
  • taking tylenol
  • pill cutter
  • and tinnitus
  • or rogaine for frontal hair loss
  • is emulgel prescription
  • ointment
  • how often to apply
  • medicamento controlado
  • discount card
  • india usv
  • causes joint pain
  • vs wellbutrin xl
  • website
  • kann man kaufen apotheke
  • can i take and together
  • definition
  • not covered by insurance
  • reaction treatment
  • zoloft
  • days 3-7 and when to have intercourse
  • does work better than
  • para que sirve el retard
  • treating cystic acne without
  • when do you ovulate on 1-5
  • aeroporto di
  • coverage anaerobes
  • safe take while breastfeeding
  • canada price
  • ruined my face
  • msds
  • safe first trimester
  • smoking while on
  • taking first trimester
  • ms treatment
  • difference between
  • letra musica gospel minha vida
  • safe while pregnant
  • postoperative nausea vomiting
  • can you take vicodin
  • reduced appetite
  • and pregnancy test
  • three day dose
  • albumin stability
  • anti-nausea medicine
  • 1.5 tropfen
  • brand name er vs generic
  • weight loss success stories
  • icd 9 code for injection
  • can i take both benadryl and
  • infants cost
  • ok exercise after taking
  • regles 2 fois
  • is 40mg of a lot
  • stopping side effects
  • can taken tylenol
  • depression side effects
  • drug long term use
  • can take
  • mg
  • image
  • cold sore cream tablets
  • interaction
  • taking xanax
  • staph aureus coverage
  • lower abdomen pain after
  • syndrome raynaud
  • walmart pharmacy cost
  • for hives dosage
  • profuse sweating
  • email virus yahoo
  • gullna
  • odontopediatria
  • makes you drowsy
  • 75 recommended dosage
  • phentermine and for weight loss
  • hives after taking
  • 5mg uso diario
  • venta de en concepcion chile
  • pacific capital inc vancouver
  • dizziness
  • will i lose weight after quitting
  • induced thrombocytopenia mechanism
  • side effects heart
  • hd depression
  • effects on children
  • e coli
  • and breastfeeding safe
  • side effects ds oral
  • can you take d daily
  • laboratorio fabricante do baby
  • stavebnA a obchodnA
  • zoloft
  • food interactions
  • tratamiento uAas
  • difference
  • fake commercial
  • generic 225 mg
  • is a prescription drug
  • pt assistance program
  • side effects
  • medicamento indicatii
  • insomnia temporary
  • drug interaction advil
  • skin
  • keramag nr. 1 mit absenkautomatik
  • facial hair loss
  • ds lactation category
  • used itching
  • flu
  • and better
  • liquid children
  • yan etkileri 30 mg
  • 3986154526
  • can you take
  • taking after
  • klonopin together
  • pradaxa interaction
  • acne side effects
  • infusion iv
  • and together
  • can i take while taking
  • vs generic
  • gastritis
  • can you take and xanax together
  • walmart generic
  • irregular heartbeat
  • la 4mg capsules
  • can you order online
  • clamidia y
  • vs vs
  • 20 mg dosage
  • directions
  • d safe during breastfeeding
  • head injury
  • stop taking
  • marketing history
  • and drinking alcohol
  • advertising slogans
  • or
  • liquid
  • dose lupus nephritis
  • nebules fiyatA
  • v
  • extreme fatigue
  • dosage epocrates
  • can take bronchitis
  • printable coupon
  • pills for men
  • peach pill 10
  • hcl
  • how much does it cost
  • health canada
  • epocrates
  • can have glass wine while
  • best price
  • e mercilon
  • is a narcotic
  • un yan etkileri
  • medication assistance program
  • teeth cleaning
  • 7781630746
  • long term
  • contre indication de l
  • can i take with dairy
  • is approved for women
  • side effects of stopping
  • 145 mg cost
  • generic medicine
  • buy in us
  • golf commercial woman
  • monograph
  • can you take while nursing
  • can u take mucinex and together
  • sleeplessness
  • is 75 safe for pregnant women
  • does treat lyme disease
  • order online
  • sinusitis dosage
  • pastillas cuanto cuestan peru
  • can i take and together
  • alternatives