Running A/B split tests is a common practice for the internet marketer.  The approach is fairly straight forward:  Run two concurrent versions of you ad / landing page / creative / etc,  one being a control (current version) the other your treatment (version that you think will do better).  While the approach is fairly simple, quite a few people get hung up on how to determine the correct sample size.  There are plenty of resources out there that give recommendations ranging from always using 100 conversions to complex statistical formulas.  While you can pull a number out of a hat or run some complex math, there is a much easier method:  The Convergence Method

It's pretty simple.  Let's break it down:

In a typical test you run the following:

• A - Control
• B - Treatment

You stop the test once you hit your predetermined sample size you spent hours calculating.

Using the Convergence Method, you run the following:

• A1 - Control #1
• A2 - Control #2
• B - Your treatment

By monitoring the convergence in performance of the two exact same controls you're able to understand when you've hit your minimum sample size.  In other words, when the cumulative performance of A1 and A2 come within your required variance the overall results of your test are statistically valid.   An expected variance might be .5% or whatever you choose.

Here's what it looks like:

As you can see above, the the cumulative conversion variance between A1 and A2 decreases over time and eventually converges into a relatively steady variance of approximately +-.1% around hour 13.  It's at this point in time that we consider the overall results of the test valid.  It's also important to note that your measure must be cumulatively computed over time, not at discrete time intervals.

In addition to monitoring the convergence of the controls, you'll also want to consider the length of the test based on potential variability in performance over a given time period.  In this test I chose 24 hours.  You may want to run it longer if you feel day of the week would affect the performance of your control and treatment at independent levels.  Don't get too worried about this though.  Odds are that most time of day and day of week variances will affect your control and treatments equally.

A lot of material out there regarding calculating sample sizes assumes you need calculate it before you run the test.  In the offline world this is usually true.  You need to know how many mailers to print or survey candidates to find.  With internet marketing tests you generally have access to real-time performance measures, which is what enables this method to work.

So put down that stats book and calculator and use convergence in your next test.

Minimum Viable Product (MVP) is a fundamental component to Eric Ries' Lean Startup methodology.  Eric gives a great definition in his March 2009 interview with Venture Hacks.  Here's my take on MVP and why it's so effective.

MVP is the concept of completing the minimal amount of work in order to prove two things:

1. People want your product
2. There's a business model

You're trying to learn whether or not your assumptions about the world needing your product are true.  The focus is learning.   Not profitability, scalability or defensibility.  Those things come once you've proven that early indicators or measurements support you have a market and model.  The disciple of differentiating features between your market/model and other objectives is where the true value of this process lies.  The process forces you to think objectively about how each feature is going to help you better understand your product's viability in order of importance.  Very often this process is skipped leading to wasted effort on features that do nothing to prove or disprove your assumptions.

For example, you decide to include the ability for users to "remember my login".  You end up spending 1 day of development and testing to complete this functionality.  However, after the first release  to market you realize that users are simply not signing up for your product at the rate you assumed.  You then move into your second iteration focusing in on the sign up process.  In retrospect the one day spent developing the “remember my login” functionality did nothing to better understand the market or model.  It was a great feature, users loved it, but that didn't matter.  Users didn't even like the idea of the product enough to sign up to use the great feature.

So you’re probably thinking, ok fine, eliminate that feature, but what if the signup rates were spectacular and user retention was horrible due to the missing feature?  I'd say that's an excellent problem to have.  You've learned your priority #1 assumption is holding true, now it’s time to focus on retention or whatever your secondary assumptions might be.

Ok, fine.  But what if those individuals I converted into users never come back?  They might not.  That’s ok.  Odds are with your limited test marketing budget you still have the majority of the world’s population left convert into a user.

But what if I receive bad press from those initial users?  Any press at all is a good thing.  Its feedback.  Receiving no feedback is worse.  You don't have a clue why users hate your product.  Great example of this is Twitter.  Twitter received a massive amount of bad press regarding the stability of their platform in their early days and even still today.  I didn't really matter.  Users still used their product because they loved it.  They had the MVP nailed.

This concept can be extremely difficult to do, but the results are worth it.  Removing all kinds of stuff you really think is cool is painful.  It may even completely change the makeup of the product.  But that’s the point.  The increased level of pain up front pays dividends in the long run by getting your product to market faster allowing you to learn and iterate towards a product people love.