django-experiments 0.1

Experiments and Alternatives

At this point it normally helps if you know what AB testing is, if not, check out [wikipedia] (http://en.wikipedia.org/wiki/A/B_testing).

The experiment is manually created in your nexus admin.*

An experiment allows you to test the effect of various design alternatives on user interaction. Nexus Experiments is designed to work from within django templates, to make it easier for designers. We begin by loading our module:

{% load experiments %}

and we then define our first experiment and alternative, using the following syntax:

{% experiment EXPERIMENT ALTERNATIVE %}

We are going to run an experiment called “register_text” to see what registration link text causes more users to complete the registration process. Our first alternative must always be the “control” alternative. This is our fallback if the experiment is disabled.

{% experiment register_text control %}
    <a href = "register.html">Register now.</a>
{% endexperiment %}

So while the experiment is disabled, users will see a register link saying “Register now”. Let’s define another, more polite alternative:

{% experiment register_text polite %}
    <a href = "register.html">Please register!</a>
{% endexperiment %}

While experiment is disabled, users will still see the “control” alternative, and their registration link will say “Register now”. When the experiment is enabled, users will be randomly assigned to each alternative. This information is stored in the enrollment, a unique combination of the user, the experiment and which alternative they are assigned to.

*Experiments will be dynamically created by default if they are defined in a template but not in the admin. This can be overridden in settings.

Goals

Goals allow us to acknowledge when a user hits a certain page. You specify them in the EXPERIMENTS_GOALS tuple in your settings. Given the example above, we would want a goal to be triggered once the user has completed the registration process.

Our setting:

EXPERIMENTS_GOALS = ("registration")

Our registration successful page will contain our goal, “registration”:

{% experiment_goal "registration" %}

This will be fired when the user loads the page. There are two alternate ways of using goals, using either a JavaScript onclick event, or cookies. These allow for onclick goals, for example:

<button onclick="experiments.goal('registration')">Complete Registration</button>

or

<span data-experiments-goal="registration">Complete Registration</span>

The goal is independant from the experiment as many experiments can all have the same goal. The goals are defined in the settings.py file for your project.

Confirming Human

The framework can distinguish between humans and bots. By including

{% include "experiments/confirm_human.html" %}

at some point in your code (we recommend you put it in your base.html file), unregistered users will then be confirmed as human. This can be quickly overridden in settings, but be careful - bots can really mess up your results!

The States

Control - The experiment is essentially disabled. All users will see the control alternative, and no data will be collected.

Enabled - The experiment is enabled globally, for all users.

Gargoyle - If a switch_key is specified, the experiment will rely on the gargoyle switch to determine if the user is included in the experiment. More on this below.

Using Gargoyle

Gargoyle lets you toggle features to selective sets of users based on a set of conditions. Connecting an experiment to a gargoyle “switch” allows us to run targeted experiments - very useful if we don’t want to expose everyone to it. For example, we could specify to run the result to 10% of our users, or only to staff.