How Do I A/B Test My Giving Page?

A/B testing in Virtuous Raise lets you run controlled experiments on your donation pages to identify which version drives the most revenue, no third-party tools required. Create two versions of any donation page, split visitor traffic between them, and use statistical analysis to determine a winner. This article covers how to set up, run, and complete an A/B test in Raise.

Table of Contents

1. What is A/B Testing?
2. Create an Experiment
3. How Your Variant Split Affects Traffic to Your Page
4. Viewing A/B Test Progress
5. How Does Raise Select a Winner?

What is A/B Testing?

When performing an A/B test on your giving page, Raise presents two versions (A and B) of your giving page to supporters. Version A, your Control page, displays the content and layout you might typically publish. In the alternate page, Version B, you make a change, either to the layout, the call to action, graphics, or whatever else you'd like to test. It's best to focus on making just one change, as this helps determine if that particular change drives more engagement. After a winning page is determined, Raise prompts you to approve the winner and make it the live version for all web visitors.

Creating an Experiment

To create a new A/B Test for a Raise giving page:

1. Navigate to the Experiences tab, and open the published page you want to test. You can run one experiment per donation page at a time.
2. Select Actions and Create A/B Test to create a new experiment. 

3. On the pop-up form, enter Test Name and an optional Test Hypothesis for your experiment. For example, your hypothesis may describe using a mission-centric video on one page vs an image on the other.

4. Select a Winning Metric. Note: You cannot change this selection after the experiment starts. The metric options include:
   1. Revenue Per Visitor (default). This is best for evaluating overall page performance.
   2. Highest Conversion Rate. This is best for maximizing the total number of donors.
   3. Highest Average Gift. This is best for maximizing gift size.
   4. Highest Recurring Gift Conversion. This is best for growing monthly donors.
5. Set your Variant splits for how your pages will be presented to supporters. See more below on how the Variant splits work.
6. To have the winning version promoted without manual approval, enable Automatically promote winner.
7. Enter email address(es) for any staff to Send test performance notifications to.
8. Select Create Test to begin configuring your two pages used in the experiment. 
9. In the page builder, toggle between the two Variants at the top to make changes to either one. Select the Sync Variants icon to copy changes from one Variant to the other. Select Save Variants to save your changes and come back to your test draft at any time. Both versions support full page editing, including layout, imagery, copy, ask amounts, and form fields. Because the A/B tests run at the whole-page level, not on individual page elements, we recommend only testing a single variable on each experiment. 

10. When your test Variant is created and you’re ready to publish to supporters, select Start Experiment. Once you start the test, you can no longer edit it. 

How Your Variant Split Affects Traffic to Your Page

Every experiment includes two page versions to use in your tests:

1. Variant A (Control): Your original page is the baseline. It functions as both Variant A and the Control.
2. Variant B: Build the challenger page by duplicating Variant A and changing an element, or starting from a blank canvas.

The traffic split you set determines what percentage of visitors sees each Variant versus the Control. Supporters who fall into the Control audience will not participate in the experiment or affect the winning page. The Variant test participants will then be split evenly in half. For example, setting the Overall Traffic Allocation split to 80% total Variant traffic and 20% Control traffic means:

• 40% of the traffic sees Variant A and engagement influences the winner.
• 40% of the traffic sees Variant B and engagement influences the winner.
• 20% of the traffic sees the Control page and engagement does not affect the winner.

Viewing A/B Test Progress

Any A/B tests running will have an indicator next to the Page Name under the Experiences tab. 

After you start your experiment, you can view progress, pause, or end the test at any time before a winner is selected. To do so, 

1. Under the Experiences tab, select the page running your experiment.
2. Select the A/B Tests tab to find your experiment.

3. Select the name of your experiment to view metrics and traffic data for each variant. From here, you can also take the following actions:
   1. Select Pause to temporarily stop visitors from being assigned to a variant.
   2. Select Complete Test to end the experiment early and manually select a winner to publish as the live version.

How Does Raise Select a Winner?

Raise uses statistical analysis to determine a winner. All of the following thresholds must be met before a winner is declared:

• The experiment has run for 7 or more days.
• The test has received 1,000 or more total visitors.
• Each Variant has met the threshold for your selected winning metric:
  • Conversion Rate: 1,000 or more visitors
  • Revenue Per Visitor: 1,000 or more visitors and 100 or more donations
  • Highest Average Gift: 100 or more donations
  • Recurring Conversion Rate: 100 or more donations

Once all thresholds are met, Raise begins calculating the confidence level. A version is declared the winner when it reaches 95% or higher probability of outperforming the other. 

When your experiment reaches a conclusion for the test, it has a Ready for Review status, and you will receive an email notification. If you enabled Automatically promote winner in the experiment setup, Raise automatically makes this page the live page for all traffic.  If not, select the suggested winner to approve it, or choose the other version instead.