A/B testing on Sitefinity CMS platform: how to run regular experiments

Tags: ,

We keep sharing useful experience of Digital Design working with the Sitefinity CMS platform. We have already told about the content personalization and conversion optimization using Sitefinity’s built-in tools, and in this article, we will talk about the A/B testing.

Sitefinity module for A/B testing allows to evaluate ideas for improving the user experience based on hard data and experiment with any part of the page: layout, position of page elements, content elements, images or CTA button design. Sitefinity also provides the ability to calculate not only the result and statistical significance of the test, but also the potential increase in conversion for the next 30 days for each of the audience segments.

At the preliminary stage, you choose a test hypothesis (a problem, a supposed cause and expected result of the changes), determine the goal (a set of metrics which will be used to evaluate the result) and the optimized element/page. These are further steps of A/B testing on Sitefinity:

Step 1: A/B test creation

Information that must be specified:

  • A/B test original page.
  • Experiment run time.
  • Number of page variants participating in the experiment.
  • Traffic distribution between the original page and its versions.
  • Campaign goals.

You can create your own goals for the test or choose from the following list:

  • Next viewed page.
  • Form submission.
  • Sitefinity Insight Conversions (previously configured in the Conversions module).

For example, an A/B test for a page with a “Contact Us” form lasts 30 days, 50% of visitors see its alternative version, and the other 50% see the current version. The “Contact Us” form is moved to a different position on the alternative page. The main goal is the submission of the form.

A/B testing on Sitefinity CMS platform: how to run regular experiments - 1

Step 2: Development of alternative variants of designs or content

The most important source of ideas for changing content or widgets as part of the A/B test is the analysis of site conversions and attributions. For example, if we are able to find a touch point that significantly increases the probability of a conversion, we can experiment with adding an appropriate link to the site pages within the customer journey.

If you want to create an alternative variation, you can change any part of the page, including such elements as:

  • Page titles.
  • Forms.
  • Content.
  • Page layout and widgets.
  • Text, colour, size or position of CTA or any other button.
  • Links.
  • Images.

A/B testing on Sitefinity CMS platform: how to run regular experiments - 2

Step 3: A/B test start

Sitefinity provides the following A/B test management options:

  • Launch.
  • Editing.
  • Deleting.
  • Status change (Active/Stopped/Completed).
  • Continuation.
  • Completion (i.e. select the winning version and publish it on the page).

A/B testing on Sitefinity CMS platform: how to run regular experiments - 3

Step 4: Implementation analysis

After the start of the A/B test, Sitefinity Insight collects data and generates detailed reports on its results:

  • Number of visitors to each of the page variations within the test.
  • Number of achieved goals and goal achievement rate for each page variation (goal completion rate graph shows how it changes over time).
  • Increasing the conversion rate for each option.
  • Improvement rate (may be negative if the version of the page performs worse than its original).
  • Statistical significance (must be calculated using p-value, target confidence level is at least 95%, i.e. p-value 0.05 is the standard of test result significance).
  • Winner option, if any (can be determined only if there are results with high statistical significance and not earlier than two weeks after the start of the test).

You can also download the results of the test and conduct a more detailed analysis yourself.

A/B testing on Sitefinity CMS platform: how to run regular experiments - 4

After the test is completed, based on the goal achievement metrics and the statistical significance of the results, the winning option is selected and it becomes the default page.

In the following articles, we will touch on the topics of creating customer profiles for group of users, analytics, SEO and triggered e-mail campaigns on Sitefinity.