How Optimizely Web Experimentation enables fast validation of design and UX changes

In this article, we'll demonstrate how we use a tool called Optimizely Web Experimentation to validate design changes and increase conversion rates.

We are currently working with Karndean Designflooring, an international luxury vinyl flooring provider who recently launched a new B2C brand named Palio Express aimed at the UK DIY flooring market. Following a user research programme that we undertook to help Karndean get a deeper understanding of its target audience's needs when researching and buying hard flooring online, we came away with a set of insights and recommendations to improve conversion rates on the Palio Express website.

This article focusses on a number of changes we made to the product detail page (PDP) of the Palio Express website to give you an idea of how Optimizely Web Experimentation works in practice.

Proposed changes to validate

Below are the six potential high-impact and low-effort changes to test on the product detail page (PDP).

1. Swap the default product close-up image of the flooring SKU to the contextual in-room image. 
2. Give the PayPal “3 interest free payments” more prominence by moving it higher up the page.
3. Move the “How to measure my room” link so it’s within context when calculating how many flooring packs are required for a space.
4. Open the product description accordion by default.
5. Move the product guarantee to the top of the page.
6. Embed the video for “How to install the product yourself” directly on the page so that users don’t need to navigate away from the page to learn more about installation (part of a section further down the page).

Using Optimizely Web Experimentation to validate changes

Optimizely Web Experimentation is perfectly suited to testing design and UX changes before committing to development lifecycles. It enables us to run as many A/B or multivariate tests with real users as we need to, until we have enough data to clearly identify the most impactful changes. Once we’ve identified the best changes to make, only then do we take them on the full development lifecycle to be rolled out permanently across the entire web platform.

Web Experimentation is a platform and tech stack agnostic product, which means it can be used stand-alone on top of any website. Optimizely CMS or Commerce is not a requirement to run the same type of website experiments.

Installation is incredibly simple and can be done in a matter of minutes — by simply adding one line of code to the website's html file.

Applying scope to the experiments

In order to validate these six design changes, we’ll be targeting one specific product’s detail page; this helps us keep the scope of the change small, so we don’t cause any unforeseen issues on other variant pages, which in this implementation all use the same underlying page layout.

If we wanted to limit exposure even further, we could also create custom audience segments to only show our experiments to a subset of users. These segments can be defined by characteristics like browser, device, location, language, time of day, traffic source, specific cookies, and further custom attributes specific to the implementation.

Setting up the experiments

For this example, we will be creating an A/B test with the six different variants and our control (the current live page).

We then give the experiment some basic details before we get started with the real fun.

By default, we’re greeted with two variations: the control (original version) and a placeholder variant.

To keep things nice and tidy we can rename the default “Variation #1” to something that has more meaning for what we’re doing. It doesn’t have to be anything fancy as these names exist only so we can tell the variations apart in the reporting.

We start out with a variation that’s not got any changes inside it.

To add changes, we can click on the “Create” button and we get a list of possible change types.

Making changes to existing elements

For our first variation we’re going to be changing the default image that’s shown to the second product image in the image gallery. This requires two HTML element changes using Web Experimentation. The first variation is to change the “selected” border for the two small images below the main image, and the second change is the URL of the main image loaded.

Next, we swap the existing image URL in the HTML source four times with different image sizes so that the image still loads responsively on all devices.

Reordering existing elements

The second variation — where we move the PayPal “3 interest free payments” section — is much simpler to setup. As usual we need to select the element.

Then we can use the “Layout” controls to place it directly before our existing price element on the page.

Our third and fourth variation both follow the same approach as the above: a simple html change to expand the accordion by default and a simple “Layout” panel rearrange to move the “How to measure my room” section, which we will omit for brevity.

Insert entirely new HTML

The fifth variation for highlighting the product guarantee is more interesting as we’ve inserted completely new HTML.

The last variation follows the same approach, so we won’t show that for brevity.

Live preview of variants and changes

We can at any time preview our variants and test them in browser to make sure everything still works like it should by using the gadget injected by Web Experimentation only visible to us.

Using this gadget, we can also override the experiment/variant we are previewing to quickly cycle between them.

With all our variants setup we can now add a metric.

Choosing metrics

Before an experiment can be started, we need to add at least one metric. This is the “goal” we tell Web Experimentation is the desired outcome or activity we want to increase.

There are a few possible options available out-of-the-box, but we’re going to be using our custom metric for adding a product to the basket. 

Creating the custom metric was almost as simple as installing Web Experimentation in the first place. All we needed to do was paste in a provided snippet of code into our shared add to basket front-end component so that it can report back to Web Experimentation when users perform the action.

It is also possible to add secondary metrics for even richer tests and results, but we will concentrate on the primary metrics for the purpose of this demonstration. 

We are now ready to start the experiment and examine the results.

Results

After a couple of minutes, the results will start updating in near real-time as users interact with our variants.

Here’s an example showing how the “PayPal payment” variant and the “Highlight product guarantee” variant leaped ahead in revenue per visitor while “Expanding the product guarantee” had some uptick and adding the video for “How to install the product yourself” led to an increase in full size sample orders.

When we leave the test running for long enough so that enough users have interacted with it, Web Experimentation will give us the statistically relevant best results. As with any dataset, anomalies can occur with small samples sizes.

With statistically relevant results we now have changes validated with real users in the real world, so we know which of the six potential variants make the biggest impact, and we can focus our development efforts on those first.

Wrapping up

We’ve shown at a high level how Optimizely Web Experimentation can be used to improve conversion rates quickly and efficiently. In doing so, we have extracted meaningful data in the process, to help focus development time on the areas where the biggest impact can be made.

If you would like to learn more about how Optimizely Web Experimentation can be used to increase conversion rates, customer acquisition, retention and ongoing engagement don’t hesitate to get in touch.