Users experienced difficulties setting up and managing multivariate test use cases using MTRIBES’ existing scheduling and traffic allocation features. The two key issues encountered were:

1. Unintuitive process of allocating traffic across test variants.

2. Difficulty comparing and analysing data across test variants.

We had four key clients express their need for multivariate testing. In discovery sessions, they outlined their specific use cases:

• Sports Client 1 - Test conversion rate of three different pricing packages across different regions.

• Sports Client 2 - Test personalised content variations with different user segments

• Broadcasting Client 1- Test personalised content on three separate devices over three months

• Broadcasting Client 2 - Test a variety of button colours for an upcoming seasonal promotion

Leverage existing mtribes features and data to provide customers with the ability to create tests of their Experience and compare the performance of multiple variants without redesigning mtribes into a dedicated testing platform.

I was responsible for owning this feature end-to-end; conducting research, dictating the design direction and validation process.

To better understand the UX and functionality of multivariate testing features, I conducted a detailed analysis of four testing platforms. Focusing in particular on their setup flows, breadth of functionality (considering whether or not testing was a primary or secondary function of the app) and how they integrate the feature into their IA. This gave me a good indication of user expectations of a testing feature and would better inform my design decisions when ideating.

The platforms I investigated were: 

• Launch Darkly: Focus on feature control, experiment feature is a secondary function 

• Optimizely Full Stack: Focus on feature control, experiment feature is a primary function 

• Amplitude Experiment: Focus on analytics, experiment feature is a primary function 

• Split: Focus on audience segmentation, experiment feature is a primary function

With a tight timeline for release, the focus was to leverage as many existing components and functionality to minimise development time. The intention was to address the two key user problems without introducing additional multivariate testing analytics for mvp, such as statistical significance and confidence intervals.

I explored two different approaches through lofi Axure prototyping and user flows.

Definitions:

Experience: represents a component or element.

Scenario: a variant of an Experience that has been personalised for a specific user group.

Experiences can be changed to an “experiment” mode, using any existing Scenarios as test variants. Would require a significant UI change to allow variant performance comparison and variant control.

Benefits

• ✅ Low risk of affecting experiment data accuracy when making adjustments.

• ✅ Maintains existing Experience > Scenario hierarchy.

Risks

• ❌ Removes fundamental Experience functionality.

• ❌ Low use-case flexibility for the non-developer user.

A new "experiment" type Scenario containing test variants and dedicated testing functionality. It can be scheduled alongside regular Scenarios or other experiments on the same Experience.

Benefits

• ✅ Maintains fundamental Experience functionality.

• ✅ Flexible control for the non-developer user.

Risks

• ❌ Risk of affecting experiment data accuracy when making adjustments to the Experience.

• ❌ Requires some changes in the backend to how users are allocated a Scenario.

I ran testing sessions with seven participants with varying roles and experience with MTRIBES:

Customer

1. Product Owner - experienced, infrequent user

2. Lead Content Analyst - experienced, frequent user

3. Content Analyst - inexperienced, frequent user

Deltatre

1. Operations Manager - inexperienced, frequent user

2. Client Service Lead Developer - experienced, frequent user

3. Client Service Developer - experienced, frequent user

4. Product Engagement Lead - experienced, frequent user

The session covered the end-to-end flow of setting up and monitoring an experiment. I primarily wanted to test the functionality of the traffic allocation feature and the clarity of comparing variant performance.

Overall the feature tested positively, with 6 of our 7 participants completing setting up and managing an experiment.

Key Quotes

“ [It's] really easy to use, I can see what traffic I’ve got where, what’s being served and the different elements of each test ” -  Client Lead Analyst

“ [It's] easy, a lot of the other tools I’ve used are not so easy, so that’s great ” - Client PO

Most participants were successful in identifying and understanding the experiment traffic allocation feature. The only issue of note was a relatively lengthy completion time (average 21s) due to the feature's location in the "targeting" tab, instead of the "activity" tab or “edit variants” modal as initially expected by 71% of participants.

Despite this, 79% successfully located and understood the feature, as reflected in positive feedback.

“This has made it much simpler in terms of having [allocation] in one place” - Client Lead Analyst

When testing the comparison of variant performance, we had similar success to the traffic allocation task, however with the same notable time on task issue. This averaged 19s due to 43% of users initially expecting performance data to show by clicking into the variants table, rather than using the dropdown menus to change the metrics.

71% of participants completed the task in the end, and feedback highlighted expectations for future features, including the expectation of a confidence statistic.

“ In other testing tools that I’ve used, there is often a confidence statistic [...] is it statistically significant? ” - Client PO

The success of this feature's development considering its relatively short timeframe can be attributed to the clear communication and understanding of both the business and user goals by the design and development teams. Our understanding of development restrictions ensured we weren't designing more functionality than we should, and awareness of the root cause of our user's problems ensured the feature we released could still provide maximum impact within the scope of an MVP feature.