A/B tests are a great way to accurately measure the impact that Chameleon is having and / or compare different versions of a product tour.

Chameleon makes it easy to run a control group (Chameleon vs. nothing) or AB test (two different Chameleon Tours). We recommend starting with a control group test and iterating until your Tour is effective, before testing Tour variations. 

A/B testing is most relevant at scale (>10k MAUs) and is available on Growth plans; contact us to discuss. 

Control Group testing

There are two ways to deliver Chameleon Tours (read more here) and testing here is relevant for Automatic Tours only. 

Setting a Control Group

Once you have selected the Automatic option, you will be able to turn on a Control Group and set the proportion of users that should be part of the holdback / out / control group.

The image means that a random 20% of the target audience users will not see this Chameleon Tour. 

Analyzing results

Whenever a user within the Target Audience is liable to see this Tour, an event ("Chameleon Experiment entered") will be logged. This will be available within all connected analytics integrations

Note: Within Mixpanel, the event name will be "Experiment Started" to better match with Mixpanel's experimentation analysis framework.

For this event the following properties will also be logged:

  • Group 
  • Testing ID
  • Tour name
  • Page URL
  • ...

The "Group" property values are either:

  • Control (Out) -- user is part of the control group and will not see the Tour
  • Test (In) -- user is part of the test group and can see the Tour

Other events (e.g. "Tour started") will also be logged as normal once a user starts interacting with the Chameleon experience. You will see all these within your analytics platform* (e.g. Amplitude, Heap, Mixpanel, Google Analytics) and can use this to further analyze conversion or relative impact of your Chameleon experiment. 

*Updated reporting within the Chameleon dashboard to measure experiment success and goal conversion is coming soon.

A/B and multivariate testing 

Chameleon also lets you more manually show an Experience to a random proportion of your target audience. You can use this to show two different Tours to different groups, or to group multiple tests to the same random sample. 

Manual set test group

To create a test group, simply add an extra filter ("sampling filter") to your Target Audience segment that selects a random sample of users of the desired size. 

This means you can still target users based on other conditions, such as user properties, events, data sources etc. 

To add the sampling filter:

  1. Select 'Default properties' as the type of filter
  2. Select 'Percentage' in the next dropdown
  3. Use 'more than' or 'less than' to define the range of users
  4. Set the boundary number for this "Percentage value"

In the above example, users that have a "Percentage value" between 50 and 100 would be targeted. This would constitute 50% of the users within the group defined by the other segment filters. 

To target 10% of users, you could use either:

  • Percentage more than 90
  • Percentage less than 10

You could also use multiple filters to target another 10% group (e.g. Percentage more than 10, and Percentage less than 20). 

You can use the same filter configuration in another segment (by re-creating the filter) to target the same user group. This means you can run multiple A/B tests to the same user group. 

How to A/B test Tour variations

To test two variations of a tour:

  1. Create the control variation of the tour, including the target audience, with the sampling filter above
  2. Duplicate the control tour, update the tour name (using the variant name / label) and then re-create the target audience. This time use the opposite sampling filter, to target the alternative user group
  3. Set both tours live

You will be able to analyze the results through Chameleon, or within the analytics software you have connected. 

You will be able to use the tour name to segregate the data, as each Chameleon event contains additional attributes, such as Tour name, User ID, URL etc. 

How it works

Chameleon automatically appends a random value ("Percentage value") between 0 and 100 to each of your users. This will be persistent and you can leverage this to target users. 

Did this answer your question?