Manage experiments
After activating an experiment, you can monitor its performance, manage its state, and evaluate results. Managing an experiment's state works in a similar way to managing a campaign's state.
An experiment's schedule affects its state similarly to that of a campaign schedule.
Be aware of overlap when running multiple experiments or campaigns that target the same customer group. Overlapping promotions can dilute the statistical significance of your results, making it difficult to distinguish which incentive is driving customer behavior.
Evaluate experiment results
You can monitor the performance of an experiment in the experiment dashboard.
To open the experiment dashboard:
- From the left-side menu, click Experiments to open the experiment list.
- Click the name of an experiment.
- On the left-side menu of the experiment, click Dashboard.
To the right of the experiment's name, the experiment's state is displayed, for example, Running.
While Talon.One provides data-driven recommendations, we recommend reviewing the detailed metrics before making business decisions. For example, you can compare metrics, such as revenue gains versus discount costs.
The following details are shown below the experiment name, along with an AI-generated summary:
- Experiment ID: The ID of the experiment.
- Campaign ID: The ID of the campaign that was generated upon experiment creation.
- Created: The date and time when the experiment was created.
- Start date: The date and time when the experiment starts, as set under Schedule.
- End date: The date and time when the experiment ends, as set under Schedule.
Understand the summary
The summary is an AI-generated explanation of your experiment's results.
The AI model that generates the summary analyzes the metrics in the evaluation table that express averages, and the confidence scores for these metrics. It reports whether there's a winning variant, a trade-off between variants, or an inconclusive result. It also highlights any metrics with a significant change, and can provide recommendations based on the experiment's outcome.
- Talon.One enables the AI-generated summary by default. If your organization's policy does not allow AI-powered features, contact your Customer Success Manager to disable this feature.
- The AI model behind the summary only receives the metrics and confidence scores shown in the evaluation table. It does not receive individual order values or personally identifiable information.
Use the summary to, for example, decide whether to roll out a winning variant as a campaign. You can also use it to understand the experiment's outcome without analyzing the data yourself.
To ensure your summary reflects the latest data collected for your experiment, click Regenerate. This option becomes available after your experiment has collected enough data.
Understand the evaluation table
The evaluation table displays the metrics and values used to evaluate your experiment's performance and generate data-driven recommendations. It includes the following columns:
| Column | Description |
|---|---|
| Metrics | The metrics that Talon.One evaluates for the experiment. For a description of each one, see Experiment metrics. |
| Confidence | Values that indicate whether the performance difference between Variant A and Variant B is a statistically significant and repeatable result, or is due to random chance. The confidence score values are calculated using Welch's two-sided t-test and displayed as percentages. Note: You can only view the confidence score for metrics that express averages: Gross average order value (AOV), Net average order value (AOV), and Average units per order (UPO). For the other metrics, this column shows N/A. To see how these scores have changed over the duration of the experiment, refer to the Confidence chart below the evaluation table. |
| Variant A/B | The respective variant's values for the data points listed under Metrics. These values are displayed as either currency or integers, depending on the metric type. The experiment's goal determines the winning variant. When the goal sets a primary metric, the variant that reaches statistical significance on that metric is displayed as the Winning variant. When the Other goal type is used, if one variant has a best overall score, that variant is displayed as the Winning variant. |
| Difference (B vs A) | The difference in results between the variants. In the left column, the difference values are displayed as either currency or integers, depending on the metric type. In the right column, all values are displayed as percentages. Note: If Variant A has the best overall score, these columns appear next to the Variant A column and their name is shown as Difference (A vs B). |
Experiment metrics
The evaluation table lists the following metrics for each variant:
| Metric | Description |
|---|---|
| Gross average order value (AOV) | The average gross revenue from orders with at least one effect applied. |
| Net average order value (AOV) | The average net revenue from orders with at least one effect applied. |
| Average units per order (UPO) | The average number of items purchased per order with at least one effect applied. |
| Orders | The total number of closed sessions with at least one effect applied. |
| Gross revenue | The total pre-discount value of orders with at least one effect applied. |
| Discounts | The total value of applied discounts from all orders. |
| Coupons | The total number of redeemed coupons from all orders. |
Evaluate confidence scores
The Confidence chart shows how the confidence score values for your experiment have changed over its duration. Use it to check whether your results have stabilized before you decide to stop the experiment.
The Confidence chart appears only after the experiment has collected enough data.
The chart displays a trend line for each of the three metrics that express averages.
The confidence score, shown as a percentage, is tracked over the duration of the experiment starting on the start date, with a dashed line marking the 90% threshold that Talon.One recommends reaching for reliable results. For a running experiment, the confidence score values ends on the current date. For a disabled experiment, it ends on the date the experiment was disabled.
To see the confidence scores for a specific date, hover over the chart.
Use the shape of the trend lines to decide whether to stop the experiment:
- If the confidence scores have leveled off above the 90% threshold, your results are stable and you can stop the experiment.
- If the confidence scores are still climbing, let the experiment continue until they stabilize.
- If the confidence scores fluctuate, your results are not yet reliable. Let the experiment run for longer to gather more data.
Evaluate segment insights
The Segment insights table finds customer segments where one variant outperforms the other for a specific metric, with high confidence. This can reveal groups of customers that behave in a particular way, even when the overall experiment result is inconclusive.
To learn more about the available segments and how to read the table, see Segment insights.