WhatCounts uses Revenue Lift as the primary metric to determine the effectiveness of your Campaign efforts. More specifically, Windsor Circle determines that a campaign is successful only if generates more revenue than would be observed in its absence. This is discovered through control group testing, which observes and measures the behavior of treated versus controlled customers. For a more detailed explanation, please read Lift Estimation Methodology help article.
Why Control Group Testing?
How Does Control Group Testing Work?
It is worth noting that customers may be eligible for multiple automators at the same time. For example, a customer may be eligible for treatment from a Best Customer automator and a Cart Recovery automator concurrently. Because customers are assigned to treatment or control groups at the automator level and as they become eligible, a customer may be a member of the control group for one automator and the treatment group for another.
Please remember: Not everybody intended to be treated is actually treated because some emails don’t reach their recipient. For this reason, the effective split between the treated and the controls is not always 90/10, but more often something less than 90 and something more than 10. We consider only those recipients whose emails show up as sent successfully as treated.
When Do You Know It's Working?
From there, as your orders in both groups build up, we measure for statistical significance, which tells us whether we have enough data to conclusively — with 95% confidence — say with your lift is positive or negative. Until we reach that point, we will say that your campaigns (or overall Windsor Circle campaign efforts) are building toward statistical significance.
Understanding Your Lift Report
- Overall Lift Summary: Indicates whether your overall lift is positive, negative, or still building toward statistical significance
- A gray bell curve indicates your overall lift is still building toward statistical significance
- A green bell curve indicates your overall lift is statistically significant and positive
- A red bell curve indicates your overall lift is statistically significant and negative
- Campaign Lift Summary: For each Windsor Circle campaign you are running, we will determine whether it's positive or negative with statistical significance
- A green bell curve indicates your campaign's lift is statistically significant and positive
- A red bell curve indicates your campaign's lift is statistically significant and negative
- Campaigns Building Toward Statistical Significance: Campaigns that have not yet achieved statistical significance can be found in the bottom section of your Lift Report.
We express Lift as a rounded percentage for a quick and ready idea of the relative difference that an automator makes, but the Revenue Lift is often the more important figure because an automator with a small lift that affects a large revenue base is more interesting — and meaningful to the marketer — than one that shows a large percent lift but moves few actual dollars.
Requirements to Seeing Revenue Lift
- You will only ever see Revenue Lift for Windsor Circle Campaigns. If you are syncing segments to your ESP and launching campaigns from there, we are unable to calculate revenue lift for these campaigns.
- Revenue Lift does not happen overnight. We need to observe a statistically significant amount of orders in both your treatment and control groups in order to determine whether Revenue Lift is occurring. Depending on order volume, this can take a few months from when a campaign is first launched.