Follow

Lift

WhatCounts uses Revenue Lift as the primary metric to determine the effectiveness of your Campaign efforts. More specifically, Windsor Circle determines that a campaign is successful only if generates more revenue than would be observed in its absence. This is discovered through control group testing, which observes and measures the behavior of treated versus controlled customers. For a more detailed explanation, please read Lift Estimation Methodology help article.

Why Control Group Testing? 

We believe control group testing is the only definitive way to measure the success of a campaign. Generally, marketers will look at two common methods for attribution and ROI modeling: Last-touch and multi-touch attribution. These can be imperfect methods for really getting to the meat of what a specific marketing effort or tool is doing for your business because they tend not to tell the whole story and force marketers to draw conclusions from interactions that may or may not have had a tangible effect on a purchase. Moreover, drawing conclusions from those interactions can be risky.
 

How Does Control Group Testing Work? 

As your customers become eligible to receive campaign emails, Windsor Circle randomly assigns them to one of two groups: a treatment group or a control group. Their assignment determines whether they will receive the Windsor Circle campaign emails they are eligible for or whether they will never receive a Windsor Circle treatment for that campaign. Either way, their behavior is observed. By default, Windsor Circle will assign 90% of all eligible customers to a treatment group and the remaining 10% to a control group, but this breakdown can be reconfigured upon request.

It is worth noting that customers may be eligible for multiple automators at the same time. For example, a customer may be eligible for treatment from a Best Customer automator and a Cart Recovery automator concurrently. Because customers are assigned to treatment or control groups at the automator level and as they become eligible, a customer may be a member of the control group for one automator and the treatment group for another.

Please remember: Not everybody intended to be treated is actually treated because some emails don’t reach their recipient. For this reason, the effective split between the treated and the controls is not always 90/10, but more often something less than 90 and something more than 10. We consider only those recipients whose emails show up as sent successfully as treated.

When Do You Know It's Working? 

The Windsor Circle platform measures transactional purchase data. We observe the aforementioned treatment and control groups and assess whether there is a difference in spending patterns. We also handle outliers through a process called winsorization, which essentially caps outliers above the 99th percentile at exactly the 99th percentile. This represents our systemized way of trimming the excessively large values in the data set so they don't skew things. For example, if the 99th percentile is $700, then all order totals in the input data that are higher than this are overwritten with $700. Extremes at the low end are left unchanged.

From there, as your orders in both groups build up, we measure for statistical significance, which tells us whether we have enough data to conclusively — with 95% confidence — say with your lift is positive or negative. Until we reach that point, we will say that your campaigns (or overall Windsor Circle campaign efforts) are building toward statistical significance.

Understanding Your Lift Report 

Now that you understand how we're calculating Revenue Lift, let's break down your Lift Report, which is broken down into as many as three sections:
  • Overall Lift Summary: Indicates whether your overall lift is positive, negative, or still building toward statistical significance
    • A gray bell curve indicates your overall lift is still building toward statistical significance
    • A green bell curve indicates your overall lift is statistically significant and positive
    • A red bell curve indicates your overall lift is statistically significant and negative
  • Campaign Lift Summary: For each Windsor Circle campaign you are running, we will determine whether it's positive or negative with statistical significance
    • A green bell curve indicates your campaign's lift is statistically significant and positive
    • A red bell curve indicates your campaign's lift is statistically significant and negative
  • Campaigns Building Toward Statistical Significance: Campaigns that have not yet achieved statistical significance can be found in the bottom section of your Lift Report.

We express Lift as a rounded percentage for a quick and ready idea of the relative difference that an automator makes, but the Revenue Lift is often the more important figure because an automator with a small lift that affects a large revenue base is more interesting — and meaningful to the marketer — than one that shows a large percent lift but moves few actual dollars.

Requirements to Seeing Revenue Lift 

It is important to remember a two important things about Revenue Lift:
  1. You will only ever see Revenue Lift for Windsor Circle Campaigns. If you are syncing segments to your ESP and launching campaigns from there, we are unable to calculate revenue lift for these campaigns.
  2. Revenue Lift does not happen overnight. We need to observe a statistically significant amount of orders in both your treatment and control groups in order to determine whether Revenue Lift is occurring. Depending on order volume, this can take a few months from when a campaign is first launched.
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk