Test setting: email to retailer mailing list

Unit: email address

Treatments: email version A, email version B, holdout

Reponse: open, click and 1-month purchase ($)

Selection: all active customers

Assignment: randomly assigned (1/3 each)

d %>% group_by((days_since < 60), group) %>% summarize(mean(open), mean(click), mean(purch))

## # A tibble: 6 x 5
## # Groups:   (days_since < 60) [2]
##   `(days_since < 60)` group   `mean(open)` `mean(click)` `mean(purch)`
##   <lgl>               <fct>          <dbl>         <dbl>         <dbl>
## 1 FALSE               ctrl           0            0               6.80
## 2 FALSE               email_A        0.582        0.106          17.1 
## 3 FALSE               email_B        0.503        0.0715         17.0 
## 4 TRUE                ctrl           0            0              18.5 
## 5 TRUE                email_A        0.865        0.160          34.8 
## 6 TRUE                email_B        0.812        0.117          35.4

• The email seems to produce a stronger effect on purchases for recently active customers.

d %>% filter(email==TRUE) %>% ggplot(aes(y=purch, x=group)) + 
  geom_dotplot(binaxis='y', stackdir='center', stackratio=0.1, dotsize=0.1, binwidth=0.1) +
  ylab("30-Day Purchases ($)") + xlab("") + scale_y_log10()

t.test(purch ~ email, data=d[d$days_since < 60,])

## 
##  Welch Two Sample t-test
## 
## data:  purch by email
## t = -33.51, df = 50513, p-value < 2.2e-16
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -17.5776 -15.6350
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            18.48809            35.09439

t.test(purch ~ email, data=d[d$days_since > 60,])

## 
##  Welch Two Sample t-test
## 
## data:  purch by email
## t = -30.257, df = 56220, p-value < 2.2e-16
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -10.752048  -9.443798
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            6.792411           16.890335

Anyone who keeps historic data on customers or visitors has lots of baseline variables available for slicing and dicing:

• data on previous website visits
• sign-ups
• geographic location
• source
• past purchase (by category)
• recency
• frequency

Re-analyze the opens, clicks and purchases for people who have bought syrah in the past.

summary(d$syrah > 0)

##    Mode   FALSE    TRUE 
## logical   88359   35629

mean(d$syrah > 0)

## [1] 0.2873585

Slicing and dicing will reveal two things about subgroups of customers.

1. Subgroups will vary in how much they engage in behaviors
   • Recently active tend to have higher average purchases after the email
2. Subgroups vary in how they respond to treatments
   • Recently active are more affected by the email

“Experiments are used because they provide credible estimates of the effect of an intervention for a sample population. But underlying this average effect for a sample may be substantial variation in how particular respondents respond to treatments: there may be heterogeneous treatment effects.”

– Athey and Imbens, 2015

Marketers should be interested in heterogeneous treatment effects when there is opportunity to apply different treatments to each subgroup (ie targeting).

email \(\rightarrow\) high potential for targeting

website \(\rightarrow\) less potential for targeting

We use a regression model to define a relationship between the response (\(y\)) and the treatment (\(x\)).

\(y = a + b \times x + \varepsilon\)

The model literally says that we get the average response by multiplying the treatment indicator \(x\) by \(b\) and adding that to \(a\). When we fit a model, we use data to estimate \(a\) and \(b\).

In R, we shorthand the model equation with an R formula:

purch ~ email

This means exactly the same thing as:

purch \(= a + b \times\) email \(+ \varepsilon\)

where we estimate \(a\) and \(b\) from data.

To incorporate heterogeneous treatment effects, we need an interaction between the treatment effect (\(x\)) and a baseline variable (\(z\)).

When we interact to terms, we are defining a model that multiplies the two terms:

\(y = a + b x + c z + d (x z) + \varepsilon\)

The R formula for this model is:

purch ~ email + (days_since < 60) + email:(days_since < 60)

or equivalently

purch ~ email*(days_since < 60)

If you have someone who wasn’t in the test, but you know their baseline variables, you can use an uplift model to predict likely treatment effect.

new_cust <- data.frame(past_purch=rep(38.12,2), days_since=rep(19,2), visits=rep(3,2))
(pred <- predict(m4, cbind(email=c(TRUE, FALSE), new_cust)))

##        1        2 
## 15.40060 11.05818

(lift <- pred[1] - pred[2])

##        1 
## 4.342422

This new customer is predicted to buy $13.03 if they get an email or $12.40 without, for a uplift of $0.63.

new_cust <- data.frame(past_purch=rep(127.88,2), days_since=rep(19,2), visits=rep(40,2))
(pred <- predict(m4, cbind(email=c(TRUE, FALSE), new_cust)))

##        1        2 
## 43.86908 22.47098

(lift <- pred[1] - pred[2])

##        1 
## 21.39809

This is a better target with an uplift of 11.61.

For costly treatments (eg catalogs, discounts) we should target customers customers that we predict will have a positive effect that exceeds costs.

Source: Predictive Analytics Times

We can also build an uplift model for click probability, but we should use a logistic regression for binary outcomes.

m5 <- glm(click ~ group*(days_since < 60) + group*(past_purch > 50) + group*(visits > 3) +
                  group*(syrah > 0) + group*(cab > 0) + 
                  group*(sav_blanc > 0) + group*(chard > 0),
          family = binomial,
         data=d[d$group != "ctrl",])

Uplift models can include many, many baseline variables. Creating these variables from source data (CRM, web analytics data, etc) is called feature engineering.

Causal forests are an alternative to regression for identifying heterogeneous treatment effects and scoring customers based on predicted treatment effect uplift.

Where regression models predict customer outcomes with a linear equation, cart trees predict customer outcomes using a tree structure. CARTs are estimated by finding the tree structure that seems to classify people correctly most of the time.

Random forests are collections of different CARTs each fit to a subset of the data. Each tree in the forest classifies customers slightly differently. Unlike a regression, a random forest can pick up non-linear relationships.

Causal forests are random forests designed to categorize customers according to their treatment effect in an experiment. The customers in each leaf are assumed to have homogeneous treatment effects, with heterogeneous treatment effects between leaves.

Advantages
- Works well with a large number of baseline variables
- Doesn’t require the analyst to define cut-offs for continuous baseline variables
- Will fit non-linear relationships between baseline variables and uplift

treat <- d$email
response <- d$purch
baseline <- d[, c("days_since", "past_purch", "visits", "chard", "sav_blanc", "syrah", "cab")]
cf <- causal_forest(baseline, response, treat)
print(cf)

## GRF forest object of type causal_forest 
## Number of trees: 2000 
## Number of training samples: 123988 
## Variable importance: 
##     1     2     3     4     5     6     7 
## 0.165 0.667 0.093 0.023 0.031 0.012 0.008

average_treatment_effect(cf, method="AIPW")

##   estimate    std.err 
## 13.2946030  0.2876234

This is similar to the estimate from our simple regression which was 6.42 (0.30).

Just like any uplift model, we can use the model to predict the email effect for new customers.

new_cust <- data.frame(chard=38.12, sav_blanc=0, syrah=0, cab=0,  
                       past_purch=38.12, days_since=19, visits=3)
predict(cf, new_cust, estimate.variance = TRUE)

##   predictions variance.estimates
## 1  -0.3514193           10.04156

hist(predict(cf)$predictions, 
     main="Histogram of Purchase Lift", 
     xlab="Purchase Lift for Email", ylab="Customers")

trans_gray <- rgb(0.1, 0.1, 0.1, alpha=0.1)
plot(d$past_purch, predict(cf)$predictions, cex=0.5, col=trans_gray,
     xlab="Past Purchase Amount ($)", ylab="Predicted Treatment Effect ($)")

trans_gray <- rgb(0.1, 0.1, 0.1, alpha=0.1)
plot(d$days_since, predict(cf)$predictions, cex=0.5, col=trans_gray,
     xlab="Days Since Last Active", ylab="Predicted Treatment Effect ($)")

• Large sample \(\rightarrow\) look for heterogeneous treatment effects using baseline variables
• Three ways to find heterogeneous treatment effects
  • Slicing and dicing: filter down then analyze sub-test
  • Uplift modeling
    • Build a regression with interactions between x and z’s
    • Use logistic regression for binary response
  • Causal forests