Summary

Design, launch, and statistical analysis of a multifactorial advertising experiment to evaluate the impact of foreign language marketing on perceived value of wine. Analysis used experimental techniques and multiple types of linear regression models.

Design

This experiment was used to analyze multiple effects of marketing environment on the perception of product value. The multifactorial design includes three treatment variables: page language, flavor description length, and country of origin. While a product’s country of origin is non-mutable for marketing purposes, this was an important feature to analyze any interaction with the page format. In addition to French and English, German was added as a language format to observe the effects of a foreign language that’s not usually associated with the product. The description length was be used both as a feature and a metric of dosage for the language format.

drawing The experiment was conducted as a survey. Participants first answer demographic questions, then were prompted with the treatment. The multiple treatments were administered simultaneously in the form of an image of an online marketing page. Each variable appeared on the page

Wine_sample

The participants were then asked to guess the value of the wine advertised to them; this provided a continuous outcome variable. They were then asked if they would purchase the wine at three descending price points ($50, $35, $20), which would be used as binary variables to calculate purchase likelihood.

drawing

The initial launch (after a pilot) showed a need for currency familiarity and/or engagement management. To achieve this while maintaining comparability, a baseline question was asked after the treatment and outcome steps- guess the cost of a gallon of milk. For the test group, responses that priced milk above $10 were filtered out. Filtering for duplicate IP addresses and geographical location were conducted to avoid inclusion of bots, clickfarms, and the risk of treatment spillover. This resulted in only 5% total attrition.

Analysis Methods

Initial analysis was conducted on the training set of data in order to establish hypotheses before launching the test set. While p-value penalties could have been used with a single launch, the baseline question for re-launch was critical, so a training/test set approach was used to prevent p-hacking. This allowed the use of multiple statistical methods across many hypotheses and interaction hypotheses to be tested. In addition to hypothesis testing, statistical methods were also used to examine heterogeneous treatment effects for specific demographic categories.

The following statistical methods were tested with and without demographic covariates. The price guesses were processed as a continuous outcome variable, while the willingness to purchase responses were processed as binary variables, using logistic regression to evaluate probability of purchase as a percentage.

Random inference with Sharp Null hypothesis (on bottom 95% of guesses)
Linear regression
Robust linear regression (Huber method to control for outlier influence)

Simple treatment
Paired treatment (2 variables)
Treatment track (3 variables)
Interaction models

Same origin and page format
Two-pronged interactions
Three-pronged interactions
Fully saturated model

Binary logistic regression (including above models and interactions)

Findings

Treatment signals were undetectable using random inference (on the bottom 95% of price guesses) and the sharp-null and simple linear regression. Statistical significance was observed in binary logistic regressions with french layout * french origin at 5% statistical significance in the training set and 10% in the test set. It should be noted that further data acquisition could increase the intensity of this signal.

Sample Images from Report

Winecosts
WineOLS

Project Git