How to Interpret a Decile Analysis

After building a predictive model, there are several ways to determine how well the model is describing your data. One visual way to get an idea of how well a model is fitting your data is by taking a look at the decile analysis. Here we’ll take a look at what the decile analysis represents, how it’s created, and how to spot a good model.

What a Decile Analysis Represents

After building a statistical model, a decile analysis is created to test the model’s ability to predict the intended outcome. Each column in the decile analysis chart represents a collection of records that have been scored using the model. The height of each column represents the average of those records’ actual behavior.

How the Decile Analysis is Calculated

1. The hold-out or validation sample is scored according to the model being tested.

2. The records are sorted by their predicted scores in descending order and divided into ten equal-sized bins or deciles. The top decile contains the 10% of the population most likely to respond and the bottom decile contains the 10% of the population least likely to respond, based on the model scores.

3. The deciles and their actual response rates are graphed on the x and y axes, respectively. 

After the decile analysis is built, you’ll want to take a look at the height of the bars in relation to one another. Deciding whether a model is worth moving forward with depends on the pattern you see when viewing the decile analysis. 

Ideal Situation: The Staircase Effect

When you’re looking at a decile analysis, you want to see a staircase effect; that is, you’ll want the bars to descend in order from left to right, as shown below. 

This is telling you that the model is “binning” your constituents correctly from most likely to respond to least likely to respond. A model exhibiting a good staircase decile analysis is one you can consider moving forward with.

Not-So-Ideal Situations

In contrast, if the bars seem to be out of order (as shown below), the decile analysis is telling you that the model is not doing a very good job of predicting actual responses.

 If the bars seem to be the same height, or the decile analysis looks “flat”, the decile analysis is telling you that the model isn’t performing any better than randomly binning people into deciles would. In both cases, your model should be improved before moving forward with it.  

-Caitlin Garrett, Statistical Analyst at Rapid Insight

On Target: Predicting Pregnancy

Call me biased, but I think creative uses for predictive analytics are pretty cool.  Target’s “pregnancy-prediction model”, explained Thursday in a The New York Times Magazine article, is a great example.  It should inspire all of us to take a fresh look at our data and consider what more we can accomplish with a powerful predictive analysis tool (like RI Analytics) and a little bit of creative thinking.

Target’s journey to predicting pregnancy started with an idea conceived by its marketing department. The department had previously conducted surveys which indicated that once a consumer’s shopping habits are ingrained, it can be hard to change them – except during certain brief periods of a person’s life, like after a marriage or the birth of a child, where shopping patterns and brand loyalties often change.  The birth of a child represents a new grocery and household goods list for new parents, as well as the opportunity for Target to sell things like cribs, rugs, furniture, car seats, and other items that a person or couple would not usually buy. Because birth records are public information it was already common practice for companies to send promotional items to new parents; so, to stay one step ahead of competitors, marketers at Target wanted to see if there was a way to predict pregnancy during the second trimester.

Target reviewed the shopping habits of women who had a baby-shower registry as they approached their due dates. Eventually they were able to identify about 25 different products that were indicators of pregnancy, including items like unscented lotion, vitamin supplements, hand sanitizers and washcloths. By treating the purchase of each item as a variable, they were able to create a model that assigned each shopper a pregnancy prediction score based on their purchases. This score was then used to send out relevant coupons and advertisements tailored to each woman at a specific point in her pregnancy – before other retailers even knew she was pregnant. Needless to say, sales in Target’s Mom and Baby department skyrocketed.

This is one way that a creative use of data, combined with some predictive analytics, yields some pretty cool results. Target had the data they needed all along –they just needed the right person to ask the right question. 

-Caitlin Garrett, Statistical Analyst at Rapid Insight