Four Years of Predictive Modeling and Lessons Learned

I recently got the chance to talk with Dr. Michael Johnson, Director of Institutional Research at Dickinson College, about his experiences with predictive modeling over the past four years. Dr. Johnson will be presenting a free webinar, "Four Years of Predictive Modeling and Lessons Learned", on Thursday, January 31st at 2pm EST in which he'll provide a more in-depth look at his experiences. 

Can you give us an example of a lesson you’ve learned through your experiences with predictive modeling?

I’ve learned that predictive modeling is good but predictive modeling in real time is just more extremely beneficial. When we picked up Rapid Insight, we moved a five day turnaround time to an eight minute turnaround. That’s one of the biggest changes we’ve made, and the effects have been very apparent.

If there was another thing I’ve learned, it is to automate absolutely every process possible to remove the opportunity for human error.

What types of predictive models will you be discussing during your webinar?

The enrollment management model is our primary model but a close cousin to that is the one we’ve been using for retention. The dataset is basically the same only slightly enhanced. It’s good to use essentially the same dataset to solve two different problems.

How has predictive modeling changed the way you operate?

It is the primary tool that we use to make decisions on the incoming class. This last week has been an incredible example of that. We’re taking a look at our early action pool and asking questions: What does it look like? How does it compare with previous years? What if we swap out some people; how does that change our incoming class?

What do you hope attendees will take away from your webinar?

There’s really no need to reinvent the wheel, so I’ll share some ideas that I’ve picked up. I hope that others come on board and share their successes as well. We all have the same problem set, so it will be nice for others to take away a few things that I’ve seen that have given me success. It would be great if they had ideas that they wanted to share with others as well. 

To register for Dr. Johnson's free webinar, "Four Years of Predictive Modeling and Lessons Learned", or for more information, please click here. 

To read a case study about how Dickinson College uses predictive modeling for strategic enrollment management, please click here. 

How to Interpret a Decile Analysis

After building a predictive model, there are several ways to determine how well the model is describing your data. One visual way to get an idea of how well a model is fitting your data is by taking a look at the decile analysis. Here we’ll take a look at what the decile analysis represents, how it’s created, and how to spot a good model.

What a Decile Analysis Represents

After building a statistical model, a decile analysis is created to test the model’s ability to predict the intended outcome. Each column in the decile analysis chart represents a collection of records that have been scored using the model. The height of each column represents the average of those records’ actual behavior.

How the Decile Analysis is Calculated

1. The hold-out or validation sample is scored according to the model being tested.

2. The records are sorted by their predicted scores in descending order and divided into ten equal-sized bins or deciles. The top decile contains the 10% of the population most likely to respond and the bottom decile contains the 10% of the population least likely to respond, based on the model scores.

3. The deciles and their actual response rates are graphed on the x and y axes, respectively. 

After the decile analysis is built, you’ll want to take a look at the height of the bars in relation to one another. Deciding whether a model is worth moving forward with depends on the pattern you see when viewing the decile analysis. 

Ideal Situation: The Staircase Effect

When you’re looking at a decile analysis, you want to see a staircase effect; that is, you’ll want the bars to descend in order from left to right, as shown below. 

This is telling you that the model is “binning” your constituents correctly from most likely to respond to least likely to respond. A model exhibiting a good staircase decile analysis is one you can consider moving forward with.

Not-So-Ideal Situations

In contrast, if the bars seem to be out of order (as shown below), the decile analysis is telling you that the model is not doing a very good job of predicting actual responses.

 If the bars seem to be the same height, or the decile analysis looks “flat”, the decile analysis is telling you that the model isn’t performing any better than randomly binning people into deciles would. In both cases, your model should be improved before moving forward with it.  

-Caitlin Garrett, Statistical Analyst at Rapid Insight

Guide to Rapid Insight Resources

Networking:
We have created several opportunities for networking with other Rapid Insight users, including a Rapid Insight LinkedIn customers-only group, and several more subject-specific subgroups.

Rapid Insight LinkedIn Page

• Strategic Enrollment Management Network
• Fundraising and Advancement Network
• Institutional Research Network

Webinars:
Check out our list of upcoming webinars here. These are a few you might want to check out:

Predictive Modeling (PM) for Higher Ed ** PM for Fundraising

Dashboards and Reporting for Higher Ed ** PM for Healthcare 

Training Resources:

• To set up a training session with one of our analysts, please email support@rapidinsightinc.com
• Check out our training videos
• Browse the Veera and Analytics user manual downloads
• Got questions? Post or browse for answers in our forum
• For more resources, see the Training page of our website

Blog:
Here are links to a couple of recently featured series:

• Customer Tips: tips from customers on ways to make your life easier.
• Creating Variables: on how any why to augment your dataset by creating additional variables using Veera.
• The Forgotten Tabs: on the benefits of utilizing some of Analytics’ lesser talked about tabs.

Conference:
Rapid Insight is proud to host an annual User Conference each summer. Information about the conference will be available on our User Conference page as the conference draws near. 

If you have any additional questions about Rapid Insight resources or products, please feel free to contact me directly at caitlin.garrett@rapidinsightinc.com. 

-Caitlin Garrett, Statistical Analyst

On Automated Mining

One of the things I love the most about using statistical modeling software (especially Analytics) is that so much of the process is automated. Although automation has made the lives of statisticians much easier (calculating individual standard errors by hand would take hours for each variable), it is still important to be familiar with the methods and thinking that go into the variable selection process. One tab that does a lot of statistical heavy lifting for us is the Automated Mining tab, and I thought it would be good to explore some of the tests that are being used in that tab.

The function of the Automated Mining tab is to determine, variable by variable, which variables are statistically related to the selected y-variable, and which are not. The statistical test will vary from pair to pair depending on the types of variables being compared. One thing that is important to note is that we’re not doing any modeling or looking at the relationships between x-variables yet. The Automated Mining tab and its tests are only deciding which variables have the possibility of being in the predictive model, not which ones will be.

Depending on the types of x- and y-variables involved, one of three tests will be used to decide how related each variable pair is. These possibilities include a Chi-Square test, a Z-test, or an F-test. 

	Variable Under Evaluation
Y-Variable	Binary	Continuous	Categorical
Binary	Z-Test	Decile-Chi-Square	Z-Test
Continuous	Z-Test	Decile-F-Test/ANOVA	Z-Test
Categorical	n/a	n/a	n/a

Chi-Square Test

A chi-square test is performed for any continuous x-variables used to predict a binary y-variable. In our Automated Mining tab, this test is performed on each of 10 deciles to determine whether or not the ‘ones’ are randomly distributed across the deciles. This test is more robust than using a linear correlation, as it captures non-linear relationships as well as relationships that are not well fit by a curve or line.

Z-Test

A Z-test is used for any binary or categorical predictors, regardless of the type of y-variable they’re trying to predict. It tests whether any category is significantly different in terms of the Y (relative to all other categories).

F-Test

An F-test is used whenever you have a continuous x-variable trying to predict for a continuous y-variable. In our Automated Mining tab, the data is sorted into deciles and an ANOVA test is run on these deciles to determine if the means are statistically different. This is more robust than a linear correlation, as it captures non-linear relationships and those that do not fit a standard curve.

Once each of these tests is performed at the specified level of significance, we have a narrowed-down dataset of only variables that are statistically related to our y-variable, which brings us one step closer to figuring out which variables historically have the most influence on our y-variable and will end up in our final predictive model. 

-Caitlin Garrett, Statistical Analyst at Rapid Insight 

How to Score a Dataset Using Analytics Only

Since we’ve already covered how to score a dataset using Veera, it’s only fair that we show you how to score using the Analytics Scoring program. We’ll start at the point where you save your scoring model within Analytics. After memorizing your model in the Model tab, you’ll want to move down to the Compare Models tab. This tab allows you to compare any two models side-by-side. Once you’ve decided which model you like better, you’re ready to save it by selecting the model and clicking the “Save Scoring Model” as button, as shown below.

Analytics will prompt you to navigate to where you’d like the file to be saved, and will save it with a .rism (Rapid Insight Scoring Model) extension. After saving the .rism file, you’ll want to open the Analytics Scoring Module by going to your Start Menu and navigating to Rapid Insight Inc. -> Analytics -> Scoring, as shown below. 

Once inside the scoring module, you’ll need to click the “Select Dataset” button and navigate to where the dataset you’d like to score is located on your machine. After loading in your dataset, you’ll see all of the variables within it populate the ‘Dataset Variables’ window. Next, you’ll need to click the “Select Scoring Model” button and navigate to where the scoring model (.rism) file you’d like to use is located. Once you find the model, its equation will show up in the corresponding window.

Before you start the scoring process, you have a couple of options detailing how you’d like the model to be scored. The first option, shown above in the green box, allows you to validate the model by looking at the decile analysis resulting from the scoring process. The second option, shown in the blue box, allows you to output the scores as well as the corresponding deciles or percentiles. After you’ve selected the appropriate options, click on the “Start Scoring” button, decide where you’d like your scores to output, and Analytics will score your dataset in the way that you request. 

-Caitlin Garrett, Statistical Analyst at Rapid Insight

Predictive Modeling Mantras

Whether you're new to predictive modeling, or you dream in decile analyses, here are some things to keep in mind as you're embarking on your next modeling project: 

Data preparation makes ALL the difference.
Simply put, if you use junk data to create a model, chances are that your model’s output will be junk too. Thus, it’s very important to clean up your data before building your predictive model. During the data clean-up process, you’ll want to think about things like how to handle any missing values, possibilities for new variables to be added to your dataset, how to handle outliers, and make sure that your data is as error-free as possible.

A complex model isn’t the same as a good model.
More often than not, the best model is a simple one. Although you can almost always find a new variable to add, or new way to slice your data, you want to avoid the trap of overfitting. You want your model to be specific, but not so specific that you sacrifice reliability when scoring a new dataset.

A good model validates what you know while revealing what you don’t .

Don’t be surprised if some of your “common sense” variables outperform the more exotic ones. Although it’s always nice to pick up on some new variables and insights, building a predictive model can also boost your confidence in the rest of your data.  

If a model looks perfect, it’s lying.

As exciting as getting a great model fit statistic can be, there is the possibility of too good to be true when it comes to model building. If you build a particularly great model, you’ll want to double and triple check each of the variables in the model to be sure they make sense. One of the most common reasons for a great model is an anachronistic variable – a variable you would have available only after your y-outcome was decided.

Persistence is a virtue (because building models is an iterative process).

After you’ve taken a first pass at a model, maybe you’ll think of a related variable that would be predictive. Maybe you take a second look at some of the relationships between variables and decide to bin or re-map some of your continuous or categorical variables. Maybe the outputted variables are the opposite of what you expected, so you decide to tweak the way your dataset is set up. The point here is that your first model will likely not be your final model. Be ready.

Trust and verify.
The modeling process doesn’t end after you finish building your model. After implementing your predictive model, you want to be sure that it’s correctly predicting your y-variable over time. To do this, you’ll need to compare your model scores with actual results once they are available. If your model is correctly predicting the desired outcome, you can continue to use it (but still must validate as time goes by); otherwise, you’ll need to take a few steps back to see where you can make improvements.

-Caitlin Garrett, Statistical Analyst at Rapid Insight
[photo credit]

How to Score a Dataset Using Veera

After you’ve ‘memorized’ the predictive model you’d like to use, you’re ready to start the scoring process. There are actually two ways to score a dataset using the Rapid Insight software suite. In this post, we’ll talk about how to import your scoring model into Veera and quickly score your dataset.

We’ll start at the point where you save your scoring model within Analytics. After memorizing your model in the Model tab, you’ll want to move down to the Compare Models tab. This tab allows you to compare any two models side-by-side. Once you’ve decided which model you like better, you’re ready to save it by selecting the model and clicking the “Save Scoring Model” as button, as shown below.

Analytics will prompt you to navigate to where you’d like the file to be saved, and will save it with a .rism (Rapid Insight Scoring Model) extension. Once you’ve saved the file, you’re ready to move into Veera to score your dataset.

In Veera, you’ll want to create a new job for scoring. In that job, bring in your input file (the file you’d like to score), and connect it to an output files. When configuring the output file, you can choose to write your scores to a file, spreadsheet, or back to a database table. Once the input and output files are connected, you’ll be importing the scoring model between them. To do so, right-click on the line connecting the two files and select Wizard -> Import Scoring Model, as shown below:

You will need to navigate to where you saved your scoring (.rism) file and select it to finish the import. Once you’ve done so, you’ll see four or five new nodes populate on the line between your input and output file. These nodes, shown below, are Analytics’ way of communicating the scoring process to Veera.

One very important thing about this scoring process is that your model is not a black-box model – you can explore each step to see how your data is scored. Feel free to open each of the nodes and see what they are accomplishing. If you open the “Create New Variables” node, you’ll be able to see any of the transformations used in your predictive model; you can also access the model formula itself by opening the “Calculate Probability” node. To get the probability scores, go ahead and run your job. The scores will be outputted as a new column called “Probability” in your output file or database. 

-Caitlin Garrett, Statistical Analyst at Rapid Insight

Fall Pricing Promotion

For both new and existing customers - now until November 16th!

Not a customer yet? Rapid Insight is offering 10% off your software purchase through November 16th, 2012. 

Already feeling the love? We're offering existing customers the "add a license" promotion: add a license of either Rapid Insight Analytics or Veera for only $3,000 each. 

We hope this is what you've been waiting for!



For more information, please contact Sheryl Kovalik at 
sheryl.kovalik@rapidinsightinc.com or (603) 447-0240 ext. 7568

The Forgotten Tabs: Profiling Analysis

The final installment of the Forgotten Tabs Series is focused on the Profiling Analysis tab. The Profiling Analysis tab allows us to compare the two groups of a binary variable by generating an output of all of the variables in a dataset for which those two groups are significantly statistically different. Once in the tab, simply select the binary variable you’d like to profile, and you’ll get an output like this:

Here you can see that we are comparing students who enrolled at our institution to students who did not enroll to see what the major differences are between the two populations.

If your y-variable is binary, this tab provides great insight into how the populations that fall into the two possible categories of your y-variable differ. This also provides another way to look at your data. One common question I am asked is something along the lines of “Without knowing the scores, how do I know which variables I should be looking at?”, meaning that though the scores are helpful in decision making, sometimes knowing the differences between the variables that make up those scores can be just as helpful. The Profiling Analysis tab directs you to only those variables for which the two populations differ significantly.

One great use of the profiling tab in higher education is to compare the differences between graduates and non-graduates. The following illustrates what this analysis might look like: 

Using an output like this, we can see that students who graduated generally lived closer to campus, had a higher HS GPA, applied earlier, and had higher SAT Math scores than non-graduates. Highlighting these differences and placing an average value on each variable for both graduates and non-graduates allows us greater insights into the differences between these two populations. The Profiling Analysis tab is a great resource whenever you want to compare two populations to see how and where they differ statistically. 

-Caitlin Garrett, Statistical Analyst at Rapid Insight

The Forgotten Tabs: Frequency Analysis

During this year’s User Conference, I gave a presentation called “Analytics: The Forgotten Tabs”, which I’ve decided to expand into a blog series. The purpose of this series will be to explain how and why to use four of the lesser-known tabs in Analytics – Frequency Analysis, Means Analysis, Correlation Analysis, and Profiling Analysis. Each entry will focus on one of these tabs and we’ll start with the Frequency Analysis tab.

The Frequency Analysis tab’s output is actually fairly simple; it gives you the frequency of occurrence for any binary or categorical variable. For a single variable, it will output counts and percentages for each value of that variable. It is also capable of creating two-way cross frequencies, which output raw numbers, as well as row, column, and total percentages. 

While Frequency Analysis isn’t actually performing any statistical test – its functions are simple summing and percentage operations – it is providing valuable information about the number and percentage of observations that fall into each sub-category of a binary or categorical variable. Using this tab gives you a quick by-the-numbers glance at variables like “Ethnicity” or “Department”, which allows you to instantaneously compare subcategories without doing any manual addition or division. This is particularly useful when you’re working with a variable such as “Department” that may have a lot of sub-categories.

One other little-known fact about the output from Frequency Analysis (and other tabs) is that you can save it to the Report Bar the same way you would a graph or chart. To do so, click on the ‘Reports’ section of the taskbar and select ‘Launch Report Bar’. 

The report bar will float over your analysis; you can save things to it by dragging the outputs you wish to save into the bar itself. Saving things to the report bar allows you to export them from Analytics in a few different ways. If you select the ‘PPoint’ option before clicking ‘Export’, Analytics will create a PowerPoint such that each of the graphs our outputs you saved will become their own slide in the presentation. The other option you have is to save the information you’re interested in to the Reports tab in Analytics (by selecting the ‘Report’ option on the Report Bar), which allows you to create custom reports within the program and export these reports as Word Documents to be used later on. In any case, there are a number of ways to take the information that you’re getting from Analytics and use it in a presentation or report down the line.  

-Caitlin Garrett, Statistical Analyst at Rapid Insight