Predicting Retention for Online Students: Where to Start

With the rise of enrollment in online programs and MOOCs, we’re seeing more and more students forego traditional classroom experiences in favor of more flexible online programs. With this shift comes a whole new set of guidelines for enrollment management, financial aid, and retention programs. Retention, in particular, has seen a significant downward trend as learning moves from in-person to online classrooms.

My interest lies in figuring out what variables might be worth including in an analysis attempting to predict online student retention. I did a bit of research and was hoping to find a list of variables online that had worked in the past but couldn’t find any comprehensive resource, so I’ve started to build my own. In the sections below, I’ve listed the type of information that I think would be worth analyzing broken out into four separate categories. Some of these are variables in and of themselves, and some can be broken down different ways; for example, “age” can be used by itself, but creating a “non-traditional age” flag is useful as well. Realistically, not all schools will have all of this information, so this list is meant to be a good starting point of what to shoot for when collecting data.

Also, if you have any variables to add (and I’m sure there are some I’ve missed), I’d love to hear about them in the comments. 

Student Demographic Information

• Socioeconomic status / financial aid information
• FAFSA info, Pell eligibility, any scholarship or award info
• Ethnicity
• Minority Status
• Gender
• Home state
• Distance from physical campus (if applicable)
• Age; traditional or non-traditional?
• Military background?
• Have children?
• Currently employed full-time?
• First generation college student?
• Legacy student? (Did a parent/grandparent/sibling attend?)

Student Online Learning History

• Registered for classes online or in person?
• How many days did they register before the start of the term?
• Ever attended a class on-campus?
• Do they plan to attend both online and on-campus classes?
• Did they attend any type of orientation?
• Number of previous online courses taken
• First-time online learner?

Student Academic History

• GPA
• SAT/ACT scores
• Degree hours completed
• Degree hours attempted
• Taking developmental courses?
• Transfer student?
• Degree program / major 
• Program level (Associate, Bachelors, Masters, etc.)
• Number of program or major changes (if applicable)
• Any previous degrees?

Course- and Program- Related

• Amount of text vs. interactive content 
• Lessons with immediate feedback?
• Any peer-to-peer forum for interaction?
• Lessons in real time or recorded?
• Amount of teacher interaction with students
• Chat, email exchange, turn-around time on assignments

Closing notes:

Getting course-related data might be difficult, but the variables I listed above are derived from studies about how to improve online courses as being areas to focus on; my thinking is that the more engaged a student is, both with peers and instructors, the better their chances of online success are. If you have the data available, it would be worth trying to incorporate it into your model dataset to see whether or not it is predictive.

Rather than using retention as a y-variable when building these models, we typically create an attrition variable (exactly the opposite of retention) and use that as our y instead. This way, we're getting more directly at the characteristics of a student who is likely to leave rather than stay.

Typically when building attrition models, I create separate models for freshmen and upperclassmen. I’d suggest doing that here as well, since previous online coursework will probably be a good indicator of future online coursework. In that case, you’d want to take out many of the variables listed above when modeling freshmen retention.

Finally, it’s important to keep in mind that student success has different meanings for different institutions. You could be basing success on # of credits completed, transitions from semester to semester, or a particular GPA cutoff, among other indicators. When building these different types of student success models, you will probably need to tailor some of these variables to fit the model you're building.

-Caitlin Garrett is a Statistical Analyst at Rapid Insight

The Forgotten Tabs: Means Analysis

Continuing with the Forgotten Tabs series, the next tab we’ll be focusing on is the Means Analysis tab. The Means Analysis tab provides the mean, number of observations, maximum values, and minimum values for any of the variables in your dataset. You also have the option to take “means by” and “subclass by” to view the means of multiple subcategories across variable combinations.

 In this case, we’re comparing the attrition rates of legacy and non-legacy students by whether or not they received financial aid. For each of these four possible categories, we are able to see the mean attrition rate, the number of observations, and the min and max values. Doing so allows us to see the differences in attrition rate over a couple of different characteristics.

The Means Analysis tab can be also useful in comparing data from different cohorts or years in order to spot trends. In the example below, we’re comparing attrition rates by year, which allows us to pick up on any trends or changes that are occurring from year to year. If for some reason we were noticing a year that had a much higher or lower attrition rate than the other years, this gives us the opportunity to pick up on that and investigate further as to why that might be. 

You might also note that beyond looking at ‘Attrition’, we are also looking at a variable called ‘Predicted Attrition’. This variable represents the predicted attrition probabilities that we’ve assigned to each student. In this case, we’ve grouped these values by year to get an idea of how well we’re predicting attrition for that year compared to the actual attrition rate. Comparing our predicted values to actual values gives us a sense of any weaknesses from year to year that our predictive model might have. If we do find any weakness in predictive ability, we have the opportunity to go back and further fine-tune our model in order to incorporate our findings. 

-Caitlin Garrett, Statistical Analyst at Rapid Insight

Creating Variables: Retention and Attrition

Now that we’ve highlighted some basic variables in our Creating Variables series, I think we’re ready to move on to a variable that is a little trickier to create: retention. The retention variable we’ll create will represent whether or not a freshman is retained from one fall semester to the next. Because of the number of different factors you can choose to include or exclude, I hope that you’ll use these instructions as a general guide rather than a hard-and-fast manual on how to create a retention variable.

In order to create a fall-to-fall retention variable, we need to be looking at enrollment data from several consecutive fall semesters. In order to focus just on freshmen, we’ll start by placing a filter on our first semester:

Within the filter, you’ll want to focus on the field in your dataset that represents student year and filter down to just those students who are freshmen. If you’d like to include other filters here, such as making sure you’re focusing on first-time freshmen rather than transfer students, or undergraduate rather than graduate students, this is the time to do so.

The next step is to merge this dataset and filter with the next consecutive fall semester:

Inside the merge, you’ll want to connect the datasets on an identifying field, such as student ID. In doing so, you’ll want to take all of the information about each student from only the first column, so your merge should look something like this:

Here’s where the magic happens. After you’ve set up the merge, you need to do one more step before exiting the merge node. See the section titled “Source Table Flag Columns” in the bottom right corner? That’s where we’re headed. This section allows you to create a flag variable that tells you which dataset your information came from. In this case, we’ll create a flag to let us know whenever a student shows up in both datasets and we’ll call this variable “Retained” (double click on the existing name to rename it). It should look something like this:

Now we’re getting close, but we’re not done yet: those of you who have done some predictive modeling probably know that usually when we talk about retention, we’re actually planning to model attrition. (So, instead of modeling each student’s likelihood of leaving, we’d model each student’s likelihood of staying.)Modeling attrition rather than retention is helpful because we’re focusing on a smaller population of students and honing in on the characteristics of students who are likely to leave more directly. To accomplish this, we’ll need to add a transform in order to create a new attrition variable.

In the transform, we’re going to set up the attrition variable to be the opposite of the “retained” variable. To do this, we’ll want to select the “retained” variable to work with and type the following into the Enter a Formula window: IF (A=1, 0, 1). This basically turns zeros to ones and ones to zeros. Set the result type to binary and be sure to name the variable something like “attrition” before creating and exiting… And there you have it, a state-of-the-art attrition variable to add to your collection!

-Caitlin Garrett, Statistical Analyst at Rapid Insight