Promise 2016-2019 Academic Evaluation
Jason Whittle
April 30, 2019
Summary of Results
This analysis looks at the impacts of Salt Lake Community College’s (SLCC) Promise program on three primary student outcomes: term GPA, term Credits and Fall to Fall retention. The Promise program was designed to encourage Pell qualifying students to act like ‘good’ students. To qualify for SLCC’s Promise in addition to qualifying for Pell, a student needs to maintain a passing GPA, enroll in at least 12 credits in a term, and have a Degreeworks plan on file. Because of Promise’s requirements both term GPA and term Credits will be difficult to assess due to extreme self-selection bias (Promise’s requirements automatically select likely good students).
Promise is a relatively small program in terms of student involvement with 3,071 total awards to 1775 total students since Fall 2016.
- Fall to Fall (F-F) retention impact: There appears to be no certain Promise effect on Fall-Fall retention (consistent with prior Promise study results). A statistical model estimates a maximum impact of receiving Promise ranging between -1.1% to 4.9% change in F-F retention.
- Term credits earned impact: There appears to be a small positive Promise effect with some possible issues that will be discussed later. A statistical model estimates Promise students took an additional 3 credits per term (consistent with prior Promise study results). There is a high likelihood of self-selection bias with this result that will be discussed later which may render these differences meaningless.
- Term passing GPA impact: There appears to be a small positive Promise effect with some possible issues that will be discussed later. A statistical model estimated a maximum impact of receiving Promise ranging between 1.1% and 6.5%. It appears Promise students are very good students, I find it hard to believe that they are very good students because of Promise, merely there is a strong association with Promise students.
- Completion impact: Promise is still too new of a program to assess completion in manner consistent with other SLCC studies (6-year completion time frame).
It is my opinion that it is unlikely that SLCC’s Promise program had any meaningful effect on changing student behavior and was likely used primarily by students who were already ‘good’ students. The only impact from this program that can be seen as being significantly different from zero, in both the initial study for Fall 2016 students and this analysis, are variables which are part of the requirements to receive promise funds (Term credits and maintaining a passing GPA).
I believe Fall-to-Fall retention to be one of the most telling outcomes since it is not a requirement of the Promise program and thus not susceptible to self-selection in the same manner. Another sign that Promise was less effective than hoped is the number of students participating in Promise has remained relatively small. Currently as of writing this only 381 awards have been given to students in Spring 2019. If Promise were an effective and meaningful program students would try to get all they can out of SLCC before transferring to other institutions, it appears this is not the case.
Data Used for the Analysis
The data used for this analysis was pulled out of SLCC’s data warehouse. The data pulled consisted of all enrolled Pell eligible students from Fall 2015 (one year prior to SLCC Promise) to Fall 2018. Since Pell eligibility is necessary to be considered for Promise funds it is an important filter placed on our data from the beginning in order to make apples to apples comparisons to determine Promise’s true effects.
The data was grouped in two different sets for the three statistical models used. The first grouping consisted of only Fall semesters ending in Fall 2017. The Fall semester subset (F-F data) was half the size of the other data set used as it only consisted of Fall semesters (excluding spring and summer) and was stopped in Fall 2017 since Fall 2018 students would not have had the opportunity to continue Fall-to-Fall until Fall 2019.
The other data set which was only limited to enrolled Pell eligible students was used to evaluate the percentage of students with passing term GPAs (> 2.0) and term credits earned.
Study Results
The next few sections will summarizes the differences between Promise students and non-Promise Pell students. There will be several tables and explanations of why the statistical models in the next section are used to adjust for student characteristics.
Raw Results Fall-to-Fall Retention
Promise students retain F-F at a higher rate than the average Pell student. The table below displays the raw differences between Promise students and non-Promise Pell students. There is an eight percent difference between Promise and non-Promise Pell students but this difference is inaccurate since students who take more credits generally retain at higher percentages.
| Promise | F-F Retained % |
|---|---|
| No | 46 |
| Yes | 54 |
The following two tables show how the eight percent difference shrinks considerably when placing only the modest additional control of adjusting for term credits attempted on the F-F data. The first table shows a difference of only five percent when you limit the data set to those students who attempted at least 6 credits in the semester.
| Promise | F-F Retained % |
|---|---|
| No | 49 |
| Yes | 54 |
The table below performs the same adjustment to the F-F data as the previous table but increases the credits attempted to 12 (in line with the Promise requirement). The difference has fallen to just 3% only by controlling for credits attempted in an unsophisticated (statistically) manner. Tables like these and the fact that Promise is not randomly assigned will necessitate the full statistical model presented in the next section in order to assess Promise’s impacts while attempting to control for self-selection bias (good students utilizing Promise rather than Promise being the impetus for student behavioral change).
| Promise | F-F Retained % |
|---|---|
| No | 50 |
| Yes | 53 |
Raw Results Term Passing (GPA > 2.0)
For term passing GPA (> 2.0 for the term) there is an 11 percent advantage to the Promise students compared to the non-Promise Pell students when no additional controls are placed on the data. This result will decay with additional controls, just like F-F retention did. The decline however, is not nearly as much as with F-F giving us more confidence going into the statistical model that this is related to Promise students and not just noise.
| Promise | Pass % |
|---|---|
| No | 72 |
| Yes | 83 |
Even when filtered to only look at students who attempted more than 12 term credit hours there is still a 6% advantage for Promise student with this academic outcome. There are Promise students who fall below the 12 credit hour limit but still received the funds as I understand it (technically counted as a Promise student but are failing to meet the requirements). This explains the small drop in Promise students pass rates. Promise students are very high achieving students based on these differences.
| Promise | Pass % |
|---|---|
| No | 76 |
| Yes | 82 |
Raw Results Term Credits Earned
Term credits earned (i.e. passed credits) is the hardest impact of the Promise program to assess. While maintaining a Passing GPA is a requirement of Promise it is also something the student might not fully control or know going into the semester. The student might assume they are going to pass all their classes (otherwise they probably wouldn’t take the course they believed they would fail). For passing GPA we can also try to group students based on the number of credits they attempt in the semester to ‘match’ students who are more similar in other academic behaviors. For example, a student who takes one course might not be interested in academic goals such as a degree and probably should not be directly compared to a Promise student.
But term credits earned is something the student knows with a fair degree of certainty going into the semester. You can’t earn 12 credits unless you enroll in at least 12 credits and you can’t receive Promise funds without enrolling in 12 credits. There is not an easy way to assess the question of Promise’s impact on term credits taken while simultaneously controlling for credits attempted (it is literally one of the self-selection methods for Promise). In other words, it is quite hard to separate out those students who do not have academic intentions at heart or are really on the fence about college. It is unlikely quantitative methods will ever be able to determine if a student who enrolled in 12 credits would have only take 3 credits without Promise (although this narrative seems really optimistic and not at all consistent with everything else we know about Promise).
Term credits earned at SLCC is a very skewed and spiky distribution as the figure below shows. With a maximum peak at 0 earned credits and other peaks around 3, 6, 9, and 12. There is very little difference between the percent of students who earn 3 - 6 credits (lower dedication/focused students) and 9 - 12 credits (more dedicated/focused students). I include this plot to demonstrate how uneven and skewed this distribution is. These types of distributions, combined with the known self-selection into Promise make me question the full extra course result seen in the raw numbers and the statistical model output below.
As would be expected by the requirements of the program, Promise students earn more credits than non-Promise Pell students. These raw differences and the differences from the statistical model explained later are consistent from the Promise student on the initial Fall 2016 cohort. It appears Promise student complete an additional 4 credit hour on average (Prior research on Promise showed an additional 3 credit hour course).
| Promise | Average Term Credits |
|---|---|
| No | 8 |
| Yes | 12 |
Model Result Details
There were two derivatives of Multilevel regression Models (MLM) used to evaluate the impact of Promise (2 logitistic regression and 1 linear regression). The results for all three models are below with the estimated effects of Promise on the first line. The first two models from left to right are in Log-odds scale and are thus not directly interpretable without some calculations (however the appropriate interpretation for the estimated Promise effects were summarized in the Results Summary section). The third model from left to right is a linear model and is directly interpretable on a term credits level. GPA scores for both High School and Prior Undergrad GPA in the (4,5] range represents a missing value and should be interpreted as a Null value (they were coded this way for modeling purposes after they were pulled out of SLCC’s data warehouse).
Ethnicity and Terms were adjusted for using random effects for all of these models. The ethnicity results are presented in the next subsection. Term was held as a random effect to control for serially correlated variables and not presented here since it is beyond the scope of this analysis.
| Dependent variable: | |||
| F_F | passed | TERM_UG_CREDITS | |
| generalized linear | generalized linear | linear | |
| mixed-effects | mixed-effects | mixed-effects | |
| (1) | (2) | (3) | |
| PROMISE | 0.076 (0.061) | 0.153*** (0.054) | 3.508*** (0.083) |
| scale(CREDITS_ATTEMTPED) | 0.141*** (0.011) | 0.274*** (0.009) | |
| scale(AGE_ON_FIRST_DAY) | 0.014 (0.013) | -0.050*** (0.010) | -0.577*** (0.019) |
| GENDERMale | -0.033 (0.021) | -0.230*** (0.016) | -0.002 (0.030) |
| married | -0.118*** (0.027) | 0.239*** (0.022) | 0.234*** (0.038) |
| COLLEGE_READY_ENGLISHY | -0.001 (0.023) | -0.030 (0.019) | 0.252*** (0.034) |
| COLLEGE_READY_MATHY | 0.152*** (0.024) | 0.087*** (0.020) | 0.480*** (0.034) |
| FIRST_GENERATION_INDY | -0.048** (0.021) | -0.069*** (0.017) | -0.060** (0.030) |
| FIRST_TERM_NON_CONCURRENT_INDY | 0.001 (0.036) | -0.048 (0.031) | 2.138*** (0.056) |
| EVER_CONCURRENT_INDY | -0.037 (0.027) | 0.239*** (0.022) | -0.789*** (0.038) |
| scale(PRIOR_UG_CREDITS) | -0.346*** (0.014) | 0.083*** (0.011) | 0.042** (0.019) |
| slco | 0.126*** (0.026) | -0.030 (0.020) | -0.167*** (0.036) |
| HIGH_SCHOOL_GPA(1,2] | -1.420* (0.796) | -0.471 (0.565) | 0.577 (1.062) |
| HIGH_SCHOOL_GPA(2,2.5] | -1.375* (0.792) | -0.258 (0.562) | 0.888 (1.056) |
| HIGH_SCHOOL_GPA(2.5,3] | -1.096 (0.791) | 0.008 (0.562) | 1.484 (1.054) |
| HIGH_SCHOOL_GPA(3,3.5] | -1.093 (0.790) | 0.257 (0.561) | 1.798* (1.054) |
| HIGH_SCHOOL_GPA(3.5,4] | -1.075 (0.790) | 0.921 (0.562) | 2.039* (1.054) |
| HIGH_SCHOOL_GPA(4,5] | -1.289 (0.790) | 0.120 (0.561) | 1.314 (1.053) |
| PRIOR_UG_GPA(1,2] | 0.461*** (0.101) | 0.464*** (0.063) | 1.204*** (0.126) |
| PRIOR_UG_GPA(2,2.5] | 0.912*** (0.097) | 0.790*** (0.061) | 2.198*** (0.121) |
| PRIOR_UG_GPA(2.5,3] | 1.167*** (0.095) | 1.278*** (0.060) | 3.123*** (0.118) |
| PRIOR_UG_GPA(3,3.5] | 1.242*** (0.095) | 2.008*** (0.061) | 4.311*** (0.117) |
| PRIOR_UG_GPA(3.5,4] | 1.275*** (0.094) | 2.775*** (0.062) | 5.225*** (0.117) |
| PRIOR_UG_GPA(4,5] | 0.578*** (0.095) | 1.243*** (0.062) | 1.228*** (0.121) |
| Constant | 0.015 (0.798) | -0.841 (0.569) | 1.365 (1.064) |
| Observations | 40,622 | 88,986 | 88,986 |
| Log Likelihood | -27,239.010 | -45,648.240 | -256,900.900 |
| Akaike Inf. Crit. | 54,532.030 | 91,350.470 | 513,855.800 |
| Bayesian Inf. Crit. | 54,764.550 | 91,604.170 | 514,109.500 |
| Note: | p<0.1; p<0.05; p<0.01 | ||
Random Effect: Ethnicity
Below is a breakdown of the random effects differences between ethnicities at SLCC for F-F retention, Passing Term GPA and Term Credit Earned for both Promise and non-Promise Pell students at SLCC (this is just extra infromation we obtain about SLCC students in general while testing Promise). The black dot represents the group mean and the black horizontal line represents the likely mean estimate error. The x-axis is not directly interpretable so it is best to use these plots to note differences between groups that are large enough to be considered statistically significant (when the black horizontal lines indicating each groups error do not cross, the difference between these two groups can be considered statistically significant). Larger groups typically will have smaller error as the group mean is based on many more observations (note how small White and Hispanic group errors are as they are the two largest student ethnicity groups in this study). The first two plots x-axis display relative effects and are not directly interpretable without additional calculation.
Pacific Islander for instance are lagging far behind their peers in all three and should be focused on for achievement gap initiatives. For F-F retention there is a noticeable gap between ‘Pacific Islanders or Native Hawaiian’ and the rest of SLCC. The F-F gap indicates that this group lags behind all other groups at SLCC in a statistically significant (likely non-random) manner.
The x-axis values for the final plot, ‘Term Credits Earned’, are in term credit level since this output was generated from a linear model. For this plot we see that while there are differences based on ethnicity these differences are at a maximum difference of less than 1 term credit earned.
Financial Aid Interviews
There were several notable points about the success of the Promise program at SLCC that were the result of interviews with staff from Financial Aid (FAS). Promise has remained a small program at SLCC despite a continuous marketing effort across campus, according to FAS there were several reason in their minds for this.
- Students are increasingly receiving enough Pell money and there are fewer and fewer students in the “Pell gap” range.
- There are many other funding sources for students that fall into the need based aid category that do not require meeting the standards of Promise.
- The degree plan as a requirement was seen as a hurdle that many students, even after being notified of Promise eligibility, were not willing to overcome (i.e. meet with an divisor and file a plan).
- Promise is administratively ‘high maintenance’. There is a lot of manual checking of students to see if they meet the requirements throughout the semester.
Data checks
This section will display some data quality checks and some pre-modeling processing steps take for future replication purposes and will not be discussed.
Correlation plot visually displays the correlation matrix.
Near-zero Variance check. GENDERUnkown was removed from the analysis as a variable.
| freqRatio | percentUnique | zeroVar | nzv | |
|---|---|---|---|---|
| F_F | 1.487936 | 0.0022475 | FALSE | FALSE |
| passed | 2.597720 | 0.0022475 | FALSE | FALSE |
| TERM_UG_CREDITS | 1.125380 | 0.0516935 | FALSE | FALSE |
| AGE_ON_FIRST_DAY | 1.058193 | 0.0775403 | FALSE | FALSE |
| GENDERMale | 1.201098 | 0.0022475 | FALSE | FALSE |
| GENDERUnknown | 350.723320 | 0.0022475 | FALSE | TRUE |
| FIRST_GENERATION_INDU | 5.305697 | 0.0022475 | FALSE | FALSE |
| FIRST_GENERATION_INDY | 1.089903 | 0.0022475 | FALSE | FALSE |
| married | 3.464703 | 0.0022475 | FALSE | FALSE |
| COLLEGE_READY_MATHU | 16.527280 | 0.0022475 | FALSE | FALSE |
| COLLEGE_READY_MATHY | 1.863035 | 0.0022475 | FALSE | FALSE |
| COLLEGE_READY_ENGLISHU | 11.491016 | 0.0022475 | FALSE | FALSE |
| COLLEGE_READY_ENGLISHY | 1.086914 | 0.0022475 | FALSE | FALSE |
| PRIOR_UG_CREDITS | 5.016490 | 1.1541141 | FALSE | FALSE |
| FIRST_TERM_NON_CONCURRENT_INDY | 5.121767 | 0.0022475 | FALSE | FALSE |
| EVER_CONCURRENT_INDY | 2.560722 | 0.0022475 | FALSE | FALSE |
| slco | 3.494242 | 0.0022475 | FALSE | FALSE |
| north | 12.764269 | 0.0022475 | FALSE | FALSE |
| CREDITS_ATTEMTPED | 1.091541 | 0.0842829 | FALSE | FALSE |
| ETHNICITY | 2.740225 | 0.0101140 | FALSE | FALSE |
| TERM_CODE | 1.065546 | 0.0078664 | FALSE | FALSE |
| HIGH_SCHOOL_GPA | 7.037909 | 0.0078664 | FALSE | FALSE |
| PRIOR_UG_GPA | 1.065487 | 0.0078664 | FALSE | FALSE |
All variable histograms
