Predicting hospital admission, re-admission, and death

Using two predictive models

Predictive Model #1

Machine Learning Enhanced Predictions of Hospital Readmission or Death in Heart Failure

Background

Heart failure is a leading cause of hospitalization
The LACE risk score is an established index to quantify the risk of readmission or death¹
There are few tools to accurately identify patients at high risk for unplanned admissions
So we built Predictive Model #1: a machine learning (ML)-generated risk score for readmissions or deaths

Objective

Use Machine Learning to predict hospital readmissions.

Methods

Included HF patients in the OM1 Real-World Data Cloud
Index date was the hospital admission
Training (67%) and validation (33%) patient sets
Predictive features were developed by Machine Learning
A key predictive feature is the OM1 Medical Burden Index (OMBI^®), which is a standardized measure generated from extensive analysis of OM1’s longitudinal patient cohort (n>175M)
OM1 HF readmission risk score (on 0-100 scale) was developed, and compared with the LACE risk score

REFERENCES

LACE Index Scoring Tool for Risk Assessment of Death and Readmission, Ottawa Hospital Research Institute

Results

This study included a total of 14,065 patients with HF. Of these, 3,502 were readmitted within 30 days after the index admission, and 10,563 were not readmitted.

The single most predictive feature was the OMBI^® score, derived from machine learning. The mean OMBI^® admission was 72 among the patients who were readmitted, and this compares to the mean of 28 among the patients who were not readmitted (Table 1). A few other variables were also predictive but not as strong as the OMBI^®.

Table 1. Patient Characteristics Pre-admission through Discharge from the Index Admission

Results (continued)

Figure 1 shows the ROC curves for the published LACE index and the OMBI score*. In this study, the LACE index score has an area under the ROC of 0.69, and the OMBI score has a higher area under the ROC of 0.77. The area under the ROC is also known as the C-Statistic.

Because there is often a trade-off between precision and sensitivity, we compared the sensitivity of the LACE index and the OMBI score by matching the precision. Results are shown in Figure 2. When fixing the precision at 36%, the OM1 score had a sensitivity of 81%, and the LACE index had 71%. The OMBI score had a 10% higher sensitivity than the LACE index.

*Formerly called OM1 risk score

Figure 1. Receiver Operating Characteristic (ROC) Curves for the LACE index (C Statistic 0.69) and the OMBI Score (C Statistic 0.77)

Figure 2. When matched by precision to the LACE Score (36%) the OMBI Score (81%) was more sensitive than the LACE Score (71%)

Results (continued)

Figure 3 shows that the OMBI score* was closely correlated with the observed readmission rate. There was a clear linear trend. From the slope, we can estimate that for every 10 unit increase in the risk score, the readmission rate increases by about 12%.

Figure 4 shows a clinical application of the OMBI score. For example, based on Mr. Smith’s patient profile, OM1 analytics can produce a score. In this example, the score is 70. This score indicates a high risk of readmission. Appropriate level of patient intervention can then be applied.

*Formerly called OM1 risk score

Figure 3. The OMBI Score was Closely Correlated with the Observed Readmission Rate

Figure 4. Clinical Application of OMBI Score

Conclusions

In this study, we presented a new model to predict mortality and readmission at 30 days after an index admission for HF.
The new model had superior performance than the published LACE index. Future work includes refining the model by including lab results and more unstructured data.
Integrating these predictive models into clinical workflow will permit timely interventions in high risk patients.

View the full poster here for more details and a list of references.

Predictive Model #2

Machine Learning Generated Risk Model to Predict Unplanned Hospital Admission in Heart Failure

Background

Heart failure is a leading cause of hospitalization. There are few tools to accurately identify patients at high risk for unplanned admission in the outpatient setting. We used machine learning (ML) on outpatient electronic medical records and medical claims to develop a HF specific predictive model.

The design of Predictive Model #2 was similar to Model #1, except that the outcome in this model was an unplanned admission instead of a readmission.

Objective

The objective is to develop a Machine Learning model to predict unplanned Heart Failure admissions.

Methods

Machine Learning methods included Random Forest, Xgboost and Treenet models.
The key predictive feature is the OM1 medical burden index (OMBI^®), which is the same feature as used in Model #1.
This model included a 12-month observation period, and a 6-month prediction period. A total of 48,761 patients were included. Of these, 2,770 patients were admitted due to HF, and the remaining were not admitted during the 6-month prediction period.

Results

The distribution of gender was similar between the admission and non-admission groups (Table 1).

Patients in the admission group were about 2 years younger.

There was a big difference in OMBI^®, the mean was 24 in the admission group, and 10 in the non-admission group.

The admission group also had higher Charlson comorbidity index with a mean of 3.9, as compared to 2.7 in the non-admission group.

Table 1. Patient characteristics during the 12-month observation period, stratified by the admission status during the 6-month prediction period

Results (continued)

Table 2 shows that the OMBI^® was the single most predictive variable in an univariate analysis. With OMBI^® being the only variable in the model, the model achieved a C-statistic (or area under the ROC) of 0.72. The odds ratio was 1.14, which means for every 10% increase in the OMBI^®, the odds of being admitted due to HF increases by 14%.

The next single most important predictor was the number of services in the past 12 months, which had a C-statistic of 0.71. The number of services here means the number of medical procedures. Other strong predictors included the number of hospitalizations in the past 12 months, time since last hospitalization and Charlson comorbidity index.

Table 2. The OMBI^® is the single most predictive variable in univariate analysis

Figure 1 shows that starting with the OMBI^®, the sequential addition of other machine learning-generated predictive features to the OMBI^® improved the predictive performance. The C-statistics increased from 0.72 to 0.79. The final model with C-statistic of 0.79 included all of the predictors listed in Figure 1.

Figure 1. Addition of other machine learning-generated predictive features to the OMBI^® improved the predictive performance (C statistic: 0.72 to 0.79)

Results (continued)

Figure 2 shows the odds ratios for the most influential predictive features in a multivariable analysis. Results showed that with all other variables in the model, the OMBI^® was still very significant. This means the predictive effect of the OMBI^® cannot be replaced by other variables. Please note the contributions of predictors cannot be simply measured by the odds ratios. In this figure, the number of hospitalizations had the highest odds ratio, around 1.9. However, many patients did not have a hospitalization during the 12-month observation period. Thus, although it had the largest odds ratio, it was not the most predictive feature.

Figure 2. Odds ratios for the most influential predictive features in multivariable analysis

Figure 3 shows the final model with all the predictors listed in Figure 2. The model had an C-statistic of 0.79. This is considered a very high C-statistic in predicting HF admissions.

Figure 3. Receiver operating characteristics of the machine learning model (C statistic: 0.79)

Conclusions

We demonstrated the utility of machine learning in leveraging variables already available in an outpatient EMR and medical claims to predict hospitalizations in 8 out of 10 patients. When integrated into the clinical workflow, such tools may offer the ability to focus resources on patients at highest risk for unplanned admission.

View the full poster here for more details and a list of references.