Transferable Healthcare Models

Making Hospital AI Models More Portable: New Research on Transfer Learning in Healthcare

Healthcare AI has made impressive strides in recent years, particularly in predicting patient outcomes in intensive care units (ICUs). However, there’s a catch - most of these AI models are trained using data from large hospitals with substantial resources. What about smaller hospitals that can’t develop their own models from scratch?

The Problem: One Hospital’s Model Doesn’t Fit All

We identified a crucial issue: medical data varies significantly across different regions of the United States. For instance, we found that:

• Hospitals in the Northeast measure Glasgow Coma Scores more frequently than those in the South
• Some vital measurements, like FiO2 (fraction of inspired oxygen), show markedly different patterns between hospitals
• Certain measurements that are routine in one region might be rarely recorded in another

This variation means that an AI model trained at one hospital might perform poorly when used at another - a serious problem if we want to democratize access to AI-powered healthcare tools.

A New Way to Test AI Model Transferability

We developed a benchmark to evaluate how well medical AI models can adapt when moved from one hospital to another. We framed this as a “domain incremental learning” problem - essentially, how can a model learn new patterns without forgetting what it already knows?

The benchmark tests models on four critical tasks:

• Predicting in-hospital mortality
• Identifying patient decompensation (sudden deterioration)
• Estimating length of stay
• Detecting various medical conditions (phenotyping)

Innovative Solutions

We explored several approaches to make models more transferable, including:

1. Elastic Weight Consolidation (EWC): This technique helps preserve important features the model learned from its original hospital while adapting to new data.
2. Data Replay: The model periodically reviews a small sample of data from its original training, helping it maintain critical knowledge.
3. A Novel Combined Approach: We developed a new method that merges EWC and data replay, showing better performance than either technique alone.

Key Findings

The results were enlightening:

• The combined method showed superior performance on simpler tasks like mortality prediction
• More complex tasks, like predicting length of stay, proved challenging for all approaches
• Different regions required different strategies for optimal performance
• We highlighted specific areas where further innovation is needed

Why This Matters

This research is crucial for several reasons:

1. Healthcare Equity: Making AI models more portable could help smaller hospitals access advanced predictive tools without massive investments.
2. Resource Efficiency: Instead of every hospital developing its own models, institutions could build upon and adapt existing ones.
3. Better Patient Care: More reliable transfer of AI models means more hospitals can benefit from advanced predictive capabilities.

Looking Forward

While the research shows promising directions for making medical AI more portable, it also highlights remaining challenges. We note that future work could focus on developing methods that require even less data transfer between hospitals, addressing privacy concerns. This work represents an important step toward making healthcare AI more accessible and effective across different hospital settings. As these techniques continue to evolve, we might see a future where any hospital, regardless of size or resources, can benefit from advanced AI-powered clinical prediction tools.

Explore the full paper on GitHub or on OpenReview