Multi-modal co-learning for Earth observation: enhancing single-modality models via modality collaboration

This work introduces a general multi-modal co-learning framework for Earth Observation (EO) that improves single-modality models by exploiting complementary information from multiple sensor modalities during training. The approach addresses practical EO scenarios where only one modality is available at inference time, avoiding the need for modality-specific architectures or task-dependent solutions. By combining contrastive and modality-discriminative learning, the framework encourages models to capture both shared and modality-specific information. Extensive experiments across four EO benchmarks covering classification and regression tasks demonstrate consistent improvements over state-of-the-art methods from machine learning, computer vision, and EO, highlighting the potential of multi-modal collaboration for robust single-modality inference.

Recommended citation: Mena, Francisco, et al. "Multi-modal co-learning for Earth observation: enhancing single-modality models via modality collaboration." Machine Learning 114.12 (2025): 279.
Download Paper