Self-supervised spatio-temporal representation learning of Satellite Image Time Series

Iris Dumeur (1) , Silvia Valero (1) , Jordi Inglada (1, 2)

1 CESBIO - Centre d'études spatiales de la biosphère
2 CNES - Centre National d'Études Spatiales [Toulouse]

In this paper, a new self-supervised strategy for learning meaningful representations of complex optical Satellite Image Time Series (SITS) is presented. The methodology proposed named U-BARN, a Unet-BERT spAtio-temporal Representation eNcoder, exploits irregularly sampled SITS. The designed architecture allows learning rich and discriminative features from unlabeled data, enhancing the synergy between the spatio-spectral and the temporal dimensions. To train on unlabeled data, a time series reconstruction pretext task inspired by the BERT strategy is proposed. A Sentinel-2 large-scale unlabeled data-set is used to pre-train U-BARN. To demonstrate its feature learning capability, representations of SITS encoded by U-BARN are then fed into a shallow classifier to generate semantic segmentation maps. Experimental results are conducted on a labeled data-set (PASTIS). Two ways of exploiting U-BARN pre-training are considered: either U-BARN weights are frozen (named U-BARN FR) or fine-tuned (U-BARN FT). The obtained results demonstrate that representations of SITS given by U-BARN FR are more efficient for land cover classification than those of a supervised-trained linear layer. Then, we observe in scenarios with scarce reference data-set that the fine-tuning brings a significative performance gain compared to fully-supervised approaches. We also investigate the influence of the percentage of element masked during pre-training on the quality of the SITS representation. Eventually, semantic segmentation performances show that the fully supervised U-BARN architecture reaches slightly better performances than the spatio-temporal baseline (U-TAE).

Mots clés: Satellite Image Time Series SITS; Transformer; Self-supervised learning; Spatio-Temporal Network; Unet; Representation Learning

Domaines: Intelligence artificielle [cs.AI]; Interfaces continentales, environnement; Méthodologie [stat.ME]; Machine Learning [stat.ML]

Iris Dumeur, Silvia Valero, Jordi Inglada. Self-supervised spatio-temporal representation learning of Satellite Image Time Series. 2023. ⟨hal-04084839v2⟩