New Discoveries #26
A Neural Plasticity-Inspired Foundation Model, A Satellite Band Selection Framework, Sen2Fire, UV6K & SICKLE datasets, and MapYourCity challenge
Welcome to the 26th edition of the newsletter. I'm delighted to share that the newsletter now has 9,305 subscribers 🔥
DOFA: A Neural Plasticity-Inspired Foundation Model
One of the key challenges in developing EO foundation models is how to cope with multi-sensor data. Early foundation models utilised a single data source (typically Sentinel-2 and/or Landsat), or a specific range of spatial and spectral resolutions. Over time the range of sensors and spectral bands supported has increased. For example, Satlas comprises a large-scale dataset from various sensors, with individual pre-trained models provided for each sensor. However these approaches do not tap into the vast potential offered by the optimum fusion of complementary information provided by multimodal data.
DOFA is an adaptive foundation model which aims to overcome the inefficiencies and complexities of employing separate vision encoders for each modality by drawing inspiration from the concept of neuroplasticity.
Neuroplasticity is the ability of the brain to adapt its neural connections in response to varying stimuli and environmental changes. Dynamic One-For-All (DOFA) is designed to emulate the dynamic mechanisms of neuroplasticity (illustrated above) by adaptively altering its network weights in response to novel data modalities.
DOFA is evaluated on diverse real-world tasks, and surpasses current leading foundation models in 12 out of 13 datasets. This performance, achieved with a singular network using identical pretraining, indicates DOFA’s superior handling of multimodal EO data. DOFA mirrors the dynamic learning of the human brain for continuous model improvement from diverse data sources, making it highly adaptable for remote sensing’s broad data spectrum. The experimental results showcase DOFA’s versatility and effectiveness and identify DOFA as a novel foundation model for remote sensing data.
A Satellite Band Selection Framework
This study employs the Univariate Marginal Distribution Algorithm (UMDA) to select spectral bands from Landsat-8 imagery to optimise the performance of a DeepLabv3+ segmentation model for deforestation detection. Experimental results identify band combinations that achieved superior balanced accuracy compared to commonly utilised combinations. Moreover, the optimal band combinations identified by the UMDA-based approach enabled the DeepLabv3+ model to surpass state-of-the-art approaches compared in this study.
📖 Paper
Sen2Fire dataset
The Sen2Fire dataset is curated from Sentinel-2 multi-spectral data and Sentinel-5P aerosol product, comprising a total of 2466 image patches. Each patch has a size of 512×512 pixels with 13 bands. Results presented in the associated paper suggest that, in contrast to using all bands for wildfire detection, selecting specific band combinations yields superior performance.
Urban Vehicle Segmentation Dataset (UV6K)
Urban Vehicle Segmentation Dataset (UV6K) is a high-resolution remote sensing urban vehicle segmentation dataset. It consists of 6,313 images and contains 245,141 vehicles. The RGB images have a resolution of 0.1m and an image size of 1024x1024 pixels.
📀 Dataset
SICKLE: A Multi-Sensor Satellite Imagery Dataset Annotated with Multiple Key Cropping Parameters
SICKLE is a time-series dataset of multi-resolution imagery from 3 satellites: Landsat-8, Sentinel-1 and Sentinel-2. Temporal sequences cover locations where farmers are primarily engaged in paddy cultivation in the Cauvery Delta region of Tamil Nadu, India. Annotations cover key cropping parameters at multiple resolutions (3m, 10m and 30m). There parameters include the variety of paddy, its growing season and productivity in terms of per-acre yields. The authors benchmark SICKLE on three tasks: crop type, crop phenology (sowing, transplanting, harvesting), and yield prediction.
MapYourCity challenge
MapYourCity challenges entrants to leverage a dataset containing street-view imagery, Medium Resolution Sentinel 2 imagery and VHR resolution top-view imagery (Airbus Pleiades) to train a multi-modal deep-learning system capable of accurately estimating the construction year of any given building.
🖥️ Website
💰 1st prize: 2500 Euros
🗓️ Ends: 14 July 2024
