New discoveries #6
SatMAE, WeatherFusionNet, SatlasPretrain & SRSDD datasets, Lance
Welcome to the 6th edition of the newsletter. I am delighted to share that the newsletter now has 1642 subscribers 🥳
SatMAE: Pretraining Transformers for Temporal and Multi-Spectral Satellite Imagery
Fresh from NeurIPS 2022 is the paper "SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery". Unsupervised pre-training methods have been shown to enhance performance on downstream supervised tasks, and SatMAE is a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). In addition to encoding the spatial position of patches, SatMAE also encodes temporal and spectral information within the positional embedding. This approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to 14%) and semantic segmentation.
Authors: Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, Stefano Ermon.
WeatherFusionNet: Predicting Precipitation from Satellite Data
WeatherFusionNet achieved 1st place in the NeurIPS 2022 Weather4Cast challenge. The challenge was for participants to produce high resolution short-term predictions of rainfall from low resolution satellite data. WeatherFusionNet uses three different networks to process the satellite data; extracting rain information from the current frames (sat2rad), predicting future satellite frames (PhyDNet), and combining the input sequence directly to predict rainfall (U-Net). Using this approach, WeatherFusionNet can predict severe rain up to eight hours in advance. Note that the three models are not trained end-to-end due to memory requirements, but the authors propose to do this in the future.
Authors: Jiří Pihrt, Rudolf Raevskiy, Petr Šimánek, Matej Choma
SatlasPretrain dataset
SatlasPretrain is a large-scale pre-training dataset for tasks that involve understanding satellite images. Regularly-updated satellite data is publicly available for much of the Earth through sources such as Sentinel-2 and NAIP, and can inform numerous applications from tackling illegal deforestation to monitoring marine infrastructure.
SRSDD dataset
A High-Resolution SAR Rotation Ship Detection Dataset. All data are from GF-3 Spotlight (SL) mode with a 1-m resolution and each image has 1024 × 1024 pixels. Compared with other existing datasets, the dataset contains multiple categories, namely a total of six categories of 2884 ships
Lance
Lance is an open source python tool for ‘Blazing fast exploration and analysis of computer vision data using SQL and DuckDB, backed by an Apache-Arrow compatible data format’. Lance offers functionality for dataset versioning, debugging, and model reproducibility. Given the familiarity of SQL to many people it is exciting to see a tool which uses it to work with computer vision datasets
Poll
Last week I asked ‘Which ML framework do you use the most?’, and the clear winner was Pytorch, with almost half the votes. Keras/Tensorflow was slightly behind and Sikit learn trailed. The result correlates with what I have observed in the literature at least; a tendency towards Pytorch usage, at least amongst researchers. This week I am interested to know if you have a machine/deep learning model in production?
Congratulations that the newsletter is growing. Thanks for sharing these discoveries Robin!