January 2025, Amini

Summary

We build on the channel-independence and patch-segmentation concepts introduced by PatchTST, adapting them to remote-sensing pixel time series for self-supervised representation learning. In our approach, each spectral band/index (e.g., Red, NDWI) is treated as a univariate sequence, split into “patch tokens,” then processed by a ~600k-parameter Transformer. We randomly mask a substantial fraction of these patch tokens and train the model to reconstruct the missing segments, capturing both short sub-seasonal dynamics within each patch and multi-month context across the entire time series.

In this blog, we discuss:

1. Patch-based masking for remote-sensing pixel time series,
2. Experimental results on Uasin Gishu County, covering land-cover classification and anomaly detection,
3. Future directions, including foundation-scale training and integration of additional sensors modalities.

Method Overview

Pixel-Time-Series Setup

Let {x1(m).…xL(m)} represent a univariate time series for spectral channel m (e.g., the Red band in Sentinel-2). For biweekly composites, L≈26 data points per year, but L can grow to hundreds of time steps when spanning multiple years. Our goal is to learn an embedding function Φ mapping each pixel-level sequence to a latent representation that captures meaningful spatial-temporal semantics.

Patch-Based Segmentation & Channel Independence

To reduce computational overhead while incorporating local temporal context, each univariate series is divided into patches of length P, typically with stride S≤P. For instance, P=16 and S=8 might produce about L/S patch tokens in a length-L sequence. Next, instead of early channel fusion, each channel (Red, NIR, NDWI, etc) is processed independently via the same Transformer encoder. This approach avoids mixing different noise profiles across channels, promotes parameter efficiency and helps learn universal time-series representations more effectively.

Masked Autoencoding Objective

We randomly mask ~ 40% of the patch tokens in each channel by replacing them with mask tokens. A 600K parameter transformer encoder then processes both unmasked and masked patches, producing patch-level embeddings of shape (X, n_patches, embedding_size). This patch-wise representation enables the model to capture local sub-seasonal patterns within each patch as well as broader multi-season context across patches. Finally, a lightweight projection head reconstructs only the masked patches under a mean-squared error (MSE) loss, ensuring the model integrates global temporal information rather than just relying on short-range interpolation.

Experiments

Dataset & Preprocessing

We tested this approach on Sentinel-2 biweekly composites over 4 years from Uasin Gishu County, Kenya, a region known for its diverse land-cover types and environmental conditions. The preprocessing steps included:

• Composite Formation: For each biweekly interval, multiple Sentinel-2 acquisitions are merged. Cloud masks are applied, median reflectance computed, and data converted to a pixel time series format for temporal processing.
• Spectral Channels: A set of 10 spectral bands/indices [‘Green’, ‘Blue’, ‘Red’, ‘NIR’, ‘SWIR1’, ‘SWIR2’, ‘NDMI’, ‘NDWI’, ‘CI’, ‘NDVI’] is used to form a comprehensive multi-spectral pixel time series used for pretraining.
• Data Splits: We use spatial splits for training, validation, and test sets, minimizing overlap to ensure that the model generalizes to unseen regions.
• Land-Cover Labels (Weak Labels): To evaluate the effectiveness of the learnt embeddings, we leverage ESRI LULC data as “weak labels” in a downstream classification task. Although these labels are not a perfect ground truth, their broad coverage and easy availability make them practical for experimentation. By accepting some degree of label noise, we can still gauge how well the embeddings separate high-level land-cover categories (e.g., water vs. vegetation vs. built-up areas).

Results & Discussion

Pretraining Dynamics & Reconstruction Performance

The ~600k-parameter model converges smoothly under the masked-autoencoder objective, demonstrating a strong ability to reconstruct missing patches. Over the course of training, the training loss steadily declines (from ~0.1715 to ~0.0232), while validation loss stabilizes around 0.0864 –0.0211 signifying limited overfitting.

Land Cover Classification & Confusion Matrix

To evaluate the effectiveness of the learned representations, we leverage the pretrained transformer’s frozen weights to extract patch-level embeddings. These embeddings are averaged across the patch dimension to obtain a single global representation for each pixel’s temporal sequence. The resulting fixed-dimensional vectors are then fed into a linear classifier trained to distinguish five land cover classes derived from ESRI LULC labels: water, trees, crops, built area, and rangeland.

Despite the class imbalance in our test set and label noise that might be inherent in the ESRI dataset, diagonal dominance from the confusion matrix indicates the model’s learnt representations effectively distinguish these major land-cover types.

Anomaly Detection in Maize Fields

To further evaluate the quality of the learned representations, we conducted an anomaly detection experiment focusing on maize field pixels from this dataset. We first extracted 128-dimension embeddings for each pixel by aggregating the patch-based representations from the pre-trained transformer. These embeddings were then used to train an Isolation Forest algorithm, which identifies anomalous patterns labelled as “-1” and normal temporal behaviours labelled as “1”. The figure below shows a 2D PCA visualization of the embeddings:

Conclusion & Future Directions

Overall, we’ve shown that patch-based masking stands as a flexible and powerful method for self-supervised representation learning on remote-sensing pixel time series, balancing both computational efficiency and local scale dynamics that are critical for real -world earth observation challenges. While our present experiment focuses on a single county, we are actively developing a foundation scale training pipeline targeting a larger geographic scope and additional sensor modalities. We will share our findings. In the meantime, we invite you to explore our pretrained model , trained on Uasin Gishu County on Hugging Face and share any feedback you may have through our Community Channel.

Read & Download our Research Paper

Self-Supervised Representation Learning on Remote Sensing Pixel Time Series with Patch-Based… was originally published in The Sovereign Stack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Climate change and variability continue to exert immense pressure on the African continent. As recorded by the Intergovernmental Panel on Climate Change (IPCC), Africa’s contribution is among the lowest of historical greenhouse gas (GHG) emissions responsible for human-induced climate change, yet it remains the most vulnerable continent in the face of climate change.

This change, coupled with Africa’s high dependence on climate-sensitive sectors such as agriculture, poses significant challenges to livelihoods, food security, and socio-economic development. However, the advent of machine learning (ML) and earth observation (EO) technologies offers a glimmer of hope, providing data-driven solutions that foster resilience and adaptability.

Fostering Climate Resilience in Agriculture

Agriculture is the backbone of many African economies, providing livelihoods for a significant proportion of the population. More than 60 percent of the population of sub-Saharan Africa is smallholder farmers, and about 23 percent of sub-Saharan Africa’s GDP comes from agriculture. According to a report by McKinsey, agriculture accounts for about 23% of Africa’s GDP and about 60% of its jobs. However, due to the sector’s reliance on rainfall and changing weather patterns, climate change poses a significant threat to agriculture and, by extension, food security.

ML and EO can predict weather patterns with increased accuracy, aiding farmers in making informed decisions regarding planting and harvesting times. For example, satellite-based remote sensing can provide real-time information on soil moisture, crop health, and weather conditions. This information is crucial for smallholder farmers who rely on rain-fed agriculture.

Through these technologies, crop yields can also be predicted more accurately. A study conducted by a group of researchers including the Climate Hazards Center, Department of Geography, University of California et al has indicated that machine learning algorithms showed an impressive accuracy rate of over 85% in some regions. Such predictive capabilities could significantly enhance food security and contribute to economic stability in the region. It allows for better planning, from predicting potential food shortages to providing valuable data for agricultural policy development.

Another aspect of climate resilience in agriculture is the ability to monitor and adapt to land use changes. Africa is experiencing rapid urbanization, which often comes at the expense of agricultural lands. EO technologies can track these changes, allowing for more effective land use planning and helping ensure that agricultural productivity is not undermined.

Furthermore, ML and EO can also play a vital role in pest and disease management, a challenge exacerbated by climate change. By identifying signs of pest infestations or disease outbreaks early, preventative measures can be implemented promptly, thereby protecting crop yields and the livelihoods of farmers.

Biodiversity Protection

Biodiversity plays a vital role in ensuring the overall health of the planet and has a direct impact on human lives. However, the biodiversity crisis we are facing is as alarming as the climate crisis. The African continent is home to a wealth of biodiversity, including approximately 30% of the planet’s remaining rainforest, such as the Congo Basin, the second-largest rainforest in the world.

Through EO technology, changes in land cover — a significant driver of biodiversity loss — can be detected and analyzed in real-time. This allows for rapid responses to threats such as deforestation, illegal logging, and encroachment. For example, the United Nations has shown that Africa loses 3.9 million hectares of forest per year, a trend that can be combated more effectively with the use of these technologies.

Additionally, machine learning and earth observation technologies provide powerful tools for monitoring and conserving biodiversity. Satellite imagery, combined with machine learning algorithms, can track changes in ecosystems, identify threats to endangered species, and monitor protected areas’ effectiveness. This provides invaluable data for crafting and implementing effective conservation strategies.

Natural disasters such as droughts, floods, and cyclones are becoming increasingly frequent and intense due to climate change, posing a significant threat to African communities. Timely and effective disaster management can reduce these impacts, saving lives, and preserving livelihoods. Here too, machine learning and earth observation technologies can be of immense value. By analyzing historical and real-time data, ML algorithms can be used to predict the likelihood of natural disasters, providing crucial early warnings to communities at risk.

Once a natural disaster has occurred, ML and EO can also aid in damage assessment and recovery efforts. Satellite imagery can quickly provide an overview of the affected areas, identifying infrastructure damage, flooded areas, or landslides. Machine learning algorithms can then process this data, aiding in the efficient allocation of resources where they are most needed.

Conclusion

In this transformative landscape, startups like Amini are taking the lead. Amini has established itself as the single source of truth for African environmental data, addressing the critical issue of Africa’s environmental data scarcity. Through a holistic solution that harnesses AI and space technologies, Amini aims to bridge this data gap, foster economic inclusivity for farmers, and enhance supply chain resilience across the continent and beyond.

Significant investments, policy support, and capacity building remain essential to fully exploiting the potential of these technologies. Leveraging these powerful tools can enable us to mitigate the impacts of climate change, fostering a more resilient and sustainable future for Africa. As the African proverb says, “Tomorrow belongs to the people who prepare for it today.” By acting now and embracing the transformative potential of these technologies, we can prepare for and secure a resilient tomorrow for Africa. The promise of this future depends on the actions we take today.

Machine Learning and Earth Observation — A Beacon of Hope for a Climate-Resilient Africa was originally published in The Sovereign Stack on Medium, where people are continuing the conversation by highlighting and responding to this story.