[SIGSPATIAL '26] rs-embed: Solving the Fragmentation of Remote Sensing Foundation Models

TL;DR

The remote sensing field is currently in a "Wild West" phase of Foundation Models (RSFMs). While models like SatMAE, Prithvi, and RemoteCLIP offer powerful representations, their APIs and data requirements are a mess of inconsistency. rs-embed is a new Python library that provides a unified, "Any Model, Any Place, Any Time" interface, enabling researchers to fetch embeddings for any coordinate and timeframe with just one line of code.

The Bottleneck: Engineering Friction in Earth Observation

If you want to compare two different remote sensing models today, you usually face three hurdles:

1. Deployment Chaos: One model is on Hugging Face, another is a custom PyTorch script requiring a specific environment.
2. Data Sourcing: You have to manually query Google Earth Engine (GEE) or Planetary Computer, handle cloud masking, and align spectral bands (e.g., 6-band vs. 12-band inputs).
3. Inconsistency: Some authors release precomputed embeddings on specific grids, while others release raw weights. This makes "Fair Benchmarking" an expensive engineering nightmare.

Methodology: The rs-embed Architecture

The authors solve this by decoupling Data Sourcing from Model Inference through a four-stage parallel pipeline.

1. Specification & Provider Layer

The user defines a SpatialSpec (BBox or PointBuffer) and a TemporalSpec. The Provider Layer then talks to backends like GEE to fetch, resample, and composite (median/mosaic) the raw satellite tensors. This ensures that the model always gets the exact data format it expects, regardless of the user's raw data source.

2. Embedder Layer

This is the core "Wrapper." Whether a model is On-the-fly (runs local inference) or Precomputed (fetches existing vectors from a database), the get_embedding API remains identical. It handles the internal band mapping and normalization automatically.

3. High-Performance Orchestration

To handle large-scale datasets, the library implements a Prefetch-Inference-Export cycle. It deduplicates data requests, uses thread pools for I/O, and reuses model instances in memory to avoid the overhead of constant weight reloading.

Experiments: Maize Yield Mapping

To prove its utility, the authors conducted a "real-world" test: predicting maize yield in Illinois.

Using rs-embed, they quickly extracted embeddings from multiple models (AgriFM, SatMAE, etc.) and fed them into a Random Forest regressor. AgriFM emerged as the winner in terms of $R^2$, but the real victory was the ease of the experiment—switching between 16 different foundation models required only a parameter change in the code, rather than weeks of refactoring.

Deep Insight: Beyond Just a Wrapper

What makes rs-embed significant is its Policy Enforcement. It pre-validates spatial and temporal settings before hitting expensive cloud APIs. It also introduces OutputSpec.grid(), which preserves spatial context—critical for tasks like segmentation where "pooled" vectors lose too much information.

Visualizing how different models "see" the same river and land structures via the grid output mode.

Conclusion & Future Outlook

rs-embed is a crucial step toward an "Open Geospatial" ecosystem. By standardizing the "Embedding Layer," it allows the community to focus on downstream science rather than data plumbing.

Limitations: Currently, it is heavily reliant on GEE for backends. Future versions expanding to more "local-first" data or private satellite constellations would make it even more robust for industry use cases.

Takeaway: If you are working on Remote Sensing downstream tasks, rs-embed is likely to become the "Scikit-Learn" of RSFM embeddings.