Enriching matchup metadata

Enriching matchup metadata#

After a catalogue has been populated by find_and_catalogue(), each matchup STAC Item carries a small set of core properties (time of overpass, bounding box, sensor pair, …). The enrichment system lets you compute additional properties — area, time difference, solar elevation, land fraction, and anything you define yourself — and write them back to the item JSON so they persist in the catalogue and can be used to filter matchups.

Overview#

An enricher is any callable with the signature:

def my_enricher(matchup: Matchup) -> Dict[str, Any]: ...

It receives a Matchup and returns a plain dict of property names to values. enrich() iterates all matchup items, reconstructs the Matchup domain object for each, calls every enricher, and merges the resulting dicts back into item.properties. If the item has a self_href the updated JSON is written to disk immediately so the properties survive the process and can be ingested into the central catalogue.

Quick start#

from eomatch.mu_stac import MatchupCatalogue
from eomatch.enrich import enrich
from eomatch.enrich.time_diff import time_diff
from eomatch.enrich.geometric import geometric
from eomatch.enrich.cloud_cover import cloud_cover

In [1]: In [1]: catalogue = MatchupCatalogue.open("/data/my_catalogue")

In [2]: In [2]: enrich(catalogue, enrichers=[time_diff, geometric, cloud_cover])

In [3]: Out[2]: 42

enrich returns the number of matchup items it enriched.

Or from the command line:

eomatch-enrich \
    --catalogue /data/my_catalogue \
    --enricher eomatch.enrich.time_diff.time_diff \
    --enricher eomatch.enrich.geometric.geometric \
    --enricher eomatch.enrich.solar_elevation.solar_elevation \
    --enricher eomatch.enrich.land_fraction.land_fraction \
    --enricher eomatch.enrich.cloud_cover.cloud_cover

Add --overwrite to replace properties that were written in an earlier run. Add -v / --verbose for debug-level logging.

Built-in enrichers#

All built-in enrichers live as submodules of eomatch.enrich. solar_elevation and land_fraction require the optional enrich extra:

pip install 'eomatch[enrich]'

Module	Property added	Notes
`time_diff`	`time_diff_s`	Signed seconds: `products[1].start_time − products[0].start_time`.
`geometric`	`collocation_area_km2`, `collocation_centroid_lon`, `collocation_centroid_lat`	Area on the WGS-84 ellipsoid via `pyproj`; falls back to degree² if `pyproj` is absent.
`solar_elevation`	`solar_elevation_deg`	Solar elevation at the collocation centroid midpoint. Positive = daytime, negative = night-time. Requires `pysolar`.
`land_fraction`	`land_fraction`	Fraction of the collocation region over land (0.0 = ocean, 1.0 = land), computed from Natural Earth 110 m polygons. Requires `geopandas` and `geodatasets`.
`cloud_cover`	`cloud_cover`	Highest `cloud_fraction` among the matchup’s products (maximum, not average, so one heavily-clouded product isn’t masked by a clearer one). No property is written if no product has cloud cover data.
`insitu_time_diff`	`insitu_time_diff_before_s`, `insitu_time_diff_after_s`, `insitu_time_diff_nearest_s`	Gap in seconds to the nearest valid in-situ sample before/after the satellite crossover (`None` if there is none on that side). Reads the in-situ product’s real data, not just its declared time bounds. Satellite-to-in-situ matchups only.
`roi_cloud_cover`	`roi_cloud_cover`	Percentage of the collocation region flagged as cloud by the satellite product’s own classification mask (Sentinel-2 L1C/L2A; for Planet SuperDove, any pixel not flagged clear). Unlike `cloud_cover` this is specific to the region, not the whole scene. Reads the product.
`roi_valid_fraction`	`roi_valid_fraction`	Fraction (0–1) of the collocation region with valid, non-NaN data. Use it to drop matchups where the image is empty over the target. Reads one band (or the RGB group) of the product.

Enrichers that read the product#

roi_cloud_cover and roi_valid_fraction (and insitu_time_diff) need the product itself, so they download it on demand if it is not already local. Run download_products() first to control when that happens. To avoid downloading products that can never be useful, skip matchups using cheap properties that are already known:

min_collocation_area_km2 skips matchups whose collocation region is smaller than a threshold.
gate takes the same {property: {operator: value}} filter as get_events(), e.g. only look at matchups whose scene cloud cover is already low.

import functools
from eomatch.enrich.roi_valid_fraction import roi_valid_fraction

In [4]: In [1]: gate = {"cloud_cover": {"lt": 50}}

In [5]: In [2]: enricher = functools.partial(roi_valid_fraction, min_collocation_area_km2=0.1, gate=gate)

In [6]: In [3]: enrich(catalogue, enrichers=[enricher])

In [7]: Out[3]: 17

A skipped matchup simply gets no property. roi_cloud_cover also accepts a roi polygon to use (intersected with the collocation region) instead of the whole collocation region.

Keyword arguments can be bound from the command line by appending :key=value to the enricher path. Values are parsed as Python literals, and dict or list values are allowed; quote the whole argument:

eomatch-enrich \
    --catalogue /data/my_catalogue \
    --enricher 'eomatch.enrich.roi_valid_fraction.roi_valid_fraction:min_collocation_area_km2=0.1,gate={"cloud_cover": {"lt": 50}}'

Enriching a subset#

Enriching a large catalogue re-runs every enricher on every matchup, which is expensive for the enrichers above. Pass matchup_ids to enrich only some matchups, for example the ones a run has just added:

In [8]: In [1]: enrich(catalogue, enrichers=[cloud_cover], matchup_ids=["mu-a", "mu-b"])

In [9]: Out[1]: 2

Filtering by enriched properties#

Once properties have been written to the catalogue, pass a properties filter to get_events() to restrict results to matchups that meet your criteria:

# Keep only matchups with an overpass time difference under 15 minutes
events = catalogue.get_events(
    properties={"time_diff_s": {"lt": 900}}
)

# Daytime ocean matchups with a large overlap area
events = catalogue.get_events(
    properties={
        "solar_elevation_deg": {"gt": 0},
        "land_fraction": {"lt": 0.05},
        "collocation_area_km2": {"gt": 5000},
    }
)

Supported filter operators:

Operator	Meaning
`lt`	property < threshold
`lte`	property ≤ threshold
`gt`	property > threshold
`gte`	property ≥ threshold
`eq`	property == value (also the default when the condition is a plain value)
`ne`	property != value
`in`	property is a member of a list

Writing a custom enricher#

Any callable that accepts a Matchup and returns a dict qualifies as an enricher. To make it available on the CLI, place it in an importable module and pass the dotted path:

# my_package/enrichers.py
from typing import Any, Dict

# Platforms excluded from this analysis, e.g. a known calibration issue
# for a specific date range -- not something eomatch could know about.
_EXCLUDED_PLATFORMS = {"S2B"}

def custom_qc_flag(matchup) -> Dict[str, Any]:
    """Flag matchups involving a platform excluded from this analysis."""
    excluded = any(p.platform in _EXCLUDED_PLATFORMS for p in matchup.products)
    return {"qc_excluded_platform": excluded}

eomatch-enrich \
    --catalogue /data/my_catalogue \
    --enricher my_package.enrichers.custom_qc_flag

Enrichers that raise an exception are logged as warnings and skipped for that item — other enrichers in the same run are unaffected. Pass raise_error=True to enrich() to raise instead, which is useful when developing a new enricher.

The `overwrite` flag#

By default, enrich() skips keys that already exist in item.properties. Pass overwrite=True (Python) or --overwrite (CLI) to replace existing values:

# Re-run after updating the land_fraction enricher
enrich(catalogue, enrichers=[land_fraction], overwrite=True)

Ingesting enriched properties into the central catalogue#

Enriched properties are written to the local item.properties dict and persisted to the on-disk JSON immediately. The next time you run eomatch-ingest the updated Items — including all new properties — are pushed to the central pgSTAC catalogue, where they become queryable via the CQL2 filter extension:

# Local enrichment
eomatch-enrich \
    --catalogue /data/my_catalogue \
    --enricher eomatch.enrich.time_diff.time_diff \
    --enricher eomatch.enrich.land_fraction.land_fraction

# Push to central catalogue
eomatch-ingest --catalogue /data/my_catalogue