STAC catalogue

STAC catalogue#

EOMatch persists matchup events and matchup items to a STAC catalogue on disk. The catalogue is managed through MatchupCatalogue and follows this structure:

catalogue/
├── catalog.json
├── {collection-1}/
│   ├── collection.json
│   └── YYYY/MM/DD/{item-id}.json                            # one per source product
├── {collection-2}/
│   └── …
├── matchup-events-{col-1}-{platform-1}-vs-{col-2}-{platform-2}/
│   ├── collection.json
│   └── YYYY/MM/DD/{item-id}.json                            # one per MatchupEvent
└── {col-1}-{platform-1}-vs-{col-2}-{platform-2}/
    ├── collection.json
    └── YYYY/MM/DD/{item-id}.json                            # one per Matchup

Items are organised by date rather than by per-item subdirectory, so a busy collection stays flat and browsable. Collection IDs include the platform name for each sensor so that matchups between different satellite platforms within the same collection are stored separately (e.g. LANDSAT_C2L1-Landsat-8 vs LANDSAT_C2L1-Landsat-9). Pairs are sorted alphabetically by collection name for stability, giving IDs such as LANDSAT_C2L1-Landsat-9-vs-S2_MSI_L1C-S2A. Repeated (collection, platform) pairs are collapsed, so two observations from the same sensor give the same ID as one, and a matchup with a single distinct sensor ends in -vs-None (e.g. S2_MSI_L1C-S2A-vs-None).

Each matchup Item links back to its source products via derived_from links and to its parent event via a related link (matchup:role=event).

Running the find-and-catalogue pipeline#

The recommended way to populate the catalogue is via find_and_catalogue(), which runs the Sat2SatMUFinder and saves everything in one step:

from eomatch import EOMatchContext
from eomatch.find_and_catalogue import find_and_catalogue

ctx = EOMatchContext("my_config.yaml")
catalogue = find_and_catalogue(context=ctx, path="/data/my_catalogue")

Or from the command line:

eomatch-find --config my_config.yaml --path /data/my_catalogue

Add --verbose / -v for debug-level logging.

The catalogue path can also be set in the config file:

matchup_catalogue:
  path: /data/my_catalogue
  id: my-matchup-catalogue
  description: "Matchups for S2A vs S3A, June 2023"

Opening an existing catalogue#

from eomatch.mu_stac import MatchupCatalogue

cat = MatchupCatalogue.open("/data/my_catalogue/catalog.json")

Querying the catalogue#

Use get_events() to retrieve events and their matchups, with optional filtering:

import datetime as dt

events = cat.get_events(
    collections=["S2_MSI_L1C", "S3_EFR"],
    start_time=dt.datetime(2023, 6, 1),
    stop_time=dt.datetime(2023, 6, 30),
    bbox=[-10.0, 40.0, 30.0, 70.0],
)

for event in events:
    print(event)
    for mu in event.matchup_set:
        print("  ", mu)

To restrict to events whose source products have already been downloaded:

events = cat.get_events(products_downloaded=True)

To find matchups that still need an asset generated (e.g. a thumbnail), or conversely to restrict to ones that already have it:

events = cat.get_events(missing_assets=["thumbnail"])
events = cat.get_events(has_assets=["thumbnail"])

Handling corrupted or missing matchup files#

An item’s JSON file can end up missing or corrupted on disk — e.g. a script that errored partway through a write, or a file deleted/truncated outside eomatch entirely. get_events() and download_products() both tolerate this: an item that can’t be read is skipped with a logged warning rather than raising and aborting the whole call, so one bad matchup doesn’t take down everything else in the catalogue.

To actually clean a broken matchup out of the catalogue, use remove_matchup():

removed = cat.remove_matchup("LANDSAT_C2L1-Landsat-9-vs-S2_MSI_L1C-S2A_20230615T103245_ab12cd34")

This works even when the matchup’s own item file is exactly what’s missing or corrupted — the matchup is located by matching link references, not by reading the item itself. It also removes the item’s link from its owning collection and the corresponding link from its parent event, so a later get_events() call doesn’t keep warning about the same dangling reference. Source products the matchup referenced are left alone, since they can be shared by other matchups.

By default, a matchup found to be unreadable is only warned about, not removed automatically — set matchup_catalogue.auto_remove_corrupted: true in the config (see Configuration) to have get_events() call remove_matchup() on a broken matchup automatically, instead of leaving cleanup as a manual step.

Downloading products#

download_products() downloads all source products for a set of events and registers a "data" asset on each product Item so that the download state is tracked in the catalogue:

cat.download_products(event_set=events)

Products that are already present on disk are registered without being re-downloaded. The updated Item JSON is written to disk after each product.

Managing products from the command line#

The eomatch-download and eomatch-remove console scripts provide a convenient way to bulk-download or remove source products without writing any Python. Both commands accept the same filtering flags:

# Download all products for S2 vs Landsat matchups in June 2023
eomatch-download \
    --path /data/my_catalogue \
    --collections S2_MSI_L1C,LANDSAT_C2L1 \
    --start-time 2023-06-01 \
    --stop-time 2023-06-30

# Remove those products from disk (keeps catalogue metadata intact)
eomatch-remove \
    --path /data/my_catalogue \
    --collections S2_MSI_L1C,LANDSAT_C2L1 \
    --start-time 2023-06-01 \
    --stop-time 2023-06-30

# Remove asset references only, leave the files on disk
eomatch-remove --path /data/my_catalogue --keep-files

Pass --verbose / -v for debug-level logging. The catalogue path can be omitted if matchup_catalogue.path is set in your config file; pass --config to load a non-default config.

Available filter flags (shared by both commands):

Flag	Description
`--path PATH`	Catalogue root directory or `catalog.json` path.
`--collections C1,C2`	Comma-separated collection names (e.g. `S2_MSI_L1C,LANDSAT_C2L1`).
`--platforms P1,P2`	Comma-separated platform names.
`--start-time DATETIME`	ISO 8601 start-time; events ending before this are excluded.
`--stop-time DATETIME`	ISO 8601 stop-time; events starting after this are excluded.
`--bbox LON_MIN,LAT_MIN,LON_MAX,LAT_MAX`	Spatial bounding-box filter.

eomatch-remove additionally accepts --keep-files to remove the "data" asset from the catalogue without deleting the local files.

Managing products from Python#

The same functionality is available as Python functions in eomatch.manage_products:

import datetime as dt
from eomatch import EOMatchContext
from eomatch.manage_products import (
    download_catalogue_products,
    remove_catalogue_products,
)

ctx = EOMatchContext("my_config.yaml")

# Download source products for matching events
paths = download_catalogue_products(
    context=ctx,
    path="/data/my_catalogue",
    collections=["S2_MSI_L1C", "LANDSAT_C2L1"],
    start_time=dt.datetime(2023, 6, 1),
    stop_time=dt.datetime(2023, 6, 30),
)
print(f"Handled {len(paths)} product(s)")

# Remove downloaded products from disk (and deregister from catalogue)
n = remove_catalogue_products(
    context=ctx,
    path="/data/my_catalogue",
    collections=["S2_MSI_L1C", "LANDSAT_C2L1"],
    delete_files=True,  # set False to keep files, remove asset reference only
)
print(f"Removed {n} product asset(s)")

Both functions open the catalogue, apply the filters, and then act on every matching event. Products that appear in multiple matchups are processed only once. download_catalogue_products() registers a "data" STAC asset on each product Item after downloading so that the catalogue tracks which products are present on disk.

Attaching assets#

You can attach arbitrary STAC assets to any Item or Collection in the catalogue — useful for storing processing outputs alongside the raw products. Item-level methods (add_matchup_asset, add_event_asset, add_product_asset, add_asset_by_id) build the pystac.Asset internally from a file path, auto-detecting the media type from the file extension unless one is given:

# Attach to a single matchup Item
cat.add_matchup_asset(mu, asset_key="statistics", file_path="/data/results/mu_stats.csv")

# Attach to a matchup event Item, with explicit metadata
cat.add_event_asset(
    event,
    asset_key="thumbnail",
    file_path="/data/results/event_thumb.png",
    media_type="image/png",
    title="Event footprint thumbnail",
    roles=["thumbnail"],
)

Collection-level assets (a report covering every matchup of a sensor pair, say) are added by collection ID with add_collection_asset(), which also builds the asset from a file path:

In [1]: In [1]: collection_id = "LANDSAT_C2L1-Landsat-9-vs-S2_MSI_L1C-S2A"

In [2]: In [2]: cat.add_collection_asset(collection_id, "report", "/data/results/report.pdf", media_type="application/pdf")

In [3]: Out[2]: True

It returns False if no such collection exists. The add_matchup_collection_asset and add_event_collection_asset variants instead take a matchup or event and a pre-built pystac.Asset.

Corresponding remove_* methods (including remove_collection_asset()) are available at every level and will optionally delete the local file from disk.

To generate and register the same kind of asset (e.g. a thumbnail) across many matchups or events in one call, see generate_missing_assets() in Registering analysis results. To remove an asset from many matchups or events at once, see remove_assets() in the same page.