Writing a dataset

Writing a dataset #

eoio.write() writes an xarray Dataset or DataTree to disk. It is the counterpart to eoio.read().

Basic usage #

import eoio

ds = eoio.read("/path/to/product")
eoio.write(ds, "output.nc", writer="netcdf")

write returns the path it wrote, as a string.

Warning

An existing file or store at the path is overwritten. The one exception is appending to a Zarr store, below.

The output format is not guessed from the file extension. Reading detects the format because it is a property of the file being opened; writing is a choice, and stating it avoids silently producing something other than what you meant. It also does not generalise — a directory-style product such as .SAFE has no extension to inspect.

Write parameters #

write_params controls how the data are written. Which are accepted depends on the writer — inspect them with eoio.writer_options():

>>> eoio.writer_options("zarr")["write_params"]["append_dim"]
'dimension name to append along, or None (not supported for a DataTree)'

Every writer accepts these:

Parameter	Default	Description
`encode_attrs`	`True`	Encode attributes the format cannot store natively — nested dictionaries, datetimes, geometries — so they survive the round trip
`on_nonconformance`	`"warn"`	What to do when the dataset does not conform to CF-1.8 / ACDD / the controlled vocabulary: `"warn"`, `"error"` or `"ignore"`
`update_history`	`True`	Record the write in `history` and `eoio:processing_steps`
`create_parents`	`True`	Create missing parent directories of the output path

Format-specific parameters include complevel and engine for netCDF, and append_dim, chunks and zarr_format for Zarr.

Attributes #

Readers put rich Python objects in attrs — nested metadata dictionaries, datetime objects, geometries — none of which netCDF or Zarr can store. write encodes them automatically, so a dataset that reads will write.

Reading a netCDF file back with eoio.read decodes them for you:

ds = eoio.read("output.nc", read_params={"metadata_level": "original"})
ds.attrs["product_metadata"]["datatake"]["id"]

metadata_level="original" returns the file’s attributes as they were written. At the default level, "all", they are kept under ds.attrs["product_metadata"] instead, alongside eoio’s standard attributes for the read.

Anywhere else - a file opened with xarray directly, or a Zarr store - decode after reading:

from eoio.serialisation import decode_attrs

ds = decode_attrs(xr.open_dataset("output.nc"))

Nested dictionaries are written as dot-separated attribute names (product_metadata.datatake.id), so they stay readable in ncdump and any generic netCDF viewer.

Building up a time series #

The Zarr writer can append along a dimension, which is what you want when observations arrive one at a time and the stack grows. The same call handles the first write and every later one — the first creates the store, the rest extend it:

for scene in scenes:
    eoio.write(
        scene,
        "stack.zarr",
        writer="zarr",
        write_params={"append_dim": "time"},
    )

Only the new data is written; existing chunks are left alone. Memory stays flat however large the stack gets.

The appended data must have the same variables and the same shape in every dimension other than append_dim, or the write is refused. Appending a coordinate value the store already holds, as when re-running a partly failed series, is also refused by default; pass on_overlap="skip" to write only what is new. Values are concatenated in the order written, so sort beforehand if order matters.

Note

Appending is not supported for a DataTree — xarray raises NotImplementedError.

Compression #

netCDF output is compressed by default (zlib, level 4); adjust with complevel or turn it off with compress=False. Zarr compresses natively, so there is nothing extra to configure — pass encoding to change it.

Variables under 16 KB are left uncompressed for netCDF, since the per-variable overhead would make them larger. Set compress_min_bytes=0 to compress everything.

Processing before writing #

processors applies a processing pipeline before writing, in the same form eoio.read() takes, so read-transform-write is two calls:

ds = eoio.read("/path/to/product")
eoio.write(ds, "output.nc", writer="netcdf", processors={"stack": {...}})

Name	Writes
`"netcdf"`	A netCDF4 file. A `DataTree` becomes netCDF4 groups
`"zarr"`	A Zarr store. A `DataTree` becomes Zarr groups. Needs the optional zarr package: `pip install eoio[zarr]`

Writing a dataset

Contents