rework catalog/collections/archives customizations

[ungarj]

fixes #9

• core

  • driver configuration: replace archive with source
  • Sentinel-2 driver configuration:
    • removed archive, cat_baseurl and max_cloud_cover parameters
    • Sentinel2Source: new configuration element to customize a source
    • EarthSearch sentinel-2-c1-l2a collection is now the default
  • removed archives, known_catalogs, platforms.sentinel2.archives modules
  • io.items.get_item_property(): added default kwarg which behaves similar to dict.get(); property now also accepts a tuple of strings which are checked in that order and the first matching property is being returned
  • io.path.asset_mpath(): asset now accepts a tuple of strings and the first matching asset path is being returned
  • platforms.sentinel2: added preconfigured_sources module which holds all custom mapper functions between collections and data/metadata archives
  • removed platforms.sentinel2.path_mappers module
  • removed geometry module
  • make time parameter optional
  • settings: add lazy_load_stac_items option

• packaging

  • added cql2 to dependencies

[Scartography]

• source_mappers.py in sentinel2 platform; the collections are immutable for now when set here, mapchete config wont get passed
• query cant be just string, it should be a List[str] to be accepted via pystac_client search kwargs :)
• ...

[Scartography]

The KNOWN_SOURCES and the ingestion into Source base class still feels like it has defined params and overall it feels confusing how it is used as there are now multiple configs/settings for sentinel-2 Driver alone. The whole Sentinel2DriverConfig and source now feels more separated from the baseclasses, which is good, however still feels like a "sentient" config, which does more than it should (idk)

[Scartography]

• assets and or eo_band configs should be validated, get, parsed and co. before reading products so that they are matched/parsed from collection or items before reading products as xarrays, numpy arrays, currently quite nested in the IO/read modules, functions.

[codecov]

Codecov Report

❌ Patch coverage is 89.34754% with 80 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mapchete_eo/source.py	77.41%	12 Missing and 2 partials ⚠️
...nel2/preconfigured_sources/metadata_xml_mappers.py	84.33%	13 Missing ⚠️
...ms/sentinel2/preconfigured_sources/item_mappers.py	87.09%	6 Missing and 2 partials ⚠️
mapchete_eo/platforms/sentinel2/source.py	87.87%	4 Missing and 4 partials ⚠️
...e_eo/platforms/sentinel2/metadata_parser/models.py	85.41%	3 Missing and 4 partials ⚠️
mapchete_eo/search/stac_static.py	77.41%	5 Missing and 2 partials ⚠️
mapchete_eo/io/items.py	81.48%	2 Missing and 3 partials ⚠️
mapchete_eo/search/utm_search.py	33.33%	4 Missing ⚠️
...apchete_eo/platforms/sentinel2/_mapper_registry.py	94.11%	1 Missing and 1 partial ⚠️
.../platforms/sentinel2/metadata_parser/s2metadata.py	87.50%	1 Missing and 1 partial ⚠️
... and 10 more

📢 Thoughts on this report? Let us know!

[Scartography]

Query as List[str] ? for pystac_client from the get go? If string will need a parser, ideally here as well

[Scartography]

When passing whole huge polygons to stac search as area the STAC API on the other side rejects it with InternalServerError; fix in my PR to this via just making bounds from area area=box(*area.bounds),

[Scartography]

With this thee mhub job workers have quite large RAM usage, also tasks dont seem to be passed to scheduler proprly (might be too large entities); this would indicate potencial materializing of some metadata objects.

Also investigating if the search returns proper number of products, seeing quite large numbers.

[ungarj]

With this thee mhub job workers have quite large RAM usage, also tasks dont seem to be passed to scheduler proprly (might be too large entities); this would indicate potencial materializing of some metadata objects.

Also investigating if the search returns proper number of products, seeing quite large numbers.

Please investigate, I'll do the same.

[Scartography]

Buffer also applies to footprint mask it feels which should not be the case (https://github.com/mapchete/mapchete-eo/blob/main/mapchete_eo/platforms/sentinel2/product.py#L556).

[ungarj]

Buffer also applies to footprint mask it feels which should not be the case (https://github.com/mapchete/mapchete-eo/blob/main/mapchete_eo/platforms/sentinel2/product.py#L556).

true. especially as we are already provide a mechanism to buffer the footprint separately.

ideally, we should buffer each individual mask before combining them but until we have found a more performant (i.e. raster based) implementation for buffer_array(), i opt against fixing this now.