Parallelize / cache `collections/` logic?

[C-Loftus]

Is your feature request related to a problem? Please describe.

Currently when a user goes to collections/ pygeoapi iterates through all the collections and iterates through them, instantiating the class, one by one serially. This becomes a bottleneck in servers with many collections.

Here is the function:

pygeoapi/pygeoapi/api/__init__.py

Lines 930 to 933 in fa3d129

	@jsonldify
	def describe_collections(api: API, request: APIRequest,
	dataset=None) -> Tuple[dict, int, str]:

And here is one of the loops:

pygeoapi/pygeoapi/api/__init__.py

Lines 966 to 968 in fa3d129

	for k, v in collections_dict.items():
	if v.get('visibility', 'default') == 'hidden':
	LOGGER.debug(f'Skipping hidden layer: {k}')

This is generally fine for local / offline data but if you need to proxy a remote ESRI endpoint or have another provider (i.e. xarray with S3) that makes a network request to get the metadata like fields, it can significantly slow down the entire collections/ response, particularly if there are multiple of these types of collections.

Describe the solution you'd like

You could solve this in two ways:

1. Parallelism/Concurrency:

• Ideally this would be a good fit for async. However, since there are some challenges with async Starlette server fails with top level async-await #1964 (and I don't think async would work with flask regardless) it seems like a threadpool might be the easiest option in the standard library. Threadpool would add some additional overhead at the start but decrease latency overall.

2. Caching

• You could wrap the collections endpoint with functools.cache and it would become nearly instant without needing to worry about overhead from parallelism. However the downside for this would be dealing with cache invalidation. It could be an option to enable caching collections in the pygeoapi config (i.e. not have it enabled by default) and then have an option to supply a specific header to break the cache if needed.

Describe alternatives you've considered

You can use other green thread libraries like https://www.gevent.org/ but this would introduce another dependency. This would be lighter weight than threading.

Additional context

To my understanding there is no way to limit what data gets returned in collections. If this was possible and one could skip fetching the data that one may need to use network calls for (i.e. fields for EDR) this would be less of an issue. However, I don't believe that is the case.

[C-Loftus]

And to clarify this would be something I would be happy to contribute a PR for

[dghouse-ca]

This would be something my organization would be very interested in. From my naive perspective, would this just reduce to making caching and connection pooling available in your data provider? My own use-cases involve RDBMS for the moment.

[C-Loftus]

This would be something my organization would be very interested in. From my naive perspective, would this just reduce to making caching and connection pooling available in your data provider? My own use-cases involve RDBMS for the moment.

This issue particularly is just about speeding up the collections/ response. I think there are other opportunities for caching and pooling in provider-specific code but for the sake of simplicity this issue just focuses on the specific function which generates the response for collections/. collections/ is necessary to drive a lot of UIs which summarize all the data in the endpoint and having it load each provider serially each time slows things down a lot.

[JohannesSchnell]

Hello,
regarding this matter, we were wondering why the collections/ endpoint needs to read from every data source on each request. Wouldn’t it be sufficient to retrieve data only when needed—e.g., after selecting a specific Collection, or when requesting its fields through the schema link?
In that scenario, we would only need to read from a single resource rather than all of them.

Additionally, if one collection is faulty (e.g., its data source is unreachable or an error occurs while reading), it prevents the entire collections/ endpoint from rendering. We find this behavior counter intuitive.

We would like to understand whether a specific design pattern or architectural reason motivated this workflow. If not, we would like to propose moving the data-source reading logic to a later stage in order to improve scalability. To give you an estimate we have around 60 ZARR Collections and another 60 SQL based Vector Collections, all in a cloud/S3 setup, and it takes about 50 seconds for the collections/ endpoint to load

We are also happy to contribute to this effort.

[C-Loftus]

regarding this matter, we were wondering why the collections/ endpoint needs to read from every data source on each request. Wouldn’t it be sufficient to retrieve data only when needed—e.g., after selecting a specific Collection, or when requesting its fields through the schema link?

@JohannesSchnell , there are probably other reasons, particularly for purely OGC API Features collections, but I know that for EDR collections, to generate the data_queries and parameter_names fields in the JSON response (i.e. with the xarray provider I presume you are using), pygeoapi instantiates the provider and calls the provider and the get_fields method each time. This is another thing that could be tweaked for better performance (i.e. reading and storing the fields once at startup and then just reusing them for subsequent calls).

i.e. get_fields returns the parameters as in this image

[C-Loftus]

@tomkralidis just curious, could I open a PR on this?

I know from #2098 you don't want to change internals which I respect so my plan is to submit a PR without changing any existing function definitions or output behavior of the internal API. My thought would be:

1. I will use a threadpool in describe_collections to generate the collection metadata in parallel
   • This can default to min(4, len(collections) or whatever you prefer for threads
   • I am happy to keep this disabled by default and add a pygeoapi-config.yml setting to enable if preferred
2. All of the logic for describe_collections will remain the same and I will add any necessary tests to ensure this
3. I will avoid any caching or behavior that requires invalidation since I presume that is out of scope at the moment

This will greatly reduce latency and prevent /collections from being a blocker to scaling a pygeoapi instance which I think is useful to have before any 1.0 release.

---

I have a test already running for this in my fork which has significant speedup but I would prefer to keep this in sync with the project.

> time curl -H "Accept: text/html" "$PYGEOAPI_MASTER/collections" 

________________________________________________________
Executed in   14.32 secs      fish           external
   usr time   16.23 millis  147.00 micros   16.08 millis
   sys time   12.55 millis  936.00 micros   11.61 millis

> time curl -H "Accept: text/html" "$FORK/collections"

________________________________________________________
Executed in    1.25 secs      fish           external
   usr time    3.15 millis  155.00 micros    3.00 millis
   sys time    4.67 millis  879.00 micros    3.79 millis

[C-Loftus]

As another comment, long async tasks without await yields block the async threadloop. Because of the fact that collections is ran as async without internal async support, it will consistently block the threadloop and eliminate any benefit from async in the rest of the starlette application.

pygeoapi/pygeoapi/starlette_app.py

Line 537 in 46eaaf5

async def collections(request: Request,

[webb-ben]

It is fairly easy to patch the load_plugin with a wrapper like: https://github.com/internetofwater/pygeoapi/blob/bee3893cf5fcd748357c819aa0afcd27015b8e09/pygeoapi/plugin.py#L110-L130

@doublebyte1 @tomkralidis not sure this if you think this type of behavior is suited for core

[tomkralidis]

This behaviour is suited for downstream applications, not core.