Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/gcs-integration.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: GCS BlobStore Integration

on:
push:
tags:
- "**"
pull_request:
paths:
- ".github/workflows/gcs-integration.yml"
- "pom.xml"
- "geowebcache/pom.xml"
- "geowebcache/core/**"
- "geowebcache/gcsblob/**"

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
fake-gcs-server:
name: Fake GCS Server container
runs-on: ubuntu-latest
strategy:
matrix:
java-version: [ 17, 21 ]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: ${{ matrix.java-version }}
cache: 'maven'

- name: Tests against Fake GCS Server TestContainers
run: |
mvn verify -f geowebcache/pom.xml -pl :gcs-blob -am \
-Ponline \
-DskipTests=true \
-DskipITs=false -B -ntp

- name: Remove SNAPSHOT jars from repository
run: |
find .m2/repository -name "*SNAPSHOT*" -type d | xargs rm -rf {}
59 changes: 54 additions & 5 deletions documentation/en/user/source/configuration/storage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ Storage
Cache
-----

Starting with version 1.8.0, there are two types of persistent storage mechanisms for tiles:
Starting with version 1.8.0, GeoWebCache supports multiple persistent storage mechanisms for tiles:

* File blob store: stores tiles in a directory structure consisting of various image files organized by layer and zoom level.
* S3 blob store: stores tiles in an `Amazon Simple Storage Service <http://aws.amazon.com/s3/>`_ bucket, as individual "objects" following a
* File blob store: stores tiles in a directory structure consisting of various image files organized by layer and zoom level.
* S3 blob store: stores tiles in an `Amazon Simple Storage Service <http://aws.amazon.com/s3/>`_ bucket, as individual "objects" following a
`TMS <http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification>`_-like key structure.
* Google Cloud Storage blob store: stores tiles in a GCS bucket using the same TMS-like structure as S3.
* Azure blob store, MBTiles blob store, Swift blob store: additional storage backends described below.

Zero or more blobstores can be configured in the configuration file to store tiles at different locations and on different storage back-ends.
One of the configured blobstores will be the **default** one. Meaning that it will be used to store the tiles of every layer whose configuration
Expand Down Expand Up @@ -287,7 +289,54 @@ GeoServer ``topp:states`` sample layer on a fictitious ``my-geowebcache-bucket``
zoom: 2
})
});



Google Cloud Storage (GCS) Blob Store
+++++++++++++++++++++++++++++++++++++

This blob store allows to configure a cache for layers on a Google Cloud Storage bucket with the same TMS-like key structure as S3:

[prefix]/<layer id>/<gridset id>/<format id>/<parameters hash | "default">/<z>/<x>/<y>.<extension>

Configuration example:

.. code-block:: xml

<GoogleCloudStorageBlobStore default="false">
<id>myGcsCache</id>
<enabled>true</enabled>
<bucket>my-gwc-bucket</bucket>
<prefix>test-cache</prefix>
<projectId>my-gcp-project</projectId>
<useDefaultCredentialsChain>true</useDefaultCredentialsChain>
</GoogleCloudStorageBlobStore>

Properties:

* **bucket**: Mandatory. The name of the GCS bucket where to store tiles.
* **prefix**: Optional. A prefix path to use as the "root folder" to store tiles at.
* **projectId**: Optional. The GCP project ID. Can be omitted if using service account credentials that already specify the project.
* **quotaProjectId**: Optional. Project to bill for quota when using requester-pays buckets.
* **endpointUrl**: Optional. Custom endpoint URL for use with GCS emulators or compatible services.
* **useDefaultCredentialsChain**: Optional. Set to ``true`` to use Application Default Credentials. This will look for credentials in the following order: environment variable GOOGLE_APPLICATION_CREDENTIALS pointing to a service account key file, GCE/GKE metadata service, or gcloud CLI credentials.
* **apiKey**: Optional. API key for authentication. If both apiKey and useDefaultCredentialsChain are provided, apiKey takes precedence.

**Note**: Like S3, all configuration properties support environment variable expansion using the ``${VARIABLE_NAME}`` syntax:

.. code-block:: xml

<bucket>${GCS_BUCKET}</bucket>
<projectId>${GCS_PROJECT_ID}</projectId>

Authentication options:

* **Application Default Credentials** (recommended): Set ``useDefaultCredentialsChain`` to ``true``. This works automatically on GCE/GKE and when GOOGLE_APPLICATION_CREDENTIALS points to a service account key.
* **API Key**: Set the ``apiKey`` property. Less secure, mainly for testing.
* **No auth**: For use with emulators only. Leave both auth options unset.

Implementation notes:

Delete operations run asynchronously in a background thread pool. When deleting tile ranges or layers, tiles are removed in batches using the GCS batch API for efficiency. The thread pool is sized based on available processors and shuts down gracefully on blob store destruction.

Microsoft Azure Blob Store
+++++++++++++++++++++++++++++++++++++++++++++
Expand Down Expand Up @@ -861,4 +910,4 @@ Additional Information:

* The package makes use of the open source multi-cloud toolkit `jclouds <https://jclouds.apache.org/>`_
* Jclouds documentation for `getting started with Openstack <https://jclouds.apache.org/guides/openstack/>`_
* Jclouds documentation for `OpenStack Keystone V3 Support <https://jclouds.apache.org/blog/2018/01/16/keystone-v3/>`_ used in config
* Jclouds documentation for `OpenStack Keystone V3 Support <https://jclouds.apache.org/blog/2018/01/16/keystone-v3/>`_ used in config
68 changes: 68 additions & 0 deletions geowebcache/gcsblob/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# GCS Blob Store

BlobStore implementation for Google Cloud Storage.

## Overview

This module provides a `BlobStore` that stores tiles in a GCS bucket. Tiles are organized using the standard TMS key structure: `<prefix>/<layer>/<gridset>/<format>/<parameters>/<z>/<x>/<y>.<ext>`

## Components

- `GoogleCloudStorageBlobStore` - Main BlobStore implementation
- `GoogleCloudStorageClient` - Low-level GCS operations, handles batch deletes via background thread pool
- `GoogleCloudStorageBlobStoreInfo` - XStream-serializable configuration
- `GoogleCloudStorageConfigProvider` - Spring integration for config management

## Building

```bash
mvn clean install
```

## Testing

Unit tests run against a fake GCS server via testcontainers:

```bash
mvn test
```

Integration tests use the same fake GCS server but run through the failsafe plugin:

```bash
mvn verify -Ponline
```

## Configuration

Supports environment variable expansion in all config parameters. Example:

```xml
<GoogleCloudStorageBlobStore>
<id>gcs-store</id>
<enabled>true</enabled>
<bucket>${GCS_BUCKET}</bucket>
<prefix>gwc</prefix>
<projectId>${GCS_PROJECT_ID}</projectId>
<useDefaultCredentialsChain>true</useDefaultCredentialsChain>
</GoogleCloudStorageBlobStore>
```

### Parameters

- `bucket` (required) - GCS bucket name
- `prefix` (optional) - Path prefix within the bucket. If not set, operates at bucket root
- `projectId` (optional) - GCP project ID
- `quotaProjectId` (optional) - Project to bill for quota (for requester-pays buckets)
- `endpointUrl` (optional) - Custom endpoint URL for emulators or GCS-compatible services
- `useDefaultCredentialsChain` (optional) - Set to `true` to use Application Default Credentials
- `apiKey` (optional) - API key for authentication

Authentication options (pick one):
- `useDefaultCredentialsChain` - Uses Application Default Credentials
- `apiKey` - API key for simple auth
- No auth specified - Anonymous access (useful for emulators)

## Notes

Delete operations run asynchronously on a background thread pool sized to available processors. The pool shuts down gracefully on blob store destruction with a 60s timeout.
90 changes: 90 additions & 0 deletions geowebcache/gcsblob/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.geowebcache</groupId>
<artifactId>geowebcache</artifactId>
<version>1.28-SNAPSHOT</version>
</parent>
<artifactId>gcs-blob</artifactId>
<name>Google Cloud Storage blob store</name>

<properties>
<google-cloud-storage.version>2.55.0</google-cloud-storage.version>
<testcontainers-fake-gcs-server.version>0.2.0</testcontainers-fake-gcs-server.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.aiven</groupId>
<artifactId>testcontainers-fake-gcs-server</artifactId>
<version>${testcontainers-fake-gcs-server.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.geowebcache</groupId>
<artifactId>gwc-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>${google-cloud-storage.version}</version>
</dependency>

<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.geowebcache</groupId>
<artifactId>gwc-core</artifactId>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.aiven</groupId>
<artifactId>testcontainers-fake-gcs-server</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<profiles>
<profile>
<id>online</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/**
* This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General
* Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
* later version.
*
* <p>This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
* warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
*
* <p>You should have received a copy of the GNU Lesser General Public License along with this program. If not, see
* <http://www.gnu.org/licenses/>.
*
* @author Gabriel Roldan, Camptocamp, Copyright 2025
*/
package org.geowebcache.storage.blobstore.gcs;

import org.geowebcache.mime.MimeType;

/**
* A record to hold the components of a tile's cache identity.
*
* <p>Note {@code layerName} is used to provide the layer name to callbacks (see
* {@link GoogleCloudStorageBlobStore#sendTileDeleted(TileLocation, long)}, may the layer id be different than the layer
* name like in GeoServer tile layers. For all other purposes, {@code layerId} uniquely identifies the layer (e.g. for
* layer cache prefixes)
*
* @param layerId The unique identifier of the layer.
* @param layerName The name of the layer.
* @param gridsetId The identifier of the gridset.
* @param format The MIME type of the tile.
* @param parametersId The identifier for the tile's parameter set, can be {@code null}.
* @since 1.28
*/
record CacheId(String layerId, String layerName, String gridsetId, MimeType format, String parametersId) {}
Loading