|
1 | | -# indy-metadata-service |
| 1 | +# Indy Metadata Service |
2 | 2 |
|
| 3 | +A Quarkus-based microservice for managing metadata operations in the Indy artifact repository system. This service handles metadata file management, special path operations, and provides REST APIs for metadata-related operations. |
3 | 4 |
|
4 | | -## Prerequisite |
5 | | -1. jdk11 |
6 | | -2. mvn 3.6.2+ |
| 5 | +## Overview |
7 | 6 |
|
8 | | -## Configure services |
| 7 | +The Indy Metadata Service is part of the CommonJava Indy ecosystem and provides: |
9 | 8 |
|
10 | | -To make it run, Infinispan and Kafka need to be configured. Please see the example in `application.yaml`. |
| 9 | +- **Metadata Management**: Automatic cleanup and management of metadata files (e.g., `maven-metadata.xml`, `package.json`) |
| 10 | +- **Group-Aware Operations**: Metadata updates propagate through repository group hierarchies |
| 11 | +- **Event-Driven Architecture**: Reactive processing of file events via Kafka |
| 12 | +- **RESTful APIs**: HTTP endpoints for metadata operations |
| 13 | +- **Multi-Package Support**: Support for Maven, NPM, and other package types |
11 | 14 |
|
12 | | -## Try it |
| 15 | +## Architecture |
13 | 16 |
|
14 | | -### 1. Start the cache server and the message broker |
| 17 | +### Core Components |
15 | 18 |
|
16 | | -It needs a remote cache server(Infinispan) as well as the message broker(Kafka) for event handler. To ease the setup, we have provided a `docker-compose.yml` file which start the containers and bind the network ports. |
| 19 | +- **MetadataController**: Main business logic controller |
| 20 | +- **MetadataHandler**: Handles metadata deletion and cleanup operations |
| 21 | +- **FileEventConsumer**: Processes file events from Kafka for automatic metadata cleanup |
| 22 | +- **MetadataResource**: JAX-RS REST endpoints |
17 | 23 |
|
18 | | -``` |
19 | | -docker-compose up |
20 | | -``` |
| 24 | +## API Endpoints |
21 | 25 |
|
22 | | -### 2. Start gateway in debug mode |
23 | | -``` |
24 | | -$ mvn quarkus:dev |
| 26 | +### Metadata Operations |
| 27 | + |
| 28 | +```http |
| 29 | +GET /api/metadata/{packageType}/{type}/{name}/{path} |
| 30 | +DELETE /api/metadata/{packageType}/{type}/{name}/{path} |
| 31 | +GET /api/metadata/{packageType}/{type}/{name} |
25 | 32 | ``` |
26 | 33 |
|
27 | | -### 3. Verify the installation |
| 34 | +**Parameters:** |
| 35 | +- `packageType`: Package type (maven, npm, etc.) |
| 36 | +- `type`: Store type (hosted, group, remote) |
| 37 | +- `name`: Store name |
| 38 | +- `path`: File path within the store |
| 39 | + |
| 40 | +## Configuration |
| 41 | + |
| 42 | +### Application Properties |
| 43 | + |
| 44 | +Key configuration options in `application.yaml`: |
| 45 | + |
| 46 | +```yaml |
| 47 | +quarkus: |
| 48 | + http: |
| 49 | + port: 8081 |
| 50 | + read-timeout: 30m |
| 51 | + limits: |
| 52 | + max-body-size: 500M |
| 53 | + |
| 54 | +kafka: |
| 55 | + bootstrap: |
| 56 | + servers: "localhost:9092" |
28 | 57 |
|
| 58 | +storage-service-api/mp-rest/url: http://localhost |
| 59 | +repo-service-api/mp-rest/url: http://localhost |
29 | 60 | ``` |
30 | | -http://localhost:8080/swagger-ui/ |
| 61 | +
|
| 62 | +### Environment Variables |
| 63 | +
|
| 64 | +- `KAFKA_BOOTSTRAP_SERVERS`: Kafka broker addresses |
| 65 | +- `STORAGE_SERVICE_API_MP_REST_URL`: Storage service URL |
| 66 | +- `REPO_SERVICE_API_MP_REST_URL`: Repository service URL |
| 67 | + |
| 68 | +## Development |
| 69 | + |
| 70 | +### Prerequisites |
| 71 | + |
| 72 | +- Java 11+ |
| 73 | +- Maven 3.6+ |
| 74 | +- Docker & Docker Compose (for local development) |
| 75 | + |
| 76 | +### Building |
| 77 | + |
| 78 | +```bash |
| 79 | +# Compile and run tests |
| 80 | +mvn clean compile test |
| 81 | +
|
| 82 | +# Package the application |
| 83 | +mvn package |
| 84 | +
|
| 85 | +# Run in development mode |
| 86 | +mvn quarkus:dev |
31 | 87 | ``` |
32 | 88 |
|
33 | | -### 4. Check the entries on cache server |
| 89 | +### Docker Development |
34 | 90 |
|
| 91 | +```bash |
| 92 | +# Build the Docker image |
| 93 | +docker build -f src/main/image/Dockerfile.jvm -t indy-metadata-service . |
| 94 | +
|
| 95 | +# Run with Docker Compose (includes Kafka) |
| 96 | +docker-compose up |
35 | 97 | ``` |
36 | | -http://localhost:11222/console/ |
| 98 | + |
| 99 | +### Local Development Setup |
| 100 | + |
| 101 | +1. Start Kafka and Zookeeper: |
| 102 | + ```bash |
| 103 | + docker-compose up zookeeper kafka |
| 104 | + ``` |
| 105 | + |
| 106 | +2. Configure external services (storage and repository services) |
| 107 | + |
| 108 | +3. Run the application: |
| 109 | + ```bash |
| 110 | + mvn quarkus:dev |
| 111 | + ``` |
| 112 | + |
| 113 | +## Event Processing |
| 114 | + |
| 115 | +The service processes two types of Kafka events and maintains consistency across repository group hierarchies: |
| 116 | + |
| 117 | +> **Important**: When a metadata file is updated in any repository, the service automatically propagates these changes up through the entire group hierarchy. For example, if a POM file is uploaded to a hosted repository, the metadata will be regenerated not only for that repository but also for all groups that contain it, all the way up to the root group. |
| 118 | + |
| 119 | +### File Events (`file-event` topic) |
| 120 | +- Triggers metadata cleanup when POM files or package tarballs are uploaded/deleted |
| 121 | +- Automatically clears corresponding metadata files in hosted repositories |
| 122 | +- **Group Propagation**: Updates cascade through all repository groups containing the affected repository |
| 123 | + |
| 124 | +### Promote Events (`promote-complete` topic) |
| 125 | +- Handles promotion completion events |
| 126 | +- Manages metadata updates across repository groups |
| 127 | + |
| 128 | + |
| 129 | +## Monitoring & Observability |
| 130 | + |
| 131 | +- **OpenTelemetry**: Distributed tracing support |
| 132 | +- **Structured Logging**: JSON-formatted logs with configurable levels |
| 133 | +- **Health Checks**: Built-in Quarkus health endpoints |
| 134 | +- **Metrics**: Application metrics via Micrometer |
| 135 | + |
| 136 | +## Testing |
| 137 | + |
| 138 | +```bash |
| 139 | +# Run unit tests |
| 140 | +mvn test |
| 141 | +
|
| 142 | +# Run integration tests |
| 143 | +mvn verify |
| 144 | +
|
| 145 | +# Run specific test class |
| 146 | +mvn test -Dtest=MetadataResourceTest |
37 | 147 | ``` |
38 | 148 |
|
| 149 | +## Dependencies |
| 150 | + |
| 151 | +- **Quarkus**: Reactive framework and runtime |
| 152 | +- **RESTEasy**: JAX-RS implementation |
| 153 | +- **SmallRye Reactive Messaging**: Kafka integration |
| 154 | +- **OpenTelemetry**: Observability |
| 155 | +- **Maven Repository Metadata**: Maven-specific metadata handling |
39 | 156 |
|
| 157 | +## License |
40 | 158 |
|
| 159 | +Licensed under the Apache License, Version 2.0. See the [LICENSE](LICENSE) file for details. |
0 commit comments