Skip to content

Commit 323e222

Browse files
silviasdfgsilviasdfg
authored
Merge pull request #82 from CitComAI-Hub/develop
Update main with latest develop changes
2 parents a153403 + 4cb5061 commit 323e222

3 files changed

Lines changed: 147 additions & 105 deletions

File tree

‎docs/services/img/level2_miais.png‎

118 KB
Loading

‎docs/services/waste_collection.md‎

Lines changed: 137 additions & 105 deletions
Original file line numberDiff line numberDiff line change
@@ -6,45 +6,30 @@
66
</figure>
77

88
## Introduction
9-
This guide shows how to deploy an AI-based service to optimize city waste collection using context information and the [Openroute](https://openrouteservice.org/) optimization service. As a limited example and possible starting point to build your own, it illustrates a **Minimal Interoperable AI Service (MIAIS)** that follows [MIMs embraced by the CitCom.ai project](https://oasc.gitbook.io/mims-2024).
10-
11-
!!! abstract "Scenario"
12-
Different sensors are deployed throughout the city to monitor the fill levels of waste containers. These sensors periodically collect data on the fill levels and send it to the TEF site data platform. The goal is to use this context information to create optimal truck waste collection routes. The solution will only consider the current waste container filling level, their location, available trucks, and start and end location.
13-
14-
Do not worry if your TEF site lacks some components from the described scenario. As a demo, we offer a docker instance with all the necessary components to quickly test the service, including some dummy data. Further sections explain how to adapt and deploy the service on your TEF site. To successfully deploy the service in a real environment, **you must meet the minimum requirements** below or at least be close to them.
9+
The **MAIS** is a conceptual and technical demonstration of how AI can be deployed across heterogeneous smart city platforms. Its purpose is to validate AI integration under [different levels of interoperability maturity](./../getting_started/interoperability.md/#interoperability-levels).
10+
Below, we outline how the MAIS, a waste collection route optimization service, can be deployed in environments adhering to **Level 1** and **Level 2** interoperability models.
1511

1612
### Minimal requirements
17-
Below are the minimum requirements for deploying the service on your TEF site. Please remember that you can always [test the service via the demo](#deploying-the-demo) if you still need to meet them.
18-
19-
#### NGSI-LD for Context Information (MIM1)
20-
```mermaid
21-
graph LR
22-
A["🏛️ TEF site data platform"] ---|NGSI-LD| B["🤖 AI service"];
23-
B --- C["📈 Dashboard"];
24-
C["📈 Dashboard"] --- D["👥 Users"];
25-
```
2613

27-
The above image shows the overall architecture: The AI service gets the necessary information from the TEF site data platform using the **NGSI-LD** specification compliant with MIM1. In the future, once [the data space connector](./../documentation/data_space_connectors/fiware/index.md) is deployed, the AI service will get the data through it.
14+
In Level 0 environments, the MAIS can not be deployed. However, as an example, it includes a Docker instance with all the necessary components to easily simulate a Level 1 environment, including some dummy data.
2815

29-
An intermediary adapter may be required in cases where the city data platform does not comply with the proposed NGSI-LD standard. If your current data platform uses the NGSIv2 specification, check the [documentation section](./../documentation/index.md) for more details about using [Lepus](../documentation/data_federation/ngsiv2_to_ld/lepus.md) or [connecting an NGSI-V2 broker with an NGSI-LD broker through subscriptions](../documentation/data_federation/ngsiv2_to_ld/iot_agent.md).
16+
At Level 1, interoperability is established by adopting **shared data standards and APIs**. The foundational components include **NGSI-LD (MIM1)** as the standard interface for context information exchange and **Smart Data Models (MIM2)** as a shared vocabulary for entities and attributes (e.g., [WasteContainer](https://github.com/smart-data-models/dataModel.WasteManagement/tree/master/WasteContainer), [Vehicle](https://github.com/smart-data-models/dataModel.Transportation/tree/master/Vehicle)).
3017

31-
The AI service will use the gathered information to offer an interactive service through a web dashboard. Once the user provides a desired config the AI service will produce an optimal solution.
18+
Level 2 enhances the Level 1 foundation by introducing **secure, governed, and federated data exchange mechanisms through a [data space connector](./../documentation/data_space_connectors/index.md)**.
3219

33-
#### SmartDataModels for entities (MIM2)
34-
The following entities are retrieved from the TEF site data platform and used in the service: [WasteContainer](https://github.com/smart-data-models/dataModel.WasteManagement/tree/master/WasteContainer) and [Vehicle](https://github.com/smart-data-models/dataModel.Transportation/tree/master/Vehicle). Feel free to click on them and explore their corresponding [Smart Data Model](https://smartdatamodels.org) specifications.
3520

36-
#### Openroute API key
21+
### Openroute API key
3722
[Openroute](https://openrouteservice.org/) offers a free vehicle routing optimization service based on the [Vroom](https://github.com/VROOM-Project/vroom) project. The MIAIS uses this service to provide and optimal solution. To access the service you will need a valid API key, so go over to [openrouteservice.org](https://openrouteservice.org) and get one; you will need it later. If you want to learn more, the API and parameters specification are explained [on the Vroom repository](https://github.com/VROOM-Project/vroom/blob/master/docs/API.md).
3823

39-
## Deploying the demo
40-
As mentioned, a demo example with some dummy data has been provided so partners can quickly test the service without worrying about the minimal requirements. Below, you will find step-by-step instructions on deploying the minimal interoperable service for waste collection using docker.
24+
## Running MIAIS locally
25+
As mentioned, a demo example with some dummy data has been provided so partners can quickly test the service without worrying about the interoperability level. Below, you will find step-by-step instructions on deploying the MIAIS:
4126

4227
1. Clone the repository and navigate to its root folder:
4328
```bash
4429
git clone https://github.com/CitComAI-Hub/waste-collection-demo.git && cd waste-collection-demo
4530
```
4631

47-
2. Init git submodules with the following command. This will clone and install a dead simple [ngsi-ld client library](https://github.com/CitComAI-Hub/ngsild-client) in `lib` folder. Please note that the library is for testing purposes only and lacks most functionality. However, it quickly allows you to implement your own methods to interact with the context broker.
32+
2. Init git submodules with the following command. This will clone and install a dead simple [ngsi-ld client library](https://github.com/CitComAI-Hub/ngsild-client) in `lib` folder. Please **note that the library is for testing purposes only and lacks most functionality**. However, it quickly allows you to implement your own methods to interact with the context broker.
4833
```bash
4934
git submodule init && git submodule update
5035
```
@@ -91,8 +76,129 @@ python3 upsert_fake_data.py
9176
flask --app server run
9277
```
9378

94-
## Deploying in your TEF
95-
If your TEF site meets all minimum requirements, you can go over deploying the MIAIS in your city. Start by changing your `.env` variables so they point to your real data platform. However, some changes, such as implementing an authentication method, may be required. The Minimal Interoperable AI Service is a starting point; therefore, feel free to explore and edit the project to start building it up on your own. Here are some tips that can help you adapt this example to your needs:
79+
## Running MIAIS in Level 1
80+
As described above, the MIAIS gets the necessary information from the TEF site data platform using NGSI-LD (MIM1) as the standard interface for context information exchange and Smart Data Models (MIM2) as a shared vocabulary for entities and attributes. An intermediary adapter may be required in cases where the city data platform does not comply with the latest NGSI specification. If your current data platform uses the NGSIv2 specification, check the [documentation section](./../documentation/index.md) for more details about using [Lepus](../documentation/data_federation/ngsiv2_to_ld/lepus.md) or [connecting an NGSI-V2 broker with an NGSI-LD broker through subscriptions](../documentation/data_federation/ngsiv2_to_ld/iot_agent.md).
81+
82+
<figure>
83+
```mermaid
84+
graph LR
85+
A["**TEF Context Broker**"] ---|NGSI-LD| B["🤖 **AI Service**"];
86+
```
87+
<figcaption>Figure 1: MIAIS integration in Level 1</figcaption>
88+
</figure>
89+
90+
91+
If your TEF site meets all minimum requirements, you can proceed with deploying the MIAIS in your city. Follow [previous instructions](#running-miais-locally) to get it up and running (except for the commands aimed at creating the context broker and inserting dummy data). Just remember that you must update the `.env` variables file so they point to the TEF data platform. If everything goes well, the example should work. Of course, some `WasteContainer` and `Vehicle` **entities must be available in your context broker**. Otherwise, use and adapt the `upsert_fake_data.py`script to your needs.
92+
93+
### Authentication
94+
When working with brokers in a production state, authentication is often required. The [`ngsild-client`](https://github.com/CitComAI-Hub/ngsild-client) library included in the example does not come with authentication support. However, it is quite straightforward to extend it to meet authentication requirements.
95+
96+
As an example, check the following code of the [Valencia TEF site](./../tef/south_connect/valencia.md) implementation, which integrates [authentication for their NGSIv2 context broker](https://github.com/CitComAI-Hub/ngsild-client/blob/master/Authv2.py).
97+
98+
```python
99+
from lib.ngsildclient.Auth import Authv2
100+
from lib.ngsildclient.Client import Client
101+
102+
103+
# Define service & subservice
104+
service = "tef_city"
105+
subservice = "/containers"
106+
107+
# Authenticate
108+
auth = Authv2()
109+
token = auth.get_auth_token_subservice(service, subservice)
110+
111+
# Ngsi-ld broker client
112+
client = Client()
113+
114+
# Fetch WasteContainer entities
115+
context = os.environ.get("WASTECONTAINERS_CONTEXT")
116+
containers = client.get_all_entities_by_type("WasteContainer", context, 100, 0, service, subservice, token).json()
117+
```
118+
119+
Environment variables in `.env` file:
120+
121+
```bash
122+
AUTH_PROTOCOL="https"
123+
ENDPOINT_KEYSTONE="auth.tef.com:15000"
124+
AUTH_USER="xxxxx"
125+
AUTH_PASSWORD="xxxxx"
126+
```
127+
128+
## Running MIAIS in Level 2
129+
At Level 2, deploying the MAIS requires integration with the **data space infrastructure**. The AI service no longer connects directly to raw NGSI-LD endpoints; instead, it accesses data via the standardized interface provided by the connector. This approximation guarantees that all data exchanges are **traceable, governed, and policy-compliant**.
130+
131+
<figure markdown>
132+
![Level 2 MIAIS](img/level2_miais.png){ loading=lazy }
133+
<figcaption>Figure 2: MIAIS integration at Level 2</figcaption>
134+
</figure>
135+
136+
To illustrate the case (see Figure 2), a simple data space structure is assumed, composed of a **trust anchor, a data space connector in the provider role, and another in the role of consumer**. The data provider and the consumer are **registered in the trust anchor**, establishing a trust relationship. Moreover, the **consumer is registered in the provider's Trusted Issuer List**, which allows the consumer to issue credentials to third parties with permissions to access the provider's data.
137+
138+
In addition, the AI service has a **wallet identity** consisting of a **Decentralized Identifier (DID)** and its associated private key, enabling the service to authenticate itself and sign verifiable credentials. Finally, the AI service must authenticate against the consumer's **Keycloak** identity server to get the corresponding access token.
139+
140+
This authentication process leverages the **OpenID for Verifiable Presentations (OID4VP)** protocol, allowing the AI service to obtain a verifiable credential from the consumer's identity provider and present it as cryptographic proof of authorization. As a wallet, the service generates a verifiable presentation signed with its private key and submits it to the identity server. After successful verification, the service receives an access token, enabling secure and trusted interaction with the provider. For more details about this process, refer to the [Data Space Connectors documentation](./../documentation/data_space_connectors/index.md).
141+
142+
143+
Below, you will find step-by-step instructions on deploying the MIAIS:
144+
145+
1. Clone the repository and navigate to its root folder:
146+
```bash
147+
git clone https://github.com/CitComAI-Hub/waste-collection-demo.git && cd waste-collection-demo
148+
```
149+
150+
2. Switch the branch to `mvds`:
151+
```bash
152+
git checkout mvds
153+
```
154+
155+
3. Prepare wallet-identity:
156+
```bash
157+
mkdir wallet-identity
158+
chmod o+rw wallet-identity
159+
docker run -v $(pwd)/wallet-identity:/cert quay.io/wi_stefan/did-helper:0.1.1
160+
# unsecure, only do that for testing
161+
sudo chmod -R o+rw wallet-identity/private-key.pem
162+
```
163+
164+
4. Create wallet identity secret
165+
```bash
166+
kubectl create secret generic wallet-identity-secret \
167+
--from-file=did.json=wallet-identity/did.json \
168+
--from-file=private-key.pem=wallet-identity/private-key.pem \
169+
-n consumer
170+
171+
# Check
172+
kubectl get secrets -n consumer
173+
kubectl describe secret wallet-identity-secret -n consumer
174+
```
175+
176+
5. Create ORS API key secret
177+
```bash
178+
kubectl create secret generic ors-api-key --from-literal=OPENROUTESERVICE_API_KEY=your_api_key -n consumer
179+
```
180+
181+
6. Create keycloak login secret
182+
```bash
183+
kubectl create secret generic keycloak-credentials \
184+
--from-literal=KEYCLOAK_USER='test-user' \
185+
--from-literal=KEYCLOAK_PASSWORD='test' \
186+
--from-literal=KEYCLOAK_CLIENT_ID='admin-cli' \
187+
-n consumer
188+
189+
# Check
190+
kubectl get secrets -n consumer
191+
kubectl describe secret keycloak-credentials -n consumer
192+
```
193+
194+
7. Deploy the MIAIS in your Kubernetes cluster:
195+
```bash
196+
kubectl apply -f flask-app.yaml -n consumer
197+
kubectl get pods -n consumer
198+
```
199+
200+
## Extending MIAIS
201+
The Minimal Interoperable AI Service is a starting point; therefore, feel free to explore and edit the project to start building it up on your own.
96202

97203
??? tip "Project structure"
98204
- `static/`: Frontend folder.
@@ -109,83 +215,9 @@ If your TEF site meets all minimum requirements, you can go over deploying the M
109215

110216
Moreover, maybe your situation needs to consider some time restrictions or priorities. Check out the [Openroute service API specification](https://github.com/VROOM-Project/vroom/blob/master/docs/API.md), which is powerful and includes many parameters to fit your optimization needs. To change/add additional query parameters, go over [`Optimization.py`](https://github.com/CitComAI-Hub/waste-collection-demo/blob/mvs-orionld/services/Optimization.py) and [`Optimizer.js`](https://github.com/CitComAI-Hub/waste-collection-demo/blob/mvs-orionld/static/modules/Optimizer.js) files.
111217

112-
??? tip "Level 1: Authentication"
113-
When working with brokers in a production state, authentication is often required. The [`ngsild-client`](https://github.com/CitComAI-Hub/ngsild-client) library included in the example does not come with authentication support. However, it is quite straightforward to extend it to meet authentication requirements. As an example, see the following code from the [Valencia](https://github.com/CitComAI-Hub/waste-collection-demo/tree/valencia) TEF site implementation, which implementes [authentication for their NGSIv2 setup](https://github.com/CitComAI-Hub/ngsild-client/blob/master/Authv2.py).
114-
115-
```python
116-
from lib.ngsildclient.Auth import Authv2
117-
from lib.ngsildclient.Client import Client
118-
119-
120-
# Define service & subservice
121-
service = "tef_city"
122-
subservice = "/containers"
123-
124-
# Authenticate
125-
auth = Authv2()
126-
token = auth.get_auth_token_subservice(service, subservice)
127-
128-
# Ngsi-ld broker client
129-
client = Client()
130-
131-
# Fetch WasteContainer entities
132-
context = os.environ.get("WASTECONTAINERS_CONTEXT")
133-
containers = client.get_all_entities_by_type("WasteContainer", context, 100, 0, service, subservice, token).json()
134-
```
135-
136-
Environment variables in `.env` file:
137-
138-
```bash
139-
AUTH_PROTOCOL="https"
140-
ENDPOINT_KEYSTONE="auth.tef.com:15000"
141-
AUTH_USER="xxxxx"
142-
AUTH_PASSWORD="xxxxx"
143-
```
144-
145-
??? tip "Level 2: Data Space Connector Authentication"
146-
If your TEF has a data space connector deployed, you can use the `fdsauth` python library to authenticate and retrieve the corresponding token. To install `fdsauth`, simply use `pip`:
147-
```bash
148-
pip install fdsauth
149-
```
150-
Next, a DID (Decentralized Identifier) and the corresponding key-material is required. You can create such via:
151-
```bash
152-
mkdir certs && cd certs
153-
docker run -v $(pwd):/cert quay.io/wi_stefan/did-helper:0.1.1
154-
```
155-
156-
Then, use the following example code to obtain your authentication token:
157-
```python
158-
from fdsauth import Consumer
159-
import requests
160-
161-
consumer = Consumer(
162-
keycloak_protocol="http",
163-
keycloak_endpoint="keycloak.consumer-a.local",
164-
keycloak_realm_path="realms/test-realm/protocol",
165-
keycloak_user_name="test-user",
166-
keycloak_user_password="test",
167-
apisix_protocol="http",
168-
apisix_endpoint="apisix-proxy.provider-a.local",
169-
certs_path="./certs",
170-
)
171-
172-
token = consumer.get_data_service_access_token()
173-
174-
try:
175-
# Attempt to access data using the obtained service token. Get entities of type EnergyReport.
176-
url = f"http://apisix-proxy.provider-a.local/ngsi-ld/v1/entities?type=EnergyReport"
177-
headers = {
178-
"Accept": "application/json",
179-
"Authorization": f"Bearer {token}",
180-
}
181-
response = requests.get(url, headers=headers)
182-
response.raise_for_status()
183-
print(response.json())
184-
except Exception as req_err:
185-
print(f"Request error occurred: {req_err}")
186-
```
187-
For more details, check out the [fdsauth repository](https://github.com/CitComAI-Hub/fdsauth/).
188-
189-
## Track and status of known problems
218+
## Issues
219+
Did you find any problem? [Create a new issue](https://github.com/CitComAI-Hub/waste-collection-demo/issues/new).
220+
221+
### Track and status of known problems
190222
- [X] Openroute optimization service has a maximum limit of 70 locations. This can be solved by [deploying your own Openroute instance](https://giscience.github.io/openrouteservice/getting-started).
191-
- [ ] Solutions offered by the AI service should also be provided following MIM1 and MIM2 recommendations. Eg: using Smart data models format like ([FleetVehicle](), [FleetVehicleOperation](), [Road]() and [RoadSegment]()).
223+
- [ ] Output from MIAIS service should be provided following MIM1 and MIM2 recommendations. Eg: using Smart data models format like ([FleetVehicle](), [FleetVehicleOperation](), [Road]() and [RoadSegment]()).

‎docs/stylesheets/extra.css‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,13 @@ footer .footer-tagline p {
5252
margin: 0;
5353
padding: 0;
5454
}
55+
56+
figure {
57+
max-width: 100% !important;
58+
width: auto !important;
59+
}
60+
61+
figure > svg {
62+
width: 100% !important;
63+
height: auto !important;
64+
}

0 commit comments

Comments
 (0)