diff --git a/docs/en/architecture/architecture.mdx b/docs/en/architecture/architecture.mdx index 338ba4b..6c1631c 100644 --- a/docs/en/architecture/architecture.mdx +++ b/docs/en/architecture/architecture.mdx @@ -7,67 +7,19 @@ sourceSHA: a693536535f48f4ec77b829836a78a804dd12d6a552db81460fa4666b0170dad ![](../assets/log.png) -## Overall Architecture Description - -The logging system consists of the following core functional modules: - -1. Log Collection - - Provided based on the open-source component filebeat - - Log collection: Supports the collection of standard output logs, file logs, Kubernetes events, and audits. - -2. Log Storage - - Two different log storage solutions are provided based on the open-source components Clickhouse and ElasticSearch. - - Log Storage: Supports long-term storage of log files. - - Log Storage Time Management: Supports management of log storage duration at the project level. - -3. Log Visualization - - Provides convenient and reliable log querying, log exporting, and log analysis capabilities. - ## Log Collection -### Component Installation Method - -nevermore is installed as a daemonset in the cpaas-system namespace of each cluster. This component consists of 4 containers: - -| Name | Function | -| -------------------- | ------------------------------ | -| audit | Collects audit data | -| event | Collects event data | -| log | Collects log data (including standard output and file logs) | -| node-problem-detector | Collects abnormal information on nodes | - -### Data Collection Process - -After nevermore collects audit/event/log information, it sends the data to the log storage cluster, undergoing authentication by Razor before being ultimately stored in ElasticSearch or ClickHouse. - -## Log Consumption and Storage - -### Razor - -Razor is responsible for authentication and receiving and forwarding log messages. - -- After Razor receives requests sent by nevermore from various workload clusters, it first authenticates using the Token in the request. If authentication fails, the request is denied. -- If the installed log storage component is ElasticSearch, it writes the corresponding logs into the Kafka cluster. -- If the installed log storage component is Clickhouse, it passes the corresponding logs to Vector, which are ultimately written into Clickhouse. - -### Lanaya - -Lanaya is responsible for consuming and forwarding log data in the ElasticSearch log storage link. - -- Lanaya subscribes to topics in Kafka. After receiving the messages from the subscription, it decompresses the messages. -- After decompression, it preprocesses the messages by adding necessary fields, transforming fields, and splitting data. -- Finally, it stores the messages in the corresponding index of ElasticSearch based on the message's time and type. - -### Vector - -Vector is responsible for processing and forwarding log data in the Clickhouse log storage link, ultimately storing the logs in the corresponding table in Clickhouse. +- Marketplace - Cluster Plugins - Alauda Container Platform Log Collector enables the log collection component. The log types include: Audit, Events, Kubernetes logs, Product logs, System logs, and Application logs. +- The Nevermore pod runs multiple containers (audit, events, log, node-problem-detector) responsible for collecting different types of logs. By entering the corresponding container, executing `ps -ef` allows viewing the loaded configuration files. +- Kubernetes-event-exporter is responsible for monitoring events generated by kube-apiserver and writing them to the local disk file `/cpaas/event/events.log`, which is then collected by Nevermore. -## Log Visualization +## Log Storage -1. Users can query the audit/event/log query URLs from the product UI interface for display: +- Nevermore collects logs from `/cpaas/{log,audit,event}` and sends them to the ALB of the corresponding log storage cluster. (1) If logs are stored within the same cluster, data is sent via the razor service, using the path `http://razor:8080/v4/callback`; (2) If logs from a business cluster are stored in the global cluster, they are sent to razor using the platform access address, with the path `https:///v4/callback`; (3) If logs are stored in another business cluster, they are sent to razor through that cluster's ALB on port 11780, using the path `http://alb:11780/v4/callback`. (The destination address for Nevermore to send logs to razor can be queried using `kubectl get cm -n cpaas-system nevermore-config -oyaml`). +- If Elasticsearch is deployed in the log storage cluster, razor writes the received logs to Kafka. The `ALAUDA_AUDIT_TOPIC` stores audit logs, `ALAUDA_EVENT_TOPIC` stores event logs, and `ALAUDA_LOG_TOPIC` stores Kubernetes logs, product logs, system logs, and application logs. Lanaya consumes logs from Kafka and writes them to Elasticsearch. +- If ClickHouse is deployed in the log storage cluster, dependencies on Kafka, Zookeeper, Lanaya, and Razor are removed. Vector is introduced as a log aggregator to write directly to ClickHouse, reducing the load on ClickHouse. -- Log Query /platform/logging.alauda.io/v1 -- Event Query /platform/events.alauda.io/v1 -- Audit Query /platform/audits.alauda.io/v1 +## Log Query -2. The requests are processed by the advanced API component Courier, which queries the log data from the log storage clusters ElasticSearch or Clickhouse and returns it to the page. +- The product UI interface queries logs via the following URLs (Log Query: `/platform/logging.alauda.io/v2`, Event Query: `/platform/events.alauda.io/v1`, Audit Query: `/platform/audits.alauda.io/v1`). +- The ALB invokes the `courier-api` component in the global cluster (if the log storage plugin is deployed in a business cluster, the `courier-api` component runs in the global cluster) to query logs from the log storage cluster's Elasticsearch or ClickHouse and returns the results to the front end. (The courier-api's YAML does not contain the Elasticsearch request address; this is recorded in the `feature` resource of `log-center` and is loaded by retrieving the feature after startup). diff --git a/docs/en/assets/log.png b/docs/en/assets/log.png index 9836eb3..96931c7 100644 Binary files a/docs/en/assets/log.png and b/docs/en/assets/log.png differ