Feature/postgresql support#369
Closed
fabioluciano wants to merge 299 commits into
Closed
Conversation
Signed-off-by: Ji Bin <matrixji@live.com> Signed-off-by: Ji Bin <matrixji@live.com>
Support setting log level in helm chart to enable debug log for debugging some issues
…rties for lake/ui/grafana (apache#52) There are resources/nodeSelector/affinity/tolerations config in values.yaml but no used in related yamls
- each merges to the main needs to promote the version if charts/ changed - chart version should align with the app version Signed-off-by: Ji Bin <matrixji@live.com>
…urce envs, and add smoke-test CI (apache#350) * Add custom MySQL DSN query params feature - Introduce `DB_CUSTOM_PARAMS` in `configmap.yaml`. - Enhance `DB_URL` in `deployments.yaml` to include custom params. - Define `mysql.extraParams` in `values.yaml` for configuration. Signed-off-by: kahirokunn <okinakahiro@gmail.com> * Enable optional DB URL auto-assembly for MySQL Signed-off-by: kahirokunn <okinakahiro@gmail.com> * fix: Pass MySQL env vars to Grafana for datasource setup - Introduced `connectionConfigmapName` to optionally override the default ConfigMap name for better configurability. Signed-off-by: kahirokunn <okinakahiro@gmail.com> * Add Helm smoke test workflow Signed-off-by: kahirokunn <okinakahiro@gmail.com> * Link Secret and ConfigMap names in values.yaml Signed-off-by: kahirokunn <okinakahiro@gmail.com> * Add customizable probes for MySQL containers Signed-off-by: kahirokunn <okinakahiro@gmail.com> * fix(mysql): Resolve environment-dependent startup failures by switching MySQL probes to TCP Signed-off-by: kahirokunn <okinakahiro@gmail.com> --------- Signed-off-by: kahirokunn <okinakahiro@gmail.com>
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
* feat: update v1.0.2-beta6 * v1.0.3 beta5
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
* feat: update v1.0.2-beta6 * v1.0.3 beta6
- Bump release Co-authored-by: Kostas Petrakis <kostas.petrakis@leanix.net>
Signed-off-by: drfaust92 <ilia.lazebnik@gmail.com>
…he#364) ### Summary This PR removes the commented `pgsql` configuration block from the Helm chart `values.yaml`. ### Motivation The chart currently exposes commented PostgreSQL configuration that appears to suggest PostgreSQL is supported, but: - The chart only implements MySQL configuration paths. - The `pgsql` values are not wired into templates. - Setting `option.database=pgsql` does not produce a working deployment. This creates confusion for users and leads to time spent attempting to configure an unsupported path. ### Change - Removes the commented `pgsql` configuration block from values.yaml. - Leaves MySQL as the clearly supported database option. ### Benefit - Reduces ambiguity around supported database backends. - Prevents users from attempting a non-functional configuration. - Improves chart clarity and maintainability. If/when PostgreSQL support becomes fully implemented, the configuration can be reintroduced with complete wiring and documentation. Happy to help reintroduce when full support exists.
Add separate StatefulSet templates for MySQL and PostgreSQL databases. Replace single statefulsets.yaml with type-specific implementations. - Add statefulset-mysql.yaml with MySQL-specific configuration - Add statefulset-postgresql.yaml with PostgreSQL-specific configuration - Remove generic statefulsets.yaml template
Update core configuration to support both database types. - Update Chart version to 2.0.0 - Refactor helpers with database-specific functions - Update configmap for multi-database support - Enhance secrets with database type handling - Update services for MySQL and PostgreSQL - Improve validation for database configuration - Update values.yaml with database.type option - Add .gitignore entry
Add external secrets, autoscaling, network security, and monitoring. - Add External Secrets Operator integration - Add Horizontal Pod Autoscaler for lake and ui - Add NetworkPolicy for database, lake, and ui components - Add ServiceMonitor for Prometheus integration - Add ServiceAccount with configurable annotations - Add pre-install validation job - Add Helm test suite for connectivity validation - Update deployments with HPA and service account support - Add Prometheus metrics annotations to ingresses
Add backup automation, pod disruption budgets, and schema validation. - Add backup CronJob for automated database backups - Add PodDisruptionBudget for high availability - Add values.schema.json for Helm values validation - Enhance NOTES.txt with deployment information and next steps
Add dedicated Job resource for PostgreSQL database migrations. - Add postgresql-migration-job.yaml with configurable migration settings - Support pre-install and pre-upgrade hooks - Include resource limits and security context - Add service label updates for lake and ui components
There was a problem hiding this comment.
Pull request overview
This PR upgrades the DevLake Helm chart to v2.0.0 and introduces a database.type selector with separate MySQL and PostgreSQL StatefulSet templates, alongside a broader refactor of values, helpers, validation, and operational add-ons.
Changes:
- Refactor database configuration from
mysql.*/option.databasetodatabase.*withdatabase.type: mysql|postgresql, adding dedicated MySQL/PostgreSQL StatefulSet templates. - Add new operational/security features (NetworkPolicies, External Secrets Operator integration, ServiceAccount template usage, ServiceMonitor, PDB/HPA scaffolding, backup CronJob, pre-install validation Job).
- Update chart/app versions and adjust defaults (e.g., Grafana disabled by default; new validation/schema enforcement).
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| charts/devlake/values.yaml | Major values refactor to database.*, plus new security/ops settings (NetworkPolicy, ExternalSecrets, backups, PDB/HPA, resource defaults). |
| charts/devlake/values.schema.json | Adds JSON schema validation for key values (database/lake/ui/ingress/networkPolicy/backup). |
| charts/devlake/templates/validate.yaml | Adds template-time validation for db type, secrets, resources, Grafana/Postgres compatibility, and hostNetwork restriction. |
| charts/devlake/templates/tests/test-connection.yaml | Adds a Helm test Pod to validate DB connectivity and HTTP endpoints for lake/ui. |
| charts/devlake/templates/statefulsets.yaml | Removes the legacy generic StatefulSet template. |
| charts/devlake/templates/statefulset-postgresql.yaml | Adds PostgreSQL-specific embedded StatefulSet implementation. |
| charts/devlake/templates/statefulset-mysql.yaml | Adds MySQL-specific embedded StatefulSet implementation. |
| charts/devlake/templates/services.yaml | Refactors DB service naming/ports by database.type; adds metrics scrape annotations; normalizes port names to http. |
| charts/devlake/templates/servicemonitor.yaml | Adds ServiceMonitor resources for lake and ui when enabled. |
| charts/devlake/templates/serviceaccount.yaml | Adds chart-managed ServiceAccount creation (when enabled). |
| charts/devlake/templates/secrets.yaml | Refactors DB secret generation to type-specific keys; gates internal Secret creation behind externalSecrets.enabled=false. |
| charts/devlake/templates/pre-install-validation.yaml | Adds a pre-install/pre-upgrade hook Job to surface config checks and warnings. |
| charts/devlake/templates/postgresql-migration-job.yaml | Adds a post-install/post-upgrade hook Job to apply PostgreSQL migration fixes. |
| charts/devlake/templates/poddisruptionbudget.yaml | Adds PDBs for lake and ui using a helper. |
| charts/devlake/templates/NOTES.txt | Expands install notes with access instructions, security guidance, and troubleshooting. |
| charts/devlake/templates/networkpolicy-ui.yaml | Adds optional NetworkPolicy for UI ingress. |
| charts/devlake/templates/networkpolicy-lake.yaml | Adds optional NetworkPolicy for lake ingress/egress. |
| charts/devlake/templates/networkpolicy-database.yaml | Adds optional NetworkPolicy limiting DB ingress to lake (and optionally Grafana). |
| charts/devlake/templates/ingresses.yaml | Adds optional nginx rate-limit annotation wiring. |
| charts/devlake/templates/hpa.yaml | Adds optional HPAs for lake and ui. |
| charts/devlake/templates/external-secret.yaml | Adds ExternalSecret resource to populate a Kubernetes Secret via External Secrets Operator. |
| charts/devlake/templates/deployments.yaml | Refactors lake/ui images to helper-based resolution; switches to chart ServiceAccount helper; adds anti-affinity/topology spread hooks; adjusts lake env wiring to new db config. |
| charts/devlake/templates/configmap.yaml | Refactors DB config ConfigMap generation to database.type-specific env keys. |
| charts/devlake/templates/backup-cronjob.yaml | Adds optional backup CronJob that runs mysqldump or pg_dump into a PVC-mounted path. |
| charts/devlake/templates/_helpers.tpl | Adds new helpers for db image/port/uid, digest pinning, PDB/anti-affinity, and renames secret/configmap helpers. |
| charts/devlake/Chart.yaml | Bumps chart to 2.0.0 and appVersion to v1.0.3-beta12. |
| .gitignore | Ignores **/.omc. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| {{- end }} | ||
|
|
||
| {{- if .Values.lake.hostNetwork }} | ||
| {{- fail "SECURITY VIOLATION: hostNetwork is not allowed. Remove lake.hostNetwork or use LoadBalancer/NodePort service type for external access. hostNetwork bypasses NetworkPolicies and exposes the node's network namespace." }} |
| serviceAccountName: {{ include "devlake.serviceAccountName" . }} | ||
| containers: | ||
| - name: postgresql-migration-fix | ||
| image: postgres:15-alpine |
Comment on lines
30
to
+34
| # Chart version | ||
| version: 1.0.3-beta10 | ||
| version: 2.0.0 | ||
|
|
||
| # devlake version | ||
| appVersion: v1.0.3-beta10 | ||
| appVersion: v1.0.3-beta12 |
Comment on lines
+69
to
+70
| {{- else -}} | ||
| {{- include "devlake.fullname" . }}-sa |
Comment on lines
+41
to
+48
| # Database access | ||
| - to: | ||
| - podSelector: | ||
| matchLabels: | ||
| devlakeComponent: {{ if eq .Values.database.type "mysql" }}mysql{{ else }}postgresql{{ end }} | ||
| ports: | ||
| - protocol: TCP | ||
| port: {{ if eq .Values.database.type "mysql" }}3306{{ else }}5432{{ end }} |
Comment on lines
+54
to
+70
| seccompProfile: | ||
| type: {{ .Values.database.securityContext.seccompProfile.type }} | ||
| initContainers: | ||
| - name: fix-permissions | ||
| image: {{ include "database.image" . }} | ||
| command: ['sh', '-c'] | ||
| args: | ||
| - | | ||
| chown -R 70:70 /var/lib/postgresql/data | ||
| chmod 700 /var/lib/postgresql/data | ||
| volumeMounts: | ||
| - name: data | ||
| mountPath: /var/lib/postgresql/data | ||
| securityContext: | ||
| runAsUser: 0 | ||
| runAsNonRoot: false | ||
| {{- with .Values.database.initContainers}} |
Comment on lines
+47
to
+48
| - name: backup | ||
| image: "{{ .Values.backup.image.repository }}:{{ .Values.backup.image.tag }}" |
| volumes: | ||
| - name: backup | ||
| persistentVolumeClaim: | ||
| claimName: {{ .Values.backup.pvc.existingClaim | default (printf "%s-backup" (include "devlake.fullname" .)) }} |
- Implemented PostgreSQL support in backup cronjob, statefulsets, and other templates. - Added common annotations for better resource management and identification. - Introduced external database CIDR configuration for network policies. - Enhanced validation checks for external database usage and resource requests. - Created a new PVC template for backup storage. - Updated values.yaml and values.schema.json to accommodate new configurations.
Author
|
@klesh Reviewed the annotations and add some stuff |
Contributor
|
I am not familiar with helm-chart. Would you like to help fix the failed tests? |
Author
|
I will. |
- Updated README.md to reflect the new project name and added comprehensive sections on features, prerequisites, installation, configuration, and operations. - Removed outdated ReleaseSOP.md file as it was no longer necessary. - Added .helmignore to exclude unnecessary files from Helm packaging. - Created a new README.md for the devlake chart with detailed requirements and values. - Introduced README.md.gotmpl for templating the README file. - Added licensing information to helper templates for compliance. - Cleaned up unnecessary whitespace in deployment and extra resources templates. - Ensured all templates include licensing information for consistency.
fabioluciano
force-pushed
the
feature/postgresql-support
branch
from
d234bdc to
044a99b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add separate StatefulSet templates for MySQL and PostgreSQL databases.
Replace single statefulsets.yaml with type-specific implementations.