From 0b59c49ef35d1220f7d9f30e2d3e7fae4e3b9731 Mon Sep 17 00:00:00 2001 From: Adam Gutglick Date: Mon, 8 Jun 2026 12:43:42 +0100 Subject: [PATCH 1/5] CI 2 proposal Signed-off-by: Adam Gutglick --- rfcs/0061-ci-2-0.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 rfcs/0061-ci-2-0.md diff --git a/rfcs/0061-ci-2-0.md b/rfcs/0061-ci-2-0.md new file mode 100644 index 0000000..99b9d26 --- /dev/null +++ b/rfcs/0061-ci-2-0.md @@ -0,0 +1,23 @@ +- Start Date: 2026-06-08 +- Authors: @AdamGS +- RFC PR: [vortex-data/rfcs#61](https://github.com/vortex-data/rfcs/pull/61) + +# CI 2.0 + +## Summary + +## Motivation + +## Design + +## Compatibility + +## Drawbacks + +## Alternatives + +## Prior Art + +## Unresolved Questions + +## Future Possibilities From b35542d1864f2272b517481f7f602f276447e23b Mon Sep 17 00:00:00 2001 From: Adam Gutglick Date: Mon, 8 Jun 2026 13:16:09 +0100 Subject: [PATCH 2/5] start Signed-off-by: Adam Gutglick --- rfcs/0061-ci-2-0.md | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/rfcs/0061-ci-2-0.md b/rfcs/0061-ci-2-0.md index 99b9d26..cf6de30 100644 --- a/rfcs/0061-ci-2-0.md +++ b/rfcs/0061-ci-2-0.md @@ -6,7 +6,46 @@ ## Summary -## Motivation +This RFC intends to propose a cohesive architecture for the project's CI and workflow system. There have been many conversations (both public and private) on this topic, +and I would like to try and propose a holistic vision that can be discussed and iterated in a public forum. + +## Current state + +The current solution includes (but is not limited) to CI, publishing and benchmarking. It is fully implemented using Github Actions workflows and apps, and relies on +a mix [runs-on](https://runs-on.com/) powered self-hosted runners and GitHub hosted runners. + +### Infrastructure + +Our self-hosted runners use a custom AMI, that we build using a private repo only accessible to members of the GitHub organization. + +### Permissions + +There are four different scopes of permissions: + +1. Admins - very limited group of users that can manage the organization-wide settings, and the core repo settings. +2. Maintainers - All TSC members, can push branches and open pull requests directly from within the core repo, and approve and merge pull requests. + CI runs automatically for them, and they can trigger CI for external contributions. +3. Members - Non-TSC members that are explicitly trusted. They have the same development experience a maintainer has, aside from the ability to approve and merge pull requests. +4. Everyone else - have to open PRs using a fork in a classic GH flow. Requires a maintainer to run any actions-based workflow, label their PR etc. + +### Benchmarking + +TODO + +### Testing and CI + +TODO + +### Versioning and publishing + +Every pull request must include a label indicating the sort of change it includes (feature, fix, chore, performance, ci or skip). +Those changes are accumulated into a draft release. What that draft is published, a workflow is triggers that gradually publishes all rust crates +and all language or framework bindings in order. Those include: + +1. Rust crates (including integrations like `vortex-datafusion`). +2. Python package +3. JNI bindings +4. Spark bindings ## Design From 6ad691a94be847ee01c491394dd07650b59bec0f Mon Sep 17 00:00:00 2001 From: Adam Gutglick Date: Mon, 8 Jun 2026 13:23:14 +0100 Subject: [PATCH 3/5] more Signed-off-by: Adam Gutglick --- rfcs/0061-ci-2-0.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/rfcs/0061-ci-2-0.md b/rfcs/0061-ci-2-0.md index cf6de30..58605ab 100644 --- a/rfcs/0061-ci-2-0.md +++ b/rfcs/0061-ci-2-0.md @@ -38,7 +38,7 @@ TODO ### Versioning and publishing -Every pull request must include a label indicating the sort of change it includes (feature, fix, chore, performance, ci or skip). +Every pull request must include a label indicating the sort of change it includes (feature, break, fix, chore, performance, ci or skip). Those changes are accumulated into a draft release. What that draft is published, a workflow is triggers that gradually publishes all rust crates and all language or framework bindings in order. Those include: @@ -47,16 +47,16 @@ and all language or framework bindings in order. Those include: 3. JNI bindings 4. Spark bindings -## Design +## Issues -## Compatibility +### Versioning is tightly coupled -## Drawbacks +Our versioning is tightly coupled between the rust implementation and external integrations. Fixing any issue in the Java bindings or DataFusion integration +requires a full release, which might include many unrelated changes. -## Alternatives +This also makes the meaning of the version itself murky. We currently vaguely follow semantic versioning, but we effectively bump or minor version on every release, which means that +a breaking change is always acceptable. -## Prior Art - -## Unresolved Questions +### Permissions -## Future Possibilities +TODO From 72919010ac241e6be6820e0633de5e3a8fc2d8b6 Mon Sep 17 00:00:00 2001 From: Adam Gutglick Date: Mon, 8 Jun 2026 14:00:35 +0100 Subject: [PATCH 4/5] text Signed-off-by: Adam Gutglick --- rfcs/0061-ci-2-0.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/rfcs/0061-ci-2-0.md b/rfcs/0061-ci-2-0.md index 58605ab..8b75f53 100644 --- a/rfcs/0061-ci-2-0.md +++ b/rfcs/0061-ci-2-0.md @@ -18,7 +18,7 @@ a mix [runs-on](https://runs-on.com/) powered self-hosted runners and GitHub hos Our self-hosted runners use a custom AMI, that we build using a private repo only accessible to members of the GitHub organization. -### Permissions +### Permission Model There are four different scopes of permissions: @@ -28,13 +28,19 @@ There are four different scopes of permissions: 3. Members - Non-TSC members that are explicitly trusted. They have the same development experience a maintainer has, aside from the ability to approve and merge pull requests. 4. Everyone else - have to open PRs using a fork in a classic GH flow. Requires a maintainer to run any actions-based workflow, label their PR etc. -### Benchmarking +### Testing and CI -TODO +Our current test suite includes many workflows, I think they can be grouped into the following categories: -### Testing and CI +1. Linters - these include rustfmt, clippy, cargo-deny and other language-specific tooling. +2. Core tests - the full test suite, coverage tests +3. Integration-focused tests - Python, Java/JNI, SLT, C/C++ +4. Environment tests - making sure Vortex works on various environments like wasm and Windows. + +### Benchmarking -TODO +Benchmarks run on dedicated runs-on runners, orchestrated by Github, they run both on a nightly schedule, on every commit and can be triggered as-required by a maintainer +labeling the PR. ### Versioning and publishing @@ -52,11 +58,13 @@ and all language or framework bindings in order. Those include: ### Versioning is tightly coupled Our versioning is tightly coupled between the rust implementation and external integrations. Fixing any issue in the Java bindings or DataFusion integration -requires a full release, which might include many unrelated changes. +requires a full release, which might include many unrelated changes. It also means that changes in the periphery of the project gets included in the changelog AND +the versioning scheme for the core of the project. This also makes the meaning of the version itself murky. We currently vaguely follow semantic versioning, but we effectively bump or minor version on every release, which means that a breaking change is always acceptable. ### Permissions -TODO +Everyone on the maintainers group is automatically subscribed to every repo event, which creates a lot of noise in both emails and notifications. The distinction between them +is hard to communicate externally. From 0832275dc2c5d2c0a86c7e8a047dc43037045e08 Mon Sep 17 00:00:00 2001 From: Adam Gutglick Date: Mon, 8 Jun 2026 14:55:24 +0100 Subject: [PATCH 5/5] text Signed-off-by: Adam Gutglick --- rfcs/0061-ci-2-0.md | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/rfcs/0061-ci-2-0.md b/rfcs/0061-ci-2-0.md index 8b75f53..8fa665f 100644 --- a/rfcs/0061-ci-2-0.md +++ b/rfcs/0061-ci-2-0.md @@ -9,15 +9,13 @@ This RFC intends to propose a cohesive architecture for the project's CI and workflow system. There have been many conversations (both public and private) on this topic, and I would like to try and propose a holistic vision that can be discussed and iterated in a public forum. +I've tried to include the current state of the project to the best of my understanding, and the issues I see that motivate my proposed changes. + ## Current state The current solution includes (but is not limited) to CI, publishing and benchmarking. It is fully implemented using Github Actions workflows and apps, and relies on a mix [runs-on](https://runs-on.com/) powered self-hosted runners and GitHub hosted runners. -### Infrastructure - -Our self-hosted runners use a custom AMI, that we build using a private repo only accessible to members of the GitHub organization. - ### Permission Model There are four different scopes of permissions: @@ -28,6 +26,9 @@ There are four different scopes of permissions: 3. Members - Non-TSC members that are explicitly trusted. They have the same development experience a maintainer has, aside from the ability to approve and merge pull requests. 4. Everyone else - have to open PRs using a fork in a classic GH flow. Requires a maintainer to run any actions-based workflow, label their PR etc. +Everyone on the maintainers group is automatically subscribed to every repo event, which creates a lot of noise in both emails and notifications. The distinction between the two +groups is hard to communicate externally. + ### Testing and CI Our current test suite includes many workflows, I think they can be grouped into the following categories: @@ -42,6 +43,8 @@ Our current test suite includes many workflows, I think they can be grouped into Benchmarks run on dedicated runs-on runners, orchestrated by Github, they run both on a nightly schedule, on every commit and can be triggered as-required by a maintainer labeling the PR. +Any user that can run CI has potential unbounded access to the AWS account, allowing them (or a 3rd party that took control of their account) to hijack resources. + ### Versioning and publishing Every pull request must include a label indicating the sort of change it includes (feature, break, fix, chore, performance, ci or skip). @@ -53,10 +56,6 @@ and all language or framework bindings in order. Those include: 3. JNI bindings 4. Spark bindings -## Issues - -### Versioning is tightly coupled - Our versioning is tightly coupled between the rust implementation and external integrations. Fixing any issue in the Java bindings or DataFusion integration requires a full release, which might include many unrelated changes. It also means that changes in the periphery of the project gets included in the changelog AND the versioning scheme for the core of the project. @@ -64,7 +63,20 @@ the versioning scheme for the core of the project. This also makes the meaning of the version itself murky. We currently vaguely follow semantic versioning, but we effectively bump or minor version on every release, which means that a breaking change is always acceptable. +We also don't have clear and consistent definitions of what each kind of changes means, making the changelog inconsistent and somewhat confusing. + +## Suggested changes + ### Permissions -Everyone on the maintainers group is automatically subscribed to every repo event, which creates a lot of noise in both emails and notifications. The distinction between them -is hard to communicate externally. +I suggest we get rid of the two-tier member status, moving the GitHub's built-in permission model that allows users that contributed to the project in the past to run CI by-default, which will make the committers group redundant. + +Their CI flow will not run on the project's runs-on infrastructure, which we can limit by making CI check `github.event.pull_request.head.repo.full_name` instead of `github.repository`. We can further reduce the risk of external contributors by adding more restrictions attempted changes to the `.github` directory. + +### Benchmarks Bot + +For benchmarks, we'll create a new dedicated bot that will have a limited list of allowed users that can trigger benchmark run using a comment (or any other preferred control flow). The bot will be hosted in a different repo, so the surface area for runs-on for external contributors is reduced. Hosting the bot externally will also make it easier to have it run on PRs coming from forks, and it'll be easier to give it a more flexible scheduling system and more powerful permissions. + +### Versioning + +I suggest we move some of the binding into their own