From 54c34bba928bb20936c6ed27866a034b16ce2284 Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Wed, 3 Apr 2024 13:10:25 +0200 Subject: [PATCH 1/7] Propose queue changeTaskRunPriority --- README.md | 1 + rfcs/0190-queue-change-task-run-priority.md | 55 +++++++++++++++++++++ rfcs/README.md | 1 + 3 files changed, 57 insertions(+) create mode 100644 rfcs/0190-queue-change-task-run-priority.md diff --git a/README.md b/README.md index 79afd0f..8ba8c71 100644 --- a/README.md +++ b/README.md @@ -68,4 +68,5 @@ See [mechanics](mechanics.md) for more detail. | RFC#180 | [Github cancel previous tasks](rfcs/0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](rfcs/0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](rfcs/0189-batch-task-apis.md) | +| RFC#190 | [Queue change task run priority](rfcs/0190-queue-change-task-run-priority.md) | | RFC#191 | [Worker Manager launch configurations](rfcs/0191-worker-manager-launch-configs.md) | diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md new file mode 100644 index 0000000..38903c7 --- /dev/null +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -0,0 +1,55 @@ +# RFC 0190 - Queue change task run priority +* Comments: [#0190](https://github.com/taskcluster/taskcluster-rfcs/pull/190) +* Proposed by: @lotas + +## Summary + +Since we refactored queue internals it became possible to see what tasks are pending for a particular +worker pool / task queue. + +This RFC proposes new API method that would allow to change the priority of the existing task run. + +## Motivation + +There might be several use-cases when some worker pool has a lot of pending tasks and there is a need +to move some more important task to the top of the waiting list, or also move down less important ones. + +Instead of cancelling existing tasks we can instead choose to change task run priorities. + +## Details + +I propose we implement this on a task run level and not the task itself: + +* tasks are supposed to be immutable +* changing the priority makes more sense in the context of a specific run, not the whole task + +Queue service will expose new method `queue.changeTaskRunPriority(taskId, runId, newPriority)`. + +This will try to do the following: + +* update `task.runs[runId]` in the database to reflect priority change (possibly extra field to keep the original) +* update `priority` field in the `queue_pending_tasks` table + +If any of this fails (for example task is being changed by some other process), this method will also fail. + +This will also require introduction of the new scopes (anyOf): + +* `queue:change-task-run-priority-in-queue:` +* `queue:change-task-run-priority:` + + +Current order for picking up tasks is based on the task priority and insertion time (FIFO). +This RFC proposes to change the priority only, and leave the insertion time as is. + + +### Other considerations + +Chain of Trust validation might be broken if we change too much of the task definition or run. +We need to make sure we don't break it with this change. + + + + +## Implementation + +_pending_ \ No newline at end of file diff --git a/rfcs/README.md b/rfcs/README.md index 44c4580..fb901dd 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -56,4 +56,5 @@ | RFC#180 | [Github cancel previous tasks](0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](0189-batch-task-apis.md) | +| RFC#190 | [Queue change task run priority](0190-queue-change-task-run-priority.md) | | RFC#191 | [Worker Manager launch configurations](0191-worker-manager-launch-configs.md) | From 729ad6c93e6aaa9120bc05e3b27a94505ad9b984 Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 28 May 2024 14:49:29 +0200 Subject: [PATCH 2/7] Include more-granular scopes, typo fix --- rfcs/0190-queue-change-task-run-priority.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 38903c7..4a7a18d 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -12,7 +12,7 @@ This RFC proposes new API method that would allow to change the priority of the ## Motivation There might be several use-cases when some worker pool has a lot of pending tasks and there is a need -to move some more important task to the top of the waiting list, or also move down less important ones. +to move some more important tasks to the top of the waiting list, or also move down less important ones. Instead of cancelling existing tasks we can instead choose to change task run priorities. @@ -36,20 +36,17 @@ This will also require introduction of the new scopes (anyOf): * `queue:change-task-run-priority-in-queue:` * `queue:change-task-run-priority:` - +* `queue:lower-task-run-priority:` to only allow lowering the priority +* `queue:raise-task-run-priority:` to only allow raising the priority Current order for picking up tasks is based on the task priority and insertion time (FIFO). This RFC proposes to change the priority only, and leave the insertion time as is. - ### Other considerations Chain of Trust validation might be broken if we change too much of the task definition or run. We need to make sure we don't break it with this change. - - - ## Implementation _pending_ \ No newline at end of file From 361422dce8579445bd8bac80dccb22aa1f1a93bb Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 28 May 2024 15:16:20 +0200 Subject: [PATCH 3/7] Update the proposal from task-run level to the task and task group levels --- README.md | 2 +- rfcs/0190-queue-change-task-run-priority.md | 56 ++++++++++++++------- rfcs/README.md | 2 +- 3 files changed, 41 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 8ba8c71..22b33b5 100644 --- a/README.md +++ b/README.md @@ -68,5 +68,5 @@ See [mechanics](mechanics.md) for more detail. | RFC#180 | [Github cancel previous tasks](rfcs/0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](rfcs/0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](rfcs/0189-batch-task-apis.md) | -| RFC#190 | [Queue change task run priority](rfcs/0190-queue-change-task-run-priority.md) | +| RFC#190 | [Queue change task / task group priority](rfcs/0190-queue-change-task-run-priority.md) | | RFC#191 | [Worker Manager launch configurations](rfcs/0191-worker-manager-launch-configs.md) | diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 4a7a18d..29a9982 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -1,4 +1,4 @@ -# RFC 0190 - Queue change task run priority +# RFC 0190 - Queue change task / task group priority * Comments: [#0190](https://github.com/taskcluster/taskcluster-rfcs/pull/190) * Proposed by: @lotas @@ -7,45 +7,67 @@ Since we refactored queue internals it became possible to see what tasks are pending for a particular worker pool / task queue. -This RFC proposes new API method that would allow to change the priority of the existing task run. +This RFC proposes new API method that would allow to change the priority of the existing task run / task group. ## Motivation There might be several use-cases when some worker pool has a lot of pending tasks and there is a need to move some more important tasks to the top of the waiting list, or also move down less important ones. -Instead of cancelling existing tasks we can instead choose to change task run priorities. +Instead of cancelling some existing tasks we can change task run priorities. + +Use-case examples: + +* We have a high-priority patch running tasks on Try. We want to bump the priority of all tasks in this task group +* We have a chemspill release or high priority nightly to get out. We want to bump the priority of all tasks in one or more groups +* We are waiting for a specific task in a low resource pool (eg: mac hardware). We want to bump the priority of that specific task +* There are several tasks with the same highest priority and one of them is more important. We want to lower the priority of the less important task ## Details -I propose we implement this on a task run level and not the task itself: +Queue service will expose new methods: + +* `queue.changeTaskPriority(taskId, newPriority)` +* `queue.changeTaskGroupPriority(taskGroupId, newPriority)` + +New priority would be stored along the task definition for a given task or all tasks within the task group. -* tasks are supposed to be immutable -* changing the priority makes more sense in the context of a specific run, not the whole task +The process to change single task will be as follows: -Queue service will expose new method `queue.changeTaskRunPriority(taskId, runId, newPriority)`. +* new task priority will be stored in the database as `task.priority_override` column +* if task was scheduled already, its priority in the `queue_pending_tasks` table will be updated +* if the tasks run fails and new one is created, it will check if `task.priority_override` is set and use it instead of the original priority -This will try to do the following: +The process to change the whole task group will be as follows: -* update `task.runs[runId]` in the database to reflect priority change (possibly extra field to keep the original) -* update `priority` field in the `queue_pending_tasks` table +* all tasks in the task group will be updated to store the new priority in the `task.priority_override` column +* all currently scheduled tasks would be attempted to be updated in the `queue_pending_tasks` table +* all unscheduled tasks and tasks that will be restarted will use the new priority -If any of this fails (for example task is being changed by some other process), this method will also fail. +### New scopes This will also require introduction of the new scopes (anyOf): -* `queue:change-task-run-priority-in-queue:` -* `queue:change-task-run-priority:` -* `queue:lower-task-run-priority:` to only allow lowering the priority -* `queue:raise-task-run-priority:` to only allow raising the priority +* `queue:change-task-priority-in-queue:` +* `queue:change-task-priority:` +* `queue:lower-task-priority:` to only allow lowering the priority +* `queue:raise-task-priority:` to only allow raising the priority + +And for task group: + +* `queue:change-task-group-priority:/` (for consistency with existing task group scopes) +* `queue:lower-task-group-priority:` to only allow lowering the priority +* `queue:raise-task-group-priority:` to only allow raising the priority + +### Priority calculation Current order for picking up tasks is based on the task priority and insertion time (FIFO). This RFC proposes to change the priority only, and leave the insertion time as is. ### Other considerations -Chain of Trust validation might be broken if we change too much of the task definition or run. -We need to make sure we don't break it with this change. +To insure that Chain of Trust validation is not affected, we aim to keep the original task definition +and store the new priority in a separate column. ## Implementation diff --git a/rfcs/README.md b/rfcs/README.md index fb901dd..2064076 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -56,5 +56,5 @@ | RFC#180 | [Github cancel previous tasks](0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](0189-batch-task-apis.md) | -| RFC#190 | [Queue change task run priority](0190-queue-change-task-run-priority.md) | +| RFC#190 | [Queue change task / task group priority](0190-queue-change-task-run-priority.md) | | RFC#191 | [Worker Manager launch configurations](0191-worker-manager-launch-configs.md) | From c671add4d3dc42161daa25c66d68fae2cfe31b57 Mon Sep 17 00:00:00 2001 From: Yarik Date: Tue, 4 Jun 2024 11:09:08 +0200 Subject: [PATCH 4/7] Update rfcs/0190-queue-change-task-run-priority.md Co-authored-by: Ben Hearsum (he/him) --- rfcs/0190-queue-change-task-run-priority.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 29a9982..0a063db 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -66,7 +66,7 @@ This RFC proposes to change the priority only, and leave the insertion time as i ### Other considerations -To insure that Chain of Trust validation is not affected, we aim to keep the original task definition +To ensure that Chain of Trust validation is not affected, we aim to keep the original task definition and store the new priority in a separate column. ## Implementation From f3144c2dd5abeea28378432f9c112bd2601ca925 Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 4 Jun 2024 11:30:50 +0200 Subject: [PATCH 5/7] Include info about cot ignore_keys, update phrasing --- rfcs/0190-queue-change-task-run-priority.md | 23 +++++++++------------ 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 0a063db..b82d3d2 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -7,7 +7,7 @@ Since we refactored queue internals it became possible to see what tasks are pending for a particular worker pool / task queue. -This RFC proposes new API method that would allow to change the priority of the existing task run / task group. +This RFC proposes a new API method that would allow changing the priority of the existing task run / task group. ## Motivation @@ -30,19 +30,21 @@ Queue service will expose new methods: * `queue.changeTaskPriority(taskId, newPriority)` * `queue.changeTaskGroupPriority(taskGroupId, newPriority)` -New priority would be stored along the task definition for a given task or all tasks within the task group. +New priority would replace the original one for a given task or all tasks within the task group. The process to change single task will be as follows: -* new task priority will be stored in the database as `task.priority_override` column -* if task was scheduled already, its priority in the `queue_pending_tasks` table will be updated -* if the tasks run fails and new one is created, it will check if `task.priority_override` is set and use it instead of the original priority +* new task priority will be stored in the database as `task.priority` (overriding existing value) +* additionally, if task was scheduled already, its priority will be updated in the `queue_pending_tasks` table The process to change the whole task group will be as follows: -* all tasks in the task group will be updated to store the new priority in the `task.priority_override` column -* all currently scheduled tasks would be attempted to be updated in the `queue_pending_tasks` table -* all unscheduled tasks and tasks that will be restarted will use the new priority +* all tasks in the task group will be updated to store the new priority in the `task.priority` column +* additionally, all currently scheduled tasks would be attempted to be updated in the `queue_pending_tasks` table + +Tasks would only be updated if they are not resolved yet and before their deadlines. + +> **Note**: To allow changing the priority of the task without breaking CoT validations, it would be necessary to add `"priority"` to the [`ignore_keys`](https://github.com/mozilla-releng/scriptworker/blob/454c4dd0bae7958140ea8d19adf3670e705ace09/src/scriptworker/cot/verify.py#L910) ### New scopes @@ -64,11 +66,6 @@ And for task group: Current order for picking up tasks is based on the task priority and insertion time (FIFO). This RFC proposes to change the priority only, and leave the insertion time as is. -### Other considerations - -To ensure that Chain of Trust validation is not affected, we aim to keep the original task definition -and store the new priority in a separate column. - ## Implementation _pending_ \ No newline at end of file From 95a7b610b93dad78c82c6bfcec01da0351420028 Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 4 Jun 2024 11:46:00 +0200 Subject: [PATCH 6/7] Include pulse events --- rfcs/0190-queue-change-task-run-priority.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index b82d3d2..b071011 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -46,6 +46,17 @@ Tasks would only be updated if they are not resolved yet and before their deadli > **Note**: To allow changing the priority of the task without breaking CoT validations, it would be necessary to add `"priority"` to the [`ignore_keys`](https://github.com/mozilla-releng/scriptworker/blob/454c4dd0bae7958140ea8d19adf3670e705ace09/src/scriptworker/cot/verify.py#L910) +### Events + +Since Taskcluster doesn't have any built-in auditing tools, it would be beneficial to have events for the priority change. + +Following events would be emitted: + +* `task-priority-changed` with `taskId`, `existingPriority` and `newPriority` +* `task-group-priority-changed` with `taskGroupId`, `newPriority` + +When the task group priority is changed, the event will be emitted for each task in the group additionally. + ### New scopes This will also require introduction of the new scopes (anyOf): From 8d4e0c146c531fd182a707598ffa48aeac9a01bb Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Thu, 6 Nov 2025 12:25:09 +0100 Subject: [PATCH 7/7] update line about audits and scopes --- rfcs/0190-queue-change-task-run-priority.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index b071011..c2e325c 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -48,9 +48,8 @@ Tasks would only be updated if they are not resolved yet and before their deadli ### Events -Since Taskcluster doesn't have any built-in auditing tools, it would be beneficial to have events for the priority change. - -Following events would be emitted: +Taskcluster has internal audit system and we can track those events by emitting audit events. +Additionally, following events would be emitted: * `task-priority-changed` with `taskId`, `existingPriority` and `newPriority` * `task-group-priority-changed` with `taskGroupId`, `newPriority` @@ -61,6 +60,7 @@ When the task group priority is changed, the event will be emitted for each task This will also require introduction of the new scopes (anyOf): +* `queue:create-task::` * `queue:change-task-priority-in-queue:` * `queue:change-task-priority:` * `queue:lower-task-priority:` to only allow lowering the priority @@ -79,4 +79,4 @@ This RFC proposes to change the priority only, and leave the insertion time as i ## Implementation -_pending_ \ No newline at end of file +_pending_