Refactor Job manager routes #296

ryuwd · 2024-09-16T15:17:22Z

Renamed from job_manager to jobs
Split into multiple modules
Removed some endpoints in favour of expressing the desired operation(s) as bulk job status changes
Major refactor of reschedule_job to respect the job state machine
significantly reduce the number of statements/queries generated in bulk operations

closes #22, #341
TODO:

Tests passing
Check rescheduling and resetJob
fix bug in job state machine when rescheduling in JobDB
make rescheduling and reset job tests a bit more rigorous to check I did not break anything
regenerate client methods with autorest
use config object to pull out MaxRescheduling from the CS instead of making a property in JobDB

diracx-db/src/diracx/db/sql/utils/job_status.py

ryuwd · 2024-09-23T08:50:30Z

65da423 is the start of using bulk insert/update statements in the job manager, and are supposed to be significantly faster than lots of single insert/update statements. I started with rescheduling and job status updates.

diracx-db/src/diracx/db/sql/utils/job_status.py

ryuwd · 2024-09-23T10:15:59Z

@fstagni do you have any view on things to watch out for with bulk inserts and updates, based on previous experience in DIRAC? I can foresee some risks of things becoming inconsistent, which could be mitigated by reducing the batch sizes or carefully checking the input and results of each statement execution.

diracx-core/src/diracx/core/config/schema.py

diracx-core/src/diracx/core/exceptions.py

diracx-testing/src/diracx/testing/__init__.py

diracx-routers/src/diracx/routers/jobs/submission.py

diracx-db/src/diracx/db/sql/job/db.py

diracx-core/src/diracx/core/config/schema.py

diracx-db/src/diracx/db/sql/job/db.py

…routes for jobs router

Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com>

Working tests

Regenerated REST client with autorest

…e comment)

whoops

regenerate client

…schedule_job, eventually will extend to set_job_statuses)

… where possible hold on to comments for later

…ID generation.

Overwrite correctly

ryuwd · 2024-12-16T13:49:53Z

@chrisburr @aldbr once the tests fully pass I think this is ready to merge

extensions/gubbins/gubbins-db/src/gubbins/db/sql/jobs/db.py

for more information, see https://pre-commit.ci

diracx-routers/tests/test_job_manager.py

diracx-routers/src/diracx/routers/jobs/status.py

diracx-client/src/diracx/client/generated/_client.py

diracx-db/src/diracx/db/sql/job/db.py

diracx-db/src/diracx/db/sql/utils/job_status.py

…routers.

ryuwd force-pushed the roneil-jobmgr branch from 6240cda to 0f71084 Compare September 16, 2024 15:20

chrisburr reviewed Sep 16, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/utils/job_status.py Outdated Show resolved Hide resolved

ryuwd force-pushed the roneil-jobmgr branch from 634dd67 to 22449b1 Compare September 17, 2024 11:43

fstagni linked an issue Sep 17, 2024 that may be closed by this pull request

JobManagerHandler conversion #22

Closed

8 tasks

ryuwd mentioned this pull request Sep 18, 2024

Re-organize the code #268

Closed

ryuwd commented Sep 23, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/utils/job_status.py Outdated Show resolved Hide resolved

aldbr reviewed Dec 13, 2024

View reviewed changes

ryuwd force-pushed the roneil-jobmgr branch from 5fdbf7a to 25798b4 Compare December 13, 2024 10:40

ryuwd marked this pull request as ready for review December 15, 2024 13:19

ryuwd commented Dec 16, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/job/db.py Outdated Show resolved Hide resolved

ryuwd commented Dec 16, 2024

View reviewed changes

diracx-core/src/diracx/core/config/schema.py Outdated Show resolved Hide resolved

ryuwd commented Dec 16, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/job/db.py Outdated Show resolved Hide resolved

ryuwd and others added 4 commits December 16, 2024 14:36

job_manager --> jobs

673769d

split jobs router into multiple jobs router modules; simplify status …

6b4895d

…routes for jobs router

Get result correctly from tasks

5b26ffa

Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com>

Moved to avoid clashing wrong route being matched

305b3bc

ryuwd force-pushed the roneil-jobmgr branch from 2aeec6a to e93e851 Compare December 16, 2024 13:36

ryuwd added 11 commits December 16, 2024 14:39

Fixed some tests

3f8f8fe

Working tests

Update doc for remove bulk

f959cdc

Regenerated REST client with autorest

Rescheduling including TODO regarding job state machine bug (check th…

030fdf1

…e comment)

Major refactoring of job rescheduling

420c138

whoops

Make reset_jobs optional

e99b3d1

Improved job rescheduling and the test

e278df3

Add missing MaxRescheduling to with_config_repo

1071b7e

regenerate client

Start refactoring things to use generally less queries (start with re…

8286072

…schedule_job, eventually will extend to set_job_statuses)

Evolve set_job_status to do bulk operations

37adf07

refactored job submission to reduce the number of statements executed…

e3b82cb

… where possible hold on to comments for later

use _bulk function

09dbdb8

ryuwd added 2 commits December 16, 2024 14:39

Cleanup insert_bulk - use taskgroups, and single row inserts for job …

ad7e5e8

…ID generation.

overwrite the correct method ...

1fb0f3a

Overwrite correctly

ryuwd force-pushed the roneil-jobmgr branch from e93e851 to 1fb0f3a Compare December 16, 2024 13:40

aldbr reviewed Dec 16, 2024

View reviewed changes

extensions/gubbins/gubbins-db/src/gubbins/db/sql/jobs/db.py Show resolved Hide resolved

ryuwd and others added 2 commits December 16, 2024 15:31

Fixed date truncation to avoid escaping issues in datetime formatting

97d37c3

[pre-commit.ci] auto fixes from pre-commit.com hooks

76e0fc3

for more information, see https://pre-commit.ci

aldbr reviewed Dec 16, 2024

View reviewed changes

diracx-routers/tests/test_job_manager.py Show resolved Hide resolved

aldbr reviewed Dec 16, 2024

View reviewed changes

diracx-routers/src/diracx/routers/jobs/status.py Outdated Show resolved Hide resolved

Remove stray comment

83dad27

chrisburr reviewed Dec 16, 2024

View reviewed changes

diracx-client/src/diracx/client/generated/_client.py Outdated Show resolved Hide resolved

Regenerate diracx client

4c0346d

aldbr mentioned this pull request Dec 16, 2024

Split test_job_manager.py into smaller and more specific test files #345

Closed

aldbr reviewed Dec 16, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/job/db.py Show resolved Hide resolved

aldbr reviewed Dec 16, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/job/db.py Outdated Show resolved Hide resolved

aldbr reviewed Dec 16, 2024

View reviewed changes

diracx-db/src/diracx/db/sql/utils/job_status.py Show resolved Hide resolved

ryuwd added 5 commits December 17, 2024 15:20

Moved non-DB related job submission logic out of diracx-db to diracx-…

80388dc

…routers.

Move job status functions from diracx-db to diracx-routers

f0291b0

This is less than ideal

9b273c9

Skip tests.

1625a40

refix tests

4df7c99

ryuwd force-pushed the roneil-jobmgr branch from fb7b8ff to 4df7c99 Compare December 17, 2024 15:12

ryuwd added 5 commits December 17, 2024 16:57

We agreed to move these functions back into diracx-db utils module

8016b6c

Gubbins hiccups

828e8c2

cast to list

da1b717

nearly..

f34c957

I affirm that the tests will pass

57e610c

aldbr approved these changes Dec 18, 2024

View reviewed changes

aldbr merged commit 3dd59d5 into DIRACGrid:main Dec 18, 2024
24 checks passed

ryuwd deleted the roneil-jobmgr branch October 14, 2025 17:39

Refactor Job manager routes #296

Refactor Job manager routes #296

Uh oh!

Conversation

ryuwd commented Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ryuwd commented Sep 23, 2024

Uh oh!

Uh oh!

ryuwd commented Sep 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryuwd commented Dec 16, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryuwd commented Sep 16, 2024 •

edited

Loading