MultiBackendJobManager: run_jobs , provider property not working

[tho-hole]

The MultiBackendJobManager has a provider property/column in the jobs DataFrame that should determine which backend a specific job is submitted to. However, when multiple providers (e.g., CDSE and VITO) are added to the DataFrame and the JobManager is launched, it ignores the provider field and instead balances jobs across all registered backends regardless of what is specified in the jobs DataFrame.

Expected Behavior:
Jobs should be submitted to the backend specified in the provider field of each row in the jobs DataFrame.

Actual Behavior:
Jobs are distributed/balanced across all registered backends, ignoring the provider field value.

Suggested Fix:
The MultiBackendJobManager should check the provider field for each job and submit it to the corresponding backend, only falling back to load balancing if the field is empty/null.

[soxofaan]

Jobs are distributed/balanced across all registered backends, ignoring the provider field value.

I think this is the intended design of MultiBackendJobManager:

• jobs are automatically distributed across available backends
• "provider" is an "output field" to keep track of where the job was sent to, it is not an "input field"

if you want to send jobs to a particular backend, you should use a dedicated MultiBackendJobManager with only that backend enabled