Skip to content

Investigate slow ML tasks on the AsyncCompute pool #381

@YukumoHunter

Description

@YukumoHunter

Currently, we run all ML on the Compute pool, but as slow models can run over multiple cycles, we ideally want them to be executed from the AsyncCompute pool. This in theory also makes sure that the Compute pool keeps low contention and we miss our control cycle deadline (82Hz/12ms) less. However, when we actually make this change we see our cycle time shoot up to almost 30ms in Rerun.

Current ideas:

  • It could be that the AsyncCompute pool is out of threads and these blocking tasks create a giant backlog. Would it make sense to instead spawn them inside the blocking crate's blocking::unblock?

Investigation tips:

  • sindri run --timings flag to look at profiler output
  • Task pool thread division can be configured in bevy

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions