smarter tokenization pool #2
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
router/src/lib.rs
Outdated
| // fall-back to num_cpus - 1 to leave some CPU for the backend, and at most 64 workers. | ||
| let tokenization_workers = | ||
| tokenization_workers.unwrap_or_else(|| std::cmp::min(std::cmp::max(2, num_cpus::get() - 1), 64)); |
There was a problem hiding this comment.
When tokenization_workers is not set, the new fallback min(max(2, num_cpus::get() - 1), 64) still spawns at least two tokenization threads even on 1–2 CPU hosts, leaving no core free for the backend despite the intent expressed in the preceding comment. On such small instances this change can regress inference throughput by competing with the backend for all available CPU, unlike the previous default of matching the core count. Consider leaving one core free without forcing a minimum of two workers.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
fair, i'll change to 1 minium.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
What does this PR do?
Fixes # (issue)
Before submitting
instasnapshots?Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.