[Scheduler] Defer block recycling to accelerate LRU node freeing#7885
[Scheduler] Defer block recycling to accelerate LRU node freeing#7885liyonghua0910 wants to merge 3 commits into
Conversation
|
Thanks for your contribution! |
liyonghua0910
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7885 +/- ##
==========================================
Coverage ? 67.60%
==========================================
Files ? 467
Lines ? 65105
Branches ? 9983
==========================================
Hits ? 44014
Misses ? 18292
Partials ? 2799
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览有 2 个 required 失败任务需要处理（1 个环境问题 + 1 个待审批），建议处理后合并。
2 任务状态汇总2.1 Required任务 : 7/10 通过
2.2 可选任务 — 28/32 通过
3 失败详情（仅 required）xpu_8cards_case_test / run_xpu_8cards_cases — 基础设施（置信度: 高）xpu_8cards_case_test / run_xpu_8cards_cases
失败用例:
根因详情: 关键日志: 修复建议:
关联变更: Approval — 需要人工审批该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
liyonghua0910
force-pushed
the
develop+20260521_free_blocks
branch
from
3c777a7 to
9522e2c
Compare
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-28 10:53:39
📋 Review 摘要
PR 概述：将 LRU 驱逐循环中的 recycle_gpu_blocks 调用延迟到循环结束后批量执行，减少堆操作开销，并修复父节点重复入堆问题。
变更范围：fastdeploy/cache_manager/prefix_cache_manager.py
影响面 Tag：[KVCache]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🟡 建议 | prefix_cache_manager.py:1352 |
[node.block_id] 包装方式需确认，None 可能混入回收列表 |
📝 PR 规范检查
标题 Tag [Scheduler] 与实际变更路径不符——改动文件位于 fastdeploy/cache_manager/，按影响面判断表应使用 [KVCache]。此外，Checklist 中 "Add unit tests" 已勾选，但 diff 中未见测试文件，PR 描述也未说明不需要单测的原因。
标题建议（可直接复制）：
[KVCache] Defer block recycling to accelerate LRU node freeing
PR 描述建议（点击展开，可直接复制）
## Motivation
In the LRU eviction loop of `free_block_ids_async`, each iteration calls `recycle_gpu_blocks` individually, which causes frequent heap operations and slows down the overall freeing process. This PR defers block recycling to a single batch call after the loop completes.
## Modifications
- Defer `recycle_gpu_blocks` calls inside the LRU freeing loop to a single batch call after the loop, reducing the overhead of repeated heap operations.
- Add `defer_recycle` parameter to `_handle_free_gpu_node_without_cpu` to support deferred block recycling.
- Fix the LRU leaf node freeing logic: disconnect the child node from its parent first, then check whether the parent should be added to the LRU heap, avoiding duplicate freeing.
- Add warning logs to help diagnose duplicate node issues in the LRU heap.
## Usage or Command
No additional configuration required. The optimization takes effect automatically.
## Accuracy Tests
Only affects KV Cache block recycling timing, no impact on model output accuracy.
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [x] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [x] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
整体优化思路清晰，延迟批量回收可有效减少堆操作频次，父节点重复入堆的 bug fix 也是必要修正。建议确认 recycle_gpu_blocks 对 flat list 的兼容性及 node.block_id 的 None 边界，并补充单测或在 PR 中说明不需要单测的原因。
| logger.info(f"_handle_free_gpu_node_without_cpu: free node {node.node_id}") | ||
|
|
||
| self.recycle_gpu_blocks(node.reverved_dec_block_ids) | ||
| blocks_to_recycle = list(node.reverved_dec_block_ids) + [node.block_id] |
There was a problem hiding this comment.
🟡 建议 [node.block_id] 包装方式需确认
原代码分两次调用 recycle_gpu_blocks：第一次传 list，第二次传单个值 node.block_id。新代码将两者合并为 list(...) + [node.block_id]，始终传 list。
需确认：
recycle_gpu_blocks是否支持接收 flat list（原第二次调用传的是单值）node.block_id是否可能为None（若存在无 GPU block 的节点，None会混入回收列表）
建议在此处加 guard：
blocks_to_recycle = list(node.reverved_dec_block_ids)
if node.block_id is not None:
blocks_to_recycle.append(node.block_id)3 participants
Motivation
In the LRU eviction loop of
free_block_ids_async, each iteration callsrecycle_gpu_blocksindividually, which causes frequent heap operations and slows down the overall freeing process. This PR defers block recycling to a single batch call after the loop completes.Modifications
blocks_deferred_to_recyclelist infree_block_ids_async. Instead of callingrecycle_gpu_blocksper node inside the while-loop, blocks are appended to this list and recycled in a single batch call after the loop exits, reducing the overhead of repeated heap operations.defer_recycleparameter to_handle_free_gpu_node_without_cpu: Whendefer_recycle=True, the method returns the list of block IDs to recycle (reverved_dec_block_ids + [block_id]) without callingrecycle_gpu_blocksitself, allowing the caller to batch the recycling. Whendefer_recycle=False, behavior is unchanged (immediate recycling).node.reverved_dec_block_idsin defer path: Whendefer_recycle=True, the node'sreverved_dec_block_idsis now cleared after copying to the return list, preventing double recycling if the node object is accessed again before GC.node.cache_status = CacheStatus.CPU: This line was accidentally removed during the refactor. Restored to keep consistency with_handle_free_gpu_node_with_cpu.parentvariable for clarity, and add a dedup check with warning log when the parent is already ingpu_lru_leaf_set.shared_count != 0, log a warning to aid diagnosis.Usage or Command
No additional configuration required. The optimization takes effect automatically.
Accuracy Tests
Only affects KV Cache block recycling timing, no impact on model output accuracy.
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.