[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

heheda12345 · 2025-04-30T14:48:58Z

In the future hybrid allocator, the KVCacheManager output would be list[list[list[KVCacheBlocks]], which is much more complex than the current list[KVCacheBlocks].

To hide the complexity, this pr introduces KVCacheBlocks to save the KVCacheManager output so that scheduler do not need to parse the internal structure of KVCacheManager output.

Splitted from #16101

Signed-off-by: Chen Zhang <[email protected]>

github-actions · 2025-04-30T14:49:06Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

mergify · 2025-04-30T19:38:04Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @heheda12345.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Chen Zhang <[email protected]>

WoosukKwon

LGTM! Left minor comments.

WoosukKwon · 2025-05-06T05:16:48Z

vllm/v1/core/kv_cache_manager.py


    def allocate_slots(
        self,
        request: Request,
        num_tokens: int,
-        new_computed_blocks: Optional[list[KVCacheBlock]] = None,
+        new_computed_blocks_obj: Optional[KVCacheBlocks] = None,


Q: Why rename this?

Becase it is with type KVCacheBlocks and we need to unwrap it to list[KVCacheBlock] before using. Prefer to call the object with type list[KVCacheBlock] "new_computed_blocks" so I rename this object to "new_computed_blocks_obj"

But when would we ever use list[KVCacheBlock] directly? The _obj suffix feels a bit awkward to me.

list[KVCacheBlock] is used inside allocate_slots

I thought we kinda decided to blur the distinction when introducing the KVCacheBlocks class. I don't understand why this particular variable name is a concern for you.

Otherwise I need to change new_computed_blocks in this function to another name.

I personally prefer that, because I really feel new_computed_blocks_obj awkward. But I'm fine if you want to stick to this.

OK. I can change it later.

vllm/v1/core/kv_cache_manager.py

Signed-off-by: Chen Zhang <[email protected]>

heheda12345 · 2025-05-06T06:47:06Z

@WoosukKwon I've updated this PR. Can you take another look?

Signed-off-by: Chen Zhang <[email protected]>

eryugey · 2025-05-06T14:01:16Z

vllm/v1/core/kv_cache_manager.py

@@ -109,7 +128,7 @@ def get_computed_blocks(
        """


Update the doc string as well about the return type?

* [Model] Add GraniteMoeHybrid 4.0 model (vllm-project#17497) Signed-off-by: Thomas Ortner <[email protected]> Signed-off-by: Stanislaw Wozniak <[email protected]> Co-authored-by: Thomas Ortner <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> * [easy] Fix logspam on PiecewiseBackend errors (vllm-project#17138) Signed-off-by: rzou <[email protected]> * [Bugfix] Fixed prompt length for random dataset (vllm-project#17408) Signed-off-by: Mikhail Podvitskii <[email protected]> * [Doc] Update notes for H2O-VL and Gemma3 (vllm-project#17219) Signed-off-by: DarkLight1337 <[email protected]> * [Misc] Fix ScalarType float4 naming (vllm-project#17690) Signed-off-by: Lucas Wilkinson <[email protected]> * Fix `dockerfilegraph` pre-commit hook (vllm-project#17698) Signed-off-by: Harry Mellor <[email protected]> * [Bugfix] Fix triton import with local TritonPlaceholder (vllm-project#17446) Signed-off-by: Mengqing Cao <[email protected]> * [V1] Enable TPU V1 backend by default (vllm-project#17673) Signed-off-by: mgoin <[email protected]> * [V1][PP] Support PP for MultiprocExecutor (vllm-project#14219) Signed-off-by: jiang1.li <[email protected]> Signed-off-by: jiang.li <[email protected]> * [v1] AttentionMetadata for each layer (vllm-project#17394) Signed-off-by: Chen Zhang <[email protected]> * [Feat] Add deprecated=True to CLI args (vllm-project#17426) Signed-off-by: Aaron Pham <[email protected]> * [Docs] Use gh-file to add links to tool_calling.md (vllm-project#17709) Signed-off-by: windsonsea <[email protected]> * [v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager (vllm-project#17479) Signed-off-by: Chen Zhang <[email protected]> * [doc] Add RAG Integration example (vllm-project#17692) Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]> * [Bugfix] Fix modality limits in vision language example (vllm-project#17721) Signed-off-by: DarkLight1337 <[email protected]> * Make right sidebar more readable in "Supported Models" (vllm-project#17723) Signed-off-by: Harry Mellor <[email protected]> * [TPU] Increase block size and reset block shapes (vllm-project#16458) * [Misc] Add Next Edit Prediction (NEP) datasets support in `benchmark_serving.py` (vllm-project#16839) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Signed-off-by: dtransposed <> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> * [Bugfix] Fix for the condition to accept empty encoder inputs for mllama (vllm-project#17732) Signed-off-by: Gregory Shtrasberg <[email protected]> * [Kernel] Unified Triton kernel that doesn't distinguish between prefill + decode (vllm-project#16828) Signed-off-by: Thomas Parnell <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> --------- Signed-off-by: Thomas Ortner <[email protected]> Signed-off-by: Stanislaw Wozniak <[email protected]> Signed-off-by: rzou <[email protected]> Signed-off-by: Mikhail Podvitskii <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: Mengqing Cao <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: jiang1.li <[email protected]> Signed-off-by: jiang.li <[email protected]> Signed-off-by: Chen Zhang <[email protected]> Signed-off-by: Aaron Pham <[email protected]> Signed-off-by: windsonsea <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Signed-off-by: dtransposed <> Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Thomas Parnell <[email protected]> Signed-off-by: [email protected] <[email protected]> Co-authored-by: Stan Wozniak <[email protected]> Co-authored-by: Thomas Ortner <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Richard Zou <[email protected]> Co-authored-by: Mikhail Podvitskii <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Chen Zhang <[email protected]> Co-authored-by: Aaron Pham <[email protected]> Co-authored-by: Michael Yao <[email protected]> Co-authored-by: Reid <[email protected]> Co-authored-by: reidliu41 <[email protected]> Co-authored-by: Jevin Jiang <[email protected]> Co-authored-by: d.transposed <[email protected]> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Gregory Shtrasberg <[email protected]> Co-authored-by: Thomas Parnell <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]>

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <[email protected]> Signed-off-by: Mu Huai <[email protected]>

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <[email protected]>

update test

e484ba9

Signed-off-by: Chen Zhang <[email protected]>

heheda12345 requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners April 30, 2025 14:48

mergify bot added the v1 label Apr 30, 2025

mergify bot added the needs-rebase label Apr 30, 2025

Merge branch 'main' of github.com:vllm-project/vllm into kv_cache_blocks

579c07a

Signed-off-by: Chen Zhang <[email protected]>

mergify bot removed the needs-rebase label May 1, 2025

WoosukKwon approved these changes May 6, 2025

View reviewed changes

heheda12345 added 2 commits May 5, 2025 23:41

cls

93b985c

Signed-off-by: Chen Zhang <[email protected]>

rename

4f1c40a

Signed-off-by: Chen Zhang <[email protected]>

Merge branch 'main' of github.com:vllm-project/vllm into kv_cache_blocks

986f19c

WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label May 6, 2025

fix

7789dac

Signed-off-by: Chen Zhang <[email protected]>

eryugey suggested changes May 6, 2025

View reviewed changes

WoosukKwon merged commit aabcd2c into vllm-project:main May 6, 2025
49 checks passed

mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCac…

909425e

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <[email protected]>

This was referenced May 14, 2025

[Bug]: V0 Scheduler is incapable of the newest KVCacheManager interface in vllm main branch code vllm-project/vllm-ascend#861

Open

[WIP] Fixed: v0 style Scheduler broken issue in vllm main branch. vllm-project/vllm-ascend#862

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

heheda12345 commented Apr 30, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Apr 30, 2025

mergify bot commented Apr 30, 2025

WoosukKwon left a comment

WoosukKwon May 6, 2025

heheda12345 May 6, 2025

WoosukKwon May 6, 2025

heheda12345 May 6, 2025

WoosukKwon May 6, 2025 •

edited

Loading

heheda12345 May 6, 2025

WoosukKwon May 6, 2025

heheda12345 May 6, 2025

heheda12345 commented May 6, 2025

eryugey May 6, 2025

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

Conversation

heheda12345 commented Apr 30, 2025 • edited by github-actions bot Loading

github-actions bot commented Apr 30, 2025

mergify bot commented Apr 30, 2025

WoosukKwon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WoosukKwon May 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heheda12345 commented May 6, 2025

Choose a reason for hiding this comment

heheda12345 commented Apr 30, 2025 •

edited by github-actions bot

Loading

WoosukKwon May 6, 2025 •

edited

Loading