[Core][V1][TPU] Enable structured decoding on TPU V1 #16499

Chenyaaang · 2025-04-11T16:24:16Z

Enable structured decoding on V1 TPU.
The workflow changes from model forward -> select hidden states -> sample from hidden (compute logits + sampler) to
model forward -> select hidden states -> compute logits -> structured decode -> sampler

The change has been verified no recompilation.

pre-compile time

Adding structured deocoding takes around 25 seconds to compile additional 10 xla graphs.
previous:

stage	#xla graphs	time(s)
backbone	24	103.87
select_hidden_states	52	12.10
sample_from_hidden	20	16.53

now:

stage	#xla graphs	time(s)
backbone	24	103.83
select_hidden_states	52	12.19
compute_logits	10	2.31
structured_decoding	10	25.12
sample_from_logits	15	14.12

How to run structured deocoding

unit test

pytest -s tests/v1/entrypoints/llm/test_struct_output_generate.py::test_structured_output

benchmark

server: VLLM_USE_V1=1 VLLM_XLA_CHECK_RECOMPILATION=1 VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct --disable-log-requests --gpu-memory-utilization 0.98 --max-num-batched-tokens 2048 --max-num-seqs 128 --max-model-len 4096 --tensor-parallel-size 1 --no-enable-prefix-caching
client: python benchmarks/benchmark_serving_structured_output.py --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --dataset choice --structured-output-ratio 1.0 --structured-output-backend auto --request-rate 10 --num-prompts 500

Signed-off-by: Chenyaaang <[email protected]>

github-actions · 2025-04-11T16:24:27Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

mergify · 2025-04-11T16:25:03Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Chenyaaang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Chenyaaang <[email protected]>

yaochengji

Thanks for your contribution!

What's the impact on the execution time when splitting sample_from_hidden to two parts?

How does the performance compare with and without structured decoding?

vllm/v1/worker/tpu_model_runner.py

russellb · 2025-04-16T12:30:33Z

vllm/v1/worker/tpu_model_runner.py

+        logger.info(
+            "Compiling structured_decoding with different input shapes.")


Do you want to keep this and other logs in this method? They seem more like debug logs to me, so perhaps logger.debug if you want to keep them?

actually I'm fine with both, but since _get_token_paddings is also using logger.info, so I'd like to keep it as it is. In my understanding, it logs the preparation steps.

Chenyaaang · 2025-04-16T23:00:32Z

What's the impact on the execution time when splitting sample_from_hidden to two parts?

The benchmark_serving results are almost the same (qps, latency).

How does the performance compare with and without structured decoding?

I ran benchmark_serving_structured_output.py and benchmark_serving.pyon both TPU and GPU to compute performance degradation

server: VLLM_USE_V1=1 VLLM_XLA_CHECK_RECOMPILATION=1 vllm serve meta-llama/Llama-3.1-8B-Instruct --disable-log-requests --gpu-memory-utilization 0.98 --max-num-batched-tokens 2048 --max-num-seqs 128 --max-model-len 4096 --tensor-parallel-size 1 --no-enable-prefix-caching

benchmark_serving: python benchmarks/benchmark_serving.py --model meta-llama/Llama-3.1-8B-Instruct --dataset-name sonnet --dataset-path benchmarks/sonnet_4x.txt --sonnet-input-len 1101 --sonnet-output-len 1 --ignore_eos --percentile-metrics ttft,tpot,itl,e2el

benchmark_serving_structured_output: python benchmarks/benchmark_serving_structured_output.py --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --dataset choice --structured-output-ratio 1.0 --structured-output-backend auto --num-prompts 1000 --percentile-metrics ttft,tpot,itl,e2el

	w/o structured decoding	w structured decoding
H100	Qps: 38.41mean e2el: 13.16s	Qps: 33.87Mean e2el: 15.24s
V6e	Qps: 24.44mean e2el: 21.03s	Qps: 21.82Mean e2el: 23.45s

yaochengji · 2025-04-16T23:09:41Z

I ran benchmark_serving_structured_output.py and benchmark_serving.py, both using input-len 1100 and output-len 1, qps is 7.15 vs 8.19, mean e2el is 70s vs 61s

I think the performance drop is acceptable. Although there might be some performance optimization chances but we can handle it in the following PRs.

Signed-off-by: Chenyaaang <[email protected]>

yaochengji

LGTM, thanks for the contribution.

Final nit: Per the contributing guide, could you change the title to
[Core][V1][TPU] Enable structured decoding?

mergify · 2025-04-17T18:20:27Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Chenyaaang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Chenyaaang <[email protected]>

NickLucche

Nice work! Left some comments, take a look when you find the time.

vllm/v1/worker/tpu_model_runner.py

mergify · 2025-04-22T10:01:53Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Chenyaaang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Chenyaaang <[email protected]>

…ruct_output_generate to the top Signed-off-by: Chenyaaang <[email protected]>

Signed-off-by: Chenyaaang <[email protected]>

mgoin

LGTM, great work!

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Frieda (Jingying) Huang <[email protected]>

* [BugFix] Remove default multiproc executor `collective_rpc` timeout (vllm-project#17000) Signed-off-by: Nick Hill <[email protected]> * [Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499) Signed-off-by: Chenyaaang <[email protected]> * [Bugfix] validate urls object for multimodal content parts (vllm-project#16990) Signed-off-by: Guillaume Calmettes <[email protected]> * add Dockerfile build vllm against torch nightly (vllm-project#16936) Signed-off-by: Yang Wang <[email protected]> * [Kernel][ROCM] Upstream prefix prefill speed up for vLLM V1 (vllm-project#13305) Signed-off-by: Sage Moore <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Aleksandr Malyshev <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: maleksan85 <[email protected]> Signed-off-by: <> Co-authored-by: Sage Moore <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Aleksandr Malyshev <[email protected]> Co-authored-by: qli88 <[email protected]> Co-authored-by: root <[email protected]> * [V1][DP] More robust DP/EP dummy request coordination (vllm-project#16277) Signed-off-by: Nick Hill <[email protected]> * [BugFix] Revert ROCm Custom Paged Attention Env Flag Check (vllm-project#17022) Signed-off-by: vllmellm <[email protected]> * Revert "[Misc] Add S3 environment variables for better support of MinIO." (vllm-project#17021) * [misc] tune some env vars for GB200 (vllm-project#16992) Signed-off-by: youkaichao <[email protected]> * [INTEL-HPU][v0] Port delayed sampling to upstream (vllm-project#16949) Signed-off-by: Michal Adamczyk <[email protected]> Signed-off-by: Chendi Xue <[email protected]> Co-authored-by: Michal Adamczyk <[email protected]> * [doc] add download path tips (vllm-project#17013) Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]> * [Bugfix] Triton FA function takes no keyword arguments (vllm-project#16902) Signed-off-by: vllmellm <[email protected]> * [V1] Avoid socket errors during shutdown when requests are in in-flight (vllm-project#16807) Signed-off-by: Nick Hill <[email protected]> * [BugFix] llama4 fa3 fix - RuntimeError: scheduler_metadata must have shape (metadata_size) (vllm-project#16998) Signed-off-by: Lucas Wilkinson <[email protected]> * [Misc] Improve readability of get_open_port function. (vllm-project#17024) Signed-off-by: gitover22 <[email protected]> * [Bugfix] Fix AssertionError: skip_special_tokens=False is not supported for Mistral tokenizers (vllm-project#16964) Signed-off-by: chaunceyjiang <[email protected]> * [CI] Run v1/test_serial_utils.py in CI (vllm-project#16996) Signed-off-by: Russell Bryant <[email protected]> * Mistral-format support for compressed-tensors (vllm-project#16803) Signed-off-by: mgoin <[email protected]> * Categorize `tests/kernels/` based on kernel type (vllm-project#16799) Signed-off-by: mgoin <[email protected]> * [Doc] Add top anchor and a note to quantization/bitblas.md (vllm-project#17042) Signed-off-by: windsonsea <[email protected]> * Ensure that `pid` passed to `kill_process_tree` is `int` for `mypy` (vllm-project#17051) Signed-off-by: Harry Mellor <[email protected]> * [CI] Update structured-output label automation (vllm-project#17055) Signed-off-by: Russell Bryant <[email protected]> * Improve Transformers backend model loading QoL (vllm-project#17039) Signed-off-by: Harry Mellor <[email protected]> * `CacheConfig.block_size` should always be `int` when used (vllm-project#17052) Signed-off-by: Harry Mellor <[email protected]> * Use `@property` and private field for `data_parallel_rank_local` (vllm-project#17053) Signed-off-by: Harry Mellor <[email protected]> * [Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (vllm-project#15949) Signed-off-by: Travis Johnson <[email protected]> * [BugFix][V1] Fix int32 token index overflow when preparing input ids (vllm-project#16806) * [V1][Spec Decode] Always use argmax for sampling draft tokens (vllm-project#16899) Signed-off-by: Woosuk Kwon <[email protected]> * [CI/Build] workaround for CI build failure (vllm-project#17070) Signed-off-by: csy1204 <[email protected]> Co-authored-by: Michael Goin <[email protected]> * [Quantization]add prefix for commandA quantized model (vllm-project#17017) * [Minor] Use larger batch sizes for A100/B100/B200/MI300x (vllm-project#17073) Signed-off-by: Woosuk Kwon <[email protected]> * [Bugfix] Enable V1 usage stats (vllm-project#16986) Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Nick Hill <[email protected]> * More informative error when using Transformers backend (vllm-project#16988) Signed-off-by: Harry Mellor <[email protected]> * Addendum Fix to support FIPS enabled machines with MD5 hashing (vllm-project#17043) Signed-off-by: sydarb <[email protected]> * [Bugfix][Core] add seq_id_to_seq_group clearing to avoid memory leak when s… (vllm-project#16472) Signed-off-by: 开哲 <[email protected]> Co-authored-by: 开哲 <[email protected]> * [V1] Update structured output (vllm-project#16812) Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]> * [doc] update to hyperlink (vllm-project#17096) Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]> * Add docs for runai_streamer_sharded (vllm-project#17093) Signed-off-by: Omer Dayan (SW-GPU) <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> * [Chore] Remove Sampler from Model Code (vllm-project#17084) Signed-off-by: Woosuk Kwon <[email protected]> * Disable enforce_eager for V1 TPU sampler and structured output tests (vllm-project#17016) Signed-off-by: mgoin <[email protected]> * Simplify `TokenizerGroup` (vllm-project#16790) Signed-off-by: Harry Mellor <[email protected]> * Fix OOT registration test (vllm-project#17099) Signed-off-by: Harry Mellor <[email protected]> * [V1][PP] Optimization: continue scheduling prefill chunks (vllm-project#17080) Signed-off-by: Rui Qiao <[email protected]> * [Misc] Remove OLMo2 config copy (vllm-project#17066) Signed-off-by: Isotr0py <[email protected]> * Improve static type checking in `LoRAModelRunnerMixin` (vllm-project#17104) Signed-off-by: Harry Mellor <[email protected]> * [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning (vllm-project#16954) Signed-off-by: shen-shanshan <[email protected]> * [Frontend] Using matryoshka_dimensions control the allowed output dimensions. (vllm-project#16970) * Add missing rocm_skinny_gemms kernel test to CI (vllm-project#17060) Signed-off-by: mgoin <[email protected]> * [Misc] refactor example series - structured outputs (vllm-project#17040) Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]> * [V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <[email protected]> * [CI] Add automation for the `tool-calling` github label (vllm-project#17118) Signed-off-by: Russell Bryant <[email protected]> * Updating builkite job for IBM Power (vllm-project#17111) Signed-off-by: Aaruni Aggarwal <[email protected]> * existing torch installation pip command fix for docs (vllm-project#17059) * Molmo Requirements (vllm-project#17026) Signed-off-by: Eyshika Agarwal <[email protected]> Signed-off-by: eyshika <[email protected]> * Add `:markdownhelp:` to `EngineArgs` docs so markdown docstrings render properly (vllm-project#17124) Signed-off-by: Harry Mellor <[email protected]> * Improve configs - `LoRAConfig` + `PromptAdapterConfig` (vllm-project#16980) Signed-off-by: Harry Mellor <[email protected]> * [Docs] Generate correct github links for decorated functions (vllm-project#17125) Signed-off-by: Russell Bryant <[email protected]> * Add collective_rpc to llm engine (vllm-project#16999) Signed-off-by: Yinghai Lu <[email protected]> * Add chat template for Llama 4 models (vllm-project#16428) Signed-off-by: Max de Bayser <[email protected]> * [Misc] Add example to run DeepSeek with Ray Serve LLM (vllm-project#17134) Signed-off-by: Rui Qiao <[email protected]> * Better error message for missing mistral params.json (vllm-project#17132) Signed-off-by: mgoin <[email protected]> * Use custom address for listening socket (vllm-project#15988) Signed-off-by: Jens Glaser <[email protected]> * [FEAT] [ROCm]: AITER Fused MOE V1 Support (vllm-project#16752) Signed-off-by: vllmellm <[email protected]> Co-authored-by: tjtanaa <[email protected]> * [Attention] FA3 decode perf improvement - single mma warp group support for head dim 128 (vllm-project#16864) Signed-off-by: Lucas Wilkinson <[email protected]> * fix float16 support for kimi-vl (vllm-project#17156) Co-authored-by: zhouzaida <[email protected]> * [Doc] V1 : Update LoRA status (vllm-project#17133) Signed-off-by: varun sundar rabindranath <[email protected]> Co-authored-by: varun sundar rabindranath <[email protected]> * [Docs] Fix True->true in supported_models.md (vllm-project#17141) * Move missed `SchedulerConfig` args into scheduler config group in `EngineArgs` (vllm-project#17131) Signed-off-by: Harry Mellor <[email protected]> * [Misc] Clean up redundant code in uniproc_executor.py (vllm-project#16762) Signed-off-by: Lifu Huang <[email protected]> * [Bugfix][Misc] Use TritonPlaceholderModule to defensively import triton (vllm-project#15099) Signed-off-by: Mengqing Cao <[email protected]> * [Misc] Benchmark Serving Script Support Appending Results (vllm-project#17028) Signed-off-by: Lucas Wilkinson <[email protected]> * [Perf]Optimize rotary_emb implementation to use Triton operator for improved inference performance (vllm-project#16457) Signed-off-by: cynthieye <[email protected]> Co-authored-by: MagnetoWang <[email protected]> * [Bugfix] remove fallback in guided_json (int range, patterns) (vllm-project#16725) Signed-off-by: csy1204 <[email protected]> Co-authored-by: 조상연[플레이스 AI] <[email protected]> * [Quantization][FP8] Add support for FP8 models with input_scale for output projection and QK quantization (vllm-project#15734) Signed-off-by: Randall Smith <[email protected]> Signed-off-by: Luka Govedič <[email protected]> Co-authored-by: Luka Govedič <[email protected]> * [Doc] Add headings to improve gptqmodel.md (vllm-project#17164) Signed-off-by: windsonsea <[email protected]> * Only turn on FastIncrementalDetokenizer when tokenizers >= 0.21.1 (vllm-project#17158) * [Doc] Add two links to disagg_prefill.md (vllm-project#17168) Signed-off-by: windsonsea <[email protected]> * [Doc] Move todo out of beam search docstring (vllm-project#17183) Signed-off-by: Alex-Brooks <[email protected]> * [Bugfix] Fix mistral model tests (vllm-project#17181) Signed-off-by: DarkLight1337 <[email protected]> * [Bugfix] Fix Mistral ChatCompletionRequest Body Exception (vllm-project#16769) Signed-off-by: Jasmond Loh <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> * Fix API typo and remove FP8 on V1 restriction --------- Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Guillaume Calmettes <[email protected]> Signed-off-by: Yang Wang <[email protected]> Signed-off-by: Sage Moore <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Aleksandr Malyshev <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: maleksan85 <[email protected]> Signed-off-by: <> Signed-off-by: vllmellm <[email protected]> Signed-off-by: youkaichao <[email protected]> Signed-off-by: Michal Adamczyk <[email protected]> Signed-off-by: Chendi Xue <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: gitover22 <[email protected]> Signed-off-by: chaunceyjiang <[email protected]> Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: windsonsea <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: Travis Johnson <[email protected]> Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: csy1204 <[email protected]> Signed-off-by: sydarb <[email protected]> Signed-off-by: 开哲 <[email protected]> Signed-off-by: Omer Dayan (SW-GPU) <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: shen-shanshan <[email protected]> Signed-off-by: Mark McLoughlin <[email protected]> Signed-off-by: Aaruni Aggarwal <[email protected]> Signed-off-by: Eyshika Agarwal <[email protected]> Signed-off-by: eyshika <[email protected]> Signed-off-by: Yinghai Lu <[email protected]> Signed-off-by: Max de Bayser <[email protected]> Signed-off-by: Jens Glaser <[email protected]> Signed-off-by: varun sundar rabindranath <[email protected]> Signed-off-by: Lifu Huang <[email protected]> Signed-off-by: Mengqing Cao <[email protected]> Signed-off-by: cynthieye <[email protected]> Signed-off-by: Randall Smith <[email protected]> Signed-off-by: Luka Govedič <[email protected]> Signed-off-by: Alex-Brooks <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Jasmond Loh <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Chenyaaang <[email protected]> Co-authored-by: Guillaume Calmettes <[email protected]> Co-authored-by: Yang Wang <[email protected]> Co-authored-by: Aleksandr Malyshev <[email protected]> Co-authored-by: Sage Moore <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Aleksandr Malyshev <[email protected]> Co-authored-by: qli88 <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: vllmellm <[email protected]> Co-authored-by: Chauncey <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Chendi.Xue <[email protected]> Co-authored-by: Michal Adamczyk <[email protected]> Co-authored-by: Reid <[email protected]> Co-authored-by: reidliu41 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: huafeng <[email protected]> Co-authored-by: Russell Bryant <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Michael Yao <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: Travis Johnson <[email protected]> Co-authored-by: Yong Hoon Shin <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]> Co-authored-by: Sangyeon Cho <[email protected]> Co-authored-by: Chen Xia <[email protected]> Co-authored-by: Areeb Syed <[email protected]> Co-authored-by: 张宇 <[email protected]> Co-authored-by: 开哲 <[email protected]> Co-authored-by: omer-dayan <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Rui Qiao <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: Shanshan Shen <[email protected]> Co-authored-by: wang.yuqi <[email protected]> Co-authored-by: Mark McLoughlin <[email protected]> Co-authored-by: Aaruni Aggarwal <[email protected]> Co-authored-by: Atilla <[email protected]> Co-authored-by: Eyshika Agarwal <[email protected]> Co-authored-by: Yinghai Lu <[email protected]> Co-authored-by: Maximilien de Bayser <[email protected]> Co-authored-by: jglaser <[email protected]> Co-authored-by: tjtanaa <[email protected]> Co-authored-by: Zaida Zhou <[email protected]> Co-authored-by: zhouzaida <[email protected]> Co-authored-by: Varun Sundar Rabindranath <[email protected]> Co-authored-by: varun sundar rabindranath <[email protected]> Co-authored-by: Lifu Huang <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: yexin(叶鑫) <[email protected]> Co-authored-by: MagnetoWang <[email protected]> Co-authored-by: 조상연[플레이스 AI] <[email protected]> Co-authored-by: rasmith <[email protected]> Co-authored-by: Luka Govedič <[email protected]> Co-authored-by: Lu Fang <[email protected]> Co-authored-by: Alex Brooks <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Jasmond L <[email protected]>

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Zijing Liu <[email protected]>

) Signed-off-by: Chenyaaang <[email protected]>

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Agata Dobrzyniewicz <[email protected]>

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Mu Huai <[email protected]>

add structured decoding to tpu model runner

ceb6eb3

Signed-off-by: Chenyaaang <[email protected]>

Chenyaaang requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners April 11, 2025 16:24

mergify bot added ci/build v1 tpu Related to Google TPUs labels Apr 11, 2025

mergify bot added the needs-rebase label Apr 11, 2025

Merge remote-tracking branch 'origin/main' into tpu-structured-decoding1

bc6dddb

Signed-off-by: Chenyaaang <[email protected]>

mergify bot removed the needs-rebase label Apr 11, 2025

mgoin self-requested a review April 11, 2025 21:19

yaochengji reviewed Apr 12, 2025

View reviewed changes

vllm/v1/worker/tpu_model_runner.py Show resolved Hide resolved

vllm/v1/worker/tpu_model_runner.py Outdated Show resolved Hide resolved

vllm/v1/worker/tpu_model_runner.py Outdated Show resolved Hide resolved

russellb added the structured-output label Apr 16, 2025

russellb reviewed Apr 16, 2025

View reviewed changes

Chenyaaang closed this Apr 16, 2025

Chenyaaang reopened this Apr 16, 2025

Chenyaaang force-pushed the tpu-structured-decoding1 branch from c0774ba to 48adba8 Compare April 17, 2025 00:06

fix comments

2a8c28b

Signed-off-by: Chenyaaang <[email protected]>

Chenyaaang force-pushed the tpu-structured-decoding1 branch from 48adba8 to 2a8c28b Compare April 17, 2025 00:11

yaochengji approved these changes Apr 17, 2025

View reviewed changes

Chenyaaang changed the title ~~[Core] Enable structured decoding on V1 TPU~~ [Core][V1][TPU] Enable structured decoding on TPU V1 Apr 17, 2025

mergify bot added the needs-rebase label Apr 17, 2025

fix lint error

809073a

Signed-off-by: Chenyaaang <[email protected]>

yaochengji added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 19, 2025

fix tpu test

dc169ed

Signed-off-by: Chenyaaang <[email protected]>

WoosukKwon requested a review from russellb April 21, 2025 16:07

Merge remote-tracking branch 'origin/main' into tpu-structured-decoding1

8b9ea03

NickLucche suggested changes Apr 22, 2025

View reviewed changes

vllm/v1/worker/tpu_model_runner.py Outdated Show resolved Hide resolved

vllm/v1/worker/tpu_model_runner.py Outdated Show resolved Hide resolved

vllm/v1/worker/tpu_model_runner.py Outdated Show resolved Hide resolved

vllm/v1/worker/tpu_model_runner.py Outdated Show resolved Hide resolved

mergify bot added the needs-rebase label Apr 22, 2025

NickLucche mentioned this pull request Apr 22, 2025

debug test sampler inconsistency #16939

Closed

loosen test_sampler

fe043e4

Signed-off-by: Chenyaaang <[email protected]>

Chenyaaang force-pushed the tpu-structured-decoding1 branch from 7309f4a to fe043e4 Compare April 22, 2025 17:01

mergify bot removed the needs-rebase label Apr 22, 2025

fix comments

24a3f9f

Signed-off-by: Chenyaaang <[email protected]>

russellb added this to Structured Output Apr 22, 2025

Chenyaaang added 2 commits April 22, 2025 20:06

update tpu model runner to include slidingwindowSpec and move test_st…

d882f6e

…ruct_output_generate to the top Signed-off-by: Chenyaaang <[email protected]>

move test_struct_output_generate back to bottom

f152c0a

Signed-off-by: Chenyaaang <[email protected]>

mgoin approved these changes Apr 22, 2025

View reviewed changes

mgoin merged commit 83d9337 into vllm-project:main Apr 23, 2025
43 checks passed

github-project-automation bot moved this to Done in Structured Output Apr 23, 2025

frieda-huang pushed a commit to frieda-huang/vllm that referenced this pull request Apr 23, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

ffd2e3d

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Frieda (Jingying) Huang <[email protected]>

russellb mentioned this pull request Apr 23, 2025

[Feature]: guided decoding on TPU #11104

Closed

1 task

liuzijing2014 pushed a commit to liuzijing2014/vllm that referenced this pull request Apr 25, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

5cca099

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Zijing Liu <[email protected]>

liuzijing2014 pushed a commit to liuzijing2014/vllm that referenced this pull request Apr 25, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

07f611c

) Signed-off-by: Chenyaaang <[email protected]>

wuisawesome pushed a commit to character-tech/vllm that referenced this pull request Apr 28, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

77db7e2

) Signed-off-by: Chenyaaang <[email protected]>

jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Apr 29, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

06fa94f

) Signed-off-by: Chenyaaang <[email protected]>

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

9351a59

) Signed-off-by: Chenyaaang <[email protected]>

adobrzyn pushed a commit to HabanaAI/vllm-fork that referenced this pull request Apr 30, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

46297cb

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Agata Dobrzyniewicz <[email protected]>

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

[Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499

3590331

) Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: Mu Huai <[email protected]>

ckhordiasma mentioned this pull request May 14, 2025

nm vllm ent 0.8.5 sync red-hat-data-services/vllm#139

Merged

		logger.info(
		"Compiling structured_decoding with different input shapes.")

Uh oh!

[Core][V1][TPU] Enable structured decoding on TPU V1 #16499

[Core][V1][TPU] Enable structured decoding on TPU V1 #16499

Uh oh!

Conversation

Chenyaaang commented Apr 11, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pre-compile time

How to run structured deocoding

unit test

benchmark

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

mergify bot commented Apr 11, 2025

Uh oh!

yaochengji left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

russellb Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chenyaaang Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chenyaaang commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaochengji commented Apr 16, 2025

Uh oh!

yaochengji left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Apr 17, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Apr 22, 2025

Uh oh!

mgoin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Chenyaaang commented Apr 11, 2025 •

edited by github-actions bot

Loading

russellb Apr 16, 2025 •

edited

Loading

Chenyaaang Apr 16, 2025 •

edited

Loading

Chenyaaang commented Apr 16, 2025 •

edited

Loading

yaochengji left a comment •

edited

Loading