Vol. 6 No. 2 (2026): July−December 2026
Articles

Quantifying the Throughput–Tail-Latency Trade-off of Continuous Batching and Chunked-Prefill Scheduling under Production LLM Traces: An Empirical Comparison

View PDF
  • Yixin Zheng,
  • Ruming Tang
Yixin Zheng
Information Science, University of Michigan, Ann Arbor, MI, USA
Ruming Tang
Data Science, Fudan University, Shanghai, China

Published 2026-07-07

Keywords

  • LLM inference serving,
  • continuous batching,
  • chunked prefill,
  • tail latency

How to Cite

Zheng, Y., & Tang, R. (2026). Quantifying the Throughput–Tail-Latency Trade-off of Continuous Batching and Chunked-Prefill Scheduling under Production LLM Traces: An Empirical Comparison. Spectrum of Research, 6(2). Retrieved from https://spectrumofresearch.com/index.php/sr/article/view/58

Copyright (c) 2026 Spectrum of Research

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Interactive large language model (LLM) services must satisfy tight latency targets while sustaining high request throughput, and the batching and scheduling policy of the serving engine governs where a deployment lands on this trade-off. This work reports a controlled empirical comparison of three scheduling and memory factors—static versus continuous (iteration-level) batching, chunked-prefill chunk size, and key-value (KV) cache block size—measured on a single accelerator under request traces drawn from production LLM services. We replay arrival timestamps together with prompt and generation lengths from public Azure and BurstGPT traces, and we report time-to-first-token (TTFT), time-per-output-token (TPOT), tail percentiles, and service-level-objective (SLO) attainment. Continuous batching raises sustained throughput from 4.2 to 11.0 requests per second while lowering P99 TTFT from 2,150 to 890 ms relative to static batching. Chunked prefill with a 512-token chunk lowers P99 TPOT by 40.6% at a moderate 16.7% increase in median TTFT, exposing a tunable latency balance. KV block size shows modest sensitivity, with a 16-token block giving the best measured throughput. Under bursty arrivals the combined configuration attains 88.9% SLO compliance. The study quantifies practical operating points rather than proposing new machinery.

View PDF

References

  1. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI '22) (pp. 521–538). USENIX Association.
  2. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP '23) (pp. 611–626). ACM.
  3. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., & Ramjee, R. (2024). Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) (pp. 117–134). USENIX Association.
  4. Wang, Y., Chen, Y., Li, Z., Kang, X., Fang, Y., Zhou, Y., Zheng, Y., Tang, Z., He, X., Guo, R., Wang, X., Wang, Q., Zhou, A. C., & Chu, X. (2025). BurstGPT: A real-world workload dataset to optimize LLM serving systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '25) (pp. 5831–5841). ACM.
  5. Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI '23) (pp. 663–679). USENIX Association.
  6. Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., & Lin, W. (2024). Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) (pp. 173–191). USENIX Association.
  7. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., & Chen, B. (2023). H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).
  8. Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2024). Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR 2024).
  9. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., & Zhang, H. (2024). DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) (pp. 193–210). USENIX Association.
  10. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., & Sheng, Y. (2024). SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) (pp. 62557–62583).
  11. Patel, P., Choukse, E., Zhang, C., Goiri, Í., Shah, A., Maleki, S., & Bianchini, R. (2024). Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA '24) (pp. 118–132). IEEE.
  12. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., & Zhang, H. (2024). LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset. In International Conference on Learning Representations (ICLR 2024).
  13. Jin, Y., Wu, C.-F., Brooks, D., & Wei, G.-Y. (2023). S3: Increasing GPU utilization during generative inference for higher throughput. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).
  14. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022).
  15. Wu, B., Zhong, Y., Zhang, Z., Liu, S., Liu, F., Sun, Y., Huang, G., Liu, X., & Jin, X. (2023). Fast distributed inference serving for large language models. arXiv. https://arxiv.org/abs/2305.05920
  16. Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., & Panwar, A. (2025). vAttention: Dynamic memory management for serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '25) (pp. 1133–1150). ACM.
  17. Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., & Xu, X. (2025). Mooncake: Trading more storage for less computation — A KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST '25) (pp. 155–170). USENIX Association.