Vol. 6 No. 1 (2026): January−June 2026
Articles

Position Bias and Verbosity Bias in LLM-as-a-Judge: An Empirical Cross-Benchmark Comparison

View PDF
  • Hongbo Wang
Hongbo Wang
Computer Science, University of Southern California, CA, USA

Published 2026-06-24

Keywords

  • LLM-as-a-Judge,
  • position bias,
  • verbosity bias,
  • evaluation reliability

How to Cite

Wang, H. (2026). Position Bias and Verbosity Bias in LLM-as-a-Judge: An Empirical Cross-Benchmark Comparison. Spectrum of Research, 6(1). Retrieved from https://spectrumofresearch.com/index.php/sr/article/view/53

Copyright (c) 2026 Spectrum of Research

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Using large language models as automatic judges has become a routine substitute for costly human evaluation, yet the reliability of these judges is rarely examined with the same rigor applied to the systems they assess. This study consolidates and compares published evidence on two pervasive distortions, position bias and verbosity bias, across five public meta-evaluation benchmarks, namely MT-Bench, FairEval, LLMBar, RewardBench, and JudgeBench. Rather than introducing a new evaluator, the work re-organizes reported results under a single set of reliability metrics: position consistency, conflict rate, length-controlled agreement, and robustness rate. The consolidated picture shows that position bias is systematic rather than random, that it is modulated more strongly by the quality gap between candidates than by their length, and that verbosity bias inflates agreement with human preference until length is explicitly controlled. Length control raises the rank correlation between an automatic benchmark and a crowd-sourced arena from 0.94 to 0.98, while adversarial instruction-following subsets reduce some judge configurations to near-random accuracy. The comparison indicates that no single debiasing strategy is universally sufficient, and that pairing order-swapped double scoring with length control and explicit robustness reporting offers the most dependable practice. The synthesis clarifies where automatic judging can be trusted and where it still demands caution before its verdicts are propagated to downstream model selection.

View PDF

References

  1. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track.
  2. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), 2511–2522.
  3. Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. Proceedings of the First Conference on Language Modeling (COLM 2024).
  4. Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2024). Judging the judges: A systematic study of position bias in LLM-as-a-judge. arXiv preprint arXiv:2406.07791.
  5. Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Long Papers, 9440–9450.
  6. Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., & Seo, M. (2024). Prometheus: Inducing fine-grained evaluation capability in language models. Proceedings of the 12th International Conference on Learning Representations (ICLR 2024).
  7. Wang, Y., Yu, Z., Yao, W., Zeng, Z., Yang, L., Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., Ye, W., Zhang, S., & Zhang, Y. (2024). PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. Proceedings of the 12th International Conference on Learning Representations (ICLR 2024).
  8. Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024). Benchmarking cognitive biases in large language models as evaluators. Findings of the Association for Computational Linguistics: ACL 2024, 517–545.
  9. Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., Chawla, N. V., & Zhang, X. (2025). Justice or prejudice? Quantifying biases in LLM-as-a-judge. Proceedings of the 13th International Conference on Learning Representations (ICLR 2025).
  10. Chen, G. H., Chen, S., Liu, Z., Jiang, F., & Wang, B. (2024). Humans or LLMs as the judge? A study on judgement bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), 8301–8327.
  11. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., & Chen, D. (2024). Evaluating large language models at evaluating instruction following. Proceedings of the 12th International Conference on Learning Representations (ICLR 2024).
  12. Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., & Hajishirzi, H. (2025). RewardBench: Evaluating reward models for language modeling. Findings of the Association for Computational Linguistics: NAACL 2025, 1755–1797.
  13. Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., & Stoica, I. (2025). JudgeBench: A benchmark for evaluating LLM-based judges. Proceedings of the 13th International Conference on Learning Representations (ICLR 2025)