Vol. 6 No. 2 (2026): July−December 2026
Articles

Road-Geometry, Scenario-Diversity, and Failure-History Prioritization for BeamNG Lane-Keeping Simulation Regression Tests under Fixed Simulation-Time Budgets: An Empirical Comparison

View PDF
  • Jianuo Wang,
  • Yi Guo
Jianuo Wang
Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA
Yi Guo
Computer and Information Science, University of Pennsylvania, PA, USA

Published 2026-08-24

Keywords

  • autonomous-driving testing,
  • regression-test prioritization,
  • simulation-time budget,
  • failure history

How to Cite

Wang, J., & Guo, Y. (2026). Road-Geometry, Scenario-Diversity, and Failure-History Prioritization for BeamNG Lane-Keeping Simulation Regression Tests under Fixed Simulation-Time Budgets: An Empirical Comparison. Spectrum of Research, 6(2). Retrieved from https://spectrumofresearch.com/index.php/sr/article/view/59

Copyright (c) 2026 Spectrum of Research

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Simulation enables repeatable testing of automated driving functions, yet continuous regression campaigns may contain more executable scenarios than a development cycle can afford. This study evaluates whether inexpensive pre-execution information can allocate a fixed simulation-time budget toward failure-revealing BeamNG lane-keeping tests. The evaluation uses 5,638 complete records from the TRAVEL SDC-Prioritizer archive, including 2,543 unsafe executions and 69.96 hours of recorded simulation. A leakage-controlled rolling protocol evaluates 4,641 tests on four future dates after fitting history-dependent methods only on earlier records. Five prioritization strategies are compared at 10%, 25%, and 50% of each fold's execution time: random ordering, a road-geometry score, farthest-first geometric diversity, failure probability learned from prior executions, and failure probability divided by predicted duration. At the 25% budget, random ordering recalls 25.0% of unsafe tests, geometry recalls 33.9%, history recalls 43.3%, and cost-aware history recalls 44.2%; diversity recalls 21.3%. The advantage of cost adjustment over unadjusted history is only 0.9 percentage points. Exact paired tests across four temporal folds do not establish statistical significance after multiplicity correction. The evidence supports historical failure information as a useful scheduling signal for this archive, while exposing limited sample size, a single safety oracle, and the absence of consecutive software revisions.

View PDF

References

  1. Birchler, C., Khatiri, S., Derakhshanfar, P., Panichella, S., & Panichella, A. (2023). Single and multi-objective test cases prioritization for self-driving cars in virtual environments. ACM Transactions on Software Engineering and Methodology, 32(2), 1-30. https://doi.org/10.1145/3533818
  2. Birchler, C., Khatiri, S., Bosshard, B., Gambi, A., & Panichella, S. (2023). Machine learning-based test selection for simulation-based testing of self-driving cars software. Empirical Software Engineering, 28(3), Article 71. https://doi.org/10.1007/s10664-023-10286-y
  3. Arcaini, P., & Cetinkaya, A. (2026). DETOUR: A tool for regression testing of autonomous driving systems. Science of Computer Programming, 253, Article 103490. https://doi.org/10.1016/j.scico.2026.103490
  4. Birchler, C., Rohrbach, C., Kim, H., Gambi, A., Liu, T., Horneber, J., Kehrer, T., & Panichella, S. (2023). TEASER: Simulation-based CAN bus regression testing for self-driving cars software. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (pp. 2058-2061). IEEE. https://doi.org/10.1109/ASE56229.2023.00154
  5. Birchler, C., Klikovits, S., Fazzini, M., & Panichella, S. (2025). ICST Tool Competition 2025-Self-Driving Car Testing Track. In 2025 IEEE Conference on Software Testing, Verification and Validation (pp. 801-804). IEEE. https://doi.org/10.1109/ICST62969.2025.10989045
  6. Grano, G., Laaber, C., Panichella, A., & Panichella, S. (2021). Testing with fewer resources: An adaptive approach to performance-aware test case generation. IEEE Transactions on Software Engineering, 47(11), 2332-2347. https://doi.org/10.1109/TSE.2019.2946773
  7. Arrieta, A., Wang, S., Arruabarrena, A., Markiegi, U., Sagardui, G., & Etxeberria, L. (2018). Multi-objective black-box test case selection for cost-effectively testing simulation models. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 1411-1418). ACM. https://doi.org/10.1145/3205455.3205490
  8. Arrieta, A., Wang, S., Sagardui, G., & Etxeberria, L. (2019). Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. Journal of Systems and Software, 149, 1-34. https://doi.org/10.1016/j.jss.2018.09.055
  9. Luo, Q., Moran, K., & Poshyvanyk, D. (2016). A large-scale empirical comparison of static and dynamic test case prioritization techniques. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 559-570). ACM. https://doi.org/10.1145/2950290.2950344
  10. Rocklage, E., Kraft, H., Karatas, A., & Seewig, J. (2017). Automated scenario generation for regression testing of autonomous vehicles. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (pp. 476-483). IEEE. https://doi.org/10.1109/ITSC.2017.8317919
  11. Gambi, A., Müller, M., & Fraser, G. (2019). Automatically testing self-driving cars with search-based procedural content generation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 318-328). ACM. https://doi.org/10.1145/3293882.3330566
  12. Koren, M., Alsaif, S., Lee, R., & Kochenderfer, M. J. (2018). Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (pp. 1-7). IEEE. https://doi.org/10.1109/IVS.2018.8500400
  13. Li, G., Li, Y., Jha, S., Tsai, T., Sullivan, M. B., Hari, S. K. S., Kalbarczyk, Z., & Iyer, R. (2020). AV-FUZZER: Finding safety violations in autonomous driving systems. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (pp. 25-36). IEEE. https://doi.org/10.1109/ISSRE5003.2020.00012
  14. Birchler, C., Ganz, N., Khatiri, S., Gambi, A., & Panichella, S. (2022). Cost-effective simulation-based test selection in self-driving cars software with SDC-Scissor. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (pp. 164-168). IEEE. https://doi.org/10.1109/SANER53432.2022.00030
  15. Huai, Y., Almanee, S., Chen, Y., Wu, X., Chen, Q. A., & Garcia, J. (2023). scenoRITA: Generating diverse, fully mutable, test scenarios for autonomous vehicle planning. IEEE Transactions on Software Engineering, 49(10), 4656-4676. https://doi.org/10.1109/TSE.2023.3309610
  16. Corso, A., Du, P., Driggs-Campbell, K., & Kochenderfer, M. J. (2019). Adaptive stress testing with reward augmentation for autonomous vehicle validation. In 2019 IEEE Intelligent Transportation Systems Conference (pp. 163-168). IEEE. https://doi.org/10.1109/ITSC.2019.8917242
  17. Kim, S., Liu, M., Rhee, J. J., Jeon, Y., Kwon, Y., & Kim, C. H. (2022). DriveFuzz: Discovering autonomous driving bugs through driving quality-guided fuzzing. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (pp. 1753-1767). ACM. https://doi.org/10.1145/3548606.3560558
  18. Sun, Y., Poskitt, C. M., Sun, J., Chen, Y., & Yang, Z. (2022). LawBreaker: An approach for specifying traffic laws and fuzzing autonomous vehicles. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (pp. 1-12). ACM. https://doi.org/10.1145/3551349.3556897
  19. Zhong, Z., Kaiser, G., & Ray, B. (2023). Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles. IEEE Transactions on Software Engineering, 49(4), 1860-1875. https://doi.org/10.1109/TSE.2022.3195640
  20. Xu, C., Ding, W., Lyu, W., Liu, Z., Wang, S., He, Y., Hu, H., Zhao, D., & Li, B. (2022). SafeBench: A benchmarking platform for safety evaluation of autonomous vehicles. Advances in Neural Information Processing Systems, 35, 25667-25682. https://doi.org/10.52202/068431-1861
  21. Pronovost, E., Ganesina, M. R., Hendy, N., Wang, Z., Morales, A., Wang, K., & Roy, N. (2023). Scenario diffusion: Controllable driving scenario generation with diffusion. Advances in Neural Information Processing Systems, 36, 68873-68894. https://doi.org/10.52202/075280-3014
  22. Liu, Y., Peng, Z., Cui, X., & Zhou, B. (2025). Adv-BMT: Bidirectional motion transformer for safety-critical traffic scenario generation. Advances in Neural Information Processing Systems, 38, 61573-61598. https://doi.org/10.52202/085713-1849
  23. Liu, J., Guo, Y., Zhong, F., Zhang, T., Jing, Z., Liang, S., Wang, J., Zhang, M., Liu, A., & Liu, X. (2026). Adversarial generation and collaborative evolution of safety-critical scenarios for autonomous vehicles. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45), 38926-38934. https://doi.org/10.1609/aaai.v40i45.41238
  24. Huang, Y., Chen, Y., Zhang, R., Sun, Y., Ruan, S., Wu, Z., Dong, Y., & Wei, X. (2026). Any2Critical: Safety-critical scenario generation from arbitrary real-world driving contexts. Proceedings of the AAAI Conference on Artificial Intelligence, 40(42), 35509-35517. https://doi.org/10.1609/aaai.v40i42.40861
  25. Gambi, A., Huynh, T., & Fraser, G. (2019). Generating effective test cases for self-driving cars from police reports. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 257-267). ACM. https://doi.org/10.1145/3338906.3338942