Vol. 6 No. 2 (2026): July−December 2026
Articles

Post-hoc Calibration of Classical and Shallow Neural Classifiers for Handwritten Digit Recognition: An Empirical Comparison under Limited Calibration Data

View PDF
  • Yinchu Wu,
  • Jiale Gao
Yinchu Wu
Computer Technology, Shanghai Jiao Tong University, Shanghai, China
Jiale Gao
Applied Analytics, Columbia University, New York, NY, USA

Published 2026-09-25

Keywords

  • Confidence calibration,
  • Handwritten digit recognition,
  • Post-hoc calibration,
  • Classifier comparison

How to Cite

Wu, Y., & Gao, J. (2026). Post-hoc Calibration of Classical and Shallow Neural Classifiers for Handwritten Digit Recognition: An Empirical Comparison under Limited Calibration Data. Spectrum of Research, 6(2). Retrieved from https://spectrumofresearch.com/index.php/sr/article/view/66

Copyright (c) 2026 Spectrum of Research

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Post-hoc calibration is the standard remedy for unreliable classifier confidence, yet its evidence base comes mainly from deep networks with large calibration sets. This paper compares five post-hoc maps (temperature scaling, vector scaling, regularized matrix scaling, histogram binning and isotonic regression) applied to eight classical and shallow neural classifiers on the UCI handwritten digits data, with calibration budgets of 5 to 45 images per class 20 stratified splits and paired signed-rank tests over six calibration measures. Before calibration, accuracy lies between 91.88% and 97.55%, while the expected calibration error ranges from 2.21 to 38.24 percentage points: support vector machines and random forests are strongly underconfident, and Gaussian naive Bayes is overconfident. Temperature scaling achieves the best average rank over six measures at 5, 10 and 20 images per class and the lowest negative log-likelihood at every budget. Regularized matrix scaling ranks first only at 45 images per class, and one-vs-rest histogram binning lowers accuracy by about seven points. The results indicate that a single temperature is the safest post-hoc choice for small recognizers when labeled calibration data are scarce, while richer maps pay off only with larger budgets.

View PDF

References

  1. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. https://doi.org/10.1109/5.726791
  2. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330). PMLR. https://proceedings.mlr.press/v70/guo17a.html
  3. Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., & Lucic, M. (2021). Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems (Vol. 34). Curran Associates. https://proceedings.neurips.cc/paper/2021/hash/8420d359404024567b5aefda1231af24-Abstract.html
  4. Galil, I., Dabbah, M., & El-Yaniv, R. (2023). What can we learn from the selective prediction and uncertainty estimation performance of 523 ImageNet classifiers? In International Conference on Learning Representations. https://openreview.net/forum?id=p66AzKi6Xim
  5. Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 29, No. 1). AAAI Press.
  6. Gupta, K., Rahimi, A., Ajanthan, T., Mensink, T., Sminchisescu, C., & Hartley, R. (2021). Calibration of neural networks using splines. In International Conference on Learning Representations. https://openreview.net/forum?id=eQe8DEWNN2W
  7. Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 694–699). ACM.
  8. Kull, M., Perello Nieto, M., Kängsepp, M., Silva Filho, T., Song, H., & Flach, P. (2019). Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In Advances in Neural Information Processing Systems (Vol. 32). Curran Associates. https://proceedings.neurips.cc/paper/2019/hash/8ca01ea920679a0fe3728441494041b9-Abstract.html
  9. Patel, K., Beluch, W. H., Yang, B., Pfeiffer, M., & Zhang, D. (2021). Multi-class uncertainty calibration via mutual information maximization-based binning. In International Conference on Learning Representations. https://arxiv.org/abs/2006.13092
  10. Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., & Dokania, P. K. (2020). Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems (Vol. 33, pp. 15288–15299). Curran Associates.
  11. Müller, R., Kornblith, S., & Hinton, G. E. (2019). When does label smoothing help? In Advances in Neural Information Processing Systems (Vol. 32). Curran Associates. https://arxiv.org/abs/1906.02629
  12. Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., & Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems (Vol. 32). Curran Associates. https://arxiv.org/abs/1905.11001
  13. Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (pp. 625–632). ACM. https://doi.org/10.1145/1102351.1102430
  14. Alpaydin, E., & Kaynak, C. (1998). Optical recognition of handwritten digits [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C50P49
  15. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
  16. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems (Vol. 30, pp. 3146–3154). Curran Associates.
  17. Wang, D.-B., Feng, L., & Zhang, M.-L. (2021). Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. In Advances in Neural Information Processing Systems (Vol. 34, pp. 11809–11820). Curran Associates.
  18. Gupta, C., & Ramdas, A. (2022). Top-label calibration and multiclass-to-binary reductions. In International Conference on Learning Representations. https://arxiv.org/abs/2107.08353
  19. Tao, L., Zhu, Y., Guo, H., Dong, M., & Xu, C. (2024). A benchmark study on calibration. In International Conference on Learning Representations. https://openreview.net/forum?id=GzNhzX9kVa
  20. Rahimi, A., Shaban, A., Cheng, C.-A., Hartley, R., & Boots, B. (2020). Intra order-preserving functions for calibration of multi-class neural networks. In Advances in Neural Information Processing Systems (Vol. 33, pp. 13456–13467). Curran Associates. https://papers.nips.cc/paper/2020/hash/9bc99c590be3511b8d53741684ef574c-Abstract.html
  21. Kumar, A., Liang, P., & Ma, T. (2019). Verified uncertainty calibration. In Advances in Neural Information Processing Systems (Vol. 32). Curran Associates. https://proceedings.neurips.cc/paper/2019/hash/f8c0c968632845cd133308b1a494967f-Abstract.html