Threshold-based Abstention for Indonesian Hoax Claim Verification using IndoNLI

Arrizal Cahyadi, Harits Ar Rosyid, M. Zainal Arifin

Abstract


Automated hoax claim verification systems need to abstain from making predictions when the available evidence is insufficient or inconclusive. However, Indonesian-language Natural Language Inference (NLI) models generally produce predictions for every input. This study investigates whether threshold-based abstention using the maximum softmax probability can improve the reliability of NLI predictions. An Indonesian RoBERTa model was trained on the IndoNLI dataset using three random seeds, with the abstention threshold calibrated on the validation set to achieve a target coverage level. Evaluation was conducted on the IndoNLI test_lay and test_expert sets, as well as 758 claim pairs collected from TurnBackHoax (270 hoax claims labeled as contradiction) and Antara (488 valid news claims labeled as entailment). At a target coverage of 90%, accuracy among the retained predictions increased by 4.32 ± 0.48 percentage points on test_lay (from 73.39% to 77.71%) and by 3.60 ± 0.28 percentage points on test_expert (from 60.27% to 63.87%). In the case study, accuracy without abstention was only 62.14 ± 9.78% (ranging from 55.41% to 73.35% across seeds), which was below the majority-class baseline of 64.4%. Abstention increased accuracy by 7.36 ± 1.98 percentage points at an actual coverage of 71.5% and rejected 49% to 65% of hoax claims incorrectly predicted as entailment. However, 39% to 65% of misclassified hoax claims still passed through the system without being rejected. The hypothesis-only baseline achieved 59.3 ± 0.3% accuracy on the IndoNLI validation set, compared with 36.7% for majority-class prediction. These findings indicate that threshold-based abstention improves prediction accuracy on IndoNLI but remains insufficiently reliable for direct deployment in real-world hoax claim verification. The case study labels were derived from the source articles and did not include the neutral class; therefore, the findings should not be generalized beyond this evaluation setting.

Keywords


abstention; confidence threshold; hoax claim verification; IndoNLI; natural language inference

Full Text:

PDF

References


F. Rochman, “Kemkomdigi Identifikasi 1.923 Konten Hoaks Sepanjang 2024,” ANTARA News Gorontalo, Jan. 8, 2025. [Online]. Available:

https://gorontalo.antaranews.com/berita/292006/kemkomdigi-identifikasi-1923-konten-hoaks sepanjang-2024

D. J. Akhir, “Laporan Mafindo Ungkap Pertamina jadi Sasaran Hoaks, ini Daftarnya,” Okezone, Oct. 23, 2025. [Online]. Available:

https://economy.okezone.com/read/2025/10/23/320/3178861/laporan-mafindo-ungkap-pertamina-jadi-sasaran-hoaks-ini-daftarnya-nbsp

Z. Guo, M. Schlichtkrull, and A. Vlachos, “A Survey on Automated Fact-Checking,” Trans. Assoc. Comput. Linguist., Vol. 10, pp. 178–206, 2022, DOI: 10.1162/tacl_a_00454.

T. Schuster, A. Fisch, and R. Barzilay, “Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence,” in Proc. 2021 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 2021, pp. 624–643, DOI: 10.18653/v1/2021.naacl-main.52.

O. Honovich et al., “TRUE: Re-Evaluating Factual Consistency Evaluation,” in Proc. 2022 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA, 2022, pp. 3905–3920, DOI: 10.18653/v1/2022.naacl-main.287.

R. Mahendra, A. F. Aji, S. Louvan, F. Rahman, and C. Vania, “IndoNLI: A Natural Language Inference Dataset for Indonesian,” in Proc. 2021 Conf. Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, 2021, pp. 10511–10527, DOI: 10.18653/v1/2021.emnlp-main.821.

J. Fawaid, A. Awalina, R. Y. Krisnabayu, and N. Yudistira, “Indonesia’s Fake News Detection using Transformer Network,” in Proc. 6th Int. Conf. Sustainable Information Engineering and Technology, 2021, pp. 247–251, DOI: 10.1145/3479645.3479666.

D. Y. Yefferson, V. Lawijaya, and A. S. Girsang, “Hybrid model: IndoBERT and Long Short-Term Memory for Detecting Indonesian Hoax News,” IAES Int. J. Artif. Intell., Vol. 13, No. 2, pp. 1913–1924, 2024, DOI: 10.11591/ijai.v13.i2.pp1913-1924.

C. Jocelynne, I. G. N. L. Wijayakusuma, and L. P. I. Harini, “Detection of Political Hoax News using Fine-Tuning IndoBERT,” J. Appl. Informatics Comput., Vol. 9, No. 2, pp. 354–360, 2025, DOI: 10.30871/jaic.v9i2.8989.

J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: A Large-Scale Dataset for Fact Extraction and VERification,” in Proc. 2018 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA, 2018, pp. 809–819, DOI: 10.18653/v1/N18-1074.

P. Atanasova, J. G. Simonsen, C. Lioma, and I. Augenstein, “Fact Checking with Insufficient Evidence,” Trans. Assoc. Comput. Linguist., Vol. 10, pp. 746–763, 2022, DOI: 10.1162/tacl_a_00486.

J. Xin, R. Tang, Y. Yu, and J. Lin, “The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing,” in Proc. 59th Annu. Meeting of the Association for Computational Linguistics and 11th Int. Joint Conf. Natural Language Processing (Volume 1: Long Papers), Online, 2021, pp. 1040–1051, DOI: 10.18653/v1/2021.acl-long.84.

B. Wen et al., “Know Your Limits: A Survey of Abstention in Large Language Models,” Trans. Assoc. Comput. Linguist., Vol. 13, pp. 529–556, 2025, DOI: 10.1162/tacl_a_00754.

D. Hendrycks and K. Gimpel, “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks,” in Proc. Int. Conf. Learning Representations, 2017.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” in Proc. 34th Int. Conf. Machine Learning, Vol. 70, 2017, pp. 1321–1330.

N. Varshney, S. Mishra, and C. Baral, “Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings,” in Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, 2022, pp. 1995–2002, DOI: 10.18653/v1/2022.findings-acl.158.

M. Mosbach, M. Andriushchenko, and D. Klakow, “On the Stability of Fine-Tuning BERT: Misconceptions, Explanations, and Strong Baselines,” in Proc. Int. Conf. Learning Representations, 2021.

X. Bouthillier et al., “Accounting for Variance in Machine Learning Benchmarks,” in Proc. Machine Learning and Systems, Vol. 3, 2021, pp. 747–769.

W. Wongso, “Indonesian-Roberta-Base-Indonli,” Hugging Face model card. [Online]. Available: https://huggingface.co/w11wo/indonesian-roberta-base-indonli [Accessed: Oct. 7, 2026]

R. Arrahmi, I. Budi, and E. Yulianti, “Improving Indonesian Claim Verification and Explainability via Evidence-Grounded LLM Reasoning,” Int. J. Intell. Eng. Syst., Vol. 19, No. 6, pp. 649–660, 2026, DOI: 10.22266/ijies2026.0630.34.

A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme, “Hypothesis only Baselines in Natural Language Inference,” in Proc. 7th Joint Conf. Lexical and Computational Semantics, New Orleans, LA, USA, 2018, pp. 180–191, DOI: 10.18653/v1/S18-2023.

S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith, “Annotation Artifacts in Natural Language Inference Data,” in Proc. 2018 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LA, USA, 2018, pp. 107–112, DOI: 10.18653/v1/N18-2017.

C. Chow, “On Optimum Recognition Error and Reject Tradeoff,” IEEE Trans. Inf. Theory, Vol. 16, No. 1, pp. 41–46, 1970, DOI: 10.1109/TIT.1970.1054406.

Y. Geifman and R. El-Yaniv, “Selective Classification for Deep Neural Networks,” in Advances in Neural Information Processing Systems, Vol. 30, 2017, pp. 4878–4887.

Z. Jiang, J. Araki, H. Ding, and G. Neubig, “How can We Know When Language Models Know? On the Calibration of Language Models for Question Answering,” Trans. Assoc. Comput. Linguist., Vol. 9, pp. 962–977, 2021, DOI: 10.1162/tacl_a_00407.

S. Feng, W. Shi, Y. Wang, W. Ding, V. Balachandran, and Y. Tsvetkov, “Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration,” in Proc. 62nd Annu. Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, 2024, pp. 14664–14690, DOI: 10.18653/v1/2024.acl-long.786.

Masyarakat Anti Fitnah Indonesia (Mafindo), “TurnBackHoax.” [Online]. Available: https://turnbackhoax.id [Accessed: Oct. 6, 2026]

ANTARA News, “Berita.” [Online]. Available: https://www.antaranews.com [Accessed: Oct. 6, 2026]

Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv preprint arXiv:1907.11692, 2019.

T. Wolf et al., “Transformers: State-of-the-Art Natural Language Processing,” in Proc. 2020 Conf. Empirical Methods in Natural Language Processing: System Demonstrations, Online, 2020, pp. 38–45, DOI: 10.18653/v1/2020.emnlp-demos.6.




DOI: https://doi.org/10.32520/stmsi.v15i11.7129

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.