Benchmarking IndoBERT and IndoBERTweet on Emotion and Sentiment of Indonesian App Reviews

Natalie Grace Widjaja Kuswanto, Rinabi Tanamal

Abstract


Indonesian mobile application reviews are informal free-text feedback. Their sentiment and emotion labels are costly to obtain and only moderately reliable. This study determines which pre-trained encoder, IndoBERT or IndoBERTweet, classifies emotion and sentiment more reliably in Indonesian app reviews, and tests whether combining or routing between the two encoders improves on the reference single model. IndoBERT was pre-trained on general Indonesian text and IndoBERTweet on Indonesian tweets. Both were fine-tuned for six-class emotion and three-class sentiment classification on 21,065 de-duplicated reviews. Five random seeds, McNemar tests, and paired bootstrap confidence intervals evaluated confidence-based and task-based routing, probability averaging, sentiment derived from emotion, and a lexical register analysis. IndoBERTweet outperformed IndoBERT on emotion, reaching a macro-F1 of 0.627 against 0.596, with the advantage concentrated on reviews containing slang. Because IndoBERTweet performed at least as well on both formal and informal reviews, confidence-based routing gave no additional gain. Averaging the probabilities of ten models raised emotion macro-F1 significantly to 0.652, an improvement of 0.024 over the reference single model. Sentiment derived from the predicted emotion was never worse than a dedicated sentiment head. It performed significantly better for a single IndoBERTweet model, reaching 0.807 against 0.796, eliminated 10.2 to 14.6% of reviews with contradictory separate heads, and halved inference cost. Single-model modifications that did not degrade performance converged near 0.62 to 0.63, consistent with annotation reliability limiting attainable performance. These findings give practitioners an evidence-based default configuration: IndoBERTweet as the encoder, with sentiment read directly from the predicted emotion label.

Keywords


annotation agreement; emotion classification; IndoBERTweet; model routing; sentiment analysis

Full Text:

PDF

References


J. Dąbrowski, E. Letier, A. Perini, and A. Susi, “Analysing App Reviews for Software Engineering: A Systematic Literature Review,” Empir. Softw. Eng., Vol. 27, No. 2, p. 43, Mar. 2022, DOI: 10.1007/s10664-021-10065-7.

E. V. A. Karyoto, Y. T. Wiranti, and M. I. A. Putera, “Pengaruh Behavioral Intention terhadap Use Behavior pada Penggunaan Aplikasi Gojek,” Teknika, Vol. 13, No. 1, pp. 109–119, Mar. 2024, DOI: 10.34148/teknika.v13i1.761.

M. D. Darullah and R. Tanamal, “Faktor yang Paling mempengaruhi Keputusan dalam menggunakan TikTok Shop,” J. Econ. Manag. Account. Technol., Vol. 6, No. 2, pp. 166–175, Aug. 2023, DOI: 10.32500/jematech.v6i2.2846.

Y. Asri, D. Kuswardani, W. N. Suliyanti, Y. O. Manullang, and A. R. Ansyari, “Sentiment Analysis based on Indonesian Language Lexicon and IndoBERT on User Reviews PLN Mobile Application,” Indones. J. Electr. Eng. Comput. SCI., Vol. 38, No. 1, p. 677, Apr. 2025, DOI: 10.11591/ijeecs.v38.i1.pp677-688.

Tarwoto, R. Nugroho, N. Azka, and W. S. R. Graha, “Analisis Sentimen Ulasan Aplikasi Mobile JKN di Google PlayStore menggunakan IndoBERT,” J. JTIK J. Teknol. Inf. Dan Komun., Vol. 9, No. 2, pp. 495–505, Jan. 2025, DOI: 10.35870/jtik.v9i2.3340.

K. C. Pradhisa and R. Fajriyah, “Analisis Sentimen Ulasan Pengguna E-Commerce di Google Play Store menggunakan Metode IndoBERT,” Build. Inform. Technol. SCI. BITS, Vol. 6, No. 1, Jun. 2024, DOI: 10.47065/bits.v6i1.5247.

A. C. Saputra, A. S. Saragih, and D. Ronaldo, “Prediksi Emosi dalam Teks Bahasa Indonesia menggunakan Model IndoBERT,” J. Teknol. Inf. J. Keilmuan dan Apl. Bid. Tek. Inform., Vol. 19, No. 1, pp. 1–15, Jan. 2025.

A. S. Rizky and E. Y. Hidayat, “Emotion Classification in Indonesian Text using IndoBERT,” Comput Eng Appl J, Vol. 14, No. 1, pp. 1–11, 2025.

Riccosan and K. E. Saputra, “Multilabel Multiclass Sentiment and Emotion Dataset from Indonesian Mobile Application Review,” Data Brief, Vol. 50, p. 109576, Oct. 2023, DOI: 10.1016/j.dib.2023.109576.

I. Nurjannah and S. M. Siwi, “Guidelines for Analysis on Measuring Interrater Reliability of Nursing Outcome Classification,” Int. J. Res. Med. SCI., Vol. 5, No. 4, p. 1169, Mar. 2017, DOI: 10.18203/2320-6012.ijrms20171220.

B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Suzhou, China: Association for Computational Linguistics, 2020, pp. 843–857. DOI: 10.18653/v1/2020.aacl-main.85.

F. Koto, J. H. Lau, and T. Baldwin, “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021, pp. 10660–10668. DOI: 10.18653/v1/2021.emnlp-main.833.

I. N. Aqiilah, R. Saptono, and A. Syaifuddin, “Sentence-Level Sentiment Analysis of Indonesian App Reviews using IndoBERTweet,” J. Comput. Theor. Appl., Vol. 4, No. 1, pp. 74–91, Jun. 2026, DOI: 10.62411/jcta.16240.

H. Jayadianti, W. Kaswidjanti, A. T. Utomo, S. Saifullah, F. A. Dwiyanto, and R. Drezewski, “Sentiment Analysis of Indonesian Reviews using Fine-Tuning IndoBERT and R-CNN,” Ilk. J. Ilm., Vol. 14, No. 3, pp. 348–354, Dec. 2022, DOI: 10.33096/ilkom.v14i3.1505.348-354.

C. Hayat, “Sentiment Analysis of User Perceptions Toward the Gemini Application using the Naïve Bayes Algorithm,” Sist. J Sist Inf, Vol. 15, No. 7, pp. 2456–2469, 2026.

S. A. Nugraha, C. Lestari, K. B. Sanjaya, R. A. Naya, and J. Jolie, “Comparative Analysis of IndoBERT and mBERT for Online Gambling Comment Detection in Indonesian Social Media,” J. Tek. Inform. Jutif, Vol. 7, No. 2, pp. 1931–1943, Apr. 2026, DOI: 10.52436/1.jutif.2026.7.2.5677.

Riccosan and K. E. Saputra, “Multilabel Classification Sentiment Analysis on Indonesian Mobile App Reviews,” IAES Int. J. Artif. Intell. IJ-AI, Vol. 14, No. 5, pp. 4226–4234, Oct. 2025, DOI: 10.11591/ijai.v14.i5.pp4226-4234.

G. Z. Nabiilah, I. N. Alam, E. S. Purwanto, and M. F. Hidayat, “Indonesian Multilabel Classification using IndoBERT Embedding and MBERT Classification,” Int. J. Electr. Comput. Eng. IJECE, Vol. 14, No. 1, p. 1071, Feb. 2024, DOI: 10.11591/ijece.v14i1.pp1071-1078.

W. Christian, D. Adamlu, A. Yu, and D. Suhartono, “Leveraging IndoBERT and DistilBERT for Indonesian Emotion Classification in e-Commerce Reviews,” Procedia Comput. SCI., Vol. 269, pp. 321–330, 2025, DOI: 10.1016/j.procs.2025.08.284.

L. D. Cahya, A. Luthfiarta, J. I. T. Krisna, S. Winarno, and A. Nugraha, “Improving Multi-Label Classification Performance on Imbalanced Datasets Through SMOTE Technique and Data Augmentation using IndoBERT Model,” J. Nas. Teknol. dan Sist. Inf., Vol. 9, No. 3, pp. 290–298, Jan. 2024, DOI: 10.25077/TEKNOSI.v9i3.2023.290-298.

W. Tang, J. Hu, H. Zhang, P. Wu, and H. He, “Kappa Coefficient: A Popular Measure of Rater Agreement,” Shanghai Arch. Psychiatry, Vol. 27, No. 1, pp. 62–67, 2015, DOI: 10.11919/j.issn.1002-0829.215010.

R. Artstein and M. Poesio, “Inter-Coder Agreement for Computational Linguistics,” Comput. Linguist., Vol. 34, No. 4, pp. 555–596, Dec. 2008, DOI: 10.1162/coli.07-034-R2.

V. Bobicev and M. Sokolova, “Inter-Annotator Agreement in Sentiment Analysis: Machine Learning Perspective,” in RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning, Incoma Ltd. Shoumen, Bulgaria, Nov. 2017, pp. 97–102. DOI: 10.26615/978-954-452-049-6_015.

C. G. Northcutt, L. Jiang, and I. L. Chuang, “Confident Learning: Estimating Uncertainty in Dataset Labels,” Aug. 22, 2022, arXiv: arXiv:1911.00068. DOI: 10.48550/arXiv.1911.00068.

T. G. Dietterich, “Ensemble Methods in Machine Learning,” in Multiple Classifier Systems, Lecture Notes in Computer Science, Vol. 1857, Berlin, Heidelberg: Springer Berlin Heidelberg, 2000, pp. 1–15. DOI: 10.1007/3-540-45014-9_1.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” May 24, 2019, arXiv: arXiv:1810.04805. DOI: 10.48550/arXiv.1810.04805.

R. Dror, G. Baumer, S. Shlomov, and R. Reichart, “The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia: Association for Computational Linguistics, 2018, pp. 1383–1392. DOI: 10.18653/v1/P18-1128.

T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-Generation Hyperparameter Optimization Framework,” Jul. 25, 2019, arXiv: arXiv:1907.10902. DOI: 10.48550/arXiv.1907.10902.

A. Conneau et al., “Unsupervised Cross-Lingual Representation Learning at Scale,” Apr. 08, 2020, arXiv: arXiv:1911.02116. DOI: 10.48550/arXiv.1911.02116.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal Loss for Dense Object Detection,” Feb. 07, 2018, arXiv: arXiv:1708.02002. DOI: 10.48550/arXiv.1708.02002.




DOI: https://doi.org/10.32520/stmsi.v15i10.6998

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.