Comparative Evaluation of Modern Embeddings and Ensemble Learning for Sentiment Analysis of Mobile Banking Application Reviews

Abstract

User reviews of mobile banking applications provide an important source of information for evaluating the quality of digital services. However, conventional text representations such as Term Frequency–Inverse Document Frequency (TF-IDF) have limitations in capturing semantic context, while class imbalance can affect sentiment classification performance. This study aims to comparatively evaluate text representations and classification algorithms for sentiment analysis of mobile banking application reviews. The dataset consists of 9,885 user reviews of the Livin’ by Mandiri application collected from the Google Play Store. The study compares three text representation methods, namely TF-IDF, Multilingual E5 Large Instruct, and Nomic Embed v2, with four classification algorithms: Naive Bayes, Support Vector Machine (SVM), XGBoost, and CatBoost, resulting in 12 model combinations. Borderline-SMOTE was applied to the training data to address class imbalance, while hyperparameter optimization was performed using BayesSearchCV. Model performance was evaluated using accuracy, precision, recall, F1-score, and macro-F1 through stratified 5-fold cross-validation on the training data and evaluation on the test set. The results show that Multilingual E5 Large Instruct achieved the highest average macro-F1 score of 0.950. The combination of CatBoost and Multilingual E5 Large Instruct achieved the best individual performance, with an accuracy of 98% and a macro-F1 score of 0.97. These findings demonstrate that the choice of text representation and classification algorithm plays an important role in determining the performance of sentiment analysis for Indonesian-language mobile banking application reviews.