Llama and Gemma vs IndoBERT and SBERT: Zero-Shot and Few-Shot Sentiment Analysis

Haydar Yahya Asya'bani Nur, Muhammad Iqbal Akbar

Abstract


Generative Large Language Models (LLMs), such as Llama and Gemma, offer zero-shot and few-shot capabilities without requiring fine-tuning. However, few studies have directly compared these models with embedding-similarity-based encoder models, such as IndoBERT and SBERT, under equivalent evaluation conditions, particularly for Indonesian–English multilingual sentiment analysis. This study aims to evaluate and compare the performance of these four models under zero-shot and few-shot settings without fine-tuning. The study uses the parallel NusaX-senti dataset, which was stratified and divided into an 80% training pool and a 20% test pool, with six few-shot examples and 100 evaluation scenarios. Model performance was assessed using accuracy, precision, recall, and macro-F1 scores, supplemented by confusion matrix analysis. Differences across experimental conditions were examined using McNemar's test and bootstrap confidence intervals. The results show that few-shot prompting improved F1 scores across all models, with statistically significant improvements in four of the seven model–language combinations. The largest improvement was observed for IndoBERT, whose F1 score increased from 0.297 to 0.616, whereas SBERT showed a smaller improvement (ΔF1 ≈ 0.07) that was not statistically significant. The highest F1 score was achieved by Llama-3.2-3B under the English few-shot setting (0.780; 95% CI: 0.696–0.857). However, 23–24% of Llama's few-shot responses were truncated due to the 60-token limit, although sentiment ratings could still be extracted from all responses. SBERT exhibited the lowest variation in F1 scores across experimental conditions (0.511–0.619), although this stability was accompanied by lower performance than Llama. Performance differences between Indonesian and English were not statistically significant for any of the models. These findings provide insights into the development of computationally efficient Indonesian sentiment analysis systems without requiring model training.

Keywords


analisis sentimen, few-shot learning, model bahasa generatif, embedding kemiripan semantik, multibahasa Indonesia-Inggris

Full Text:

PDF

References


F. Koto, N. Aisyah, H. Li, and T. Baldwin, "Large Language Models only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU," in Proc. 2023 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2023, pp. 12359–12374, DOI: 10.18653/v1/2023.emnlp-main.760.

Q. Dong et al., "A Survey on in-Context Learning," in Proc. 2024 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2024, pp. 1107–1128, DOI: 10.18653/v1/2024.emnlp-main.64.

H. Steck, C. Ekanadham, and N. Kallus, "Is Cosine-Similarity of Embeddings Really about Similarity?," in Companion Proc. ACM Web Conference 2024, 2024, DOI: 10.1145/3589335.3651526.

K. Zhou, K. Ethayarajh, D. Card, and D. Jurafsky, "Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words," in Proc. 60th Annual Meeting of the Association for Computational Linguistics (ACL), Vol. 2, 2022, pp. 401–423, DOI: 10.18653/v1/2022.acl-short.45.

T. Gao, X. Yao, and D. Chen, "SimCSE: Simple Contrastive Learning of Sentence Embeddings," in Proc. 2021 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 6894–6910, DOI: 10.18653/v1/2021.emnlp-main.552.

A. Buscemi and D. Proverbio, "ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis," arXiv preprint arXiv:2402.01715, 2024, DOI: 10.48550/arXiv.2402.01715.

M. A. Hasan et al., "Zero- and Few-Shot Prompting with LLMs: A Comparative Study with Fine-Tuned Models for Bangla Sentiment Analysis," in Proc. 2024 Joint Int. Conf. Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy, 2024, pp. 17808–17818.

D. Cielen, K. W. De Bock, and L. Florès, "Exploring Zero-Shot SLM Ensembles as an Alternative to LLMs for Sentiment Analysis," Information Fusion, Vol. 126, pt. B, art. No. 103666, 2026, DOI: 10.1016/j.inffus.2025.103666.

D. Di Palma, A. De Bellis, G. Servedio, V. W. Anelli, F. Narducci, and T. Di Noia, "LLaMAs have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models through Probing," in Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Vol. 1, 2025, pp. 6124–6142, DOI: 10.18653/v1/2025.acl-long.306.

M. Cherakhloo, A. Abbasi, M. S. Sarafraz, and B. V. Vahdat, "Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning," arXiv preprint arXiv:2510.12807, 2025, DOI: 10.48550/arXiv.2510.12807.

G. I. Winata et al., "NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages," in Proc. 17th Conf. European Chapter of the Association for Computational Linguistics (EACL), 2023, pp. 815–834, DOI: 10.18653/v1/2023.eacl-main.57.

M. Ikonić Nešić, S. Petalinkar, M. Škorić, R. Stanković, and B. Rujević, "Advancing Sentiment Analysis in Serbian Literature: A Zero and Few-Shot Learning Approach using the Mistral Model," in Proc. 6th Int. Conf. Computational Linguistics in Bulgaria (CLIB 2024), 2024, pp. 58–70.

S. Münker, K. Kugler, and A. Rettinger, "Zero-Shot Prompt-based Classification: Topic Labeling in Times of Foundation Models in German Tweets," in Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Student Research Workshop, 2025, pp. 53–63, DOI: 10.18653/v1/2025.acl-srw.4.

S. Schulhoff et al., "The Prompt Report: A Systematic Survey of Prompt Engineering Techniques," arXiv preprint arXiv:2406.06608, 2024, DOI: 10.48550/arXiv.2406.06608.

Y. Tang, D. Tuncel, C. Koerner, and T. Runkler, "The Few-Shot Dilemma: Over-Prompting Large Language Models," in Proc. 3rd Int. Conf. Foundation and Large Language Models (FLLM), 2025, pp. 134–141, DOI: 10.1109/FLLM67465.2025.11391015.

B. P. Allen, F. Polat, and P. Groth, "SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-based Classification for Hallucination Detection," in Proc. 18th Int. Workshop on Semantic Evaluation (SemEval-2024), 2024, pp. 839–844, DOI: 10.18653/v1/2024.semeval-1.120.

L. Geni, E. Yulianti, and D. I. Sensuse, "Sentiment Analysis of Tweets before the 2024 Elections in Indonesia using BERT Language Models," Jurnal Ilmiah Teknik Elektro Komputer dan Informatika, Vol. 9, No. 3, pp. 746–757, 2023, DOI: 10.26555/jiteki.v9i3.26490.

D. G. Mandhasiya, H. Murfi, and A. Bustamam, "The Hybrid of BERT and Deep Learning Models for Indonesian Sentiment Analysis," Indonesian Journal of Electrical Engineering and Computer Science, Vol. 33, No. 1, pp. 591–602, 2024, DOI: 10.11591/ijeecs.v33.i1.pp591-602.

T. D. Purnomo and J. Sutopo, "Comparison of Pre-Trained BERT-based Transformer Models for Regional Language Text Sentiment Analysis in Indonesia," International Journal Science and Technology, Vol. 3, No. 3, pp. 11–21, 2024, DOI: 10.56127/ijst.v3i3.1739.

F. N. Khaqqi, L. Alfat, and I. Nurhaida, "Sentiment Analysis of US-China Tariffs using IndoBERT and Economic Impact on Indonesia," Journal of Applied Informatics and Computing, Vol. 9, No. 6, pp. 3000–3011, 2025, DOI: 10.30871/jaic.v9i6.11544.

A. Bello, S.-C. Ng, and M.-F. Leung, "A BERT Framework to Sentiment Analysis of Tweets," Sensors, Vol. 23, No. 1, art. No. 506, 2023, DOI: 10.3390/s23010506.

B. Wilie et al., "IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding," in Proc. 1st Conf. Asia-Pacific Chapter of the Association for Computational Linguistics and 10th Int. Joint Conf. on Natural Language Processing (AACL-IJCNLP), 2020, pp. 843–857, DOI: 10.18653/v1/2020.aacl-main.85.

V. R. Joseph, "Optimal Ratio for Data Splitting," Statistical Analysis and Data Mining: The ASA Data Science Journal, Vol. 15, No. 4, pp. 531–538, 2022, DOI: 10.1002/sam.11583.

J. Snell, K. Swersky, and R. Zemel, "Prototypical Networks for Few-Shot Learning," in Advances in Neural Information Processing Systems (NeurIPS), Vol. 30, 2017.

L. Pacchiardi, L. G. Cheke, and J. Hernández-Orallo, "100 Instances is All You Need: Predicting the Success of a New LLM on Unseen Data by Testing on a Few Instances," arXiv preprint arXiv:2409.03563, 2024, DOI: 10.48550/arXiv.2409.03563.

A. Grattafiori et al., "The Llama 3 Herd of Models," arXiv preprint arXiv:2407.21783, 2024, doi: 10.48550/arXiv.2407.21783.

Gemma Team, "Gemma 2: Improving Open Language Models at a Practical Size," arXiv preprint arXiv:2408.00118, 2024, DOI: 10.48550/arXiv.2408.00118.

N. Reimers and I. Gurevych, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks," in Proc. 2019 Conf. Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992, DOI: 10.18653/v1/D19-1410.

N. Reimers and I. Gurevych, "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation," in Proc. 2020 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 4512–4525, DOI: 10.18653/v1/2020.emnlp-main.365.

T. G. Dietterich, "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms," Neural Computation, Vol. 10, No. 7, pp. 1895–1923, 1998, DOI: 10.1162/089976698300017197.

S. Holm, "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian Journal of Statistics, Vol. 6, No. 2, pp. 65–70, 1979.

B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman & Hall, 1993.




DOI: https://doi.org/10.32520/stmsi.v15i11.7018

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.