Comparing LoRA on Qwen2.5 and DeepSeek-R1 with IndoBERT Fine-Tuning for Indonesian Sentiment

Afito Indra Permana, Muhammad Iqbal Akbar

Abstract


Indonesian sentiment analysis has largely relied on encoder-based models, such as IndoBERT, through full fine-tuning. Meanwhile, open-weight generative large language models (LLMs) adapted using Low-Rank Adaptation (LoRA) offer a potentially more resource-efficient alternative. This study compares Qwen2.5-1.5B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B using LoRA, with identical adaptation hyperparameters, against IndoBERT-base-p1 using full fine-tuning for three-class sentiment classification on the SmSA dataset from IndoNLU, with a re-split stratified into 70:15:15 training, validation, and test sets. Across three random seeds (42, 43, and 44), evaluated on 1,914 test samples, the mean macro-F1 scores were 0.9242 ± 0.0048 for IndoBERT, 0.9195 ± 0.0039 for Qwen, and 0.8779 ± 0.0048 for DeepSeek. In an earlier run using a different training configuration, Qwen achieved 0.9211 ± 0.0023. DeepSeek performed significantly worse than both competing models across all three seeds, based on McNemar's test with Holm correction and paired bootstrap analysis. In contrast, the performance gap between Qwen and IndoBERT was small (approximately 0.5 percentage points), and the confidence intervals included zero. Thus, IndoBERT showed only a slight tendency toward better performance, while comparability between the models can only be claimed within a predefined equivalence margin of ±0.02 macro-F1. The backbones of the two LLMs were not identical. An additional single-seed control experiment using LoRA-adapted Qwen2.5-Math-1.5B yielded a macro-F1 score of 0.8842, which was not significantly different from DeepSeek. This suggests that DeepSeek's performance gap may be more closely associated with its base model than with reasoning distillation. Minimal preprocessing did not produce a statistically significant difference in IndoBERT's performance. IndoBERT was also substantially more efficient at inference, requiring 6.12 ms per sample, compared with 154.81–160.80 ms for the two LoRA-based models (seed 42). These findings indicate that applying LoRA to generative LLMs does not automatically outperform full fine-tuning of encoder-based models in in-distribution sentiment classification tasks such as SmSA.

Keywords


sentiment analysis; IndoBERT; LoRA; parameter-efficient fine-tuning; SmSA

Full Text:

PDF

References


W. J. Kusoema and I. Ibrahim, “Sentiment Analysis on the PT Pertamina Corruption Case using IndoBERT and RCNN Methods,” Sistemasi: Jurnal Sistem Informasi, Vol. 14, No. 5, pp. 2246–2257, 2025, DOI: 10.32520/stmsi.v14i5.5392.

C.-H. Lin and U. Nuha, “Sentiment Analysis of Indonesian Datasets based on a Hybrid Deep-Learning Strategy,” Journal of Big Data, Vol. 10, No. 1, p. 88, 2023, DOI: 10.1186/s40537-023-00782-9.

A. H. Nasution, A. Onan, Y. Murakami, W. Monika, and A. Hanafiah, “Benchmarking Open-Source Large Language Models for Sentiment and Emotion Classification in Indonesian Tweets,” IEEE Access, Vol. 13, pp. 94009–94025, 2025, DOI: 10.1109/ACCESS.2025.3574629.

B. Wilie, K. Vincentio, G. I. Winata, S. Cahyawijaya, X. Li, Z. Y. Lim, S. Soleman, R. Mahendra, P. Fung, S. Bahar, and A. Purwarianti, “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proc. 1st Conf. Asia-Pacific Chapter ACL and 10th Int. Joint Conf. Natural Language Processing (AACL-IJCNLP), 2020, pp. 843–857, DOI: 10.18653/v1/2020.aacl-main.85.

F. Koto, J. H. Lau, and T. Baldwin, “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” in Proc. 2021 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 10660–10668, DOI: 10.18653/v1/2021.emnlp-main.833.

D. Suharjanto and M. Mustakim, “Optimizing Emotion Recognition in Indonesian Product Reviews using LoRA on Transformer,” Jurnal Sains, Nalar, dan Aplikasi Teknologi Informasi, Vol. 5, No. 2, pp. 118–127, 2026, DOI: 10.20885/snati.v5.i2.48248.

F. M. Al Simabua and L. Alfat, “Optimization of IndoBERT-Lite Fine-Tuning for Spam Detection in Digital Customer Services,” Sistemasi: Jurnal Sistem Informasi, Vol. 15, No. 5, pp. 1886–1899, 2026, DOI: 10.32520/stmsi.v15i5.6398.

Qwen Team, “Qwen2.5 Technical Report,” arXiv preprint arXiv:2412.15115, 2024, DOI: 10.48550/arXiv.2412.15115.

D. Guo et al., “DeepSeek-R1 Incentivizes Reasoning in LLMs Through Reinforcement Learning,” Nature, Vol. 645, pp. 633–638, 2025, DOI: 10.1038/s41586-025-09422-z.

Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey,” arXiv preprint arXiv:2403.14608, 2024, DOI: 10.48550/arXiv.2403.14608.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. 10th Int. Conf. Learning Representations (ICLR), 2022, URL: https://openreview.net/forum?id=nZeVKeeFYf9.

X. Wang and A. Azman, “LoRA-based Efficient Fine-Tuning of Transformer Models for Short Text Classification,” in Advances in Visual Informatics (IVIC 2025), Lecture Notes in Computer Science, Vol. 16215. Singapore: Springer, 2026, pp. 168–179, DOI: 10.1007/978-981-95-3808-9_13.

K. I. Roumeliotis, N. D. Tselikas, and D. K. Nasiopoulos, “Optimizing Airline Review Sentiment Analysis: A Comparative Analysis of LLaMA and BERT Models through Fine-Tuning and Few-Shot Learning,” Computers, Materials & Continua, Vol. 82, No. 2, pp. 2769–2792, 2025, DOI: 10.32604/cmc.2025.059567.

H. Jin, N. Ashrafi, A. Abdollahi, W. Liu, J. Wang, G. Gui, M. Pishgar, and H. Feng, “LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA,” arXiv preprint arXiv:2509.00731, 2025, DOI: 10.48550/arXiv.2509.00731.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” in Proc. 2019 Conf. North American Chapter ACL: Human Language Technologies, Vol. 1, 2019, pp. 4171–4186, DOI: 10.18653/v1/N19-1423.

M. A. A. Saputra, A. Alamsyah, D. P. Ramadhani, T. S. Siadari, and H. Fakhrurroja, “IndoBERT-Sentiment: Context-Conditioned Sentiment Classification for Indonesian Text,” arXiv preprint arXiv:2604.07057, 2026, DOI: 10.48550/arXiv.2604.07057.

V. R. Joseph, “Optimal Ratio for Data Splitting,” Statistical Analysis and Data Mining: The ASA Data Science Journal, Vol. 15, No. 4, pp. 531–538, 2022, DOI: 10.1002/sam.11583.

J. Opitz, “A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice,” Transactions of the Association for Computational Linguistics, Vol. 12, pp. 820–836, 2024, DOI: 10.1162/tacl_a_00675.

S. Henning, W. Beluch, A. Fraser, and A. Friedrich, “A Survey of Methods for Addressing Class Imbalance in Deep-Learning based Natural Language Processing,” in Proc. 17th Conf. European Chapter ACL (EACL), 2023, pp. 523–540, DOI: 10.18653/v1/2023.eacl-main.38.

Q. McNemar, “Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages,” Psychometrika, Vol. 12, No. 2, pp. 153–157, 1947, DOI: 10.1007/BF02295996.

B. Efron, “Bootstrap Methods: Another Look at the Jackknife,” The Annals of Statistics, Vol. 7, No. 1, pp. 1–26, 1979, DOI: 10.1214/aos/1176344552.

Y. Bestgen, “Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status,” in Proc. 13th Language Resources and Evaluation Conf. (LREC), 2022, pp. 5956–5962, URL: https://aclanthology.org/2022.lrec-1.640.

Y. Kim, M. Lee, and S. Lee, “A Survey on Memory-Efficient Fine-Tuning for Large Language Models,” Transactions of the Association for Computational Linguistics, Vol. 14, pp. 960–982, 2026, DOI: 10.1162/tacl.a.692.

N. T. Bui, G. Savova, and L. Wang, “Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models,” arXiv preprint arXiv:2503.07329, 2025, DOI: 10.48550/arXiv.2503.07329.

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” in Advances in Neural Information Processing Systems 36 (NeurIPS), 2023, DOI: 10.48550/arXiv.2305.14314.

S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “DoRA: Weight-Decomposed Low-Rank Adaptation,” in Proc. 41st Int. Conf. Machine Learning (ICML), 2024, pp. 32100–32121, DOI: 10.48550/arXiv.2402.09353.




DOI: https://doi.org/10.32520/stmsi.v15i11.7015

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.