Improving Digital Fraud Detection Performance Using SMOTE and Comparative Analysis of Classification Algorithms

Niko Akbar, Yandi Anzhari, Oki Dahwanu, Nurul Abdillah

Abstract


The digital transformation of social media platforms, particularly X (formerly Twitter), has been accompanied by an increase in digital fraud cases. Machine learning approaches are widely used for fraud detection; however, they face the challenge of class imbalance, in which fraudulent posts are substantially outnumbered by legitimate posts. This study aims to evaluate the effect of the Synthetic Minority Over-sampling Technique (SMOTE) on fraud detection performance and compare three classification algorithms: Naive Bayes, Support Vector Machine (SVM), and Logistic Regression. The data were obtained from the Kaggle dataset “ScamTags: Indonesian NER for Fraud Identification,” with features extracted using Term Frequency–Inverse Document Frequency (TF-IDF). To prevent data leakage, training and test data were strictly separated within the cross-validation procedure, ensuring that SMOTE was applied exclusively to the training folds and never to the test data. The experimental results show that, for this small and imbalanced dataset, SMOTE did not consistently improve model performance. Naive Bayes was the only model that effectively detected fraudulent posts, achieving the highest recall of 90.00% and an F-measure of 60.76% in the scenario without SMOTE. In contrast, SVM failed to detect the fraud class, yielding a recall of 0%, despite its seemingly high accuracy. These findings underscore the importance of leakage-free evaluation and highlight how accuracy can be misleading when dealing with imbalanced data. Consequently, minority-class recall and F-measure provide more informative measures for evaluating fraud detection systems. In practical terms, this study suggests that a simple model such as Naive Bayes can serve as a reliable initial option for screening Indonesian-language fraudulent posts, particularly when training data are limited. Nevertheless, these findings should be interpreted in light of the dataset's small size and single-source origin. Further validation using larger and more diverse datasets is required to establish the generalizability of the results.

Keywords


data leakage; fraud detection; imbalanced data; naive bayes; SMOTE; twitter

Full Text:

PDF

References


H. Awan Irawan, “Deteksi Fraud Kartu Kredit dengan Logistic Regression, Random Forest, dan Gradient Boosting,” Jurnal Informatika Terpadu, Vol. 11, No. 2, pp. 92–97, 2025, [Online]. Available: https://journal.nurulfikri.ac.id/index.php/JIT

Maulana Malik Ibrahim and Sony Alfauzan, “Analisis Kinerja Model Machine Learning untuk Mendeteksi Transaksi Fraud pada Sistem Pembayaran Online,” Jurnal Ilmiah Nusantara, Vol. 2, No. 3, pp. 35–49, Apr. 2025, DOI: 10.61722/jinu.v2i3.4276.

E. Ileberi, Y. Sun, and Z. Wang, “Performance Evaluation of Machine Learning Methods for Credit Card Fraud Detection using SMOTE and AdaBoost,” IEEE Access, Vol. 9, pp. 165286–165294, 2021, DOI: 10.1109/ACCESS.2021.3134330.

Reyhand Ardhitha, Revifal Anugerah, and Tata Sutabri, “Analisis Penerapan Machine Learning dan Algoritma Anomali untuk Deteksi Penipuan pada Transaksi Digital,” Repeater : Publikasi Teknik Informatika dan Jaringan, Vol. 3, No. 1, pp. 80–90, Jan. 2025, DOI: 10.62951/repeater.v3i1.345.

L. Mahya, T. Tarjo, Z. M. Sanusi, and F. A. Kurniawan, “Intelligent Automation of Fraud Detection and Investigation:A Bibliometric Analysis Approach,” Jurnal Reviu Akuntansi dan Keuangan, Vol. 13, No. 3, pp. 588–613, Oct. 2023, DOI: 10.22219/jrak.v13i3.28487.

M. Reyhan, D. Ramadhani Ahmad, N. Ari Ramadhan, R. N. Hidayat, and I. Respati Kusumasari, “Penggunaan Data Analisis dan Big Data dalam Strategi Pengambilan Keputusan Keuangan,” 2024. [Online]. Available: https://economics.pubmedia.id/index.php/jampk

A. C. Umami and Y. Yuniseffendri, “Implementasi Makna Intensi dalam Cuitan Akun Twitter Pejabat Negara (Kajian Semantik),” BAPALA, Vol. 10, No. 3, 2023.

A. Z. Dilivia, L. Febriyanto, and B. Ginanjar, “Pergeseran Makna Kosakata Flora dalam Metafora di Media Sosial Twitter: Kajian Semantik,” Linguistik Indonesia, Vol. 41, No. 1, pp. 75–88, 2023.

M. Oktaviana, Z. A. Achmad, H. Arviani, and K. Kusnarto, “Budaya Komunikasi Virtual di Twitter dan Tiktok: Perluasan Makna Kata Estetik,” Satwika: Kajian Ilmu Budaya dan Perubahan Sosial, Vol. 5, No. 2, pp. 173–186, 2021.

A. Tri Arsanto, A. Faizin, M. Lutfi, Z. Nikmatus Saadah, T. informatika, and U. Yudharta Pasuruan, “Sistemasi: Jurnal Sistem Informasi Optimasi Algoritma Naive Bayes dengan Kombinasi SMOTETomek untuk Imbalance Class Fraud Detection Optimization of the Naive Bayes Algorithm with SMOTETomek Combination for Imbalance Class Fraud Detection.” [Online]. Available: http://sistemasi.ftik.unisi.ac.id

T. Berhane, T. Melese, A. Walelign, and A. Mohammed, “A Hybrid Convolutional Neural Network and Support Vector Machine‐based Credit Card Fraud Detection Model,” Math. Probl. Eng., Vol. 2023, No. 1, Jan. 2023, DOI: 10.1155/2023/8134627.

J. R. Sari, K. Sadik, A. M. Soleh, and C. Suhaeni, “Evaluasi Model Klasifikasi dalam Deteksi Penipuan Transaksi: Studi Kasus pada Data Tidak Seimbang,” Jurnal Gaussian, Vol. 14, No. 2, pp. 565–576, Dec. 2025, DOI: 10.14710/j.gauss.14.2.565-576.

R. Bounab, K. Zarour, B. Guelib, and N. Khlifa, “Enhancing Medicare Fraud Detection Through Machine Learning: Addressing Class Imbalance with SMOTE-ENN,” IEEE Access, Vol. 12, pp. 54382–54396, 2024, DOI: 10.1109/ACCESS.2024.3385781.

Z. Zhao and T. Bai, “Financial Fraud Detection and Prediction in Listed Companies using SMOTE and Machine Learning Algorithms,” Entropy, Vol. 24, No. 8, Aug. 2022, DOI: 10.3390/e24081157.

D. R. K. Saputra, Y. V. Via, and A. N. Sihananto, “Deteksi Anomali menggunakan Ensemble Learning dan Random Oversampling pada Penipuan Transaksi Keuangan,” Jurnal Informatika dan Teknik Elektro Terapan, Vol. 12, No. 3, Aug. 2024, DOI: 10.23960/jitet.v12i3.4910.

A. M. Syahbani, W. Firdaus, and K. A. Musodo, “A Comparative Study of Data Mining Algorithms for Fraud Detection in Financial Transactions,” Sinkron, Vol. 9, No. 2, pp. 814–821, Apr. 2025, DOI: 10.33395/sinkron.v9i2.14645.




DOI: https://doi.org/10.32520/stmsi.v15i10.6813

Article Metrics

Abstract view : 270 times
PDF - 31 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.