Accuracy Evaluation of the Naïve Bayes Classifier for Sentiment Classification of Diploma Authenticity Issues using Orange

Abstract

The rapid growth of digital text data has increased the demand for effective methods to extract meaningful information, particularly for understanding public opinion. Sentiment analysis is widely used to classify opinions into positive, negative, and neutral categories. However, challenges such as linguistic ambiguity, subjectivity, and class imbalance often degrade classification performance. This study aims to evaluate the performance of the Naïve Bayes algorithm for sentiment classification on the issue of diploma authenticity using a publicly available dataset, while examining the impact of data distribution on model performance. A quantitative experimental approach was employed using an original dataset of 1,014 instances and an oversampled dataset of 1,767 instances. The data were processed through preprocessing, Bag-of-Words feature extraction, and sentiment classification using Orange Data Mining with 10-fold cross-validation. Model performance was evaluated using accuracy and the Area Under the Receiver Operating Characteristic Curve (AUC). The results indicate that the Naïve Bayes model achieved an accuracy of 37.2% and an AUC of 0.704 on the original imbalanced dataset, reflecting relatively poor classification performance. After applying oversampling to balance the class distribution, the model's accuracy increased substantially to 82.1%, while the AUC improved to 0.970. These findings demonstrate that class distribution has a significant impact on the performance of the Naïve Bayes algorithm in sentiment classification and highlight the importance of addressing class imbalance to achieve more reliable classification results.