Zero-Shot Sentiment Analysis on Student Feedback: A Comparative Study of Multilingual and Translate-Test Approaches in Indonesian Higher Education
Authors
-
Adam Bachtiar
Universitas Pendidikan Mandalika
-
Ahmad Ashril Rizal
Universitas Islam Negeri Mataram
-
Tuning Ridha Addhiny
Universitas Islam Negeri Mataram
DOI:
https://doi.org/10.36312/e-saintika.v9i3.4002Keywords:
Academic Service Evaluation, Politeness Bias, Sentiment Analysis, Translate-Test Approach, XLM-RoBERTa, Zero-Shot ClassificationAbstract
The evaluation of academic services in Indonesian higher education is often hindered by the scarcity of labeled datasets (cold start problem) and the complexity of culturally implicit feedback. This study evaluates the efficacy of Zero-Shot Classification by benchmarking two distinct inference paradigms: the Direct Multilingual approach (XLM-RoBERTa-large-xnli) and the Translate-Test approach (Facebook/BART-large-mnli). Using a dataset of 280 student reviews validated by human annotators ( =0.844), the research reveals a significant performance trade-off. While XLM-RoBERTa greater robustness in maintaining global performance equilibrium (Macro F1-Score: 0.67), it exhibits a pronounced ‘Politeness Bias’, frequently failing to detect negative reviews masked by courteous language (Recall: 0.48). Conversely, the Translate-Test approach (BART) shows higher sensitivity in capturing negative sentiments (Recall: 0.77). Qualitative analysis suggests that the translation process potentially functions as a dual-mechanism: acting as a cultural decontextualization filter that isolates implicit criticism and a denoising layer that normalizes informal slang and typographical errors. However, this enhanced sensitivity results in an approximate 2.6x increase in computational latency and weaker neutral class detection. These findings indicate that while XLM-RoBERTa offers balanced generalization for broad analysis, the Translate-Test strategy is highly effective for accurately uncovering latent student grievances obscured by local linguistic styles.
Downloads
References
Adulyasak, Y., Benomar, O., Chaouachi, A., Cohen, M. C., & Khern-am-nuai, W. (2023). Using AI to detect panic buying and improve products distribution amid pandemic. AI & SOCIETY, 39(4), 2099–2128. (pub.1157275219). https://doi.org/10.1007/s00146-023-01654-9
Ahmat, A., Yang, Y., Ma, B., Dong, R., Lu, K., & Wang, L. (2023). WAD-X: Improving Zero-shot Cross-lingual Transfer via Adapter-based Word Alignment. In ACM Transactions on Asian and Low-Resource Language Information Processing (Vol. 22, Issue 9). Association for Computing Machinery. https://doi.org/10.1145/3610289
An, B. (2023). Prompt-based for Low-Resource Tibetan Text Classification. In ACM Transactions on Asian and Low-Resource Language Information Processing (Vol. 22, Issue 8). Association for Computing Machinery. https://doi.org/10.1145/3603168
Aras, N. B., Risawandi, R., & Rosnita, L. (2023). Analisis sentimen kepuasan customer terhadap ekspedisi tiki, sicepat express dan ninja express menggunakan algoritma naive bayes. Journal of Informatics and computer Science, 9(1), 53. https://doi.org/10.33143/jics.v9i1.2943
Araújo, M., Pereira, A., & Benevenuto, F. (2020). A comparative study of machine translation for multilingual sentence-level sentiment analysis. Information Sciences, 512, 1078–1102. https://doi.org/10.1016/j.ins.2019.10.031
Astia, I. (2020). Politeness Strategy in Interlanguage Pragmatics of Complaints by International Students. IJELTAL (Indonesian Journal of English Language Teaching and Applied Linguistics), 4(2), 349. https://doi.org/10.21093/ijeltal.v4i2.528
Boudad, N., Faizi, R., & Oulad Haj Thami, R. (2023). Multilingual, monolingual and mono-dialectal transfer learning for Moroccan Arabic sentiment classification. Social Network Analysis and Mining, 14(1), 3. https://doi.org/10.1007/s13278-023-01159-9
Budisantoso, E. A., Darmawan, G., & Pravitasari, A. A. (2025). Improving Accuracy with Hyperparameter Tuning for Sarcasm Detection in Twitter Comments using BiLSTM. In IAENG International Journal of Applied Mathematics (Vol. 55, Issue 7, pp. 2042–2050). International Association of Engineers.
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8440–8451. https://doi.org/10.18653/v1/2020.acl-main.747
De Raedt, M., Bitew, S. K., Godin, F., Demeester, T., & Develder, C. (2023). Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: An Exploratory Study. Proceedings of the 3rd Workshop on Multi-Lingual Representation Learning (MRL), 50–66. https://doi.org/10.18653/v1/2023.mrl-1.5
Dwiyono, A., Abdiansah, A., & Fachrurrozi, M. (2024). Analisis Perbandingan Klasifikasi Intent Chatbot Menggunakan Deep Learning BERT, RoBERTa, dan IndoBERT. Journal of Information System Research (JOSH), 6(1), 595–606. https://doi.org/10.47065/josh.v6i1.6051
Edalati, M., Imran, A. S., Kastrati, Z., & Daudpota, S. M. (2022). The Potential of Machine Learning Algorithms for Sentiment Classification of Students’ Feedback on MOOC. In K. Arai (Ed.), Lecture Notes in Networks and Systems (Vol. 296, pp. 11–22). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-82199-9_2
Fang, H., Xu, G., Long, Y., & Tang, W. (2022). An Effective ELECTRA-Based Pipeline for Sentiment Analysis of Tourist Attraction Reviews. Applied Sciences, 12(21), 10881. https://doi.org/10.3390/app122110881
Fata, M. A. K., Sumpeno, S., Wibawa, A. D., & Feryando, D. A. (2023). Evaluating the Sentiment Analysis from Auto-Generated Summary Text Using IndoBERT Fine-Tuning Model in Indonesian News Text. 2023 IEEE 15th International Conference on Computational Intelligence and Communication Networks (CICN), 822–829. https://doi.org/10.1109/CICN59264.2023.10402345
Fergan, E., & Tashu, T. M. (2023). Course Review Sentiment Analysis: A Comparative Study of Machine Learning and Deep Learning Methods. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/BESC59560.2023.10386515
Horibe, N., & Fujihira, K. (2022). Stability of a Multilingual Sentiment Analysis based on Word-to-Word Translations. International Journal of Service and Knowledge Management, 6(2), 1. https://doi.org/10.52731/ijskm.v6.i2.664
Hote, A. & Pandey, D. R. (2023). OPEN-AMZPRE: Optimized Preprocessing with Ensemble Classification for Amazon Product Reviews Sentiment Prediction. International Journal of Scientific Research in Science and Technology, 385–401. https://doi.org/10.32628/IJSRST52310672
Huertas‐Tato, J., Martín, A., & Camacho, D. (2023). BERTuit: Understanding Spanish language in Twitter with transformers. Expert Systems, 40(9), e13404. https://doi.org/10.1111/exsy.13404
Islam, K. I., Kar, S., Islam, M. S., & Amin, M. R. (2021). SentNoB: A Dataset for Analysing Sentiment on Noisy Bangla Texts. Findings of the Association for Computational Linguistics: EMNLP 2021, 3265–3271. https://doi.org/10.18653/v1/2021.findings-emnlp.278
Jacqmin, L., Marzinotto, G., Gromada, J., Szczekocka, E., Kołodyński, R., Chauvière, A., & Damnati, Ǵeraldine. (2021). SpanAlign: Efficient Sequence Tagging Annotation Projection into Translated Data applied to Cross-Lingual Opinion Mining (W. Xu, A. Ritter, T. Baldwin, & A. Rahimi, Eds.; pp. 238–248). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2021.wnut-1.27
Jazuli, A., Widowati, W., & Kusumaningrum, R. (2023). Aspect-based sentiment analysis on student reviews using the Indo-Bert base model. In R. R. Isnanto, Hadiyanto, & B. Warsito (Eds.), E3S Web of Conferences (Vol. 448). EDP Sciences. https://doi.org/10.1051/e3sconf/202344802004
Jazuli, A., Widowati, W., & Kusumaningrum, R. (2025). Optimizing Aspect-Based Sentiment Analysis Using BERT for Comprehensive Analysis of Indonesian Student Feedback. In Applied Sciences (Switzerland) (Vol. 15, Issue 1). Multidisciplinary Digital Publishing Institute (MDPI). https://doi.org/10.3390/app15010172
Jones, A., Wang, W. Y., & Mahowald, K. (2021). A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5833–5847. https://doi.org/10.18653/v1/2021.emnlp-main.471
Kastrati, Z., Dalipi, F., Imran, A. S., Pireva Nuci, K., & Wani, M. A. (2021). Sentiment Analysis of Students’ Feedback with NLP and Deep Learning: A Systematic Mapping Study. Applied Sciences, 11(9), 3986. https://doi.org/10.3390/app11093986
Kastrati, Z., Imran, A. S., & Kurti, A. (2020). Weakly Supervised Framework for Aspect-Based Sentiment Analysis on Students’ Reviews of MOOCs. In IEEE Access (Vol. 8, pp. 106799–106810). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ACCESS.2020.3000739
Kiziltepe, R. S., Ezin, E., Yentür, Ö., Basbrain, A. M., & Karakus, M. (2025). Advancing Sentiment Analysis for Low-Resource Languages Using Fine-Tuned LLMs: A Case Study of Customer Reviews in Turkish Language. In IEEE Access (Vol. 13, pp. 77382–77394). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ACCESS.2025.3566000
Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP. Proceedings of the 28th International Conference on Computational Linguistics, 757–770. https://doi.org/10.18653/v1/2020.coling-main.66
Lakoro, D. Y., Utami, E., & Ariatmanto, D. (2023). Sentiment Analysis of BNI Mobile Application Using The K-Nearest Neighbor Algorithm (KNN) With Particle Swarm Optimization (PSO) Feature Selection. INTECOMS: Journal of Information Technology and Computer Science, 6(2), 948–953. https://doi.org/10.31539/intecoms.v6i2.7912
Li, B., Sun, Q., Xia, H., Cao, Q., Rong, W., & Chen, C. (2024). Dual-Track Aspect-Level Sentiment Analysis for Alleviating Cold Start in MOOC Course Reviews (pp. 174–181). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/HPCC64274.2024.00033
Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., & Oard, D. W. (2022). Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models. In M. Hagen, S. Verberne, C. Macdonald, C. Seifert, K. Balog, K. Nørvåg, & V. Setty (Eds.), Advances in Information Retrieval (Vol. 13185, pp. 382–396). Springer International Publishing. https://doi.org/10.1007/978-3-030-99736-6_26
Perwira, R. I., Permadi, V. A., Purnamasari, D. I., & Agusdin, R. P. (2025). Domain-Specific Fine-Tuning of IndoBERT for Aspect-Based Sentiment Analysis in Indonesian Travel User-Generated Content. Journal of Information Systems Engineering and Business Intelligence, 11(1), 30–40. https://doi.org/10.20473/jisebi.11.1.30-40
Quan, Z., & Pu, L. (2022). An improved accurate classification method for online education resources based on support vector machine (SVM): Algorithm and experiment. Education and Information Technologies, 28(7), 8097–8111. (pub.1153602403). https://doi.org/10.1007/s10639-022-11514-6
Ranasinghe, T., Plum, A., Purschke, C., & Zampieri, M. (2023). Publish or Hold? Automatic Comment Moderation in Luxembourgish News Articles. Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings, 968–978. https://doi.org/10.26615/978-954-452-092-2_104
Rau, G., & Shih, Y.-S. (2021). Evaluation of Cohen’s kappa and other measures of inter-rater agreement for genre analysis and other nominal data. Journal of English for Academic Purposes, 53, 101026. https://doi.org/10.1016/j.jeap.2021.101026
Saetia, C., Thonglong, A., Amornchaiteera, T., Chalothorn, T., Taerungruang, S., & Buabthong, P. (2024). Streamlining event extraction with a simplified annotation framework. Frontiers in Artificial Intelligence, 7, 1361483. https://doi.org/10.3389/frai.2024.1361483
Shaik, T. B., Tao, X. D., Dann, C. E., Quadrelli, C., Li, Y., & O’Neill, S. (2022). Educational Decision Support System Adopting Sentiment Analysis on Student Feedback (J. Zhao, Y. Fan, E. Bagheri, N. Fuhr, & A. Takasu, Eds.; pp. 377–383). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/WI-IAT55865.2022.00062
Sunar, A. S., & Khalid, M. S. (2024). Natural Language Processing of Student’s Feedback to Instructors: A Systematic Review. In IEEE Transactions on Learning Technologies (Vol. 17, pp. 741–753). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/TLT.2023.3330531
Troiano, E., Velutharambath, A., & Klinger, R. (2023). From theories on styles to their transfer in text: Bridging the gap with a hierarchical survey. Natural Language Engineering, 29(4), 849–908. https://doi.org/10.1017/S1351324922000407
Wang, G., & Jaber, M. M. (2025). A Deep Learning Approach to Sentiment Analysis of Hotel Reviews: Comparing BERT and LSTM Models. International Journal of Advances in Artificial Intelligence and Machine Learning, 2(2), 67–75. https://doi.org/10.58723/ijaaiml.v2i2.403
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., & Purwarianti, A. (2020). IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, 843–857. https://doi.org/10.18653/v1/2020.aacl-main.85
Wilson, G., & Cook, D. J. (2020). A Survey of Unsupervised Deep Domain Adaptation. ACM Transactions on Intelligent Systems and Technology, 11(5), 1–46. https://doi.org/10.1145/3400066
Wongso, W., Joyoadikusumo, A., Buana, B. S., & Suhartono, D. (2023). Many-to-Many Multilingual Translation Model for Languages of Indonesia. IEEE Access, 11, 91385–91397. https://doi.org/10.1109/ACCESS.2023.3308818
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10, 291–306. https://doi.org/10.1162/tacl_a_00461
Zaikis, D., & Vlahavas, I. (2023). From Pre-Training to Meta-Learning: A Journey in Low-Resource-Language Representation Learning. In IEEE Access (Vol. 11, pp. 115951–115967). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ACCESS.2023.3326337
Zyout, I., & Zyout, M. (2024). Sentiment analysis of student feedback using attention-based RNN and transformer embedding. In IAES International Journal of Artificial Intelligence (Vol. 13, Issue 2, pp. 2171–2182). Institute of Advanced Engineering and Science. https://doi.org/10.11591/ijai.v13.i2.pp2173-2184
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Adam Bachtiar, Ahmad Ashril Rizal, Tuning Ridha Addhiny
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with Jurnal Penelitian dan Pengkajian Ilmu Pendidikan: e-Saintika agree to the following terms:
- For all articles published in Jurnal Penelitian dan Pengkajian Ilmu Pendidikan: e-Saintika, copyright is retained by the authors. Authors give permission to the publisher to announce the work with conditions. When the manuscript is accepted for publication, the authors agrees to implement a non-exclusive transfer of publishing rights to the journals.
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
