Auditable Crowdsourced Digital Dictionary for Low-Resource Toulour with Gamification and Functional Evaluation

Kenneth Palilingan, Salaki Joshua, Salvius Lengkong

Abstract


Toulour, also documented as Tondano/Toundano, is a low-resource Minahasan language whose digital lexical resources require accessible retrieval and accountable community maintenance. This study designs and evaluates an auditable crowdsourced and gamified Toulour–Indonesian digital dictionary in which participation incentives are separated from lexical authority. Using Design Science Research Methodology (DSRM), the artifact integrates bidirectional lexical access, contributor proposals, validator-controlled publication, persistent before/after provenance, experience points, quizzes, badges, feedback, and role-based administration. Functional evaluation used 50 predefined black-box cases structured by ISO/IEC 25010:2023 Functional Suitability and informed by ISO/IEC/IEEE 29119-4:2021; 49 cases passed (98.0%), with one oversized-query boundary failure retained. Supplementary evaluation included a pilot user study, automated application-surface security scanning, and staged local load testing. Of 40 questionnaire submissions, 34 eligible adults who had tried the application were analyzed; mean System Usability Scale (SUS) was 77.79, while descriptive perceived usefulness, perceived ease of use, and behavioral intention means were 4.10, 4.07, and 3.90 on a 1–5 scale. Final context-scoped OWASP ZAP verification reported 0 High, 3 Medium, 3 Low, and 4 Informational alert types. k6 testing at 1, 10, 25, and 50 virtual users observed no HTTP request failures, although latency rose substantially at higher concurrency and throughput gains flattened between 25 and 50 virtual users. The results support an operational integration of governed crowdsourcing, reconstructable provenance, participation-linked incentives, and complementary evaluation while retaining explicit pilot, local-deployment, and residual-security limitations.

Keywords


controlled crowdsourcing, digital dictionary, gamification, lexical provenance, low-resource language

References


S. Bird, “Local Languages, Third Spaces, and Other High-Resource Scenarios,” in Proc. 60th Annual Meeting of the Association for Computational Linguistics, 2022, pp. 7817-7829, DOI: 10.18653/v1/2022.acl-long.539.

A. Wantalangi, G. J. Tendean, S. Wuntu, A. Kawengian, and E. Kaligis, Kamus Tondano-Indonesia. Jakarta, Indonesia: Pusat Pembinaan dan Pengembangan Bahasa, 1985.

T. C. Brickell, Tondano (Toundano): A Grammar Sketch of an Endangered Minahasan Language. Routledge, 2022, DOI: 10.4324/9780429508165.

G. I. Winata et al., “NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages,” in Proc. 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 815-834, DOI: 10.18653/v1/2023.eacl-main.57.

S. Cahyawijaya et al., “NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages,” in Proc. IJCNLP-AACL, 2023, pp. 921-945, DOI: 10.18653/v1/2023.ijcnlp-main.60.

S. Cahyawijaya et al., “NusaCrowd: Open Source Initiative for Indonesian NLP resources,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 13745-13818, DOI: 10.18653/v1/2023.findings-acl.868.

M. Maxwell and A. Bills, “Endangered Data for Endangered Languages: Digitizing Print Dictionaries,” in Proc. 2nd Workshop on the Use of Computational Methods in the Study of Endangered Languages, 2017, pp. 85-91, DOI: 10.18653/v1/W17-0112.

M. Saltzman, “Jejueo Talking Dictionary: A Collaborative Online Database for Language Revitalization,” in Proc. 2nd Workshop on the Use of Computational Methods in the Study of Endangered Languages, 2017, pp. 122-129, DOI: 10.18653/v1/W17-0117.

N. F. Hasan and M. S. A. Iribaram, “Digitalisasi Kamus Bahasa Daerah Papua menggunakan Metode Rapid Application Development,” MATRIK, Vol. 21, No. 3, pp. 710-720, 2022, DOI: 10.30812/matrik.v21i3.1688.

B. Hunt, L. Schwartz, S. Schreiner, and E. Chen, “Community Consultation and the Development of an online Akuzipik-English Dictionary,” in Proc. AmericasNLP, 2023, pp. 134-143, DOI: 10.18653/v1/2023.americasnlp-1.14.

C. Pramartha, G. P. W. Rajeg, M. Dalrymple, and I. W. Arka, “Preserving the Enggano language: A Digital Dictionary Approach,” in Proc. Australasian Conf. Information Systems, 2024, paper 9.

A. L. Daigneault and G. Anderson, “Citizen Linguists and Decolonial Lexicography: Co-Creative Dictionary-Building in Grassroots Digital Language Documentation,” in Proc. 8th Workshop on the Use of Computational Methods in the Study of Endangered Languages, 2025, pp. 20-28.

H. Khalilia, J. Otterbacher, G. Bella, S. Darma, and F. Giunchiglia, “Crowdsourcing Lexical Diversity,” Frontiers in Artificial Intelligence, Vol. 8, 2025, DOI: 10.3389/frai.2025.1648073.

J. Janetzki, G. De Melo, J. Nemecek, and D. Whitenack, “GUIDE: Creating Semantic Domain Dictionaries for Low-Resource Languages,” in Proc. 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP (SIGTYP), 2024, pp. 10-24, DOI: 10.18653/v1/2024.sigtyp-1.2.

S. Walker, W. Wu, B. Torres Fischer, and L. Kimura, “Kuene: A Web Platform for Facilitating Hawaiian Word Neologism,” in Proc. 8th Workshop on the Use of Computational Methods in the Study of Endangered Languages, 2025, pp. 182-187. [Online]. Available: https://aclanthology.org/2025.computel-main.21/

K. Narayanan R et al., “Field to Model: Pairing Community Data Collection with Scalable NLP through the LiFE Suite,” in Proc. 4th Workshop on NLP Applications to Field Linguistics, 2025, pp. 76-84. [Online]. Available: https://aclanthology.org/2025.fieldmatters-1.7/

B. Morschheuser, J. Hamari, J. Koivisto, and A. Maedche, “Gamified Crowdsourcing: Conceptualization, Literature Review, and Future Agenda,” International Journal of Human-Computer Studies, Vol. 106, pp. 26-43, 2017, DOI: 10.1016/j.ijhcs.2017.04.005.

H. Ogawa, H. Nishikawa, T. Tokunaga, and H. Yokono, “Gamification Platform for Collecting Task-Oriented Dialogue Data,” in Proc. 12th Language Resources and Evaluation Conference, 2020, pp. 7084-7093.

G. Eryiğit, A. Şentaş, and J. Monti, “Gamified Crowdsourcing for Idiom Corpora Construction,” Natural Language Engineering, Vol. 29, No. 4, pp. 909-941, 2023, DOI: 10.1017/S1351324921000401.

F. Genovese, M. M. Bolognesi, A. Di Iorio, and F. Vitali, “The Advantages of Gamification for Collecting Linguistic Data: A Case Study using Word Ladders,” Online Journal of Communication and Media Technologies, Vol. 14, No. 2, e202426, 2024, DOI: 10.30935/ojcmt/14443.

D. Torunoğlu Selamet, A. Şentaş, and G. Eryiğit, “Gamified Crowd-Sourcing for Word Sense Disambiguation of Turkish,” ACM Transactions on Asian and Low-Resource Language Information Processing, Vol. 24, No. 11, Art. No. 130, pp. 1-23, 2025, DOI: 10.1145/3767160.

K. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “A Design Science Research Methodology for Information Systems Research,” Journal of Management Information Systems, Vol. 24, No. 3, pp. 45-77, 2007, DOI: 10.2753/MIS0742-1222240302.

J. Venable, J. Pries-Heje, and R. Baskerville, “FEDS: A Framework for Evaluation in Design Science Research,” European Journal of Information Systems, Vol. 25, No. 1, pp. 77-89, 2016, DOI: 10.1057/ejis.2014.36.

ISO/IEC 25010:2023, Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—Product Quality Model, 2nd ed. Geneva, Switzerland: International Organization for Standardization, 2023.

ISO/IEC/IEEE 29119-4:2021, Software and Systems Engineering—Software Testing—Part 4: Test Techniques, 2nd ed. Geneva, Switzerland: International Organization for Standardization, 2021.

J. Brooke, “SUS: A ‘Quick and Dirty’ Usability Scale,” in Usability Evaluation in Industry, P. W. Jordan, B. Thomas, I. L. McClelland, and B. Weerdmeester, Eds. London, U.K.: Taylor & Francis, 1996, pp. 189–194.

F. D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly, Vol. 13, No. 3, pp. 319–340, 1989, DOI: 10.2307/249008.

ZAP by Checkmarx, “ZAP 2.17.0,” 2025. [Online]. Available: https://www.zaproxy.org/docs/desktop/releases/2.17.0/

Grafana Labs, “Grafana k6 Documentation: Load Testing Websites,” [Online]. Available: https://grafana.com/docs/k6/latest/testing-guides/load-testing-websites/




DOI: https://doi.org/10.32520/stmsi.v15i9.6936

Article Metrics

Abstract view : 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.