Automated systems for assessing student knowledge based on artificial intelligence: development and empirical verification
DOI:
https://doi.org/10.46502/issn.1856-7576/2026.20.03.9Keywords:
automated student assessment, dominance analysis, higher education, learning analytics, personalized feedback.Abstract
The study aimed to develop a three-level conceptual model of an AI-based automated knowledge assessment system and to provide empirical evidence regarding its performance within a higher education context. For this purpose, a quasi-experimental design (pretest–posttest) was used with the participation of 250 students of humanities specialties (bachelor's: n = 163; master's: n = 87). The toolkit consisted of parallel forms of the academic achievement test (40 tasks, α = 0.83), an adapted TAM scale. For analysis, paired t-test, confirmatory factor analysis, hierarchical multiple regression and dominance analysis were used. Students demonstrated higher post-test academic achievement following implementation of the AKS model. The analysis showed a hierarchy of relative importance of levels (R² = .61): semantic-discursive level (50.8%) > personalized feedback (31.1%) > syntactic-formal (18.1%). Students positively perceived the system on all TAM dimensions: PU = 3.91, PEOU = 3.67, OS = 3.78. Therefore, the hierarchy of predictors became an important recommendation for developers of the ASZ regarding the priority of investments in the semantic level. The findings suggest that a phased implementation strategy may represent a pedagogically promising approach. The results are relevant for higher education institutions operating in a distance or blended format.
References
Aquino, M. (2026). From Freire’s critical pedagogy to the age of Artificial Intelligence: Continuities, ruptures, and new educational challenges. Science Insights Education Frontiers, 33(2), 5387–5401. https://doi.org/10.15354/sief.26.re124
Basilotta-Gómez-Pablos, V., Matarranz, M., Casado-Aranda, L.-A., & Otto, A. (2022). Teachers’ digital competencies in higher education: A systematic literature review. International Journal of Educational Technology in Higher Education, 19(1), 8. https://doi.org/10.1186/s41239-021-00312-8
Byrka, M., Sushchenko, A., Luchko, V., Perun, G., & Luchko, V. (2024). Algorithmic thinking in higher education: Determining observable and measurable content. Information Technologies and Learning Tools, 104(6), 1–13. https://doi.org/10.33407/itlt.v104i6.5750
Carvalho, R., Lins, L. F., Rodrigues, L., Miranda, P., Oliveira, H., Cordeiro, T., Bittencourt, I. I., Isotani, S., & Mello, R. F. (2024). Exploring NLP and embedding for automatic essay scoring in the Portuguese. In A. M. Olney, I.-A. Chounta, Z. Liu, O. C. Santos, & I. I. Bittencourt (Eds.), Artificial intelligence in education. Posters and late breaking results, workshops and tutorials, industry and innovation tracks, practitioners, doctoral consortium and blue sky (pp. 228–233). Springer. https://doi.org/10.1007/978-3-031-64315-6_18
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2, 100027. https://doi.org/10.1016/j.caeai.2021.100027
Cechinel, C., Ochoa, X., Lemos dos Santos, H., Carvalho Nunes, J. B., Rodés, V., & Marques Queiroga, E. (2020). Mapping learning analytics initiatives in Latin America. British Journal of Educational Technology, 51(4), 892–914. https://doi.org/10.1111/bjet.12941
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.
da Silva, L. M., Dias, L. P. S., Barbosa, J. L. V., Rigo, S. J., dos Anjos, J. C. S., Geyer, C. F. R., & Leithardt, V. R. Q. (2022). Learning analytics and collaborative groups of learners in distance education: A systematic mapping study. Informatics in Education, 21(1), 113–146. https://doi.org/10.15388/infedu.2022.05
Deroncele-Acosta, A., Sayán-Rivera, R. M. E., Mendoza-López, A. D., & Norabuena-Figueroa, E. D. (2025). Generative artificial intelligence and transversal competencies in higher education: A systematic review. Applied System Innovation, 8(3), 83. https://doi.org/10.3390/asi8030083
Dos Santos, R. F., Da Silva, J. d. M. G., De Souza, M. H., & Borges, P. C. R. (2026). Development of an artificial intelligence model to support teaching in dynamic educational environments. Remunom, 13(03), 1–25. https://doi.org/10.66104/fkfg6q38
Duarte Ehlert, A., Da Rocha Junqueira, J., Astrogildo de Freitas, L., & Brisolara Corrêa, U. (2025). A Review of recent advances on automatic question generation for the portuguese language. The International FLAIRS Conference Proceedings, 38. https://doi.org/10.32473/flairs.38.1.138961
Dung, D. A., Toan, N. T., Minh, N. D., & Anh, N. Q. (2026). Reimagining quality: Artificial intelligence, governance and the politics of data in higher education. Higher Education for the Future, 13(1), 50–73 https://doi.org/10.1177/23476311251386123
Durso, S. d. O., & Arruda, E. P. (2022). Artificial intelligence in distance education: A systematic literature review of Brazilian studies. Problems of Education in the 21st Century, 80(5), 679–692. https://doi.org/10.33225/pec/22.80.679
Fajardo-Ramos, D. C., Chiappe, A., & Mella-Norambuena, J. (2025). Human-in-the-loop assessment with AI: Implications for teacher education in Ibero-American universities. Frontiers in Education, 10, 1710992. https://doi.org/10.3389/feduc.2025.1710992
Falcão, T. P., Mello, R. F., Rodrigues, R. L., Diniz, J. R. B., Tsai, Y.-S., & Gašević, D. (2020). Perceptions and expectations about learning analytics from a brazilian higher education institution. In LAK '20: 10th International Conference on Learning Analytics and Knowledge. ACM. https://doi.org/10.1145/3375462.3375478
Feitosa, R. G. F., Campos, G. A. L. d., Santos, I. d. S., Gonçalves, C. H. R., Serra, A. d. B., Oliveira, A. R. d., Feitosa, P. L. P., Santos, Y. D., Bispo, E. L., & Esmeraldo, G. Á. R. M. (2025). Competency and skill-based educational recommendation system. International Journal of Artificial Intelligence in Education, 35(1), 135-154. https://doi.org/10.1007/s40593-024-00423-z
Fernandes, R. M., Nagata, V. d. M. N., Leal Filho, W., Rampasso, I. S., Avila, L. V., da Silva, J. M. N., Anholon, R., & Martins, V. W. B. (2026). Proposal of a framework to enhance the adoption of artificial intelligence in higher education considering sustainable development: a hybrid Lawshe-TOPSIS approach. International Journal of Educational Management, 40(1-2), 8–28. https://doi.org/10.1108/ijem-08-2024-0441
Ferreira, H. N. M., Brant-Ribeiro, T., Araujo, R. D., Dorca, F. A., & Cattelan, R. G. (2017). An automatic and dynamic knowledge assessment module for adaptive educational systems. In 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT). IEEE. https://doi.org/10.1109/icalt.2017.86
Garcia, S., Marques, E. C. M., Mello, R. F., Gašević, D., & Falcão, T. P. (2021). Aligning expectations about the adoption of learning analytics in a brazilian higher education institution. У Lecture Notes in Computer Science (с. 173–177). Springer International Publishing. https://doi.org/10.1007/978-3-030-78270-2_31
Guardia-Paniura, C. H., Cueva-Luza, T., Cruz-Carpio, F. M., Ito-Díaz, R. R., Apaza-Paco, D. V., Rosas-Rojas, N., Mamani-Mamani, B., Terrero-Pérez, Á., Yaedú, R. Y. F., & Peralta-Mamani, M. (2026). Human and AI-generated feedback in higher education: A systematic review of effectiveness and student perceptions. Contemporary Educational Technology, 18(1), ep623. https://doi.org/10.30935/cedtech/17863
Guerrero-Quiñonez, A. J., Bedoya-Flores, M. C., Mosquera-Quiñonez, E. F., Mesías-Simisterra, Á. E., & Bautista-Sánchez, J. V. (2023). Artificial Intelligence and its scope in Latin American higher education. Ibero-American Journal of Education & Society Research, 3(1), 264–271. https://doi.org/10.56183/iberoeds.v3i1.627
Hilliger, I., Ortiz‐Rojas, M., Pesántez‐Cabrera, P., Scheihing, E., Tsai, Y., Muñoz‐Merino, P. J., Broos, T., Whitelock‐Wainwright, A., Gašević, D., & Pérez‐Sanagustín, M. (2020). Towards learning analytics adoption: A mixed methods study of data‐related practices and policies in Latin American universities. British Journal of Educational Technology, 51(4), 915–937. https://doi.org/10.1111/bjet.12933
Hu, L.-t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
Kosasi, S., Vedyanto, Kasma, U., & Ayu Eka Yuliani, I. D. (2020). The mediating role of learning analytics to improve student academic performance. У 2020 2nd International Conference on Cybernetics and Intelligent System (ICORIS). IEEE. https://doi.org/10.1109/icoris50180.2020.9320802
Leal Filho, W., Ribeiro, P. C. C., Mazutti, J., Lange Salvia, A., Bonato Marcolin, C., Lima Silva Borsatto, J. M., Sharifi, A., Sierra, J., Luetz, J., Pretorius, R., & Viera Trevisan, L. (2024). Using artificial intelligence to implement the UN sustainable development goals at higher education institutions. International Journal of Sustainable Development & World Ecology, 31(6), 726–745. https://doi.org/10.1080/13504509.2024.2327584
Lima, Y. O. d., Silveira Braga, C. A., & Pereira, I. F. (2026). Estimating the impact of industry 4.0 automation on curricular competence indicators in brazilian vocational education and training: A mixed-methods ai-supported analysis. International Journal for Research in Vocational Education and Training, 13(2), 211–236. https://doi.org/10.13152/ijrvet.13.2.3
Lin, F., Morland, R., & Yan, H. (2024). QuizMaster: An adaptive formative assessment system. In Generative Intelligence and Intelligent Tutoring Systems (p. 55–67). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-63028-6_5
Lourenço, A. A., & Valente, S. (2026). Artificial Intelligence in learning: An integrative framework for education 4.0. Creative Education, 17(3), 393–422. https://doi.org/10.4236/ce.2026.173025
Martinho, V. R. D. C., Nunes, C., & Minussi, C. R. (2013). An intelligent system for prediction of school dropout risk group in higher education classroom based on artificial neural networks. In 2013 IEEE 25th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE. https://doi.org/10.1109/ictai.2013.33
Mello, R. F., Neto, R., Fiorentino, G., Alves, G., Arêdes, V., Silva, J. V. G. F., Falcão, T. P., & Gašević, D. (2022). Enhancing instructors’ capability to assess open-response using natural language processing and learning analytics. In Lecture Notes in Computer Science (pp. 102–115). Springer International Publishing. https://doi.org/10.1007/978-3-031-16290-9_8
Oksanen, A., Osma, T., Heiskari, M., Cvetkovic, A., Ruokosuo, E. S., Koike, M., Arriaga, P., & Savolainen, I. (2026). Mapping AI learning readiness self-efficacy worldwide: Scale validation and cross-continental patterns. Computers in Human Behavior: Artificial Humans, 7, 100251. https://doi.org/10.1016/j.chbah.2026.100251
Ramírez-Montoya, M. S., Castillo-Martínez, I. M., Sanabria-Z, J., & Miranda, J. (2022). Complex thinking in the framework of education 4.0 and open innovation—a systematic literature review. Journal of Open Innovation: Technology, Market, and Complexity, 8(1), 4. https://doi.org/10.3390/joitmc8010004
Rodrigues, L., Toda, A. M., Palomino, P. T., Oliveira, W., & Isotani, S. (2020). Personalized gamification: A literature review of outcomes, experiments, and approaches. In TEEM'20: Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality. ACM. https://doi.org/10.1145/3434780.3436665
Ruiz-Casado, M., Alfonseca, E., & Castells, P. (2005). Automatic extraction of semantic relationships for Wordnet by means of pattern learning from Wikipedia. In Natural Language Processing and Information Systems (pp. 67–79). Springer Berlin Heidelberg. https://doi.org/10.1007/11428817_7
Salas‐Pilco, S. Z., & Yang, Y. (2020). Learning analytics initiatives in Latin America: Implications for educational researchers, practitioners and decision makers. British Journal of Educational Technology, 51(4), 875–891. https://doi.org/10.1111/bjet.12952
Skrypnyk, A., Lytvyn, N., Kholod, I., Didenko, N., & Ivashchuk, A. (2024). Linguistic imagology as a new approach to the analysis of linguistic images: Methods and theoretical aspects. Amazonia Investiga, 13(73), 220–230. https://doi.org/10.34069/ai/2024.73.01.18
Toda, A. M., Valle, P. H. D., & Isotani, S. (2018). The dark side of gamification: An overview of negative effects of gamification in education. In Communications in Computer and Information Science (pp. 143–156). Springer International Publishing. https://doi.org/10.1007/978-3-319-97934-2_9
UNESCO. (2023). New UNESCO global survey reveals impact of COVID-19 on higher education.
https://www.unesco.org/en/articles/new-unesco-global-survey-reveals-impact-covid-19-higher-education
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Volodymyr Luchko, Iryna Nazarenko, Alla Horobets, Olga Serhiienko, Olena Bazyl
This work is licensed under a Creative Commons Attribution 4.0 International License.
