Рақамли технологияларнинг назарий ва амалий масалалари 9-том 2-нөмір (2026) · 45-53-беттер
Development and Evaluation of the ModernUzBERT Semantic Embedding Model with Extended Context for the Uzbek Language
Хужаяров, И.Ш., Очилов, М.М., Холматов, О.А., Жуманов, В.И.
Аңдатпа
This research is dedicated to the development of ModernUzBERT, an advanced embedding model for the Uzbek language based on contemporary Transformer architecture with an extended context window of 8192 tokens. In low-resource and agglutinative languages such as Uzbek, the limited capacity for processing comprehensive context significantly hampers model performance. To address this challenge, a large-scale text corpus in the Uzbek Latin script comprising over 125 million tokens was curated, and an optimized vocabulary of 52,000 sub-word units was developed. The model training process was executed in two primary stages: initially, the model was pre-trained from scratch using the Masked Language Modeling (MLM) objective; subsequently, it was adapted for semantic search via Supervised Fine-Tuning (SFT) using a dataset of 30,000 question-answer pairs. Through the implementation of Flash Attention 2 and Unpadding technologies, GPU resource utilization efficiency was substantially enhanced. The 8192-token context window enables the analysis of large-scale documents while preserving semantic integrity. Upon conclusion of the study, the developed model and all its components were released for open-access to the scientific community on the Hugging Face platform.
NLPузбекский языкModernBERTсемантические эмбеддингиинформационный поискдлинный контекстNLPuzbek languageModernBERTsemantic embedding
Метадеректер дереккөзі: журналдың OAI-PMH архиві · Sindex толық мәтінді сақтамайды, дереккөзге сілтеме береді.