Czert – Czech BERT-like Model for Language Representation

Jakub Sido and Ondřej Pražák and Pavel Přibáň and Miloslav Konopík
RANLP (2021)



This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pretrained and fine-tuned models freely for the research community



@article{sido2021czert, title={Czert--Czech BERT-like Model for Language Representation}, author={Sido, Jakub and Pra{\v{z}}{\'a}k, Ond{\v{r}}ej and P{\v{r}}ib{\'a}{\v{n}}, Pavel and Pa{\v{s}}ek, Jan and Sej{\'a}k, Michal and Konop{\'\i}k, Miloslav}, journal={arXiv preprint arXiv:2103.13031}, year={2021} }
Back to Top