Czert – Czech BERT-like Model for Language Representation
RANLP (2021)
Research topics:
Language Modeling | Sentiment Analysis | Document Classification | Named Entity RecognitionAbstract
This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pretrained and fine-tuned models freely for the research community
Authors
BibTex
@article{sido2021czert, title={Czert--Czech BERT-like Model for Language Representation}, author={Sido, Jakub and Pra{\v{z}}{\'a}k, Ond{\v{r}}ej and P{\v{r}}ib{\'a}{\v{n}}, Pavel and Pa{\v{s}}ek, Jan and Sej{\'a}k, Michal and Konop{\'\i}k, Miloslav}, journal={arXiv preprint arXiv:2103.13031}, year={2021} }
