A comparative study of cross-lingual sentiment analysis
Expert Systems with Applications (2024)
Research topics:
Sentiment AnalysisAbstract
This paper presents a detailed comparative study of the zero-shot cross-lingual sentiment analysis. Namely, we use modern multilingual Transformer-based models and linear transformations combined with CNN and LSTM neural networks. We evaluate their performance in Czech, French, and English. We aim to compare and assess the models’ ability to transfer knowledge across languages and discuss the trade-off between their performance and training/inference speed. We build strong monolingual baselines comparable with the current SotA approaches, achieving state-of-the-art results in Czech (96.0% accuracy) and French (97.6% accuracy). Next, we compare our results with the latest large language models (LLMs), i.e., Llama 2 and ChatGPT. We show that the large multilingual Transformer-based XLM-R model consistently outperforms all other cross-lingual approaches in zero-shot cross-lingual sentiment classification, surpassing them by at least 3%. Next, we show that the smaller Transformer-based models are comparable in performance to older but much faster methods with linear transformations. The best-performing model with linear transformation achieved an accuracy of 92.1% on the French dataset, compared to 90.3% received by the smaller XLM-R model. Notably, this performance is achieved with just approximately 0.01 of the training time required for the XLM-R model. It underscores the potential of linear transformations as a pragmatic alternative to resource-intensive and slower Transformer-based models in real-world applications. The LLMs achieved impressive results that are on par or better, at least by 1%–3%, but with additional hardware requirements and limitations. Overall, this study contributes to understanding cross-lingual sentiment analysis and provides valuable insights into the strengths and limitations of cross-lingual approaches for sentiment analysis.
Authors
BibTex
@article{PRIBAN2024123247, title = {A comparative study of cross-lingual sentiment analysis}, journal = {Expert Systems with Applications}, volume = {247}, pages = {123247}, year = {2024}, issn = {0957-4174}, doi = {https://doi.org/10.1016/j.eswa.2024.123247}, url = {https://www.sciencedirect.com/science/article/pii/S095741742400112X}, author = {Pavel Přibáň and Jakub Šmíd and Josef Steinberger and Adam Mištera}, keywords = {Sentiment analysis, Zero-shot cross-lingual classification, Linear transformation, Transformers, Large language models, Transfer learning}, abstract = {This paper presents a detailed comparative study of the zero-shot cross-lingual sentiment analysis. Namely, we use modern multilingual Transformer-based models and linear transformations combined with CNN and LSTM neural networks. We evaluate their performance in Czech, French, and English. We aim to compare and assess the models’ ability to transfer knowledge across languages and discuss the trade-off between their performance and training/inference speed. We build strong monolingual baselines comparable with the current SotA approaches, achieving state-of-the-art results in Czech (96.0% accuracy) and French (97.6% accuracy). Next, we compare our results with the latest large language models (LLMs), i.e., Llama 2 and ChatGPT. We show that the large multilingual Transformer-based XLM-R model consistently outperforms all other cross-lingual approaches in zero-shot cross-lingual sentiment classification, surpassing them by at least 3%. Next, we show that the smaller Transformer-based models are comparable in performance to older but much faster methods with linear transformations. The best-performing model with linear transformation achieved an accuracy of 92.1% on the French dataset, compared to 90.3% received by the smaller XLM-R model. Notably, this performance is achieved with just approximately 0.01 of the training time required for the XLM-R model. It underscores the potential of linear transformations as a pragmatic alternative to resource-intensive and slower Transformer-based models in real-world applications. The LLMs achieved impressive results that are on par or better, at least by 1%–3%, but with additional hardware requirements and limitations. Overall, this study contributes to understanding cross-lingual sentiment analysis and provides valuable insights into the strengths and limitations of cross-lingual approaches for sentiment analysis.} }
