Poster De Conférence Année : 2025

Measuring the reliability of LLM Annotations in albanian political discourse

Résumé

The rapid growth of digital communication has created an enormous amount of textual data, especially in the political sphere. Citizens now debate, criticize, and share their opinions on politics across social media, news comments, blogs, and online forums. For researchers, policymakers, and governments, analyzing this type of discourse is crucial for understanding public opinion, identifying emerging topics, and tracking societal reactions to political events. Two key tasks play an essential role in this process: sentiment analysis and topic detection. Research on sentiment analysis and topic detection has grown a lot in recent years, especially with the rise of large language models (LLMs). One of the most relevant datasets for political text is AgoraSpeech, which provides dual annotations for both sentiment and 33 topic categories defined by data journalists. The dataset is based on Greek texts, which are then translated into English for annotation purposes. The authors have noted several challenges: achieving high inter-annotator agreement was difficult for ambiguous or nuanced sentences, and certain topics were harder to classify consistently. When it comes to sentiment analysis in low-resource languages like Albanian, the situation is even more challenging. Existing NLP tools are limited. Furthermore, there are very few annotated datasets available, no dataset related to political discourse and topic detection is even less explored. There is growing recognition that large language models (LLMs) are not perfect annotators, especially in low-resource settings. specifically evaluated LLMs on tasks such as sentiment analysis, news classification, and hate speech detection, finding that even advanced models underperformed compared to fine-tuned baselines. The objective of this study is to evaluate how accurately an LLM can classify both sentiment and topics in albanian political discourse compared to human annotators. Political discourse is often rich in irony, sarcasm, rhetorical devices, and culturally loaded references, all of which are challenging even for humans to interpret consistently. As a result, evaluating how well AI can replicate human judgment in this domain is both timely and necessary.

Fichier principal
Vignette du fichier
Poster-Template ATALA. Decembre 2025.pdf (654.84 Ko) Télécharger le fichier
Poster ATALA_journée. Decembre 2025.pdf (122.3 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05411051 , version 1 (07-01-2026)

Licence

Identifiants

  • HAL Id : hal-05411051 , version 1

Citer

Ueda Qorrasi, Aude Grezka, Eglantina Gishti, Nathalie Pernelle. Measuring the reliability of LLM Annotations in albanian political discourse. Journée d'études AFIA-ATALA : Technologies linguistiques pour les langues peu dotées, Dec 2025, Paris, France. ⟨hal-05411051⟩
176 Consultations
60 Téléchargements

Partager

  • More