The rapid growth of digital communication has created an enormous amount
of textual data, especially in the political sphere. Citizens now debate, criticize,
and share their opinions on politics across social media, news comments, blogs,
and online forums. For researchers, policymakers, and governments, analyzing
this type of discourse is crucial for understanding public opinion, identifying
emerging topics, and tracking societal reactions to political events. Two key tasks
play an essential role in this process: sentiment analysis and topic detection.
Research on sentiment analysis and topic detection has grown a lot in recent
years, especially with the rise of large language models (LLMs). One of the
most relevant datasets for political text is AgoraSpeech, which provides
dual annotations for both sentiment and 33 topic categories defined by data
journalists. The dataset is based on Greek texts, which are then translated into
English for annotation purposes. The authors have noted several challenges:
achieving high inter-annotator agreement was difficult for ambiguous or nuanced
sentences, and certain topics were harder to classify consistently.
When it comes to sentiment analysis in low-resource languages like Albanian,
the situation is even more challenging. Existing NLP tools are limited.
Furthermore, there are very few annotated datasets available, no dataset related
to political discourse and topic detection is even less explored. There is growing
recognition that large language models (LLMs) are not perfect annotators, especially
in low-resource settings. specifically evaluated LLMs on tasks such as
sentiment analysis, news classification, and hate speech detection, finding that
even advanced models underperformed compared to fine-tuned baselines.
The objective of this study is to evaluate how accurately an LLM can classify
both sentiment and topics in albanian political discourse compared to human
annotators. Political discourse is often rich in irony, sarcasm, rhetorical devices,
and culturally loaded references, all of which are challenging even for humans to
interpret consistently. As a result, evaluating how well AI can replicate human
judgment in this domain is both timely and necessary.