Beyond Questionnaires: Automating the Polarity Analysis of Spontaneous User Postings

TL;DR

Evaluating User Experience (UX) has traditionally been a manual, labor-intensive process involving interviews and surveys. This paper explores a more authentic alternative: Postings Related to the Use (PRUs)—spontaneous comments users leave while interacting with a social academic system. By comparing lexical tools with machine learning, the authors demonstrate that Naive Bayes (84.6% accuracy) is far more effective than generic dictionary tools at decoding the technical frustrations and praises of users.

Contextualizing the Problem: The Illusion of Accuracy in Lexicons

In the realm of Human-Computer Interaction (HCI), the "Observer Effect" is a persistent hurdle. When you ask a user a question about a system, their answer is colored by the question itself. PRUs solve this by capturing users "in the wild." However, the sheer volume of text data makes manual analysis impossible.

The authors identify a critical gap: standard sentiment analysis tools often fail in technical domains. For example, a phrase like "Google Chrome is much better" is objectively positive, but if a system only supports Firefox, that sentence is actually a negative critique of the system's limitations.

Methodology: Lexicon vs. Machine Learning

The research analyzed 1,345 postings from SIGAA, a Brazilian academic system. After filtering for relevance, 832 sentences were processed through two distinct pipelines:

1. Lexical (SentiStrength): Uses a pre-defined dictionary where words have "strength" values (e.g., +5 for extremely positive).
2. Machine Learning (Naive Bayes): Uses a probabilistic approach, training the model on pre-labeled data to understand the specific relationship between words and sentiments in this specific academic system.

Typical PRUs labeled with and without sentiment.

Key Findings: Why Generic Tools Fail

The experimental results were stark. SentiStrength struggled significantly with negative feedback (20% recall), likely because technical frustration is often expressed through functionality-specific nouns rather than generic "bad" adjectives.

Figure 1: Naive Bayes significantly outperformed SentiStrength, especially in identifying negative and neutral sentiments.

The study also extracted a "Keyword Map" for each polarity:

• Positive: "better," "looks," "cool," "like."
• Negative: "module," "enrollment," "new" (indicating resistance to change), "problem."
• Neutral: "teacher," "history," "grade."

Critical Insight: The Context Problem

One of the paper's most salient points is the contextual shift. In an academic social system, the word "grade" (nota) or "enrollment" (matrícula) are usually neutral technical terms. However, when users discuss a "new module," the frequency analysis shows it shifts toward the negative polarity. Generic sentiment dictionaries cannot "see" that a technical update might be causing friction unless they are specifically tuned to that lifecycle phase of the software.

Conclusion and Future Outlook

While Naive Bayes showed high performance, the authors admit a major limitation: supervised learning requires manual labeling, which is "onerous." To move toward a truly automatic textual evaluation framework (like their proposed UUX-Post tool), we need hybrid systems that can learn technical jargon in a zero-shot or few-shot manner.

This work serves as a foundational step in moving HCI from reactive surveys to proactive, automated sentiment monitoring of social systems.

Takeaway for Practitioners: If you are building a feedback sentiment pipeline for your product, don’t rely solely on off-the-shelf sentiment APIs. Your users’ most critical "negative" feedback likely involves neutral-sounding technical terms used in a context of frustration.