Style over Substance? Predicting Answer Quality via Surface Linguistics

TL;DR

In the world of Community Question Answering (CQA), not all answers are created equal. This paper argues that we don't need complex semantic analysis or years of "vote" history to identify a good answer. Instead, surface linguistic features—the mechanical "skeleton" of writing like sentence length and vocabulary variety—can predict answer quality with remarkable accuracy, often outperforming user engagement data.

Background: The Metadata Trap

Traditionally, platforms like Yahoo! Answers or StackOverflow rely on non-textual features: How many people clicked this? How many "upvotes" did it get?

However, the authors point out a critical flaw: Data Sparseness and Recency Bias. A high-quality answer posted five minutes ago has zero clicks and zero votes. If search engines rely solely on engagement, this "hidden gem" remains buried under older, popular, yet potentially inferior content. The solution? Look at the text itself, but keep it computationally "cheap."

Methodology: The Power of Proxes

The core philosophy of this work is based on the concept of Proxes (computer approximations) for Trins (intrinsic human variables). For instance:

• Fluency can be approximated by text length.
• Diction can be approximated by the variation in word length.
• Readability can be approximated by lexical density.

The authors extracted 13 specific features, including:

• Chars/Words: Basic volume.
• Sents: Number of sentences.
• Lexdens: Ratio of unique words to total words.
• LW1-4: Counts of "long" words (more than 1-4 characters).

These features were fed into a Maximum Entropy (MaxEnt) Classifier, chosen for its ability to handle mutually dependent features and provide a clean probability score.

Figure 1: The system takes an answer, extracts surface features, and uses a classifier to output a probability of "High Quality."

Experimental Insights

The team tested their approach on 2,589 manually annotated Q&A pairs from Naver.

1. Linguistic vs. Non-Textual Features

The most striking finding was that surface linguistic features alone provided better ranking performance than non-textual features like click-through counts (when length was excluded).

2. The "Length" Paradox

Previous literature (like the PEG model for essay grading) suggested that simple word count was the king of quality features. However, this study found that the number of sentences (Sents) was actually the most salient feature.

Figure 2: The Recall-Precision graph clearly shows the linguistic classifier (solid line) dominating the random baseline, proving that "style" effectively signals "quality."

3. Feature Salience

While the "StyleSet" (lexical density, word lengths) is useful, the "LengthSet" (character/word/sentence counts) remains the most powerful predictor. Interestingly, adding a single surface feature (length) to engagement-based models causes a massive jump in their performance—proving that even "dumb" textual metrics are essential.

Table 3: Comparison between the "LengthSet" and "StyleSet" shows that volume-related metrics deliver higher average precision (0.9726).

Critical Analysis & Takeaways

The brilliance of this paper lies in its computational pragmatism. In a commercial web setting, running deep syntactic parsing or large-scale NLP on every user comment is prohibitively expensive. Surface features offer a "free lunch"—they are trivial to compute but highly correlated with human judgment of "informativeness" and "sincerity."

Limitations:

• The model can be "gamed." A spammer could theoretically write a very long, grammatically structured but nonsensical paragraph to fool the classifier.
• It measures style, not truth. An eloquent lie might rank higher than a blunt, poorly-formatted truth.

Future Outlook

The authors suggest that the next frontier is the integration of these quality scores into Language Modeling for IR. Instead of just matching keywords, search engines can use these quality probabilities as "prior probabilities" to ensure that the best-written answers always rise to the top of the results page.