CHITL: Solving the "Unknown Unknowns" in Relation Extraction with Human-in-the-Loop Denoising

TL;DR

Relation Extraction (RE) under distant supervision is plagued by "Unknown Unknowns" (UUs)—noisy labels where models are confidently wrong. The CHITL (Crowdsourcing Based Human-In-The-Loop) framework tackles this by using an attention-based LSTM to "sniff out" suspicious data points and employing human crowd-workers to fix them, leading to a significant 10% jump in precision for standard CNN models.

The "Confidence Trap": Why Distant Supervision Fails

In the world of NLP, labeling data is expensive. Distant Supervision (DS) was the industry's "free lunch"—it automatically labels text by aligning it with Knowledge Bases (like Freebase). For example, if (Trump, USA) has a "nationality" relation in a database, DS assumes every sentence mentioning both entities expresses that relation.

However, this creates two deadly types of noise:

1. Wrong Labels: "Trump visited the Apple Inc. in the US" (No actual nationality relation expressed).
2. Missing Labels: The KB doesn't know "Trump was born in New York," so it labels it as 'NA'.

These are Unknown Unknowns (UUs). They are particularly dangerous because current SOTA models (RL and Adversarial-based) often just "guess" which samples are noisy without any objective ground truth to correct them.

Methodology: The CHITL Framework

The core philosophy of CHITL is simple: Automate the discovery, humanize the correction. It splits the task into three distinct modules:

1. The Intelligent Selector (LSTM + Attention)

Instead of checking every sentence, the framework uses an attention-based LSTM. It calculates Entity-pair Embedding ($V_{e1}-V_{e2}$) to focus the model's attention on words that actually define the relationship. If the model's prediction differs from the KB label with high confidence, it flags it as a "Potential UU."

2. Strategic Crowdsourcing

To prevent human error and keep costs low, CHITL uses two filters:

• Participant Quality: Mixing "Golden Data" (pre-verified answers) into the tasks to screen out low-quality workers.
• Majority Vote: Ensuring consensus before a label is updated.

3. Iterative Feedback Loop

The newly cleaned data is fed back into the training set. This doesn't just help the final classifier; it also helps the Selector get better at finding more subtle UUs in the next epoch.

Experiments: Superior Denoising Performance

The authors tested CHITL on the classic New York Times (NYT) dataset. The results prove that even a small amount of human intervention can outperform "pure" algorithmic denoising.

Key Performance Gains:

• Precision @ 100: Jumped from 75.2% to 85.2% for CNN models.
• Versatility: When added to existing denoising frameworks like ADV (Adversarial) or RL (Reinforcement Learning), CHITL consistently boosted their AUC and F1 scores.

The table below highlights that for the /people/person/place_lived relation, the selector achieved an impressive hit rate—identifying 162 real UUs out of 280 potential flags.

Critical Insight: Data Quality over Model Complexity

The most striking takeaway is that human-in-the-loop (HITL) frameworks are model-independent. Whether you are using a simple CNN or a complex BiRNN, cleaning the "UUs" provides a performance floor that no amount of hyperparameter tuning can match.

Moreover, CHITL identifies "hidden" relations (Missing Labels) that the model would previously have ignored, effectively expanding the Knowledge Base through the extraction process itself.

Conclusion & Future Outlook

While CHITL effectively bridges the gap between machine confidence and human truth, the reliance on crowdsourcing still incurs a financial cost. Future research could look into Active Learning to further minimize the number of human queries needed or leverage Large Language Models (LLMs) as "proxy humans" to label these UUs at scale.

Keywords: Relation Extraction, Distant Supervision, Human-in-the-loop, Crowdsourcing, Unknown Unknowns, Denoising.