Filling the Gaps: Intelligent Sentence Complementation for Real-Time Crowdsourcing

TL;DR

In the high-stakes world of real-time interpretation (transcribing lectures or sign language), workers often leave sentences unfinished due to time pressure. This paper proposes a method to "repair" these incomplete task results by borrowing words from other workers. By analyzing word co-occurrence and sentence similarity, the system fills in the blanks, turning fragmented notes into high-quality candidates for a final majority vote.

Background: The "Incompleteness" Problem in Crowdsourcing

Crowdsourcing is a powerful tool for tasks like transcription, but its real-time application faces a significant hurdle: Time Pressure. When a worker is transcribing a live lecture or interpreting sign language, they might miss words, resulting in "incomplete task results."

Standard quality control methods, such as Majority Voting, work well for multiple-choice questions but fail miserably for free-text inputs. If three workers all provide different, broken versions of the same sentence, who is right? The authors argue that we shouldn't just pick the "best" broken sentence; we should synthesize a complete one.

Methodology: The Logic of Complementation

The authors model the problem by treating missing information as a wildcard (*). Their goal is to replace that wildcard with a word found in another worker's result for the same task.

1. The Core Insight: Co-occurrence Matters

The system doesn't just pick a random word. It follows a hypothesis that "a good complement result resembles the original task result without disrupting existing word relationships."

2. The Selection Process

The algorithm filters candidates through four conditions:

• Condition 1 (Stability): The chosen word should not drastically change the co-occurrence patterns already present in the sentence.
• Condition 2 (Similarity): The resulting sentence should be as similar as possible to the collective pool of results (using Jaccard Similarity and 2-gram units).
• Condition 3 & 4 (Heuristics): Particles or auxiliary verbs are ignored as candidates, and words already present in the sentence are excluded to avoid redundancy.

Figure 1: The workflow of receiving Japanese interpreter sentences and outputting complemented candidates.

Experiments: Consistency is Key

The researchers tested their method on two distinct datasets:

1. Data set A (Sign Language Interpreter): High consistency, vocabulary follow Japanese word order.
2. Data set B (English-to-Japanese Translation): Lower consistency, varied vocabulary (e.g., proper nouns not unified).

Key Results

The results were striking in their disparity:

• In Data set A, nearly 70% of sentences were improved.
• In Data set B, only 25% improved, while 66% actually became worse.

Table 1: Evaluation of individual interpreter sentences across different datasets.

Why the gap?

The method thrives on glossarially-controlled environments. If workers use wildly different synonyms for the same concept, the co-occurrence logic breaks down. Furthermore, morphological analysis (splitting words) sometimes failed—for example, treating "Italian" as two separate entities ("Italy" + "Language"), preventing a successful match.

Critical Insight & Future Outlook

This work demonstrates that we can "bootstrap" quality from a crowd without needing an external training corpus. By leveraging the diversity of the crowd, we can fill the gaps left by individual failure.

However, the heavy reliance on Jaccard Similarity and 2-grams makes the system fragile when faced with creative or inconsistent language. The takeaway for future research? Integrating modern NLP embeddings or LLMs could likely solve the "synonym" problem that hindered Data set B, allowing the system to understand that two different words might mean the same thing, thus making the complementation process much more robust.

Conclusion

The proposed method is a significant step toward real-time quality assurance in crowdsourcing. While it currently excels in structured environments like sign-language-to-text, its "collaborative repair" philosophy offers a blueprint for more resilient human-in-the-loop AI systems.