TAUCONS: Engineering Serendipity Through Multi-Source NLP Interest Extraction

TL;DR

Can we program "luck"? This paper investigates the feasibility of using NLP to extract Twitter users' interests—specifically their peripheral interests—to suggest serendipitous connections. By analyzing not just a user's tweets, but also the web pages they link to and the hashtags they share, the authors developed TAUCONS, a system capable of identifying topics that are both relevant and surprising to the user.

Background Positioning

In the landscape of social computing, most recommendation engines focus on high-probability matches (central interests). This paper resides in the niche but vital area of Serendipitous Discovery, shifting the focus from "more of the same" to "valuable unexpectedness." It treats Twitter as a real-time sensor for raw, unrationalized human thought.

Problem & Motivation: The Paradox of Choice

Current social media analysis primarily relies on metadata (hashtags, follower counts) rather than deep semantic content. The authors argue that:

1. Central vs. Peripheral: You won't find surprising connections in your core research area; serendipity lives on the periphery.
2. The Noise Barrier: Tweets are short (140 characters at the time of study), ungrammatical, and full of typos, making standard NLP tools fail.
3. The Goal: To find "unsought findings"—connections that the user didn't know they wanted until they saw them.

Methodology: The TAUCONS Architecture

The authors propose that interests aren't monolithic. To capture the full spectrum, they utilize three distinct data streams:

• User Tweets (UT): The source for core/central interests.
• Expanded Tweets (ET): Collecting tweets from others using the same hashtags to find "collective" but potentially surprising interests.
• Web Pages (WP): Deep-diving into the URLs shared by the user for rich, formal semantic data.

System Architecture

The system uses a pipeline involving a C-value term extractor (to identify multi-word concepts) and OpenNLP for Named Entity Recognition (NER).

The C-value method is particularly important here as it moves beyond single-word nouns, capturing complex concepts that better represent a user's specific domain of interest.

Experiments & Results: Where Surprise Resides

The researchers tested the system on three types of users: a research-focused account (A) and two personal accounts (B and C). Users ranked results on two 10-point scales: Interesting and Surprising.

Key Findings:

• UT (User's own tweets) provided the most interesting terms (67.5% for User A) but the fewest surprising ones.
• ET (Expanded collection) and WP (Web pages) were the engines of serendipity, providing significantly higher "surprisingness" scores (up to 47.5%).
• For User A, 27.5% of all extracted items were deemed truly "serendipitous" (scoring high on both metrics).

Performance Comparison Table (User A)

The following table demonstrates how different sources contribute to "Interesting" vs. "Surprising" metrics:

Critical Analysis & Conclusion

Takeaway

The research confirms that automating serendipity is feasible but requires looking "outside the box" of a user's immediate timeline. The expansion to hashtags and linked content is critical for moving beyond the "echo chamber."

Limitations

• Subjectivity: Serendipity is in the eye of the beholder. What one user finds "surprising," another might find "irrelevant."
• NLP Sensitivity: Standard NER tools still struggle with the extreme noise of social media.
• Scalability: Real-time crawling and analysis of external web pages (WP) is computationally expensive.

Future Work

The next frontier involves integrating these semantic "peripheral interests" into graph-based recommendation systems to facilitate real-time, serendipitous social discovery.