From News to Facts: Architecture for Big Data Social Graph Analysis

TL;DR

The paper details a robust industrial-academic collaboration to build a Big Data pipeline that digests millions of news articles and spits out interactive social graphs. By combining Hadoop’s distributed power with Cogito’s deep semantic analysis, the system identifies not just who is in the news, but how they are connected through specific facts and linguistic co-occurrences.

Motivation: The "Story" Beyond the Document

Analysts in fields like security and business intelligence don't just need to find a document; they need to find a story. Traditional semantic search tells you that "Company A" appears in 5,000 articles. It doesn't tell you that "Company A" is increasingly linked to "Person B" in "Country C" via "Criminal Actions" or "Economic Interconnections" across those same articles.

The authors identified that the bottleneck wasn't just volume (Big Data), but the velocity of insight. They realized that to discover hidden patterns, they needed to move from keyword indexing to Relationship Extraction.

Methodology: Semantic Intelligence at Scale

The architecture is a masterclass in the "Hadoop Ecosystem" of the mid-2010s, utilizing a specialized pipeline to turn raw text into structured nodes and edges.

1. The Semantic Engine (Cogito API)

At the heart of the system is the Sensigrafo, a semantic network of over 1 million concepts. Unlike simple Named Entity Recognition (NER), Cogito performs:

• Disambiguation: Distinguishing if "Arthur Andersen" is a person or an organization based on context.
• Co-occurrence Extraction: Identifying triples like (Entity A, Correlation, Entity B).
• Fact Mining: Grouping entities into domain-specific events (e.g., "Construction and Property").

2. The Distributed Pipeline

To handle the scale, the system uses a multi-stage approach:

• Sqoop: Pulls news from RDBMS into HDFS.
• MapReduce: Distributes the CPU-intensive Cogito API calls across the cluster.
• Impala/Pig: Handles the interactive and batch querying of the resulting facts.

Fig 1: The architecture flow from raw data acquisition to graph visualization.

From Extraction to Social Graphs

The most innovative part of the work is the mapping of semantic outputs to a Social Graph.

• Nodes: Represent entities (People, Places, Orgs) or specific "Facts."
• Edges: Labeled with the type of correlation (Communicative, Criminal, Movement, etc.).
• Intuition: By aggregating these across millions of documents, a "Generic" relationship that appears 100 times signifies a much stronger tie than a single mention.

Fig 2: The detailed breakdown of the Unstructured Analysis System showing the integration of Giraph and Mahout.

Experimental Results & Insights

While the system was tested in a virtualized environment, the focus was on Recall. The authors argue that in intelligence gathering, it is better to suggest multiple potential connections to a human analyst than to filter them out prematurely.

Key results included:

• Scalability: By deploying the API within Map functions, they achieved linear scaling by adding nodes.
• Interactivity: Using Cloudera Impala allowed analysts to query these massive graphs with "sub-second" response times, a significant upgrade over traditional batch processing.

Fig 3: A generated social graph focused on the entity "Naftomar," illustrating its links to Athens and Egypt via specific activities.

Critical Analysis & Conclusion

The Takeaway: This paper provides a blueprint for "Turning News into Facts." It demonstrates that Big Data isn't just about storage; it's about the semantic enrichment that happens during the processing phase.

Limitations:

• The work is largely "preliminary" regarding graph optimization (using Giraph/Spark GraphX).
• The reliance on a proprietary API (Cogito) makes it difficult to replicate exactly without that specific software.

Future Work: The authors suggest moving toward Apache Spark for better graph handling and implementing Streaming operations to analyze news in real-time as it breaks. This work laid the groundwork for modern automated Knowledge Graph construction from global news feeds.