The Query Network: Scaling Datalog to Millions of User-Defined Rules in Social Networks

TL;DR

In a future where every social network user is an autonomous agent with their own connection logic, we face a "Large Query" problem where the query set is as massive as the data itself. This paper introduces the Query Network model and two core algorithms—Backward-Radius Triggering (BRT) and Divide and Conquer (DAC)—to efficiently evaluate recursive Datalog queries across millions of nodes without hitting the traditional optimization wall.

Background: When the Query is as Big as the Data

Traditionally, database research assumes a clear hierarchy: the schema and queries are tiny, while the data (EDB) is massive. However, Web 2.0 and the rise of personal agents invert this. Imagine if 100 million Facebook users each had a specific rule: "If two of my friends know Person X and Person X lives in Haifa, add Person X as a friend."

Mathematically, this represents a Datalog program with millions of rules. Standard optimizers, like those using Magic Sets, would struggle because they were never designed for rule sets that scale linearly with the number of participants.

Methodology: Exploiting Locality

The authors identify two ways to bypass the exhaustive "round-robin" evaluation of rules:

1. Backward-Radius Triggering (BRT)

Instead of checking every user's rule in every iteration, BRT uses the "Radius" of a query—the maximum path length the query traverses to make a decision. When a new edge (u, v) is added, only nodes within a specific "Backward Radius" of u could potentially trigger new results.

Figure: A query graph defining the relationship between node n and potential new friends.

2. Divide and Conquer (DAC)

Social networks are naturally clustered. DAC partitions the network into sub-graphs, evaluates them independently, and merges them. The crucial "Matchmaking" step pairs clusters with the most crossing edges to minimize the ripple effect of new connections during the merge phase.

Figure: The DAC process: Partitioning, local evaluation, and hierarchical merging.

Experimental Insights

The researchers tested their system against both synthetic clusters and real-world collaboration data from DBLP.

Key Findings:

• The IDB/EDB Ratio: The effectiveness of DAC is highly correlated with how many new edges (IDB) are created compared to existing ones (EDB). When the "friendship explosion" is high, DAC is significantly faster than BRT.
• Cycle Preservation: The paper proves a "Preservation of Cycles" property, ensuring that the recursive nature of Datalog doesn't create infinite loops if the original input was a Directed Acyclic Graph (DAG).

Figure: Performance results showing DAC and BRT dwarfing the Basic evaluation time as the network complexity grows.

Critical Analysis & Future Outlook

While the paper provides a robust foundation for large-scale recursive querying, it leaves a few doors open:

• Dynamic Deletion: The current model focuses on adding edges. Handling the removal of connections requires a more complex "truth maintenance" logic to retract derived edges.
• Heterogeneous Data: The authors suggest extending the model to include XPath predicates, allowing rules to trigger based on the actual content of a user's profile (XML) rather than just the graph structure.

Final Takeaway

The "Query Network" shifts Datalog from a centralized data-processing language to a distributed coordination protocol. For engineers building the next generation of social platforms or decentralized autonomous organizations (DAOs), the BRT and DAC algorithms provide a blueprint for letting millions of users run custom logic at scale.