[System Architecture] Oases: Scaling Online Spam Detection via DHT-Based Progressive Aggregation

TL;DR

Social media spam is not just annoying; it's high-velocity and structurally adaptive. Oases is a decentralized system that solves the "spam drift" problem by moving detection from central servers to a scalable, DHT-based peer-to-peer overlay. By using progressive aggregation trees, it achieves near-SOTA classification performance (96.2% F1) while maintaining the scalability to handle millions of posts across hundreds of distributed agents.

The "Content Drifting" Bottleneck

Why can't we just use a massive centralized Random Forest classifier? The authors identify two critical flaws in the current ecosystem:

1. Temporal Decay: Spam content "drifts" rapidly. A classifier trained on last week's data is often useless against today's "Charlie Hebdo" shooting rumors or trending malware links.
2. Centralized Chokepoints: OSN logs are generated across globally distributed web servers. Shipping all this raw data to a central cluster for "offline" analysis creates massive latency and a single point of failure.

As shown in the figure below, social rumors peak and evolve within extremely short windows, necessitating an online approach that can update models on the fly.

Methodology: The Power of Two Trees

Oases doesn't reinvent machine learning algorithms; instead, it reinvents the infrastructure that hosts them. Built on top of the Pastry DHT and Scribe group management protocol, Oases orchestrates agents into two functional hierarchies.

1. The Model Construction Tree

To solve the "drift" problem, the Oases root must push updated training sets to all agents. Instead of $N$ point-to-point connections, it uses a multicast tree where the dataset is copied and forwarded down the branches in $O(\log N)$ hops.

2. The Spam Processing Tree (Aggregation Flow)

Once leaf agents classify local streams, they don't just send "Spam/Ham" flags. They send (ID, label) pairs. Intermediate nodes "roll up" these results. If multiple agents across the network flag the same hashed content ID, the confidence score increases, allowing for a robust, consensus-based detection mechanism across distributed servers.

Experimental Validation

The system was stress-tested on a cluster of 800 agents using 3 million real-world tweets.

Classification Prowess

While Oases is algorithm-agnostic, the implementation of Random Forest far outperformed SVM, KNN, and Naive Bayes in this distributed context, striking the best balance between False Positive Rate (0.26) and F1-score.

Scalability & Elasticity

A key innovation is the Self-adjustable Tree. Depending on the application's needs, Oases can tune the "tree bit" (fan-out).

• High Latency Sensitivity? Increase fan-out to decrease tree depth.
• High Fault Tolerance? Decrease fan-out to create a deeper tree where individual node failures have localized impacts.

Figure: The linear growth in processing time relative to depth proves the system avoids the exponential bottlenecks typical of centralized aggregators.

Critical Insight: Why it Works

The brilliance of Oases lies in its Inductive Bias toward decentralization. By utilizing the "leaf set" mechanism in Pastry, Oases nodes can detect parent failures via heartbeats and autonomously reform the tree. This "self-healing" nature is essential for production social networks where nodes (web servers) frequently recycle or go offline.

Conclusion & Future Work

Oases demonstrates that we don't need to sacrifice detection accuracy for scalability. By combining DHT-based routing with progressive aggregation, the system handles the "velocity" and "variety" of modern spam. Future iterations aim to include more complex specification APIs and automated checkpointing for even higher availability in massive-scale deployments.