[IEEE ICME] Two-Branch Collaborative GNNs: Unmasking Hidden Marketing Intentions via Multimodal Graph Fusion

TL;DR

In the era of "native advertising," marketing content is often cleverly camouflaged within normal social media posts. This paper from the University of Technology Sydney proposes a Two-Branch Collaborative GNN that treats images and text as graphs. By calculating a "graph center" for each modality and injecting it into the other branch after every convolution layer, the model "unmasks" marketing intentions that might be invisible when looking at images or text in isolation.

Problem & Motivation: The Camouflage of Digital Marketing

Modern marketing is no longer just a loud "Buy Now" banner. It hides in movie reviews, folk science articles, and lifestyle blogs. Detecting these automatically is difficult for two reasons:

1. Heterogeneity: A single post might have three sentences but five images. Traditional CNNs or RNNs struggle with this variable relational data.
2. Weak Correlation at Surface Level: An image might look like a harmless film still (see Figure 3), but the text contains promotional hooks. If the model doesn't "fuse" these signals deeply, it misses the intent.

Existing SOTA methods often use two-branch networks but keep them "siloed"—only combining them at the very end (Late Fusion). This paper argues that collaboration must happen early and often.

Methodology: Collaborative Graph Representation

The core innovation lies in how the authors handle the interaction between the Image Branch and the Text Branch.

1. Graph Construction

Instead of a fixed-length vector, the authors represent news as:

• ImageGraph ($I_G$): Nodes are images (ResNet-18 features); edges represent cosine similarity.
• TextGraph ($T_G$): Nodes are sentences (Word2Vec features); edges represent semantic similarity.

2. The Collaborative Fusion Layer

Unlike typical GNNs, this model introduces a cross-pollination step between GCN layers.

• Step 1: Calculate the Graph Center of the Image Branch ($h_l^{(I)}$) by averaging all node embeddings.
• Step 2: Update the Text Branch nodes by adding this center: $H^{(S{)}^{\prime }}=H^{(S)}+\mu {\overline{H}}^{(I)}$.
• Step 3: Repeat the process for the Image branch using the Text center.

This ensures that by the time the data reaches the next convolution layer, the text nodes have been "informed" by the overall visual context, and vice versa.

Fig 1. The Collaborative Fusion Strategy: Crosswise accumulation of graph centers helps synchronize multimodal semantics.

Experiments & Results

The authors tested their approach on the SOHU dataset (50,000 news samples). Their "Two-Branch Collaborative" approach consistently beat all baselines.

Key Comparisons:

• Single Modality: Text-only (0.689) performed much better than Image-only (0.645), confirming that text remains the primary carrier of intent.
• Simple Fusion: A standard two-branch GNN reached 0.707.
• Our Model: By adding the iterative fusion layer, the accuracy jumped to 0.740.

Fig 2. The full framework includes GCN layers, fusion layers, and Self-Attention Graph Pooling (SAGPool) for refined representation.

The results (Table 2 in the paper) show that the LogLoss also dropped significantly (from 0.558 to 0.534), indicating the model became more confident in its predictions.

Critical Analysis & Conclusion

Why it Works

The "Graph Center" acts as a global summary. When you add the image summary to every sentence node, the GNN effectively performs a global-to-local alignment. If the images show a specific product and the text discusses "life tips," the fusion layer helps the model identify the underlying promotional link between those disparate "nodes."

Limitations & Future Work

• Edge Construction: The current similarity-based edge construction (top 50%) is somewhat heuristic. Learning the graph structure dynamically (e.g., via Graph Attention) could be a natural next step.
• Scalability: While GNNs handle variable nodes well, the computational cost of building dense similarity edges for very large documents might be high.

Final Takeaway

This work proves that for complex multimodal tasks like intention detection, how you fuse is just as important as what you fuse. Moving beyond simple concatenation to iterative graph-based collaboration is a powerful path forward for social media analysis.