Beyond Text: Leveraging Factorized Multi-modal Attention for Cultural Heritage Discovery
Discovering Latent Information from Noisy Sources in the Cultural Heritage Domain
The paper introduces a multi-modal framework for Named Entity Recognition (NER) and latent information discovery in the Cultural Heritage domain. It proposes a novel "Factorizing Attention Module" (FAM) that integrates BERT-based textual embeddings with VGG16 visual features to extract entities from noisy, short-form social media data.
TL;DR
Social media is a goldmine for cultural heritage data, but it's buried under "noise"—typos, lack of context, and cryptic brevity. This research proposes moving beyond text-only analysis by using a Factorizing Attention Module (FAM). By treating images and text as interacting entities through second-order factorizing, the model recovers latent information that traditional NER systems miss.
Background: The Boon and Curse of Social Media
In the Cultural Heritage domain, we are caught between two worlds: structured museum catalogs and the chaotic, rich flow of social media. While Twitter can reveal latent relationships (e.g., an artist being influenced by a specific movement in real-time discussions), the data is:
- Sparse: Tweets are extremely short.
- Noisy: Grammatical errors and bot-generated structures are the norm.
- Ambiguous: Without context, identifying if "Starry Night" refers to the painting, a song, or a local event is difficult.
The author's insight is simple: Approximately 30% of these tweets contain images. These visuals are the "missing context" that can ground the ambiguous text.
Methodology: The Factorizing Attention Module (FAM)
Most multi-modal models simply concatenate text and image vectors or use standard cross-attention. However, this paper argues that for sparse data, we need to explicitly model the interactions between features.
The Architecture
The proposed framework compares three approaches:
- Model A (Text Only): Uses BERT embeddings followed by a linear classification layer.
- Model B (Standard Attention): Combines BERT and VGG16 features using traditional attention mechanisms.
- Model C (Proposed FAM): Uses a Factorizing Attention Module. Inspired by Factorization Machines, FAM accounts for second-order interactions within and across modalities. It looks for "feature weights" in the combined sparse space of pixels and sub-words.
Figure: Comparison of Text-only (A), Standard Attention (B), and the proposed Factorizing Attention (C).
Experiments: Real vs. Ideal Context
The author tested the models on a dataset labeled via Mechanical Turk using the BIO tagging scheme. The entities included Artists (A), Movements (M), Venues (V), and Artworks (W).
Key findings from the experimental results:
- The Content Gap: Using real-world images (from VGG16) initially showed lower F1 scores than text-only models due to the "noise" in the images themselves (e.g., snapshots of posters instead of art).
- The Potential of FAM: When "Ideal Features" (synthetic visual markers) were used, the FAM model outperformed the text-only baseline across every category.
- Scaling Efficiency: As the training data size increased from 1,000 to 1,500 samples, the performance gap between text+images and text-only narrowed significantly, suggesting that the multi-modal approach scales better with data volume.
Table: F1 scores showing the performance boost when multi-modal interactions are effectively captured.
Critical Insight: Why Factorization?
Standard attention often struggles when one modality (text) is much "stronger" or more structured than the other (unlabeled images). By using a Factorizing approach, the model doesn't just attend to the image; it calculates how specific visual features alter the meaning of specific words. This second-order logic is crucial for "latent" discovery—finding links that aren't explicitly stated but are implied by the co-occurrence of a visual style and a textual mention.
Conclusion
This PhD work highlights that the future of information extraction in specialized domains like Cultural Heritage lies in cross-modal grounding. While the study currently relies on older backbones (BERT and VGG16), the introduction of the Factorizing Attention Module provides a unique architectural path for handling noisy, sparse data.
Future Outlook: The next step involves scaling the corpus and potentially moving towards "n-shot learning" to handle the long-tail of obscure artists and movements that appear in social media but lack massive training sets.