NowAndThen: Moving Beyond Dates to Visual Context in Social Reminiscence
NowAndThen: a social network-based photo recommendation tool supporting reminiscence
NowAndThen is a multimodal photo recommendation tool designed to support spontaneous reminiscence on Social Network Sites (SNS). By utilizing Bag-of-Visual-Words (BoVW) and RootSIFT features, it retrieves and recommends past photos from both users and their friends that share common visual subjects with current posts, achieving a Mean Average Precision of 82.4%.
TL;DR
"On This Day" notifications often feel like a roll of the dice—sometimes sweet, sometimes painful. NowAndThen shifts the paradigm of social media reminiscence from time-based triggers to context-based triggers. By analyzing the visual features of photos you are currently sharing, the system surfaces relevant memories from your and your friends' pasts, turning a simple upload into a meaningful journey through time.
Background: The Nostalgia Gap
In the age of Instagram and Facebook, we are drowning in digital memories, yet we rarely revisit them unless prompted. Current systems often fail because they lack contextual intelligence. A photo from three years ago might be a high-school graduation (positive) or a breakup (negative). NowAndThen operates on a core research intuition: if you are sharing a photo of a mountain today, you are likely in a state of mind to enjoy a similar mountain vista from five years ago.
Methodology: Under the Hood of Visual Retrieval
The researchers didn't just rely on tags (which only 17% of users use consistently); they built a robust visual engine.
1. The Retrieval Pipeline
The system treats images like documents. It extracts "visual words" using the Hessian-affine detector and SIFT descriptors, quantizing them into a massive vocabulary of 1 million visual words using Approximate K-Means (AKM).
Figure: The Offline Indexing Stage - converting millions of features into an efficient inverted index.
2. Spatial Verification & Metadata
To ensure that a photo of a building isn't confused with a similarly colored cat, the system employs RANSAC spatial verification. It checks if the geometric arrangement of visual features matches. Finally, it layers in social metadata—tags and friend relationships—to rank the most "socially relevant" memories at the top.
Human-Centric Design Insights
The team developed a mobile prototype that offers three modes:
- F1 (Across-time): Matches your current post to your own past.
- F2 (Between-friends): Matches your post to a friend's old photo.
- F3 (Photo viewing): Suggests related memories while you browse the feed.
Figure: The "Then and Now" side-by-side view (Top) and the recommendation carousel (Bottom).
Experimental Findings:
- Emotional Resonance: Participants reported feeling "touched" and "surprised" by the accuracy of the memories.
- Privacy & Ownership: Users were happy to share their own old photos side-by-side with new ones but preferred a "separate" view for friends' photos to respect ownership and maintain focus on their own current activity.
- The "Uncanny" Accuracy: In some cases, the system matched different people in the same location due to similar compositions, highlighting the system's reliance on visual structure (as seen below).
Figure: A "failure" case where visual similarity overrode semantic reality—two different people in different places with the same composition.
Critical Analysis & Future Outlook
While the RootSIFT and BoVW approach was SOTA for 2016, modern Vision Transformers (ViT) and CLIP embeddings would likely solve the "semantic mismatch" seen in Figure 7.
The Takeaway: NowAndThen proves that reminiscence tools should be context-aware. By aligning triggers with a user's current interests, we can maximize the psychological benefits of nostalgia (increased self-esteem and social connection) while minimizing the risk of "memory landmines."
Future systems should look toward sentiment analysis of comments to ensure that the memories surfaced are not just visually similar, but emotionally positive.