[IEEE T-MM] RSBM: Decoding the Core of Family Kinship through Tri-Subject Verification

TL;DR

While most AI systems try to determine if two people are related (Bi-Subject), this work shifts the focus to the nuclear family unit: verifying a child against both parents simultaneously (Tri-Subject). By introducing the Relative Symmetric Bilinear Model (RSBM) and the TSKinFace dataset, the researchers achieved over 85% accuracy, proving that "knowing both parents" is the key to solving the hardest kinship puzzles.

Background: Beyond One-on-One

Kinship verification is notoriously difficult due to the "appearance gap"—a daughter doesn't look exactly like her father due to age and gender differences. However, biology gives us a hint: a child inherits 50% of DNA from each parent. Most prior SOTA methods ignored this "tri-relationship," treating a family as a collection of independent pairs. This paper argues that the child-parents relationship is the "core of a family" and must be modeled as a single, dependent structure.

The Problem: The Partial Resemblance Paradox

Current methods fail because they assume a child looks "equally" like both parents. In reality, a child often favors one parent’s features (e.g., "having the father's eyes").

• Prior Work Limitation: Rigid metric learning (like ITML or LMNN) forces a distance that doesn't account for this asymmetric resemblance.
• The Gap: How do we model the conditional similarity where one parent "referees" the resemblance of the other?

Methodology: The RSBM Framework

The authors propose a three-stage pipeline:

1. Feature Extraction: Overlapping patches are encoded using SIFT.
2. Spatially Voted Feature Selection (FS): Instead of the usual Group Lasso (where patches are forced together), the authors use "competition before grouping." Individual features compete, and patches with the most "winning" features are selected.
3. RSBM Core: The model uses a bilinear function $s(x,x_c)=x^{T}W{x}_c$. Crucially, it uses a softmax function to weight the similarity: if the system detects the child looks more like the father, it increases the weight of the father-child similarity score in the final verification.

Figure 1: The overall architecture showing patch extraction, feature selection, and the bilinear similarity modeling.

Experimental Results & SOTA Comparison

The authors curated the TSKinFace database, the largest of its kind with over 1,000 families. The results were clear:

• Accuracy: The RSBM-block-FS model reached 85.4% avg, compared to just 53.4% for a standard SVM.
• Gender Bias: Verification for sons (FM-S) was consistently more accurate than for daughters (FM-D), aligning with psychological theories that kin signals in daughters are often more subtle.
• Boosting 1-vs-1: Interestingly, if you are trying to verify a Mother-Daughter relationship, adding the Father's image as a "reference" (even if he isn't the primary subject) significantly improves accuracy.

Table III: Comparison against SOTA. Notice the significant jump from ABM to RSBM-block-FS.

Deep Insight: Visualizing "Relative Similarity"

One of the most compelling parts of the study is the visualization. By multiplying the learned priors back into the images, we can see the model "learning" which parent provides the primary visual evidence for kinship in a specific family.

Figure 6: Visualizing the "resemblance prior"—the model effectively highlights which parent's features are more dominant for a given child.

Conclusion & Perspective

This work proves that Tri-Subject verification is not just a sub-field, but a more robust way to handle the noise of age and gender in computer vision.

• Takeaway: Future AI for missing children or social media tagging shouldn't look for pairs—they should look for families.
• Future Work: The authors suggest extending this to "General Family Structures" (grandparents, siblings).
• Limitation: The model currently relies on SIFT/Hand-crafted features; a transition to end-to-end Deep Bilinear Pooling could potentially push these accuracies toward the 95%+ range required for law enforcement applications.