PaveSAM: Revolutionizing Pavement Distress Segmentation via Zero-Shot Learning

TL;DR

Pavement maintenance is shifting from manual inspection to AI-driven automation. However, the bottleneck remains the "annotation tax"—the high cost of pixel-level labeling. PaveSAM breaks this barrier by fine-tuning the Segment Anything Model (SAM) with just 180 images. It allows engineers to generate sub-pixel accurate crack masks using simple bounding boxes, achieving a 34.9% performance boost over the original SAM and outperforming specialized UNet variants.

The "Annotation Tax" Problem in Civil Engineering

In pavement management, knowing where a crack is (Bounding Box) isn't enough; we need to know its area and width (Segmentation) to calculate the Pavement Condition Index (PCI).

• The Pain Point: Supervised learning requires pixel-level masks which take nearly 8x longer to create than bounding boxes.
• The Gap: While Meta's SAM changed the game for natural images, it struggles with the irregular, "spider-web" geometry of alligator cracks and the noisy texture of asphalt.

Methodology: Tuning the Giant

Instead of training a model from scratch, the authors leveraged the "knowledge" stored in SAM's massive ViT-based image encoder.

1. Architecture Choice

PaveSAM keeps the heavy Image Encoder (ViT-H) frozen to preserve general features. The team only fine-tuned the Mask Decoder. This strategy allows the model to map specific "pavement textures" to the hierarchical prompt embeddings provided by bounding box coordinates.

Figure 1: PaveSAM adapts SAM’s prompt-based architecture for infrastructure-specific distress types.

2. Prompt Engineering

The authors discovered that "Segment Everything" mode (automatic mask generation) creates irrelevant noise on roads. Bounding box prompts provide the necessary spatial prior, focusing the model's attention on the distress area (e.g., longitudinal or transverse cracks).

Experiments and Benchmarking

PaveSAM was tested against a "Who's Who" of segmentation models: UNet, ResUNet++, TransResUNet, and DeepLabv3.

Performance on Public Data (Crack500)

When evaluated on the Crack500 dataset, PaveSAM didn't just win—it dominated. It achieved an F1-score of 0.691, significantly higher than the base SAM's 0.148. This demonstrates the critical importance of domain-specific fine-tuning for "edge-case" geometries like cracks.

Figure 2: Training IoU scores showing PaveSAM (green line) maintaining superior convergence and stability.

Visual Evidence

The qualitative results show PaveSAM capturing the fine, branch-like details of cracks that UNet variants often "blur" or miss entirely.

Figure 3: Semantic masks generated by PaveSAM vs Ground Truth and other baselines. Note the precision in capturing thin crack branches.

Critical Insight: The Complexity Trade-off

While PaveSAM is a performance beast, it comes with a high computational cost:

• The Heavyweight: With 136M parameters and 487 GFLOPs, it is significantly slower (6.28 FPS) than specialized VGG-based UNets (183 FPS).
• The Bottleneck: The image encoder is the primary culprit for the slow inference speed.

Conclusion and Future Outlook

PaveSAM is a pioneering step in bringing Foundation Models to the pavement industry. Its ability to convert existing bounding-box datasets into high-fidelity segmentation masks is a massive win for data-starved researchers.

Future Directions:

• Tiny PaveSAM: Moving from ViT-H to more efficient encoders (like MobileSAM or FastSAM) to achieve real-time performance on inspection vehicles.
• Text-to-Mask: Using CLIP-based text prompts (e.g., "fine transverse crack") to eliminate the need for even bounding boxes.

PaveSAM proves that even the most "general" AI can become a domain expert with the right prompts and a small, high-quality dataset.