Why neighboring rows break trellised-crop segmentation
In trellised systems such as vineyards and modern orchards, a single side-view image can contain several vegetation rows that look almost identical. A segmentation model asked to find leaves or fruits therefore tends to detect organs from neighboring rows together with the target row, which is a problem when the goal is to characterize only the foreground crop row. Phenotyping images are also acquired at high resolution to preserve small structures such as leaves, fruits, and disease symptoms, and direct full-image inference becomes difficult for many vision models at that scale.
FrontVeg V2 addresses both issues at once. It does not replace the underlying foundation models; it adds a processing framework that makes their zero-shot predictions more suitable for structured agricultural scenes. The current implementation uses Depth Anything V2 for monocular depth estimation and SAM3 for zero-shot instance segmentation, with model paths supplied externally rather than fixed in the source code.
Valley-Aware Depth Thresholding sets the foreground mask
The first branch of the pipeline extracts a binary foreground mask from depth. The depth model estimates a relative depth map from the RGB image, and the software converts that map into a one-dimensional histogram over 0 to 255, smoothed with a Gaussian filter to suppress local fluctuations. Prominent local maxima form a set of modes, and because the first mode generally corresponds to the distant background, the module discards it.
For each consecutive pair of modes, the algorithm defines the valley as the minimum of the smoothed histogram between them. To avoid selecting thresholds from distant regions, the search is limited to the first four candidate valleys, and the optimal threshold is the valley with the lowest smoothed value. Thresholding the depth map with that value directly yields the binary foreground mask. This is the step that encodes scene geometry as a prior, rather than relying on appearance alone.
Tiling and graph assembly recover small, fragmented targets
In parallel, the second branch processes the original RGB image. To preserve small plant structures, the image is divided into overlapping square tiles, by default 640×640 with an overlap factor of 0.5, giving a stride of tile size times one minus overlap. Each tile is processed independently by the vision foundation model using the user's textual prompt, and fragments smaller than 50 pixels or covering more than 55% of a tile are discarded.
Overlapping tiles generate redundant and fragmented predictions, so a Graph-Based Mask Assembly module represents each valid fragment as a node in a graph. Edges are established in two stages: candidate pairs are first identified by bounding-box IoU above 0.05, then validated by mask IoU above 0.10. Connected components form reconstructed object instances, and overlapping pixels belonging to different instances are resolved by a winner-takes-all procedure based on confidence scores. Finally, the foreground mask is aligned to the segmentation resolution and combined by a pixel-wise logical AND, which removes predictions outside the foreground vegetation region without altering the shape of detected objects.
FrontVeg V2 leads zero-shot baselines on every evaluated trait
The performance impact was evaluated on a vineyard benchmark against existing zero-shot pipelines. FrontVeg V2 outperformed the compared tools across all evaluated traits, with the largest gaps on complex targets. For fruits it reached Dice 0.92 versus 0.58 for INSID3, and for disease symptoms the competing methods failed with Dice below 0.20 while FrontVeg V2 reached 0.61 for black rot and 0.73 for downy mildew. Leaves were already well handled by baselines, and FrontVeg V2 still improved to Dice 0.95.
These are mean values with standard deviations reported on the vineyard benchmark, so the comparison is specific to that benchmark and those traits rather than a general claim about all crops or all imaging conditions.
Tiling and foreground filtering each remove a different failure
An ablation study isolated the contribution of each component. Using SAM3 alone yielded poor results on small or complex targets, including Dice 0.00 on black rot. Adding tiling and non-maximum suppression raised leaves to Dice 0.90, fruits to 0.90, black rot to 0.50, and downy mildew to 0.72, showing that tiled inference and mask assembly significantly improve instance retrieval.
Applying the geometry-aware foreground filter alone left results essentially unchanged from SAM3 alone, which indicates that the filter is not a detector: it removes background noise rather than finding objects. Combining tiling, non-maximum suppression, and the foreground filter produced the highest overall accuracy, and the full FrontVeg V2 configuration reached Dice 0.95 for leaves, 0.92 for fruits, 0.61 for black rot, and 0.73 for downy mildew. The two components address different failure modes, and the gain comes from their integration.
The practical gain is measured in annotation hours
Beyond raw segmentation accuracy, the paper frames the contribution as resolving the manual annotation bottleneck in digital phenotyping. Creating the 158-image evaluation dataset required approximately 200 hours of manual expert annotation, producing around 10,000 manually delineated object instances. By offering zero-shot inference without fine-tuning, FrontVeg V2 is estimated to save 1,300 hours of manual labor per 1,000 images.
That estimate is a projection from the annotation effort of this dataset rather than a measured outcome of a deployment, so it should be read as an order-of-magnitude argument for the workflow rather than a precise accounting.
Where the framework applies and where inference still stalls
The evaluation dataset comprises 158 images from six geographic locations in France, Spain, Italy, Belgium, and Switzerland, covering grapevine, apricot, and apple, with leaf and fruit segmentation for all three crops and black rot and downy mildew segmentation for grapevine. Images were acquired with different camera systems, illumination conditions, and viewpoints, so the benchmark spans heterogeneous field conditions rather than a single controlled setup. The software currently targets leaves, fruits, and those two grapevine diseases across grapevine, apple, and apricot orchards.
Runtime is the clearest operational boundary. The pipeline runs on consumer-grade hardware, an Intel i7-11850H with 32 GB RAM and an RTX 3080 16 GB, without cloud dependency. Valley-Aware Depth Thresholding preprocessing takes about 1 second per image, while sliding-window SAM3 inference remains the main bottleneck at roughly 2 minutes per class. That asymmetry, not segmentation quality, is what currently limits real-time or edge deployment.
Sources used in this answer
FrontVeg V2: A Training-Free Software Framework for Foreground-Aware Zero-Shot Plant Trait Segmentation in High-Resolution Images of Trellised Crops
On a vineyard benchmark, FrontVeg V2 reached Dice 0.95 for leaves, 0.92 for fruits, 0.61 for black rot, and 0.73 for downy mildew, outperforming compared zero-shot pipelines across all evaluated traits without task-specific retraining.