We present real-time all-purpose segmentation to segment and recognize objects for image, video, and interactive inputs. In addition to benchmarking, we also propose a simple yet effective baseline, named RAP-SAM, which achieves the best accuracy and speed trade-off among three different tasks. Both real time panoptic segmentation and video instance segmentation are shown at the right. Advanced by transformer architecture, vision foundation models VFMs achieve remarkable progress in performance and generalization ability.
However, most VFMs cannot run in real-time, which makes it difficult to transfer them into several products. Thus, this work explores a new real-time segmentation setting, named all-purpose segmentation in real-time, to transfer VFMs in real-time deployment. It contains three different tasks, including interactive segmentation, panoptic segmentation, and video segmentation.
We aim to use one model to achieve the above tasks in real time. We first benchmark several strong baselines. It contains an efficient encoder and an efficient decoupled decoder to perform prompt-driven decoding. Moreover, we further explore different training strategies and tuning methods to boost co-training performance further. Our method contains three visual inputs: image, video, and visual prompts. Utilizing positional encoding, we generate prompt queries from these visual prompts.
The learnable object queries, alongside the prompt queries and the feature map F , are directed to the multi-stage decoder. This process generates multi-stage predictions and refined queries.
These refined queries engage in cross-attention with F , resulting in the final prediction. The interactive segmentation results with a single-point prompt shown in green color. The visualization result on YouTube-ViS Comparison of Segmentation Methods.
