User: The first chair violinist of an orchestra is a vital musical leader with widely ranging responsibilities, from tuning the orchestra to working closely with the conductor. Who is most likely the first violinist in the image? Please output the segmentation mask.

User: Please identify the activity people are doing in the image and segment those individuals who have already demonstrated excellent body strength and stability through some kind of professional actions. Lastly, output the mask that includes these individuals.

User: I am looking for a gym equipment to do weight training on my arms muscles. Which equipment would most likely draw my attention when I walk into a gym? Please output the segmentation mask.

User: Which objects are composed of a short bar with a weight on each end, typically used for weight training? Please find the unracked ones and output the segmentation mask.

User: Please segment the person who is shooting in this NBA All-Star game.

User: Please segment the food with the highest amount of vitamins.

User: A person would like to recharge on some vitamins with fruit, however after a long day of school, they don't want to eat anything that would take too long to prepare, if we were to rank from most easy to hardest, what would be his last choice?

Previous Next

Abstract

Reasoning segmentation is an emerging vision-language task that requires generating a segmentation mask from implicit and often ambiguous language queries, enabled by recent advances in Multimodal Large Language Models (MLLMs). However, state-of-the-art training-based approaches often fail in challenging cases that demand higher-level reasoning or external knowledge. In this work, we introduce CoT-Seg, a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction. Instead of fine-tuning, CoT-Seg leverages the inherent reasoning ability of pre-trained MLLMs (e.g. GPT-4o) to decompose queries into meta-instructions, extract fine-grained semantics from images, and identify target objects even under implicit or complex prompts. Crucially, CoT-Seg incorporates a self-correction stage: the model evaluates its own segmentation against the original query and reasoning trace, identifies mismatches, and iteratively refines the mask. This tight integration of reasoning and correction significantly improves reliability and robustness, especially in ambiguous or error-prone cases. Furthermore, we extend CoT-Seg with retrieval-augmented reasoning, enabling the system to access external knowledge when the input lacks sufficient information, further enhancing segmentation accuracy. Extensive experiments on ReasonSeg and RefCOCO demonstrate that CoT-Seg consistently outperforms existing baselines while remaining training-free. Our results highlight that combining chain-of-thought reasoning, self-correction, and retrieval augmentation offers a powerful paradigm for advancing reasoning-driven segmentation.