Abstract: For the issue of multi-object robust tracking, a type of watershed segmentation and Scale-Invariant Feature Transform (SIFT) feature points based full-automatic tracking algorithm is presented. To avoid flat area while do watershed segmentation on the image, a regular gradient image is added to the source image. After the Gaussian blurred process is done on the added image in float field, field minimal points are selected as object feature points as well as seed points to do watershed segmentation. Moving object is detected through short time points trajectories derived from watershed region mapping relationship between current and backward image. SIFT feature pool is built and updated based on object occlusion occurred or not and watershed segmentation. With the help of watershed region mapping and feature matching with the SIFT feature pool, object is robustly tracked. Actual tests show that the algorithm can track multi-object well and with a better performance of mutual occlusion robustness.