Abstract: In order to solve the problems of color similar target, scale change and occlusion in target tracking, this paper proposes an adaptive fragments-based target tracking method fusing color histogram and SIFT features. Taking color projection and imaging angle as the standard, the adaptive fragment method can ensure not only the distinctive features of fragments but also the number. Each fragment is represented by color histogram and SIFT features. Adaptive scale selection can be obtained by computing the scale change of SIFT features. The weighting factor and corresponding template of each fragment are updated during tracking and the whole target template will be divided again if the target appearance changes significantly. Experimental results demonstrate that the method can track the target accurately and effectively and has advantage in color similar target tracking, adaptive scale selection and occlusion handling.