Abstract: Mean shift, an iterative procedure that shifts each data point to the average of data points in its neighborhood, has been applied to object tracking. However, with the changing structure of object in video sequences, traditional mean shift tracker by isotropic kernel often loses the object, especially when object structure varies fast. This paper implements object tracking with anisotropic kernel mean shift in which the shape, scale, and orientation of the kernels adapt to the changing object structure. The algorithm ensures tracking robust and real-time. Experimental results show it is effective.