Abstract: Recently, a class of tracking techniques called tracking by detection receive much attention in computer vision. These methods train a discriminative classsifier to separate the object from the background. The classifier bootstraps itself by using the current tracker state to extract positive and negative examples from the current frame. Slight inaccuracies in the tracker lead to incorrectly labeled training examples, which degrade the classifier and cause drift. In this paper, an effective algorithm is proposed to overcome the target drift. It takes the framework of tracking by detection. Median Flow (MF) is used as a tracker to improve the reliability of the tracking point; the detector is constituted with several weak classifiers of random ferns to cascade, and it is updated with online Multiple Instance Learning (MIL). Finally the detector and tracking results are integrated to get the target location. Experiments on a number of challenging video clips show that the proposed method outperforms some state-of-the-art tracking methods, especially for fast motion and drifts.