Abstract: A novel method is proposed for segmentation of moving object using motion feature of edge in infrared videos. At first, a new index Motion Saliency of Edge (MSE) is defined. MSE can reflect the motion feature of edge points of an image based on the spatial-temporal characteristic. The higher the MSE of an edge point is, the more likely it belongs to a moving object. The edge point with a high MSE value is extracted by using Otsus thresholding. The results obtained by Otsu are updated by using historical data. The edge points which are extracted and updated can be considered to be the seed of moving objects. At last, the segmented masks of moving objects are grown from the seeds by using a region growing method of layer-by-layer. The proposed method is successfully tested over three infrared image sequences and compared with the other two methods. The experiment results demonstrate that the proposed method has better performance of moving object segmentation with less effect of object-background misclassification in infrared videos.