Abstract: According to the problem of camera jittering under natural environments when detecting moving objects, a background adaptive scheme is proposed in the paper. First, the Harris operator is used to detect corners in the region-of-interest for background and foreground image respectively. A correlation and relaxation method is also applied to a small region to obtain several stable matched points. Then, the camera jitter parameter is estimated with offsets of these matched points and used to recover background image to match against the current image. At last, background difference algorithm based on the multi-resolution pyramid is adopted to detect moving object. It can remove the environment dynamic background noises and some small offset estimation errors caused by image blurring. The proposed algorithm is verified with camera jittering sequence of the public test image and compared with several state-of-the-art algorithms qualitatively and quantitatively. Experimental results demonstrate that the proposed algorithm can solve the problem of camera jittering in natural environment effectively. The detected effect evaluation parameter is better than the current algorithms.