Abstract: Color-based moving object detection performs poorly when illumination changes or shadow exists. Depth-based moving object detection is affected by the high level of depth-data noise at object boundaries, and it fails when foreground objects move close to the background. For these reasons, a novel approach that establishes color and depth classifier for each pixel is presented by making full use of color information obtained by CCD camera and depth information obtained by TOF camera. In order to realize the effective detection, different weights are assigned adaptively for each output of the classifier by considering foreground detections in the previous frames and the depth feature. Multi video sequences are captured to verify the proposed method, and the experimental results show that the proposed approach can effectively solve the limitations of color-based or depth-based detection and realize the effective detection.