Abstract: This paper proposes a novel model based on the hierarchical representation using heterogeneous descriptors for multi-class generic object detection in real-world scenario. Following the idea of part-based object detection, the model extracts complementary features of object class at different levels and represents them with a unified Conditional Random Field (CRF) framework, in which the individual part and its local features correspond to a unary node and the interactions (edges) between pairwise nodes reflect the underlying geometrical structure of the object class. To improve the discrimination and flexibility of the proposed model, Support Vector Machine (SVM) classifier and the learning of edge structure are combined into CRF according to the geometrical topology of object class. Experimental results on UIUC multi-scale dataset and PASCAL VOC 2007 dataset show that the proposed model can not only effectively represent a variety of complex object classes, also successfully detect objects with pose, scale, illumination variations as well as partial occlusions.