Abstract: The maximum number of targets that can be uniquely identified by the traditional MIMO radar is limited by the number of virtual sensors. To alleviate this issue, a novel antenna array in MIMO radar which is based on the concept of nested arrays is designed in this paper, and a modified spatial sparsity-based Direction-Of-Arrival (DOA) estimation method is proposed. First, the effect of nested sampling with application to virtual array of traditional MIMO radar on the DOA estimation performance is analyzed. Second, the method to design antenna array of nested MIMO radar is proposed. It is proven that nested MIMO radar can detect more targets than traditional MIMO radar when they share the same number of virtual sensors. Finally, a modified spatial sparsity-based approach to DOA estimation in nested MIMO radar is proposed based on noise subspace weighted minimization problem, which can increase resolution and effectively suppress spurious peaks. Extensive simulation results demonstrate the effectiveness and superiority of the proposed methods.