Abstract: A novel algorithm called Dual Latent Variable Spaces Local Particle Search (DLVSLPS) is proposed. It can estimate the 3D human motion sequence from silhouettes of multi-view image sequence more accurately. Gaussian Process Dynamical Models (GPDM) is used to reduce the dimension to build the dual latent variable spaces and the mapping from low dimensional latent variable data to high dimensional data. Then, the low dimensional particles are searched in these spaces by the method called Neighbor Weight Prior Condition Search (NWPCS). The better high dimensional data are generated from the mapping to estimate the 3D human motion of the corresponding frame. The proposed algorithm aims to solve the problem of traditional particle filters. The problem is that sampling in high dimensional data space can not get the valid and correct data to estimate the 3D human motion. The simulating experiments show the proposed algorithm has better performance than the traditional particle filters. The better performance includes the multi-view and discontinuous frame estimation, overcoming the silhouette ambiguity and reducing the estimation error.