Abstract: Fractional calculus is widely used in control system theory. Owing to introducing of fractional dynamic system model, searching solution method of fractional state estimation is an urgent issue. Starting from fractional nonlinear dynamic system model, fractional unscented Kalman filter is derived based on probability theory. The filter is applied to two typical nonlinear systems, Univariate Nonstationary Growth Model (UNGM) model and Reentry Vehicle Tracking (RVT) model. Experiment results prove the performance of fractional unscented Kalman filter given in this paper is better than that of the unscented Kalman filter in the context of reasonable setting of the fractional order.