Abstract: As to the difficulty of confidence measure estimation regarding to Automatic Speech Recognition (ASR), a strategy resorting to multi-source knowledge combination to improve the confidence measure is proposed in this paper. More specially, the knowledge come from acoustic level, linguistic level and semantic level are firstly selected and then combined by different ways by held-out validation. And then, these multi-source knowledge are integrated under the framework of Hidden-units Conditional Random Fields (HuCRFs). Lastly, the conditional probability computed from HuCRFs is used to be a new estimation procedure of confidence measure for recognition candidate. Experiments show that the discriminative ability of conditional probability of HuCRFs is superior to the conventional posterior computed from lattice. Furthermore, a lattice rescoring is carried out by utilizing the conditional probabilities of HuCRFs to search the best hypotheses and resulted in a significant reduction on Character Error Rate (CER) by about 2% absolutely on a benchmark corpus. Simultaneously, a performance comparison between the performances of long-distance language model based lattice rescoring and conditional probability of HuCRFs based lattice rescoring is also performed and it is further proved that HuCRFs is a better alternative to the estimation of confidence measure in ASR.