Automatic Segmentation of Chinese using overlaying ambiguity examining method and statistics language model

Wang Xianfang; Du Limin

Volume 25 Issue 9

Sep. 2003

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2003 > 25(9): 1168-1173

Wang Xianfang, Du Limin. Automatic Segmentation of Chinese using overlaying ambiguity examining method and statistics language model[J]. Journal of Electronics & Information Technology, 2003, 25(9): 1168-1173.

Citation:

Wang Xianfang, Du Limin. Automatic Segmentation of Chinese using overlaying ambiguity examining method and statistics language model[J]. Journal of Electronics & Information Technology, 2003, 25(9): 1168-1173.

Citation:

PDF( 538 KB)

Automatic Segmentation of Chinese using overlaying ambiguity examining method and statistics language model

Received Date: 2002-03-19
Rev Recd Date: 2002-08-09
Publish Date: 2003-09-19

Abstract

Abstract

In this paper, the question of Chinese automatic segmentation is discussed using overlaying ambiguity examining method and statistics language model. The multi-time iterative method is applied to train language model, which can produce a better model. The process of training language model is described in detail. The result shows that the perplexity of language model is reduced. The accuracy of segmentation changes with different language model and the reason is analyzed.

FullText(HTML)

References(1)

References

刘开瑛,中文文本自动分词和标注,上海,商务印书馆,2000,30-41[2]陈小菏,现代汉语自动分析,北京,北京语言文化大学出版社,1999,60-62.[3]马晏,基于评价的汉语自动分词系统的研究及实现,语言处理专论,北京,清华大学出版社,1996,80-105.[4]侯敏,孙建军,陈肇雄,汉语自动分词的歧义问题,计算语言学进展与应用,北京,清华大学出版社,1995,40-43.[5]沈达阳,孙茂松,基于统计的汉语分词模型及其实现方法,BYTE China,重庆,1998,2(2),38-40[6]孙茂松等,高频最大交集型歧义切分字段在汉语自动分词中的运用,中文信息学报,1999,13(1),60-62.[7]王雪松,汉语语言的多层面优化统计语言模型研究,[硕士论文],中科院声学所,1997,13-15.[8]张瑞强,用于汉语连续语音识别中的语言模型的研究,[博士论文],清华大学,1997,20-27.

Relative Articles

Supplements(0)

Cited By

Proportional views