Abstract: When Web images are indexed, the textual information in the hosting web pages are usually used as approximate image description. However, such information is not accurate enough. In this paper, a framework is proposed to utilize the visual content, the textual context, and the semantic relations between keywords to refine the image annotation. Experiments on large-scale dataset demonstrate the effectiveness of the proposed method.