Abstract: A new SAR image segmentation model with edge penalty is constructed, which uses oriented edge strength information and a minimized hierarchical region merging algorithm is proposed in this paper. The edge strength information is extracted by using Multi-Direction Ratio Edge Detector (MDRED), based on which a high quality initial over-segmentation is obtained using watershed transformation. In order to extract the directions of boundaries of regions, polygons are used to approximate them, and a penalty term whose power is in inverse proportion to edge strength is obtained by incorporating Oriented Edge Strength Map (OESM) into the term. A hierarchical region merging algorithm driven by image features is obtained through graduated increased edge penalty. In order to accelerate the region merging, the Region Adjacency Graph (RAG) is used to represent the image segmentation. The experimental results show that the proposed method has advantages in performance and efficiency, and obtains better segmentation results with respect to other methods.