Abstract: H.264 takes objective metric as distortion criteria to perform Rate Control (RC) and Rate Distortion Optimization (RDO)-based mode decision, which can not acquire optimal subjective quality. Base on our previous research results, this paper applies Structural SIMilarity (SSIM) based subjective distortion to RDO-based inter mode decision in H.264 video coding, and further proposes an analytic MacroBlock (MB) layer Lagrange multiplier adjustment scheme to adaptively balance the tradeoff between rate and SSIM distortion better. Experimental results show that, at given target bit rate, the proposed method encodes image structural information more effectively, and thus acquires better subjective RDO performance and image quality compared with objective quality based encoding scheme and SSIM-based RC method (without performing SSIM-based RDO inter prediction).