Abstract: In multi-antenna amplify-and-forward two-way relay system, the closed form expression of the MSE optimal relay processing matrix with low complexity based on Minimum Sum Mean Square Error (MSMSE) criterion is derived. To utilize the spatial diversity and frequency diversity comprehensively, the resource allocation strategy in OFDM two-way relay system is investigated, and the layered subcarrier pairing with low complexity and power optimized allocation strategy is proposed. Simulation results show that, the proposed relay processing scheme outperforms other two-way relaying schemes in the performances of system sum rate and bit error ratio, which improve with the number of relay antennas, and the layered subcarrier pairing strategy combined with power allocation can improve the system sum rate dramatically and its performance approaches the optimal strategy.