Abstract: The characteristics of coherent radiation produced by a cylindrical electron beam passing through a rectangular waveguide and linearly polarized wig-gler is studied. The instability analysis is based on the linearized Vlasov-Maxwell equations for the perturbations about a self-consistent beam equilibrium. The dispersion equation of TMmn mode is deduced and by making use of numerical calculation the radiation frequency and growth rate as a function of electron beam energy and radius, axial magnetic field, wiggler field and wave lengthare presented and discus sed.