Circularly polarized optical excitation of plasmonic nanostructures causes coherent circulating motion of their electrons, which in turn gives rise to strong optically induced magnetization, a phenomenon known as the inverse Faraday effect (IFE). In this study we report how the IFE also significantly decreases plasmon damping. By modulating the optical polarization state incident on achiral plasmonic nanostructures from linear to circular, we observe reversible increases of reflectance by up to 8% and simultaneous increases of optical field concentration by 35.7% under 109 W/m2 continuous wave (CW) optical excitation. These signatures of decreased plasmon damping were also monitored in the presence of an external magnetic field (0.2 T). We rationalize the observed decreases in plasmon damping in terms of the Lorentz forces acting on the circulating electron trajectories. Our results outline strategies for actively modulating intrinsic losses in the metal via optomagnetic effects encoded in the polarization state of incident light.