Drift-induced deceleration of Solar Energetic Particles