Quasiparticle scattering from vortices in d-wave superconductors. II. Berry phase contribution