The stellar populations of quiescent ultra-diffuse galaxies from optical to mid-infrared spectral energy distribution fitting