The evolution of the number density of compact galaxies