Mass assembly in quiescent and star-forming galaxies since z=4 from UltraVISTA