Fundamental limits to attractive and repulsive Casimir--Polder forces