Can environmental effects spoil precision gravitational-wave astrophysics?