Impact of the ISM magnetic field on GRB afterglow polarization