Robust field-level inference with dark matter halos