Robustness of cosmological axion mass limits