Resilience of the quantum Rabi model in circuit QED