Robustifying the Multi-Scale Representation of Neural Radiance Fields