Less is More: Pseudo-Label Filtering for Continual Test-Time Adaptation