Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representations