Deep factorization for speech signal