Fragility of surface states in non-Wigner Dyson topological insulators