Magnetospheric Flows in X-ray Pulsars I: Instability at super-Eddington regime of accretion