Phase Drifts of Sub-pulses during the 2004 Giant Flare of SGR 1806-20 and Settling of the Magnetic Fields