Unlocking ultra-deep wide-field imaging with sidereal visibility averaging