Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap