Running Late: Testing Delayed Supermassive Black Hole Growth Models Against the Quasar Luminosity Function