Baryonic clues to the puzzling diversity of dwarf galaxy rotation curves