Learning to Switch Between Machines and Humans