Planetesimals to Protoplanets II: Effect of Debris on Terrestrial Planet Formation