Finite groups with large Chermak-Delgado lattices