Fractional topological charges and the lowest Dirac modes