Quantum signatures of the mixed classical phase space for three interacting particles in a circular trap