General Relativistic effects and the NIR variability of Sgr A* II: A systematic approach to temporal asymmetry