Higher-order congruence relations on affine moment graphs I