Generalization of the three-term recurrence formula and its applications