Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models