In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model