Physical limits of ultra-high-finesse optical cavities: Taming two-level systems of glassy metal oxides