Robustness of topological states with respect to lattice instability in the nonsymmorphic topological insulator KHgSb

We report a polarized Raman scattering study of nonsymmorphic topological insulator KHgSb with hourglasslike electronic dispersion. Supported by theoretical calculations, we show that the lattice of the previously assigned space group $P{6}₃/mmc$ (No. 194) is unstable in KHgSb. While we observe one of two calculated Raman active ${\mathrm{E}}_2g$ phonons of space group $P{6}₃/mmc$ at room temperature, an additional ${\mathrm{A}}_1g$ peak appears at 99.5 ${\mathrm{cm}}^{\ensuremath{-}1}$ upon cooling below ${T}^*=150$ K, which suggests a lattice distortion. Several weak peaks associated with two-phonon excitations emerge with this lattice instability. We also show that the sample is very sensitive to high temperature and high laser power, conditions under which it quickly decomposes, leading to the formation of Sb. Our first-principles calculations indicate that space group $P{6}₃mc$ (No. 186), corresponding to a vertical displacement of the Sb atoms with respect to the Hg atoms that breaks the inversion symmetry, is lower in energy than the presumed $P{6}₃/mmc$ structure and preserves the glide-plane symmetry necessary to the formation of hourglass fermions.