Interplay of topology and interactions in quantum Hall topological insulators: U(1) symmetry, tunable Luttinger liquid, and interaction-induced phase transitions

We consider a class of quantum Hall topological insulators: topologically nontrivial states with zero Chern number at finite magnetic field, in which the counterpropagating edge states are protected by a symmetry (spatial or spin) other than time-reversal. HgTe-type heterostructures and graphene are among the relevant systems. We study the effect of electron interactions on the topological properties of the system. We particularly focus on the vicinity of the topological phase transition, marked by the crossing of two Landau levels, where the system is a strongly interacting quantum Hall ferromagnet. We analyze the edge properties using the formalism of the nonlinear $\ensuremath{\sigma}$-model. We establish the symmetry requirement for the topological protection in this interacting system: effective continuous U(1) symmetry with respect to uniaxial isospin rotations must be preserved. If U(1) symmetry is preserved, the topologically nontrivial phase persists; its edge is a helical Luttinger liquid with highly tunable effective interactions. We obtain explicit analytical expressions for the parameters of the Luttinger liquid in the quantum-Hall-ferromagnet regime. However, U(1) symmetry may be broken, either spontaneously or by U(1)-asymmetric interactions. In either case, interaction-induced transitions occur to the respective topologically trivial phases with gapped edge charge excitations.