Effects of anisotropy in spin molecular-orbital coupling on effective spin models of trinuclear organometallic complexes

We consider layered decorated honeycomb lattices at two-thirds filling, as realized in some trinuclear organometallic complexes. Localized $S=1$ moments with a single-spin anisotropy emerge from the interplay of Coulomb repulsion and spin molecular-orbit coupling (SMOC). Magnetic anisotropies with bond-dependent exchange couplings occur in the honeycomb layers when the direct intracluster exchange and the spin molecular-orbital coupling are both present. We find that the effective spin exchange model within the layers is an XXZ + ${120}^{\ensuremath{\circ}}$ honeycomb quantum compass model. The intrinsic nonspherical symmetry of the multinuclear complexes leads to very different transverse and longitudinal spin molecular-orbital couplings, which greatly enhances the single-spin and exchange coupling anisotropies. The interlayer coupling is described by an XXZ model with anisotropic biquadratic terms. As the correlation strength increases the system becomes increasingly one-dimensional. Thus, if the ratio of SMOC to the interlayer hopping is small this stabilizes the Haldane phase. However, as the ratio increases there is a quantum phase transition to the topologically trivial ``$D$ phase.'' We also predict a quantum phase transition from a Haldane phase to a magnetically ordered phase at sufficiently strong external magnetic fields.