Kitaev spin-orbital bilayers and their moiré superlattices

Abstract

We determine the phase diagram of a bilayer, Yao-Lee spin-orbital model with inter-layer interactions (J), for several stackings and moiré superlattices. For AA stacking, a gapped ${{\mathbb{Z}}}_{2}$ quantum spin liquid phase emerges at a finite J_c. We show that this phase survives in the well-controlled large-J limit, where an isotropic honeycomb toric code emerges. For moiré superlattices, a finite-q inter-layer hybridization is stabilized. This connects inequivalent Dirac points, effectively ‘untwisting’ the system. Our study thus provides insight into the spin-liquid phases of bilayer spin-orbital Kitaev materials.

Introduction

Quantum spin liquids (QSLs) are disordered phases of magnetic systems with emergent exotic properties arising from their underlying topological character^1,2,3,4,5. The Kitaev model on the honeycomb lattice^6,7 is of particular significance as the first member of a family of exactly-solvable models. Recent years witnessed experimental progress in identifying candidate materials which include a number of iridates⁸ and α-RuCl₃⁹. Kitaev interactions can also be strong in other van der Waals (vdW) materials such as CrI₃^10,11. Moreover, vdW materials can be arranged in stacking patterns and twisted to form moiré superlattices, potentially leading to new phases. Indeed, recent theoretical studies^{12,13,14,15,16,17} predict several magnetic phases in twisted vdW magnets, partially realized experimentally^18,19.

We study the zero-temperature phase diagram of bilayer versions of Kitaev spin-orbital models, initially proposed by Yao and Lee²⁰, with additional inter-layer Heisenberg interactions. Spin-orbital models are generalizations of the original Kitaev model with extra local orbital degrees of freedom (DOF) and Kugel-Khomskii interactions for spin and orbital sectors^{20,21,22,23,24,25,26,27,28,29,30}. Much like Kitaev’s original proposal, spin and orbital DOF can each be represented in terms of three-flavored sets of Majorana fermions. The Yao-Lee model exhibits an emergent ${{\mathbb{Z}}}_{2}$ gauge symmetry with gapped flux excitations (visons) defined exclusively in terms of the orbital DOF²⁰. The inter-layer spin-exchange interactions commute with the intra-layer flux operators, in contrast to the Kitaev model and subsequent bilayer realizations^31,32,33,34. We take advantage of this unique feature by considering only the lowest-energy, zero-flux sector. Furthermore, we treat the spin-exchange interactions in the Hartree approximation. This introduces an effective inter-layer hybridization for the itinerant Majorana fermions associated with the spin DOF. A non-zero expectation value indicates the formation of inter-layer spin-singlets, as shown in Supplementary Note 1. The conservation of the fluxes in the Yao-Lee bilayer, which are defined exclusively in terms the orbital DOF, stands in clear contrast to bilayers based on Kitaev’s original model. As shown below, this leads to distinct phase diagrams and to an enhanced stability of topological QSL phases in Yao-Lee bilayers.

We focus on AA stacking and moiré superlattices, which exhibit fully-gapped spectra, but also briefly cover the gapless, AB stacking case. For AA stacking, the effective hybridization becomes non-zero at a finite value of the inter-layer exchange coupling, and opens a gap in the itinerant Majorana fermion spectrum. This signals a topological phase transition to a gapped ${{\mathbb{Z}}}_{2}$ QSL. We support our Hartree approximation with two additional considerations. First, we show that the bilayer model is equivalent to an attractive Hubbard model with three flavors of complex fermions, for our choice of gauge. Previous quantum Monte Carlo (QMC) studies have shown that the Hubbard model exhibits a single transition to a charge density wave (CDW) phase³⁵, which is equivalent to the bilayer with a non-zero inter-layer hybridization. Secondly, we show that in the limit of large inter-layer interactions, the bilayer model maps onto Kitaev’s toric code³⁶, which is gapped and exhibits topological order. This naturally suggests that the gapped phase predicted by the Hartree approximation is adiabatically connected to the toric code. However, first-order transitions, possibly involving changes in the flux configurations, cannot be completely excluded. For AB stacking, the formation of inter-layer spin singlets leaves the itinerant Majorana fermions gapless with quadratic band touching, in analogy with bilayer graphene³⁷. For moiré superlattices, we consider both uniform (q = 0) and modulated inter-layer effective hybridizations (q ≠ 0). In contrast to the q = 0 case, the finite-q hybridization connects inequivalent Dirac points, effectively ‘untwisting’ the system, and opening a gap. This leads to the emergence of a gapped ${{\mathbb{Z}}}_{2}$ QSL, as for AA stacking.

Kitaev spin-orbital models can be realized in spin-orbit coupled 4d and 5d Mott insulators, as predicted by several recent studies^26,38,39,40. For instance, an enhanced SU(4) symmetry⁴¹ has been advanced for α-ZrCl₃.

Results

Model

Our models include intra-layer Yao-Lee²⁰ interactions on a honeycomb lattice (H_ν), and inter-layer, antiferromagnetic Heisenberg interactions (H_I):

$$H={H}_{\nu }+{H}_{{{{\rm{I}}}}}$$

(1)

$${H}_{\nu }=\mathop{\sum}\limits_{\alpha -{{{\rm{links}}}},\langle ij\rangle }{K}^{(\alpha )}\left({\tau }_{\nu ,i}^{(\alpha )}{\tau }_{\nu ,j}^{(\alpha )}\right)\left({{{{\boldsymbol{\sigma }}}}}_{\nu ,i}\cdot {{{{\boldsymbol{\sigma }}}}}_{\nu ,j}\right)$$

(2)

$${H}_{{{{\rm{I}}}}}=\mathop{\sum}\limits_{ij}{J}_{ij}{{{{\boldsymbol{\sigma }}}}}_{1i}\cdot {{{{\boldsymbol{\sigma }}}}}_{2j}.$$

(3)

We first focus on H_ν, where K^(α) is the nearest neighbor coupling (NN) constant for type-α links (α ∈ {x, y, z}) (Fig. 1a, b). The lattice sites are labeled by i and j, while ν ∈ {1, 2} denotes the two layers. An exact solution is obtained by introducing Majorana fermion representations for the spin and orbital DOF in each layer: ${\sigma }_{\nu ,j}^{(\alpha )}=-{{{\rm{i}}}}{\epsilon }^{\alpha \beta \gamma }{c}_{\nu ,j}^{(\beta )}{c}_{\nu ,j}^{(\gamma )}/2$ and ${\tau }_{\nu ,j}^{(\alpha )}=-{{{\rm{i}}}}{\epsilon }^{\alpha \beta \gamma }{d}_{\nu ,j}^{(\beta )}{d}_{\nu ,j}^{(\gamma )}/2$²⁰. Note that we use a normalization convention for the Majorana fermions where $\{{c}_{\mu ,i}^{(\alpha )},{c}_{\nu ,j}^{(\beta )}\}=2{\delta }_{\alpha \beta }{\delta }_{\mu \nu }{\delta }_{ij}$, and similarly for the b’s. These representations are redundant and the physical states in each layer must be restricted to the eigenstates of ${D}_{\nu ,i}=-{{{\rm{i}}}}{c}_{\nu ,i}^{(x)}{c}_{\nu ,i}^{(y)}{c}_{\nu ,i}^{(z)}{d}_{\nu ,i}^{(x)}{d}_{\nu ,i}^{(y)}{d}_{\nu ,i}^{(z)}$ operators with eigenvalues 1. As in Kitaev’s model, these constraints can be imposed via projection operators P_ν = ∏_i(1 + D_ν,i)/2. The intra-layer Hamiltonians in the Majorana representation can be expressed as ${H}_{\nu }={P}_{\nu }{{{{\mathcal{H}}}}}_{\nu }{P}_{\nu }$, where

$${{{{\mathcal{H}}}}}_{\nu }=\mathop{\sum}\limits_{\langle ij\rangle }{K}^{(\alpha )}{u}_{\nu ,ij}^{\alpha }\left[{{{\rm{i}}}}{c}_{\nu ,i}^{(x)}{c}_{\nu ,j}^{(x)}+{{{\rm{i}}}}{c}_{\nu ,i}^{(y)}{c}_{\nu ,j}^{(y)}+{{{\rm{i}}}}{c}_{\nu ,i}^{(z)}{c}_{\nu ,j}^{(z)}\right].$$

(4)

The bond operators ${u}_{\nu ,ij}^{(\alpha )}=-{{{\rm{i}}}}{d}_{\nu ,i}^{(\alpha )}{d}_{\nu ,j}^{(\alpha )}$, where i, j are on the A and B sublattices, respectively, commute with ${{{{\mathcal{H}}}}}_{\nu }$, and are therefore conserved with eigenvalues ± 1. Both ${{{{\mathcal{H}}}}}_{\nu }$ are invariant under separate ${{\mathbb{Z}}}_{2}$ gauge transformations ${c}_{\nu ,i}^{(\alpha )}\to -{c}_{\nu ,i}^{(\alpha )};$ ${u}_{ij}^{(\alpha )}\to -{u}_{ij}^{(\alpha )}$ with flux operators which are defined by the product of the ${u}_{ij}^{(\alpha )}$ around hexagonal plaquettes.

Fig. 1: Yao-Lee bilayer without twisting.

Illustration of the model for (a) AA and (b) AB stacking patterns. K and J are the intra-layer Kitaev and inter-layer Heisenberg exchange terms, respectively. (c) Effective inter-layer hybridization for AA stacking. Finite $\left\langle {\chi }_{{{{\rm{AA}}}}}\right\rangle$ indicates the formation of inter-layer singlets and leads to gapped itinerant Majorana fermions. (d) Same for AB stacking, leading to quadratic band touching.

Lieb’s theorem⁴² predicts that the ground state lies in the zero-flux sector, with a finite vison gap. We can obtain the itinerant Majorana spectrum by choosing a gauge where u_ij = 1 ∀ 〈ij〉 in both layers. Unless otherwise stated, we use this choice throughout. The three flavors of itinerant Majorana fermions have identical spectra, which are gapless for K_x + K_y > K_z. We consider the symmetric, gapless case with K_x = K_y = K_z = K.

We now consider the inter-layer interactions in H_I. Unlike in the Kitaev model, the visons in the Yao-Lee model are defined exclusively in terms of the orbital DOF, while the itinerant Majorana excitations stem from the spin DOF alone. Consequently, additional terms involving the spin DOF only, including a bilayer coupling, commute with the flux operators. The resulting spectrum can a priori preserve the gapped flux excitations, in contrast to the original Kitaev model^31,33,34. Consequently, we consider Yao-Lee bilayers coupled via inter-layer antiferromagnetic Heisenberg interactions in H_I. Note that we allow for general inter-layer J_ij coupling beyond NN. The bilayer Hamiltonian in the Majorana representation is ${{{\mathcal{H}}}}={\sum }_{\nu }{{{{\mathcal{H}}}}}_{\nu }+{{{{\mathcal{H}}}}}_{{{{\rm{I}}}}}$ where

$${{{{\mathcal{H}}}}}_{{{{\rm{I}}}}}=\mathop{\sum}\limits_{i,j;\alpha \ne \beta }\frac{{J}_{ij}}{2}\left({c}_{1i}^{(\alpha )}{c}_{2j}^{(\alpha )}{c}_{1i}^{(\beta )}{c}_{2j}^{(\beta )}\right).$$

(5)

Self-consistent solutions for AA and AB stacking patterns

The inter-layer interactions are bi-quadratic in the itinerant Majorana operators, thus precluding a closed-form solution. Instead, we treat the inter-layer interactions within a Hartree approximation. This approach is supported by additional considerations, as discussed below.

For the purpose of illustration, we restrict the inter-layer coupling to NN pairs. We do not expect that weaker couplings beyond NNs will alter our conclusions. The on-site mean-field (MF) parameters $\langle {\chi }_{i}^{(\alpha )}\rangle =\langle {{{\rm{i}}}}{c}_{1i}^{(\alpha )}{c}_{2i}^{(\alpha )}\rangle$ preserve the SO(3) spin symmetry, and, we drop the corresponding flavor indices for most of the following discussion.

Before proceeding with a detailed presentation of the results, we first clarify the nature of the MF parameters. In the absence of intra-layer Yao-Lee interactions (K^(α) = 0), the decoupled, inter-layer, spin-singlet states for overlapping sites can equally be described by two eigenstates of ${\chi }_{i}^{(\alpha )}$, with eigenvalues ± 1 for each α, as shown in Supplementary Note 1. The Ising-like nature of these states stems from a redundancy in the representation of the decoupled singlets in terms of the c Majorana fermions. Once the intra-layer interactions are turned on, and a set of bond variables (${u}_{ij}^{(\alpha )}$) is chosen, we obtain a unique MF solution with $\langle {\chi }_{i}^{(\alpha )}\rangle\, \ne \,0$, which is identical for the three flavors. These finite MF parameters likewise indicate the formation of inter-layer spin-singlets in the physical ground-state (GS). However, the Ising-like nature of these parameters is not immediately physical, since the non-trivial phases that we find are not described in terms of a local order parameter. We further elucidate these aspects in the following.

As previously mentioned, we carry out the Hartree approximation in a gauge where all u_ij = 1 in both layers, and obtain the GS

$$\left\vert \Psi \right\rangle =\left\vert \forall \,{u}_{\nu ,ij}^{(\alpha )}=1\right\rangle \otimes \left\vert \left\langle {\chi }_{i}^{(\alpha )}\ne 0\right\rangle \right\rangle .$$

(6)

Importantly, ${\langle {\chi }_{i}^{(\alpha )}\rangle }_{\Psi }$ is not a well-defined, Landau-Ginzburg order parameter for the bilayer. Indeed, any gauge transformation, implemented for instance by ${D}_{\nu ,i}\left\vert \Psi \right\rangle$, changes the sign of the associated $\langle {\chi }_{i}^{(\alpha )}\rangle$ together with those of the three bonds extending from i in layer ν. Furthermore, the physical GS is obtained by applying the projector P to $\left\vert \Psi \right\rangle$ as

$${\left\vert \Psi \right\rangle }_{{{{\rm{Phys}}}}}=P\left\vert \Psi \right\rangle .$$

(7)

${\left\vert \Psi \right\rangle }_{{{{\rm{Phys}}}}}$ amounts to a linear superposition of all gauge-symmetrized states which preserve a net zero flux, as shown in Supplementary Note 2 A. States with finite $\pm | \langle {\chi }_{i}^{(\alpha )}\rangle |$ occur with equal weight, implying that ${\langle {\chi }_{i}^{(\alpha )}\rangle }_{{{{\rm{Phys}}}}}=0$.

In order to characterize transitions in the physical GS, we instead consider a gauge-invariant correlator

$${\left\langle {C}_{ij}^{(\alpha )}\right\rangle }_{{{{\rm{Phys}}}}}={\left\langle \left(\mathop{\prod }\limits_{\alpha -{{{\rm{links}}}},\left\langle {i}^{{\prime} }{j}^{{\prime} }\right\rangle }^{{\prime} }{u}_{1,{i}^{{\prime} }{j}^{{\prime} }}^{(\alpha )}{u}_{2,{i}^{{\prime} }{j}^{{\prime} }}^{(\alpha )}\right){\chi }_{i}^{(\alpha )}{\chi }_{j}^{(\alpha )}\right\rangle }_{{{{\rm{Phys}}}}},$$

(8)

where the strings of bonds connect operators at the end sites i, j. In Supplementary Note 2 B, we show that the expectation value of ${C}_{ij}^{(\alpha )}$ in the physical GS matches that of a two-point correlator for χ^(α) in $\left\vert \Psi \right\rangle$.

$${\left\langle {C}_{ij}^{(\alpha )}\right\rangle }_{{{{\rm{Phys}}}}}={\left\langle {\chi }_{i}^{(\alpha )}{\chi }_{j}^{(\alpha )}\right\rangle }_{\Psi }.$$

(9)

From this expression, long-range order in $\left\vert \Psi \right\rangle$ is equivalent to ${\langle {\chi }_{i}^{(\alpha )}\rangle }_{\Psi }\,\ne\, 0$. It follows that non-vanishing MF parameters imply a finite ${\langle {C}_{ij}^{(\alpha )}\rangle }_{{{{\rm{Phys}}}}}$, in the limit of infinite separation. In Supplementary Note 2 C, we express ${\langle {C}_{ij}^{(\alpha )}\rangle }_{{{{\rm{Phys}}}}}$ in terms of the spin and orbital operators of the bilayer, and show that it signals a topological phase transition to a gapped, ${{\mathbb{Z}}}_{2}$ QSL for the AA-stacked case, which involves the formation of inter-layer spin-singlets. We note that all subsequent conclusions regarding the MF parameters, obtained in the Hartree approximation and in a fixed gauge, are to be understood in the present context.

We now discuss our results in the Hartree approximation. For the AA stacking pattern, the A and B sublattice sites overlap (Fig. 1a). The inter-layer interactions involve two pairs of sites per unit cell: ${{{{\mathcal{H}}}}}_{{{{\rm{I}}}}}=-2J[{\sum }_{i\in A,\alpha }\langle {\chi }_{{{{\rm{AA}}}}}\rangle ({{{\rm{i}}}}{c}_{1i}^{\alpha }{c}_{2i}^{\alpha })+{\sum }_{j\in B,\alpha }\langle {\chi }_{{{{\rm{BB}}}}}\rangle ({{{\rm{i}}}}{c}_{1j}^{\alpha }{c}_{2j}^{\alpha })]$. Solutions which are both uniform and symmetric in the sublattice index ($\left\langle {\chi }_{{{{\rm{AA}}}}}\right\rangle =\left\langle {\chi }_{{{{\rm{BB}}}}}\right\rangle$) amount to gapless itinerant Majorana fermions, with shifted Dirac cones. In contrast, when the hybridization has an alternating sign on the two sublattices (〈χ_AA〉 = − 〈χ_BB〉), the spectrum is gapped, leading to a lower ground-state energy. Our self-consistent solutions are shown in Fig. 1c as functions of J/K. We find that the critical value for this transition is J_c/K = 0.55.

To establish the stability of our solutions beyond the Hartree approximation, we map ${{{\mathcal{H}}}}$ to an equivalent form by using complex fermions ${f}_{i}^{\alpha }=({c}_{1i}^{\alpha }+{{{\rm{i}}}}{c}_{2i}^{\alpha })/2$:

$${{{\mathcal{H}}}}=2K\mathop{\sum}\limits_{\langle ij\rangle ,\alpha }({{{\rm{i}}}}{f}_{A,i}^{\alpha {\dagger} }{f}_{B,j}^{\alpha }+{{{\rm{H.c.}}}})-2J\mathop{\sum}\limits_{i}{\left({n}_{i}-\frac{3}{2}\right)}^{2},$$

(10)

where ${n}_{i}={\sum }_{\alpha }{f}_{i}^{\alpha {\dagger} }{f}_{i}^{\alpha }$. For J > 0, Eq. (10) describes an attractive Hubbard model with three flavors of complex fermions. This model exhibits a single, broken-symmetry CDW phase with finite 〈n_A〉 = − 〈n_B〉, as determined by QMC³⁵. This Ising order parameter acts as a mass term for the complex fermions, and gaps their spectrum. It is equivalent to a solution in which $\left\langle \chi \right\rangle$ alternates between sublattices in the Yao-Lee bilayer. Importantly, the Hubbard model and CDW order parameter were obtained by fixing the gauge. While the CDW breaks inversion symmetry in the Hubbard model, the same cannot be said of the physical GS of the bilayer model. As previously mentioned, the order parameters obtained in a fixed gauge are physically meaningful only in relation to the gauge-invariant correlator ${\langle {C}_{ij}^{(\alpha )}\rangle }_{{{{\rm{Phys}}}}}$ defined in Eq. (9).

The GS obtained in the Hartree approximation for AA stacking has a fourfold topological degeneracy, as shown in Supplementary Note 9. This result is corroborated by the perturbative analysis in the large-J limit discussed in the following.

For AB stacking, the A sublattice sites of layer 1 lie directly on top of the B sublattice sites of layer 2, with a single bond per unit cell, (Fig. 1b). Therefore, for finite $\left\langle \chi \right\rangle$ beyond J_c/K ≃ 1.1, the itinerant Majorana spectrum is similar to that of AB-stacked bilayer graphene with quadratic band touching³⁷. The self-consistent solutions for 〈χ_AB〉 are shown in Fig. 1d. A mapping to an equivalent model as in the AA case is not apparent here.

The choice of uniform u_ij = 1 for both layers implies that the system persists in a zero-flux sector. This is supported by additional MF calculations with several distinct non-zero flux patterns (Supplementary Note 3), which indicate that the zero-flux states are lower in energy. Furthermore, the effective Hamiltonian in the large-J limit (see below) similarly prefers this configuration.

We comment on the stability of the phases obtained in the Hartree approximation in the presence of additional inter-layer, NN, spin-exchange interactions, which we realistically expect to be subleading. For the AA-stacked bilayer, the gapped phase obtained for J > J_c is stable with respect to additional, infinitesimal, NN interactions. For the AB-stacked bilayer with J > J_c, our Hartree approximation predicts quadratic band touching, which implies a finite density of states for the itinerant Majorana fermions at zero energy. Additional inter-layer, NN interactions are therefore likely relevant in a renormalization-group sense. Establishing the nature of the low-energy phases in these cases requires further analysis, at Hartree level and beyond, and we reserve such questions for future study.

The GSs obtained in the fixed gauge survive projection onto the physical sector, as shown in Supplementary Note 2 A.

Limit of large inter-layer interactions with AA stacking pattern

We consider the AA stacking pattern in the limit of large J/K. To zeroth order in the intra-layer (K) terms, the GS manifold consists of a collection of independent inter-layer spin singlets with degenerate orbital states. We derive an effective Hamiltonian on the GS manifold, perturbatively up to 6^th order in K/J

$$\begin{array}{ll}{H}_{{{{\rm{eff}}}}}\,=\,\mathop{\sum}\limits_{\begin{array}{l}{\scriptstyle\alpha -{{{\rm{links}}}}}\\ {\displaystyle\langle ij}\rangle \end{array}}\left({g}_{2}{\tau }_{1i}^{(\alpha )}{\tau }_{2i}^{(\alpha )}{\tau }_{1j}^{(\alpha )}{\tau }_{2j}^{(\alpha )}+{g}_{3}{\sum \limits_{\nu =1,2}}{\tau }_{\nu i}^{(\alpha )}{\tau }_{\nu j}^{(\alpha )}\right)\\ \qquad\quad\,+\,{g}_{6}{\sum\limits_{\hexagon,\nu }}{W}_{p}^{\nu }\end{array}$$

(11)

where ${W}_{p}^{1(2)}$ is the flux operator defined on the honeycomb plaquettes on layer 1(2) as ${W}_{p}^{\nu }={\tau }_{\nu i}^{(z)}{\tau }_{\nu j}^{(y)}{\tau }_{\nu k}^{(x)}{\tau }_{\nu l}^{(z)}{\tau }_{\nu m}^{(y)}{\tau }_{\nu n}^{(y)}$ (Fig. 2d). Please consult Supplementary Notes 4 and 5 for additional details. The coupling constants are g₂ = − K²/4J, g₃ = − K³/J², and g₆ = − K⁶/(8J)⁵. The g₂ term describes Kitaev interactions around inter-layer plaquettes while the g₃ term is a standard Kitaev interaction in each layer.

Fig. 2: Effective Hamiltonian in the large-J limit.

a The bond operator ${r}_{ij}={{{\rm{sgn}}}}({p}_{i}^{(\alpha )})={{{\rm{sgn}}}}({p}_{j}^{(\alpha )})$, which is defined in the ground-state manifold of the g₂ terms in Eq. (11). Red (black) solid lines correspond to +(−) bonds. Conversely, bond configurations which include NNs with ${{{\rm{sgn}}}}({p}_{i}^{(\alpha )})\,\ne\, {{{\rm{sgn}}}}({p}_{j}^{(\alpha )})$, shown here with blue dashed lines, are not labeled by r_ij bonds. These configurations correspond to excited states. b A ground-state manifold configuration which minimizes the g₂ terms and which also obeys the local product constraint, equivalent to an Ising Gauss’s law. c g₃ terms on sites i, j flip four adjacent bonds resulting in an excited state. d g₆ terms change bond configurations from r_ij = 1 to r_ij = − 1 around each plaquette and thus connect configurations which belong to the Ising Gauss' Law manifold.

Note that g₆ terms promote uniform ${W}_{p}^{1(2)}=1$ corresponding to a zero-flux low-energy manifold. This configuration is preserved by the remaining terms which commute with the ${W}_{p}^{1(2)}=1$.

We first focus on the the leading g₂ terms, and define new operators ${p}_{i}^{(\alpha )}={\tau }_{i1}^{(\alpha )}{\tau }_{2i}^{(\alpha )}$, which unlike the τ’s, all commute with each other. Furthermore their product amounts to ${p}_{i}^{(x)}{p}_{i}^{(y)}{p}_{i}^{(z)}=-1$. Therefore, we use local basis states which are eigenstates of all p operators and which also satisfy the product rule: $\{\left\vert -,-,-\right\rangle ,\left\vert -,+,+\right\rangle ,\left\vert +,-,+\right\rangle ,\left\vert +,+,-\right\rangle \}$ where ± denotes the eigenvalue of p^(α), (α = x, y, z). The g₂ < 0 terms favor equal-p^(α) states on NN sites. Therefore, in the GS manifold of the g₂ terms, it is possible to define bond variables r_ij = ± 1 for pairs of ( + , + ) and ( − , − ) eigenvalues of ${p}_{i/j}^{(\alpha )}$, respectively. For configurations that do not minimize the g₂ terms, r_ij is not defined (Fig. 2a). In addition to minimizing the g₂ terms, the GS manifold must also satisfy the local constraint due to ${p}_{i}^{(x)}{p}_{i}^{(y)}{p}_{i}^{(z)}=-1$. Taken together, these conditions are equivalent to bond configurations which obey an Ising Gauss’s law ${G}_{i}^{P}={\prod }_{{\mathsf{Y}}}{r}_{ij}=-1$ (Fig. 2b). We stress that the r_ij bond variables and Gauss’s law are only defined in the GS manifold of the g₂ terms.

Next, we examine the effect of g₃ and g₆ terms acting on the GS manifold obtained from the combined effects of the g₂ terms and local product constraints. Each ${\tau }_{1,2}^{(\alpha )}$ acting on $\vert {p}_{x},{p}_{y},{p}_{z}\rangle$ preserves the corresponding α eigenvalues but flips the remaining two (see Supplementary Note 5). Therefore, the g₆ terms acting on a plaquette flips all of the r_ij bond variables therein (Fig 2d), leading to an effective term

$$-\kappa {\sum \limits_{\hexagon}}\left(\left\vert \hexagon\right\rangle \left\langle \bar{\hexagon}\right\vert +{{{\rm{H.c.}}}}\right)$$

(12)

In contrast, the single g₃ term on sites 〈ij〉, connects a ground-state configuration to excited states (Fig. 2 (c)). Consecutive application of g₃ terms around a plaquette leads to plaquette flips, but these processes are subdominant with respect to those due to the g₆ term.

As shown in Supplementary Note 6, the resonance term in Eq. (12), along with Gauss’ law, describe Kitaev’s toric code³⁶ on a honeycomb lattice. We thus conclude that the bilayer model in the limit of large inter-layer spin exchange interactions is in a gapped abelian ${{\mathbb{Z}}}_{2}$ topological QSL phase.

Self-consistent solutions for moiré superlattices

We generalize the Hartree approximation to include the effects of small-angle twists. We follow Ref. ⁴³ to derive a low-energy theory defined on the moiré extended BZ, as shown in Supplementary Note 7.

To allow for non-vanishing inter-layer interactions under arbitrary, small twist angles, we extend the former beyond overlapping NN pairs and allow for an implicit decay with increasing pair separation. In general, this entails a decay of the Fourier components J(k) with ∣k∣, and involves interactions which are delocalized in the extended BZ. In the low-energy limit, the interactions are naturally limited to the vicinity of a discrete set of equivalent crystal momenta throughout the extended BZ. In practice, we keep only J(k) with ∣k∣ ⪅ ∣2K₀₀∣, or twice the distance from the origin to the nearest Dirac point (Eq. 59 in Supplementary Note 7). We also assume that the retained Fourier components are all comparable in magnitude. The restrictions on the values of J(k) allow us to explicitly consider the Yao-Lee bilayer analogs of flat bands in twisted bilayer graphene⁴³. However, our conclusions are independent of this approximation, as discussed in the following. We also limit the hybridization to states in the vicinity of Dirac points in neighboring moiré reciprocal unit cells. This truncation is justified in the low-energy limit, where small-momentum scattering processes are dominant.

The intra-layer terms amount to the usual Dirac fermions for the two layers, which are shifted with respect to each other due to twisting. The inter-layer interactions together with the approximations discussed previously can be written as

$${{{{\mathcal{H}}}}}_{{{{\rm{I}}}}}=-\frac{4J}{N}\mathop{\sum}\limits_{nm}\left[\left\langle {\chi }_{00}^{{\dagger} }({{{\bf{q}}}})\right\rangle {\chi }_{nm}({{{\bf{q}}}})+\left\langle {\chi }_{00}({{{\bf{q}}}})\right\rangle {\chi }_{nm}(-{{{\bf{q}}}})\right] +\,{{\mbox{H.c.}}}$$

(13)

where

$$\begin{array}{ll}{\chi }_{nm}({{{\bf{q}}}})\,=\,{{{\rm{i}}}}\mathop{\sum}\limits_{{{{\bf{k}}}}}\left[{c}_{1,\alpha ;nm}^{{\dagger} ,(\mu )}({{{\bf{k}}}}){c}_{2,\beta ;nm}^{(\mu )}({{{\bf{k}}}}-{{{\bf{q}}}})\right.\\ \quad\qquad\qquad +\,{{{{\rm{e}}}}}^{-{{{\rm{i}}}}{{{{\mathcal{G}}}}}_{2}\cdot ({{{{\boldsymbol{\tau }}}}}_{\alpha }-{{{{\boldsymbol{\tau }}}}}_{\beta })}{c}_{1,\alpha ;nm}^{{\dagger} ,(\mu )}({{{\bf{k}}}}){c}_{2,\beta ;n+1m}^{(\mu )}({{{\bf{k}}}}-{{{\bf{q}}}})\\ \left. \quad\qquad\qquad+\,{{{{\rm{e}}}}}^{-{{{\rm{i}}}}{{{{\mathcal{G}}}}}_{3}\cdot ({{{{\boldsymbol{\tau }}}}}_{\alpha }-{{{{\boldsymbol{\tau }}}}}_{\beta })}{c}_{1,\alpha ;nm}^{{\dagger} ,(\mu )}({{{\bf{k}}}}){c}_{2,\beta ;nm+1}^{(\mu )}({{{\bf{k}}}}-{{{\bf{q}}}})\right]\end{array},$$

(14)

while

$${c}_{\alpha ,1/2}^{{\dagger} ,(\mu )}({{{{\bf{K}}}}}_{00}+{{{\bf{k}}}}-n{{{{\bf{b}}}}}_{{{{\bf{2}}}}}-m{{{{\bf{b}}}}}_{3})={c}_{\alpha ,1/2;nm}^{{\dagger} ,(\mu )}({{{\bf{k}}}})$$

(15)

are states with an effective Dirac dispersion which is shifted by the moiré reciprocal vectors b_2,3 with respect to the Dirac point centered on the moiré first BZ at n = m = 0. K₀₀ is the position of the Dirac point of layer 1 in the first BZ while ${{{{\mathcal{G}}}}}_{2,3}$ are the reciprocal unit vectors of layer 1. The sums over momenta k cover the extended moiré BZ, with an implicit cutoff. The vectors τ_α,β denote the shift of the A, B sublattices in layers 1 and 2, respectively. q is a vector contained within a single moiré reciprocal unit cell. As already mentioned, our approximations, and the cutoff for J(k) in particular, ensure that the form of the effective hybridization in Eqs. (13) and (14) bears a close resemblance to that of twisted bilayer graphene⁴³. For more details on the MF procedure, please see Supplementary Note 7.

We consider two cases, one for q = 0 corresponding to a uniform inter-layer hybridization, and another for finite q = q₁ where ${{{{\bf{q}}}}}_{1}=-8\pi /3\sin (\theta /2)\hat{y}$⁴³ which denotes the shift between the Dirac points in layers 1 and 2 in the first BZ due to twisting (see Fig. 3d). In both cases, 〈χ_AA〉 = − 〈χ_BB〉 acquire finite expectation values whereas 〈χ_AB〉 and 〈χ_BA〉 remain pinned to zero. Our self-consistent calculations indicate that the critical coupling J_c/K for the q = q₁ solution is below its q = 0 counterpart for the entire range of twist angles (Fig. 3a), indicating that the modulated hybridization is energetically favored. A finite-q hybridization connects states near inequivalent Dirac points in the moiré BZ and gaps the spectrum, as illustrated in Fig. 3b, effectively ‘untwisting’ the system. In contrast, for q = 0, the spectrum remains gapless, (Fig. 3c). Consequently, the finite-q solution is preferred for any non-zero twist angle. The two solutions merge smoothly as θ → 0 since q₁ vanishes in this limit, at which point the low-energy sectors match the self-consistent solutions of the un-twisted bilayer with AA stacking.

Fig. 3: Moiré superlattices of the Yao-Lee bilayer.

a Critical inter-layer exchange ${J}_{c}\left(\theta \right)$, as a function of twisting angle θ, and for q = 0 and q = q₁ in the effective interlayer hybridization $\left\langle {\chi }_{00}({{{\bf{q}}}})\right\rangle$ (Eq. (13)), in units of ${J}_{c}\left(\theta =0\right)$. The q = q₁ solution is energetically preferred. The spectrum for (b) q = q₁ and (c) q = 0 in units of the intra-layer coupling K (Eq. (2)). d Moiré reciprocal unit cells for J < J_c (black) and J > J_c (red). 1 and 2 denote the Dirac points of the two layers which are separated by q₁ for J < J_c, corresponding to vanishing inter-layer hybridization. When the latter acquires a finite value for J > J_c and q = q₁, the two Dirac points are shifted to the Γ point of the folded BZ, and are subsequently gapped. e Schematic illustration of the moiré lattice vectors for J < J_c (black) and J > J_c (red) corresponding to zero and non-zero (q = q₁) hybridization, respectively.

At the level of the Hartree approximation, our results indicate that the gap remains open as the small-angle twisting is turned on. Within the same approximation, we conclude that resulting phases are adiabatically connected with the AA-stacked bilayer in the large-J limit. Our results suggest that, beyond the Hartree approximation, the gap in the spin excitations of the bilayer survives, and that the GS remains in a net zero-spin state for small-angle twisting. We expect that the intra-layer interactions lift the extensive degeneracy of the orbitals, resulting in a gapped, ${{\mathbb{Z}}}_{2}$ QSL, as for the case with AA stacking.

For the gauge choice of uniform and identical bonds in both layers, the q = q₁ incommensurate, inter-layer hybridization breaks the translation symmetry of simple moiré pattern but preserves all other symmetries. It consequently triples the size of the moiré unit cell (Fig. 3e). Figure 3d shows the moiré (black) and folded (red) BZ’s, respectively. The rotated Dirac cones at the corners of the moiré BZ are folded onto the Γ point. However, since the effective hybridization is not gauge invariant, this does not imply a true translation symmetry breaking, but instead demonstrates that small-angle twisting preserves the gapped ${{\mathbb{Z}}}_{2}$ QSL.

We note that the main conclusion of the preceding paragraphs, that twisting the AA-stacked bilayer by small angles preserves the gapped spectrum, does not rely on our assumptions concerning the cutoff in J(k). Indeed, keeping only the leading J(0) terms in the expression for the self-consistent hybridization (Eqs. 82-84 in Supplementary Note 7), which likewise connect pairs of Dirac points in the moiré BZ, leads to a similar conclusion in the Hartree approximation.

Discussion

It is instructive to contrast the bilayer Yao-Lee model considered here with the bilayer Kitaev models of earlier works. For a bilayer Kitaev model, a mean-field study predicts gapped QSL and trivial dimer phases for intermediate and large values of the inter-layer coupling, respectively³¹. However, an exact diagonalization study³³ finds that a single phase transition between gapless QSL and trivial dimer phases occurs at a substantially weaker coupling J/K ~ 0.06. Our results indicate that the QSL phase remains stable in Yao-Lee bilayers for large but finite intra-layer couplings, while the trivial dimer phase emerges only in the absence of intra-layer terms (K = 0). The stability of the gapped QSL in the Yao-Lee bilayer can be attributed to the effect of the spin operators on the zero-flux GSs of the decoupled layers. In the Kitaev model, the spin operators create two visons, as shown in Supplementary Note 10. By contrast, the spin operators in the Yao-Lee model preserve the zero-flux GS manifold, since the spin and flux operators are associated with different DOF.

We studied the zero-temperature phase diagram of a bilayer Yao-Lee model with inter-layer interactions. For AA stacking, we determined that finite inter-layer singlet correlations gap the itinerant Majorana fermion spectrum. We also derived an effective Hamiltonian in the limit of large J/K, and demonstrated that it maps onto the toric code. In the absence of any additional transitions which close the gap, we concluded that the solutions obtained via the Hartree approximation are adiabatically connected to the large inter-layer interaction limit, leading to the stability of a topological gapped ${{\mathbb{Z}}}_{2}$ QSL. This phase persists for moiré superlattices under small-angle twisting. Detailed studies of the AB stacked phases and of the toric code models in the large inter-layer coupling limit are clearly desirable.

Methods

For more details on the methods used to obtain the results shown in Figs. 1–3 please consult Supplementary Notes 7 and 8.