Concomitants of ternary quartics and vector-valued Siegel and Teichmüller modular forms of genus three

Abstract

We show how one can use the representation theory of ternary quartics to construct all vector-valued Siegel modular forms and Teichmüller modular forms of degree 3. The relation between the order of vanishing of a concomitant on the locus of double conics and the order of vanishing of the corresponding modular form on the hyperelliptic locus plays an important role. We also determine the connection between Teichmüller cusp forms on $\overline{\mathcal {M}}_{g}$ and the middle cohomology of symplectic local systems on ${\mathcal {M}}_{g}\,$. In genus 3, we make this explicit in a large number of cases.

1 Introduction

This paper contains two main results. Firstly, we show how the representation theory associated to ternary quartics can be used to describe and construct all vector-valued Siegel and Teichmüller modular forms of degree 3 (Theorem 11.6). This uses the classical notion of concomitants, of which invariants, covariants, and contravariants are special cases. Secondly, we describe for arbitrary g the precise relation between certain spaces of Teichmüller cusp forms on $\overline{\mathcal {M}}_{g}$ and the middle cohomology of the standard symplectic local systems on ${\mathcal {M}}_{g}$ (Theorem 13.1). We illustrate the main results by a substantial number of examples, focusing on genus 3 for the second theorem, and obtain several other results of independent interest.

Let ${\mathcal {A}}_{3}$ be the moduli space of principally polarized abelian varieties of dimension 3. Over the complex numbers the orbifold ${\mathcal {A}}_{3}({\mathbb {C}})$, associated to the moduli space ${\mathcal {A}}_{3}\,$, can be written as an arithmetic quotient $\Gamma _3\backslash \mathfrak {H}_3\,$, where the Siegel modular group $\Gamma _3={\mathrm {Sp}}({6},{\mathbb {Z}})$ of degree 3 acts on the Siegel upper half space $\mathfrak {H}_3$ of degree 3 in the usual way. The moduli stack ${\mathcal {A}}_{3}$ carries a natural vector bundle ${\mathbb {E}}$ of rank 3, the Hodge bundle with fibre $H^0(X,\Omega _X^1)$ over the point [X] of ${\mathcal {A}}_{3}\,$. Over ${\mathbb {C}}$ it can be given as a quotient $\Gamma _3\backslash (\mathfrak {H}_3\times {\mathbb {C}}^3)$ where the action on ${\mathbb {C}}^3$ corresponds to the standard representation of $\mathrm{GL}(3)$. In a similar way, for each irreducible representation $\rho$ of $\mathrm{GL}(3)$ there is an associated vector bundle ${\mathbb {E}}_{\rho }$ that can be constructed from ${\mathbb {E}}$ by using a Schur functor. Sections of powers of the determinant bundle $\det ({\mathbb {E}})$ on ${\mathcal {A}}_{3}$ can be identified with scalar-valued Siegel modular forms, while sections of ${\mathbb {E}}_{\rho }$ can be identified with vector-valued Siegel modular forms. The vector bundle ${\mathbb {E}}$, and more generally all ${\mathbb {E}}_{\rho }$, extend in a natural way over the standard smooth compactification of ${\mathcal {A}}_{3}$. By the Koecher principle holomorphic sections of ${\mathbb {E}}_{\rho }$ extend over this compactification.

Let ${\mathcal {M}}_{3}$ denote the moduli space of curves of genus 3. The Torelli morphism $t:{\mathcal {M}}_{3}\rightarrow {\mathcal {A}}_{3}$ is a morphism of algebraic stacks of degree 2 ramified along the hyperelliptic locus. By pullback under t we obtain the Hodge bundle ${\mathbb {E}}^{\prime }$ on ${\mathcal {M}}_{3}$ and for each irreducible representation $\rho$ of $\mathrm{GL}(3)$ a vector bundle ${\mathbb {E}}^{\prime }_{\rho }$ on ${\mathcal {M}}_{3}$. Sections of such a bundle ${\mathbb {E}}^{\prime }_{\rho }$ are called Teichmüller modular forms of degree 3. The vector bundle ${\mathbb {E}}^{\prime }$ and hence all the ${\mathbb {E}}^{\prime }_{\rho }$ can be extended in a natural way over the Deligne–Mumford compactification $\overline{\mathcal M}_3$ and we show that a holomorphic section of ${\mathbb {E}}^{\prime }_{\rho }$ automatically extends to a holomorphic section of the extended bundle.

The first Teichmüller modular form that is not (a pullback under the Torelli map of) a Siegel modular form is the scalar-valued form $\chi _9$ of weight 9 whose existence was proven by Ichikawa [22, 23]. There is an involution on the space of Teichmüller forms such that a Teichmüller modular form F that is invariant is the pullback of a Siegel modular form, while an anti-invariant form is divisible by $\chi _9$, with quotient the pullback of a Siegel modular form. The study of Teichmüller modular forms of degree 3 reduces therefore to that of Siegel modular forms (Sect. 7).

A nonhyperelliptic curve of genus 3 has as canonical image a quartic curve in ${\mathbb {P}}^2$ and thus the open part ${\mathcal M}_3^\mathrm{nh}$ of ${\mathcal {M}}_{3}$ that corresponds to nonhyperelliptic curves has a description as the quotient of an open part of the space of ternary quartics under the action of $\mathrm {GL}(3)$. Thus the representation theory of ternary quartics enters, i.e., the decomposition of ${\mathrm {Sym}}^d({\mathrm {Sym}}^4(\mathbb {C}^3))$ into irreducible representations of $\mathrm {GL}(3)$. The classical notion of concomitants of ternary quartics makes this decomposition explicit (see  [7]). Our first main result is a complete description of Siegel (and Teichmüller) modular forms of degree 3 in terms of concomitants. We associate to a concomitant a meromorphic Siegel or Teichmüller modular form which is holomorphic outside the hyperelliptic locus on ${\mathcal {M}}_{3}$ or ${\mathcal {A}}_{3}$ (see Sect. 10). This is analogous to our description of Siegel modular forms of degree 2 in terms of covariants of the action of $\mathrm {GL}(2)$ on the space of binary sextics, see [8].

The most basic concomitant is the universal ternary quartic f. It defines a meromorphic Teichmüller modular form $\chi _{4,0,-1}$ of weight $(4,0,-1)$. Multiplication by $\chi _9$ makes it into a holomorphic Siegel modular form $\chi _{4,0,8}\,$, a section of ${\mathrm {Sym}}^4({\mathbb {E}}) \otimes \det ^8({\mathbb {E}})$, the ‘first’ Siegel cusp form of degree 3 (cf. Sect. 4).

The fact that in general concomitants define meromorphic modular forms that become holomorphic after multiplication with a suitable power of $\chi _9$ forces us to analyze the order of vanishing of the modular form associated to a concomitant along the hyperelliptic locus. We express this order of vanishing in terms of the order of vanishing of the concomitant along the locus of double conics in the space of ternary quartics. Remarkably, this enables us to identify the spaces of concomitants of ternary quartics with given order of vanishing along the locus of double conics with the spaces of Siegel modular forms with given order of vanishing along the divisor at infinity (Theorem 11.6).

Instead of working with $\chi _{4,0,-1}$ and multiplying with a power of $\chi _9\,$, we can also work with $\chi _{4,0,8}$ and obtain from a concomitant a holomorphic Siegel modular form, which may be divisible by a power of $\chi _9\,$. In order to use this in an efficient way, we need to know the Fourier expansion of $\chi _{4,0,8}$ rather well. We obtain it by analyzing the Schottky form, a scalar-valued Siegel cusp form of degree 4 and weight 8, along the ‘diagonally embedded’ $\mathfrak {H}_3 \times \mathfrak {H}_1 \subset \mathfrak {H}_4$ in the Siegel upper half space of degree 4 (see Sects. 4–5).

To demonstrate our approach, we construct a substantial number of Siegel cusp forms and we compute some of their Hecke eigenvalues, finding agreement with  [4].

In [3], Bergström and two of the present authors studied the cohomology of symplectic local systems on ${\mathcal {A}}_{3}\,$. The same method can also be used to study the cohomology of the corresponding local systems on ${\mathcal {M}}_{3}\,$. (As is well-known, this cohomology is very closely related to the cohomology of the moduli spaces ${\mathcal {M}}_{3,n}$ of n-pointed curves of genus 3.) If the local system is even, its cohomology on ${\mathcal {M}}_{3}$ equals that on $t({\mathcal {M}}_{3})={\mathcal {A}}_{3}-{\mathcal {A}}_{2,1}\,$, the moduli space of indecomposable principally polarized abelian threefolds. However, in the odd case, the cohomology can not be explained in terms of ${\mathcal {A}}_{3}\,$. It is here that Teichmüller modular forms enter. Our second main result (Theorem 13.1) gives the precise relationship between certain spaces of Teichmüller cusp forms and the middle cohomology of symplectic local systems on ${\mathcal {M}}_{g}\,$. This is a partial analogue of the results of Faltings and Chai  [15] for ${\mathcal {A}}_{g}\,$. We then specialize to genus 3, where we have determined these spaces of Teichmüller cusp forms in a substantial number of cases, via computations with concomitants. Finally, we discuss how these results match perfectly with conjectural formulas for the ‘motivic’ Euler characteristics of the symplectic local systems of weight at most 20 on ${\mathcal {M}}_{3}\,$, obtained from counts of curves over finite fields, as in  [3].

2 Siegel modular forms

Let ${\mathbb {Z}}^{2g}$ be the symplectic lattice of rank 2g with basis elements $e_1,\ldots , e_g,f_1,\ldots ,f_g$ and with the symplectic pairing given by $\langle e_i,e_j\rangle =0=\langle f_i,f_j\rangle$ and $\langle e_i,f_j\rangle =\delta _{ij}$. We write $\Gamma _g= \mathrm{Sp}(2g,{\mathbb {Z}})=\mathrm{Aut}({\mathbb {Z}}^{2g}, \langle \, , \, \rangle )$ for the Siegel modular group of degree g. An element $\gamma \in \Gamma _g$ can be given as a $2\times 2$ matrix of $g\times g$ blocks with respect to the basis $e_1,\ldots ,e_g,f_1,\ldots ,f_g$. We will denote by ${\mathcal {A}}_{g}$ the moduli stack of principally polarized abelian varieties and by ${\mathcal {M}}_{g}$ the moduli stack of curves of genus g (for $g>1$). Over the complex numbers the orbifold ${\mathcal {A}}_{g}({\mathbb {C}})$, associated to the moduli space ${\mathcal {A}}_{g}$, can be written as $\Gamma _g\backslash \mathfrak {H}_g$, where the Siegel modular group $\Gamma _g={\mathrm {Sp}}({2g},{\mathbb {Z}})$ of degree g acts on the Siegel upper half space

$$\begin{aligned} \mathfrak {H}_g=\{ \tau \in \mathrm{Mat}(g \times g, {\mathbb {C}}): \tau ^t=\tau , \mathrm{Im}(\tau ) > 0\} \end{aligned}$$

of degree g in the usual way:

$$\begin{aligned} \tau \mapsto \gamma \cdot \tau =(a\tau +b)(c\tau +d)^{-1} \qquad \text { for } \gamma =\left( \begin{matrix} a &{} b \\ c &{} d \\ \end{matrix} \right) \in \Gamma _g. \end{aligned}$$

The moduli space ${\mathcal {A}}_{g}$ carries a rank g vector bundle, the Hodge bundle ${\mathbb {E}}$. The induced bundle on ${\mathcal {A}}_{g}({\mathbb {C}})$ corresponds to the factor of automorphy

$$\begin{aligned} j(\gamma ,\tau )=c\tau +d \, . \end{aligned}$$

For an irreducible representation $\rho$ of ${\mathrm {GL}({g},{\mathbb {C}})}$ of highest weight $(\rho _1, \ldots , \rho _g)$ with $\rho _1 \ge \rho _2 \ge \cdots \ge \rho _g$, we have a corresponding vector bundle ${\mathbb {E}}_{\rho }$ on ${\mathcal {A}}_{g}$; the Hodge bundle corresponds to the standard representation with highest weight $(1,0,\ldots ,0)$ and its determinant has highest weight $(1,\ldots ,1)$. If $\rho : {\mathrm {GL}({g},{\mathbb {C}})} \rightarrow \mathrm{GL}(W)$ is a finite-dimensional complex representation, the induced vector bundle on ${\mathcal {A}}_{g}({\mathbb {C}})$ is defined by the factor of automorphy

$$\begin{aligned} j(\gamma , \tau )= \rho (c\tau +d)\, . \end{aligned}$$

A scalar-valued Siegel modular form of degree $g>1$ and weight k is a holomorphic function $f :\mathfrak {H}_g \rightarrow {\mathbb {C}}$ satisfying

$$\begin{aligned} f(\gamma \cdot \tau )= \det (c\tau +d)^k f(\tau ) \end{aligned}$$

for all $\gamma \in \Gamma _g$, while for $g=1$ we also need a growth condition at infinity. If W is a finite-dimensional complex vector space and $\rho : {\mathrm {GL}({g},{\mathbb {C}})} \rightarrow \mathrm{GL}(W)$ a representation, then a vector-valued Siegel modular form of degree $g>1$ and weight $\rho$ is a holomorphic map $f: \mathfrak {H}_g \rightarrow W$ such that for all $\gamma \in \Gamma _g$

$$\begin{aligned} f(\gamma \cdot \tau )= \rho (c\tau +d) f(\tau ) \, . \end{aligned}$$

Siegel modular forms of weight $\rho$ can be interpreted as sections of the vector bundle ${\mathbb {E}}_{\rho }\,$, and conversely. Sections of the kth power $L^k$ of the determinant line bundle $L=\det ({\mathbb {E}})$ correspond to scalar-valued Siegel modular forms of degree g and weight k. The vector bundle ${\mathbb {E}}$ and the bundles ${\mathbb {E}}_{\rho }$ extend in a canonical way to Faltings–Chai type toroidal compactifications of ${\mathcal {A}}_{g}$ and the Koecher principle says that their sections do so too.

A Siegel modular form F of weight $\rho$ admits a Fourier expansion

$$\begin{aligned} F= \sum _{n} a(n) \, q^n \qquad \text { with } q^n=e^{2\pi i \, \mathrm{Tr} (n \tau )}, \end{aligned}$$

where n runs over the half-integral symmetric positive semi-definite $g\times g$ matrices and $a(n) \in W$. (Half-integral means that 2n is integral with even entries on the diagonal.)

We are interested in the case $g=3$. For an irreducible representation $\rho$ of highest weight $(\rho _1,\rho _2,\rho _3)$ of ${\mathrm {GL}({3},{\mathbb {C}})}$ we denote the weight of the corresponding Siegel modular forms by

$$\begin{aligned} (i,j,k)= w(\rho )= (\rho _1-\rho _2,\rho _2-\rho _3,\rho _3)\, . \end{aligned}$$

(1)

The vector space of Siegel modular forms of weight (i, j, k) on $\Gamma _3$ is denoted by $M_{i,j,k}$. The space of cusp forms is denoted by $S_{i,j,k}$. For scalar-valued Siegel modular forms we often abbreviate the weight (0, 0, k) by k.

Scalar-valued Siegel modular forms of degree 3 form a graded ring:

$$\begin{aligned} R=\oplus _{k=0}^{\infty } M_{0,0,k}\, . \end{aligned}$$

Vector-valued modular forms of degree 3 form a graded module $M=\oplus _{i,j,k} M_{i,j,k}$ over the ring R. The ring R was described by Tsuyumine [39]. He gave 34 generators and the generating function of R. His work used results by Igusa [27] and Shioda [37]. Igusa showed that there is an exact sequence

$$\begin{aligned} 0 \rightarrow \chi _{18} \, R \rightarrow R \, {\buildrel r \over \longrightarrow } \, I(2,8)\, , \end{aligned}$$

where $\chi _{18}$ is a cusp form of weight 18 (see Sect. 3) and I(2, 8) is the ring of invariants of binary octics. The map r is induced by the restriction map to the zero locus in ${\mathcal {A}}_{3}$ of $\chi _{18}$. This locus is the closure of the image of the hyperelliptic locus ${\mathcal H}_3$ under the Torelli map. Shioda determined the ring of invariants of binary octics [37]. In the recent paper [26], Lercier and Ritzenthaler reduce the number of generators of R to 19.

For Siegel modular forms, both the notion of degree and that of genus are used; we tend to use degree for Siegel modular forms and genus for Teichmüller modular forms, but are not strict in this respect.

3 The scalar-valued Siegel modular form $\chi _{18}$

The scalar-valued Siegel modular form $\chi _{18}$ of degree 3 is up to a normalization defined as the product of the 36 even theta constants. It is a cusp form of weight 18. Its Fourier expansion starts with

$$\begin{aligned} \chi _{18}=-\left( \frac{\sigma _3-\sigma _2+\sigma _1-1}{\sigma _3}\right) ^2 \, (\sigma _3^2-2\sigma _3\sigma _1+8\, \sigma _3+\sigma _1^2-4\, \sigma _2) \, q_1^2q_2^2q_3^2 +\cdots \end{aligned}$$

where we use

$$\begin{aligned} q_j=e^{2 \pi i \tau _{jj}}, \quad u=e^{2 \pi i \tau _{12}}, \quad v=e^{2 \pi i \tau _{13}}\quad \text {and}\quad w=e^{2 \pi i \tau _{23}} \, , \end{aligned}$$

(2)

and $\sigma _i$ is the ith elementary symmetric function in u, v, w. Thus $\chi _{18}$ vanishes with multiplicity 2 at infinity. It is well-known that the divisor of $\chi _{18}$ in the toroidal compactification $\tilde{{\mathcal {A}}_{3}}$ is $H+2D$ with H the hyperelliptic locus and D the divisor of $\tilde{\mathcal A}_3$ at infinity:

$$\begin{aligned} \mathrm{div}(\chi _{18})= H +2\, D \, . \end{aligned}$$

(3)

We now give a direct proof that $\chi _{18}$ is up to a scalar the unique cusp form of weight 18 vanishing twice at infinity. This fact also follows from the result of Harris and Morrison [19, Corollary 0.5] on the slope of effective divisors on $\overline{\mathcal {M}}_3$.

Lemma 3.1

The space of Siegel cusp forms of degree 3 and weight 18 vanishing with multiplicity $\ge 2$ on D is generated by $\chi _{18}$.

Proof

The dimension of $S_{0,0,18}$ is 4. In order to construct a basis we consider the Eisenstein series $E_4$ and $E_6$ whose Fourier expansions start with

$$\begin{aligned} E_4= 1+240(q_1+q_2+q_3) +\cdots , \qquad E_6=1-504(q_1+q_2+q_3) + \cdots \end{aligned}$$

and cusp forms $F_{12}$ and $F_{14}$ of weight 12 and 14; note that $\dim S_{0,0,12}=1=\dim S_{0,0,14}$ (cf.  [39]). We normalize these forms such that the following table gives their first Fourier coefficients:

n	$\det (2n)$	$F_{12}$	$F_{14}$
$1_3$	8	164	20
$A_1(1/2)\oplus A_2(1/2)$	6	18	$-6$
$A_3(1/2)$	4	1	1

Here we write the Fourier series of a modular form as before as

$$\begin{aligned} \sum _{n\ge 0} a(n)\, q^n \qquad \text { (using } q^n= e^{2 \pi i \mathrm{Tr}( n\tau )}) \end{aligned}$$

with n running over the half-integral positive semi-definite symmetric matrices; $A_i$ refers to the Gram matrix of the standard root lattice. Recall that for a scalar-valued modular form we have $a(n)=a(u^t n u)$ for all $u \in {\mathrm {GL}({3},{\mathbb {Z}})}$. We look at the induced action of ${\mathrm {GL}({3},{\mathbb {Z}})}$. The set ${\mathcal N}$ of half-integral positive definite symmetric matrices with 1’s on the diagonal contains elements from three distinct orbits: the orbit of $1_3$ (one element), the orbit of $A_1(1/2)\oplus A_2(1/2)$ (sixteen elements), and the orbit of $A_3(1/2)$ (six elements). This implies that we can read off the Fourier coefficient of $q_1q_2q_3$ of $F_{12}$ and $F_{14}$ from the table above. We claim that a basis of $S_{0,0,18}$ is given by

$$\begin{aligned} \chi _{18}, \, \chi _{18}{|T_2}, \, E_4F_{14}, \, E_6F_{12} \end{aligned}$$

with $T_2$ the Hecke operator of the prime 2. We can calculate the Fourier coefficient of $\chi _{18}{|T_2}$ using [10]. We get the following Fourier coefficients:

n	$\det (2n)$	$E_6\, F_{12}$	$E_4\, F_{14}$	$\chi _{18}\vert T_2$	$\chi _{18}$
$1_3$	8	164	20	108	0
$A_1(1/2)\oplus A_2(1/2)$	6	18	$-6$	0	0
$A_3(1/2)$	4	1	1	$-1$	0

This shows that the four cusp forms of weight 18 are linearly independent and that $\chi _{18}$ is up to a nonzero scalar the unique cusp form of weight 18 that vanishes twice at infinity. $\square$

Remark 3.2

In Sect. 11, we will obtain a considerably more general result, with a different method of proof (Theorem 11.6 and Corollary 11.8).

4 The vector-valued Siegel modular form $\chi _{4,0,8}$

The modular form generating $S_{4,0,8}$ is the unique nonzero cusp form (up to scalar) for which $i+2j+3k$ is minimal (equal to 28) (cf.  [38]). This form $\chi _{4,0,8}$ will play a central role in this paper. Its Fourier expansion can be obtained as in  [10] by taking the Schottky form of weight 8 on $\Gamma _4$ and developing it in the normal directions to $\mathfrak {H}_1 \times \mathfrak {H}_3$ in $\mathfrak {H}_4$. The lowest order term gives a nonzero multiple of $\Delta \otimes \chi _{4,0,8}$ in $S_{12}(\Gamma _1) \otimes S_{4,0,8}(\Gamma _3)$. We normalize $\chi _{4,0,8}$ so that its Fourier expansion starts as follows

$$\begin{aligned} \left( \begin{array}{c} 0\\ 0 \\ 0 \\ (v-1)^2(w-1)^2/vw \\ (u-1)(v-1)(w-1)(-1+1/vw+1/uw-1/uv) \\ (u-1)^2(w-1)^2/uw \\ 0\\ (u-1)(v-1)(w-1)(-1+1/vw-1/uw+1/uv) \\ (u-1)(v-1)(w-1)(-1-1/vw+1/uw+1/uv)\\ 0\\ 0 \\ 0 \\ (u-1)^2(v-1)^2/uv \\ 0\\ 0\\ \end{array} \right) q_1q_2q_3+\cdots \, , \end{aligned}$$

where we use the same variables as in (2). This modular form is a lift of $\Delta =\sum \tau (n)q^n \in S_{12}(\Gamma _1)$ and its Hecke eigenvalue at a prime p is by the results of  [3] predicted to be

$$\begin{aligned} \tau (p)\, (p^5+\tau (p)+p^6). \end{aligned}$$

We embed $\mathfrak {H}_2\times \mathfrak {H}_1$ in $\mathfrak {H}_3$ via

$$\begin{aligned} (\tau ^{\prime },\tau ^{\prime \prime }) \mapsto \left( \begin{matrix} \tau ^{\prime } &{} 0 \\ 0 &{} \tau ^{\prime \prime }\\ \end{matrix} \right) \end{aligned}$$

and consider the vanishing of $\chi _{4,0,8}$ along this locus.

Lemma 4.1

Let ${\mathcal I} \subset {\mathcal O}_{\mathfrak {H}_3}$ be the ideal sheaf of $\mathfrak {H}_2\times \mathfrak {H}_1$ in $\mathfrak {H}_3$. The coordinates of the form $\chi _{4,0,8}$ lie in ${\mathcal I}^2$, but not all in ${\mathcal I}^3$.

Proof

Consider the Schottky form $J_8$, a scalar-valued Siegel cusp form of weight 8 and degree 4. If we restrict it to $\mathfrak {H}_2\times \mathfrak {H}_2$ we find as first term in its development in the normal directions to $\mathfrak {H}_2\times \mathfrak {H}_2$ the tensor product $\chi _{6,8} \otimes \chi _{6,8}$, where $\chi _{6,8}$ is a Siegel cusp form of weight (6, 8) in degree 2. If we develop $J_8$ along $\mathfrak {H}_3\times \mathfrak {H}_1$ then we find as first term in the normal directions $\chi _{4,0,8}\otimes \Delta$ with $\Delta$ the elliptic modular cusp form of weight 12. We refer to [10] for these facts. Comparing the degrees of vanishing along $\mathfrak {H}_3\times \mathfrak {H}_1$ and $\mathfrak {H}_2\times \mathfrak {H}_2$, see  [10, Section 5], we see that $\chi _{4,0,8}$ vanishes with multiplicity 2 along $\mathfrak {H}_2 \times \mathfrak {H}_1$.

Another way to see this is by looking at the expansion of $\chi _{4,0,8}$ given in [10, p. 39]. We thus see that the coordinates of $\chi _{4,0,8}$ lie in ${\mathcal I}^2$, but not all in ${\mathcal I}^3$. $\square$

We can trivialize the pullback of ${\mathbb {E}}$ to $\mathfrak {H}_3$ as $\mathfrak {H}_3 \times {\mathbb {C}}^3$. We choose coordinates $z_1,z_2,z_3$ on ${\mathbb {C}}^3$. Since the pullback of ${\mathrm {Sym}}^2({\mathbb {E}})$ can be identified with the cotangent bundle of $\mathfrak {H}_3$ the coordinates $\tau _{ij}$ correspond to $z_i z_j$. In particular, a basis of ${{\mathrm {Sym}}}^4({\mathbb {E}})$ corresponds to the monomials of degree 4 in $z_1,z_2$ and $z_3$. Therefore, we can write $\chi _{4,0,8}$ as

$$\begin{aligned} \chi _{4,0,8}=\sum _I n_I \alpha _I \, z^I \, , \end{aligned}$$

where for a multi-index $I=(i_1,i_2,i_3)$ we inserted a normalization factor $n_I={4!}/{i_1!i_2!i_3!}$. We take the sum over the lexicographically ordered monomials $z^I=z_1^{i_1}z_2^{i_2}z_3^{i_3}$ of degree 4 and $\alpha _I$ is a holomorphic function on $\mathfrak {H}_3$ which we can present as a Fourier series living in ${\mathbb {C}}[u^{\pm 1},v^{\pm 1}, w^{\pm 1}][[q_1,q_2,q_3]]$. The $z^I$ are just dummy variables to indicate the coordinates $n_I \, \alpha _I$ of the vector-valued modular form $\chi _{4,0,8}$.

The symmetric group $\mathfrak {S}_3$ acts on ${\mathbb {C}}^3$ by permuting $z_1,z_2, z_3$ and hence induces an action on $\mathfrak {H}_3$ via

$$\begin{aligned} (\tau _{ij}) \mapsto (\tau _{\sigma (i) \sigma (j)}), \qquad \sigma \in \mathfrak {S}_3\, . \end{aligned}$$

(4)

This action of $\mathfrak {S}_3$ derives from an action of a subgroup of $\Gamma _3$ by taking a $3\times 3$ permutation matrix $a=(a^{-1})^t$ and using $({\begin{matrix} a &{} 0 \\ 0 &{} a \\ \end{matrix}})$. To give the action on the Fourier expansion, note that we have an induced action of $\mathfrak {S}_3$ on $q_1,q_2,q_3$ and on u, v, w.

Lemma 4.2

The action of $\mathfrak {S}_3$ on $\{1,2,3\}$ induces an action on the coordinates $\alpha _I$ of $\chi _{4,0,8}$ given by

$$\begin{aligned} \alpha _I(q_1,q_2,q_3,u,v,w) \mapsto \alpha _{\sigma (I)} \left( q_{\sigma (1)},q_{\sigma (2)}, q_{\sigma (3)},\sigma (u),\sigma (v),\sigma (w) \right) \, . \end{aligned}$$

The action on the fifteen coordinates has one orbit of length 6 and three orbits of length 3. The reader may check that the action of $(23)\in \mathfrak {S}_3$ on ${{\mathrm {Sym}}}^4({\mathbb {E}})$ is given by sending the transpose of $v_I=(v_1,v_2,\ldots ,v_{15})$ to the transpose of

$$\begin{aligned} (v_1,v_3,v_2,v_6,v_5,v_4,v_{10},v_9,v_8,v_7,v_{15},v_{14},v_{13},v_{12},v_{11}) \end{aligned}$$

and that of $(13)\in \mathfrak {S}_3$ is given by

$$\begin{aligned} (v_{15},v_{14},v_{10},v_{13}, v_{9},v_{6},v_{12},v_8,v_5,v_3,v_{11}, v_7,v_4,v_2,v_1). \end{aligned}$$

For $\chi _{4,0,8}$ it thus suffices to give the coordinates $v_1,v_2,v_4$ and $v_5$ only; the other coordinates can then be calculated by using the action of Lemma 4.2.

5 The Fourier–Jacobi expansion of the Schottky form

In the preceding section the form $\chi _{4,0,8}$ was defined by developing the Schottky form of degree 4 and weight 8 along $\mathfrak {H}_1 \times \mathfrak {H}_3$ and observing that the first nonzero term is $\Delta \otimes \chi _{4,0,8}$ in $S_{12}(\Gamma _1) \otimes S_{4,0,8}(\Gamma _3)$. For our application we need to be able to calculate the Fourier expansion of $\chi _{4,0,8}$ quite far and for that we need the first Fourier–Jacobi coefficient of the scalar-valued form $J_8$.

Recall that the Schottky form $J_8$ can be expressed as follows:

$$\begin{aligned} J_8=(R_{00}^2+R_{01}^2+R_{10}^2-2(R_{00}R_{01}+R_{00}R_{10}+R_{01}R_{10}))/2^{16}\, ; \end{aligned}$$

here we write

$$\begin{aligned} R_{\mu \nu }(\tau )=\prod _{\alpha ,\, \beta ,\, \gamma \in \left\{ 0,1\right\} } \vartheta _ { \left[ {\begin{matrix} \mu &{} 0 &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta &{} \gamma \end{matrix}} \right] } (\tau ), \quad (\tau \in \mathcal {H}_4) \end{aligned}$$

where we use the theta series with characteristics $\left[ {\begin{matrix} a \\ b \end{matrix}} \right]$ with a and b row vectors of size $g\in \mathbb {Z}_{\ge 1}$ with coordinates in $\mathbb {Z}$:

$$\begin{aligned} \vartheta _{ \left[ {\begin{matrix} a \\ b \end{matrix}} \right] } (\tau ,z)= \sum _{l\in \mathbb {Z}^g} e^{\pi i (l+a/2) \tau (l+a/2)^t} e^{2 \pi i (l+a/2)\cdot (z+b/2)^t}, \quad \tau \in \mathcal {H}_g, \quad z=(z_1,\ldots ,z_g)\in \mathbb {C}^g. \end{aligned}$$

When $z=0$, we denote $\vartheta _{ \left[ {\begin{matrix} a \\ b \end{matrix}} \right] } (\tau ,0)$ simply by $\vartheta _{ \left[ {\begin{matrix} a \\ b \end{matrix}} \right] } (\tau ).$ We thus need to calculate the first Fourier–Jacobi coefficient of these theta functions. For any $\tau =(\tau _{ij}) \in \mathfrak {H}_4$ we write

$$\begin{aligned} \tau = \left( \begin{matrix} \tau ' &{} z^t \\ z &{} \tau _4\\ \end{matrix}\right) \quad \text {where }\, \tau ' \in \mathfrak {H}_3 \quad \text {and } z=(\tau _{14},\tau _{24},\tau _{34}) \end{aligned}$$

and we abbreviate $\tau _{ii}$ as $\tau _i$. We write the Fourier–Jacobi expansion of $\vartheta _ { \left[ {\begin{matrix} \mu &{} 0 &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta &{} \gamma \end{matrix}} \right] }$ as

$$\begin{aligned} \vartheta _{ \left[ {\begin{matrix} \mu &{} 0 &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta &{} \gamma \end{matrix}} \right] } (\tau )= \sum _{l\in \mathbb {Z}} (-1)^{l\, \gamma } \vartheta _{ \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ',l\, z) e^{\pi i \, l^2 \tau _4}. \end{aligned}$$

(5)

So we get

$$\begin{aligned} \vartheta _{ \left[ {\begin{matrix} \mu &{} 0 &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta &{} \gamma \end{matrix}} \right] } (\tau )= \vartheta _{ \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ')+ (-1)^{\gamma } ( \vartheta _{ \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ',z)+ \vartheta _{ \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ',-z) )\, q_4^{1/2}+O(q_4^2) \, , \end{aligned}$$

where $q_4=e^{2\pi i \tau _4}$. These first terms correspond to $l=0$ and $l=\pm 1$ in formula (5) and the next term is given by $l=\pm 2$ which gives rise to $O(q_4^2)$. Since we are dealing with even characteristics, the latter formula simplifies to

We introduce for $\tau \in \mathfrak {H}_3$ and $z \in {\mathbb {C}}^3$ the notation

$$\begin{aligned} r_{\mu \nu }(\tau )=\prod _{\alpha ,\, \beta \in \left\{ 0,1\right\} } \vartheta _ { \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ), \qquad s_{\mu \nu }(\tau ,z)=\sum _{\alpha ,\, \beta \in \left\{ 0,1\right\} } \left( \frac{ \vartheta _ { \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ,z) }{ \vartheta _ { \left[ {\begin{matrix} \mu &{} 0 &{} 0 \\ \nu &{} \alpha &{} \beta \end{matrix}} \right] } (\tau ) }\right) ^2 \quad \, . \end{aligned}$$

By using formulas (5) and $(5')$ and this notation we get the following result.

Lemma 5.1

The Fourier–Jacobi expansion of $R_{\mu \nu }$ starts with

$$\begin{aligned} R_{\mu \nu }(\tau )= r_{\mu \nu }(\tau ')^2 ( 1-4\, s_{\mu \nu }(\tau ',z)\, q_4 )+O(q_4^2). \end{aligned}$$

Since we are dealing with explicit theta series this allows us to compute the first Fourier–Jacobi coefficient of $J_8\,$. We write the Fourier–Jacobi development as $J_8(\tau )=\sum _{m=0}^{\infty } \varphi _{8,m}(\tau ^{\prime },z) e^{2\pi i m \tau _4}$.

Proposition 5.2

The first nonzero Fourier–Jacobi coefficient of the Schottky form $J_8$ is given by

$$\begin{aligned} \varphi _{8,1}(\tau ',z)=\frac{1}{2^{12}}\, (r_{00}r_{01}r_{10})\, ( -r_{00}s_{00}+ r_{01}s_{01}+ r_{10}s_{10} )(\tau ',z). \end{aligned}$$

Proof

We have

$$\begin{aligned} J_8(\tau )= \frac{1}{2^{16}}(\varphi _{8,0}(\tau ')+\varphi _{8,1}(\tau ',z)\, q_4+O(q_4^2)). \end{aligned}$$

The Fourier–Jacobi coefficient $\varphi _{8,0}$ is a modular form of weight 8 on $\Gamma _3$ but, since $J_8$ is a cusp form, we have $\varphi _{8,0}\in S_8(\Gamma _3)$ and this space is (0), so $\varphi _{8,0}=0$. Computing its expression in terms of the $r_{\mu \nu }$ gives

$$\begin{aligned} \varphi _{8,0}(\tau ')&= (r_{00}^4+r_{01}^4+r_{10}^4-2(r_{00}^2r_{01}^2+r_{00}^2r_{10}^2+r_{01}^2r_{10}^2))(\tau ')\\&=(r_{00}-r_{01}-r_{10})(r_{00}+r_{01}+r_{10})(r_{00}+r_{10}-r_{01})(r_{00}+r_{01}-r_{10})(\tau ')\\&=0 \end{aligned}$$

since we have the following relation

$$\begin{aligned} r_{00}-r_{01}-r_{10}=0. \end{aligned}$$

(6)

The Fourier–Jacobi coefficient $\varphi _{8,1}$ is a Jacobi cusp form of weight 8 and index 1 on $\Gamma _3$ and we have

$$\begin{aligned} \varphi _{8,1}&=8 (- r_{00}^2(r_{00}^2-r_{01}^2-r_{10}^2)s_{00}+ r_{01}^2(r_{00}^2-r_{01}^2+r_{10}^2)s_{01}\\&+ r_{10}^2(r_{00}^2+r_{01}^2-r_{10}^2)s_{10})=16\, r_{00}r_{01}r_{10}(-r_{00}s_{00}+ r_{01}s_{01}+ r_{10}s_{10}) \end{aligned}$$

since the relation (6) among the $r_{\mu \nu }$ implies

$$\begin{aligned}&r_{00}^2-r_{01}^2-r_{10}^2=2\, r_{01}r_{10}, \qquad r_{00}^2-r_{01}^2+r_{10}^2=2\, r_{00}r_{10},\\&\qquad r_{00}^2+r_{01}^2-r_{10}^2=2\, r_{00}r_{01}. \end{aligned}$$

$\square$

Recall now that we write $\chi _{4,0,8}$ as $\sum n_I \alpha _I \, z^I$ where the sum is over the fifteen lexicographically ordered multi-indices of the monomials of degree 4 in the three variables $z_1,z_2,z_3$ and each $\alpha _I$ is a holomorphic function on $\mathfrak {H}_3$.

Proposition 5.3

The coefficient $n_I \alpha _I$ for a multi-index $I=(i_1,i_2,i_3)$ with $i_1+i_2+i_3=4$ of the cusp form $\chi _{4,0,8}$ of weight (4, 0, 8) on $\Gamma _3$ is given by

$$\begin{aligned} \alpha _I{=} \frac{1}{2^{19}\, 3} r_{00}r_{01}r_{10}(\tau ) \left( {-}r_{00}(\tau )\frac{\partial ^4}{\partial _I} s_{00}(\tau ,0) {+}r_{01}(\tau )\frac{\partial ^4}{\partial _I} s_{01}(\tau ,0) {+}r_{10}(\tau )\frac{\partial ^4}{\partial _I} s_{10}(\tau ,0) \right) , \end{aligned}$$

where ${\partial ^4}/{\partial _I}$ stands for $\partial ^4/ \partial {\tau }_{14}^{i_1} \partial {\tau }_{24}^{i_2} \partial {\tau }_{34}^{i_3}\,$ and $n_I= 24/{i_1! \, i_2! \, i_3!}\,$.

Using Lemma 4.2 and the remarks thereafter we see that we need to calculate the coefficients for the multi-indices $I=(1,1,1,1)$, (1, 1, 1, 2), (1, 1, 2, 2) and (1, 1, 2, 3) only.

As a check on our calculations we use the fact that the Hecke eigenvalues of $\chi _{4,0,8}$ for a prime p are by the results of  [3] predicted to be

$$\begin{aligned} \lambda _p= \tau (p) \, (p^5+\tau (p)+p^6) \end{aligned}$$

with $\Delta =\sum \tau (n)q^n \in S_{12}(\Gamma _1)$. By applying the explicit formulas for the Hecke operators for $\Gamma _3$ given in [10] we can calculate the Fourier coefficients a(n) and hence the eigenvalues for small p; for example with the notation used in  [10, p. 45] we have for $n_0=[1\, 1\, 1; 0\, 0\, 0]$ and $p=2$

$$\begin{aligned} a_2(n_0)&= a(2\, n_0)+ 2^{15}{\mathrm {Sym}}^{-4} \left( {\begin{matrix} 1 &{} 0 &{} 1 \\ 0 &{} 1 &{} 1 \\ 0 &{} 0 &{} 2 \end{matrix}} \right) a([1\, 1\, 2; 1\, 2\, 2]) \\&\quad + 2^{9}({\mathrm {Sym}}^{-4} \left( {\begin{matrix} 1 &{} 0 &{} 1 \\ 0 &{} 2 &{} 0 \\ 0 &{} 0 &{} 2 \end{matrix}} \right) a([1\, 2\, 2; 2\, 0\, 0]) \\&\quad +{\mathrm {Sym}}^{-4} \left( {\begin{matrix} 0 &{} 1 &{} 1 \\ 2 &{} 0 &{} 0 \\ 0 &{} 0 &{} 2 \end{matrix}} \right) a([1\, 2\, 2; 0\, 2\, 0])). \end{aligned}$$

The Fourier coefficients needed are the initial one

$$\begin{aligned} a(n_0)=a([1\, 1\, 1; 0\, 0\, 0])&=[0, 0, 0, 4, 0, 4, 0, 0, 0, 0, 0, 0, 4, 0, 0]^t \end{aligned}$$

and

$$\begin{aligned} a([2\, 2\, 2; 0\, 0\, 0])&=[-512, 0, 0, -2816, 0, -2816, 0, 0, 0, 0, -512, 0, -2816, 0, -512]^t \\ a([1\, 1\, 2; 1\, 2\, 2])&=[0, 0, 0, 0, 1, 1, 0, 1, 3, 2, 0, 0, 1, 2, 1]^t \\ a([1\, 2\, 2; 0\, 2\, 0])&=[0, 0, 0, -24, 0, -48, 0, -48, 0, -96, 48, 0, -48, 0, -48]^t \\ a([1\, 2\, 2; 2\, 0\, 0])&=[0, 0, 0, -48, 0, -24, -96, 0, -48, 0, -48, 0, -48, 0, 48]^t; \end{aligned}$$

this gives $\lambda _2(\chi _{4,0,8})=-1728=-24\, (2^5-24+2^6)$ as expected. We similarly checked the agreement for $\lambda _3$ and $\lambda _5\,$.

6 Another expression for $\chi _{4,0,8}$

In this section we derive another expression for $\chi _{4,0,8}$ that connects it to the geometry of curves of genus three. For a principally polarized abelian variety $(X,\Theta )$ of dimension g we consider the space

$$\begin{aligned} \Gamma _{00}(X,\Theta ):=\{ s \in H^0(X, {\mathcal O}(2\Theta )): m_0(s) \ge 4\} \, , \end{aligned}$$

where $m_0$ means the order of vanishing at the origin, see [40]. For a principally polarized abelian variety $(X, \Theta )$ that is not decomposable (that is, not a product of nontrivial lower-dimensional principally polarized abelian subvarieties) it is known that

$$\begin{aligned} \dim \Gamma _{00}(X, \Theta )=2^g-\frac{g(g+1)}{2}-1 \, . \end{aligned}$$

If for $\sigma \in ({\mathbb {Z}}/2{\mathbb {Z}})^g$ we define the second order theta function $\Theta _{\sigma }(\tau ,z)$ on $\mathfrak {H}_g \times {\mathbb {C}}^g$ by

$$\begin{aligned} \Theta _{\sigma }(\tau ,z)= \vartheta _{[{\begin{matrix} \sigma \\ 0 \\ \end{matrix}}]}(2\, \tau ,2\, z) \end{aligned}$$

then for fixed $\tau$ the $\Theta _{\sigma }(\tau ,z)$ with $\sigma \in ({\mathbb {Z}}/2{\mathbb {Z}})^g$ give a basis of $H^0(X_{\tau },{\mathcal O}(2\Theta ))$ with $X_{\tau }={\mathbb {C}}^g/\Lambda _{\tau }$ the abelian variety associated to $\tau$. Now $s=\sum _{\sigma } a_{\sigma } \Theta _{\sigma }(\tau ,z)$ belongs to $\Gamma _{00}$ if and only if

$$\begin{aligned} \sum _{\sigma } a_{\sigma } \Theta _{\sigma }(\tau ,0)=0, \quad \sum _{\sigma } a_{\sigma } \frac{\partial ^2 \Theta _{\sigma }}{\partial z_i \partial z_j}(\tau ,0)=0 \qquad \text { for all } 1\le i, j \le g. \end{aligned}$$

For an indecomposable principally polarized abelian threefold the space $\Gamma _{00}$ has dimension 1. In [16] Frobenius constructed a non-zero element of $\Gamma _{00}$ by defining a function $\varphi (\tau ,z)$ on $\mathfrak {H}_3\times {\mathbb {C}}^3$. A somewhat different construction, leading to the same result, is as follows. Let $\Gamma _3[2,4]$ be the usual congruence subgroup of $\Gamma _3$ of level (2, 4).

The theta functions of the second order $\Theta _{\sigma }(\tau ,0)$ on $\mathfrak {H}_3$ define a morphism

$$\begin{aligned} \mathrm{Th} :{A}_3[2,4]=\Gamma _3[2,4]\backslash \mathfrak {H}_3 \rightarrow {\mathbb {P}}^7\, , \end{aligned}$$

which is an injective map and an immersion along the locus $A^0_3[2,4]$ of indecomposable abelian varieties, see [35, Theorem 3.2]. The closure of the image is a hypersurface given by an explicit equation $F(\ldots ,x_{\sigma },\ldots )=0$ of degree 16 in the coordinates $x_{\sigma }$ of ${\mathbb {P}}^7$, see [40, p. 632]. Then the expression

$$\begin{aligned} \phi (\tau ,z) := \sum _{\sigma } \frac{\partial F}{\partial x_{\sigma }} (\ldots , \Theta _{\sigma }(\tau ,0), \ldots ) \, \Theta _{\sigma }(\tau ,z) \end{aligned}$$

equals up to a non-zero multiplicative constant the function constructed by Frobenius, see [40, p. 624]. In fact, the functions $\varphi (\tau ,z)$ and $\phi (\tau ,z)$ differ multiplicatively by a function that depends only on $\tau$ and is invariant under $\Gamma _3[2,4]$ and descends to a holomorphic function on $A^0_3[2,4]$ and thus is constant since the locus of decomposable abelian threefolds has codimension 2. For a $\tau$ that corresponds to the periods of a smooth projective nonhyperelliptic curve Y the first term of $\phi$ in the Taylor expansion as a function in $z_1,z_2,z_3$ gives the quartic polynomial that defines the canonical image of Y as Frobenius showed [16, p. 37]. In fact, the zero locus of $\phi$ on the abelian variety $\mathrm{Jac}(Y)={\mathbb {C}}^3/\Lambda _{\tau }$ is the surface $\{(x-y): x, y \in Y\}$. Note that for $g=3$ the divisor $\{ (x-y)\in \mathrm{Jac}(Y): x,y \in Y\}$ belongs to $|2\Theta |$. This surface is singular since under the map $Y\times Y \rightarrow \mathrm{Jac}(Y)$ given by $(x,y) \mapsto (x-y)$ the diagonal is contracted and the tangent cone to the singularity is the cone over the canonical image of Y, see [40]. For yet another description of $\phi (\tau ,z)$ we refer to [17, Proposition 1].

It follows from the properties of the theta functions $\Theta _{\sigma }$ and the fact that $\phi$ vanishes to order 4 along the zero section in the universal abelian threefold ${\mathcal {X}}_{3}$ over ${\mathcal {A}}_{3}$ that the degree 4 term in the normal expansion of $\phi$ along the zero section defines a section of ${\mathrm {Sym}}^4({\mathbb {E}})\otimes \det ^8({\mathbb {E}})$.

Proposition 6.1

The function $\phi$ defines a holomorphic Siegel modular cusp form of weight (4, 0, 8) and for a nonhyperelliptic Jacobian $\mathrm{Jac}(Y)$ the naturally associated vector of length 15 gives the coefficients of the quartic defining the canonical image of Y.

Proof

Over the locus of indecomposable abelian threefolds the section $\phi$ vanishes along the zero section of ${\mathcal X}_3\rightarrow {\mathcal {A}}_{3}$ to order 4. Since its zero locus for a smooth curve Y is the surface $\{(x-y): x,y\in Y\}$ it vanishes with exact order 4. Therefore the first term in the development is a non-zero section of ${\mathrm {Sym}}^4({\mathbb {E}}) \otimes \det ^8({\mathbb {E}})$ over this locus. Since the complement has codimension two, the section extends over all of ${\mathcal {A}}_{3}$ to give a holomorphic modular form. It vanishes on the locus of decomposable abelian varieties, hence it defines a cusp form. $\square$

7 Teichmüller modular forms

The Torelli map is a morphism $t=t_g:{\mathcal {M}}_{g} \rightarrow {\mathcal {A}}_{g}$ of Deligne–Mumford stacks obtained by associating to a smooth projective curve of genus g its Jacobian. It can be extended to a morphism from the Deligne–Mumford compactification $\overline{\mathcal {M}}_{g}$ to the toroidal compactification defined by the second Voronoi fan  [31]. In the case of genus 3 this compactification $\tilde{\mathcal A}_3$ is the standard compactification of ${\mathcal {A}}_{3}$ (cf.  [12]).

The moduli space $\overline{\mathcal {M}}_{3}$ carries a vector bundle ${\mathbb {E}}^{\prime }$ of rank 3, the Hodge bundle. It is the pullback under the Torelli morphism of the Hodge bundle ${\mathbb {E}}$ on $\tilde{\mathcal A}_3$. Moreover, for each irreducible representation $\rho$ of $\mathrm{GL}(3)$ of highest weight $(\rho _1,\rho _2,\rho _3)$ we have the vector bundle ${\mathbb {E}}_{\rho }^{\prime }$ on $\overline{\mathcal {M}}_{3}$, again pulled back from $\tilde{\mathcal A}_3$.

A Teichmüller modular form of weight $w(\rho )=(\rho _1-\rho _2,\rho _2-\rho _3,\rho _3)$ and degree 3 is a section of ${\mathbb {E}}_{\rho }^{\prime }$ on ${\mathcal {M}}_{3}$. In Proposition 7.3 below, we show that a Teichmüller modular form extends to $\overline{\mathcal {M}}_{3}$. We denote the space of Teichmüller modular forms of weight $w(\rho )$ by $T_{w(\rho )}$. If $w(\rho )=(0,0,k)$ we are dealing with scalar-valued Teichmüller modular forms (of weight k). Scalar-valued Teichmüller modular forms form a ring $T=\oplus _k T_{0,0,k}$.

Ichikawa showed that there exists a Teichmüller modular form $\chi _{9}$ of weight (0, 0, 9) on $\overline{\mathcal {M}}_{3}$. It is a square root of the pullback of $\chi _{18}$. In particular, $\chi _9$ vanishes on the hyperelliptic locus, with class $h\sim 9\lambda -\delta _0-3\delta _1$. The divisor of $\chi _9$ is therefore known:

$$\begin{aligned} \mathrm{div}(\chi _9)=h+\delta _0+3\, \delta _1\, . \end{aligned}$$

(7)

Ichikawa also showed that the ring T is generated over the ring R of scalar-valued Siegel modular forms of degree 3 by $\chi _9$. See [22,23,24,25].

The map $\mathcal {M}_3 \rightarrow \mathcal {A}_3$ of stacks is a double cover of its image, the locus of indecomposable abelian threefolds. It is ramified along the hyperelliptic locus. The covering involution $\iota$ can be understood by using a fine moduli space ${\mathcal {M}}_{3}[N]$ of curves with level-N structure ($N\ge 3$): it sends the isomorphism class of a pair $(C,\alpha )$ of a curve C and a level structure $\alpha$ to the class of $(C,-\alpha )$. Note that $(A,\alpha )\cong (A,-\alpha )$ for a polarized abelian variety A.

Further, $\iota$ preserves the Hodge bundle $\mathbb {E}'$ and acts as $-1$ on the fiber. Hence the bundle $\mathbb {E}_{\rho }'$ is preserved as well and the action on the fiber is as $(-1)^{\rho _1-\rho _2+\rho _3}=(-1)^{\rho _1+\rho _2+\rho _3}$.

Call $\rho$ odd (resp. even) when $\rho _1+\rho _2+\rho _3$ is odd (resp. even). Let us also call a Teichmüller (or Siegel) modular form of weight $w(\rho )$ odd (resp. even) when $\rho$ is odd (resp. even). It is clear that an odd Siegel modular form is identically zero and that an odd Teichmüller modular form vanishes on the hyperelliptic locus. Further, a Siegel modular form of weight $w(\rho )$ pulls back to a Teichmüller modular form of the same weight. Moreover, we have the following result.

Lemma 7.1

An even Teichmüller modular form is the pullback of a Siegel modular form.

Proof

Note that the complement of the image of ${\mathcal {M}}_{3}$ in ${\mathcal {A}}_{3}$ is ${\mathcal {A}}_{2,1}$, the locus of products of principally polarized abelian varieties, of codimension two in ${\mathcal {A}}_{3}$. An even Teichmüller modular form of weight $w(\rho )$ is invariant under $\iota$ and descends to a section of $\mathbb {E}_{\rho }$ over $t({\mathcal {M}}_{3})$. Since ${\mathcal {A}}_{3}$ is a smooth stack and the codimension of ${\mathcal {A}}_{2,1}$ is two, this section extends to ${\mathcal {A}}_{3}$ and yields the Siegel modular form that pulls back to the Teichmüller modular form. $\square$

We also have the following extension of Ichikawa’s result.

Lemma 7.2

An odd Teichmüller modular form is divisible by $\chi _9$, i.e., it is the product of $\chi _9$ and an even Teichmüller modular form.

Proof

Let f be an odd Teichmüller modular form. Consider $g=f/\chi _9$, a priori a meromorphic Teichmüller modular form with possibly a simple pole along ${\mathcal {H}}_{3}$. Since f vanishes on ${\mathcal {H}}_{3}$, it follows that g is a regular Teichmüller modular form, which clearly is even. (So g is the pullback of a Siegel modular form.) $\square$

A Teichmüller modular form extends holomorphically to $\overline{\mathcal {M}}_{3}$:

Proposition 7.3

Let f be a Teichmüller modular form of weight $w(\rho )$. Then f extends to a holomorphic section of ${\mathbb {E}}^{\prime }_{\rho }$ on $\overline{\mathcal {M}}_{3}$.

Proof

If f is even, then it is the pullback of a Siegel modular form F. By the Koecher principle, F extends to $\tilde{\mathcal {A}}_3$. The pullback of the extension gives the extension of f to $\overline{\mathcal {M}}_{3}$. If f is odd, it is the product of $\chi _9$ and an even Teichmüller modular form. Both extend, so f extends as well. $\square$

Remark 7.4

In the appendix (Sect. 14), we prove that a Teichmüller modular form extends to $\overline{\mathcal {M}}_{g}$ for any $g\ge 3$.

Corollary 7.5

An odd Teichmüller modular form is a cusp form, i.e., it vanishes along $\delta _0$. It also vanishes with multiplicity at least three along $\delta _1$. If it is nonzero and of weight $w(\rho )$, then $\rho _3\ge 9$.

Consider the modular form $\chi _{4,0,8}$. We define a Siegel modular cusp form of weight (4i, 0, 8i) by taking the ith symmetric power of $\chi _{4,0,8}$ and projecting:

$$\begin{aligned} \chi ^{(i)}= \mathrm{pr}_{[4i,0,8i]}({\mathrm {Sym}}^i(\chi _{4,0,8})), \end{aligned}$$

where $\mathrm{pr}_{[4i,0,8i]}$ is the projection of the ith symmetric power of ${\mathrm {Sym}}^4(\mathbb {E})\otimes \det ^8(\mathbb {E})$ onto ${\mathrm {Sym}}^{4i}(\mathbb {E})\otimes \det ^{8i}(\mathbb {E})$.

Lemma 7.6

The form $\chi ^{(i)}$ is not identically zero.

Proof

We use that $\chi _{4,0,8}\otimes \Delta$ is obtained as the first term of the degree 4 scalar-valued Siegel cusp form $J_8$ of weight 8 when developed in the normal directions of $\mathfrak {H}_3\times \mathfrak {H}_1$. By developing $J_8^i$ we find $\chi ^{(i)}\otimes \Delta ^{i}$, since the order of vanishing of $J_8^i$ along $\mathfrak {H}_3\times \mathfrak {H}_1$ is i times the order of $J_8$. $\square$

Note that the order of $\chi ^{(i)}$ along D is at least i. We now give a result on the orders of vanishing of $\chi _{18}$ and $\chi _{4,0,8}$.

Lemma 7.7

The orders of vanishing of $\chi _{18}$ and $\chi _{4,0,8}$ along the hyperelliptic locus H, the divisor D at infinity, and the locus $\mathcal {A}_{2,1}$ of products are given in the following table.

F	$\mathrm{ord}_D$	$\mathrm{ord}_{H}$	$\mathrm{ord}_{\mathcal {A}_{2,1}}$
$\chi _{18}$	2	1	6
$\chi _{4,0,8}$	1	0	2

Proof

The vanishing orders of $\chi _{18}$ along D and H were given in Sect. 3, the order along $\mathcal {A}_{2,1}$ is given in [10, Proposition 5.2]. If the form $\chi ^{(2)}$ would vanish on the hyperelliptic locus, we could divide it by $\chi _{18}$, obtaining a holomorphic Siegel modular form of weight $(8,0,-2)$, which has to be zero. So $\chi ^{(2)}$ does not vanish along H. It follows that $\mathrm{ord}_{H}\chi _{4,0,8}=0$. For the remaining entries, see Sect. 4, in particular Lemma 4.1. $\square$

8 Invariants and concomitants of ternary quartics

Let V be a 3-dimensional vector space over ${\mathbb {C}}$ generated by elements x, y and z. We will denote the space of ternary quartics by ${\mathrm {Sym}}^4(V)$ and we write a ternary quartic as

$$\begin{aligned} f&=a_0\, x^4+ 4\, a_1\, x^3y+4\, a_2\, x^3z+6\, a_3\, x^2y^2+12\, a_4\, x^2yz +6\, a_5\, x^2z^2\\&\quad +4\, a_6\, xy^3 +12\, a_7\, xy^2z+12\, a_8\, xyz^2+4\, a_9 \, xz^3 + a_{10}\, y^4\\&\quad +4\, a_{11} \, y^3z+ 6\, a_{12}\, y^2z^2+4\, a_{13}\, yz^3+a_{14}z^4 \, . \end{aligned}$$

Note that we order the monomials lexicographically and normalize $x^ay^bz^c$ by the factor $4!/a!\, b!\, c!$.

An element $A=(a_{ij})$ of the group $\mathrm{GL}(3,{\mathbb {C}})$ acts on ${\mathrm {Sym}}^4(V)$ via

$$\begin{aligned} f(x,y,z) \mapsto f(a_{11}x+a_{21}y+a_{31}z,a_{12}x+a_{22}y+a_{32}z,a_{13}x+a_{23}y+a_{33}z) \, . \end{aligned}$$

We also consider the induced actions of $\mathrm{PGL}(3,{\mathbb {C}})$ and $\mathrm{SL}(3,{\mathbb {C}})$ on ${\mathcal Q} ={\mathbb {P}}({\mathrm {Sym}}^4(V))$. We take $(a_0,a_1,\ldots ,a_{14})$ as coordinates on ${\mathrm {Sym}}^4(V)$ and ${\mathcal Q}$. The natural ample line bundle ${\mathcal L}= {\mathcal O}_{\mathcal Q}(1)$ on ${\mathcal Q}$ admits an action of $\mathrm{SL}(3,{\mathbb {C}})$ compatible with the action on the projectivized space ${\mathcal Q}$ of ternary quartics, cf. [30].

By definition an invariant is an element of the ring

$$\begin{aligned} I= \oplus _{n\ge 0} \mathrm{H}^0({\mathcal Q}, {\mathcal L}^n)^{\mathrm{SL}(3,{\mathbb {C}})}\, , \end{aligned}$$

that is, it is a polynomial in the coefficients $a_0,\ldots ,a_{14}$ of f which is invariant under the action of $\mathrm{SL}(3,{\mathbb {C}})$. By the work of G. Salmon, T. Shioda and J. Dixmier (see [34, 37] and [11]), see also the unpublished work of T. Ohno [32], we know the structure of the ring of invariants of ternary quartics (see [11, Theorem 3.2]: seven algebraically independent generators of degrees 3, 6, 9, 12, 15, 18, 27 which form a system of parameters of I and six more basic invariants of degrees 9, 12, 15, 18, 21, 21). (See also [13] and the recent paper [26].) The Poincaré series of this graded ring was determined by T. Shioda (see [37]):

$$\begin{aligned} \sum _{n=0}^{\infty } \dim (I_n)\, t^n= \frac{P(t)}{(1-t^3)(1-t^6)(1-t^9)(1-t^{12})(1-t^{15})(1-t^{18})(1-t^{27})}\, , \end{aligned}$$

where $I_n$ denotes the graded piece of I of degree n and P is the palindromic polynomial

$$\begin{aligned}&t^{75}+t^{66}+t^{63}+t^{60}+2\,t^{57}+3\,t^{54}+2\,t^{51} +3\,t^{48}+4\,t^{45}+3\,t^{42}+4\,t^{39} \\&\quad +4\,t^{36}+3\,t^{33} +4\,t^{30}+3\,t^{27}+2\,t^{24}+3\,t^{21}+2\,t^{18}+ t^{15}+t^{12}+t^{9}+1 \, . \end{aligned}$$

This generating series $\sum _{n=0}^{\infty } \dim (I_n) \, t^n$ starts as follows

$$\begin{aligned} 1+t^{3}+2\,t^{6}+4\,t^{9}+7\,t^{12}+11\,t^{15}+19\,t^{18} +29\,t^{21}+44\,t^{24}+67\,t^{27}+\cdots \end{aligned}$$

Before we come to the notion of concomitant, we will fix coordinates on $\wedge ^2 V$. We take

$$\begin{aligned} \hat{x}=y\wedge z, \qquad \hat{y}=z \wedge x, \qquad \hat{z}=x \wedge y\, ; \end{aligned}$$

(8)

then a basis of ${\mathrm {Sym}}^i(V) \otimes {\mathrm {Sym}}^j(\wedge ^2 V)$ is given by the lexicographically ordered monomials of degree i in x, y, z and degree j in $\hat{x}, \hat{y},\hat{z}$.

We recall the notion of concomitants for ternary quartics. Write $W=V^{\vee }$. Consider an equivariant inclusion of ${\mathrm {SL}}({3},{\mathbb {C}})$-representations

$$\begin{aligned} \varphi :A \rightarrow {\mathrm {Sym}}^d ({\mathrm {Sym}}^4(W)), \end{aligned}$$

or equivalently

$$\begin{aligned} {\mathbb {C}} \hookrightarrow {\mathrm {Sym}}^d ({\mathrm {Sym}}^4(W))\otimes A^{\vee }\, , \end{aligned}$$

where $\mathbb {C}$ denotes the trivial representation; write $\Phi$ for the image of 1 under this map. If $A=W[\rho ]$ is the irreducible representation of highest weight $\rho$ we call $\Phi$ a concomitant of type $(d,\rho )$. It can be viewed as a form of degree d in the coordinates $a_i$ of the ternary quartic and of degree $\rho _1-\rho _2$ (resp. $\rho _2-\rho _3$) in x, y, z (resp. $\hat{x},\hat{y},\hat{z}$). In essence, we follow here Chipalkatti’s set-up in [7], but the notation is the dual one and our basis of ${\mathrm {Sym}}^4(V)$ includes the multinomial coefficients. Also, although we work with ${\mathrm {SL}}({3},{\mathbb {C}})$ here, we have used the notation introduced earlier for the irreducible representations of ${\mathrm {GL}({3},{\mathbb {C}})}$, since we will soon need to work with the latter group.

The simplest nontrivial example is the case where $d=1$ and $A={\mathrm {Sym}}^4(W)$ and the inclusion is the identity. Then the corresponding concomitant $\Phi$ is the universal ternary quartic f given in the beginning of this section.

For $d=2$ we have the decomposition

$$\begin{aligned} {\mathrm {Sym}}^2({\mathrm {Sym}}^4(W))=W[8,0,0]+W[6,2,0]+W[4,4,0]\, . \end{aligned}$$

The concomitants corresponding to the first and last isotypical component are $f^2$ and a classical concomitant denoted $\sigma$ by Salmon (see [34, p. 264]). It is a quartic polynomial in $\hat{x},\hat{y},\hat{z}$ with coefficients of degree 2 in the $a_i$, see Sect. 12.

The concomitants form a module C over the ring I of invariants.

9 Quartic curves and moduli

Let ${\mathcal {M}}_{3}$ be the moduli space of curves of genus 3 and ${\mathcal M}_3^\mathrm{nh}= {\mathcal {M}}_{3}-\mathcal {H}_{3}$ the open part of nonhyperelliptic curves. The canonical image of a nonhyperelliptic curve is a quartic curve in ${\mathbb {P}}^2$.

To describe ${\mathcal {M}}_{3}^\mathrm{nh}$ we consider a 3-dimensional vector space V over a field k, say generated by x, y, z. We may view ${\mathrm {Sym}}^4(V)$ as the space of ternary quartics. There is a natural left action of $\mathrm{GL}(V)$ on ${\mathrm {Sym}}^4(V)$: view $f\in {\mathrm {Sym}}^4(V)$ as a homogeneous polynomial map from the dual $V^{\vee }$ to k of degree 4; the action of $A\in \mathrm {GL}(V)$ on f is defined as

$$\begin{aligned} A\cdot f = f \circ A^t. \end{aligned}$$

(Recall: for B a linear map from $V_1$ to $V_2$, the transpose $B^t$ is the natural map from $V_2^{\vee }$ to $V_1^{\vee }$ obtained by composition with B.)

In fact, for our purposes it is better to twist ${\mathrm {Sym}}^4(V)$ by the inverse of the determinant, so we consider

$$\begin{aligned} Q={\mathrm {Sym}}^4(V)\otimes {\det }^{-1}(V), \end{aligned}$$

the irreducible representation of $\mathrm {GL}(V)$ of highest weight $(3,-1,-1)$. The point is that $c\cdot \mathrm{Id}_V$ acts on Q as $c\cdot \mathrm{Id}_Q$. Let $U\subset Q$ be the $\mathrm {GL}(V)$-invariant open subset corresponding to nonsingular projective plane quartics.

Proposition 9.1

There exists an isomorphism of algebraic stacks

$$\begin{aligned}{}[U/\mathrm {GL}(V)] \buildrel {\sim } \over \longrightarrow {\mathcal M}_3^\mathrm{nh} \, . \end{aligned}$$

Proof

The standard construction of the coarse moduli space $M_3^\mathrm{nh}$ is as the quotient of $\mathbb {P}(U)$ by $\mathrm {PGL}(V)$. Since the embedding is canonical, the stack quotient $[\mathbb {P}(U)/\mathrm {PGL}(V)]$ gives the stack ${\mathcal M}_3^\mathrm{nh}$. By the above, the stack quotient of U by the center of $\mathrm {GL}(V)$ may be identified with $\mathbb {P}(U)$. This gives the result. $\square$

Actually, we need an extension of this result to an open subset of Q with complement of codimension at least two. Let $U'\subset Q$ be the $\mathrm {GL}(V)$-invariant open subset of quartics that are either nonsingular or have one singularity, an ordinary double point. Correspondingly, let

$$\begin{aligned} {\mathcal M}_3^{\mathrm{nh},*} =\overline{\mathcal {M}}_3 - \overline{\mathcal {H}}_3 - \Delta _1 - \Delta _{00} \end{aligned}$$

be the partial compactification of ${\mathcal M}_3^\mathrm{nh}$ consisting of nonhyperelliptic stable curves of genus 3 with at most one node, which is nondisconnecting. (Here $\Delta _1$ is the boundary component of curves with a disconnecting node and $\Delta _{00}$ is the closure of the boundary stratum of irreducible curves with exactly two nodes.) Essentially the same proof as above gives:

Proposition 9.2

There exists an isomorphism of algebraic stacks

$$\begin{aligned}{}[U'/\mathrm {GL}(V)] \buildrel {\sim } \over \longrightarrow {\mathcal M}_3^{\mathrm{nh},*} \, . \end{aligned}$$

For $f\in U'$, the elements x, y, z give a basis of the space of global sections of the dualizing sheaf on the quartic in ${\mathbb {P}}^2$ defined by f. This globalizes and we obtain the following result.

Corollary 9.3

The pullback to $U'$ under the quotient map $q:U'\rightarrow {\mathcal M}_3^{\mathrm{nh},*}$ of the Hodge bundle $\mathbb {E}^{\prime }$ is the $\mathrm {GL}(V)$-equivariant bundle $U'\times V$.

Hence the pullback under q of ${\mathrm {Sym}}^4(\mathbb {E}^{\prime })\otimes \det ^{-1}(\mathbb {E}^{\prime })$ is the $\mathrm {GL}(V)$-equivariant bundle $U'\times Q$, which admits the diagonal section, the universal (at most 1-nodal) ternary quartic.

Corollary 9.4

The diagonal section of $U'\times Q$ descends to a canonical section of ${\mathrm {Sym}}^4(\mathbb {E}^{\prime })\otimes \det ^{-1}(\mathbb {E}^{\prime })$ over ${\mathcal M}_3^{\mathrm{nh},*}$.

We revisit the notion of concomitants for ternary quartics. Working with ${\mathrm {GL}({3},{\mathbb {C}})}$ instead of ${\mathrm {SL}}({3},{\mathbb {C}})$, it is best to work with Q or its dual R, as explained above. We consider an equivariant inclusion of ${\mathrm {GL}({3},{\mathbb {C}})}$-representations

$$\begin{aligned} \varphi :A \rightarrow {\mathrm {Sym}}^d (R), \end{aligned}$$

or equivalently

$$\begin{aligned} {\mathbb {C}} \hookrightarrow {\mathrm {Sym}}^d (R)\otimes A^{\vee }\, , \end{aligned}$$

where $\mathbb {C}$ denotes the trivial representation; write $\Phi$ for the image of 1 under this map. If $A=W[\rho ]$ is the irreducible representation of highest weight $\rho$ we call $\Phi$ a concomitant of type $(d,\rho )$. It can be viewed as a form of degree d in the coordinates $a_i$ of the twisted ternary quartic and of degree $\rho _1-\rho _2$ (resp. $\rho _2-\rho _3$, resp. $\rho _3$) in x, y, z (resp. $\hat{x},\hat{y},\hat{z}$, resp. $x\wedge y\wedge z$). Note that $d=\rho _1+\rho _2+\rho _3$, so d is determined by $\rho$ in the ${\mathrm {GL}({3},{\mathbb {C}})}$-setting. Sometimes, we will speak of concomitants of type $\rho$.

E.g., for $d=2$ we have the decomposition

$$\begin{aligned} {\mathrm {Sym}}^2(R)=W[6,-2,-2]+W[4,0,-2]+W[2,2,-2]\, ; \end{aligned}$$

compared with the earlier decomposition of ${\mathrm {Sym}}^2({\mathrm {Sym}}^4(W))$, in each summand all entries have been lowered by $d=2$.

10 Invariants, concomitants and modular forms

We consider a scalar-valued Teichmüller modular form F of weight k, that is, a section of ${\det ^k({\mathbb {E}}')}$ on $\overline{\mathcal {M}}_3$, cf. Proposition 7.3. By Corollary 9.3, the pullback of $\det ^k({\mathbb {E}}')$ to $U'$ under q is $\det ^k(V)$. The pullback of F is a section of $\det ^k(V)$ over $U'$ that extends to a section of $\det ^k(V)$ over Q (since the complement of $U'$ has codimension two). The corresponding irreducible representation of $\mathrm {GL}(V)$ has highest weight (k, k, k); it occurs in ${\mathrm {Sym}}^{3k}(Q)$. Thus F defines an invariant $\iota _F \in I$ of degree 3k (for $\mathrm{SL}(3,{\mathbb {C}})$).

Conversely, if we have a homogeneous invariant, necessarily of degree 3k with $k\in \mathbb {N}$, it gives a section of $\det ^{k}(V)$. It descends to a holomorphic section of $\det ^k(\mathbb {E}')$ over the open subset $\mathcal {M}_3^\mathrm{nh}$ that extends to a meromorphic section over ${\mathcal {M}}_{3}$. Since $\chi _9$ vanishes exactly once on the hyperelliptic locus, we conclude that we get injections

$$\begin{aligned} {T} \longrightarrow I \longrightarrow {T}_{\chi _{_{9}}}\, \end{aligned}$$

(9)

with T the graded ring of scalar-valued Teichmüller modular forms and ${T}_{\chi _{_{9}}}$ the ring obtained by inverting $\chi _9$. Under the first map above, the modular form $\chi _9$ maps to (a nonzero multiple of) the discriminant, an invariant of degree 27. (In the recent paper [26], Lercier and Ritzenthaler study the relation between scalar-valued Siegel modular forms and invariants of ternary quartics in detail. Their approach differs from ours.)

We now generalize this to concomitants for ${\mathrm {GL}({3},{\mathbb {C}})}$. A concomitant $\Phi$ of type $(d, \rho )$ gives an equivariant section of $V_{\rho }$. It descends to a holomorphic section of ${\mathbb {E}}^{\prime }_{\rho }$ on ${\mathcal M}_3^\mathrm{nh}$ that extends to a meromorphic section $\gamma (\Phi )$ on ${\mathcal {M}}_{3}$. After multiplication with the rth power of $\chi _9$ for r large enough, it becomes a holomorphic section of ${\mathbb {E}}^{'}_{\rho }\otimes \det ^{9r}({\mathbb {E}}')$ (as we shall see $r=d$ suffices). Conversely, if F is a Teichmüller modular form of weight $\rho$, that is, a section of ${\mathbb {E}}^{\prime }_{\rho }$, then by pulling it back from ${\mathcal M}_3^{\mathrm{nh},*}$ to $U'$ we get an equivariant section of $V_{\rho }$ that extends to Q: a concomitant $\beta (F)$ of type $\rho$.

We thus find a commutative diagram

where $\Sigma =\oplus _{\rho } T_{\rho }$ (with $\rho$ running over the irreducible representations of $\mathrm{GL}(3,\mathbb {C})$) is the T-module of vector-valued Teichmüller modular forms, $\Sigma _{\chi _9}$ denotes the module obtained by inverting $\chi _9$ and C is the I-module of concomitants. A modular form F of weight (i, j, k) is sent to a concomitant of type

$$\begin{aligned} (d,\rho )=(i+2j+3k,[i+j+k,j+k,k]) \end{aligned}$$

and a concomitant of type $(d,\rho )$ to a (meromorphic) form of weight

$$\begin{aligned} (\rho _1-\rho _2,\rho _2-\rho _3, \rho _3). \end{aligned}$$

Note that the parity of the form equals the parity of the degree d of the concomitant.

A basic observation is the following.

Proposition 10.1

The image of the universal ternary quartic $f \in C$ under $\gamma$ equals up to a nonzero multiplicative constant the meromorphic Teichmüller modular form

$$\begin{aligned} \chi _{4,0,-1}=\chi _{4,0,8}/\chi _9\, . \end{aligned}$$

Proof

The tautological f defines an element $\gamma (f)$ of $\Sigma _{\chi _9}$, a meromorphic Teichmüller modular form of weight $(4,0,-1)$. In order to identify it we consider the Siegel (or Teichmüller) modular form $\chi _{4,0,8}$. By Proposition 6.1 we know that on the open set ${\mathcal M}_3^\mathrm{nh}$ the two sections $\gamma (f)$ and $\chi _{4,0,8}$ differ by a nowhere vanishing holomorphic function. Therefore they differ by a power of $\chi _9$. Since the weights are $(4,0,-1)$ and (4, 0, 8), it follows that up to a nonzero multiplicative constant we have $\gamma (f)=\chi _{4,0,8}/\chi _9$. $\square$

Then the map $\gamma : C \rightarrow \Sigma _{\chi _{9}}$ can be written as a substitution

$$\begin{aligned} c \mapsto c \circ \chi _{4,0,-1} \end{aligned}$$

in the following sense. Recall that we write our ternary quartic as $f=\sum _I n_I a_I x^I$ where I runs over the indices $(i_1,i_2,i_3)$ with $i_1+i_2+i_3=4$, $n_I= 4!/(i_1!i_2!i_3!)$ and $x^I=x^{i_1}y^{i_2}z^{i_3}$. We write $\chi _{4,0,8}$ in a similar way as a vector

$$\begin{aligned} \chi _{4,0,8}= \sum _{I} n_I\alpha _I x^I\, . \end{aligned}$$

(11)

Then each $\alpha _I$ is a holomorphic function on $\mathfrak {H}_3$ or also on Teichmüller space ${\mathcal T}_3$. The map $\gamma$ is given by substituting in a concomitant the meromorphic functions $\alpha _I/\chi _9$ on ${\mathcal T}_3$ for the $a_I$.

Sometimes we prefer to work with holomorphic modular forms. Then we don’t use the map $\gamma$, but a slightly adapted map $\gamma '$ that maps the tautological f to $\gamma (f)\chi _9$ and that is defined by substituting in a concomitant c of degree d the $\alpha _I$ for the $a_I$. The result is a holomorphic vector-valued modular form

$$\begin{aligned} \gamma ^{\prime }(c)=\gamma (c)\chi _9^d=c\circ \chi _{4,0,8}. \end{aligned}$$

For a concomitant of type $(d,\rho )$ it is a form of weight $(\rho _1-\rho _2,\rho _2-\rho _3,\rho _3+9d)$, i.e., a section of ${\mathbb {E}}_{\rho } \otimes \det ^{9d}{\mathbb {E}}$. We can calculate the Fourier expansion of $\gamma '(c)$ from the Fourier expansion of $\chi _{4,0,8}$.

Depending on the parity of $\rho$, the form $\gamma (c)$ is a (meromorphic) Siegel or Teichmüller modular form, but $\gamma '(c)$ is (the pullback of) a holomorphic Siegel modular form. If it vanishes along the hyperelliptic locus, we can divide it by $\chi _{18}\,$. Note that the order of divisibility by $\chi _{18}$ is bounded above by 1/2 of the order along $\delta _0$ and also by 1/6 of the order along $\delta _1$.

11 The order along the locus of double conics

Inside the space ${\mathbb {P}}^{14}$ of ternary quartics there is the locus DC of double conics. The order of a concomitant c of ternary quartics along the locus DC is determined as the order in the parameter $t\in \mathbb {C}$ of the evaluation of c on the quartic $tf+g^2$, where f is a general ternary quartic and g a general ternary conic.

Proposition 11.1

Let c be a concomitant of degree d vanishing with order v along the locus of double conics. Then the order of $\gamma (c)$ along the hyperelliptic locus ${\mathcal {H}}_{3}\subset {\mathcal {M}}_{3}$ equals $2v-d$. Hence the order of $\gamma '(c)$ along ${\mathcal {H}}_{3}$ equals 2v and the corresponding Siegel modular form has order v along $H\subset {\mathcal {A}}_{3}$.

Proof

The proof is completely analogous to that of Theorem 1 in [9], so we only mention the necessary changes. The meromorphic form $\gamma (c)$ is obtained by substituting $\chi _{4,0,-1}$ in c and $\chi _{4,0,-1}$ has a simple pole along ${\mathcal {H}}_{3}$, so the result holds for the universal f (with $d=1$ and $v=0$). After multiplying with f or $f^2$ if necessary, we may assume that d is divisible by three, equal to 3e. We now let A be the invariant of degree 3:

$$\begin{aligned} A= a_0a_{10}a_{14}-4a_0a_{11}a_{13}+3a_0a_{12}^2-4a_1a_6a_{14}+{12}a_1a_7a_{13}-{12}a_1a_8a_{12} +\cdots . \end{aligned}$$

One easily checks that A doesn’t vanish along DC, so $\gamma (A)$ has a pole of order 3 along ${\mathcal {H}}_{3}$. We write $\gamma (c)$ as $\gamma (c/A^e)\cdot \gamma (A)^e$, where $c/A^e$ is a meromorphic concomitant of degree 0. Its components are meromorphic functions that descend to the components of $\gamma (c/A^e)$. Now recall that the coarse moduli space $M_3$ may be constructed by blowing up the locus of double conics in the projective space of ternary quartics, deleting the proper transform of the discriminant hypersurface, and taking the quotient by ${\mathrm {PGL}({3},{\mathbb {C}})}$. The order of $\gamma (c/A^e)$ along ${\mathcal {H}}_{3}$ equals twice the order of $c/A^e$ along the exceptional divisor; this reflects the difference between the stack ${\mathcal {M}}_{3}$ and $M_3$. On the other hand, the order along the exceptional divisor equals the order along DC. This proves the first result. The other results are immediate consequences. $\square$

Corollary 11.2

Same hypothesis. If d is even, then the order along H of the meromorphic Siegel modular form corresponding to $\gamma (c)$ equals $v-d/2$. If d is odd, then the order along H of the meromorphic Siegel modular form corresponding to $\chi _9\gamma (c)$ equals $v-(d-1)/2$.

Corollary 11.3

Let F be a Teichmüller modular form of type $\rho$ that has order m along ${\mathcal {H}}_{3}$. Then $\beta (F)$ is a concomitant of type $\rho$, so of degree $d=\rho _1+\rho _2+\rho _3$, that has order $\tfrac{1}{2}(m+d)$ along the locus of double conics. In particular, m and d have the same parity (which equals that of F).

Proof

This follows from the proposition, since the composition $\gamma \circ \beta$ of the maps in diagram (10) above is the identity map onto $\Sigma \subset \Sigma _{\chi _9}$. That the parities agree was already known: it is obvious in the even case and follows in the odd case from the fact that $\chi _9$ has multiplicity 1 along ${\mathcal {H}}_{3}$. $\square$

Example 11.4

Let $\Xi$ be the discriminant of ternary quartics, an invariant of degree 27. Then $\gamma (\Xi )=\chi _9$ and $\chi _9\gamma (\Xi )=\chi _{18}$. We know that $\chi _{18}$ has order 1 along H and find that $v=14$, in agreement with a result of Aluffi–Cukierman [1].

Notation 11.5

Let $m\ge 0$ be an integer. We denote by $C_{d,\rho }(-m\, DC)$ the vector space of concomitants of type $(d,\rho )$ that have order $\ge m$ along the locus DC of double conics.

We denote by $S_{i,j,k}^m$ the vector space of Siegel modular forms of type (i, j, k) that have order $\ge m$ along the boundary divisor D. So $S_{i,j,k}^0=M_{i,j,k}$ and $S_{i,j,k}^1=S_{i,j,k}$.

These notations also make sense when $m<0$, but in most cases this doesn’t give anything new, since we consider regular or holomorphic concomitants and modular forms here. However, see Corollary 11.8 below.

Theorem 11.6

Notation as above. There exists an isomorphism

$$\begin{aligned} \varphi :C_{d,\rho }(-mDC) \buildrel \sim \over \longrightarrow S_{\rho _1-\rho _2, \rho _2-\rho _3,\rho _3+9n}^n \, , \qquad c \mapsto \gamma (c)\chi _9^n \, , \end{aligned}$$

where $n=d-2m$.

Proof

Let $c\in C_{d,\rho }(-mDC)$. By Proposition 11.1, the order along ${\mathcal {H}}_{3}$ of $\gamma (c)$ is at least $2m-d$. So $\varphi (c)$ is regular along ${\mathcal {H}}_{3}$. Since d and n have the same parity, $\varphi (c)$ is (the pullback of) a Siegel modular form. It follows immediately that $\varphi (c)\in S_{\rho _1-\rho _2, \rho _2-\rho _3,\rho _3+9n}^n \,$. Now $\varphi$ is certainly injective, but also surjective: take $F\in S_{\rho _1-\rho _2, \rho _2-\rho _3,\rho _3+9n}^n\,$, then $\beta (F)/\Xi ^n$ is a meromorphic concomitant that has order at least m along DC and order at least 0 along the locus $\Xi =0$ of singular quartics. So it is in fact a regular concomitant and an element of $C_{d,\rho }(-mDC)$. $\square$

Corollary 11.7

If $i+2j+4k<36n$, then $S_{i,j,k}^n=0$.

Proof

Trivially, $C_{d,\rho }=0$ when $\rho _3<-d$. Since $i=\rho _1-\rho _2$, $j=\rho _2-\rho _3$, and $k=\rho _3+9n$, we get $d=i+2j+3k-27n$ and obtain the result. $\square$

We also obtain the following generalization of the main result of Sect. 3; this can also be deduced from the result of Harris–Morrison [19, Corollary 0.5] on slopes of effective divisors on $\overline{\mathcal M}_3$.

Corollary 11.8

Let k be a positive integer. The space $S_{0,0,18k}^{2k}$ of cusp forms of weight 18k of order at least 2k along D is generated by $\chi _{18}^k$.

Proof

Take $d=0$, so $\rho =(0,0,0)$, and $m=-k$, so $n=2k$. $\square$

We illustrate Theorem 11.6 with a number of examples.

Example 11.9

• (1) Let $d=1$ and $m=0$. Then $\rho =(3,-1,-1)$ and $n=1$. We get an identification between the space of concomitants of degree 1, so of type $(3,-1,-1)$ and the space of cusp forms $S_{4,0,8}$ : the universal ternary quartic f is mapped to $\chi _{4,0,8}$. We also see that $S_{2,1,8}$, $S_{0,2,8}$, and $S_{1,0,9}$ all vanish (which was known).

• (2) Let $d=2$ and take $\rho =(4,0,-2)$ and $m=0$, so $n=2$. We get an identification between the space of concomitants of degree 2 and type $(4,0,-2)$ and the space $S_{4,2,16}^2=M_{4,2,16}(-2D)$ of Siegel modular forms vanishing at least twice at the cusp. Hence the latter space is one-dimensional. Analogously, $S_{8,0,16}^2$ and $S_{0,4,16}^2$ are also one-dimensional, whereas the vector spaces $S_{i,j,k+16}^2$ with $i+2j+3k=8$ are zero in all (seven) other cases.

• (3) Let $d=3$ and take $\rho =(6,0,-3)$ and $m=1$, so $n=1$. We get an isomorphism $C_{3,(6,0,-3)}(-DC) \cong S_{6,3,6}$. The representation $W[6,0,-3]$ occurs with multiplicity 1 in ${\mathrm {Sym}}^3(R)$ and the associated concomitant vanishes on the locus of double conics. Correspondingly, $\dim S_{6,3,6}=1$. Completely analogously, $\rho =(4,1,-2)$ yields $\dim S_{3,3,7}=1$.

  There are seven other nonzero concomitants of degree 3; they don’t vanish along DC. Taking $m=0$, we find nine one-dimensional spaces of cusp forms with order at least three along D (two from the cusp forms just mentioned). Finally, seventeen spaces $S_{i,j,k+6}$ and ten spaces $S^3_{i,j,k+24}$ with $i+2j+3k=12$ vanish.

• (4) Take $\rho =(5,4,-4)$, so $d=5$, and $m=2$, so $n=1$. There is an isomorphism $C_{5,(10,9,1)}(-2DC) \cong S_{1,8,5}$. The space $S_{1,8,5}$ is one-dimensional and a generator of it has been constructed in the paper of Ibukiyama and Takemori [21]. There is a unique concomitant of type $\rho$ and indeed it vanishes to order 2 along DC. We give more details in Sect. 12.3.

• (5) For $d=27$ and $\rho =(9,9,9)$, take $m=14$, so $n=-1$. On the right, the space $S_{0,0,0}^{-1}$ is just $\mathbb {C}$. Hence the space of invariants of degree 27 with order at least 14 along DC is one-dimensional; it is generated by the discriminant $\Xi$, which is mapped to 1, since $\gamma (\Xi )=\chi _9$.

12 Constructing modular forms from concomitants

12.1 Degree 1 and 2

In this section, we will use the ${\mathrm {SL}}({3},{\mathbb {C}})$-notation for concomitants as in Sect. 8, mainly because it is easier to read. As long as the degree d is known, this should not lead to confusion. As mentioned in Sect. 9, the ${\mathrm {GL}({3},{\mathbb {C}})}$-notation for a concomitant of degree d is obtained by lowering the three entries by d.

There is one concomitant of degree $d=1$, the universal ternary quartic. As we have seen, the image under the map $\gamma '$ of the basic concomitant f is $\chi _{4,0,8}\,$.

For $d=2$ we have the decomposition

$$\begin{aligned} {\mathrm {Sym}}^2({\mathrm {Sym}}^4(W))=W[8,0,0]+W[6,2,0]+W[4,4,0]\, . \end{aligned}$$

The last isotypical component defines a covariant denoted $\sigma$ by Salmon (see [34, p. 264]). It describes the curve in the dual ${\mathbb {P}}^2$ of lines intersecting the curve defined by f equianharmonically. Its first terms are given by (using $u_0=\hat{x}, u_1=\hat{y}, u_2=\hat{z}$)

$$\begin{aligned}&\sigma = (a_{10}a_{14}-4a_{11}a_{13}+3a_{12}^2)u_0^4+ (4a_9a_{11}-12a_8a_{12}+12a_7a_{13}-4a_6a_{14})u_0^3u_1\\&\quad +(-4a_9a_{10}+12a_8a_{11}-12a_7a_{12}+4a_6a_{13})u_0^3u_2\\&\quad +(6a_5a_{12}-12a_4a_{13}+6a_3a_{14}-12a_7a_9+12a_8^2)u_0^2u_1^2+\cdots \\ \end{aligned}$$

Under the map $\gamma '$ the three concomitants corresponding to the three terms in the decomposition of ${\mathrm {Sym}}^2({\mathrm {Sym}}^4(W))$ give rise to Siegel modular forms of weights $\chi _{8,0,16}$, $\chi _{4,2,16}$ and $\chi _{0,4,16}$ respectively. The form $\chi _{8,0,16}$ is the (symmetric) square of $\chi _{4,0,8}$.

Since $\chi _{4,0,8}$ vanishes with multiplicity 1 along D, these three modular forms vanish with multiplicity $\ge 2$ along D. In order to give the beginning of the Fourier expansion of $\chi _{0,4,16}\,$, we notice that it suffices to give the coordinates $v_1,v_2,v_4$ and $v_5\,$, since under the action of $\mathfrak {S}_3$ given in (4) the 15 coordinates satisfy the same relations as those of $\chi _{4,0,8}$. We have for the entries of the coefficient of $q_1^2q_2^2q_3^2$ :

$$\begin{aligned} v_1= & {} 3\, (u-1)^4(v-1)^4/u^2v^2, \qquad v_2= 12\, (u-1)^3(v-1)^3(w-1)\\&\quad (uvw+u-v-w)/u^2v^2w,\\ v_4= & {} 6\, (u-1)^2(v-1)^2(w-1)^2 \, (2\sigma _3^2-4\sigma _3\sigma _1-2\sigma _3+2\sigma _1^2-8\sigma _2\\&+9(u^2+1)vw) /\sigma _3^2 \end{aligned}$$

and

$$\begin{aligned} v_5= -12\, (u-1)^2(v-1)^2(w^2-1) \, (\sigma _3^2-2\sigma _3\sigma _1+8\sigma _3+\sigma _1^2-4\sigma _2)/\sigma _3^2\, , \end{aligned}$$

where as before $\sigma _i$ is the elementary symmetric function of degree i in u, v, w.

Summing up, from Theorem 11.6 and Example 11.9, we have isomorphisms

$$\begin{aligned} C_{2,(8,0,0)} \cong S_{8,0,16}^2, \qquad C_{2,(6,2,0)} \cong S_{4,2,16}^2, \qquad C_{2,(4,4,0)} \cong S_{0,4,16}^2, \end{aligned}$$

(12)

given by $c \longleftrightarrow (c\circ \chi _{4,0,-1}) \chi _{18}$.

Note that the dimensions of $S_{8,0,16}$ (resp. $S_{4,2,16}$, $S_{0,4,16}$) are 26 (resp. 25, 12) (cf.  [38]).

12.2 Degree 3

For $d=3$ we have the multiplicity-free decomposition

$$\begin{aligned} {\mathrm {Sym}}^3({\mathrm {Sym}}^4(W))&=W[12,0,0]+W[10,2,0]\\&\quad +W[9,3,0]+W[8,4,0]+W[8,2,2]\\&\quad +W[7,4,1]+W[6,6,0]+W[6,4,2]+W[4,4,4]\, . \end{aligned}$$

The covariant that corresponds to W[12, 0, 0] is given by the form of degree 12 that is the third power of f, while W[8, 2, 2] corresponds to the covariant given by the Hessian of f. The contravariant that corresponds to W[6, 6, 0] is given by the dual sextic of lines intersecting f in a 4-tuple with j-invariant 1728. Finally, W[4, 4, 4] corresponds to the invariant of degree 3.

By the map $c \mapsto (c \circ \chi _{4,0,-1})\chi _9^3$ these concomitants yield modular forms of weights

$$\begin{aligned}&(12,0,24), (8,2,24), (6,3,24), (4,4,24), (6,0,26), (3,3,25),\\&\quad (0,6,24), (2,2,26), (0,0,28). \end{aligned}$$

In fact, we have isomorphisms between these spaces of concomitants and spaces of cusp forms vanishing with multiplicity $\ge 3$ along D:

$$\begin{aligned} C_{3,(a,b,c)} \cong S_{a-b,b-c,c+24}^3 \qquad \mathrm{via} \qquad c \mapsto (c \circ \chi _{4,0,-1})\chi _9^3\, , \end{aligned}$$

and also

$$\begin{aligned} C_{3,(a,b,c)}(-DC) \cong S_{a-b,b-c,c+6} \qquad \mathrm{via} \qquad c \mapsto (c \circ \chi _{4,0,-1})\chi _9\, . \end{aligned}$$

Only for $\rho =[9,3,0]$ and [7, 4, 1] we get nonzero spaces $S_{a-b,b-c,c+6}$ ; namely, $\dim S_{6,3,6}=\dim S_{3,3,7}=1$, in perfect agreement with the fact that the concomitants of degree 3 that vanish on the locus of double conics are those corresponding to [9, 3, 0] and [7, 4, 1].

As an example, we take a closer look at the case of the concomitant c provided by W[7, 4, 1] in ${\mathrm {Sym}}^3({\mathrm {Sym}}^4(W))$. The modular form $\gamma '(c)$ is divisible by $\chi _{18}$ and yields a cusp form $\chi _{3,3,7}=\sum a(N) q^N$ of weight (3, 3, 7) on $\Gamma _3$. We can calculate some of its Fourier coefficients: we find the irreducible representation of highest weight (13, 10, 7) inside ${\mathrm {Sym}}^{3}(W)\otimes {\mathrm {Sym}}^3(\wedge ^2(W))\otimes {\det }^7(W)$ with W the standard representation of $\mathrm{GL}(3)$. For example, using the shorthand $N=[n_{11},n_{22},n_{33};2n_{12},2n_{13},2n_{23}]$ for the half-integral matrix $N=(n_{ij})$, we have for $N_1=[1,1,1;1,1,1]$

$$\begin{aligned} a(N_1)=[6,-20,-20,0,40,0,0,0,0,0,0,-27,-30,15,90, \ldots ]^t \end{aligned}$$

and for $a(2N_1)$ we find

$$\begin{aligned} 8\, [1050,-2380,-2380,720,3320,720,0,0,0,0,1560,-4725,-2490,-15,1430,\ldots ]^t \end{aligned}$$

and for a([3, 2, 2, 4, 4, 2]) we find the vector

$$\begin{aligned} 4 [ -480, 860, 860, -430, -1060, -430, 0, 315, 315, 0, -1140, 2160, 2040, -1275, -2610, \ldots ]^t \end{aligned}$$

and then the formula for the Hecke operator for $(i,j,k)=(3,3,7)$

$$\begin{aligned} a_2(N_1)=&\, a(2\, N_1) + 2^{i+2j+k-3} {\mathrm {Sym}}^{-i} \left( {\begin{matrix} 1 &{} 1 &{} 1 \\ 0 &{} 2 &{} 0 \\ 0 &{} 0 &{} 2 \end{matrix}} \right) {\mathrm {Sym}}^{-j}(\wedge ^2( \left( {\begin{matrix} 1 &{} 1 &{} 1 \\ 0 &{} 2 &{} 0 \\ 0 &{} 0 &{} 2 \end{matrix}} \right) ) a([3\, 2\, 2; 4\, 4\, 2]) \end{aligned}$$

gives the eigenvalue $\lambda _2=1080$. The eigenvalues can be checked against the data provided by [4].

The invariant $\iota$ of type (3, (4, 4, 4)) is given by

$$\begin{aligned}&a_{0}\, a_{10}\, a_{14} {-}4\, a_{0}\, a_{11}\, a_{13} {+}3\,a_{0}\, a_{12}^2{+} 4\, a_{1}\, a_{11}\, a_{9} {-}12\, a_{1}\, a_{12}\, a_{8} {+}12\, a_{1}\, a_{13}\, a_{7} {-}4\, a_{1}\,a_{14}\, a_{6}\\&\quad -4\, a_{10}\, a_{2}\, a_{9} {+}3\, a_{10}\,a_{5}^2 {+}12\, a_{11}\, a_{2}\, a_{8} {-}12\, a_{11}\, a_{4}\, a_{5} -12\, a_{12}\, a_{2}\, a_{7} {+}6\, a_{12}\, a_{3}\,a_{5} {+}12\, a_{12}\, a_{4}^2\\&\quad +4\, a_{13}\, a_{2}\, a_{6} -12\, a_{13}\, a_{3}\, a_{4} +3\, a_{14}\, a_{3}^2 -12\,a_{3}\, a_{7}\, a_{9} +12\, a_{3}\, a_{8}^2 +12\, a_{4}\, a_{6}\, a_{9}\\&\quad -12\, a_{4}\, a_{7}\, a_{8} -12\, a_{5}\,a_{6}\, a_{8} +12\, a_{5}\, a_{7}^2 \end{aligned}$$

in the coefficients $a_i$ of the ternary quartic f. It defines a cusp form $\chi _{28}$ of weight 28 vanishing with multiplicity 3 at D. Using the pairing induced by the pairing of W and $\wedge ^2 W$, we get $\langle f, \sigma \rangle =\iota$ or in terms of modular forms

$$\begin{aligned} \langle \chi _{4,0,8}, \chi _{0,4,16} \rangle = \chi _{28}\, , \end{aligned}$$

where the pairing of $\chi _{4,0,8}=\sum _I n_I \alpha _I x^I$ and $\chi _{0,4,16}=\sum _I \beta _I \hat{x}^I$ is $\sum _I n_I \alpha _I \beta _I$. The Fourier expansion of $\chi _{28}$ starts with

$$\begin{aligned} F(\tau )=(\frac{(u-1)^2(v-1)^2(w-1)^2}{144\,u^3v^3w^3} (u^4v^4w^4+u^4v^4w^3+\cdots )) q_1^3q_2^3q_3^3+\cdots . \end{aligned}$$

This Siegel modular form vanishes with order 3 at infinity and along ${\mathcal A}_{2,1}$ with order 8. Indeed, expanding F as a Taylor series along $\mathfrak {H}_2 \times \mathfrak {H}_1$ as done in [10, Proposition 2.1] we get as first term a tensor product $F' \otimes F^{\prime \prime }\in S_{8,28}(\Gamma _2)\otimes S_{36}(\Gamma _1)$ because $F^{\prime \prime }$ will be a cusp form on $\Gamma _1$ vanishing with multiplicity 3 at the cusp and the first such form is $\Delta ^3$. By looking around $\mathfrak {H}_1^3$ we see that $F^{\prime }$ vanishes along $\mathfrak {H}_1^2$ with multiplicity 4; dividing $F^{\prime }$ by $\chi _{10}^2$ we get a cusp form of weight (8, 8) on $\Gamma _2$. Now $S_{8,8}(\Gamma _2)$ has dimension 1 and is generated by the form $\chi _{8,8}$ (see [8, p. 11]) and we find $F^{\prime }= (1/104230)^2 \chi _{10}^2 \chi _{8,8}\,$.

We thus see that the modular form F, viewed as Teichmüller form, vanishes with order 3 along $\delta _0$ and order 8 along $\delta _1$. It can be seen as a section of ${\mathcal O}(28\lambda -3\, \delta _0-8 \, \delta _1)$; see the last remark of [33, p. 1766].

12.3 Degree 5

We have the isotypical decomposition of ${\mathrm {Sym}}^5({\mathrm {Sym}}^4(W))$ as

$$\begin{aligned}&2\, W[8,6,6] + 2\, W[8,8,4] + W[9,6,5] + W[9,7,4] + W[9,8,3] + 4\, W[10,6,4] + 2\, W[10,7,3] \\&\quad + 3\, W[10,8,2] + W[10,9,1] + W[10,10,0] + W[11,5,4] + 3\, W[11,6,3] + 2\, W[11,7,2]\\&\quad + W[11,8,1] + 3\, W[12,4,4] + W[12,5,3] + 4\, W[12,6,2] + W[12,7,1]+ 2\, W[12,8,0] \\&\quad + 2\, W[13,4,3] + 2\, W[13,5,2]+ 2\, W[13,6,1] + W[13,7,0] + 3\, W[14,4,2]+ W[14,5,1] \\&\quad + 2\, W[14,6,0] + W[15,3,2] + W[15,4,1] + W[15,5,0] + W[16,2,2] + 2\, W[16,4,0] \\&\quad + W[17,3,0] + W[18,2,0] + W[20,0,0]. \end{aligned}$$

The concomitant c provided by W[10, 9, 1] occurring in ${\mathrm {Sym}}^5({\mathrm {Sym}}^4(W))$ vanishes with order 2 along the locus of double conics, so the modular form $\gamma '(c)$ is divisible by $\chi _{18}^2$ and yields a cusp form $\chi _{1,8,5} \in S_{1,8,5}$. We calculate a few Fourier coefficients. We represent these inside the representation $W\otimes {\mathrm {Sym}}^8(\wedge ^2(W))\otimes {\det }^5(W)$ with W the standard representation of $\mathrm{GL}(3)$. With $N_1 = [1,1,1;1,1,1]$ we have

$$\begin{aligned} a(N_1)=1536 \, [0, 0, 0, 0, 0, 0, 0, -1, 1, 0, 0, 2, 0, -2, 0, 0, -1, -2, \ldots ]^t \end{aligned}$$

and $a(2N_1)$ is given by

$$\begin{aligned} 24\, [0, -4, 4, 8, 0, -8, -4, 17, -17, 4, 0, -50, 0, 50, 0, 4, 25, 50,\ldots ]^t \end{aligned}$$

while a([3, 2, 2; 4, 4, 2]) is given by

$$\begin{aligned} 192 \, [0, 8, -8, -36, 0, 36, 68, 74, -74, -68, -70, -160, 0, 160, 70, 42, 155, 110,\ldots ]^t. \end{aligned}$$

This gives the Hecke eigenvalue at $p=2$: $\lambda _2=-2880=-24(216+2^2(-24))$, in agreement with the fact that this cusp form is predicted to be a lift from $\Gamma _1$ with Hecke eigenvalues of the shape $\lambda _p=\tau (p)(b(p)+p^2\tau (p))$, where $\tau (p)$ is the Fourier coefficient of $\Delta$ at p, while b(p) is that of the unique normalized cusp form of weight 16 on $\Gamma _1$. This modular form is also given in [21, §5.6], where it is constructed using theta functions.

In the case of the components

$$\begin{aligned} W[10,6,4],\quad W[10,8,2],\quad W[11,6,3],\quad W[12,4,4],\quad W[12,6,2],\quad W[13,4,3],\quad W[13,6,1],\quad W[14,4,2] \end{aligned}$$

which occur with multiplicity $\ge 2$ one can find a nonzero concomitant c vanishing with multiplicity $\ge 2$ on the locus of double conics, and then $\gamma ^{\prime }(c)/\chi _{18}^2$ defines a holomorphic cusp form of weight

$$\begin{aligned} (4,2,8),\quad (2,6,6),\quad (5,3,7),\quad (8,0,8),\quad (6,4,6),\quad (9,1,7),\quad (7,5,5),\quad (10,2,6) \end{aligned}$$

respectively, and in these cases we checked that the eigenvalue for the Hecke operator at $p=2$ agrees with the data given in [4]. Also the case W[8, 6, 6] gives a concomitant c vanishing with order $\ge 2$ on the locus of double conics. Then $\gamma ^{\prime }(c)/\chi _{18}^2$ yields a cusp form in $S_{2,0,10}$. Its Fourier expansion starts with

$$\begin{aligned} \frac{1}{2308\, uvw} \left( \begin{matrix} c_1 \\ \vdots \\ c_6 \end{matrix} \right) q_1q_2q_3 +\cdots \end{aligned}$$

with

$$\begin{aligned} c_1&=u^2v^2w^2+u^2v^2w+u^2vw^2+uv^2w^2-6\, u^2vw+-6\, uv^2w\\&\quad + 14\, uvw^2 +u^2v+u^2w\\&\quad +uv^2 -20\, uvw+uw^2+v^2w+vw^2+u^2+14\, uv\\&\quad -6\, uw+v^2-6\, vw+w^2+u+v+w \\ \end{aligned}$$

and

$$\begin{aligned} c_2= & {} u^2\, v^2w^2+u^2v^2w+u^2vw^2-6\, u^2vw+u^2v+u^2w-v^2w-vw^2\\&+u^2-v^2+6\, vw-w^2-v-w \end{aligned}$$

and this determines the other coordinates by Lemma 4.2.

12.4 Degree 6

The so-called catalecticant is an invariant of degree $d=6$ and is associated to W[8, 8, 8] occurring in ${\mathrm {Sym}}^6({\mathrm {Sym}}^4(W))$. It is given as

$$\begin{aligned} i_6= \left| \begin{matrix} a_0 &{} a_3 &{} a_5 &{} a_4 &{} a_2 &{} a_1\\ a_3 &{} a_{10} &{} a_{12} &{} a_{11} &{} a_7 &{} a_6 \\ a_5 &{} a_{12} &{} a_{14} &{} a_{13} &{} a_9 &{} a_8 \\ a_4 &{} a_{11} &{} a_{13} &{} a_{12} &{} a_8 &{} a_7 \\ a_2 &{} a_7 &{} a_9 &{} a_8 &{} a_5 &{} a_4 \\ a_1 &{} a_6 &{} a_8 &{} a_7 &{} a_4 &{} a_3 \end{matrix} \right| \end{aligned}$$

and gives rise to a Siegel modular form of weight 56 vanishing with order 6 at D and with order at least 16 along ${\mathcal A}_{2,1}$ since the first cusp form vanishing with order 6 at $\infty$ on $\Gamma _1$ is $\Delta ^6$ of weight 72 and $16=72-56$. This modular form can be interpreted as a section of ${\mathcal O}(56\, \lambda -6\, \delta _0-16\, \delta _1)$ on $\overline{\mathcal M}_3$, in agreement with [33, p. 1766]. Another description of this form can be found in [26, Proposition 4.5].

Remark 12.1

Some of the modular forms constructed above have a geometric meaning. Chipalkatti proves in [7] that a ternary quartic f is the sum of the fourth powers of s linear forms $f=\ell _1^4+\ell _2^4+\cdots +\ell _s^4$ for $1\le s \le 5$ if and only if the concomitants in a certain set $U_s$ vanish, where the set $U_s$ is given by

$$\begin{aligned} U_1&=\{ c(2;4,2), c(2;0,4)\}, \qquad U_2=\{ c(3;6,0), c(3;0,6), c(3;3,3), c(3,2,2), c(3;0,0)\}, \\ U_3&=\{ c(4;4,0), c(4;2,4), c(4;1,3), c(4;0,2) \},\\ U_4&=\{ c_1(5;0,4)-c_2(5;0,4), c(5;2,0) \},\qquad U_5=\{ 3c(6,0,0)-c(3;0,0)^2\}; \end{aligned}$$

the concomitant c(d; m, n) corresponds to the irreducible representation $W[m_1,m_2,m_3]$ occurring in ${\mathrm {Sym}}^d({\mathrm {Sym}}^4(W))$ with $m=m_1-m_2$ and $n=m_2-m_3$ (and $4d=m_1+m_2+m_3$), see [7, Theorem 4.1]. For example, the vanishing of the modular forms $\chi _{4,2,16}$ and $\chi _{0,4,16}$ signalizes this property for $s=1$.

13 Teichmüller modular forms and the cohomology of local systems

It is well-known that Siegel modular forms of degree g occur in the cohomology of local systems on the moduli space ${\mathcal {A}}_{g}$ of principally polarized abelian varieties of dimension g. Denoting by $\pi : {\mathcal X}_g \rightarrow {\mathcal A}_g$ the universal abelian variety, we let $\mathbb {V}=R^1\pi _* {\mathbb {Q}}_{\ell }$ be the standard local system of rank 2g on ${\mathcal {A}}_{g}$. This comes with a symplectic pairing ${\mathbb {V}} \times {\mathbb {V}} \rightarrow {\mathbb {Q}}_{\ell }(-1)$. For every irreducible representation of the symplectic group $\mathrm{GSp}(2g,{\mathbb {Q}})$ with highest weight $\mu$, we have a local system $\mathbb {V}_{\mu }$ obtained from $\mathbb {V}$ by applying a Schur functor. We consider the ‘motivic’ Euler characteristic

$$\begin{aligned} e_c({\mathcal {A}}_{g};{\mathbb {V}_{\mu }})=\sum _{i=0}^{g(g+1)}(-1)^i [H^i_c({\mathcal {A}}_{g},\mathbb {V}_{\mu })] \end{aligned}$$

of compactly supported cohomology. The cohomology group $H^i_c({\mathcal {A}}_{g}\otimes {\mathbb {C}}, \mathbb {V}_{\mu }\otimes {\mathbb {C}})$ (resp. $H^i({\mathcal {A}}_{g}\otimes {\mathbb {C}},\mathbb {V}_{\mu }\otimes {\mathbb {C}})$) is provided with a mixed Hodge structure of weights $\le |\mu |+i$ (resp. $\ge |\mu |+i$) and the sums of the elements of the $2^g$ subsets of $\{ \mu _g+1,\mu _{g-1}+2,\ldots , \mu _1+g\}$ yield the degrees at which nontrivial steps in the Hodge filtration may occur, see [15] or [3] and references there. So the last step is $F^{|\mu |+g(g+1)/2}$ and it is here that we find Siegel modular forms: there is an isomorphism

$$\begin{aligned} S_{n(\mu )}= H^0(\overline{\mathcal {A}}_{g}\otimes {\mathbb {C}}, {\mathbb {E}}_{\mu }\otimes {\det }^{g+1}({\mathbb {E}})(-D)) \cong F^{|\mu |+g(g+1)/2} H^{g(g+1)/2}_c({\mathcal {A}}_{g}\otimes {\mathbb {C}}, {\mathbb {V}_{\mu }}\otimes {\mathbb {C}})\nonumber \\ \end{aligned}$$

(13)

where

$$\begin{aligned} n(\mu )=(\mu _1-\mu _2,\mu _2-\mu _3,\ldots , \mu _{g-1}-\mu _g, \mu _g +g+1)\, . \end{aligned}$$

We denote by $H^i_{!}({\mathcal {A}}_{g},\mathbb {V}_{\mu })$ the image of $H^i_c({\mathcal {A}}_{g},\mathbb {V}_{\mu })\rightarrow H^i({\mathcal {A}}_{g},\mathbb {V}_{\mu })$. It is known that if $\mu$ is regular then $H_{!}^i({\mathcal {A}}_{g},\mathbb {V}_{\mu })=(0)$ if $i\ne g(g+1)/2$. The above results are due to Faltings [14] and Faltings–Chai [15]; a key role is played by the (dual) BGG-complex.

We are interested in a similar interpretation of Teichmüller modular forms for $g\ge 2$. So far, we have only considered these for $g=3$. With a Teichmüller modular form of type $\rho$ (or weight $w(\rho )$), we mean here a section over $\overline{\mathcal {M}}_{g}$ of $\mathbb {E}^{\prime }_{\rho }$, the bundle obtained by applying the Schur functor associated to an irreducible representation $\rho$ of $\mathrm{GL}(g)$ to the Hodge bundle $\mathbb {E}^{\prime }$.

By pulling back under the Torelli map t, we obtain local systems $\mathbb {V}_{\mu }^{\prime }=t^*{\mathbb {V}_{\mu }}$ on ${\mathcal {M}}_{g}$ and ${\mathcal {M}}_{g}^c=\overline{\mathcal {M}}_{g}-\Delta _0$, the moduli space of curves of compact type. We consider the motivic Euler characteristic

$$\begin{aligned} e_c({\mathcal {M}}_{g};\mathbb {V}_{\mu }^{\prime })= \sum _{i=0}^{6g-6}(-1)^i [H^i_c({\mathcal {M}}_{g},\mathbb {V}_{\mu }^{\prime })] \end{aligned}$$

and similarly we can consider $e_c({\mathcal {M}}_{g}^c;\mathbb {V}_{\mu }^{\prime })$.

We obtain the following partial analogue of the above results.

Theorem 13.1

For the natural mixed Hodge structures on the middle cohomology groups, we have the following isomorphisms:

$$\begin{aligned}&F^mH^{3g-3}({\mathcal {M}}_{g},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})\cong H^0(\overline{\mathcal {M}}_{g},\mathbb {E}^{\prime }_{\mu }\otimes \mathcal {O}(13\lambda -\delta )), \end{aligned}$$

(14)

$$\begin{aligned}&F^mH^{3g-3}({\mathcal {M}}_{g}^c,{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})\cong H^0(\overline{\mathcal {M}}_{g},\mathbb {E}^{\prime }_{\mu }\otimes \mathcal {O}(13\lambda +\delta _0-2\delta )), \qquad \text { and} \end{aligned}$$

(15)

$$\begin{aligned}&F^mH^{3g-3}_c({\mathcal {M}}_{g},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}) \cong F^mH^{3g-3}_c({\mathcal {M}}_{g}^c,{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}) \cong H^0(\overline{\mathcal {M}}_{g},\mathbb {E}^{\prime }_{\mu }\otimes \mathcal {O}(13\lambda -2\delta )), \end{aligned}$$

(16)

where $m=3g-3+|\mu |$ is the maximum possible Hodge degree.

Proof

It is not clear whether the BGG-complex can be adapted to this setting and we resort to the logarithmic de Rham-complex. The local system ${\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}$ on ${\mathcal {M}}_{g}$ (or ${\mathcal {M}}_{g}^c$) corresponds to a holomorphic vector bundle ${\mathcal {V}}_{\mu }$ with flat connection $\nabla$ (the Gauss–Manin connection). The boundary $D=\overline{\mathcal {M}}_{g}-{\mathcal {M}}_{g}$ is a divisor with normal crossings. The bundle ${\mathcal {V}}_{\mu }$ admits a canonical extension ${\overline{\mathcal {V}}_{\mu }}$ to $\overline{\mathcal {M}}_{g}$ and $\nabla$ extends to a map

$$\begin{aligned} \nabla :{\overline{\mathcal {V}}_{\mu }}\rightarrow {\overline{\mathcal {V}}_{\mu }}\otimes \Omega ^1_{\overline{\mathcal {M}}_{g}}(\log D) \end{aligned}$$

with nilpotent polar part. Then the hypercohomology of the logarithmic de Rham-complex computes the cohomology of ${\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}$ on ${\mathcal {M}}_{g}$ :

$$\begin{aligned} H^p({\mathcal {M}}_{g}, {\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}) \cong \mathbb {H}^p(\overline{\mathcal {M}}_{g},{\overline{\mathcal {V}}_{\mu }}\otimes \Omega ^{\bullet }_{\overline{\mathcal {M}}_{g}}(\log D)). \end{aligned}$$

Cf.  [36]. Twisting with $\mathcal {O}(-D)$, we obtain a complex that computes the compactly supported cohomology:

$$\begin{aligned} H^p_c({\mathcal {M}}_{g},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})\cong \mathbb {H}^p(\overline{\mathcal {M}}_{g}, {\overline{\mathcal {V}}_{\mu }}\otimes \Omega ^{\bullet }_{\overline{\mathcal {M}}_{g}}(\log D)\otimes \mathcal {O}(-D)). \end{aligned}$$

Analogous statements hold for ${\mathcal {M}}_{g}^c\,$, after replacing D by $\Delta _0$.

These complexes admit natural Hodge filtrations, which induce the Hodge filtrations of the mixed Hodge structures on $H^p_{(c)}({\mathcal {M}}_{g},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})$ and $H^p_{(c)}({\mathcal {M}}_{g}^c,{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})$. Let $m=3g-3+|\mu |$ be the top Hodge degree. The complex $F^m({\overline{\mathcal {V}}_{\mu }}\otimes \Omega ^{\bullet }_{\overline{\mathcal {M}}_{g}}(\log D))$ consists of the sheaf $\mathbb {E}^{\prime }_{\mu }\otimes \Omega ^{3g-3}_{\overline{\mathcal {M}}_{g}}(\log D)$, considered as a complex supported in degree $3g-3$. As is well-known  [20], the canonical bundle of the stack $\overline{\mathcal {M}}_{g}$ equals $\mathcal {O}(13\lambda -2\delta )$. Since the spectral sequence associated to the Hodge filtration degenerates at $E_1$, we obtain the stated isomorphisms. $\square$

Thus the final steps in the Hodge filtrations on the middle cohomology groups are isomorphic to spaces of Teichmüller modular forms of highest weight $\mu +(13,13,\dots ,13)$ with prescribed vanishing behaviour along components of the boundary. This is entirely analogous to the relation between Siegel modular forms and the cohomology of ${\mathcal {A}}_{g}\,$. In particular, we have in genus 2 the isomorphism ${\mathcal {M}}_{2}^c\cong {\mathcal {A}}_{2}$ and the relation $10\lambda \sim \delta _0+2\delta _1$; the isomorphisms for ${\mathcal {M}}_{2}^c$ above agree with those for ${\mathcal {A}}_{2}$ discussed earlier. However, note that the Teichmüller modular forms found here are cusp forms in a strong sense: they need to vanish (at least) once or twice along each component of the boundary.

The isomorphisms continue to hold if we change the cohomological degrees on both sides by the same amount; in particular, $F^mH^p_{(c)}=(0)$ for $p<3g-3$. As a trivial example, take $\mu =0$, so $m=3g-3$; the top compactly supported cohomology of ${\mathcal {M}}_{g}$ and ${\mathcal {M}}_{g}^c$ is spanned by $L^{3g-3}$ and coincides with $F^{3g-3}$; on the other hand, $H^{3g-3}(\overline{\mathcal {M}}_{g},\mathcal {O}(13\lambda -2\delta ))\cong \mathbb {C}$, as follows from Serre duality.

We now return to $g=3$. Assume first that $\mu =(a,b,c)$ is even, i.e., $a+b+c$ is even. Then $\mu +(13,13,13)$ is odd and so are the Teichmüller modular forms in the isomorphisms above. As we saw in Sect. 7, odd Teichmüller modular forms of genus 3 are divisible by $\chi _9$, and the even quotients are pullbacks of Siegel modular forms. So for $g=3$ and $\mu$ even, the isomorphisms above may be rewritten as follows:

$$\begin{aligned}&F^mH^{6}({\mathcal {M}}_{3},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})\cong F^mH^{6}({\mathcal {M}}_{3}^c,{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})\cong M_{a-b,b-c,c+4}, \end{aligned}$$

(17)

$$\begin{aligned}&F^mH^{6}_c({\mathcal {M}}_{3},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}) \cong F^mH^{6}_c({\mathcal {M}}_{3}^c,{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}) \cong S_{a-b,b-c,c+4}, \end{aligned}$$

(18)

with $m=a+b+c+6$. Therefore, we also have isomorphisms with $F^mH^6({\mathcal {A}}_{3},\mathbb {V}_{\mu })$ resp. $F^mH^6_c({\mathcal {A}}_{3},\mathbb {V}_{\mu })$.

An interesting particular case of equation (17) is obtained by taking $\mu =0$. We find that $F^6H^6({\mathcal {M}}_{3},\mathbb {C})$ and $F^6H^6({\mathcal {M}}_{3}^c,\mathbb {C})$ are onedimensional and naturally isomorphic to $M_{0,0,4}$, which is spanned by the Eisenstein series $\alpha _4$ (see  [39]). This is Looijenga’s class of type (6, 6) (cf.  [28]). (The dual class in $H^6_c$ is of type (0, 0) and $F^6H^6_c=(0)$, since $\alpha _4$ is not a cusp form.)

Assume next that $\mu$ is odd, so $\nu =\mu +(13,13,13)$ is even. Equations (14)–(16) say the following. The classes in $F^{a+b+c+6}$ of the middle cohomology of ${\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}}$ on ${\mathcal {M}}_{3}$ correspond to Teichmüller forms of type $\nu$ that vanish along the entire boundary. Only those corresponding to forms that vanish at least twice along the boundary divisors parametrizing reducible curves are restrictions from ${\mathcal {M}}_{3}^c$. Only the forms vanishing at least twice along the entire boundary correspond to compactly supported classes.

For concrete examples, note that Bergström  [2] has determined $e_c({\mathcal {M}}_{3},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})$, hence $e({\mathcal {M}}_{3},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})$ for all $\mu$ with $|\mu |=a+b+c\le 7$. E.g., for $\mu =(1,1,1)$, so $m=9$, one has $e_c=-L^7-L^2+L+1$, so $e=L^9+L^8-L^7-L^2$. Then $F^me=L^9$, but it is not yet clear which cohomology groups contribute to this. Since the virtual cohomological dimension of ${\mathcal {M}}_{3}$ equals 7 (cf.  [18]), only classes in $H^6$ and $H^7$ can contribute, and there must be at least one class in $F^9H^6$. Now

$$\begin{aligned} F^9H^6({\mathcal {M}}_{3},{\mathbb {V}}_{\mu }^{\prime }\otimes {\mathbb {C}})\cong H^0(\overline{\mathcal {M}}_{3},\mathcal {O}(14\lambda -\delta )). \end{aligned}$$

According to  [39], the space $S_{0,0,14}$ is spanned by the cusp form $\beta _{14}$. We conclude that $\beta _{14}$ vanishes along ${\mathcal {A}}_{2,1}$ and that $F^9H^7=(0)$. In fact, by loc. cit., $\beta _{14}$ vanishes twice along ${\mathcal {A}}_{2,1}$, so the corresponding class is a restriction from ${\mathcal {M}}_{3}^c$.

In  [3], Bergström and two of the present authors have counted curves of genus 2 and 3 over finite fields and their numbers of points and in this way determined the $\mathbb {F}_q$-traces of $e_c({\mathcal {M}}_{3},\mathbb {V}_{\mu }^{\prime })$ and $e_c({\mathcal {A}}_{3},\mathbb {V}_{\mu })$ for $q\le 25$. As discussed in loc. cit., this has led to a complete conjecture for these motivic Euler characteristics in the case of ${\mathcal {A}}_{3}$. In the case of ${\mathcal {M}}_{3}$, we have obtained precise conjectures for nearly all $\mu$ with $|\mu |\le 20$ (for three $\mu$ with $|\mu |=19$, some information is missing). The work of Chenevier–Renard [6], Chenevier–Lannes [5], Taïbi [38], and Mégarbané [29] has played a particularly important role here. Below, we use some of the results to provide further examples (in effect, we obtain further evidence for the conjectures). Note that the ‘motivic’ conjectures tell us in particular where terms of the maximum possible Hodge degree are to be expected.

We first consider scalar-valued Teichmüller modular forms, so $\mu =(k,k,k)$. Here are three more even cases:

• (1) $k=2$, $m=12$: $e_c=1$, $e=L^{12}$; this corresponds to the Eisenstein series $\alpha _6$ ;

• (2) $k=4$, $m=18$: $e_c=L^5+1$, $e=L^{18}+L^{13}$; $F^me$ corresponds to the Eisenstein series $\alpha _4^2$ ;

• (3) $k=6$, $m=24$: $F^me=L^{24}+L^7S_{18}$ corresponds to the space spanned by the modular forms $\alpha _4\alpha _6$ and $\alpha _{10}\,$.

More interesting are two more odd cases:

• (1) $k=3$, $m=15$: $F^me=L^{15}$; indeed, $H^0(\overline{\mathcal {M}}_{3},\mathcal {O}(16\lambda -\delta ))$ is spanned by $\beta _{16}$, since no nontrivial linear combination of the other two generators $\alpha _4\alpha _{12}$ and $\alpha _{16}-\tfrac{27}{4}\alpha _6\alpha _{10}$ of the space of cusp forms of weight 16 vanishes along ${\mathcal {A}}_{2,1}$ ;

• (2) $k=5$, $m=21$: in this case, we conjecture that

  $$\begin{aligned} e_c({\mathcal {M}}_{3},\mathbb {V}_{5,5,5}^{\prime })= L^7+2L^6+5L^5+6L^4+4L^3+3L^2+2L+1+S[4,10], \end{aligned}$$

  so $F^me= L^{21}+F^mS[4,10]\cong \mathbb {C}^2$. On the other hand, $H^0(\overline{\mathcal {M}}_{3},\mathcal {O}(18\lambda -\delta ))$ is spanned by $\alpha _4\beta _{14}$ and $\chi _{18}$, since no linear combination of $\alpha _6\alpha _{12}$ and $\alpha _{18}-27\alpha _4^2\alpha _{10}$ (the other generators of the space of cusp forms of weight 18) vanishes along ${\mathcal {A}}_{2,1}\,$. Note also that $F^me_c=F^mS[4,10]\cong \mathbb {C}$. This corresponds to $H^0(\overline{\mathcal {M}}_{3},\mathcal {O}(18\lambda -2\delta ))=\mathbb {C}\cdot \chi _{18}$.

More generally, for $\mu =(a,b,c)$ odd, it is natural to consider

$$\begin{aligned} H^0(\overline{\mathcal {M}}_{3},\mathbb {E}^{\prime }_{\mu }\otimes \mathcal {O}(13\lambda -2\delta )), \end{aligned}$$

since this corresponds to compactly supported cohomology by equation (16) and thus to the data obtained by counting curves.

This space is a subspace of

$$\begin{aligned} S_{a-b,b-c,c+13}^2, \end{aligned}$$

which by Theorem 11.6 is isomorphic to

$$\begin{aligned} C_{d,(a-5,b-5,c-5)}(-\tfrac{d-2}{2}DC), \end{aligned}$$

where $d=a+b+c-15$ is even. In particular, there are no contributions for $|\mu | < 15$.

More precisely, the subspace consists of those forms vanishing to order at least 2 along ${\mathcal {A}}_{2,1}$ (or along $\delta _1$ when considered as Teichmüller forms).

Now the isomorphism is given by $c\mapsto \gamma (c)\chi _{18}$ and we know that $\chi _{18}$ has order 6 along ${\mathcal {A}}_{2,1}$. So whenever $\gamma (c)$ has order at least $-4$ along $\delta _1$, we obtain an element of the subspace of interest. Since $\chi _{4,0,-1}$ has a simple pole along $\delta _1$ (by Lemma 4.1 and Proposition 10.1), this is automatically the case for $d\le 4$.

The case $d=0$, hence $\mu =(5,5,5)$ has been discussed already: $m=21$ and

$$\begin{aligned} F^me_c(\mathbb {V}_{5,5,5}^{\prime }\otimes \mathbb {C}) =F^mS[4,10] \end{aligned}$$

corresponds to the space spanned by $\chi _{18}\,$.

In the case $d=2$, there are three nonzero spaces of concomitants (Sect. 12), each of dimension 1. The corresponding $\mu$ are (11, 3, 3), (9, 5, 3), and (7, 7, 3). The associated (irreducible) ‘motives’ are M[23, 13, 5], S[8, 9], and M[23, 15, 3], respectively. Here, S[8, 9] is the motive associated to Siegel cusp forms of degree 2 (!) of type ${\mathrm {Sym}}^8\det ^9$. The other two motives are not associated to Siegel cusp forms. They are 6-dimensional motives of weight 23, first identified by Chenevier–Renard [6]; later, Mégarbané [29] computed their $\mathbb {F}_q$-traces for many small q, in complete agreement with the counting data of [3]. In denoting these motives, we essentially follow the notation of [6] and [29]; e.g., the Hodge degrees of M[23, 13, 5] are

$$\begin{aligned} 0,\quad 5,\quad 9,\quad 14,\quad 18,\quad 23 \end{aligned}$$

(so the successive widths are 23, 13, and 5).

We have explicitly computed all (spaces of) concomitants of ternary quartics of degree at most 6. We have also determined the subspaces of concomitants vanishing to order at least r along DC, for each $r\ge 0$. (The computations are quite involved and we will discuss them in a future paper.)

In particular, for $d=4$, we find nonzero concomitants vanishing along DC exactly in the 13 following cases:

$$\begin{aligned}&\mu =(14,4,1),\quad (13,5,1),\quad (12,5,2),\quad (12,4,3),\quad (11,7,1),\quad (11,6,2),\quad (11,5,3),\\&(10,7,2),\quad (10,6,3),\quad (10,5,4), \quad (9,7,3),\quad (9,5,5), \quad (8,7,4). \end{aligned}$$

The corresponding motives have been identified in 10 of the 13 cases. Their dimensions are 4, 6, or 8. The work of Mégarbané [29] has played a crucial role here. There are reasons to believe that in the 3 remaining cases the motives will be 12-dimensional; in any case, these motives have not yet been identified.

The conjectural formulas for the Euler characteristics $e_c({\mathcal {M}}_{3},{\mathbb {V}}_{\mu }^{\prime })$ with $\mu$ odd are rather complicated and contain many more terms than just the motives mentioned above. The techniques and computations that were used to obtain the formulas, and the formulas themselves, will be discussed elsewhere.

Appendix: Teichmüller modular forms extend to \(\overline{\mathcal {M}}_{g}\) for \(g\ge 3\)

The result below was found by Farkas, Pandharipande, and the second author, after a talk in Berlin (January 2019), in which Proposition 7.3 and its proof were presented. The visit to Berlin was supported by the Einstein Stiftung.

Let ${\mathcal {M}}_{g}$ denote the moduli space of curves of genus $g\ge 2$. The Torelli morphism $t:{\mathcal {M}}_{g}\rightarrow {\mathcal {A}}_{g}$ is a morphism of algebraic stacks. By pullback under t we obtain the Hodge bundle ${\mathbb {E}}^{\prime }$ on ${\mathcal {M}}_{g}$ and for each irreducible representation $\rho$ of $\mathrm{GL}(g)$ a vector bundle ${\mathbb {E}}^{\prime }_{\rho }$ on ${\mathcal {M}}_{g}\,$. Sections of such a bundle ${\mathbb {E}}^{\prime }_{\rho }$ are called Teichmüller modular forms of degree g. The vector bundle ${\mathbb {E}}^{\prime }$ and hence all the ${\mathbb {E}}^{\prime }_{\rho }$ extend in a natural way over the Deligne–Mumford compactification $\overline{\mathcal M}_g\,$. The purpose of this note is to show that a holomorphic section of ${\mathbb {E}}^{\prime }_{\rho }$ automatically extends to a holomorphic section of the extended bundle, for $g\ge 3$. (As Ichikawa already observed, the result fails for $g=2$: Igusa’s cusp form $\chi _{10}$ vanishes exactly along $\Delta _0$ and (doubly) along $\Delta _1\,$, so its inverse is a regular section of $\det ^{-10}({\mathbb {E}}^{\prime })$ on ${\mathcal {M}}_{2}$ which does not extend to $\overline{\mathcal {M}}_{2}\,$.)

Proposition 14.1

Let $g\ge 3$ be an integer. Let $\rho$ be an irreducible representation of $\mathrm{GL}(g)$ and let ${\mathbb {E}}^{\prime }_{\rho }$ be the bundle on $\overline{\mathcal {M}}_{g}$ arising from the Hodge bundle ${\mathbb {E}}^{\prime }$ of rank g by applying the Schur functor corresponding to $\rho$. Then a section s of ${\mathbb {E}}^{\prime }_{\rho }$ over ${\mathcal {M}}_{g}$ extends to a regular section of ${\mathbb {E}}^{\prime }_{\rho }$ over $\overline{\mathcal {M}}_{g}\,$.

Proof

We first show that s extends over the boundary divisors parametrizing reducible curves. Let $\Delta _i$ with $0<i\le g/2$ be such a divisor. Let D be the open boundary stratum of $\Delta _i\,$. So $D\cong {\mathcal {M}}_{i,1}\times {\mathcal {M}}_{g-i,1}$ or its quotient by $S_2$ when $i=g/2$. Let $X={\mathcal {M}}_{g}\cup D$. We show that the global sections of ${\mathbb {E}}^{\prime }_{\rho }$ on ${\mathcal {M}}_{g}$ agree with those on X, as follows.

On X, we have the following exact sequences:

$$\begin{aligned}&0\rightarrow {\mathcal O}(-D)\rightarrow {\mathcal O}\rightarrow {\mathcal O}_D\rightarrow 0,\\&0\rightarrow {\mathcal O}\rightarrow {\mathcal O}(D)\rightarrow {\mathcal O}(D)_D\rightarrow 0,\\&0\rightarrow {\mathbb {E}}^{\prime }_{\rho }\rightarrow {\mathbb {E}}^{\prime }_{\rho }(D) \rightarrow ({\mathbb {E}}^{\prime }_{\rho }(D))|_D\rightarrow 0. \end{aligned}$$

We want

$$\begin{aligned} H^0(X,{\mathbb {E}}^{\prime }_{\rho })=H^0(X,{\mathbb {E}}^{\prime }_{\rho }(D)) \end{aligned}$$

and obtain this by showing that

$$\begin{aligned} H^0(X,({\mathbb {E}}^{\prime }_{\rho }(D))|_D)=H^0(D,{\mathbb {E}}^{\prime }_{\rho }(D))=0. \end{aligned}$$

For the latter vanishing, consider the complete curve B in D obtained by varying the attachment point on a given smooth curve C of genus $g-i$ (since $g\ge 3$ we have $g-i\ge 2$). On such a curve, ${\mathbb {E}}^{\prime }$ and thus ${\mathbb {E}}^{\prime }_{\rho }$ are constant, so a direct sum of trivial line bundles. The line bundle ${\mathcal O}(D)|_B$ is $-K_C\,$, of strictly negative degree, so its global sections on B are zero. The same holds then for the global sections of ${\mathbb {E}}^{\prime }_{\rho }(D)$ on B and then also on D, since the curves B fill out D. Thus

$$\begin{aligned} H^0(X,{\mathbb {E}}^{\prime }_{\rho })=H^0(X,{\mathbb {E}}^{\prime }_{\rho }(D)) \end{aligned}$$

as desired. By twisting more with ${\mathcal O}(D)$ we find similarly that

$$\begin{aligned} H^0(X,{\mathbb {E}}^{\prime }_{\rho })=H^0(X,{\mathbb {E}}^{\prime }_{\rho }(kD)) \end{aligned}$$

for an arbitrary positive integer k. Thus a global section of ${\mathbb {E}}^{\prime }_{\rho }$ on ${\mathcal {M}}_{g}\,$, which a priori could have a pole along D of some order, is in fact regular along D and extends to X.

Next, we consider $\Delta _0\,$. Let D be the open part consisting of the open boundary stratum and the stratum corresponding to smooth curves of genus $g-1$ with a singular elliptic tail attached. So $D\cong ({\mathcal C}_{g-1}^2) /S_2\,$. Again, let $X={\mathcal {M}}_{g}\cup D$. We proceed as above, but for the argument on D we may as well work on the double cover $D_2\cong {\mathcal C}_{g-1}^2\,$. Let B in D be the complete curve obtained by varying the (second) point Q on a given one-pointed smooth curve (C, P) of genus $g-1$. (When $Q=P$, we get a stable curve in the smaller stratum, of course.) Now ${\mathcal O}(D)|_B$ is $-K_C(2P)$, again of negative degree. On $D_2\,$, the Hodge bundle ${\mathbb {E}}^{\prime }$ sits in an exact sequence

$$\begin{aligned} 0\rightarrow {\mathbb {F}}^{\prime } \rightarrow {\mathbb {E}}^{\prime } \,{\buildrel r \over \longrightarrow }\, {\mathcal O}_{D_2}\rightarrow 0, \end{aligned}$$

where ${\mathbb {F}}^{\prime }$ is the Hodge bundle pulled back from genus $g-1$ and the map r to ${\mathcal O}_{D_2}$ is obtained by taking the residue at P. The restriction of ${\mathbb {F}}^{\prime }$ to B is trivial. A global section t of ${\mathbb {E}}^{\prime }(D)|_{D_2}$ on B gives a global section of ${\mathcal O(D)}_{D_2}\,$ on B, which must be trivial, so t comes from a section of ${\mathbb {F}}^{\prime }(D)|_{D_2}$ on B, which must be trivial as well. To show that a global section of ${\mathbb {E}}^{\prime }_{\rho }(D)$ on B also must vanish, we argue as follows. Firstly, global sections of $(({\mathbb {E}}^{\prime })^{\otimes n})(D)$ on B vanish for every n; e.g., for $n=3$ a global section t gives via $r\otimes r\otimes r$ a global section of ${\mathcal O}(D)$, which vanishes; so t comes from a section of the subbundle

$$\begin{aligned} ({\mathbb {F}}^{\prime }\otimes {\mathbb {E}}^{\prime } \otimes {\mathbb {E}}^{\prime } + {\mathbb {E}}^{\prime }\otimes {\mathbb {F}}^{\prime } \otimes {\mathbb {E}}^{\prime } + {\mathbb {E}}^{\prime }\otimes {\mathbb {E}}^{\prime } \otimes {\mathbb {F}}^{\prime })(D) \end{aligned}$$

of rank $g^3-1$; applying $r\otimes r\otimes 1$, $r\otimes 1\otimes r$, and $1\otimes r\otimes r$, we find t must come from a section of

$$\begin{aligned} ({\mathbb {F}}^{\prime }\otimes {\mathbb {F}}^{\prime } \otimes {\mathbb {E}}^{\prime } + {\mathbb {F}}^{\prime }\otimes {\mathbb {E}}^{\prime } \otimes {\mathbb {F}}^{\prime } + {\mathbb {E}}^{\prime }\otimes {\mathbb {F}}^{\prime } \otimes {\mathbb {F}}^{\prime })(D); \end{aligned}$$

applying $r\otimes 1\otimes 1$, $1\otimes r\otimes 1$, and $1\otimes 1\otimes r$, we find t comes from a section of

$$\begin{aligned} ({\mathbb {F}}^{\prime }\otimes {\mathbb {F}}^{\prime } \otimes {\mathbb {F}}^{\prime })(D), \end{aligned}$$

which must vanish. Next, on $D_2$ we have that $\det ({\mathbb {E}}^{\prime })$ is trivial. So with $\rho =(\rho _1\ge \rho _2\ge \cdots \ge \rho _g)$, we may and will assume that $\rho _g=0$. Finally, ${\mathbb {E}}^{\prime }_{\rho }$ occurs in the decomposition of $({\mathbb {E}}^{\prime })^{\otimes n}$ as a summand, with $n=|\rho |=\rho _1+\cdots +\rho _{g-1}$. More precisely, the bundle of $\mathrm{GL}(g)$-equivariant homomorphisms from ${\mathbb {E}}^{\prime }_{\rho }$ to $({\mathbb {E}}^{\prime })^{\otimes n}$ is free of rank equal to the multiplicity of the former in the decomposition of the latter. Choose a nonzero section u of this bundle; via u, a section of ${\mathbb {E}}^{\prime }_{\rho }$ gives a section of $({\mathbb {E}}^{\prime })^{\otimes n}$, by taking the zero section of the other isotypical components. It follows that a global section of ${\mathbb {E}}^{\prime }_{\rho }(D)$ on B necessarily vanishes, as claimed. The same holds then for the global sections on D and then also for ${\mathbb {E}}^{\prime }_{\rho }(kD)$ for positive k.

We conclude that a section of ${\mathbb {E}}^{\prime }_{\rho }$ over ${\mathcal {M}}_{g}$ extends over nonempty open parts of each boundary divisor. This is enough, since the complement is of codimension 2 and $\overline{\mathcal {M}}_{g}$ is a smooth stack. $\square$