Exercise 12, § 4

Assume that the form $x.y = B_M(x, y)$ is non-degenerate and that W is infinite. Identify E with its dual $E^*$ by means of $B_M$; in particular, denote by $(e_s^*)$ the basis of E dual to the basis $(e_s)$, and by C the interior of the simplicial cone $\overline{C}$ generated by the $e_s^*$. Let G be the orthogonal group of $B_M$, and let $\mu$ be a Haar measure on G; the group G is unimodular and contains W.

a) Show that, if $\nu(W \setminus G) < \infty$ (where $\nu$ is a non-zero positive measure invariant under G), the form $B_M$ is of signature $(n-1, 1)$, with

and that $x.x < 0$ for all $x \in C$.

(Let $x \in C$ be such that $x.x \neq 0$, and let $L_x$ be the hyperplane orthogonal to $x$. Show, by using Exerc. 10, that the restriction of $B_M$ to $L_x$ is either positive or negative; show that the second case implies that $B_M$ is of signature $(1, n-1)$ and that this is impossible. Deduce that $x.x \leq 0$ for all $x \in C$, and hence $x.x < 0$ since C is open.)

b) Conversely, assume that $B_M$ is of signature $(n-1,1)$ and $x.x < 0$ for all $x \in C$ (in which case $W$, or $(W,S)$, or the corresponding Coxeter graph, is said to be of hyperbolic type). Let $H$ be the set of $x \in E$ such that $x.x < 0$, and let $H_+$ be the connected component of $H$ containing $C$. Show that $H$ is the disjoint union of $H_+$ and $H_- = -H_+$, and that $H_-$ is contained in the simplicial cone generated by $(e_s)_{s \in S}$ (use the fact that $H_-$ is the polar of $H_+$). Show that $H_+$ and $H_-$ are stable under $W$.

c) Retain the notation and assumptions of $b$. Let $f$ be the linear form on $E$ defined by $f(e_s) = 1$ for all $s \in S$. If $x \in H$, put $\varphi(x) = f(x)^2/(x.x)$. If $\mathbf{PH}$ denotes the image of the cone $H$ in the projective space $\mathbf{P}(E)$, the function $\varphi$ defines a function $\tilde{\varphi}$ on $\mathbf{PH}$. Show that the map

is proper. Deduce that, for all $x \in H_+$, the functions $w \mapsto \varphi(w.x)$ and $w \mapsto f(w.x)$ attain their maximum for a value $w_1 \in W$ (use the fact that $G$ acts properly on $\mathbf{PH}$, cf. Exerc. 11, and that $W$ is discrete in $G$); show that these properties are equivalent to $w_1(x) \in \overline{C}$. Deduce that $\overline{C} \cap H_+$ is a fundamental domain for the action of $W$ in $H_+$, and that the image of this fundamental domain in $\mathbf{PH}$ has finite measure for the invariant measure $\nu$ on $\mathbf{PH}$ (use Exerc. 11). Conclude that $\nu(W \setminus G) < \infty$. Show that $W \setminus G$ is compact if and only if $\overline{C}$ is contained in $H_+$, i.e. if $e_s^*.e_s^* < 0$ for all $s \in S$ (in which case $W$, or $(W,S)$, or the corresponding Coxeter graph, is said to be of compact hyperbolic type).