§ 4. Geometric representation of a Coxeter group
In this paragraph, all the vector spaces considered are real vector spaces.
1. Form associated to a Coxeter group
Let S be a set and let M = be a Coxeter matrix (Chap. IV, § 1, no. 9) of type S. Recall that this means:
(1) The elements of M are integers or . (2) M is symmetric. (3) for all s. (4) for .
Let , let be the canonical basis of E, and let be the bilinear form on E such that
The form is symmetric. It is called the associated form of the matrix M. We have
Let and let be the linear form . We denote by the pseudo-reflection defined by the pair (cf. § 2, no. 1); since , it is a reflection (, no. 2). We have
and in particular
Since is not isotropic for , the space E is the direct sum of the line and the hyperplane orthogonal to . Since is equal to on and to 1 on , it follows that preserves the form . When S is finite and is non-degenerate (a case to which we shall return in no. 8), it follows that is an orthogonal reflection (, no. 3).
2. The plane and the group generated by and
In this number, we denote by and two elements of with . Put and denote by the plane .
Proposition 1
The restriction of to is positive, and it is non-degenerate if and only if is finite.
Let , with , be an element of . We have
The reflections and leave stable. We are going to determine the order of the restriction of to . We distinguish two cases:
.
Let . We have , so is invariant under and . Moreover,
is finite.
The form provides with the structure of a euclidean plane. Since the scalar product of and is equal to , we can orient so that the angle between the half-lines and is equal to . If D and D' denote the lines orthogonal to and ,
We now return to E:
Proposition 2
The subgroup of generated by and is a dihedral group of order *.
Since and are of order 2, and are distinct, it is enough to show that their product is of order . When is infinite, this follows from above. When is finite, it follows from Prop. 1 that E is the direct sum of and its orthogonal complement ; since and are the identity on , and since the restriction of to is of order by , the order of is indeed equal to .
3. Group and representation associated to a Coxeter matrix
We retain the notation of the preceding numbers. Let be the group defined by the family of generators and the relations¹
Proposition 3
There exists a unique homomorphism
such that for all . The elements of preserve the bilinear form .
To prove the existence and uniqueness of , it is enough to show that if . Now, if , this follows from the fact that is of order 2; if , it follows from what we proved in no. 2. Finally, since the reflections preserve , so do the elements of .
Remark 1
We shall see in no. 4 that is injective; the group W can thus be identified with the subgroup of generated by the .
Proposition 4
a)* The map from to is injective.
b) Each of the is of order 2.
c) If , is of order .
Assertion a) follows from the fact that the composite map
from to is injective.
For b) (resp. c)), we remark that the order of (resp. the order of ) is at most 2 (resp. at most ). Since we have seen in no. 2 that the order of (resp. of ) is 2 (resp. ), we must have equality.
In view of a), can be identified with a subset of by means of the map .
¹ This means that, if denotes the free group on , is the quotient of by the smallest normal subgroup of containing the elements , for .
Corollary
The pair* (W, S) is a Coxeter system with matrix M.
This is simply the content of properties b) and c), together with the definition of W.
Remark 2
We have thus shown that every Coxeter matrix corresponds to a Coxeter group.
4. Contragredient representation
Let E* be the dual of E. Since W acts on E via σ, it also acts, by transport of structure, on E*. The corresponding representation
is called the contragredient representation of σ. We have
If x* ∈ E* and w ∈ W, we denote by w(x*) the transform of x* by σ*(w).
If s ∈ S, denote by A_s the set of x* ∈ E* such that x*(e_s) > 0. Let C be the intersection of the A_s, s ∈ S. When S is finite, C is an open simplicial cone in E* (§ 1, no. 6).
Theorem 1 (Tits)
If w ∈ W and C ∩ w(C) ≠ ∅, then w = 1*.
We indicate immediately several consequences of this theorem:
Corollary 1
The group* W acts simply-transitively on the set of w(C) for w ∈ W.
This is clear.
Corollary 2
The representations* σ and σ* are injective.
Indeed, if σ*(w) = 1, then w(C) = C, so w = 1 by the theorem. The injectivity of σ follows from that of σ*.
Corollary 3
If S is finite, σ(W) is a discrete subgroup of* \mathbf{GL}(E) provided with its canonical Lie group structure; similarly, σ(W) is a discrete subgroup of* \mathbf{GL}(E^*) *.
Let x* ∈ C. The set U of g ∈ \mathbf{GL}(E^*) such that g(x*) ∈ C is a neighbourhood of the identity element in \mathbf{GL}(E^*); by the theorem,
thus, σ*(W) is a discrete subgroup of \mathbf{GL}(E^*). By transport of structure, it follows that σ(W) is discrete in \mathbf{GL}(E).
Proof of Theorem 1.
If , denote by the length of with respect to (Chap. IV, § 1, no. 1).
We are going to prove the following assertions, where denotes an integer :
(P_n) Let , with , and . Then: either ; or and .
(Q_n) Let , with , and . Let be the subgroup of generated by and . There exists such that
These assertions are trivial for . We prove them by induction on , according to the scheme
Proof that .
Let , with , and . We can write in the form with and . If , applied to shows that , hence and . If , applied to shows that there exists such that
Lemma 1
Let , with , and let . Then is contained in either or in , and in the second case .
The proof will be given in no. 5.
We apply the lemma to the element . There are two possibilities: either
Proof that ((P_{n+1}) and (Q_n)) \Longrightarrow (Q_{n+1}).
Let w \in W, with l(w) = n + 1, and s, s' \in S, s \neq s'. If w(C) is contained in A_s \cap A_{s'}, condition (Q_{n+1}) is satisfied with u = 1. For if not, suppose for example that w(C) is not contained in A_s. By (P_{n+1}), w(C) \subset s(A_s) and l(sw) = n. By (Q_n), applied to sw, there exists v \in W_{s,s'} such that
sw(C) \subset v(A_s \cap A_{s'}) \quad \text{and} \quad l(sw) = l(v) + l(v^{-1}sw).
Then
w(C) \subset sv(A_s \cap A_{s'})
and
l(w) = 1 + l(sw) = 1 + l(v) + l(v^{-1}sw) \geq l(sv) + l((sv)^{-1}w) \geq l(w),
so the inequalities above must be equalities. It follows that (Q_{n+1}) is satisfied with u = sv.
Proof of the theorem.
Let w \in W, with w \neq 1. We can write w in the form sw' with s \in S and l(w') = l(w) - 1. By (P_n), applied to w' and n = l(w'), we have w'(C) \subset A_s, since the case w'(C) \subset s(A_s) is excluded because l(sw') = l(w) = l(w') + 1. So w(C) = sw'(C) \subset s(A_s), and since A_s and s(A_s) are disjoint, we have C \cap w(C) = \varnothing. Q.E.D.
5. Proof of lemma 1
Let E_{s,s'}^* be the dual of the plane E_{s,s'} = \mathbf{R}e_s \oplus \mathbf{R}e_{s'} (no. 2). The transpose of the injection E_{s,s'} \to E is a surjection
that commutes with the action of the group W_{s,s'}. It is clear that A_s, A_{s'} and A_s \cap A_{s'} are the inverse images under p of corresponding subsets of E_{s,s'}^* (considered as the space of the contragredient representation of the Coxeter group W_{s,s'}). Moreover, since the length of an element of W_{s,s'} is the same with respect to \{s, s'\} and with respect to S (Chap. IV, § 1, no. 8), we are reduced finally to the case where S = \{s, s'\}; if m = m(s, s'), the group W is then a dihedral group of order 2m.
We now distinguish two cases: a) m = +\infty. Let (\varepsilon, \varepsilon') be the dual basis of (e_s, e_{s'}). Then
Let D be the affine line of containing and ; the formulas above show that D is stable under s and and that the restriction of s (resp. ) to D is the reflection with respect to the point (resp. ). Let
be the affine bijection . Let be the image under of the open interval , and let be the union of the for . Then ; moreover, by the Remark of § 2, no. 4, applied to the affine space D, the are permuted simply-transitively by W; hence so are the . If , is equal to one of the , hence is contained in if and in if . In the second case, and are on opposite sides of the point ; hence (loc. cit.).
b) is finite.
The form is then non-degenerate (no. 2) so we can identify with E. We have seen that E can be oriented so that the angle between the half-lines and is equal to . Let D (resp. ) be the half-line obtained from (resp. ) by a rotation of (resp. ), cf. Fig. 2. The chamber C is the set of whose scalar product with and is ; this is the open angular sector with origin and extremity D. By the Remark of § 2, no 5, every element of W transforms C into an angular sector that is on the same side of D as C (i.e. is contained in ) or on the opposite side (i.e. contained in ), and in the latter case
which completes the proof of the lemma.

Fig. 2
6. Fundamental domain of w in the union of the chambers
We retain the notation of no. 4. For , denote by the hyperplane of orthogonal to , by the set of such that and by the intersection of the for . For the weak topology defined by the duality between and (Topological Vector Spaces, Chap. II, §6, no. 2), the are closed half-spaces and is a closed convex cone. Moreover, is the closure of ; indeed, if and , then for every real number and .
For , put
We have , and . The sets , for , form a partition of .
On the other hand, recall (Chap. IV, §1, no. 8) that denotes the subgroup of generated by . Clearly for and .
Proposition 5
Let and . If , then , and .
We are reduced immediately to the case . The proof is by induction on the length of . If , the assertion is clear. If , there exists such that and then (cf. the end of no. 4) , hence . Since , it follows that
Then for all , and a fortiori for all
Consequently, the relation implies on the one hand that , and hence that , and on the other hand that . The induction hypothesis then implies that and , so and since . It follows that and that .
Corollary
Let be a subset of and an element of . The stabiliser of in is .
Now let be the union of the for , and let be the set of subsets of of the form , with and . By the above, is a partition of .
Proposition 6
(i) The cone U is convex.
(ii) Every closed segment contained in U meets only finitely many elements of .
(iii) The cone is a fundamental domain for the action of W on U.
To prove (iii), it is enough to show that, if and are such that , then . Now there exist two subsets X and Y of S such that and ; we have , and Prop. 5 shows that and , which implies that .
Now let ; we shall show that the segment is covered by finitely many elements of , which will establish both (i) and (ii). By transforming and by the same element of W, we can assume that . Let be such that . We argue by induction on the length of . For , the relation is equivalent to and hence to (cf. no. 4). Prop. 7 of no. 8 of Chap. IV, §1 now implies that there exist only finitely many such that . The set T of such that is thus finite. On the other hand, the intersection is a closed segment . If , that is if , there exist subsets X and Y of S such that and . The open segment is then contained in , so . If , there exists an such that . Then and . The induction hypothesis thus implies that the segment is covered by a finite number of elements of , and hence so is
since .
7. Irreducibility of the geometric representation of a Coxeter group
We retain the notation of the preceding nos., and assume that S is finite.
Proposition 7
Assume that* is irreducible (Chap. IV, §1, no. 9). Let be the subspace of E orthogonal to E with respect to . The group W acts trivially on , and every subspace of E distinct from E and stable under W is contained in .
If , then for all . Since W is generated by S, it follows that W acts trivially on .
Let be a subspace of E stable under W. Let be two elements that are linked in the graph of (Chap. IV, §1, no. 9); recall that this means that . Suppose that . Then and since the coefficient of in is non-zero, we have . Since is connected, it follows that, if contains one of the it contains them all and coincides with E. Except in this case, it follows from Prop. 3 of §2, no. 2 that, for all , is contained in the hyperplane orthogonal to . Since the intersection of the is equal to , this proves the proposition.
Corollary
Assume that* is irreducible. Then: a) If is non-degenerate, the W-module is absolutely simple. b) If is degenerate, the W-module is not semi-simple.
In case a), Prop. 7 shows that is simple, hence also absolutely simple (, no. 1, Prop. 1).
In case b), , (since ), and Prop. 7 shows that has no complement stable under ; thus, the W-module is not semi-simple.
8. Finiteness criterion
We retain the notation of the preceding nos., and assume that is finite.
Theorem 2
The following properties are equivalent*: (1) is finite. (2) The form is positive and non-degenerate.
(1) (2). Let be the decomposition of into connected components (Chap. IV, , no. 9), and let be the corresponding decomposition of . The space can be identified with the direct sum of the spaces , and can be identified with the direct sum of the corresponding forms . We are thus reduced to the case when is irreducible. Since is assumed to be finite, is a semi-simple W-module (Appendix, Prop. 2). By the Cor. to Prop. 5, it follows that is absolutely simple. Let be a positive non-degenerate form on , and let be the sum of its transforms under . Since is invariant under , it is proportional to (, no. 1, Prop. 1); since for all , the coefficient of proportionality is , and since is positive so is , which proves (2).
(2) (1). If is positive non-degenerate, the orthogonal group is compact (Integration, Chap. VII, , no. 1). Since is a discrete subgroup of (Cor. 3 of Th. 1), it follows that is finite, hence so is . Q.E.D.
The following result has been established in the course of the proof:
Corollary
If* is irreducible and finite, E is an absolutely simple W-module.
The finiteness criterion provided by Th. 2 permits the classification of all finite Coxeter groups (cf. Chap. VI, ). We restrict ourselves here to the following preliminary result:
Proposition 8
If* is finite, the graph of is a forest (Chap. IV, Appendix).
If not, this graph would contain a circuit , . Putting , , and , this means that for all . Let
Corollary
If* is irreducible and finite, its graph is a tree.
Indeed, a connected forest is a tree.
Comparison with the results of § 3.
First of all let be a finite Coxeter group. Denote by the form ; by Th. 2, this is a scalar product on . For all , let be the hyperplane associated to the orthogonal reflection , and let be the family of hyperplanes , for , . Let be the set of such that for all . Finally, identify (by means of ) with a subgroup of the orthogonal group of the space .
Proposition 9
With the preceding notations*, is the subgroup of generated by the reflections with respect to the hyperplanes of . It is an essential group (, no. 7) and is a chamber of relative to .
The first assertion is trivial. On the other hand, if is invariant under , it is orthogonal to all the , and hence is zero; this shows that is essential. Finally, the isomorphism defined by transforms to the set of no. 4; the property proved there shows that, for all
The roots of the equation are 1 and . Thus and consequently . Note also that , hence
and all , does not meet . It follows that is contained in the complement of the union of the hyperplanes of , and since is connected, open and closed in , it is a chamber of relative to . Q.E.D.
We can apply to and all the properties proved in § 3. In particular, is a fundamental domain for the action of on (in other words, the cone defined in no. 6 is equal to the whole of ).
Conversely, let be a finite dimensional real vector space, provided with a scalar product and let be an essential finite group of displacements of leaving 0 fixed; assume that is generated by reflections. Let be a chamber of with respect to (cf. § 3), and let be the set of orthogonal reflections relative to the walls of . Then is a finite Coxeter system (, no. 2, Th. 1). Moreover, if , denote by the wall of corresponding to , and denote by the unit vector orthogonal to and on the same side of as . If denotes the Coxeter matrix of , Props. 3 and 7 of show that
and that the form a basis of . *The natural representation of on can thus be identified with the representation of no. 3.
9. Case in which is positive and degenerate
In this number, we assume that is finite, that is irreducible, and that the form is positive and degenerate.
Lemma 2
The orthogonal complement of for is of dimension 1; it is generated by an element with for all .
This follows from Lemma 4 of , no. 5, applied to the matrix with entries .
Let be the vector satisfying the conditions of Lemma 2 and such that , and let be the affine hyperplane of consisting of the such that . If denotes the orthogonal complement of in , has a natural structure of an affine space with space of translations . Moreover, the form defines by passage to the quotient a non-degenerate scalar product on , hence also on its dual ; this gives a euclidean structure on the affine space (Algebra, Chap. IX, § 6, no. 6).
Let be the subgroup of consisting of the automorphisms leaving and invariant; if , the contragredient map leaves and stable, and defines by restriction to a displacement of (cf. § 3). It is immediate that this gives an isomorphism from to the group of displacements of . Moreover, the stabiliser of a point of can be identified with the orthogonal group of the Hilbert space T and is thus compact. On the other hand, G is a locally compact group countable at infinity and A is a Baire space: it follows (Integration, Chap. VII, Appendix, Lemma 2) that the map defines a homeomorphism from to A. Thus G acts properly on A (General Topology, Chap. III, § 4, no. 2, Prop. 5). Since W is a subgroup of G, it can be identified with a group of displacements of A. We are going to show that this group satisfies the assumptions of § 3. More precisely:
Proposition 10
The group W provided with the discrete topology acts properly on A; it is generated by orthogonal reflections; it is infinite, irreducible and essential* (§ 3, no. 7). The intersection C ∩ A is a chamber of A for W. If Ls denotes the hyperplane of A formed by the intersection of A with the hyperplane of E* orthogonal to e_s, the L_s for s ∈ S are the walls of C ∩ A. If is the unit vector of T orthogonal to L_s on the same side of L_s as C ∩ A, then (for s, t ∈ S) and the Coxeter matrix of W (§ 3, no. 4) is identified with M.
By Cor. 3 of Th. 1, W is discrete in , and hence in G, and acts properly on A. Let . Since Card S ≥ 2, the hyperplane of E* orthogonal to is not orthogonal to v and its intersection L_s with A is indeed a hyperplane. The displacement corresponding to s is thus a displacement of order 2 leaving fixed all the points of L_s: it is necessarily the orthogonal reflection associated to L_s. It follows that W is generated by orthogonal reflections. Th. 2 now shows that it is infinite and Prop. 7 that it is essential and irreducible.
Since C is an open simplicial cone, whose walls are the hyperplanes with equations (for ), the intersection C ∩ A is a convex, hence connected, open and closed subset of the complement of the union of the L_s in A. Moreover, C ∩ A is non-empty, for if we have and . It follows that C ∩ A is a chamber of A relative to the system of the L_s. Moreover, for all (cf. no. 4, property (P_n)) and it follows that C ∩ A is a chamber of A relative to the system consisting of the transforms of the L_s by the elements of W; by the Cor. of Th. 1 of § 3, no. 2, it follows that C ∩ A is a chamber of A relative to W.
Let be the vertex of the simplex C ∩ A not in L_s. We have
for and , and
Let be the vector in T defined by the relations:
The vector is orthogonal to and is on the same side of the hyperplane as . Moreover
which shows that is the image of the class of under the isomorphism from to given by the quadratic form . It follows that
Consequently is a unit vector and the last assertion of Prop. 10 is proved.
The euclidean affine space provided with the group is called the space associated to the Coxeter matrix and we denote it by .
Proposition 10 admits a converse:
Proposition 11
Let be a group of displacements of a euclidean affine space , satisfying the assumptions of § 3. Assume that is infinite, essential and irreducible. Then the form attached to the Coxeter matrix of is positive degenerate and there exists a unique isomorphism from the affine space associated to to , commuting with the action of . This isomorphism transforms the scalar product of into a multiple of the scalar product of .
Let be a chamber of and let be the set of orthogonal reflections with respect to the walls of . If denotes the unit vector orthogonal to the hyperplane associated to the reflection and on the same side of as (§ 3, Prop. 3), the form is such that for . It is thus positive. Since the are linearly dependent (§ 3, no. 9, Prop. 8), it is degenerate.
We can thus apply the preceding constructions to . With the same notation as above, and there exists a unique isomorphism of Hilbert spaces from to the space of translations of such that . Let and be two distinct vertices of and the reflection in such that . Put and let be the affine bijection from to defined by
It is then immediate that for all and that transforms the scalar product of into a multiple of that on . It follows at once that commutes with the action of . Finally, the uniqueness of is evident, for for example is the unique point of invariant under the reflections .
Exercises
In the exercises below, denotes a Coxeter system. We assume that is finite; its cardinal is called the rank of . We identify with a subgroup of by means of (cf. nos. 3 and 4).
See the exercises for § 4.