Lemma 8

Let E be a finite-dimensional complex Hilbert space of dimension $n$.

a) The groups $\mathbf{S}\mathbf{U}(E)$ and $\mathbf{U}(E)$ are connected;

b) The derived group of $\mathbf{S}\mathbf{U}(E)$ is equal to $\mathbf{S}\mathbf{U}(E)$;

c) The derived group of $\mathbf{U}(E)$ is equal to $\mathbf{S}\mathbf{U}(E)$.

We may suppose that $n\geqslant 1$ and that $E =\mathbf{C}^n$.

Let A be the subgroup of $\mathbf{U}(E)$ formed by the diagonal matrices; it is homeomorphic to $\mathbf{U}^n$, and therefore connected (TG, VI, p. 11, cor. 2 and I, p. 83, prop. 8). Its intersection with $\mathbf{S}\mathbf{U}(E)$ is homeomorphic to $\mathbf{U}^{n-1}$, and is therefore likewise connected.

By Theorem 1 of IV, p. 149, the group $\mathbf{U}(E)$ (resp. $\mathbf{S}\mathbf{U}(E)$) is the union of the connected subsets $gAg^{-1}$ for $g\in G$ (resp. of the connected subsets $g(A\cap \mathbf{S}\mathbf{U}(E))g^{-1}$); since the identity element belongs to each of these sets, the space $\mathbf{U}(E)$ (resp. $\mathbf{S}\mathbf{U}(E)$) is connected (TG, I, p. 81, prop. 2).

Let us prove b). The assertion is true when $n= 1$, since $\mathbf{S}\mathbf{U}(E)$ is then reduced to the identity element. Suppose therefore that $n\geqslant 2$. The Lie algebra of $\mathbf{S}\mathbf{U}(E)$ is then simple (LIE, IX, p. 20, § 3, No.$^o4$) and hence the derived group of $\mathbf{S}\mathbf{U}(E)$ has finite index in $\mathbf{S}\mathbf{U}(E)$. The result follows, since $\mathbf{S}\mathbf{U}(E)$ is connected by a).

Assertion c) follows from b), since the derived group of $\mathbf{U}(E)$ is contained in $\mathbf{S}\mathbf{U}(E)$.

Let us now prove Theorem 3. Let $n$ denote the dimension of the Hilbert space E. Since the representation $\pi$ is faithful, we may suppose that G is a compact subgroup of $\mathbf{U}(E)$. The group G is closed in the real Lie group $\mathbf{U}$(E), and is therefore a compact real Lie group (LIE, III, §8, No.$^o2$, Theorem 2). By hypothesis, G is infinite, and therefore of dimension $\geqslant 1$. Let $\mathfrak{g}$ be its Lie algebra; it is non-zero.

Let us prove a). Suppose that the derived group D(G) is infinite. We have the G-invariant decomposition End(E) $=\mathbf{C}1_E\oplus \mathfrak{s}\mathfrak{l}(E)$. The representations of G on $\mathbf{C}1_E$ and on $\mathfrak{s}\mathfrak{l}(E)$ are not isomorphic, since dim $\mathfrak{s}\mathfrak{l}(E)\geqslant 2$. By Lemma 5, we therefore have $M_4(\pi )\geqslant 2$ with equality if and only if the representation of G in $\mathfrak{s}\mathfrak{l}(E)$ is irreducible. Now, the derived group of G is contained in $\mathbf{S}\mathbf{L}$(E), and the complexification of the Lie algebra of D(G) may therefore be identified with a subspace of $\mathfrak{s}\mathfrak{l}$(E), which is a subrepresentation of the representation of G in $\mathfrak{s}\mathfrak{l}(E)$ (Lemma 7). This subrepresentation is non-zero, since D(G) is infinite. We therefore have $M_4(\pi ) = 2$ if and only if $(\mathscr{D}\mathfrak{g})_{(\mathbf{C})}=\mathfrak{s}\mathfrak{l}(E)$. Since D(G) is contained in $\mathbf{S}\mathbf{U}$(E), and since $\mathbf{S}\mathbf{U}(E) = D(\mathbf{S}\mathbf{U}(E))$ (Lemma 8), this condition is equivalent to $\mathbf{S}\mathbf{U}(E)\subset G$.

For the proof of assertion b), we resume the notation of LIE, VIII, §13, No.$^o$ 2–4.

Suppose that $\pi$ is of symplectic type and that dim(E) $\geqslant 3$. Let $q$ be a non-zero G-invariant alternating bilinear form on E; it is nondegenerate (LIE, IX, p. 103, App. 2, prop. 3) and G is a compact subgroup of the symplectic group $\mathbf{S}\mathbf{p}(q)$. Let us denote by $q^*\in \wedge^2E$ the element with which the alternating form $q$ is identified. The complexified Lie algebra $\mathfrak{g}_{(\mathbf{C})}$ of G is contained in $\mathfrak{s}\mathfrak{p}(q)$. Now, since dim(E) $\geqslant 3$, the representation of $\mathfrak{s}\mathfrak{p}(q)$ in $E\otimes E$ admits the direct-sum decomposition

where $E_2$ is the second fundamental representation of $\mathfrak{s}\mathfrak{p}(q)$ (LIE, VIII, p. 202, §13, No.$^o3$, (IV)). The representations $E_2$ and $\mathbf{C}q^*$ of $\mathfrak{s}\mathfrak{p}(q)$ are irreducible (loc. cit.) and the representation $\mathsf{S}^2(E)$ is non-zero.

After lemma 5, one therefore has $M_4(\pi )\geqslant 3$ with equality if and only if the representations of G in $\mathsf{S}^2$(E), $E_2$ and $\mathbf{C}q^*$ are irreducible and pairwise non-isomorphic.

Suppose that $M_4(\pi ) = 3$. Since the representation $\mathsf{S}^2(E)$ is identified with the adjoint representation of $\mathfrak{s}\mathfrak{p}(q)$ (lemma 6), it contains as subrepresentation the complexification of the adjoint representation of G (lemma 7). One therefore has $\mathfrak{g}_{(\mathbf{C})}=\mathfrak{s}\mathfrak{p}(q)$.

Conversely, suppose that $\mathfrak{g}_{(\mathbf{C})}=\mathfrak{s}\mathfrak{p}(q)$. The adjoint representation of $\mathfrak{s}\mathfrak{p}(q)$ and the representation $E_2$ of $\mathfrak{s}\mathfrak{p}(q)$ are irreducible, of dimensions $n(n+ 1)/2$ and $n(n-1)/2-1$ respectively (LIE, VIII, p. 202, §13, No.$^o3$, (IV)), which are different and $\geqslant 2$ since $n\geqslant 3$, whence $M_4(\pi ) = 3$.

Finally, suppose that $\pi$ is of real type and that dim(E) $\geqslant 3$ and dim(E) $\not = 4$. Let $q$ be a non-zero symmetric bilinear form, G-invariant on E; it is separating (LIE, IX, p. 103, App. 2, prop. 3) and G is a compact subgroup of the orthogonal group $\mathbf{O}(q)$. Let us denote by $q^*\in \mathsf{S}^2(E)$ the element with which $q$ is identified.

The complexified Lie algebra $\mathfrak{g}_{(\mathbf{C})}$ of G is contained in $\mathfrak{s}\mathfrak{o}(q)$. The representation of $\mathfrak{s}\mathfrak{o}(q)$ in $E\otimes E$ admits the decomposition into a direct sum

where $\mathsf{S}^2_0(E)$ is the orthogonal of $q^*$ in $\mathsf{S}^2(E)$. These representations are of dimension at least 2 since dim(E) $\geqslant 3$. After lemma 5, one has $M_4(\pi )\geqslant 3$ with equality if and only if the representations of G in $\wedge^2E$ and $\mathsf{S}^2_0(E)$ are irreducible and non-isomorphic.

Suppose that $M_4(\pi ) = 3$. Since the representation $\wedge^2E$ is identified with the adjoint representation of $\mathfrak{s}\mathfrak{o}(q)$ (lemma 6), it contains as subrepresentation the complexification of the adjoint representation of G (lemma 7). The condition $M_4(\pi ) = 3$ therefore implies that $\mathfrak{g}_{(\mathbf{C})}=\mathfrak{s}\mathfrak{o}(q)$.

Conversely, suppose that $\mathfrak{g}_{(\mathbf{C})}=\mathfrak{s}\mathfrak{o}(q)$. The adjoint representation of $\mathfrak{s}\mathfrak{o}(q)$ is irreducible, since dim(E) $\not = 4$ (LIE, VIII, § 13, p. 193, (I) and p. 206, (I)), and of dimension $n(n-1)/2$. The representation $\mathsf{S}^2_0(E)$ of $\mathfrak{s}\mathfrak{o}(q)$ has highest weight $2\varpi_1$. By comparing its dimension with that of the irreducible representation of $\mathfrak{s}\mathfrak{o}(q)$ of highest weight $2\varpi_1$, calculated by Weyl's formula (cf. LIE, VIII, §9, n$^o2$, th. 2 and LIE, VI, plates II and IV), one verifies that $\mathsf{S}^2_0(E)$ is irreducible. It follows therefore that $M_4(\pi ) = 3$ if $\mathfrak{g}_{(\mathbf{C})}=\mathfrak{s}\mathfrak{o}(q)$.