Proposition 10

Let G be a topological group. Let $\pi$ be an irreducible unitary representation of G in a complex Hilbert space $E_{\pi}$ and let $\varrho$ be a unitary representation of G in a complex Hilbert space E.

There exist families $(E_i)_{i\in I}$ of closed invariant subspaces of E such that the subrepresentation of $\varrho$ in $E_i$ is isomorphic to $\pi$ for every $i\in I$ and such that $M_{\pi}(\varrho )$ is the Hilbert sum of the spaces $E_i$. Moreover, the cardinal of I is independent of the family $(E_i)_{i\in I}$ satisfying these properties.

Let us prove the existence of families satisfying the indicated conditions. Let $\mathscr{O}$ be the set of subsets C of Hom$_G(\pi , \varrho )-\{0\}$ such that the subspaces $u(E_{\pi})$ for $u\in C$ are pairwise orthogonal. The set $\mathscr{O}$ is ordered by inclusion. It is nonempty since the empty set is one element, and it is of finite character (E, III, p. 34, Def. 2) since C belong to $\mathscr{O}$ if and only if the sets containing two elements of C belong to $\mathscr{O}$. By E, III, p. 35, Theorem 1, there exists a maximal element C of $\mathscr{O}$. Let F be the closure of the subspace generated by the spaces $u(E_{\pi})$ for $u\in C$; it is the Hilbert sum of the spaces $u(E_{\pi})$ for $u\in C$. We shall prove that $F = M_{\pi}(\varrho )$, which will prove that the family $(u(E_{\pi}))_{\pi\in C}$ has the required properties.

By definition, $u(E_{\pi})\subset M_{\pi}(\varrho )$ for every $u\in C$, hence F is contained in $M_{\pi}(\varrho )$. To prove the converse inclusion, it is enough to prove that if $v$ is a G-morphism of $\pi$ into $\varrho$, then its image is contained in F. Let $p$ be the orthogonal projector of E with image $F^{\circ}$; it is a G-morphism, since F is a subrepresentation of E (Proposition 4 of V, p. 383). The image of the G-morphism $p\circ v$ is orthogonal to F; it is therefore zero (otherwise $C\cup \{p\circ v\} \in \mathscr{O}$, which contradicts the maximality of C), and consequently the image of $v$ is contained in F.

Let now $(E_i)_{i\in I}$ and $(F_j)_{j\in J})$ be families of closed invariant subspaces of E, pairwise orthogonal, such that the subrepresentation of G in $E_i$ (resp. in $F_j$) is isomorphic to $\pi$ for every $i\in I$ (resp. for every $j\in J$), and such that $M_{\pi}(\varrho )$ is the Hilbert sum of the family $(E_i)_{i\in I}$ and of the family $(F_j)_{j\in J}$.

If I is finite, then

dim Hom$_G(\pi ,M_{\pi}(\varrho )) =$ dim Hom$_G(E_{\pi},\bigoplus_{i\in I}E_i)=$ Card(I)

(formula (1) of V, p. 377 and Corollary 2 of V, p. 387). For every finite subset L of J, one then has

Card(L) = dim Hom$_G(E_{\pi},\bigoplus_{i\in L}F_j)$

$\leqslant$ dim Hom$_G(\pi ,M_{\pi}(\varrho )) =$ Card(I)

(loc. cit.). This proves that J is then finite and that Card(I) = Card(J), as desired. Analogously, if J is finite, then I is finite and has the same cardinal.

Suppose now that I and J are infinite. For $j\in J$, let $p_j$ denote the orthogonal projector of E with image $F_j$. For $i\in I$, let $x_i$ be a nonzero element of $E_i$; it is a cyclic vector of $E_i$. Observe that since $p_j$ induces by passing to the subspaces a G-morphism of $E_i$ into $F_j$, the space $p_j(E_i)$ is zero if and only if $p_j(x_i) = 0$ (in fact, the space $E_i\cap$Ker($p_j$) is either zero, or equal to $E_i$).

For every $j\in J$, there exists an element $f(j)\in I$ such that $p_j(E_{f(j)})$ is not reduced to 0 (otherwise, the orthogonal projector $p_j$ would be zero on $M_{\pi}(\varrho )$). Thus a mapping $f$ of J into I has been defined. For

$-1$

every $i\in I$, the set $f(i)$ is countable. In fact, this set is contained in the set of $j\in J$ such that $p_j(E_i)$ is nonzero, that is, such that $p_j(x_i)\not = 0$. Now (EVT, V, p. 20, cor. 2), one has

therefore the set of $j\in J$ such that $p_j(x_i)\not = 0$ is countable. By E, III, p. 50, prop. 4, it follows that Card(J) = Card($f(J)$)$\leqslant$ Card(I). By interchanging the roles of I and J, one concludes that Card(I) = Card(J).