Exercise 13, § 3

Let $\varrho$ be an irreducible unitary representation of G in a Hilbert space E, and $\pi$ a unitary representation of G in F. Let $x\in E$ be of norm 1. Suppose that $\varrho \leqslant \pi ($cf. exercise 11).

Let $A\subset \mathscr{C}(G)$ denote the set of normalized diagonal matrix coefficients of $\pi$, and $\widetilde{A}$ the closed convex hull of A in $L^{\infty}(G)$ for the weak topology.

Let $x\in E$ be of norm 1 and $f:g\mapsto \langle x|\varrho (g)x\rangle$ the associated diagonal matrix coefficient.

a) The set $\widetilde{A}$ is a compact convex set in $L^{\infty}(G)$ and $f$ is an extreme point of $\widetilde{A}$.

b) The function $f$ belongs to the closure of A in $\widetilde{A}$ for the weak topology.

c) The function $f$ belongs to the closure of A in $\mathscr{C}(G)$ endowed with the topology of compact convergence.

d) The trivial representation of G in $\mathbf{C}$ is weakly contained in $\pi$ if and only if, for every compact subset C of G and every real $\varepsilon >0$, there exists $x\in F$ of norm 1 such that

sup$_{g\in C}\|\pi (g)x-x\|< \varepsilon$.