Exercise 17, § 3

Let $n\geqslant 2$ be an integer. Consider the real Lie group $G =\mathbf{R}^n\rtimes$ $\mathbf{S}\mathbf{L}(n,\mathbf{R})$, and denote by H the subgroup $\mathbf{R}^n\times \{e\}$ of G, identified with $\mathbf{R}^n$.

Let $\pi$ be a unitary representation of G in a complex Hilbert space E. Suppose that for every compact subset C of G and every $\varepsilon >0$, there exists $x\in E$ nonzero such that

sup$_{g\in C}\|\pi (g)x-x\|< \varepsilon \|x\|$. a) There exists a sequence $(x_m)$ of vectors of norm 1 in E such that the functions $\varphi_m(g) =\langle x_m|\pi (g)x_m\rangle$ converge to the constant function 1 in $\mathscr{C}(G)$ endowed with the topology of compact convergence. (Consider a sequence $(C_m)_{m\in\mathbf{N}}$ of compact subsets whose union is equal to G, and take $x_m$ satisfying the condition above for $(C_m,1/(m+ 1)).$)

b) For every $m\in \mathbf{N}$, there exists a positive measure $\mu_m$ of total mass 1 on $\mathbf{R}^n$ whose Fourier transform is identified with the restriction of $\varphi_m$ to H. c) Suppose that $\mu_m(\{0\}) = 0$ for every $m\in \mathbf{N}$. Then identify $\mu_m$ with a positive measure of total mass 1 on $\mathbf{R}^n-\{0\}$, and denote by $\nu_m$ the image measure of $\mu_m$ on the projective space $\mathbf{P}_{n-1}(\mathbf{R})$. There exists a subsequence $(\nu_{n_k})_{k\in\mathbf{N}}$ converging weakly to a positive measure $\nu$ of total mass 1 on $\mathbf{P}_{n-1}(\mathbf{R})$; the measure $\nu$ is $\mathbf{S}\mathbf{L}(n,\mathbf{R}$)-invariant. Deduce a contradiction.

d) Let $m\in \mathbf{N}$ be such that $\mu_m(\{0\})>0$. Let $\psi$ be the restriction of $\varphi_m$ to H. There exists $c >0$ such that $\psi =c+\widetilde{\psi}$ where $\widetilde{\psi}$ is a function of positive type on $\mathbf{R}^n$. e) The representation $\pi$ contains H-invariant vectors. (Use exercise 2.) One says that the pair $(G,H)$ has relative property ]T[.