Proposition 11

Let $\varphi \in$ Pos$_1(G)$ and let $(\varrho , x)$ be a cyclic Hilbertian realization of $\varphi$. Then $\varrho$ is an irreducible representation of G if and only if $\varphi$ is an extreme point (EVT, II, p. 57, def. 1) of Pos$_1(G)$.

Let E be the space of $\varrho$. Suppose first that $\varrho$ is not irreducible. Let F be a nonzero closed subspace of E, stable under $\varrho$, and different from E. Let us write $x=x_1+x_2$, where $x_1\in F$ and $x_2\in F^{\circ}$. Then one has $1 =\|x\|^2=\|x_1\|^2+\|x_2\|^2$. The subrepresentation of E generated by $x_1$ is contained in F, hence $x_1\not =x$ since $x$ is a cyclic vector of $\varrho$, whence $x_2\not = 0$. Analogously one verifies that $x_1\not = 0$.

For $j= 1$ and $j= 2$, let us denote by $\varphi_j$ the continuous function on G such that

for all $g\in G$. One has $\varphi_j\in$ Pos$_1(G)$(V, p. 443, example). Since $\varphi =\|x_1\|^2\varphi_1+\|x_2\|^2\varphi_2$, it is enough to verify that $\varphi_1\not =\varphi_2$ in order to prove that $\varphi$ is not an extreme point of Pos$_1(G)$; for this it is enough to prove that $\varphi \not =\varphi_1$.

Arguing by contradiction, suppose that $\varphi =\varphi_1$. Since one has $\langle x_1|\varrho (g)x_2\rangle = 0$ for all $g\in G$, it would follow that

for all $g\in G$, whence $\langle x_1|y\rangle =\langle \|x_1\|^2x|y\rangle$ for every element $y$ of the vector subspace of E generated by the elements $\varrho (g)x$ for $g\in G$, hence for all $y\in E$ since $x$ is a cyclic vector of $\varrho$. This would imply that $x_1=\|x_1\|^2x$ is also a cyclic vector of $\varrho$, which is a contradiction, whence the assertion.

Now suppose that $\varrho$ is irreducible, and prove that $\varphi$ is an extreme point of Pos$_1(G)$. Let $\varphi_1\not =\varphi_2$ be elements of Pos$_1(G)$ and let $t_1,t_2\in [0,1]$ be such that $t_1+t_2= 1$ and $\varphi =t_1\varphi_1+t_2\varphi_2$. For $j\in \{1,2\}$, let $(\varrho_j, x_j)$ denote a cyclic Hilbert realization of $\varphi_j$, and let $E_j$ denote the space of $\varrho_j$. Let $x_3=t^{1/2}_1x_1+t^{1/2}_2x_2$. Then $(\varrho_1\oplus \varrho_2, x_3)$ is a Hilbert realization of $\varphi$. Since $(\varrho , x)$ is cyclic, there exists an isometric G-morphism $u: E\rightarrow E_1\oplus E_2$ such that $u(x) =x_3$ (Prop. 10).

Let $j= 1$ or $j= 2$. Since $\varrho$ is irreducible, there exists $\lambda_j\geqslant 0$ such that the G-morphism $u_j=$ pr$_j\circ u$ of E into $E_j$ satisfies

for all $y$ and $y'$ in E (Cor. 5 of V, p. 388, if $u_j\not = 0$, and one may take $\lambda_j= 0$ otherwise). For all $g\in G$, one has

Since $\varphi_j(e) =\varphi (e) = 1$, it follows that $\varphi_j=\varphi$ if $t_j\not = 0$. The assumption $\varphi_1\not =\varphi_2$ therefore implies that $t_1$ or $t_2$ must be zero, which proves that $\varphi$ is an extreme point of Pos$_1(G)$.