Proposition 1

— Let G be a subgroup of Aut(E). The following three properties are equivalent:

(i) G is relatively compact in End(E); (ii) G is relatively compact in Aut(E); (iii) G leaves invariant a nondegenerate$^1$ positive hermitian form on E.

(iii) $\Rightarrow$ (i): Suppose that $G$ leaves invariant a nondegenerate positive hermitian form $\Psi$. Let $(e_1, \ldots, e_n)$ be an orthonormal basis for $\Psi$ (Alg., Ch. IX, § 6, No. 1, Cor. 1 of Th. 1). For every $u \in G$, let $(u_{ij})$ be its matrix with respect to $(e_i)$. For any $j$, we have $\sum_{i=1}^n |u_{ij}|^2 = 1$, thus $|u_{ij}| \leq 1$ for all $i$ and $j$, which proves (i).

(i) $\Rightarrow$ (ii): This follows from GT, X, § 3, No. 5, Cor. of Th. 4, taking into account the fact that the topology of $\mathrm{End}(E)$ is that of compact convergence.

(ii) $\Rightarrow$ (iii): Suppose that the closure $\overline{G}$ of $G$ in $\mathrm{Aut}(E)$ is compact. Let $\Phi$ be a nondegenerate positive hermitian form on $E$. If the field of scalars is $\mathbf{R}$ or $\mathbf{C}$, the giving of $\Phi$ makes $E$ a finite-dimensional Hilbert space, and condition (iii) will result from the following lemma:

$^1$ Non dégénérée; in EVT, the term is replaced by séparante, subsequently translated as "separating" (TVS, V, § 1, No. 1).