DISCRETE DECOMPOSABILITY OF THE RESTRICTION OF

A_{𝔮}(λ) WITH RESPECT TO REDUCTIVE SUBGROUPS II

— micro-local analysis and asymptotic K-support

Toshiyuki KOBAYASHI

University of Tokyo

Abstract. Let G′ ⊂ G be real reductive Lie groups. This paper offers a criterion on the triplet (G, G′, π) that the irreducible unitary representation π of G splits into a discrete sum of irreducible unitary representations of a subgroup G′ when restricted to G′, each of finite multiplicity. Furthermore, we shall give an upper estimate of the multiplicity of an irreducible unitary representation of G′ occurring in π|_{G′}.

§1. Introduction

\underline{1.1.} Our object of study is the restriction of an irreducible unitary representation of a real reductive Lie group with respect to its reductive subgroup with focus on a discrete decomposition with finite multiplicity.

\underline{1.2.} Let G be a real reductive linear Lie group, and G′ a subgroup that is reductive in G. We write Ĝ (resp. Ĝ′) for the unitary dual of G (resp. G′). If π ∈ Ĝ, then the restriction π|_{G′} is decomposed into the direct integral of irreducible unitary representations of G′:

π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{G′}(τ : π|_{G′}) τ dμ(τ),

where dμ is a Borel measure on Ĝ′ and m_{G′}(· : π|_{G′}): Ĝ′ → ℕ ∪ {∞} is the multiplicity defined almost everywhere with respect to dμ.

We say that the restriction π|_{G′} is G′-admissible if the measure dμ is discrete and if the multiplicity m_{G′}(τ : π|_{G′}) < ∞ for all τ ∈ Ĝ′. It is a famous result due to Harish-Chandra that this is always the case if G′ is a maximal compact subgroup of G. Some other examples of the admissible restriction π|_{G′} were previously known in the setting where π is a highest weight module such as the Segal-Shale-Weil representation (cf. [4]) and a holomorphic discrete series representation (cf. [22], [21],

Typeset by AMS-TEX

1991 Mathematics Subject Classification. Primary 22E46; Secondary 32A45, 22E45.

[7]) for suitable G′. In [13], we gave a sufficient condition for the G′-admissibility of the restriction π|_{G′} under the assumption that

π  : a Zuckerman-Vogan’s derived functor module A_{𝔮}(λ),
(G, G′) : a reductive symmetric pair.

A distinguishing feature in this setting is that G′ is not compact and A_{𝔮}(λ) is not always a highest weight module. In this paper, the above setting is generalized to:

π : any irreducible unitary representation of G,
G′ : any closed subgroup that is reductive in G.

Furthermore, we give an upper estimate of the multiplicity m_{G′}(τ : π|_{G′}).

Our approach, based on micro-local analysis, is different and simpler in the special setting that was treated in the first part [13], and only a complete replacement of the machinery gave the base for such a generalization. Conversely, a necessary condition for discrete decomposability will be discussed in the third part [16] based on the study of the associated varieties by purely algebraic methods.

\underline{1.3.} Our main results (see Theorem 2.9) are interesting not only from representation theoretic view points (e.g. leading us to an analog of Howe’s dual pair correspondence of the minimal unipotent representation of SO(p, q) [19]), but also give new applications in other branches of mathematics such as:

1) Vanishing theorem of modular symbols defined by the arithmetic quotients of Riemannian symmetric spaces, which especially solves affirmatively a conjecture of T. Oda in connection with the theory of automorphic forms [18].

2) Construction of new discrete series representations for non-symmetric pseudo-Riemannian homogeneous manifolds of reductive type [15].

3) The finite dimensionality theorem of the space of the global solutions of certain system of partial differential equations (e.g. a generalization of the Gauss-Aomoto-Gelfand hypergeometric equations to higher order) by using a (generalized) Penrose construction (see [24], §7).

\underline{1.4.} Our idea goes back to fundamental results ([10], [5]) of Kashiwara, Vergne and Howe in the late ’70s, which are based on micro-local analysis (e.g. [23], [6], [9]). The novelty of our work is to establish a framework of “the admissible restriction” with a number of new examples of interest, which rely on the breakthrough in the theory of the unitary dual in the ’80s (e.g. [27], [28], [31], [2]).

\underline{1.5.} The main results of this paper were announced in [14]. I am grateful to R. Howe and M. Kashiwara for helpful discussions. I also expresses my hearty thanks to the the staff and the organizers, in particular, M. Flensted-Jensen and G. ’Olafsson, of the special year on “Analysis on Lie groups” at the Mittag-Leffler Institute, where most part of the manuscript was prepared.

§2. Asymptotic K-support and the restriction of representations.

\underline{2.1.} In this section, building on the work of Kashiwara and Vergne [10] (see also [5]), we study the restriction of an infinite dimensional representation with respect to a reductive subgroup from the micro-local point of view. Furthermore, we employ the minimal K-type theory of Vogan [25] to obtain an upper estimate of the multiplicity of each irreducible component occurring in the restriction formula. The main result is Theorem 2.9.

\underline{2.2.} Let K be a Lie group with Lie algebra 𝔨_{0}. We give a trivialization of the tangent bundle T K by the right translation:

K × 𝔨_{0} →^{∼} T K, (x, X) ↦ \frac{d}{dt}|_{t=0} exp(tX)x.

Then the cotangent bundle T^{∗}K is naturally identified with K × 𝔨_{0}^{∗}. With these identifications, the translation L_{a}R_{b}: K → K, x ↦ axb (a, b ∈ K) induces bundle isomorphisms:

(L_{a}R_{b})_{∗}: T K → T K, (x, X) ↦ (axb, Ad(a)X),
(L_{a}R_{b})^{∗}: T^{∗}K → T^{∗}K, (axb, λ) ↦(x, Ad^{∗}(a)^{−1}λ).

\underline{2.3.} Suppose that K is a connected compact Lie group. We write 𝔨 = 𝔨_{0} ⊗_{ℝ} ℂ for the complexification of 𝔨_{0}. We fix a Cartan subalgebra 𝔱_{0} of 𝔨_{0}, and a positive system Δ^{+}(𝔨, 𝔱). We write C̄ (⊂ \sqrt{−1}𝔱_{0}^{∗}) for the dominant Weyl chamber, Λ (⊂ \sqrt{−1}𝔱_{0}^{∗}) for the weight lattice and put

Λ_{+} := Λ ∩ C̄.

We fix an Ad^{∗}(K)-invariant inner product on \sqrt{−1}𝔨_{0}^{∗}, and then regard \sqrt{−1}𝔱_{0}^{∗} as a subspace of \sqrt{−1}𝔨_{0}^{∗}. We denote by |λ| the induced norm for λ ∈ \sqrt{−1}𝔱_{0}^{∗}.

The Cartan-Weyl theory classifies the equivalence classes of finite dimensional irreducible representations of K by means of highest weight vectors. This map will be denoted by

Λ_{+} →^{∼} K̂, λ ↦ (τ_{λ}, F (K, λ)).

The Peter-Weyl theorem gives the irreducible decomposition of the regular representation L^{2}(K) as K × K-modules:

L^{2}(K) ≃ ∑_{λ∈Λ_{+}}^{⊕}F (K, λ) ⊠ F (K, λ)^{∗}.

The Paley-Wiener type theorem characterizes functions φ on K in terms of the development φ = ∑_{λ∈Λ_{+}} φ̂_{λ} with φ̂_{λ} ∈ F (K, λ) ⊠ F (K, λ)^{∗} as follows:

(2.3.1) φ ∈ B(K) ⇔ sup_{λ∈Λ_{+}} ∥φ̂_{λ}∥_{L^{2}(K)} exp(−ε|λ|) < ∞ for any ε > 0,
(2.3.2) φ ∈ D′(K) ⇔ sup_{λ∈Λ_{+}} ∥φ̂_{λ}∥_{L^{2}(K)}(1 + |λ|^{n})^{−1} < ∞ for some n > 0,
(2.3.3) φ ∈ C^{∞}(K) ⇔ sup_{λ∈Λ_{+}} ∥φ̂_{λ}∥_{L^{2}(K)}(1 + |λ|^{n}) < ∞ for any n > 0,
(2.3.4) φ ∈ A(K) ⇔ sup_{λ∈Λ_{+}} ∥φ̂_{λ}∥_{L^{2}(K)} exp(ε|λ|) < ∞ for some ε > 0.

Here, A, B, and D′ denote the sheaf of analytic functions, (Sato’s) hyperfunctions, and (Schwartz’s) distributions, respectively.

\underline{2.4.} The singularity spectrum (or singular spectrum) S. S. φ of a hyperfunction φ ∈ B(K) is a closed cone in \sqrt{−1}T^{∗}K. The books [8], [9] have been an indispensable guide into the analytic background. We need:

Fact 2.4. (Theorem 4.5 in [10]). Let φ = ∑_{λ∈Λ_{+}} φ̂_{λ} ∈ B(K) and let Y be a closed cone in \sqrt{−1}𝔱_{0}^{∗} which is contained in C̄. Then the following conditions are equivalent:

i) S. S. φ ⊂ K × (− Ad^{∗}(K) · Y) (⊂ \sqrt{−1}T^{∗}K ≃ K × \sqrt{−1}𝔨_{0}^{∗}). ii) If a closed cone Y′ ⊂ C̄ satisfies Y ∩ Y′ = ∅, then there exist constants R_{Y′} > 0 and ε_{Y′} > 0 such that ∥φ̂_{λ}∥_{L^{2}(K)} ≤ R_{Y′} exp(−ε_{Y′}|λ|) for any λ ∈ Λ_{+} ∩ Y′.

Here, a closed cone Y in ℝ^{n} means that Y is a closed subset of ℝ^{n} \ {0} such that λY ⊂ Y for any λ > 0.

\underline{2.5.} Let S be a subset of a real vector space ℝ^{n}. Following [9], Definition 2.4.3, we define a closed cone S∞ in ℝ^{n} \ {0} by

S∞ := {y ∈ ℝ^{n} \ {0} : there exists a sequence (y_{n}, ε_{n}) ∈ S × ℝ_{+} such that lim_{n→∞} ε_{n}y_{n} = y and lim_{n→∞} ε_{n} = 0}.

The following lemma is straightforward.

Lemma 2.5. Let S be a subset of a real vector space ℝ^{n}, and Y a closed cone in ℝ^{n}. Then the following two conditions are equivalent.

i) S∞ ∩ Y = ∅. ii) There exists an open cone V containing Y such that S ∩V is relatively compact.

\underline{2.6.} There are several equivalent definitions of the singularity spectrum of a hyperfunction (see [3] and references therein). One of them is given by the support of the Fourier transform, which is also the idea of the analytic wave front set for distributions (e.g. [6]). That is, the Fourier transform f̃(x) := ∫_{ℝ^{n}} f(y) exp(\sqrt{−1}⟨x, y⟩) dy of a function f ∈ L_{loc}^{1}(ℝ^{n}) with the infra-exponential growth condition:

sup_{y∈ℝ^{n}} |f(y)| exp(−ε|y|) < ∞ for any ε > 0

has the singularity spectrum estimated by

S. S. f̃ ⊂ ℝ^{n} × \sqrt{−1}(Supp f)∞ (⊂ \sqrt{−1} T^{∗}ℝ^{n}).

Similar estimates hold for a hyperfunction on a compact Lie group K: Given φ = ∑_{λ∈Λ_{+}} φ̂_{λ} ∈ B(K) where φ̂_{λ} ∈ F (K, λ) ⊠ F (K, λ)^{∗}, we define

(2.6.1)  Supp(φ̂) := \{λ ∈ Λ_{+} : φ̂_{λ} ≠ 0\} ⊂ \sqrt{−1}𝔱_{0}^{∗}.

It seems that the following Lemma 2.6 (1) and Proposition 2.7 (1) are implicit in [10], but we prove them for the sake of completeness.

Lemma 2.6. 1) Retain the above notation. We have:

S. S. φ ⊂ K × (− Ad^{∗}(K) · (Supp(φ̂)∞)) (⊂ \sqrt{−1}T^{∗}K ≃ K × \sqrt{−1}𝔨_{0}^{∗}).

2) Furthermore, we assume ∥φ̂_{λ}∥_{L^{2}(K)} ∈ ℕ for any λ ∈ Λ_{+}. If Y ⊂ C̄ is a closed cone such that

S. S. φ ⊂ K × (− Ad^{∗}(K) · Y) ,

then we have Supp(φ̂)∞ ⊂ Y.

Proof. 1) We define a closed cone by Y := Supp(φ̂)∞ ⊂ \sqrt{−1}𝔱_{0}^{∗}. Let Y′ be an arbitrary closed cone of \sqrt{−1}𝔱_{0}^{∗} such that Y′ ∩ Y = ∅. By Lemma 2.5 (i) ⇒ (ii), we find an open cone V containing Y′ such that V ∩ Supp(φ̂) is relatively compact. In particular, Y′ ∩ Supp(φ̂) is a finite set because Supp(φ̂) (⊂ Λ_{+}) is discrete. Hence there exist R_{Y′} > 0 and ε_{Y′} > 0 such that ∥φ̂_{λ}∥_{L^{2}(K)} ≤ R_{Y′} exp(−ε_{Y′}|λ|) for any λ ∈ Λ_{+} ∩ Y′. Now the first statement follows from Fact 2.4 (ii) ⇒ (i).

2) If Supp(φ̂)∞ ⊄ Y, then we could find closed cones Y′ and Y′′ and an open cone V such that

∅ ≠ Y′′ ⊂ V ⊂ Y′ ⊂ C̄, (2.6.2)(a)

Y′′ ⊂ Supp(φ̂)∞, (2.6.2)(b)

Y′ ∩ Y ⊂ {0}. (2.6.2)(c)

It follows from Fact 2.4 (i) ⇒ (ii) that there exist R_{Y′} > 0 and ε_{Y′} > 0 such that ∥φ̂_{λ}∥_{L^{2}(K)} ≤ R_{Y′} exp(−ε_{Y′}|λ|) for any λ ∈ Λ_{+} ∩ Y′. Because ∥φ̂_{λ}∥_{L^{2}(K)} ∈ ℕ, we have

♯ (Supp(φ̂) ∩ Y′) < ∞.

In particular, Supp(φ̂) ∩ V is relatively compact. It follows from Lemma 2.5 (ii) ⇒ (i) that Y′′ ∩ Supp(φ̂)∞ = ∅. This contradicts to (2.6.2)(b) and Y′′ ≠ ∅. Hence, we have Supp(φ̂)∞ ⊂ Y. □

\underline{2.7.} Suppose (π, H) is a representation of K. The K-multiplicity is defined by

m_{K}(· : π): K̂ → ℕ ∪ {∞}, τ ↦ m_{K}(τ : π) := dim Hom_{K}(τ, π).

The asymptotic K-support of (π, H) (see [10]), denoted by AS_{K}(π), is defined by

(2.7.1)(a)  Supp_{K}(π) := {λ ∈ Λ_{+} : m_{K}(τ_{λ} : π) ≠ 0},
(2.7.1)(b)  AS_{K}(π) := Supp_{K}(π)∞.

Then AS_{K}(π) is a closed cone contained in C̄ ⊂ \sqrt{−1}𝔱_{0}^{∗} because Supp_{K}(π) ⊂ Λ_{+} and C̄ = Λ_{+}∞.

We say that m_{K}(· : π) is of infra-exponential growth, if

(2.7.2)  sup_{λ∈Λ_{+}} m_{K}(F (K, λ) : π) exp(−ε|λ|) < ∞ for any ε > 0;

and of polynomial growth, if there exist constants C and N such that

(2.7.3)  m_{K}(F (K, λ) : π) ≤ C(1 + |λ|)^{N} for any λ ∈ Λ_{+}.

Proposition 2.7. 1) Assume that m_{K}(· : π) is of infra-exponential growth (resp. of polynomial growth). Then the K-character of (π, H) given by

(2.7.4)  Θ_{π}^{K} := ∑_{τ∈K̂} m_{K}(τ : π) Trace τ

is well-defined as a hyperfunction on K with the singularity spectrum (resp. as a distribution on K with the analytic wave front set):

S. S. Θ_{π}^{K} ⊂ K × (Ad^{∗}(K) · AS_{K}(π)), (⊂ \sqrt{−1}T^{∗}K ≃ K × \sqrt{−1}𝔨_{0}^{∗}).

2) Conversely, with the same assumption in (1), if W ⊂ \sqrt{−1}C̄ is a closed cone such that S. S. Θ_{π}^{K} ⊂ K × Ad^{∗}(K) · W, then AS_{K}(π) ⊂ W.

Proof. 1) The first statement follows from (2.3.1). We put φ := \overline{Θ_{π}^{K}} the complex conjugate of Θ_{π}^{K} and φ̂_{λ} := m_{K}(τ_{λ} : π)\overline{Trace τ_{λ}} for λ ∈ Λ_{+}. Then we have:

φ = ∑_{λ∈Λ_{+}} φ̂_{λ} ∈ B(K), with φ̂_{λ} ∈ F (K, λ) ⊠ F (K, λ)^{∗},

Supp(φ̂) = Supp(ˆ\overline{Θ_{π}^{K}}) = Supp_{K}(π), (see (2.6.1) and (2.7.1)(a) for definition).

It follows from Lemma 2.6 (1) and definition (2.7.1)(b) that

S. S. Θ_{π}^{K} = − S. S. φ ⊂ K × Ad^{∗}(K)(Supp(φ̂)∞) = K × Ad^{∗}(K) · AS_{K}(π).

2) Because m_{K}(τ_{λ} : π) ∈ ℕ, φ satisfies the assumption of Lemma 2.6 (2). Therefore, it follows from S. S. φ = − S. S. Θ_{π}^{K} ⊂ K × (− Ad^{∗}(K) · W) that

AS_{K}(π) = Supp_{K}(π)∞ = Supp(φ̂)∞ ⊂ W.

Hence, we have proved the Proposition. □

\underline{2.8.} Suppose H is a closed subgroup of K. We write pr_{𝔨→𝔥}: 𝔨_{0}^{∗} → 𝔥_{0}^{∗} for the projection dual to the inclusion of Lie algebras 𝔥_{0} ↪ 𝔨_{0}. The same notation pr_{𝔨→𝔥} is used for the complexified map 𝔨^{∗} → 𝔥^{∗}. Let 𝔥_{0}^{⊥} := Ker(pr_{𝔨→𝔥}: 𝔨_{0}^{∗} → 𝔥_{0}^{∗}). For μ ∈ Ĥ, we put a subset of Λ_{+} ≃ K̂ by

Λ_{+}(μ) := {λ ∈ Λ_{+} : [τ_{λ}|_{H} : μ] ≠ 0} = Supp_{K}(L^{2}(K ×_{H} μ)).

Here, L^{2}(K ×_{H} μ) denotes the Hilbert space consisting of L^{2}-sections for the Kequivariant vector bundle K ×_{H} μ → K/H associated to μ ∈ Ĥ. We note that we have naturally a unitary representation of K on L^{2}(K ×_{H} μ) and that the left regular unitary representation of K on L^{2}(K) decomposes as follows:

L^{2}(K) ≃ _{\begin{matrix}∑ & ⊕ \\ μ∈Ĥ \end{matrix}}L^{2}(K ×_{H} μ) ⊗ μ^{∗} ≃ _{\begin{matrix}∑ & ⊕ \\ μ∈Ĥ \end{matrix}} (_{\begin{matrix}∑ & ⊕ \\ λ∈Λ_{+}(μ) \end{matrix}}τ_{λ} ⊗ Hom_{H}(τ_{λ}|_{H}, μ)) ⊗ μ^{∗}.

Theorem 2.8. Let H be a closed subgroup of a connected compact Lie group K. Let π be a representation of K. We assume

(2.8.1)  AS_{K}(π) ∩ \sqrt{−1} Ad^{∗}(K)𝔥_{0}^{⊥} = \{0\}.

1) Then we have

(2.8.2)  ♯ (Λ_{+}(μ) ∩ Supp_{K}(π)) < ∞,

for each μ ∈ Ĥ. If the K-multiplicity m_{K}(τ : π) < ∞ for any τ ∈ K̂, then the H-multiplicity m_{H}(μ : π|_{H}) := dim Hom_{H}(μ, π|_{H}) < ∞ for any μ ∈ Ĥ.

2) If the K-multiplicity function m_{K}(· : π) : K̂ → ℕ is of infra-exponential growth (see (2.7.2)), then so is the H-multiplicity function m_{H}(· : π|_{H}): Ĥ → ℕ. Furthermore, the restriction of Θ_{π}^{K} to an analytic submanifold H ↪ K is welldefined as a hyperfunction, and the resulting hyperfunction Θ_{π}^{K}|_{H} ∈ B(H) coincides with

Θ_{π|_{H}}^{H} := ∑_{μ∈Ĥ} m_{H}(μ : π|_{H}) Trace μ ∈ B(H).

In particular, Θ_{π|_{H}}^{H} has the singularity spectrum estimated by

S. S.(Θ_{π|_{H}}^{H}) ⊂ H × pr_{𝔨→𝔥} (Ad^{∗}(K) AS_{K}(π)) (⊂ \sqrt{−1}T^{∗}H ≃ H × \sqrt{−1}𝔥_{0}^{∗}).

3) If the K-multiplicity function m_{K}(· : π): K̂ → ℕ is of polynomial growth, then so is the H-multiplicity function m_{H}(· : π|_{H}): Ĥ → ℕ.

Proof. 1) Because Ad^{∗}(K)𝔥_{0}^{⊥} is a closed cone satisfying (2.8.1), we can find a closed cone T in \sqrt{−1}𝔱_{0}^{∗} such that

AS_{K}(π) ⋐ T ⊂ C̄, (2.8.3)(a)

T ∩ \sqrt{−1} Ad^{∗}(K)𝔥_{0}^{⊥} ⊂ \{0\}. (2.8.3)(b)

We write ∁(T^{∘}) for the complement in \sqrt{−1}𝔱_{0}^{∗} of the interior set T^{∘} of T. Because

AS_{K}(π) ∩ ∁(T^{∘}) = Supp_{K}(π)∞ ∩ ∁(T^{∘}) = ∅, there exists an open cone V containing

∁(T^{∘}) such that Supp_{K}(π)∩V is relatively compact by Lemma 2.5 (i)⇒(ii). Because

V ⊃ ∁(T^{∘}) ⊃ ∁T and because Supp_{K}(π) ⊂ Λ_{+} is discrete, we conclude that

(2.8.4)  ♯ (Supp_{K}(π) \ T) < ∞.

If φ = ∑_{λ∈Λ_{+}(μ)} φ̂_{λ} ∈ L^{2}(K ×_{H} μ), then φ is regarded as an L^{2}-function on K with value in the representation space of μ (∈ Ĥ) satisfying

φ(kh) = μ(h)^{−1}φ(k) (k ∈ K, h ∈ H).

In particular,

S. S. φ ⊂ \{(k, ξ) : Ad^{∗}(k)^{−1}ξ ∈ \sqrt{−1}𝔥_{0}^{⊥}\} ⊂ K × \sqrt{−1} Ad^{∗}(K)𝔥_{0}^{⊥}.

It follows from (2.8.3)(b) and from Fact 2.4 (i)⇒(ii) that there exist constants R_{T} > 0 and ε_{T} > 0 such that

(2.8.5)  ∥φ̂_{λ}∥_{L^{2}} ≤ R_{T} exp(−ε_{T} |λ|) for any λ ∈ Λ_{+} ∩ T.

In order that an estimate of the form (2.8.5) holds for any φ = ∑_{λ∈Λ_{+}(μ)} φ̂_{λ} ∈ L^{2}(K ×_{H} μ), we must have ♯ (Λ_{+}(μ) ∩ T) < ∞. Then, combining with (2.8.4) we have proved (2.8.2).

Assume furthermore that m_{K}(τ : π) < ∞ for any τ ∈ K̂. By definition, m_{K}(τ_{λ} : π) ≠ 0 iff λ ∈ Supp_{K}(π) and [τ_{λ}|_{H} : μ] ≠ 0 iff λ ∈ Λ_{+}(μ). Because (2.8.6)

m_{H}(μ : π|_{H}) = ∑_{λ∈Λ_{+}} m_{K}(τ_{λ} : π) [τ_{λ}|_{H} : μ] = ∑_{λ∈Supp_{K}(π)∩Λ_{+}(μ)} m_{K}(τ_{λ} : π) [τ_{λ}|_{H} : μ]

is a finite sum by (2.8.2), we have m_{H}(μ : π|_{H}) < ∞ for any μ ∈ Ĥ.

2) The embedding ι: H ↪ K defines the conormal bundle T_{H}^{∗}K → H by

T_{H}^{∗}K := Ker(ι^{∗}: T^{∗}K|_{H} → T^{∗}H) ≃ Ker ((H × 𝔨_{0}^{∗}) → (H × 𝔥_{0}^{∗})) ≃ H × 𝔥_{0}^{⊥}.

Then the natural projection

ρ: (H ×_{K} \sqrt{−1}T^{∗}K) \ \sqrt{−1}T_{H}^{∗}K → \sqrt{−1}T^{∗}H, (h, \sqrt{−1}ξ) ↦ (h, \sqrt{−1}ι^{∗}(ξ))

is identified with

ρ: H × \sqrt{−1}(𝔨_{0}^{∗} \ 𝔥_{0}^{⊥}) → H × \sqrt{−1}𝔥_{0}^{∗}, (h, \sqrt{−1}ξ) ↦ (h, \sqrt{−1}pr_{𝔨→𝔥}(ξ)),

respectively. It follows from Proposition 2.7 and (2.8.1) that

S. S. Θ_{π}^{K} ∩ \sqrt{−1}T_{H}^{∗}K ⊂ (K × Ad^{∗}(K) · AS_{K}(π)) ∩ (H × \sqrt{−1}𝔥_{0}^{⊥})
⊂ H × \{0\} (⊂ \sqrt{−1}T^{∗}K ≃ K × \sqrt{−1}𝔨_{0}^{∗}).

Therefore, it follows from [9], Theorem 3.1.3 that the restriction Θ_{π}^{K}|_{H} is welldefined as a hyperfunction on H and that its singularity spectrum is estimated by

S. S. (Θ_{π}^{K}|_{H}) ⊂ ρ ((H ×_{K} \sqrt{−1}T^{∗}K) ∩ S. S. Θ_{π}^{K})
⊂ H × pr_{𝔨→𝔥} (Ad^{∗}(K) · AS_{K}(π)) (⊂ \sqrt{−1}T^{∗}H ≃ H × \sqrt{−1}𝔥_{0}^{∗}).

Because Θ_{π}^{K} is an Ad(K)-invariant (especially, Ad(H)-invariant) hyperfunction on K, the restriction Θ_{π}^{K}|_{H} is also an Ad(H)-invariant hyperfunction on H. Therefore, there exists uniquely a function a : Ĥ → ℂ of infra-exponential growth such that

Θ_{π}^{K}|_{H} = ∑_{μ∈Ĥ} a(μ) Trace μ,

where the right side converges in the weak topology of B(H) ≃ A′(H). Hence, in order to complete the proof of the second statement, it suffices to show a(μ) = m_{H}(μ : π|_{H}) for each μ ∈ Ĥ, namely, to show

(2.8.7)  ∫_{H} Θ_{π}^{K}|_{H}(h) \overline{Trace μ(h)} dh = m_{H}(μ : π|_{H}),

where the Haar measure on a compact group H is normalized so that the volume of H is 1. Let us prove (2.8.7). We fix μ ∈ Ĥ once and for all. We put

(2.8.8)(a)  Θ_{1} := ∑_{λ∈Λ_{+}(μ)} m_{K}(τ_{λ} : π) Trace τ_{λ} ∈ B(K),
(2.8.8)(b)  Θ_{2} := ∑_{λ∉Λ_{+}(μ)} m_{K}(τ_{λ} : π) Trace τ_{λ} ∈ B(K).

Then we have Θ_{π}^{K} = Θ_{1} +Θ_{2}. In view of (2.8.2), the sum in (2.8.8)(a) is taken over the finite set Λ_{+}(μ) ∩ Supp_{K}(π). Therefore, Θ_{1} ∈ A(K) and we have

∫_{H} Θ_{1}|_{H}(h) \overline{Trace μ(h)} dh = ∑_{λ∈Λ_{+}(μ)∩Supp_{K}(π)} m_{K}(τ_{λ} : π) [τ_{λ}|_{H} : μ],

from the orthogonal relation of characters of compact groups. By (2.8.6), the proof of (2.8.7) will be complete if we show

(2.8.7′)  ∫_{H} Θ_{2}|_{H}(h) \overline{Trace μ(h)} dh = 0.

We choose τ′ ∈ K̂ such that [τ′|_{H} : μ] ≠ 0. We take a function φ_{0} ∈ τ′ ⊠ (τ′)^{∗} ⊂ A(K) satisfying

∫_{H} φ_{0}(h)\overline{Trace μ(h)} dh = 1.

We set

φ(k) := ∫_{H} φ_{0}(kh)\overline{Trace μ(h)} dh.

Then φ ∈ τ′ ⊠ (τ′)^{∗} ⊂ A(K) and φ(kh) = φ(k) Trace μ(h) for k ∈ K and h ∈ H. We define an analytic map by

Φ : K × H → K, (k, h) ↦ kh.

With the isomorphisms in §2.2, Φ_{(k,h)∗} : T_{(k,h)}(K × H) → T_{kh}K is given by the linear map 𝔨_{0} + 𝔥_{0} → 𝔨_{0}, (X, Y) ↦ (X, Ad(k)Y), and then the pullback Φ_{(k,h)}^{∗} : T_{kh}^{∗}K → T_{(k,h)}^{∗}(K × H) is by 𝔨_{0}^{∗} → 𝔨_{0}^{∗} + 𝔥_{0}^{∗}, ξ ↦ (ξ, pr_{𝔨→𝔥} (Ad^{∗}(k)^{−1}ξ)). Because Φ : K×H → K is a submersion, the pullback of hyperfunctions on K is well-defined. We put

F := Φ^{∗}(Θ_{2}\overline{φ}) ∈ B(K × H).

In view of

S. S.(Θ_{2}\overline{φ}) ⊂ S. S.(Θ_{2}) = S. S.(Θ_{π}^{K}) ⊂ K × Ad^{∗}(K) · AS_{K}(π),

we have by [9], Theorem 3.1.7

S. S. F ⊂ {(k, h; ξ, pr_{𝔨→𝔥}(Ad^{∗}(k)^{−1}ξ)) : k ∈ K, h ∈ H, ξ ∈ Ad^{∗}(K) · AS_{K}(π)}.

We put

V (k) := ∫_{H} F (k, h) dh ∈ B(K).

If ξ ∈ Ad^{∗}(K) · AS_{K}(π) satisfies pr_{𝔨→𝔥}(Ad^{∗}(k)^{−1}ξ) = 0 then ξ = 0 by (2.8.1). Therefore, we have

(2.8.9)  S. S. F ∩ (\sqrt{−1}T^{∗}K × (H × \{0\})) = ∅.

It follows from (2.8.9) that S. S. V = ∅ (see [9], Theorem 3.2.1), namely, V is real analytic, and also that

V (e) = ∫_{H} F (e, h) dh (= ∫_{H} Θ_{2}|_{H}(h) \overline{Trace μ(h)} dh).

Hence, (2.8.7′) will be proved if we show V ≡ 0. We shall show V ≡ 0 by proving ∫_{K} V (k)f(k) dk = 0 for any f ∈ τ_{1} ⊠ τ_{1}^{∗} (⊂ L^{2}(K)) and for any τ_{1} ∈ K̂. We fix such f and τ_{1}. Then

∫_{K} V (k)f(k) dk = ∫_{K} ∫_{H} Θ_{2}(kh)\overline{φ(kh)}f(k) dh dk
= ∫_{H} ∫_{K} Θ_{2}(kh)\overline{φ(kh)}f(k) dk dh.

Because S. S. F ∩ ((K × \{0\}) × \sqrt{−1}T^{∗}H) = ∅ and f is real analytic, we have

∫_{K} Θ_{2}(kh)\overline{φ(kh)}f(k) dk = ∫_{K} (Θ_{2}(kh)\overline{φ(kh)}f(k)) |_{K} dk

= ∫_{K} ∑_{τ∉Λ_{+}(μ)} m_{K}(τ : π) Trace τ(kh)\overline{φ(kh)}f(k) dk,

where the first integration is that for B(K ×H) along the fiber ([9], Theorem 3.2.1), while the second and third ones are for B(K) with h ∈ H fixed and the resulting function is real analytic on H. Because the sum in the last formula converges in the weak topology of B(K) ≃ A′(K) and because φ ∈ τ′ ⊠ (τ′)^{∗} and and f ∈ τ_{1} ⊠ τ_{1}^{∗}, the above integral equals

= ∑_{\begin{matrix}τ∉Λ_{+}(μ) \\ τ∈Supp_{K}(τ_{1}^{∗}⊗τ′) \end{matrix}} ∫_{K} m_{K}(τ : π) Trace τ(kh)\overline{φ(kh)}f(k) dk.

Now that the sum is taken over a finite set, we have

∫_{K} V (k)f(k) dk
= ∑_{\begin{matrix}τ∉Λ_{+}(μ) \\ τ∈Supp_{K}(τ_{1}^{∗}⊗τ′) \end{matrix}} m_{K}(τ : π) ∫_{K} (∫_{H} Trace τ(kh)\overline{Trace μ(h)} dh) \overline{φ(k)}f(k) dk

=0.

Here, the last equality follows from [τ|_{H} : μ] = 0 for any τ ∉ Λ_{+}(μ). Thus, we have proved V ≡ 0 and so (2.8.7′). Hence the proof of the second statement is complete.

3) The third statement is proved if we employ the analytic wave front set for Schwartz’s distributions instead of the singularity spectrum. □

Remark. The equation Θ_{π}^{K}|_{H} = Θ_{π|_{H}}^{H} is true as a formal character. In fact, it follows from (Trace τ) |_{H} = ∑_{μ∈Ĥ}[τ|_{H} : μ] μ that we have formally

( ∑_{τ∈K̂} m_{K}(τ : π) Trace τ) |_{H} = ∑_{τ∈K̂} m_{K}(τ : π) ( ∑_{μ∈Ĥ}[τ|_{H} : μ] Trace μ)

= ∑_{μ∈Ĥ} ( ∑_{τ∈K̂} m_{K}(τ : π)[τ|_{H} : μ] Trace μ) .

However, the equation Θ_{π}^{K}|_{H} = Θ_{π|_{H}}^{H} in B(H) under the assumption (2.8.1) does not follow directly from a general theory of hyperfunctions (e.g. [9], [23]), as one observes the following: Let a sequence of analytic functions ψ_{j} on K converge to ψ ∈ B(K) in the weak topology of B(K) ≃ A′(K). If S. S. ψ ∩ \sqrt{−1}T_{H}^{∗}K ⊂ \{0\} then the restriction ψ|_{H} is a well-defined hyperfunction on H, but ψ_{j}|_{H} may not converge to ψ|_{H} in the weak topology of B(H) ≃ A′(H). Likewise for distributions (cf. [6], Theorem 8.2.3 and Definition 8.2.2 (ii)).

\underline{2.9.} We are ready to state our main theorem. Let G be a real reductive Lie group, K a maximal compact subgroup of G, and θ the corresponding Cartan involution of G. Suppose G′ is a θ-stable closed subgroup of G with finitely many connected components. Then G′ is also a reductive Lie group with a maximal compact subgroup K′ := G′ ∩ K and we have a Cartan decomposition 𝔤′_{0} = 𝔨′_{0} + 𝔭′_{0}. We fix a θ-stable Cartan subalgebra 𝔧′_{0} of 𝔤′_{0}. We put

𝔧′_{ℝ} := \sqrt{−1}(𝔧′_{0} ∩ 𝔨′_{0}) + (𝔧′_{0} ∩ 𝔭′_{0}).

Let 𝔧′_{ℝ}^{∗} := Hom_{ℝ}(𝔧′_{ℝ}, ℝ) and 𝔧′^{∗} = Hom_{ℂ}(𝔧, ℂ). Then 𝔧′^{∗} = 𝔧′_{ℝ}^{∗}⊗ℂ. Correspondingly, the map taking the real part is denoted by

Re : 𝔧′^{∗} → 𝔧′_{ℝ}^{∗}.

We fix a norm | · | on 𝔧′_{ℝ}^{∗} which is invariant under the Weyl group W (𝔤′) of the root system Δ(𝔤′, 𝔧′). For an irreducible representation τ of G′, we denote by χ_{τ} ∈ 𝔧′^{∗} the Z(𝔤′)-infinitesimal character of the underlying (𝔤′, K′)-module of τ via the Harish-Chandra isomorphism Hom_{ℂ-alg.}(Z(𝔤′), ℂ) ≃ 𝔧′^{∗}/W (𝔤′).

Theorem 2.9. Assume that π ∈ Ĝ satisfies

(2.9.1)  AS_{K}(π) ∩ Ad^{∗}(K)\sqrt{−1}𝔨′_{0}^{⊥} = \{0\}.

Then we have the following.

1) The restriction π|_{K′} is K′-admissible. In particular, the restriction π|_{G′} is G′-admissible. Namely, we have a unitary equivalence of G′-modules:

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈Ĝ′ \end{matrix}}m_{G′}(τ : π|_{G′}) τ (a discrete Hilbert sum),

where m_{G′}(τ : π|_{G′}) := dim Hom_{G′}(τ, π|_{G′}) < ∞ for each τ ∈ Ĝ′.

2) If τ ∈ Ĝ′ is a discrete summand of the restriction π|_{G′}, then we have an upper estimate of the asymptotic K-support of τ by

AS_{K′}(τ|_{K′}) ⊂ pr_{𝔨→𝔨′} (Ad^{∗}(K) AS_{K}(π)) .

3) There exist A > 0 and N ≥ 0 such that the G′-multiplicity satisfies

(2.9.2)  m_{G′}(τ : π|_{G′}) ≤ A(1 + | Re χ_{τ} |)^{N}  for any τ ∈ Ĝ′.

We note that the choice of 𝔧′ and the norm affect (2.9.2) only by a constant multiple of A.

Remark 2.10. In Theorem 2.9 (3), N = 0 means that the multiplicity is uniformly bounded with respect to τ. We note that this is not the case in general. We have recently proved sufficient conditions for the uniform boundedness of the multiplicity m_{G′}(τ : π|_{G′}) in the following two cases (i) and (ii).

i) ([17]) π is any highest weight module, and (G, G′) is a reductive symmetric pair. ii) (in a forthcoming paper with T. Oshima) π ∈ Ĝ is arbitrary and (G, G′) is a real form of (SL(n, ℂ), GL(n − 1, ℂ)) or (O(n, ℂ), O(n − 1, ℂ)). In this case, the multiplicity m_{G′}(τ : π|_{G′}) is uniformly bounded with respect to both τ ∈ Ĝ′ and π ∈ Ĝ.

\underline{2.11.} Proof of Theorem 2.9. The subquotient theorem (ref. [32], Theorem 3.5.6) asserts that π can be realized as a subquotient of some principal series representation of G. In particular,

m_{K}(τ : π|_{K}) ≤ m_{K}(τ : L^{2}(K)) = dim τ

for any τ ∈ K̂. Let δ_{K} ∈ \sqrt{−1}𝔱_{0}^{∗} be half the sum of elements in Δ^{+}(𝔨, 𝔱). By Weyl’s dimension formula, we have

dim τ_{λ} = \frac{∏_{α∈Δ^{+}(𝔨,𝔱)}⟨λ + δ_{K}, α⟩}{∏_{α∈Δ^{+}(𝔨,𝔱)}⟨δ_{K}, α⟩}.

We note that

♯Δ^{+}(𝔨, 𝔱) = \frac{1}{2}(dim_{ℂ} 𝔨 − dim_{ℂ} 𝔱) = \frac{1}{2}(dim K − rank K)

Hence, there is a constant C > 0 such that

(2.11.1)  m_{K}(τ_{λ} : π|_{K}) ≤ C (1 + |λ|)^{\frac{1}{2} (dim K−rank K)}

for any π ∈ Ĝ and λ ∈ Λ_{+} ≃ K̂. Thus, the multiplicity function m_{K}(· : π|_{K}) : K̂ → ℕ is of polynomial growth.

1) Theorem 2.8 shows that the restriction π|_{K′} is K′-admissible. Because K′ ⊂ G′, it follows from [13], Theorem 1.2 that the restriction π|_{G′} is G′-admissible, too.

2) With the notation in Theorem 2.8,

Θ_{π|_{K′}}^{K′} = ∑_{μ∈K̂′} m_{K′}(μ : π|_{K′}) Trace μ

is a well-defined hyperfunction on K′ with the singularity spectrum

S. S.(Θ_{π|_{K′}}^{K′}) ⊂ K′ × pr_{𝔨→𝔨′} (Ad^{∗}(K) AS_{K}(π)) .

Because ∥m_{K′}(μ : π|_{K′}) Trace μ∥_{L^{2}(K′)} = m_{K′}(μ : π|_{K′}) ∈ ℕ, we can apply Lemma 2.6 (2) to obtain

AS_{K′}(π|_{K′}) = Supp_{K′}(π|_{K′})∞ = Supp( ; \overline{Θ_{π|_{K′}}^{K′}})∞ ⊂ pr_{𝔨→𝔨′} (Ad^{∗}(K) AS_{K}(π)) .

Since τ is a subrepresentation of π, we have Supp_{K′}(τ|_{K′}) ⊂ Supp_{K′}(π|_{K′}). Therefore, AS_{K′}(τ|_{K′}) ⊂ AS_{K′}(π|_{K′}). Hence, AS_{K′}(τ|_{K′}) ⊂ pr_{𝔨→𝔨′} (Ad^{∗}(K) AS_{K}(π)) .

3) We recall the minimal K-type theory due to Vogan [25]. We may assume that 𝔧′ is a fundamental Cartan subalgebra of 𝔤′, that is, 𝔱′_{0} := 𝔧′_{0}∩𝔨′_{0} is a Cartan subalgebra of 𝔨′_{0}. Fix a positive system Δ^{+}(𝔨′, 𝔱′). We write δ_{K′} ∈ \sqrt{−1}𝔱′_{0}^{∗} for half the sum of elements in Δ^{+}(𝔨′, 𝔱′). A minimal K-type of a (𝔤′, K′)-module τ is a K′-type F (K′, μ) for which |μ + 2δ_{K′}| is minimal among all K′-types occurring in τ. It follows from [11], Theorem 10.26 that there exists a constant C_{1}(≥ 0) depending only on 𝔤 with the following property: if τ is an irreducible (𝔤′, K′)-module with infinitesimal character χ_{τ} ∈ 𝔧′^{∗} and with minimal K-type F (K′, μ_{τ}) ∈ K̂′, then

(2.11.2)  | Re χ_{τ} | ≥ |μ_{τ} + 2δ_{K′}| − C_{1} ≥ |μ_{τ} | − C_{1}.

Then we have

(2.11.3)  m_{G′}(τ, π|_{G′}) ≤ dim Hom_{K′}(F (K′, μ_{τ}), π|_{K′}).

Because m_{K}(· : π|_{K}): K̂ → ℕ is of polynomial growth, so is m_{K′}(· : π|_{K′}): K̂′ → ℕ from Theorem 2.8 (3). Thus,

(2.11.4)  dim Hom_{K′}(F (K′, μ_{τ}), π|_{K′}) ≤ C(1 + |μ_{τ} |)^{N}.

It follows from (2.11.2) that there exists A > 0 (e.g. A = C(1 + C_{1})^{N}) such that

(2.11.5)  C(1 + |μ_{τ} |)^{N} ≤ A(1 + | Re χ_{τ} |)^{N}.

Combining (2.11.3), (2.11.4) and (2.11.5), we have

m_{G′}(τ, π|_{G′}) ≤ A(1 + | Re χ_{τ} |)^{N}.

This is what we wanted to prove. □

Example 2.12. If G′ = K, then K′ = K and 𝔨′_{0}^{⊥} = {0}. Therefore, the assumption (2.9.1) is automatically satisfied. This case corresponds to a result due to Harish-Chandra asserting that π|_{K} is K-admissible for any π ∈ Ĝ (e.g. [32], Theorem 3.4.10). In this case, we can take N = \frac{1}{2}(dim K − rank K) for the constant N in Theorem 2.9 (see (2.11.1)).

Example 2.13. Suppose that 𝔨′_{0} is an ideal of 𝔨_{0}, or equivalently, K is locally isomorphic to the direct product of K′ and a compact normal subgroup K′′. Then the assumption (2.9.1) is equivalent to

(2.9.1′)  AS_{K}(π) ∩ \sqrt{−1}(𝔨′′_{0})^{∗} = \{0\}.

Indeed, according to the direct sum decomposition 𝔨_{0} = 𝔨′_{0} ⊕ 𝔨′′_{0}, we have 𝔨_{0}^{∗} = (𝔨′_{0})^{∗} ⊕ (𝔨′′_{0})^{∗}. Therefore, Ad^{∗}(K)\sqrt{−1}(𝔨′_{0})^{⊥} = Ad^{∗}(K)\sqrt{−1}(𝔨′′_{0})^{∗} = \sqrt{−1}(𝔨′′_{0})^{∗}.

Example 2.14. Suppose that (K, K′) is a symmetric pair defined by an involution σ of K. Take a maximal abelian subspace 𝔞_{0} in {X ∈ 𝔨_{0} : σX = −X} and extend it to a Cartan subalgebra 𝔱_{0} of 𝔨_{0}. We choose compatible positive systems Δ^{+}(𝔨, 𝔱) and Σ^{+}(𝔨, 𝔞). We regard 𝔞_{0}^{∗} ⊂ 𝔱_{0}^{∗} according to the direct sum decomposition 𝔱_{0} = (𝔱_{0} ∩𝔨′_{0})⊕𝔞_{0}. Put C̄_{𝔞} := C̄ ∩\sqrt{−1}𝔞_{0}^{∗}. Then we have C̄ ∩Ad^{∗}(K)\sqrt{−1}𝔨′_{0}^{⊥} = C̄_{𝔞}. Hence the assumption (2.9.1) amounts to

(2.9.1′′)  AS_{K}(π) ∩ C̄_{𝔞} = {0}

because AS_{K}(π) ⊂ C̄.

§3. Computation of the asymptotic K-support

In this section we compute the asymptotic K-support AS_{K}(π) for typical irreducible unitary representations, namely,

1) π = A_{𝔮}(λ), Vogan-Zuckerman’s derived functor module (§3.2).

2) π is a unitary highest weight module (§3.3).

3) π is a unipotent representation of SO(2p, 2q) (§3.4).

\underline{3.1.} We begin with the following:

Lemma 3.1. Suppose π is a K-module (possibly of infinite dimensional).

1) If π′ is a K-submodule of π, then AS_{K}(π′) ⊂ AS_{K}(π).

2) If σ is a (non-zero) finite dimensional K-module, then AS_{K}(π ⊗ σ) = AS_{K}(π).

Proof. 1) Obvious from the inclusion Supp_{K}(π′) ⊂ Supp_{K}(π).

2) Let Q ⊂ Λ (⊂ \sqrt{−1}𝔱_{0}^{∗}) be the set of 𝔱_{0}-weights of the representation (σ, U). As is well-known (e.g. [26], Proposition 3.2.12), if λ ∈ Λ_{+}, then any irreducible Ksubmodule of F (K, λ) ⊗ U is of the form F (K, λ + ν) for some ν ∈ Q. Therefore, we have

Supp_{K}(π ⊗ σ) ⊂ Supp_{K}(π) + Q := \{μ + ν ∈ \sqrt{−1}𝔱_{0}^{∗} : μ ∈ Supp_{K}(π), ν ∈ Q\}.

Since Q is a finite set, we have (Supp_{K}(π) + Q)∞ = Supp_{K}(π)∞. Hence,

(3.1.1)  AS_{K}(π ⊗ σ) = Supp_{K}(π ⊗ σ)∞ ⊂ Supp_{K}(π)∞ = AS_{K}(π).

Similarly, we have AS_{K}(π ⊗ σ ⊗ σ^{∗}) ⊂ AS_{K}(π ⊗ σ). Since π is a K-submodule of π ⊗ σ ⊗ σ^{∗}, we have AS_{K}(π) ⊂ AS_{K}(π ⊗ σ ⊗ σ^{∗}) by (1). Hence, we have

(3.1.2)  AS_{K}(π) ⊂ AS_{K}(π ⊗ σ).

Now, (2) follows from (3.1.1) and (3.1.2). □

\underline{3.2.} First, we consider Zuckerman-Vogan’s derived functor modules. We start with a brief review of them. Standard references are [26], [32], and [11].

Given λ ∈ C̄ (⊂ \sqrt{−1}𝔱_{0}^{∗}), we define a θ-stable parabolic subalgebra of 𝔤 by 𝔮 ≡ 𝔮(λ) := 𝔩 + 𝔲, where 𝔩 and 𝔲 are stable under ad(𝔥) with weights given by

Δ(𝔩, 𝔥) := {α ∈ Δ(𝔤, 𝔥) : ⟨α, λ⟩ = 0},
Δ(𝔲, 𝔥) := {α ∈ Δ(𝔤, 𝔥) : ⟨α, λ⟩ > 0}.

Let L := Z_{G}(λ), and L̃ a metaplectic covering of L defined by the character of L acting on ∧^{top}𝔲 ≃ ℂ_{2ρ(𝔲)}. Here we write the character ℂ_{2ρ(𝔲)} in an additive way. We note that 𝔩 is the complexified Lie algebra of L (and also of L̃). The elliptic orbit Ad(G)X ≃ G/L carries a G-invariant complex structure, with the canonical line bundle Ω := ∧^{top}T^{∗}(G/L) ≃ G×_{L}ℂ_{2ρ(𝔲)}. Suppose W is a finite dimensional metaplectic representation of L̃. Then the L-module W ⊗ℂ_{ρ(𝔲)} defines a G-homogeneous holomorphic vector bundle W := G×_{L}(W ⊗ ℂ_{ρ(𝔲)}). As an algebraic analog of the Dolbeault cohomology group Hj ; ¯∂(G/L, W) (see [33]), Zuckerman introduced the co-

homological parabolic induction R_{𝔮}^{j} ≡ (R_{𝔮}^{𝔤})^{j} (j ∈ ℕ), which is a covariant functor from the category of metaplectic (𝔩, (L ∩ K)^{∼})-modules to that of (𝔤, K)-modules. If the Z(𝔩)-infinitesimal character γ ∈ 𝔥^{∗} of an irreducible metaplectic representation W of L̃ lies in the good range in the sense of [29], Definition 2.5, namely, if Re⟨γ, α⟩ > 0 for any α ∈ Δ(𝔲, 𝔥^{c}), then R_{𝔮}^{j}(W) = 0 for any j ≠ S := dim_{ℂ}(𝔲 ∩ 𝔨) and R_{𝔮}^{S}(W) is an irreducible (𝔤, K)-module. Furthermore, if dim W = 1, we sometimes write A_{𝔮}(λ) for R_{𝔮}^{S}(W) where ℂ_{λ} := W ⊗ ℂ_{−ρ(𝔲)} is a character of L. Here, we follow the normalization in [28], Definition 6.20 for R_{𝔮}^{j}; and the one in [30], §5 for A_{𝔮}. We define a closed cone (see [13]) by

(3.2.1)  ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ := \{ ∑_{β∈Δ(𝔲∩𝔭,𝔱)} n_{β} β : n_{β} ≥ 0\} (⊂ \sqrt{−1}𝔱_{0}^{∗}).

Example 3.2. With notation as above, suppose W is a finite dimensional metaplectic representation of L̃. Then we have

AS_{K}(R_{𝔮}^{S}(W)) ⊂ ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ C̄.

Proof of Example 3.2. We write σ_{1} ≤_{K} σ_{2} if σ_{1} and σ_{2} are K-modules such that

dim Hom_{K}(π, σ_{1}) ≤ dim Hom_{K}(π, σ_{2}) for any π ∈ K̂. Then, as is proved in [11], Theorem 5.35, we have the following estimate as K-modules:

R_{𝔮}^{j}(W)|_{K} ≤_{K} Hj ; ¯∂(K/L ∩ K, S(𝔲 ∩ 𝔭) ⊗ W ⊗ ℂ_{ρ(𝔲)}), j ∈ ℕ.

If ν − 2ρ(𝔲 ∩ 𝔨) ∈ Λ_{+}, then we have

HS ; ¯∂(K/L ∩ K, F (L ∩ K, ν)) = F (K, ν − 2ρ(𝔲 ∩ 𝔨))

from the Borel-Weil-Bott theorem. Therefore, if we define a finite set Q by Q := {δ + ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨) : δ is a highest weight of (L ∩ K)^{∼} type occurring in W }, then, by using the notation analogous to (3.2.1) (ℝ_{+} replaced by ℕ), we have

Supp_{K}(R_{𝔮}^{S}(W)) ⊂ Supp_{K}(HS ; ¯∂(K/L ∩ K, S(𝔲 ∩ 𝔭) ⊗ W ⊗ ℂ_{ρ(𝔲)}))
⊂ (ℕ⟨𝔲 ∩ 𝔭⟩ + Q) ∩ C̄

because any irreducible representation of L ∩ K occurring in the symmetric tensor algebra S(𝔲 ∩ 𝔭) is of the form F (L ∩ K, ν) with ν ∈ ℕ⟨𝔲 ∩ 𝔭⟩. Thus, we have AS_{K}(R_{𝔮}^{S}(W)) = Supp_{K}(R_{𝔮}^{S}(W))∞ ⊂ ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ C̄. □

\underline{3.3.} Next, we consider unitary highest weight modules.

Suppose that G/K is an irreducible Hermitian symmetric space. Let 𝔠(𝔨_{0}) be the center of 𝔨_{0}. Then, it is known that there exists Z ∈ 𝔠(𝔨_{0}) such that

𝔤 = 𝔨 ⊕ 𝔭^{+} ⊕ 𝔭^{−}

are 0, \sqrt{−1} and −\sqrt{−1} eigenspaces of ad(Z). An irreducible (𝔤, K)-module is said to be a highest weight module if there exists a non-zero vector annihilated by 𝔭^{+}.

Example 3.3. If V is a unitary highest weight (𝔤, K)-module, then

(3.3.1)  AS_{K}(V) ⊂ ℝ_{+}⟨𝔭^{−}⟩ ∩ C̄.

Proof of Example 3.3. Let G_{ℂ} be a connected complex Lie group with Lie algebra 𝔤, P_{−} a maximal parabolic subgroup of G_{ℂ} with Lie algebra 𝔨+𝔭^{−}. We equip G/K with a G-invariant complex structure through the Borel embedding:

G/K ↪ G_{ℂ}/P_{−}.

Then we can realize V as a (𝔤, K)-submodule of O(G ×_{K} U), where

U := V^{𝔭^{+}} = {v ∈ V_{K} : dπ(Y)v = 0 for any Y ∈ 𝔭^{+}} (≠ {0})

is an irreducible representation of K. Since the K-structure of the underlying (𝔤, K)-module of O(G ×_{K} U) is given by O(G ×_{K} U)_{K} ≃ S(𝔭^{−}) ⊗ U, we have

AS_{K}(V) ⊂ AS_{K}(S(𝔭^{−}) ⊗ U) = AS_{K}(S(𝔭^{−})) ⊂ ℝ_{+}⟨𝔭^{−}⟩ ∩ C̄.

Here, we have used Lemma 3.1 (1) and (2). Thus the proof is completed. □

Remark. Following [1], Definition 1.6, we say that a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 is anti-holomorphic if 𝔲 ∩ 𝔭 ⊂ 𝔭^{−}. If 𝔮 is anti-holomorphic, then any irreducible (𝔤, K)-subquotient V of R_{𝔮}^{S}(W) is a highest weight module for any finite dimensional metaplectic representation W of L̃ ([1], Lemma 1.7). In this case, Example 3.2 gives an estimate

AS_{K}(V) ⊂ ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ C̄ (⊂ ℝ_{+}⟨𝔭^{−}⟩ ∩ C̄),

which is a refinement of (3.3.1).

\underline{3.4.} Third, we give an example of a “unipotent representation”. Similar computation works for other “small representations” of other reductive groups in an elementary way, especially when its K-type formula is of “ladder type”.

Example 3.4. Let G = SO_{o}(2p, 2q), for p ≥ 1, q ≥ 1. Choose a basis {f_{i} : 1 ≤ i ≤ p + q\} on \sqrt{−1}𝔱_{0}^{∗} such that a positive root system of 𝔨 is given by

Δ^{+}(𝔨, 𝔱) = {f_{i} ± f_{j} : 1 ≤ i < j ≤ p, or p + 1 ≤ i < j ≤ p + q}.

Kostant, Binegar and Zierau constructed an irreducible unitary representation (π, H) of G (also that of SO_{0}(2p + 1, 2q + 1)), which is “associated with” the minimal nilpotent coadjoint orbit [20], [2] (different constructions are also known by Torasso, Huang-Zhu, and [19]). This representation (π, H) is not a highest (or lowest) weight module if p > 1 and q > 1, and has the K-structure of the so called “ladder” type:

H_{K} ≃ ⨁_{\begin{matrix}a,b∈ℕ \\ a+p=b+q \end{matrix}} F (SO(2p), af_{1}) ⊠ F (SO(2q), bf_{p+1}).

Thus, Supp_{K}(π) = {af_{1} + bf_{p+1} : a, b ∈ ℕ, a + p = b + q} and so

AS_{K}(π) = ℝ_{+}(f_{1} + f_{p+1}).

If G′ = SO(2p, q′) × SO(q′′) (q′ + q′′ = 2q) then 𝔞 = ∑_{j=1}^{l} ℝf_{p+j} with notation in Example 2.14, where l := min(q′, q′′). If G′ = U(p, q), then 𝔞 is of the form

\{(x_{1}, x_{1}, x_{2}, x_{2}, …, x_{[ \frac{p}{2} ]−1}, (−1)^{[ \frac{p}{2} ]}x_{[ \frac{p}{2} ]}, (0)) : x_{i} ∈ ℝ\}
⊕\{y_{1}, y_{1}, y_{2}, y_{2}, …, y_{[ \frac{q}{2} ]−1}, (−1)^{[ \frac{q}{2} ]}y_{[ \frac{q}{2} ]}, (0)) :y_{j} ∈ ℝ\}.

Therefore, both of the restrictions π|_{SO(2p,q′)×SO(q′′)} and π|_{U(p,q)} are discretely decomposable. Explicit branching laws will be presented in a forthcoming paper joint with B. Ørsted, giving an analog of Howe’s correspondence ([19]).

Remark 3.5. Theorem 2.9 involves two invariants, namely, Ad^{∗}(K)\sqrt{−1}𝔨′_{0}^{⊥} and AS_{K}(π), of which typical examples are computed explicitly in Examples 2.12, 2.13, 2.14 and in Examples 3.2, 3.3, 3.4, respectively. The combination of these examples gives rise to both new results and a unified treatment of previous results. We collect them for the convenience of the reader.

1) Example 2.12 corresponds to a result of Harish-Chandra asserting that any irreducible unitary representation of G is K-admissible. The explicit branching law in the setting of the combination of Example 2.12 and Example 3.2 is called the generalized Blattner conjecture (theorem) proved by H. Hecht and W. Schmid (discrete series case) and by D. Vogan (general case) (see [26]).

2) The combination of Example 2.13 (applied to the case 𝔨′_{0} is the center of 𝔨_{0}) and Example 3.3 (applied to holomorphic discrete series representations) gives a theorem of Martens [22] and of Lipsman and Wallach ([21], Theorem 4.2). The decomposition formula was studied by Jakobsen and Vergne [7].

3) The combination of Example 2.13 and Example 3.2 leads to [12], Proposition 4.1.3, which was used to determine the non-vanishing condition of unitary cohomologically induced representations R_{𝔮}^{S}(ℂ_{λ}) with singular parameters λ.

4) The combination of Example 2.14 and Example 3.2 yields the main theorem of [13] (see Theorem 3.2 loc. cit.). In [16], Theorem 4.2, we prove that our condition is also necessary to be K′-admissible in this case.

5) A special case of Example 2.14 (that is, (G, G′) = (SO(2, 2n), SO(1, 2n))) combined with Example 3.2 leads us to a very explicit formula of the middle Hodge component of certain modular symbols that are totally real and totally geodesic cycles in the arithmetic quotients of bounded symmetric domain of type IV [18].

6) The combination of Example 2.14 and Example 3.4 presents new examples of an analog of Howe’s dual pair correspondence for minimal unipotent representations [19].

References

[1] J. Adams, Unitary highest weight modules, Adv. in Math. 63 (1987), 113–137.

[2] B. Binegar and R. Zierau, Unitarization of a singular representation of SO(p,q), Commun. Math. Phys. 138 (1991), 245–258.

[3] J. M. Bony, Equivalence des diverses notions de spectre singulier analytique, Sém. Goulaouic-Schwartz (1976/77), Ecole Polytechnique, Exposé. 3.

[4] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33 (1979), Amer. Math. Soc., 275–285.

[5] , Wave front sets of representations of Lie groups, Automorphic forms, representation theory, and arithmetic (1981), 117–140.

[6] L. Hörmander, The Analysis of Linear Partial Differential Operators I, Grundlehren, vol. 256, Springer-Verlag, Berlin, 1983, ISBN 0-387-12104-8.

[7] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29-53.

[8] A. Kaneko, Introduction to Hyperfunctions, Kluwer Academic Publ., 1988, ISBN 90-277-2837-2.

[9] M. Kashiwara, T. Kawai and T. Kimura, Foundations of Algebraic Analysis, Princeton Math. Series, vol. 37, Princeton Univ. Press, New Jersey, 1986, ISBN 0-691-08413-0.

[10] M. Kashiwara and M. Vergne, K-types and singular spectrum, Lect. Notes. in Math., vol. 728, Springer-Verlag, 1979, pp. 177–200.

[11] A. Knapp and D. Vogan, Cohomological Induction and Unitary Representations, Princeton Math. Series, vol. 45, Princeton University Press, New Jersey, 1995, ISBN 0-691-03756-6.

[12] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), Memoirs of Amer. Math. Soc., vol. 462, Providence R. I., 1992, ISBN 0-8218-2524-0.

[13] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205.

[14] , The restriction of A_{𝔮}(λ) to reductive subgroups II, Proc. Japan Acad. 71 (1995), 24–26.

[15] , Discrete series representations for the orbit spaces arising from two involutions of real reductive Lie groups, J. Funct. Anal. (to appear).

[16] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. (to appear).

[17] , Multiplicity free branching laws for unitary highest weight modules, preprint.

[18] T. Kobayashi and T. Oda, Vanishing of modular symbols on locally symmetric spaces, Commentarii Mathematici Helvetici (to appear).

[19] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, preprint.

[20] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory (Connes et al, eds.), vol. 92, Birkhäuser, Boston, 1990, pp. 85–124.

[21] R. Lipsman, Restrictions of principal series to a real form, Pacific J. Math. 89 (1980), 367–390.

[22] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. USA 72 (1975), 3275–3276.

[23] M. Sato, T. Kawai and M. Kashiwara, Microfunctions and pseudo-differential equations, Lecture Notes in Math. 287 (1973), Springer-Verlag, 265–529.

[24] H. Sekiguchi, The Penrose Transform for certain non-compact homogeneous manifolds of U(n, n), ph. D. dissertation, University of Tokyo, 1995 January; published in J. Math. Sci. Univ. Tokyo 3 (1996), 655–697.

[25] D. A. Vogan, Jr., The algebraic structure of the representations of semisimple Lie group I, Ann. of Math. 109 (1979), 1–60.

[26] , Representations of real reductive Lie groups, Progress in Math., vol. 15, Birkhäuser, 1981.

[27] , Unitarizability of certain series of representations, Ann. of Math. 120 (1984), 141– 187.

[28] , Unitary Representations of Reductive Lie Groups, Ann. of Math. Stud., vol. 118, Princeton University Press, New Jersey, 1987, ISBN 0-691-08482-3.

[29] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191–221.

[30] D. A. Vogan, Jr. and G. J. Zuckerman, Unitary representations with non-zero cohomology, Compositio Math. 53 (1984), 51–90.

[31] N. Wallach, On the unitarizability of derived functor modules, Invent. Math. 78 (1984), 131–141.

[32] , Real Reductive Groups I, II, Pure and Appl. Math., vol. 132, Academic Press, San Diego, 1988, 1992, ISBN 0-12-732960-9, 0-12-732961-7.

[33] H. Wong, Dolbeault cohomologies and Zuckerman modules associated with finite rank representations, ph. D. dissertation, Harvard University, 1992; published in J. Funct. Anal. 129 (1995), 428–454.

Department of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153, Tokyo, Japan E-mail address: toshi@ms.u-tokyo.ac.jp