The Entropy Gain of Linear Systems and Some of Its Implications

Abstract

We study the increase in per-sample differential entropy rate of random sequences and processes after being passed through a non minimum-phase (NMP) discrete-time, linear time-invariant (LTI) filter G. For LTI discrete-time filters and random processes, it has long been established by Theorem 14 in Shannon’s seminal paper that this entropy gain, $\mathcal{G}\left(G\right)$, equals the integral of $\log \left|G\right({\mathrm{e}}^{j\omega }\left)\right|$. In this note, we first show that Shannon’s Theorem 14 does not hold in general. Then, we prove that, when comparing the input differential entropy to that of the entire (longer) output of G, the entropy gain equals $\mathcal{G}\left(G\right)$. We show that the entropy gain between equal-length input and output sequences is upper bounded by $\mathcal{G}\left(G\right)$ and arises if and only if there exists an output additive disturbance with finite differential entropy (no matter how small) or a random initial state. Unlike what happens with linear maps, the entropy gain in this case depends on the distribution of all the signals involved. We illustrate some of the consequences of these results by presenting their implications in three different problems. Specifically: conditions for equality in an information inequality of importance in networked control problems; extending to a much broader class of sources the existing results on the rate-distortion function for non-stationary Gaussian sources, and an observation on the capacity of auto-regressive Gaussian channels with feedback.

1. Introduction

We study the difference between the differential entropy rate of a random process ${\mathrm{u}}_1^{\infty }=\{{\mathrm{u}}_1,{\mathrm{u}}_2,\dots \}$ entering a discrete-time linear time-invariant (LTI) system G and the differential entropy rate of its (possibly noisy) output ${\mathrm{y}}_1^{\infty }$, as depicted in Figure 1.

Recall that the differential entropy rate of a random process ${\mathrm{x}}_1^{\infty }$ is given by $\overline{h}\left({\mathrm{x}}_1^{\infty }\right)\triangleq {\lim }_{n\to \infty }{n}^{-1}h({\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_n)$, provided the limit exists, where $h({\mathrm{x}}_1,\dots ,{\mathrm{x}}_n)=\mathrm{E}\left[-\log f({\mathrm{x}}_1,\dots ,{\mathrm{x}}_n)\right]$ is the differential entropy of the ensemble ${\mathrm{x}}_1,\dots ,{\mathrm{x}}_n$ with probability density function (PDF) f [1]. The system G is supposed to satisfy the following:

Assumption 1.

The LTI system G in Figure 1 is causal and stable and such that

1.

G has a rational p-th order transfer function $G\left(z\right)$ with m zeros ${\left\{{\rho }_i\right\}}_{i=1}^m$ outside the unit circle, i.e., non-minimum-phase (NMP) zeros, where $m\in \{0,1,\dots ,p\}$, indexed in non-increasing magnitude order, i.e., $|{\rho }_1|\ge |{\rho }_2|\ge \cdots \ge |{\rho }_m|>1$.

2.

The unit-impulse response of G, say, $g_0,g_1,\dots$ satisfies $|g_0|=1$.

In this general setup, G may have a random initial state vector ${\mathbf{x}}_0\in {\mathbb{R}}^p$, $p\in \mathbb{N}$, and a real-valued random output disturbance ${\mathrm{z}}_1^{\infty }$. Our main purpose is to characterize the limit

$$\begin{array}{c}\mathcal{G}(G,{\mathbf{x}}_0,{\mathrm{u}}_1^{\infty },{\mathrm{z}}_1^{\infty })\triangleq \underset{n\to \infty }{\lim }\frac{1}{n}\left(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right)\right),\end{array}$$

(1)

evaluating the possible effect produced by ${\mathbf{x}}_0$ and ${\mathrm{z}}_1^{\infty }$. This difference can be interpreted as the entropy gain (entropy amplification or entropy boost) introduced by the filter G and (as apparent from the other variables in the argument of $\mathcal{G}$) the statistics of ${\mathbf{x}}_0,{\mathrm{u}}_1^{\infty },{\mathrm{z}}_1^{\infty }$. We shall refer to the special case in which ${\mathbf{x}}_0$ and ${\mathrm{z}}_1^{\infty }$ are both zero (or deterministic) as the noise-less case, and write $\mathcal{G}(G,0,{\mathrm{u}}_1^{\infty },0)$ accordingly.

The earliest reference related to this problem corresponds to a noise-less continuous-time counterpart considered by Shannon. In his seminal 1948 paper [2], Shannon gave a formula for the change in differential entropy per degree of freedom that a continuous-time random process ${\mathrm{u}}_c$, band-limited to a frequency range $[0,B)$ (in Hz), experiences after passing through an LTI continuous-time filter $G_c$ (without considering a random initial state or an output disturbance). Such entropy per degree of freedom is defined in terms of uniformly taken samples as

$$\begin{array}{cc}\overline{h}\left({\mathrm{u}}_c\right)& \triangleq \underset{n\to \infty }{\lim }\frac{1}{n}h({\mathrm{u}}_c\left(T\right),{\mathrm{u}}_c\left(2T\right),\dots ,{\mathrm{u}}_c\left(nT\right)),\hfill \end{array}$$

(2)

with $T\triangleq 1/\left(2B\right)$. In this formula, if the LTI filter has frequency response $G_c\left(\xi \right)$ (with $\xi$ in Hz), then the resulting differential entropy rate of the output process ${\mathrm{y}}_c$ is given by the following theorem:

Theorem 1

(Reference [2], Theorem 14). If an ensemble having an entropy $\overline{h}\left({\mathrm{u}}_c\right)$ per degree of freedom in band B is passed through a filter with characteristic $G_c\left(\xi \right)$ the output ensemble has an entropy

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_c\right)=\overline{h}\left({\mathrm{u}}_c\right)+\frac{2}{B}\underset{0}{\overset{B}{\int }}\log \left|G_c\left(\xi \right)\right|d\xi .\end{array}$$

(3)

Shannon arrived at (3) by arguing that an LTI filter can be seen as a linear operator that selectively scales its input signal along infinitely many frequencies, each of them representing an orthogonal component of the source. He then obtained the result by writing down the determinant of the Jacobian of this operator as the product of the squared frequency response magnitude of the filter over n frequency bands, applying logarithm, dividing by n, and then taking the limit as n tends to infinity.

Remark 1.

There is a factor of two in excess in the integral on the right-hand side (RHS) of (3). To see this, consider a filter with a constant gain a over $[0,B)$ (i.e., a simple multiplicative factor). In such case, the entropy rate of ${\mathrm{y}}_c$ should exceed that of ${\mathrm{u}}_c$ by $\log \left|a\right|$ [1]. However, (3) yields an entropy gain equal to $2\log \left|a\right|$. This error arises because the determinant of the Jacobian of the transformation is actually the product of $\left|G_c\right|$ over the n frequency bands considered in Shannon’s argument. Such excess factor of two is also present in the entropy losses appearing in Reference [2], Table 1.

Theorem 14 in Reference [2] has found application in works ranging from traditional themes, such as linear prediction [3] and source coding [4], to molecular communication systems [5,6].

The available literature treating the phenomenon itself of the entropy gain (loss, boost, or amplification) induced by LTI systems seems to be rather scarce. This is not surprising given that (3) was published in Reference [2], Theorem 14, the work which gave birth to Information Theory.

The following publication concerned with this problem is Reference [7], following a time-domain analysis for the corresponding discrete-time problem. In this approach, one can obtain ${\mathrm{y}}_1^n\triangleq \{\mathrm{y}\left(1\right),\mathrm{y}\left(2\right),\dots ,\mathrm{y}\left(n\right)\}$ as a function of ${\mathrm{u}}_1^n$, for every $n\in \mathbb{N}$, and evaluate the difference between the limits $\overline{h}\left({\mathrm{y}}_1^{\infty }\right)$ and $\overline{h}\left({\mathrm{u}}_1^{\infty }\right)$, obtained by letting $n\to \infty$. More precisely, for an LTI discrete-time filter G with impulse response $g_0^{\infty }=\{g_0,g_1,\dots \}$, we can write

$$\begin{array}{c}{\mathbf{y}}_n^1=\underset{{\mathit{G}}_n}{\underbrace{\left(\begin{array}{cccc}{g}_0& 0& \cdots & 0\\ g_1& g_0& \cdots & 0\\ ⋮& & \ddots & ⋮\\ g_{n-1}& g_{n-2}& \cdots & g_0\end{array}\right)}}{\mathbf{u}}_n^1,\end{array}$$

(4)

where we adopt the notation ${\mathbf{y}}_n^1$ for column vectors to avoid the abuse of notation incurred by treating the sequence ${\mathrm{y}}_1^n$ as a vector, and because, by writing ${\mathbf{y}}_n^1$, it is easier to remember that its samples are ordered from top to bottom. ${\mathbf{y}}_n^1\triangleq {[\mathrm{y}\left(1\right)\mathrm{y}\left(2\right)\cdots \mathrm{y}\left(n\right)]}^T$ and the random vector ${\mathbf{u}}_n^1$ is defined likewise. From this, it is clear (see, e.g., the corollary after Theorem 8.6.4 in Reference [1]) that

$$\begin{array}{c}h\left({\mathbf{y}}_n^1\right)=h\left({\mathbf{u}}_n^1\right)+\log \left|\det \left({\mathit{G}}_n\right)\right|,\end{array}$$

(5)

where $\det \left({\mathit{G}}_n\right)$ (or simply $\det {\mathit{G}}_n$) stands for the determinant of ${\mathit{G}}_n$. This result is utilized in Reference [7] to show that no entropy gain is produced by a stable minimum phase LTI system G if and only if the first sample in its impulse response has unit magnitude.

In Reference [8], p. 568, the entropy gain of a discrete-time LTI system G (the noise-less version of the setup depicted in Figure 1) is found to be

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_1^{\infty }\right)=\overline{h}\left({\mathrm{u}}_1^{\infty }\right)+\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega ,\end{array}$$

(6)

where ${\mathrm{y}}_1^{\infty }$ is the filter’s discrete-time output process (without the effect of random initial state or an output disturbance) and

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_1^{\infty }\right)\triangleq \underset{n\to \infty }{\lim }\frac{1}{n}h\left({\mathrm{y}}_1^n\right).\end{array}$$

(7)

This result was obtained starting from the fact that, for a Gaussian stationary process ${\mathrm{u}}_1^{\infty }$ with power spectral density (PSD) $S_{\mathrm{u}}\left({\mathrm{e}}^{j\omega }\right)$, $\overline{h}\left({\mathrm{u}}_1^{\infty }\right)=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }{S}_{\mathrm{u}}\left({\mathrm{e}}^{j\omega }\right)d\omega$. If ${\mathrm{u}}_1^{\infty }$ enters a discrete-time LTI system with frequency response $G\left({\mathrm{e}}^{j\omega }\right)$, then the PSD of its output ${\mathrm{y}}_1^{\infty }$ is $S_{\mathrm{y}}\left({\mathrm{e}}^{j\omega }\right)=S_{\mathrm{u}}\left({\mathrm{e}}^{j\omega }\right){\left|G\left({\mathrm{e}}^{j\omega }\right)\right|}^2$; thus, it is argued that (6) follows for Gaussian stationary inputs. Then, Reference [8] extends the result for non-Gaussian inputs with a proof sketch which uses a time-domain relation, like (4), to point out that the filter is a linear operator and, as such, the differential entropy of its output exceeds that of its input by a quantity that is independent of the input distribution. (It is worth noting that (6) is the discrete-time equivalent of (3) (without its wrong factor of 2), which follows directly from the correspondence between sampled band-limited continuous-time systems and discrete-time systems.)

It is in Reference [9], Section II-C, where, for the first time, it is shown that, for a stationary Gaussian input ${\mathrm{u}}_1^{\infty }$, the full entropy gain predicted by (6) takes place if the system output ${\mathrm{y}}_1^{\infty }$ is contaminated by an additive output disturbance of length p and positive definite covariance matrix, where p is the order of $G\left(z\right)$.

The integral $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega$ can be related to the structure of the filter G. It is well known (from Jensen’s formula) that if G has a causal and stable rational transfer function $G\left(z\right)$ and an impulse response with its first sample $g_0\triangleq {\lim }_{z\to \infty }G\left(z\right)$, then

$$\begin{array}{c}\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega =\log \left|g_0\right|+\sum _{i:\left|{\rho }_i\right|>1}\log \left|{\rho }_i\right|,\end{array}$$

(8)

where $\left\{{\rho }_i\right\}$ are the zeros of $G\left(z\right)$ (see, e.g., References [10,11]). This provides a straightforward formula to evaluate $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega$ of a given LTI filter with rational transfer function $G\left(z\right)$. When combined with (6), this equation also reveals that if the entropy gain $\mathcal{G}({\mathrm{u}}_1^{\infty },{\mathrm{y}}_1^{\infty })$ is negative (i.e., if it corresponds to an entropy loss), then $|g_0|<1$ (with the corresponding change of variables, this is the case in all the examples given by Shannon in Reference [2], Table 1). More importantly, (8) allows us to concentrate, without loss of generality, on LTI systems $G\left(z\right)$, whose first impulse-response sample has unit magnitude, as required by Assumption 1. Under the latter condition, (8) shows that the entropy gain is greater than zero if and only if $G\left(z\right)$ has zeros outside the unit disk $\mathbb{D}\triangleq \{\rho \in \mathbb{C}:\left|\rho \right|\le 1\}$. A system with the latter property is said to be non-minimum phase (NMP); conversely, a system with all its zeros inside $\mathbb{D}$ is said to be minimum phase (MP) [11].

1.1. Main Contributions of this Paper

The main contributions of this paper can be summarized as follows:

• Our first main result is showing that (6) and (3) do not hold for a large class of continuous-time filters and inputs. To see this, notice that

  $$\begin{array}{c}|g_0|=1⟹|\det \left({\mathit{G}}_n\right)|=1,\forall n\in \mathbb{N},\end{array}$$

  (9)

  which, in view of (5), is equivalent to $h\left({\mathbf{y}}_n^1\right)=h\left({\mathbf{u}}_n^1\right),\forall n\in \mathbb{N}$. In turn, this implies that $\overline{h}\left({\mathrm{y}}_1^{\infty }\right)-\overline{h}\left({\mathrm{u}}_1^{\infty }\right)=0$, regardless of whether $G\left(z\right)$ (i.e., the polynomial $g_0+g_1{z}^{-1}+\cdots$) has zeros with magnitude greater than one (choose, for example, $g_0=1,g_1=2$, and $g_k=0$ for $k\ge 2$). This reveals that (4) holds if and only $G\left(z\right)$ is MP. But (6) and (3) are equivalent (correcting for the in excess factor of 2 discussed in Remark 1); thus, Theorem 14 in Reference [2] also does not hold for a class of continuous-time filters. However, the transfer function $G_c\left(s\right)$ of a band-limited continuous-time filter $G_c$ is defined only for imaginary values of s (because the bilateral Laplace transform of $\sin \left(t\right)/t$ converges only on the imaginary axis), so one cannot classify such filters as MP or NMP. Instead, we consider a class of continuous-time filters limited to the frequencies in the band $[0,B)$, where $B>0$ is in [Hz], defined by having a unit-impulse response of the form

  $$\begin{array}{c}g\left(t\right)\triangleq \sum _{k=0}^{\eta }{g}_k{\varphi }_k\left(t\right),\end{array}$$

  (10)

  for some absolutely summable sequence of real-valued coefficients ${\left\{g_i\right\}}_{i=0}^{\eta }$, $\eta =1,2,\dots$, where the sinc functions

  $$\begin{array}{cc}{\varphi }_k\left(t\right)& \triangleq \frac{\sin \left(2\pi B\right[t-k/\left(2B\right)\left]\right)}{\pi [t-k/(2B\left)\right]}.\hfill \end{array}$$

  (11)

  Since every such g satisfies $g(k/(2B\left)\right)=0$ for $k<0$, it makes sense to refer to such filters as “sample-wise causal”. For this class of band-limited filters, we show that Theorem 14 holds if and only if the z-transform of ${\left\{g_i\right\}}_{i=0}^{\eta }$ is MP:

Theorem 2.

Suppose $G_c$ is a low-pass continuous-time filter with unit-impulse response as in (10). Let the continuous-time random input of $G_c$ be

$$\begin{array}{c}{\mathrm{u}}_c\left(t\right)=\frac{1}{2B}\sum _{k=1}^{\infty }\mathrm{u}\left(k\right){\varphi }_k\left(t\right),\end{array}$$

(12)

for some random sequence ${\left\{\mathrm{u}\left(k\right)\right\}}_{k=1}^{\infty }$, with ${\varphi }_k$ as in (11), and denote its output as ${\mathrm{y}}_c$. Then,

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_c\right)-\overline{h}\left({\mathrm{u}}_c\right)=\log \left|g_0\right|=\log \left|\frac{1}{B}{\int }_0^B\Re \left\{G_c\left(\xi \right)\right\}d\xi \right|\stackrel{\left(a\right)}{\le }\frac{1}{B}{\int }_0^B\log \left|G_c\left(\xi \right)\right|d\xi ,\end{array}$$

(13)

with equality in $\left(a\right)$ if and only if the polynomial $g_0+g_1{z}^{-1}+g_2{z}^{-2}\cdots$ has no roots outside the unit circle.

2.

We show that $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega$ actually corresponds to the entropy gain introduced by G but considering the new notion of effective differential entropy rate of ${\mathrm{y}}_1^{\infty }$ proposed in this paper, defined next.

Definition 1

(The Effective Differential Entropy). Let $\mathbf{y}\in {\mathbb{R}}^{\ell }$ be a random vector. If $\mathbf{y}$ can be written as a linear transformation $\mathbf{y}=\mathit{S}\mathbf{u}$, for some $\mathbf{u}\in {\mathbb{R}}^n$ ($n\le \ell$) with bounded differential entropy, $\mathit{S}\in {\mathbb{R}}^{\ell \times n}$, then the effective differential entropy of $\mathbf{y}$ is defined as

$$\begin{array}{c}\breve{h}\left(\mathbf{y}\right)\triangleq h\left(\mathit{A}\mathbf{y}\right),\end{array}$$

(14)

where $\mathit{S}={\mathit{A}}^T\mathit{T}\mathit{C}$ is an SVD for $\mathit{S}$, with $\mathit{T}\in {\mathbb{R}}^{n\times n}$.

We can now state our second main result, the proof of which is in Appendix A:

Theorem 3.

Let ${\mathrm{u}}_1^{\infty }$ be the input of an LTI system G with transfer function $G\left(z\right)$ without zeros on the unit circle and with an absolutely summable unit impulse response ${\left\{g_i\right\}}_{i=0}^{\eta -1}$, with $\eta =\infty$ if G has an infinite impulse response. Denote the output of G as ${\mathrm{y}}_1^{\infty }$. Suppose $\left|h\left({\mathrm{u}}_1^n\right)\right|<\infty$ for every finite n. Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\left(\breve{h}\left({\mathrm{y}}_1^{n+\eta }\left({\mathrm{u}}_1^n\right)\right)-h\left({\mathrm{u}}_1^n\right)\right)=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega ,\end{array}$$

(15)

where ${\mathrm{y}}_1^{n+\eta }\left({\mathrm{u}}_1^n\right)$ denotes the entire response of G to the input ${\mathrm{u}}_1^n$.

Theorem 3 states that, when considering the full-length output of a system, the effective entropy gain is introduced by the system itself.

Section 4 provides a geometrical description of the phenomenon behind Definition 1 and Theorem 3.

2.

We show that $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega$ is a tight upper bound to the entropy gain of G (as defined in (1)), when the output is contaminated by some additional additive signal, such as a random initial state (represented by ${\mathbf{x}}_0$ in Figure 1) or an output disturbance (such as ${\mathrm{z}}_1^{\infty }$ in Figure 1), with sufficiently many degrees of freedom (a condition formally stated in Assumption 2 below). Moreover, we show that an entropy gain equal to the latter upper bound can appear even when these disturbances or random initial state have infinitesimally small variances. To the best of our knowledge, the latter phenomenon has been discussed in the literature first (and only) in Reference [9], Section II-C, for Gaussian stationary inputs and an LTI filter. We go beyond the latter result by explicitly and fully characterizing the entropy gain of LTI systems for a large class of not necessarily Gaussian nor stationary random input. We refer to this class as entropy-balanced processes, formally specified in the following definition:

Definition 2.

A random process ${\left\{\mathrm{v}\left(k\right)\right\}}_{k=1}^{\infty }$ is said to be entropy balanced if the following two conditions are satisfied:

(i) 

Its sample variances ${\sigma }_{\mathrm{v}\left(n\right)}^2$ are finite for finite n and

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\log \left({\sigma }_{\mathrm{v}\left(n\right)}^2\right)=0.\end{array}$$

(16)

(ii) 

For every $\nu \in \mathbb{N}$ and for every sequence of matrices ${\left\{{\mathsf{\Phi }}_n\right\}}_{n=\nu +1}^{\infty }$, ${\mathsf{\Phi }}_n\in {\mathbb{R}}^{(n-\nu )\times n}$ with orthonormal rows,

$$\begin{array}{cc}\underset{n\to \infty }{\lim }\frac{1}{n}& \left(h\left({\mathsf{\Phi }}_n{\mathbf{v}}_n^1\right)-h\left({\mathbf{v}}_n^1\right)\right)=0.\hfill \end{array}$$

(17)

The second condition guarantees that projecting an entropy-balanced process onto any subspace having finitely fewer dimensions yields a process with the same differential entropy rate.

The entropy gain induced by finite-length output disturbances is characterized by our next theorem.

Theorem 4.

In the system of Figure 1, let G satisfy Assumption 1 and suppose that ${\mathrm{u}}_1^{\infty }$ is entropy balanced. Suppose the random output disturbance ${\mathrm{z}}_1^{\infty }$ is such that $\mathrm{z}\left(i\right)=0,\forall i>\kappa$, and that $\left|h\right({\mathrm{z}}_1^{\kappa }\left)\right|<\infty$. Let $\overline{\kappa }\triangleq \min \{\kappa ,m\}$, where m is the number of NMP zeros of $G\left(z\right)$. Then,

$$\sum _{i=m-\overline{\kappa }+1}^m\log |{\rho }_i|\le {\lim \sup }_{n\to \infty }\frac{1}{n}(h\left({\mathbf{y}}_n^1\right)-h\left({\mathbf{u}}_n^1\right))\le \sum _{i=1}^{\overline{\kappa }}\log \left|{\rho }_i\right|\stackrel{\left(a\right)}{\le }\frac{1}{2\pi }\underset{-\pi }{\overset{\pi }{\int }}\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega$$

(18)

with equality in $\left(a\right)$ if and only if $\kappa \ge m$.

The proof is presented in Section 6.4, and we provide geometrical insight explaining the phenomenon underlying Definition 2 and Theorem 4 in Section 5.1.

2.

We illustrate the relevance of the results summarized above by applying them to three problems in three areas, namely:

(a)

Networked Control: We show that equality holds in the inequality stated in Reference [12], Lemma 3.2 (a fundamental piece for the performance limitation results further developed in Reference [13]), under very general conditions. In addition, we extend the validity of a related equality for the perfect-feedback case, given by Reference [14], Theorem 14, for Gaussian signals, to the much larger class of entropy-balanced processes.

(b)

The rate-distortion function for non-stationary Gaussian sources: This problem has been previously solved in References [15,16,17]. We provide a simpler proof based upon the results described above. This proof extends the result stated in References [16,17] to a broader class of non-stationary sources.

(c)

Gaussian channel capacity with feedback: We show that capacity results based on using a short random sequence as channel input and relying on a feedback filter which boosts the entropy rate of the end-to-end channel noise (such as the one proposed in Reference [9]), crucially depend upon the complete absence of any additional disturbance anywhere in the system. Specifically, we show that the information rate of such capacity-achieving schemes drops to zero in the presence of any such additional disturbance. As a consequence, the relevance of characterizing the robust (i.e., in the presence of disturbances) feedback capacity of Gaussian channels, which appears to be a fairly unexplored problem, becomes evident.

1.2. Paper Outline

The remainder of this paper begins with some necessary definitions and preliminary results in Section 2. It continues with our detailed exposition in Section 3 of why Shannon’s reasoning fails to yield the right expression for the entropy gain. We present an intuitive discussion leading to the definition of effective differential entropy in Section 4, which is ended by the proof of Theorem 3. Section 5 gives a geometric interpretation of how an arbitrarily small additive perturbation is able to boost the differential entropy rate of the process coming out of an NMP LTI filter. This exposition helps understanding and justifies the introduction of entropy-balanced random processes, which are also characterized there. Section 6 and Section 7 contain our results for the entropy gain produced by an output disturbance and a random initial state, respectively. Our illustrative application results are presented in Section 8, followed by our conclusions in Section 9. Except when presented right after a statement or in its own section, all proofs are given in Appendix B.

2. Preliminaries

2.1. Notation

The sets of natural, real and complex numbers are denoted $\mathbb{N}$, $\mathbb{R}$, and $\mathbb{C}$, respectively. For a complex x, $\Re \left\{x\right\}$ is the real part of x. For a set $\mathcal{S}$, the indicator function $1_{\mathcal{S}}\left(x\right)$ equals 1 if $x\in \mathcal{S}$ and 0 otherwise. For any LTI system G, the transfer function $G\left(z\right)$ corresponds to the z-transform of the impulse response $g_0,g_1,\dots$, i.e., $G\left(z\right)={\sum }_{i=0}^{\infty }{g}_i{z}^{-i}$. For a transfer function $G\left(z\right)$, we denote by ${\mathit{G}}_n\in {\mathbb{R}}^{n\times n}$ the lower triangular Toeplitz matrix having ${[g_0\cdots g_{n-1}]}^T$ as its first column. We write $x_1^n$ as a shorthand for the sequence $\{x_1,\dots ,x_n\}$, and, when convenient, we write $x_1^n$ in vector form as ${\mathit{x}}_n^1\triangleq {[x_1{x}_2\cdots x_n]}^T$, where ${\left(\right)}^T$ denotes transposition. Random scalars (vectors) are denoted using non-italic characters, such as x (non-italic and boldface characters, such as $\mathbf{x}$). The notation $\mathbf{x}{\displaystyle ⫫}\mathrm{y}$ means $\mathbf{x}$ and y are independent. If x and z are conditionally independent given y, we write $\mathrm{x}⟷\mathrm{y}⟷\mathrm{z}$. For matrices, we use upper-case boldface symbols, such as $\mathit{A}$. We write ${\lambda }_i\left(\mathit{A}\right)$ to denote the i-th eigenvalue of $\mathit{A}$ sorted in increasing magnitude. If $\mathit{A}\in {\mathbb{C}}^{m\times n}$, ${\mathit{A}}^H$ is its conjugate transpose, and ${\sigma }_i\left(\mathit{A}\right)\triangleq \sqrt{{\lambda }_i\left({\mathit{A}}^H\mathit{A}\right)}$, if $m\ge n$, and ${\sigma }_i\left(\mathit{A}\right)\triangleq \sqrt{{\lambda }_i\left(\mathit{A}{\mathit{A}}^H\right)}$, if $m<n$. We define ${\sigma }_{\min }\left(\mathit{A}\right)\triangleq {\sigma }_1\left(\mathit{A}\right)$ and ${\sigma }_{\max }\left(\mathit{A}\right)\triangleq {\sigma }_{\min \{m,n\}}\left(\mathit{A}\right)$. The term ${\mathit{A}}_{i,j}$ denotes the entry in the intersection between the i-th row and the j-th column. If $\mathit{A}\in {\mathbb{C}}^{m\times n}$, then ${\mathit{A}}^T$ and ${\mathit{A}}^{*}$ denote the transpose and conjugate transpose of $\mathit{A}$, respectively. We write ${\left[\mathit{A}\right]}_{i_2}^{i_1}$, with $1\le i_1\le i_2\le m$, to refer to the matrix formed by selecting the rows $i_1$ to $i_2$ of $\mathit{A}$. Likewise, for $1\le j_1\le j_2\le n$, $\stackrel{j_1{j}_2}{{\displaystyle \overbracket{\mathit{A}}}}$ is the matrix built with columns $j_1$ to $j_2$ of $\mathit{A}$. The expression ${}^{m_1}{\left[\mathit{A}\right]}_{m_2}$ corresponds to the square sub-matrix along the main diagonal of $\mathit{A}$, with its top-left and bottom-right corners on ${\mathit{A}}_{m_1,m_1}$ and ${\mathit{A}}_{m_2,m_2}$, respectively. A diagonal matrix whose entries are the elements in a set $\mathcal{D}$ (wherein elements may be repeated) is denoted as $\mathrm{diag}\mathcal{D}$. If $\mathit{A}\in {\mathbb{R}}^{n\times m_1}$ and $\mathit{B}\in {\mathbb{R}}^{n\times m_2}$, we write $\left[\mathit{A}\right|\mathit{B}]\in {\mathbb{R}}^{n\times (m_1+m_2)}$ to denote the augmented matrix built by placing the columns of $\mathit{A}$ followed by those of $\mathit{B}$.

2.2. Mutual Information and Differential Entropy

Let $\mathbf{x}$, y, and z be random variables with joint PDF $f_{\mathbf{x},\mathrm{y},\mathrm{z}}$, and marginal PDFs $f_{\mathrm{x}}$, $f_{\mathrm{y}}$, and $f_{\mathrm{z}}$, respectively. The mutual information between x and y is defined as $I(\mathrm{x};\mathrm{y})\triangleq \int f_{\mathrm{x},\mathrm{y}}(x,y)\log \left(\frac{f_{\mathrm{x},\mathrm{y}}(x,y)}{f_{\mathrm{x}}\left(x\right)f_{\mathrm{y}}\left(y\right)}\right)dxdy$. The conditional mutual information between x and y given z is defined as $I(\mathrm{x};\mathrm{y}|\mathrm{z})\triangleq \int f_{\mathrm{x},\mathrm{y},\mathrm{z}}(x,y,z)\log \left(\frac{f_{\mathrm{x},\mathrm{y}|\mathrm{z}}(x,y|z)}{f_{\mathrm{x}|\mathrm{z}}\left(x\right|z)f_{\mathrm{y}|\mathrm{z}}\left(y\right|z)}\right)dxdydz$, where $f_{\mathrm{x},\mathrm{y}|\mathrm{z}}$ is the joint PDF of x and y given z, and $f_{\mathrm{x}|\mathrm{z}}$, $f_{\mathrm{y}|\mathrm{z}}$ are defined likewise. The conditional differential entropy of x given y is defined as $h\left(\mathrm{x}\right|\mathrm{y})\triangleq -\int f_{\mathrm{x},\mathrm{y}}(x,y)\log \left(f_{\mathrm{x}|\mathrm{y}}\left(x\right|y)\right)dxdy$.

From these definitions, it is easy to verify the following properties Reference [1], Sections 2.4–2.6 and 8.4–8.6:

• Shift invariance: for every deterministic function f,

  $$\begin{array}{c}h(\mathrm{x}+f(\mathrm{y}\left)\right|\mathrm{y})=h(\mathrm{x}\left|\mathrm{y}\right).\end{array}$$

  (19)
• Non-negativity:

  $$\begin{array}{c}I(\mathrm{x};\mathrm{y})\ge 0,\end{array}$$

  (20)

  with equality if and only if x and y are independent.
• Chain Rule:

  $$\begin{array}{c}I(\mathrm{x};\mathrm{y},\mathrm{z})=I(\mathrm{x};\mathrm{y})+I(\mathrm{x};\mathrm{z}|\mathrm{y}).\end{array}$$

  (21)
• Relationship with entropy:

  $$\begin{array}{c}I(\mathrm{x};\mathrm{y})=h\left(\mathrm{x}\right)-h\left(\mathrm{x}\right|\mathrm{y})=h(\mathrm{y})-h(\mathrm{y}\left|\mathrm{x}\right).\end{array}$$

  (22)

2.3. System Model and Assumptions

Consider the discrete-time system depicted in Figure 1. In this setup, the block G satisfies Assumption 1.

It is worth noting that there is no loss of generality in considering $g_0=1$, since one can otherwise write $G\left(z\right)$ as $G^{\prime }\left(z\right)=g_0·(G\left(z\right)/g_0)$; thus, the entropy gain introduced by $G^{\prime }\left(z\right)$ would be $\log |g_0|$ plus the entropy gain due to $G\left(z\right)/g_0$ (in agreement with (6)), which has an impulse response with its first sample equal to 1.

The following assumption is made about the output disturbance ${\mathrm{z}}_1^{\infty }$:

Assumption 2.

The disturbance ${\mathrm{z}}_1^{\infty }$ is independent of ${\mathrm{u}}_1^{\infty }$ and belongs to a κ-dimensional linear subspace, for some finite $\kappa \in \mathbb{N}$. This subspace is spanned by the κ orthonormal columns of a matrix $\mathsf{\Phi }\in {\mathbb{R}}^{\left|\mathbb{N}\right|\times \kappa }$ (where $\left|\mathbb{N}\right|$ stands for the countably infinite size of $\mathbb{N}$), such that $\left|h\right({\mathsf{\Phi }}^T{\mathbf{z}}_{\infty }^1\left)\right|<\infty$. Moreover, ${\mathbf{z}}_{\infty }^1=\mathsf{\Phi }{\mathbf{s}}_{\kappa }^1$, where the random vector ${\mathbf{s}}_{\kappa }^1\triangleq {\mathsf{\Phi }}^T{\mathbf{z}}_{\infty }^1$ has finite differential entropy, its covariance matrix ${\mathit{K}}_{{\mathbf{s}}_{\kappa }^1}$ satisfies ${\lambda }_{\max }\left({\mathit{K}}_{{\mathbf{s}}_{\kappa }^1}\right)<\infty$, and it is independent of ${\mathbf{u}}_{\infty }^1$.

3. Revisiting Theorem 14 in Reference Shannon et al.

In this section, after presenting the proof of Theorem 2, we develop Shannon’s approach into a more detailed and formal exposition. This allows us to explain why, for part of the continuous-time filters considered in Theorem 2, the approach chosen by Shannon to prove Theorem 14 in Reference [2] is unable to predict the correct value for the entropy gain.

3.1. Proof of Theorem 2

To begin with, the Fourier transform of ${\varphi }_k$ is

$$\begin{array}{c}{\Phi }_k\left(\xi \right)\triangleq {\int }_{-\infty }^{\infty }{\varphi }_k\left(t\right){\mathrm{e}}^{-j2\pi \xi t}dt=1_{[-B,B]}\left(\xi \right){\mathrm{e}}^{-j2\pi \xi k/\left(2B\right)}.\end{array}$$

(23)

It is easy to verify that the functions ${\varphi }_k$ satisfy the following orthogonality property:

$$\begin{array}{c}{\int }_{-\infty }^{\infty }{\varphi }_k\left(t\right){\varphi }_i\left(t\right)dt=\left\{\begin{array}{cc}2B\hfill & ,k=i\hfill \\ 0\hfill & ,k\ne i\hfill \end{array}\right.\end{array}$$

(24)

and

$$\begin{array}{c}{\varphi }_k(-t)={\varphi }_{-k}\left(t\right).\end{array}$$

(25)

Notice that $\mathrm{u}\left(k\right)={\mathrm{u}}_c\left(\frac{k}{2B}\right)$, $k\in \mathbb{N}$.

The output of $G_c$ sampled at time $t=\ell /\left(2B\right)$, $\ell \in \mathbb{N}$, is

$$\begin{array}{c}\mathrm{y}(\ell )\triangleq {\mathrm{y}}_c\left(\frac{\ell }{2B}\right)={\int }_{-\infty }^{\infty }g\left(\tau \right){\mathrm{u}}_c(\frac{\ell }{2B}-\tau )d\tau \end{array}$$

(26)

$$\begin{array}{cc}& =\frac{1}{2B}{\sum }_{k=1}^{\infty }{\sum }_{i=0}^{\eta }{g}_i\mathrm{u}\left(k\right){\int }_{-\infty }^{\infty }{\varphi }_i\left(\tau \right){\varphi }_k(\frac{\ell }{2B}-\tau )d\tau \hfill \end{array}$$

(27)

$$\begin{array}{cc}& =\frac{1}{2B}{\sum }_{k=1}^{\infty }{\sum }_{i=0}^{\eta }{g}_i\mathrm{u}\left(k\right){\int }_{-\infty }^{\infty }{\varphi }_i\left(\tau \right){\varphi }_{\ell -k}\left(\tau \right)d\tau \hfill \end{array}$$

(28)

$$\begin{array}{cc}& ={\sum }_{i=0}^{\eta }{g}_i\mathrm{u}(\ell -i),\hfill \end{array}$$

(29)

with $\mathrm{u}\left(k\right)=0$ for $k\le 0$. This means that the output samples ${\mathrm{y}}_1^{\infty }$ are the discrete-time convolution between ${\mathrm{u}}_1^{\infty }$ and the filter coefficients ${\left\{g_i\right\}}_{i=0}^{\eta }$. Therefore, the matrix relation (4) holds. We then obtain that $\overline{h}\left({\mathrm{y}}_c\right)=\overline{h}\left({\mathrm{u}}_c\right)+\log \left|g_0\right|$.

The frequency response of $G_c$ is given by

$$\begin{array}{c}{G}_c\left(\xi \right)={\int }_{-\infty }^{\infty }g\left(t\right){\mathrm{e}}^{-j2\pi \xi t}dt=\sum _{k=0}^{\eta }{g}_k{\Phi }_k\left(\xi \right)=\sum _{k=0}^{\eta }{g}_k{\mathrm{e}}^{-j\pi \xi k/B},\end{array}$$

(30)

where $\xi$ is in [Hz]. This means that

$$\begin{array}{c}{g}_0=\frac{1}{2B}{\int }_{-B}^B{G}_c\left(\xi \right)d\xi =\frac{1}{B}{\int }_0^B\Re \left\{G_c\left(\xi \right)\right\}d\xi ,\end{array}$$

(31)

where the last equality holds because $G_c\left(\xi \right)$ is conjugate symmetric. Thus, the entropy gain introduced by $G_c$ is the right-hand side of (13), concluding the proof.   □

3.2. Formalizing Shannon’s Argument

In the approach followed by Shannon, it is argued that the entropy gain is the limit as $n\to \infty$ of $n^{-1}{\sum }_{r=0}^{n-1}\log \left|G_c\left({\xi }_r\right)\right|$ over uniformly spaced frequencies ${\xi }_0,\dots ,{\xi }_{n-1}$. Here, we show that this summation corresponds to $\log \left|\det \right({\tilde{\mathit{G}}}_n\left)\right|$, where ${\tilde{\mathit{G}}}_n$ is an n-by-n Toeplitz circulant matrix. Moreover, the sequences of Hermitian matrices ${\left\{{\mathit{G}}_n{\mathit{G}}_n^{*}\right\}}_{n=1}^{\infty }$ and ${\left\{{\tilde{\mathit{G}}}_n{\tilde{\mathit{G}}}_n^{*}\right\}}_{n=1}^{\infty }$ are asymptotically equivalent (as defined in Reference [18], Section 2.3), which would yield ${\lim }_{n\to \infty }{n}^{-1}\log |\det \left({\mathit{G}}_n\right)|={\lim }_{n\to \infty }{n}^{-1}\log \left|\det \left({\tilde{\mathit{G}}}_n\right)\right|$ if the eigenvalues of ${\mathit{G}}_n{\mathit{G}}_n^{*}$ were bounded between constants $0<{\zeta }_m<{\zeta }_M<\infty$ for all $n\in \mathbb{N}$. However, if $G\left(z\right)$ (the z-transform of ${\left\{g_k\right\}}_{k=0}^{\infty }$) has NMP zeros, then ${\mathit{G}}_n{\mathit{G}}_n^{*}$ has eigenvalues tending to zero exponentially as $n\to \infty$, which precludes these two limits to coincide.

To prove the above claims, we first apply the change of variable $\omega \triangleq \pi \xi /B$, with which (30) becomes

$$\begin{array}{c}{G}_c(B\omega /\pi )=G\left({\mathrm{e}}^{j\omega }\right)\triangleq \sum _{k=0}^{\eta }{g}_k{\mathrm{e}}^{-j\omega k},\end{array}$$

(32)

where $G\left({\mathrm{e}}^{j\omega }\right)$ is the frequency response of the discrete-time filter G with unit-impulse response ${\left\{g_i\right\}}_{i=1}^{\eta }$ and $\omega$ is in radians per second. Now, following Shannon’s approach, we uniformly sample $G\left({\mathrm{e}}^{j\omega }\right)$ at n frequencies

$$\begin{array}{c}{\omega }_r\triangleq \left\{\begin{array}{cc}r\frac{2\pi }{n}\hfill & ,r/n\le 0.5\hfill \\ r\frac{2\pi }{n}-2\pi \hfill & ,r/n>0.5\hfill \end{array}\right.,r=0,1,\dots ,n-1,\end{array}$$

(33)

which, from (32), yields the spectral samples

$$\begin{array}{cc}G\left({\mathrm{e}}^{-j{\omega }_r}\right)& =\sum _{k=0}^{\eta }{g}_k{\mathrm{e}}^{-j\frac{2\pi }{n}rk}.\hfill \end{array}$$

(34)

We will cast the reason why (3) fails to coincide with the correct expression for the entropy gain provided by (5) as a disagreement between the asymptotic behavior of the logarithm of the determinant of two sequences of asymptotically equivalent matrices. For that purpose, since (34) coincides with Reference [18], Equation 4.34, we have that the spectral samples ${\left\{G\left({\mathrm{e}}^{-j{\omega }_r}\right)\right\}}_{r=0}^{n-1}$ are the eigenvalues of the Toeplitz circulant matrix (Reference [18], Chapter 3)

$$\begin{array}{cc}{\tilde{\mathit{G}}}_n\triangleq \left(\begin{array}{cccc}{\tilde{g}}_{n,0}& {\tilde{g}}_{n,n-1}& \cdots & {\tilde{g}}_{n,1}\\ {\tilde{g}}_{n,1}& \ddots & \ddots & ⋮\\ ⋮& \ddots & \ddots & {\tilde{g}}_{n,n-1}\\ {\tilde{g}}_{n,n-1}& \ddots & \ddots & {\tilde{g}}_{n,0}\end{array}\right)& ={\mathit{U}}_n^{*}\mathrm{diag}\{G\left({\mathrm{e}}^{-j{\omega }_0}\right),\dots ,G\left({\mathrm{e}}^{-j{\omega }_{n-1}}\right)\}{\mathit{U}}_n,\hfill \end{array}$$

(35)

where ${\mathit{U}}_n\in {\mathbb{C}}^{n\times n}$ is the n-point discrete Fourier transform (DFT) matrix, defined as

$$\begin{array}{c}{\left[{\mathit{U}}_n\right]}_{k,r}\triangleq \frac{1}{\sqrt{n}}{\mathrm{e}}^{-j\frac{2\pi }{n}kr},k,r=0,1,\dots ,n-1.\end{array}$$

(36)

From Reference [18], Lemma 4.5, ${\tilde{g}}_{n,k}\triangleq {\sum }_{i\in {\mathbb{N}}_0:k+ni\le \eta }{g}_{k+in}$, corresponding to the (possibly) aliased impulse response $g_0,g_1,\dots ,g_{\eta }$ as a result of sampling in frequency.

We can now see that the discrepancy between the entropy gain predicted by (3) and (5) is the disagreement between the following limits:

$$\begin{array}{cc}\underset{n\to \infty }{\lim }\frac{1}{2n}\log \left(\det \left({\mathit{G}}_n{\mathit{G}}_n^{*}\right)\right)& \stackrel{\left(31\right)}{=}\log \left|\frac{1}{B}{\int }_0^B\Re \left\{G_c\left(\xi \right)\right\}d\xi \right|,\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\underset{n\to \infty }{\lim }\frac{1}{2n}\log \left(\det \left({\tilde{\mathit{G}}}_n{\tilde{\mathit{G}}}_n^{*}\right)\right)& \stackrel{\phantom{\left(\right)}}{=}\frac{1}{B}{\int }_0^B\log \left|G_c\left(\xi \right)\right|d\xi ,\hfill \end{array}$$

(37b)

where, due to (8), the expressions on both right-hand sides differ if and only if $G\left(z\right)$ has NMP zeros. According to Reference [18], Lemma 4.6, the sequences ${\left\{{\mathit{G}}_n\right\}}_{n=1}^{\infty }$ and ${\left\{{\tilde{\mathit{G}}}_n\right\}}_{n=1}^{\infty }$ are asymptotically equivalent, which is written as ${\mathit{G}}_n\sim {\tilde{\mathit{G}}}_n$. Then, from Reference [18], Theorem 2.1, the Hermitian matrices ${\mathit{G}}_n{\mathit{G}}_n^{*}\sim {\tilde{\mathit{G}}}_n{\tilde{\mathit{G}}}_n^{*}$, which, from Reference [18], Theorem 2.4, implies that

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\sum _{i=1}^{n}f\left({\lambda }_i\left({\mathit{G}}_n{\mathit{G}}_n^{*}\right)\right)=\underset{n\to \infty }{\lim }\frac{1}{n}\sum _{i=1}^{n}f\left({\lambda }_i\left({\tilde{\mathit{G}}}_n{\tilde{\mathit{G}}}_n^{*}\right)\right),\end{array}$$

(38)

for any function f continuous over a finite interval $[{\zeta }_m,{\zeta }_M]$ such that

$$\begin{array}{c}{\zeta }_m\le {\lambda }_i\left({\mathit{G}}_n{\mathit{G}}_n^{*}\right),{\lambda }_i\left({\tilde{\mathit{G}}}_n{\tilde{\mathit{G}}}_n^{*}\right)\le {\zeta }_M,i=1,2,\dots ,n,n=1,2,\dots .\end{array}$$

(39)

However, when $G\left(z\right)$ has m NMP zeros, Lemma 7 (in Section 6.3) establishes that there are exactly m eigenvalues of ${\mathit{G}}_n$ that tend to zero exponentially as $n\to \infty$. Crucially, $\log (·)$ is discontinuous at 0, which precludes the limits in (37) from coinciding.

4. The Effective Differential Entropy

Theorem 3 establishes that the effective differential entropy rate of the entire or complete output of an LTI system exceeds that of the (shorter) input sequence by the RHS of (15). This section provides a geometrical interpretation of this problem and intuition about the effective differential entropy already introduced in Definition 1.

Consider the random vectors $\mathbf{u}\triangleq {\left[{\mathrm{u}}_1{\mathrm{u}}_2\right]}^T$ and $\mathbf{y}\triangleq {\left[{\mathrm{y}}_1{\mathrm{y}}_2{\mathrm{y}}_3\right]}^T$ related via

$$\begin{array}{c}\left[\begin{array}{c}{\mathrm{y}}_1\\ {\mathrm{y}}_2\\ {\mathrm{y}}_3\end{array}\right]=\underset{\triangleq {\breve{\mathit{G}}}_2}{\underbrace{\left(\begin{array}{cc}1& 0\\ 2& 1\\ 0& 2\end{array}\right)}}\left[\begin{array}{c}{\mathrm{u}}_1\\ {\mathrm{u}}_2\end{array}\right].\end{array}$$

(40)

Suppose $\mathbf{u}$ is uniformly distributed over $[0,1]\times [0,1]$. Applying the conventional definition of differential entropy of a random sequence, we would have that

$$\begin{array}{c}h({\mathrm{y}}_1,{\mathrm{y}}_2,{\mathrm{y}}_3)=h({\mathrm{y}}_1,{\mathrm{y}}_2)+h\left({\mathrm{y}}_3\right|{\mathrm{y}}_1,{\mathrm{y}}_2)=-\infty \end{array}$$

(41)

because ${\mathrm{y}}_3$ is a deterministic function of ${\mathrm{y}}_1$ and ${\mathrm{y}}_2$:

$${\mathrm{y}}_3=\left[02\right]{\left[{\mathrm{u}}_1{\mathrm{u}}_2\right]}^T=\left[02\right]{\left(\begin{array}{cc}1& 0\\ 2& 1\end{array}\right)}^{-1}\left[\begin{array}{c}{\mathrm{y}}_1\\ {\mathrm{y}}_2\end{array}\right].$$

In other words, the problem lies in that, although the output is a three-dimensional vector, it only has two degrees of freedom, i.e., it is restricted to a 2-dimensional subspace of ${\mathbb{R}}^3$. This is illustrated in Figure 2, where the set $[0,1]\times [0,1]$ is shown (coinciding with the u-v plane), together with its image through ${\breve{\mathit{G}}}_2$ (as defined in (40)).

As can be seen in this figure, the image of the square ${[0,1]}^2$ through ${\breve{\mathit{G}}}_2$ is a 2-dimensional rhombus over which $\{{\mathrm{y}}_1,{\mathrm{y}}_2,{\mathrm{y}}_3\}$ distributes uniformly. Since the intuitive notion of differential entropy of an ensemble of random variables relates to the size of the region spanned by the associated random vector (and determines how difficult it is to compress it in a lossy fashion with a given precision), one could argue that the differential entropy of $\{{\mathrm{y}}_1,{\mathrm{y}}_2,{\mathrm{y}}_3\}$, far from being $-\infty$, should be somewhat larger than that of $\{{\mathrm{u}}_1,{\mathrm{u}}_2\}$ (since the rhombus ${\breve{\mathit{G}}}_2{[0,1]}^2$ has a larger area than ${[0,1]}^2$). So, what does it mean that (and why should) $h({\mathrm{y}}_1,{\mathrm{y}}_2,{\mathrm{y}}_3)=-\infty$? Simply put, the differential entropy relates to the volume spanned by the support of the probability density function. For $\mathbf{y}$ in our example, the latter (three-dimensional) volume is clearly zero.

From the above discussion, the comparison between the differential entries of $\mathbf{y}\in {\mathbb{R}}^3$ and $\mathbf{u}\in {\mathbb{R}}^2$ of our previous example should take into account that $\mathbf{y}$ actually lives in a two-dimensional subspace of ${\mathbb{R}}^3$. Indeed, since the multiplication by a unitary matrix does not alter differential entries, we could consider the differential entropy of

$$\begin{array}{c}\left[\begin{array}{c}\tilde{\mathbf{y}}\\ 0\end{array}\right]\triangleq \left(\begin{array}{c}\breve{\mathit{Q}}\\ {\overline{\mathit{q}}}^T\end{array}\right)\mathbf{y},\end{array}$$

(42)

where ${\breve{\mathit{Q}}}^T$ is the $3\times 2$ matrix with orthonormal rows in the SVD of ${\breve{\mathit{G}}}_2$

$$\begin{array}{c}{\breve{\mathit{G}}}_2={\breve{\mathit{Q}}}^T\breve{\mathit{D}}\breve{\mathit{R}},\end{array}$$

(43)

and $\overline{\mathit{q}}$ is a unit-norm vector orthogonal to the rows of $\breve{\mathit{Q}}$ (and thus orthogonal to $\mathbf{y}$, as well). We are now able to compute the differential entropy in ${\mathbb{R}}^2$ for $\tilde{\mathbf{y}}$, corresponding to the rotated version of $\mathbf{y}$ such that its support is now aligned with ${\mathbb{R}}^2$.

The preceding discussion motivates the use of a modified version of the notion of differential entropy for a random vector $\mathbf{y}\in {\mathbb{R}}^n$ which considers the number of dimensions actually spanned by $\mathbf{y}$ instead of its length.

It is worth mentioning that Shannon’s differential entropy of a vector $\mathbf{y}\in {\mathbb{R}}^{\ell }$, whose support’s ℓ-volume is greater than zero, arises from considering it as the difference between its (absolute) entropy and that of a random variable uniformly distributed over an ℓ-dimensional, unit-volume region of ${\mathbb{R}}^{\ell }$. More precisely, if in this case the probability density function (PDF) of $\mathbf{y}={[{\mathrm{y}}_1{\mathrm{y}}_2\cdots {\mathrm{y}}_{\ell }]}^T$ is Riemann integrable, then [1], Thm. 9.3.1,

$$\begin{array}{c}h\left(\mathbf{y}\right)=\underset{\Delta \to 0}{\lim }\left[H\left({\mathbf{y}}^{\Delta }\right)+\ell \log \Delta \right],\end{array}$$

(44)

where ${\mathbf{y}}^{\Delta }$ is the discrete-valued random vector resulting when $\mathbf{y}$ is quantized using an ℓ-dimensional uniform quantizer with ℓ-cubic quantization cells with volume ${\Delta }^{\ell }$. However, if we consider a variable $\mathbf{y}$ whose support belongs to an n-dimensional subspace of ${\mathbb{R}}^{\ell }$, $n<\ell$ (i.e., $\mathbf{y}=\mathit{S}\mathbf{u}={\mathit{A}}^T\mathit{T}\mathit{C}\mathbf{u}$, as in Definition 1), then the entropy of its quantized version in ${\mathbb{R}}^{\ell }$, say $H_{\ell }\left({\mathbf{y}}^{\Delta }\right)$, is distinct from $H_n\left({\left(\mathit{A}\mathbf{y}\right)}^{\Delta }\right)$, the entropy of $\mathit{A}\mathbf{y}$ in ${\mathbb{R}}^n$. Moreover, it turns out that, in general,

$$\begin{array}{c}\underset{\Delta \to 0}{\lim }\left(H_{\ell }\left({\mathbf{y}}^{\Delta }\right)-H_n\left({\left(\mathit{A}\mathbf{y}\right)}^{\Delta }\right)\right)\ne 0,\end{array}$$

(45)

despite the fact that $\mathit{A}$ has orthonormal rows. Thus, the definition given by (44) does not yield consistent results for the case wherein a random vector has a support’s dimension (i.e., its number of degrees of freedom) smaller that its length (The mentioned inconsistency refers to (45).), which reveals that the asymptotic behavior $H_{\ell }\left({\mathbf{y}}^{\Delta }\right)$ changes if $\mathbf{y}$ is rotated. (If this were not the case, then we could redefine (44) replacing ℓ by n, in a spirit similar to the one behind Renyi’s d-dimensional entropy [19].) To see this, consider the case in which $\mathbf{u}\in \mathbb{R}$ distributes uniformly over $[0,1]$ and $\mathbf{y}={\left[11\right]}^T\mathbf{u}/\sqrt{2}$. Clearly, $\mathbf{y}$ distributes uniformly over the unit-length segment connecting the origin with the point $(1,1)/\sqrt{2}$. Then,

$$\begin{array}{c}{H}_2\left({\mathbf{y}}^{\Delta }\right)=-⌊\frac{1}{\Delta \sqrt{2}}⌋\Delta \sqrt{2}\log \left(\Delta \sqrt{2}\right)-\left(1-⌊\frac{1}{\Delta \sqrt{2}}⌋\sqrt{2}\Delta \right)\log \left(1-⌊\frac{1}{\Delta \sqrt{2}}⌋\sqrt{2}\Delta \right).\end{array}$$

(46)

On the other hand, since, in this case, $\mathit{A}\mathbf{y}=\mathbf{u}$, we have that

$$\begin{array}{c}{H}_1\left({\left(\mathit{A}\mathbf{y}\right)}^{\Delta }\right)=H_1\left({\mathbf{u}}^{\Delta }\right)=-⌊\frac{1}{\Delta }⌋\Delta \log \Delta -(1-⌊\frac{1}{\Delta }⌋\Delta )\log (1-⌊\frac{1}{\Delta }⌋\Delta ).\end{array}$$

(47)

Thus, the d-dimensional entropy would not generally be equal to the effective differential entropy, that is:

$$\begin{array}{c}\underset{\Delta \to 0}{\lim }\left(H_1({\left(\mathit{A}\mathbf{y}\right)}^{\Delta })-H_2\left({\mathbf{y}}^{\Delta }\right)\right)=\underset{\Delta \to 0}{\lim }\left(⌊\frac{1}{\Delta \sqrt{2}}⌋\Delta \sqrt{2}\log \left(\Delta \sqrt{2}\right)-⌊\frac{1}{\Delta }⌋\Delta \log \Delta \right)=\log \sqrt{2}.\end{array}$$

(48)

The latter example further illustrates why the notion of effective entropy is appropriate in the setup considered in this section, where the effective dimension of the random sequences does not coincide with their length (it is easy to verify that the effective entropy of $\mathbf{y}$ does not change if one rotates $\mathbf{y}$ in ${\mathbb{R}}^{\ell }$).

We finish this section with an example to illustrate the usefulness of the notion of effective differential entropy beyond the context of entropy gain.

Application Example: Shannon Lower Bound

The rate-distortion function (RDF) $R\left(D\right)$ is the infimum, among all codes, of the expected number of bits per sample necessary to reconstruct a given random source with distortion not greater than D [1]. Let the source and reconstruction be the vectors ${\mathbf{x}}_{\ell }^1$ and ${\mathbf{x}}_{\ell }^1+{\mathbf{v}}_{\ell }^1$, respectively, and suppose the distortion is assessed using the mean-squared error (MSE) $d\left({\mathbf{v}}_{\ell }^1\right)\triangleq \mathrm{E}[\parallel {\mathbf{v}}_{\ell }^1{\parallel }^2]$. Then, restricting our attention to uniquely-decodable codes Reference [1], p. 105), the Shannon Lower Bound (SLB) [20] establishes that

$$\begin{array}{c}\ell R\left(D\right)\ge h\left({\mathbf{x}}_{\ell }^1\right)-\underset{d\left({\mathbf{v}}_{\ell }^1\right)\le D}{\max }h\left({\mathbf{v}}_{\ell }^1\right),\end{array}$$

(49)

provided $h\left({\mathbf{x}}_{\ell }^1\right)$ is bounded. Therefore, if ${\mathrm{x}}_{\ell }^1$ is the entire forced response of an FIR filter G of order p to an input ${\mathrm{u}}_1^n$, then $\ell =n+p$ and $h\left({\mathbf{x}}_{\ell }^1\right)$ is minus infinity, which precludes one from using (49). We will show next that, in this case, the SLB can still be stated by using the effective differential entropy $\breve{h}\left({\mathbf{x}}_{\ell }^1\right)$ instead of $h\left({\mathbf{x}}_{\ell }^1\right)$. Following Definition 1, we can write the source vector as ${\mathbf{x}}_{\ell }^1={\mathit{A}}^T\mathit{T}\mathit{C}{\mathbf{u}}_n^1$, where $\mathit{A}\in {\mathbb{R}}^{n\times \ell }$ has orthonormal rows, $\mathit{T}\in {\mathbb{R}}^{n\times n}$ is diagonal with non-negative entries, and $\mathit{C}\in {\mathbb{R}}^{n\times n}$ is unitary. Let $\mathit{H}\triangleq \left[{\mathit{A}}^T\right|{\overline{\mathit{A}}}^T]\in {\mathbb{R}}^{\ell \times \ell }$ be a unitary matrix, which means that $\overline{\mathit{A}}{\mathit{A}}^T={\mathbf{0}}_{p\times \ell }$. Then,

$$\begin{array}{cc}\ell R\left(D\right)& \stackrel{\left(a\right)}{\ge }I({\mathbf{x}}_{\ell }^1;{\mathbf{x}}_{\ell }^1+{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(50)

$$\begin{array}{cc}& \stackrel{\phantom{\left(a\right)}}{=}I(\mathit{H}{\mathbf{x}}_{\ell }^1;\mathit{H}{\mathbf{x}}_{\ell }^1+\mathit{H}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(51)

$$\begin{array}{cc}& \stackrel{\phantom{\left(a\right)}}{=}I(\mathit{A}{\mathbf{x}}_{\ell }^1;\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{H}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(52)

$$\begin{array}{cc}& \stackrel{\phantom{\left(a\right)}}{=}I(\mathit{A}{\mathbf{x}}_{\ell }^1;\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{A}{\mathbf{v}}_{\ell }^1,\overline{\mathit{A}}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(53)

$$\begin{array}{cc}& \stackrel{\left(21\right)}{=}I(\mathit{A}{\mathbf{x}}_{\ell }^1;\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{A}{\mathbf{v}}_{\ell }^1),+I(\mathit{A}{\mathbf{x}}_{\ell }^1;\overline{\mathit{A}}{\mathbf{v}}_{\ell }^1|\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{A}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(54)

$$\begin{array}{cc}& \stackrel{\left(20\right)}{\ge }I(\mathit{A}{\mathbf{x}}_{\ell }^1;\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{A}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(55)

$$\begin{array}{cc}& \stackrel{\left(22\right)}{=}h\left(\mathit{A}{\mathbf{x}}_{\ell }^1\right)-h\left(\mathit{A}{\mathbf{x}}_{\ell }^1\right|\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{A}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(56)

$$\begin{array}{cc}& \stackrel{\left(19\right)}{=}h\left(\mathit{A}{\mathbf{x}}_{\ell }^1\right)-h\left(\mathit{A}{\mathbf{v}}_{\ell }^1\right|\mathit{A}{\mathbf{x}}_{\ell }^1+\mathit{A}{\mathbf{v}}_{\ell }^1),\hfill \end{array}$$

(57)

$$\begin{array}{cc}& \stackrel{\left(b\right)}{\ge }h\left(\mathit{A}{\mathbf{x}}_{\ell }^1\right)-h\left(\mathit{A}{\mathbf{v}}_{\ell }^1\right),\hfill \end{array}$$

(58)

$$\begin{array}{cc}& \stackrel{\phantom{\left(a\right)}}{\ge }h\left(\mathit{A}{\mathbf{x}}_{\ell }^1\right)-\underset{{\mathbf{w}}_n^1:d\left({\mathbf{w}}_n^1\right)\le D}{\max }h\left({\mathbf{w}}_n^1\right),\hfill \end{array}$$

(59)

$$\begin{array}{cc}& \stackrel{\left(c\right)}{=}\breve{h}\left({\mathbf{x}}_{\ell }^1\right)-\underset{{\mathbf{w}}_n^1:d\left({\mathbf{w}}_n^1\right)\le D}{\max }h\left({\mathbf{w}}_n^1\right),\hfill \end{array}$$

(60)

where $\left(a\right)$ stems from Reference [1], Theorems 5.4.1 and 5.5.1 and Equations (10).58-10.61, $\left(b\right)$ holds because conditioning does not increase entropy and $\left(c\right)$ is from the definition of effective differential entropy.

5. Entropy-Balanced Processes: Geometric Interpretation and Properties

In the first part of this section, we provide a geometric interpretation of the effect that a non-minimum phase LTI system has on its input random process. This will give an intuitive meaning to the notion of an entropy-balanced random process (introduced in Definition 2 above) and provide insights into why and how the entropy gain defined in (1) arises as a consequence of an output random disturbance or a random initial state (the themes of Section 6 and Section 7, respectively).

The second part of this section identifies several entropy-balanced processes and establishes two properties satisfied by this class of processes.

5.1. Geometric Interpretation

We begin our discussion with a simple example.

Example 1.

Suppose that G in Figure 1 is a finite impulse response (FIR) filter with impulse response $g_0=1,g_1=2,g_i=0,\forall i\ge 2$. Notice that this choice yields $G\left(z\right)=(z-2)/z$; thus, $G\left(z\right)$ has one non-minimum phase zero, at $z=2$. The associated matrix ${\mathit{G}}_n$ for $n=3$ is

$${\mathit{G}}_3=\left(\begin{array}{ccc}1& 0& 0\\ 2& 1& 0\\ 0& 2& 1\end{array}\right),$$

whose determinant is clearly one (indeed, all its eigenvalues are 1). Hence, as discussed in the introduction, $h\left({\mathit{G}}_3{\mathbf{u}}_3^1\right)=h\left({\mathbf{u}}_3^1\right)$; thus, ${\mathit{G}}_3$ (and ${\mathit{G}}_n$, in general) does not introduce an entropy gain by itself. However, an interesting phenomenon becomes evident by looking at the SVD of ${\mathit{G}}_3$, given by ${\mathit{G}}_3={\mathit{Q}}_3^T{\mathit{D}}_3{\mathit{R}}_3$, where ${\mathit{Q}}_3$ and ${\mathit{R}}_3$ are unitary matrices, and ${\mathit{D}}_3\triangleq \mathrm{diag}\{d_1,d_2,d_3\}$. In this case, ${\mathit{D}}_3=\mathrm{diag}\{0.19394,1.90321,2.70928\}$; thus, one of the singular values of ${\mathit{G}}_3$ is much smaller than the others (although the product of all singular values yields 1, as expected). As will be shown in Section 6, for a stable $G\left(z\right)$, such uneven distribution of singular values arises only when $G\left(z\right)$ has non-minimum phase zeros. The effect of this can be visualized by looking at the image of the cube ${[0,1]}^3$ through ${\mathit{G}}_3$, shown in Figure 3.

If the input ${\mathbf{u}}_3^1$ were uniformly distributed over this cube (of unit volume), then ${\mathit{G}}_3{\mathbf{u}}_3^1$ would distribute uniformly over the unit-volume parallelepiped depicted in Figure 3; hence, $h\left({\mathit{G}}_3{\mathbf{u}}_3^1\right)=h\left({\mathbf{u}}_3^1\right)$.

Now, if we add to ${\mathit{G}}_3{\mathbf{u}}_3^1$ a disturbance ${\mathbf{z}}_3^1=\mathsf{\Phi }\mathrm{s}$, with scalar s uniformly distributed over $[-0.5,0.5]$ independent of ${\mathbf{u}}_3^1$, and with $\mathsf{\Phi }\in {\mathbb{R}}^{3\times 1}$, the effect would be to “thicken” the support over which the resulting random vector ${\mathbf{y}}_3^1={\mathit{G}}_3{\mathbf{u}}_3^1+{\mathbf{z}}_3^1$ is distributed, along the direction pointed by $\mathsf{\Phi }$. If $\mathsf{\Phi }$ is aligned with the direction along which the support of ${\mathit{G}}_3{\mathbf{u}}_3^1$ is thinnest (given by ${\mathit{q}}_{3,1}$, the first row of ${\mathit{Q}}_3$), then the resulting support would have its volume significantly increased, which can be associated with a large increase in the differential entropy of ${\mathbf{y}}_3^1$ with respect to ${\mathbf{u}}_3^1$. Indeed, a relatively small variance of s and an approximately aligned $\mathsf{\Phi }$ would still produce a significant entropy gain.

The above example suggests that the entropy gain from ${\mathbf{u}}_n^1$ to ${\mathbf{y}}_n^1$ appears as a combination of two factors. The first of these is the uneven way in which the random vector ${\mathit{G}}_n{\mathbf{u}}_n^1$ is distributed over ${\mathbb{R}}^n$. The second factor is the alignment of the disturbance vector ${\mathbf{z}}_n^1$ with respect to the span of the subset ${\left\{{\mathit{q}}_{n,i}\right\}}_{i\in {\Omega }_n}$ of columns of ${\mathit{Q}}_n$, associated with the smallest singular values of ${\mathit{G}}_n$, indexed by the elements in the set ${\Omega }_n$. As we shall discuss in the next section, if G has m non-minimum phase zeros, then, as n increases, there will be m singular values of ${\mathit{G}}_n$ going to zero exponentially. Since the product of the singular values of ${\mathit{G}}_n$ equals 1 for all n, it follows that ${\prod }_{i\notin {\Omega }_n}{d}_{n,i}$ must grow exponentially with n, where $d_{n,i}$ is the i-th diagonal entry of ${\mathit{D}}_n$. This implies that ${\mathit{G}}_n{\mathbf{u}}_n^1$ expands with n along the span of ${\left\{{\mathit{q}}_{n,i}\right\}}_{i\notin {\Omega }_n}$, compensating its shrinkage along the span of ${\left\{{\mathit{q}}_{n,i}\right\}}_{i\in {\Omega }_n}$, thus keeping $h\left({\mathit{G}}_n{\mathbf{u}}_n^1\right)=h\left({\mathbf{u}}_n^1\right)$ for all n. Thus, as n grows, any small disturbance distributed over the span of ${\left\{{\mathit{q}}_{n,i}\right\}}_{i\in {\Omega }_n}$, added to ${\mathit{G}}_n{\mathbf{u}}_n^1$, will keep the support of the resulting distribution from shrinking along this subspace. Consequently, the expansion of ${\mathit{G}}_n{\mathbf{u}}_n^1$ with n along the span of ${\left\{{\mathit{q}}_{n,i}\right\}}_{i\notin {\Omega }_n}$ is no longer compensated, yielding an entropy increase proportional to $\log ({\prod }_{i\notin {\Omega }_n}{d}_{n,i})$.

The above analysis allows one to anticipate a situation in which no entropy gain would take place even when some singular values of ${\mathit{G}}_n$ tend to zero as $n\to \infty$. Since the increase in entropy is made possible by the fact that, as n grows, the support of the distribution of ${\mathit{G}}_n{\mathbf{u}}_n^1$ shrinks along the span of ${\left\{{\mathit{q}}_{n,i}\right\}}_{i\in {\Omega }_n}$, no such entropy gain should arise if the support of the distribution of the input ${\mathbf{u}}_n^1$ expands accordingly along the directions pointed by the rows ${\left\{{\mathit{r}}_{n,i}\right\}}_{i\in {\Omega }_n}$ of ${\mathit{R}}_n$.

An example of such situation can be easily constructed as follows: Let $G\left(z\right)$ in Figure 1 have non-minimum phase zeros and suppose that ${\mathrm{u}}_1^{\infty }$ is generated as $G^{-1}{\tilde{\mathrm{u}}}_1^{\infty }$, where ${\tilde{\mathrm{u}}}_1^{\infty }$ is an i.i.d. random process with bounded entropy rate. Since the determinant of ${\mathit{G}}_n^{-1}$ equals 1 for all n, we have that $h\left({\mathbf{u}}_n^1\right)=h\left({\tilde{\mathbf{u}}}_n^1\right)$, for all n. On the other hand, ${\mathbf{y}}_n^1={\mathit{G}}_n{\mathit{G}}_n^{-1}{\tilde{\mathbf{u}}}_n^1+{\mathbf{z}}_n^1={\tilde{\mathbf{u}}}_n^1+{\mathbf{z}}_n^1$. Since ${\mathbf{z}}_n^1={\left[\mathsf{\Phi }\right]}_n^1{\mathbf{s}}_{\kappa }^1$ for some finite $\kappa$ (recall Assumption 2), it is easy to show that ${\lim }_{n\to \infty }\frac{1}{n}h\left({\mathbf{y}}_n^1\right)={\lim }_{n\to \infty }\frac{1}{n}h\left({\tilde{\mathbf{u}}}_n^1\right)={\lim }_{n\to \infty }\frac{1}{n}h\left({\mathbf{u}}_n^1\right)$; thus, no entropy gain appears.

The preceding discussion reveals that the entropy gain produced by G in the situation shown in Figure 1 depends on the distribution of the input and on the support and distribution of the disturbance. This stands in stark contrast with the well known fact that the increase in differential entropy produced by an invertible linear operator depends only on its Jacobian, and not on the statistics of the input [2]. We have also seen that the distribution of a random process along the different directions within the Euclidean space which contains it plays a key role, as well. This motivates the need to specify a class of random processes which distribute more or less evenly over all directions. This is precisely the intuitive meaning of an entropy-balanced process.

The following section identifies a large family of processes belonging to this class, as well as two properties which greatly expands this family.

5.2. Characterization of Entropy-Balanced Processes

We have defined the notion of an “entropy-balanced” process in Section 1.1. In words, the first condition in this definition allows one to guarantee that the orthogonal projection of an entropy-balanced process onto any $\nu$-dimensional linear subspace has a differential entropy whose magnitude remains bounded or grows at most sub-linearly with n. The second condition states that the projection of an entropy-balanced process ${\mathrm{v}}_1^{\infty }$ onto any linear subspaces having $\nu$ fewer dimensions has the same differential entropy rate as the original process. This condition is equivalent to requiring that every unitary transformation on ${\mathrm{v}}_1^n$ yields a random sequence ${\mathrm{y}}_1^n$ such that ${\lim }_{n\to \infty }\frac{1}{n}h\left({\mathrm{y}}_{n-\nu +1}^n\right|{\mathrm{y}}_1^{n-\nu })=0$. This property of the resulting random sequence ${\mathrm{y}}_1^n$ means that one cannot predict its last $\nu$ samples with arbitrary accuracy by using its previous $n-\nu$ samples, even if n goes to infinity.

We now characterize a large family of entropy-balanced random processes and establish some of their properties. Although intuition may suggest that most random processes (such as i.i.d. or stationary processes) should be entropy balanced, that statement seems rather difficult to prove. In the following, we show that the entropy-balanced condition is met by i.i.d. processes with per-sample probability density function (PDF) being uniform, piece-wise constant or Gaussian. It is also shown that adding to an entropy-balanced process an independent random processes independent of the former yields another entropy-balanced process, and that filtering an entropy-balanced process by a stable and minimum phase filter yields an entropy-balanced process, as well. The proofs can be found in Appendix B.

Lemma 1.

Let ${\mathrm{u}}_1^{\infty }$ be a Gaussian random process with independent elements having positive and bounded variance, i.e., there exist $0<{\check{\sigma }}^2\le {\widehat{\sigma }}^2<\infty$ such that ${\check{\sigma }}^2\le {\sigma }_{\mathrm{u}\left(n\right)}^2\le {\widehat{\sigma }}^2$, $n\in \mathbb{N}$. Then, ${\mathrm{u}}_1^{\infty }$ is entropy balanced.

Lemma 2.

Let ${\mathrm{u}}_1^{\infty }$ be a random process with independent elements satisfying Condition i) in Definition 2, in which each ${\mathrm{u}}_i$ is distributed according to a (possibly different) piece-wise constant PDF such that each interval where this PDF is constant has measure less than θ and greater than ϵ, for some constants $0<ϵ<\theta <\infty$. Then, ${\mathrm{u}}_1^{\infty }$ is entropy balanced.

Lemma 3.

Let ${\mathrm{u}}_1^{\infty }$ and ${\mathrm{v}}_1^{\infty }$ be mutually independent random processes. If ${\mathrm{u}}_1^{\infty }$ is entropy balanced, and ${\mathrm{w}}_1^{\infty }\triangleq {\mathrm{u}}_1^{\infty }+{\mathrm{v}}_1^{\infty }$ satisfies ${\sigma }_{\mathrm{w}\left(n\right)}^2<\infty$ for finite n and ${\lim }_{n\to \infty }{n}^{-1}\log \left({\sigma }_{\mathrm{w}\left(n\right)}^2\right)=0$, then ${\mathrm{w}}_1^{\infty }$ is also entropy balanced.

The proof of Lemma 3 is on page 33. The working behind this lemma can be interpreted intuitively by noting that adding to a random process another independent random process can only increase the “spread” of the distribution of the former, which tends to balance the entropy of the resulting process along all dimensions in Euclidean space. In addition, it follows from Lemma 3 that all i.i.d. processes having a per-sample PDF which can be constructed by convolving uniform, piece-wise constant or Gaussian PDFs as many times as required are entropy balanced. It also implies that one can have non-stationary processes which are entropy balanced, since Lemma 3 imposes no requirements for the process ${\mathrm{v}}_1^{\infty }$.

The next lemma related to the properties of entropy-balanced processes shows that filtering by a stable and minimum phase LTI filter preserves the entropy balanced condition of its input.

Lemma 4.

Let ${\mathrm{u}}_1^{\infty }$ be an entropy-balanced process and G an LTI stable and minimum-phase filter. Then, the output ${\mathrm{w}}_1^{\infty }\triangleq G{\mathrm{u}}_1^{\infty }$ is also an entropy-balanced process.

This result implies that any stable moving-average auto-regressive process constructed from entropy-balanced innovations is also entropy balanced, provided the coefficients of the averaging and regression correspond to a stable MP filter.

The last lemma of this section states a crucial property of entropy-balanced processes (the proof is in Appendix B, page 34).

Lemma 5.

Let ${\mathrm{u}}_1^{\infty }$ be an entropy balanced process. Consider a disturbance ${\mathrm{z}}_1^{\infty }$ satisfying Assumption 2 and define ${\mathrm{y}}_1^{\infty }\triangleq {\mathrm{u}}_1^{\infty }+{\mathrm{z}}_1^{\infty }$. Then, ${\lim }_{n\to \infty }{n}^{-1}(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right))=0$.

We finish this section by pointing out two examples of processes which are non-entropy-balanced, namely the output of a NMP-filter to an entropy-balanced input and the output of an unstable filter to an entropy-balanced input. The first of these cases plays a central role in the next section.

6. Entropy Gain Due to External Disturbances

In this section, we formalize the ideas which were qualitatively outlined in the previous section. Specifically, for the system shown in Figure 1 we will characterize the entropy gain $\mathcal{G}(G,{\mathbf{x}}_0,{\mathrm{u}}_1^{\infty },{\mathrm{z}}_1^{\infty })$ defined in (1) for the case in which the initial state ${\mathbf{x}}_0$ is zero (or deterministic) and there exists a random disturbance of (possibly infinite length) ${\mathrm{z}}_1^{\infty }$ which satisfies Assumption 2.

6.1. Input Disturbances Do Not Produce Entropy Gain

In this section, we show that random disturbances satisfying Assumption 2, when added to the input ${\mathrm{u}}_1^{\infty }$ (i.e., before G), do not introduce entropy gain. This result can be obtained from Lemma 6, as stated in the following theorem:

Theorem 5

(Input Disturbances do not Introduce Entropy Gain). Let G and ${\mathrm{z}}_1^{\infty }$ satisfy Assumptions 1 and 2, respectively. Suppose that ${\mathrm{u}}_1^{\infty }$ is entropy balanced and consider the output

$$\begin{array}{c}{\mathrm{y}}_1^{\infty }=G({\mathrm{u}}_1^{\infty }+{\mathrm{z}}_1^{\infty }).\end{array}$$

(61)

Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\left(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right)\right)=0\end{array}$$

(62)

Proof. 

From Lemma 5, the differential entropy rate of ${\mathrm{u}}_1^{\infty }$ equals that of ${\mathrm{u}}_1^{\infty }+{\mathrm{z}}_1^{\infty }$. The proof is completed by recalling that G yields no entropy gain for its input ${\mathrm{u}}_1^{\infty }+{\mathrm{z}}_1^{\infty }$ because it corresponds to the noise-less scenario.    □

6.2. The Entropy Gain Introduced by Output Disturbances when G is MP is Zero

The results from the previous section yield the following corollary, which states that an LTI system with transfer function $G\left(z\right)$ without zeros outside the unit circle (i.e., an MP transfer function) cannot introduce entropy gain.

Corollary 1

(Minimum Phase Filters do not Introduce Entropy Gain). Consider the system shown in Figure 1 wherein the input ${\mathrm{u}}_1^{\infty }$ is an entropy-balanced random process and the output disturbance ${\mathrm{z}}_1^{\infty }$ satisfies Assumption 2. Besides Assumption 1, suppose that $G\left(z\right)$ is minimum phase. Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\left(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right)\right)=0.\end{array}$$

(63)

Proof. 

Since $G\left(z\right)$ is minimum phase and stable, the result follows directly from Lemmas 4 and 5.    □

6.3. The Entropy Gain Introduced by Output Disturbances when $G\left(z\right)$ is NMP

We show here that the entropy gain of an LTI system with transfer function $G\left(z\right)$ and an output disturbance is at most the sum of the logarithm of the magnitude of the zeros of $G\left(z\right)$ outside the unit circle.

The following lemma will be instrumental for that purpose.

Lemma 6.

Consider the system in Figure 1, and suppose ${\mathrm{z}}_1^{\infty }$ satisfies Assumption 2, and that the input process ${\mathrm{u}}_1^{\infty }$ is entropy balanced. Let ${\mathit{G}}_n={\mathit{Q}}_n^T{\mathit{D}}_n{\mathit{R}}_n$ be the SVD of ${\mathit{G}}_n$, where ${\mathit{D}}_n=\mathrm{diag}\{d_{n,1},\dots ,d_{n,n}\}$ are the singular values of ${\mathit{G}}_n$, with $d_{n,1}\le d_{n,2}\le \cdots \le d_{n,n}$, such that $|\det {\mathit{G}}_n|=1$$\forall n$. Let m be the number of these singular values which tend to zero exponentially as $n\to \infty$. Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\left(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right)\right)=\underset{n\to \infty }{\lim }\frac{1}{n}\left(-\sum _{i=1}^m\log d_{n,i}+h\left({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{u}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\mathbf{z}}_n^1\right)\right).\end{array}$$

(64)

The proof of this lemma can be found on page 34, in Appendix B.

Lemma 6 leaves the need to characterize the asymptotic behavior of the singular values of ${\mathit{G}}_n$. This is accomplished in the following lemma, which relates these singular values to the zeros of $G\left(z\right)$. It is a generalization of the unnumbered lemma in the proof of Reference [16], Theorem 1 (restated in Appendix C as Lemma A3), which holds for FIR transfer functions, to the case of infinite-impulse response (IIR) transfer functions (i.e., transfer functions having poles).

Lemma 7.

For a transfer function $G\left(z\right)$ satisfying Assumption 1, where its zeros ${\left\{{\rho }_i\right\}}_{i=1}^p$ satisfy $|{\rho }_1|\ge \cdots \ge |{\rho }_m|>1\ge |{\rho }_{m+1}|\ge \cdots \ge |{\rho }_p|$. Then,

$$\begin{array}{c}{\lambda }_l\left({\mathit{G}}_n{\mathit{G}}_n^T\right)=\left\{\begin{array}{cc}{\alpha }_{n,l}^2{\left|{\rho }_l\right|}^{-2n}\hfill & ,\mathrm{if}l\le m,\hfill \\ {\alpha }_{n,l}^2\hfill & ,otherwise,\hfill \end{array}\right.\end{array}$$

(65)

where the elements in the sequence $\left\{{\alpha }_{n,l}\right\}$ are positive and increase or decrease at most polynomially with n.

(The proof of this lemma can be found in Appendix B, page 36).

Lemma 6 also precisely formulates the geometric idea outlined in Section 5.1. To see this, notice that no entropy gain is obtained if the output disturbance vector ${\mathbf{z}}_n^1$ becomes orthogonal (with probability 1) to the space spanned by the first m columns of ${\mathit{Q}}_n$ sufficiently fast as $n\to \infty$. Recalling from Assumption 2 that

$$\begin{array}{c}{\mathbf{z}}_n^1={\left[\mathsf{\Phi }\right]}_n^1{\mathbf{s}}_{\kappa }^1,\end{array}$$

(66)

where the matrix $\mathsf{\Phi }$ has $\kappa$ orthonormal columns of infinite length, such orthogonality condition can be formally stated by defining

$$\begin{array}{c}{\kappa }_n\triangleq \mathrm{rank}\left({\left[{\mathit{Q}}_n\right]}_m^1{\left[\mathsf{\Phi }\right]}_n^1\right)\end{array}$$

(67)

$$\begin{array}{c}{\widehat{\kappa }}_{\infty }\triangleq \underset{n\to \infty }{\lim \sup }{\kappa }_n\end{array}$$

(68)

$$\begin{array}{c}{\check{\kappa }}_{\infty }\triangleq \underset{n\to \infty }{\lim \inf }{\kappa }_n\end{array}$$

(69)

as $\widehat{\kappa }=0$.

If this were the case, then the disturbance would not be able fill the subspace along which ${\mathit{G}}_n{\mathbf{u}}_n^1$ is shrinking exponentially. Indeed, if ${\kappa }_n=0$ for all n, then $h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{u}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\mathbf{z}}_n^1)=h({}^1{\left[{\mathit{D}}_n\right]}_m{\left[{\mathit{R}}_n\right]}_m^1{\mathbf{u}}_n^1)={\sum }_{i=1}^m\log d_{n,i}+h({\left[{\mathit{R}}_n\right]}_m^1{\mathbf{u}}_n^1)$, and the latter sum cancels out the one on the RHS of (64), while ${\lim }_{n\to \infty }\frac{1}{n}h({\left[{\mathit{R}}_n\right]}_m^1{\mathbf{u}}_n^1)=0$ since ${\mathrm{u}}_1^{\infty }$ is entropy balanced. On the contrary (and loosely speaking), if the projection of the support of ${\mathbf{z}}_n^1$ onto the subspace spanned by the first m rows of ${\mathit{Q}}_n$ is of dimension m (i.e., if ${\kappa }_n=m$) for all n, then $h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{u}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\mathbf{z}}_n^1)$ remains bounded for all n, and the entropy limit of the sum ${\lim }_{n\to \infty }\frac{1}{n}(-{\sum }_{i=1}^m\log d_{n,i})$ on the RHS of (64) yields the largest possible entropy gain. Notice that $-{\sum }_{i=1}^m\log d_{n,i}={\sum }_{i=m+1}^n\log d_{n,i}$ (because $\det \left({\mathit{G}}_n\right)=1$); thus, this entropy gain stems from the uncompensated expansion of ${\mathit{G}}_n{\mathbf{u}}_n^1$ along the space spanned by the rows of ${\left[{\mathit{Q}}_n\right]}_n^{m+1}$. Beyond these extreme cases (i.e., for general values of $\check{\kappa }$ and $\widehat{\kappa }$), the following theorem provides tight bounds on the entropy gain.

Theorem 6.

In the system of Figure 1, suppose that ${\mathrm{u}}_1^{\infty }$ is entropy balanced, and that $G\left(z\right)$ and ${\mathrm{z}}_1^{\infty }$ satisfy Assumptions 1 and 2, respectively, where the zeros ${\left\{{\rho }_i\right\}}_{i=1}^p$ of $G\left(z\right)$ satisfy $|{\rho }_1|\ge \cdots \ge |{\rho }_m|>1\ge |{\rho }_{m+1}|\ge \cdots \ge |{\rho }_p|$. For each $n\in \mathbb{N}$, let ${\mathit{Q}}_n^T\in {\mathbb{R}}^{n\times n}$ be the unitary matrix holding the left singular vectors of ${\mathit{G}}_n\in {\mathbb{R}}^{n\times n}$ (as in Lemma 6), where ${\mathit{G}}_n$ is as defined in (4).

1.

Then,

$$\begin{array}{c}0\le \underset{n\to \infty }{\lim \inf }\frac{1}{n}\left(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right)\right)\le \underset{n\to \infty }{\lim \sup }\frac{1}{n}(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right))\le \sum _{i=1}^{{\widehat{\kappa }}_{\infty }}\log \left|{\rho }_i\right|\stackrel{\left(8\right)}{\le }\frac{1}{2\pi }\underset{-\pi }{\overset{\pi }{\int }}\log \left|G\left({\mathrm{e}}^{j\omega }\right)\right|d\omega .\end{array}$$

(70)

The bounds on both extremes are tight. Moreover, the lower bound is reached if ${\widehat{\kappa }}_{\infty }=0$.

2.

If ${\lim \inf }_{n\to \infty }\frac{1}{n}\log ({\sigma }_{\min }\left({\left[{\mathit{Q}}_n\right]}_m^1{\left[\mathsf{\Phi }\right]}_n^1\right)=0$, then

$$\begin{array}{c}\sum _{i=m-{\check{\kappa }}_{\infty }+1}^m\log \left|{\rho }_i\right|\le \underset{n\to \infty }{\lim \inf }\frac{1}{n}\left(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right)\right).\end{array}$$

(71)

Thus, the rightmost upper bound in (70) is achieved if ${\check{\kappa }}_{\infty }=m$.

Proof. 

See Appendix B, page 37.    □

The next technical result is very useful for finding conditions under which the requirements of point 2 in Theorem 6 are satisfied (the proof is in Appendix B, page 39).

Lemma 8.

Let F be an FIR LTI causal system of order m such that the m zeros of $F\left(z\right)$ are NMP, and ${\mathit{F}}_n={\mathit{Q}}_n^T{\mathit{D}}_n{\mathit{R}}_n$ be an SVD for ${\mathit{F}}_n$, for every $n\in \{m,m+1,\dots \}$. For each $\kappa \in \{1,\dots ,n\}$, define

$$\begin{array}{c}{\kappa }_n\triangleq \mathrm{rank}\left(\stackrel{1\hspace{1em}\hspace{1em}\kappa }{\overbracket{{\left[{\mathit{Q}}_n\right]}_m^1}}\right),\end{array}$$

(72)

and $\overline{\kappa }\triangleq \min \{m,\kappa \}$. Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }{\sigma }_{\min }\left(\stackrel{1\hspace{1em}\hspace{1em}\kappa }{\overbracket{{\left[{\mathit{Q}}_n\right]}_m^1}}\right)>0,\end{array}$$

(73)

and ${\lim }_{n\to \infty }{\kappa }_n=\overline{\kappa }$.

Now, we can prove Theorem 4.

6.4. Proof of Theorem 4

Factorize $G\left(z\right)$ as $G\left(z\right)=F\left(z\right)\tilde{G}\left(z\right)$, where $\tilde{G}\left(z\right)$ is stable and minimum phase and $F\left(z\right)$ is a stable FIR transfer function with all the m non-minimum-phase zeros of $G\left(z\right)$. Letting ${\tilde{\mathbf{u}}}_n^1\triangleq {\tilde{\mathit{G}}}_n{\mathbf{u}}_n^1$, we have that $h\left({\mathbf{y}}_n^1\right)=h({\mathit{F}}_n{\tilde{\mathbf{u}}}_n^1+{\mathbf{z}}_n^1)$, $h\left({\tilde{\mathbf{u}}}_n^1\right)=h\left({\mathbf{u}}_n^1\right)$, and that ${\tilde{\mathrm{u}}}_1^{\infty }$ is entropy balanced (from Lemma 4). Thus,

$$\begin{array}{c}h\left({\mathbf{y}}_n^1\right)-h\left({\mathbf{u}}_n^1\right)=h({\mathit{G}}_n{\mathbf{u}}_n^1+{\mathbf{z}}_n^1)-h\left({\mathbf{u}}_n^1\right)=h({\mathit{F}}_n{\tilde{\mathbf{u}}}_n^1+{\mathbf{z}}_n^1)-h\left({\tilde{\mathbf{u}}}_n^1\right).\end{array}$$

(74)

This means that the entropy gain of G due to the output disturbance ${\mathrm{z}}_1^{\infty }$ corresponds to the entropy gain of F due to the same output disturbance.

Clearly, ${\mathbf{u}}_1^{\infty }$, $F\left(z\right)$, and ${\mathrm{z}}_1^{\infty }$ satisfy the assumptions of Theorem 6 with $\mathsf{\Phi }={\left[{\mathit{I}}_{\kappa }\right|\mathbf{0}]}^T$ (see Assumption 2). Therefore,

$$\begin{array}{c}{\left[{\mathit{Q}}_n\right]}_m^1{\left[\mathsf{\Phi }\right]}_n^1=\stackrel{1\hspace{1em}\hspace{1em}\kappa }{\overbracket{{\left[{\mathit{Q}}_n\right]}_m^1}}.\end{array}$$

(75)

Combining this with Lemma 8, it readily follows that, for every $\kappa \ge 1$, the condition in point 2 of Theorem 6 is met, and also ${\lim }_{n\to \infty }{\kappa }_n=\overline{\kappa }$. The proof is then completed by substituting ${\lim \inf }_{n\to \infty }{\kappa }_n={\lim \sup }_{n\to \infty }{\kappa }_n=\overline{\kappa }$ into (70) and (71).

7. Entropy Gain Due to a Random Initial State

Here, we analyze the scenario illustrated by Figure 1 for the case in which there exists a random initial state ${\mathbf{x}}_0$ independent of the input ${\mathrm{u}}_1^{\infty }$, and zero (or deterministic) output disturbance.

The treatment of an initial state of the LTI system G requires one to first define an internal model for it. For this purpose, in this section, we consider the state-space realization of G in the Kalman canonical form, given by

$$\begin{array}{c}\mathbf{x}\left(k\right)\triangleq \left[\begin{array}{c}{\mathbf{x}}_{co}\left(k\right)\\ {\mathbf{x}}_{\overline{c}o}\left(k\right)\\ {\mathbf{x}}_{c\overline{o}}\left(k\right)\\ {\mathbf{x}}_{\overline{c}\overline{o}}\left(k\right),\end{array}\right]=\left(\begin{array}{cccc}{\mathit{A}}_{co}& {\mathit{A}}_{12}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathit{A}}_{\overline{c}o}& \mathbf{0}& \mathbf{0}\\ {\mathit{A}}_{31}& {\mathit{A}}_{32}& {\mathit{A}}_{c\overline{o}}& {\mathit{A}}_{34}\\ \mathbf{0}& {\mathit{A}}_{42}& \mathbf{0}& {\mathit{A}}_{\overline{c}\overline{o}}\end{array}\right)\left[\begin{array}{c}{\mathbf{x}}_{co}(k-1)\\ {\mathbf{x}}_{\overline{c}o}(k-1)\\ {\mathbf{x}}_{c\overline{o}}(k-1)\\ {\mathbf{x}}_{\overline{c}\overline{o}}(k-1),\end{array}\right]+\left[\begin{array}{c}{\mathit{b}}_{co}\\ \mathbf{0}\\ {\mathit{b}}_{c\overline{o}}\\ \mathbf{0}\end{array}\right]\mathrm{u}\left(k\right)\end{array}$$

(76a)

$$\begin{array}{c}\mathrm{y}\left(k\right)=\left[\begin{array}{cccc}{\mathit{c}}_{co}^T& {\mathit{c}}_{\overline{c}o}^T& \mathbf{0}& \mathbf{0}\end{array}\right]\mathbf{x}(k-1)+\mathrm{u}\left(k\right),\end{array}$$

(76b)

(see, e.g., Reference [21] or Reference [22], Chapter 6) where the column state vectors ${\mathbf{x}}_{co}\left(k\right)$, ${\mathbf{x}}_{\overline{c}o}\left(k\right)$, ${\mathbf{x}}_{c\overline{o}}\left(k\right)$, ${\mathbf{x}}_{\overline{c}\overline{o}}\left(k\right)$ are, respectively, controllable and observable, non-controllable and observable, controllable and non-observable, and non-controllable and non-observable. There is no loss of generality in choosing this state-space representation, because every state-space representation consistent with a rational transfer function $G\left(z\right)$ can be written in this form (Reference [22], Theorem 6.7).

Since our interest is on the effect of the random initial state of G on its output, we only need to consider the observable subsystem within (76) and without its input, given by

$$\begin{array}{cc}{\mathbf{x}}_o\left(k\right)\triangleq \left[\begin{array}{c}{\mathbf{x}}_{co}\left(k\right)\\ {\mathbf{x}}_{\overline{c}o}\left(k\right)\end{array}\right]& =\underset{{\mathit{A}}_o}{\underbrace{\left(\begin{array}{cc}{\mathit{A}}_{co}& {\mathit{A}}_{12}\\ \mathbf{0}& {\mathit{A}}_{\overline{c}o}\end{array}\right)}}\left[\begin{array}{c}{\mathbf{x}}_{co}(k-1)\\ {\mathbf{x}}_{\overline{c}o}(k-1)\end{array}\right],\hfill \end{array}$$

(77a)

$$\begin{array}{cc}\tilde{\mathrm{y}}\left(k\right)& =\underset{{\mathit{c}}_o^T}{\underbrace{\left[{\mathit{c}}_{co}^T{\mathit{c}}_{\overline{c}o}^T\right]}}{\mathbf{x}}_o(k-1),\hfill \end{array}$$

(77b)

where $\tilde{\mathrm{y}}$ is the natural response of G to its initial state ${\mathbf{x}}_o\left(0\right)$ and ${\mathbf{x}}_{co}\in {\mathbb{R}}^p$ and ${\mathbf{x}}_{\overline{c}o}\in {\mathbb{R}}^q$. We shall decompose $\tilde{\mathrm{y}}$ as

$$\begin{array}{c}{\tilde{\mathrm{y}}}_1^{\infty }={\left[{\tilde{\mathrm{y}}}_{\overline{c}o}\right]}_1^n+{\left[{\tilde{\mathrm{y}}}_{co}\right]}_1^{\infty },\end{array}$$

(78)

where ${\tilde{\mathrm{y}}}_{\overline{c}o}$ and ${\tilde{\mathrm{y}}}_{co}$ are the natural responses of G to initial states ${\left[{\mathbf{0}}_{1\times p}{\mathbf{x}}_{\overline{c}o}{\left(0\right)}^T\right]}^T$ and ${\left[{\mathbf{x}}_{co}{\left(0\right)}^T{\mathbf{0}}_{1\times q}\right]}^T$, respectively. The natural response component ${\tilde{\mathrm{y}}}_{co}$ can be generated by the following minimal state-space representation of $G\left(z\right)$, without the effect of its input u:

$$\begin{array}{c}\underset{{\mathbf{x}}_{co}\left(k\right)}{\underbrace{\left[\begin{array}{c}{\mathrm{x}}_{co,1}\left(k\right)\\ {\mathrm{x}}_{co,2}\left(k\right)\\ {\mathrm{x}}_{co,3}\left(k\right)\\ ⋮\\ {\mathrm{x}}_{co,p}\left(k\right)\end{array}\right]}}=\underset{{\mathit{A}}_{co}}{\underbrace{\left(\begin{array}{ccccc}{b}_1& b_2& b_3& \cdots & b_p\\ 1& 0& 0& \cdots & 0\\ 0& 1& 0& \ddots & ⋮\\ ⋮& \ddots & \ddots & \ddots & 0\\ 0& \cdots & 0& 1& 0\end{array}\right)}}\underset{{\mathbf{x}}_{co}}{\underbrace{\left[\begin{array}{c}{\mathrm{x}}_{co,1}(k-1)\\ {\mathrm{x}}_{co,2}(k-1)\\ {\mathrm{x}}_{co,3}(k-1)\\ ⋮\\ {\mathrm{x}}_{co,p}(k-1)\end{array}\right],}}\end{array}$$

(79)

$$\begin{array}{c}{\tilde{\mathrm{y}}}_{co}\left(k\right)=\underset{{\mathit{a}}^T}{\underbrace{[a_1{a}_2\cdots a_p]}}{\mathbf{x}}_{co}(k-1)+\underset{{\mathit{p}}^T}{\underbrace{[b_1{b}_2\cdots b_p]}}{\mathbf{x}}_{co}(k-1).\end{array}$$

(80)

Now, we can state and prove the main result of this section:

Theorem 7.

Suppose G satisfies Assumption 1 and ${\mathrm{u}}_1^{\infty }$ is entropy balanced. Assume that ${\mathbf{x}}_o\left(0\right)$ (the observable part of the initial state of G) is independent of the input ${\mathrm{u}}_1^{\infty }$, $\left|h\right({\mathbf{x}}_o\left(0\right)\left)\right|<\infty$ and that $\mathrm{tr}\left\{{\mathit{K}}_{{\mathbf{x}}_o\left(0\right)}\right\}<\infty$. Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}(h\left({\mathrm{y}}_1^n\right)-h\left({\mathrm{u}}_1^n\right))={\sum }_{i=1}^m\log \left|{\rho }_i\right|.\end{array}$$

(81)

Proof. 

Both G and ${\mathrm{u}}_1^{\infty }$ satisfy the conditions of Theorem 6. Thus, as in its statement, we write $G\left(z\right)=F\left(z\right)\tilde{G}\left(z\right)$, where $\tilde{G}\left(z\right)$ is stable and minimum phase and $F\left(z\right)$ is a stable FIR transfer function with only the m non-minimum-phase zeros of $G\left(z\right)$.

Defining ${\mathbf{w}}_n^1\triangleq {\tilde{\mathit{G}}}_n{\mathbf{u}}_n^1$, we have

$$\begin{array}{c}{\mathbf{y}}_n^1={\mathit{F}}_n{\tilde{\mathit{G}}}_n{\mathbf{u}}_n^1+{\tilde{\mathbf{y}}}_n^1={\mathit{F}}_n{\mathbf{w}}_n^1+{\tilde{\mathbf{y}}}_n^1,\end{array}$$

(82)

$$\begin{array}{c}h\left({\mathbf{w}}_n^1\right)=h\left({\mathbf{u}}_n^1\right),\end{array}$$

(83)

and ${\tilde{\mathbf{y}}}_n^1{\displaystyle ⫫}{\mathbf{w}}_n^1$. In addition, the fact that G is stable guarantees that the sample second moment of ${\tilde{\mathrm{y}}}_1^{\infty }$ decays exponentially, which means that ${\tilde{\mathrm{y}}}_1^{\infty }$ satisfies Assumption 2. Thus, the conditions of Lemma 6 are met considering ${\mathit{G}}_n={\mathit{F}}_n$, where now ${\mathit{F}}_n={\mathit{Q}}_n^T{\mathit{D}}_n{\mathit{R}}_n$ is the SVD for ${\mathit{F}}_n$, and $d_{n,1}\le d_{n,2}\le \cdots \le d_{n,n}$. Consequently, the proof would be completed if we can show that ${\lim }_{n\to \infty }\frac{1}{n}h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{w}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\tilde{\mathbf{y}}}_n^1)=0$. But all the involved variables have bounded variance, while ${\mathit{R}}_n$ is unitary, ${\left[{\mathit{Q}}_n\right]}_m^1$ has orthonormal rows and the entries of ${\left[{\mathit{D}}_n\right]}_m^1$ decay exponentially with n. This implies that ${\lim }_{n\to \infty }\frac{1}{n}h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{w}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\tilde{\mathbf{y}}}_n^1)\le 0$. Therefore, it is only left to prove that

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{w}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\tilde{\mathbf{y}}}_n^1)\ge 0.\end{array}$$

(84)

Recalling (78), let us decompose ${\left[{\tilde{\mathbf{y}}}_{co}\right]}_n^1$ so that

$$\begin{array}{cc}{\tilde{\mathbf{y}}}_n^1& ={\mathit{F}}_n{\tilde{\mathit{P}}}_n{\mathbf{x}}_{co}\left(0\right)+{\mathit{P}}_n{\mathbf{x}}_{co}\left(0\right)+{\left[{\tilde{\mathbf{y}}}_{\overline{c}o}\right]}_n^1,\hfill \end{array}$$

(85)

where ${\tilde{\mathit{P}}}_n,{\mathit{P}}_n\in {\mathbb{R}}^{n\times (p+q)}$, the sequences ${\mathit{F}}_n{\tilde{\mathit{P}}}_n{\mathbf{x}}_{co}\left(0\right)$ and ${\mathit{P}}_n{\mathbf{x}}_{co}\left(0\right)$, respectively, are the natural responses of $\tilde{G}$ and F to the controllable and observable initial state ${\mathbf{x}}_{co}$, and ${\left[{\tilde{\mathbf{y}}}_{\overline{c}o}\right]}_n^1$ is the natural response of G to the non-controllable and observable initial state ${\mathbf{x}}_{\overline{c}o}\left(0\right)$. Then,

$$\begin{array}{cc}h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{w}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\tilde{\mathbf{y}}}_n^1)& \stackrel{\left(a\right)}{\ge }h({\left[{\mathit{Q}}_n\right]}_m^1{\tilde{\mathbf{y}}}_n^1)\stackrel{\left(85\right)}{=}h({\left[{\mathit{Q}}_n\right]}_m^1({\mathit{F}}_n{\tilde{\mathit{P}}}_n{\mathbf{x}}_{co}\left(0\right)+{\mathit{P}}_n{\mathbf{x}}_{co}\left(0\right)+{\left[{\tilde{\mathbf{y}}}_{\overline{c}o}\right]}_n^1)),\hfill \end{array}$$

(86)

$$\begin{array}{c}\stackrel{\left(b\right)}{=}h({\left[{\mathit{Q}}_n\right]}_m^1({\mathit{F}}_n{\tilde{\mathit{P}}}_n{\mathbf{x}}_{co}\left(0\right)+{\mathit{P}}_n{\mathbf{x}}_{co}\left(0\right))|{\mathbf{x}}_{\overline{c}o}\left(0\right))=h({\left[{\mathit{Q}}_n\right]}_m^1({\mathit{F}}_n{\tilde{\mathit{P}}}_n+{\mathit{P}}_n){\mathbf{x}}_{co}\left(0\right)|{\mathbf{x}}_{\overline{c}o}\left(0\right)),\hfill \end{array}$$

(87)

where $\left(a\right)$ is from the entropy-power inequality [1] and $\left(b\right)$ holds because conditioning does not increase entropy and ${\left[{\tilde{\mathbf{y}}}_{\overline{c}o}\right]}_n^1$ is a deterministic function of ${\mathbf{x}}_{\overline{c}o}\left(0\right)$. Let the SVD of ${\left[{\mathit{Q}}_n\right]}_m^1({\mathit{F}}_n{\tilde{\mathit{P}}}_n+{\mathit{P}}_n)$ be

$$\begin{array}{c}{\left[{\mathit{Q}}_n\right]}_m^1({\mathit{F}}_n{\tilde{\mathit{P}}}_n+{\mathit{P}}_n)={\mathit{S}}_n{\mathit{T}}_n{\mathit{H}}_n,n=m,m+1,\dots ,\end{array}$$

(88)

where ${\mathit{S}}_n\in {\mathbb{R}}^{m\times m}$ is unitary, ${\mathit{T}}_n=\mathrm{diag}\{t_1,t_2,\dots ,t_m\}$ holds the singular values of ${\left[{\mathit{Q}}_n\right]}_m^1({\mathit{F}}_n{\tilde{\mathit{P}}}_n+{\mathit{P}}_n)$ and ${\mathit{H}}_n\in {\mathbb{R}}^{m\times p}$ has orthonormal rows. Substituting this SVD into (87) we obtain

$$\begin{array}{c}h({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{w}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\tilde{\mathbf{y}}}_n^1)\ge h\left({\mathit{S}}_n{\mathit{T}}_n{\mathit{H}}_n{\mathbf{x}}_{co}\left(0\right)\right|{\mathbf{x}}_{\overline{c}o}\left(0\right))=\log \left(\det \left({\mathit{T}}_n\right)\right)+h\left({\mathit{H}}_n{\mathbf{x}}_{co}\left(0\right)\right|{\mathbf{x}}_{\overline{c}o}\left(0\right)).\end{array}$$

(89)

This last differential entropy is bounded because $\left|h\right({\mathbf{x}}_o\left)\right|<\infty$ and $\mathrm{tr}\left\{{\mathit{K}}_{{\mathbf{x}}_o}\right\}<\infty$, which implies (thanks to Proposition A1) that $\left|h\right({\mathit{H}}_n{\mathbf{x}}_{co},{\mathbf{x}}_{\overline{c}o}\left)\right|<\infty$, and by the chain rule of entropy,

$$\begin{array}{c}h({\mathit{H}}_n{\mathbf{x}}_{co},{\mathbf{x}}_{\overline{c}o})=h\left({\mathbf{x}}_{\overline{c}o}\right)+h\left({\mathit{H}}_n{\mathbf{x}}_{co}\left(0\right)\right|{\mathbf{x}}_{\overline{c}o}\left(0\right)),\end{array}$$

(90)

so $\left|h\right({\mathit{H}}_n{\mathbf{x}}_{co}\left(0\right)\left|{\mathbf{x}}_{\overline{c}o}\left(0\right)\right)|<\infty$ because $\left|h\right({\mathbf{x}}_{\overline{c}o}\left(0\right)\left)\right|<\infty$ (again from Proposition A1). Thus, in view of (89) and (84), all that remains to prove is that

$$\begin{array}{c}\underset{n\to \infty }{\lim }{\sigma }_{\min }\left({\left[{\mathit{Q}}_n\right]}_m^1\left({\mathit{F}}_n{\tilde{\mathit{P}}}_n+{\mathit{P}}_n\right)\right)>0,\end{array}$$

(91)

For that purpose, notice that ${\left[{\mathit{Q}}_n\right]}_m^1\left({\mathit{F}}_n{\tilde{\mathit{P}}}_n+{\mathit{P}}_n\right)={}^1{\left[{\mathit{D}}_n\right]}_m{\left[{\mathit{R}}_n\right]}_m^1{\tilde{\mathit{P}}}_n+{\left[{\mathit{Q}}_n\right]}_m^1{\mathit{P}}_n$. Therefore, from Lemma A4 (in Appendix C), it follows that (91) holds if

$$\begin{array}{c}\underset{n\to \infty }{\lim }{\sigma }_{\min }\left({\left[{\mathit{Q}}_n\right]}_m^1{\mathit{P}}_n\right)>0,\end{array}$$

(92)

and

$$\begin{array}{c}\underset{n\to \infty }{\lim }{\sigma }_{\max }\left({}^1{\left[{\mathit{D}}_n\right]}_m{\left[{\mathit{R}}_n\right]}_m^1{\tilde{\mathit{P}}}_n\right)=0.\end{array}$$

(93)

To prove (93), recall that the entries in the diagonal matrix ${}^1{\left[{\mathit{D}}_n\right]}_m$ decay exponentially with n. On the other hand, the rows of ${\left[{\mathit{R}}_n\right]}_m^1$ are orthonormal. Finally, the fact that $\tilde{G}$ is stable implies that the $p+q$ columns of ${\tilde{\mathit{P}}}_n$ have norms which are bounded for all n. These three observations readily yield that (93) holds.

To prove that (92) holds, write the rational transfer function of G (described by (80)) as

$$\begin{array}{c}G\left(z\right)=\frac{1+a_1{z}^{-1}+\cdots +a_p{z}^{-p}}{1+b_1{z}^{-1}+\cdots +b_p{z}^{-p}}=\underset{F\left(z\right)}{\underbrace{(1+f_1{z}^{-1}+\cdots +f_m{z}^{-m})}}\underset{\tilde{G}\left(z\right)}{\underbrace{\frac{1+{\tilde{a}}_1{z}^{-1}+\cdots +{\tilde{a}}_{\tilde{m}}{z}^{-\tilde{m}}}{1+b_1{z}^{-1}+\cdots +b_p{z}^{-p}}}},\end{array}$$

(94)

where $\tilde{m}\triangleq p-m$. The coefficients in the numerator of $G\left(z\right)$ are related to those of $F\left(z\right)$ and $\tilde{G}\left(z\right)$ by the convolution

$$\begin{array}{c}{a}_i={\sum }_{j=0}^m{f}_j{\tilde{a}}_{i-j},i=1,\dots ,p,\end{array}$$

(95)

where ${\tilde{a}}_0=f_0=1$.

Denote the natural response of F (up to time n) to its initial state ${\mathbf{x}}_F\left(0\right)$ (which is a linear function of ${\mathbf{x}}_{co}\left(0\right)$) as

$$\begin{array}{c}{\ddot{\mathbf{y}}}_n^1\triangleq {\mathit{P}}_n{\mathbf{x}}_{co}\left(0\right).\end{array}$$

(96)

Let ${\tilde{\mathbf{w}}}_n^1\triangleq {\tilde{\mathit{P}}}_n{\mathbf{x}}_{co}\left(0\right)$ be the natural response of $\tilde{G}$ to its initial state ${\mathbf{x}}_{co}\left(0\right)$. Following the structure of (80), $\tilde{\mathrm{w}}\left(k\right)$ can be written as

$$\begin{array}{c}\tilde{\mathrm{w}}\left(k\right)=[{\tilde{a}}_1\cdots {\tilde{a}}_{\tilde{m}}0\cdots 0]{\mathbf{x}}_{co}(k-1)+{\mathit{p}}^T{\mathbf{x}}_{co}(k-1),k=1,2,\dots ,\end{array}$$

(97)

where ${\mathbf{x}}_{co}$ satisfies (79). Considering the following minimal state-space representation of F

$$\begin{array}{c}{\mathbf{x}}_F\left(k\right)\triangleq \left[\begin{array}{c}{\mathrm{x}}_{F,1}\left(k\right)\\ {\mathrm{x}}_{F,2}\left(k\right)\\ {\mathrm{x}}_{F,3}\left(k\right)\\ ⋮\\ {\mathrm{x}}_{F,m}\left(k\right)\end{array}\right]=\underset{{\mathit{A}}_F}{\underbrace{\left(\begin{array}{ccccc}0& 0& \cdots & \cdots & 0\\ 1& 0& 0& \cdots & 0\\ 0& 1& 0& \ddots & ⋮\\ ⋮& \ddots & \ddots & \ddots & 0\\ 0& \cdots & 0& 1& 0\end{array}\right)}}\left[\begin{array}{c}{\mathrm{x}}_{F,1}(k-1)\\ {\mathrm{x}}_{F,2}(k-1)\\ {\mathrm{x}}_{F,3}(k-1)\\ ⋮\\ {\mathrm{x}}_{F,m}(k-1)\end{array}\right]+\left[\begin{array}{c}1\\ 0\\ 0\\ ⋮\\ 0\end{array}\right]\mathrm{w}\left(k\right),\end{array}$$

(98)

$$\begin{array}{c}{\tilde{\mathrm{y}}}_{co}\left(k\right)=\underset{{\mathit{c}}_F^T}{\underbrace{[f_1{f}_2\cdots f_m]}}{\mathbf{x}}_F(k-1)+\tilde{\mathrm{w}}\left(k\right),\end{array}$$

(99)

it can be seen that the natural response of F to its own initial state ${\mathbf{x}}_F\left(0\right)$ can be written as

$$\begin{array}{c}\ddot{\mathrm{y}}\left(k\right)={\tilde{\mathrm{y}}}_{co}\left(k\right)-\tilde{\mathrm{w}}\left(k\right)-“\mathrm{the}\mathrm{effect}\mathrm{of}{f}_1,\dots ,f_{k-1}”.\end{array}$$

(100)

But, from (80) and (95),

$$\begin{array}{cc}{\tilde{\mathrm{y}}}_{co}\left(k\right)& ={\tilde{\mathit{a}}}^T\mathrm{diag}{\left\{[f_1\cdots f_m]\right\}}_{\tilde{m}+1}{\mathbf{x}}_{co}(k-1)+\underset{\tilde{\mathrm{w}}\left(k\right)}{\underbrace{[{\tilde{a}}_1\cdots {\tilde{a}}_{\tilde{m}}0\cdots 0]{\mathbf{x}}_{co}(k-1)+{\mathit{p}}^T{\mathbf{x}}_{co}(k-1)}},\hfill \end{array}$$

(101)

where $\tilde{\mathit{a}}\triangleq [1\tilde{a}\cdots {\tilde{a}}_{\tilde{m}}]$, and

(102)

therefore,

$$\begin{array}{c}\ddot{\mathrm{y}}\left(1\right)={\tilde{\mathit{a}}}^T\mathrm{diag}{\left\{[f_1{f}_2\cdots f_m]\right\}}_{\tilde{m}+1}{\mathbf{x}}_{co}\left(0\right),\end{array}$$

(103)

$$\begin{array}{c}\ddot{\mathrm{y}}\left(2\right)={\tilde{\mathit{a}}}^T\mathrm{diag}{\left\{[0f_2\cdots f_m]\right\}}_{\tilde{m}+1}{\mathit{A}}_{co}{\mathbf{x}}_{co}\left(0\right),\end{array}$$

(104)

$$\begin{array}{cc}& ={\tilde{\mathit{a}}}^T\mathrm{diag}{\left\{[f_2\cdots f_{m}0]\right\}}_{\tilde{m}+1}{\mathbf{x}}_{co}\left(0\right)\hfill \end{array}$$

(105)

$$\begin{array}{cc}& ⋮\hfill \end{array}$$

(106)

$$\begin{array}{c}\ddot{\mathrm{y}}\left(m\right)={\tilde{\mathit{a}}}^T\mathrm{diag}{\left\{[0\cdots 0f_m]\right\}}_{\tilde{m}+1}{\mathit{A}}_{co}^{m-1}{\mathbf{x}}_{co},\end{array}$$

(107)

$$\begin{array}{cc}& ={\tilde{\mathit{a}}}^T\mathrm{diag}{\left\{[f_{m}0\cdots 0]\right\}}_{\tilde{m}+1}{\mathbf{x}}_{co}\left(0\right),\hfill \end{array}$$

(108)

with $\ddot{\mathrm{y}}\left(k\right)=0$ for $k>m$. Therefore,

$$\begin{array}{c}{\ddot{\mathbf{y}}}_m^1=\mathit{E}{\mathbf{x}}_{co}\left(0\right)=\underset{\mathit{E}}{\underbrace{\left[\mathit{M}\right|\mathit{N}]}}{\mathbf{x}}_{co}\left(0\right),\end{array}$$

(109)

where $\mathit{M}\in {\mathbb{R}}^{m\times (p-m)}$ and $\mathit{N}\in {\mathbb{R}}^{m\times m}$ is a lower anti-triangular Toeplitz matrix with ${\tilde{a}}_{\tilde{m}}{f}_m$ along its main anti diagonal.

This implies that ${\mathit{P}}_n={\left[{\mathit{E}}^T\right|{\mathbf{0}}_{p\times (n-m)}]}^T$ and

$$\begin{array}{c}{\sigma }_{\min }\left(\mathit{E}\right)>0.\end{array}$$

(110)

Thus, resuming the reasoning before (94), we have that

$$\begin{array}{c}{\left[{\mathit{Q}}_n\right]}_m^1{\mathit{P}}_n={}^1{\left[{\mathit{Q}}_n^{\left(p\right)}\right]}_m\mathit{E}.\end{array}$$

(111)

It then follows from (110) and Lemma 8 that

$$\begin{array}{c}\underset{n\to \infty }{\lim }{\sigma }_{\min }\left({\left[{\mathit{Q}}_n\right]}_m^1{\mathit{P}}_n\right)=\underset{n\to \infty }{\lim }{\sigma }_{\min }\left({}^1{\left[{\mathit{Q}}_n\right]}_m\mathit{E}\right).>0.\end{array}$$

(112)

Hence, (91) is satisfied. Substituting (91) into (89) and the latter into (84) yields

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}h\left({\left[{\mathit{D}}_n\right]}_m^1{\mathit{R}}_n{\mathbf{w}}_n^1+{\left[{\mathit{Q}}_n\right]}_m^1{\overline{\mathbf{y}}}_n^1\right)=0.\end{array}$$

(113)

The proof is completed by invoking Lemma (6).   □

Theorem 7 allows us to formalize the effect that the presence or absence of a random initial state has on the entropy gain using arguments similar to those utilized in Section 6.

8. Some Implications

The purpose of this section is to illustrate how the results obtained in the previous section can be applied to other problems. To do so, we present next some of the implications of these results on three different problems previously addressed in the literature, namely finding the rate-distortion function for non-stationary processes, an inequality in networked control theory, and the feedback capacity of Gaussian stationary channels. The common feature in these three problems is that, in all of them, non-minimum phase transfer functions play a role (either explicitly or implicitly).

8.1. Networked Control

The analysis developed in Reference [13] considers an LTI system P within a noisy feedback loop, as the one depicted in Figure 4. In this scheme, C represents a causal feedback channel which combines the output of P with an exogenous (noise) random process ${\mathrm{c}}_1^{\infty }$ to generate its output. The process ${\mathrm{c}}_1^{\infty }$ is assumed independent of the initial state of P, represented by the random vector ${\mathbf{x}}_0$, which has finite differential entropy.

For this system, it is shown in Reference [13], Theorem 4.2, that

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_1^{\infty }\right)\ge \overline{h}\left({\mathrm{u}}_1^{\infty }\right)+\underset{n\to \infty }{\lim }\frac{1}{n}I({\mathbf{x}}_0;{\mathrm{y}}_1^n),\end{array}$$

(114a)

where $I({\mathbf{x}}_0;{\mathrm{y}}_1^n)$ is the mutual information (see Reference [1], Section 8.5) between ${\mathbf{x}}_0$ and ${\mathrm{y}}_1^n$, with equality if w is a deterministic function of v. Furthermore, it is shown in Reference [12], Lemma 3.2, that, if $\left|h\right({\mathbf{x}}_0\left)\right|<\infty$ and the steady state variance of system P remains asymptotically bounded as $k\to \infty$, then

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}I({\mathbf{x}}_0;{\mathrm{y}}_1^n)\ge {\sum }_{p_i:\left|p_i\right|>1}\log \left|p_i\right|,\end{array}$$

(114b)

where $\left\{p_i\right\}$ are the poles of P. Thus, for the (simplest) case in which $\mathrm{w}=\mathrm{v}$, the output ${\mathrm{y}}_1^{\infty }$ is the result of filtering ${\mathrm{u}}_1^{\infty }$ by a filter $G=\frac{1}{1-P}$ (as shown in Figure 4 right), and the resulting entropy rate of ${\mathrm{y}}_1^{\infty }$ will exceed that of ${\mathrm{u}}_1^{\infty }$ only if there is a random initial state with bounded differential entropy (see (114a)). Moreover, if $\mathrm{w}=\mathrm{v}$ and $G\left(z\right)$ is stable, (114) (as well as Reference [13], Lemma 4.3) implies that this entropy gain is lower bounded by the right-hand side (RHS) of (8), which is greater than zero if and only if G is NMP. However, both [12,13] do not provide conditions under which this lower bound is reached.

In Reference [14], Theorem 14, it is shown that, when there is perfect feedback (i.e., when $\mathrm{v}=\mathrm{w}$), as in Figure 4 right, with P being the concatenation of a stabilizing LTI controller and an LTI plant, and assuming ${\mathrm{u}}_1^{\infty }$ is Gaussian i.i.d. and a Gaussian initial state, then

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_1^{\infty }\right)-\overline{h}\left({\mathrm{u}}_1^{\infty }\right)={\sum }_{p_i:\left|p_i\right|>1}\log \left|p_i\right|.\end{array}$$

(115)

Notice that this implies reaching equality in both (114a) and (114b).

By using the results obtained in Section 7 we show next that equality holds in (114b) provided the feedback channel satisfies the following assumption:

Assumption 3.

The feedback channel in Figure 4 can be written as

$$\begin{array}{c}\mathrm{w}=AB\mathrm{v}+BF\left(\mathrm{c}\right),\end{array}$$

(116)

where:

1.

A and B are stable rational transfer functions such that $AB$ is biproper, $ABP$ has the same unstable poles as P, and the feedback $AB$ stabilizes the plant P.

2.

F is any (possibly non-linear) operator such that $\tilde{\mathrm{c}}\triangleq F\left(\mathrm{c}\right)$ has finite variance ${\sigma }_{\tilde{\mathrm{c}}\left(n\right)}^2$ for finite n, ${\lim }_{n\to \infty }{n}^{-1}\log ({\sigma }_{\tilde{\mathrm{c}}\left(n\right)}^2)=0$, and

3.

${\mathrm{c}}_1^{\infty }{\displaystyle ⫫}{\mathbf{x}}_0$.

We also extend Reference [14], Theorem 14, to situations including a feedback channel satisfying Assumption 3. For the perfect-feedback case, this extends the validity of (115) to a much larger class of distributions for ${\mathrm{u}}_1^{\infty }$.

An illustration of the class of feedback channels satisfying this assumption is depicted on top of Figure 5. Trivial examples of channels satisfying Assumption 5 are a Gaussian additive channel preceded and followed by linear operators [23]. Indeed, when F is an LTI system with a strictly causal transfer function, the feedback channel that satisfies Assumption 3 is widely known as a noise shaper with input pre and post filter, used in, e.g., References [24,25,26,27].

Theorem 8.

In the networked control system of Figure 4, suppose that the feedback channel satisfies Assumption 3, that the plant $P\left(z\right)$ has poles ${\left\{p_i\right\}}_i^p$, and that the input ${\mathrm{u}}_1^{\infty }$ is entropy balanced. If the random initial states of $AB$ and P, namely ${\mathbf{s}}_0\in {\mathbb{R}}^q$ and ${\mathbf{x}}_0\in {\mathbb{R}}^p$, respectively, are independent, have finite variance and $\left|h\right({\mathbf{x}}_0\left)\right|<\infty$, then

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}I({\mathbf{x}}_0;{\mathrm{y}}_1^n)={\sum }_{\left|p_i\right|>1}\log \left|p_i\right|.\end{array}$$

(117a)

Moreover,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}(h\left({\mathrm{y}}_1^n\right)-h\left({\tilde{\mathrm{u}}}_1^n\right))={\sum }_{\left|p_i\right|>1}\log \left|p_i\right|,\end{array}$$

(117b)

where $\tilde{\mathrm{u}}\triangleq \mathrm{u}+B\tilde{\mathrm{c}}$ (see Figure 5 bottom).

Proof. 

Let $P\left(z\right)=N\left(z\right)/G\left(z\right)$ and $T\left(z\right)\triangleq A\left(z\right)B\left(z\right)=\Gamma \left(z\right)/\Theta \left(z\right)$. We will first show that the output ${\mathbf{y}}_n^1$ can be written as

$$\begin{array}{c}{\mathbf{y}}_n^1={\mathit{G}}_n{\tilde{\mathit{G}}}_n{\tilde{\mathbf{u}}}_n^1+{\mathit{G}}_n{\tilde{\mathit{P}}}_n{\left[{\mathbf{x}}_0^T{\mathbf{s}}_0^T\right]}^T+{\mathit{P}}_n{\mathbf{x}}_0,\end{array}$$

(118)

where $\tilde{G}$ is the stable LTI system with biproper and MP transfer function

$$\begin{array}{c}\tilde{G}\left(z\right)\triangleq \frac{\Theta \left(z\right)}{\Theta \left(z\right)G\left(z\right)+N\left(z\right)\Gamma \left(z\right)},\end{array}$$

(119)

with ${\mathbf{s}}_0\in {\mathbb{R}}^q$, ${\mathbf{x}}_0\in {\mathbb{R}}^p$ and ${\left[{\mathbf{x}}_0^T{\mathbf{s}}_0^T\right]}^T$ being the random initial states of T, G, and $\tilde{G}$, respectively, and

$$\begin{array}{c}\tilde{\mathrm{u}}\triangleq \mathrm{u}+B\tilde{\mathrm{c}}\end{array}$$

(120)

(see Figure 5 bottom). The matrices ${\tilde{\mathit{P}}}_n\in {\mathbb{R}}^{n\times p}$ and ${\mathit{P}}_n\in {\mathbb{R}}^{n\times (p+q)}$. From Figure 5, it is clear that the transfer function from $\tilde{\mathrm{u}}$ to y is $G\left(z\right)\frac{\Theta \left(z\right)}{\Theta \left(z\right)G\left(z\right)+N\left(z\right)\Gamma \left(z\right)}$, validating the first term on the RHS of (118). In addition, it is evident that the initial state of $\tilde{G}$ is a linear combination of ${\mathbf{x}}_0$ and ${\mathbf{s}}_0$, justifying the term ${\tilde{\mathit{P}}}_n{\left[{\mathbf{x}}_0^T{\mathbf{s}}_0^T\right]}^T$ as the natural response of $\tilde{G}$. Thus, it is only left to prove that the initial state of G is ${\mathbf{x}}_0$. For that purpose, let $G\left(z\right)=1-{\sum }_{i=1}^p{g}_i{z}^{-i}$ and $N\left(z\right)={\sum }_{i=1}^p{n}_i{z}^{-i}$. Define the following variables:

$$\begin{array}{c}\mathrm{o}\triangleq \frac{1}{G}\mathrm{y},\mathrm{w}\triangleq N\mathrm{o}.\end{array}$$

(121)

Then, the recursion corresponding to $P\left(z\right)$ is

$$\begin{array}{c}{\mathrm{o}}_k={\sum }_{i=1}^p{g}_i{\mathrm{o}}_{k-i}+{\mathrm{y}}_k,k\ge 1,\end{array}$$

(122)

$$\begin{array}{c}{\mathrm{w}}_k={\sum }_{i=1}^p{n}_i{\mathrm{o}}_{k-i},k\ge 1.\end{array}$$

(123)

This reveals that the initial state of $P\left(z\right)$ corresponds to

$$\begin{array}{c}{\mathbf{x}}_0=[{\mathrm{o}}_{1-p}{\mathrm{o}}_{2-p}\cdots {\mathrm{o}}_0].\end{array}$$

(124)

But, from (121), o is also the output of $\tilde{G}$ to the input $\tilde{\mathrm{u}}$, and

$$\begin{array}{cc}{\mathrm{y}}_k& ={\mathrm{o}}_k-{\sum }_{i=1}^p{f}_i{\mathrm{o}}_{k-i},k\ge 1,\hfill \end{array}$$

(125)

which means that the initial state of G is ${\mathbf{x}}_0$.

Now, using (118), we have that

$$\begin{array}{c}I({\mathbf{x}}_0;{\mathbf{y}}_n^1)=h\left({\mathbf{y}}_n^1\right)-h\left({\mathbf{y}}_n^1\right|{\mathbf{x}}_0),\end{array}$$

(126)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h\left({\mathit{F}}_n[{\tilde{\mathit{G}}}_n{\tilde{\mathbf{u}}}_n^1+{\tilde{\mathit{P}}}_n{\mathbf{s}}_0]\right),\hfill \end{array}$$

(127)

$$\begin{array}{cc}& =h({\mathit{F}}_n{\tilde{\mathbf{u}}}_n^1+{\mathit{P}}_n{\mathbf{x}}_0)-h\left({\tilde{\mathbf{u}}}_n^1\right),\hfill \end{array}$$

(128)

where the first equality is because ${\mathbf{s}}_0{\displaystyle ⫫}{\mathbf{x}}_0$ and ${\overline{\mathbf{u}}}_n^1\triangleq {\tilde{\mathit{G}}}_n{\tilde{\mathbf{u}}}_n^1+{\tilde{\mathit{P}}}_n{\mathbf{s}}_0$. The last equality holds since the first sample of the unit-impulse response of G is 1. Since ${\mathrm{u}}_1^{\infty }$ is entropy balanced, $\tilde{G}\left(z\right)$ is biproper, stable, and MP, and both ${\tilde{\mathrm{c}}}_1^{\infty }$ and ${\tilde{\mathit{P}}}_n{\mathbf{s}}_0$ have finite variance, it follows from Lemmas 3 and 4 that ${\overline{\mathrm{u}}}_1^{\infty }$ is entropy balanced, as well. Thus, the proof of the first claim is completed by direct application of Theorem 7.

For the second claim,

$$\begin{array}{c}h\left({\mathbf{y}}_n^1\right)-h\left({\tilde{\mathbf{u}}}_n^1\right)\stackrel{\left(a\right)}{=}h\left({\mathbf{y}}_n^1\right)-h({\tilde{\mathit{G}}}_n{\tilde{\mathbf{u}}}_n^1)=h\left({\mathbf{y}}_n^1\right)-h\left({\overline{\mathbf{u}}}_n^1\right)+(h({\tilde{\mathit{G}}}_n{\tilde{\mathbf{u}}}_n^1)-h({\tilde{\mathbf{u}}}_n^1)),\end{array}$$

(129)

where $\left(a\right)$ holds because the first sample of the unit-impulse response of $\tilde{G}$ is ${\tilde{g}}_0={\lim }_{z\to \infty }\tilde{G}\left(z\right)=1$. Then,

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}(h\left({\mathbf{y}}_n^1\right)-h\left({\tilde{\mathbf{u}}}_n^1\right))=\underset{n\to \infty }{\lim }\frac{1}{n}(h\left({\mathbf{y}}_n^1\right)-h\left({\overline{\mathbf{u}}}_n^1\right))+\underset{n\to \infty }{\lim }\frac{1}{n}(h\left({\tilde{\mathit{G}}}_n{\tilde{\mathbf{u}}}_n^1\right)-h\left({\tilde{\mathbf{u}}}_n^1\right)),\end{array}$$

(130)

$$\begin{array}{cc}& \stackrel{\left(a\right)}{=}\underset{n\to \infty }{\lim }\frac{1}{n}(h\left({\mathbf{y}}_n^1\right)-h\left({\overline{\mathbf{u}}}_n^1\right)),\hfill \end{array}$$

(131)

$$\begin{array}{cc}& \stackrel{\left(b\right)}{=}{\sum }_{\left|p_i\right|>1}\log \left|p_i\right|,\hfill \end{array}$$

(132)

where $\left(a\right)$ holds because $\tilde{\mathit{G}}\tilde{\mathrm{u}}$ is entropy balanced (from Lemma 4), and ${\tilde{\mathit{P}}}_n{\mathbf{s}}_0$ has finite variance, allowing us to apply Proposition A3. In turn, $\left(b\right)$ follows from (128) and (117a). This completes the proof.   □

Remark 2.

If $A\left(z\right)$ had poles outside the unit circle, then Theorem 8 can still be applied by associating those poles to P.

Remark 3.

Under the conditions of Theorem 8, one has that if either $\overline{h}\left({\mathrm{u}}_1^{\infty }\right)$ or $\overline{h}\left({\tilde{\mathrm{c}}}_1^{\infty }\right)$ exists, then the other entropy rate exists too. In that case, if $\mathrm{c}{\displaystyle ⫫}\mathrm{u}$ and defining $\overline{\mathrm{c}}\triangleq B\tilde{\mathrm{c}}$, (117) yields

$$\begin{array}{c}\overline{h}\left({\mathrm{y}}_1^{\infty }\right)-\overline{h}\left({\mathrm{u}}_1^{\infty }\right)-\overline{h}\left({\overline{\mathrm{c}}}_1^{\infty }\right)=\underset{n\to \infty }{\lim }\frac{1}{n}I({\mathbf{x}}_0;{\mathrm{y}}_1^n)={\sum }_{\left|p_i\right|>1}\log \left|p_i\right|,\end{array}$$

(133)

revealing that the gap in (114a) is exactly $\overline{h}\left({\overline{\mathrm{c}}}_1^{\infty }\right)$. In addition, in the perfect-feedback scenario, Theorem 8 extends the validity of (115) from the Gaussian i.i.d. u and Gaussian ${\mathbf{x}}_0$ considered in Reference [14], Theorem 14, to an entropy-balanced u and an ${\mathbf{x}}_0$ with finite variance and finite differential entropy.

8.2. Rate Distortion Function for Non-Stationary Processes

In this section, we obtain a simpler proof of a result by Gray, Hashimoto and Arimoto [15,16,17], which compares the rate distortion function (RDF) of a non-stationary auto-regressive Gaussian process ${\mathrm{x}}_1^{\infty }$ (of a certain class to be defined shortly) to that of a corresponding stationary version, under MSE distortion. Our proof is based upon the ideas developed in the previous sections, and extends the class of non-stationary sources for which the results in References [15,16,17] are valid.

To be more precise, let ${\left\{a_i\right\}}_{i=1}^{\infty }$ and ${\left\{{\tilde{a}}_i\right\}}_{i=1}^{\infty }$ be, respectively, the impulse responses of two linear time-invariant filters A and $\tilde{A}$ with rational transfer functions

$$\begin{array}{cc}A\left(z\right)& =\frac{z^M}{{\prod }_{i=1}^M(z-p_i)},\hfill \end{array}$$

(134)

$$\begin{array}{cc}\tilde{A}\left(z\right)& =\frac{z^M}{{\prod }_{i=1}^M\left|p_i^{*}\right|(z-1/p_i^{*})},\hfill \end{array}$$

(135)

where $\left|p_i\right|>1$, $\forall i=1,\dots ,M$. From these definitions, it is clear that $A\left(z\right)$ is unstable, $\tilde{A}\left(z\right)$ is stable, and

$$\begin{array}{c}|A\left({\mathrm{e}}^{j\omega }\right)|=|\tilde{A}\left({\mathrm{e}}^{j\omega }\right)|,\forall \omega \in [-\pi ,\pi ].\end{array}$$

(136)

Notice also that ${\lim }_{\left|z\right|\to \infty }A\left(z\right)=1$ and ${\lim }_{\left|z\right|\to \infty }\tilde{A}\left(z\right)=1/{\prod }_{i=1}^M\left|p_i\right|$; thus,

$$\begin{array}{ccc}{a}_0=1,& \hfill & {\tilde{a}}_0=\prod _{i=1}^M{\left|p_i\right|}^{-1}.\end{array}$$

(137)

Consider the non-stationary random sequence (source) ${\mathrm{x}}_1^{\infty }$ and the asymptotically stationary source ${\tilde{\mathrm{x}}}_1^{\infty }$ generated by passing a stationary Gaussian process ${\mathrm{w}}_1^{\infty }$ through $A\left(z\right)$ and $\tilde{A}\left(z\right)$, respectively, which can be written as

$$\begin{array}{c}{\mathbf{x}}_n^1={\mathit{A}}_n{\mathbf{w}}_1^n,n=1,\dots ,\end{array}$$

(138)

$$\begin{array}{c}{\tilde{\mathbf{x}}}_n^1={\tilde{\mathit{A}}}_n{\mathbf{w}}_1^n,n=1,\dots .\end{array}$$

(139)

(A block-diagram associated with the construction of x is presented in Figure 6.)

Define the rate-distortion functions for these two sources as

$$\begin{array}{ccc}{R}_{\mathrm{x}}\left(D\right)\triangleq \underset{n\to \infty }{\lim }{R}_{\mathrm{x},n}\left(D\right),& R_{\mathrm{x},n}\left(D\right)& \triangleq \min \frac{1}{n}I({\mathrm{x}}_1^n;{\mathrm{x}}_1^n+{\mathrm{u}}_1^n),\hfill \end{array}$$

(140)

$$\begin{array}{ccc}{R}_{\tilde{\mathrm{x}}}\left(D\right)\triangleq \underset{n\to \infty }{\lim }{R}_{\tilde{\mathrm{x}},n}\left(D\right),& R_{\tilde{\mathrm{x}},n}\left(D\right)& \triangleq \min \frac{1}{n}I({\tilde{\mathrm{x}}}_1^n;{\tilde{\mathrm{x}}}_1^n+{\tilde{\mathrm{u}}}_1^n),\hfill \end{array}$$

(141)

where, for each n, the minima are taken over all the conditional probability density functions $f_{{\mathrm{u}}_1^n|{\mathrm{x}}_1^n}$ and $f_{{\tilde{\mathrm{u}}}_1^n|{\tilde{\mathrm{x}}}_1^n}$ yielding $\mathrm{E}[\parallel {\mathbf{u}}_n^1{\parallel }^2]/n\le D$ and $\mathrm{E}[\parallel {\tilde{\mathbf{u}}}_n^1{\parallel }^2]/n\le D$, respectively.

The above rate-distortion functions have been characterized in References [15,16,17] for the case in which ${\mathrm{w}}_1^{\infty }$ is an i.i.d. Gaussian process. In particular, it is explicitly stated in References [16,17] that, for that case,

$$\begin{array}{c}{R}_{\mathrm{x}}\left(D\right)-R_{\tilde{\mathrm{x}}}\left(D\right)=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log |A^{-1}\left({\mathrm{e}}^{j\omega }\right)|d\omega ={\sum }_{i=1}^M\log \left|p_i\right|.\end{array}$$

(142)

We will next provide an alternative and simpler proof of this result, and extend its validity for general (not-necessarily stationary) Gaussian ${\mathrm{w}}_1^{\infty }$, using the entropy gain properties of non-minimum phase filters established in Section 6. Indeed, the approach in References [15,16,17] is based upon asymptotically-equivalent Toeplitz matrices in terms of the signals’ covariance matrices. This restricts ${\mathrm{w}}_1^{\infty }$ to be Gaussian and i.i.d. and $A\left(z\right)$ to be an all-pole unstable transfer function, and then, the only non-stationarity allowed is that arising from unstable poles. For instance, a cyclo-stationary innovation followed by an unstable filter $A\left(z\right)$ would yield a source which cannot be treated using Gray and Hashimoto’s approach. By contrast, the reasoning behind our proof lets ${\mathrm{w}}_1^{\infty }$ be any entropy-balanced Gaussian process with bounded differential entropy rate, and then let the source be $A\mathrm{w}$, with $A\left(z\right)$ having unstable poles (and possibly zeros and stable poles, as well).

The statement is as follows:

Theorem 9.

Let ${\mathrm{w}}_1^{\infty }$ be any Gaussian entropy-balanced process with bounded differential entropy rate, and let ${\mathrm{x}}_1^{\infty }$ and ${\tilde{\mathrm{x}}}_1^{\infty }$ be as defined in (138) and (139), respectively. Then, (142) holds.

Thanks to the ideas developed in the previous sections, it is possible to give an intuitive outline of the proof of this theorem (given in Appendix B, page 40) by using a sequence of block diagrams. More precisely, consider the diagrams shown in Figure 7.

In the top diagram in this figure, suppose that $\mathrm{y}=C\mathrm{x}+\mathrm{u}$ realizes the RDF for the non-stationary source x. The sequence u is independent of $\mathbf{x}$, and the linear filter $C\left(z\right)$ is such that the error $(\mathrm{y}-\mathrm{x}){\displaystyle ⫫}\mathrm{y}$ (a necessary condition for minimum MSE optimality). The filter $B\left(z\right)$ is the Blaschke product of $A\left(z\right)$ (see (A83) in Appendix B) (a stable, NMP filter with unit frequency response magnitude such that $\tilde{\mathrm{x}}=B\mathrm{x}$).

If one moves the filter $B\left(z\right)$ towards the source, then the middle diagram in Figure 7 is obtained. By doing this, the stationary source $\tilde{\mathrm{x}}$ appears with an additive error signal $\tilde{\mathrm{u}}$ that has the same asymptotic variance as u, reconstructed as $\tilde{\mathrm{y}}=C\tilde{\mathrm{x}}+\tilde{\mathrm{u}}$. From the invertibility of $B\left(z\right)$, it also follows that the mutual information rate between $\tilde{\mathrm{x}}$ and $\tilde{\mathrm{y}}$ equals that between x and y. Thus, the channel $\tilde{\mathrm{y}}=C\tilde{\mathrm{x}}+\tilde{\mathrm{u}}$ has the same rate and distortion as the channel $\mathrm{y}=C\mathrm{x}+\mathrm{u}$.

However, if one now adds a short disturbance d to the error signal $\tilde{\mathrm{u}}$ (as depicted in the bottom diagram of Figure 7), then the resulting additive error term $\overline{\mathrm{u}}=\tilde{\mathrm{u}}+\mathrm{d}$ will be independent of $\tilde{\mathrm{x}}$ and will have the same asymptotic variance as $\tilde{\mathrm{u}}$. Nonetheless, the differential entropy rate of $\overline{\mathrm{u}}$ will exceed that of $\tilde{\mathrm{u}}$ by the RHS of (142). This will make the mutual information rate between $\tilde{\mathrm{x}}$ and $\overline{\mathrm{y}}$ to be less than that between $\tilde{\mathrm{x}}$ and $\tilde{\mathrm{y}}$ by the same amount. Hence, $R_{\tilde{\mathbf{x}}}\left(D\right)$ is at most $R_{\mathbf{x}}\left(D\right)-{\sum }_{i=1}^m\log \left|p_i\right|$. A similar reasoning can be followed to prove that $R_{\mathbf{x}}\left(D\right)-R_{\tilde{\mathbf{x}}}\left(D\right)\le {\sum }_{i=1}^m\log \left|p_i\right|$.

8.3. The Feedback Channel Capacity of (Non-White) Gaussian Channels

Consider a non-white additive Gaussian channel of the form

$$\begin{array}{c}{\mathrm{y}}_k={\mathrm{x}}_k+{\mathrm{z}}_k,\end{array}$$

(143)

where the input x is subject to the power constraint

$$\begin{array}{c}\underset{n\to \infty }{\lim }\frac{1}{n}\mathrm{E}[\parallel {\mathbf{x}}_n^1{\parallel }^2]\le P,\end{array}$$

(144)

and ${\mathrm{z}}_1^{\infty }$ is a stationary Gaussian process.

The feedback information capacity of this channel is realized by a Gaussian input x, and is given by

$$\begin{array}{c}{C}_{\mathrm{FB}}=\underset{n\to \infty }{\lim }\underset{{\mathit{K}}_{{\mathbf{x}}_n^1}:\frac{1}{n}\mathrm{tr}\left\{{\mathit{K}}_{{\mathbf{x}}_n^1}\right\}\le P}{\max }I({\mathbf{x}}_n^1;{\mathbf{y}}_n^1),\end{array}$$

(145)

where ${\mathit{K}}_{{\mathbf{x}}_n^1}$ is the covariance matrix of ${\mathbf{x}}_n^1$, and, for every $k\in \mathbb{N}$, the input ${\mathbf{x}}_k$ is allowed to depend upon the channel outputs ${\mathrm{y}}_1^{k-1}$ (since there exists a causal, noise-less feedback channel with one-step delay).

In Reference [9], it was shown that if z is an auto-regressive moving-average process of M-th order, then $C_{\mathrm{FB}}$ can be achieved by the scheme shown in Figure 8. In this system, B is a strictly causal and stable finite-order filter and ${\mathrm{v}}_1^{\infty }$ is Gaussian with ${\mathrm{v}}_k=0$ for all $k>M$ and such that ${\mathbf{v}}_n^1$ is Gaussian with a positive-definite covariance matrix ${\mathit{K}}_{{\mathbf{v}}_M^1}$.

Here, we use the ideas developed in Section 6 to show that the information rate achieved by the capacity-achieving scheme proposed in Reference [9] drops to zero if there exists any additive disturbance of length at least M and finite differential entropy affecting the output, no matter how small.

To see this, notice that, in this case, and for all $n>M$,

$$\begin{array}{c}I({\mathrm{x}}_1^n;{\mathrm{y}}_1^n)=I({\mathrm{v}}_1^M;{\mathrm{y}}_1^n)=h\left({\mathbf{y}}_n^1\right)-h\left({\mathbf{y}}_n^1\right|{\mathbf{v}}_n^1),\end{array}$$

(146)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{v}}_n^1|{\mathbf{v}}_M^1),\hfill \end{array}$$

(147)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h\left(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1\right|{\mathbf{v}}_M^1),\hfill \end{array}$$

(148)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h\left(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1\right)=h\left({\mathbf{y}}_n^1\right)-h\left({\mathbf{z}}_n^1\right),\hfill \end{array}$$

(149)

$$\begin{array}{cc}& =h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{v}}_n^1)-h\left({\mathbf{z}}_n^1\right),\hfill \end{array}$$

(150)

since $\det ({\mathit{I}}_n+{\mathit{B}}_n)=1$. From Theorem 4, this gap between differential entries is precisely the entropy gain introduced by ${\mathit{I}}_n+{\mathit{B}}_n$ to an input ${\mathbf{z}}_n^1$ when the output is affected by the disturbance ${\mathbf{v}}_M^1$. Thus, from Theorem 4, the capacity of this scheme will correspond to $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log \left|1+B\left({\mathrm{e}}^{j\omega }\right)\right|d\omega ={\sum }_{\left|{\rho }_i\right|>1}\log \left|{\rho }_i\right|$, where ${\left\{{\rho }_i\right\}}_{i=1}^M$ are the zeros of $1+B\left(z\right)$, which is precisely the result stated in Reference [9], Theorem 4.1.

However, if the output is now affected by an additive disturbance ${\mathrm{d}}_1^{\infty }$ not passing through $B\left(z\right)$ such that ${\mathrm{d}}_k=0$, $\forall k>M$ and $\left|h\right({\mathbf{d}}_M^1\left)\right|<\infty$, with ${\mathrm{d}}_1^{\infty }{\displaystyle ⫫}({\mathrm{v}}_1^M,{\mathrm{z}}_1^{\infty })$, then we will have

$$\begin{array}{c}{\mathbf{y}}_n^1={\mathbf{v}}_n^1+({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{d}}_n^1.\end{array}$$

(151)

In this case,

$$\begin{array}{c}I({\mathrm{x}}_1^n;{\mathrm{y}}_1^n)=I({\mathrm{v}}_1^M;{\mathrm{y}}_1^n)=h\left({\mathbf{y}}_n^1\right)-h\left({\mathbf{y}}_n^1\right|{\mathbf{v}}_n^1),\end{array}$$

(152)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{v}}_n^1+{\mathbf{d}}_n^1|{\mathbf{v}}_M^1),\hfill \end{array}$$

(153)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{d}}_n^1|{\mathbf{v}}_M^1),\hfill \end{array}$$

(154)

$$\begin{array}{cc}& =h\left({\mathbf{y}}_n^1\right)-h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{d}}_n^1).\hfill \end{array}$$

(155)

But ${\lim }_{n\to \infty }\frac{1}{n}(h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{v}}_n^1+{\mathbf{d}}_n^1)-h(({\mathit{I}}_n+{\mathit{B}}_n){\mathbf{z}}_n^1+{\mathbf{d}}_n^1))=0,$ which follows directly from applying Theorem 4 to each of the differential entries. Notice that this result holds irrespective of how small the power of the disturbance may be.

Thus, the capacity-achieving scheme proposed in Reference [9] (and further studied in Reference [28]), although of groundbreaking theoretical importance, would yield zero rate in any practical situation, since in every physically implemented scheme, signals are unavoidably affected by some amount of noise.

9. Conclusions

We have provided an intuitive explanation and a rigorous characterization of the entropy gain of a linear time-invariant (LTI) system, defined as the difference between the differential entropy rates of its output and input random signals. The continuous-time version of this problem, considered by Shannon in Theorem 14 of his 1948 landmark paper, involves an LTI system $G_c$ band limited to B [Hz]. For this scenario, we restricted our attention to systems such that the samples of its unit-impulse response, taken ${\left(2B\right)}^{-1}$ seconds apart, correspond to the unit-impulse response $g_0,g_1,\dots$ of a causal and stable discrete-time system G. We show that the entropy gain in this case is $\log |g_0|$, which implies that, for this class of systems, Shannon’s Theorem 14 holds if and only if $G_c$ has a corresponding discrete-time G that is minimum phase (MP).

For the discrete-time case, we introduced a new notion referred to as effective differential entropy, which quantifies the amount of uncertainty in vector signals that are confined to subspaces of lower dimensionality than that of the signals themselves. (Note that this is not possible by the conventional notion of differential entropy, which simply diverges to minus infinity.) It turns out that the difference in effective differential entropy rate between an n-length input to an LTI discrete-time system with frequency response $G\left({\mathrm{e}}^{j\omega }\right)$, and its full length output, as n tends to infinity, equals $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\left|G\right({\mathrm{e}}^{j\omega }\left)\right|d\omega$.

When comparing input and output sequences of equal length, our analysis revealed that, in the absence of external random disturbances, the entropy gain of a discrete-time LTI system G with unit-impulse response $g_0,g_1,\dots$ is simply $\log |g_0|$. An entropy gain greater than $\log |g_0|$ can be obtained only if a random signal is added to the output of G and if such output process has statistical properties that make it susceptible to the added random signal. In order to characterize the role of G, its input has been assumed to be entropy balanced (EB), a notion introduced herein. Crucially, the differential entropy rate of an EB process is not susceptible to random signals. EB processes constitute a large family that includes Gaussian processes with bounded, non-vanishing variance. We also show that (i) the sum of an EB process and any bounded variance process is EB, too, and (ii) passing an EB process by a stable MP filter yields an EB process. When the input is EB, we show that if G has NMP zeros ${\rho }_1,{\rho }_2,\dots ,{\rho }_m$, then the largest possible entropy gain is $|g_0|+{\sum }_{i=1}^m\log \left|{\rho }_i\right|$, which equals $\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\left|G\right({\mathrm{e}}^{j\omega }\left)\right|d\omega$. This upper bound is achieved by adding a finite-length output disturbance with finite variance and bounded differential entropy if and only if its length is at least m, no matter how tiny its variance may be. The same entropy gain is also obtained if G has a random initial state with bounded differential entropy and finite variance.

We used these fundamental insights about when the entropy gain occurs in order to establish a new and more general proof of the quadratic rate-distortion function for non-stationary Gaussian sources. Moreover, we demonstrated that the information rate of the capacity-achieving scheme proposed in Reference [9] for the auto regressive Gaussian channel with feedback drops to zero in the presence of any additive disturbance in the channel input or output of sufficient (finite) length, no matter how small it may be. This has crucial implications in any physical setup, where noise is unavoidable.