734 INDEX
encoding function, 193, 264, 305, 359, 571,
583, 599
encrypted text, 506
energy, 261, 265, 272, 273, 294, 424
England, 82
English, 104, 168–171, 174, 175, 182, 360,
470, 506
entropy rate, 159, 182
models of, 168
entanglement, 56
entropy, xvii, 3, 4, 13–56, 87, 659, 671,
686
average, 49
axiomatic definition, 14, 54
base of logarithm, 14, 15
bounds, 663
chain rule, 23
concavity, 33, 34
conditional, 16, 51, see conditional
entropy
conditioning, 42
cross entropy, 55
differential, see differential entropy
discrete, 14
encoded bits, 156
functions, 45
grouping, 50
independence bound, 31
infinite, 49
joint, 16, 47, see joint entropy
mixing increase, 51
mixture, 46
and mutual information, 21
properties of, 42
relative, see relative entropy
Renyi, 676
sum, 47
thermodynamics, 14
entropy and relative entropy, 12, 28
entropy power, xviii, 674, 675, 678, 679,
687
entropy power inequality, xx, 298, 657,
674–676, 678, 679, 687
entropy rate, 4, 74, 71–101, 114, 115, 134,
151, 156, 159, 163, 167, 168, 171,
175, 182, 221, 223, 259, 417, 419,
420, 423–425, 428–462, 613, 624,
645, 667, 669
differential, 416
English, 168, 170, 174, 175
Gaussian process, 416
Hidden Markov model, 86
Markov chain, 77
subsets, 667
envelopes, 182
Ephremides, A., 611, 699
Epimenides liar paradox, 483
equalization, 611
Equitz, W., xxiii, 699
erasure, 188, 226, 227, 232, 235, 527, 529,
594
erasure channel, 219, 235, 433
ergodic, 69, 96, 167, 168, 175, 297, 360,
443, 444, 455, 462, 557, 613, 626,
644, 646, 647, 651
ergodic process, xx, 11, 77, 168, 444, 446,
451, 453, 644
ergodic source, 428, 644
ergodic theorem, 644
ergodic theory, 11
Erkip, E., xxi, xxiii
Erlang distribution, 661
error correcting code, 205
error detecting code, 211
error exponent, 4, 376, 380, 384, 385, 388,
399, 403
estimation, xviii, 255, 347, 392, 425, 508
spectrum, 415
estimator, 39, 40, 52, 255, 392, 393,
395–397, 401, 402, 407, 417, 500, 663
bias, 393
biased, 401
consistent in probability, 393
domination, 393
efficient, 396
unbiased, 392, 393, 395–397, 399, 401,
402, 407
Euclidean distance, 514
Euclidean geometry, 378
Euclidean space, 538
Euler’s constant, 153, 662
exchangeable stocks, 653
expectation, 14, 167, 281, 306, 321, 328,
393, 447, 479, 617, 645, 647, 669, 670
expected length, 104
exponential distribution, 256, 661
extension of channel, 193
extension of code, 105