H(X|Y ) − H(X) = −
X
x∈X
X
y∈Y
p(x, y) log p(x|y) +
+
X
x∈X
X
y∈Y
p(x, y) log p(x) =
=
X
x∈X
X
y∈Y
p(x, y) log
p(x)
p(x|y)
≤
≤
X
x∈X
X
y∈Y
p(x, y)
µ
p(x)
p(x|y)
− 1
¶
log e =
=
X
x∈X
X
y∈Y
p(y)p(x)−
X
x∈X
X
y∈Y
p(x, y)
log e =
= 0.
¤
p(x, y) = p(x)p(y|x) = p(y)p(x|y),
p(x
1
, ..., x
n
) = p(x
1
)p(x
2
|x
1
)...p(x
n
|x
1
, ..., x
n−1
).
¤
XY Z =
{(x, y, z), p(x, y, z)} y ∈ Y
p(x, z|y) p(x|y)
H(X|y, Z) = M
XZ|y
[−log p(x|yz)],
H(X|y) = M
X|y
[−log p(x|y)].
H(X|y, Z) ≤ H(X|y).
y ¤