X = {a, b, c, d} p(a) =
1/2 p(b) = 1/4 p(c) = p(d) = 1/8
I(a) = 1 I(b) = 2 I(c) = I(d) = 3
X = {a, b} p(a) = 0.05
p(b) = 0 .95 I(a) = 4 , 322 I(b) = 0, 216
x X = {x, p(x)}
X = {x, p(x)}
X = {x, p(x)}
H(X) = M [−log p(x)] = −
X
x∈X
p(x) log p(x) .
H(X) ≥ 0.
H(X) ≤ log |X|.
X