Properties of Entropy

Shannon’s theory Theorem: \[ H(X,Y)\leq H(X)+H(Y)\\ let\ H(X)+H(Y)-H(X,Y)\\ =\sum_{x_{i}}Pr(x_{i})lb(\frac{1}{Pr(x_{i})})+\sum_{y_{j}}Pr(y_{j})lb(\frac{1}{Pr(y_{j})}-H(X,Y))\\ =\sum_{x_{i}}lb(\frac{1}{Pr(x_{i})})\sum_{y_{j}}Pr(x_{i},y_{j})+\sum_{y_{j}}lb(\frac{1}{P(y_{j})})\sum_{x_{i}}Pr(x_{i},y_{j})-H(X,Y)\\ =\sum_{x_{i},y_{j}}Pr(x_{i},y_{j})lb(\frac{1}{Pr(x_{i})Pr(y_{j})})-\sum_{x_{i},y_{j}}Pr(x{i},y_{j})lb(\frac{1}{Pr(x_{i},y_{j})})\\ =\sum_{x_{i},y_{j}}Pr(x_{i},y_{j})lb(\frac{Pr(x_{i},y_{j})}{Pr(x_{i})Pr(y_{j})})\\ =-\sum_{x_{i},y_{j}}Pr(x_{i},y_{j})lb(\frac{Pr(x_{i})Pr(y_{j})}{Pr(x_{i},y_{j})})\\ \]     Read more
tigertang's avatar
tigertang Apr 02, 2017