Is it a disorder, uncertainty or surprise?
The idea of entropy is confusing at first because so many words are used to describe it: disorder, uncertainty, surprise, unpredictability, amount of information and so on. If you’ve got confused with the word “entropy”, you are in the right place. I am going to demystify it for you.
Who Invented Entropy and Why?
In 1948, Claude Shannon introduced the concept of information entropy in his paper “A Mathematical Theory of Communication”.
Source: https://en.wikipedia.org/wiki/Claude_Shannon
Shannon was looking for a way to efficiently send messages without losing any information.
Shannon measured the efficiency in terms of average message length. Therefore, he was thinking about how to encode the original message into the smallest possible data structure. At the same time, he required that there should be no information loss. As such, the decoder at the destination must be able to restore the original message losslessly.
Shannon defined the entropy as the smallest possible average size of lossless encoding of the messages sent from…
