Lesson 43 of 48
Entropy, cross-entropy, KL divergence
Measuring surprise, and the price of being wrong about it.
Entropy quantifies uncertainty; cross-entropy quantifies the cost of using the wrong beliefs. The loss function you will write at lesson 45 is cross-entropy, so this lesson is not background — it is the objective.
Do this
Compute the entropy of several distributions and show it is largest when the distribution is uniform. Then show numerically that cross-entropy is never below entropy.
The question that unlocks the next lesson
Which distribution over 4 outcomes has the highest entropy?
- A(0.97, 0.01, 0.01, 0.01)
- B(0.25, 0.25, 0.25, 0.25)
- C(1, 0, 0, 0)
- D(0.5, 0.5, 0, 0)