Pawel Flajszer Slip-boxNotebooksWorkAbout
written
rests on
1
cited by
1

The model will sacrifice errors the loss considers cheap in order to minimize the errors the loss considers expensive

In other words, the model (specifically, the optimizer) will cause cheap errors to grow when that growth allows the expensive ones to shrink. This isn’t a choice, but happens when following gradient descent. Because expensive errors contribute more steeply to the loss, the gradient points more strongly away from them. Following the gradient disproportionately reduces expensive errors, at the cost of letting cheap errors grow.

Rests on

Cited by