Attention: Can we formalize it?

FavoriteLoadingAdd to favorites

In statistics the bias-variance tradeoff is a core concept. Roughly speaking, bias is how well the best hypothesis in your hypothesis class would perform in reality, whereas variance is how much performance degradation is introduced from having finite training data.Last century both data and compute were relatively scarce so models that had high bias but low variance (and low computational…
Original post: Attention: Can we formalize it?
Source: Machined Learnings