To understand transformers, you must understand attention.
Many explanations focus on the how rather than the why, making terms like queries, keys, and the underlying math feel arbitrary. You can memorize the attention formula and still have no idea why it actually works. But once you grasp the core intuition behind each step, attention becomes easy to remember and reason about.
In this article, we’ll explore both the how and the why.
This article discusses the derivative of the softmax function, providing an introductory example suitable for readers with a basic understanding of single-variable calculus. It outlines the mathematical principles necessary for computing this derivative and presents a concise expression for the softmax function’s Jacobian matrix, showcasing its application in probability distributions.
1 Introduction Understanding how functions change in different directions is crucial in many fields. For example in the context of neural networks where gradients are used to update weights during
The simoid function, $\sigma(x)$, is also called the logistic function, or expit \textcite{wiki_logit}. It is the inverse of the logit function. It’s function definition is:\begin{equation}\sigma(x) = \frac{1}{(1+e^{-x})}%\tag{sigmoid function}\label{eqn:sigmoid}\end{equation} Let’s get
The term logit has different meanings in math and in the TensorFlow library. In Ten-sorFlow it means “Per-label activations, typically a linear output. These activation energies are interpreted as unnormalized
