FTRL (Follow The Regularized Leader)

FTRL is an abbreviation for “Follow-The-Regularized-Leader” and is an optimization algorithm specifically designed for online learning scenarios. It was developed to work efficiently with large-scale and sparse data. This algorithm, proposed by Google, is widely used in ad prediction, recommendation systems, and real-time learning systems.

FTRL Optimization Algorithm

Core Approach

Unlike classical gradient descent methods, FTRL updates parameters based on a combination of accumulated gradients from previous steps and regularization terms. In particular, it incorporates <span class="katex"><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">L</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>

FTRL enables online and sparse learning by accounting for past gradients and regularization terms.