banditbased

Bandit-based refers to approaches and systems that base their decision-making on the multi-armed bandit framework. In this framework, an agent repeatedly selects from a finite set of actions (arms) and receives a reward from an unknown distribution. The goal is to maximize cumulative reward over time by balancing exploration of less-known actions with exploitation of actions that currently appear best. Bandit-based methods emphasize online learning from feedback rather than relying on pre-collected labeled data.

There are several variants of the bandit problem. Stochastic bandits assume each arm has a fixed reward distribution, while adversarial bandits make no probabilistic assumptions about rewards and aim to perform well against worst-case sequences. Contextual bandits extend the basic model by incorporating side information, or context, to tailor decisions to the current situation. This makes bandit-based methods particularly suitable for personalized or adaptive tasks.

Common algorithms include epsilon-greedy, which selects random arms with a fixed probability, and upper confidence bound (UCB) strategies that favor arms with high estimated value and high uncertainty. Thompson sampling uses Bayesian reasoning to sample actions from the posterior distribution. For adversarial settings, algorithms such as EXP3 are used. Bandit-based methods are valued for sample efficiency and strong regret guarantees under suitable assumptions.

Applications span online advertising, A/B testing, recommendation systems, adaptive experimentation, clinical trials, and robotics, where decisions must be made sequentially with limited feedback. Limitations include sensitivity to model assumptions, challenges with large or rapidly changing action spaces, and the need to carefully manage exploration versus exploitation under non-stationarity.