TDlambda

TDlambda, commonly written as TD(λ), is a family of temporal-difference learning algorithms used in reinforcement learning to estimate value functions. It blends bootstrapping from TD methods with information gathered from longer sequences through eligibility traces, governed by the trace-decay parameter λ in the interval [0, 1]. The approach updates estimates online as an agent interacts with an environment and encompasses a spectrum: it reduces to TD(0) when λ is 0 and approaches Monte Carlo methods as λ approaches 1.

The concept has a forward and a backward view. The forward view uses λ-returns, a weighted sum of n-step returns that balance immediate bootstrapping against full returns. The backward view implements this online via eligibility traces. At each time step t, the TD error δ_t = R_{t+1} + γ V(S_{t+1}) − V(S_t) is computed. An eligibility trace vector e_t accumulates credit for recently visited states or features, updated as e_t = γ λ e_{t-1} + φ(S_t) for linear function approximation, where φ(S_t) is the current feature representation. The value function is then updated by moving toward the target with V(S_t) ← V(S_t) + α δ_t e_t.

TD(λ) includes on-policy and off-policy variants. On-policy examples include Sarsa(λ) for action-value estimation, while off-policy variants such as Q(λ) exist with appropriate corrections. TD(λ)'s appeal lies in improved data efficiency and better handling of delayed rewards by distributing credit over past states and actions.

Limitations include sensitivity to the choice of λ and potential instability with certain function-approximation settings, particularly in off-policy or nonlinear approaches. Nonetheless, TD(λ) remains a foundational method for balancing bias and variance in value estimation.