Qérték

Qérték, commonly referred to as the Q-value, is a central concept in reinforcement learning. The Q-value function Qπ(s,a) assigns to each state s and action a the expected return, i.e., the expected sum of discounted rewards, when the agent takes action a in state s and subsequently follows a given policy π. Formally, Qπ(s,a) = Eπ[ ∑_{t=0}^∞ γ^t R_{t+1} | S0 = s, A0 = a ], where γ ∈ [0,1) is the discount factor. The optimal Q-value Q*(s,a) corresponds to the maximum obtainable return, and the greedy policy π*(s) = argmax_a Q*(s,a) is optimal under mild conditions.

Q-learning, a model-free algorithm introduced by Watkins and Dayan, iteratively updates Q-values toward the observed reward and the estimated value of the next state: Q(s,a) ← Q(s,a) + α [ r + γ max_{a'} Q(s',a') − Q(s,a) ]. The on-policy counterpart SARSA uses the actually chosen next action: Q(s,a) ← Q(s,a) + α [ r + γ Q(s', a') − Q(s,a) ].

In practice, Q-values can be stored in a table for small problems; large or continuous spaces require function approximators, notably neural networks in Deep Q-Networks (DQN). Training employs mechanisms such as experience replay and, in some variants, target networks to stabilize learning. Q-values underpin policy extraction: a policy can be derived by selecting the action with the highest Q-value in each state.