Actionvalues

Actionvalues, often referred to as action-value functions or Q-values, are a central concept in reinforcement learning. They assign to each state-action pair (s, a) the expected return that an agent can obtain by taking action a in state s and thereafter following a given policy. The action-value function for a policy π is denoted Qπ(s, a) and defined as the expected discounted sum of future rewards: Qπ(s, a) = E[ sum_{t=0}^∞ γ^t R_{t+1} | S0 = s, A0 = a, π ], where γ ∈ [0,1) is the discount factor.

The action-value function complements the state-value function Vπ(s) = E[ sum_{t=0}^∞ γ^t R_t | S0 = s ]. Qπ(s, a) provides a direct measure of the value of choosing action a in state s under policy π and can be used to derive an optimal policy by selecting actions that maximize Q*(s, a). The optimal action-value function Q*(s, a) satisfies the Bellman optimality equation: Q*(s, a) = E[ R_{t+1} + γ max_{a'} Q*(S_{t+1}, a') | S_t = s, A_t = a ].

In practice, action-values are learned in model-free algorithms through temporal-difference or Monte Carlo methods. For example, SARSA updates use the observed next action A' to form the target, while Q-learning uses the maximum over possible next actions. In deep reinforcement learning, neural networks approximate Q-values, as in Deep Q-Networks, often using experience replay and target networks to improve stability.

Applications include guiding decision making in games, robotics, and other sequential tasks where evaluating the value of actions informs policy improvement.