macroactions

Macroactions are temporally extended actions that bundle a sequence of primitive actions into a single decision unit. They are used in planning and learning to provide temporal abstraction, enabling agents to operate over longer time scales in domains such as robotics, game AI, and cognitive architectures.

In hierarchical reinforcement learning, macro-actions are often formalized as options. An option includes an initiation set, an internal policy to execute while active, and a termination condition. When a macro-action is activated, the agent follows its policy until termination, at which point the higher-level decision process resumes. This structure helps with credit assignment across time and can reduce learning complexity.

The main advantages of macroactions include shorter effective planning horizons, improved sample efficiency, and the reuse or transfer of skills across tasks that share subgoals. They also simplify policy representations by abstracting away low-level details that are not needed at the higher level.

Design and discovery of macroactions can be manual or automated. Hand-crafted macro-actions rely on domain knowledge, while learned macro-actions are discovered through methods such as subgoal discovery or option-critic frameworks. The choice of initiation conditions and termination criteria significantly affects performance and safety, particularly in real-world systems.

Challenges include selecting meaningful subgoals, aligning termination with rewards, and ensuring robust credit assignment across extended durations. There is also the risk of suboptimal behavior if macro-actions constrain exploration or overlook alternative approaches. Applications span robotics (navigating, grasping), video games (patrolling, attacking sequences), and software automation (multi-step commands).