Chapter 6: Learning in Physical Systems

Why Learning is Essential in Physical AI

Rule-based control systems, while effective in structured and predictable environments, fail to generalize in the presence of environmental uncertainty, unmodeled dynamics, or changing task requirements. Physical AI systems—particularly humanoid robots operating in human-centric environments—must contend with high-dimensional sensorimotor spaces, partial observability, and continuous interaction with non-stationary surroundings. These conditions render exhaustive pre-specification of behaviors infeasible.

Learning provides a mechanism for systems to adapt their internal models or policies based on experience. Unlike pre-programmed behavior, which is static and brittle, learning enables agents to refine their strategies in response to observed outcomes, thereby improving performance over time. This adaptive capability is essential for achieving robust autonomy in real-world physical settings where analytical models are either unavailable or insufficiently accurate.

Supervised Learning in Robotics

Supervised learning in robotics primarily addresses perception tasks by mapping raw sensor inputs to structured outputs using labeled datasets. These datasets typically consist of input–output pairs such as images annotated with object labels, depth maps paired with 3D poses, or tactile readings associated with material properties.

In humanoid robotics, supervised learning is commonly applied to visual perception problems, including object recognition and 6D pose estimation. Convolutional neural networks (CNNs) trained on large-scale annotated image corpora enable robots to identify objects and estimate their positions and orientations relative to the agent. The success of such methods depends critically on the fidelity and diversity of the training data, as performance degrades when test distributions diverge significantly from training conditions.

Reinforcement Learning for Physical Agents

Reinforcement learning (RL) enables physical agents to learn control policies through trial-and-error interaction with an environment. The agent observes states, selects actions, and receives scalar rewards that signal task progress. The objective is to maximize cumulative reward over time, typically by learning a policy that maps states to actions or by estimating value functions that predict long-term returns.

A central challenge in RL for physical systems is the exploration–exploitation trade-off: the agent must balance trying new actions to discover potentially better strategies (exploration) against using known actions that yield high rewards (exploitation). Additionally, sample efficiency is a critical concern. Real-world trials are time-consuming, costly, and may risk hardware damage, making algorithms that require millions of interactions impractical without simulation or other acceleration techniques.

Imitation and Learning from Demonstration

Imitation learning, or learning from demonstration (LfD), leverages human-provided examples to bootstrap robot behaviors. This approach is particularly useful when designing reward functions for RL is difficult or when expert demonstrations encode nuanced, task-relevant priors.

Human-guided learning can take several forms. Kinesthetic teaching involves physically moving a robot’s limbs to demonstrate trajectories, which the system then attempts to reproduce. Alternatively, visual demonstrations—recorded via external cameras or motion capture—can be used to extract action sequences or state trajectories. Behavior cloning, a common supervised approach to imitation, trains a policy to map observed states to demonstrated actions by minimizing prediction error on a dataset of expert trajectories. While simple and effective in constrained settings, behavior cloning suffers from compounding errors due to distributional shift between training and deployment.

Sim-to-Real Transfer

Training in simulation offers a scalable and safe alternative to real-world data collection, enabling extensive policy or model learning without hardware wear or safety risks. However, policies trained purely in simulation often fail when deployed on physical systems due to the reality gap—the mismatch between simulated and real dynamics, sensor characteristics, and environmental conditions.

Domain randomization is a widely adopted technique to mitigate this gap. By randomizing physical parameters (e.g., mass, friction, lighting, camera noise) during simulation, the learned policy becomes robust to a range of plausible real-world variations. Other approaches include system identification to calibrate simulators, adaptive transfer methods that fine-tune policies on real data, and the use of neural rendering to better match visual appearances across domains. Despite these advances, achieving zero-shot sim-to-real transfer remains challenging for complex, contact-rich tasks.

Challenges and Limitations

Learning in physical systems faces several persistent challenges. Data efficiency is a primary constraint: unlike virtual agents, physical robots cannot generate vast datasets rapidly, limiting the applicability of data-intensive methods. Safety during learning is another critical concern—unconstrained exploration may result in collisions, hardware damage, or harm to humans, necessitating safe exploration strategies or constrained policy updates.

Computational constraints further limit on-board learning. Real-time control often requires low-latency inference, which conflicts with the high computational demands of training deep models or maintaining large replay buffers. These factors collectively restrict the deployment of end-to-end learning approaches on resource-limited humanoid platforms, motivating hybrid architectures that combine learned components with model-based or modular reasoning.