Deep Reinforcement Learning (John Schulman, OpenAI)

Deep reinforcement learning can be overkill for problems with few parameters or where a simulator is available for derivative-free optimization. If your problem can be approximated as a contextual bandit or if operations research methods suffice, those might be simpler and more established alternatives.

Deep RL has achieved significant breakthroughs in playing Atari games using raw screen images, defeating a Go champion by combining various techniques, learning robotic manipulation tasks in real-time, and achieving complex robotic locomotion.

An MDP is the fundamental framework for reinforcement learning. It's defined by a state space, action space, and a probability distribution that determines the next state and reward given the current state and action. The goal is typically to maximize cumulative reward.

Policy gradient methods explicitly represent the policy (how the agent chooses actions) and optimize it directly using gradient descent. They work by collecting trajectories, identifying successful actions/trajectories, and increasing the probability of those good actions.

This estimator allows calculating the gradient of an expectation with respect to a parameterized probability distribution. It involves taking the expectation of the function value multiplied by the gradient of the log probability of a sample, enabling optimization even with non-differentiable components.

Key strategies include using future rewards instead of all rewards, introducing a baseline (subtracting the expected return), and applying a discount factor to rewards, which ignores very delayed effects and can make learning more stable.

Q-function methods learn a 'Q-function' that estimates the value of state-action pairs. They don't directly optimize the policy but implicitly derive it from the learned Q-function. These methods are often more sample-efficient but can be harder to debug when they fail.

The Bellman backup is an iterative operation that updates a Q-function estimate. It uses the right-hand side of the Bellman equation, plugging in the current Q-function estimate, and repeatedly applying this operation converges to the true Q-function.

DQN uses a replay pool to store and sample past experiences, providing a more representative dataset for training. It also employs a target network, a lagged copy of the Q-function, to stabilize the learning process during Bellman backups, similar to value iteration.

Q-function methods (like DQN) are generally more sample-efficient but less robust and harder to diagnose when they fail. Policy gradient methods are more general, better understood, and more robust, but can be less sample-efficient. The choice often depends on the specific problem.

Yes, but it often requires significant patience due to the time needed for data collection and learning. For example, learning robotic locomotion that took weeks of real-time simulation could be attempted in the real world, though challenges remain.