From the course: Deep Learning with Python: Optimizing Deep Learning Models

Unlock this course with a free trial

Join today to access over 25,500 courses taught by industry experts.

Root Mean Square Propagation (RMSProp)

Root Mean Square Propagation (RMSProp) - Python Tutorial

From the course: Deep Learning with Python: Optimizing Deep Learning Models

Start my 1-month free trial Buy for my team

Root Mean Square Propagation (RMSProp)

“

- [Instructor] RMSProp, which stands for root mean squared propagation, was developed to address the diminishing learning rate problem observed in AdaGrad. It modifies AdaGrad by introducing an exponential decay average, or moving average, of the squared gradients. Instead of accumulating all past squared gradients, RMSProp keeps a running average that decays over time. This allows the algorithm to forget older gradients and focus on more recent ones. One of the significant benefits of RMSProp is its ability to maintain adaptive learning rates without the issue of the learning rates decaying too quickly. By using an exponential moving average, RMSProp ensures that the accumulated squared gradients do not grow indefinitely, preventing the learning rates from becoming too small. RMSProp is particularly effective in training models on non-stationary objectives, where the underlying data distribution changes over time. It also handles noisy and sparse gradients well, making it suitable…

Contents