From the course: Deep Learning with Python: Optimizing Deep Learning Models
Join today to access over 25,500 courses taught by industry experts.
Gradient clipping - Python Tutorial
From the course: Deep Learning with Python: Optimizing Deep Learning Models
Gradient clipping
- [Instructor] Gradient Clipping is a technique used in deep learning to prevent the gradients of a model from becoming excessively large during training. This phenomenon known as exploding gradients occurs when the gradients of the loss function respectively model's parameters grow excessively large during back propagation, this can destabilize training cause numerical issues and prevent the model from converging to an optimal solution. Gradient clipping solves this problem by putting a limit or cap on how large the gradients can get. There are two primary approaches to gradient clipping, clipping by value and clipping by norm. In clipping by value, each individual gradient is clipped so that it doesn't exceed a specific minimum value or maximum value. Let's walk through a simple example of how this works. Suppose we have a gradient vector G with values two, negative six, eight, negative three, and five with a gradient threshold C set to four. This means that the minimum greater…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.
Contents
-
-
-
-
-
-
-
(Locked)
Batch normalization3m 5s
-
(Locked)
Applying batch normalization to a deep learning model2m 55s
-
(Locked)
Gradient clipping5m 10s
-
(Locked)
Applying gradient clipping to a deep learning model3m
-
(Locked)
Early stopping and checkpointing3m 23s
-
(Locked)
Learning rate scheduling4m 56s
-
(Locked)
Training a deep learning model using callbacks6m 13s
-
(Locked)
-