[Previous] [Next]

Finding Local Minima using Gradient Descent

• Suppose we have a plastic sheet and we want to find the low points.
• One possibility would be to place a drop of mercury on the sheet and watch as it rolls down the sheet.
• The point where it stops is low (although possibly not the lowest on the entire sheet).
• The idea of the state of a system (position of the mercury) descending on an energy surface (the plastic sheet) is a recuring one in the field of neural networks - both for describing changes to activations as a consequence of cycling and changes to weights as a consequence of learning.