[Previous]

Why does energy decrease?

• The Hebbian learning rule produces weight patterns that are symmetric. w_ij = a_i a_j = a_j a_i = w_ji
• So energy can be written as:
  H = C - _(ij) w_ij a_i a_j

  where (ij) means all of the distinct pairs ij - so the pair 12 is the same as pair 21. The constant term C accumulates the (ii) pairs and is a constant which only depends of the number of units.

• Now consider the affect of an activation update on the energy. Suppose a_i' is the new value of a_i. If a_i' = a_i then the energy does not change. If a_i' = - a_i we can pick out the terms invovling a_i' and express the change in energy as:

  H' - H	= C - ai' ¬ j = i wij aj - (C - ai ¬ j = i wij aj)
  	= 2 ai ¬ j = i wij aj
  	= 2 ai j wij aj - 2 wii

  
  
  
  Now by the activation rule:
  a_i = sgn(_j w_ija_i)
  So:
  - a_i = sgn(_j w_ija_i)
  And 2 a_{i j} w_ij a_j must be negative. Also w_ii is equal to the number of patterns, so - 2 w_ii is also negative so the energy must decrease.