Introduction to Neural Networks

Copyright © 1997 Simon Dennis

1. What is a Neural Network?
2. Architectures
3. How a Neural Network Computes
4. How a Neural Network Learns
5. References
6. Slides

What is a Neural Network?

The rapidly developing field of artificial neural networks emphasizes biologically inspired approaches to problem solving. Artificial neural networks process information differently from traditional computers. Computation occurs in parallel across large numbers of simple processing units, rather than in the serial fashion of traditional computer architectures. Similarly, information is distributed across the entire network rather than being located in one specific place or address. In fact, neural computation is sometimes called parallel distributed processing to emphasize how it differs from traditional computing. In addition, simple learning algorithms can be defined which alter the connections between units (and therefore processing) as a result of experience. As we shall see in the remainder of this workbook, these properties have direct consequences for the way one thinks about computation, cognition and the mind. As a consequence, artificial neural networks enjoy multidisciplinary appeal. For computer scientists and engineers, neural networks provide a paradigm for solving problems which is often very successful particularly in domains that are poorly understood or subject to uncertainty. For linguists, cognitive scientists, psychologists and philosophers, neural networks provide a metaphor for the way in which cognitive processes such as perception, attention, learning, memory, language, reasoning and thinking occur. And for neuroscientists the mathematical simplification of the physiological processes allows for the analysis of large networks of units and provides insight into how the myriad interactions of neurons result in overt behaviour.

In the human brain, there are approximately 10 billion neurons, each of which is connected to about 10,000 other neurons. A single neuron (Figure 1) consists of a cell body called the soma, a number of spine-like extensions called dendrites, and a single nerve fibre called an axon which branches out from the soma and connects to other neurons. Neurons combine input signals from these connections or synapses to determine if and when it will transmit a signal to other neurons through the connecting dendrites and synapses. The synapses modulate the input signals before they are combined, and the system learns by changing the modulation at each synapse. Neurons inter-connected by axons and dendrites form the basic neural network.

Figure 1: A schematic of a typical neuron.

Figure 2 shows an artificial neural network in the BrainWave simulator. The squares are units, which represent a mathematical simplification of biological neurons. The overall level of activity of a unit is called its activation, which is represented by the size and colour of the squares (red being positive and blue being negative). The arrows connecting the units are weights, which represent the synapses. Weights can either be excitatory (red) or inhibitory (blue). Units connected to active units by excitatory weights will become more active, while units connected to active units by inhibitory weights will become less active.
Figure 2: The Jets and Sharks Network (McClelland, 1981)

To see a neural network in operation click on the "Art" unit. It should turn red indicating that it is now active. Then click on thethe "Cycle" button. Notice how the activations of the units alter as a function of the activations of the units to which they are connected.

Architectures

Artificial neural networks (and real neural networks for that matter) come in many different shapes and sizes (see Figure 3). In feedforward architectures, the activations of the input units are set and then propagated through the network until the values of the output units are determined. The network acts as a vector-valued function taking one vector on the input and returning another vector on the output. For instance, the input vector might represent the characteristics of a bank customer and the output might be a prediction of whether that customer is likely to default on a loan. Or the inputs might represent the characteristics of a gang member and the output might be a prediction of the gang to which that person belongs.

Figure 3: Some artificial neural network connection structures.

Feedforward networks may have a single layer of weights, where the inputs are directly connected to the outputs, or multiple layers with intervening sets of hidden units (see Figure 3). Neural networks use hidden units to create internal representations of the input patterns. In fact, it has been shown that given enough hidden units it is possible to approximate arbitrarily closely almost any function with a simple feedfoward network. This result has encouraged people to use neural networks to solve many kinds of problems.

The competitive network is similar to a single-layered feedforward network except that there are connections, usually negative, between the output nodes. Because of these connections the output nodes tend to compete to represent the current input pattern. Sometimes the output layer is completely connected and sometimes the connections are restricted to units that are close to each other (in some neighbourhood). With the appropriate learning algorithm the latter sort of network can be made to organize itself topologically. In a topological map, neurons near each other represent similar input patterns. Networks of this kind have been used to explain the formation of topological maps that occurs in many animal sensory systems including vision, audition, touch and smell (see Chapter Four for an example of a competitive network that develops topological maps).

The fully-recurrent network (see Figure 3) is perhaps the simplest of neural network architectures. All units are connected to all other units and every unit is both an input and an output. Typically, a set of patterns is instantiated on all of the units one at a time. As each pattern is instantiated the weights are modified. When a degraded version of one of the patterns is presented, the network attempts to reconstruct the pattern.

Recurrent networks are also useful in that they allow networks to process sequential information. Processing in recurrent networks depends on the state of the network at the last timestep. Consequently, the response to the current input depends on previous inputs. Figure 3 shows two such networks: the simple recurrent network and the Jordan network.

Having taken a birds eye view of the different sorts of connection structures that have been used in artifical neural networks, we will now zoom in on a single unit and discuss the processing that occurs there.

How a Neural Network Computes

Associated with each unit is a transfer function which determines how that unit's value (or activation) is updated. Typically, the transfer function multiplies each weight projecting to the unit by the activations of the (input) units which the weights project from. The sum of the weight inputs is added to a baseline or bias value to calculate the net input to the unit. Then a very simple activation function is applied to the net input (see Figure 4).

Figure 4: The Transfer Function

Figure 5: Some common activation functions.

Figure 5 shows some of the most common activation functions. In each case, the x-axis is the value of the net input and the y-axis is the output from the unit. To see how a neural network computes a function, consider the following network, which is designed to implement the logical function AND.

Figure 6: AND Network.

The AND network has two input units and a single output unit. The output unit employs a threshold activation function. If its net input is greater than zero the units activation will be one; if the net input is less than zero the unit's activation will be zero. Both weights are set to 1 and the bias is set to -1.5.

The AND function is represented by the numbers next to the units where each number represents a possible input. There are four possible combinations: 00, 01, 10, and 11. When both input units are 0 the output should be 0, when either unit one or unit two is zero the output should be zero, but when both inputs are 1 the output should be 1.

Consider the case where both inputs are zero. The net input is:

net input = 0x1 + 0x1 - 1.5 = -1.5

which is less than the threshold of zero, so the activation of the output unit will be 0. When unit 1 is set to 1 and unit 2 is set to 0 the net input is:

net input = 1x1 + 0x1 - 1.5 = -0.5

Similarly, if the unit 2 is set to one and unit 1 is set to zero net input is:

net input = 0x1 + 1x1 - 1.5 = -0.5

In both of these cases, the net input is less than threshold of zero, so the output of the network is 0. However, when both unit 1 and unit 2 are set to 1, the net input is:

net input = 1x1 + 1x1 - 1.5 = 0.5,

which is greater than the threshold, and the output of the network is 1. Given these weights the network implements the logical AND function.

Exercise 1: Alter the weights of the network to implement the logical OR function (see following table).

The OR Function
Unit 1	Unit 2	Output Unit
0	0	0
1	0	1
0	1	1
1	1	1

Exercise 2: Alter the weights of the network to implement the logical XOR function (see following table).

The XOR Function
Unit 1	Unit 2	Output Unit
0	0	0
1	0	1
0	1	1
1	1	0

Now lets consider a larger example. Suppose we want develop a network to recognize the letters "I", "L", and "T". Firstly, we need to choose a input representation. Figure 7 shows one possible artificial "retina". The units are arranged in a three by three grid. An active unit (=1) represents the foreground colour, while an inactive unit represents the background colour.

Figure 7: ILT Input Representations.

The task is to identify each of these letters. So we will have three output units representing "I", "L", and "T" respectively. The "I" unit should be active when the "I" pattern is presented and should be inactive when either the "L", or the "T" patterns are presented. Similarly, the "L" and "T" units should be active only when the corresponding pattern is presented.

To recognize the "T" we can connect the units that will be active in the "T" input pattern to the "T" output unit with weights of positive 1 and then set the bias to 4.5 (see Figure 8). When the "T" is presented the net input will be 5 which will exceed the bias and the "T" unit will active. Note that neither of the other two letters have 5 active units and hence they will never be able to create a netinput over 4.5 that will activate the "T" unit.

Figure 8: T Weights.

To recognize the "L" we can connect the active units to the "L" output unit and set the bias to 3.5 (see Figure 9). When the "L" input pattern is present the netinput will be 4 which will exceed the bias and activate the "L" output unit. Note that the "I" and "T" patterns only overlap on three units, so the netinput generated by these two letters will only be 3 which is below the bias.

Figure 9: L Weights.

The "I" pattern is somewhat more difficult. Because the "I" input representation is a subset of the "L" and "T" patterns we cannot just use bias to ensure that the "I" output unit does not become active when "L" or "T" are presented. Instead we connect negative weights from the units that are active in the "T" and "L" inputs but not in the "I" input. When these patterns are presented the negative weights will decrease the netinput taking it below the 2.5 bias (see Figure 10).

Figure 10: I Weights.

We have now successfully constructed a network to identify the letters "I", "L" and "T". While it is possible in a simple example such as this to select the weights by hand, it quickly becomes infeasible as the size of the network and the number of patterns (c.f. letters) increases. In the next section, we describe the main kinds of learning algorithms that have been developed to automatically select weights.

How a Neural Network Learns

The output of a network, given its input, depends on the connection structure and the weights. In general, the connection structure is held constant and the weights are modified to allow the network to implement different functions. How the weights are modified depends on the objective of the network and the information available to the learning rule. There are three main learning paradigms:

1. Supervised Learning with a Teacher: The network is provided with a set of inputs and the appropriate outputs for those inputs.
2. Supervised Learning with Reinforcement: The network is provided with a evaluation of its output given the input and alters the weights to try to increase the reinforcement it receives.
3. Unsupervised Learning: The network receives no external feedback but has an internal criterion that it tries to fulfil given the inputs that it faces.

Figure 7: Supervised Learning with a Teacher

Figure 7 depicts supervised learning with a teacher. In this paradigm the learning algorithm is given a set of input/output pattern pairs. The weights are adjusted so that the network will produce the required output in future. In the example of figure 7 the algorithm would be given a set of pictures of animals that the teacher classifies as spider, insect, lizard or other. If the network is shown a spider, but classifies it as a lizard then the weights are adjusted to make the network respond "spider".

Figure 8: Supervised Learning with Reinforcement

Supervised learning with reinforcement (Figure 8, also known as learning with a critic) is similar to learning with a teacher except that instead of being told the appropriate output given the input, the learning algorithm guesses at the right output and is told if it is correct. The algorithm updates the weights to maximize the number of inputs on which it is correct. In the example of figure 8, the network is told that it is incorrect when it classifies the spider as a lizard - but it isn't told what the correct classification would be. For this reason, learning with a critic is often more difficult and takes longer. Sometimes, however, we don't know what the correct output given an input should be and learning with a critic is the only learning that is possible. For instance, if you were to construct a neural network to control a chemical plant and you wanted to make sure that the temperature of the plant did not exceed a certain bound, you may not know what settings of the valves should be used at a given time. If, however, the temperature rises above the bound you know that what the network just did wasn't good. That is, you can supply critic information, but not teacher information, so a reinforcement algorithm could be applied.

Figure 9: Unsupervised Learning

The final learning paradigm, depicted in Figure 9, is unsupervised learning. Unsupervised learning does not receive information from either a teacher or critic. Instead, it relies on an internal criterion to guide learning. For instance, in Figure 9 the objective is to create an output representation in which similar inputs activate output units that are close to one another (i.e. forming a topological map, see above). The network is shown a series of animals and gradually changes the weights so that similar animals are mapped to adjacent units. From an intial random assignment, the spiders end up in the bottom left hand corner - near the insects, which are similar. The lizards and frogs end up at the opposite extreme of the map (the top right hand corner).

We have now considered the main types of learning paradigms. Within these general structures there are many variations in exactly how the weights are updated. In Connectionist Models of Cognition we will examine some of the major algorithms. Now, however, it is time to get our hands dirty running and constructing neural networks using the BrainWave simulator.