Engelbrecht Andries P. Computational Intelligence: An Introduction

Подождите немного. Документ загружается.

18 2. The Artificial Neuron

Artiﬁcial neurons that compute the net input signal as the weighted sum of input

signals are referred to as summation units (SU). An alternative to compute the net

input signal is to use product units (PU) [222], where

net =



i=1

(2.5)

Product units allow higher-order combinations of inputs, having the advantage of

increased information capacity.

2.2 Activation Functions

The function f

receives the net input signal and bias, and determines the output (or

ﬁring strength) of the neuron. This function is referred to as the activation function.

Diﬀerent types of activation functions can be used. In general, activation functions

are monotonically increasing mappings, where (excluding the linear function)

(−∞)=0 or f

(−∞)=−1 (2.6)

and

(∞) = 1 (2.7)

Frequently used activation functions are enumerated below:

1. Linear function (see Figure 2.2(a) for θ =0):

(net − θ)=λ(net − θ) (2.8)

where λ is the slope of the function. The linear function produces a linearly

modulated output, where λ is a constant.

2. Step function (see Figure 2.2(b) for θ>0):

(net − θ)=



if net ≥ θ

if net < θ

(2.9)

The step function produces one of two scalar output values, depending on the

value of the threshold θ. Usually, a binary output is produced for which γ

and γ

= 0; a bipolar output is also sometimes used where γ

=1andγ

= −1.

3. Ramp function (see Figure 2.2(c) for θ>0):

(net − θ)=







γ if net − θ ≥ 

net − θ if −<net− θ<

−γ if net − θ ≤−

(2.10)

The ramp function is a combination of the linear and step functions.

2.2 Activation Functions 19

(net − θ)

net −θ

(a) Linear function

(net − θ)

net − θ

(b) Step function

net − θθ

(net − θ)

−γ

net − θ

θ =0

(net − θ)

(d) Sigmoid function

net − θ

(net − θ)

−1

θ =0

(e) Hyperbolic tangent function

net − θ

(net − θ)

(f) Gaussian function

Figure 2.2 Activation Functions

20 2. The Artificial Neuron

4. Sigmoid function (see Figure 2.2(d) for θ =0):

(net − θ)=

1+e

−λ(net−θ)

(2.11)

The sigmoid function is a continuous version of the ramp function, with

(net) ∈ (0, 1). The parameter λ controls the steepness of the function.

Usually, λ =1.

5. Hyperbolic tangent (see Figure 2.2(e) for θ =0):

(net − θ)=

λ(net−θ)

− e

−λ(net−θ)

λ(net−θ)

+ e

−λ(net−θ)

(2.12)

or also approximated as

(net − θ)=

1+e

−λ(net−θ)

− 1 (2.13)

The output of the hyperbolic tangent is in the range (−1, 1).

6. Gaussian function (see Figure 2.2(f) for θ =0):

(net − θ)=e

−(net−θ)

/σ

(2.14)

where net − θ is the mean and σ the standard deviation of the Gaussian distri-

bution.

2.3 Artiﬁcial Neuron Geometry

Single neurons can be used to realize linearly separable functions without any error.

Linear separability means that the neuron can separate the space of I-dimensional in-

put vectors yielding an above-threshold response from those having a below-threshold

response by an I-dimensional hyperplane. The hyperplane forms the boundary be-

tween the input vectors associated with the two output values. Figure 2.3 illustrates

the decision boundary for a neuron with the ramp activation function. The hyperplane

separates the input vectors for which



−θ>0 from the input vectors for which



− θ<0.

Figure 2.4 shows how two Boolean functions, AND and OR, can be implemented

using a single perceptron. These are examples of linearly separable functions. For

such simple functions, it is easy to manually determine values for the bias and the

weights. Alternatively, given the input signals and a value for θ, the weight values v

can easily be calculated by solving

vZ = θ (2.15)

where Z is the matrix of input patterns as given in the truth tables.

An example of a Boolean function that is not linearly separable is the XOR as illus-

trated in Figure 2.5. A single perceptron can not implement this function. If a single

2.4 Artificial Neuron Learning 21

net − θ =0

net −θ<0

net −θ>0

Figure 2.3 Artiﬁcial Neuron Boundary

perceptron is used, then the best accuracy that can be obtained is 75%. To be able

to learn functions that are not linearly separable, a layered NN of several neurons is

required. For example, the XOR function requires two input units, two hidden units

and one output unit.

2.4 Artiﬁcial Neuron Learning

The question that now remains to be answered is whether an automated approach

exists for determining the values of the weights v

and the threshold θ? As illustrated

in the previous section, it is easy to calculate these values for simple problems. But

suppose that no prior knowledge exists about the function – except for data – how can

the v

and θ values be computed? The answer is through learning. The AN learns the

best values for the v

and θ from the given data. Learning consists of adjusting weight

and threshold values until a certain criterion (or several criteria) is (are) satisﬁed.

There are three main types of learning:

• Supervised learning, where the neuron (or NN) is provided with a data set

consisting of input vectors and a target (desired output) associated with each

input vector. This data set is referred to as the training set. The aim of super-

vised training is then to adjust the weight values such that the error between the

real output, o = f(net −θ), of the neuron and the target output, t, is minimized.

• Unsupervised learning, where the aim is to discover patterns or features in

the input data with no assistance from an external source. Many unsupervised

learning algorithms basically perform a clustering of the training patterns.

• Reinforcement learning, where the aim is to reward the neuron (or parts of

a NN) for good performance, and to penalize the neuron for bad performance.

22 2. The Artificial Neuron

The Artificial Neuron

Truth Table

Decision Boundary

θ =2

AND z

000

100

010

111

net − θ>0

net − θ<0

(a) AND Perceptron

Truth Table

The Artificial Neuron

Decision Boundary

OR z

111

θ =2

OR z

net − θ>0

net − θ<0

(b) OR Perceptron

Figure 2.4 Linear Separable Boolean Perceptrons

2.4 Artificial Neuron Learning 23

Truth Table Decision Boundary

XOR z

000

Figure 2.5 XOR Decision Boundaries

Several learning rules have been developed for the diﬀerent learning types. Before con-

tinuing with these learning rules, we simplify our AN model by introducing augmented

vectors.

2.4.1 Augmented Vectors

An artiﬁcial neuron is characterized by its weight vector v, threshold θ and activation

function. During learning, both the weights and the threshold are adapted. To simplify

learning equations, the input vector is augmented to include an additional input unit,

I+1

, referred to as the bias unit.Thevalueofz

I+1

is always -1, and the weight v

I+1

serves as the value of the threshold. The net input signal to the AN (assuming SUs)

is then calculated as

net =



i=1

− θ



i=1

+ z

I+1



i=1

(2.16)

where θ = z

I+1

= −v

I+1

In the case of the step function, an input vector yields an output of 1 when



I+1

i=1

≥

0, and 0 when



I+1

i=1

< 0.

The rest of this chapter considers training rules for single neurons.

24 2. The Artificial Neuron

2.4.2 Gradient Descent Learning Rule

While gradient descent (GD) is not the ﬁrst training rule for ANs, it is possibly the

approach that is used most to train neurons (and NNs for that matter). GD requires

the deﬁnition of an error (or objective) function to measure the neuron’s error in

approximating the target. The sum of squared errors

E =



p=1

− o

)

(2.17)

is usually used, where t

and o

are respectively the target and actual output for the

p-th pattern, and P

is the total number of input-target vector pairs (patterns)inthe

training set.

The aim of GD is to ﬁnd the weight values that minimize E.Thisisachievedby

calculating the gradient of E in weight space, and to move the weight vector along the

negative gradient (as illustrated for a single weight in Figure 2.6).

Error

Minimum

Weight

Figure 2.6 Gradient Descent Illustrated

Given a single training pattern, weights are updated using

(t)=v

(t − 1) + ∆v

(t) (2.18)

with

∆v

(t)=η(−

∂E

∂v

) (2.19)

where

∂E

∂v

= −2(t

− o

)

∂f

∂net

i,p

(2.20)

and η is the learning rate (i.e. the size of the steps taken in the negative direction of

the gradient). The calculation of the partial derivative of f with respect to net

(the

net input for pattern p) presents a problem for all discontinuous activation functions,

2.5 Assignments 25

such as the step and ramp functions; z

i,p

is the i-th input signal corresponding to

pattern p. The Widrow-Hoﬀ learning rule presents a solution for the step and ramp

functions, while the generalized delta learning rule assumes continuous functions that

are at least once diﬀerentiable.

2.4.3 Widrow-Hoﬀ Learning Rule

For the Widrow-Hoﬀ learning rule [907], assume that f = net

.Then

∂f

∂net

=1,

giving

∂E

∂v

= −2(t

− o

i,p

(2.21)

Weights are then updated using

(t)=v

(t − 1) + 2η(t

− o

i,p

(2.22)

The Widrow-Hoﬀ learning rule, also referred to as the least-means-square (LMS) al-

gorithm, was one of the ﬁrst algorithms used to train layered neural networks with

multiple adaptive linear neurons. This network was commonly referred to as the

Madaline [907, 908].

2.4.4 Generalized Delta Learning Rule

The generalized delta learning rule is a generalization of the Widrow-Hoﬀ learning rule

that assumes diﬀerentiable activation functions. Assume that the sigmoid function

(from equation (2.11)) is used. Then,

∂f

∂net

= o

(1 − o

) (2.23)

giving

∂E

∂v

= −2(t

−o

(1 − o

i,p

(2.24)

2.4.5 Error-Correction Learning Rule

For the error-correction learning rule it is assumed that binary-valued activation func-

tions are used, for example, the step function. Weights are only adjusted when the

neuron responds in error. That is, only when (t

− o

)=1or(t

− o

)=−1, are

weights adjusted using equation (2.22).

2.5 Assignments

1. Explain why the threshold θ is necessary. What is the eﬀect of θ, and what will

the consequences be of not having a threshold?

26 2. The Artificial Neuron

2. Explain what the eﬀects of weight changes are on the separating hyperplane.

3. Explain the eﬀect of changing θ on the hyperplane that forms the decision bound-

ary.

4. Which of the following Boolean functions can be realized with a single neuron

that implements a SU? Justify your answer by giving weight and threshold values.

(a) z

(b) z

+ z

where z

denotes (z

AND z

); z

+ z

denotes (z

OR z

); z

denotes

(NOT z

5. Is it possible to use a single PU to learn problems that are not linearly separable?

6. In the calculation of error, why is the error per pattern squared?

7. Can errors be calculated as |t

− o

| instead of (t

− o

)

if gradient descent is

used to adjust weights?

8. Is the following statement true or false: A single neuron can be used to approx-

imate the function f(z)=z

? Justify your answer.

9. What are the advantages of using the hyperbolic tangent activation function

instead of the sigmoid activation function?

Chapter 3

Supervised Learning Neural

Networks

Single neurons have limitations in the type of functions they can learn. A single neuron

(implementing a SU) can be used to realize linearly separable functions only. As soon

as functions that are not linearly separable need to be learned, a layered network of

neurons is required. Training these layered networks is more complex than training a

single neuron, and training can be supervised, unsupervised or through reinforcement.

This chapter deals with supervised training.

Supervised learning requires a training set that consists of input vectors and a target

vector associated with each input vector. The NN learner uses the target vector

to determine how well it has learned, and to guide adjustments to weight values to

reduce its overall error. This chapter considers diﬀerent NN types that learn under

supervision. These network types include standard multilayer NNs, functional link

NNs, simple recurrent NNs, time-delay NNs, product unit NNs, and cascade networks.

These diﬀerent architectures are ﬁrst described in Section 3.1. Diﬀerent learning rules

for supervised training are then discussed in Section 3.2. The chapter ends with a

short discussion on ensemble NNs in Section 3.4.

3.1 Neural Network Types

Various multilayer NN types have been developed. Feedforward NNs such as the stan-

dard multilayer NN, functional link NN and product unit NN receive external signals

and simply propagate these signals through all the layers to obtain the result (output)

of the NN. There are no feedback connections to previous layers. Recurrent NNs, on

the other hand, have such feedback connections to model the temporal characteristics

of the problem being learned. Time-delay NNs, on the other hand, memorize a window

of previously observed patterns.

Computational Intelligence: An Introduction, Second Edition A.P. Engelbrecht

2007 John Wiley & Sons, Ltd