Rescorla-Wagner theory

9.031 Neural Basis of Learning and Memory: Lecture 9

Rescorla-Wagner reinforcement learning theory

Basic ideas

US processing - US leads to UR dependent upon reinforcement which can be + or –
Recorla-Wagner – predict US based on the CS

CS and US must occur at the same time

An association strength V relates the US to the CS
Learning is a change in the association strength and is defined by

Delta V = (level of US processing) x (level of CS processing)

Level of US processing -> Reinforcement

Level of CS processing -> Eligibility for change (eg. attention, salience, rehearsal)

Mackintosh advanced the notion of competition for CS eligibility to explain overshadowing
Rescorla-Wagner used competition for US processing

Overshadowing

Pairing two CSs A and B (eg. Light and tone) with a US leads to a stronger association between A or B and the US. That is one of the CSs will overshadow the other.

train

AB – US

test

A -> weak CR

B -> strong CR

Mackintosh proposed that A and B compete for limited CS processing resources when presented together (e.g. limited attention available to process both CSs simultaneously)

Note that this is not a simple consequence of differential stimulus associability (see association bias) since either A or B paired with US alone can lead to strong association

A-US Aà strong CR

B-US B-> strong CR

Rescorla-Wagner proposed competition for limited US processing

As A and B become associated with the US, the US is less unexpected, one of the CSs begins to predict the US and therefore less US processing is available for the other CS.

Blocking

Pretraining with one CS can block subsequent association with a 2^nd CS

train

A-US

AB-US

test

A->strong CR

B->weak CR

Mackintosh explains this as CS processing bias. Pretraining makes A stronger and thus biases subsequent level of CS processing resources available for B
Rescorla-Wagner explain this as US processing bias. Pretraining reduces the unexpectedness of the US and thus there is less for B to predict.

Learning occurs whenever events violate expections, i.e. when actual US level (lambda) differs from expected level (Vbar)

Vbar is the prediction based on total association strength of CSs

If US is present lambda = US value

If US is absent lambda = 0

With training Vbar will approach lambda

Starting from the basic equation for learning

Delta V = reinforcement(+,-) x eligibility(+,0)

Delta V is proportional to (lambda – Vbar)

Delta V = beta(lambda – Vbar)

With (lambda – Vbar) as the level of reinforcement and beta a coefficient of learning

With a dependence on the presence of the CS

Each CS defined as Xi

If CS is present Xi = 1

If CS is absent Xi = 0

With alphai a coefficient describing the salience or strength of the CS

Alphai Xi is the eligibility

So the net learning for each CS is

DeltaVi = beta(lambda – Vbar) alphai Xi

With Vbar = SUM(Vi Xi)

[walkthough of an example of blocking]

While this formulation can explain simple conditioning and blocking, it fails to explain second order conditioning

train

A - US

B - A

Test

B -> CR

[walkthrough of an example of second order conditioning]

Why does Rescorla-Wagner fail?

During training with A – US, A comes to predict the US by strengthening Vi for A.

During training with B – A, there is no US so lambda = 0

Since deltaVi for B is beta(lambda – Vbar) alphai Xi

With lambda = 0, deltaVi is at best 0, i.e. no learning should occur for B since there is no reinforcement being given and hence nothing to predict

To overcome this, lambda must be proportional to A, that is

The CS must produce reinforcement as well as a CR

A limitation of Rescorla-Wagner is that is does not specify how you get lambda
Additionally, assuming that lamda as a level of reinforcement does not explain this experimental result

train

………………………_____________________________________

US _____________|………………………………………………….. |___________

……………….____

CS1 _______|…… |_______________________________________________

……………………………………………………………………..____

CS2 _____________________________________________|….. |___________

test

CS1 -> positive association with US (e.g. shock)

CS2 -> negative association with US

Which suggests that

Changes in US level determine reinforcement rather that level itself

Applying this idea to the above experiment and using the results of 2^nd order conditioning that demonstrate that CSs can generate reinforcement as well as Uss

Assume that the reinforcement of each CS occurs at both onset and offset with values

+Vi at onset

-Vi at offset

and the US has values

+Vus at onset

-Vus at offset

let Y = sum of association strengths of ALL stimuli (CSs and USs)

and Ydot = Y(t) – Y(t-deltat)

Ydot is therefore the time-dependent reinforcement value

Replace lambda – Vbar in the standard Rescorla-Wagner with Ydot to get the Sutto Barton model

DeltaVi = beta Ydot alphai Xi