Theory of linear networks

Sebastian Seung

9.641 Lecture 5: September 20, 2000

1  Linear filters vs. linear networks

• Our phenomenological model of a retinal ganglion cell took the form of the linear filter O_i = å_j D_ij I_j.
• This could also be interpreted as a mechanistic model, in which I_j is the output of the jth photoreceptor, O_i is the output of the ith ganglion cell, and D_ij is the strength of the synaptic connection between them. (In the vertebrate retina, there are other types of neurons between the photoreceptors and the ganglion cells, so this is too simplistic, but let's ignore that fact for the moment).
• But more than one network is consistent with the same linear filter. More generally, we could write the network

  
  

  t . O   i  +Oi = å j  WijOj + å j  FijIj

  which has lateral connections W between the output neurons, and feedforward connections F from the input to the output.

• The steady state response is given by O = (I-W)^-1FI, from which we read off

  
  

  D = (I-W)-1F

  In other words, any network with W and F satisfying this equation is consistent with a linear filter D.

2  Feedback in networks

• Earlier we introduced the network equations

  
  

  t . x   i  +xi = f æ è å j  Wijxj+bi ö ø

  which contain both nonlinearity and feedback.

• Today we will consider the equations

  
  

  t . x   i  +xi = å j  Wijxj+bi

  which have feedback but no nonlinearity. The feedforward connections are implicit in b. They'll be ignored, so that we can focus on the role of the recurrent connections W.

3  Single feedback loop

• rearrange t[(x)\dot] + x = W x + b to

  
  

  t 1-W . x   + x = b 1-W

  • steady state gain is [1/(1-W)]
  • effective time constant is [(t)/(1-W)]
• stable vs. unstable

  • stable (W < 1): exponential convergence to the steady state.
  • unstable (W > 1): exponential divergence from the steady state.
  • borderline (W = 1)

    • b = 0: infinitely many steady states, marginally stable for b = 0
    • no steady state exists
• positive vs. negative feedback

  • positive feedback (W > 0): amplification, slowdown
  • negative feedback (W < 0): attenuation, speedup
• gain-bandwidth product is constant
• This model could be applied to a single neuron in culture, which grows synaptic connections onto itself (autapses). Autapses are also found in vivo.

4  Mutual inhibition

• Two linear neurons with mutual inhibition of strength b:

  
  

  t . x   1  + x1 = -bx2 + b1 (1) t . x   2  + x2 = -bx1 + b2 (2)

• common mode x_c = x₁+x₂ is attenuated by negative feedback

  
  

  t d dt (x1+x2) + (x1+x2) = -b(x1+x2) + (b1+b2)

• differential mode x_d = x₁ - x₂ is amplified by positive feedback

  
  

  t d dt (x1-x2) + (x1-x2) = b(x1-x2) + (b1-b2)

• This is the simplest example of lateral inhibition, which we will discuss in greater depth later. When generalized to a larger network, lateral inhibition can yield the center-surround response discussed before.
• The responses of the neuron can be reconstructed from the common and differential modes, e.g., x₁ = (x_c+x_c)/2.
• In the case of the autapse (single feedback loop), excitation corresponded to positive feedback, and inhibition to negative feedback. In a general network, the correspondence is not direct. Here inhibition leads to both positive and negative feedback, depending on which mode is considered. Individual connections (elements of the matrix W) are excitatory or inhibitory, whereas positive and negative feedback cannot be localized to a single synapse. As we shall see, they are quantified by the eigenvalues of W, which are global quantities.

5  General case

• Let x¹,¼,x^N be the right eigenvectors of W:

  
  

  å j  Wij xmj = ~ W   m   xmi

• Let h¹,¼,h^N be the left eigenvectors of W:

  
  

  å j  hm i Wij = ~ W   m   hmj

• The left and right eigenvectors can be chosen so as to satisfy orthogonality constraints

  
  

  å i  hmi xni = dmn (3) å m  xmi hmj = dij (4)

  This means that, when considered as matrices, x and h are inverses of each other.

• We can write x in the x basis as

  
  

  xi = å j  dij xj (5) = å j  å m  xmi hmj xj (6) = å m  xmi å j  hmj xj (7) º å m  hmi ~ x   m   (8)

  So [(x)\tilde]^m, the mth component of x in the x basis, is obtained by projecting onto h^m. A similar equation holds for b.

• To obtain the dynamical equations in the eigenvector basis, we left multiply by h

  
  

  t d dt ~ x   m   + ~ x   m   = å ij  himWijxj + ~ b   m

• Now expand the remaining x_j in terms of its eigenvectors

  
  

  t d dt ~ x   m   + ~ x   m   = å n  å ij  himWijxnj ~ x   n   + ~ b   m

  and observe that

  
  

  å ij  him Wijxnj = ~ W   n   å i  hm i xni = ~ W   n   dmn

  Finally we obtain

  
  

  t d dt ~ x   m   + ~ x   m   = ~ W   m   ~ x   m   + ~ b   m

• In the eigenvector basis, the dynamics decouples into N single feedback loops. The amplification/attenuation for each eigenmode is determined by the corresponding eigenvalue. The gain-bandwidth product is constant for each eigenmode.

6  General case: matrix form

• Suppose that the x^m are the columns of S and h^m the rows of S^-1, x_i^m = S_im and h_i^m = (S^-1)_mi. Then WS = SL and S^-1W = L S^-1.
• The orthogonality conditions are equivalent to the identities S^-1S = I and SS^-1 = I.
• The transformation to the eigenvector basis is [(x)\tilde] = S^-1x.
• The dynamical equations take the form

  
  

  t d dt ~ x   + ~ x   = L ~ x   + ~ b