jang et al: add distinction from TD, and constraint satisfaction / search refs

Author: rcoreilly

citedrefs.json

@@ -19,7 +19,7 @@ There have been a variety of proposals for how error backpropagation could be im
 
 [[#figure_bidir-err]] illustrates this in the context of a simple three-layer network, with two distinct _phases_ of neural activity, starting with an initial _prediction_ or _minus_ phase that reflects the impact of a given _input_ pattern presented over the Input layer of simulated neuron-like processing units. Subsequently, the _outcome_ or _plus_ phase of activity arises when the _actual_ outcome (i.e., correct or target) activity pattern is driven onto the Prediction layer. Remarkably, the simple subtraction of these activity states (_plus -- minus_ or _outcome -- prediction_, i.e., the _temporal derivative_ or _temporal difference_) at any neuron anywhere in such a network provides a good approximation to the error gradient that would otherwise be computed by error backpropagation ([[@OReilly96]]).
 
-Thus, the direct biological prediction from this type of error-driven learning is that the direction of synaptic plasticity should be a function of the change in activity (i.e., temporal derivative) across a time window that would encompass this transition between the prediction and outcome phases. 
+Thus, the direct biological prediction from this type of error-driven learning is that the direction of synaptic plasticity should be a function of the change in activity (i.e., temporal derivative) across a time window that would encompass this transition between the prediction and outcome phases. Note that despite both being based on changes over time, this neocortical learning mechanism is entirely distinct from the _TD_ (_temporal difference_) reinforcement learning algorithm that describes the behavior of dopamine neurons in the midbrain ([[@SuttonBarto98]]; [[@MontagueDayanSejnowski96]]). In TD, dopamine neurons represent the temporal difference _explicitly_ in their firing rates. By contrast, in neocortical temporal derivative learning the error gradient remains _implicit_ in the changes in neural firing over time, and yet this temporal derivative drives synaptic plasticity locally everywhere. This implicit representation of the error gradient has critical advantages in simplifying neural computation as elaborated in the discussion.
 
 {id="figure_pulv-conns" style="height:15em"}
 ![Connectivity between the neocortex and the pulvinar nucleus of the thalamus, in the case of primary and secondary visual areas, that is uniquely well suited for driving predictive error-driven learning. The numerous and relatively weaker projections from layer 6 (VI) neurons are well-suited for activating a prediction over the pulvinar, that integrates the signals from multiple cortical areas and neurons to synthesize the prediction, which improves over the course of learning throughout the neocortex and in these final projections into the pulvinar. By contrast, the strong, focal driver inputs from layer 5 (V) intrinsic bursting (5IB) neurons can activate an outcome representation that is essentially an unlearned copy of the activity pattern in lower cortical layers (e.g., V1 trains V2 predictions in this case). The periodic bursting of the 5IB neurons ensures that this outcome activity is only phasically present (i.e., the plus phase), with a complete prediction -- outcome learning cycle occuring within roughly 200 ms (i.e., theta frequency, 5 Hz). Diagram based on [[@^ShermanGuillery06]].](media/fig_pulvinar_connectivity.png)
@@ -169,13 +169,15 @@ Thus, this difference in overall integration rate stands as a prediction from th
 
 ### Cortical dynamics versus predictive coding
 
-The error-driven learning supported by the corticothalamic prediction vs. outcome mechanism ([[#figure_pulv-conns]]; [[@OReillyRussinZolfagharEtAl21]]) represents an alternative to the widely-discussed Bayesian predictive coding framework (e.g., [[@RaoBallard99]]; [[@Friston09]]). The temporal derivative basis for this alternative enables important differences in the cortical dynamics necessary to support predictive learning, which align better with the available data.
+The error-driven learning supported by the corticothalamic prediction vs. outcome mechanism ([[#figure_pulv-conns]]; [[@OReillyRussinZolfagharEtAl21]]) represents an alternative to the widely-discussed Bayesian predictive coding framework (e.g., [[@RaoBallard99]]; [[@Friston09]]). The temporal derivative basis for this alternative greatly simplifies the cortical dynamics necessary to support predictive learning, which aligns better with the available data.
 
-The Bayesian model requires that a sub-population of neurons directly represent the _prediction error,_ by subtracting a top-down prediction from the bottom-up actual outcome. Thus, different populations of neurons must be somehow segregated so that they can represent fundamentally distinct information. Furthermore, all three of these different signals (prediction, outcome, error) should in principle be communicated across layers, in different directions, requiring strongly segregated pathways.
+The Bayesian model requires that a sub-population of neurons explicitly represent the _prediction error,_ by subtracting a top-down prediction from the bottom-up actual outcome. Thus, different populations of neurons must be somehow segregated so that they can represent fundamentally distinct information. Furthermore, all three of these different signals (prediction, outcome, error) should in principle be communicated across layers, in different directions, requiring strongly segregated pathways.
 
-By contrast, in the temporal derivative model, the entire network is always _coherent_ and _synergistic_ at any given point in time: all layers and neurons are fundamentally cooperating to represent a consistent interpretation of the state of the world, which alternates between representing the prediction versus the outcome.  
+By contrast, in the temporal derivative model, the entire network is always _coherent_ and _synergistic_ at any given point in time: all layers and neurons are fundamentally cooperating to represent a consistent interpretation of the _current_ state of the world. This current state just alternates over time between representing the prediction versus the outcome. If the outcome matches the prediction, then there is no change, which would typically be the situation in a mature, well-trained system: a stable and accurate representation of the world. However, earlier in developmental learning, and in relatively novel or challenging situations in the mature system, unexpected outcomes can drive learning to improve the accuracy of the prediction states.
 
-The available neural evidence is consistent with the coherent, synergistic, redundant encoding of information across all levels of the cortex, with no significant evidence of the kind of structural segregation required by the classical model ([[@WalshMcGovernClarkEtAl20]]; [[@HeilbronChait18]]). The primary evidence that has been found, a suppression of neural activity for expected outcomes relative to unexpected ones, is compatible with the alternative temporal derivative model in conjunction with well-established neural adaptation / accommodation mechanisms ([[@KokLange15]]; see [[@OReillyRussinZolfagharEtAl21]] for detailed discussion).
+This form of learning thus allows for all levels in the network to work together to drive parallel _constraint satisfaction_ processing, integrating top-down and bottom-up constraints, to drive coherent interpretations of the current state ([[@HopfieldTank85]]; [[@OReillyWyatteHerdEtAl13]]). This represents a powerful form of [[search]] through representation space, operating as a kind of inner-loop optimization within the outer-loop of error backpropagation search through synaptic weight space to improve the predictive accuracy of the system. Computational models reported in [[@^OReillyRussinZolfagharEtAl21]] and extensively on [compcogneuro.org](https://compcogneuro.org) demonstrate the efficacy of this form of learning and processing, using biologically-realistic spiking neurons.
+
+The available neural evidence is consistent with the coherent, synergistic, redundant encoding of information across all levels of the cortex, with no significant evidence of the kind of structural segregation required by the Bayesian model ([[@WalshMcGovernClarkEtAl20]]; [[@HeilbronChait18]]). The primary positive evidence that has been found, a suppression of neural activity for expected outcomes relative to unexpected ones, is compatible with the alternative temporal derivative model in conjunction with well-established neural adaptation / accommodation mechanisms ([[@KokLange15]]; see [[@OReillyRussinZolfagharEtAl21]] for detailed discussion).
 
 Thus, the temporal derivative framework supports the widely-accepted idea that the neocortex learns by generating top-down predictions of what will happen next, in a way that appears to be more compatible with available neural evidence at multiple levels of analysis.