What Is Wrong With Backpropagation

The Forward-Forward Algorithm: Some Preliminary Investigations

Geoffrey Hinton [Google Brain]

27 Dec 2022

What Is Wrong With Backpropagation
 Despite the mathematical advantages we've obtained thanks to backpropagation, the paper maintains that backpropagation is an implausible method when we consider how the actual cortex is trained. Cortex does not mirror bottom-up connections like backpropagation does. Instead it forms loops through cortical layers (Let's not go too deep on neuroscience). This doesn't mean that the representations in earlier stages being affected by the representations in later stages is nonsense. However, the way it executes the feedback process should be top-down, not bottom up. Furthermore, the inference and learning process should be held in real time without stopping to perform backpropagation.

Forward-Forward Algorithm
 The forward-forward algorithm is an immature architecture for preliminary investigation which proffers a contrary idea about how artificial neural networks should have been. While backpropagation uses one forward pass and one backward pass, forward-forward algorithm uses two forward passes. The former pass is composed with positive (i.e. real) data while the latter one is composed with negative data. Every hidden is influenced by the data in terms of the measure called 'goodness'. Goodness can be adjusted in various ways on many different tasks.

  The crucial limitation backpropagation has is that it requires full knowledge about the forward pass. So, whenever we weren't able to perfectly calculate the derivatives, it tends to rely heavily on reinforcement learning. However reinforcement learning procedures suffer from high variance. The main point of this paper is to show that neural networks containing unknown non-linearities do not need to resort to reinforcement learning. Two key advantages the forward-forward algorithm provides are as follows:

1. Able to use despite the black box setting on forward computation.
2. Able to learn while pipelining sequential data without storing or stopping.

 

 The rest of the contents are about how forward-forward algorithm could stay competitive to the existing backpropagation algorithm. Forward-forward algorithm might be superior because the model resembles the way how cortex learns and it does not require a reinforcement learning in the setting of very low-power analog hardware.

 

Artificial Neural Network vs. Cortex

 Artificial intelligence is made to resemble the way human beings think. Thus practically the ultimate goal of the artificial neural nets is the cortex. Even though yet the performance of the forward-forward algorithm isn't satisfactory and the concept seems abstract and vague, the algorithm might be an important preliminary investigation for thousands of subsequent papers.