TLDR: Researchers have introduced FAUST (Forward-forward Algorithm Unified with Similarity-based Tuplet loss), a novel algorithm that redefines the Forward-Forward (FF) algorithm using a similarity-based objective. This approach significantly improves FF’s accuracy and inference efficiency, narrowing the performance gap with traditional backpropagation. FAUST eliminates the need for multiple forward passes during inference by leveraging anchor-positive-negative relationships and tuplet loss, achieving competitive results on datasets like MNIST, Fashion-MNIST, and CIFAR-10.
Artificial neural networks have achieved remarkable success in various fields, largely thanks to an algorithm called backpropagation. However, backpropagation has some notable limitations. For instance, it requires storing all intermediate calculations during training, which can be memory-intensive. More importantly, its mechanism of global error propagation and ‘backward locking’ isn’t considered biologically realistic, meaning it doesn’t align well with how the human brain is thought to learn.
To address these issues, the Forward-Forward (FF) algorithm was proposed as a more biologically plausible alternative. Instead of a backward pass to adjust weights, FF uses an additional forward pass. While this eliminates the need for backward locking and uses local learning signals, it has historically lagged behind backpropagation in accuracy. Furthermore, its optimal form requires multiple forward passes during inference, making it less efficient for real-world applications.
A new research paper, “Reshaping the Forward-Forward Algorithm with a Similarity-Based Objective”, introduces a novel approach called FAUST (Forward-forward Algorithm Unified with Similarity-based Tuplet loss) that significantly improves upon the traditional FF algorithm. This work integrates FF with similarity learning frameworks, leading to enhanced accuracy and eliminating the need for multiple forward passes during inference.
How FAUST Works
Traditional FF operates by feeding both ‘positive’ (correctly labeled) and ‘negative’ (incorrectly labeled) data through a network layer. It then calculates a ‘goodness’ score, aiming to increase this score for positive inputs and decrease it for negative ones. During inference, it tests all possible labels to find the one that yields the highest goodness score across all layers, which is computationally expensive.
FAUST fundamentally redefines this objective. Instead of goodness scores, it uses a similarity-based objective. In this framework, the model is presented with an ‘anchor’ image, a ‘positive’ image from the same class, and one or more ‘negative’ images from different classes. The goal is to train the network to produce embeddings (numerical representations) where the anchor is closer to the positive image than to any negative images.
The researchers explored different loss functions for this similarity learning: the Triplet Margin Loss, which considers one negative class at a time, and the more generalized (N+1)-tuplet Loss, which can handle multiple negative classes. They developed three variations of FAUST:
- FAUST-vanilla triplet: Uses the triplet loss with random sampling.
- FAUST-vanilla tuplet: Uses the (N+1)-tuplet loss with random sampling.
- FAUST-representative tuplet: A more efficient approach where one image from each class is selected as a ‘representative.’ During training, each image in a batch acts as an anchor, compared against its corresponding positive representative and all other negative representatives. This significantly reduces computational overhead and simplifies the optimization problem.
A key advantage of FAUST is its inference efficiency. Unlike the optimal form of traditional FF, which requires a separate forward pass for each possible class, FAUST only needs a single forward pass per test input. This is achieved by classifying an input based on its proximity to pre-computed class centroids or representative embeddings.
Impressive Results
Empirical evaluations on standard datasets like MNIST, Fashion-MNIST, and CIFAR-10 showed that FAUST substantially improves accuracy compared to existing FF algorithms. FAUST-vanilla tuplet, for example, achieved up to 7.03% higher accuracy than FAUST-vanilla triplet. The FAUST-representative tuplet variant consistently outperformed the vanilla tuplet version on more complex datasets like Fashion-MNIST and CIFAR-10, confirming the benefits of using fixed representatives.
While backpropagation still holds a slight edge on complex datasets, FAUST significantly narrows the performance gap. On CIFAR-10, FAUST achieved 56.22% accuracy with a simple multi-layer perceptron, closely approaching backpropagation’s benchmark of 57.63% accuracy. The research also demonstrated that deeper layers in FAUST learn progressively more informative and discriminative representations, and its convergence rate is comparable to backpropagation.
Also Read:
- Rethinking Neural Network Training: Why Gradient Direction Matters More Than Magnitude
- Machine Unlearning: A Targeted Approach to Eradicating Bias in AI Vision Models
Looking Ahead
FAUST represents a significant step forward in developing biologically plausible and efficient alternatives to backpropagation. By integrating similarity learning, it overcomes key limitations of the original Forward-Forward algorithm, offering improved accuracy and single-pass inference. Future work will focus on applying FAUST to more challenging tasks and larger network architectures, including convolutional neural networks, and exploring other methods for formulating the similarity objective.
