TLDR: A study on fNIRS classification found that symmetrical activation functions like Tanh and Absolute consistently outperform common functions like ReLU in deep neural networks. This suggests that aligning activation function properties with fNIRS signal characteristics, which include meaningful bidirectional variations, is crucial for improving model accuracy.
Functional Near-Infrared Spectroscopy (fNIRS) is a non-invasive brain imaging technique that measures brain activity by detecting changes in blood oxygenation. It’s increasingly used with deep neural networks (DNNs) for tasks like classifying brain signals. However, a crucial component of these networks, known as activation functions, has not been thoroughly studied for its impact on fNIRS classification performance.
The Challenge of Activation Functions in fNIRS
Activation functions introduce non-linearity into neural networks, allowing them to learn complex patterns. While many functions exist, such as the widely used Rectified Linear Unit (ReLU), their effectiveness can vary significantly across different data types and network architectures. In the fNIRS domain, signals often have low signal-to-noise ratios and high variability, making the choice of activation function particularly important. Despite this, there hasn’t been a systematic investigation into which activation functions work best for fNIRS classification.
A Systematic Evaluation
A recent study aimed to fill this gap by evaluating a range of conventional and specialized activation functions for fNIRS classification tasks. The researchers tested seven different activation functions: ReLU, Exponential Linear Unit (ELU), Swish, Sigmoid, Hyperbolic Tangent (Tanh), Square, and the Absolute function. These were evaluated across multiple deep learning architectures, including fNIRSNet, AbsoluteNet, MDNN, and ShallowConvNet (used as a baseline). All tests were conducted on a single dataset recorded during an auditory task, ensuring a fair comparison through standardized preprocessing and consistent training parameters.
Understanding Activation Function Properties
The study highlighted key properties of activation functions that influence their performance:
- Symmetry: Functions mirrored around the y-axis or origin, potentially beneficial for data centered around zero.
- Boundedness: Outputs remain within a finite range, helping prevent exploding activations.
- Monotonicity: Functions that either never increase or never decrease, simplifying optimization.
- Smoothness: Continuously differentiable functions, aiding gradient-based optimization.
Key Findings: The Power of Symmetry
The results consistently showed that symmetrical activation functions, particularly Tanh and the Absolute value function (Abs(x)), often outperformed commonly used functions like ReLU. This was observed across various network architectures. For instance, Tanh achieved the highest test accuracy in fNIRSNet, AbsoluteNet, and ShallowConvNet, while the Absolute function excelled in MDNN and showed strong, stable results in AbsoluteNet and fNIRSNet.
The study also introduced a Modified Absolute Function (MAF) to specifically analyze the role of symmetry. This analysis further supported the effectiveness of symmetrical activation functions. When the MAF was configured to behave like ReLU (ignoring negative inputs), performance consistently dropped. This suggests that for fNIRS data, which can contain meaningful information in both positive and negative signal variations, preserving both polarities through symmetrical activation functions is crucial for better model performance.
Conversely, functions like Square generally yielded the weakest results across the tested networks, indicating their unsuitability for this domain without further tuning.
Also Read:
- Unpacking Neural Network Efficiency: How Dense Circuits Solve the Universal-AND Problem
- Bridging the Performance Gap: Enhancing Spiking Neural Networks with Cross-Modal Knowledge Transfer
Implications for fNIRS Research
These findings underscore the critical importance of carefully selecting activation functions that align with the unique signal characteristics of fNIRS data. Rather than defaulting to standard choices like ReLU, researchers and developers in the fNIRS domain should consider symmetrical functions such as Tanh and Absolute. This tailored approach can significantly enhance the accuracy and reliability of deep learning models in fNIRS-based classification tasks.
While this study provides valuable insights, the authors note limitations, including the use of homogeneous activation strategies (all layers using the same function) and testing on a single dataset. Future research will explore hybrid activation designs and evaluate performance across multiple datasets to further generalize these findings.
For more detailed information, you can read the full research paper here.
