TLDR: Researchers have developed the In-Memory Computing Emulator (IMCE), a distributed and expandable system for rapid prototyping and testing of AI integrated circuits that use in-memory computing. This emulator allows for early analysis, microcode testing, and application deployment, significantly shortening development time and ensuring high accuracy for complex AI models like ResNet and YOLO.
The rapid advancement of artificial intelligence, especially in edge computing devices, relies heavily on efficient hardware. A key technology in this area is In-Memory Computing (IMC), which allows AI devices to perform calculations directly within memory, leading to significantly lower power consumption and faster processing of complex matrix-based functions. However, developing and integrating these specialized AI chips into larger systems is a time-consuming process, often taking months or even years, especially since extensive testing and verification are needed before the physical chip is even available.
To address this challenge, researchers have developed a novel solution: the In-Memory Computing Emulator (IMCE). This distributed and expandable emulation system is designed for the rapid prototyping of integrated circuits based on in-memory computing technologies. It provides a real-time environment where various system aspects can be analyzed, microcode can be tested, and applications can be deployed, all before the actual chip is manufactured. This significantly shortens the development cycle and allows for in-depth analysis and debugging, even enabling the intentional introduction of noise to simulate real-world conditions in non-volatile memory systems.
Understanding the IMCE Architecture
The IMCE system is built upon three main components: a Front End unit (IMCE-FE), multiple Processing Units (IMCE-PUs), and a Configuration and Data Analytics server (IMCE-CDA). These units communicate over high-speed links, typically 1 or 10 Gbps Ethernet, and are organized into Neural Processing Unit (NPU) clusters, which can be interconnected to form a large emulation environment capable of supporting extensive Deep Neural Networks (DNNs).
The IMCE-FE acts as the interface to the emulated chip’s electrical connections, allowing the emulator to integrate with existing testing platforms. The IMCE-CDA server is the brain for setup and analysis, using custom software to map AI models onto the IMCE’s internal architecture. It also generates data for verification and collects statistical information from all processing nodes with minimal latency, enabling comprehensive data analysis.
The core computational power comes from the IMCE-PUs, which are implemented on individual FPGA boards. There are two types of PUs: Analog Processing Units (AnPUs) and Digital Processing Units (DiPUs. Each PU features multiple Arm Cortex processors and a custom hardware accelerator. The PUs operate on a “compute-and-forward” principle, processing data as it arrives and immediately sending outputs to the next units, maximizing pipelining and processing rates.
Specialized Accelerators: AnPU and DiPU
The AnPU is designed to emulate analog Matrix-Vector Multiplications (MVMs) and activation operations, which are crucial for DNNs. It can also handle convolution operations by transforming them into a series of MVMs using a technique called im2col. This unit is highly optimized for parallel processing, utilizing numerous DSP units for efficient Multiply-Accumulate (MAC) operations.
Conversely, the DiPU functions as a Data Processing Unit, handling a variety of digital operations. These include additions (with or without ReLU activation), Sigmoid Linear Unit (SiLU) functions, Max/Average Pooling for dimensionality reduction, concatenation to merge tensors, and splitting to divide them. The DiPU employs custom hardware engines and dedicated DMA for fast data transfers, ensuring high processing rates for these essential digital tasks.
The Software Ecosystem
To deploy AI models on the IMCE, a sophisticated software stack is in place. This stack begins with AI models developed in various frameworks, which are then exported in the ONNX format. The AI Framework2IMCE Software Kit includes High-level Transformation Tools (HTT) that optimize and quantize the AI model according to IMCE characteristics, generating parameters for each node. Hardware-related Configuration Tools (HCT) then map these nodes to the available FPGA boards and create the necessary configuration files. The system also supports the introduction of non-volatile memory (NVM) noise models for realistic testing. Finally, the RunTime application manages the model’s inference on the IMCE-PUs.
Underpinning this is the Diolkos software stack, which facilitates communication between the CDA server and the IMCE-PUs, enabling efficient data exchange and statistical data collection.
Also Read:
- NANOMIND: Bringing Advanced Multimodal AI to Small, Battery-Powered Devices
- Living Neurons Power AI: A New Era for Biological Computing
Demonstrated Performance
The usefulness of the IMCE has been demonstrated through experimental results using popular AI models such as ResNet8, ResNet18, and YOLOv8n. For instance, ResNet8, a classification model, showed an accuracy of 88.18% on the IMCE, closely matching the 88.25% achieved by the ONNX Runtime with INT8 quantization. Similarly, ResNet18 achieved 93.10% accuracy on the IMCE compared to 93.17% with ONNX Runtime (INT8). For object detection, YOLOv8n on the IMCE achieved a mAP50-95 of 0.344, very close to the theoretical INT8 model’s 0.345. These results confirm that the IMCE delivers high accuracy consistent with theoretical models, making it a reliable tool for studying and validating DNN models.
In conclusion, the IMCE provides a flexible, expandable, and accurate emulation environment that significantly accelerates the development and testing of AI integrated circuits based on in-memory computing. By supporting a wide range of analog and digital functions and offering a comprehensive software toolchain, it paves a clear path from DNN models to efficient inference execution. For more detailed information, you can read the full research paper here.
