TLDR: A new research paper introduces a framework that uses Multi-modal Large Language Models (MLLM) to enhance the Fireworks Algorithm (FWA) for complex optimization tasks. By incorporating visual information alongside text, the framework improves FWA’s ability to solve challenging problems like the Traveling Salesman Problem (TSP) and Electronic Design Automation (EDA). The approach, which includes a ‘Critical Part’ concept, demonstrates superior performance and resource efficiency, outperforming existing methods on many benchmarks. The study also analyzes how MLLMs design algorithms when given visual cues, noting that visual information can lead to more customized solutions for low-dimensional problems but may amplify commonalities in high-dimensional tasks.
A recent research paper introduces a groundbreaking framework that significantly enhances the capabilities of the Fireworks Algorithm (FWA) in tackling complex optimization challenges. This novel approach integrates Multi-modal Large Language Models (MLLM) to improve how optimization algorithms are designed, especially for problems that are traditionally difficult to solve due to their complexity, high dimensionality, or black-box nature.
The paper, titled “Using Multi-modal Large Language Model to Boost Fireworks Algorithm’s Ability in Settling Challenging Optimization Tasks,” was authored by Shipeng Cen and Ying Tan from Peking University. Their work addresses the limitations of conventional optimization methods and even previous LLM-assisted algorithm designs, which often struggle with fully utilizing available optimization information and extending to very complex problems.
The Core Idea: MLLM and the Critical Part
At the heart of this new framework is the concept of a “Critical Part” (CP), which allows the MLLM to focus its design efforts on specific aspects of the FWA. The FWA itself is a swarm intelligence optimization algorithm, known for its simplicity and effectiveness in various scenarios, including those without gradient information. The MLLM, with its advanced language understanding and code generation abilities, is tasked with designing or refining the operators within the FWA (like explosion, mutation, and selection operators).
What makes this framework unique is the introduction of multi-modal information, particularly visual data, into the algorithm design process. Unlike previous methods that relied solely on text-based interactions, this approach feeds various visualizations of the optimization process to the MLLM. This gives the MLLM a richer, multi-perspective view of the task, allowing it to make more informed design decisions.
The Critical Part concept also broadens the applicability of FWA. For problems that are well-suited for global optimization, the CP module directly deploys FWA. However, for highly complex and high-dimensional tasks, such as those found in electronic design automation, the framework evolves a local FWA to refine specific rules, like step-size adjustments within existing optimizers.
Tackling the Traveling Salesman Problem (TSP)
The researchers validated their framework on two highly challenging NP-hard problems: the Traveling Salesman Problem (TSP) and the Electronic Design Automation (EDA) problem.
For the TSP, a classic combinatorial optimization problem, the FWA was used as a global optimizer. Each individual in the FWA population represented a unique sequence of city visits. The MLLM was guided through mutation and crossover operations, where it redesigned or combined FWA operators. Crucially, the MLLM received not only textual information (FWA codes and tour lengths) but also visual information. This visual input included route visualizations with convex hulls, path crossing heatmaps, and route density analyses. These visualizations helped the MLLM intuitively understand path configurations and identify deficiencies.
The experimental results for TSP were impressive. The algorithm generated by the new framework consistently outperformed leading TSP heuristics and algorithms across various TSPLIB instances. In some cases, it even found paths that were numerically better than the known optimal solutions in TSPLIB when evaluated with floating-point arithmetic. An interesting finding was that visual information, while not always guaranteeing better performance in rounded results, led to the generation of more customized and heterogeneous FWA operators, suggesting the MLLM was better able to capture unique geometric features of the paths.
Innovating Electronic Design Automation (EDA)
The EDA problem, specifically integrated circuit placement, presents a different kind of challenge due to its extremely high dimensionality (millions of variables). Here, the framework didn’t use FWA as a global optimizer but rather leveraged its optimization ideas to design heuristic rules for adjusting the step size within the open-source DreamPlace framework. The DreamPlace framework, while powerful, often suffers from instability due to its reliance on a simple Barzilai-Borwein step-size heuristic.
The MLLM was tasked with designing a “firework_step_redesign” function to provide a better step for the placement process. Similar to TSP, visual information in the form of placement visualizations was provided to the MLLM. The framework achieved state-of-the-art performance on 6 out of 8 ISPD2005 benchmarks, with significantly reduced wirelength (wHPWL) values, all while using very limited computational resources (a single 12GB GPU). This contrasts sharply with other LLM-based approaches like EOAGP, which required distributed GPU clusters.
However, the analysis of MLLM behavior in EDA revealed a different pattern. While visual information was beneficial for high-dimensional instances, it didn’t always guarantee better performance. Furthermore, due to the vastness and complexity of EDA layouts, the visual information tended to amplify surface commonalities between different instances rather than highlighting unique features, leading to higher similarity in the generated FWA codes compared to the TSP case.
Also Read:
- Optimizing LLM Collaboration: A Graph-Based Approach to Test-Time Scaling
- Unpacking How AI Models Resolve Conflicting Information from Images and Text
Conclusion and Future Outlook
This research demonstrates a powerful new paradigm for optimization algorithm design. By integrating multi-modal information and the concept of a Critical Part, the framework significantly extends the applicability of swarm intelligence algorithms like FWA to a broader range of complex problems. It achieves superior performance in both combinatorial (TSP) and physical design (EDA) challenges, often with remarkable resource efficiency. The study also provides valuable insights into how MLLMs behave when presented with visual optimization information, highlighting that the impact of visual data can vary depending on the problem’s characteristics and dimensionality. For more details, you can read the full paper here.
