TLDR: This research demonstrates that providing Large Language Models (LLMs) with structured supplementary information, such as user and business average ratings, significantly improves sentiment analysis accuracy. By using JSON-formatted prompts, even lightweight 3-billion parameter LLMs can achieve competitive performance without fine-tuning. The study confirms that LLMs engage in genuine contextual reasoning, utilizing these reference points for relative evaluation rather than simply proxying labels, and highlights the importance of structured input for effective information utilization.
Large Language Models (LLMs) have become indispensable tools across various industries, including marketing research, where sentiment analysis plays a crucial role in understanding consumer preferences. Traditionally, sentiment analysis in Natural Language Processing (NLP) primarily focuses on analyzing review text in isolation. However, established marketing theories, such as prospect theory and expectation-disconfirmation theory, suggest that consumer evaluations are shaped not only by their direct experience but also by additional ‘reference points’. These reference points can include past purchasing patterns, prior experiences with a business, or comparative evaluations against competitors.
This research paper, titled “REFERENCE POINTS IN LLM S ENTIMENT ANALYSIS : THE ROLE OF STRUCTURED CONTEXT”, investigates how incorporating such supplementary information, and the format in which it’s presented, impacts sentiment analysis using LLMs. The study, conducted by Junichiro Niimi, explores the effectiveness of natural language (NL) versus JSON-formatted prompts when providing additional context to a lightweight 3-billion parameter LLM, making it suitable for practical marketing applications and resource-constrained edge devices.
The researchers conducted experiments on two Yelp categories: Restaurant and Nightlife. They compared models that received no supplementary information, those that received it in natural language, and those that received it in a structured JSON format. The supplementary information included user’s average past ratings, business’s average ratings, and other contextual factors like restaurant name, operating hours, and open days.
The findings were significant. The model using JSON-formatted prompts with all additional information (JSON-UBO) consistently outperformed all baselines, including those without fine-tuning. For the Restaurant dataset, Macro-F1 score increased by 4.3% and RMSE (Root Mean Square Error) decreased by 16.44% compared to the baseline LLM without supplementary information. In the Nightlife category, the improvements were even more substantial, with a 20.7% increase in Macro-F1 and a 15.8% reduction in RMSE. This clearly demonstrates that supplying reference points in a machine-friendly format helps the model capture complex relationships and improve prediction accuracy.
Interestingly, while JSON prompts showed consistent performance gains as more information was added, natural language prompts did not follow this pattern. This suggests that even with large context windows, LLMs can struggle to effectively utilize complex contextual factors when embedded as plain text, highlighting the critical role of prompt design.
A follow-up analysis addressed a key concern: whether these reference points were merely acting as proxies for the actual sentiment labels. The study found that prediction accuracy actually improved when the review scores deviated significantly from the average reference points. This indicates that the LLM was engaging in genuine contextual reasoning, using the reference points for relative evaluation rather than simply copying them as labels.
Furthermore, the research explored how interactions between different reference points affect prediction accuracy. It was observed that accuracy tended to improve when user’s past evaluations aligned with other consumers’ average ratings (e.g., both high or both low). Conversely, conflicting reference points often indicated inherently challenging cases for prediction. This insight has practical implications, allowing companies to develop adaptive inference strategies where samples with aligned reference points can be processed efficiently on-device, while more complex, conflicting cases can be routed to larger, cloud-based models for further analysis.
Also Read:
- Dynamic Self-Awareness for Efficient Large Language Model Reasoning
- Inclusion Arena: Advancing AI Model Evaluation Through Real-World Application Feedback
In conclusion, this study provides compelling evidence that incorporating structured supplementary information, guided by marketing theories, can significantly enhance the performance of LLM-based sentiment analysis. The use of JSON-formatted prompts enables even smaller, lightweight models to achieve competitive accuracy without the need for extensive fine-tuning, offering a practical and computationally efficient alternative for real-world applications. For more details, you can read the full research paper here.
