TLDR: The REAL (Real Estate Agent Large Language Model Evaluation) benchmark has been introduced to assess the capabilities of Large Language Models (LLMs) in housing transactions and services. Comprising 5,316 entries across memory, comprehension, reasoning, and hallucination topics, REAL evaluates LLMs on their domain-specific knowledge and skills. Initial experiments show that while LLMs have potential, they still require significant improvement, especially in reasoning and avoiding hallucinations, before they can effectively serve as real estate agents.
Large Language Models (LLMs) have significantly advanced the development of chatbots across many sectors, from legal and financial advice to disease diagnosis. A critical question arises: can these LLMs effectively serve as agents in complex fields like housing transactions and services, performing as well as human experts?
To address this, researchers have introduced the Real Estate Agent Large Language Model Evaluation (REAL), the first comprehensive evaluation suite specifically designed to assess LLMs’ capabilities in the real estate domain. This benchmark aims to determine if LLMs possess the necessary knowledge and skills for scenarios involving housing transactions and services.
REAL is a robust evaluation tool, comprising 5,316 high-quality evaluation entries. These entries are organized across four key topics: memory, comprehension, reasoning, and hallucination. These topics are further broken down into 14 distinct categories, covering various aspects of real estate expertise.
Understanding the Core Topics of REAL
The benchmark evaluates LLMs on several crucial abilities:
Memory: This section assesses an LLM’s knowledge of real estate blocks, including inherent attributes like green space ratios and parking availability, as well as information about surrounding facilities and their distances. Questions might ask an LLM to match a block with specific attributes or a business district.
Comprehension: This topic tests an LLM’s ability to understand natural language descriptions related to real estate. For instance, it might require an LLM to associate a detailed description with the most consistent block entity or recommend blocks based on specific client demands.
Reasoning: Considered an advanced skill, reasoning evaluates an LLM’s capacity to infer client preferences. This involves analyzing past behavior data, such as browsing and clicking information on real estate websites, to predict which type of house a client is most likely to purchase.
Hallucination: This unique topic specifically targets a common LLM problem: generating non-existent or fabricated information. Unlike human agents who wouldn’t invent details, LLMs might. REAL uses queries with fictional names (e.g., fictional block or area names) to see if the model generates responses that acknowledge the non-existence or if it “hallucinates” information about them. This is crucial for ensuring reliability in online service scenarios.
Also Read:
- J1-ENVS: A New Frontier for Legal AI Evaluation
- Rethinking LLM Evaluation: Why ‘Answer Matching’ Outperforms Multiple Choice
Data and Evaluation
The data for REAL is meticulously collected from Beike corporation, a major real estate company, and includes client queries from the internet. All data is processed to remove sensitive information and ensure quality. It’s important to note that, for now, the scope of REAL’s entries is limited to Beijing-specific information.
Experiments using REAL have evaluated several advanced LLMs, including various GPT-4 models, Qwen, DeepSeek, GLM, Baichuan, and Yi, alongside internal models from Beike Inc. The results indicate that while some models, particularly Beike’s internal models, show promising performance, LLMs generally still have significant room for improvement before they can be directly applied as agents in the real estate field. For example, the accuracy of all models in reasoning tasks was below 50%, highlighting the difficulty of this skill for current LLMs. Furthermore, the hallucination evaluation revealed that LLMs frequently generate hallucinatory responses when presented with fictional names, often introducing information about them rather than questioning their existence.
The development of REAL is a significant step towards creating more capable and reliable AI agents for the real estate sector. Researchers plan to expand and optimize the benchmark to include data from other cities in the future. For more in-depth details, you can refer to the full research paper here.
