TLDR: AdvisingWise is a multi-agent AI system designed to assist academic advisors in higher education by automating information retrieval and drafting responses. It ensures reliability through institutional resources, personalizes advice by adapting to student backgrounds, and maintains human oversight by requiring advisor validation for all responses. Evaluations show high accuracy and positive reception from advisors, demonstrating the potential of human-AI collaboration in academic advising.
Academic advising is a cornerstone of student success in higher education, guiding students through their academic journeys and career development. However, universities often face significant challenges, including high student-to-advisor ratios and a lack of formal training for many faculty advisors in complex institutional policies. These issues can lead to advisors feeling overwhelmed, making it difficult to provide timely and personalized support, especially during busy periods.
Recent advancements in Large Language Models (LLMs) offer promising avenues to enhance the advising process. While tools like ChatGPT show potential, they often fall short in high-stakes scenarios due to issues like factual inaccuracies (hallucinations) and a lack of personalization. Effective advising demands reliable, tailored guidance that considers each student’s unique circumstances, and it also requires a human touch, as emotions and trust play a crucial role.
Introducing AdvisingWise: A Human-in-the-Loop Multi-Agent System
To address these challenges, researchers from Tufts University have developed AdvisingWise, a multi-agent system designed to automate time-consuming tasks such as information retrieval and drafting responses, all while preserving essential human oversight. AdvisingWise is built on three core principles: reliability, personalization, and an advisor-in-the-loop approach.
Reliability: The system ensures accuracy by leveraging authoritative institutional resources and employing a sophisticated three-phase architecture for query processing, information collection, and quality-controlled response generation. This includes a “ReAct-style” information retrieval mechanism, where specialized agents iteratively reason about what information is needed and then take actions to retrieve it from knowledge bases, course databases, or even the web.
Personalization: AdvisingWise maintains student profiles based on conversation history. When a student asks a question, the system adaptively prompts them for additional academic background if needed, allowing it to generate responses tailored to their specific situation, such as their major, completed courses, or degree requirements.
Advisor-in-the-Loop: Crucially, all draft responses generated by AdvisingWise undergo human advisor validation before being delivered to students. Advisors receive detailed answers, summaries, and lists of cited sources, along with “advisor-only notes” that flag any uncertainties or limitations in the draft. This allows advisors to review, edit, and approve responses, ensuring human judgment and connection remain central to the process.
How AdvisingWise Works for Students and Advisors
Students interact with AdvisingWise through a familiar chat-based environment, like Rocket.Chat. When a student submits a question, AdvisingWise processes it and sends a notification to the designated advisor with an AI-generated draft response. This draft is presented as a collapsible “thread” within the chat interface. Advisors can then easily review the response, make any necessary edits, and approve it for delivery to the student. If the system needs more information for personalization, it will ask the student follow-up questions directly, streamlining the process and reducing back-and-forth communication.
Evaluating AdvisingWise: Promising Results
The system was evaluated using a mixed-methods approach, including expert evaluation of responses, an LLM-as-a-judge comparison of its information retrieval strategy, and a user study with eight academic advisors from a computer science department at a private U.S. university. The results were highly positive:
- High Accuracy: Expert validation showed that 84.2% of AdvisingWise’s responses achieved high accuracy scores. The system was designed to be conservative, often declining to answer when uncertain rather than providing incorrect information.
- Superior Information Retrieval: In comparisons using an LLM-as-a-judge framework, AdvisingWise’s ReAct-style retrieval system outperformed a standard Retrieval-Augmented Generation (RAG) baseline with a 3:1 preference ratio, especially for complex queries where information was not explicitly stated in source documents.
- Positive Advisor Perceptions: Advisors reported increasingly positive perceptions after using AdvisingWise. Initial concerns about reliability and personalization diminished as they observed the system successfully handle complex, context-dependent queries with accurate and nuanced responses. Many expressed optimism about integrating such a tool into their daily work, seeing it as a way to answer simple questions quickly and free up their time for more complex student needs.
Also Read:
- AI Agents Collaborate to Uncover New Scientific Machine Learning Methods
- New AI System ED2D Uses Evidence-Based Debate to Combat Misinformation
Key Insights and Future Directions
The study revealed several important implications for integrating AI into academic advising:
- Design Drives Trust: Addressing advisor concerns about accuracy and personalization directly in the system’s design was crucial for fostering trust and adoption.
- Conciseness in Uncertainty: When information is unavailable or uncertain, advisors preferred a brief acknowledgment of limitations over lengthy, speculative responses, even with disclaimers. “Less is more” in these situations.
- Accommodating Diverse Styles: Advisors have varied preferences for response length, tone, and detail. Future systems could benefit from customization options to match individual advising styles.
- The “Confidence Illusion”: While AI can be highly accurate, advisors were reluctant to delegate tasks involving emotionally sensitive situations (e.g., failing a course). This highlights that advising involves emotional support and relationship-building, which AI cannot replace. Human oversight remains vital for these nuanced interactions.
While AdvisingWise shows great promise, the researchers acknowledge limitations, including a relatively small sample size in the user study and the use of hypothetical scenarios. Future work will involve larger, more diverse participant groups, real-world deployment studies, and exploring student perspectives on AI-advising integration.
AdvisingWise represents a significant step towards a human-AI synergy in academic advising, demonstrating how AI can effectively support information-intensive tasks while preserving the irreplaceable human element of mentorship and emotional support. You can read the full research paper here.
