TLDR: TVS Sidekick is an in-house AI assistant developed by TVS SCS UK, a logistics provider, to enhance internal operations using Large Language Models (LLMs) and Retrieval Augmented Generation (RAG). The paper details the technical implementation, including data ingestion and query processing, and outlines the ethical, regulatory, and sociotechnical challenges encountered during its deployment. It highlights the company’s efforts to comply with regulations like the EU AI Act and ISO/IEC 42001 through an AI Management System, and shares practical insights from monitoring usage and gathering user feedback.
In today’s rapidly evolving business landscape, enterprises are increasingly turning to Artificial Intelligence (AI) to boost competitiveness and efficiency. However, the journey to integrate AI, especially advanced Large Language Models (LLMs), into an organization is fraught with ethical, regulatory, and practical challenges. A recent research paper, TVS Sidekick: Challenges and Practical Insights from Deploying Large Language Models in the Enterprise, sheds light on these complexities through a real-world case study at TVS Supply Chain Solutions (TVS SCS UK).
Introducing TVS Sidekick: An In-House AI Assistant
TVS SCS UK, a leading third-party logistics provider, embarked on an ambitious AI transformation by developing an in-house AI team and its flagship product, TVS Sidekick. This AI assistant is designed to enhance internal operational efficiency by leveraging LLMs to answer queries using relevant enterprise data. Unlike many companies that rely on third-party integrators, TVS SCS UK chose an in-house approach to maintain agility and foster innovation.
At its core, TVS Sidekick is built upon the principles of Retrieval Augmented Generation (RAG). This means it doesn’t just generate answers; it first retrieves relevant information from the company’s internal cloud-based systems. These documents, ranging from SharePoint files and Azure DevOps data to code repositories and the TVS website, are vectorized and compared to user queries. The LLM then extracts information and provides answers, often with custom prompting. Users interact with Sidekick through a Microsoft Teams extension, making it easily accessible within their existing workflow.
A notable technical contribution of Sidekick is its ability to handle both text and code-related queries. For code, files are logically split, and an LLM generates descriptions of their purpose, structure, and functions. Both code and text fragments are stored in a vector database, allowing Sidekick to provide relevant source code lines as context when responding to technical questions.
Navigating the Regulatory Maze
Deploying LLMs in an enterprise environment means navigating a complex legal and regulatory landscape. The European Union Artificial Intelligence Act (EU AIA) and standards like ISO/IEC 42001 for AI Management Systems are central to this. TVS SCS UK is proactively addressing these by working towards AI standardization, recognizing its importance for responsible AI development and adoption.
The company has adopted an AI Management System (AIMS) framework, aligning ISO/IEC 42001 requirements with their existing management systems, such as ISO/IEC 27001 for information security. This strategic mapping helped identify key focus areas for AIMS implementation, including:
- AI Risks: Establishing a Knowledge Base of academic research and case studies to stay informed about current AI risks and inform development.
- Monitoring & Measuring: Developing a monitoring system for Sidekick that tracks usage indicators like interaction volume, response time, and user engagement, providing pragmatic measures of effectiveness.
- Continuous Improvement: Implementing processes for ongoing evaluation and technical documentation, including periodic feedback interviews with target user groups to understand needs and support a culture of ethical AI.
Sociotechnical Challenges and User Insights
The deployment of Sidekick has also brought to light various sociotechnical challenges. Quantitative statistics from a four-month monitoring period (March-June 2025) showed continued use, with over 500 prompts in the first two months. The most active users were primarily in technology and business roles, with SharePoint data queries being the most common.
Qualitative feedback gathered from 24 interviews within the IT department provided deeper insights:
- Enhanced Retrieval: Users highly valued Sidekick’s ability to provide specific TVS data, finding it faster and more readable than manual searches, especially for SharePoint tasks.
- Good for Business Logic: Sidekick proved particularly helpful for business analysts in understanding the context of TVS data and key definitions within business processes.
- Not Enough Technical Detail: Developers expressed a need for more in-depth domain knowledge, especially for legacy programming languages with limited documentation. While Sidekick offered a good starting point, it sometimes fell short on highly specific technical queries.
- Keen to Engage with AI: Overall, staff were enthusiastic about using Sidekick for tasks like code standardization, reducing duplication, and onboarding new starters. They also proposed new features, such as learning from user prompts.
- Privacy/Commercially Sensitive Questions/Other Concerns: While no major concerns were raised regarding Sidekick’s use with proper information feeding and access levels, some staff expressed general concerns about job security and distrust in AI tools. The importance of using Sidekick internally to prevent disclosure of client information was also emphasized.
Also Read:
- Fairness Under Scrutiny: How Minor Prompt Changes Uncover Bias in RAG Systems
- Unlocking Data Insights: A Framework for Transparent and Evaluable AI Agents
Lessons Learned and Future Directions
The experience with TVS Sidekick offers valuable practical insights into deploying LLMs in a real-world business context. It highlights the importance of an in-house AI team for agility, the effectiveness of RAG for enterprise data, and the necessity of a robust AI Management System to navigate regulatory and ethical considerations. The continuous monitoring and feedback loops are crucial for addressing challenges and driving the ongoing development of AI solutions that truly meet organizational needs while adhering to responsible AI principles.
