AI-Powered AR Assistant: Empowering Inclusive Customer Service

This work represents ongoing research in assistive technology, applications of large language models (LLMs), and augmented reality (AR) interfaces for accessibility in customer service scenarios. Visitors to our exhibit will experience integrating AR and LLM-driven assistant technologies designed to enhance customer service interactions, especially by supporting neurodivergent service representatives. Upon approaching our booth, visitors can participate in an interactive demonstration. They'll take on the role of either a customer or a service representative. The service representative will wear our AR glasses while engaging in a simulated customer service scenario. Our system captures customer speech through microphones and converts speech to text in real-time. Based on the input, the system generates clear, simple guidance through the LLM API that appears in the AR glasses display, helping the service representative navigate the interaction effectively. Visitors will observe how the AR glasses provide assistance without disrupting the natural flow of conversation. The system is designed to identify the speech and adjust its guidance accordingly. When faced with complex customer inquiries, the system presents straightforward, step-by-step guidance in the display. We'll have a demonstration station where visitors can: Try on the AR glasses and experience the guidance system in different conversation scenarios View how the system differentiates between speakers Examine a monitor showing the processing workflow Our exhibit highlights how technology can create more inclusive workplaces by providing customized support for service representatives with different needs. We'll demonstrate how the system can be adjusted to accommodate various cognitive abilities, ensuring clear, accessible guidance. For technically-minded visitors, we'll explain the components of our research prototype, including the web server architecture, voice detection algorithms, speaker identification methods, and integration with LLMs. This project represents our research into how AR technology might serve as an assistive tool for individuals preparing for customer-facing roles. Visitors can learn how the AI-powered AR system can create a supportive learning environment where trainees receive aid from real-time guidance and support. The system helps users develop customer service skills in a structured, confidence-building environment.