VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications
The advent of immersive Virtual Reality applications has transformed various domains, yet their integration with advanced artificial intelligence technologies like Visual Language Models remains underexplored. This study introduces a pioneering approach utilizing VLMs within VR environments to enhance user interaction and task efficiency. Leveraging the Unity engine and a custom-developed VLM, our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions. The incorporation of speech-to-text and text-to-speech technologies allows for seamless communication between the user and the VLM, enabling the system to guide users through complex tasks effectively. Preliminary experimental results indicate that utilizing VLMs not only reduces task completion times but also improves user comfort and task engagement compared to traditional VR interaction methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSpeech-to-Texttext-to-speechText to SpeechUnitySimilar Papers 제목 키워드 기반
Text2VR: Automated instruction Generation in Virtual Reality using Large language Models for Assembly Task
Virtual Reality (VR) has emerged as a powerful tool for workforce training, offering immersive, interactive, and risk-free environments that enhance skill acquisition, decision-making, and confidence. Despite its advanta…
Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality
Selection of occluded objects is a challenging problem in virtual reality, even more so if multiple objects are involved. With the advent of new artificial intelligence technologies, we explore the possibility of leverag…
Language ModelingLanguage ModellingLarge Language ModelArtificial Intelligence-based algorithms in medical image scan seg-mentation and intelligent visual-content generation -- a concise overview
Recently, Artificial Intelligence (AI)-based algorithms have revolutionized the medical image segmentation processes. Thus, the precise segmentation of organs and their lesions may contribute to an efficient diagnostics …
Image SegmentationMedical Image SegmentationSegmentationSemantic SegmentationTowards Everyday Virtual Reality through Eye Tracking
With developments in computer graphics, hardware technology, perception engineering, and human-computer interaction, virtual reality and virtual environments are becoming more integrated into our daily lives. Head-mounte…
Privacy PreservingA virtual reality-based method for examining audiovisual prosody perception
Prosody plays a vital role in verbal communication. Acoustic cues of prosody have been examined extensively. However, prosodic characteristics are not only perceived auditorily, but also visually based on head and facial…