Enhancing metal additive manufacturing training with the advanced vision language model: A pathway to immersive augmented reality training for non-experts
Author links open overlay panel

https://doi.org/10.1016/j.jmsy.2024.0...
Get rights and content Under a Creative Commons license open access
Highlights
• Developed an AR training system for metal AM, integrating VLM, YOLO, and CRAFT in a DT framework.
• Provides immersive training for metal AM, overcoming traditional limitations.
• Demonstrates effective use of VLM for multimodal data processing.
• Shows AR and VLM synergy in training, offering interactive environments and expert guidance.
• Opens broader avenues in smart manufacturing beyond metal AM.

Abstract
This paper introduces an innovative
training system for the Renishaw AM400 metal printer, leveraging the synergy of the advanced Vision Language Model (VLM) with Augmented Reality (AR) within the Digital Twins (DT) framework. Aimed at overcoming the limitations of conventional training methods in metal additive manufacturing (AM), our system integrates AR to provide an immersive learning environment, enhancing the real-world experience with interactive digital overlays. The core of the system lies in its use of VLM, which, pre-trained on diverse datasets, excels in processing multi-modal data, thereby offering nuanced and contextually relevant guidance for trainees. Key experiments demonstrate the system’s effectiveness, particularly highlighting the usage of VLM as an Artificial Intelligence (AI) agent to integrate external tools like YOLO-v7 for valve state classification and CRAFT for control panel text recognition. This approach significantly improves recognition accuracy, operational understanding, and human–machine interaction, especially for non-expert users, making complex metal AM operations more accessible. The research not only showcases the potential of AR and VLM in industrial training but also sets a new standard for smart manufacturing practices, indicating broader applications in various industrial domains.
=========
Extracted from https://www.sciencedirect.com/science...