XR and AI based Human in the Loop Assistance for Industry X.0 and above

Abstract

The current generation of Industry 4.0 integrates automation and data in manufacturing, while the upcoming generation, Industry 5.0, aims to enable seamless interaction between humans, robots, and machines for sustainable and personalized product and service creation. This requires manufacturing components according to customer wishes, leading to frequent changes in the production line. In this context, automation proves unsustainable and could be addressed through a human-in-the-loop system. The effectiveness of the human-in-the-loop system depends on how well the information transfers to them. The recent evaluation of eXtended Reality (XR) technology allows a new way of information exchange. In this thesis, we focus on developing an XR-based human-in-the-loop system for various complex industry tasks. We introduce a new way of information rendering using deep learning and machine learning algorithms. Additionally, we propose an effective method of data generation to train deep learning models, which reduces the time and manpower required to develop the dataset. The proposed method of data generation improves the model's performance. We develop the Mixed Reality (MR) environment for manual assembly guidance and propose a machine learning and deep learning-based markerless instruction rendering method. We analyze the MR-based manual assembly guidance with simple and complex assembly tasks. The assembly instructions are given to the user through a multimodal user interface that provides instructions based on the user's hand and eye position. The proposed multimodal user interface effectively guides the user to complete the assembly task. However, some assembly tasks require human-robot cooperation due to human limitations. When humans and robots work cooperatively, the effectiveness of task completion depends on information exchange. Therefore, we develop an XR-based multimodal user interaction to improve Human-Robot Interaction (HRI) with haptic feedback, which assists users in remotely performing tasks. The haptic feedback, generated using the potential field method, aids users in controlling robots without collisions. We introduce a multimodal XR-based framework for effective communication between humans and robots, utilizing vision models to provide information about the remote side to local users. Seamless communication between humans and robots is facilitated through multiple modalities such as haptic, speech, visual, and text. The system's effectiveness in pick-and-place activities and adaptation for complex industry use cases such as peg in the hole and welding is further evaluated. The results demonstrate the system's efficacy in two specific human-robot collaborative scenarios in industries: arc welding and assembly tasks. The XR-based multimodal instruction method effectively guides the user in performing manual assembly tasks and enables a new way of human-robot cooperation.