The Value of Extended Reality Techniques to Improve Remote Collaborative Maintenance Operations: A User Study Corentin Coupry LARIS, France Paul Richard LARIS, France David Bigaud LARIS, France Sylvain Noblecourt LINEACT, France David Baudry LINEACT, France This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.03

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

In the Architecture, Engineering and Construction (AEC) sector, data extracted from building information modelling (BIM) can be used to create a digital twin (DT). The algorithms of a BIM-based DT can facilitate the retrieval of information, which can then be used to improve building operation and maintenance procedures. However, with the increased complexity and automation of the building, maintenance operations are likely to become more complex and may require expert intervention. Collaboration and interaction between the operator and the expert may be limited as the latter may not be on site or within the company. Recently, extended reality (XR) technologies have proven to be effective in improving collaboration during maintenance operations, through data display and shared interactions. This paper presents a new collaborative solution using these technologies to enhance collaboration during remote maintenance operations. The proposed approach consists of a mixed reality (MR) set-up for the operator, a virtual reality (VR) set-up for the remote expert and a shared Digital Model of a heat exchanger. The MR set-up is used for tracking and displaying specific information, provided by the VR module. A user study was carried out to compare the efficiency of our solution with a standard audio-video collaboration. Our approach demonstrated substantial enhancements in collaborative inspection, resulting in a significative reduction in both the overall completion time of the inspection and the frequency of errors committed by the operators

 Virtual Reality; Mixed Reality; Operation & Maintenance; Collaboration; Digital Twin

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.03

References Anton, D., Kurillo, G., & Bajcsy, R. (2018). User experience and interaction performance in 2D/3D telecollaboration. Future Generation Computer Systems, 82, 77–88. 10.1016/j.future.2017.12.055 Bai, H., Sasikumar, P., Yang, J., & Billinghurst, M. (2020). A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13. 10.1145/3313831.3376550 Chenechal, M. L., Duval, T., Gouranton, V., Royan, J., & Arnaldi, B. (2016). Vishnu: Virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality. 2016 IEEE Third VR International Workshop on Collaborative Virtual Environments (3DCVE), 9–12. 10.1109/3DCVE.2016.7563559 Coupry, C., Noblecourt, S., Richard, P., Baudry, D., & Bigaud, D. (2021). BIM-Based Digital Twin and XR Devices to Improve Maintenance Procedures in Smart Buildings: A Literature Review. Applied Sciences, 11(15), 6810. 10.3390/app11156810 Fakourfar, O., Ta, K., Tang, R., Bateman, S., & Tang, A. (2016). Stabilized Annotations for Mobile Remote Assistance. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 1548–1560. 10.1145/2858036.2858171 Gao, L., Bai, H., Lee, G., & Billinghurst, M. (2016). An oriented point-cloud view for MR remote collaboration. SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications, 1–4. 10.1145/2999508.2999531 Gartner Top 10 Strategic Technology Trends for 2023. (2023). Gartner. https://www.gartner.com/en/articles/gartner-top-10-strategic-technology-trends-for-2023 Grandi, J. G., Debarba, H. G., & Maciel, A. (2019). Characterizing Asymmetric Collaborative Interactions in Virtual and Augmented Realities. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 127–135. 10.1109/VR.2019.8798080 Jamwal, A., Agrawal, R., Sharma, M., & Giallanza, A. (2021). Industry 4.0 Technologies for Manufacturing Sustainability: A Systematic Review and Future Research Directions. Applied Sciences, 11(12), 5725. 10.3390/app11125725 Karl-Bridge-Microsoft. (2023, June 2). Windows Device Portal overview—UWP applications. https://learn.microsoft.com/en-us/windows/uwp/debug-test-perf/device-portal Kolkmeier, J., Harmsen, E., Giesselink, S., Reidsma, D., Theune, M., & Heylen, D. (2018). With a little help from a holographic friend: The OpenIMPRESS mixed reality telepresence toolkit for remote collaboration systems. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, 1–11. 10.1145/3281505.3281542 Ladwig, P. (2019). Remote guidance for machine maintenance supported by physical LEDs and virtual reality. ACM International Conference Proceeding Series, Query date: 2022-06-03 15:39:34, 255–262. 10.1145/3340764.3340780 Microsoft. (n.d.-a). Microsoft HoloLens. Retrieved April 22, 2021, from https://www.microsoft.com/en-us/hololens Microsoft. (n.d.-b). Microsoft Teams. Retrieved June 28, 2023, from https://www.microsoft.com/en-us/microsoft-teams/group-chat-software Microsoft. (2022, October 4). QR code tracking overview—Mixed Reality. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/qr-code-tracking-overview Oda, O., Elvezio, C., Sukan, M., Feiner, S., & Tversky, B. (2015). Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 405–415. 10.1145/2807442.2807497 OpenXR. (2016, December 6). The Khronos Group. https://www.khronos.org/openxr/ Photon Unity Networking. (n.d.). Retrieved June 22, 2023, from https://www.photonengine.com/pun/ Pierce, J. S., Stearns, B. C., & Pausch, R. (1999). Voodoo dolls: Seamless interaction at multiple scales in virtual environments. Proceedings of the 1999 Symposium on Interactive 3D Graphics - SI3D ’99, 141–145. 10.1145/300523.300540 Piumsomboon, T., Day, A., Ens, B., Lee, Y., Lee, G., & Billinghurst, M. (2017). Exploring enhancements for remote mixed reality collaboration. SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA ’17, 1–5. 10.1145/3132787.3139200 Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The Effects of Sharing Awareness Cues in Collaborative Mixed Reality. Frontiers in Robotics and AI, 6, 5. 10.3389/frobt.2019.00005 Piumsomboon, T., Lee, G. A., Irlitti, A., Ens, B., Thomas, B. H., & Billinghurst, M. (2019). On the Shoulder of the Giant: A Multi-Scale Mixed Reality Collaboration with 360 Video Sharing and Tangible Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–17. 10.1145/3290605.3300458 Serubugo, S. (2018). Self-overlapping maze and map design for asymmetric collaboration in room-scale virtual reality for public spaces. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, 229(Query date: 2022-06-02 15:15:28), 194–203. 10.1007/978-3-319-76908-0_19 Teo, T., Lawrence, L., Lee, G. A., Billinghurst, M., & Adcock, M. (2019). Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–14. 10.1145/3290605.3300431 Wang, P., Wang, Y., Billinghurst, M., Yang, H., Xu, P., & Li, Y. (2023). BeHere: A VR/SAR remote collaboration system based on virtual replicas sharing gesture and avatar in a procedural task. Virtual Reality, 27(2), 1409–1430. 10.1007/s10055-023-00748-5 Wang, P., Zhang, S., Bai, X., Billinghurst, M., He, W., Sun, M., Chen, Y., Lv, H., & Ji, H. (2019). 2.5DHANDS: A gesture-based MR remote collaborative platform. The International Journal of Advanced Manufacturing Technology, 102(5–8), 1339–1353. 10.1007/s00170-018-03237-1 WebRTC. (n.d.). WebRTC. Retrieved May 10, 2022, from https://webrtc.org/?hl=fr