Integrating Real-Time Object Detection into an AR-Driven Task Assistance Prototype: An Approach Towards Reducing Specific Motions in Therbligs Theory Xiang Yuan University of Alberta, Canada Qipei Mei University of Alberta, Canada Xinming Li University of Alberta, Canada This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.12

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

Due to challenges in filling vacant positions and the heightened demands posed on existing staff, employers and project managers are progressively considering the recruitment of inexperienced individuals and seeking strategies to swiftly provide them with essential job-specific knowledge. The potential of industrial AR has been widely researched to support workers in overcoming skill-related knowledge and enhancing industrial processes. However, most studies focus on demonstrating technology usability across different processes and overcoming engineering hurdles on a case-by-case basis. There is no direct benefit analysis on how AR assists construction tasks at human motion level, and how to eliminate the ineffective motions and reduce the duration of effective motions. To fill this gap, this paper first establishes an AR-based near real-time object detection system of small tools and components involved in task processes for egocentric perception of workers in the construction industry. Later, the Standard Operating Procedure (SOP) for scaffolding assembly activities is deconstructed from a manual process into Therbligs-based elemental motions. Finally, this research conducted a comparative study of two prototypes across four dimensions of evaluation. As a step forward in this direction, this paper renews the connotations of Therbligs theory under industry 5.0 era, rethinks the AR-assisted construction task processes, and applies appropriate technologies enhancing the adaptability of AR technology for construction workers’ needs

 Augmented Reality (AR); Microsoft HoloLens 2; Object Detection; Task Assistance; Therbligs

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.12

References Butaslac, I. M., Fujimoto, Y., Sawabe, T., Kanbara, M., & Kato, H. (2022). Systematic Review of Augmented Reality Training Systems. IEEE Transactions on Visualization and Computer Graphics, 1–20. 10.1109/TVCG.2022.3201120 Büttner, S., Prilla, M., & Röcker, C. (2020). Augmented Reality Training for Industrial Assembly Work—Are Projection-based AR Assistive Systems an Appropriate Tool for Assembly Training? Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12. Honolulu HI USA: ACM. 10.1145/3313831.3376720 Casano, D. (2021). HoloHelp: HoloLens Detection for a Guided Interaction. University of Catania. David, D. (2000) Therbligs: The Keys to Simplifying Work, The Gilbreth Network: Therbligs. Available at: https://gilbrethnetwork.tripod.com/therbligs.html (Accessed: 26 July 2023). de Souza Cardoso, L. F., Mariano, F. C. M. Q., & Zorzal, E. R. (2020). A survey of industrial augmented reality. Computers & Industrial Engineering, 139, 106159. 10.1016/j.cie.2019.106159 Eswaran, M., & Bahubalendruni, M. V. A. R. (2022). Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: A state of the art review. Journal of Manufacturing Systems, 65, 260–278. (36). 10.1016/j.jmsy.2022.09.016 Farasin, A., Peciarolo, F., Grangetto, M., Gianaria, E., & Garza, P. (2020). Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens: Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 165–172. Valletta, Malta: SCITEPRESS - Science and Technology Publications. 10.5220/0008877901650172 Fuglseth, S. S. (2022). Object Detection with HoloLens 2 using Mixed Reality and Unity a proof-of-concept (Bachelor thesis, Høgskolen i Molde - Vitenskapelig høgskole i logistikk). Høgskolen i Molde - Vitenskapelig høgskole i logistikk. Retrieved from https://himolde.brage.unit.no/himolde-xmlui/handle/11250/3023916 George, R. (2021). Using Object Recognition on Hololens 2 for Assembly (M.S.). Retrieved from https://www.proquest.com/docview/2621280160/abstract/34D5D9DBA2ED4869PQ/1 Ghasemi, Y., Jeong, H., Choi, S. H., Park, K.-B., & Lee, J. Y. (2022). Deep learning-based object detection in augmented reality: A systematic review. Computers in Industry, 139, 103661. 10.1016/j.compind.2022.103661 Government of Canada, S. C. (2022). Retrieved from https://www.statcan.gc.ca/en/subjects-start/labour_/labour-shortage-trends-canada#shr-pg0 Grubert, J., Hamacher, D., Mecke, R., Böckelmann, I., Schega, L., Huckauf, A., … Tümler, J. (2010). Extended investigations of user-related issues in mobile industrial AR. 2010 IEEE International Symposium on Mixed and Augmented Reality, 229–230. 10.1109/ISMAR.2010.5643581 Ke, Y. (2018). Research on the Chinese Industrialized Construction Migrant Workers from the Perspective of Complex Adaptive System: Combining the Application of SWARM Computer Simulation Technology. Wireless Personal Communications, 102(4), 2469–2481. 10.1007/s11277-018-5266-8 Khan, N., Saleem, M. R., Lee, D., Park, M.-W., & Park, C. (2021). Utilizing safety rule correlation for mobile scaffolds monitoring leveraging deep convolution neural networks. Computers in Industry, 129. Scopus. 10.1016/j.compind.2021.103448 Kim, J., Olsen, D., & Renfroe, J. (2022). Construction Workforce Training Assisted with Augmented Reality. 2022 8th International Conference of the Immersive Learning Research Network (iLRN), 1–6. 10.23919/iLRN55037.2022.9815960 Lee, K., Jeon, C., & Shin, D. H. (2023). Small Tool Image Database and Object Detection Approach for Indoor Construction Site Safety. KSCE Journal of Civil Engineering, 27(3), 930–939. 10.1007/s12205-023-1011-2 Liao, W., Iseley, T., & Behbahani, S. (2022). Industry/University Cooperative Research Centers (IUCRC): A Critical Component for Addressing Underground Infrastructure Challenges. 56–66. 10.1061/9780784484289.007 Łysakowski, M., Żywanowski, K., Banaszczyk, A., Nowicki, M. R., Skrzypczyński, P., & Tadeja, S. K. (2023, June 6). Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8. arXiv. Retrieved from http://arxiv.org/abs/2306.03537 Niebel, B., & Freivalds, A. (2013). Niebel’s Methods, Standards, & Work Design. McGraw-Hill Education. Ninjatacoshell. (2012). English: The 18 therbligs. Own work. Retrieved from https://commons.wikimedia.org/wiki/File:Therblig_(English).svg Oyekan, J., Hutabarat, W., Turner, C., Arnoult, C., & Tiwari, A. (2020). Using Therbligs to embed intelligence in workpieces for digital assistive assembly. Journal of Ambient Intelligence and Humanized Computing, 11(6), 2489–2503. 10.1007/s12652-019-01294-2 PatrickFarley. (2023, July 18). What is Custom Vision? - Azure AI services. Retrieved October 10, 2023, from https://learn.microsoft.com/en-us/azure/ai-services/custom-vision-service/overview Peñaloza, G. A., Saurin, T. A., & Formoso, C. T. (2020). Monitoring complexity and resilience in construction projects: The contribution of safety performance measurement systems. Applied Ergonomics, 82, 102978. 10.1016/j.apergo.2019.102978 Qin, Y., Wang, S., Zhang, Q., Cheng, Y., Huang, J., & He, W. (2023). Assembly training system on HoloLens using embedded algorithm. Third International Symposium on Computer Engineering and Intelligent Communications (ISCEIC 2022), 12462, 121–128. SPIE. 10.1117/12.2660940 Sung, R. C. W., Ritchie, J. M., Lim, T., & Medellin, H. (2009). Assembly planning and motion study using virtual reality. 31–38. Scopus. 10.1115/WINVR2009-713 Tao, W., Lai, Z.-H., Leu, M. C., Yin, Z., & Qin, R. (2019). A self-aware and active-guiding training & assistant system for worker-centered intelligent manufacturing. Manufacturing Letters, 21, 45–49. 10.1016/j.mfglet.2019.08.003 The Home Depot. (2022). Retrieved from https://www.homedepot.ca/product/metaltech-scaffold-bench-multipurpose-4-in-1-6-ft-baker-scaffold/1001160246 Trinh, M. T., & Feng, Y. (2020). Impact of Project Complexity on Construction Safety Performance: Moderating Role of Resilient Safety Culture. Journal of Construction Engineering and Management, 146(2), 04019103. 10.1061/(ASCE)CO.1943-7862.0001758 Ungureanu, D., Bogo, F., Galliani, S., Sama, P., Duan, X., Meekhof, C., … Pollefeys, M. (2020, August 25). HoloLens 2 Research Mode as a Tool for Computer Vision Research. arXiv. 10.48550/arXiv.2008.11239 Wang, B., Zhang, Z., Jiang, C., Zhao, Y., Ding, S., Xu, F., & Niu, J. (2021). A Novel Approach Combined with Therbligs and VACP Model to Evaluate the Workload During Simulated Maintenance Task. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12771 LNCS, 164–173. Scopus. 10.1007/978-3-030-77074-7_13 Wolf, J., Wolfer, V., Halbe, M., Maisano, F., Lohmeyer, Q., & Meboldt, M. (2021). Comparing the effectiveness of augmented reality-based and conventional instructions during single ECMO cannulation training. International Journal of Computer Assisted Radiology and Surgery, 16(7), 1171–1180. 10.1007/s11548-021-02408-y Wu, S., Hou, L., & Chen, H. (n.d.). Measuring the impact of Augmented Reality warning systems on onsite construction workers using object detection and eye-tracking. Wu, S., Hou, L., Zhang, G. (Kevin), & Chen, H. (2022). Real-time mixed reality-based visual warning for construction workforce safety. Automation in Construction, 139, 104252. 10.1016/j.autcon.2022.104252 Wu, Z., Zhao, T., & Nguyen, C. (2020). 3D Reconstruction and Object Detection for HoloLens. 2020 Digital Image Computing: Techniques and Applications (DICTA), 1–2. 10.1109/DICTA51227.2020.9363378 Zhang, Y., Xuan, Y., Yadav, R., Omrani, A., & Fjeld, M. (2023, February 3). Playing with Data: An Augmented Reality Approach to Interact with Visualizations of Industrial Process Tomography. arXiv. 10.48550/arXiv.2302.01686