Seamless Indoor/Outdoor Marker-Less Augmented Reality Registration Supporting Facility Management Operations Leonardo Messi Università Politecnica delle Marche, Italy Francesco Spegni Università Politecnica delle Marche, Italy Massimo Vaccarini Università Politecnica delle Marche, Italy Alessandra Corneli Università Politecnica delle Marche, Italy Leonardo Binni Università Politecnica delle Marche, Italy This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.11

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

Augmented reality (AR) still struggles to be widely used in real processes in the construction industry despite its great potential. This is partly due to the difficulties that exist in aligning holograms and maintaining their stability, especially for outdoor applications. In addition, being indoor-outdoor interactions crucial for built environment management, it would be important that AR apps can work seamlessly. Alignment in indoor environments cannot make use of methods such as GNSS, nor can all environments be assumed to have been previously initialized with AR tools. Thus, marker-less AR registration is crucial for indoor applications. This paper presents an approach for marker-less AR registration seamlessly in both outdoor and indoor environments. Real-time kinematic positioning (RTK) and Inertial Measurement Units (IMU) technologies have been chosen for outdoor registration, while image comparison based on convolutional neural networks (CNN) for indoor registration. In this research, the application of these two technologies and their integration have been studied and tested on site on a real Facility Management use case related to a university campus. The proposed approach has shown very promising results in displaying BIM elements of the electrical system seamlessly superimposed through AR to their physical counterparts in mixed indoor-outdoor environments

 Augmented Reality Seamless Registration Feature Matching Pose Estimation Real-Time Kinematic Facility Management

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.11

References Ahmad, A., Claudio, P., Alizadeh Naeini, A., & Sohn, G. (2020). Wi-fi rss fingerprinting for indoor localization using augmented reality. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 5(4), 57–64. 10.5194/isprs-Annals-V-4-2020-57-2020 Albahbah, M., Kıvrak, S., & Arslan, G. (2021). Application areas of augmented reality and virtual reality in construction project management: A scoping review. Journal of Construction Engineering, 4, 151–172. 10.31462/jcemi.2021.03151172 Ashour, Z., Shaghaghian, Z., & Yan, W. (2022). BIMxAR: BIM-Empowered Augmented Reality for Learning Architectural Representations. 10.48550/arXiv.2204.03207 Baek, F., Ha, I., & Kim, H. (2019). Augmented reality system for facility management using image-based indoor localization. Automation in Construction, 99, 18–26. 10.1016/j.autcon.2018.11.034 Cao, B., Araujo, A., & Sim, J. (2020). Unifying Deep Local and Global Features for Image Search. http://arxiv.org/abs/2001.05027 Carbonari, A., Franco, C., Naticchia, B., Spegni, F., & Vaccarini, M. (2022). A Mixed Reality Application for the On-Site Assessment of Building Renovation: Development and Testing. Sustainability (Switzerland), 14(20). 10.3390/su142013239 Chen, K., Chen, W., Fellow, P., Li, C. T., Student, M., & Cheng, J. C. P. (2019). A BIM-based location aware AR collaborative framework for facility maintenance management. In Journal of Information Technology in Construction (ITcon) (Vol. 24). http://www.itcon.org/2019/19 Cheng, J. C. P., Chen, K., & Chen, W. (2020). State-of-the-Art Review on Mixed Reality Applications in the AECO Industry. Journal of Construction Engineering and Management, 146(2). 10.1061/(asce)co.1943-7862.0001749 Costanza, E., Kunz, A., & Fjeld, M. (2009). Mixed Reality: A Survey. Lecture Notes in Computer Science, 5440. 10.1007/978-3-642-00437-7_3 Cyrus, J., Krcmarik, D., Moezzi, R., Koci, J., & Petru, M. (2019). Hololens used for precise position tracking of the third party devices - Autonomous vehicles. Communications - Scientific Letters of the University of Žilina, 21(2), 18–23. 10.26552/com.c.2019.2.18-23 DeTone, D., Malisiewicz, T., & Rabinovich, A. (2017). SuperPoint: Self-Supervised Interest Point Detection and Description. http://arxiv.org/abs/1712.07629 El Barhoumi, N., Hajji, R., Bouali, Z., Ben Brahim, Y., & Kharroubi, A. (2022). Assessment of 3D Models Placement Methods in Augmented Reality. Applied Sciences (Switzerland), 12(20). 10.3390/app122010620 Ethan Rublee, Vincent Rabaud, Kurt Konolige, & Gary Bradski. (2011). ORB: An efficient alternative to SIFT or SURF. 2011 International Conference on Computer Vision. 10.1109/ICCV.2011.6126544 Google LLC. (2023). ARCore. https://developers.google.com/ar?hl=it Guarese, R. L. M., & Maciel, A. (2019). Development and Usability Analysis of a Mixed Reality GPS Navigation Application for the Microsoft HoloLens. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11542 LNCS, 431–437. 10.1007/978-3-030-22514-8_41 Hansen, L. H., Fleck, P., Stranner, M., Schmalstieg, D., & Arth, C. (2021). Augmented Reality for Subsurface Utility Engineering, Revisited. IEEE Transactions on Visualization and Computer Graphics, 27(11), 4119–4128. 10.1109/TVCG.2021.3106479 He, Z., Xia, Z., Chang, Y., Chen, W., Hu, J., & Wei, X. (2006). Research on underground pipeline augmented reality system based on ARToolKit. In H. Wu & Q. Zhu (Eds.), Proc. SPIE 6421, Geoinformatics 2006: Geospatial Information Technology (p. 642112). 10.1117/12.713123 Humenberger, M., Cabon, Y., Guerin, N., Morat, J., Leroy, V., Revaud, J., Rerole, P., Pion, N., de Souza, C., & Csurka, G. (2020). Robust Image Retrieval-based Visual Localization using Kapture. http://arxiv.org/abs/2007.13867 Jurado, D., Jurado, J. M., Ortega, L., & Feito, F. R. (2021). Geuinf: Real-time visualization of indoor facilities using mixed reality. Sensors (Switzerland), 21(4), 1–21. 10.3390/s21041123 Kaizu, Y., & Choi, J. (2012). Development of a tractor navigation system using augmented reality. Engineering in Agriculture, Environment and Food, 5(3), 96–101. 10.1016/S1881-8366(12)80021-8 Kim, C., Park, T., Lim, H., & Kim, H. (2013). On-site construction management using mobile computing technology. Automation in Construction, 35, 415–423. 10.1016/j.autcon.2013.05.027 Kneip, L., Scaramuzza, D., & Siegwart, R. (2011). A Novel Parametrization of the Perspective-Three-Point Problem for a Direct Computation of Absolute Camera Position and Orientation. 10.1109/CVPR.2011.5995464 Koch, C., Neges, M., König, M., & Abramovici, M. (2014). Natural markers for augmented reality-based indoor navigation and facility maintenance. Automation in Construction, 48, 18–30. 10.1016/j.autcon.2014.08.009 Kuo, C., Jeng, T., & Yang, I. (2013). An invisible head marker tracking system for indoor mobile augmented reality. Automation in Construction, 33, 104–115. 10.1016/j.autcon.2012.09.011 Lee, S., & Akin, Ö. (2011). Augmented reality-based computational fieldwork support for equipment operations and maintenance. Automation in Construction, 20(4), 338–352. 10.1016/j.autcon.2010.11.004 Li, Y., Snavely, N., Huttenlocher, D., & Fua, P. (2012). Worldwide Pose Estimation using 3D Point Clouds. 10.1007/978-3-642-33718-5_2 Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023). LightGlue: Local Feature Matching at Light Speed. http://arxiv.org/abs/2306.13643 Ling, F. F., Elvezio, C., Bullock, J., Henderson, S., & Feiner, S. (2019). A Hybrid RTK GNSS and SLAM Outdoor Augmented Reality System. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 1044–1045. 10.1109/VR.2019.8798315 Marchand, E., Uchiyama, H., & Spindler, F. (2016). Pose estimation for augmented reality: a hands-on survey. 10.1109/TVCG.2015.2513408 Microsoft. (2023). World Locking Tools documentation. https://learn.microsoft.com/en-us/mixed-reality/world-locking-tools/ Naticchia, B., Vaccarini, M., Corneli, A., Messi, L., & Carbonari, A. (2021). Leveraging Extended Reality technologies with RFID to enhance on field maintenance of buildings. http://itc.scix.net/paper/w78-2021-paper-038 Park, C. S., Lee, D. Y., Kwon, O. S., & Wang, X. (2013). A framework for proactive construction defect management using BIM, augmented reality and ontology-based data collection template. Automation in Construction, 33, 61–71. 10.1016/j.autcon.2012.09.010 PTC Products. (2023). Vuforia Enterprise AR Software. https://www.ptc.com/en/products/vuforia Roberts, G. W., Evans, A., Dodson, A. H., Denby, B., Cooper, S., & Hollands, R. (2002). The Use of Augmented Reality, GPS and INS for Subsurface Data Visualisation. https://doi.org/https://api.semanticscholar.org/CorpusID:112931894 Salman, A., & Ahmad, W. (2023). Implementation of augmented reality and mixed reality applications for smart facilities management: a systematic review. Smart and Sustainable Built Environment. 10.1108/SASBE-11-2022-0254 Sarlin, P.-E., Cadena, C., Siegwart, R., & Dymczyk, M. (2018). From Coarse to Fine: Robust Hierarchical Localization at Large Scale. http://arxiv.org/abs/1812.03506 Sarlin, P.-E., Debraine, F., Dymczyk, M., Siegwart, R., & Cadena, C. (2018). Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization. http://arxiv.org/abs/1809.01019 Schönberger, J. L., & Frahm, J.-M. (2016). Structure-from-Motion Revisited. https://github.com/colmap/colmap. Teruggi, S., & Fassi, F. (2022). Mixed Reality Content Alignment in Monumental Environments. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 43(B2-2022), 901–908. 10.5194/isprs-archives-XLIII-B2-2022-901-2022 Trimble Inc. (2023). Site Vision. https://sitevision.trimble.com/ Vaccarini, M., Carbonari, A., Spegni, F., & Giretti, A. (2022). Enhancing BIM through Mixed Reality for Facility Management. In Building Information Modeling - A Sustainable Approach and Emerging Technologies [Working Title]. IntechOpen. 10.5772/intechopen.106186 vGIS Inc. (2023). Engineering-grade AR for AEC. https://www.vgis.io/augmented-reality-bim-gis-ar-aec-civil-construction-engineering-bentley-autodesk-esri/ Zhao, S., Chen, Y., & Farrell, J. A. (2016). High-Precision Vehicle Navigation in Urban Environments Using an MEM’s IMU and Single-Frequency GPS Receiver. IEEE Transactions on Intelligent Transportation Systems, 17(10), 2854–2867. 10.1109/TITS.2016.2529000