Semi-Automated Visual Quality Control Inspection During Construction or Renovation of Railways Using Deep Learning Techniques and Augmented Reality Visualization Apostolia Gounaridou Centre for Research and Technology Hellas, Greece Evangelia Pantraki Centre for Research and Technology Hellas, Greece Vasileios Dimitriadis Centre for Research and Technology Hellas, Greece Athanasios Tsakiris Centre for Research and Technology Hellas, Greece Dimosthenis Ioannidis Centre for Research and Technology Hellas, Greece Dimitrios Tzovaras Centre for Research and Technology Hellas, Greece This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.86

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

The construction industry stands to greatly benefit from the technological advancements in deep learning and computer vision, which can automate time-consuming tasks such as quality control. In this paper, we introduce a framework that incorporates two advanced tools - the Visual Quality Control (VQC) tool and the Digital Twin visualization with Augmented Reality (DigiTAR) tool - to perform semi-automated visual quality control in the construction site during the execution phase of the project. The VQC tool is a backend service that detects potential defects on images captured on-site using the Mask R-CNN algorithm trained on annotated images of concrete and railway defects. The surveyor, aided by the Augmented Reality (AR) technology through the DigiTAR tool, can in-situ confirm/reject the detected defects and propose remedial actions. All the quality control results are recorded in the relevant BIM model and can be viewed on-site overlaid on the physical construction elements. This solution offers a semi-automated visual inspection that can speed up and simplify the quality control process, especially in case of large linear infrastructures, illustrating the added value of AR-based applications in Digital Twins

 BIM Augmented Reality AR in Construction Deep Learning Computer Vision Visual Inspection Digital Twins

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.86

References Abdulla. (2017). Mask R-CNN for object detection and segmentation using TensorFlow 2.0. In GitHub repository. https://github.com/ahmedfgad/Mask-RCNN-TF2 Atha, D. J., & Jahanshahi, M. R. (2018). Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection. Structural Health Monitoring, 17(5), 1110–1128. 10.1177/1475921717737051 Attard, L., Debono, C. J., Valentino, G., Castro, M. Di, Masi, A., & Scibile, L. (2019). Automatic crack detection using Mask R-CNN. 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), 152–157. 10.1109/ISPA.2019.8868619 Brien, D. O., Osborne, J. A., Perez-Duenas, E., Cunningham, R., & Li, Z. (2023). Automated crack classification for the CERN underground tunnel infrastructure using deep learning. Tunnelling and Underground Space Technology, 131. 10.1016/j.tust.2022.104668 Cao, X., Xie, W., Ahmed, S. M., & Li, C. R. (2020). Defect detection method for rail surface based on line-structured light. Measurement: Journal of the International Measurement Confederation, 159. 10.1016/j.measurement.2020.107771 Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk, O. (2018). Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering, 33(9), 731–747. 10.1111/mice.12334 Chi, H. L., Kim, M. K., Liu, K. Z., Thedja, J. P. P., Seo, J., & Lee, D. E. (2022). Rebar inspection integrating augmented reality and laser scanning. Automation in Construction, 136. 10.1016/j.autcon.2022.104183 COGITO project. (n.d.). Retrieved September 29, 2023, from https://cogito-project.eu/ Concrete Crack Segmentation Dataset. (n.d.). Retrieved September 29, 2023, from https://www.kaggle.com/datasets/motono0223/concrete-crack-segmentation-dataset Crack Segmentation Dataset. (n.d.). Retrieved September 29, 2023, from https://www.kaggle.com/datasets/lakshaymiddha/crack-segmentation-dataset?select=crack_segmentation_dataset Gan, J., Li, Q., Wang, J., & Yu, H. (2017). A hierarchical extractor-based visual rail surface inspection system. IEEE Sensors Journal, 17(23), 7935–7944. 10.1109/JSEN.2017.2761858 García-Pereira, I., Portalés, C., Gimeno, J., & Casas, S. (2020). A collaborative augmented reality annotation tool for the inspection of prefabricated buildings. Multimedia Tools and Applications, 79(9–10), 6483–6501. 10.1007/s11042-019-08419-x Guo, F., Qian, Y., Rizos, D., Suo, Z., & Chen, X. (2021). Automatic rail surface defects inspection based on mask r-cnn. In Transportation Research Record (Vol. 2675, Issue 11, pp. 655–668). SAGE Publications Ltd. 10.1177/03611981211019034 He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. http://arxiv.org/abs/1703.06870 Kumar, P., Sharma, A., & Kota, S. R. (2021). Automatic multiclass instance segmentation of concrete damage using deep learning model. IEEE Access, 9, 90330–90345. 10.1109/ACCESS.2021.3090961 Kwon, O. S., Park, C. S., & Lim, C. R. (2014). A defect management system for reinforced concrete work utilizing BIM, image-matching and augmented reality. Automation in Construction, 46, 74–81. 10.1016/j.autcon.2014.05.005 Laxman, K. C., Tabassum, N., Ai, L., Cole, C., & Ziehl, P. (2023). Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning. Construction and Building Materials, 370. 10.1016/j.conbuildmat.2023.130709 Liang, Z., Zhang, H., Liu, L., He, Z., & Zheng, K. (2019). Defect detection of rail surface with deep convolutional neural networks. Proceedings of the World Congress on Intelligent Control and Automation (WCICA), 2018-July, 1317–1322. 10.1109/WCICA.2018.8630525 Lockley, S., Benghi, C., & Černý, M. (2017). Xbim.Essentials: A library for interoperable building information applications. The Journal of Open Source Software, 2(20), 473. 10.21105/joss.00473 Meng, S., Gao, Z., Zhou, Y., He, B., & Djerrad, A. (2023). Real-time automatic crack detection method based on drone. Computer-Aided Civil and Infrastructure Engineering, 38(7), 849–872. 10.1111/mice.12918 Microsoft HoloLens2. (n.d.). Retrieved September 29, 2023, from https://www.microsoft.com/en-us/hololens/ Papamarkou, T., Guy, H., Kroencke, B., Miller, J., Robinette, P., Schultz, D., Hinkle, J., Pullum, L., Schuman, C., Renshaw, J., & Chatzidakis, S. (2021). Automated detection of corrosion in used nuclear fuel dry storage canisters using residual neural networks. Nuclear Engineering and Technology, 53(2), 657–665. 10.1016/j.net.2020.07.020 Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173. 10.1007/s11263-007-0090-8 Vuforia Engine. (n.d.). Retrieved September 29, 2023, from https://developer.vuforia.com/downloads/SDK Wei, F., Yao, G., Yang, Y., & Sun, Y. (2019). Instance-level recognition and quantification for concrete surface bughole based on deep learning. Automation in Construction, 107. 10.1016/j.autcon.2019.102920 Wei, X., Yang, Z., Liu, Y., Wei, D., Jia, L., & Li, Y. (2019). Railway track fastener defect detection based on image processing and deep learning techniques: A comparative study. Engineering Applications of Artificial Intelligence, 80, 66–81. 10.1016/j.engappai.2019.01.008 Xu, X., Zhao, M., Shi, P., Ren, R., He, X., Wei, X., & Yang, H. (2022). Crack detection and comparison study based on faster R-CNN and Mask R-CNN. Sensors, 22(3). 10.3390/s22031215 Xu, Y., Li, D., Xie, Q., Wu, Q., & Wang, J. (2021). Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN. Measurement: Journal of the International Measurement Confederation, 178. 10.1016/j.measurement.2021.109316 Xue, Y., Cai, X., Shadabfar, M., Shao, H., & Zhang, S. (2020). Deep learning-based automatic recognition of water leakage area in shield tunnel lining. 10.17632/xz2nykszbs.1 Xue, Y., & Li, Y. (2018). A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects. Computer-Aided Civil and Infrastructure Engineering, 33(8), 638–654. 10.1111/mice.12367 Zhang, Z., Liang, M., & Wang, Z. (2021). A deep extractor for visual rail surface inspection. IEEE Access, 9, 21798–21809. 10.1109/ACCESS.2021.3055512 Zhang, Z., Yu, S., Yang, S., Zhou, Y., & Zhao, B. (2021). Rail-5k: A real-world dataset for rail surface defects detection. http://arxiv.org/abs/2106.14366 Zheng, D., Li, L., Zheng, S., Chai, X., Zhao, S., Tong, Q., Wang, J., & Guo, L. (2021). A defect detection method for rail surface and fasteners based on deep convolutional neural network. Computational Intelligence and Neuroscience, 2021. 10.1155/2021/2565500 Zhou, Y., Luo, H., & Yang, Y. (2017). Implementation of augmented reality for segment displacement inspection during tunneling construction. Automation in Construction, 82, 112–121. 10.1016/j.autcon.2017.02.007