A Framework for Realistic Virtual Representation for Immersive Training Environments. Caolan Plumb Teesside University, United Kingdom Farzad Pour Rahimian Teesside University, United Kingdom Diptangshu Pandit Teesside University, United Kingdom Hannah Thomas The Faraday Centre LTD, United Kingdom Nigel Clark The Faraday Centre LTD, United Kingdom This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.26

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

As mixed-reality (XR) technology becomes more available, virtually simulated training scenarios have shown great potential in enhancing training effectiveness. Realistic virtual representation plays a crucial role in creating immersive experiences that closely mimic real-world scenarios. With reference to previous methodological developments in the creation of information-rich digital reconstructions, this paper proposes a framework encompassing key components of the 3D scanning pipeline. While 3D scanning techniques have advanced significantly, several challenges persist in the field. These challenges include data acquisition, noise reduction, mesh and texture optimisation, and separation of components for independent interaction. These complexities necessitate the search for an optimised framework that addresses these challenges and provides practical solutions for creating realistic virtual representations in immersive training environments. The following exploration acknowledges and addresses challenges presented by the photogrammetry and laser-scanning pipeline, seeking to prepare scanned assets for real-time virtual simulation in a games-engine. This methodology employs both a camera and handheld laser-scanner for accurate data acquisition. Reality Capture is used to combine the geometric data and surface detail of the equipment. To clean the scanned asset, Blender is used for mesh retopology and reprojection of scanned textures, and attention given to correct lighting details and normal mapping, thus preparing the equipment to be interacted with by Virtual Reality (VR) users within Unreal Engine. By combining these elements, the proposed framework enables realistic representation of industrial equipment for the creation of training scenarios that closely resemble real-world contexts

 Digital twin; 3D reconstruction; Virtual reality; Laser scanning; Photogrammetry; Training simulation; Unreal Engine

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.26

References Abulrub, A. H. G., Attridge, A. N., & Williams, M. A. (2011, 4-6 April 2011). Virtual reality in engineering education: The future of creative learning. 2011 IEEE Global Engineering Education Conference (EDUCON), Adami, P., Rodrigues, P. B., Woods, P. J., Becerik-Gerber, B., Soibelman, L., Copur-Gencturk, Y., & Lucas, G. (2021). Effectiveness of VR-based training on improving construction workers’ knowledge, skills, and safety behavior in robotic teleoperation. Advanced Engineering Informatics, 50, 101431. 10.1016/j.aei.2021.101431 Alexander, O., Rogers, M., Lambeth, W., Chiang, M., & Debevec, P. (2009, 12-13 Nov. 2009). Creating a Photoreal Digital Actor: The Digital Emily Project. 2009 Conference for Visual Media Production, Bot, J. A., Irschick, D. J., Grayburn, J., Lischer-Katz, Z., Golubiewski-Davis, K., & Ikeshoji-Orlati, V. (2019). Using 3D photogrammetry to create open-access models of live animals: 2D and 3D software solutions. Grayburn et al., eds. D, 3, 54-72. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4), 834-848. Cowie, N., & Alizadeh, M. (2022). The Affordances and Challenges of Virtual Reality for Language Teaching. International Journal of TESOL Studies, 4(3). Cui, B., Tao, W., & Zhao, H. (2021). High-Precision 3D Reconstruction for Small-to-Medium-Sized Objects Utilizing Line-Structured Light Scanning: A Review. Remote Sensing, 13(21), 4457. https://www.mdpi.com/2072-4292/13/21/4457 Dewez, T. J. B., Girardeau-Montaut, D., Allanic, C., & Rohmer, J. (2016). FACETS : A CLOUDCOMPARE PLUGIN TO EXTRACT GEOLOGICAL PLANES FROM UNSTRUCTURED 3D POINT CLOUDS. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLI-B5, 799-804. 10.5194/isprs-archives-XLI-B5-799-2016 Farella, E. M., Morelli, L., Rigon, S., Grilli, E., & Remondino, F. (2022). Analysing Key Steps of the Photogrammetric Pipeline for Museum Artefacts 3D Digitisation. Sustainability, 14(9), 5740. https://www.mdpi.com/2071-1050/14/9/5740 Frost, A., Mirashrafi, S., Sánchez, C. M., Vacas-Madrid, D., Millan, E. R., & Wilson, L. (2023). Digital Documentation of Reflective Objects: A Cross-Polarised Photogrammetry Workflow for Complex Materials. In 3D Research Challenges in Cultural Heritage III: Complexity and Quality in Digitisation (pp. 131-155). Springer. Han, E., Nowak, K. L., & Bailenson, J. N. (2022). Prerequisites for Learning in Networked Immersive Virtual Reality. Kang, M. S., & An, Y.-K. (2021). Deep Learning-Based Automated Background Removal for Structural Exterior Image Stitching. Applied Sciences, 11(8), 3339. https://www.mdpi.com/2076-3417/11/8/3339 Leberl, F., Irschara, A., Pock, T., Meixner, P., Gruber, M., Scholz, S., & Wiechert, A. (2010). Point Clouds. Photogrammetric Engineering & Remote Sensing, 76(10), 1123-1134. 10.14358/PERS.76.10.1123 Loosemore, M., & Malouf, N. (2019). Safety training and positive safety attitude formation in the Australian construction industry. Safety Science, 113, 233-243. 10.1016/j.ssci.2018.11.029 Moolman, J., Corkery, G., Walsh, J., & Morrissey-Tucker, S. (2022). THE USE OF COLLABORATIVE VIRTUAL ENVIRONMENTS (CVES) FOR ENGINEERING EDUCATION IN HIGHER EDUCATION INSTITUTIONS. EDULEARN22 Proceedings, Newton, S., Wang, R., & Lowe, R. (2015). Blended reality and presence. International Journal of Design Sciences & Technology, 21(2). Noya, N. C., García, Á. L., & Ramírez, F. C. (2015). Combining photogrammetry and photographic enhancement techniques for the recording of megalithic art in north-west Iberia. Digital Applications in Archaeology and Cultural Heritage, 2(2), 89-101. 10.1016/j.daach.2015.02.004 Pan, Y., Braun, A., Brilakis, I., & Borrmann, A. (2022). Enriching geometric digital twins of buildings with small objects by fusing laser scanning and AI-based image recognition. Automation in Construction, 140, 104375. 10.1016/j.autcon.2022.104375 Porter, S. T., Roussel, M., & Soressi, M. (2016). A Simple Photogrammetry Rig for the Reliable Creation of 3D Artifact Models in the Field: Lithic Examples from the Early Upper Paleolithic Sequence of Les Cottés (France). Advances in Archaeological Practice, 4(1), 71-86. 10.7183/2326-3768.4.1.71 Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Sacks, R., Perlman, A., & Barak, R. (2013). Construction safety training using immersive virtual reality. Construction Management and Economics, 31(9), 1005-1017. 10.1080/01446193.2013.828844 Schaich, M., & Fritsch, D. (2013). Combined 3D scanning and photogrammetry surveys with 3D database support for archaeology and cultural heritage. A practice report on ArcTron’s information system aSPECT3D. Photogrammetric Week’13, 233-246. Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549-3557. Stefan, H., Mortimer, M., Horan, B., & Kenny, G. (2023). Evaluating the preliminary effectiveness of industrial virtual reality safety training for ozone generator isolation procedure. Safety Science, 163, 106125. 10.1016/j.ssci.2023.106125 Triantafyllou, V., Kotsopoulos, K. I., Tsolis, D., & Tsoukalos, D. (2022, 18-20 July 2022). Practical Techniques for Aerial Photogrammetry, Polygon Reduction and Aerial 360 Photography for Cultural Heritage Preservation in AR and VR Applications. 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA), Wang, J., Li, Z., Hu, W., Shao, Y., Wang, L., Wu, R., Ma, K., Zou, D., & Chen, Y. (2019). Virtual reality and integrated crime scene scanning for immersive and heterogeneous crime scene reconstruction. Forensic Science International, 303, 109943. 10.1016/j.forsciint.2019.109943 Wang, R., Peethambaran, J., & Chen, D. (2018). LiDAR Point Clouds to 3-D Urban Models$:$ A Review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(2), 606-627. 10.1109/JSTARS.2017.2781132 White, W. W., & Jung, M. J. (2022). Three-Dimensional Virtual Reality Spinal Cord Stimulator Training Improves Trainee Procedural Confidence and Performance. Neuromodulation: Technology at the Neural Interface. 10.1016/j.neurom.2022.03.005