Improving Sense-Making for Construction Planning Tasks Using Visual and Haptic Stimuli in Virtual Reality Environments Ivan Mutis Illinois Institute of Technology, United States Marina Oberemok Illinois Institute of Technology, United States Nishanth Purushotham Illinois Institute of Technology, United States This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.14

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

Design documents, drawings, and specifications are visual representations that are fundamental and prevalent in today’s construction engineering practice. Construction specialties (e.g., structural, mechanical) rely on these visual representations to express and draw meaning during collaborations. Construction engineering and management (CEM) students must acquire the knowledge, skills, and abilities — a key example of which is perceptual competence —for interpreting visual representations to facilitate efficient task execution, such as planning. Empowering learners with new technology using robust real-world immersion and interactive features is a significant step towards this target. The presented research explores new human-machine interactions to determine the best way for CEM students to learn through the combined senses of sight and touch. The approach merges visual and haptic interactions within an immersive environment to enhance perception and reasoning skills. The research demonstrates how CEM learners interact with and interpret the meanings of information within a planning task. It explores how VR and haptic technology augment the ability to recognize meanings — a new type of representational competency — for improved interpretation of information related to components with respect to engineering disciplines and sub-systems in a CEM, and investigates learners’ problem-solving ability by using perception-rich enhanced virtual reality (VR) and haptic affordances

 haptic cues human-computer-interaction design interpretations

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.14

References Adami, P., Rodrigues, P. B., Woods, P. J., Becerik-Gerber, B., Soibelman, L., Copur-Gencturk, Y., & Lucas, G. (2022). Impact of VR-Based Training on Human–Robot Interaction for Remote Operating Construction Robots. Journal of Computing in Civil Engineering, 36(3), 04022006. 10.1061/(ASCE)CP.1943-5487.0001016 Adilkhanov, A., Rubagotti, M., & Kappassov, Z. (2022). Haptic Devices: Wearability-Based Taxonomy and Literature Review. IEEE Access, 10, 91923-91947. 10.1109/access.2022.3202986 Alakhawand, N., Frier, W., & Lepora, N. F. (2022). Mapping Mid-Air Haptics With a Low-Cost Tactile Robot. IEEE Robotics and Automation Letters, 7, 7873-7880. Antonenko, P., & Mutis, I. (2017). Using Unmanned Aerial Systems to Bring STEM Field Experiences to the Classroom National Association for Research in Science Teaching (NARST), San Antonio, TX. http://www.narst.org/annualconference/2017conference.cfm Antonenko, P. D., & Mutis, I. (2017). Empowering learning through remote visualizations using unmanned aerial systems: Perspectives of education and industry experts 2017 American Educational Research Association, San Antonio, TX. http://www.aera.net/Publications Bluteau, J., Coquillart, S., Payan, Y., & Gentaz, E. (2008). Haptic Guidance Improves the Visuo-Manual Tracking of Trajectories. PLOS ONE, 3(3), e1775. 10.1371/journal.pone.0001775 Christiand, & Yoon, J. (2011). Assembly simulations in virtual environments with optimized haptic path and sequence. Robotics and Computer-Integrated Manufacturing, 27(2), 306-317. 10.1016/j.rcim.2010.07.015 Chryssa, T., & Julie-Ann, S. (2020). From Video-Conferencing to Holoportation and Haptics: How Emerging Technologies Can Enhance Presence in Online Education? 10.1007/978-981-15-0618-5_16 Coffey, M., & Pierson, A. (2022, 2022). Collaborative Teleoperation with Haptic Feedback for Collision-Free Navigation of Ground Robots. Cooper, N., Milella, F., Pinto, C., Cant, I., White, M., & Meyer, G. (2018). The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment. PLOS ONE, 13(2), e0191846. 10.1371/journal.pone.0191846 Enriquez, M., Maclean, K., & Chita, C. (2006, 2006). Haptic phonemes Basic - Building Blocks of Haptic Communication. Enriquez, M. J., & MacLean, K. E. (2003, 22-23 March 2003). The hapticon editor: a tool in support of haptic communication research. 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings., Feygin, D., Keehner, M., & Tendick, R. (2002). Haptic guidance: experimental evaluation of a haptic training method for a perceptual motor skill. Fulkerson, M. (2020). Touch (E. N. Zalta, Ed. Summer 2020 ed.). Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/sum2020/entries/touch/ Hatzfeld, C., Kern, T. A., Opitz, T., Neupert, C., Kassner, S., Matysek, M., Rausch, J., Meckel, O., Rettig, A., Sindlinger, S., & Haus, H. (2015). Engineering Haptic Devices - Table of Contents and Sample Pages. Hu, W., & Zhang, X. (2012, 2012). A Rapid Development Method of Virtual Assembly Experiments Based on 3D Game Engine. Huang, K., Chitrakar, D., Rydén, F., & Chizeck, H. J. (2019). Evaluation of haptic guidance virtual fixtures and 3D visualization methods in telemanipulation—a user study. Intelligent Service Robotics, 12(4), 289-301. 10.1007/s11370-019-00283-w James, J., R, B., & Neamtu, G. (2019). Design of a bi-manual haptic interface for skill acquisition in surface mount device soldering. Soldering & Surface Mount Technology, 31. 10.1108/SSMT-01-2018-0001 Jong, T. d. (2014). Emerging Representation Technologies for Problem Solving. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of Research on Educational Communications and Technology (pp. 809-816). Springer New York. 10.1007/978-1-4614-3185-5_65 Jose, J., Unnikrishnan, R., Marshall, D., & Bhavani, R. R. (2016, 2016). Haptics enhanced multi-tool virtual interfaces for training carpentry skills. Kortum, P. (2008). HCI beyond the GUI: Design for haptic, speech, olfactory, and other nontraditional interfaces. Elsevier. Kreimeier, J., Hammer, S., Friedmann, D., Karg, P., Bühner, C., Bankel, L., & Götzelmann, T. (2019, 2019). Evaluation of different types of haptic feedback influencing the task-based presence and performance in virtual reality. Li, Z., Wang, J., Anwar, M. S., & Zheng, Z. (2020). An efficient method for generating assembly precedence constraints on 3D models based on a block sequence structure. Computer-Aided Design, 118, 102773. 10.1016/j.cad.2019.102773 Liu, X., Dodds, G., Hinds, B. K., & McCartney, J. (2003). Virtual DesignWorks: Designing 3D CAD Models Via Touch Interaction. ASME 2003 International Mechanical Engineering Congress and Exposition, Luo, J., Yang, C., Su, H., & Liu, C. (2019). A Robot Learning Method with Physiological Interface for Teleoperation Systems. Applied Sciences, 9(10), 2099. 10.3390/app9102099 Manchanda, N., Jha, S., & Mukherjee, S.-h. (2017). Overview: Human-Computer Interaction an Globally Uses Technique in Society. Mastrolembo Ventura, S., Castronovo, F., Nikolić, D., & Ciribini, A. L. C. (2022). Implementation of virtual reality in construction education: a content-analysis based literature review. Journal of Information Technology in Construction, 27, 705-731. 10.36680/j.itcon.2022.035 Medellín-Castillo, H., Gonzalez-Badillo, G., Govea, E., Espinosa Castañeda, R., & Gallegos-Nieto, E. (2015). Development of Haptic-Enabled Virtual Reality Applications for Engineering, Medicine and Art. 10.1115/IMECE2015-52770 Mugge, W., Kuling, I. A., Brenner, E., & Smeets, J. B. J. (2016). Haptic Guidance Needs to Be Intuitive Not Just Informative to Improve Human Motor Accuracy. PLOS ONE, 11(3), e0150912. 10.1371/journal.pone.0150912 Müller, T. (2020). Designing with Haptic Feedback Umeå University]. http://umu.diva-portal.org/smash/get/diva2:1445032/FULLTEXT01.pdf Mutis, I. (2014). Enhancing spatial and temporal cognitive ability in construction education through augmented reality and artificial visualizations. The International Conference for Computing in Civil and Building Engineering (Icccbe) and CIB-W78, 2014, Orlando Fl. Mutis, I. (2015). Enhancing spatial and temporal cognitive ability in construction education through the effect of artificial visualizations. 2015 ASCE International Workshop on Computing in Civil Engineering, Austin, Tx. Mutis, I. (2018a). Spatial-temporal cognitive ability: Coupling representations to situations and contexts for coordinating activities in the construction project environment. In Transforming engineering education: Innovative computer-mediated learning technologies (pp. 5-24). Mutis, I. (2018b). Spatial-Temporal Cognitive Ability: coupling representations to situations and contexts for coordinating activities in the construction project environment. In I. Mutis, R. Fruchter, & C. Menassa (Eds.), Transforming Engineering Education through Innovative Computer Mediated Learning Technologies (pp. 14 ). ASCE, American Society of Civil Engineers. NSF, N. S. F. (2020). STEM Education for the Future: a visioning report. National Science Foundation. OED. (2020). Oxford English dictionary. https://www.google.com/search?q=haptics+definition&rlz=1C5CHFA_enUS1021US1021&oq=haptics+&aqs=chrome.0.35i39j69i57j35i39j0i131i433i512j0i512j69i60l3.5443j1j4&sourceid=chrome&ie=UTF-8 Pavlik, R. A., Vance, J. M., & Luecke, G. R. (2013). Interacting With a Large Virtual Environment by Combining a Ground-Based Haptic Device and a Mobile Robot Base. Prabhakaran, A., Mahamadu, A.-M., & Mahdjoubi, L. (2022). Understanding the challenges of immersive technology use in the architecture and construction industry: A systematic review. Automation in Construction, 137, 104228. 10.1016/j.autcon.2022.104228 Rahimian, F. P., & Ibrahim, R. (2011). Impacts of VR 3D sketching on novice designers’ spatial cognition in collaborative conceptual architectural design. Design Studies, 32, 255-291. Ranjith, R., Akshay, N., Unnikrishnan, R., & Bhavani, R. R. (2014, 2014). Do It Yourself Educational Kits for Vocational Education and Training. Sanfilippo, F., Blazauskas, T., Salvietti, G., Ramos, I., Vert, S., Radianti, J., Majchrzak, T. A., & Oliveira, D. (2022). A Perspective Review on Integrating VR/AR with Haptics into STEM Education for Multi-Sensory Learning. Robotics, 11(2), 41. 10.3390/robotics11020041 Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285. 10.1016/0364-0213(88)90023-7 Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive Architecture and Instructional Design: 20 Years Later. Educational Psychology Review, 31(2), 261-292. 10.1007/s10648-019-09465-5 Takahashi, C., Diedrichsen, J., & Watt, S. J. (2009). Integration of vision and haptics during tool use. Journal of Vision, 9(6), 3-3. 10.1167/9.6.3 Teranishi, A., Korres, G., Park, W., & Eid, M. A. (2018). Combining Full and Partial Haptic Guidance Improves Handwriting Skills Development. IEEE Transactions on Haptics, 11, 509-517. Tran, C., Smith, B., & Buschkuehl, M. (2017). Support of mathematical thinking through embodied cognition: Nondigital and digital approaches. Cognitive Research: Principles and Implications, 2(1). 10.1186/s41235-017-0053-8 Tytler, R. (2020). STEM Education for the Twenty-First Century. In (pp. 21-43). 10.1007/978-3-030-52229-2_3 Williams Ii, R. L., Chen, M.-Y., & Seaton, J. M. (2001). Haptics-Augmented High School Physics Tutorials. International Journal of Virtual Reality, 5(1), 167-184. 10.20870/ijvr.2001.5.1.2678 Williams, N. L., Li, J., & Lin, M. C. (2023). A Framework for Active Haptic Guidance Using Robotic Haptic Proxies. arXiv preprint arXiv:2301.05311. Yeh, S.-C., Hwang, W.-Y., Wang, J.-L., & Zhan, S.-Y. (2013). Study of co-located and distant collaboration with symbolic support via a haptics-enhanced virtual reality task. Interactive Learning Environments, 21, 184 - 198. Yuan, M. L., Ong, S. K., & Nee, A. Y. C. (2008). Augmented reality for assembly guidance using a virtual interactive tool. International Journal of Production Research, 46, 1745 - 1767.