Predicting Mental Workload of Using Exoskeletons for Construction Work: A Deep Learning Approach Abiola Akanmu Virginia Tech/Myers Lawson School of Construction, United States Adedeji Afolabi Virginia Tech/Myers Lawson School of Construction, United States Akinwale Okunola Virginia Tech/Myers Lawson School of Construction, United States This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.69

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

Exoskeletons are gaining attention as a potential solution for addressing low back injury in the construction industry. However, use of active back-support exoskeletons in construction can trigger unintended consequences which could increase mental workload of users while working with exoskeletons. Prolonged increase in mental workload could impact workers’ wellbeing and productivity. Prediction of mental workload during exoskeleton-use could inform strategies to mitigate the triggers. This study investigates a machine-learning framework for predicting mental workload of workers while using active back-support exoskeletons for construction work. Laboratory experiments were conducted wherein Electroencephalography (EEG) data were collected from participants wearing active back-support exoskeletons to perform flooring task. The EEG data underwent preprocessing, including band filtering, notch filtering, and independent component analysis, to remove artifacts and ensure data quality. A regression-based Long Short-Term Memory network was trained to forecast future time steps of the processed EEG data. The performance of the network was evaluated using root mean square error (RMSE) and r-squared (R2). A RMSE of 0.1527 and R2 of 0.9665 indicating good fit and strong correlation, respectively, were observed between the predicted and actual EEG data. Results of the comparison between the actual and predicted mental workload also show strong correction with an R2 of 0.8692. The findings motivate research directions into real-time monitoring of mental workload of workers during exoskeleton-use. The study has significant implications for stakeholders, enabling them to gain a deeper understanding of the impact of mental workload while using exoskeletons thereby providing opportunities for mitigation

 Work-related musculoskeletal disorders Exoskeleton Mental workload Electroencephalogram Long Short-Term Memory Flooring task

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.69

References Alabdulkarim, S., Kim, S., & Nussbaum, M. A. (2019). Effects of exoskeleton design and precision requirements on physical demands and quality in a simulated overhead drilling task. Applied Ergonomics, 80, 136-145. 10.1016/j.apergo.2019.05.014 Alemi, M. M., Madinei, S., Kim, S., Srinivasan, D., & Nussbaum, M. A. (2020). Effects of Two Passive Back-Support Exoskeletons on Muscle Activity, Energy Expenditure, and Subjective Assessments During Repetitive Lifting. Hum Factors, 62(3), 458-474. 10.1177/0018720819897669 Baltrusch, S. J., Houdijk, H., van Dieen, J. H., & de Kruif, J. T. C. M. (2021). Passive Trunk Exoskeleton Acceptability and Effects on Self-efficacy in Employees with Low-Back Pain: A Mixed Method Approach. Journal of Occupational Rehabilitation, 31(1), 129-141. 10.1007/s10926-020-09891-1 Bequette, B., Norton, A., Jones, E., & Stirling, L. (2020). Physical and Cognitive Load Effects Due to a Powered Lower-Body Exoskeleton. Human Factors, 62(3), 411-423. 10.1177/0018720820907450 Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews, 44, 58-75. 10.1016/j.neubiorev.2012.10.003 Chen, D., Huang, H., Bao, X., Pan, J., & Li, Y. (2023). An EEG-based attention recognition method: fusion of time domain, frequency domain, and non-linear dynamics features. Frontiers in Neuroscience, 17. 10.3389/fnins.2023.1194554. Chen, J. Y., Song, X. Y., & Lin, Z. H. (2016). Revealing the "Invisible Gorilla" in construction: Estimating construction safety through mental workload assessment. Automation in Construction, 63, 173-183. 10.1016/j.autcon.2015.12.018 Chen, J. Y., Taylor, J. E., & Comu, S. (2017). Assessing Task Mental Workload in Construction Projects: A Novel Electroencephalography Approach. Journal of Construction Engineering and Management, 143(8). 10.1061/(ASCE)CO.1943-7862.0001345 Coulibaly, P., & Baldwin, C. K. (2005). Nonstationary hydrological time series forecasting using nonlinear dynamic methods. Journal of Hydrology, 307(1-4), 164-174. 10.1016/j.jhydrol.2004.10.008 Cumplido-Trasmonte, C., Barquin-Santos, E., Garces-Castellote, E., Gor-Garcia-Fogeda, M. D., Plaza-Flores, A., Hernandez-Melero, M., Gutierrez-Ayala, A., Cano-de-la-Cuerda, R., Lopez-Moron, A. L., & Garcia-Armada, E. (2023). Safety and usability of the MAK exoskeleton in patients with stroke. Physiotherapy Research International. 10.1002/pri.2038 de Looze, M. P., Bosch, T., Krause, F., Stadler, K. S., & O'Sullivan, L. W. (2016). Exoskeletons for industrial application and their potential effects on physical work load. Ergonomics, 59(5), 671-681. 10.1080/00140139.2015.1081988 Fan, J., & Smith, A. P. (2017). The Impact of Workload and Fatigue on Performance. Communications in Computer and Information Science, 726, 90-105. 10.1007/978-3-319-61061-0_6 Fox, S., Aranko, O., Heilala, J., & Vahala, P. (2020). Exoskeletons Comprehensive, comparative and critical analyses of their potential to improve manufacturing performance. Journal of Manufacturing Technology Management, 31(6), 1261-1280. 10.1108/Jmtm-01-2019-0023 Frølich, L., & Dowding, I. (2018). Removal of muscular artifacts in EEG signals: a comparison of linear decomposition methods. Brain informatics, 5(1), 13-22. 10.1007/s40708-017-0074-6. Gonsalves, N., Akanmu, A., Gao, X. H., Agee, P., & Shojaei, A. (2023). Industry Perception of the Suitability of Wearable Robot for Construction Work. Journal of Construction Engineering and Management, 149(5). 10.1061/JCEMD4.COENG-12762 Gonsalves, N. J., Ogunseiju, O. R., Akanmu, A. A., & Nnaji, C. A. (2021). Assessment of a Passive Wearable Robot for Reducing Low Back Disorders during Rebar Work. Journal of Information Technology in Construction, 26, 936-952. 10.36680/j.itcon.2021.050 Gorgey, A. S. (2018). Robotic exoskeletons: The current pros and cons. World Journal of Orthopedics, 9(9), 112-119. 10.5312/wjo.v9.i9.112 Guo, H. L., Zhang, Z. T., Yu, R., Sun, Y. K., & Li, H. (2023). Action Recognition Based on 3D Skeleton and LSTM for the Monitoring of Construction Workers' Safety Harness Usage. Journal of Construction Engineering and Management, 149(4). 10.1061/JCEMD4.COENG-12542 Hernandez, C., Slaton, T., Balali, V., & Akhavian, R. (2019). A Deep Learning Framework for Construction Equipment Activity Analysis. Computing in Civil Engineering 2019: Data, Sensing, and Analytics, 479-486. 10.1061/9780784482438.061. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. 10.1162/neco.1997.9.8.1735. Jaiswal, A., Ramesh Babu, A., Zaki Zadeh, M., Wylie, G., & Makedon, F. (2023). Detecting Cognitive Fatigue in Subjects with Traumatic Brain Injury from FMRI Scans Using Self-Supervised Learning. Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, 83-90. 10.1145/3594806.3594868. Jebelli, H., Choi, B., & Lee, S. (2019). Application of wearable biosensors to construction sites. I: Assessing workers’ stress. Journal of Construction Engineering and Management, 145(12), 04019079. 10.1061/(ASCE)CO.1943-7862.0001729. Jebelli, H., Hwang, S., & Lee, S. (2018a). EEG-based workers' stress recognition at construction sites. Automation in Construction, 93, 315-324. 10.1016/j.autcon.2018.05.027. Jebelli, H., Hwang, S., & Lee, S. (2018b). EEG signal-processing framework to obtain high-quality brain waves from an off-the-shelf wearable EEG device. Journal of Computing in Civil Engineering, 32(1), 04017070. 10.1061/(ASCE)CP.1943-5487.0000719. Ke, J., Zhang, M., Luo, X., & Chen, J. (2021). Monitoring distraction of construction workers caused by noise using a wearable Electroencephalography (EEG) device. Automation in Construction, 125, 103598. 10.1016/j.autcon.2021.103598 Kim, S., Moore, A., Srinivasan, D., Akanmu, A., Barr, A., Harris-Adamson, C., Rempel, D. M., & Nussbaum, M. A. (2019). Potential of Exoskeleton Technologies to Enhance Safety, Health, and Performance in Construction: Industry Perspectives and Future Research Directions. Iise Transactions on Occupational Ergonomics & Human Factors, 7(3-4), 185-191. 10.1080/24725838.2018.1561557 Liu, M. Z., Xu, X., Hu, J., & Jiang, Q. N. (2022). Real time detection of driver fatigue based on CNN‐LSTM. IET Image Processing, 16(2), 576-595. 10.1049/ipr2.12373. Liu, P. K., Chi, H. L., Li, X., & Li, D. S. (2020). Development of a Fatigue Detection and Early Warning System for Crane Operators: A Preliminary Study. Construction Research Congress 2020: Computer Applications, 106-115. 10.1061/9780784482865.012. Liu, Y., Li, X. L., Lai, J. R., Zhu, A. B., Zhang, X. D., Zheng, Z. M., Zhu, H. J., Shi, Y. Y., Wang, L., & Chen, Z. Y. (2021). The Effects of a Passive Exoskeleton on Human Thermal Responses in Temperate and Cold Environments. International Journal of Environmental Research and Public Health, 18(8). 10.3390/ijerph18083889 Mantini, D., Franciotti, R., Romani, G. L., & Pizzella, V. (2008). Improving MEG source localizations: an automated method for complete artifact removal based on independent component analysis. NeuroImage, 40(1), 160-173. 10.1016/j.neuroimage.2007.11.022. Marchand, C., De Graaf, J. B., & Jarrasse, N. (2021). Measuring mental workload in assistive wearable devices: a review. Journal of Neuroengineering and Rehabilitation, 18(1). 10.1186/s12984-021-00953-w Massardi, S., Pinto-Fernandez, D., Babic, J., Dezman, M., Trost, A., Grosu, V., Lefeber, D., Rodriguez, C., Bessler, J., Schaake, L., Prange-Lasonder, G., Veneman, J. F., & Torricelli, D. (2023). Relevance of hazards in exoskeleton applications: a survey-based enquiry. J Neuroeng Rehabil, 20(1), 68. 10.1186/s12984-023-01191-y Mastropietro, A., Pirovano, I., Marciano, A., Porcelli, S., & Rizzo, G. (2023). Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines. Sensors (Basel), 23(3). 10.3390/s23031367 Mehmood, I., Li, H., Qarout, Y., Umer, W., Anwer, S., Wu, H., Hussain, M., & Antwi-Afari, M. F. (2023). Deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data. Advanced Engineering Informatics, 56, 101978. 10.1016/j.aei.2023.101978. Missonnier, P., Deiber, M. P., Gold, G., Millet, P., Pun, M. G. F., Fazio-Costa, L., Giannakopoulos, P., & Ibanez, V. (2006). Frontal theta event-related synchronization: comparison of directed attention and working memory load effects. Journal of Neural Transmission, 113(10), 1477-1486. 10.1007/s00702-005-0443-9 Miyamoto, K., Tanaka, H., & Nakamura, S. (2022). Online EEG-Based Emotion Prediction and Music Generation for Inducing Affective States. Ieice Transactions on Information and Systems, E105d(5), 1050-1063. 10.1587/transinf.2021EDP7171 Nussbaum, M. A., Lowe, B. D., de Looze, M., Harris-Adamson, C., & Smets, M. (2019). An Introduction to the Special Issue on Occupational Exoskeletons. Iise Transactions on Occupational Ergonomics & Human Factors, 7(3-4), 153-162. 10.1080/24725838.2019.1709695 Ogunseiju, O., Akinniyi, A., Gonsalves, N., Khalid, M., & Akanmu, A. (2023). Detecting Learning Stages within a Sensor-Based Mixed Reality Learning Environment Using Deep Learning. Journal of Computing in Civil Engineering, 37(4). 10.1061/JCCEE5.CPENG-5169 Ogunseiju, O., Olayiwola, J., Akanmu, A., & Olatunji, O. A. (2022). Evaluation of postural-assist exoskeleton for manual material handling. Engineering Construction and Architectural Management, 29(3), 1358-1375. 10.1108/Ecam-07-2020-0491 Poliero, T., Lazzaroni, M., Toxiri, S., Di Natali, C., Caldwell, D. G., & Ortiz, J. (2020). Applicability of an Active Back-Support Exoskeleton to Carrying Activities. Frontiers in Robotics and Ai, 7. 10.3389/frobt.2020.579963 Qin, Y. M., & Bulbul, T. (2023). Electroencephalogram-based mental workload prediction for using Augmented Reality head mounted display in construction assembly: A deep learning approach. Automation in Construction, 152. 10.1016/j.autcon.2023.104892 Renaud, O., & Victoria-Feser, M.-P. (2010). A robust coefficient of determination for regression. Journal of Statistical Planning and Inference, 140(7), 1852-1862. 10.1016/j.jspi.2010.01.008. Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991-1009. 10.1016/j.ergon.2005.04.005 Simon, M., Schmidt, E. A., Kincses, W. E., Fritzsche, M., Bruns, A., Aufmuth, C., Bogdan, M., Rosenstiel, W., & Schrauf, M. (2011). EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions. Clinical Neurophysiology, 122(6), 1168-1178. 10.1016/j.clinph.2010.10.044 Theurel, J., Desbrosses, K., Roux, T., & Savescu, A. (2018). Physiological consequences of using an upper limb exoskeleton during manual handling tasks. Applied Ergonomics, 67, 211-217. 10.1016/j.apergo.2017.10.008 Wang, F., Xuan, Z., Zhen, Z., Li, K., Wang, T., & Shi, M. (2020). A day-ahead PV power forecasting method based on LSTM-RNN model and time correlation modification under partial daily pattern prediction framework. Energy Conversion and Management, 212, 112766. 10.1016/j.enconman.2020.112766. Wei, W., Zha, S., Xia, Y., Gu, J., & Lin, X. (2020). A hip active assisted exoskeleton that assists the semi-squat lifting. Applied Sciences, 10(7), 2424. 10.3390/app10072424. Welch, P. (1967). The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacoustics, 15(2), 70-73. 10.1109/TAU.1967.1161901. Xing, X., Zhong, B., Luo, H., Rose, T., Li, J., & Antwi-Afari, M. F. (2020). Effects of physical fatigue on the induction of mental fatigue of construction workers: A pilot study based on a neurophysiological approach. Automation in Construction, 120, 103381. 10.1016/j.autcon.2020.103381. Yang, Y. Q., Ye, Z. H., Easa, S. M., Feng, Y., & Zheng, X. Y. (2023). Effect of driving distractions on driver mental workload in work zone's warning area. Transportation Research Part F-Traffic Psychology and Behaviour, 95, 112-128. 10.1016/j.trf.2023.03.018 Young, M. S., Brookhuis, K. A., Wickens, C. D., & Hancock, P. A. (2015). State of Science: Mental Workload in Ergonomics. Ergonomics, 58, 1–17. 10.1080/00140139.2014.956151. Zhu, F., Kern, M., Fowkes, E., Afzal, T., Contreras-Vidal, J.-L., Francisco, G. E., & Chang, S.-H. (2021). Effects of an exoskeleton-assisted gait training on post-stroke lower-limb muscle coordination. Journal of Neural Engineering, 18(4), 046039. 10.1088/1741-2552/abf0d5