Scheduling Optimization of Electric Ready Mixed Concrete Vehicles Using an Improved Model-Based Reinforcement Learning Zhengyi Chen University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong Changhao Song University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong Xiao Zhang University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong Jack C. P. Cheng University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong This is a section of CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality (DOI: 10.36253/979-12-215-0289-3) by Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi Firenze University Press Florence 2023 https://doi.org/10.36253/10.36253/979-12-215-0289-3.74

Available for academic research purposes

Open Access

Copyright Author(s)

Content licence CC BY-NC 4.0

Metadata licence CC0 1.0

This is original content, published for academic research purposes

 Digital edition XML powered by Booksflow

Decarbonizing the construction sector has become an imperative global agenda, with electric machinery playing a pivotal role in realizing this objective. This research concentrates on devising an operational scheduling optimization method for electric ready-mixed concrete vehicles (ERVs) – a groundbreaking, eco-friendly intervention for the construction sector. We commence by outlining a systematic problem definition for the ERV operational process, considering the distinctive characteristics of electric vehicles and ready-mixed concrete (RMC) delivery tasks. The entire process is then conceptualized as a Markov decision problem (MDP), which enables sequential decision-making. We subsequently develop an enhanced model-based reinforcement learning technique, named parallel-masked-decaying Monte Carlo Tree Search (PMD-MCTS), for efficient resolution of the MDP. The entire system is authenticated via a real-world case study, and the PMD-MCTS's performance is juxtaposed against existing benchmarks. The results demonstrate the appropriateness of the proposed MDP formulation for tackling RMC delivery tasks. The PMD-MCTS algorithm and one of its ablation algorithms (PM-MCTS) have demonstrated superior performance compared to other benchmarks in either cost reduction or delay minimization, with PMD-MCTS requiring 30% less computation time than PM-MCTS

 Electric vehicle Ready-mixed concrete delivery; Scheduling optimization; Model-based reinforcement learning; Monte Carlo Tree Search

It is available online at https://doi.org/10.36253/10.36253/979-12-215-0289-3.74

References Bogachkova, L. Y., Guryanova, L. S., & Usacheva, N. Y. (2022). Decarbonization Trends in the Largest Post-soviet Countries and the Specifics of Their Inclusion in the Global Climate Agenda. New Technology for Inclusive and Sustainable Growth: Perception, Challenges and Opportunities, 77-88. 10.1007/978-981-16-9804-0_7 Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., . . . Colton, S. (2012). A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1), 1-43. 10.1109/TCIAIG.2012.2186810 Gan, V. J., Chan, C. M., Tse, K., Lo, I. M., & Cheng, J. C. (2017). A comparative analysis of embodied carbon in high-rise buildings regarding different design parameters. Journal of Cleaner Production, 161, 663-675. 10.1016/j.jclepro.2017.05.156 Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2), 100-107. 10.1109/TSSC.1968.300136 Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. 10.1162/neco.1997.9.8.1735 Huang, B., Boularias, A., & Yu, J. (2022). Parallel Monte Carlo Tree Search with Batched Rigid-body Simulations for Speeding up Long-Horizon Episodic Robot Planning. Paper presented at the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Huang, S., & Ontañón, S. (2020). A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14171. 10.32473/flairs.v35i.130584 Karakatič, S. (2021). Optimizing nonlinear charging times of electric vehicle routing with genetic algorithm. Expert Systems with Applications, 164, 114039. 10.1016/j.eswa.2020.114039 Lin, P.-C., Wang, J., Huang, S.-H., & Wang, Y.-T. (2010). Dispatching ready mixed concrete trucks under demand postponement and weight limit regulation. Automation in Construction, 19(6), 798-807. 10.1016/j.autcon.2010.05.002 Lin, T., Lin, Y., Ren, H., Chen, H., Chen, Q., & Li, Z. (2020). Development and key technologies of pure electric construction machinery. Renewable and Sustainable Energy Reviews, 132, 110080. 10.1016/j.rser.2020.110080 Liu, A., Chen, J., Yu, M., Zhai, Y., Zhou, X., & Liu, J. (2018). Watch the unobserved: A simple approach to parallelizing monte carlo tree search. arXiv preprint arXiv:1810.11755. 10.48550/arXiv.1810.11755 Liu, Z., Zhang, Y., & Li, M. (2014). Integrated scheduling of ready-mixed concrete production and delivery. Automation in Construction, 48, 31-43. 10.1016/j.autcon.2014.08.004 Liu, Z., Zhang, Y., Yu, M., & Zhou, X. (2017). Heuristic algorithm for ready-mixed concrete plant scheduling with multiple mixers. Automation in Construction, 84, 1-13. 10.1016/j.autcon.2017.08.013 Luo, F.-M., Xu, T., Lai, H., Chen, X.-H., Zhang, W., & Yu, Y. (2022). A survey on model-based reinforcement learning. arXiv preprint arXiv:2206.09328. 10.48550/arXiv.2206.09328 Olanrewaju, O. I., Edwards, D. J., & Chileshe, N. (2020). Estimating on-site emissions during ready mixed concrete (RMC) delivery: a methodology. Case Studies in Construction Materials, 13, e00439. 10.1016/j.cscm.2020.e00439 Palaniappan, S., Bashford, H., Li, K., Fafitis, A., & Stecker, L. (2009). Carbon emissions based on transportation for post-tensioned slab foundation construction: A production home building study in the greater phoenix arizona area. International journal of construction education and research, 5(4), 236-260. 10.1080/15578770903355533 Sinha, R. K., & Chaturvedi, N. D. (2019). A review on carbon emission reduction in industries and planning emission limits. Renewable and Sustainable Energy Reviews, 114, 109304. 10.1016/j.rser.2019.109304 Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction: MIT press. Tan, S., Yang, J., Zhao, X., Hai, T., & Zhang, W. (2018). Gear ratio optimization of a multi-speed transmission for electric dump truck operating on the structure route. Energies, 11(6), 1324. 10.3390/en11061324 Tong, Z., Jiang, Y., Tong, S., Zhang, Q., & Wu, J. (2023). Hybrid drivetrain with dual energy regeneration and collaborative control of driving and lifting for construction machinery. Automation in Construction, 150, 104806. 10.1016/j.autcon.2023.104806 Turan, B., Pedarsani, R., & Alizadeh, M. (2020). Dynamic pricing and fleet management for electric autonomous mobility on demand systems. Transportation Research Part C: Emerging Technologies, 121, 102829. 10.1016/j.trc.2020.102829 Volvo Trucks delivers the first heavy-duty electric concrete mixer truck to CEMEX. (2023). Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., . . . Ba, J. (2019). Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.02057. 10.48550/arXiv.1907.02057 Zhang, X., Zhang, J., Liu, Z., Cui, Q., Tao, X., & Wang, S. (2020). MDP-based task offloading for vehicular edge computing under certain and uncertain transition probabilities. IEEE Transactions on Vehicular Technology, 69(3), 3296-3309. 10.1109/TVT.2020.2965159