At construction sites, as-built management is generally conducted by taking pictures or surveying with total stations and comparing the images or survey data with design drawings or Building Information Modeling (BIM) models. Since this work is time-consuming and error-prone, more efficient and accurate methods using advanced Information and Communication Technology (ICT) are desired. Therefore, this research proposes a method that can efficiently capture the progress of construction by detecting each constructed structural member, such as beams, columns, connections, etc. In this proposed method, construction engineers first take many pictures of the construction site and conduct automatic image segmentation using a pre-trained Convolutional Neural Network (CNN) model. Next, point cloud data is generated from taken pictures by using Structure from Motion (SfM). Then, the point cloud data is semantically segmented by overlapping the segmented images and point cloud data using the pin-hole camera technique. Finally, the design BIM model and segmented point cloud data are overlapped, and constructed parts of the BIM model can be detected, which can be reported as as-built parts. A prototype system was developed and applied to an actual railway construction project in Osaka, Japan for testing the accuracy and performance of the system