The goal of this project is to implement technique known as “space carving" to reconstruct the shape of a 3D object from multiple photographs taken at known but arbitrarily distributed viewpoints. An object is placed on top of a rotating plate together with a custom-designed fiducal marker (see the following sections for details). The background is made of a uniform-colored material to be easily distinguished from the target object. A calibrated camera is placed in front of the object capturing the scene throughout an entire rotation.
The volume occupied by the object is represented by a discrete set of voxels distributed on a cube of size N x N x N. A voxel can be seen as the 3D extension of a “pixel” describing a property of a certain region of space. In this case, the property is being either occupied or not by our target object.
At each frame, a set of 3D rays exit the camera starting from the optical center and passing through each pixel of the image. Every ray may intersect some voxels before reaching either the background or the object itself. If a ray reaches the background without touching the object, all the intersected voxels can be “carved” (ie. removed) as they represent empty space. On the contrary, if a ray reaches the object, at least one of the intersected voxels is part of the object, so they must not be removed from the set.
For the space-carving technique being effective, two sub-problems must be solved at each frame i:
- Estimate the position of the camera with respect to the object (ie. the camera pose Ri, Ti)
- Find the “silhouette” of the object by clustering the pixels as part of the background or foreground
The complete algorithm works as follows:
- Let V be an initial set of voxels distributed in a N x N x N cube
- For each image:
- Compute the camera Projection matrix P = K [Ri Ti]
- Project all the voxels onto the image
- Check if the projection of the voxel is inside or outside the silhouette. In the latter case, remove the voxel from V
When all the frames have been processed, the remaining set of voxels define the volume occupied by the object.
A package containing 4 different video sequences (plus the calibration) can be downloaded at the following URL. After having download the compressed folder extract it to the data directory.
The project structure is divided into three subprojects:
- Background foreground detection (1_background_foreground_detection)
- Markers Detector (2_markers_detector)
- Pose estimation (3_pose_estimation)
Each mini project is a task that the students could decide to develop during the course or to carry out the final project as I did. The additional work in the project is mainly summarized in the implementation of the .ply file in order to export the object mesh.
The final project can be examined in the space_carving folder.
pip install numpy
pip install opencv-python
The first step in order to start the project is to run the camera_calibration program to obtain the camera matrix and distortion parameters
cd space_carving
python camera_calibration.py
Then to run the space_carving program you have also to specify the following positional argument:
- voxel_cube_edge_dim: which is the dimension of one voxel cube edge
And you can add the following option:
- --hd_laptop: if you have HD screen resolution
In this console example I run the space_carving program using a dimension of a voxel cube edge of 2 wrt the marker reference coordinates dimension
python space_carving.py 2
cd 3_pose_estimation
python camera_calibration.py
cd 1_background_foreground_detection
python back_fore_undist_segmentation.py
cd 2_markers_detector
python marker_detector.py
cd 3_pose_estimation
python pose_estimation.py
The Mesh results of each object are stored in the optput_project folder, I recommend using MeshLab as software to deeply analyse the object shape.
Space Carving of obj01.mp4...
DONE
Average FPS is: 0.6086838923118594
Average Reprojection RMS Pixel Error is: 0.9412538925287559
Saving PLY file...
DONE
Space Carving of obj02.mp4...
DONE
Average FPS is: 0.47182103639343614
Average Reprojection RMS Pixel Error is: 0.896294098953868
Saving PLY file...
DONE
Space Carving of obj03.mp4...
DONE
Average FPS is: 0.2545594350406094
Average Reprojection RMS Pixel Error is: 0.9178120944262036
Saving PLY file...
DONE
Space Carving of for obj04.mp4...
DONE
Average FPS is: 0.6229241890497109
Average Reprojection RMS Pixel Error is: 0.9080787727231838
Saving PLY file...
DONE