Overview
General object grasping is a fundamental problem in robotics and has many applications in industry.
In this project, we designed an interactive, general grasping framework. Users can complete the task of grasping objects
with a robotic arm by simply clicking on the target object in the interactive interface.
The system pipeline is as follows: interactive segmentation is achieved through the SAM model with the input color and user prompt.
Using the generated mask and the input depth, the 3D point cloud of the target object is obtained.
Candidate grippers are then generated using the general grasping network GraspNet,
and finally, the optimal gripper is selected to grasp the target.
We selected six different objects to verify the effectiveness of the general object grasping. From top to bottom are the input image, interactive segmentation results, candidate grippers, and the optimal gripper. The results show that the system can accurately segment the target object and generate a reasonable gripper.
Single Object Grasp
We deployed the system in a real-world setting and tested it with different grasping objects. The system was able to accurately achieve target grasping in all cases.
Object 1
Object 2
Object 3
Object 4
Object 5
Object 6
Multi-object Grasp
We demonstrated tests of continuous grasping of multiple objects. The system can accurately grasp each target object.
Grasp Accuracy
We uniformly selected 6 fixed points within the camera’s working range to test the grasping accuracy of different objects. The test results are shown in the table below, indicating that different objects can be accurately grasped at different points.
| Bowl | Cup | Flower | Cube | Eggplant | Grape | Doll | |
|---|---|---|---|---|---|---|---|
| Position1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Position2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Position3 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Position4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Position5 | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Position6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Additionally, we conducted quantitative tests on the accuracy of continuous grasping. For each object, we randomly performed 100 grasps within the camera’s working range. The accuracy are shown in the table below, demonstrating that different objects all achieve very high accuracy.
| Bowl | Cup | Flower | Cube | Eggplant | Grape | Doll | |
|---|---|---|---|---|---|---|---|
| Accuracy | 100% | 94% | 100% | 100% | 96% | 100% | 100% |
Reference
[1] Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[2] Fang, Hao-Shu, et al. "Graspnet-1billion: A large-scale benchmark for general object grasping." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.