Abstract

Three-dimensional reconstruction technology is used in a wide variety of applications. Automatically creating accurate pointclouds for large parts with complex geometries usually requires expensive metrology instruments. We are interested in using low-cost depth cameras mounted on commonly available industrial robots to create accurate pointclouds for large parts automatically. Manufacturing applications require fast cycle times. Therefore, we are interested in speeding up the 3D reconstruction process. We present algorithmic advances in 3D reconstruction that achieve a sub-millimeter accuracy using a low-cost depth camera. Our system can be used to determine a pointcloud model of large and complex parts. Advances in camera calibration, cycle time reduction for pointcloud capturing, and uncertainty estimation are made in this work. We continuously capture pointclouds at an optimal camera location with respect to part distance during robot motion execution. The redundancy in pointclouds achieved by the moving camera significantly reduces errors in measurements without increasing cycle time. Our system produces sub-millimeter accuracy.

References

1.
Malhan
,
R.
,
Joseph
,
R. J.
,
Bhatt
,
P.
,
Shah
,
B.
, and
Gupta
,
S. K.
,
2021
, “
Fast, Accurate, and Automated 3D Reconstruction Using a Depth Camera Mounted on An Industrial Robot
,”
ASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, IDETC/CIE 2021
,
Virtual
,
Aug. 17–19
,
American Society of Mechanical Engineers
.
2.
Glorieux
,
E.
,
Franciosa
,
P.
, and
Ceglarek
,
D.
,
2020
, “
Coverage Path Planning With Targetted Viewpoint Sampling for Robotic Free-Form Surface Inspection
,”
Rob. Comput.-Integr. Manuf.
,
61
, p.
101843
.
3.
Bircher
,
A.
,
Kamel
,
M.
,
Alexis
,
K.
,
Oleynikova
,
H.
, and
Siegwart
,
R.
,
2018
, “
Receding Horizon Path Planning for 3D Exploration and Surface Inspection
,”
Auton. Rob.
,
42
(
2
), pp.
291
306
.
4.
Vasquez-Gomez
,
J. I.
,
Sucar
,
L. E.
,
Murrieta-Cid
,
R.
, and
Lopez-Damian
,
E.
,
2014
, “
Volumetric Next-best-view Planning for 3D Object Reconstruction With Positioning Error
,”
Int. J. Adv. Rob. Syst.
,
11
(
10
), p.
159
.
5.
Raffaeli
,
R.
,
Mengoni
,
M.
,
Germani
,
M.
, and
Mandorli
,
F.
,
2013
, “
Off-Line View Planning for the Inspection of Mechanical Parts
,”
Int. J. Interact. Des. Manuf. (IJIDeM)
,
7
(
1
), pp.
1
12
.
6.
Jing
,
W.
,
2017
, “
Coverage Planning for Robotic Vision Applications in Complex 3D Environment
,” Ph.D. thesis,
Carnegie Mellon University
,
Pittsburgh, PA
.
7.
González-Banos
,
H.
,
2001
, “
A Randomized Art-Gallery Algorithm for Sensor Placement
,”
Proceedings of the Seventeenth Annual Symposium on Computational Geometry
,
Medford, MA
,
June 3–5
,
Association for Computational Machinery
, pp.
232
240
.
8.
Devrim Kaba
,
M.
,
Gokhan Uzunbas
,
M.
, and
Nam Lim
,
S.
,
2017
, “
A Reinforcement Learning Approach to the View Planning Problem
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Honolulu, HI
,
July 21–26
,
Institute for Electrical and Electronics Engineers
, pp.
6933
6941
.
9.
Landgraf
,
C.
,
Meese
,
B.
,
Pabst
,
M.
,
Martius
,
G.
, and
Huber
,
M. F.
,
2021
, “
A Reinforcement Learning Approach to View Planning for Automated Inspection Tasks
,”
Sensors
,
21
(
6
), p.
2030
.
10.
Almadhoun
,
R.
,
Taha
,
T.
,
Seneviratne
,
L.
,
Dias
,
J.
, and
Cai
,
G.
,
2016
, “
A Survey on Inspecting Structures Using Robotic Systems
,”
Int. J. Adv. Rob. Syst.
,
13
(
6
), p.
1729881416663664
.
11.
Dong
,
S.
,
Xu
,
K.
,
Zhou
,
Q.
,
Tagliasacchi
,
A.
,
Xin
,
S.
,
Nießner
,
M.
, and
Chen
,
B.
,
2019
, “
Multi-robot Collaborative Dense Scene Reconstruction
,”
ACM Trans. Graph.
,
38
(
4
), pp.
1
16
.
12.
Papadopoulos
,
G.
,
Kurniawati
,
H.
, and
Patrikalakis
,
N. M.
,
2013
, “
Asymptotically Optimal Inspection Planning Using Systems with Differential Constraints
,”
International Conference on Robotics and Automation
,
Karlsruhe, Germany
,
May 6–10
,
IEEE
, pp.
4126
4133
.
13.
Janoušek
,
P.
, and
Faigl
,
J.
,
2013
, “
Speeding Up Coverage Queries in 3D Multi-Goal Path Planning
,”
International Conference on Robotics and Automation
,
Karlsruhe, Germany
,
May 6–10
,
IEEE
, pp.
5082
5087
.
14.
Englot
,
B.
, and
Hover
,
F.
,
2012
, “
Sampling-Based Coverage Path Planning for Inspection of Complex Structures
,”
Proceedings of the International Conference on Automated Planning and Scheduling
,
Atibaia, São Paulo, Brazil
,
June 25–29
, AAAI Publications, Vol.
22
, p.
392
.
15.
Englot
,
B.
, and
Hover
,
F. S.
,
2012
, “
Sampling-Based Sweep Planning to Exploit Local Planarity in the Inspection of Complex 3D Structures
,”
International Conference on Intelligent Robots and Systems
,
Vilamoura, Algarve, Portugal
,
Oct. 7–12
,
IEEE/RSJ
, pp.
4456
4463
.
16.
Helsgaun
,
K.
,
2000
, “
An Effective Implementation of the Lin–Kernighan Traveling Salesman Heuristic
,”
Eur. J. Oper. Res.
,
126
(
1
), pp.
106
130
.
17.
LaValle
,
S. M.
,
Kuffner
,
J. J.
, and
Donald
,
B.
,
2001
, “
Rapidly-Exploring Random Trees: Progress and Prospects
,”
Algorithmic and Comput. Rob.: New Directions
,
5
, pp.
293
308
.
18.
Kavraki
,
L. E.
,
Svestka
,
P.
,
Latombe
,
J. -C.
, and
Overmars
,
M. H.
,
1996
, “
Probabilistic Roadmaps for Path Planning in High-Dimensional Configuration Spaces
,”
Trans. Rob. Autom.
,
12
(
4
), pp.
566
580
.
19.
Bai
,
S.
,
Chen
,
F.
, and
Englot
,
B.
,
2017
, “
Toward Autonomous Mapping and Exploration for Mobile Robots Through Deep Supervised Learning
,”
International Conference on Intelligent Robots and Systems (IROS)
,
Vancouver, BC, Canada
,
Sept. 24–28
,
IEEE
, pp.
2379
2384
.
20.
Chen
,
F.
,
Bai
,
S.
,
Shan
,
T.
, and
Englot
,
B.
,
2019
, “
Self-Learning Exploration and Mapping for Mobile Robots Via Deep Reinforcement Learning
,”
AIAA Scitech Forum
,
San Diego, CA
,
Jan. 7–11
, AAAI Publications, p.
0396
.
21.
Jain
,
R.
,
1996
, “
Building An Environment Model Using Depth Information
,”
Computer
,
22
(
06
), pp.
85
88
.
22.
Moravec
,
H.
,
1996
, “
Robot Spatial Perceptionby Stereoscopic Vision and 3d Evidence Grids
,”
Perception
.
23.
Hornung
,
A.
,
Wurm
,
K. M.
,
Bennewitz
,
M.
,
Stachniss
,
C.
, and
Burgard
,
W.
,
2012
, “
Octomap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees
,”
Auton. Rob.
,
34
, pp.
189
206
.
24.
Khan
,
S.
,
Dometios
,
A.
,
Verginis
,
C.
,
Tzafestas
,
C.
,
Wollherr
,
D.
, and
Buss
,
M.
,
2014
, “
RMAP: a Rectangular Cuboid Approximation Framework for 3D Environment Mapping
,”
Auton. Rob.
,
37
(
3
), pp.
261
277
.
25.
Hilton
,
A.
,
Stoddart
,
A. J.
,
Illingworth
,
J.
, and
Windeatt
,
T.
,
1996
, “
Reliable Surface Reconstruction From Multiple Range Images
,”
European Conference on Computer Vision
,
Cambridge, UK
,
Apr. 15–18
,
Springer
, pp.
117
126
.
26.
Curless
,
B.
, and
Levoy
,
M.
,
1996
, “
A Volumetric Method for Building Complex Models From Range Images
,”
Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques
,
New York
,
ACM
, pp.
303
312
.
27.
Wheeler
,
M. D.
,
Sato
,
Y.
, and
Ikeuchi
,
K.
,
1998
, “
Consensus Surfaces for Modeling 3D Objects From Multiple Range Images
,”
Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271)
,
Bombay, India
,
January
,
IEEE
, pp.
917
924
.
28.
Nießner
,
M.
,
Zollhöfer
,
M.
,
Izadi
,
S.
, and
Stamminger
,
M.
,
2013
, “
Real-Time 3D Reconstruction At Scale Using Voxel Hashing
,”
ACM Trans. Graph. (ToG)
,
32
(
6
), pp.
1
11
.
29.
Newcombe
,
R. A.
,
Izadi
,
S.
,
Hilliges
,
O.
,
Molyneaux
,
D.
,
Kim
,
D.
,
Davison
,
A. J.
,
Kohli
,
P.
,
Shotton
,
J.
,
Hodges
,
S.
, and
Fitzgibbon
,
A. W.
,
2011
, “
Kinectfusion: Real-Time Dense Surface Mapping and Tracking
,”
10th International Symposium on Mixed and Augmented Reality
,
Basel, Switzerland
,
IEEE
, pp.
127
136
.
30.
Mescheder
,
L.
,
Oechsle
,
M.
,
Niemeyer
,
M.
,
Nowozin
,
S.
, and
Geiger
,
A.
,
2019
, “
Occupancy Networks: Learning 3D Reconstruction in Function Space
,”
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
,
Long Beach, CA
,
June 16–20
, pp.
4460
4470
.
31.
Jiang
,
C.
,
Sud
,
A.
,
Makadia
,
A.
,
Huang
,
J.
,
Nießner
,
M.
,
Funkhouser
,
T.
,
2020
, “
Local Implicit Grid Representations for 3D Scenes
,”
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
,
Seattle, WA
,
June 13–19
, pp.
6001
6010
.
32.
Wong
,
Y.-S.
,
Li
,
C.
,
Nießner
,
M.
, and
Mitra
,
N. J.
,
2021
, “
Rigidfusion: Rgb-D Scene Reconstruction With Rigidly-Moving Objects
,”
Comput. Graph. Forum
,
40
(
2
).
33.
Božič
,
A.
,
Zollhöfer
,
M.
,
Theobalt
,
C.
, and
Nießner
,
M.
,
2020
, “
Deepdeform: Learning Non-Rigid Rgb-D Reconstruction With Semi-Supervised Data
,”
Proceedings of the Computer Vision and Pattern Recognition (CVPR)
,
Seattle, WA
,
June 13–19
,
IEEE
, pp.
7002
7012
.
34.
Spong
,
M. W.
,
Hutchinson
,
S.
, and
Vidyasagar
,
M.
,
2006
,
Robot Modeling and Control
.
35.
Malhan
,
R. K.
,
Kabir
,
A. M.
,
Shah
,
B.
, and
Gupta
,
S. K.
,
2019
, “
Identifying Feasible Workpiece Placement with Respect to Redundant Manipulator for Complex Manufacturing Tasks
,”
2019 International Conference on Robotics and Automation (ICRA)
,
Montreal, Canada
,
IEEE
, pp.
5585
5591
.
36.
Kabir
,
A. M.
,
Kanyuck
,
A.
,
Malhan
,
R. K.
,
Shembekar
,
A. V.
,
Thakar
,
S.
,
Shah
,
B. C.
, and
Gupta
,
S. K.
,
2019
, “
Generation of Synchronized Configuration Space Trajectories of Multi-Robot Systems
,”
International Conference on Robotics and Automation (ICRA)
,
Montreal, Canada
,
IEEE
, pp.
8683
8690
.
37.
Croes
,
G. A.
,
1958
, “
A Method for Solving Traveling-Salesman Problems
,”
Oper. Res.
,
6
(
6
), pp.
791
812
.
38.
Lane
,
D.
,
2003
,
Online Statistics Education: A Multimedia Course of Study
,
Association for the Advancement of Computing in Education (AACE)
.
39.
Malhan
,
R. K.
,
Shembekar
,
A. V.
,
Kabir
,
A. M.
,
Bhatt
,
P. M.
,
Shah
,
B.
,
Zanio
,
S.
,
Nutt
,
S.
, and
Gupta
,
S. K.
,
2021
, “
Automated Planning for Robotic Layup of Composite Prepreg
,”
Rob. Comput.-Integr. Manuf.
,
67
, p.
102020
.
You do not currently have access to this content.