1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> 2<!-- saved from url=(0042)https://research.nvidia.com/labs/par/calm/ --> 3<html><head><meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
3<script src="" type="text/javascript"></script>
vendor: 1 bytes, line 3
3
4<script type="text/javascript">google.load("jquery", "1.3.2");</script>
vendor: 1 bytes, line 4
4
5<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
vendor: 1 bytes, line 5
5
6<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js"></script>
6 7 8<style type="text/css"> 9body { 10 font-family: "Titillium Web", "HelveticaNeue-Light", "Helvetica Neue Light", "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; 11 font-weight: 300; 12 font-size: 17px; 13 margin-left: auto; 14 margin-right: auto; 15 width: 980px; 16} 17h1 { 18 font-weight:300; 19 line-height: 1.15em; 20} 21 22h2 { 23 font-size: 1.75em; 24} 25a:link,a:visited { 26 color: #1367a7; 27 text-decoration: none; 28} 29a:hover { 30 color: #208799; 31} 32h1, h2, h3 { 33 text-align: center; 34} 35h1 { 36 font-size: 40px; 37 font-weight: 500; 38} 39h2 { 40 font-weight: 400; 41 margin: 16px 0px 4px 0px; 42} 43.paper-title { 44 padding: 16px 0px 16px 0px; 45} 46section { 47 margin: 32px 0px 32px 0px; 48 text-align: justify; 49 clear: both; 50} 51.col-6 { 52 width: 16.6%; 53 float: left; 54} 55.col-5 { 56 width: 20%; 57 float: left; 58} 59.col-4 { 60 width: 25%; 61 float: left; 62} 63.col-3 { 64 width: 33%; 65 float: left; 66} 67.col-2 { 68 width: 50%; 69 float: left; 70} 71.row, .author-row, .affil-row { 72 overflow: auto; 73} 74.author-row, .affil-row { 75 font-size: 20px; 76} 77.row { 78 margin: 16px 0px 16px 0px; 79} 80.authors { 81 font-size: 18px; 82} 83.affil-row { 84 margin-top: 16px; 85} 86.teaser { 87 max-width: 100%; 88} 89.text-center { 90 text-align: center; 91} 92.screenshot { 93 width: 256px; 94 border: 1px solid #ddd; 95} 96.screenshot-el { 97 margin-bottom: 16px; 98} 99hr { 100 height: 1px; 101 border: 0; 102 border-top: 1px solid #ddd; 103 margin: 0; 104} 105.material-icons { 106 vertical-align: -6px; 107} 108p { 109 line-height: 1.25em; 110} 111.caption { 112 font-size: 16px; 113 /*font-style: italic;*/ 114 color: #666; 115 text-align: left; 116 margin-top: 8px; 117 margin-bottom: 8px; 118} 119video { 120 display: block; 121 margin: auto; 122} 123figure { 124 display: block; 125 margin: auto; 126 margin-top: 10px; 127 margin-bottom: 10px; 128} 129#bibtex pre { 130 font-size: 14px; 131 background-color: #eee; 132 padding: 16px; 133} 134.blue { 135 color: #2c82c9; 136 font-weight: bold; 137} 138.orange { 139 color: #d35400; 140 font-weight: bold; 141} 142.flex-row { 143 display: flex; 144 flex-flow: row wrap; 145 justify-content: space-around; 146 padding: 0; 147 margin: 0; 148 list-style: none; 149} 150.paper-btn { 151 position: relative; 152 text-align: center; 153 154 display: inline-block; 155 margin: 8px; 156 padding: 8px 8px; 157 158 border-width: 0; 159 outline: none; 160 border-radius: 2px; 161 162 background-color: #1367a7; 163 color: #ecf0f1 !important; 164 font-size: 20px; 165 width: 100px; 166 font-weight: 600; 167} 168 169.supp-btn { 170 position: relative; 171 text-align: center; 172 173 display: inline-block; 174 margin: 8px; 175 padding: 8px 8px; 176 177 border-width: 0; 178 outline: none; 179 border-radius: 2px; 180 181 background-color: #1367a7; 182 color: #ecf0f1 !important; 183 font-size: 20px; 184 width: 150px; 185 font-weight: 600; 186} 187 188.paper-btn-parent { 189 display: flex; 190 justify-content: center; 191 margin: 16px 0px; 192} 193.paper-btn:hover { 194 opacity: 0.85; 195} 196.container { 197 margin-left: auto; 198 margin-right: auto; 199 padding-left: 16px; 200 padding-right: 16px; 201} 202.venue { 203 color: #1367a7; 204} 205 206.kitti { 207 display: flex; 208 justify-content: center; 209 margin: 16px 0px; 210} 211 212.topnav { 213 overflow: hidden; 214 background-color: #EEEEEE; 215} 216 217.topnav a { 218 float: left; 219 color: black; 220 text-align: center; 221 padding: 14px 16px; 222 text-decoration: none; 223 font-size: 16px; 224} 225 226 227 228</style> 229 230 231</head><body><div class="topnav" id="myTopnav"> 232 <a href="https://www.nvidia.com/"><img width="100%" src="./files/nvidia.svg"></a> 233 <a href="https://www.nvidia.com/en-us/research/"><strong>NVIDIA Research</strong></a> 234</div> 235 236<!-- End : Google Analytics Code -->
237<script type="text/javascript" src="./files/style/hidebib.js"></script>
237 238<link href="./files/style/css" rel="stylesheet" type="text/css"> 239 240 <title>Point-Cloud Completion with Pretrained Text-to-image Diffusion Models</title> 241 <meta property="og:description" content="-Cloud Completion with Pretrained Text-to-image Diffusion Models"> 242 <link href="./files/style/css2" rel="stylesheet"> 243 244<!-- Global site tag (gtag.js) - Google Analytics -->
245<script async="" src="./files/style/js"></script>
vendor: 1 bytes, line 245
245
246<script> 247
vendor: 139 bytes, lines 247-250
247window.dataLayer = window.dataLayer || []; 248 function gtag(){dataLayer.push(arguments);} 249 gtag('js', new Date()); 250 gtag('config', '
250G-6HHDEXF452
vendor: 4 bytes, line 250
250');
251</script>
251 252 253 254 255 256 257<div class="container"> 258 <div class="paper-title"> 259 <h1>Point-Cloud Completion with Pretrained Text-to-image Diffusion Models</h1> 260 </div> 261 262 263 <div id="authors"> 264 <div class="author-row"> 265 <div class="col-3 text-center"><a href="https://ykasten.github.io/">Yoni Kasten</a><sup>1</sup></div> 266 <div class="col-3 text-center"><a href="https://ohad204.github.io/ohadrahamim.github.io/">Ohad Rahamim</a><sup>2</sup></div> 267 <div class="col-3 text-center"><a href="https://research.nvidia.com/person/gal-chechik">Gal Chechik</a><sup>1,2</sup></div> 268 </div> 269 270 <div class="affil-row"> 271 <div class="venue text-center"><sup>1</sup>NVIDIA</div> 272 </div> 273 <div class="affil-row"> 274 <div class="venue text-center"><sup>2</sup>Bar-Ilan University</div> 275 </div> 276 <div class="affil-row"> 277 <div class="venue text-center"><b>NeurIPS 2023</b></div> 278 </div> 279 280 <div style="clear: both"> 281 <div class="paper-btn-parent"> 282 <a class="supp-btn" href="https://arxiv.org/pdf/2306.10533.pdf"> 283 <span class="material-icons"> </span> 284 Paper 285 </a> 286 <a class="supp-btn" href="https://github.com/NVlabs/sds-complete"> 287 <span class="material-icons"> </span> 288 Code 289 </a> 290 </div></div> 291 </div> 292 293 <section id="teaser"> 294 <figure style="width: 100%;"> 295 <a href="./files/teaser.png"> 296 <img width="100%" src="./files/teaser.png"> 297 </a> 298 <p class="caption" style="margin-bottom: 1px;"> 299 We present SDS-Complete: A test-time optimization method for completing point clouds captured by depth sensors, leveraging pre-trained text-to-image diffusion model. The inputs to our method are an incomplete point cloud (blue) along with a textual description of the object. The output is a complete surface (gray) that is consistent with the input points (blue). The method works well on a variety of objects captured by real-world point-cloud sensors. 300 </p> 301 </figure> 302 </section> 303 304 <section id="abstract"> 305 <h2>Abstract</h2> 306 <hr> 307 <p> 308 Point-cloud data collected in real-world applications are often incomplete, because objects are being observed from specific viewpoints, which only capture one perspective. Data can also be incomplete due to occlusion and low-resolution sampling. Existing approaches to completion rely on training models with datasets of predefined objects to guide the completion of point clouds. Unfortunately, these approaches fail to generalize when tested on objects or real-world setups that are poorly represented in their training set. Here, we leverage recent advances in text-guided 3D shape generation, showing how to use image priors for generating 3D objects. We describe an approach called SDS-Complete that uses a pre-trained text-to-image diffusion model and leverages the text semantics of a given incomplete point cloud of an object, to obtain a complete surface representation. SDS-Complete can complete a variety of objects using test-time optimization without expensive collection of 3D data. We evaluate SDS-Complete on a collection of incomplete scanned objects, captured by real-world depth sensors and LiDAR scanners. We find that it effectively reconstructs objects that are absent from common datasets, reducing Chamfer loss by about 50% on average compared with current methods. 309 </p> 310 </section> 311 312 <hr> 313 <h1>Overview</h1> 314 <hr> 315 316 <section id="teaser-videos"> 317 <div class="flex-row"> 318 <figure style="width: 100%;"> 319 <video width="90%" controls="" muted="" loop="" autoplay=""> 320 <source src="./files/00034.mp4" type="video/mp4"> 321 Your browser does not support the video tag. 322 </video> 323 </figure> 324 <div style="width: 100%;"> 325 <br><br> 326 <p> 327 Here, we address the challenge of completing 3D objects in the wild from real-world partial point 328 clouds. This is achieved by leveraging priors about object shapes that are encoded in pretrained 329 text-to-image diffusion models. Our key idea is that since text-to-image diffusion models were trained 330 on a vast number of diverse objects, they contain a strong prior about the shape and texture of objects,
331 and that prior can be used for completing object missing parts. For example, given a partial point 332 cloud, knowing that it corresponds to a chair can guide the completion process, because objects from 333 this class are expected to exhibit some types of symmetries and parts that are captured in 2D images. 334 </p> 335 336 <p> 337 A similar intuition has been used for generating 3D objects âfrom scratch" (DreamFusion). 338 DreamFusion uses the SDS loss, which measures agreement between 2D model prior and renderings 339 of the 3D shape. Unfortunately, naively applying the SDS loss to our problem of point cloud 340 completion fails. This is because, as we show below, it does not combine well the hard constraints 341 implied by the points collected from the sensor with the prior embedded in the diffusion model. 342 </p> 343 344 <p> 345 To address these challenges, we introduce SDS-Complete: a method to complete a given partial 346point cloud using several considerations. First, we use a Signed Distance Function (SDF) surface 347representation, and constrain the zero level set of the SDF to go through the input 348points. Second, we use information about areas with no collected points, to rule out object parts 349in these areas. Third, we use a prior about camera position and orientation and a curriculum of 350out-painting when sampling camera positions. Finally, we use the SDS loss to incorporate prior 351guided by the class of an object on the rendered images. 352 </p> 353 354 <p>
355 We demonstrate that SDS-Complete generates completions for various objects with different shape 356types from two real-world datasets: the Redwood dataset, which contains various incomplete 357real-world depth camera partial scans, and the KITTI dataset, which contains object LiDAR 358scans from driving scenarios. In both cases, SDS-Complete outperforms the current state-of-the-art 359methods. 360 </p> 361 </div> 362 </div> 363 </section> 364 365 <section id="pipeline"> 366 367 <h2>Pipeline</h2> 368 <hr> 369 370 <figure style="width: 100%;"> 371 <video width="90%" controls="" muted="" loop="" autoplay=""> 372 <source src="./files/SDS-Complete_pipeline_video.mp4" type="video/mp4"> 373 Your browser does not support the video tag. 374 </video> 375 <p class="caption" style="margin-bottom: 1px;"> 376 <div class="mathjax"> 377 378 379 SDS-loss optimizes two neural functions: A signed distance function \(f_\theta\) representing the surface and a volumetric coloring function \(\mathbf{c}_\varphi\). Together, \((\mathbf{c}_\varphi,f_\theta)\) define a radiance field, which is used to render novel image views \(Im_0,\ldots Im_n\). The SDS loss is applied to the renderings to encourage them to be compatible with the input text \(\mathbf{y}\). Three sensor-compatibility losses verify that the reconstructed surface is compatible with the sensor observations in various aspects. </p> 380</div> 381 </p> 382 </figure> 383 384 <h2>Results</h2> 385 <hr> 386 Our primary goal is to evaluate our SDS-Complete and baseline method in real-world scenarios. This is in contrast to evaluating test splits from the synthetic datasets that were used for training the baseline methods. To achieve relevant evaluation datasets, we based the evaluation on partial real-world point clouds obtained from depth images and LiDAR scans. 387 <h2>Redwood dataset</h2> 388 <hr> 389 390 <section id="teaser"> 391 <figure style="width: 100%;"> 392 <a href="./Example_titles.png"> 393 <img width="100%" src="./files/Example_titles.png"> 394 </a> 395 </figure> 396 </section> 397 398 <figure style="display: inline; width: 33%;"> 399 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 400 <source src="./files/01184.mp4" type="video/mp4"> 401 Your browser does not support the video tag. 402 </video> 403 </figure> 404 <figure style="display: inline; width: 49%;"> 405 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 406 <source src="./files/04797.mp4" type="video/mp4"> 407 Your browser does not support the video tag. 408 </video> 409 </figure> 410 <figure style="display: inline; width: 33%;"> 411 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 412 <source src="./files/09484.mp4" type="video/mp4"> 413 Your browser does not support the video tag. 414 </video> 415 </figure> 416 417 418 <figure style="display: inline; width: 33%;"> 419 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 420 <source src="./files/06127.mp4" type="video/mp4"> 421 Your browser does not support the video tag. 422 </video> 423 </figure> 424 <figure style="display: inline; width: 33%;"> 425 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 426 <source src="./files/06145.mp4" type="video/mp4"> 427 Your browser does not support the video tag. 428 </video> 429 </figure> 430 431 432 <figure style="display: inline; width: 33%;"> 433 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 434 <source src="./files/07306.mp4" type="video/mp4"> 435 Your browser does not support the video tag. 436 </video> 437 </figure> 438 <figure style="display: inline; width: 33%;"> 439 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 440 <source src="./files/09639.mp4" type="video/mp4"> 441 Your browser does not support the video tag. 442 </video> 443 </figure> 444 445 <figure style="display: inline; width: 33%;"> 446 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 447 <source src="./files/09424.mp4" type="video/mp4"> 448 Your browser does not support the video tag. 449 </video> 450 </figure> 451 <figure style="display: inline; width: 33%;"> 452 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 453 <source src="./files/04154.mp4" type="video/mp4"> 454 Your browser does not support the video tag. 455 </video> 456 </figure> 457 <figure style="display: inline; width: 33%;"> 458 <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay=""> 459 <source src="./files/04457.mp4" type="video/mp4"> 460 Your browser does not support the video tag. 461 </video> 462 463 <p class="caption" style="margin-bottom: 1px;"> 464 Example arranged in two columns, where each column has the following structure: 465 Left: The partial scans that are given as input to our model. Middle: the completed surface. Right: the completed surface together with the input points. 466 For more examples, please refer to the paper. 467 </p> 468 </figure> 469 470 471 <h2>KITTI dataset</h2> 472 <hr> 473 474 <figure style="display: inline; width: 33%;"> 475 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
476 <source src="./files/KITTI/motorcycle5_ours.mp4" type="video/mp4"> 477 Your browser does not support the video tag. 478 </video> 479 </figure> 480 <figure style="display: inline; width: 33%;"> 481 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 482 <source src="./files/KITTI/car_ours.mp4" type="video/mp4"> 483 Your browser does not support the video tag. 484 </video> 485 </figure> 486 <figure style="display: inline; width: 33%;"> 487 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 488 <source src="./files/KITTI/motorcycle7_ours.mp4" type="video/mp4"> 489 Your browser does not support the video tag. 490 </video> 491 </figure> 492 493 <figure style="display: inline; width: 33%;"> 494 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 495 <source src="./files/KITTI/car2_ours.mp4" type="video/mp4"> 496 Your browser does not support the video tag. 497 </video> 498 </figure> 499 <figure style="display: inline; width: 33%;"> 500 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 501 <source src="./files/KITTI/motorcycle10_ours.mp4" type="video/mp4"> 502 Your browser does not support the video tag. 503 </video> 504 </figure> 505 <figure style="display: inline; width: 33%;"> 506 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 507 <source src="./files/KITTI/truck_ours.mp4" type="video/mp4"> 508 Your browser does not support the video tag. 509 </video> 510 </figure> 511 512 <figure style="display: inline; width: 33%;"> 513 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 514 <source src="./files/KITTI/truck7_ours.mp4" type="video/mp4"> 515 Your browser does not support the video tag. 516 </video> 517 </figure> 518 <figure style="display: inline; width: 33%;"> 519 <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay=""> 520 <source src="./files/KITTI/truck13_ours.mp4" type="video/mp4"> 521 Your browser does not support the video tag. 522 </video> 523 524 <p class="caption" style="margin-bottom: 1px;"> 525 The completed surface (gray) together with the input points (blue). 526 </p> 527 </figure> 528 529 </section> 530 531 <section id="bibtex"> 532 <h2>Citation</h2> 533 <pre><code> 534@article{kasten2023point, 535 title={Point-Cloud Completion with Pretrained Text-to-image Diffusion Models}, 536 author={Kasten, Yoni and Rahamim, Ohad and Chechik, Gal}, 537 journal={arXiv preprint arXiv:2306.10533}, 538 year={2023} 539} 540 </code></pre> 541 </section> 542 543 544 545<br> 546 547 548</div> 549 550 551</body></html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.