PageSourceSearch

https://sds-complete.github.io/

html sds-complete.github.io collected 2026-10-03 09:55:16 UTC 20,343 bytes, 551 lines download raw bytes

1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<!-- saved from url=(0042)https://research.nvidia.com/labs/par/calm/ -->
3<html><head><meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
3<script src="" type="text/javascript"></script>
vendor: 1 bytes, line 3
3
4<script type="text/javascript">google.load("jquery", "1.3.2");</script>
vendor: 1 bytes, line 4
4
5<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
vendor: 1 bytes, line 5
5
6<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js"></script>
6
7
8<style type="text/css">
9body {
10    font-family: "Titillium Web", "HelveticaNeue-Light", "Helvetica Neue Light", "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif;
11    font-weight: 300;
12    font-size: 17px;
13    margin-left: auto;
14    margin-right: auto;
15    width: 980px;
16}
17h1 {
18    font-weight:300;
19    line-height: 1.15em;
20}
21
22h2 {
23    font-size: 1.75em;
24}
25a:link,a:visited {
26    color: #1367a7;
27    text-decoration: none;
28}
29a:hover {
30    color: #208799;
31}
32h1, h2, h3 {
33    text-align: center;
34}
35h1 {
36    font-size: 40px;
37    font-weight: 500;
38}
39h2 {
40    font-weight: 400;
41    margin: 16px 0px 4px 0px;
42}
43.paper-title {
44    padding: 16px 0px 16px 0px;
45}
46section {
47    margin: 32px 0px 32px 0px;
48    text-align: justify;
49    clear: both;
50}
51.col-6 {
52     width: 16.6%;
53     float: left;
54}
55.col-5 {
56     width: 20%;
57     float: left;
58}
59.col-4 {
60     width: 25%;
61     float: left;
62}
63.col-3 {
64     width: 33%;
65     float: left;
66}
67.col-2 {
68     width: 50%;
69     float: left;
70}
71.row, .author-row, .affil-row {
72     overflow: auto;
73}
74.author-row, .affil-row {
75    font-size: 20px;
76}
77.row {
78    margin: 16px 0px 16px 0px;
79}
80.authors {
81    font-size: 18px;
82}
83.affil-row {
84    margin-top: 16px;
85}
86.teaser {
87    max-width: 100%;
88}
89.text-center {
90    text-align: center;  
91}
92.screenshot {
93    width: 256px;
94    border: 1px solid #ddd;
95}
96.screenshot-el {
97    margin-bottom: 16px;
98}
99hr {
100    height: 1px;
101    border: 0; 
102    border-top: 1px solid #ddd;
103    margin: 0;
104}
105.material-icons {
106    vertical-align: -6px;
107}
108p {
109    line-height: 1.25em;
110}
111.caption {
112    font-size: 16px;
113    /*font-style: italic;*/
114    color: #666;
115    text-align: left;
116    margin-top: 8px;
117    margin-bottom: 8px;
118}
119video {
120    display: block;
121    margin: auto;
122}
123figure {
124    display: block;
125    margin: auto;
126    margin-top: 10px;
127    margin-bottom: 10px;
128}
129#bibtex pre {
130    font-size: 14px;
131    background-color: #eee;
132    padding: 16px;
133}
134.blue {
135    color: #2c82c9;
136    font-weight: bold;
137}
138.orange {
139    color: #d35400;
140    font-weight: bold;
141}
142.flex-row {
143    display: flex;
144    flex-flow: row wrap;
145    justify-content: space-around;
146    padding: 0;
147    margin: 0;
148    list-style: none;
149}
150.paper-btn {
151  position: relative;
152  text-align: center;
153
154  display: inline-block;
155  margin: 8px;
156  padding: 8px 8px;
157
158  border-width: 0;
159  outline: none;
160  border-radius: 2px;
161  
162  background-color: #1367a7;
163  color: #ecf0f1 !important;
164  font-size: 20px;
165  width: 100px;
166  font-weight: 600;
167}
168
169.supp-btn {
170  position: relative;
171  text-align: center;
172
173  display: inline-block;
174  margin: 8px;
175  padding: 8px 8px;
176
177  border-width: 0;
178  outline: none;
179  border-radius: 2px;
180  
181  background-color: #1367a7;
182  color: #ecf0f1 !important;
183  font-size: 20px;
184  width: 150px;
185  font-weight: 600;
186}
187
188.paper-btn-parent {
189    display: flex;
190    justify-content: center;
191    margin: 16px 0px;
192}
193.paper-btn:hover {
194    opacity: 0.85;
195}
196.container {
197    margin-left: auto;
198    margin-right: auto;
199    padding-left: 16px;
200    padding-right: 16px;
201}
202.venue {
203    color: #1367a7;
204}
205
206.kitti {
207    display: flex;
208    justify-content: center;
209    margin: 16px 0px;
210}
211
212.topnav {
213  overflow: hidden;
214  background-color: #EEEEEE;
215}
216
217.topnav a {
218  float: left;
219  color: black;
220  text-align: center;
221  padding: 14px 16px;
222  text-decoration: none;
223  font-size: 16px;
224}
225
226
227
228</style>
229
230
231</head><body><div class="topnav" id="myTopnav">
232  <a href="https://www.nvidia.com/"><img width="100%" src="./files/nvidia.svg"></a>
233  <a href="https://www.nvidia.com/en-us/research/"><strong>NVIDIA Research</strong></a>
234</div>
235
236<!-- End : Google Analytics Code -->
237<script type="text/javascript" src="./files/style/hidebib.js"></script>
237
238<link href="./files/style/css" rel="stylesheet" type="text/css">
239
240    <title>Point-Cloud Completion with Pretrained Text-to-image Diffusion Models</title>
241    <meta property="og:description" content="-Cloud Completion with Pretrained Text-to-image Diffusion Models">
242    <link href="./files/style/css2" rel="stylesheet">
243
244<!-- Global site tag (gtag.js) - Google Analytics -->
245<script async="" src="./files/style/js"></script>
vendor: 1 bytes, line 245
245
246<script>
247    
vendor: 139 bytes, lines 247-250
247window.dataLayer = window.dataLayer || [];
248    function gtag(){dataLayer.push(arguments);}
249    gtag('js', new Date());
250    gtag('config', '
250G-6HHDEXF452
vendor: 4 bytes, line 250
250');
251</script>
251
252
253
254
255
256 
257<div class="container">
258    <div class="paper-title">
259      <h1>Point-Cloud Completion with Pretrained Text-to-image Diffusion Models</h1>
260    </div>
261
262    
263    <div id="authors">
264        <div class="author-row">
265            <div class="col-3 text-center"><a href="https://ykasten.github.io/">Yoni Kasten</a><sup>1</sup></div>
266            <div class="col-3 text-center"><a href="https://ohad204.github.io/ohadrahamim.github.io/">Ohad Rahamim</a><sup>2</sup></div>
267            <div class="col-3 text-center"><a href="https://research.nvidia.com/person/gal-chechik">Gal Chechik</a><sup>1,2</sup></div>
268        </div>
269
270        <div class="affil-row">
271            <div class="venue text-center"><sup>1</sup>NVIDIA</div>
272        </div>
273        <div class="affil-row">
274            <div class="venue text-center"><sup>2</sup>Bar-Ilan University</div>
275        </div>
276        <div class="affil-row">
277            <div class="venue text-center"><b>NeurIPS 2023</b></div>
278        </div>
279
280        <div style="clear: both">
281            <div class="paper-btn-parent">
282            <a class="supp-btn" href="https://arxiv.org/pdf/2306.10533.pdf">
283                <span class="material-icons">  </span> 
284                 Paper
285            </a>
286            <a class="supp-btn" href="https://github.com/NVlabs/sds-complete">
287                <span class="material-icons">  </span> 
288                  Code 
289            </a>
290        </div></div>
291    </div>
292
293    <section id="teaser">
294            <figure style="width: 100%;">
295                <a href="./files/teaser.png">
296                    <img width="100%" src="./files/teaser.png">
297                </a>
298                <p class="caption" style="margin-bottom: 1px;">
299                    We present SDS-Complete: A test-time optimization method for completing point clouds captured by depth sensors, leveraging pre-trained text-to-image diffusion model. The inputs to our method are an incomplete point cloud (blue) along with a textual description of the object. The output is a complete surface (gray) that is consistent with the input points (blue). The method works well on a variety of objects captured by real-world point-cloud sensors.
300                </p>
301            </figure>
302    </section>
303
304    <section id="abstract">
305        <h2>Abstract</h2>
306        <hr>
307        <p>
308            Point-cloud data collected in real-world applications are often incomplete, because objects are being observed from specific viewpoints, which only capture one perspective. Data can also be incomplete due to occlusion and low-resolution sampling. Existing approaches to completion rely on training models with datasets of predefined objects to guide the completion of point clouds. Unfortunately, these approaches fail to generalize when tested on objects or real-world setups that are poorly represented in their training set. Here, we leverage recent advances in text-guided 3D shape generation, showing how to use image priors for generating 3D objects. We describe an approach called SDS-Complete that uses a pre-trained text-to-image diffusion model and leverages the text semantics of a given incomplete point cloud of an object, to obtain a complete surface representation. SDS-Complete can complete a variety of objects using test-time optimization without expensive collection of 3D data. We evaluate SDS-Complete on a collection of incomplete scanned objects, captured by real-world depth sensors and LiDAR scanners. We find that it effectively reconstructs objects that are absent from common datasets, reducing Chamfer loss by about 50% on average compared with current methods.
309        </p>
310    </section>
311
312    <hr>
313    <h1>Overview</h1>
314    <hr>
315
316    <section id="teaser-videos">
317        <div class="flex-row">
318        <figure style="width: 100%;">
319            <video width="90%" controls="" muted="" loop="" autoplay="">
320                <source src="./files/00034.mp4" type="video/mp4">
321                Your browser does not support the video tag.
322            </video>
323        </figure>
324            <div style="width: 100%;">
325                <br><br>
326                <p>
327                    Here, we address the challenge of completing 3D objects in the wild from real-world partial point
328                    clouds. This is achieved by leveraging priors about object shapes that are encoded in pretrained
329                    text-to-image diffusion models. Our key idea is that since text-to-image diffusion models were trained
330                    on a vast number of diverse objects, they contain a strong prior about the shape and texture of objects,
331                    and that prior can be used for completing object missing parts. For example, given a partial point
332                    cloud, knowing that it corresponds to a chair can guide the completion process, because objects from
333                    this class are expected to exhibit some types of symmetries and parts that are captured in 2D images.
334                </p> 
335
336                <p>
337                    A similar intuition has been used for generating 3D objects “from scratch" (DreamFusion).
338                    DreamFusion uses the SDS loss, which measures agreement between 2D model prior and renderings
339                    of the 3D shape. Unfortunately, naively applying the SDS loss to our problem of point cloud
340                    completion fails. This is because, as we show below, it does not combine well the hard constraints
341                    implied by the points collected from the sensor with the prior embedded in the diffusion model.
342                </p>
343                
344                                <p>
345                                    To address these challenges, we introduce SDS-Complete: a method to complete a given partial
346point cloud using several considerations. First, we use a Signed Distance Function (SDF) surface
347representation, and constrain the zero level set of the SDF to go through the input
348points. Second, we use information about areas with no collected points, to rule out object parts
349in these areas. Third, we use a prior about camera position and orientation and a curriculum of
350out-painting when sampling camera positions. Finally, we use the SDS loss to incorporate prior
351guided by the class of an object on the rendered images.
352                </p>   
353                
354                <p>
355                    We demonstrate that SDS-Complete generates completions for various objects with different shape
356types from two real-world datasets: the Redwood dataset, which contains various incomplete
357real-world depth camera partial scans, and the KITTI dataset, which contains object LiDAR
358scans from driving scenarios. In both cases, SDS-Complete outperforms the current state-of-the-art
359methods.
360                </p>
361            </div>
362        </div>
363    </section>
364
365    <section id="pipeline">
366
367        <h2>Pipeline</h2>
368        <hr>
369
370        <figure style="width: 100%;">
371            <video width="90%" controls="" muted="" loop="" autoplay="">
372                <source src="./files/SDS-Complete_pipeline_video.mp4" type="video/mp4">
373                Your browser does not support the video tag.
374            </video>
375            <p class="caption" style="margin-bottom: 1px;">
376                <div class="mathjax">
377    
378
379    SDS-loss optimizes two neural functions: A signed distance function \(f_\theta\) representing the surface and a volumetric coloring function \(\mathbf{c}_\varphi\). Together, \((\mathbf{c}_\varphi,f_\theta)\) define a radiance field, which is used to render novel image views \(Im_0,\ldots Im_n\). The SDS loss is applied to the renderings to encourage them to be compatible with the input text \(\mathbf{y}\). Three sensor-compatibility losses verify that the reconstructed surface is compatible with the sensor observations in various aspects. </p>
380</div>
381            </p>
382        </figure>
383
384        <h2>Results</h2>
385        <hr>
386        Our primary goal is to evaluate our SDS-Complete and baseline method in real-world scenarios. This is in contrast to evaluating test splits from the synthetic datasets that were used for training the baseline methods. To achieve relevant evaluation datasets, we based the evaluation on partial real-world point clouds obtained from depth images and LiDAR scans.
387        <h2>Redwood dataset</h2>
388        <hr>
389        
390        <section id="teaser">
391            <figure style="width: 100%;">
392                <a href="./Example_titles.png">
393                    <img width="100%" src="./files/Example_titles.png">
394                </a>
395            </figure>
396        </section>
397
398        <figure style="display: inline; width: 33%;">
399            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
400                <source src="./files/01184.mp4" type="video/mp4">
401                Your browser does not support the video tag.
402            </video>
403        </figure>
404        <figure style="display: inline; width: 49%;">
405            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
406                <source src="./files/04797.mp4" type="video/mp4">
407                Your browser does not support the video tag.
408            </video>
409        </figure>
410        <figure style="display: inline; width: 33%;">
411            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
412                <source src="./files/09484.mp4" type="video/mp4">
413                Your browser does not support the video tag.
414            </video>
415        </figure>
416        
417
418        <figure style="display: inline; width: 33%;">
419            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
420                <source src="./files/06127.mp4" type="video/mp4">
421                Your browser does not support the video tag.
422            </video>
423        </figure>
424        <figure style="display: inline; width: 33%;">
425            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
426                <source src="./files/06145.mp4" type="video/mp4">
427                Your browser does not support the video tag.
428            </video>
429        </figure>
430        
431
432        <figure style="display: inline; width: 33%;">
433            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
434                <source src="./files/07306.mp4" type="video/mp4">
435                Your browser does not support the video tag.
436            </video>
437        </figure>
438        <figure style="display: inline; width: 33%;">
439            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
440                <source src="./files/09639.mp4" type="video/mp4">
441                Your browser does not support the video tag.
442            </video>
443        </figure>
444        
445        <figure style="display: inline; width: 33%;">
446            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
447                <source src="./files/09424.mp4" type="video/mp4">
448                Your browser does not support the video tag.
449            </video>
450        </figure>
451        <figure style="display: inline; width: 33%;">
452            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
453                <source src="./files/04154.mp4" type="video/mp4">
454                Your browser does not support the video tag.
455            </video>
456        </figure>
457        <figure style="display: inline; width: 33%;">
458            <video style="display: inline;" width="49.5%" controls="" muted="" loop="" autoplay="">
459                <source src="./files/04457.mp4" type="video/mp4">
460                Your browser does not support the video tag.
461            </video>
462            
463            <p class="caption" style="margin-bottom: 1px;">
464            Example arranged in two columns, where each column has the following structure: 
465                Left: The partial scans that are given as input to our model. Middle: the completed surface. Right: the completed surface together with the input points.
466                 For more examples, please refer to the paper.
467            </p>
468        </figure>
469        
470        
471        <h2>KITTI dataset</h2>
472        <hr>
473        
474        <figure style="display: inline; width: 33%;">
475            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
476                <source src="./files/KITTI/motorcycle5_ours.mp4" type="video/mp4">
477                Your browser does not support the video tag.
478            </video>
479        </figure>
480        <figure style="display: inline; width: 33%;">
481            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
482                <source src="./files/KITTI/car_ours.mp4" type="video/mp4">
483                Your browser does not support the video tag.
484            </video>
485        </figure>
486        <figure style="display: inline; width: 33%;">
487            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
488                <source src="./files/KITTI/motorcycle7_ours.mp4" type="video/mp4">
489                Your browser does not support the video tag.
490            </video>
491        </figure>
492
493        <figure style="display: inline; width: 33%;">
494            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
495                <source src="./files/KITTI/car2_ours.mp4" type="video/mp4">
496                Your browser does not support the video tag.
497            </video>
498        </figure>
499        <figure style="display: inline; width: 33%;">
500            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
501                <source src="./files/KITTI/motorcycle10_ours.mp4" type="video/mp4">
502                Your browser does not support the video tag.
503            </video>
504        </figure>
505        <figure style="display: inline; width: 33%;">
506            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
507                <source src="./files/KITTI/truck_ours.mp4" type="video/mp4">
508                Your browser does not support the video tag.
509            </video>
510        </figure>
511        
512        <figure style="display: inline; width: 33%;">
513            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
514                <source src="./files/KITTI/truck7_ours.mp4" type="video/mp4">
515                Your browser does not support the video tag.
516            </video>
517        </figure>
518        <figure style="display: inline; width: 33%;">
519            <video style="display: inline;" width="24%" controls="" muted="" loop="" autoplay="">
520                <source src="./files/KITTI/truck13_ours.mp4" type="video/mp4">
521                Your browser does not support the video tag.
522            </video>
523            
524                        <p class="caption" style="margin-bottom: 1px;">
525                The completed surface (gray) together with the input points (blue).
526            </p>
527        </figure>
528
529    </section>
530    
531    <section id="bibtex">
532        <h2>Citation</h2>
533        <pre><code>
534@article{kasten2023point,
535  title={Point-Cloud Completion with Pretrained Text-to-image Diffusion Models},
536  author={Kasten, Yoni and Rahamim, Ohad and Chechik, Gal},
537  journal={arXiv preprint arXiv:2306.10533},
538  year={2023}
539}
540        </code></pre>
541    </section>
542
543    
544
545<br>
546
547
548</div>
549
550
551</body></html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.