1<!DOCTYPE html> 2<html> 3 4<head> 5 <!-- Google tag (gtag.js) -->
vendor: 66 bytes, lines 5-6
5 6 <script async src="https://www.googletagmanager.com/gtag/js?id=
6G-55V1E709SK
vendor: 14 bytes, lines 6-7
6"></script> 7
7<script> 8
vendor: 140 bytes, lines 8-12
8window.dataLayer = window.dataLayer || []; 9 function gtag(){dataLayer.push(arguments);} 10 gtag('js', new Date()); 11 12 gtag('config', '
12G-55V1E709SK
vendor: 6 bytes, lines 12-13
12'); 13
13</script>
13 14 <meta charset="utf-8"> 15 <meta name="description" 16 content="ReconX, unleashes the strong generative prior of video diffusion model for sparse-view 3D scene reconstruction. "> 17 <meta name="keywords" content="ReconX, Gaussian Splatting, Video Diffusion Model"> 18 <meta name="viewport" content="width=device-width, initial-scale=1"> 19 <title>ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model 20 </title> 21 22 <!-- Global site tag (gtag.js) - Google Analytics -->
vendor: 66 bytes, lines 22-23
22 23 <script async src="https://www.googletagmanager.com/gtag/js?id=
23G-PYVRSFMDRL
vendor: 14 bytes, lines 23-24
23"></script> 24
24<script> 25
vendor: 155 bytes, lines 25-33
25window.dataLayer = window.dataLayer || []; 26 27 function gtag() { 28 dataLayer.push(arguments); 29 } 30 31 gtag('js', new Date()); 32 33 gtag('config', '
33G-PYVRSFMDRL
vendor: 6 bytes, lines 33-34
33'); 34
34</script>
34 35 36 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> 37 38 <link rel="stylesheet" href="./static/css/bulma.min.css"> 39 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 40 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 41 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 42 <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 43 <link rel="stylesheet" href="./static/css/index.css"> 44 45
45<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
45 46
46<script defer src="./static/js/fontawesome.all.min.js"></script>
46 47
47<script src="./static/js/bulma-carousel.min.js"></script>
47 48
48<script src="./static/js/bulma-slider.min.js"></script>
48 49
49<script src="./static/js/index.js"></script>
49 50
50<script src="./static/js/script.js"></script>
50 51 52</head> 53 54<body> 55 56 57 58 <section class="hero"> 59 <div class="hero-body"> 60 <div class="container is-max-desktop"> 61 <div class="columns is-centered"> 62 <div class="column has-text-centered"> 63 <h1 class="title is-1 publication-title"><span 64 style="color: #48a7b5 ;font-weight: bolder;">ReconX</span>: Reconstruct Any Scene from Sparse Views with Video Diffusion Model</h1> 65 <div class="is-size-5 publication-authors"> 66 <span class="author-block"> 67 <a href="https://liuff19.github.io/">Fangfu Liu</a><sup>1</sup><sup>*</sup>,</span> 68 <span class="author-block"> 69 <a href="https://github.com/wenqsun">Wenqiang Sun</a><sup>2</sup><sup>*</sup>,</span> 70 <span class="author-block"> 71 <a href="https://hanyang-21.github.io/">Hanyang Wang</a><sup>1</sup><sup>*</sup>,</span> 72 <span class="author-block"> 73 <a href="https://yikaiw.github.io/">Yikai Wang</a><sup>1</sup>,</span> 74 <span class="author-block"> 75 <a href="https://github.com/sunboy0617">Haowen Sun</a><sup>1</sup>,</span><br> 76 <span class="author-block">
77 <a href="https://jamesyjl.github.io/">Junliang Ye</a><sup>1</sup>,</span> 78 <span class="author-block"> 79 <a href="https://eejzhang.people.ust.hk/home.html">Jun Zhang</a><sup>2</sup>,</span> 80 <span class="author-block"> 81 <a href="https://duanyueqi.github.io/">Yueqi Duan</a><sup>1</sup></span> 82 </div> 83 84 <!-- affiliations --> 85 <div class="subtitle is-size-5"> 86 <!-- <div class="is-size-5 publication-authors"> --> 87 <sup>1</sup> Tsinghua University 88 <sup>2</sup> HKUST <br> 89 <sup>*</sup> Equal Contribution 90 <!-- <span class="author-block"><sup>1</sup>The University of Texas at Austin</span> 91 <span class="author-block"><sup>2</sup>Nvidia</span> <br> 92 <span class="author-block"><sup>3</sup>Xiamen University</span> <br> 93 <span class="author-block"><sup>4</sup>Georgia Institute of Technology</span> <br> 94 <span class="author-block"><sup>5</sup>Stanford University</span> <br> 95 <span class="author-block"><sup>6</sup>University of Southern California</span> <br> 96 <span class="author-block"><sup>*</sup>denotes equal contribution</span> --> 97 </div> 98 99 <!-- TODO: update paper and video link once upload --> 100 <div class="column has-text-centered"> 101 <!-- <div class="publication-links"> --> 102 <!-- PDF Link. --> 103 <span class="link-block"> 104 <a href="https://arxiv.org/pdf/2408.16767" 105 class="external-link button is-normal is-rounded is-dark"> 106 <span class="icon"> 107 <i class="fas fa-file-pdf"></i> 108 </span> 109 <span>Paper</span> 110 </a> 111 </span> 112 <span class="link-block"> 113 <a href="https://arxiv.org/abs/2408.16767" 114 class="external-link button is-normal is-rounded is-dark"> 115 <span class="icon"> 116 <i class="ai ai-arxiv"></i> 117 </span> 118 <span>arXiv</span> 119 </a> 120 </span> 121 <!-- Video Link. --> 122 <span class="link-block"> 123 <a href="https://www.youtube.com/watch?v=UuL2nP5rJcI" class="external-link button is-normal is-rounded is-dark"> 124 <span class="icon"> 125 <i class="fab fa-youtube"></i> 126 </span> 127 <span>Video</span> 128 </a> 129 </span> 130 <!-- Code Link. --> 131 <span class="link-block"> 132 <a href="https://github.com/liuff19/ReconX" 133 class="external-link button is-normal is-rounded is-dark"> 134 <span class="icon"> 135 <i class="fab fa-github"></i> 136 </span> 137 <span>Code</span> 138 </a> 139 </span> 140 <!-- Demo Link. --> 141 <span class="link-block"> 142 <a href="https://github.com/liuff19/ReconX" 143 class="external-link button is-normal is-rounded is-dark"> 144 <span>ð¤ Demo</span> 145 </a> 146 </span> 147 </div> 148 149 150 <!-- Video Link. --> 151 <!-- <span class="link-block"> 152 <a href="https://youtu.be/470hul75bSM" class="external-link button is-normal is-rounded is-dark"> 153 <span class="icon"> 154 <i class="fab fa-youtube"></i> 155 </span> 156 <span>Video</span> 157 </a> --> 158 </span> 159 <!-- Code Link. --> 160 <!-- <span class="link-block"> 161 <a href="https://github.com/VITA-Group/Instantsplat" 162 class="external-link button is-normal is-rounded is-dark"> 163 <span class="icon"> 164 <i class="fab fa-github"></i> 165 </span> 166 <span>Code</span> 167 </a> 168 </span> --> 169 </div> 170 171 </div> 172 </div> 173 </div> 174 </div> 175 </div> 176 </section> 177 178<hr> 179<section class="hero teaser"> 180 <div class="container is-max-desktop"> 181 <div class="hero-body"> 182 <div class="columns is-centered has-text-centered"> 183 <div class="column is-four-fifths"> 184 <h2 class="title is-3">Video</h2>
185 <div class="publication-video"> 186 <iframe src="https://www.youtube.com/embed/UuL2nP5rJcI?rel=0&showinfo=0" 187 frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> 188 </div> 189 </div> 190 </div> 191 <!-- <video id="teaser" autoplay controls loop playsinline height="100%"> --> 192 <!-- <source src="static/myvideos/supp-video.mp4" type="video/mp4" /> --> 193 <!-- </video> --> 194 <h2 class="subtitle has-text-justified"> 195 <b>ReconX</b> is a novel sparse-view 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. 196 </h2> 197 </div> 198 </div> 199</section> 200 201 202 203 <section class="section"> 204 <div class="container is-max-desktop"> 205 <!-- Abstract. --> 206 <div class="columns is-centered has-text-centered"> 207 <div class="column is-full-width"> 208 <h2 class="title is-3">Abstract</h2> 209 <div class="content has-text-justified"> 210 <img src="./static/images/teaser.png"> 211 <p> 212 Advancements in 3D scene reconstruction have transformed 2D images from the real world into 3D models, producing realistic 3D results from hundreds of input photos. Despite great success in dense-view reconstruction scenarios, rendering a detailed scene from insufficient captured views is still an ill-posed optimization problem, often resulting in artifacts and distortions in unseen areas. In this paper, we propose <strong>ReconX</strong>, a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. The key insight is to unleash the strong generative prior of large pre-trained video diffusion models for sparse-view reconstruction. However, 3D view consistency struggles to be accurately preserved in directly generated video frames from pre-trained models. To address this, given limited input views, the proposed ReconX first constructs a global point cloud and encodes it into a contextual space as the 3D structure condition. Guided by the condition, the video diffusion model then synthesizes video frames that are both detail-preserved and exhibit a high degree of 3D consistency, ensuring the coherence of the scene from various perspectives. Finally, we recover the 3D scene from the generated video through a confidence-aware 3D Gaussian Splatting optimization scheme. Extensive experiments on various real-world datasets show the superiority of our ReconX over state-of-the-art methods in terms of quality and generalizability. 213 </p> 214 </div> 215 </div> 216 </div> 217 <!--/ Abstract. --> 218 </div> 219 </div> 220 </section> 221 222 223 <section class="section"> 224 <div class="container is-max-desktop"> 225 <!-- Concurrent Work. --> 226 <div class="columns is-centered has-text-centered"> 227 <div class="column is-full-width"> 228 <h2 class="title is-3">Method Overview</h2> 229 <div class="content has-text-justified"> 230 <img src="./static/images/method.png"> 231 232 <p><strong>Overall Framework of ReconX</strong>. Given sparse-view images as input, we first build a global point cloud 233 and project it into 3D context representation space as 3D structure guidance. Then we inject the 3D 234 structure guidance into the video diffusion process and guide it to generate 3D consistent video 235 frames. Finally, we reconstruct the 3D scene from the generated video through Gaussian Splatting 236 with a 3D confidence-aware and robust scene optimization scheme. In this way, we unleash the 237 strong power of the video diffusion model to reconstruct intricate 3D scenes from very sparse views.</p> 238 239 </div> 240 </div> 241 </div> 242 <!--/ Concurrent Work. --> 243 244 </div> 245 </section> 246 247 <section class="section"> 248 <div class="container is-max-desktop"> 249 <div class="columns is-centered has-text-centered"> 250 <div class="column is-full-width"> 251 <h2 class="title is-3">Real-World Results of ReconX</h2> 252 <div class="content has-text-justified"> 253 </div> 254 </div> 255 </section> 256 <section class="hero is-light is-small"> 257 <div class="hero-body"> 258 <div class="container"> 259 <div id="results-carousel" class=" carousel results-carousel"> 260 <div class="item item-family"> 261 <video poster="" id="family" autoplay controls muted loop playsinline height="100%"> 262 <source src="./static/videos/museum.mp4" 263 type="video/mp4"> 264 </video> 265 </div> 266 <div class="item item-family"> 267 <video poster="" id="family" autoplay controls muted loop playsinline height="100%"> 268 <source src="./static/videos/airport.mp4" 269 type="video/mp4"> 270 </video> 271 </div> 272 <div class="item item-horse"> 273 <video poster="" id="horse" autoplay controls muted loop playsinline height="100%"> 274 <source src="./static/videos/bighouse.mp4" 275 type="video/mp4"> 276 </video> 277 </div> 278 <div class="item item-horse"> 279 <video poster="" id="horse" autoplay controls muted loop playsinline height="100%"> 280 <source src="./static/videos/playroom.mp4" 281 type="video/mp4"> 282 </video> 283 </div> 284 <div class="item item-horse"> 285 <video poster="" id="horse" autoplay controls muted loop playsinline height="100%"> 286 <source src="./static/videos/khouse.mp4" 287 type="video/mp4"> 288 </video> 289 </div> 290 </div> 291 <p>The results shown are derived from <strong>two or three sparse views as input</strong>.</p> 292 </div> 293 </div> 294 </section> 295 296 <section class="section"> 297 <div class="container is-max-desktop "> 298 <div class=" has-text-centered"> 299 <h2 class="title is-3">Visual Comparisons</h2> 300 <div class="content has-text-justified"> 301 <p> 302 303 </p> 304 </div> 305 <div class="columns is-centered"> 306 <!-- Visual Effects. --> 307 <div class="column"> 308 <div class="content"> 309 <div class="columns is-centered"> 310 <div class="column is-full-width"> 311 <div id="example1" class="bal-container-small"> 312 313 <div class="bal-after"> 314 <img src="./static/images/comparison_n/scene1/reconx.png"> 315 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 316 ReconX [Ours] 317 </div> 318 </div> 319 320 <div class="bal-before" style="width:96.4968152866242%;"> 321 <div class="bal-before-inset" style="width: 512px;"> 322 <img src="./static/images/comparison_n/scene1/pixelsplat.png"> 323 <div class="bal-beforePosition beforeLabel"> 324 pixelSplat [Charatan 2024] 325 </div> 326 </div> 327 </div> 328 329 <div class="bal-handle" style="left:96.4968152866242%;">
330 <span class=" handle-left-arrow"></span> 331 <span class="handle-right-arrow"></span> 332 </div> 333 334 </div> 335 336 <div id="example4" class="bal-container-small"> 337 <div class="bal-after"> 338 <img src="./static/images/comparison_n/scene1/reconx.png"> 339 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 340 ReconX [Ours] 341 </div> 342 </div> 343 344 <div class="bal-before" style="width:62.10191082802548%;"> 345 <div class="bal-before-inset" style="width: 512px;"> 346 <img src="./static/images/comparison_n/scene1/mvsplat.png"> 347 <div class="bal-beforePosition beforeLabel"> 348 MVSplat [Chen 2023] 349 </div> 350 </div> 351 </div> 352 353 <div class="bal-handle" style="left:62.10191082802548%;"> 354 <span class=" handle-left-arrow"></span> 355 <span class="handle-right-arrow"></span> 356 </div> 357 358 </div> 359 360 </div> 361 </div> 362 </div> 363 </div> 364 365 366 <!--/ Visual Effects. --> 367 368 <!-- Matting. --> 369 <div class="column"> 370 <div class="columns is-centered"> 371 <div class="column content"> 372 373 374 <div id="example2" class="bal-container-small"> 375 376 <div class="bal-after"> 377 <img src="./static/images/comparison_n/scene-gen/reconx.png"> 378 <div class="bal-afterPosition afterLabel"> 379 ReconX [Ours] 380 </div> 381 </div> 382 383 <div class="bal-before" style="width: 50%;"> 384 <div class="bal-before-inset" style="width: 512px;"> 385 <img src="./static/images/comparison_n/scene-gen/pixelsplat.png"> 386 <div class="bal-beforePosition beforeLabel"> 387 pixelSplat [Charatan 2024] 388 </div> 389 </div> 390 </div> 391 392 <div class="bal-handle" style="left: 50%;"> 393 <span class=" handle-left-arrow"></span> 394 <span class="handle-right-arrow"></span> 395 </div> 396 </div> 397 398 399 <div id="example5" class="bal-container-small"> 400 <div class="bal-after"> 401 <img src="./static/images/comparison_n/scene-gen/reconx.png"> 402 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 403 ReconX [Ours] 404 </div> 405 </div> 406 407 <div class="bal-before" style="width:56.77179962894249%;"> 408 <div class="bal-before-inset" style="width: 512px;"> 409 <img src="./static/images/comparison_n/scene-gen/mvsplat.png"> 410 <div class="bal-beforePosition beforeLabel"> 411 MVSplat [Chen 2023] 412 </div> 413 </div> 414 </div> 415 416 <div class="bal-handle" style="left:56.77179962894249%;"> 417 <span class=" handle-left-arrow"></span> 418 <span class="handle-right-arrow"></span> 419 </div> 420 421 </div> 422 423 </div> 424 </div> 425 </div> 426 427 <div class="column"> 428 <div class="columns is-centered"> 429 <div class="column content"> 430 431 432 <div id="example3" class="bal-container-small"> 433 434 <div class="bal-after"> 435 <img src="./static/images/comparison_n/scene3/reconx.png"> 436 <div class="bal-afterPosition afterLabel"> 437 ReconX [Ours] 438 </div> 439 </div> 440 441 <div class="bal-before" style="width: 50%;"> 442 <div class="bal-before-inset" style="width: 512px;"> 443 <img src="./static/images/comparison_n/scene3/pixelsplat.png"> 444 <div class="bal-beforePosition beforeLabel"> 445 pixelSplat [Charatan 2024] 446 </div> 447 </div> 448 </div> 449 450 <div class="bal-handle" style="left: 50%;">
451 <span class=" handle-left-arrow"></span> 452 <span class="handle-right-arrow"></span> 453 </div> 454 </div> 455 456 <div id="example6" class="bal-container-small"> 457 <div class="bal-after"> 458 <img src="./static/images/comparison_n/scene3/reconx.png"> 459 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 460 ReconX [Ours] 461 </div> 462 </div> 463 464 <div class="bal-before" style="width:56.77179962894249%;"> 465 <div class="bal-before-inset" style="width: 512px;"> 466 <img src="./static/images/comparison_n/scene3/mvsplat.png"> 467 <div class="bal-beforePosition beforeLabel"> 468 MVSplat [Chen 2023] 469 </div> 470 </div> 471 </div> 472 473 <div class="bal-handle" style="left:56.77179962894249%;"> 474 <span class=" handle-left-arrow"></span> 475 <span class="handle-right-arrow"></span> 476 </div> 477 </div> 478 </div> 479 </div> 480 </div> 481 </div> 482 </div> 483 484 485 </div> 486 </section> 487 488 489 490 491
491<script> 492 new BeforeAfter({ 493 id: '#example1' 494 }); 495 new BeforeAfter({ 496 id: '#example2' 497 }); 498 new BeforeAfter({ 499 id: '#example3' 500 }); 501 new BeforeAfter({ 502 id: '#example4' 503 }); 504 new BeforeAfter({ 505 id: '#example5' 506 }); 507 new BeforeAfter({ 508 id: '#example6' 509 }); 510 511 512 </script>
512 513 514 <!-- 515<section class="section"> 516 <div class="container is-max-desktop"> 517 <div class="columns is-centered"> 518 <div class="column is-full-width"> 519 <h2 class="title is-3">Related Links</h2> 520 521 <div class="content has-text-justified"> 522 <p> 523 Needs update 524 </p> 525 526 </div> 527 </div> 528 </div> 529 530 </div> 531</section> --> 532 533 534Â <section class="section" id="BibTeX"> 535Â Â <div class="container is-max-desktop content"> 536Â Â Â <h2 class="title">BibTeX</h2> 537Â Â Â <pre><code>@misc{liu2024reconxreconstructscenesparse, 538Â Â Â Â title={ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model}, 539Â Â Â Â author={Fangfu Liu and Wenqiang Sun and Hanyang Wang and Yikai Wang and Haowen Sun and Junliang Ye and Jun Zhang and Yueqi Duan}, 540Â Â Â Â year={2024}, 541Â Â Â Â eprint={2408.16767}, 542Â Â Â Â archivePrefix={arXiv}, 543Â Â Â Â primaryClass={cs.CV}, 544 url={https://arxiv.org/abs/2408.16767}, 545Â Â Â }</code></pre> 546Â Â </div> 547Â </section> 548 549 550 <!-- TODO: update the arxiv link and github --> 551 <footer class="footer"> 552 <div class="container"> 553 <div class="content has-text-centered"> 554 <!-- <a class="icon-link" href=""> 555 <i class="fas fa-file-pdf"></i> 556 </a> --> 557 <!-- <a class="icon-link" href="" class="external-link" disabled> 558 <i class="fab fa-github"></i> --> 559 </a> 560 </div> 561 <div class="columns is-centered"> 562 <div class="column is-8"> 563 <div class="content"> 564 <p> 565 This website is licensed under a <a rel="license" 566 href="http://creativecommons.org/licenses/by-sa/4.0/">Creative 567 Commons Attribution-ShareAlike 4.0 International License</a>. 568 </p> 569 <p> 570 We thank the authors of 571 We thank the authors of <a href="https://nerfies.github.io/">Nerfies</a> that kindly open sourced the 572 template of this website. 573 </p> 574 </div> 575 </div> 576 </div> 577 </div> 578 </footer> 579 580</body> 581 582</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.