1<!DOCTYPE html> 2<html> 3 4<head> 5 <meta charset="utf-8"> 6 <meta name="description" 7 content="VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment"> 8 <meta name="keywords" content="VideoLifter, video-to-3D, Gaussian Splatting, Gaussian"> 9 <meta name="viewport" content="width=device-width, initial-scale=1"> 10 <title>VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment</title> 11 12
12<script> 13
vendor: 155 bytes, lines 13-21
13window.dataLayer = window.dataLayer || []; 14 15 function gtag() { 16 dataLayer.push(arguments); 17 } 18 19 gtag('js', new Date()); 20 21 gtag('config', '
21G-PYVRSFMDRL
vendor: 6 bytes, lines 21-22
21'); 22
22</script>
22 23 24 <link href="https://fonts.googleapis.co m/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> 25 26 <link rel="stylesheet" href="./static/css/bulma.min.css"> 27 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 28 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 29 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 30 <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 31 <link rel="stylesheet" href="./static/css/index.css"> 32 <!-- <link rel="icon" href="./static/images/favicon.svg"> --> 33 34
34<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
34 35
35<script defer src="./static/js/fontawesome.all.min.js"></script>
35 36
36<script src="./static/js/bulma-carousel.min.js"></script>
36 37
37<script src="./static/js/bulma-slider.min.js"></script>
37 38
38<script src="./static/js/index.js"></script>
38 39
39<script src="./static/js/script.js"></script>
39 40 41</head> 42 43<body> 44 45 46 <section class="hero"> 47 <div class="hero-body"> 48 <div class="container is-max-desktop"> 49 <div class="columns is-centered"> 50 51 <div class="column has-text-centered"> 52 <h1 class="title is-1 publication-title">VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment</h1> 53 54 <p class="is-size-5 has-text-weight-semibold has-text-danger mt-3"> 55 ð Abstract paper won Best Paper Award @ <a href="https://ai4cc.net/">CVPR 2025 AI4CC Workshop</a> 56 </p> 57 58 <div class="is-size-5 publication-authors"> 59 <span class="author-block"> 60 <a href="https://wenyancong.com/">Wenyan Cong</a><sup>1</sup>,</span> 61 <span class="author-block"> 62 <a href="https://zhuhanqing.github.io/">Hanqing Zhu</a><sup>1</sup>,</span> 63 <span class="author-block"> 64 <a href="https://kevin-2017.github.io/">Kevin Wang</a><sup>1</sup>,</span> 65 <span class="author-block"> 66 <a href="https://www.cis.upenn.edu/~leijh/">Jiahui Lei</a><sup>2</sup>,</span> 67 <span class="author-block"> 68 <a href="https://coltonstearns.github.io/">Colton Stearns</a><sup>3</sup>,</span> 69 <span class="author-block"> 70 <a href="https://caiyuanhao1998.github.io/">Yuanhao Cai</a><sup>4</sup>,</span> 71 <span class="author-block"> 72 <!-- <a href="https://wdilin.github.io/">Dilin Wang</a><sup>5</sup>,</span> 73 <span class="author-block"> 74 <a href="https://www.linkedin.com/in/rakesh-r-3848538/">Rakesh Ranjan</a><sup>5</sup>,</span> 75 <span class="author-block"> 76 <a href="https://scholar.google.com/citations?user=A-wA73gAAAAJ&hl=en">Matt Feiszli</a><sup>5</sup>,</span> 77 <span class="author-block"> --> 78 <a href="https://profiles.stanford.edu/leonidas-guibas">Leonidas Guibas</a><sup>3</sup>,</span> 79 <span class="author-block"> 80 <a href="https://express.adobe.com/page/CAdrFMJ9QeI2y/">Zhangyang Wang</a><sup>1*</sup>,</span> 81 <span class="author-block"> 82 <!-- <a href="https://scholar.google.com/citations?user=d1-hNQoAAAAJ&hl=en">Weiyao Wang</a><sup>5*</sup>,</span> 83 <span class="author-block"> --> 84 <a href="https://zhiwenfan.github.io/">Zhiwen Fan</a><sup>1*</sup></span> 85 86 87 </div> 88 89 <div class="is-size-5 publication-authors"> 90 <span class="author-block"><sup>1</sup>The University of Texas at Austin</span> 91 <span class="author-block"><sup>2</sup>University of Pennsylvania</span> <br> 92 <span class="author-block"><sup>3</sup>Stanford University</span> 93 <span class="author-block"><sup>4</sup>Johns Hopkins University</span> 94 <!-- <span class="author-block"><sup>5</sup>Meta</span> <br> --> 95 </div> 96 97 <span class="link-block"> 98 <a href="http://arxiv.org/abs/2501.01949" 99 class="external-link button is-normal is-rounded is-dark"> 100 <span class="icon"> 101 <i class="ai ai-arxiv"></i> 102 </span> 103 <span>arXiv</span> 104 </a> 105 </span> 106 <span class="link-block"> 107 <a href="https://github.com/VITA-Group/VideoLifter" 108 class="external-link button is-normal is-rounded is-dark"> 109 <span class="icon"> 110 <i class="fab fa-github"></i> 111 </span> 112 <span>Code</span> 113 </a> 114 </span> 115 116 </div> 117 </div> 118 </div> 119 </div> 120 </section> 121 122<section class="section"> 123 <div class="container is-max-desktop"> 124 <div class="columns is-centered has-text-centered"> 125 <div class="column is-full-width"> 126 <div class="content has-text-justified"> 127 <div style="text-align: center;"> 128 <img src="./static/images/teaser.png"> 129 </div> 130 </div> 131 </div> 132 </div> 133 </div> 134</section> 135 136 137 <section class="section"> 138 <div class="container is-max-desktop"> 139 <div class="columns is-centered has-text-centered"> 140 <div class="column is-four-fifths"> 141 <h2 class="title is-3">Abstract</h2> 142 <div class="content has-text-justified"> 143 144 <p> 145 Efficient and accurate 3D reconstruction from monocular video remains a key challenge in computer vision, 146 with significant implications for applications in virtual reality, robotics, and scene understanding. 147 Existing approaches typically require pre-computed camera parameters and frame-by-frame reconstruction pipelines, 148 often leading to error propagation and substantial computational costs. 149 </p> 150 <p> 151 In contrast, we introduce <b>VideoLifter</b>, a novel method that leverages geometric priors from a learnable model 152 to incrementally optimize a globally sparse to dense 3D representation directly from video sequences. 153 VideoLifter segments the video sequence into local windows, where it matches and registers frames, 154 constructs consistent fragments, and aligns them hierarchically to produce a unified 3D model. 155 By tracking and propagating sparse point correspondences across frames and fragments, 156 VideoLifter incrementally refines camera poses and 3D structure, minimizing reprojection error for improved accuracy and robustness. 157 </p> 158 <p> 159 This approach significantly accelerates the reconstruction process, 160 reducing training time by over 82% while surpassing current state-of-the-art methods in visual fidelity and computational efficiency. 161 </p> 162 <p></p> 163 164 </div> 165 </div> 166 </div> 167 </div> 168 </section> 169 170 171 172 173 <section class="section"> 174 <div class="container is-max-desktop"> 175 <div class="columns is-centered has-text-centered"> 176 <div class="column is-full-width"> 177 <h2 class="title is-3">Method Overview</h2> 178 <div class="content has-text-justified"> 179 <img src="./static/images/network.png"> 180 <p>VideoLifter takes uncalibrated images as input and reconstructs a dense scene representation based on self-supervised photometric signals. It performs efficie
180nt sparse reconstruction through the use of ``feature points" to formulate an initial solution for globally aligned camera poses. It subsequently optimizes a set of 3D Gaussians and corrects erroneous estimations from each fragment, hierarchically aligning them into a globally coherent 3D representation.</p> 181 </div> 182 </div> 183 </div> 184 185 </div> 186 </section> 187 188 189 190 191 192 193 194 195 196 <section class="section"> 197 <div class="container "> 198 <div class=" has-text-centered"> 199 <h2 class="title is-3">Novel View Synthesis</h2> 200 <div class="content has-text-justified"> 201 202 <div class="hero-body"> 203 <p class="has-text-centered">We show novel views from a Bezier curve which is fitted to estimated camera poses.</p> 204 <div class="container"> 205 <div id="results-grid" class="columns is-multiline"> 206 <div class="column is-half"> 207 <video poster="" id="ignatius" autoplay controls muted loop playsinline height="100%"> 208 <source src="./static/videos/Ignatius_compare.mp4" type="video/mp4"> 209 </video> 210 </div> 211 <div class="column is-half"> 212 <video poster="" id="family" autoplay controls muted loop playsinline height="100%"> 213 <source src="./static/videos/Family_compare.mp4" type="video/mp4"> 214 </video> 215 </div> 216 <div class="column is-half"> 217 <video poster="" id="francis" autoplay controls muted loop playsinline height="100%"> 218 <source src="./static/videos/Francis_compare.mp4" type="video/mp4"> 219 </video> 220 </div> 221 <div class="column is-half"> 222 <video poster="" id="horse" autoplay controls muted loop playsinline height="100%"> 223 <source src="./static/videos/Horse_compare.mp4" type="video/mp4"> 224 </video> 225 </div> 226 <div class="column is-half"> 227 <video poster="" id="bench" autoplay controls muted loop playsinline height="100%"> 228 <source src="./static/videos/bench_415_57112_110099_compare.mp4" type="video/mp4"> 229 </video> 230 </div> 231 <div class="column is-half"> 232 <video poster="" id="skateboard1" autoplay controls muted loop playsinline height="100%"> 233 <source src="./static/videos/skateboard_245_26182_52130_compare.mp4" type="video/mp4"> 234 </video> 235 </div> 236 <div class="column is-half"> 237 <video poster="" id="plant" autoplay controls muted loop playsinline height="100%"> 238 <source src="./static/videos/plant_247_26441_50907_compare.mp4" type="video/mp4"> 239 </video> 240 </div> 241 <div class="column is-half"> 242 <video poster="" id="skateboard2" autoplay controls muted loop playsinline height="100%"> 243 <source src="./static/videos/skateboard_407_54965_106262_compare.mp4" type="video/mp4"> 244 </video> 245 </div> 246 <div class="column is-half"> 247 <video poster="" id="apple" autoplay controls muted loop playsinline style="height: auto; width: 150%;"> 248 <source src="./static/videos/apple_110_13051_23361_compare.mp4" type="video/mp4"> 249 </video> 250 </div> 251 <div class="column is-half"> 252 <video poster="" id="hydrant" autoplay controls muted loop playsinline height="100%"> 253 <source src="./static/videos/hydrant_106_12648_23157_compare.mp4" type="video/mp4"> 254 </video> 255 </div> 256 </div> 257 </div> 258 </div> 259 260 </div> 261 <p>We show the synthesized images from testing views.</p> 262 <div class="columns is-centered"> 263 <!-- Visual Effects. --> 264 <div class="column"> 265 <div class="content"> 266 <div class="columns is-centered"> 267 <div class="column is-full-width"> 268 <div id="example1" class="bal-container-small"> 269 270 <div class="bal-after"> 271 <img src="./static/images/co3d_106_08_ours.png"> 272 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 273 Ours 274 </div> 275 </div> 276 277 <div class="bal-before" style="width:96.4968152866242%;"> 278 <div class="bal-before-inset" style="width: 539px;">
279 <img src="./static/images/co3d_106_08_cf3dgs.png"> 280 <div class="bal-beforePosition beforeLabel"> 281 CF-3DGS 282 </div> 283 </div> 284 </div> 285 286 <div class="bal-handle" style="left:96.4968152866242%;"> 287 <span class=" handle-left-arrow"></span> 288 <span class="handle-right-arrow"></span> 289 </div> 290 291 </div> 292 293 294 <div id="example2" class="bal-container-small"> 295 296 <div class="bal-after"> 297 <img src="./static/images/horse_14_ours.png"> 298 <div class="bal-afterPosition afterLabel"> 299 Ours 300 </div> 301 </div> 302 303 <div class="bal-before" style="width: 50%;"> 304 <div class="bal-before-inset" style="width: 539px;"> 305 <img src="./static/images/horse_14_cf3dgs.png"> 306 <div class="bal-beforePosition beforeLabel"> 307 CF-3DGS 308 </div> 309 </div> 310 </div> 311 312 <div class="bal-handle" style="left: 50%;"> 313 <span class=" handle-left-arrow"></span> 314 <span class="handle-right-arrow"></span> 315 </div> 316 317 </div> 318 </div> 319 </div> 320 </div> 321 </div> 322 323 324 <!--/ Visual Effects. --> 325 326 <div class="column"> 327 <div class="columns is-centered"> 328 <div class="column content"> 329 330 <div id="example4" class="bal-container-small"> 331 <div class="bal-after"> 332 <img src="./static/images/co3d_247_23_ours.png"> 333 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 334 Ours 335 </div> 336 </div> 337 338 <div class="bal-before" style="width:62.10191082802548%;"> 339 <div class="bal-before-inset" style="width: 539px;"> 340 <img src="./static/images/co3d_247_23_cf3dgs.png"> 341 <div class="bal-beforePosition beforeLabel"> 342 CF-3DGS 343 </div> 344 </div> 345 </div> 346 347 <div class="bal-handle" style="left:62.10191082802548%;"> 348 <span class=" handle-left-arrow"></span> 349 <span class="handle-right-arrow"></span> 350 </div> 351 352 </div> 353 354 <div id="example5" class="bal-container-small"> 355 <div class="bal-after"> 356 <img src="./static/images/francis_16_ours.png"> 357 <div class="bal-afterPosition afterLabel" style="z-index:1;"> 358 Ours 359 </div> 360 </div> 361 362 <div class="bal-before" style="width:56.77179962894249%;"> 363 <div class="bal-before-inset" style="width: 628px;"> 364 <img src="./static/images/francis_16_cf3dgs.png"> 365 <div class="bal-beforePosition beforeLabel"> 366 CF-3DGS 367 </div> 368 </div> 369 </div> 370 371 <div class="bal-handle" style="left:56.77179962894249%;"> 372 <span class=" handle-left-arrow"></span> 373 <span class="handle-right-arrow"></span> 374 </div> 375 376 </div> 377 </div> 378 </div> 379 </div> 380 </div> 381 </div> 382 </div> 383 </section> 384 385
385<script> 386 new BeforeAfter({ 387 id: '#example1' 388 }); 389 new BeforeAfter({ 390 id: '#example2' 391 }); 392 new BeforeAfter({ 393 id: '#example4' 394 }); 395 new BeforeAfter({ 396 id: '#example5' 397 }); 398 399 </script>
399 400 401</body> 402 403</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.