1<!DOCTYPE html> 2<html> 3 4<head lang="en"> 5 <meta charset="UTF-8"> 6 <meta http-equiv="x-ua-compatible" content="ie=edge"> 7 8 <title>NaRCan</title> 9 10 <meta name="description" content=""> 11 <meta name="viewport" content="width=device-width, initial-scale=1"> 12 13 <meta property="og:image" content="https://koi953215.github.io/NaRCan_page/img/overview_combined.jpg"> 14 <meta property="og:image:type" content="image/png"> 15 <meta property="og:image:width" content="1296"> 16 <meta property="og:image:height" content="840"> 17 <meta property="og:type" content="website" /> 18 <meta property="og:url" content="https://koi953215.github.io/NaRCan_page/"/> 19 <meta property="og:title" content="NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing" /> 20 <meta property="og:description" content="We propose a video editing framework, NaRCan, which integrates a hybrid deformation field and a diffusion prior to generating high-quality natural canonical images to represent the input video. Our approach utilizes homography to model global motion and employs multi-layer perceptrons (MLPs) to capture local residual deformations, enhancing the model's ability to handle complex video dynamics. By introducing a diffusion prior from the early stages of training, our model ensures that the generated images retain a high-quality natural appearance, making the produced canonical images suitable for various downstream tasks in video editing, a capability not achieved by current canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA) fine-tuning and introduce a noise and diffusion prior update scheduling technique that accelerates the training process by 14 times. Extensive experimental results show that our method outperforms existing approaches in various video editing tasks and produces coherent and high-quality edited video sequences." /> 21 22 <meta name="twitter:card" content="summary_large_image" /> 23 <meta name="twitter:title" content="NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing" /> 24 <meta name="twitter:description" content="We propose a video editing framework, NaRCan, which integrates a hybrid deformation field and a diffusion prior to generating high-quality natural canonical images to represent the input video. Our approach utilizes homography to model global motion and employs multi-layer perceptrons (MLPs) to capture local residual deformations, enhancing the model's ability to handle complex video dynamics. By introducing a diffusion prior from the early stages of training, our model ensures that the generated images retain a high-quality natural appearance, making the produced canonical images suitable for various downstream tasks in video editing, a capability not achieved by current canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA) fine-tuning and introduce a noise and diffusion prior update scheduling technique that accelerates the training process by 14 times. Extensive experimental results show that our method outperforms existing approaches in various video editing tasks and produces coherent and high-quality edited video sequences." /> 25 <meta name="twitter:image" content="https://koi953215.github.io/NaRCan_page/img/overview_combined.jpg" /> 26 27 28<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>ð</text></svg>"> 29 30 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css"> 31 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css"> 32 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.css"> 33 <link rel="stylesheet" href="css/app.css"> 34 <link rel="stylesheet" href="css/fontawesome.all.min.css"> 35 <link rel="stylesheet" 36 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 37 38 <link rel="stylesheet" href="css/bootstrap.min.css"> 39 40 <!-- Google tag (gtag.js) -->
vendor: 66 bytes, lines 40-41
40 41 <script async src="https://www.googletagmanager.com/gtag/js?id=
41G-8ZERS5BVPS
vendor: 14 bytes, lines 41-42
41"></script> 42
42<script> 43
vendor: 140 bytes, lines 43-47
43window.dataLayer = window.dataLayer || []; 44 function gtag(){dataLayer.push(arguments);} 45 gtag('js', new Date()); 46 47 gtag('config', '
47G-8ZERS5BVPS
vendor: 6 bytes, lines 47-48
47'); 48
48</script>
48 49 50
50<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
50 51
51<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
51 52
52<script src="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.js"></script>
52 53
53<script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/1.5.3/clipboard.min.js"></script>
53 54
54<script type="text/javascript" async 55 src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/MathJax.js?config=TeX-MML-AM_CHTML"> 56 </script>
56 57
57<script defer src="js/fontawesome.all.min.js"></script>
57 58
58<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.5.0/Chart.min.js"></script>
58 59 60
60<script src="js/app.js"></script>
60 61
61<script src="js/video_comparison.js"></script>
61 62
62<script src="js/synced_video_selector.js"></script>
62 63 64 <style> 65 /* å½±ç容å¨çé¿æå¼è¨è¨ */ 66 .video-container2 { 67 position: relative; 68 width: 100%; 69 padding-bottom: 56.25%; /* 16:9 坬髿¯ */ 70 height: 0; 71 overflow: hidden; 72 max-width: 1200px; /* æå¤§å¯¬åº¦éå¶ */ 73 margin: 0 auto; 74 } 75 76 .video-container2 iframe { 77 position: absolute; 78 top: 0; 79 left: 0; 80 width: 100%; 81 height: 100%; 82 } 83 84 /* Bootstrap æ¬ä½èª¿æ´ */ 85 .col-md-8.col-md-offset-2 { 86 width: 100%; 87 margin-left: 0; 88 padding: 0 15px; 89 } 90 91 .publication-video { 92 margin: 20px 0; 93 } 94 95 /* ä¸è¬è¢å¹å°ºå¯¸çé¿æå¼è¨è¨ */ 96 @media (min-width: 768px) { 97 .col-md-8.col-md-offset-2 { 98 width: 66.66667%; 99 margin-left: 16.66667%; 100 } 101 } 102 </style> 103 104 105</head> 106 107<body style="padding: 1%; width: 100%"> 108 <div class="container" style="max-width: 1500px; margin: auto;" id="main"> 109 <div class="row"> 110 <h2 class="col-md-12 text-center"> 111 <b>NaRCan</b>:<br>Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing</br> 112 <small> 113 NeurIPS 2024 114 </small> 115 </h2> 116 </div> 117 <div class="row"> 118 <div class="col-md-12 text-center"> 119 <ul class="list-inline"> 120 <li> 121 <a href="https://koi953215.github.io/"> 122 Ting-Hsuan Chen 123 </a> 124 </li> 125 <li> 126 <a href=""> 127 Jiewen Chan 128 </a> 129 </li> 130 <li> 131 <a href=""> 132 Hau-Shiang Shiu 133 </a> 134 </li> 135 <li> 136 <a href=""> 137 Shih-Han Yen 138 </a> 139 </li> 140 <li> 141 <a href="https://jimmycv07.github.io/"> 142 Chang-Han Yeh 143 </a> 144 </li> 145 <li> 146 <a href="https://yulunalexliu.github.io/"> 147 Yu-Lun Liu 148 </a> 149 </li> 150 </br>National Yang Ming Chiao Tung University 151 </ul> 152 </div> 153 </div> 154 155 156 <div class="row"> 157 <div class="col-md-6 col-md-offset-3 text-center"> 158 <ul class="nav nav-pills nav-justified"> 159 <li> 160 <a href="http://arxiv.org/abs/2406.06523"> 161 <image src="img/narcan_paper_image.jpg" height="60px"> 162 <h4><strong>Paper</strong></h4> 163 </a> 164 </li> 165 <li> 166 <a href="https://drive.google.com/file/d/1Ur6HEP2MizFe3z7iSXoaMQmC5_qHUn1S/view?usp=sharing"> 167 <image src="img/database_icon.png" height="60px"> 168 <h4><strong>Evaluation results</strong></h4> 169 </a> 170 </li> 171 <!-- <li> 172 <a href="https://www.youtube.com/watch?v=xrrhynRzC8k"> 173 <image src="img/youtube_icon.png" height="60px"> 174 <h4><strong>Video</strong></h4> 175 </a> 176 </li> --> 177 <li> 178 <a href="https://github.com/koi953215/NaRCan"> 179 <image src="img/github.png" height="60px"> 180 <h4><strong>Code</strong></h4> 181 </a> 182 </li> 183 <li> 184 <a href="https://huggingface.co/spaces
184/Koi953215/NaRCan_demo"> 185 <image src="img/logos_huggingface.png" height="60px"> 186 <h4><strong>Hugging Face demo</strong></h4> 187 </a> 188 </li> 189 </ul> 190 </div> 191 </div> 192 193 194 195 <div class="row"> 196 <div class="col-md-8 col-md-offset-2"> 197 <video id="v0" width="100%" autoplay loop muted controls> 198 <source src="img/teaser.mp4" type="video/mp4" /> 199 </video> 200 </div> 201 </div> 202 203 204 <div class="row"> 205 <div class="col-md-8 col-md-offset-2"> 206 <h3> 207 Abstract 208 </h3> 209 <p class="text-justify"> 210 We propose a video editing framework, NaRCan, which integrates a hybrid deformation field and a diffusion prior to generating high-quality natural canonical images to represent the input video. Our approach utilizes homography to model global motion and employs multi-layer perceptrons (MLPs) to capture local residual deformations, enhancing the model's ability to handle complex video dynamics. By introducing a diffusion prior from the early stages of training, our model ensures that the generated images retain a high-quality natural appearance, making the produced canonical images suitable for various downstream tasks in video editing, a capability not achieved by current canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA) fine-tuning and introduce a noise and diffusion prior update scheduling technique that accelerates the training process by 14 times. Extensive experimental results show that our method outperforms existing approaches in various video editing tasks and produces coherent and high-quality edited video sequences. 211 </p> 212 </div> 213 </div> 214 215 216 <div class="row"> 217 <div class="col-md-8 col-md-offset-2"> 218 <h3>Video</h3> 219 <div class="publication-video"> 220 <div class="video-container2"> 221 <iframe src="https://www.youtube.com/embed/ufYRGldCV78?rel=0&showinfo=0" 222 frameborder="0" 223 allow="autoplay; encrypted-media" 224 allowfullscreen></iframe> 225 </div> 226 </div> 227 </div> 228 </div> 229 230 <br> 231 <div class="row"> 232 <div class="col-md-8 col-md-offset-2"> 233 <h3> 234 <b><font color="#4472c4">NaR</font><font color="#ff0000">Can</font></b> = <font color="#4472c4"><b>Na</b>tural <b>R</b>efined</font> <font color="#ff0000"><b>Can</b>onical Image</font> with <font color="#ed7d31"><b>Diffusion Prior</b></font> 235 </h3> 236 <video id="v0" width="100%" autoplay loop muted> 237 <source src="img/method_teaser.mp4" type="video/mp4" /> 238 </video> 239 <!-- <image src="img/teaser.jpg" width=100% style="display: block; margin: auto;"></image> --> 240 <br> 241 <p class="text-justify"> 242 <b>Video representation with diffusion prior.</b> Given an RGB video, we can represent the video using a canonical image. However, the canonical image and reconstruction training process focuses only on reconstruction quality and could produce an unnatural canonical image. This could cause problems with downstream tasks such as prompt-based video editing. In the bottom example, if the bear is distorted in the canonical image, the image editor, such as ControlNet, may not recognize it and could introduce an irrelevant object instead. In this paper, we propose introducing the <i>diffusion prior</i> from a LoRA fine-tuned diffusion model to the training pipeline and constraining the canonical image to be natural. Our method facilitates several downstream tasks, such as (a) video editing, (b) dynamic segmentation, and (c) video style transfer. 243 </p> 244 </div> 245 </div> 246 247 <br> 248 <div class="row"> 249 <div class="col-md-8 col-md-offset-2"> 250 <h3> 251 Our proposed framework 252 </h3> 253 <image src="img/overview_combined.jpg" width=100% style="display: block; margin: auto;"></image>
254 <p class="text-justify"> 255 Given an input video sequence, our method aims to represent the video with a <i>natural</i> canonical image, which is a crucial representation for versatile downstream applications. (a) First, we fine-tune the LoRA weights of a pre-trained latent diffusion model on the input frames. (b) Second, we represent the video using a canonical MLP and a deformation field, which consists of homography estimation and residual deformation MLP for non-rigid residual deformations. By relying entirely on the reconstruction loss, the canonical MLP often fails to represent a natural canonical image, causing problems for downstream applications. <i>E.g.</i>, image-to-image translation methods such as ControlNet may not be able to recognize that there is a train in the canonical image. (c) Therefore, we leverage the fine-tuned latent diffusion model to regularize and correct the unnatural canonical image into a natural one. Specifically, we sophistically design a noise scheduling corresponding to the frame reconstruction process. (d) The natural and artifacts-free canonical image can then be facilitated to various downstream tasks such as video style transfer, dynamic segmentation, and editing, such as adding handwritten characters. 256 </p> 257 </div> 258 </div> 259 260 <br> 261 262 <div class="row"> 263 <div class="col-md-8 col-md-offset-2"> 264 <h3> 265 Diffusion prior for canonical image refinement 266 </h3> 267 <div class="text-center"> 268 <video id="v0" width="80%" autoplay loop muted controls> 269 <source src="img/dm_prior.mp4" type="video/mp4" /> 270 </video> 271 </div> 272 <p class="text-justify"> 273 (<em>Left</em>) Without diffusion prior to regularizing the canonical image, the training process relies only on the frame reconstruction and could sacrifice the faithfulness of the canonical image. (<em>Right</em>) Our fine-tuned diffusion prior effectively corrects the canonical image to faithfully represent the input frames and results in natural canonical images. 274 </p> 275 </div> 276 </div> 277 278 <br> 279 <div class="row"> 280 <div class="col-md-8 col-md-offset-2"> 281 <h3> 282 Noise and diffusion prior update scheduling 283 </h3> 284 <image src="img/scheduling.jpg" width=100% style="display: block; margin: auto;"></image> 285 <!-- <br> --> 286 <hr style="border-top: dotted 3px;" /> 287 <div class="text-center"> 288 <video id="v0" width="80%" autoplay loop muted controls> 289 <source src="img/schedule.mp4" type="video/mp4" /> 290 </video> 291 </div> 292 <p class="text-justify"> 293 Initially, our model fits object outlines before the fields converge and without the diffusion prior, resulting in unnatural elements in the canonical image due to complex non-rigid objects. Upon introducing the diffusion prior with increased noise and update frequency, the model learns to generate natural, high-quality images, leading to convergence. Thus, the strength of noise and the update frequency will also decrease. Moreover, it's worth mentioning that update scheduling cuts training time from 4.8 hours to 20 minutes. 294 </p> 295 </div> 296 </div> 297 298 299 300 301 302 303 304 305 306 <br> 307 <div class="row"> 308 <div class="col-md-8 col-md-offset-2"> 309 <h3> 310 Text-guided video-to-video translation 311 </h3> 312 313 <div class="text-center "> 314 <ul class="nav nav-pills center-pills"> 315 <li class="method-pill" data-value="input" 316 onclick="selectCompVideo('styletransfer', this, activeScenePill)"><a>Input</a></li> 317 <li class="method-pill" data-value="hash" 318 onclick="selectCompVideo('styletransfer', this, activeScenePill)"><a>Hashing-nvd</a></li> 319 <li class="method-pill active" data-value="codef" 320 onclick="selectCompVideo('styletransfer', this, activeScenePill)"><a>CoDeF</a></li> 321 <li class="method-pill" data-value="medm" 322 onclick="selectCompVideo('styletransfer', this, activeScenePill)"><a>MeDM</a></li> 323 </ul> 324 </div> 325 326
326<script> 327 activeMethodPill = document.querySelector('.method-pill.active-pill'); 328 activeScenePill = document.querySelector('.scene-pill.active-pill'); 329 activeModePill = document.querySelector('.mode-pill.active-pill'); 330 </script>
330 331 332 <div class="text-center"> 333 <div class="video-container"> 334 <video class="video" style="max-width: 100%;" m id="compVideo0" loop playsinline autoplay muted> 335 <source src="videos/styletransfer_bear_codef_vs_ours_video.mp4" /> 336 </video> 337 <video class="video" style="max-width: 100%;" id="compVideo1" loop playsinline autoplay muted hidden> 338 <source src="videos/styletransfer_bear_codef_vs_ours_canonical.mp4" /> 339 </video> 340 </div> 341 <div class="text-center" style="color: black;" id="mode-pills"> 342 <div class="btn-group btn-group-sm"> 343 <span class="btn btn-primary mode-pill active" data-value="video" 344 onclick="selectCompVideo('styletransfer', activeMethodPill, activeScenePill, null, this)"> 345 Edited video 346 </span> 347 <span class="btn btn-primary mode-pill" data-value="canonical" 348 onclick="selectCompVideo('styletransfer', activeMethodPill, activeScenePill, null, this)"> 349 Canonical image 350 </span> 351 <span class="btn btn-primary mode-pill" data-value="canonical_edit" 352 onclick="selectCompVideo('styletransfer', activeMethodPill, activeScenePill, null, this)"> 353 Edited canonical image 354 </span> 355 </div> 356 </div> 357 358 359 <p class="text-justify" style="text-align: center;"> 360 Text prompt: <span id="textPrompt"><strong>A Van Gogh-style cartoon bear walking in the forest.</strong></span><br> 361 Baseline method (left) vs NaRCan (right). Try selecting different methods and scenes! 362 </p> 363
363<script> 364 video0 = document.getElementById("compVideo0"); 365 video1 = document.getElementById("compVideo1"); 366 video0.addEventListener('loadedmetadata', function() { 367 if (activeVidID == 0 && select){ 368 video0.play(); 369 // print video size 370 console.log(video0.videoWidth, video0.videoHeight); 371 video0.hidden = false; 372 video1.hidden = true; 373 } 374 }); 375 video1.addEventListener('loadedmetadata', function() { 376 if (activeVidID == 1 && select){ 377 video1.play(); 378 // print video size 379 console.log(video1.videoWidth, video1.videoHeight); 380 video0.hidden = true; 381 video1.hidden = false; 382 } 383 }); 384 </script>
384 385 386 <div class="pill-row scene-pills" id="scene-pills"> 387 <span class="pill scene-pill active" data-value="bear" onclick="selectCompVideo('styletransfer', activeMethodPill, this, 3)"> 388 <img class="thumbnail-img" src="thumbnails/styletransfer_bear_thumbnail.jpg" alt="DTU/scan31" width="64"> 389 </span> 390 <span class="pill scene-pill" data-value="boat" onclick="selectCompVideo('styletransfer', activeMethodPill, this, 3)"> 391 <img class="thumbnail-img" src="thumbnails/styletransfer_boat_thumbnail.jpg" alt="DTU/scan45" width="64"> 392 </span> 393 <span class="pill scene-pill" data-value="hot-air-ballon" onclick="selectCompVideo('styletransfer', activeMethodPill, this, 3)"> 394 <img class="thumbnail-img" src="thumbnails/styletransfer_hot-air-ballon_thumbnail.jpg" alt="LLFF/fern" width="64"> 395 </span> 396 <span class="pill scene-pill" data-value="overlook-the-ocean" onclick="selectCompVideo('styletransfer', activeMethodPill, this, 3)"> 397 <img class="thumbnail-img" src="thumbnails/styletransfer_overlook-the-ocean_thumbnail.jpg" alt="LLFF/horns" width="64"> 398 </span> 399 <span class="pill scene-pill" data-value="shark-ocean" onclick="selectCompVideo('styletransfer', activeMethodPill, this, 3)"> 400 <img class="thumbnail-img" src="thumbnails/styletransfer_shark-ocean_thumbnail.jpg" alt="Re10K/sofa" width="64"> 401 </span> 402 </div> 403 404
404<script> 405 activeMethodPill = document.querySelector('.method-pill.active-pill'); 406 activeScenePill = document.querySelector('.scene-pill.active-pill'); 407 activeModePill = document.querySelector('.mode-pill.active-pill'); 408 </script>
408 409 </div> 410 </div> 411 </div> 412 <br> 413 414 415 416 417 <div class="row"> 418 <div class="col-md-8 col-md-offset-2"> 419 <h3> 420 Adding handwritten characters 421 </h3> 422 423 <div class="text-center "> 424 <ul class="nav nav-pills center-pills"> 425 <li class="method-pill-handwrite" data-value="input" 426 onclick="selectCompVideoHandwrite('handwrite', this, activeScenePillHandwrite)"><a>Input</a></li> 427 <li class="method-pill-handwrite" data-value="hash" 428 onclick="selectCompVideoHandwrite('handwrite', this, activeScenePillHandwrite)"><a>Hashing-nvd</a></li> 429 <li class="method-pill-handwrite active" data-value="codef" 430 onclick="selectCompVideoHandwrite('handwrite', this, activeScenePillHandwrite)"><a>CoDeF</a></li> 431 <li class="method-pill-handwrite" data-value="medm" 432 onclick="selectCompVideoHandwrite('handwrite', this, activeScenePillHandwrite)"><a>MeDM</a></li> 433 </ul> 434 </div> 435 436
436<script> 437 activeMethodPillHandwrite = document.querySelector('.method-pill-handwrite.active-pill'); 438 activeScenePillHandwrite = document.querySelector('.scene-pill-handwrite.active-pill'); 439 activeModePillHandwrite = document.querySelector('.mode-pill-handwrite.active-pill'); 440 </script>
440 441 442 <div class="text-center"> 443 <div class="video-container"> 444 <video class="video" style="max-width: 100%;" m id="compVideo0_handwrite" loop playsinline autoplay muted> 445 <source src="videos/handwrite_train_codef_vs_ours_video.mp4" /> 446 </video> 447 <video class="video" style="max-width: 100%;" id="compVideo1_handwrite" loop playsinline autoplay muted hidden> 448 <source src="videos/handwrite_train_codef_vs_ours_canonical.mp4" /> 449 </video> 450 </div> 451 <div class="text-center" style="color: black;" id="mode-pills"> 452 <div class="btn-group btn-group-sm"> 453 <span class="btn btn-primary mode-pill-handwrite active" data-value="video" 454 onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, activeScenePillHandwrite, null, this)"> 455 Edited video 456 </span> 457 <span class="btn btn-primary mode-pill-handwrite" data-value="canonical" 458 onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, activeScenePillHandwrite, null, this)"> 459 Canonical image 460 </span> 461 <span class="btn btn-primary mode-pill-handwrite" data-value="canonical_edit" 462 onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, activeScenePillHandwrite, null, this)"> 463 Edited canonical image 464 </span> 465 </div> 466 </div> 467 468 469 <p class="text-justify" style="text-align: center;"> 470 Baseline method (left) vs NaRCan (right). Try selecting different methods and scenes! 471 </p> 472
472<script> 473 video0_handwrite = document.getElementById("compVideo0_handwrite"); 474 video1_handwrite = document.getElementById("compVideo1_handwrite"); 475 video0_handwrite.addEventListener('loadedmetadata', function() { 476 if (activeVidIDHandwrite == 0 && select){ 477 video0_handwrite.play(); 478 // print video size 479 console.log(video0_handwrite.videoWidth, video0_handwrite.videoHeight); 480 video0_handwrite.hidden = false; 481 video1_handwrite.hidden = true; 482 } 483 }); 484 video1_handwrite.addEventListener('loadedmetadata', function() { 485 if (activeVidIDHandwrite == 1 && select){ 486 video1_handwrite.play(); 487 // print video size 488 console.log(video1_handwrite.videoWidth, video1_handwrite.videoHeight); 489 video0_handwrite.hidden = true; 490 video1_handwrite.hidden = false; 491 } 492 }); 493 </script>
493 494 495 <div class="pill-row scene-pills-handwrite" id="scene-pills-handwrite"> 496 <span class="pill scene-pill-handwrite active" data-value="train" onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, this, 3)"> 497 <img class="thumbnail-img" src="thumbnails/handwrite_train_thumbnail.jpg" alt="DTU/scan31" width="64"> 498 </span> 499 <span class="pill scene-pill-handwrite" data-value="camel" onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, this, 3)"> 500 <img class="thumbnail-img" src="thumbnails/handwrite_camel_thumbnail.jpg" alt="DTU/scan45" width="64"> 501 </span> 502 <span class="pill scene-pill-handwrite" data-value="cat" onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, this, 3)"> 503 <img class="thumbnail-img" src="thumbnails/handwrite_cat_thumbnail.jpg" alt="LLFF/fern" width="64"> 504 </span> 505 <span class="pill scene-pill-handwrite" data-value="car-turn" onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, this, 3)"> 506 <img class="thumbnail-img" src="thumbnails/handwrite_car-turn_thumbnail.jpg" alt="LLFF/horns" width="64"> 507 </span> 508 <span class="pill scene-pill-handwrite" data-value="tiger" onclick="selectCompVideoHandwrite('handwrite', activeMethodPillHandwrite, this, 3)"> 509 <img class="thumbnail-img" src="thumbnails/handwrite_tiger_thumbnail.jpg" alt="Re10K/sofa" width="64"> 510 </span> 511 </div> 512 513
513<script> 514 activeMethodPillHandwrite = document.querySelector('.method-pill-handwrite.active-pill'); 515 activeScenePillHandwrite = document.querySelector('.scene-pill-handwrite.active-pill'); 516 activeModePillHandwrite = document.querySelector('.mode-pill-handwrite.active-pill'); 517 </script>
517 518 </div> 519 </div> 520 </div> 521 <br> 522 523 524 <div class="row"> 525 <div class="col-md-8 col-md-offset-2"> 526 <h3> 527 Segmentation-based tracking 528 </h3> 529 530 <div class="text-center "> 531 <ul class="nav nav-pills center-pills"> 532 <li class="method-pill-dynamic" data-value="input" 533 onclick="selectCompVideoDynamic('dynamic', this, activeScenePillDynamic)"><a>Input</a></li> 534 <li class="method-pill-dynamic" data-value="hash" 535 onclick="selectCompVideoDynamic('dynamic', this, activeScenePillDynamic)"><a>Hashing-nvd</a></li> 536 <li class="method-pill-dynamic active" data-value="codef" 537 onclick="selectCompVideoDynamic('dynamic', this, activeScenePillDynamic)"><a>CoDeF</a></li> 538 <li class="method-pill-dynamic" data-value="medm" 539 onclick="selectCompVideoDynamic('dynamic', this, activeScenePillDynamic)"><a>MeDM</a></li> 540 </ul> 541 </div> 542 543
543<script> 544 activeMethodPillDynamic = document.querySelector('.method-pill-dynamic.active-pill'); 545 activeScenePillDynamic = document.querySelector('.scene-pill-dynamic.active-pill'); 546 activeModePillDynamic = document.querySelector('.mode-pill-dynamic.active-pill'); 547 </script>
547 548 549 <div class="text-center"> 550 <div class="video-container"> 551 <video class="video" style="max-width: 100%;" m id="compVideo0_dynamic" loop playsinline autoplay muted> 552 <source src="videos/dynamic_coral-reef_codef_vs_ours_video.mp4" /> 553 </video> 554 <video class="video" style="max-width: 100%;" id="compVideo1_dynamic" loop playsinline autoplay muted hidden> 555 <source src="videos/dynamic_coral-reef_codef_vs_ours_canonical.mp4" /> 556 </video> 557 </div> 558 <div class="text-center" style="color: black;" id="mode-pills"> 559 <div class="btn-group btn-group-sm"> 560 <span class="btn btn-primary mode-pill-dynamic active" data-value="video" 561 onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, activeScenePillDynamic, null, this)"> 562 Edited video 563 </span> 564 <span class="btn btn-primary mode-pill-dynamic" data-value="canonical" 565 onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, activeScenePillDynamic, null, this)"> 566 Canonical image 567 </span> 568 <span class="btn btn-primary mode-pill-dynamic" data-value="canonical_edit" 569 onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, activeScenePillDynamic, null, this)"> 570 Segmentation mask 571 </span> 572 </div> 573 </div> 574 575 576 <p class="text-justify" style="text-align: center;"> 577 Baseline method (left) vs NaRCan (right). The segmentation mask is acquired using the <a href="https://segment-anything.com/">Segment Anything Model (SAM)</a> based on the learned canonical image and propagated to the sequence. Try selecting different methods and scenes! 578 </p> 579
579<script> 580 video0_dynamic = document.getElementById("compVideo0_dynamic"); 581 video1_dynamic = document.getElementById("compVideo1_dynamic"); 582 video0_dynamic.addEventListener('loadedmetadata', function() { 583 if (activeVidIDDynamic == 0 && select){ 584 video0_dynamic.play(); 585 // print video size 586 console.log(video0_dynamic.videoWidth, video0_dynamic.videoHeight); 587 video0_dynamic.hidden = false; 588 video1_dynamic.hidden = true; 589 } 590 }); 591 video1_dynamic.addEventListener('loadedmetadata', function() { 592 if (activeVidIDDynamic == 1 && select){ 593 video1_dynamic.play(); 594 // print video size 595 console.log(video1_dynamic.videoWidth, video1_dynamic.videoHeight); 596 video0_dynamic.hidden = true; 597 video1_dynamic.hidden = false; 598 } 599 }); 600 </script>
600 601 602 <div class="pill-row scene-pills-dynamic" id="scene-pills-dynamic"> 603 <span class="pill scene-pill-dynamic active" data-value="coral-reef" onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, this, 3)"> 604 <img class="thumbnail-img" src="thumbnails/dynamic_coral-reef_thumbnail.jpg" alt="DTU/scan31" width="64"> 605 </span> 606 <span class="pill scene-pill-dynamic" data-value="butterfly" onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, this, 3)"> 607 <img class="thumbnail-img" src="thumbnails/dynamic_butterfly_thumbnail.jpg" alt="DTU/scan45" width="64"> 608 </span> 609 <span class="pill scene-pill-dynamic" data-value="two-swan" onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, this, 3)"> 610 <img class="thumbnail-img" src="thumbnails/dynamic_two-swan_thumbnail.jpg" alt="LLFF/fern" width="64"> 611 </span> 612 <span class="pill scene-pill-dynamic" data-value="woman-drink" onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, this, 3)"> 613 <img class="thumbnail-img" src="thumbnails/dynamic_woman-drink_thumbnail.jpg" alt="LLFF/horns" width="64"> 614 </span> 615 <span class="pill scene-pill-dynamic" data-value="surf" onclick="selectCompVideoDynamic('dynamic', activeMethodPillDynamic, this, 3)"> 616 <img class="thumbnail-img" src="thumbnails/dynamic_surf_thumbnail.jpg" alt="Re10K/sofa" width="64"> 617 </span> 618 </div> 619 620
620<script> 621 activeMethodPillDynamic = document.querySelector('.method-pill-dynamic.active-pill'); 622 activeScenePillDynamic = document.querySelector('.scene-pill-dynamic.active-pill'); 623 activeModePillDynamic = document.querySelector('.mode-pill-dynamic.active-pill'); 624 </script>
624 625 </div> 626 </div> 627 </div> 628 <br> 629 630 <div class="row"> 631 <div class="col-md-8 col-md-offset-2"> 632 <h3> 633 User study 634 </h3> 635 <image src="img/user_study.jpg" width=100% style="display: block; margin: auto;"></image> 636 <p class="text-justify"> 637 Our method achieves the highest user preference ratios across all three aspects, compared with MeDM, CoDeF, Hashing-nvd. 638 </p> 639 </div> 640 </div><br> 641 642 643 644 645 646 647 648 649 650 <div class="row"> 651 <div class="col-md-8 col-md-offset-2"> 652 <h3> 653 Citation 654 </h3> 655 <div class="form-group col-md-10 col-md-offset-1"> 656 <textarea id="bibtex" class="form-control" readonly> 657@article{chen2024narcan, 658 title={NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing}, 659 author={Chen, Ting-Hsuan and Chan, Jiewen and Shiu, Hau-Shiang and Yen, Shih-Han and Yeh, Chang-Han and Liu, Yu-Lun}, 660 journal={Advances in Neural Information Processing Systems}, 661 year={2024} 662}</textarea> 663 </div> 664 </div> 665 </div> 666 667 <div class="row"> 668 <div class="col-md-8 col-md-offset-2"> 669 <h3> 670 Acknowledgements 671 </h3> 672 <p class="text-justify"> 673 This research was funded by the National Science and Technology Council, Taiwan, under Grants NSTC 112-2222-E-A49-004-MY2. The authors are grateful to Google, NVIDIA, and MediaTek Inc. for generous donations. Yu-Lun Liu acknowledges the Yushan Young Fellow Program by the MOE in Taiwan. 674 <br><br> 675 The website template was borrowed from <a href="http://mgharbi.com/">Michaël Gharbi</a> and <a href="https://dorverbin.github.io/refnerf">Ref-NeRF</a>. 676 </p> 677 </div> 678 </div> 679 </div> 680</body> 681</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.