1<!DOCTYPE html><html><head><meta charset="utf-8"> 2<meta name="viewport" content="width=device-width, initial-scale=1"> 3<title>MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance</title> 4<meta name="google-site-verification" content="I9VHWgYS9g3o8RUJ7ryB1hPoIcN9YK1bX1Ds7_uDIYE"> 5<meta property="og:title" content="MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance"> 6<meta name="description" content="MeDM utilizes pre-trained image Diffusion Models for video-to-video translation with consistent temporal flow. The proposed framework can render videos from scene position information, such as a normal G-buffer, or perform text-guided editing on videos captured in real-world scenarios. We employ explicit optical flows to construct a practical coding that enforces physical constraints on generated frames and mediates independent frame-wise scores. By leveraging this coding, maintaining temporal consistency in the generated videos can be framed as an optimization problem with a closed-form solution. To ensure compatibility with Stable Diffusion, we also suggest a workaround for modifying observed-space scores in latent-space Diffusion Models. Notably, MeDM does not require fine-tuning or test-time optimization of the Diffusion Models."> 7<meta property="og:description" content="MeDM utilizes pre-trained image Diffusion Models for video-to-video translation with consistent temporal flow. The proposed framework can render videos from scene position information, such as a normal G-buffer, or perform text-guided editing on videos captured in real-world scenarios. We employ explicit optical flows to construct a practical coding that enforces physical constraints on generated frames and mediates independent frame-wise scores. By leveraging this coding, maintaining temporal consistency in the generated videos can be framed as an optimization problem with a closed-form solution. To ensure compatibility with Stable Diffusion, we also suggest a workaround for modifying observed-space scores in latent-space Diffusion Models. Notably, MeDM does not require fine-tuning or test-time optimization of the Diffusion Models."> 8<meta property="og:image" content="https://medm2023.github.io/images/system-diagram.jpg"> 9<meta name="twitter:card" content="summary_large_image"> 10<link rel="preload" as="fetch" crossorigin="anonymous" href="/_payload.json?08a1a25d-fb84-4903-9527-0d0a8fce2643"> 11<style>.header[data-v-1f19a5f7]{margin:0 auto;max-width:1200px;padding:2rem}.title[data-v-1f19a5f7]{font-size:2.1rem;margin-bottom:1rem}.title[data-v-1f19a5f7],.venue[data-v-1f19a5f7]{font-weight:500;text-align:center}.venue[data-v-1f19a5f7]{font-size:1.5rem}.affiliation-list[data-v-1f19a5f7],.author-list[data-v-1f19a5f7]{-moz-column-gap:1.5rem;column-gap:1.5rem;display:flex;flex-wrap:wrap;font-size:1.2rem;justify-content:center;text-align:center}.author-list>div[data-v-1f19a5f7]{white-space:nowrap}@media (max-width:850px){.title[data-v-1f19a5f7] div{color:var(--color-text-mute);font-size:1.2rem}}@media (min-width:850px){.title[data-v-1f19a5f7] div{display:inline;font-weight:inherit}.title[data-v-1f19a5f7] div:before{content:": ";font-weight:inherit}}.link-list[data-v-1f19a5f7]{display:flex;flex-wrap:wrap;font-size:1.2rem;gap:1rem 2rem;justify-content:center;margin-top:2rem}.link-list>a[data-v-1f19a5f7]:after{all:initial}.link-list>a>div[data-v-1f19a5f7]{background-color:var(--color-background-accent);border-radius:100px;color:var(--color-text-accent);padding:.2rem 1.3rem;transition:transform .2s}@media (hover:hover){.link-list>a>div[data-v-1f19a5f7]:hover{transform:scale(1.05)}}.link-list>a>div[data-v-1f19a5f7]:active{transform:scale(1.05)}@media (prefers-color-scheme:dark){.link-list>a>div[data-v-1f19a5f7]{border:1px solid #fff}}i[data-v-1f19a5f7]{margin-right:.1rem}</style> 12<style>canvas[data-v-3aeb909d]{cursor:grabbing;width:100%}video[data-v-3aeb909d]{height:1px;position:absolute;top:50%;z-index:-1}.video-and-canvas[data-v-3aeb909d]{border-radius:5px;line-height:0;overflow:hidden;position:relative}.loading[data-v-3aeb909d]{align-items:center;backdrop-filter:blur(10px);-webkit-backdrop-filter:blur(10px);background-color:color-mix(in srgb,var(--color-background) 65%,transparent);display:flex;height:100%;justify-content:center;position:absolute;width:100%}.loading>i[data-v-3aeb909d]{font-size:3rem}.zoom-fade-leave-active[data-v-3aeb909d]{transition:all .5s ease-in}.zoom-fade-leave-to[data-v-3aeb909d]{opacity:0;transform:scale(2)}
12</style> 13<style>.medium-zoom-image--opened[data-v-3e6d5202],.medium-zoom-overlay[data-v-3e6d5202]{border:initial;border-radius:0;z-index:999}img[data-v-3e6d5202]{border-radius:5px;width:100%}@media (prefers-color-scheme:light){img[data-v-3e6d5202]{border:.1rem solid #ddd}}</style> 14<style>.captions[data-v-d7d48621]{align-items:flex-end;display:flex;gap:.5rem}.captions>div[data-v-d7d48621]{font-weight:500;text-align:center}.captions>div>canvas[data-v-d7d48621]{display:block;width:100%}.main[data-v-d7d48621]{display:flex;gap:.5rem}.options[data-v-d7d48621]{align-items:center;display:flex;gap:.3rem;justify-content:center;margin-top:.3rem}@media (min-width:850px){input[type=checkbox][data-v-d7d48621]{zoom:1.5}}@media (max-width:450px){input[type=checkbox][data-v-d7d48621]{zoom:.8}}</style> 15<style>svg[data-v-c136a475]{width:1rem}</style> 16<style>.tab-list[data-v-5f227498]{display:flex;justify-content:center;margin-top:1rem}.tab-list>div[data-v-5f227498]{background-color:hsla(240,4%,77%,.655);border-radius:50%;cursor:pointer;height:.4rem;margin:0 .3rem 1rem;transition:transform .1s;transition:background-color .5s;width:.4rem}.tab-list>div[data-v-5f227498]:active{transform:scale(.93)}.tab-list>div>div[data-v-5f227498]{background-color:rgba(0,0,0,.56);border-radius:50%;height:100%;width:100%}@media (prefers-color-scheme:dark){.tab-list>div>div[data-v-5f227498]{background-color:#fff}}.main[data-v-5f227498]{align-items:center;display:flex;justify-content:space-between}.video[data-v-5f227498]{align-items:flex-end;display:flex;flex-wrap:wrap;width:86%}.video .caption[data-v-5f227498]{color:var(--color-text-mute);width:100%}.navigate-button[data-v-5f227498]{align-items:center;aspect-ratio:1/1;background-color:hsla(240,4%,77%,.655);border-radius:50%;color:rgba(0,0,0,.56);cursor:pointer;display:flex;font-size:2.5vw;justify-content:center;transition:transform .1s;width:5%}@media (min-width:1280px){.navigate-button[data-v-5f227498]{font-size:32px}}.navigate-button[data-v-5f227498]:active{transform:scale(.93)}.left-button[data-v-5f227498]:after{content:"\f104";font:var(--fa-font-solid)}.right-button[data-v-5f227498]:after{content:"\f105";font:var(--fa-font-solid)}.fade-enter-active[data-v-5f227498],.fade-leave-active[data-v-5f227498]{transition:all .3s}.fade-enter-from[data-v-5f227498],.fade-leave-to[data-v-5f227498]{opacity:0}</style> 17<style>.main[data-v-eb011441]{margin:0 auto;max-width:1200px;padding:2rem}.section-title[data-v-eb011441]{font-size:2rem;font-weight:500;margin:2rem 0 .5rem;text-align:center}.caption[data-v-eb011441]{font-size:1rem;line-height:1.6;margin-top:.2rem;padding:0 2.5rem;text-align:center}.image[data-v-eb011441]{line-height:0;margin:1.5rem 0 1rem}@media (max-width:650px){.caption[data-v-eb011441]{font-size:1rem;padding:0}}p[data-v-eb011441]{margin:.5rem 0 1rem}</style> 18<style>.footer[data-v-52b75ba3]:before{background-image:url(/images/backgrounds/wave.png);background-position:bottom;content:" ";height:188px;pointer-events:none;position:absolute;top:-187px;width:100%}.footer[data-v-52b75ba3],.footer[data-v-52b75ba3]:before{background-repeat:no-repeat;background-size:cover}.footer[data-v-52b75ba3]{background-color:var(--color-background-accent);background-image:url(/images/backgrounds/footer.jpg);background-position:top;margin-top:100px;max-width:100%;min-height:300px;position:relative;width:calc(100vw - var(--scrollbarWidth))}.footer[data-v-52b75ba3],.footer a[data-v-52b75ba3]{color:var(--color-text-accent)}.content[data-v-52b75ba3]{margin:0 auto;max-width:900px;padding:2rem;position:relative;z-index:1}.logo[data-v-52b75ba3]{display:flex;padding-bottom:calc(var(--section-gap)/4);place-items:flex-start}img[data-v-52b75ba3]{width:200px}@media (min-width:900px){.footer[data-v-52b75ba3]{display:flex;place-items:center}.content[data-v-52b75ba3]{display:grid;grid-template-columns:1fr 3fr;padding:0 2rem}.logo[data-v-52b75ba3]{padding-bottom:0;padding-right:calc(var(--section-gap)/2)}}</style> 19<style>@import url("https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css");@import url("https://site-assets.fontawesome.com/releases/v6.4.2/css/all.css");:root{--vt-c-white:#fff;--vt-c-white-soft:#f8f8f8;--vt-c-white-mute:#f2f2f2;--vt-c-black:#181818;--vt-c-black-soft:#222;--vt-c-black-mute:#282828;--vt-c-indigo:#121314;--vt-c-divider-light-1:rgba(60,60,60,.29);--vt-c-divider-light-2:rgba(60,60,60,.12);--vt-c-divider-dark-1:rgba(84,84,84,.65);--vt-c-divider-dark-2:rgba(84,84,84,.48);--vt-c-text-light-1:var(--vt-c-indigo);--vt-c-text-light-2:rgba(0,0,0,.66);--vt-c-text-dark-1:var(--vt-c-white);--vt-c-text-dark-2:hsla(0,0%,92%,.64);--vt-c-asblue:#153146;--color-background:var(--vt-c-white);--color-background-soft:var(--vt-c-white-soft);--color-background-mute:var(--vt-c-white-mute);
19--color-background-accent:var(--vt-c-asblue);--color-border:var(--vt-c-divider-light-2);--color-border-hover:var(--vt-c-divider-light-1);--color-heading:var(--vt-c-text-light-1);--color-text:var(--vt-c-text-light-1);--color-text-mute:var(--vt-c-text-light-2);--color-text-accent:var(--vt-c-text-dark-1);--section-gap:160px}@media (prefers-color-scheme:dark){:root{--color-background:var(--vt-c-black);--color-background-soft:var(--vt-c-black-soft);--color-background-mute:var(--vt-c-black-mute);--color-background-accent:var(--vt-c-asblue);--color-border:var(--vt-c-divider-dark-2);--color-border-hover:var(--vt-c-divider-dark-1);--color-heading:var(--vt-c-text-dark-1);--color-text:var(--vt-c-text-dark-1);--color-text-mute:var(--vt-c-text-dark-2);--color-text-accent:var(--vt-c-text-dark-1)}}*,:after,:before{box-sizing:border-box;font-weight:400;margin:0}body{background:var(--color-background);color:var(--color-text);font-family:Inter,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif;font-size:1em;line-height:1.6;text-rendering:optimizeLegibility;transition:color .5s,background-color .5s;-webkit-font-smoothing:antialiased;-webkit-text-size-adjust:100%;-moz-osx-font-smoothing:grayscale}:root{font-size:20px}@media (max-width:850px){:root{font-size:16px}}@media (max-width:450px){:root{font-size:12px}}#app{font-weight:400;margin:0 auto;max-width:1280px;padding:2rem}a{color:var(--color-text);cursor:pointer;position:relative;text-decoration:none;white-space:nowrap}a:after{background-color:rgba(179,153,59,.5);bottom:0;content:"";left:-.1em;position:absolute;right:-.1em;top:66%;transition:top .2s cubic-bezier(0,.8,.13,1);z-index:-1}@media (hover:hover){a:hover:after{top:0}}a:active:after{top:0}@media (min-width:1024px){body{display:flex;place-items:center}#app{display:grid;grid-template-columns:1fr 1fr;padding:0 2rem}}</style> 20<link rel="stylesheet" href="/custom_assets/entry.CZQ6s-nn.css"> 21<link rel="modulepreload" as="script" crossorigin href="/custom_assets/CTQ8r4_m.js"> 22<link rel="prefetch" as="script" crossorigin href="/custom_assets/DjXeRPM2.js"> 23<link rel="prefetch" as="script" crossorigin href="/custom_assets/BwsXbxg_.js">
24<script type="module" src="/custom_assets/CTQ8r4_m.js" crossorigin></script>
24</head><body><div id="__nuxt"><!--[--><div class="header" data-v-1f19a5f7><div class="title" data-v-1f19a5f7>MeDM<div>Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance</div></div><div class="venue" data-v-1f19a5f7>AAAI 2024</div><div class="author-list" data-v-1f19a5f7><!--[--><div data-v-1f19a5f7><a href="https://ernestchu.github.io" target="_blank" data-v-1f19a5f7>Ernie Chu</a><!----></div><div data-v-1f19a5f7><!--[-->Tzuhsuan Huang<!--]--><!----></div><div data-v-1f19a5f7><!--[-->Shuo-Yen Lin<!--]--><!----></div><div data-v-1f19a5f7><a href="https://www.citi.sinica.edu.tw/pages/pullpull" target="_blank" data-v-1f19a5f7>Jun-Cheng Chen</a><!----></div><!--]--></div><div class="affiliation-list" data-v-1f19a5f7><!--[--><div data-v-1f19a5f7><!---->Research Center for Information Technology Innovation, Academia Sinica</div><!--]--></div><div class="link-list" data-v-1f19a5f7><!--[--><a href="https://doi.org/10.1609/aaai.v38i2.27899" target="_blank" data-v-1f19a5f7><div data-v-1f19a5f7><i class="ai ai-doi" data-v-1f19a5f7></i> Paper</div></a><a href="/medm-poster.pdf" target="_blank" data-v-1f19a5f7><div data-v-1f19a5f7><i class="fa-regular fa-file-pdf" data-v-1f19a5f7></i> Poster</div></a><a href="https://youtu.be/9qR88erDr6Q" target="_blank" data-v-1f19a5f7><div data-v-1f19a5f7><i class="fa-brands fa-youtube" data-v-1f19a5f7></i> Video</div></a><a href="https://arxiv.org/abs/2308.10079" target="_blank" data-v-1f19a5f7><div data-v-1f19a5f7><i class="ai ai-arxiv" data-v-1f19a5f7></i> arXiv</div></a><a href="https://github.com/aiiu-lab/MeDM" target="_blank" data-v-1f19a5f7><div data-v-1f19a5f7><i class="fa-brands fa-github" data-v-1f19a5f7></i> Code</div></a><a href="https://github.com/aiiu-lab/MeDM?tab=readme-ov-file#citation" target="_blank" data-v-1f19a5f7><div data-v-1f19a5f7><i class="fa-brands fa-github" data-v-1f19a5f7></i> Citation</div></a><!--]--></div></div><div class="main" data-v-eb011441><div class="video" data-v-eb011441><div class="video-and-canvas" data-v-eb011441 data-v-3aeb909d><!----><video poster="/videos/teaser.jpg" loop muted autoplay playsinline data-v-3aeb909d><source src="/videos/teaser.mp4" data-v-3aeb909d></video><canvas data-v-3aeb909d></canvas></div><div class="caption" data-v-eb011441>MeDM enables temporally consistent video rendering and translation using image Diffusion Models. Slide for interactive comparison. Inputs are on the left.</div></div><div class="section-title" data-v-eb011441>Abstract</div><p data-v-eb011441> MeDM utilizes pre-trained image Diffusion Models for video-to-video translation with consistent temporal flow. The proposed framework can render videos from scene position information, such as a normal G-buffer, or perform text-guided editing on videos captured in real-world scenarios. We employ explicit optical flows to construct a practical coding that enforces physical constraints on generated frames and mediates independent frame-wise scores. By leveraging this coding, maintaining temporal consistency in the generated videos can be framed as an optimization problem with a closed-form solution. To ensure compatibility with Stable Diffusion, we also suggest a workaround for modifying observed-space scores in latent-space Diffusion Models. Notably, MeDM does not require fine-tuning or test-time optimization of the Diffusion Models. </p><div class="image" data-v-eb011441><img src="/images/girl.jpg" data-v-eb011441 data-v-3e6d5202><div class="caption" data-v-eb011441>We extract a 20-pixel-wide vertical segment of pixels from each generated frame and stack them horizontally. MeDM produces fluent videos which reconstruct stripe-free images. <a data-v-eb011441><!--[-->Show<!--]--> video.</a></div></div><!----><div class="section-title" data-v-eb011441>Architecture</div><div class="image" data-v-eb011441><img src="/images/system-diagram.jpg" data-v-eb011441 data-v-3e6d5202><div class="caption" data-v-eb011441>MeDM mediates independent image score estimations after every denoising step. Inspired by the fact that video pixels are essentially views to the underlying objects, we construct an explicit pixel repository <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewbox="0 0 10 8" version="1.1" data-v-eb011441 data-v-c136a475><defs data-v-c136a475><g data-v-c136a475><symbol overflow="visible" id="glyph0-0" data-v-c136a475><path style="stroke:none;" d="M 4.03125 -6.265625 C 5.953125 -6.265625 6.453125 -5.796875 6.453125 -5.140625 C 6.453125 -4.53125 5.96875 -3.328125 4.328125 -3.265625 C 3.84375 -3.25 3.5 -2.90625 3.5 -2.796875 C 3.5 -2.734375 3.53125 -2.734375 3.54687
245 -2.734375 C 3.96875 -2.65625 4.171875 -2.46875 4.6875 -1.28125 C 5.140625 -0.234375 5.40625 0.21875 5.984375 0.21875 C 7.203125 0.21875 8.34375 -1 8.34375 -1.21875 C 8.34375 -1.296875 8.265625 -1.296875 8.234375 -1.296875 C 8.109375 -1.296875 7.71875 -1.15625 7.515625 -0.875 C 7.359375 -0.640625 7.140625 -0.328125 6.65625 -0.328125 C 6.140625 -0.328125 5.828125 -1.03125 5.5 -1.8125 C 5.28125 -2.3125 5.109375 -2.6875 4.875 -2.953125 C 6.3125 -3.484375 7.296875 -4.53125 7.296875 -5.5625 C 7.296875 -6.8125 5.625 -6.8125 4.109375 -6.8125 C 3.125 -6.8125 2.5625 -6.8125 1.71875 -6.453125 C 0.390625 -5.859375 0.203125 -5.03125 0.203125 -4.953125 C 0.203125 -4.890625 0.25 -4.875 0.3125 -4.875 C 0.46875 -4.875 0.703125 -5.015625 0.78125 -5.0625 C 0.984375 -5.203125 1.015625 -5.265625 1.078125 -5.453125 C 1.21875 -5.859375 1.5 -6.203125 2.75 -6.265625 C 2.703125 -5.65625 2.609375 -4.71875 2.265625 -3.265625 C 2 -2.15625 1.640625 -1.0625 1.203125 0.015625 C 1.140625 0.125 1.140625 0.140625 1.140625 0.15625 C 1.140625 0.21875 1.21875 0.21875 1.25 0.21875 C 1.453125 0.21875 1.859375 -0.015625 1.96875 -0.203125 C 2 -0.265625 3.265625 -3.078125 3.5625 -6.265625 Z M 4.03125 -6.265625 " data-v-c136a475></path></symbol></g></defs><g id="surface1" data-v-c136a475><g style="fill:currentColor;fill-opacity:1;" data-v-c136a475><use xlink:href="#glyph0-0" x="0.716016" y="7.766016" data-v-c136a475></use></g></g></svg> to represent the underlying world. For more details, please refer to our <a href="/medm.pdf" target="_blank" data-v-eb011441>paper</a>. </div></div><div class="section-title" data-v-eb011441>Video Rendering</div><p data-v-eb011441> MeDM is capable of efficiently rendering high quality videos solely from 3D assets, including optical flows, occlusions and position information (depth, normal). We use the lineart derived from the normal maps as the input conditions to ControlNet. 3D assets from <a href="http://sintel.is.tue.mpg.de" target="_blank" data-v-eb011441>MPI Sintel</a>. </p><span data-v-eb011441>Loading gallery...</span><p data-v-eb011441> In addition, MeDM can also begin with adding noise to the pre-rendered videos and perform the denoising process from 0.5T step following SDEdit. The generated video should be similar to the original animation and incorporating the realistic prior from the pre-trained DM (The difference is especially significant in complex texture, such as hair). More quantitative results, including comparison with prior works, can be found in our <a href="/medm.pdf" target="_blank" data-v-eb011441>paper</a>. </p><span data-v-eb011441>Loading gallery...</span><div class="section-title" data-v-eb011441>Text-Guided Video Edit</div><p data-v-eb011441> MeDM also performs well without high precision optical flows. We demonstrate this by applying text-guided video editing on real-world videos in <a href="https://davischallenge.org/davis2016/code.html" target="_blank" data-v-eb011441>DAVIS 2016</a>. </p><span data-v-eb011441>Loading gallery...</span><div class="section-title" data-v-eb011441>Video Anonymization</div><p data-v-eb011441> Finally, we demonstrate the versatility of MeDM. For example, MeDM can perform video anonymization out-of-the-box. We leverage the fact that human visual perception exhibits a remarkable sensitivity to human faces while our ability to detect and recognize other objects is not as specialized. We add noise to a video with a strength of 0.5T, which is strong enough to erase the identity while preserving other objects and the background scene, and perform denoising using MeDM to obtain the anonymized video. Text conditioning can also be injected to enable a more targeted identity modification. Celebrity videos from <a href="https://celebv-hq.github.io" target="_blank" data-v-eb011441>CelebV-HQ</a>. </p><span data-v-eb011441>Loading gallery...</span></div><div class="footer" data-v-52b75ba3><div class="content" data-v-52b75ba3><div class="logo" data-v-52b75ba3><img src="/images/logos/as-white.png" alt="" data-v-52b75ba3></div><div data-v-52b75ba3>
24 This website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International <a href="https://creativecommons.org/licenses/by-sa/4.0/" target="_blank" data-v-52b75ba3>License</a>. <br data-v-52b75ba3><br data-v-52b75ba3> This means you are free to borrow the <a href="https://github.com/medm2023/medm2023.github.io" target="_blank" data-v-52b75ba3>source code</a> of this website, we just ask that you link back to this page in the footer. However, the copyright to the logo of Academia Sinica and the footer theme is reserved. Please remove them if your are not affiliated with Academia Sinica. </div></div></div><!--]--></div><div id="teleports"></div>
24<script type="application/json" id="__NUXT_DATA__" data-ssr="true" data-src="/_payload.json?08a1a25d-fb84-4903-9527-0d0a8fce2643">[{"state":1,"once":3,"_errors":5,"serverRendered":7,"path":8,"prerenderedAt":9},["Reactive",2],{},["Reactive",4],["Set"],["Reactive",6],{},true,"/",1717078182667]</script>
vendor: 1 bytes, line 24
24
25<script>window.__NUXT__={};window.__NUXT__.config={public:{gtag:{enabled:true,id:"G-D8TKSGXPDM",initCommands:[],config:{},tags:[],loadingStrategy:"defer",url:"https://www.googletagmanager.com/gtag/js"}},app:{baseURL:"/",buildAssetsDir:"custom_assets",cdnURL:""}}</script>
25</body></html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.