1<!DOCTYPE html> 2<html lang="en-US"> 3 <head> 4 <meta charset="UTF-8"> 5 6
7<script type="text/x-mathjax-config"> 8 MathJax.Hub.Config({ 9 tex2jax: { 10 inlineMath: [ ['$','$'], ["\\(","\\)"] ], 11 processEscapes: true 12 } 13 }); 14</script>
vendor: 1 bytes, line 14
14
15<script 16 type="text/javascript" 17 charset="utf-8" 18 src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" 19> 20</script>
vendor: 1 bytes, line 20
20
21<script 22 type="text/javascript" 23 charset="utf-8" 24 src="https://vincenttam.github.io/javascripts/MathJaxLocal.js" 25> 26</script>
26 27 28 29<!-- Begin Jekyll SEO tag v2.8.0 --> 30<title>MaskViT: Masked Visual Pre-Training for Video Prediction | MaskViT</title> 31<meta name="generator" content="Jekyll v3.9.2" /> 32<meta property="og:title" content="MaskViT: Masked Visual Pre-Training for Video Prediction" /> 33<meta property="og:locale" content="en_US" /> 34<meta name="description" content="CoRL 2022 submission" /> 35<meta property="og:description" content="CoRL 2022 submission" /> 36<link rel="canonical" href="https://maskvit.github.io/" /> 37<meta property="og:url" content="https://maskvit.github.io/" /> 38<meta property="og:site_name" content="MaskViT" /> 39<meta property="og:type" content="website" /> 40<meta name="twitter:card" content="summary" /> 41<meta property="twitter:title" content="MaskViT: Masked Visual Pre-Training for Video Prediction" />
42<script type="application/ld+json"> 43{"@context":"https://schema.org","@type":"WebSite","description":"CoRL 2022 submission","headline":"MaskViT: Masked Visual Pre-Training for Video Prediction","name":"MaskViT","url":"https://maskvit.github.io/"}</script>
43 44<!-- End Jekyll SEO tag --> 45 46 <link rel="preconnect" href="https://fonts.gstatic.com"> 47 <link rel="preload" href="https://fonts.googleapis.com/css?family=Open+Sans:400,700&display=swap" as="style" type="text/css" crossorigin> 48 <meta name="viewport" content="width=device-width, initial-scale=1"> 49 <meta name="theme-color" content="#157878"> 50 <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> 51 <link rel="stylesheet" href="/assets/css/style.css?v=b23efc011b8af9ab6152f2f5df2854495d874077"> 52 <!-- start custom head snippets, customize with your own _includes/head-custom.html file --> 53 54<!-- Setup Google Analytics --> 55 56 57 58<!-- You can set your favicon here --> 59<!-- link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" --> 60 61<!-- end custom head snippets --> 62 63 </head> 64 <body> 65 <a id="skip-to-content" href="#content">Skip to the content.</a> 66 67 <header class="page-header" role="banner"> 68 <h1 class="project-name">MaskViT: <br> Masked Visual Pre-Training for Video Prediction</h1> 69 <h2 class="project-authors"><span class="nowrap"> Agrim Gupta<sup>1</sup>, </span> <span class="nowrap"> Stephen Tian<sup>1</sup> </span>, <span class="nowrap"> Yunzhi Zhang<sup>1</sup>, </span> <span class="nowrap"> Jiajun Wu<sup>1</sup>, </span> <span class="nowrap"> Roberto MartÃn-MartÃn<sup>2, 1</sup> </span>, <span class="nowrap"> Li Fei-Fei<sup>1</sup> </span></h2> 70 <h2 class="project-affiliations"><sup>1</sup> Stanford University, <sup>2</sup> Salesforce AI</h2> 71 72 <!-- <h2 class="project-tagline">CoRL 2022 submission</h2> --> 73 <!-- --> 74 75 <a href="https://arxiv.org/abs/2206.11894" class="btn">arXiV</a> 76 <a href="https://github.com/agrimgupta92/maskvit" class="btn">Code</a> 77 <a href="#qualitative-results" class="btn">Qualitative Results</a> 78 <a href="#real-robot-experiments" class="btn">Real Robot Experiments</a> 79 <!-- <a href="#video" class="btn">Video</a> --> 80 81 82 </header> 83 84 <main id="content" class="main-content" role="main"> 85 <!--  --> 86<div class="diagram_container"> 87 <video class="model_diagram" playsinline="" autoplay="" muted="" loop="" onloadstart="this.playbackRate = 1;"> 88 <source src="resources/model_sfri.mp4" type="video/mp4" /> 89 <source src="resources/model.mp4" type="video/mp4" /> 90 <source src="resources/model.webm" type="video/webm" /> 91 MaskViT model diagram (your browser couldn't load the video). 92 </video> 93</div> 94 95<h2 id="abstract">Abstract</h2> 96 97<p>The ability to predict future visual observations conditioned on past observations and motor commands can enable embodied agents to plan solutions to a variety of tasks in complex environments. This work shows that we can create good video prediction models by pre-training transformers via masked visual modeling. Our approach, named MaskViT, is based on two simple design decisions. First, for memory and training efficiency, we use two types of window attention: spatial and spatiotemporal. Second, during training, we mask a <em>variable</em> percentage of tokens instead of a <em>fixed</em> mask ratio. For inference, MaskViT generates all tokens via iterative refinement where we incrementally decrease the masking ratio following a mask scheduling function. On several datasets we demonstrate that MaskViT outperforms prior works in video prediction, is parameter efficient, and can generate high-resolution videos (\(256\times256\)). Further, we demonstrate the benefits of inference speedup (up to \(512\times\)) due to iterative decoding by using MaskViT for planning on a real robot. Our work suggests that we can endow embodied agents with powerful predictive models by leveraging the general framework of masked visual modeling with minimal domain knowledge.</p> 98 99<hr /> 100 101<h2 id="qualitative-results">Qualitative results</h2> 102<p>Here we present additional qualitative videos of results from MaskViT. Ground truth videos are on the left of each pair while predicted videos are on the right. A red border indicates that the frame is a (VQ-GAN reconstructed) context frame, and a green border indicates a predicted frame.</p> 103 104<h3 id="bair-64x64-action-free">BAIR (64x64, action-free)</h3> 105<div style="text-align: center"> 106 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 107 <source src="resources/bair/bair_indiv_0.mp4" type="video/mp4" /> 108 <source src="resources/bair/bair_indiv_0_sfri.mp4" type="video/mp4" /> 109 <source src="resources/bair/bair_indiv_0.webm" type="video/webm" /> 110 Your browser does not support the video tag. 111 </video> 112 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 113 <source src="resources/bair/bair_indiv_1.mp4" type="video/mp4" /> 114 <source src="resources/bair/bair_indiv_1_sfri.mp4" type="video/mp4" /> 115 <source src="resources/bair/bair_indiv_1.webm" type="video/webm" /> 116 Your browser does not support the video tag. 117 </video> 118 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 119 <source src="resources/bair/bair_indiv_2.mp4" type="video/mp4" /> 120 <source src="resources/bair/bair_indiv_2_sfri.mp4" type="video/mp4" /> 121 <source src="resources/bair/bair_indiv_2.webm" type="video/webm" /> 122 Your browser does not support the video tag. 123 </video> 124 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 125 <source src="resources/bair/bair_indiv_3.mp4" type="video/mp4" /> 126 <source src="resources/bair/bair_indiv_3_sfri.mp4" type="video/mp4" /> 127 <source src="resources/bair/bair_indiv_3.webm" type="video/webm" /> 128 Your browser does not support the video tag. 129 </video> 130 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 131 <source src="resources/bair/bair_indiv_4.mp4" type="video/mp4" /> 132 <source src="resources/bair/bair_indiv_4_sfri.mp4" type="video/mp4" /> 133 <source src="resources/bair/bair_indiv_4.webm" type="video/webm" /> 134 Your browser does not support the video tag. 135 </video> 136 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 137 <source src="resources/bair/bair_indiv_5.mp4" type="video/mp4" /> 138 <source src="resources/bair/bair_indiv_5_sfri.mp4" type="video/mp4" /> 139 <source src="resources/bair/bair_indiv_5.webm" type="video/webm" /> 140 Your browser does not support the video tag. 141 </video> 142 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 143 <source src="resources/bair/bair_indiv_6.mp4" type="video/mp4" /> 144 <source src="resources/bair/bair_indiv_6_sfri.mp4" type="video/mp4" /> 145 <source src="resources/bair/bair_indiv_6.webm" type="video/webm" /> 146 Your browser does not support the video tag. 147 </video> 148 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 149 <source src="resources/bair/bair_indiv_7.mp4" type="video/mp4" /> 150 <source src="resources/bair/bair_indiv_7_sfri.mp4" type="video/mp4" /> 151 <source src="resources/bair/bair_indiv_7.webm" type="video/webm" /> 152 Your browser does not support the video tag. 153 </video> 154 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 155 <source src="resources/bair/bair_indiv_8.mp4" type="video/mp4" /> 156 <source src="resources/bair/bair_indiv_8_sfri.mp4" type="video/mp4" /> 157 <source src="resources/bair/bair_indiv_8.webm" type="video/webm" /> 158 Your browser does not support the video tag. 159 </video> 160 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 161 <source src="resources/bair/bair_indiv_9.mp4" type="video/mp4" /> 162 <source src="resources/bair/bair_indiv_9_sfri.mp4" type="video/mp4" /> 163 <source src="resources/bair/bair_indiv_9.webm" type="video/webm" /> 164 Your browser does not support the video tag. 165 </video> 166 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 167 <source src="resources/bair/bair_indiv_10.mp4" type="video/mp4" /> 168 <source src="resources/bair/bair_indiv_10_sfri.mp4" type="video/mp4" /> 169 <source src="resources/bair/bair_indiv_10.webm" type="video/webm" /> 170 Your browser does not support the video tag. 171 </video> 172 <video class="results_video_64" playsinline="" autoplay="" muted="" loop=""> 173 <source src="resources/bair/bair_indiv_11.mp4" type="video/mp4" /> 174 <source src="resources/bair/bair_indiv_11_sfri.mp4" type="video/mp4" /> 175 <source src="resources/bair/bair_indiv_11.webm" type="video/webm" /> 176 Your browser does not support the video tag. 177 </video> 178 179 180</div> 181 182<hr /> 183<h3 id="kitti-256x256">KITTI (256x256)</h3> 184 185<div style="text-align: center"> 186 <div> 187 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 188 <source src="resources/kitti/kitti_indiv_0.mp4" type="video/mp4" /> 189 <source src="resources/kitti/kitti_indiv_0_sfri.mp4" type="video/mp4" /> 190 <source src="resources/kitti/kitti_indiv_0.webm" type="video/webm" /> 191 Your browser does not support the video tag. 192 </video> 193 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 194 <source src="resources/kitti/kitti_indiv_1.mp4" type="video/mp4" /> 195 <source src="resources/kitti/kitti_indiv_1_sfri.mp4" type="video/mp4" /> 196 <source src="resources/kitti/kitti_indiv_1.webm" type="video/webm" /> 197 Your browser does not support the video tag. 198 </video> 199 </div> 200 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 201 <source src="resources/kitti/kitti_indiv_2.mp4" type="video/mp4" /> 202 <source src="resources/kitti/kitti_indiv_2_sfri.mp4" type="video/mp4" /> 203 <source src="resources/kitti/kitti_indiv_2.webm" type="video/webm" /> 204 Your browser does not support the video tag. 205 </video> 206 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 207 <source src="resources/kitti/kitti_indiv_3.mp4" type="video/mp4" /> 208 <source src="resources/kitti/kitti_indiv_3_sfri.mp4" type="video/mp4" /> 209 <source src="resources/kitti/kitti_indiv_3.webm" type="video/webm" /> 210 Your browser does not support the video tag. 211 </video> 212 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 213 <source src="resources/kitti/kitti_indiv_4.mp4" type="video/mp4" /> 214 <source src="resources/kitti/kitti_indiv_4_sfri.mp4" type="video/mp4" /> 215 <source src="resources/kitti/kitti_indiv_4.webm" type="video/webm" /> 216 Your browser does not support the video tag. 217 </video> 218 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 219 <source src="resources/kitti/kitti_indiv_5.mp4" type="video/mp4" /> 220 <source src="resources/kitti/kitti_indiv_5_sfri.mp4" type="video/mp4" /> 221 <source src="resources/kitti/kitti_indiv_5.webm" type="video/webm" /> 222 Your browser does not support the video tag. 223 </video> 224 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 225 <source src="resources/kitti/kitti_indiv_6.mp4" type="video/mp4" /> 226 <source src="resources/kitti/kitti_indiv_6_sfri.mp4" type="video/mp4" /> 227 <source src="resources/kitti/kitti_indiv_6.webm" type="video/webm" /> 228 Your browser does not support the video tag. 229 </video> 230 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 231 <source src="resources/kitti/kitti_indiv_7.mp4" type="video/mp4" /> 232 <source src="resources/kitti/kitti_indiv_7_sfri.mp4" type="video/mp4" /> 233 <source src="resources/kitti/kitti_indiv_7.webm" type="video/webm" /> 234 Your browser does not support the video tag. 235 </video> 236 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 237 <source src="resources/kitti/kitti_indiv_8.mp4" type="video/mp4" /> 238 <source src="resources/kitti/kitti_indiv_8_sfri.mp4" type="video/mp4" /> 239 <source src="resources/kitti/kitti_indiv_8.webm" type="video/webm" /> 240 Your browser does not support the video tag. 241 </video> 242 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 243 <source src="resources/kitti/kitti_indiv_9.mp4" type="video/mp4" /> 244 <source src="resources/kitti/kitti_indiv_9_sfri.mp4" type="video/mp4" /> 245 <source src="resources/kitti/kitti_indiv_9.webm" type="video/webm" /> 246 Your browser does not support the video tag. 247 </video> 248 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 249 <source src="resources/kitti/kitti_indiv_10.mp4" type="video/mp4" /> 250 <source src="resources/kitti/kitti_indiv_10_sfri.mp4" type="video/mp4" /> 251 <source src="resources/kitti/kitti_indiv_10.webm" type="video/webm" /> 252 Your browser does not support the video tag. 253 </video> 254 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 255 <source src="resources/kitti/kitti_indiv_11.mp4" type="video/mp4" /> 256 <source src="resources/kitti/kitti_indiv_11_sfri.mp4" type="video/mp4" /> 257 <source src="resources/kitti/kitti_indiv_11.webm" type="video/webm" /> 258 Your browser does not support the video tag. 259 </video> 260</div> 261 262<hr /> 263<h3 id="robonet-256x256">RoboNet (256x256)</h3> 264 265<div style="text-align: center"> 266 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 267 <source src="resources/robonet/robonet_indiv_0.mp4" type="video/mp4" /> 268 <source src="resources/robonet/robonet_indiv_0_sfri.mp4" type="video/mp4" /> 269 <source src="resources/robonet/robonet_indiv_0.webm" type="video/webm" /> 270 Your browser does not support the video tag. 271 </video> 272 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 273 <source src="resources/robonet/robonet_indiv_1.mp4" type="video/mp4" /> 274 <source src="resources/robonet/robonet_indiv_1_sfri.mp4" type="video/mp4" /> 275 <source src="resources/robonet/robonet_indiv_1.webm" type="video/webm" /> 276 Your browser does not support the video tag. 277 </video> 278 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 279 <source src="resources/robonet/robonet_indiv_2.mp4" type="video/mp4" /> 280 <source src="resources/robonet/robonet_indiv_2_sfri.mp4" type="video/mp4" /> 281 <source src="resources/robonet/robonet_indiv_2.webm" type="video/webm" /> 282 Your browser does not support the video tag. 283 </video> 284 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 285 <source src="resources/robonet/robonet_indiv_3.mp4" type="video/mp4" /> 286 <source src="resources/robonet/robonet_indiv_3_sfri.mp4" type="video/mp4" /> 287 <source src="resources/robonet/robonet_indiv_3.webm" type="video/webm" /> 288 Your browser does not support the video tag. 289 </video> 290 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 291 <source src="resources/robonet/robonet_indiv_4.mp4" type="video/mp4" /> 292 <source src="resources/robonet/robonet_indiv_4_sfri.mp4" type="video/mp4" /> 293 <source src="resources/robonet/robonet_indiv_4.webm" type="video/webm" /> 294 Your browser does not support the video tag. 295 </video> 296 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 297 <source src="resources/robonet/robonet_indiv_5.mp4" type="video/mp4" /> 298 <source src="resources/robonet/robonet_indiv_5_sfri.mp4" type="video/mp4" /> 299 <source src="resources/robonet/robonet_indiv_5.webm" type="video/webm" /> 300 Your browser does not support the video tag. 301 </video> 302 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 303 <source src="resources/robonet/robonet_indiv_6.mp4" type="video/mp4" /> 304 <source src="resources/robonet/robonet_indiv_6_sfri.mp4" type="video/mp4" /> 305 <source src="resources/robonet/robonet_indiv_6.webm" type="video/webm" /> 306 Your browser does not support the video tag. 307 </video> 308 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 309 <source src="resources/robonet/robonet_indiv_7.mp4" type="video/mp4" /> 310 <source src="resources/robonet/robonet_indiv_7_sfri.mp4" type="video/mp4" /> 311 <source src="resources/robonet/robonet_indiv_7.webm" type="video/webm" /> 312 Your browser does not support the video tag. 313 </video> 314 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 315 <source src="resources/robonet/robonet_indiv_8.mp4" type="video/mp4" /> 316 <source src="resources/robonet/robonet_indiv_8_sfri.mp4" type="video/mp4" /> 317 <source
317 src="resources/robonet/robonet_indiv_8.webm" type="video/webm" /> 318 Your browser does not support the video tag. 319 </video> 320 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 321 <source src="resources/robonet/robonet_indiv_9.mp4" type="video/mp4" /> 322 <source src="resources/robonet/robonet_indiv_9_sfri.mp4" type="video/mp4" /> 323 <source src="resources/robonet/robonet_indiv_9.webm" type="video/webm" /> 324 Your browser does not support the video tag. 325 </video> 326 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 327 <source src="resources/robonet/robonet_indiv_10.mp4" type="video/mp4" /> 328 <source src="resources/robonet/robonet_indiv_10_sfri.mp4" type="video/mp4" /> 329 <source src="resources/robonet/robonet_indiv_10.webm" type="video/webm" /> 330 Your browser does not support the video tag. 331 </video> 332 <video class="results_video_256" playsinline="" autoplay="" muted="" loop=""> 333 <source src="resources/robonet/robonet_indiv_11.mp4" type="video/mp4" /> 334 <source src="resources/robonet/robonet_indiv_11_sfri.mp4" type="video/mp4" /> 335 <source src="resources/robonet/robonet_indiv_11.webm" type="video/webm" /> 336 Your browser does not support the video tag. 337 </video> 338</div> 339 340<hr /> 341 342<h2 id="real-robot-experiments">Real robot experiments</h2> 343 344<p>We demonstrate how the performance improvements afforded by MaskViT allow it to be leveraged for control through visual planning. We show examples of predicted plans and overall trajectories executed on a Sawyer robot by planning using MaskViT as an action-conditioned forward dynamics model.</p> 345 346<div> 347<div style="text-align: center"> 348 <figure style="display: inline-block"> 349 <img src="resources/robot/bowl_initial_im.jpg" width="200" /> 350 <figcaption align="center"><b>Initial observation</b></figcaption> 351 </figure> 352 <figure style="display: inline-block"> 353 <img src="resources/robot/bowl_goal_im.jpg" width="200" /> 354 <figcaption align="center"><b>Goal observation</b></figcaption> 355 </figure> 356 <figure style="display: inline-block"> 357 <video class="results_video_control" playsinline="" autoplay="" muted="" loop="" onloadstart="this.playbackRate = 1.5;"> 358 <source src="resources/robot/bowl_clip.mp4" type="video/mp4" /> 359 Your browser does not support the video tag. 360 </video> 361 <figcaption align="center"><b> Full execution (sped up) </b></figcaption> 362 </figure> 363</div> 364 365<div style="text-align: center"> 366 <figure style="display: inline-block; width: 65%"> 367 <video class="results_video_plan" playsinline="" autoplay="" muted="" loop=""> 368 <source src="resources/robot/bowl_plan.mp4" type="video/mp4" /> 369 Your browser does not support the video tag. 370 </video> 371 <figcaption align="center"><b>Visualized predictions for best plans at time step 4.</b></figcaption> 372 </figure> 373</div> 374 375<hr style="width:40%; margin:auto" /> 376 377 378<div style="text-align: center"> 379 <figure style="display: inline-block"> 380 <img src="resources/robot/hat_initial_im.jpg" width="200" /> 381 <figcaption align="center"><b>Initial observation</b></figcaption> 382 </figure> 383 <figure style="display: inline-block"> 384 <img src="resources/robot/hat_goal_im.jpg" width="200" /> 385 <figcaption align="center"><b>Goal observation</b></figcaption> 386 </figure> 387 <figure style="display: inline-block"> 388 <video class="results_video_control" playsinline="" autoplay="" muted="" loop="" onloadstart="this.playbackRate = 1.5;"> 389 <source src="resources/robot/hat_clip.mp4" type="video/mp4" /> 390 Your browser does not support the video tag. 391 </video> 392 <figcaption align="center"><b> Full execution (sped up) </b></figcaption> 393 </figure> 394</div> 395 396<div style="text-align: center"> 397 <figure style="display: inline-block; width: 65%"> 398 <video class="results_video_plan" playsinline="" autoplay="" muted="" loop=""> 399 <source src="resources/robot/hat_plan.mp4" type="video/mp4" /> 400 Your browser does not support the video tag. 401 </video> 402 <figcaption align="center"><b>Visualized predictions for best plans at time step 7.</b></figcaption> 403 </figure> 404</div> 405</div> 406 407 408 <footer class="site-footer"> 409 <!--
410 <span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a>.</span> --> 411 </footer> 412 </main> 413 </body> 414</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.