1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <meta name="description" 6 content="Efficient Human Motion Reconstruction from Monocular Videos with Physical Consistency Loss."> 7 <meta name="keywords" content="Physics, Animation, Optimization, Human Motion"> 8 <meta name="viewport" content="width=device-width, initial-scale=1"> 9 <title>Efficient Human Motion Reconstruction</title> 10 11 12 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 13 rel="stylesheet"> 14 15 <link rel="stylesheet" href="./static/css/bulma.min.css"> 16 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 17 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 18 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 19 <link rel="stylesheet" 20 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 21 <link rel="stylesheet" href="./static/css/index.css"> 22 <link rel="icon" href="./static/images/favicon.svg"> 23 24
24<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
24 25
25<script defer src="./static/js/fontawesome.all.min.js"></script>
25 26
26<script src="./static/js/bulma-carousel.min.js"></script>
26 27
27<script src="./static/js/bulma-slider.min.js"></script>
27 28
28<script src="./static/js/index.js"></script>
28 29</head> 30<body> 31 32<!-- <nav class="navbar" role="navigation" aria-label="main navigation"> 33 <div class="navbar-brand"> 34 <a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false"> 35 <span aria-hidden="true"></span> 36 <span aria-hidden="true"></span> 37 <span aria-hidden="true"></span> 38 </a> 39 </div> 40 <div class="navbar-menu"> 41 <div class="navbar-start" style="flex-grow: 1; justify-content: center;"> 42 <a class="navbar-item" href="https://hitlyn.github.io/"> 43 <span class="icon"> 44 <i class="fas fa-home"></i> 45 </span> 46 </a> 47 48 <div class="navbar-item has-dropdown is-hoverable"> 49 <a class="navbar-link"> 50 More Research 51 </a> 52 <div class="navbar-dropdown"> 53 <a class="navbar-item" href="https://hitlyn.github.io/blog/2020/08/12/Self-Adapting-Recurrent-Models-for-Object-Pushing-from-Learning-in-Simulation"> 54 Reinforcement Learning Dexterous 55 </a> 56 <a class="navbar-item" href="https://hitlyn.github.io/blog/2020/08/14/Attention-for-Reinforcement-Learning"> 57 Attention Learning 58 </a> 59 <a class="navbar-item" href="https://hitlyn.github.io/blog/2020/08/12/Self-Adapting-Recurrent-Models-for-Object-Pushing-from-Learning-in-Simulation"> 60 RMPPI 61 </a> 62 <a class="navbar-item" href="https://hitlyn.github.io/blog/2020/08/12/Self-Adapting-Recurrent-Models-for-Object-Pushing-from-Learning-in-Simulation"> 63 Reinforcement Learning Pushing 64 </a> 65 </div> 66 </div> 67 </div> 68 69 </div> 70</nav> --> 71 72 73<section class="hero"> 74 <div class="hero-body"> 75 <div class="container is-max-desktop"> 76 <div class="columns is-centered"> 77 <div class="column has-text-centered"> 78 <h1 class="title is-1 publication-title">Efficient Human Motion Reconstruction from Monocular Videos with Physical Consistency Loss</h1> 79 <div class="is-size-5 publication-authors"> 80 <span class="author-block"> 81 <a href="https://hitlyn.github.io/">Lin Cong</a><sup>1</sup>*,</span> 82 <span class="author-block"> 83 <a href="https://tams.informatik.uni-hamburg.de/people/ruppel/">Philipp Rupel</a><sup>1</sup>*,</span> 84 <span class="author-block"> 85 <a href="https://cfcs.pku.edu.cn/english/people/faculty/yizhouwang/index.htm">Yizhou Wang</a><sup>2</sup>, 86 </span> 87 <span class="author-block"> 88 <a href="https://www.zhihu.com/people/woluanxiede">Xiang Pan</a><sup>3</sup>, 89 </span> 90 <span class="author-block"> 91 <a href="https://tams.informatik.uni-hamburg.de/people/hendrich/">Norman Hendrich</a><sup>1</sup>, 92 </span> 93 <span class="author-block"> 94 <a href="https://tams.informatik.uni-hamburg.de/people/zhang/">Jianwei Zhang</a><sup>1</sup>, 95 </span> 96 </div> 97 98 <div class="is-size-5 publication-authors"> 99 <span class="author-block"><sup>1</sup>University of Hamburg</span> 100 <span class="author-block"><sup>2</sup>Peking University</span> 101 <span class="author-block"><sup>3</sup>Zhihu</span> 102 </div> 103 104 <div class="column has-text-centered"> 105 <div class="publication-links"> 106 <!-- PDF Link. --> 107 <span class="link-block"> 108 <a href="https://dl.acm.org/doi/10.1145/3610548.3618169" 109 class="external-link button is-normal is-rounded is-dark"> 110 <span class="icon"> 111 <i class="fas fa-file-pdf"></i> 112 </span> 113 <span>Paper</span> 114 </a> 115 </span> 116 <!-- Video Link. --> 117 <span class="link-block"> 118 <a href="https://www.youtube.com/watch?v=XWgKF8hXung" 119 class="external-link button is-normal is-rounded is-dark"> 120 <span class="icon"> 121 <i class="fab fa-youtube"></i> 122 </span>
123 <span>Video</span> 124 </a> 125 </span> 126 <!-- Code Link. --> 127 <span class="link-block"> 128 <a href="https://github.com/TAMS-Group/efficient-motion-reconstruction" 129 class="external-link button is-normal is-rounded is-dark"> 130 <span class="icon"> 131 <i class="fab fa-github"></i> 132 </span> 133 <span>Code</span> 134 </a> 135 </span> 136 137 </div> 138 139 </div> 140 </div> 141 </div> 142 </div> 143 </div> 144</section> 145 146<section class="hero teaser"> 147 <div class="container is-max-desktop"> 148 <div class="hero-body"> 149 <video id="teaser" autoplay muted loop playsinline height="100%"> 150 <source src="./static/videos/teaser.mp4" 151 type="video/mp4"> 152 </video> 153 <h2 class="subtitle has-text-centered"> 154 Our method can reconstruct complex human motions from monocular videos in minutes. 155 </h2> 156 </div> 157 </div> 158</section> 159 160 161<section class="hero is-light is-small"> 162 <div class="hero-body"> 163 <div class="container"> 164 <div id="results-carousel" class="carousel results-carousel"> 165 <div class="item item-steve"> 166 <video poster="" id="steve" autoplay controls muted loop playsinline height="60%"> 167 <source src="./static/videos/demo10.mp4" 168 type="video/mp4"> 169 </video> 170 </div> 171 <div class="item item-chair-tp"> 172 <video poster="" id="chair-tp" autoplay controls muted loop playsinline height="60%"> 173 <source src="./static/videos/demo9.mp4" 174 type="video/mp4"> 175 </video> 176 </div> 177 <div class="item item-shiba"> 178 <video poster="" id="shiba" autoplay controls muted loop playsinline height="60%"> 179 <source src="./static/videos/demo8.mp4" 180 type="video/mp4"> 181 </video> 182 </div> 183 <div class="item item-fullbody"> 184 <video poster="" id="fullbody" autoplay controls muted loop playsinline height="60%"> 185 <source src="./static/videos/demo7.mp4" 186 type="video/mp4"> 187 </video> 188 </div> 189 <div class="item item-blueshirt"> 190 <video poster="" id="blueshirt" autoplay controls muted loop playsinline height="60%"> 191 <source src="./static/videos/demo6.mp4" 192 type="video/mp4"> 193 </video> 194 </div> 195 <div class="item item-mask"> 196 <video poster="" id="mask" autoplay controls muted loop playsinline height="60%"> 197 <source src="./static/videos/demo5.mp4" 198 type="video/mp4"> 199 </video> 200 </div> 201 <div class="item item-coffee"> 202 <video poster="" id="coffee" autoplay controls muted loop playsinline height="60%"> 203 <source src="./static/videos/demo4.mp4" 204 type="video/mp4"> 205 </video> 206 </div> 207 <div class="item item-toby"> 208 <video poster="" id="toby" autoplay controls muted loop playsinline height="60%"> 209 <source src="./static/videos/demo3.mp4" 210 type="video/mp4"> 211 </video> 212 </div> 213 <div class="item item-toby"> 214 <video poster="" id="toby" autoplay controls muted loop playsinline height="60%"> 215 <source src="./static/videos/demo2.mp4" 216 type="video/mp4"> 217 </video> 218 </div> 219 <div class="item item-toby"> 220 <video poster="" id="toby" autoplay controls muted loop playsinline height="60%"> 221 <source src="./static/videos/demo1.mp4" 222 type="video/mp4"> 223 </video> 224 </div> 225 </div> 226 </div> 227 </div> 228</section> 229 230 231<section class="section"> 232 <div class="container is-max-desktop"> 233 <!-- Abstract. --> 234 <div class="columns is-centered has-text-centered"> 235 <div class="column is-four-fifths"> 236 <h2 class="title is-3">Abstract</h2> 237 <div class="content has-text-justified"> 238 <p> 239 Vision-only motion reconstruction from monocular videos often produces artifacts such as foot sliding and jittering. 240 Existing physics-based methods typically either simplify the problem to focus solely on foot-ground contacts, 241 or they reconstruct full-body contacts within a physics simulator, 242 necessitating the solution of a time-consuming bilevel optimization problem. 243 To overcome these limitations, we present an efficie
243nt gradient-based method for reconstructing complex human motions (including highly dynamic and acrobatic movements) with physical constraints. 244 Our approach reformulates human motion dynamics through a differentiable physical consistency loss within an augmented search space that accounts both for contacts and camera alignment. 245 This enables us to transform the motion reconstruction task into a single-level trajectory optimization problem. 246 Experimental results demonstrate that our method can reconstruct complex human motions from real-world videos in minutes, 247 which is substantially faster than previous approaches. 248 Additionally, the reconstructed results show enhanced physical realism compared to existing methods. 249 </p> 250 </div> 251 </div> 252 </div> 253 <!--/ Abstract. --> 254 255 <!-- Paper video. --> 256 <div class="columns is-centered has-text-centered"> 257 <div class="column is-four-fifths"> 258 <h2 class="title is-3">Video</h2> 259 <div class="publication-video"> 260 <iframe width="560" height="315" src="https://www.youtube.com/embed/XWgKF8hXung?si=omCRQBdQOZvxCHe7" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe> 261 </div> 262 </div> 263 </div> 264 <!--/ Paper video. --> 265 </div> 266</section> 267 268 269<section class="section"> 270 <div class="container is-max-desktop"> 271 272 <div class="columns is-centered"> 273 274 <!-- Optimization. --> 275 <div class="column"> 276 <div class="content"> 277 <h2 class="title is-3">Optimization Process</h2> 278 <p> 279 280We demonstrate the reconstruction process of several complex motions, with the majority of acrobatic movements being successfully reconstructed in under two minutes. 281 </p> 282 <video id="dollyzoom" autoplay controls muted loop playsinline height="90%"> 283 <source src="./static/videos/optimization.mp4" 284 type="video/mp4"> 285 </video> 286 </div> 287 </div> 288 <!--/ Optimization. --> 289 290 <!-- Backflip. --> 291 <div class="column"> 292 <h2 class="title is-3">Interations</h2> 293 <div class="columns is-centered"> 294 <div class="column content"> 295 <p> 296 Through the visualization of various iterations in the optimization process, we can clearly observe the significant enhancement in physical reasoning. 297 </p> 298 <video id="matting-video" controls playsinline height="100%"> 299 <source src="./static/videos/backflip.mp4" 300 type="video/mp4"> 301 </video> 302 </div> 303 304 </div> 305 </div> 306 </div> 307 <!--/ Backflip. --> 308 309 <!-- Animation. --> 310 <div class="columns is-centered"> 311 <div class="column is-full-width"> 312 <h2 class="title is-3">Animation</h2> 313 314 <!-- Unreal Rendering. --> 315 <h3 class="title is-4">Animation rendering in Unreal Engine</h3> 316 <div class="content has-text-justified"> 317 <p> 318 The optimization results can be directly used to drive the animation in Unreal Engine. 319 </p> 320 </div> 321 <div class="content has-text-centered"> 322 <video id="replay-video" 323 controls 324 muted 325 preload 326 playsinline 327 width="75%"> 328 <source src="./static/videos/animation1.mp4" 329 type="video/mp4"> 330 </video> 331 </div> 332 <!-- Unreal Rendering. --> 333 334 <!-- Multiple cameras setup. --> 335 <h3 class="title is-4">Multiple cameras setup</h3> 336 <div class="content has-text-justified"> 337 <p> 338 We also support multiple camera setups. The keypoints are collected by an open-source software <a href="https://freemocap.org/" target="_blank">FreeMocap</a>. 339 </p> 340 </div> 341 <div class="content has-text-centered"> 342 <video id="replay-video" 343 controls 344 muted 345 preload 346 playsinline 347 width="75%"> 348 <source src="./static/videos/animation2.mp4" 349 type="video/mp4"> 350 </video> 351 </div> 352 <!-- Multiple cameras setup. --> 353 354 </div> 355 </div> 356 <!--/ Animation. --> 357 358 359 360 361 </div> 362</section> 363 364 365<section class="section" id="BibTeX">
366 <div class="container is-max-desktop content"> 367 <h2 class="title">BibTeX</h2> 368 <pre><code>@inproceedings{cong2023efficient, 369 author = {Cong, Lin and Ruppel, Philipp and Wang, Yizhou and Pan, Xiang and Hendrich, Norman and Zhang, Jianwei}, 370 title = {Efficient Human Motion Reconstruction from Monocular Videos with Physical Consistency Loss}, 371 journal = {Siggraph Asia}, 372 year = {2023}, 373}</code></pre> 374 </div> 375</section> 376 377 378<footer class="footer"> 379 <div class="container"> 380 <div class="content has-text-centered"> 381 <a class="icon-link" 382 href="./static/videos/nerfies_paper.pdf"> 383 <i class="fas fa-file-pdf"></i> 384 </a> 385 </div> 386 <div class="columns is-centered"> 387 <div class="column is-8"> 388 <div class="content"> 389 <p> 390 This website is licensed under a <a rel="license" 391 href="http://creativecommons.org/licenses/by-sa/4.0/">Creative 392 Commons Attribution-ShareAlike 4.0 International License</a>. 393 </p> 394 <p> 395 Template adapted from <a href="https://github.com/nerfies/nerfies.github.io">this website</a> 396 397 </p> 398 </div> 399 </div> 400 </div> 401 </div> 402</footer> 403 404</body> 405</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.