1<!DOCTYPE html> 2<html> 3 4<head> 5 <meta charset="utf-8"> 6 <meta name="description" 7 content="ShaRPy creates personalized digital twins of hands in RGB-D sequences and estimates the uncertainty that remains in the process."> 8 <meta name="keywords" content="ShaRPy, pose and shape estimation, uncertainty, musculoskeletal diseases"> 9 <meta name="viewport" content="width=device-width, initial-scale=1"> 10 <title>ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty</title> 11 12 13 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> 14 15 <link rel="stylesheet" href="./sharpy/static/css/bulma.min.css"> 16 <link rel="stylesheet" href="./sharpy/static/css/bulma-carousel.min.css"> 17 <link rel="stylesheet" href="./sharpy/static/css/bulma-slider.min.css"> 18 <link rel="stylesheet" href="./sharpy/static/css/fontawesome.all.min.css"> 19 <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 20 <link rel="stylesheet" href="./sharpy/static/css/index.css"> 21 <!-- <link rel="icon" href="./sharpy/static/images/favicon.png"> --> 22 <link rel="icon" 23 href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%2210 0 100 100%22><text y=%22.90em%22 font-size=%2290%22>ðð»</text></svg>"> 24
24<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
24 25
25<script defer src="./sharpy/static/js/fontawesome.all.min.js"></script>
25 26
26<script src="./sharpy/static/js/bulma-carousel.min.js"></script>
26 27
27<script src="./sharpy/static/js/bulma-slider.min.js"></script>
27 28
28<script src="./sharpy/static/js/index.js"></script>
28 29</head> 30 31<body> 32 <a href="https://vwirth.github.io">Back to personal page</a> 33 34 <section class="hero"> 35 <div class="hero-body"> 36 <div class="container is-max-desktop"> 37 <div class="columns is-centered"> 38 <div class="column has-text-centered"> 39 <h1 class="title is-1 publication-title">ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D 40 with Uncertainty</h1> 41 <div class="is-size-5 publication-authors"> 42 <span class="author-block"> 43 <a href="https://vwirth.github.io">Vanessa Wirth</a><sup>1</sup>,</span> 44 <span class="author-block"> 45 <a href="https://www.medizin3.uk-erlangen.de/forschung/arbeitsgruppen/ag-dr-a-m-liphardt/">Anna-Maria 46 Liphardt</a><sup>1,2</sup>,</span> 47 <span class="author-block"> 48 <a href="https://www.ltd.tf.fau.de/ltd/institute/team/coppers-birte/">Birte Coppers</a><sup>1,2</sup>, 49 </span> 50 <span class="author-block"> 51 <a href="https://www.lhft.eei.fau.de/en/archive/person/braeunig-johanna-m-sc">Johanna 52 Bräunig</a><sup>1</sup>, 53 </span> 54 <span class="author-block"> 55 <a href="https://www.ltd.tf.fau.de/person/simon-heinrich/">Simon Heinrich</a><sup>1</sup>, 56 </span> 57 <span class="author-block"> 58 <a href="https://www.ltd.tf.fau.de/person/sigrid-leyendecker/">Sigrid Leyendecker</a><sup>1</sup>, 59 </span> 60 <span class="author-block"> 61 <a 62 href="https://www.medizin3.uk-erlangen.de/forschung/arbeitsgruppen/ag-pd-dr-a-kleyer-und-pd-dr-d-simon/">Arnd 63 Kleyer</a><sup>1,2</sup>, 64 </span> 65 <span class="author-block"> 66 <a href="https://www.fau.eu/fau/organisation-and-committees/executive-board/prof-dr-georg-schett/">Georg 67 Schett</a><sup>1,2</sup>, 68 </span> 69 <span class="author-block"> 70 <a href="https://www.lhft.eei.fau.de/archive/person/prof-dr-ing-martin-vossiek">Martin 71 Vossiek</a><sup>1</sup>, 72 </span> 73 <span class="author-block"> 74 <a href="https://eggerbernhard.ch/">Bernhard Egger</a><sup>1</sup>, 75 </span> 76 <span class="author-block"> 77 <a href="https://www.lgdv.tf.fau.de/person/marc-stamminger/">Marc Stamminger</a><sup>1</sup> 78 </span> 79 </div> 80 81 <div class="is-size-5 publication-authors"> 82 <span class="author-block"><sup>1</sup>Friedrich-Alexander-Universität Erlangen-Nürnberg,</span> 83 <span class="author-block"><sup>2</sup>University Hospital Erlangen</span> 84 </div> 85 86 <h2 style="text-align: center; font-size:1.6em; font-weight: 800;">ICCVW 2023</h2> 87 88 <div class="column has-text-centered"> 89 <div class="publication-links"> 90 <!-- PDF Link. --> 91 <span class="link-block"> 92 <a href="https://openaccess.thecvf.com/content/ICCV2023W/CVAMD/html/Wirth_ShaRPy_Shape_Reconstruction_and_Hand_Pose_Estimation_from_RGB-D_with_ICCVW_2023_paper.html" 93 class="external-link button is-normal is-rounded is-dark"> 94 <span class="icon"> 95 <i class="fas fa-file-pdf"></i> 96 </span> 97 <span>Paper</span> 98 </a> 99 </span> 100 <span class="link-block"> 101 <a href="./sharpy/docs/supplementary.pdf" class="external-link button is-normal is-rounded is-dark"> 102 <span class="icon"> 103 <i class="fas fa-file-pdf"></i> 104 </span> 105 <span>Supplementary</span> 106 </a> 107 </span> 108 <span class="link-block"> 109 <a href="./sharpy/static/docs/poster.pdf" class="external-link button is-normal is-rounded is-dark">
110 <span class="icon"> 111 <i class="fas fa-file-pdf"></i> 112 </span> 113 <span>Poster</span> 114 </a> 115 </span> 116 <span class="link-block"> 117 <a href="https://www.youtube.com/watch?v=kWdoqi2cPt8" 118 class="external-link button is-normal is-rounded is-dark"> 119 <span class="icon"> 120 <i class="fa fa-play"></i> 121 </span> 122 <span>Presentation</span> 123 </a> 124 </span> 125 <span class="link-block"> 126 <a href="https://arxiv.org/abs/2303.10042" class="external-link button is-normal is-rounded is-dark"> 127 <span class="icon"> 128 <i class="ai ai-arxiv"></i> 129 </span> 130 <span>arXiv</span> 131 </a> 132 </span> 133 <!-- Video Link. --> 134 <!-- <span class="link-block"> 135 <a href="https://www.youtube.com/watch?v=MrKrnHhk8IA" 136 class="external-link button is-normal is-rounded is-dark"> 137 <span class="icon"> 138 <i class="fab fa-youtube"></i> 139 </span> 140 <span>Video</span> 141 </a> 142 </span> --> 143 <!-- Code Link. --> 144 <span class="link-block"> 145 <a href="https://github.com/vwirth/sharpy/" class="external-link button is-normal is-rounded is-dark"> 146 <span class="icon"> 147 <i class="fab fa-github"></i> 148 </span> 149 <span>Code</span> 150 </a> 151 </span> 152 <!-- Dataset Link. --> 153 <!-- <span class="link-block"> 154 <a href="https://github.com/google/nerfies/releases/tag/0.1" 155 class="external-link button is-normal is-rounded is-dark"> 156 <span class="icon"> 157 <i class="far fa-images"></i> 158 </span> 159 <span>Data</span> 160 </a> --> 161 </div> 162 163 164 </div> 165 </div> 166 </div> 167 </div> 168 </div> 169 </section> 170 171 <section class="hero teaser"> 172 <div class="container is-max-desktop"> 173 <div class="hero-body"> 174 <video id="teaser" autoplay controls muted loop playsinline height="100%"> 175 <source src="./sharpy/static/videos/results.webm" type="video/mp4"> 176 </video> 177 <h2 class="subtitle has-text-centered"> 178 ShaRPy estimates the pose and shape parameters of a digital hand model, and estimates the uncertainty that 179 remains in those in order to discard unreliable measurements. 180 </h2> 181 </div> 182 </div> 183 </section> 184 185 186 <section class="hero is-light is-small"> 187 <div class="hero-body"> 188 <div class="container"> 189 <div id="results-carousel" class="carousel results-carousel"> 190 <div class="item item-0000"> 191 <video poster="" id="0000-rgb" autoplay muted loop playsinline height="20%"> 192 <source src="./sharpy/static/videos/0000_rgb.webm" type="video/mp4"> 193 </video> 194 <video poster="" id="0000-sharpy" autoplay muted loop playsinline height="20%"> 195 <source src="./sharpy/static/videos/0000_sharpy.webm" type="video/mp4"> 196 </video> 197 </div> 198 <!-- <div class="item item-0001"> 199 <video poster="" id="0001-rgb" autoplay muted loop playsinline height="20%"> 200 <source src="./sharpy/static/videos/0001_rgb.webm" type="video/mp4"> 201 </video> 202 <video poster="" id="0001-sharpy" autoplay muted loop playsinline height="20%"> 203 <source src="./sharpy/static/videos/0001_sharpy.webm" type="video/mp4"> 204 </video> 205 </div> --> 206 <div class="item item-0002"> 207 <video poster="" id="0002-rgb" autoplay muted loop playsinline height="20%"> 208 <source src="./sharpy/static/videos/0002_rgb.webm" type="video/mp4"> 209 </video> 210 <video poster="" id="0002-sharpy" autoplay muted loop playsinline height="20%"> 211 <source src="./sharpy/static/videos/0002_sharpy.webm" type="video/mp4"> 212 </video> 213 </div> 214 <div class="item item-0004"> 215 <video poster="" id="0004-rgb" autoplay muted loop playsinline height="20%"> 216 <source src="./sharpy/static/videos/0004_rgb.webm" type="video/mp4"> 217 </video> 218 <video poster="" id="0004-sharpy" autoplay muted loop playsinline height="20%"> 219 <source src="./sharpy/static/videos/0004_sharpy.webm" type="video/mp4"> 220 </video> 221 </div> 222 <div class="item item-0003"> 223 <video poster="" id="0003-rgb" autoplay muted loop playsinline height="20%"> 224 <source src="./sharpy/static/videos/0003_rgb.webm" type="video/mp4"> 225 </video> 226 <video poster="" id="0003-sharpy" autoplay muted loop playsinline height="20%"> 227 <source src="./sharpy/static/videos/0003_sharpy.webm" type="video/mp4"> 228 </video> 229 </div> 230 <div class="item item-0005"> 231 <video poster="" id="0005-rgb" autoplay muted loop playsinline height="20%"> 232 <source src="./sharpy/static/videos/0005_rgb.webm" type="video/mp4"> 233 </video> 234 <video poster="" id="0005-sharpy" autoplay muted loop playsinline height="20%"> 235 <source src="./sharpy/static/videos/0005_sharpy.webm" type="video/mp4"> 236 </video> 237 </div> 238 <div class="item item-0006"> 239 <video poster="" id="0006-rgb" autoplay muted loop playsinline height="20%"> 240 <source src="./sharpy/static/videos/0006_rgb.webm" type="video/mp4"> 241 </video> 242 <video poster="" id="0006-sharpy" autoplay muted loop playsinline height="20%"> 243 <source src="./sharpy/static/videos/0006_sharpy.webm" type="video/mp4"> 244 </video> 245 </div> 246 </div> 247 </div> 248 </div> 249 </section> 250 251 252 253 254 255 <section class="section"> 256 <div class="container is-max-desktop"> 257 <!-- Abstract. --> 258 <div class="columns is-centered has-text-centered"> 259 <div class="column is-four-fifths"> 260 <h2 class="title is-3">Abstract</h2> 261 <div class="content has-text-justified"> 262 <p> 263 Despite their potential, markerless hand tracking technologies are not yet applied 264 in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. 265 One reason is that the focus of most methods lies in the reconstruction of coarse, 266 plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are 267 required. 268 </p> 269 <p> 270 Therefore, we propose ShaRPy, the first RGB-D <b>Sha</b>
270pe <b>R</b>econstruction and hand <b>P</b>ose 271 tracking system, 272 which provides uncertaint<b>y</b> estimates of the computed pose, 273 e.g., when a finger is hidden or its estimate is inconsistent with the observations in the input, 274 to guide clinical decision-making. Besides pose, ShaRPy approximates a personalized hand shape, 275 promoting a more realistic and intuitive understanding of its digital twin. 276 Our method requires only a light-weight setup with a single consumer-level RGB-D camera 277 yet it is able to distinguish similar poses with only small joint angle deviations in a 278 metrically accurate space. 279 This is achieved by combining a data-driven dense correspondence predictor with traditional energy 280 minimization. 281 To bridge the gap between interactive visualization and biomedical simulation we leverage a parametric 282 hand model in 283 which we incorporate biomedical constraints and optimize for both, its pose and hand shape. 284 </p> 285 <p> 286 We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results of hand function 287 assessments for activity monitoring of musculoskeletal diseases. 288 </p> 289 290 </div> 291 </div> 292 </div> 293 <!--/ Abstract. --> 294 </div> 295 </section> 296 297 <section class="hero is-light is-small"> 298 <div class="hero-body is-light"> 299 <!-- <div class="container"> --> 300 <div class="columns"> 301 <div class="column is-one-quarter"></div> 302 <div class="column is-one-quarter"> 303 <h2 class="title is-5" style="text-align:center;">Poster</h2> 304 305 <a href="./sharpy/static/docs/poster.pdf"> 306 <img 307 style="width:70%;margin-left:auto;margin-right:auto;display:block;border-color: grey;border-style:solid" 308 src=" ./sharpy/static/docs/poster_thumbnail.png" /> 309 </a> 310 311 </div> 312 313 <div class="column is-one-quarter"> 314 <h2 class="title is-5" style="text-align:center;">Paper</h2> 315 <a href="./sharpy/static/docs/paper.pdf"> 316 <img 317 style="width:79%;margin-left:auto;margin-right:auto;display:block;border-color: grey;border-style:solid" 318 src=" ./sharpy/static/docs/paper_thumbnail.png" /> 319 </a> 320 </div> 321 </div> 322 <div class="column is-one-quarter"></div> 323 </div> 324 325 </section> 326 327 <section class="section"> 328 <div class="container is-max-desktop content"> 329 <div class="columns is-centered has-text-centered"> 330 <div class="column is-four-fifths"> 331 <h2 class="title is-3">Presentation</h2> 332 <div class="publication-video"> 333 <iframe src="https://www.youtube.com/embed/kWdoqi2cPt8?si=wGwBNubUgO0__mSM" title="YouTube video player" 334 frameborder="0" 335 allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" 336 allowfullscreen></iframe> 337 </div> 338 <!-- 339 <video id="results" controls muted loop playsinline height="100%"> 340 <source src="./sharpy/static/videos/23iccv_sharpy.webm" type="video/mp4"> 341 </video> --> 342 </div> 343 </div> 344 </div> 345 </section> 346 347 348 349 <section class="section" id="BibTeX"> 350 <div class="container is-max-desktop content"> 351 <h2 class="title">BibTeX</h2> 352 <pre><code>@InProceedings{Wirth_2023_ICCV, 353 author = {Wirth, Vanessa and Liphardt, Anna-Maria and Coppers, Birte and Br\"aunig, Johanna and Heinrich, Simon and Leyendecker, Sigrid and Kleyer, Arnd and Schett, Georg and Vossiek, Martin and Egger, Bernhard and Stamminger, Marc}, 354 title = {ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty}, 355 booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops}, 356 month = {October}, 357 year = {2023}, 358 pages = {2625-2633} 359 }</code></pre> 360 </div> 361 </section> 362 363 <section class="section" id="Acknowledgements"> 364 <div class="container is-max-desktop content"> 365 <h2 class="title">Acknowledgements</h2> 366 367 <p> 368 This work was funded by the Deutsche 369 Forschungsgemeinschaft (DFG, German Research Foundation) â SFB 1483 â Project-ID 442419336, <a 370 href="https://empkins.de">EmpkinS</a>. 371 </p> 372 <p> 373 This work used the German Research Foundation (DFG) 374 funded major instrument (reference number INST90 / 985- 375 1 FUGG) at the Institute of Applied Dynamics (Sigrid 376 Leyendecker), Friedrich-Alexander Universität Erlangen- 377 Nürnberg Germany. 378 </p> 379 <p> 380 The authors gratefully acknowledge
381 the scientific support and HPC resources provided by the 382 Erlangen National High Performance Computing Center of 383 the Friedrich-Alexander-Universität Erlangen-Nürnberg. 384 </p> 385 </div> 386 </section> 387 388 389 390 <footer class="footer"> 391 <div class="container"> 392 <div class="content has-text-centered"> 393 <a class="icon-link" href="./sharpy/static/docs/paper.pdf"> 394 <i class="fas fa-file-pdf"></i> 395 </a> 396 <a class="icon-link" href="https://github.com/vwirth" class="external-link" disabled> 397 <i class="fab fa-github"></i> 398 </a> 399 </div> 400 <div class="columns is-centered"> 401 <div class="column is-8"> 402 <div class="content"> 403 <p> 404 This website is licensed under a <a rel="license" 405 href="http://creativecommons.org/licenses/by-sa/4.0/">Creative 406 Commons Attribution-ShareAlike 4.0 International License</a>. 407 </p> 408 <p> 409 Source code of this website is stolen from the <a 410 href="https://github.com/nerfies/nerfies.github.io">nerfies</a> 411 project page. 412 </p> 413 </div> 414 </div> 415 </div> 416 </div> 417 </footer> 418 419</body> 420 421</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.