PageSourceSearch

https://vwirth.github.io/sharpy

html vwirth.github.io collected 2026-10-03 10:09:12 UTC 19,207 bytes, 421 lines download raw bytes

1<!DOCTYPE html>
2<html>
3
4<head>
5  <meta charset="utf-8">
6  <meta name="description"
7    content="ShaRPy creates personalized digital twins of hands in RGB-D sequences and estimates the uncertainty that remains in the process.">
8  <meta name="keywords" content="ShaRPy, pose and shape estimation, uncertainty, musculoskeletal diseases">
9  <meta name="viewport" content="width=device-width, initial-scale=1">
10  <title>ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty</title>
11
12
13  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
14
15  <link rel="stylesheet" href="./sharpy/static/css/bulma.min.css">
16  <link rel="stylesheet" href="./sharpy/static/css/bulma-carousel.min.css">
17  <link rel="stylesheet" href="./sharpy/static/css/bulma-slider.min.css">
18  <link rel="stylesheet" href="./sharpy/static/css/fontawesome.all.min.css">
19  <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
20  <link rel="stylesheet" href="./sharpy/static/css/index.css">
21  <!-- <link rel="icon" href="./sharpy/static/images/favicon.png"> -->
22  <link rel="icon"
23    href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%2210 0 100 100%22><text y=%22.90em%22 font-size=%2290%22>🖐🏻</text></svg>">
24  
24<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
24
25  
25<script defer src="./sharpy/static/js/fontawesome.all.min.js"></script>
25
26  
26<script src="./sharpy/static/js/bulma-carousel.min.js"></script>
26
27  
27<script src="./sharpy/static/js/bulma-slider.min.js"></script>
27
28  
28<script src="./sharpy/static/js/index.js"></script>
28
29</head>
30
31<body>
32  <a href="https://vwirth.github.io">Back to personal page</a>
33
34  <section class="hero">
35    <div class="hero-body">
36      <div class="container is-max-desktop">
37        <div class="columns is-centered">
38          <div class="column has-text-centered">
39            <h1 class="title is-1 publication-title">ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D
40              with Uncertainty</h1>
41            <div class="is-size-5 publication-authors">
42              <span class="author-block">
43                <a href="https://vwirth.github.io">Vanessa Wirth</a><sup>1</sup>,</span>
44              <span class="author-block">
45                <a href="https://www.medizin3.uk-erlangen.de/forschung/arbeitsgruppen/ag-dr-a-m-liphardt/">Anna-Maria
46                  Liphardt</a><sup>1,2</sup>,</span>
47              <span class="author-block">
48                <a href="https://www.ltd.tf.fau.de/ltd/institute/team/coppers-birte/">Birte Coppers</a><sup>1,2</sup>,
49              </span>
50              <span class="author-block">
51                <a href="https://www.lhft.eei.fau.de/en/archive/person/braeunig-johanna-m-sc">Johanna
52                  Bräunig</a><sup>1</sup>,
53              </span>
54              <span class="author-block">
55                <a href="https://www.ltd.tf.fau.de/person/simon-heinrich/">Simon Heinrich</a><sup>1</sup>,
56              </span>
57              <span class="author-block">
58                <a href="https://www.ltd.tf.fau.de/person/sigrid-leyendecker/">Sigrid Leyendecker</a><sup>1</sup>,
59              </span>
60              <span class="author-block">
61                <a
62                  href="https://www.medizin3.uk-erlangen.de/forschung/arbeitsgruppen/ag-pd-dr-a-kleyer-und-pd-dr-d-simon/">Arnd
63                  Kleyer</a><sup>1,2</sup>,
64              </span>
65              <span class="author-block">
66                <a href="https://www.fau.eu/fau/organisation-and-committees/executive-board/prof-dr-georg-schett/">Georg
67                  Schett</a><sup>1,2</sup>,
68              </span>
69              <span class="author-block">
70                <a href="https://www.lhft.eei.fau.de/archive/person/prof-dr-ing-martin-vossiek">Martin
71                  Vossiek</a><sup>1</sup>,
72              </span>
73              <span class="author-block">
74                <a href="https://eggerbernhard.ch/">Bernhard Egger</a><sup>1</sup>,
75              </span>
76              <span class="author-block">
77                <a href="https://www.lgdv.tf.fau.de/person/marc-stamminger/">Marc Stamminger</a><sup>1</sup>
78              </span>
79            </div>
80
81            <div class="is-size-5 publication-authors">
82              <span class="author-block"><sup>1</sup>Friedrich-Alexander-Universität Erlangen-Nürnberg,</span>
83              <span class="author-block"><sup>2</sup>University Hospital Erlangen</span>
84            </div>
85
86            <h2 style="text-align: center; font-size:1.6em; font-weight: 800;">ICCVW 2023</h2>
87
88            <div class="column has-text-centered">
89              <div class="publication-links">
90                <!-- PDF Link. -->
91                <span class="link-block">
92                  <a href="https://openaccess.thecvf.com/content/ICCV2023W/CVAMD/html/Wirth_ShaRPy_Shape_Reconstruction_and_Hand_Pose_Estimation_from_RGB-D_with_ICCVW_2023_paper.html"
93                    class="external-link button is-normal is-rounded is-dark">
94                    <span class="icon">
95                      <i class="fas fa-file-pdf"></i>
96                    </span>
97                    <span>Paper</span>
98                  </a>
99                </span>
100                <span class="link-block">
101                  <a href="./sharpy/docs/supplementary.pdf" class="external-link button is-normal is-rounded is-dark">
102                    <span class="icon">
103                      <i class="fas fa-file-pdf"></i>
104                    </span>
105                    <span>Supplementary</span>
106                  </a>
107                </span>
108                <span class="link-block">
109                  <a href="./sharpy/static/docs/poster.pdf" class="external-link button is-normal is-rounded is-dark">
110                    <span class="icon">
111                      <i class="fas fa-file-pdf"></i>
112                    </span>
113                    <span>Poster</span>
114                  </a>
115                </span>
116                <span class="link-block">
117                  <a href="https://www.youtube.com/watch?v=kWdoqi2cPt8"
118                    class="external-link button is-normal is-rounded is-dark">
119                    <span class="icon">
120                      <i class="fa fa-play"></i>
121                    </span>
122                    <span>Presentation</span>
123                  </a>
124                </span>
125                <span class="link-block">
126                  <a href="https://arxiv.org/abs/2303.10042" class="external-link button is-normal is-rounded is-dark">
127                    <span class="icon">
128                      <i class="ai ai-arxiv"></i>
129                    </span>
130                    <span>arXiv</span>
131                  </a>
132                </span>
133                <!-- Video Link. -->
134                <!--                 <span class="link-block">
135                  <a href="https://www.youtube.com/watch?v=MrKrnHhk8IA"
136                    class="external-link button is-normal is-rounded is-dark">
137                    <span class="icon">
138                      <i class="fab fa-youtube"></i>
139                    </span>
140                    <span>Video</span>
141                  </a>
142                </span> -->
143                <!-- Code Link. -->
144                <span class="link-block">
145                  <a href="https://github.com/vwirth/sharpy/" class="external-link button is-normal is-rounded is-dark">
146                    <span class="icon">
147                      <i class="fab fa-github"></i>
148                    </span>
149                    <span>Code</span>
150                  </a>
151                </span>
152                <!-- Dataset Link. -->
153                <!--                 <span class="link-block">
154                  <a href="https://github.com/google/nerfies/releases/tag/0.1"
155                    class="external-link button is-normal is-rounded is-dark">
156                    <span class="icon">
157                      <i class="far fa-images"></i>
158                    </span>
159                    <span>Data</span>
160                  </a> -->
161              </div>
162
163
164            </div>
165          </div>
166        </div>
167      </div>
168    </div>
169  </section>
170
171  <section class="hero teaser">
172    <div class="container is-max-desktop">
173      <div class="hero-body">
174        <video id="teaser" autoplay controls muted loop playsinline height="100%">
175          <source src="./sharpy/static/videos/results.webm" type="video/mp4">
176        </video>
177        <h2 class="subtitle has-text-centered">
178          ShaRPy estimates the pose and shape parameters of a digital hand model, and estimates the uncertainty that
179          remains in those in order to discard unreliable measurements.
180        </h2>
181      </div>
182    </div>
183  </section>
184
185
186  <section class="hero is-light is-small">
187    <div class="hero-body">
188      <div class="container">
189        <div id="results-carousel" class="carousel results-carousel">
190          <div class="item item-0000">
191            <video poster="" id="0000-rgb" autoplay muted loop playsinline height="20%">
192              <source src="./sharpy/static/videos/0000_rgb.webm" type="video/mp4">
193            </video>
194            <video poster="" id="0000-sharpy" autoplay muted loop playsinline height="20%">
195              <source src="./sharpy/static/videos/0000_sharpy.webm" type="video/mp4">
196            </video>
197          </div>
198          <!-- <div class="item item-0001">
199            <video poster="" id="0001-rgb" autoplay muted loop playsinline height="20%">
200              <source src="./sharpy/static/videos/0001_rgb.webm" type="video/mp4">
201            </video>
202            <video poster="" id="0001-sharpy" autoplay muted loop playsinline height="20%">
203              <source src="./sharpy/static/videos/0001_sharpy.webm" type="video/mp4">
204            </video>
205          </div> -->
206          <div class="item item-0002">
207            <video poster="" id="0002-rgb" autoplay muted loop playsinline height="20%">
208              <source src="./sharpy/static/videos/0002_rgb.webm" type="video/mp4">
209            </video>
210            <video poster="" id="0002-sharpy" autoplay muted loop playsinline height="20%">
211              <source src="./sharpy/static/videos/0002_sharpy.webm" type="video/mp4">
212            </video>
213          </div>
214          <div class="item item-0004">
215            <video poster="" id="0004-rgb" autoplay muted loop playsinline height="20%">
216              <source src="./sharpy/static/videos/0004_rgb.webm" type="video/mp4">
217            </video>
218            <video poster="" id="0004-sharpy" autoplay muted loop playsinline height="20%">
219              <source src="./sharpy/static/videos/0004_sharpy.webm" type="video/mp4">
220            </video>
221          </div>
222          <div class="item item-0003">
223            <video poster="" id="0003-rgb" autoplay muted loop playsinline height="20%">
224              <source src="./sharpy/static/videos/0003_rgb.webm" type="video/mp4">
225            </video>
226            <video poster="" id="0003-sharpy" autoplay muted loop playsinline height="20%">
227              <source src="./sharpy/static/videos/0003_sharpy.webm" type="video/mp4">
228            </video>
229          </div>
230          <div class="item item-0005">
231            <video poster="" id="0005-rgb" autoplay muted loop playsinline height="20%">
232              <source src="./sharpy/static/videos/0005_rgb.webm" type="video/mp4">
233            </video>
234            <video poster="" id="0005-sharpy" autoplay muted loop playsinline height="20%">
235              <source src="./sharpy/static/videos/0005_sharpy.webm" type="video/mp4">
236            </video>
237          </div>
238          <div class="item item-0006">
239            <video poster="" id="0006-rgb" autoplay muted loop playsinline height="20%">
240              <source src="./sharpy/static/videos/0006_rgb.webm" type="video/mp4">
241            </video>
242            <video poster="" id="0006-sharpy" autoplay muted loop playsinline height="20%">
243              <source src="./sharpy/static/videos/0006_sharpy.webm" type="video/mp4">
244            </video>
245          </div>
246        </div>
247      </div>
248    </div>
249  </section>
250
251
252
253
254
255  <section class="section">
256    <div class="container is-max-desktop">
257      <!-- Abstract. -->
258      <div class="columns is-centered has-text-centered">
259        <div class="column is-four-fifths">
260          <h2 class="title is-3">Abstract</h2>
261          <div class="content has-text-justified">
262            <p>
263              Despite their potential, markerless hand tracking technologies are not yet applied
264              in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases.
265              One reason is that the focus of most methods lies in the reconstruction of coarse,
266              plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are
267              required.
268            </p>
269            <p>
270              Therefore, we propose ShaRPy, the first RGB-D <b>Sha</b>
270pe <b>R</b>econstruction and hand <b>P</b>ose
271              tracking system,
272              which provides uncertaint<b>y</b> estimates of the computed pose,
273              e.g., when a finger is hidden or its estimate is inconsistent with the observations in the input,
274              to guide clinical decision-making. Besides pose, ShaRPy approximates a personalized hand shape,
275              promoting a more realistic and intuitive understanding of its digital twin.
276              Our method requires only a light-weight setup with a single consumer-level RGB-D camera
277              yet it is able to distinguish similar poses with only small joint angle deviations in a
278              metrically accurate space.
279              This is achieved by combining a data-driven dense correspondence predictor with traditional energy
280              minimization.
281              To bridge the gap between interactive visualization and biomedical simulation we leverage a parametric
282              hand model in
283              which we incorporate biomedical constraints and optimize for both, its pose and hand shape.
284            </p>
285            <p>
286              We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results of hand function
287              assessments for activity monitoring of musculoskeletal diseases.
288            </p>
289
290          </div>
291        </div>
292      </div>
293      <!--/ Abstract. -->
294    </div>
295  </section>
296
297  <section class="hero is-light is-small">
298    <div class="hero-body is-light">
299      <!-- <div class="container"> -->
300      <div class="columns">
301        <div class="column is-one-quarter"></div>
302        <div class="column is-one-quarter">
303          <h2 class="title is-5" style="text-align:center;">Poster</h2>
304
305          <a href="./sharpy/static/docs/poster.pdf">
306            <img
307              style="width:70%;margin-left:auto;margin-right:auto;display:block;border-color: grey;border-style:solid"
308              src=" ./sharpy/static/docs/poster_thumbnail.png" />
309          </a>
310
311        </div>
312
313        <div class="column is-one-quarter">
314          <h2 class="title is-5" style="text-align:center;">Paper</h2>
315          <a href="./sharpy/static/docs/paper.pdf">
316            <img
317              style="width:79%;margin-left:auto;margin-right:auto;display:block;border-color: grey;border-style:solid"
318              src=" ./sharpy/static/docs/paper_thumbnail.png" />
319          </a>
320        </div>
321      </div>
322      <div class="column is-one-quarter"></div>
323    </div>
324
325  </section>
326
327  <section class="section">
328    <div class="container is-max-desktop content">
329      <div class="columns is-centered has-text-centered">
330        <div class="column is-four-fifths">
331          <h2 class="title is-3">Presentation</h2>
332          <div class="publication-video">
333            <iframe src="https://www.youtube.com/embed/kWdoqi2cPt8?si=wGwBNubUgO0__mSM" title="YouTube video player"
334              frameborder="0"
335              allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
336              allowfullscreen></iframe>
337          </div>
338          <!-- 
339          <video id="results" controls muted loop playsinline height="100%">
340            <source src="./sharpy/static/videos/23iccv_sharpy.webm" type="video/mp4">
341          </video> -->
342        </div>
343      </div>
344    </div>
345  </section>
346
347
348
349  <section class="section" id="BibTeX">
350    <div class="container is-max-desktop content">
351      <h2 class="title">BibTeX</h2>
352      <pre><code>@InProceedings{Wirth_2023_ICCV,
353          author    = {Wirth, Vanessa and Liphardt, Anna-Maria and Coppers, Birte and Br\"aunig, Johanna and Heinrich, Simon and Leyendecker, Sigrid and Kleyer, Arnd and Schett, Georg and Vossiek, Martin and Egger, Bernhard and Stamminger, Marc},
354          title     = {ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty},
355          booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
356          month     = {October},
357          year      = {2023},
358          pages     = {2625-2633}
359      }</code></pre>
360    </div>
361  </section>
362
363  <section class="section" id="Acknowledgements">
364    <div class="container is-max-desktop content">
365      <h2 class="title">Acknowledgements</h2>
366
367      <p>
368        This work was funded by the Deutsche
369        Forschungsgemeinschaft (DFG, German Research Foundation) – SFB 1483 – Project-ID 442419336, <a
370          href="https://empkins.de">EmpkinS</a>.
371      </p>
372      <p>
373        This work used the German Research Foundation (DFG)
374        funded major instrument (reference number INST90 / 985-
375        1 FUGG) at the Institute of Applied Dynamics (Sigrid
376        Leyendecker), Friedrich-Alexander Universität Erlangen-
377        Nürnberg Germany.
378      </p>
379      <p>
380        The authors gratefully acknowledge
381        the scientific support and HPC resources provided by the
382        Erlangen National High Performance Computing Center of
383        the Friedrich-Alexander-Universität Erlangen-Nürnberg.
384      </p>
385    </div>
386  </section>
387
388
389
390  <footer class="footer">
391    <div class="container">
392      <div class="content has-text-centered">
393        <a class="icon-link" href="./sharpy/static/docs/paper.pdf">
394          <i class="fas fa-file-pdf"></i>
395        </a>
396        <a class="icon-link" href="https://github.com/vwirth" class="external-link" disabled>
397          <i class="fab fa-github"></i>
398        </a>
399      </div>
400      <div class="columns is-centered">
401        <div class="column is-8">
402          <div class="content">
403            <p>
404              This website is licensed under a <a rel="license"
405                href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
406                Commons Attribution-ShareAlike 4.0 International License</a>.
407            </p>
408            <p>
409              Source code of this website is stolen from the <a
410                href="https://github.com/nerfies/nerfies.github.io">nerfies</a>
411              project page.
412            </p>
413          </div>
414        </div>
415      </div>
416    </div>
417  </footer>
418
419</body>
420
421</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.