1<!DOCTYPE html> 2<html> 3 4<head lang="en"> 5 <meta charset="UTF-8"> 6 <meta http-equiv="x-ua-compatible" content="ie=edge"> 7 8 <title>POV</title> 9 10 <meta name="description" content=""> 11 <meta name="viewport" content="width=device-width, initial-scale=1"> 12 13 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css"> 14 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css"> 15 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.css"> 16 <link rel="stylesheet" href="css/main.css"> 17 <link rel="icon" href="images/icon.png"> 18 19
19<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
19 20
20<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
20 21
21<script src="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.js"></script>
21 22
22<script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/1.5.3/clipboard.min.js"></script>
22 23
23<script src="js/main.js"></script>
23 24</head> 25 26<body> 27<div class="container" id="main"> 28 <div class="row"> 29 <h1 class="col-md-12 text-center"> 30 (MM '23) POV: Prompt-Oriented View-Agnostic Learning for <br> Egocentric Hand-Object Interaction in the Multi-View World<br> 31 </h1> 32 <!-- <venue class="col-md-12 text-center"> 33 <font size="5"> <b>ACM Multimedia 2023 </b></font> 34 </venue> --> 35 36 </div> 37 <div class="row meta"> 38 <div class="col-md-12 text-center"> 39 <ul class="list-inline" style="padding-top: 10px; padding-bottom: 5px"> 40 <li> 41 <!-- <a href="https://xuboshen.github.io/"></a> --> 42 <font size="4"> Boshen Xu </font> 43 </li> 44 <li> 45 <font size="4"> Sipeng Zheng </font> 46 </li> 47 <li> 48 <font size="4"> Qin Jin</font> <!-- <sup>†</sup>--> 49 </li> 50 </ul> 51 <!-- <ul style="padding-top: 0px; padding-bottom: 0px"> 52 <font size="4"> <sup>†</sup>: Corresponding author</font> 53 </ul> --> 54 <ul style="padding-top: 0px; padding-bottom: 0px"> 55 <font size="4"> Renmin University of China </font> 56 </ul> 57 </div> 58 </div> 59 60 61 <div class="row meta"> 62 <div class="col-md-4 col-md-offset-4 text-center"> 63 <ul class="nav nav-pills nav-justified"> 64 <li> 65 <a href="https://dl.acm.org/doi/10.1145/3581783.3612484"> 66 <image src="images/pdf.png" height="50px"></image> 67 <br> 68 <strong>MM'23</strong> 69 </a> 70 </li> 71<!-- <li>--> 72<!-- <a href="https://youtu.be/GwQhnG6krCI">--> 73<!-- <image src="images/youtube.png" height="50px"></image>--> 74<!-- <br>--> 75<!-- <strong>Video</strong>--> 76<!-- </a>--> 77<!-- </li>--> 78 <li> 79 <a href="https://github.com/xuboshen/pov_acmmm2023"> 80 <image src="images/github.png" height="50px"></image> 81 <br> 82 <strong>Code</strong> 83 </a> 84 </li> 85 </ul> 86 </div> 87 </div> 88 89 90 <div class="row"> 91 <div class="col-md-8 col-md-offset-2 text-center"> 92 <h2> 93 Abstract 94 </h2> 95 <p class="text-justify"> 96 We humans are good at translating third-person observations of 97hand-object interactions (HOI) into an egocentric view. However, 98current methods struggle to replicate this ability of view adaptation 99from third-person to first-person. Although some approaches attempt to learn view-agnostic representation from large-scale video 100datasets, they ignore the relationships among multiple third-person 101views. To this end, we propose a Prompt-Oriented View-agnostic 102learning (POV) framework in this paper, which enables this view 103adaptation with few egocentric videos. Specifically, We introduce 104interactive masking prompts at the frame level to capture fine-grained action information, and view-aware prompts at the token 105level to learn view-agnostic representation. To verify our method, 106we establish two benchmarks for transferring from multiple third-person views to the egocentric view. Our extensive experiments 107on these benchmarks demonstrate the efficiency and effectiveness 108of our POV framework and prompt tuning techniques in terms of 109view adaptation and view generalization. 110 </p> 111 112 <table style="margin: 20px"> 113 <tr> 114 <td> 115 <image src="images/motivation.png" class="img-responsive center-block" style="max-width: 60%;"></image> 116 </td> 117 </tr> 118 </table> 119 120 121 </div> 122 </div> 123 124 125<!-- <div class="row">--> 126<!-- <div class="col-md-8 col-md-offset-2">--> 127<!-- <h2>--> 128<!-- Video--> 129<!-- </h2>--> 130<!-- <div class="text-center">--> 131<!-- <div style="position:relative;padding-top:56.5%;">--> 132<!-- <iframe src="https://www.youtube.com/embed/GwQhnG6krCI" allowfullscreen style="position:absolute;top:0;left:0;width:100%;height:100%;"></iframe>--> 133<!-- </div>--> 134<!-- </div>--> 135<!-- </div>--> 136<!-- </div>--> 137 138 139 <div class="row"> 140 <div class="col-md-8 col-md-offset-2"> 141 <h2> 142 Method 143 </h2> 144 <h3> 145 POV Framework 146 </h3> 147 <p> 148 Our prompt-oriented view-agnostic learning framework, which trains a model through two optimization 149sub-tasks and one optional sub-task: (1) <b>prompt-based action understanding</b>, which incorporates interactive masking prompts 150into frames to pre-train the entire model on third-person videos; (2) <b>view-agnostic prompt tuning</b>, where only view-aware 151prompts are fine-tuned through cross-view alignment and cross-entropy loss. (3) <b>egocentric fine-tuning</b>, where the model is 152optionally fine-tuned on limited egocentric videos. 153 <figure> 154 <image src="images/arch.png" class="img-responsive center-block" style="max-width: 100%;"></image> 155 </figure> 156 157 158 <h3> 159 Specific Pipeline 160 </h3> 161 The training and inference pipeline of our POV on Assembly101 and its view split: 162 <figure> 163 <image src="images/assembly_pipeline.png" class="img-responsive center-block" style="max-width: 100%;"></image> 164 </figure> 165 166 <p> 167 The training and inference pipeline of our POV on H2O and its view split: 168 </p> 169 <figure> 170 <image src="images/h2o_pipeline.png" class="img-responsive center-block" style="max-width: 100%;"></image> 171 </figure> 172 173 </div> 174 </div> 175 176 177 <div class="row"> 178 <div class="col-md-8 col-md-offset-2"> 179 <h2> 180 Experiment 181 </h2> 182 <h3> 183 Quantatitive Results 184 </h3>
185 <p class="text-justify"> 186 We evaluate our POV framework on two benchmarks, Assembly101 and H2O, which are established for view adaptation and view generalization, respectively. Table 1 and Table 2 shows the view adaptation results of our POV framework on Assembly101 and H2O. We can see that our POV framework achieves the best performance on both benchmarks. 187 </p> 188 <p> 189 After fine-tuning on unlabeled egocentric videos, the results of our POV on Assembly101 and H2O datasets: 190 </p> 191 <figure> 192 <image src="images/table1.png" class="img-responsive center-block" style="max-width: 97%;"></image> 193 </figure> 194 <p> 195 After fine-tuning on labeled egocentric videos, the results of our POV on Assembly101 and H2O datasets: 196 </p> 197 <figure> 198 <image src="images/table2.png" class="img-responsive center-block" style="max-width: 50%;"></image> 199 </figure> 200 <p> 201 Table 3 and Table 4 shows the view generalization results on egocentric view an third-person view, respectively. We can see that our POV framework achieves promising performance on both benchmarks. 202 </p> 203 <figure> 204 <image src="images/table3.png" class="img-responsive center-block" style="max-width: 80%;"></image> 205 </figure> 206 <figure> 207 <image src="images/table4.png" class="img-responsive center-block" style="max-width: 50%;"></image> 208 </figure> 209 210 211 <h3> 212 Quanlitative results 213 <figure> 214 <image src="images/ass_quanlitative_result.png" class="img-responsive center-block" style="max-width: 100%;"></image> 215 </figure> 216 <figure> 217 <image src="images/h2o_quanlitative_result.png" class="img-responsive center-block" style="max-width: 100%;"></image> 218 </figure> 219 220 </h3> 221 </div> 222 </div> 223 224 225 <div class="row"> 226 <div class="col-md-9 col-md-offset-2"> 227 <h2> 228 Citation 229 </h2> 230 <div class="form-group col-md-10 col-md-offset-0"> 231 <textarea id="bibtex" class="form-control" readonly> 232@inproceedings{xu2023pov, 233 title={POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-view World}, 234 author={Xu, Boshen and Zheng, Sipeng and Jin, Qin}, 235 booktitle={Proceedings of the 31st ACM International Conference on Multimedia}, 236 pages={2807--2816}, 237 year={2023} 238}</textarea> 239 </div> 240 </div> 241 </div> 242 243 244 245 246 <div class="row"> 247 <div class="col-md-8 col-md-offset-2"> 248 The website template was borrowed from <a href="https://github.com/jonbarron/website">Jon's website</a>. 249 </div> 250 </div> 251</div> 252<br> 253</body> 254</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.