1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <meta name="description" 6 content=""> 7 <meta name="keywords" content="UniTrack, object tracking"> 8 <meta name="viewport" content="width=device-width, initial-scale=1"> 9 <title>Do Different Tracking Tasks Require Different Appearance Models?</title> 10 11 12 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 13 rel="stylesheet"> 14 15 <link rel="stylesheet" href="./static/css/bulma.min.css"> 16 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 17 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 18 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 19 <link rel="stylesheet" 20 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 21 <link rel="stylesheet" href="./static/css/index.css"> 22 23
23<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
23 24
24<script defer src="./static/js/fontawesome.all.min.js"></script>
24 25
25<script src="./static/js/bulma-carousel.min.js"></script>
25 26
26<script src="./static/js/bulma-slider.min.js"></script>
26 27
27<script src="./static/js/index.js"></script>
27 28</head> 29<body> 30 31 32 33<section class="hero"> 34 <div class="hero-body"> 35 <div class="container is-max-desktop"> 36 <div class="columns is-centered"> 37 <div class="column has-text-centered"> 38 <h1 class="title is-1 publication-title">Do Different Tracking Tasks Require Different Appearance Models?</h1> 39 <div class="is-size-5 publication-authors"> 40 <span class="author-block"> 41 <a href="https://zhongdao.github.io">Zhongdao Wang</a><sup>1</sup>, 42 43 <a href="https://hszhao.github.io">Hengshuang Zhao</a><sup>2</sup>, 44 45 <a href="https://scholar.google.com/citations?user=aYLh7fsAAAAJ&hl=en&oi=ao">Ya-Li Li</a><sup>1</sup>, 46 47 <a href="https://scholar.google.com/citations?user=RgzLZZsAAAAJ&hl=en&oi=ao">Shengjin Wang</a><sup>1</sup>, 48 49 <a href="https://www.robots.ox.ac.uk/~phst/">Philip H.S. Torr</a><sup>2</sup>, 50 51 <a href="https://www.robots.ox.ac.uk/~luca">Luca Bertinetto</a><sup>3</sup> 52 <br /><sup>1</sup>Tsinghua University <sup>2</sup>University of Oxford <sup>3</sup>FIVE AI 53 <p> 54 <div class="brmod"></div>NeurIPS 2021</span> 55 </div> 56 57 58 59 60 <div class="column has-text-centered"> 61 <div class="publication-links"> 62 <!-- PDF Link. --> 63 <span class="link-block"> 64 <a href="https://arxiv.org/abs/2107.02156" 65 class="external-link button is-normal is-rounded is-dark"> 66 <span class="icon"> 67 <i class="ai ai-arxiv"></i> 68 </span> 69 <span>arXiv</span> 70 </a> 71 </span> 72 <!-- Video Link. --> 73 <!-- 74 <span class="link-block"> 75 <a href="https://www.youtube.com/" 76 class="external-link button is-normal is-rounded is-dark"> 77 <span class="icon"> 78 <i class="fab fa-youtube"></i> 79 </span> 80 <span>Video</span> 81 </a> 82 </span> 83 --> 84 <!-- Code Link. --> 85 <span class="link-block"> 86 <a href="https://github.com/Zhongdao/UniTrack" 87 class="external-link button is-normal is-rounded is-dark"> 88 <span class="icon"> 89 <i class="fab fa-github"></i> 90 </span> 91 <span>Code</span> 92 </a> 93 </span> 94 </a> 95 </div> 96 97 </div> 98 </div> 99 </div> 100 </div> 101 </div> 102</section> 103 104<section class="hero teaser"> 105 <div class="container is-max-desktop"> 106 <div class="hero-body"> 107 <img src="static/images/radar.png" 108 class="interpolation-image" 109 alt="Interpolate start reference image."/> 110 </img> 111 <br> 112 113 <h2 class="subtitle has-text-centered"> 114 <span class="dnerf"> We experiment with a large amount of self-supervised representations on five different tracking tasks and present a comprehensive comparison. A polygon with a larger area means better performance across the board. Dashed gray line indicates ImageNet pre-trained representation. 115 </h2> 116 </div> 117 </div> 118</section> 119 120 121<!-- <section class="hero is-light is-small"> 122 <div class="hero-body"> 123 <div class="container"> 124 <div id="results-carousel" class="carousel results-carousel"> 125 <div class="item item-steve"> 126 <video poster="" id="steve" autoplay controls muted loop height="100%"> 127 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/steve.mp4" 128 type="video/mp4"> 129 </video> 130 </div> 131 <div class="item item-chair-tp"> 132 <video poster="" id="chair-tp" autoplay controls muted loop height="100%"> 133 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/chair-tp.mp4" 134 type="video/mp4"> 135 </video> 136 </div> 137 <div class="item item-shiba"> 138 <video poster="" id="shiba" autoplay controls muted loop height="100%"> 139 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/shiba.mp4" 140 type="video/mp4"> 141 </video> 142 </div> 143 <div class="item item-fullbody"> 144 <video poster="" id="fullbody" autoplay controls muted loop height="100%"> 145 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/fullbody.mp4" 146 type="video/mp4"> 147 </video> 148 </div> 149 <div class="item item-blueshirt"> 150 <video poster="" id="blueshirt" autoplay controls muted loop height="100%"> 151 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/blueshirt.mp4" 152 type="video/mp4"> 153 </video> 154 </div> 155 <div class="item item-mask"> 156 <video poster="" id="mask" autoplay controls muted loop height="100%"> 157 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/mask.mp4" 158 type="video/mp4"> 159 </video> 160 </div> 161 <div class="item item-coffee"> 162 <video poster="" id="coffee" autoplay controls muted loop height="100%"> 163 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/coffee.mp4" 164 type="video/mp4"> 165 </video> 166 </div> 167 <div class="item item-toby"> 168 <video poster="" id="toby" autoplay controls muted loop height="100%"> 169 <source src="https://homes.cs.washington.edu/~kpar/nerfies/videos/toby2.mp4" 170 type="video/mp4"> 171 </video> 172 </div> 173 </div> 174 </div> 175 </div> 176</section> --> 177 178 179<section class="section"> 180 <div class="container is-max-desktop"> 181 <!-- Abstract. --> 182 <div class="columns is-centered has-text-centered"> 183 <div class="column is-four-fifths"> 184 <h2 class="title is-3">Abstract</h2> 185 <div class="content has-text-justified"> 186 <p> 187 Tracking objects of interest in a video is one of the most popular and widely applicable problems 188 in computer vision. However, with the years, a Cambrian explosion of use cases and benchmarks 189 has fragmented the problem in a multitude of different experimental setups. As a consequence, the 190 literature has fragmented too, and now the novel approaches proposed by the community are usually 191 specialised to fit only one specific setup. 192 </p> 193 194 <p> 195 To understand to what extent this specialisation is actually 196 necessary, in this work we present <b>UniTrack</b>, a solution to address five different tasks within the same 197 framework. UniTrack consists of a single and task-agnostic appearance model, which can be learned 198 in a supervised or self-supervised fashion, and multiple âheadsâ to address individual tasks and that do 199 not require training. We show how most tracking tasks can be solved within this framework, and that 200 the same appearance model can be used to obtain performance that is competitive against specialised 201 methods for all the five tasks considered. The framework also allows us to analyse appearance models 202 obtained with the most recent self-supervised methods, thus significantly extending their evaluation 203 and comparison to a larger variety of important problems. 204 </p> 205 </div> 206 </div> 207 </div> 208 <!--/ Abstract. --> 209 210 <!-- Paper video. --> 211 <!-- 212 <div class="columns is-centered has-text-centered"> 213 <div class="column is-four-fifths"> 214 <h2 class="title is-3">Video</h2>
215 <div class="publication-video"> 216 <iframe src="" 217 frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> 218 </div> 219 </div> 220 </div> 221 --> 222 <!--/ Paper video. --> 223 </div> 224</section> 225 226 227<section class="section"> 228 229 230 <div class="container is-max-desktop"> 231 232 233 <div class="columns is-centered"> 234 <div class="column is-full-width"> 235 <h2 class="title is-3">Proliferation of tracking setups</h2> 236 237 <div class="content has-text-justified"> 238 <p> 239 Various use cases and benchmarks has fragmented the tracking problem in multiple setups. 240 </p> 241 </div> 242 <div class="columns is-vcentered interpolation-panel"> 243 244 <img src="static/images/tasks.png" 245 class="interpolation-image" 246 alt="Interpolate start reference image."/> 247 </img> 248 249 </div> 250 <br/> 251 252 253 254 </div> 255 </div> 256 257 258 259 <div class="columns is-centered"> 260 261 262 <div class="column"> 263 <div class="content"> 264 <h2 class="title is-3">The UniTrack framework</h2> 265 <p> 266 To investigate what consititutes a good representation of objects in videos, we propose UniTrack, a unified tracking framework with a single shared appearance model. 267 UniTrack is divided into three "levels": 268 </p> 269 <ul> 270 <li> <b>Level-1</b>: The shared, base appearance model. We are particularly interested in <i>self-supervised</i> representations like MoCo and SimCLR.</li> 271 <li> <b>Level-2</b>: Two parameter-free algorithmic primitives: <i>propagation</i> and <i>assciation</i>, that operate on deep features extracted by the appearance model.</li> 272 <li> <b>Level-3</b>: Task-specific solutions. Used to adapt the two level-2 primitives to task-specific outputs. 273 </li> 274 </ul> 275 </div> 276 </div> 277 278 279 <div class="column"> 280 <h2 class="title is-3"></h2> 281 <div class="columns is-centered"> 282 <div class="column content"> 283 <img src="static/images/framework.png" 284 class="interpolation-image" 285 alt="Interpolate start reference image."/> 286 </img> 287 </div> 288 289 </div> 290 </div> 291 292 </div> 293</section> 294 295 296 297 298 299<section class="section"> 300 <div class="container is-max-desktop"> 301 <h2 class="title is-3">Results visulization</h2> 302 Below we show qualitative results of UniTrack on five different tracking tasks. The appearance model used is an <b> ImageNet pre-trained ResNet-18</b>. <b>No</b> task-specific training is performed. Note that we use this appearance model only for simplicity and consistency of comaprison. Many self-supervised representations are better on all task. 303 <div class="brmod"></div> 304 <div class="hero is-light is-small"> 305 306 <div class="hero-body"> 307 <div class="container"> 308 <a class="title is-5"> Single Object Tracking (SOT) @ OTB-2015</a> 309 <div id="results-carousel" class="carousel results-carousel"> 310 <div class="item item-steve"> 311 <img src="static/images/vis/sot1.gif" 312 class="interpolation-image" 313 alt="Interpolate start reference image."/> 314 </div> 315 <div class="item item-steve"> 316 <img src="static/images/vis/sot2.gif" 317 class="interpolation-image" 318 alt="Interpolate start reference image."/> 319 </div> 320 <div class="item item-steve"> 321 <img src="static/images/vis/sot3.gif" 322 class="interpolation-image" 323 alt="Interpolate start reference image."/> 324 </div> 325 <div class="item item-steve"> 326 <img src="static/images/vis/sot4.gif" 327 class="interpolation-image" 328 alt="Interpolate start reference image."/> 329 </div> 330 <div class="item item-steve"> 331 <img src="static/images/vis/sot5.gif" 332 class="interpolation-image" 333 alt="Interpolate start reference image."/> 334 </div> 335 <div class="item item-steve"> 336 <img src="static/images/vis/sot6.gif" 337 class="interpolation-image" 338 alt="Interpolate start reference image."/> 339 </div> 340 <div class="item item-steve"> 341 <img src="static/images/vis/sot7.gif" 342 class="interpolation-image" 343 alt="Interpolate start reference image."/> 344 </div> 345 <div class="item item-steve"> 346 <img src="static/images/vis/sot8.gif" 347 class="interpolation-image" 348 alt="Interpolate start reference image."/> 349 </div> 350 </div> 351 </div> 352 </div> 353 354 <div class="hero-body"> 355 <div class="container"> 356 <a class="title is-5"> Video Object Segmentation (VOS) @ DAVIS-2017</a> 357 <div id="results-carousel" class="carousel results-carousel"> 358 <div class="item item-steve"> 359 <img src="static/images/vis/vos1.gif" 360 class="interpolation-image" 361 alt="Interpolate start reference image."/> 362 </div> 363 <div class="item item-steve"> 364 <img src="static/images/vis/vos2.gif" 365 class="interpolation-image" 366 alt="Interpolate start reference image."/> 367 </div> 368 <div class="item item-steve"> 369 <img src="static/images/vis/vos3.gif" 370 class="interpolation-image" 371 alt="Interpolate start reference image."/> 372 </div> 373 <div class="item item-steve"> 374 <img src="static/images/vis/vos4.gif" 375 class="interpolation-image" 376 alt="Interpolate start reference image."/> 377 </div> 378 <div class="item item-steve"> 379 <img src="static/images/vis/vos5.gif" 380 class="interpolation-image" 381 alt="Interpolate start reference image."/> 382 </div> 383 <div class="item item-steve"> 384 <img src="static/images/vis/vos6.gif" 385 class="interpolation-image" 386 alt="Interpolate start reference image."/> 387 </div> 388 <div class="item item-steve"> 389 <img src="static/images/vis/vos7.gif" 390 class="interpolation-image" 391 alt="Interpolate start reference image."/> 392 </div> 393 <div class="item item-steve"> 394 <img src="static/images/vis/vos8.gif" 395 class="interpolation-image" 396 alt="Interpolate start reference image."/> 397 </div> 398 </div> 399 </div> 400 </div> 401 402 <div class="hero-body"> 403 <div class="container"> 404 <a class="title is-5"> Multi-Object Tracking (MOT) @ MOT-16</a> 405 <div id="results-carousel" class="carousel results-carousel"> 406 <div class="item item-steve"> 407 <img src="static/images/vis/mot1.gif" 408 class="interpolation-image" 409 alt="Interpolate start reference image."/> 410 </div> 411 <div class="item item-steve"> 412 <img src="static/images/vis/mot2.gif" 413 class="interpolation-image" 414 alt="Interpolate start reference image."/> 415 </div> 416 <div class="item item-steve"> 417 <img src="static/images/vis/mot3.gif" 418 class="interpolation-image" 419 alt="Interpolate start reference image."/> 420 </div> 421 <div class="item item-steve"> 422 <img src="static/images/vis/mot4.gif" 423 class="interpolation-image" 424 alt="Interpolate start reference image."/> 425 </div> 426 <div class="item item-steve"> 427 <img src="static/images/vis/mot5.gif" 428 class="interpolation-image" 429 alt="Interpolate start reference image."/> 430 </div> 431 <div class="item item-steve"> 432 <img src="static/images/vis/mot6.gif" 433 class="interpolation-image" 434 alt="Interpolate start reference image."/> 435 </div> 436 </div> 437 </div> 438 </div> 439 440 <div class="hero-body"> 441 <div class="container"> 442 <a class="title is-5"> Multi-Object Tracking and Segmentation (MOTS) @ MOTS</a> 443 <div id="results-carousel" class="carousel results-carousel"> 444 <div class="item item-steve"> 445 <img src="static/images/vis/mots1.gif" 446 class="interpolation-image" 447 alt="Interpolate start reference image."/> 448 </div> 449 <div class="item item-steve"> 450 <img src="static/images/vis/mots2.gif" 451 class="interpolation-image" 452 alt="Interpolate start reference image."/> 453 </div> 454 <div class="item item-steve"> 455 <img src="static/images/vis/mots3.gif" 456 class="interpolation-image" 457 alt="Interpolate start reference image."/> 458 </div> 459 <div class="item item-steve"> 460 <img src="static/images/vis/mots4.gif" 461 class="interpolation-image" 462 alt="Interpolate start reference image."/> 463 </div> 464 </div> 465 </div> 466 </div> 467 468 469 <div class="hero-body"> 470 <div class="container"> 471 <a class="title is-5"> Pose Tracking @ PoseTrack-2018 </a> 472 <div id="results-carousel" class="carousel results-carousel"> 473 <div class="item item-steve"> 474 <img src="static/images/vis/posetrack1.gif" 475 class="interpolation-image" 476 alt="Interpolate start reference image."/> 477 </div> 478 <div class="item item-steve"> 479 <img src="static/images/vis/posetrack2.gif" 480 class="interpolation-image" 481 alt="Interpolate start reference image."/> 482 </div> 483 <div class="item item-steve"> 484 <img src="static/images/vis/posetrack3.gif" 485 class="interpolation-image" 486 alt="Interpolate start reference image."/> 487 </div> 488 <div class="item item-steve"> 489 <img src="static/images/vis/posetrack4.gif" 490 class="interpolation-image" 491 alt="Interpolate start reference image."/> 492 </div> 493 <div class="item item-steve"> 494 <img src="static/images/vis/posetrack5.gif" 495 class="interpolation-image" 496 alt="Interpolate start reference image."/> 497 </div> 498 <div class="item item-steve"> 499 <img src="static/images/vis/posetrack6.gif" 500 class="interpolation-image" 501 alt="Interpolate start reference image."/> 502 </div> 503 </div> 504 </div> 505 </div> 506 507 508 </div> 509 </div> 510</section> 511 512 513 514 515<section class="section"> 516 517 518 <div class="container is-max-desktop"> 519 520 521 522 523 524 <div class="columns is-centered"> 525 526 527 <div class="column is-two-thirds"> 528 <div class="content"> 529 <h2 class="title is-3">An evaluation platform for SSL models</h2> 530 <p> 531 The "decoupled" characterazition of appearance models in UniTrack allows us to use this framework as an evaluation platform for benchmarking self
531-supervised learning (SSL) models on multiple tracking tasks. And the evaluation does not requrire any training or finetuning! 532 </p> 533 534 <p> 535 In the right we show an example: the SSL model <a href="https://github.com/facebookresearch/moco">MoCo-v1</a> is tested on five tracking tasks. Each vertex shows the ranking against the other models for a specific tracking task. The larger the area of a polygon is, the better that learned representation performs, across the board. 536 </p> 537 538 <p> 539 Resuls are shown at the <a href="#">top of the page</a>: the recent proposed <a href="https://jerryxu.net/VFS/">VFS</a> (bottom-right of the panel) dominates on all tasks except for SOT. We expect that newly-proposed representations will further improve performance across tasks. 540 541 </p> 542 </div> 543 </div> 544 545 546 <div class="column"> 547 <h2 class="title is-3"></h2> 548 <div class="columns is-centered"> 549 <div class="column content"> 550 <img src="static/images/radar_example.png" 551 class="interpolation-image" 552 alt="Interpolate start reference image."/> 553 </img> 554 </div> 555 556 </div> 557 </div> 558 559 </div> 560</section> 561 562 563<section class="section"> 564 <div class="container is-max-desktop"> 565 566 <h2 class="title is-3">Open-world applications</h2> 567 <div class="columns is-centered"> 568 569 <div class='column is-centered'> 570 <div class="content"> 571 <p> 572 In UniTrack, the appearance models are general-purpose representations. This means we can perform class-agnostic tracking in an open-world context. Below we show a demo by combining <a href="https://github.com/Megvii-BaseDetection/YOLOX"> YOLOX </a> with UniTrack. YOLOX is trained on the COCO dataset, so in this demo we can perform Multi-Object Tracking for 80 COCO classes. 573 </p> 574 <p> 575 The YOLOX detector can be replaced with other detectors if you want to track objects of other classes. 576 </p> 577 </div> 578 </div> 579 </div> 580 <!--/ Abstract. --> 581 582 <!-- Paper video. --> 583 <div class="columns is-centered has-text-centered"> 584 <div class="column"> 585 <div class="publication-video"> 586 <iframe src="static/images/unitrack_demo.mp4" 587 frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> 588 </div> 589 </div> 590 </div> 591 <!--/ Paper video. --> 592 </div> 593</section> 594 595 596 597 598 599<section class="section" id="BibTeX"> 600 <div class="container is-max-desktop content"> 601 <h2 class="title">BibTeX</h2> 602 <pre><code>@article{wang2021different, 603 author = {Wang, Zhongdao and Zhao, Hengshuang and Li, Ya-Li and Wang, Shengjin and Torr, Philip and Bertinetto, Luca}, 604 title = {Do different tracking tasks require different appearance models?}, 605 journal = {Thirty-Fifth Conference on Neural Infromation Processing Systems}, 606 year = {2021}, 607}</code></pre> 608 </div> 609</section> 610 611 612<footer class="footer"> 613 <div class="container"> 614 <div class="content has-text-centered"> 615 <a class="icon-link" 616 href="https://arxiv.org/pdf/2107.02156.pdf"> 617 <i class="fas fa-file-pdf"></i> 618 </a> 619 <a class="icon-link" href="https://github.com/Zhongdao/UniTrack" class="external-link" disabled> 620 <i class="fab fa-github"></i> 621 </a> 622 </div> 623 <div class="columns is-centered"> 624 <div class="column is-8"> 625 <div class="content has-text-centered"> 626 <p> 627 Website template borrowed from <a 628 href="https://github.com/nerfies/nerfies.github.io">Nerfies</a>. 629 </p> 630 </div> 631 </div> 632 </div> 633 </div> 634</footer> 635 636</body> 637</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.