1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <meta name="description" 6 content="ASAM2024"> 7 <meta name="keywords" content="ASAM2024"> 8 <meta name="viewport" content="width=device-width, initial-scale=1"> 9 <!-- <meta property="og:image" content="./static/img/logo.png"/> --> 10 <!-- <link rel="image_src" href="./static/img/logo.png"> --> 11 <!-- <link rel="icon" 12 type="image/x-icon" 13 href="./static/img/logo.png"/> --> 14 15 <title> 16 ASAM (CVPR2024): Boosting Segment Anything Model with Adversarial Tuning 17 </title> 18 19
19<script> 20
vendor: 134 bytes, lines 20-24
20window.dataLayer = window.dataLayer || []; 21 function gtag(){dataLayer.push(arguments);} 22 gtag('js', new Date()); 23 24 gtag('config', '
24G-EDF010G6PN
vendor: 8 bytes, lines 24-27
24'); 25 26 27
27</script>
27 28 29 30
30<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
30 31
31<script type="text/javascript" src="https://code.jquery.com/jquery-1.11.0.min.js"></script>
31 32
32<script type="text/javascript" src="https://code.jquery.com/jquery-migrate-1.2.1.min.js"></script>
32 33
33<script src="https://unpkg.com/interactjs/dist/interact.min.js"></script>
33 34 35 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 36 rel="stylesheet"> 37 38 <link rel="stylesheet" type="text/css" href="./static/slick/slick.css"/> 39 <link rel="stylesheet" type="text/css" href="./static/slick/slick-theme.css"/> 40 41 <link rel="stylesheet" href="./static/css/bulma.min.css"> 42 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 43 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 44 <link rel="stylesheet" 45 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 46 <link rel="stylesheet" href="./static/css/index.css"> 47 48
48<script defer src="./static/js/fontawesome.all.min.js"></script>
48 49
49<script src="./static/js/bulma-slider.min.js"></script>
49 50
50<script src="./static/js/index.js"></script>
50 51 52 <link rel="stylesheet" href="./static/css/gradient_color.css"> 53 54</head> 55<body> 56 57<nav class="navbar" role="navigation" aria-label="main navigation"> 58 <div class="navbar-brand"> 59 <a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false"> 60 <span aria-hidden="true"></span> 61 <span aria-hidden="true"></span> 62 <span aria-hidden="true"></span> 63 </a> 64 </div> 65 <div class="navbar-menu"> 66 <div class="navbar-start" style="flex-grow: 1; justify-content: center;"> 67 </div> 68 69 </div> 70</nav> 71 72<section class="hero"> 73 <div class="hero-body"> 74 <div class="container"> 75 <!-- <div class="columns is-centered"> 76 <div class="column is-8 has-text-centered"> 77 <img src="static/figs/diffusion.png" height="100%" alt="logo"/> 78 </div> 79 </div> --> 80 <div class="container has-text-centered"> 81 <h1 class="title is-1 publication-title"> 82 <h1 class="gradient-text">ASAM</h1> <font size="6">Boosting Segment Anything Model with Adversarial Tuning </font> 83 </h1> 84 <br> <font size="5">CVPR 2024</font> 85 <div class="is-size-5 publication-authors"> 86 <div class="author-block"> 87 <a href="https://libraboli.github.io/">Bo Li</a>, 88 </div> 89 <div class="author-block"> 90 <a href="https://scholar.google.com.hk/citations?user=bJmhMKEAAAAJ&hl=zh-CN">HaoKe Xiao</a>, 91 </div> 92 <div class="author-block"> 93 <a href="https://luckybird1994.github.io/">Lv Tang</a> 94 </div> 95 <div class="is-size-5 publication-authors"> 96 <span class="author-block">vivo Mobile Communication Co., Ltd</span> 97 </div> 98 <!-- <div class="is-size-6 publication-authors"> 99 <span class="author-block"><sup>*</sup>Most work done during internship at Adobe Research</span> 100 </div> --> 101 102 <div class="column has-text-centered"> 103 <div class="publication-links"> 104 <!-- PDF Link. --> 105 <!-- <span class="link-block"> 106 <a href="https://arxiv.org/abs/2405.00256" 107 class="external-link button is-normal is-rounded is-dark"> 108 <span class="icon"> 109 <i class="fas fa-file-pdf"></i> 110 </span> 111 <span>Paper</span> 112 </a> 113 </span> --> 114 <span class="link-block"> 115 <!-- <a href="SegGen_arxiv.pdf" --> 116 <a href="https://arxiv.org/abs/2405.00256" 117 class="external-link button is-normal is-rounded is-dark"> 118 <span class="icon"> 119 <i class="ai ai-arxiv"></i> 120 </span> 121 <span>arXiv</span> 122 </a> 123 </span> 124 <span class="link-block"> 125 <a href="https://github.com/luckybird1994/ASAM" 126 class="external-link button is-normal is-rounded is-dark"> 127 <span class="icon"> 128 <i class="fab fa-github"></i> 129 </span> 130 <span>Github</span> 131 </a> 132 </span> 133 <span class="link-block"> 134 <a href="https://huggingface.co/spaces/xhk/ASAM" 135 class="external-link button is-normal is-rounded is-dark"> 136 <span class="icon"> 137 <i class="fas fa-smile"></i> 138 </span> 139 <span>HuggingFace</span> 140 </a> 141 </span> 142 <!-- Dataset Link. --> 143 <!-- <span class="link-block"> 144 <a href="https://github.com/google/hypernerf/releases/tag/v0.1" 145 class="external-link button is-normal is-rounded is-dark"> 146 <span class="icon"> 147 <i class="far fa-images"></i> 148 </span> 149 <span>Data</span> 150 </a> 151 </span> --> 152 </div> 153 </div> 154 </div> 155 </div> 156 </div> 157</section> 158 159<section class="hero teaser"> 160 <div class="hero-body"> 161 <div class="container is-max-desktop"> 162 <!-- <video id="teaser" autoplay controls muted loop playsinline height="100%"> 163 <source src="./static/images/teaser.mp4" 164 type="video/mp4"> 165 </video> --> 166 <div class="columns is-centered"> 167 <div class="column is-8 has-text-centered"> 168 <img src="images/spider.png" height="100%" alt="logo"/> 169 </div> 170 </div> 171 <h2 class="subtitle has-text"> 172 <font color="#9e2e23"><b><i>ASAM</i></b></font> just boosts the performance of SAM without architectural modifications. ASAM is also resource friendly, since it only needs 8 A6000 GPUs without necessitating additional data (1% SA-1B data). </h2> 173 </div> 174 </div> 175</section> 176 177<section class="hero teaser"> 178 <div class="hero-body"> 179 <div class="container is-max-desktop"> 180 <!-- <video id="teaser" autoplay controls muted loop playsinline height="100%"> 181 <source src="./static/images/teaser.mp4" 182 type="video/mp4"> 183 </video> --> 184 <div class="columns is-centered"> 185 <div class="column is-8 has-text-centered"> 186 <img src="images/asam_gpu.png" height="100%" alt="logo"/> 187 </div> 188 </div> 189 <h2 class="subtitle has-text">
190 Compared to SAM (<a href="https://arxiv.org/abs/2304.02643">Segment Anything, SAM, ICCV2023 </a>) and EfficientSAM (<a href="https://arxiv.org/abs/2312.00863">EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything, ESAM, CVPR2024 </a>), our proposed ASAM and EASAM is more resource-friendly. </h2> 191 </div> 192 </div> 193</section> 194 195 196<section class="section"> 197 <div class="container is-max-desktop"> 198 199 <!-- Abstract. --> 200 <div class="columns is-centered has-text-centered"> 201 <div class="column is-full"> 202 <h2 class="title is-3">Abstract</h2> 203 <div class="content has-text-justified" style="font-size: 18px;"> 204 <p> 205 In the evolving landscape of computer vision, foundation models have emerged as pivotal tools, exhibiting exceptional adaptability to a myriad of tasks. Among these, the Segment Anything Model (SAM) by Meta AI has distinguished itself in image segmentation. However, SAM, like its counterparts, encounters limitations in specific niche applications, prompting a quest for enhancement strategies that do not compromise its inherent capabilities. This paper introduces ASAM, a novel methodology that amplifies SAM's performance through adversarial tuning. We harness the potential of natural adversarial examples, inspired by their successful implementation in natural language processing. By utilizing a stable diffusion model, we augment a subset (1\%) of the SA-1B dataset, generating adversarial instances that are more representative of natural variations rather than conventional imperceptible perturbations. Our approach maintains the photorealism of adversarial examples and ensures alignment with original mask annotations, thereby preserving the integrity of the segmentation task. The fine-tuned ASAM demonstrates significant improvements across a diverse range of segmentation tasks without necessitating additional data or architectural modifications. The results of our extensive evaluations confirm that ASAM establishes new benchmarks in segmentation tasks, thereby contributing to the advancement of foundational models in computer vision. 206 </p> 207 </div> 208 </div> 209 </div> 210 <!--/ Abstract. --> 211 212 <!-- Paper video. --> 213 <!-- <div class="columns is-centered has-text-centered"> 214 <div class="column is-four-fifths"> 215 <h2 class="title is-2">Video</h2> 216 <div class="publication-video"> 217 <iframe width="640" height="480" src="https://www.youtube.com/embed/qzgdE_ghkaI" 218 title="YouTube video player" frameborder="0" 219 allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture" 220 allowfullscreen></iframe> 221 </div> 222 </div> 223 </div> --> 224 <!--/ Paper video. --> 225 </div> 226</section> 227 228 <div class="columns is-centered has-text-centered"> 229 <h2 class="title is-3 centered">Framework</h2> 230 </div> 231 232<section class="hero teaser"> 233 <div class="hero-body"> 234 <div class="container is-max-desktop"> 235 <img src="images/framework.png" height="100%" ></img> 236 <h2 class="subtitle has-text"> 237 <b>ASAM mainly contains three step.</b> The first step is <b>Adversarial Latent Optimization.</b> The second step is <b>Controllable Adversarial Samples Generation.</b> The third step is <b>Fine-tuning SAM</b> with adversarial samples. 238 </h2> 239 </div> 240 </div> 241</section> 242 243<div class="columns is-centered has-text-centered"> 244 <h2 class="title is-3 centered">Experiments: Stronger SAM</h2> 245</div> 246 247<section class="hero teaser"> 248 <div class="hero-body"> 249 <div class="container is-max-desktop"> 250 <!-- <video id="teaser" autoplay controls muted loop playsinline height="100%"> 251 <source src="./static/images/teaser.mp4" 252 type="video/mp4"> 253 </video> --> 254 <img src="images/ASAM_performance.png" height="100%" ></img> 255 <!-- <h2 class="subtitle has-text-centered"> --> 256 <h2 class="subtitle has-text"> 257 <b><i>Stronger SAM.</i></b> Compared with PGD-Tuning SAM, DAT-Tuning SAM, DatasetDM-Tuning SAM. ASAM clearly outperforms other tuning methods and achieves performance improvements compared with original SAM across all 14 test datasets. 258 </h2> 259 </div> 260 </div> 261</section> 262 263<section class="hero is-dark is-small"> 264 <div class="hero-body"> 265 <div class="container has-text-centered"> 266 267 <div id="results-carousel" class="carousel results-carousel"> 268 269 <div> 270 <div class="results-item"> 271 <image src="images/vis.png" style="padding-right: 200px; padding-left: 200px;"></image>
272 <figcaption>Qualitative comparison of the proposed ASAM and other methods. Yellow boxes represent the box prompts. 273 </figcaption> 274 </div> 275 </div> 276 277 <div> 278 <div class="results-item"> 279 <image src="images/hugeface.png" style="padding-right: 200px; padding-left: 200px;"></image> 280 <figcaption>The demo we provided in HugeFace. 281 </figcaption> 282 </div> 283 </div> 284 285 <div> 286 <div class="results-item"> 287 <image src="images/medical1.gif" style="padding-right: 200px; padding-left: 200px;"></image> 288 <figcaption>The demo we provided in HugeFace. 289 </figcaption> 290 </div> 291 </div> 292 293 <div> 294 <div class="results-item"> 295 <image src="images/medical2.gif" style="padding-right: 200px; padding-left: 200px;"></image> 296 <figcaption>The demo we provided in HugeFace. 297 </figcaption> 298 </div> 299 </div> 300 301 </div> 302 303 <div class="columns is-centered has-text-centered"> 304 <div class="column is-three-quarters is-size-5"> 305 <p> 306 From the qualitative comparison in different scenes, such as the common scene, the medical scene, our proposed ASAM can improve the performance of SAM. 307 </p> 308 </div> 309 </div> 310 </div> 311 </div> 312</section> 313 314 315<hr/> 316 317<div class="columns is-centered has-text-centered"> 318 <h2 class="title is-3 centered">Experiments: Stronger EfficientSAM</h2> 319</div> 320 321<section class="hero teaser"> 322 <div class="hero-body"> 323 <div class="container is-max-desktop"> 324 <!-- <video id="teaser" autoplay controls muted loop playsinline height="100%"> 325 <source src="./static/images/teaser.mp4" 326 type="video/mp4"> 327 </video> --> 328 <img src="images/aesam.png" height="100%" style="padding-right: 200px; padding-left: 200px;"></img> 329 <!-- <h2 class="subtitle has-text-centered"> --> 330 <h2 class="subtitle has-text"> 331 <b><i>Stronger EfficientSAM.</i></b> AESAM achieves performance improvements compared with EfficientSAM (<a href="https://arxiv.org/abs/2312.00863">EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything, ESAM, CVPR2024 </a>) across 16 different datasets. ESAM is the latest work proposed by Meta, which is accepted with full marks at CVPR2024. 332 </h2> 333 </div> 334 </div> 335</section> 336 337<section class="hero teaser"> 338 <div class="hero-body"> 339 <div class="container is-max-desktop"> 340 <div class="columns is-centered has-text-centered"> 341 <h2 class="title is-3 centered">Experiments: Stronger HQSAM</h2> 342 </div> 343 <img src="images/HQSAM.png"/> 344 <h2 class="subtitle has-text"> 345 <b><i>Stronger HQSAM.</i></b> HQ-ASAM can achieve performance improvements compared with HQSAM (<a href="https://github.com/SysCV/sam-hq">Segment Anything in High Quality, HQSAM, NeurIPS2023</a> ) across 4 different datasets. HQSAM is the work proposed by ETH Zurich&HKUST, which achieves about 3.4k Github stars. 346 </h2> 347 </div> 348 </div> 349</section> 350 351<section class="hero teaser"> 352 <div class="hero-body"> 353 <div class="container is-max-desktop"> 354 <div class="columns is-centered has-text-centered"> 355 <h2 class="title is-3 centered">Experiments: Stronger SAM-Adapter</h2> 356 </div> 357 <img src="images/SAM-Adapter.png"/> 358 <h2 class="subtitle has-text"> 359 <b><i>Stronger SAM-Adapter.</i></b> ASAM-Adapter achieves performance improvements compared with SAM-adapter (ICCV2023 Workshop) across 2 different datasets. 360 </h2> 361 </div> 362 </div> 363</section> 364 365 366<section class="section" id="BibTeX"> 367 <div class="container content is-max-desktop"> 368 <h2 class="title">BibTeX</h2> 369 <pre><code>@article{li2024asam, 370 title={ASAM: Boosting Segment Anything Model with Adversarial Tuning}, 371 author={Li, Bo and Xiao, Haoke and Tang, Lv}, 372 journal={arXiv preprint arXiv:2405.00256}, 373 year={2024} 374} 375}</code></pre> 376 </div> 377</section> 378 379 380<section class="section" id="acknowledgements"> 381 <div class="container content is-max-desktop"> 382 The website template was adapted from 383 <a href="https://seggenerator.github.io/">SegGen</a>.
384 </div> 385</section> 386 387 388
389<script type="text/javascript" src="./static/slick/slick.min.js"></script>
389 390</body> 391</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.