1<!DOCTYPE html> 2<html> 3 4<head> 5 <meta charset="utf-8"> 6 <meta content="X-Fusion: Introducing New Modality to Frozen Large Language Models" name="description"> 7 <meta content="multimodal chatbot" name="keywords"> 8 <meta content="width=device-width, initial-scale=1" name="viewport"> 9 <meta content="./static/images/xfusion-teaser-2.gif" property="og:image" /> 10 <meta content="X-Fusion: Introducing New Modality to Frozen Large Language Models" property="og:title" /> 11 <meta content="X-Fusion: Introducing New Modality to Frozen Large Language Models" property="og:description" /> 12 <link href="jemdoc.css" rel="stylesheet" type="text/css" /> 13 <title>X-Fusion</title> 14 15 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> 16 <link href="https://cdn.jsdelivr.net/npm/[email protected]/css/bulma.min.css" rel="stylesheet"> 17 <link href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css" rel="stylesheet"> 18 <link href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css" rel="stylesheet"> 19 <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css" rel="stylesheet"> 20 <link href="./static/css/index.css" rel="stylesheet"> 21 <link href="./static/images/logo.png" rel="icon"> 22 <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"> 23 24 25
25<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
25 26
26<script defer src="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/js/all.min.js"></script>
26 27
27<script src="https://gradio.s3-us-west-2.amazonaws.com/4.16.0/gradio.js" type="module"></script>
27 28
28<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
vendor: 66 bytes, lines 28-29
28 29 <script async src="https://www.googletagmanager.com/gtag/js?id=
29G-PJGLWPJ36R
vendor: 14 bytes, lines 29-30
29"></script> 30
30<script> 31
vendor: 140 bytes, lines 31-35
31window.dataLayer = window.dataLayer || []; 32 function gtag(){dataLayer.push(arguments);} 33 gtag('js', new Date()); 34 35 gtag('config', '
35G-PJGLWPJ36R
vendor: 6 bytes, lines 35-36
35'); 36
36</script>
36 37</head> 38 39<style> 40 .expandable-card .card-text-container { 41 max-height: 200px; 42 overflow-y: hidden; 43 position: relative; 44 } 45 46 .expandable-card.expanded .card-text-container { 47 max-height: none; 48 } 49 50 .center { 51 margin-left: auto; 52 margin-right: auto; 53 } 54 55 .expand-btn { 56 position: relative; 57 display: none; 58 background-color: rgba(255, 255, 255, 0.8); 59 /* margin-top: -20px; */ 60 /* justify-content: center; */ 61 color: #510c75; 62 border-color: transparent; 63 } 64 65 .expand-btn:hover { 66 background-color: rgba(200, 200, 200, 0.8); 67 text-decoration: none; 68 border-color: transparent; 69 color: #510c75; 70 } 71 72 .expand-btn:focus { 73 outline: none; 74 text-decoration: none; 75 } 76 77 .expandable-card:not(.expanded) .card-text-container:after { 78 content: ""; 79 position: absolute; 80 bottom: 0; 81 left: 0; 82 width: 100%; 83 height: 90px; 84 background: linear-gradient(rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 1)); 85 } 86 87 .expandable-card:not(.expanded) .expand-btn { 88 margin-top: -40px; 89 } 90 91 .card-body { 92 padding-bottom: 5px; 93 } 94 95 .vertical-flex-layout { 96 justify-content: center; 97 align-items: center; 98 height: 100%; 99 display: flex; 100 flex-direction: column; 101 gap: 5px; 102 } 103 104 .figure-img { 105 max-width: 100%; 106 height: auto; 107 } 108 109 .adjustable-font-size { 110 font-size: calc(0.5rem + 2vw); 111 } 112 113 .chat-history { 114 flex-grow: 1; 115 overflow-y: auto; 116 /* overflow-x: hidden; */ 117 padding: 5px; 118 border-bottom: 1px solid #ccc; 119 margin-bottom: 10px; 120 } 121 122 #gradio pre { 123 background-color: transparent; 124 } 125</style> 126 127<body> 128 129 130<section class="hero"> 131 <div class="hero-body"> 132 <div class="container is-max-desktop"> 133 <div class="columns is-centered"> 134 <div class="column has-text-centered"> 135 <h1 class="title is-1 publication-title" style="color:#06402b;"> 136 <img src='./static/images/xfusion.gif' style="height: 60px"></img> 137 </h1> 138 <h3 class="title is-2 publication-title">Introducing New Modality to Frozen Large Language Models</h3> 139 <h5 class="subtitle is-4 publication-awards">âââ ICCV 2025 âââ</h5> 140 <div class="is-size-5 publication-authors"> 141 <span class="author-block"> 142<!-- <br>--> 143 <a href="https://sichengmo.github.io/" style="color:#f68946;font-weight:normal;">Sicheng Mo<sup>1</sup></a> 144 145 </span> 146 <span class="author-block"> 147 <!-- <br> --> 148 <a href="https://thaoshibe.github.io/" style="color:#f68946;font-weight:normal;">Thao Nguyen<sup>2</sup></a> 149 150 </span> 151 152 <span class="author-block"> 153 <a href="https://www.xunhuang.me/" style="color:#f68946;font-weight:normal;">Xun Huang<sup>3</sup></a> 154 155 </span> 156 157 <span class="author-block"> 158 <a href="https://scholar.google.com/citations?user=7uTobWwAAAAJ&hl=en" 159 style="color:#f68946;font-weight:normal;">Siddharth Srinivasan Iyer<sup>3</sup></a> 160 161 </span> 162 163 <span class="author-block"> 164 <a href="https://yijunmaverick.github.io/" 165 style="color:#f68946;font-weight:normal;">Yijun Li<sup>3</sup></a> 166 167 </span> 168 169 <span class="author-block"> 170 <a href="https://lychenyoko.github.io/" 171 style="color:#f68946;font-weight:normal;">Yuchen Liu<sup>3</sup></a> 172 173 </span> 174 175 <span class="author-block"> 176 <a href="https://tandon-a.github.io/" 177 style="color:#f68946;font-weight:normal;">Abhishek Tandon<sup>3</sup></a> 178 179 </span> 180
181 <span class="author-block"> 182 <a href="https://scholar.google.com/citations?user=B_FTboQAAAAJ&hl=en" 183 style="color:#f68946;font-weight:normal;">Eli Shechtman<sup>3</sup></a> 184 185 </span> 186 <span class="author-block"> 187 <a href="https://krsingh.cs.ucdavis.edu/" 188 style="color:#f68946;font-weight:normal;">Krishna Kumar Singh<sup>3</sup></a> 189 190 </span> 191 192 <span class="author-block"> 193 <a href="https://pages.cs.wisc.edu/~yongjaelee/" style="color:#f68946;font-weight:normal;">Yong Jae 194 Lee<sup>2</sup></a> 195 196 </span> 197 198 <span class="author-block"> 199 <a href="https://boleizhou.github.io/" style="color:#f68946;font-weight:normal;">Bolei Zhou<sup>1</sup></a> 200 201 </span> 202 203 <span class="author-block"> 204 <a href="https://yuheng-li.github.io/" 205 style="color:#F2A900;font-weight:normal;">Yuheng Li<sup>3</sup></a> 206 <!-- --> 207 </span> 208 </div> 209 <div class="is-size-5 publication-authors"> 210 <div> 211 <a href="https://www.cs.wisc.edu/" style="margin-right: 30px;"> 212 <img src="./static/images/ucla-logo.png" style="height: 40px"> 213 </a> 214 <a href="https://www.cs.wisc.edu/" style="margin-right: 15px;"> 215 <img src="./static/images/uwmadison-logo.png" style="height: 60px;"> 216 </a> 217 218 <a href="https://research.adobe.com/"> 219 <img src="./static/images/adobe-logo.png" style="height: 40px;"> 220 </a> 221 <br> 222 </div> 223 <span style="font-size: 90%;">1. University of California, Los Angeles</span> 224 225 <span style="font-size: 90%;">2. University of Wisconsin-Madison</span> 226 227 <span style="font-size: 90%;">3. Adobe Research</span> 228 <br> 229 <!-- <small><i>ð©: equal advising</i></small>--> 230 231 </div> 232 <!-- <div><a href="./static/images/yochameleon-without-bg.png"><img src='./static/images/yochameleon-without-bg.png' width="200"></a></div>--> 233 234 <div> 235<!-- <span style="font-size: 120%; color: darkred;">Best Paper @ CVPR 2025 T4V Workshop </span>--> 236<!-- <a href="https://sites.google.com/view/t4v-cvpr25/" target="_blank">--> 237<!-- <span style="font-size: 120%; color: darkred;"> ICCV 2025</span>--> 238<!-- </a>--> 239 <a href="https://sites.google.com/view/t4v-cvpr25/" target="_blank"> 240 <span style="font-size: 120%; color: darkred;"> Best Paper @ CVPR 2025 T4V Workshop</span> 241 </a> 242 </div> 243 244 <div class="column has-text-centered"> 245 <div class="publication-links"> 246 <span class="link-block"> 247 <a class="external-link button is-normal is-rounded is-dark" href="https://arxiv.org/abs/2504.20996" 248 target="_blank"> 249 <span class="icon"> 250 <i class="ai ai-arxiv"></i> 251 </span> 252 <span>Paper</span> 253 </a> 254 </span> 255 256<!-- <span class="link-block">--> 257<!-- <a class="external-link button is-normal is-rounded is-dark" href="https://arxiv.org/abs/2504.20996"--> 258<!-- target="_blank">--> 259<!-- <span class="icon">--> 260<!-- <i class="ai ai-arxiv"></i>--> 261<!-- </span>--> 262<!-- <span>Best Paper @ T4V Workshop CVPR 2025</span>--> 263<!-- </a>--> 264<!-- </span>--> 265 266<!-- <span class="link-block">--> 267<!-- <a class="external-link button is-normal is-rounded is-dark" href="Github LINK"--> 268<!-- target="_blank">-->
269<!-- <span class="icon">--> 270<!-- <i class="fab fa-github"></i>--> 271<!-- </span>--> 272<!-- <span>Code (Soon)</span>--> 273<!-- </a>--> 274<!-- </span>--> 275 <!-- <span class="link-block"> 276 <a href="https://huggingface.co/datasets/thaoshibe/YoChameleon" target="_blank" 277 class="external-link button is-normal is-rounded is-dark"> 278 <span class="icon"> 279 <i class="fas fa-database"></i> 280 </span> 281 <span>Dataset</span> 282 </a> 283 </span> --> 284 <!-- <span class="link-block"> 285 <a href="https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md" target="_blank" 286 class="external-link button is-normal is-rounded is-dark"> 287 <span class="icon"> 288 <i class="fas fa-share-square"></i> 289 </span> 290 <span>Model</span> 291 </a> 292 </span> --> 293 <!-- <span class="link-block">--> 294 <!-- <a href="POSTER" target="_blank"--> 295 <!-- class="external-link button is-normal is-rounded is-dark">--> 296 <!-- <span class="icon">--> 297 <!-- <i class="fas fa-share-square"></i>--> 298 <!-- </span>--> 299 <!-- <span>Poster (Updating)</span>--> 300 <!-- </a>--> 301 <!-- </span>--> 302 </div> 303 </div> 304 </div> 305 </div> 306 </div> 307 </div> 308</section> 309 310<section class="hero teaser"> 311 <div class="container is-max-desktop"> 312 <div class="hero-body"> 313 <!-- <h4 class="subtitle has-text-centered"> --> 314 <!-- <center><img src='./static/images/xfusion-teaser-2.gif' width="500"></center> --> 315 <center> 316 <video autoplay controls loop muted playsinline width="400" preload="auto"> 317 <source src="./static/images/xfusion-teaser.mp4" type="video/mp4" /> 318 Your browser does not support the video tag. 319 </video> 320 </center> 321 <br> 322 <center> 323 <text style="font-size:110%">We introduce X-Fusion - a novel framework that adapts pretrained LLMs (e.g., LLaMA) 324 to new modalities (e.g., vision) <br>while retaining their language capabilities and world knowledge! 325 </text> 326 </center> 327 <!-- </h4> --> 328 </div> 329 </div> 330</section> 331 332<section class="section" style="background-color:#f3f3f3"> 333 <div class="container is-max-desktop"> 334 <!-- Abstract. --> 335 <div class="columns is-centered has-text-centered"> 336 <div class="column is-six-fifths"> 337 <h2 class="title is-3">ð Abstract</h2> 338 <div class="content has-text-justified"> 339 <p> 340 341 We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks 342 while preserving their language capabilities. 343 X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLMâs parameters frozen 344 while integrating vision-specific information for both understanding and generation. 345 Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both 346 image-to-text and text-to-image tasks. 347 We find that incorporating understanding-focused data improves generation quality, reducing image data noise 348 enhances overall performance, and feature alignment accelerates convergence for smaller models but has 349 minimal impact on larger ones. 350 Our findings provide valuable insights into building efficient unified multimodal models. 351 352 </p> 353 </div> 354 </div> 355 </div> 356 </div> 357</section> 358 359</section> 360<section class="section"> 361 <div class="container is-max-desktop"> 362 <!-- Abstract. --> 363 <div class="columns is-centered has-text-centered"> 364 <div class="column is-six-fifths"> 365 <h2 class="title is-3">â¡ X-Fusion Capabilities</h2> 366 <!-- <center><img src="./static/images/xfusion-abilities.gif" width="100%"></center> --> 367 <center>
368 <video autoplay controls loop muted playsinline width="100%" preload="auto"> 369 <source src="./static/images/xfusion-abilities.mp4" type="video/mp4" /> 370 Your browser does not support the video tag. 371 </video> 372 </center> 373 374 </div> 375 </div> 376 </div> 377</section> 378<hr> 379 380<!-- # The Start of the Main Content # --> 381 382<section class="section"> 383 <div class="columns is-centered has-text-centered"> 384 <div class="column is-six-fifths"> 385 <h2 class="title is-3">ð X-Fusion </h2> 386 </div> 387 </div> 388 389 <div class="container is-max-desktop"> 390 <div class="columns is-centered"> 391 <div class="column is-full-width"> 392 <div class="column is-six-fifths"> 393 <h2 class="title is-4"> Model Architecture </h2> 394 <p> 395 We propose X-Fusion to address two key challenges: (i) retaining the language abilities of the pre-trained LLM while (ii) adapting it with image generation/understanding capabilities. 396 To achieve this, we introduce a <b>Dual-Tower</b> architecture that freezes all language weights and separate vision weights in each layer to help process visual information for the LLM. This approach aligns text and vision features not only at input or output level, but also at the intermediate processing level. 397 </p> 398 <p> 399 To show the effectiveness of our approach, we proposed alternative transformer block variants for multimodal integration, including 400 (a) Single Tower, (b) Gated Layer,and (c) Dual Projection. We find the Dual-Tower architecture achieves the best performance in both image generation and understanding tasks among them. 401 </p> 402 403 </div> 404 405 406 <div class="content has-text-justified"> 407 <center> 408 <td width="90%"> 409 <img alt="Training Pipeline" id="teaser" src="./static/images/xfusion_approach.jpg" width="98%"> 410 </td> 411 </center> 412 <div class="column is-six-fifths"> 413 Conceptual comparison of four model architecture baselines. Here, we illustrate how each layer processes the 414 sequential multi-modal feature. (a) Single Tower: Directly fine-tuning pre-trained LLM. 415 (b) Gated Layer: Duplicate Each LLM layer as the gated vision layer, 416 (c) Dual Projection: Duplicate QKV matric and MLP layer for vision modality, 417 (d) Dual Tower: Duplicated transformer block for vision modality. 418 </div> 419 </div> 420 <div class="content has-text-justified"> 421 <div class="column is-six-fifths"> 422 <h2 class="title is-4"> Training Recipe</h2> 423 Alongside of the model architecture, we also study the training recipe for X-Fusion. We focused on two key questions: 424 <ol> 425 <li> How does noise level affect performance in a joint training setting? </li> 426 <li> Does multitask training provide mutual benefits between tasks? </li> 427 </ol> 428 429 We first study the effect of noise level in the training data. 430 While diffusion-based image modeling benefits from noisy input for generation tasks, 431 excessive noise can degrade visual quality and hinder image understanding. 432 In contrast to conventional choose maximum of \( t = 50\% T \) to reduce distortion for visual understanding, 433 we find <b>clean images for visual understanding improve performance for both tasks</b>. 434 We show the performance comparison between varied the max noise level for image understanding (I2T) tasks: 435 0% (clean image), 25%, 50%, 75%, and 100% (normal generation setting) in the following figure. 436 </div> 437 </div> 438 439 <div> 440 441 <center><img src="./static/images/noise.png" width="100%"></img></center> 442 </div> 443 <br> 444 445 <div class="content has-text-justified"> 446 <div class="column is-six-fifths"> 447<!-- <h2 class="title is-4"> Training Recipe</h2>--> 448 We then study the effect of data ratio in the training data. To investigate the synergy 449 between visual understanding and generation from data's perspective, we begin with the training data composed 450 entirely of T2I tasks (100% T2I, 0% I2T, or denoted as 100/0 for short), then progressively decrease the proportion 451 of T2I data while increasing I2T data (i.e., 66/33, 50/50, 33/66) until the dataset consists solely of I2T tasks (0/100). 452 We find <b>the asymmetric relationship: understanding data benefits generation, but generation data does not enhance understanding. </b> 453 </div> 454 </div> 455 456 <div> 457 <center><img src="./static/images/data-ratio.png" width="100%"></img></center> 458 </div> 459 <br> 460 <center><b>â Please refer to the main paper for detailed ablation experiments! â</b></center> 461 </div> 462 </div> 463 </div> 464 <hr> 465</section> 466<section class="section"> 467 <div class="container is-max-desktop"> 468 <!-- Abstract. --> 469 <div class="columns is-centered has-text-centered"> 470 <div class="column is-six-fifths"> 471 <h2 class="title is-3">ð¼ï¸ Experiment Result</h2> 472 <div class="content has-text-justified"> 473 <div class="column is-six-fifths"> 474 <h2 class="title is-3"> Text-to-Image Generation</h2> 475 476 <center><img src="./static/images/xfusion_results_gen.jpg" width="90% px"></center> 477 </div> 478 </div> 479 480 481 482 <hr> 483 <br><br> 484 485 <div class="content has-text-justified"> 486 <div class="column is-six-fifths"> 487 <h2 class="title is-3"> Image Captioning</h2> 488 489 <center><img src="./static/images/xfusion_qual_2.jpg" width="90% px"></center> 490 </div> 491 </div> 492<!--–>--> 493 <hr> 494 <br><br> 495 <div class="content has-text-justified"> 496 <div class="column is-six-fifths"> 497 <h2 class="title is-3"> Fine-tuned X-Fusion on Downsteam Tasks</h2> 498 Can X-Fusion extend its capabilities to other downstream vision-and-language tasks? 499 We fine-tuned our model on four tasks simultaneouslyâincluding image editing, localization, 500 in/oupt-painting, and Visual Question Answering using internal datasets for 20k training steps. 501 In the following figure, we demonstrate that our unified X-Fusion model can handle multiple tasks without
502 creating task-specific weights. 503 </div> 504 </div> 505 506 <center><img src="./static/images/xfusion_four_tasks_web1.png" width="98% px"></center> 507 <br><br> 508 <hr> 509 <br><br> 510 511<!-- <br><br>--> 512 <div class="content has-text-justified"> 513 <div class="column is-six-fifths"> 514 <h2 class="title is-3"> Transfer from Pretrained Diffusion Model</h2> 515 516 One main advantage of our dual-tower design is that allows non-identical block designs in language and vision towers as each block in the both tower processes the entire feature sequence independently. 517 With this advantage, we could transfer the image generation capability from a large-scale pretrained diffusion model that uses diffusion transformers. 518 We trained a a variation of X-Fusion using Llama3 model as the language tower and an in-house pretrained text-to-image DiT model as its vision tower, notated as X-Fusion(Pretrained DiT), with 50K steps. 519 We show that X-Fusion(Pretrained DiT) obtained stronger image generation capability compared to the vanilla X-Fusion. 520 521 </div> 522 </div> 523 <center><img src="./static/images/xfusion-with-dit-web_v1.svg" width="96% px"></center> 524 <!-- Our X-Fusion model can also be used with pre-trained DiT model, which is a diffusion-based text-to-image generation model.--> 525 <!-- <hr>--> 526 <!-- <br><br>--> 527 <!-- <h2 class="title is-3">ð¼ï¸ Qualitative Result: Multi-Turn Generation</h2>--> 528 <!-- <center><img src="./static/images/xfusion_qual_4_conv_gen.jpg" width="90% px"></center>--> 529 530 </div> 531 </div> 532 </div> 533</section> 534<hr> 535 536 537<section class="section" id="BibTeX"> 538 <div class="container is-max-desktop content"> 539 <h2 class="title">ð¥ BibTeX</h2> 540 <pre><code> 541 @article{mo2025xfusion, 542 title={X-fusion: Introducing new modality to frozen large language models}, 543 author={Mo, Sicheng and Nguyen, Thao and Huang, Xun and Iyer, Siddharth Srinivasan and Li, Yijun and Liu, Yuchen and Tandon, Abhishek and Shechtman, Eli and Singh, Krishna Kumar and Lee, Yong Jae and others}, 544 journal={arXiv preprint arXiv:2504.20996}, 545 year={2025} 546 } 547 </code></pre> 548 </div> 549 550 <section class="section" id="Acknowledgement"> 551 <div class="container is-max-desktop content"> 552 <h2 class="title">ð Acknowledgement</h2> 553 <p> 554 This website is adapted from <a 555 href="https://github.com/nerfies/nerfies.github.io">Nerfies</a>, licensed under a <a href="http://creativecommons.org/licenses/by-sa/4.0/" 556 rel="license">Creative 557 Commons Attribution-ShareAlike 4.0 International License</a>. 558 </p> 559 <center> 560 <i>Thank you (.â á´ â.).</i> 561 <p> 562 </p> 563 </div> 564 </section> 565</body> 566 567</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.