PageSourceSearch

https://sichengmo.github.io/XFusion/

html sichengmo.github.io collected 2026-10-03 09:56:59 UTC 24,576 bytes, 567 lines download raw bytes

1<!DOCTYPE html>
2<html>
3
4<head>
5  <meta charset="utf-8">
6  <meta content="X-Fusion: Introducing New Modality to Frozen Large Language Models" name="description">
7  <meta content="multimodal chatbot" name="keywords">
8  <meta content="width=device-width, initial-scale=1" name="viewport">
9  <meta content="./static/images/xfusion-teaser-2.gif" property="og:image" />
10  <meta content="X-Fusion: Introducing New Modality to Frozen Large Language Models" property="og:title" />
11  <meta content="X-Fusion: Introducing New Modality to Frozen Large Language Models" property="og:description" />
12  <link href="jemdoc.css" rel="stylesheet" type="text/css" />
13  <title>X-Fusion</title>
14
15  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
16  <link href="https://cdn.jsdelivr.net/npm/[email protected]/css/bulma.min.css" rel="stylesheet">
17  <link href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css" rel="stylesheet">
18  <link href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css" rel="stylesheet">
19  <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css" rel="stylesheet">
20  <link href="./static/css/index.css" rel="stylesheet">
21  <link href="./static/images/logo.png" rel="icon">
22  <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
23
24
25  
25<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
25
26  
26<script defer src="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/js/all.min.js"></script>
26
27  
27<script src="https://gradio.s3-us-west-2.amazonaws.com/4.16.0/gradio.js" type="module"></script>
27
28  
28<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
vendor: 66 bytes, lines 28-29
28
29  <script async src="https://www.googletagmanager.com/gtag/js?id=
29G-PJGLWPJ36R
vendor: 14 bytes, lines 29-30
29"></script>
30  
30<script>
31    
vendor: 140 bytes, lines 31-35
31window.dataLayer = window.dataLayer || [];
32    function gtag(){dataLayer.push(arguments);}
33    gtag('js', new Date());
34
35    gtag('config', '
35G-PJGLWPJ36R
vendor: 6 bytes, lines 35-36
35');
36  
36</script>
36
37</head>
38
39<style>
40    .expandable-card .card-text-container {
41        max-height: 200px;
42        overflow-y: hidden;
43        position: relative;
44    }
45
46    .expandable-card.expanded .card-text-container {
47        max-height: none;
48    }
49
50    .center {
51        margin-left: auto;
52        margin-right: auto;
53    }
54
55    .expand-btn {
56        position: relative;
57        display: none;
58        background-color: rgba(255, 255, 255, 0.8);
59        /* margin-top: -20px; */
60        /* justify-content: center; */
61        color: #510c75;
62        border-color: transparent;
63    }
64
65    .expand-btn:hover {
66        background-color: rgba(200, 200, 200, 0.8);
67        text-decoration: none;
68        border-color: transparent;
69        color: #510c75;
70    }
71
72    .expand-btn:focus {
73        outline: none;
74        text-decoration: none;
75    }
76
77    .expandable-card:not(.expanded) .card-text-container:after {
78        content: "";
79        position: absolute;
80        bottom: 0;
81        left: 0;
82        width: 100%;
83        height: 90px;
84        background: linear-gradient(rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 1));
85    }
86
87    .expandable-card:not(.expanded) .expand-btn {
88        margin-top: -40px;
89    }
90
91    .card-body {
92        padding-bottom: 5px;
93    }
94
95    .vertical-flex-layout {
96        justify-content: center;
97        align-items: center;
98        height: 100%;
99        display: flex;
100        flex-direction: column;
101        gap: 5px;
102    }
103
104    .figure-img {
105        max-width: 100%;
106        height: auto;
107    }
108
109    .adjustable-font-size {
110        font-size: calc(0.5rem + 2vw);
111    }
112
113    .chat-history {
114        flex-grow: 1;
115        overflow-y: auto;
116        /* overflow-x: hidden; */
117        padding: 5px;
118        border-bottom: 1px solid #ccc;
119        margin-bottom: 10px;
120    }
121
122    #gradio pre {
123        background-color: transparent;
124    }
125</style>
126
127<body>
128
129
130<section class="hero">
131  <div class="hero-body">
132    <div class="container is-max-desktop">
133      <div class="columns is-centered">
134        <div class="column has-text-centered">
135          <h1 class="title is-1 publication-title" style="color:#06402b;">
136            <img src='./static/images/xfusion.gif' style="height: 60px"></img>
137          </h1>
138          <h3 class="title is-2 publication-title">Introducing New Modality to Frozen Large Language Models</h3>
139        <h5 class="subtitle is-4 publication-awards">——— ICCV 2025 ———</h5>
140          <div class="is-size-5 publication-authors">
141              <span class="author-block">
142<!--                <br>-->
143                <a href="https://sichengmo.github.io/" style="color:#f68946;font-weight:normal;">Sicheng Mo<sup>1</sup></a>
144                &nbsp;&nbsp;&nbsp;
145              </span>
146            <span class="author-block">
147                <!-- <br> -->
148                <a href="https://thaoshibe.github.io/" style="color:#f68946;font-weight:normal;">Thao Nguyen<sup>2</sup></a>
149                &nbsp;&nbsp;&nbsp;
150              </span>
151
152            <span class="author-block">
153                <a href="https://www.xunhuang.me/" style="color:#f68946;font-weight:normal;">Xun Huang<sup>3</sup></a>
154                &nbsp;&nbsp;&nbsp;
155              </span>
156
157            <span class="author-block">
158                <a href="https://scholar.google.com/citations?user=7uTobWwAAAAJ&hl=en"
159                   style="color:#f68946;font-weight:normal;">Siddharth Srinivasan Iyer<sup>3</sup></a>
160                &nbsp;&nbsp;&nbsp;
161              </span>
162
163            <span class="author-block">
164                <a href="https://yijunmaverick.github.io/"
165                   style="color:#f68946;font-weight:normal;">Yijun Li<sup>3</sup></a>
166                &nbsp;&nbsp;&nbsp;
167              </span>
168
169            <span class="author-block">
170                <a href="https://lychenyoko.github.io/"
171                   style="color:#f68946;font-weight:normal;">Yuchen Liu<sup>3</sup></a>
172                &nbsp;&nbsp;&nbsp;
173              </span>
174
175            <span class="author-block">
176                <a href="https://tandon-a.github.io/"
177                   style="color:#f68946;font-weight:normal;">Abhishek Tandon<sup>3</sup></a>
178                &nbsp;&nbsp;&nbsp;
179              </span>
180
181            <span class="author-block">
182                <a href="https://scholar.google.com/citations?user=B_FTboQAAAAJ&hl=en"
183                   style="color:#f68946;font-weight:normal;">Eli Shechtman<sup>3</sup></a>
184                &nbsp;&nbsp;&nbsp;
185              </span>
186                        <span class="author-block">
187                <a href="https://krsingh.cs.ucdavis.edu/"
188                   style="color:#f68946;font-weight:normal;">Krishna Kumar Singh<sup>3</sup></a>
189                &nbsp;&nbsp;&nbsp;
190              </span>
191
192            <span class="author-block">
193                <a href="https://pages.cs.wisc.edu/~yongjaelee/" style="color:#f68946;font-weight:normal;">Yong Jae
194                  Lee<sup>2</sup></a>
195                  &nbsp;&nbsp;&nbsp;
196              </span>
197
198            <span class="author-block">
199                <a href="https://boleizhou.github.io/" style="color:#f68946;font-weight:normal;">Bolei Zhou<sup>1</sup></a>
200                &nbsp;&nbsp;&nbsp;
201              </span>
202
203            <span class="author-block">
204                <a href="https://yuheng-li.github.io/"
205                   style="color:#F2A900;font-weight:normal;">Yuheng Li<sup>3</sup></a>
206              <!-- &nbsp;&nbsp;&nbsp; -->
207              </span>
208          </div>
209          <div class="is-size-5 publication-authors">
210            <div>
211              <a href="https://www.cs.wisc.edu/" style="margin-right: 30px;">
212                <img src="./static/images/ucla-logo.png" style="height: 40px">
213              </a>
214              <a href="https://www.cs.wisc.edu/" style="margin-right: 15px;">
215                <img src="./static/images/uwmadison-logo.png" style="height: 60px;">
216              </a>
217              &nbsp;&nbsp;&nbsp;
218              <a href="https://research.adobe.com/">
219                <img src="./static/images/adobe-logo.png" style="height: 40px;">
220              </a>
221              <br>
222            </div>
223            <span style="font-size: 90%;">1. University of California, Los Angeles</span>
224            &nbsp;&nbsp;&nbsp;
225            <span style="font-size: 90%;">2. University of Wisconsin-Madison</span>
226            &nbsp;&nbsp;&nbsp;
227            <span style="font-size: 90%;">3. Adobe Research</span>
228            <br>
229            <!--              <small><i>🚩: equal advising</i></small>-->
230
231          </div>
232          <!--            <div><a href="./static/images/yochameleon-without-bg.png"><img src='./static/images/yochameleon-without-bg.png' width="200"></a></div>-->
233
234          <div>
235<!--            <span style="font-size: 120%; color: darkred;">Best Paper @ CVPR 2025 T4V Workshop </span>-->
236<!--          <a href="https://sites.google.com/view/t4v-cvpr25/" target="_blank">-->
237<!--            <span style="font-size: 120%; color: darkred;">&nbsp;&nbsp;&nbsp;ICCV 2025</span>-->
238<!--          </a>-->
239          <a href="https://sites.google.com/view/t4v-cvpr25/" target="_blank">
240            <span style="font-size: 120%; color: darkred;">&nbsp;&nbsp;&nbsp;Best Paper @ CVPR 2025 T4V Workshop</span>
241          </a>
242          </div>
243
244          <div class="column has-text-centered">
245            <div class="publication-links">
246                <span class="link-block">
247                  <a class="external-link button is-normal is-rounded is-dark" href="https://arxiv.org/abs/2504.20996"
248                     target="_blank">
249                    <span class="icon">
250                      <i class="ai ai-arxiv"></i>
251                    </span>
252                    <span>Paper</span>
253                  </a>
254                </span>
255
256<!--                <span class="link-block">-->
257<!--                  <a class="external-link button is-normal is-rounded is-dark" href="https://arxiv.org/abs/2504.20996"-->
258<!--                     target="_blank">-->
259<!--                    <span class="icon">-->
260<!--                      <i class="ai ai-arxiv"></i>-->
261<!--                    </span>-->
262<!--                    <span>Best Paper @ T4V Workshop CVPR 2025</span>-->
263<!--                  </a>-->
264<!--                </span>-->
265
266<!--              <span class="link-block">-->
267<!--                  <a class="external-link button is-normal is-rounded is-dark" href="Github LINK"-->
268<!--                     target="_blank">-->
269<!--                    <span class="icon">-->
270<!--                      <i class="fab fa-github"></i>-->
271<!--                    </span>-->
272<!--                    <span>Code (Soon)</span>-->
273<!--                  </a>-->
274<!--                </span>-->
275              <!--                 <span class="link-block">
276                                <a href="https://huggingface.co/datasets/thaoshibe/YoChameleon" target="_blank"
277                                  class="external-link button is-normal is-rounded is-dark">
278                                  <span class="icon">
279                                    <i class="fas fa-database"></i>
280                                  </span>
281                                  <span>Dataset</span>
282                                </a>
283                              </span> -->
284              <!--                 <span class="link-block">
285                                <a href="https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md" target="_blank"
286                                  class="external-link button is-normal is-rounded is-dark">
287                                  <span class="icon">
288                                    <i class="fas fa-share-square"></i>
289                                  </span>
290                                  <span>Model</span>
291                                </a>
292                              </span> -->
293              <!--                <span class="link-block">-->
294              <!--                  <a href="POSTER" target="_blank"-->
295              <!--                    class="external-link button is-normal is-rounded is-dark">-->
296              <!--                    <span class="icon">-->
297              <!--                      <i class="fas fa-share-square"></i>-->
298              <!--                    </span>-->
299              <!--                    <span>Poster (Updating)</span>-->
300              <!--                  </a>-->
301              <!--                </span>-->
302            </div>
303          </div>
304        </div>
305      </div>
306    </div>
307  </div>
308</section>
309
310<section class="hero teaser">
311  <div class="container is-max-desktop">
312    <div class="hero-body">
313      <!-- <h4 class="subtitle has-text-centered"> -->
314      <!-- <center><img src='./static/images/xfusion-teaser-2.gif' width="500"></center> -->
315      <center>
316        <video autoplay controls loop muted playsinline width="400" preload="auto">
317          <source src="./static/images/xfusion-teaser.mp4" type="video/mp4" />
318          Your browser does not support the video tag.
319        </video>
320      </center>
321      <br>
322      <center>
323        <text style="font-size:110%">We introduce X-Fusion - a novel framework that adapts pretrained LLMs (e.g., LLaMA)
324          to new modalities (e.g., vision) <br>while retaining their language capabilities and world knowledge!
325        </text>
326      </center>
327      <!-- </h4> -->
328    </div>
329  </div>
330</section>
331
332<section class="section" style="background-color:#f3f3f3">
333  <div class="container is-max-desktop">
334    <!-- Abstract. -->
335    <div class="columns is-centered has-text-centered">
336      <div class="column is-six-fifths">
337        <h2 class="title is-3">📜 Abstract</h2>
338        <div class="content has-text-justified">
339          <p>
340
341            We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks
342            while preserving their language capabilities.
343            X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM’s parameters frozen
344            while integrating vision-specific information for both understanding and generation.
345            Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both
346            image-to-text and text-to-image tasks.
347            We find that incorporating understanding-focused data improves generation quality, reducing image data noise
348            enhances overall performance, and feature alignment accelerates convergence for smaller models but has
349            minimal impact on larger ones.
350            Our findings provide valuable insights into building efficient unified multimodal models.
351
352          </p>
353        </div>
354      </div>
355    </div>
356  </div>
357</section>
358
359</section>
360<section class="section">
361  <div class="container is-max-desktop">
362    <!-- Abstract. -->
363    <div class="columns is-centered has-text-centered">
364      <div class="column is-six-fifths">
365        <h2 class="title is-3">⚡ X-Fusion Capabilities</h2>
366        <!-- <center><img src="./static/images/xfusion-abilities.gif" width="100%"></center> -->
367        <center>
368          <video autoplay controls loop muted playsinline width="100%" preload="auto">
369            <source src="./static/images/xfusion-abilities.mp4" type="video/mp4" />
370            Your browser does not support the video tag.
371          </video>
372        </center>
373
374      </div>
375    </div>
376  </div>
377</section>
378<hr>
379
380<!--  # The Start of the Main Content # -->
381
382<section class="section">
383  <div class="columns is-centered has-text-centered">
384    <div class="column is-six-fifths">
385      <h2 class="title is-3">📐 X-Fusion </h2>
386    </div>
387  </div>
388
389  <div class="container is-max-desktop">
390    <div class="columns is-centered">
391      <div class="column is-full-width">
392        <div class="column is-six-fifths">
393          <h2 class="title is-4"> Model Architecture </h2>
394          <p>
395            We propose X-Fusion to address two key challenges:  (i) retaining the language abilities of the pre-trained LLM while (ii) adapting it with image generation/understanding capabilities.
396            To achieve this, we introduce a <b>Dual-Tower</b> architecture that freezes all language weights and separate vision weights in each layer to help process visual information for the LLM. This approach aligns text and vision features not only at input or output level, but also at the intermediate processing level.
397          </p>
398        <p>
399        To show the effectiveness of our approach, we proposed alternative transformer block variants for multimodal integration, including
400        (a) Single Tower, (b) Gated Layer,and (c) Dual Projection. We find the Dual-Tower architecture achieves the best performance in both image generation and understanding tasks among them.
401        </p>
402
403        </div>
404
405
406        <div class="content has-text-justified">
407          <center>
408            <td width="90%">
409              <img alt="Training Pipeline" id="teaser" src="./static/images/xfusion_approach.jpg" width="98%">
410            </td>
411          </center>
412          <div class="column is-six-fifths">
413          Conceptual comparison of four model architecture baselines. Here, we illustrate how each layer processes the
414          sequential multi-modal feature. (a) Single Tower: Directly fine-tuning pre-trained LLM.
415          (b) Gated Layer: Duplicate Each LLM layer as the gated vision layer,
416          (c) Dual Projection: Duplicate QKV matric and MLP layer for vision modality,
417          (d) Dual Tower: Duplicated transformer block for vision modality.
418        </div>
419        </div>
420      <div class="content has-text-justified">
421          <div class="column is-six-fifths">
422        <h2 class="title is-4"> Training Recipe</h2>
423        Alongside of the model architecture, we also study the training recipe for X-Fusion. We focused on two key questions:
424            <ol>
425              <li> How does noise level affect performance in a joint training setting? </li>
426              <li> Does multitask training provide mutual benefits between tasks? </li>
427            </ol>
428
429        We first study the effect of noise level in the training data.
430        While diffusion-based image modeling benefits from noisy input for generation tasks,
431        excessive noise can degrade visual quality and hinder image understanding.
432        In contrast to conventional choose maximum of \( t = 50\% T \) to reduce distortion for visual understanding,
433           we find  <b>clean images for visual understanding improve performance for both tasks</b>.
434        We show the performance comparison between varied the max noise level for image understanding (I2T) tasks:
435        0% (clean image), 25%, 50%, 75%, and 100% (normal generation setting) in the following figure.
436        </div>
437      </div>
438
439        <div>
440
441          <center><img src="./static/images/noise.png" width="100%"></img></center>
442        </div>
443        <br>
444
445              <div class="content has-text-justified">
446          <div class="column is-six-fifths">
447<!--        <h2 class="title is-4"> Training Recipe</h2>-->
448        We then study the effect of data ratio in the training data. To investigate the synergy
449        between visual understanding and generation from data's perspective, we begin with the training data composed
450        entirely of T2I tasks (100% T2I, 0% I2T, or denoted as 100/0 for short), then progressively decrease the proportion
451        of T2I data while increasing I2T data (i.e., 66/33, 50/50, 33/66) until the dataset consists solely of I2T tasks (0/100).
452        We find <b>the asymmetric relationship: understanding data benefits generation, but generation data does not enhance understanding. </b>
453        </div>
454      </div>
455
456        <div>
457          <center><img src="./static/images/data-ratio.png" width="100%"></img></center>
458        </div>
459        <br>
460        <center><b>❗ Please refer to the main paper for detailed ablation experiments! ❗</b></center>
461      </div>
462    </div>
463  </div>
464  <hr>
465</section>
466<section class="section">
467  <div class="container is-max-desktop">
468    <!-- Abstract. -->
469    <div class="columns is-centered has-text-centered">
470      <div class="column is-six-fifths">
471        <h2 class="title is-3">🖼️ Experiment Result</h2>
472              <div class="content has-text-justified">
473        <div class="column is-six-fifths">
474      <h2 class="title is-3">         Text-to-Image Generation</h2>
475
476                  <center><img src="./static/images/xfusion_results_gen.jpg" width="90% px"></center>
477      </div>
478                </div>
479
480
481
482        <hr>
483        <br><br>
484
485              <div class="content has-text-justified">
486        <div class="column is-six-fifths">
487      <h2 class="title is-3"> Image Captioning</h2>
488
489                  <center><img src="./static/images/xfusion_qual_2.jpg" width="90% px"></center>
490      </div>
491                </div>
492<!--&ndash;&gt;-->
493        <hr>
494        <br><br>
495                <div class="content has-text-justified">
496          <div class="column is-six-fifths">
497        <h2 class="title is-3"> Fine-tuned X-Fusion on Downsteam Tasks</h2>
498        Can X-Fusion extend its capabilities to other downstream vision-and-language tasks?
499        We fine-tuned our model on four tasks simultaneously—including image editing, localization,
500        in/oupt-painting, and Visual Question Answering using internal datasets for 20k training steps.
501        In the following figure, we demonstrate that our unified X-Fusion model can handle multiple tasks without
502        creating task-specific weights.
503        </div>
504          </div>
505
506        <center><img src="./static/images/xfusion_four_tasks_web1.png" width="98% px"></center>
507               <br><br>
508        <hr>
509        <br><br>
510
511<!--        <br><br>-->
512        <div class="content has-text-justified">
513          <div class="column is-six-fifths">
514        <h2 class="title is-3"> Transfer from Pretrained Diffusion Model</h2>
515
516        One main advantage of our dual-tower design is that allows non-identical block designs in language and vision towers as each block in the both tower processes the entire feature sequence independently.
517            With this advantage, we could transfer the image generation capability from a large-scale pretrained diffusion model that uses diffusion transformers.
518            We trained a a variation of X-Fusion using Llama3 model as the language tower and an in-house pretrained text-to-image DiT model as its vision tower, notated as X-Fusion(Pretrained DiT), with 50K steps.
519            We show that X-Fusion(Pretrained DiT) obtained stronger image generation capability compared to the vanilla X-Fusion.
520
521        </div>
522          </div>
523        <center><img src="./static/images/xfusion-with-dit-web_v1.svg" width="96% px"></center>
524        <!--          Our X-Fusion model can also be used with pre-trained DiT model, which is a diffusion-based text-to-image generation model.-->
525        <!--          <hr>-->
526        <!--          <br><br>-->
527        <!--          <h2 class="title is-3">🖼️ Qualitative Result: Multi-Turn Generation</h2>-->
528        <!--          <center><img src="./static/images/xfusion_qual_4_conv_gen.jpg" width="90% px"></center>-->
529
530      </div>
531    </div>
532  </div>
533</section>
534<hr>
535
536
537<section class="section" id="BibTeX">
538  <div class="container is-max-desktop content">
539    <h2 class="title">📥 BibTeX</h2>
540    <pre><code>
541    @article{mo2025xfusion,
542      title={X-fusion: Introducing new modality to frozen large language models},
543      author={Mo, Sicheng and Nguyen, Thao and Huang, Xun and Iyer, Siddharth Srinivasan and Li, Yijun and Liu, Yuchen and Tandon, Abhishek and Shechtman, Eli and Singh, Krishna Kumar and Lee, Yong Jae and others},
544      journal={arXiv preprint arXiv:2504.20996},
545      year={2025}
546    }
547  </code></pre>
548  </div>
549
550  <section class="section" id="Acknowledgement">
551    <div class="container is-max-desktop content">
552      <h2 class="title">💌 Acknowledgement</h2>
553      <p>
554        This website is adapted from <a
555        href="https://github.com/nerfies/nerfies.github.io">Nerfies</a>, licensed under a <a href="http://creativecommons.org/licenses/by-sa/4.0/"
556                                                                                             rel="license">Creative
557        Commons Attribution-ShareAlike 4.0 International License</a>.
558      </p>
559      <center>
560        <i>Thank you (.❛ ᴗ ❛.).</i>
561        <p>
562        </p>
563    </div>
564  </section>
565</body>
566
567</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.