1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <meta name="description" 6 content="Deformable Neural Radiance Fields creates free-viewpoint portraits (nerfies) from casually captured videos."> 7 <meta name="keywords" content="Model Fusion"> 8 <meta name="viewport" content="width=device-width, initial-scale=1"> 9 <title>Probabilistic Token Alignment for Large Language Model Fusion</title> 10 11 <!-- Global site tag (gtag.js) - Google Analytics -->
vendor: 66 bytes, lines 11-12
11 12 <script async src="https://www.googletagmanager.com/gtag/js?id=
12G-PYVRSFMDRL
vendor: 14 bytes, lines 12-13
12"></script> 13
13<script> 14
vendor: 155 bytes, lines 14-22
14window.dataLayer = window.dataLayer || []; 15 16 function gtag() { 17 dataLayer.push(arguments); 18 } 19 20 gtag('js', new Date()); 21 22 gtag('config', '
22G-PYVRSFMDRL
vendor: 6 bytes, lines 22-23
22'); 23
23</script>
23 24 25 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 26 rel="stylesheet"> 27 28 <link rel="stylesheet" href="./static/css/bulma.min.css"> 29 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 30 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 31 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 32 <link rel="stylesheet" 33 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 34 <link rel="stylesheet" href="./static/css/index.css"> 35 <link rel="icon" href="./static/images/favicon.ico"> 36 37
37<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
37 38
38<script defer src="./static/js/fontawesome.all.min.js"></script>
38 39
39<script src="./static/js/bulma-carousel.min.js"></script>
39 40
40<script src="./static/js/bulma-slider.min.js"></script>
40 41
41<script src="./static/js/index.js"></script>
41 42</head> 43<body> 44 45<nav class="navbar" role="navigation" aria-label="main navigation"> 46 <div class="navbar-brand"> 47 <a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false"> 48 <span aria-hidden="true"></span> 49 <span aria-hidden="true"></span> 50 <span aria-hidden="true"></span> 51 </a> 52 </div> 53 <div class="navbar-menu"> 54 <div class="navbar-start" style="flex-grow: 1; justify-content: center;"> 55 <a class="navbar-item" href="http://runjia.tech"> 56 <span class="icon"> 57 <i class="fas fa-home"></i> 58 </span> 59 </a> 60 61 </div> 62 63 </div> 64</nav> 65 66 67<section class="hero"> 68 <div class="hero-body"> 69 <div class="container is-max-desktop"> 70 <div class="columns is-centered"> 71 <div class="column has-text-centered"> 72 <h1 class="title is-1 publication-title">Probabilistic Token Alignment for Large Language Model Fusion</h1> 73 <div class="is-size-5 publication-authors"> 74 <span class="author-block"> 75 <a href="https://runjia.tech">Runjia Zeng</a><sup>1</sup>, 76 </span> 77 <span class="author-block"> 78 James Chenhao Liang<sup>2</sup>, 79 </span> 80 <span class="author-block"> 81 Cheng Han<sup>3</sup>, 82 </span> 83 <span class="author-block"> 84 Zhiwen Cao<sup>4</sup>, 85 </span> 86 <span class="author-block"> 87 Jiahao Liu<sup>5</sup>, 88 </span> 89 <span class="author-block"> 90 Xiaojun Quan<sup>6</sup>, 91 </span> 92 <span class="author-block"> 93 Yingjie Victor Chen<sup>7</sup>, 94 </span> 95 <span class="author-block"> 96 Lifu Huang<sup>8</sup>, 97 </span> 98 <span class="author-block"> 99 Tong Geng<sup>9,10</sup>, 100 </span> 101 <span class="author-block"> 102 Qifan Wang<sup>11</sup>, 103 </span> 104 <span class="author-block"> 105 Dongfang Liu<sup>1â </sup> 106 </span><br><br> 107 </div> 108 <p> 109 <div class="is-size-5 publication-authors"> 110 <span class="author-block"><sup>1</sup>Rochester Institute of Technology,</span> 111 <span class="author-block"><sup>2</sup>U.S. Naval Research Laboratory,</span><br> 112 <span class="author-block"><sup>3</sup>University of Missouri-Kansas City,</span> 113 <span class="author-block"><sup>4</sup>Adobe,</span> 114 <span class="author-block"><sup>5</sup>Meituan,</span><br> 115 <span class="author-block"><sup>6</sup>Sun Yat-sen University,</span> 116 <span class="author-block"><sup>7</sup>Purdue University,</span> 117 <span class="author-block"><sup>8</sup>UC Davis,</span><br> 118 <span class="author-block"><sup>9</sup>University of Rochester,</span> 119 <span class="author-block"><sup>10</sup>Rice University,</span> 120 <span class="author-block"><sup>11</sup>Meta AI,</span><br> 121 <span class="author-block"><sup>â </sup>Corresponding author</span> 122 </div> 123 124 <div class="column has-text-centered"> 125 <div class="publication-links"> 126 <!-- PDF Link. --> 127 <span class="link-block"> 128 <a href="https://arxiv.org/pdf/2509.17276" 129 class="external-link button is-normal is-rounded is-dark"> 130 <span class="icon"> 131 <i class="fas fa-file-pdf"></i> 132 </span> 133 <span>Paper</span> 134 </a> 135 </span> 136 <span class="link-block"> 137 <a href="https://arxiv.org/abs/2509.17276" 138 class="external-link button is-normal is-rounded is-dark"> 139 <span class="icon"> 140 <i class="ai ai-arxiv"></i> 141 </span> 142 <span>arXiv</span> 143 </a> 144 </span> 145 <!-- Video Link. --> 146 <!-- <span class="link-block"> 147 <a href="https://recorder-v3.slideslive.com/?share=96649&s=ddeb9230-5258-4ce1-bcad-092710eb580f" 148 class="external-link button is-normal is-rounded is-dark">
149 <span class="icon"> 150 <i class="fab fa-youtube"></i> 151 </span> 152 <span>Video</span> 153 </a> 154 </span> --> 155 <!-- Code Link. --> 156 <span class="link-block"> 157 <a href="https://github.com/runtsang/PTA-LLM" 158 class="external-link button is-normal is-rounded is-dark"> 159 <span class="icon"> 160 <i class="fab fa-github"></i> 161 </span> 162 <span>Code</span> 163 </a> 164 <!-- <span class="link-block"> 165 <a href="./static/images/VFPT.pptx" 166 class="external-link button is-normal is-rounded is-dark"> 167 <span class="icon"> 168 <i class="fas fa-file-powerpoint"></i> 169 </span> 170 <span>Slides</span> 171 </a> --> 172 </span> 173 </div> 174 175 </div> 176 </div> 177 </div> 178 </div> 179 </div> 180</section> 181 182<section class="hero teaser"> 183 <div class="container is-max-desktop"> 184 <div class="hero-body"> 185 <!-- <img src="./static/images/title.png" 186 class="image is-fullwidth" 187 alt="Interpolate start reference image."/> --> 188 <img src="./static/images/overview.jpg" 189 class="interpolation-image" 190 alt="Interpolate start reference image."/> 191 <h2 class="subtitle has-text-centered"> 192 Probabilistic Token Alignment for Large Language Model Fusion. 193 </h2> 194 </div> 195 196 </div> 197</section> 198 199<section class="section"> 200 <div class="container is-max-desktop"> 201 <!-- Abstract. --> 202 <div class="columns is-centered has-text-centered"> 203 <div class="column is-four-fifths"> 204 <h2 class="title is-3">Abstract</h2> 205 <div class="content has-text-justified"> 206 <p> 207 Training large language models (LLMs) from scratch can yield models with unique functionalities and strengths, but it is costly and often leads to redundant capabilities. A more cost-effective alternative is to fuse existing pre-trained LLMs with different architectures into a more powerful model. However, a key challenge in existing model fusion is their dependence on manually predefined vocabulary alignment, which may not generalize well across diverse contexts, leading to performance degradation in several evaluation. 208 </p> 209 <p> 210 To solve this, we draw inspiration from distribution learning and propose the probabilistic token alignment method as a general and soft mapping for alignment, named as PTA-LLM. Our approach innovatively reformulates token alignment into a classic mathematical problem: optimal transport, seamlessly leveraging distribution-aware learning to facilitate more coherent model fusion. Apart from its inherent generality, PTA-LLM exhibits interpretability from a distributional perspective, offering insights into the essence of the token alignment. Empirical results demonstrate that probabilistic token alignment enhances the target model's performance across multiple capabilities. 211 </p> 212 <p> 213 P.S. The video and slides are on the way!! 214 </p> 215 </div> 216 </div> 217 </div> 218 <!--/ Abstract. --> 219 220 <!-- Paper video. --> 221 <!-- <div class="columns is-centered has-text-centered"> 222 <div class="column is-four-fifths"> 223 <h2 class="title is-3">Video</h2> 224 <div class="publication-video"> 225 <iframe src="https://recorder-v3.slideslive.com/?share=96649&s=ddeb9230-5258-4ce1-bcad-092710eb580f" 226 frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> 227 </div> 228 </div> 229 </div> --> 230 231 <div class="columns is-centered has-text-centered"> 232 <div class="column is-four-fifths"> 233 <img src="./static/images/ptallm.jpg" 234 class="image is-fullwidth" 235 alt="Interpolate start reference image."/> 236 <img src="./static/images/poster.jpg" 237 class="image is-fullwidth" 238 alt="Interpolate start reference image."/> 239 <!-- <img src="./static/images/2.png" 240 class="image is-fullwidth" 241 alt="Interpolate start reference image."/> 242 <img src="./static/images/3.png" 243 class="image is-fullwidth" 244 alt="Interpolate start reference image."/> 245 <img src="./static/images/4.png" 246 class="image is-fullwidth" 247 alt="Interpolate start reference image."/> 248 <img src="./static/images/5.png" 249 class="image is-fullwidth" 250 alt="Interpolate start reference image."/> 251 --> 252 </div> 253 </div> 254 255 <!--/ Paper video. --> 256 </div> 257</section> 258 259 260 261 262<section class="section" id="BibTeX">
263 <div class="container is-max-desktop content"> 264 <h2 class="title">BibTeX</h2> 265 If you find our work useful, please consider citing our paper: <br><br> 266 267 <pre><code>@inproceedings{zeng2025probabilistic, 268 title={Probabilistic Token Alignment for Large Language Model Fusion}, 269 author={Zeng, Runjia and Liang, James Chenhao and Han, Cheng and Cao, Zhiwen and Liu, Jiahao and Quan, Xiaojun and Chen, Yingjie Victor and Huang, Lifu and Geng, Tong and Wang, Qifan and Liu, Dongfang}, 270 booktitle={NeurIPS}, 271 year={2025} 272} 273</code></pre> 274 </div> 275</section> 276 277 278<footer class="footer"> 279 <div class="container"> 280 <div class="content has-text-centered"> 281 <a class="icon-link" href="https://github.com/runtsang" class="external-link" disabled> 282 <i class="fab fa-github"></i> 283 </a> 284 </div> 285 <div class="columns is-centered"> 286 <div class="column is-8"> 287 <div class="content"> 288 <p> 289 This website is constructed using the templet provided by <a 290 href="https://github.com/nerfies/nerfies.github.io">Nerfies</a>. Thanks for their effort. 291 </p> 292 </div> 293 </div> 294 </div> 295 </div> 296</footer> 297 298</body> 299</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.