1<!DOCTYPE html> <html lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title> Publications | Tri Dao </title> <meta name="author" content="Tri Dao"> <meta name="description" content="Homepage of Tri Dao. "> <meta name="keywords" content="jekyll, jekyll-theme, academic-website, portfolio-website"> <meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self' 'unsafe-inline' https:; style-src 'self' 'unsafe-inline' https:; img-src 'self' data: https:; font-src 'self' data: https:; media-src 'self' https:; frame-src 'self' https:; connect-src 'self' https:;"> <link rel="stylesheet" href="/assets/css/bootstrap.min.css?v=a4b3f509e79c54a512b890d73235ef04"> <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/mdb.min.css" integrity="sha256-jpjYvU3G3N6nrrBwXJoVEYI/0zw8htfFnhT9ljN3JJw=" crossorigin="anonymous"> <link defer rel="stylesheet" href="/assets/css/academicons.min.css?v=f0b7046b84e425c55f3463ac249818f5"> <link defer rel="stylesheet" href="/assets/css/scholar-icons.css?v=62b2ac103a88034e6882a5be5f3e2772"> <link defer rel="stylesheet" type="text/css" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700|Roboto+Slab:100,300,400,500,700|Material+Icons&display=swap"> <link defer rel="stylesheet" href="/assets/css/jekyll-pygments-themes-github.css?v=591dab5a4e56573bf4ef7fd332894c99" media="" id="highlight_theme_light"> <link rel="shortcut icon" href="data:image/svg+xml,<svg%20xmlns=%22http://www.w3.org/2000/svg%22%20viewBox=%220%200%20100%20100%22><text%20y=%22.9em%22%20font-size=%2290%22>%E2%9A%9B%EF%B8%8F</text></svg>"> <link rel="stylesheet" href="/assets/css/main.css?v=d41d8cd98f00b204e9800998ecf8427e"> <link rel="canonical" href="https://tridao.me/publications/">
1<script src="/assets/js/theme.js?v=5fea5159b787642c1bbc1f334d60f883"></script>
1 <link defer rel="stylesheet" href="/assets/css/jekyll-pygments-themes-native.css?v=5847e5ed4a4568527aa6cfab446049ca" media="none" id="highlight_theme_dark">
1<script> 2 initTheme(); 3 </script>
3 </head> <body class="fixed-top-nav "> <header> <nav id="navbar" class="navbar navbar-light navbar-expand-sm fixed-top" role="navigation"> <div class="container"> <a class="navbar-brand title font-weight-lighter" href="/"> Tri Dao </a> <button class="navbar-toggler collapsed ml-auto" type="button" data-toggle="collapse" data-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar top-bar"></span> <span class="icon-bar middle-bar"></span> <span class="icon-bar bottom-bar"></span> </button> <div class="collapse navbar-collapse text-right" id="navbarNav"> <ul class="navbar-nav ml-auto flex-nowrap"> <li class="nav-item "> <a class="nav-link" href="/">About </a> </li> <li class="nav-item "> <a class="nav-link" href="/blog/">Blog </a> </li> <li class="nav-item active"> <a class="nav-link" href="/publications/">Publications <span class="sr-only">(current)</span> </a> </li> <li class="nav-item "> <a class="nav-link" href="/awards/">Awards </a> </li> <li class="nav-item "> <a class="nav-link" href="/teaching/">Teaching </a> </li> <li class="nav-item "> <a class="nav-link" href="/repositories/">Repositories </a> </li> <li class="nav-item"> <button id="search-toggle" title="Search" onclick="openSearchModal()"> <span class="nav-link">ctrl k <i class="fa-solid fa-magnifying-glass"></i></span> </button> </li> <li class="toggle-container"> <button id="light-toggle" title="Change theme"> <i class="fa-half-sun-moon" id="light-toggle-system"></i> <i class="fa-solid fa-moon" id="light-toggle-dark"></i> <i class="fa-solid fa-sun" id="light-toggle-light"></i> </button> </li> </ul> </div> </div> </nav> <progress id="progress" value="0"> <div class="progress-container"> <span class="progress-bar"></span> </div> </progress> </header> <div class="container mt-5" role="main"> <div class="post"> <header class="post-header"> <h1 class="post-title">Publications</h1> <p class="post-description"></p> </header> <article> <div class="publications"> <h2 class="bibliography">2026</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="flash4" class="col-sm-8"> <div class="title">FlashAttention-4: Algorithm and Kernel Pipelining Co-design for Asymmetric Hardware Scaling</div> <div class="author"> Ted Zadouri<sup>*</sup>, Jay Shah<sup>*</sup>, Markus Hohnerbach<sup>*</sup>, Timmy Liu, Vijay Thakkar, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In Machine Learning and Systems (MLSys)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Best Paper Honorable Mention</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Best Paper Honorable Mention</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="galanti2026addressing" class="col-sm-8"> <div class="title">Addressing the Orchestration Gap in Generalist Robots via Physical Agency</div> <div class="author"> Liane Galanti, Dhruv Shah, and <em>Tri Dao</em> </div> <div class="periodical"> <em>arXiv preprint arXiv:2607.21725</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2607.21725" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://lianegalanti.github.io/Pigey/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">HTML</a> <a href="https://github.com/lianegalanti/Pigey" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="yu2026introspective" class="col-sm-8"> <div class="title">Introspective Diffusion Language Models</div> <div class="author"> Yifan Yu, Yuqing Jian, Junxiong Wang, Zhongzhu Zhou, Donglin Zhuang, Xinyu Fang, Sri Yanamandra, Xiaoxia Wu, Qingyang Wu, Shuaiwen Leon Song, <em>Tri Dao</em>, Ben Athiwaratkun, James Zou, Fan Lai, and Chenfeng Xu </div> <div class="periodical"> <em>In Conference on Language Modeling (COLM)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2604.11035" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/Introspective-Diffusion/I-DLM" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="maheswaran2026squeeze" class="col-sm-8"> <div class="title">Squeeze Evolve: A Unified Multi-Model Orchestration Framework for Verifier-Free Evolution</div> <div class="author"> Monishwaran Maheswaran, Leon Lakhani, Zhongzhu Zhou, Shijia Yang, Junxiong Wang, Coleman Hooper, Yuezhou Hu, Rishabh Tiwari, Jue Wang, Harman Singh, Qingyang Wu, Yuqing Jian, Ce Zhang, Kurt Keutzer, <em>Tri Dao</em>, Xiaoxia Wu, Ben Athiwaratkun, James Zou, and Chenfeng Xu </div> <div class="periodical"> <em>In Conference on Language Modeling (COLM)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2604.07725" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/squeeze-evolve/squeeze-evolve" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="yang2026thought" class="col-sm-8"> <div class="title">Thought-Level Beam Search for Reasoning</div> <div class="author"> Lijie Yang, Hongyin Luo, Jiawei Zhao, <em>Tri Dao</em>, and Ravi Netravali </div> <div class="periodical"> <em>In Conference on Language Modeling (COLM)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2608.08020" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div>
3 <div id="ponnusamy2026sonicsampler" class="col-sm-8"> <div class="title">SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification</div> <div class="author"> Pragaash Ponnusamy, Shivam Sahni, Jue Wang, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In Conference on Language Modeling (COLM)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2607.20475" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="amsel2026dion3" class="col-sm-8"> <div class="title">Dion3: Full-Stack Orthogonal Updates</div> <div class="author"> Noah Amsel, Jack Zhang, Kwangjun Ahn, Ali Naeimi, Austin Feng, Berlin Chen, <em>Tri Dao</em>, and John Langford </div> <div class="periodical"> <em>arXiv preprint arXiv:2608.11612</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2608.11612" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/microsoft/dion" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="lahoti2025mamba3" class="col-sm-8"> <div class="title">Mamba-3: Improved Sequence Modeling using State Space Principles</div> <div class="author"> Aakash Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, <em>Tri Dao</em>, and Albert Gu </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Oral</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Oral</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="guo2025sonicmoe" class="col-sm-8"> <div class="title">SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations</div> <div class="author"> Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2512.14080" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="shao2025beat" class="col-sm-8"> <div class="title">Beat the long tail: Distribution-Aware Speculative Decoding for RL Training</div> <div class="author"> Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, Alpay Ariyak, Xiaoxia Wu, Ameen Patel, Jue Wang, Percy Liang, <em>Tri Dao</em>, and others </div> <div class="periodical"> <em>In Machine Learning and Systems (MLSys)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2511.13841" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="kumar2026speculative" class="col-sm-8"> <div class="title">Speculative Speculative Decoding</div> <div class="author"> Tanishq Kumar, <em>Tri Dao</em>, and Avner May </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="guo2025log" class="col-sm-8"> <div class="title">Log-Linear Attention</div> <div class="author"> Han Guo, Songlin Yang, Tarushii Goel, Eric P Xing, <em>Tri Dao</em>, and Yoon Kim </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2506.04761" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="mishra2026mrnn" class="col-sm-8"> <div class="title">M^2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling</div> <div class="author"> Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, and <em>Tri Dao</em> </div> <div class="periodical"> <em>arXiv preprint arXiv:2603.14360</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2603.14360" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="chen2026aihw2035" class="col-sm-8"> <div class="title">AI+HW 2035: Shaping the Next Decade</div> <div class="author"> Deming Chen, Jason Cong, Azalia Mirhoseini, Christos Kozyrakis, Subhasish Mitra, Jinjun Xiong, Cliff Young, Anima Anandkumar, Michael Littman, Aron Kirschen, Sophia Shao, Serge Leef, Naresh Shanbhag, Dejan Milojicic, Michael Schulte, Gert Cauwenberghs, Jerry M.
3Chow, <em>Tri Dao</em>, Kailash Gopalakrishnan, Richard Ho, Hoshik Kim, Kunle Olukotun, David Z. Pan, Mark Ren, Dan Roth, Aarti Singh, Yizhou Sun, Yusu Wang, Yann LeCun, and Ruchir Puri </div> <div class="periodical"> <em>arXiv preprint arXiv:2603.05225</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2603.05225" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="wang2026speculative" class="col-sm-8"> <div class="title">When RL Meets Adaptive Speculative Training: A Unified Training-Serving System</div> <div class="author"> Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, Zelei Shao, Yubo Wang, Yinghui Liu, Qingyang Wu, Avner May, Sri Yanamandra, Yineng Zhang, Ce Zhang, <em>Tri Dao</em>, Percy Liang, Ben Athiwaratkun, Shuaiwen Leon Song, Chenfeng Xu, and Xiaoxia Wu </div> <div class="periodical"> <em>arXiv preprint arXiv:2602.06932</em>, 2026 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2602.06932" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> </ol> <h2 class="bibliography">2025</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="pan2025marconi" class="col-sm-8"> <div class="title">Marconi: Prefix Caching for the Era of Hybrid LLMs</div> <div class="author"> Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, <em>Tri Dao</em>, Ravi Netravali, and Yida Wang </div> <div class="periodical"> <em>In Machine Learning and Systems (MLSys)</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Outstanding Paper Honorable Mention</a> <a href="http://arxiv.org/abs/2411.19379" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/ruipeterpan/marconi" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Outstanding Paper Honorable Mention</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="zadouri2025hardware" class="col-sm-8"> <div class="title">Hardware-Efficient Attention for Fast Decoding</div> <div class="author"> Ted Zadouri, Hubert Strauss, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In Conference on Language Modeling (COLM)</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2505.21487" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="po2025long" class="col-sm-8"> <div class="title">Long-context state-space video world models</div> <div class="author"> Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, <em>Tri Dao</em>, Eli Shechtman, Gordon Wetzstein, and Xun Huang </div> <div class="periodical"> <em>In International Conference on Computer Vision (ICCV)</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2505.20171" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="wang2025m1" class="col-sm-8"> <div class="title">M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models</div> <div class="author"> Junxiong Wang, Wen-Ding Li, Daniele Paliotta, Daniel Ritter, Alexander M Rush, and <em>Tri Dao</em> </div> <div class="periodical"> <em>arXiv preprint arXiv:2504.10449</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div>
3 <div id="ma2025hybridna" class="col-sm-8"> <div class="title">HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model</div> <div class="author"> Mingqian Ma, Guoqing Liu, Chuan Cao, Pan Deng, <em>Tri Dao</em>, Albert Gu, Peiran Jin, Zhao Yang, Yingce Xia, Renqian Luo, and others </div> <div class="periodical"> <em>arXiv preprint arXiv:2502.10807</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="paliotta2025thinking" class="col-sm-8"> <div class="title">Thinking slow, fast: Scaling inference compute with distilled reasoners</div> <div class="author"> Daniele Paliotta, Junxiong Wang, Matteo Pagliardini, Kevin Y Li, Aviv Bick, J Zico Kolter, Albert Gu, François Fleuret, and <em>Tri Dao</em> </div> <div class="periodical"> <em>arXiv preprint arXiv:2502.20339</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="zhang2025ladder" class="col-sm-8"> <div class="title">Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping</div> <div class="author"> Muru Zhang, Mayank Mishra, Zhongzhu Zhou, William Brandon, Jue Wang, Yoon Kim, Jonathan Ragan-Kelley, Shuaiwen Leon Song, Ben Athiwaratkun, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="oncescu2025opportunistic" class="col-sm-8"> <div class="title">Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining</div> <div class="author"> Costin-Andrei Oncescu, Qingyang Wu, Wai Tong Chung, Robert Wu, Bryan Gopal, Junxiong Wang, <em>Tri Dao</em>, and Ben Athiwaratkun </div> <div class="periodical"> <em>arXiv preprint arXiv:2511.02237</em>, 2025 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2511.02237" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> </ol> <h2 class="bibliography">2024</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="flash3" class="col-sm-8"> <div class="title">FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision</div> <div class="author"> Jay Shah<sup>*</sup>, Ganesh Bikshandi<sup>*</sup>, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2407.08608" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/Dao-AILab/flash-attention/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="mamba2" class="col-sm-8"> <div class="title">Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality</div> <div class="author"> <em>Tri Dao<sup>*</sup></em> and Albert Gu<sup>*</sup> </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2405.21060" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/state-spaces/mamba/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="mamba" class="col-sm-8"> <div class="title">Mamba: Linear-Time Sequence Modeling with Selective State Spaces</div> <div class="author"> Albert Gu<sup>*</sup> and <em>Tri Dao<sup>*</sup></em> </div> <div class="periodical"> <em>Conference on Language Modeling (COLM)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Outstanding Paper</a> <a href="http://arxiv.org/abs/2312.00752" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/state-spaces/mamba/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Outstanding Paper</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="weber2024redpajama" class="col-sm-8"> <div class="title">Redpajama: an open dataset for training large language models</div> <div class="author"> Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virgini Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, <em>Tri Dao</em>, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="wang2024mamba" class="col-sm-8"> <div class="title">The mamba in the llama: Distilling and accelerating hybrid models</div> <div class="author"> Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="hwang2024hydra" class="col-sm-8"> <div class="title">Hydra: Bidirectional state space models through generalized matrix mixers</div> <div class="author"> Sukjun Hwang, Aakash Lahoti, <em>Tri Dao</em>, and Albert Gu </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="liu2024bitdelta" class="col-sm-8"> <div class="title">Bitdelta: Your fine-tune may only be worth one bit</div> <div class="author"> James Liu, Guangxuan Xiao, Kai Li, Jason D Lee, Song Han, <em>Tri Dao</em>, and Tianle Cai </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="waleffe2024empirical" class="col-sm-8"> <div class="title">An Empirical Study of Mamba-based Language Models</div> <div class="author"> Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, <em>Tri Dao</em>, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, and others </div> <div class="periodical"> <em>arXiv preprint arXiv:2406.07887</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="lozhkov2024starcoder" class="col-sm-8"> <div class="title">Starcoder 2 and the stack v2: The next generation</div> <div class="author"> Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and others </div> <div class="periodical"> <em>arXiv preprint arXiv:2402.19173</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="schiff2024caduceus" class="col-sm-8"> <div class="title">Caduceus: Bi-directional equivariant long-range DNA sequence modeling</div> <div class="author"> Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, <em>Tri Dao</em>, Albert Gu, and Volodymyr Kuleshov </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="cai2024medusa" class="col-sm-8"> <div class="title">Medusa: Simple LLM inference acceleration framework with multiple decoding heads</div> <div class="author"> Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and <em>Tri Dao</em> </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2023flashattention2" class="col-sm-8"> <div class="title">FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning</div> <div class="author"> <em>Tri Dao</em> </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2024 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2307.08691" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/Dao-AILab/flash-attention/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> </div> </div> </li> </ol> <h2 class="bibliography">2023</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="liu2023deja" class="col-sm-8"> <div class="title">Deja vu: Contextual sparsity for efficient llms at inference time</div> <div class="author"> Zichang Liu, Jue Wang, <em>Tri Dao</em>, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen </div> <div class="periodical"> <em>In International Conference on Machine Learning</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Oral</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Oral</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="zhang2023effectively" class="col-sm-8"> <div class="title">Effectively modeling time series with simple discrete state space
3s</div> <div class="author"> Michael Zhang, Khaled K Saab, Michael Poli, <em>Tri Dao</em>, Karan Goel, and Christopher Ré </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="poli2023hyena" class="col-sm-8"> <div class="title">Hyena Hierarchy: Towards Larger Convolutional Language Models</div> <div class="author"> Michael Poli<sup>*</sup>, Stefano Massaroli<sup>*</sup>, Eric Nguyen, Daniel Y Fu, <em>Tri Dao</em>, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Oral</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Oral</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="fu2023simple" class="col-sm-8"> <div class="title">Simple Hardware-Efficient Long Convolutions for Sequence Modeling</div> <div class="author"> Daniel Y. Fu<sup>*</sup>, Elliot L Epstein<sup>*</sup>, Eric Nguyen, Armin W Thomas, Michael Zhang, <em>Tri Dao</em>, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2023hungry" class="col-sm-8"> <div class="title">Hungry Hungry Hippos: Towards Language Modeling with State Space Models</div> <div class="author"> <em>Tri Dao<sup>*</sup></em>, Daniel Y. Fu<sup>*</sup>, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In The International Conference on Learning Representations (ICLR)</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Spotlight</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Spotlight</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="li2023starcoder" class="col-sm-8"> <div class="title">StarCoder: may the source be with you!</div> <div class="author"> Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, Joao Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, <em>Tri Dao</em>, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Werra, and Harm Vries </div> <div class="periodical"> <em>Transactions on Machine Learning Research (TMLR)</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> <a href="http://arxiv.org/abs/2305.06161" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2023flashdecoding" class="col-sm-8"> <div class="title">Flash-Decoding for long-context inference</div> <div class="author"> <em>Tri Dao</em>, Daniel Haziza, Francisco Massa, and Grigory Sizov </div> <div class="periodical"> <em>PyTorch Blog</em>, 2023 </div> <div class="periodical"> </div> <div class="links"> <a href="https://pytorch.org/blog/flash-decoding/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">HTML</a> </div> </div> </div> </li> </ol> <h2 class="bibliography">2022</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2022flashattention" class="col-sm-8"> <div class="title">FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness</div> <div class="author"> <em>Tri Dao</em>, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Awards</a> <a href="http://arxiv.org/abs/2205.14135" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/Dao-AILab/flash-attention/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Best Paper award at the ICML Hardware Aware Efficient Training Workshop 2022, Inaugural Stanford Open Source Software Prize 2024</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2022monarch" class="col-sm-8"> <div class="title">Monarch: Expressive Structured Matrices for Efficient and Accurate Training</
3div> <div class="author"> <em>Tri Dao</em>, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Outstanding Paper runner-up</a> <a href="http://arxiv.org/abs/2204.00595" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">arXiv</a> <a href="https://github.com/HazyResearch/fly" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Code</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Outstanding Paper runner-up</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="yuan2022decentralized" class="col-sm-8"> <div class="title">Decentralized Training of Foundation Models in Heterogeneous Environments</div> <div class="author"> Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, <em>Tri Dao</em>, Beidi Chen, Percy Liang, Christopher Ré, and Ce Zhang </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Oral</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Oral</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="wang2022finetuning" class="col-sm-8"> <div class="title">Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees</div> <div class="author"> June Wang, Binhang Yuan, Luka Rimanic, Yongjun He, <em>Tri Dao</em>, Beidi Chen, Percy Liang, Christopher Ré, and Ce Zhang </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="poli2022transform" class="col-sm-8"> <div class="title">Transform Once: Efficient Operator Learning in Frequency Domain</div> <div class="author"> Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, <em>Tri Dao</em>, Christopher Ré, and Stefano Ermon </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="nguyen2022s4nd" class="col-sm-8"> <div class="title">S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces</div> <div class="author"> Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, <em>Tri Dao</em>, Stephen Baccus, and Christopher Ré </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="meng2022butterflyflow" class="col-sm-8"> <div class="title">ButterflyFlow: Building Invertible Layers with Butterfly Matrices</div> <div class="author"> Chenlin Meng, Linqi Zhou, Kristy Choi, <em>Tri Dao</em>, and Stefano Ermon </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2021pixelated" class="col-sm-8"> <div class="title">Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models</div> <div class="author"> <em>Tri Dao<sup>*</sup></em>, Beidi Chen<sup>*</sup>, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2022 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Spotlight</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Spotlight</p> </div> </div> </div> </li> </ol> <h2 class="bibliography">2021</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="chen2021scatterbrain" class="col-sm-8"> <div class="title">Scatterbrain: Unifying Sparse and Low-rank Attention</div> <div class="author"> Beidi Chen<sup>*</sup>, <em>Tri Dao<sup>*</sup></em>, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS)</em>, 2021 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="gu2021combining" class="col-sm-8"> <div class="title">Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers</div> <div class="author"> Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, <em>Tri Dao</em>, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>Advances in Neural Information Processing Systems</em>, 2021 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="roberts2021rethinking" class="col-sm-8"> <div class="title">Rethinking Neural Operations for Diverse Tasks</div> <div class="author"> Nicholas Roberts, Mikhail Khodak, <em>Tri Dao</em>, Liam Li, Christopher Ré, and Ameet Talwalkar </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems</em>, 2021 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="davis2021catformer" class="col-sm-8"> <div class="title">Catformer: Designing Stable Transformers via Sensitivity Analysis</div> <div class="author"> Jared Q Davis<sup>*</sup>, Albert Gu<sup>*</sup>, Krzysztof Choromanski, <em>Tri Dao</em>, Christopher Ré, Chelsea Finn, and Percy Liang </div> <div class="periodical"> <em>In International Conference on Machine Learning (ICML)</em>, 2021 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2021knowledge" class="col-sm-8"> <div class="title">Knowledge Distillation as Semiparametric Inference</div> <div class="author"> <em>Tri Dao</em>, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2021 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="chen2021mongoose" class="col-sm-8"> <div class="title">MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training</
3div> <div class="author"> Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, <em>Tri Dao</em>, Zhao Song, Anshumali Shrivastava, and Christopher Ré </div> <div class="periodical"> <em>In International Conference on Learning Representations (ICLR)</em>, 2021 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Oral</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Oral</p> </div> </div> </div> </li> </ol> <h2 class="bibliography">2020</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="gu2020hippo" class="col-sm-8"> <div class="title">HiPPO: Recurrent Memory with Optimal Polynomial Projections</div> <div class="author"> Albert Gu<sup>*</sup>, <em>Tri Dao<sup>*</sup></em>, Stefano Ermon, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In Advances in neural information processing systems (NeurIPS)</em>, 2020 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Spotlight</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Spotlight</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2019kaleidoscope" class="col-sm-8"> <div class="title">Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps</div> <div class="author"> <em>Tri Dao</em>, Nimit Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In The International Conference on Learning Representations (ICLR)</em>, 2020 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Spotlight</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Spotlight</p> </div> </div> </div> </li> </ol> <h2 class="bibliography">2019</h2> <ol class="bibliography"> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="may2019downstream" class="col-sm-8"> <div class="title">On the downstream performance of compressed word embeddings</div> <div class="author"> Avner May, Jian Zhang, <em>Tri Dao</em>, and Christopher Ré </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS) 32</em>, 2019 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Spotlight</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Spotlight</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="kuck2019approximating" class="col-sm-8"> <div class="title">Approximating the Permanent by Sampling from Adaptive Partitions</div> <div class="author"> Jonathan Kuck, <em>Tri Dao</em>, Hamid Rezatofighi, Ashish Sabharwal, and Stefano Ermon </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS) 32</em>, 2019 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="kuck2019adaptive" class="col-sm-8"> <div class="title">Adaptive Hashing for Model Counting</div> <div class="author"> Jonathan Kuck, <em>Tri Dao</em>, Shengjia Zhao, Burak Bartan, Ashish Sabharwal, and Stefano Ermon </div> <div class="periodical"> <em>In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI)</em>, 2019 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2019learning" class="col-sm-8"> <div class="title">Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations</div> <div class="author"> <em>Tri Dao</em>, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In The International Conference on Machine Learning (ICML) 36</em>, 2019 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Oral</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Oral</p> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2019kernel" class="col-sm-8"> <div class="title">A Kernel Theory of Modern Data Augmentation</div> <div class="author"> <em>Tri Dao</em>, Albert Gu, Alexander J Ratner, Virginia Smith, Christopher De Sa, and Christopher Ré </div> <div class="periodical"> <em>In The International Conference on Machine Learning (ICML) 36</em>, 2019 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> <li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="zhang2019low" class="col-sm-8"> <div class="title">Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximation</div> <div class="author"> Jian Zhang, Avner May, <em>Tri Dao</em>, and Christopher Ré </div> <div class="periodical"> <em>In The International Conference on Artificial Intelligence and Statistics (AISTATS) 22</em>, 2019 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li> </ol> <h2 class="bibliography">2018</h2> <ol class="bibliography"><li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="thomas2018learning" class="col-sm-8"> <div class="title">
3Learning Compressed Transforms with Low Displacement Rank</div> <div class="author"> Anna T Thomas, Albert Gu, <em>Tri Dao</em>, Atri Rudra, and Christopher Ré </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS) 31</em>, 2018 </div> <div class="periodical"> </div> <div class="links"> </div> </div> </div> </li></ol> <h2 class="bibliography">2017</h2> <ol class="bibliography"><li> <div class="row"> <div class="col col-sm-2 abbr"> </div> <div id="dao2017gaussian" class="col-sm-8"> <div class="title">Gaussian Quadrature for Kernel Features</div> <div class="author"> <em>Tri Dao</em>, Christopher M De Sa, and Christopher Ré </div> <div class="periodical"> <em>In Advances in Neural Information Processing Systems (NeurIPS) 30</em>, 2017 </div> <div class="periodical"> </div> <div class="links"> <a class="award btn btn-sm z-depth-0" role="button">Spotlight</a> </div> <div class="award hidden d-print-inline"> <p></p> <p>Spotlight</p> </div> </div> </div> </li></ol> </div> </article> </div> </div> <footer class="fixed-bottom" role="contentinfo"> <div class="container mt-0"> © Copyright 2026 Tri Dao. </div> </footer>
3<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/jquery.min.js" integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script>
3
3<script src="/assets/js/bootstrap.bundle.min.js"></script>
3
3<script src="https://cdn.jsdelivr.net/npm/[email protected]/js/mdb.min.js" integrity="sha256-NdbiivsvWt7VYCt6hYNT3h/th9vSTL4EDWeGs5SN3DA=" crossorigin="anonymous"></script>
3
3<script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/masonry.pkgd.min.js" integrity="sha256-Nn1q/fx0H7SNLZMQ5Hw5JLaTRZp0yILA/FRexe19VdI=" crossorigin="anonymous"></script>
3
3<script defer src="https://cdn.jsdelivr.net/npm/[email protected]/imagesloaded.pkgd.min.js" integrity="sha256-htrLFfZJ6v5udOG+3kNLINIKh2gvoKqwEhHYfTTMICc=" crossorigin="anonymous"></script>
3
3<script defer src="/assets/js/masonry.js?v=a0db7e5d5c70cc3252b3138b0c91dcaf" type="text/javascript"></script>
3
3<script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/medium-zoom.min.js" integrity="sha256-ZgMyDAIYDYGxbcpJcfUnYwNevG/xi9OHKaR/8GK+jWc=" crossorigin="anonymous"></script>
3
3<script defer src="/assets/js/zoom.js?v=85ddb88934d28b74e78031fd54cf8308"></script>
3
3<script src="/assets/js/no_defer.js?v=2781658a0a2b13ed609542042a859126"></script>
3
3<script defer src="/assets/js/common.js?v=c15de51d4bb57887caa2c21988d97279"></script>
3
3<script defer src="/assets/js/copy_code.js?v=c8a01c11a92744d44b093fc3bda915df" type="text/javascript"></script>
3
3<script defer src="/assets/js/jupyter_new_tab.js?v=d9f17b6adc2311cbabd747f4538bb15f"></script>
3
3<script async src="https://d1bxh8uas1mnw7.cloudfront.net/assets/embed.js"></script>
3
3<script async src="https://badge.dimensions.ai/badge.js"></script>
3
3<script defer type="text/javascript" id="MathJax-script" src="https://cdn.jsdelivr.net/npm/[email protected]/es5/tex-mml-chtml.js" integrity="sha256-MASABpB4tYktI2Oitl4t+78w/lyA+D7b/s9GEP0JOGI=" crossorigin="anonymous"></script>
3
3<script src="/assets/js/mathjax-setup.js?v=a5bb4e6a542c546dd929b24b8b236dfd"></script>
3
3<script defer src="https://cdnjs.cloudflare.com/polyfill/v3/polyfill.min.js?features=es6" crossorigin="anonymous"></script>
3
3<script defer src="/assets/js/progress-bar.js?v=2f30e0e6801ea8f5036fa66e1ab0a71a" type="text/javascript"></script>
3
3<script src="/assets/js/vanilla-back-to-top.min.js?v=f40d453793ff4f64e238e420181a1d17"></script>
3
3<script> 4 addBackToTop(); 5 </script>
5
5<script type="module" src="/assets/js/search/ninja-keys.min.js?v=a3446f084dcaecc5f75aa1757d087dcf"></script>
5 <ninja-keys hidebreadcrumbs noautoloadmdicons placeholder="Type to start searching"></ninja-keys>
5<script src="/assets/js/search-setup.js?v=6c304f7b1992d4b60f7a07956e52f04a"></script>
5
5<script src="/assets/js/search-data.js"></script>
5
5<script src="/assets/js/shortcut-key.js?v=ccc841c459bfc0e64c1c2b5acd10df02"></script>
5 </body> </html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.