https://hellomuffin.github.io/iterated-learning-for-vlm/

1<!DOCTYPE html>
2<html>
3<head>
4  <meta charset="utf-8">
5  <meta name="description" content="iterated learning improves compositionality in large vision-language models">
6  <meta property="og:title" content="iterated learning improves compositionality in large vision-language models">
7  <meta property="og:description" content="iterated learning improves compositionality in large vision-language models">
8  <meta property="twitter:title" content="iterated learning improves compositionality in large vision-language models">
9  <meta property="twitter:description" content="iterated learning improves compositionality in large vision-language models">
10  <meta property="og:type" content="website">
11  <meta name="keywords" content="vision-language models, LLM, compositionality, iterated learning, cultural transmission">
12  <meta name="viewport" content="width=device-width, initial-scale=1">
13  <title>IL-CLIP</title>
14
15  <!-- Google tag (gtag.js) -->

15
16  <script async src="https://www.googletagmanager.com/gtag/js?id=

16G-D65ZW4CJYF

16"></script>
17  

17<script>
18    

18window.dataLayer = window.dataLayer || [];
19    function gtag(){dataLayer.push(arguments);}
20    gtag('js', new Date());
21  
22    gtag('config', '

22G-D65ZW4CJYF

22');
23  

23</script>

23
24
25  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
26        rel="stylesheet">
27
28  <link rel="stylesheet" href="./static/css/bulma.min.css">
29  <link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
30  <link rel="stylesheet" href="./static/css/bulma-slider.min.css">
31  <link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
32  <link rel="stylesheet"
33        href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
34  <link rel="stylesheet" href="./static/css/index.css">
35
36  <!-- 

36<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>

36 -->
37  

37<script src="./static/js/jquery-3.6.4.min.js"></script>

37
38  

38<script defer src="./static/js/fontawesome.all.min.js"></script>

38
39  

39<script src="./static/js/bulma-carousel.min.js"></script>

39
40  

40<script src="./static/js/bulma-slider.min.js"></script>

40
41  

41<script src="./static/js/lazy.js"></script>

41
42  

42<script src="./static/js/faster.js"></script>

42
43  

43<script src="./static/js/index.js"></script>

43
44</head>
45<body>
46
47<section class="hero">
48  <div class="hero-body">
49    <div class="container is-max-desktop">
50      <div class="columns is-centered">
51        <div class="column has-text-centered">
52          <h1 class="title is-2 publication-title">Iterated Learning Improves Compositionality in Large Vision-Language Models</h1>
53          <div class="is-size-5 publication-authors">
54            <span class="author-block">
55              <a href="https://hellomuffin.github.io/">Chenhao Zheng</a><sup>1,2</sup>, </span>
56            <span class="author-block">
57              <a href="https://jieyuz2.github.io/">Jieyu Zhang</a><sup>1</sup>, </span>
58            <span class="author-block">
59              <a href="https://anikem.github.io/">Aniruddha Kembhavi</a><sup>1,3</sup>, </span>
60            <span class="author-block">
61              <a href="https://ranjaykrishna.com/index.html">Ranjay Krishna</a><sup>1,3</sup>, </span> 
62            <span class="author-block">
63          </div>
64
65          <div class="is-size-5 publication-authors">
66            <span class="author-block"><sup>1</sup>University of Washington &nbsp; </span>
67            <span class="author-block"><sup>2</sup>University of Michigan &nbsp; </span>
68            <span class="author-block"><sup>3</sup>Allen Institute of Artificial Intelligence</span>
69
70          </div>
71          <h1 style="font-size:24px">CVPR 2024</h1>
72
73          <div class="column has-text-centered">
74            <div class="publication-links">
75              <!-- PDF Link. -->
76              <span class="link-block">
77                <a href="https://arxiv.org/pdf/2404.02145"
78                   class="external-link button is-normal is-rounded is-dark">
79                  <span class="icon">
80                      <i class="fas fa-file-pdf"></i>
81                  </span>
82                  <span>Paper</span>
83                </a>
84              </span>
85              <!-- Video Link. -->
86              <span class="link-block">
87                <a href="https://www.youtube.com/watch?v=8UiJNqC14js"
88                   class="external-link button is-normal is-rounded is-dark">
89                  <span class="icon">
90                      <i class="fab fa-youtube"></i>
91                  </span>
92                  <span>Video</span>
93                </a>
94              </span>
95              <!-- Code Link. -->
96              <span class="link-block">
97                <a href="https://github.com/hellomuffin/iterated-learning-for-vlm"
98                   class="external-link button is-normal is-rounded is-dark">
99                  <span class="icon">
100                      <i class="fab fa-github"></i>
101                  </span>
102                  <span>Code</span>
103                  </a>
104              </span>
105            </div>
106
107          </div>
108        </div>
109      </div>
110    </div>
111  </div>
112</section>
113
114<style>
115    .video-grid {
116        display: grid;
117        grid-template-columns: repeat(1, 1fr); /* Three columns */
118        grid-template-rows: repeat(1, 1fr); /* Two rows */
119        gap: 0px 4px; /* Gap between videos */
120        width: 70%; /* Set the container width to 80% */
121        margin: 0 auto; /* Center the container horizontally */
122    }
123
124    .video-grid video {
125        width: 100%; /* Videos fill the container width */
126        height: auto;
127    }
128</style>
129
130<section>
131
132
133  <div class="video-grid">
134      <div>
135        <img src="static/images/overview.png" class="interpolation-image" alt="Interpolate start reference image." />
136      </div>
137  </div>
138  <div class="columns is-centered has-text-centered">
139    <p style="margin-top: 10px; text-align: center;">
140    We design an iterated learning algorithm that improves the compositionality in large vision-language models, inspired by cultural transmission theory in cognitive science.
141  </div>
142</section>
143
144<br>
145
146<section class="section">
147  <div class="container is-max-desktop">
148    <!-- Abstract. -->
149    <div class="columns is-centered has-text-centered">
150      <div class="column is-four-fifths">
151        <h2 class="title is-3">Abstract</h2>
152        <div class="content has-text-justified">
153          <p>
154            A fundamental characteristic common to both human
155            vision and natural language is their compositional nature.
156            Yet, despite the performance gains contributed by large vision and language pretraining, recent investigations find
157            that most—if not all—our state-of-the-art vision-language
158            models struggle at compositionality. They are unable to distinguish between images of “a girl in white facing a man in

159            black” and “a girl in black facing a man in white”. Moreover, prior work suggests that compositionality doesn't arise
160            with scale: larger model sizes or training data don't help.
161            This paper develops a new iterated training algorithm that incentivizes compositionality. We draw on decades of cognitive
162            science research that identifies cultural transmission—the
163            need to teach a new generation—as a necessary inductive
164            prior that incentivizes humans to develop compositional languages. Specifically, we reframe vision-language contrastive
165            learning as the Lewis Signaling Game between a vision agent
166            and a language agent, and operationalize cultural transmission by iteratively resetting one of the agent's weights
167            during training. After every iteration, this training paradigm
168            induces representations that become “easier to learn”, a
169            property of compositional languages: e.g. our model trained
170            on CC3M and CC12M improves standard CLIP by 4.7%,
171            4.0% respectfully in the SugarCrepe benchmark.
172          </p>
173        </div>
174      </div>
175    </div>
176    <br>
177    <!-- Paper video. -->
178    <div class="columns is-centered has-text-centered">
179      <div class="column is-four-fifths">
180        <h2 class="title is-3">Video</h2>
181        <div class="publication-video">
182          <a id="overview_video"></a>
183          <iframe
184            src="https://www.youtube.com/embed/8UiJNqC14js">
185          </iframe>
186        </div>
187      </div>
188    </div>
189  </div>
190    <!--/ Paper video. -->
191</section>
192
193<br>
194
195<style>
196    .video-grid-two-cols {
197        display: grid;
198        grid-template-columns: repeat(2, 1fr); /* Two columns */
199        gap: 10px; /* Gap between videos */
200        width: 60%; /* Set the container width to 60% */
201        margin: 0 auto; /* Center the container horizontally */
202    }
203
204    .video-grid-two-cols video {
205        width: 100%; /* Videos fill the container width */
206        height: auto;
207    }
208</style>
209
210
211<section>
212    <div class="columns is-centered has-text-centered">
213      <h2 class="title is-3">Why Iterated Learning works?</h2>
214       <a id="interactive_demo"></a>
215    </div>
216    <div class="container is-max-desktop">
217      <div class="columns is-centered has-text-centered">
218        <div class="column is-four-fifths">
219          <div class="content has-text-justified">
220            <p> Why our iterated learning algorithm works? We demonstrate it from both cognitive science point of view and machine learning point of view. 
221            </p>
222
223            <p>
224              From cognitive science point of view, we demonstrate that iterated learning produces "easy-to-learn" visual representation.  we train language encoders from scratch to align with the fixed visual encoder checkpoints (lets the vision agent be a teacher to teach language agent).  The language agents achieved significantly higher matching speed when paired with vision representations developed through iterated learning.
225            </p>
226
227            <p>
228              From a machine learning perspective, we find that iterated learning implicitly performs smoothness regularization.  As shown in the figure, the estimated upper bound of Lipschitz constant decreases as generation increases in iterated learning setting and is much smaller than the model trained with the standard scheme. 
229            </p>
230        </div>
231      </div>
232    </div>
233  </div>
234</section>
235<section class="hero">
236  <div class="hero-body">
237    <div class="">
238      <div class="columns is-centered">
239        <div class="column is-four-fifths">
240          <div class="columns is-centered" style="position: relative;">
241            <div class="column is-two-thirds image-board">
242              <img style="width: 100%;" class="clickable-image" src="static/images/cognitive_science.png" alt="Image">
243                <div id="clear-btn" class="button centered"> Plot of in-batch image-text matching accuracy for visual agents "teaches" new language agents
244                </div>
245   
246            </div>
247            <div class="column is-one-thirds image-board move">
248            <img style="width: 100%;" class="clickable-image" src="static/images/machine_learning.png" alt="Image">
249              <div id="clear-btn" class="button centered"> Estimated upper bound of Lipschitz constant</div>
250          </div>
251          </div>
252        </div>
253        </div>
254      </div>
255    </div>
256  </div>
257</section>
258
259<br>
260<br>
261<br>
262
263
264<section>
265  <div class="columns is-centered has-text-centered">
266    <h2 class="title is-3"> Codebook Interpretability</h2>
267  </div>
268    <div class="container is-max-desktop">
269      <div class="columns is-centered has-text-centered">
270      <div class="column is-four-fifths">
271        <div class="content has-text-justified">
272              <p>We visualize the learned codebook by retrieving the top 5 most relevant images for each code. We find that the codes correspond to different (somewhat) interpretable semantic concepts. After mapping the codes manually, we reverse the process and interpret which codes are selected when viewing a new image.  For example, both the “horse” and “tent” codes are assigned a higher weight when viewing an image that contains both, indicating the model’s compositional understanding.  </p>
273        </div>
274      </div>
275  </div>
276</div>
277  <br>
278  <div class="video-grid">
279    <div>
280      <img src="static/images/codebook.png" class="interpolation-image" alt="Interpolate start reference image." />
281    </div>
282  </div>
283</section>
284
285<br>
286
287
288<section class="section" id="BibTeX">

289  <div class="container is-max-desktop content">
290    <h2 class="title is-3">BibTeX</h2>
291    <pre><code>
292@inproceedings{zheng2024iterated,
293  title={Iterated learning improves compositionality in large vision-language models},
294  author={Zheng, Chenhao and Zhang, Jieyu and Kembhavi, Aniruddha and Krishna, Ranjay},
295  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
296  pages={13785--13795},
297  year={2024}
298}
299</code></pre>
300  </div>
301</section>
302
303
304
305<footer class="footer">
306  <div align="center" class="container">
307    <div class="columns is-centered">
308        <div class="content">
309            This website is borrowed from <a href="https://github.com/nerfies/nerfies.github.io">nerfies</a>.
310        </div>
311      </div>
312    </div>
313</footer>
314
315
316</body>
317</html>