1<!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
vendor: 64 bytes, line 1
1 <script async src="https://www.googletagmanager.com/gtag/js?id=
1G-LSGEXQZ74W
vendor: 12 bytes, line 1
1"></script>
1<script> 2
vendor: 134 bytes, lines 2-6
2window.dataLayer = window.dataLayer || []; 3 function gtag(){dataLayer.push(arguments);} 4 gtag('js', new Date()); 5 6 gtag('config', '
6G-LSGEXQZ74W
vendor: 4 bytes, line 6
6');
7</script>
7 <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title> Keisuke Sakaguchi | publications </title> <meta name="description" content="Keisuke Sakaguchi, Associate Professor at Tohoku University. "> <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css" rel="stylesheet" integrity="sha512-MoRNloxbStBcD8z3M/2BmnT+rg4IsMxPkXaGh2zD6LGNNFE80W3onsAhRcMAMrSoyWL9xD7Ert0men7vR8LUZg==" crossorigin="anonymous"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/mdbootstrap/4.19.1/css/mdb.min.css" integrity="sha512-RO38pBRxYH3SoOprtPTD86JFOclM51/XTIdEPh5j8sj4tp8jmQIx26twG52UaLi//hQldfrh7e51WzP9wuP32Q==" crossorigin="anonymous"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.14.0/css/all.min.css" integrity="sha512-1PKOgIY59xJ8Co8+NE6FZ+LOAZKjy+KY8iq0G4B3CyeY6wYHN3yt9PW0XpSriVlkMXe40PTKnXrLnZ9+fkDaog==" crossorigin="anonymous"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/academicons/1.9.0/css/academicons.min.css" integrity="sha512-W4yqoT1+8NLkinBLBZko+dFB2ZbHsYLDdr50VElllRcNt2Q4/GSs6u71UHKxB7S6JEMCp5Ve4xjh3eGQl/HRvg==" crossorigin="anonymous"> <link rel="preconnect" href="https://fonts.googleapis.com"> <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> <link href="https://fonts.googleapis.com/css2?family=Roboto:wght@300&display=swap" rel="stylesheet"> <link rel="stylesheet" href="/assets/css/github.css"> <link rel="icon" href="data:image/svg+xml,<svg%20xmlns=%22http://www.w3.org/2000/svg%22%20viewBox=%220%200%20100%20100%22><text%20y=%22.9em%22%20font-size=%2290%22>%F0%9F%92%A1</text></svg>"> <link rel="stylesheet" href="/assets/css/main.css"> <link rel="canonical" href="/publications/">
7<script async src="https://cdn.panelbear.com/analytics.js?site=D0uh9Msel0f"></script>
7
7<script> 8 window.panelbear = window.panelbear || function() { (window.panelbear.q = window.panelbear.q || []).push(arguments); }; 9 panelbear('config', { site: 'D0uh9Msel0f' }); 10</script>
10 </head> <body class="fixed-top-nav "> <header> <nav id="navbar" class="navbar navbar-light navbar-expand-sm fixed-top"> <div class="container"> <button class="navbar-toggler collapsed ml-auto" type="button" data-toggle="collapse" data-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar top-bar"></span> <span class="icon-bar middle-bar"></span> <span class="icon-bar bottom-bar"></span> </button> <div class="collapse navbar-collapse text-right" id="navbarNav"> <ul class="navbar-nav ml-auto flex-nowrap"> <li class="nav-item "> <a class="nav-link" href="/"> about </a> </li> <li class="nav-item active"> <a class="nav-link" href="/publications/"> publications </a> </li> <li class="nav-item "> <a class="nav-link" href="/cv/"> cv </a> </li> </ul> </div> </div> </nav> </header> <div class="container mt-5"> <div class="post"> <header class="post-header"> <h1 class="post-title">publications</h1> <p class="post-description">See also <a href="https://scholar.google.com/citations?user=6CRBF-MAAAAJ&hl=en" rel="external nofollow noopener" target="_blank">Google Scholar</a>, <a href="https://www.semanticscholar.org/author/Keisuke-Sakaguchi/2325708" rel="external nofollow noopener" target="_blank">Semantic Scholar</a>.</p> </header> <article> <div class="publications"> <h6 class="year">2026</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">IROS</a></abbr> </div> <div id="sakaguchi2026" class="col-sm-10"> <div class="title">GRACE: A Tunable Metric Unifying Accuracy and Efficiency for Robotics Evaluation</div> <div class="author"> <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)</em> Sep 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_iros_grace.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Robotics evaluation metrics generally report success rate and speed separately, ignoring the trade-off between efficie
10ncy and accuracy and making cross-benchmark comparisons difficult. To address this, we introduce GRACE (Generalized Robotics Accuracy-Completion Evaluation), a metric that unifies success rate and timing efficiency through a tunable harmonic mean. The β parameter balances accuracy against speed, and the Ï parameter controls deadline strictness. We illustrate GRACE on published benchmark results through source-audited case studies on Habitat ObjectNav, an RLBench generalization sub-benchmark, and CALVIN ABCâD. For Habitat, where wall-clock timing is not standardized, we instantiate GRACE with an explicitly declared path-efficiency proxy derived from published SR and SPL values. For RLBench and CALVIN, we use timing values reported in the same source as the success results: RLBench average task-completion time and CALVIN same-hardware per-action inference time. The results show that GRACE surfaces accuracy/efficiency trade-offs and regime-dependent ranking reversals that success-only reporting obscures, while providing more interpretable parameterization than baseline approaches. We release GRACE as an open-source toolkit to support consistent accuracy/efficiency evaluation across navigation, manipulation, long-horizon, and other robotics tasks.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi2026</span><span class="p">,</span> 11 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke}</span><span class="p">,</span> 12 <span class="na">title</span> <span class="p">=</span> <span class="s">{{GRACE: A Tunable Metric Unifying Accuracy and Efficiency for Robotics Evaluation}}</span><span class="p">,</span> 13 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}</span><span class="p">,</span> 14 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 15 <span class="na">month</span> <span class="p">=</span> <span class="nv">sep</span> 16<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="ye2026sumi" class="col-sm-10"> <div class="title">Sumi: Open Uniform Diffusion Language Model from Scratch</div> <div class="author"> Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, <em>Keisuke Sakaguchi</em>, Jun Suzuki </div> <div class="periodical"> <em>arXiv e-prints</em> Jun 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_arxiv_sumi.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">ye2026sumi</span><span class="p">,</span> 17 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Ye}, Mengyu and {Kudo}, Keito and {Ikeda}, Wataru and {Matsuda}, Ryosuke and {Sakaguchi}, Keisuke and {Suzuki}, Jun}</span><span class="p">,</span> 18 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Sumi: Open Uniform Diffusion Language Model from Scratch}}</span><span class="p">,</span> 19 <span class="na">
19journal</span> <span class="p">=</span> <span class="s">{arXiv e-prints}</span><span class="p">,</span> 20 <span class="na">keywords</span> <span class="p">=</span> <span class="s">{Computation and Language, Machine Learning}</span><span class="p">,</span> 21 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 22 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 23 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2606.19005}</span> 24<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="sasaki2026unit" class="col-sm-10"> <div class="title">Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics</div> <div class="author"> Mutsumi Sasaki, Go kamoda, Ryosuke Takahashi, Kosuke Sato, Kentaro Inui, <em>Keisuke Sakaguchi</em>, Be
24njamin Heinzerling </div> <div class="periodical"> <em>arXiv</em> Jun 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_arxiv_unit.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Quantities with measurement units, such as 110 cm and 1.2 m, require language models (LMs) to combine a numeral with a symbolic unit scale. Here, we study how LMs compare such quantities in controlled settings spanning several unit systems. We find that accuracy degrades near the comparison boundary, where small changes in value determine the correct answer. The resulting errors are systematic: linear surrogate models predict LM preferences from numerical-difference and unit-scale-difference cues, and causal interventions on subspaces aligned with these variables shift modelâs output. The results suggest that LMs compare quantities through a bag of heuristics over numerals and units, rather than first converting both expressions to an exact shared-scale representation.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">sasaki2026unit</span><span class="p">,</span> 25 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Sasaki}, Mutsumi and {kamoda}, Go and {Takahashi}, Ryosuke and {Sato}, Kosuke and {Inui}, Kentaro and {Sakaguchi}, Keisuke and {Heinzerling}, Benjamin}</span><span class="p">,</span> 26 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics}}</span><span class="p">,</span> 27 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 28 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 29 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 30 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2606.03982}</span> 31<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ICAIL</a></abbr> </div> <div id="choi2026barexam" class="col-sm-10"> <div class="title">Expert Evaluation of LLMâs Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task</div> <div class="author"> Jungmin Choi, <em>Keisuke Sakaguchi</em>, Hiroaki Yamada </div> <div class="periodical"> <em>Proceedings of the 21st International Conference on Artificial Intelligence and Law (ICAIL)</em> Apr 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_icail_barexam.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capacity for generating open-ended legal reasoning in realistic scenarios remains insufficiently explored. Notably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context. This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to identify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribution is the manual evaluation of LLMsâ generated responses by legal experts, which reveals limitations and challenges in legal reasoning. Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law. Our real exam questions, model-generated responses, and expert evaluations reveal the milestones of current LLMs in the Japanese legal domain. Our dataset and relevant resources will be available online.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">
31choi2026barexam</span><span class="p">,</span> 32 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Choi}, Jungmin and {Sakaguchi}, Keisuke and {Yamada}, Hiroaki}</span><span class="p">,</span> 33 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task}}</span><span class="p">,</span> 34 <span class="na">journal</span> <span class="p">=</span> <span class="s">{Proceedings of the 21st International Conference on Artificial Intelligence and Law (ICAIL)}</span><span class="p">,</span> 35 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 36 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 37 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2604.23730}</span> 38<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="fujinuma2026diffusion" class="col-sm-10"> <div class="title">Unlocking Prompt Infilling Capability for Diffusion Language Models</div> <div class="author"> Yoshinari Fujinuma, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>arXiv</em> Apr 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_arxiv_diffusion.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Masked diffusion language models (dLMs) generate text through bidirectional denoising, yet this capability remains locked for infilling prompts. This limitation is an artifact of the current supervised finetuning (SFT) convention of applying response-only masking. To unlock this capability, we extend full-sequence masking during SFT, where both prompts and responses are masked jointly. Once unlocked, the model infills masked portions of a prompt template conditioned on few-shot examples. We show that such model-infilled prompts match or surpass manually designed templates, transfer effectively across models, and are complementary to existing prompt optimization methods. Our results suggest that training practices, not architectural limitations, are the primary bottleneck preventing masked diffusion language models from infilling effective prompts</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">fujinuma2026diffusion</span><span class="p">,</span> 39 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Fujinuma}, Yoshinari and {Sakaguchi}, Keisuke}</span><span class="p">,</span> 40 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Unlocking Prompt Infilling Capability for Diffusion Language Models}}</span><span class="p">,</span> 41 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 42 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 43 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 44 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2604.03677}</span> 45<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="yoshida2026diagram" class="col-sm-10"> <div class="title">Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models</div> <div class="author"> Haruto Yoshida, Keito Kudo, Yoichi Aoki, Ryota Tanaka, Itsumi Saito, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>arXiv</em> Mar 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_arxiv_diagram.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements, particularly those represented by nodes and directed edges (e.g., arrows and lines). To investigate the underlying causes of this limitation, we probe the internal representation of LVLMs using a carefully constructed synthetic diagram dataset based on directed graphs. Our probing experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model. In contrast, node information and global structural features are already linearly encoded in individual hidden states of the vision encoder. These findings suggest that the stage at which linearly separable representations are formed varies depen
45ding on the type of visual information. In particular, the delayed emergence of edge representations may help explain why LVLMs struggle with relational understanding, such as interpreting edge directions, which require more abstract, compositionally integrated processes.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">yoshida2026diagram</span><span class="p">,</span> 46 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Yoshida}, Haruto and {Kudo}, Keito and {Aoki}, Yoichi and {Tanaka}, Ryota and {Saito}, Itsumi and {Sakaguchi}, Keisuke and {Inui}, Kentaro}</span><span class="p">,</span> 47 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models}}</span><span class="p">,</span> 48 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 49 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 50 <span class="na">month</span> <span class="p">=</span> <span class="nv">mar</span><span class="p">,</span> 51 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2603.02865}</span> 52<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EACL Findings</a></abbr> </div> <div id="kudo2026faithful" class="col-sm-10"> <div class="title">LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics</div> <div class="author"> Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Findings of the Association for Computational Linguistics: EACL 2026</em> Mar 2026 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2026_eacl_faithful.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This study investigates the internal information flow of large language models (LLMs) while performing chain-of-thought (CoT) style reasoning.Specifically, with a particular interest in the faithfulness of the CoT explanation to LLMsâ final answer, we explore (i) when the LLMsâ answer is (pre)determined, especially before the CoT begins or after, and (ii) how strongly the information from CoT specifically has a causal effect on the final answer.Our experiments with controlled arithmetic tasks reveal a systematic internal reasoning mechanism of LLMs.They have not derived an answer at the moment when input was fed into the model.Instead, they compute (sub-)answers while generating the reasoning chain on the fly.Therefore, the generated reasoning chains can be regarded as faithful reflections of the modelâs internal computation.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kudo2026faithful</span><span class="p">,</span> 53 <span class="na">title</span> <span class="p">=</span> <span class="s">{{LLM}s Faithfully and Iteratively Compute Answers During {C}o{T}: A Systematic Analysis With Multi-step Arithmetics}</span><span class="p">,</span> 54 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kudo, Keito and Aoki, Yoichi and Kuribayashi, Tatsuki and Sone, Shusaku and Taniguchi, Masaya and Brassard, Ana and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 55 <span class="na">editor</span> <span class="p">=</span> <span class="s">{Demberg, Vera and Inui, Kentaro and Marquez, Llu{\'i}s}</span><span class="p">,</span> 56 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026}</span><span class="p">,</span> 57 <span class="na">month</span> <span class="p">=</span> <span class="nv">mar</span><span class="p">,</span> 58 <span class="na">year</span> <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span> 59 <span class="na">address</span> <span class="p">=</span> <span class="s">{Rabat, Morocco}</span><span class="p">,</span> 60 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 61 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2026.findings-eacl.59/}</span><span class="p">,</span> 62 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2026.findings-eacl.59}</span><span class="p">,</span> 63 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1114--1153}</span><span class="p">,</span> 64 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-386-9}</span> 65<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2025</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">IJCNLP</a></abbr> </div> <div id="sasaki2025calendar" class="col-sm-10"> <div class="title">Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki</div> <div class="author"> Mutsumi Sasaki, Go Kamoda, Ryosuke Takahashi, Kosuke Sato, Kentaro Inui, <em>Keisuke Sakaguchi</em>, Benjamin Heinzerling </div> <div class="periodical"> <em>Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics</em> Dec 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_ijcnlp_calendar.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Temporal reasoning and knowledge are essential capabilities for language models (LMs).While much prior work has analyzed and improved temporal reasoning in LMs, most studies have focused solely on the Gregorian calendar.However, many non-Gregorian systems, such as the Japanese, Hijri, and Hebrew calendars, are in active use and reflect culturally grounded conceptions of time.If and how well current LMs can accurately handle such non-Gregorian calendars has not been evaluated so far.Here, we present a systematic evaluation of how well language models handle one such n
65on-Gregorian system: the Japanese *wareki*.We create datasets that require temporal knowledge and reasoning in using *wareki* dates. Evaluating open and closed LMs, we find that some models can perform calendar conversions, but GPT-4o, Deepseek V3, and even Japanese-centric models struggle with Japanese calendar arithmetic and knowledge involving *wareki* dates.Error analysis suggests corpus frequency of Japanese calendar expressions and a Gregorian bias in the modelâs knowledge as possible explanations.Our results show the importance of developing LMs that are better equipped for culture-specific tasks such as calendar understanding.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sasaki2025calendar</span><span class="p">,</span> 66 <span class="na">title</span> <span class="p">=</span> <span class="s">{Can Language Models Handle a Non-Gregorian Calendar? The Case of the {J}apanese wareki}</span><span class="p">,</span> 67 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sasaki, Mutsumi and Kamoda, Go and Takahashi, Ryosuke and Sato, Kosuke and Inui, Kentaro and Sakaguchi, Keisuke and Heinzerling, Benjamin}</span><span class="p">,</span> 68 <span class="na">editor</span> <span class="p">=</span> <span class="s">{Inui, Kentaro and Sakti, Sakriani and Wang, Haofen and Wong, Derek F. and Bhattacharyya, Pushpak and Banerjee, Biplab and Ekbal, Asif and Chakraborty, Tanmoy and Singh, Dhirendra Pratap}</span><span class="p">,</span> 69 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics}</span><span class="p">,</span> 70 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span><span class="p">,</span> 71 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 72 <span class="na">address</span> <span class="p">=</span> <span class="s">{Mumbai, India}</span><span class="p">,</span> 73 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{The Asian Federation of Natural Language Processing and The Association for Computational Linguistics}</span><span class="p">,</span> 74 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2025.ijcnlp-short.36/}</span><span class="p">,</span> 75 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2025.ijcnlp-short.36}</span><span class="p">,</span> 76 <span class="na">pages</span> <span class="p">=</span> <span class="s">{444--463}</span><span class="p">,</span> 77 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-299-2}</span> 78<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="tarek2025camellia" class="col-sm-10"> <div class="title">Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages</div> <div class="author"> Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Trung Thanh Tran, Zohaib Khan, Alan Ritter, JinYeong Bak, <em>Keisuke Sakaguchi</em>, Tanmoy Chakraborty, Yuki Arase, Wei Xu </div> <div class="periodical"> <em>arXiv</em> Oct 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_arxiv_camellia.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>As Large Language Models (LLMs) gain stronger multilingual capabilities, their ability to handle culturally diverse entities becomes crucial. Prior work has shown that LLMs often favor Western-associated entities in Arabic, raising concerns about cultural fairness. Due to the lack of multilingual benchmarks, it remains unclear if such biases also manifest in different non-Western languages. In this paper, we introduce Camellia, a benchmark for measuring entity-centric cultural biases in nine Asian languages spanning six distinct Asian cultures. Camellia includes 19,530 entities manually annotated for association with the specific Asian or Western culture, as well as 2,173 naturally occurring masked contexts for entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLM families across various tasks such as cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show a struggle by LLMs at cultural adaptation in all Asian languages, with performance differing across models developed in regions with varying access to culturally-relevant data. We further observe that different LLM families hold their distinct biases, differing in how they associate cultures with particular sentiments. Lastly, we find that LLMs struggle with context understanding in Asian languages, creating performance gaps between cultures in entity extraction.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">tarek2025camellia</span><span class="p">,</span> 79 <span class="na">author</span> <span class="p">=</span> <span class="s">
79{{Naous}, Tarek and {Savit}, Anagha and {Catalan}, Carlos Rafael and {Guo}, Geyang and {Lee}, Jaehyeok and {Lee}, Kyungdon and {Dizon}, Lheane Marie and {Ye}, Mengyu and {Kothari}, Neel and {Singh}, Sahajpreet and {Masud}, Sarah and {Patwa}, Tanish and {Thanh Tran}, Trung and {Khan}, Zohaib and {Ritter}, Alan and {Bak}, JinYeong and {Sakaguchi}, Keisuke and {Chakraborty}, Tanmoy and {Arase}, Yuki and {Xu}, Wei}</span><span class="p">,</span> 80 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages}}</span><span class="p">,</span> 81 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 82 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 83 <span class="na">month</span> <span class="p">=</span> <span class="nv">oct</span><span class="p">,</span> 84 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2510.05291}</span> 85<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">Blackbox NLP</a></abbr> </div> <div id="takahashi2025understanding" class="col-sm-10"> <div class="title">Understanding the Side Effects of Rank-One Knowledge Editing</div> <div class="author"> Ryosuke Takahashi, Go Kamoda, Benjamin Heinzerling, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP</em> Nov 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_blackboxnlp_sideeffects.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This study conducts a detailed analysis of the side effects of rank-one knowledge editing using language models with controlled knowledge. The analysis focuses on each element of knowledge triples (subject, relation, object) and examines two aspects: âknowledge that causes large side effects when editedâ and âknowledge that is affected by the side effects.â Our findings suggest that editing knowledge with subjects that have relationships with numerous objects or are robustly embedded within the LM may trigger extensive side effects. Furthermore, we demonstrate that the similarity between relation vectors, the density of object vectors, and the distortion of knowledge representations are closely related to how susceptible knowledge is to editing influences. The findings of this research provide new insights into the mechanisms of side effects in LM knowledge editing and indicate specific directions for developing more effective and reliable knowledge editing methods.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">takahashi2025understanding</span><span class="p">,</span> 86 <span class="na">title</span> <span class="p">=</span> <span class="s">{Understanding the Side Effects of Rank-One Knowledge Editing}</span><span class="p">,</span> 87 <span class="na">author</span> <span class="p">=</span> <span class="s">{Takahashi, Ryosuke and Kamoda, Go and Heinzerling, Benjamin and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 88 <span class="na">editor</span> <span class="p">=</span> <span class="s">{Belinkov, Yonatan and Mueller, Aaron and Kim, Najoung and Mohebbi, Hosein and Chen, Hanjie and Arad, Dana and Sarti, Gabriele}</span><span class="p">,</span> 89 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP}</span><span class="p">,</span> 90 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 91 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 92 <span class="na">address</span> <span class="p">=</span> <span class="s">{Suzhou, China}</span><span class="p">,</span> 93 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 94 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2025.blackboxnlp-1.11}</span><span class="p">,</span> 95 <span class="na">pages</span> <span class="p">=</span> <span class="s">{189--205}</span><span class="p">,</span> 96 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-346-3}</span> 97<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="galvansosa2025rubriks" class="col-sm-10"> <div class="title">Rubrikâs Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset</div> <div class="author"> Diana Galvan-Sosa, Gabrielle Gaudeau, Pride Kavumba, Yunmeng Li, Hongyi Gu, Zheng Yuan, <em>Keisuke Sakaguchi</em>, Paula Buttery </div> <div class="periodical"> <em>Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em> Jul 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_acl_rubriks.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>The performance and usability of Large-Language Models (LLMs) are driving their use in explanation generation tasks. However, despite their widespread adoption, LLM explanations have been found to be unreliable, making it difficult for users to distinguish good from bad explanations. To address this issue, we present Rubrikâs CUBEâan education-ins
97pired rubric and a dataset of 26k explanations, written and later quality-annotated using the rubric by both humans and six open- and closed-source LLMs. The CUBE dataset focuses on two reasoning and two language tasks, providing the necessary diversity for us to effectively test our proposed rubric. Using Rubrik, we find that explanations are influenced by both task and perceived difficulty. Low quality stems primarily from a lack of conciseness in LLM-generated explanations, rather than cohesion and word choice. The full dataset, rubric, and code are available at https://github.com/RubriksCube/rubriks_cube.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">galvansosa2025rubriks</span><span class="p">,</span> 98 <span class="na">title</span> <span class="p">=</span> <span class="s">{Rubrik{'}s Cube: Testing a New Rubric for Evaluating Explanations on the {CUBE} dataset}</span><span class="p">,</span> 99 <span class="na">author</span> <span class="p">=</span> <span class="s">{Galvan-Sosa, Diana and Gaudeau, Gabrielle and Kavumba, Pride and Li, Yunmeng and Gu, Hongyi and Yuan, Zheng and Sakaguchi, Keisuke and Buttery, Paula}</span><span class="p">,</span> 100 <span class="na">editor</span> <span class="p">=</span> <span class="s">{Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher}</span><span class="p">,</span> 101 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}</span><span class="p">,</span> 102 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 103 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 104 <span class="na">address</span> <span class="p">=</span> <span class="s">{Vienna, Austria}</span><span class="p">,</span> 105 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 106 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2025.acl-long.1160/}</span><span class="p">,</span> 107 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2025.acl-long.1160}</span><span class="p">,</span> 108 <span class="na">pages</span> <span class="p">=</span> <span class="s">{23800--23839}</span><span class="p">,</span> 109 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-251-0}</span> 110<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">AIED</a></abbr> </div> <div id="coyne2025feedback" class="col-sm-10"> <div class="title">Annotating Errors in English Learnersâ Written Language Production: Advancing Automated Written Feedback Systems</div> <div class="author"> Steven Coyne, Diana Galvan-Sosa, Ryan Spring, Camélia Guerraoui, Michael Zock, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Artificial Intelligence in Education (AIED)</em> Jul 2025 </div> <div class="award"> <img class="emoji" title=":trophy:" alt=":trophy:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f3c6.png" height="20" width="20"> <a href="https://aied2025.itd.cnr.it/index.php/program/main-track-best-papers-nominees/" rel="external nofollow noopener" target="_blank">Best Paper Nomination </a> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_aied_feedback.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/coynestevencharles/annotating-errors-wcf" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Recent advances in natural language processing (NLP) have contributed to the development of automated writing evaluation (AWE) systems that can correct grammatical errors. However, while these systems are effective at improving text, they are not optimally designed for language learning. They favor direct revisions, often with a click-to-fix functionality that can be applied without considering the reason for the correction. Meanwhile, depending on the error type, learners may benefit most from simple explanations and strategically indirect hints, especially on generalizable grammatical rules. To support the generation of such feedback, we introduce an annotation framework that models each errorâs error type and generalizability. For error type classification, we introduce a typology focused on inferring learnersâ knowledge gaps by connecting their errors to specific grammatical patterns. We collect a dataset of annotated learner errors and corresponding human-written feedback comments, each labeled as a direct correction or hint. With this data, we evaluate keyword-guided, keyword-free, and template-guided methods of generating feedback using large language models (LLMs). Human teachers examined each systemâs outputs, assessing them on grounds including relevance, factuality, and comprehensibility. We report on the development of the dataset and the performance of the systems investigated.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">coyne2025feedback</span><span class="p">,</span> 111 <span class="na">author</span> <span class="p">=</span> <span class="s">{Coyne, Steven and Galvan-Sosa, Diana and Spring, Ryan and Guerraoui, Cam{\'e}lia and Zock, Michael and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 112 <span class="na">
112editor</span> <span class="p">=</span> <span class="s">{Cristea, Alexandra I. and Walker, Erin and Lu, Yu and Santos, Olga C. and Isotani, Seiji}</span><span class="p">,</span> 113 <span class="na">title</span> <span class="p">=</span> <span class="s">{Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems}</span><span class="p">,</span> 114 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Artificial Intelligence in Education (AIED)}</span><span class="p">,</span> 115 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 116 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Springer Nature Switzerland}</span><span class="p">,</span> 117 <span class="na">pages</span> <span class="p">=</span> <span class="s">{292--306}</span><span class="p">,</span> 118 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{978-3-031-98459-4}</span><span class="p">,</span> 119 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span> 120<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ICLR</a></abbr> </div> <div id="saito2025sketchdiagram" class="col-sm-10"> <div class="title">Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches</div> <div class="author"> Itsumi Saito, Haruto Yoshida, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>The Thirteenth International Conference on Learning Representations</em> Apr 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_iclr_sketikz.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://sketikz.github.io/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We address the challenge of automatically generating high-quality vector diagrams from hand-drawn sketches. Vector diagrams are essential for communicating complex ideas across various fields, offering flexibility and scalability. While recent research has progressed in generating diagrams from text descriptions, converting hand-drawn sketches into vector diagrams remains largely unexplored due to the lack of suitable datasets. To address this gap, we introduce SketikZ, a dataset comprising 3,231 pairs of hand-drawn sketches and thier corresponding TikZ codes as well as reference diagrams. Our evaluations reveal the limitations of state-of-the-art vision and language models (VLMs), positioning SketikZ as a key benchmark for future research in sketch-to-diagram conversion. Along with SketikZ, we present ImgTikZ, an image-to-TikZ model that integrates a 6.7B parameter code-specialized open-source large language model (LLM) with a pre-trained vision encoder. Despite its relatively compact size, ImgTikZ performs comparably to GPT-4o. This success is driven by using our two data augmentation techniques and a multi-candidate inference strategy. Our findings open promising directions for future research in sketch-to-diagram conversion and broader image-to-code generation tasks. SketikZ is publicly available.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">saito2025sketchdiagram</span><span class="p">,</span> 121 <span class="na">title</span> <span class="p">=</span> <span class="s">{Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches}</span><span class="p">,</span> 122 <span class="na">author</span> <span class="p">=</span> <span class="s">{Saito, Itsumi and Yoshida, Haruto and Sakaguchi, Keisuke}</span><span class="p">,</span> 123 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{The Thirteenth International Conference on Learning Representations}</span><span class="p">,</span> 124 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 125 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://openreview.net/forum?id=KvaDHPhhir}</span><span class="p">,</span> 126 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 127 <span class="na">address</span> <span class="p">=</span> <span class="s">{Singapore}</span> 128<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NAACL Findings</a></abbr> </div> <div id="kamoda2025detok" class="col-sm-10"> <div class="title">Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference</div> <div class="author"> Go Kamoda, Benjamin Heinzerling, Tatsuro Inaba, Keito Kudo, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Findings of the Association for Computational Linguistics: NAACL 2025</em> Apr 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_naacl_detok.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>According to the stages-of-inference hypothesis, early layers of language models map their subword-tokenized input, which does not necessarily correspond to a linguistically meaningful segmentation, to more meaningful representations that form the modelâs âinner vocabularyâ.Prior analysis of this *detokenization* stage has predominantly relied on probing and interventions such as path patching, which involve selecting particular inputs, cho
128osing a subset of components that will be patched, and then observing changes in model behavior.Here, we show that several important aspects of the detokenization stage can be understood purely by analyzing model weights, without performing any model inference steps.Specifically, we introduce an analytical decomposition of first-layer attention in GPT-2.Our decomposition yields interpretable terms that quantify the relative contributions of position-related, token-related, and mixed effects.By focusing on terms in this decomposition, we discover weight-based explanations of attention bias toward close tokens and attention for detokenization.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kamoda2025detok</span><span class="p">,</span> 129 <span class="na">title</span> <span class="p">=</span> <span class="s">{Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference}</span><span class="p">,</span> 130 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kamoda, Go and Heinzerling, Benjamin and Inaba, Tatsuro and Kudo, Keito and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 131 <span class="na">editor</span> <span class="p">=</span> <span class="s">{Chiruzzo, Luis and Ritter, Alan and Wang, Lu}</span><span class="p">,</span> 132 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Findings of the Association for Computational Linguistics: NAACL 2025}</span><span class="p">,</span> 133 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 134 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 135 <span class="na">address</span> <span class="p">=</span> <span class="s">{Albuquerque, New Mexico}</span><span class="p">,</span> 136 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 137 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2025.findings-naacl.355/}</span><span class="p">,</span> 138 <span class="na">pages</span> <span class="p">=</span> <span class="s">{6324--6343}</span><span class="p">,</span> 139 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-195-7}</span> 140<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NAACL</a></abbr> </div> <div id="lee2025crest" class="col-sm-10"> <div class="title">Self-Training Meets Consistency: Improving LLMsâ Reasoning with Consistency-Driven Rationale Evaluation</div> <div class="author"> Jaehyeok Lee, <em>Keisuke Sakaguchi</em>, JinYeong Bak </div> <div class="periodical"> <em>Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)</em> Apr 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_naacl_crest.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Self-training approach for large language models (LLMs) improves reasoning abilities by training the models on their self-generated rationales. Previous approaches have labeled rationales that produce correct answers for a given question as appropriate for training. However, a single measure risks misjudging rationale quality, leading the models to learn flawed reasoning patterns. To address this issue, we propose CREST (Consistency-driven Rationale Evaluation for Self-Training), a self-training framework that further evaluates each rationale through follow-up questions and leverages this evaluation to guide its training. Specifically, we introduce two methods: (1) filtering out rationales that frequently result in incorrect answers on follow-up questions and (2) preference learning based on mixed preferences from rationale evaluation results of both original and follow-up questions. Experiments on three question-answering datasets using open LLMs show that CREST not only improves the logical robustness and correctness of rationales but also improves reasoning abilities compared to previous self-training approaches.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">lee2025crest</span><span class="p">,</span> 141 <span class="na">title</span> <span class="p">=</span> <span class="s">{Self-Training Meets Consistency: Improving {LLM}s' Reasoning with Consistency-Driven Rationale Evaluation}</span><span class="p">,</span> 142 <span class="na">author</span> <span class="p">=</span> <span class="s">{Lee, Jaehyeok and Sakaguchi, Keisuke and Bak, JinYeong}</span><span class="p">,</span> 143 <span class="na">
143editor</span> <span class="p">=</span> <span class="s">{Chiruzzo, Luis and Ritter, Alan and Wang, Lu}</span><span class="p">,</span> 144 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}</span><span class="p">,</span> 145 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 146 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 147 <span class="na">address</span> <span class="p">=</span> <span class="s">{Albuquerque, New Mexico}</span><span class="p">,</span> 148 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 149 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2025.naacl-long.528/}</span><span class="p">,</span> 150 <span class="na">pages</span> <span class="p">=</span> <span class="s">{10519--10539}</span><span class="p">,</span> 151 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-189-6}</span> 152<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NAACL</a></abbr> </div> <div id="sobhani2025categorize" class="col-sm-10"> <div class="title">Language Models can Categorize System Inputs for Performance Analysis</div> <div class="author"> Dominic Sobhani, Ruiqi Zhong, Edison Marrese-Taylor, <em>Keisuke Sakaguchi</em>, Yutaka Matsuo </div> <div class="periodical"> <em>Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)</em> Apr 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_naacl_categorize.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Language model systems are used to process diverse categories of input requests, ranging from improving creative writing to solving programming challenges. It would be useful to know which categories they are good at. However, existing evaluations compare model performance on pre-defined categories, failing to reflect a systemâs performance on finer-grained or novel ones. We propose to automatically search for finer-grained categories based on inputs where a system performs well or poorly, and describe them in natural language. To search for these categories, we propose a large number of candidate category descriptions, e.g. âCommunication Improvementâ, find the subset of inputs that match the category descriptions, and calculate the performance on these categories; then we sort these categories based on their performance, thereby highlighting those that score high or low. As one application, we apply our method to compare LLaMA 3-70B and Claude 3 Opus, which have similar Elo-ratings on Chatbot Arena; our method finds the former is weaker at making text more professional and humorous while better at providing psychological insights, depicting a more nuanced picture of model performance.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sobhani2025categorize</span><span class="p">,</span> 153 <span class="na">title</span> <span class="p">=</span> <span class="s">{Language Models can Categorize System Inputs for Performance Analysis}</span><span class="p">,</span> 154 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sobhani, Dominic and Zhong, Ruiqi and Marrese-Taylor, Edison and Sakaguchi, Keisuke and Matsuo, Yutaka}</span><span class="p">,</span> 155 <span class="na">
155editor</span> <span class="p">=</span> <span class="s">{Chiruzzo, Luis and Ritter, Alan and Wang, Lu}</span><span class="p">,</span> 156 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}</span><span class="p">,</span> 157 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 158 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 159 <span class="na">address</span> <span class="p">=</span> <span class="s">{Albuquerque, New Mexico}</span><span class="p">,</span> 160 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 161 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2025.naacl-long.317/}</span><span class="p">,</span> 162 <span class="na">pages</span> <span class="p">=</span> <span class="s">{6241--6257}</span><span class="p">,</span> 163 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-8-89176-189-6}</span> 164<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">Nature MI</a></abbr> </div> <div id="jiang2025delphi" class="col-sm-10"> <div class="title">Investigating machine moral judgement through the Delphi experiment</div> <div class="author"> Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny T. Liang, Sydney Levine, Jesse Dodge, <em>Keisuke Sakaguchi</em>, Maxwell Forbes, Jack Hessel, Jon Borchardt, Taylor Sorensen, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, Yejin Choi </div> <div class="periodical"> <em>Nature Machine Intelligence</em> 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_naturemi_delphi.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>As our society adopts increasingly powerful artificial intelligence (AI) systems for pervasive use, there are growing concerns about machine moralityâor lack thereof. Millions of users already rely on the outputs of AI systems, such as chatbots, as decision aids. Meanwhile, AI researchers continue to grapple with the challenge of aligning these systems with human morality and values. In response to this challenge, we build and test Delphi, an open-source AI system trained to predict the moral judgements of US participants. The computational framework of Delphi is grounded in the framework proposed by the prominent moral philosopher John Rawls. Our results speak to the promises and limits of teaching machines about human morality. Delphi demonstrates improved generalization capabilities over those exhibited by off-the-shelf neural language models. At the same time, Delphiâs failures also underscore important challenges in this arena. For instance, Delphi has limited cultural awareness and is susceptible to pervasive biases. Despite these shortcomings, we demonstrate several compelling use cases of Delphi, including its incorporation as a component within an ensemble of AI systems. Finally, we computationally demonstrate the potential of Rawlsâs prospect of hybrid approaches for reliable moral reasoning, inspiring future research in computational morality.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">jiang2025delphi</span><span class="p">,</span> 165 <span class="na">author</span> <span class="p">=</span> <span class="s">{Jiang, Liwei and Hwang, Jena D. and Bhagavatula, Chandra and Bras, Ronan Le and Liang, Jenny T. and Levine, Sydney and Dodge, Jesse and Sakaguchi, Keisuke and Forbes, Maxwell and Hessel, Jack and Borchardt, Jon and
165Sorensen, Taylor and Gabriel, Saadia and Tsvetkov, Yulia and Etzioni, Oren and Sap, Maarten and Rini, Regina and Choi, Yejin}</span><span class="p">,</span> 166 <span class="na">date</span> <span class="p">=</span> <span class="s">{2025/01/01}</span><span class="p">,</span> 167 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.1038/s42256-024-00969-6}</span><span class="p">,</span> 168 <span class="na">id</span> <span class="p">=</span> <span class="s">{Jiang2025}</span><span class="p">,</span> 169 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{2522-5839}</span><span class="p">,</span> 170 <span class="na">journal</span> <span class="p">=</span> <span class="s">{Nature Machine Intelligence}</span><span class="p">,</span> 171 <span class="na">number</span> <span class="p">=</span> <span class="s">{1}</span><span class="p">,</span> 172 <span class="na">pages</span> <span class="p">=</span> <span class="s">{145--160}</span><span class="p">,</span> 173 <span class="na">title</span> <span class="p">=</span> <span class="s">{Investigating machine moral judgement through the Delphi experiment}</span><span class="p">,</span> 174 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://doi.org/10.1038/s42256-024-00969-6}</span><span class="p">,</span> 175 <span class="na">volume</span> <span class="p">=</span> <span class="s">{7}</span><span class="p">,</span> 176 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span> 177<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="kobayashi2025finch" class="col-sm-10"> <div class="title">FinchGPT: a Transformer based language model for birdsong analysis</div> <div class="author"> Kosei Kobayashi, Kosuke Matsuzaki, Masaya Taniguchi, <em>Keisuke Sakaguchi</em>, Kentaro Inui, Kentaro Abe </div> <div class="periodical"> <em>arXiv</em> 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_arxiv_finch.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>The long-range dependencies among the tokens, which originate from hierarchical structures, are a defining hallmark of human language. However, whether similar dependencies exist within the sequential vocalization of non-human animals remains a topic of investigation. Transformer architectures, known for their ability to model long-range dependencies among tokens, provide a powerful tool for investigating this phenomenon. In this study, we employed the Transformer architecture to analyze the songs of Bengalese finch (Lonchura striata domestica), which are characterized by their highly variable and complex syllable sequences. To this end, we developed FinchGPT, a Transformer-based model trained on a textualized corpus of birdsongs, which outperformed other architecture models in this domain. Attention weight analysis revealed that FinchGPT effectively captures long-range dependencies within syllables sequences. Furthermore, reverse engineering approaches demonstrated the impact of computational and biological manipulations on its performance: restricting FinchGPTâs attention span and disrupting birdsong syntax through the ablation of specific brain nuclei markedly influenced the modelâs outputs. Our study highlights the transformative potential of large language models (LLMs) in deciphering the complexities of animal vocalizations, offering a novel framework for exploring the structural properties of non-human communication systems while shedding light on the computational distinctions between biological brains and artificial neural networks.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">kobayashi2025finch</span><span class="p">,</span> 178 <span class="na">title</span> <span class="p">=</span> <span class="s">{FinchGPT: a Transformer based language model for birdsong analysis}</span><span class="p">,</span> 179 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kobayashi, Kosei and Matsuzaki, Kosuke and Taniguchi, Masaya and Sakaguchi, Keisuke and Inui, Kentaro and Abe, Kentaro}</span><span class="p">,</span> 180 <span class="na">
180journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 181 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 182 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arxiv.2502.00344}</span> 183<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">COLING</a></abbr> </div> <div id="kamoda2025lfqa" class="col-sm-10"> <div class="title">Quantifying the Influence of Evaluation Aspects on Long-Form Response Assessment</div> <div class="author"> Go Kamoda, Akari Asai, Ana Brassard, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>Proceedings of the 31st International Conference on Computational Linguistics</em> Jan 2025 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2025_coling_lfqa.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/gokamoda/lfqa-weighted-eval" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Evaluating the outputs of large language models (LLMs) on long-form generative tasks remains challenging. While fine-grained, aspect-wise evaluations provide valuable diagnostic information, they are difficult to design exhaustively, and each aspectâs contribution to the overall acceptability of an answer is unclear. In this study, we propose a method to compute an overall quality score as a weighted average of three key aspects: factuality, informative- ness, and formality. This approach achieves stronger correlations with human judgments compared to previous metrics. Our analysis identifies factuality as the most predictive aspect of overall quality. Additionally, we release a dataset of 1.2k long-form QA answers annotated with both absolute judgments and relative preferences in overall and aspect-wise schemes to aid future research in evaluation practices.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kamoda2025lfqa</span><span class="p">,</span> 184 <span class="na">title</span> <span class="p">=</span> <span class="s">{Quantifying the Influence of Evaluation Aspects on Long-Form Response Assessment}</span><span class="p">,</span> 185 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kamoda, Go and Asai, Akari and Brassard, Ana and Sakaguchi, Keisuke}</span><span class="p">,</span> 186 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 31st International Conference on Computational Linguistics}</span><span class="p">,</span> 187 <span class="na">month</span> <span class="p">=</span> <span class="nv">jan</span><span class="p">,</span> 188 <span class="na">year</span> <span class="p">=</span> <span class="s">{2025}</span><span class="p">,</span> 189 <span class="na">address</span> <span class="p">=</span> <span class="s">{Abu Dhabi, UAE}</span><span class="p">,</span> 190 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 191 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2025.coling-main.588/}</span><span class="p">,</span> 192 <span class="na">pages</span> <span class="p">=</span> <span class="s">{8787--8808}</span> 193<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2024</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP</a></abbr> </div> <div id="aoki2024heuristics" class="col-sm-10"> <div class="title">First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning</div> <div class="author"> Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing</em>
193 Nov 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_emnlp_heuristics.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>"Explicit multi-step reasoning, such as chain-of-thought, is widely adopted in the community to explore the better performance of language models (LMs). We report on the systematic strategy that LMs use in this process.Our controlled experiments reveal that LMs rely more heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning when more steps are required to reach an answer. Conversely, their reliance on heuristics decreases as LMs progress closer to the final answer. This suggests that LMs track only a limited number of future steps and dynamically combine heuristic strategies with rational ones in solving tasks involving multi-step reasoning."</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">aoki2024heuristics</span><span class="p">,</span> 194 <span class="na">title</span> <span class="p">=</span> <span class="s">{First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning}</span><span class="p">,</span> 195 <span class="na">author</span> <span class="p">=</span> <span class="s">{Aoki, Yoichi and Kudo, Keito and Kuribayashi, Tatsuki and Sone, Shusaku and Taniguchi, Masaya and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 196 <span class="na">editor</span> <span class="p">=</span> <span class="s">{Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung}</span><span class="p">,</span> 197 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}</span><span class="p">,</span> 198 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 199 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 200 <span class="na">address</span> <span class="p">=</span> <span class="s">{Miami, Florida, USA}</span><span class="p">,</span> 201 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 202 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2024.emnlp-main.789/}</span><span class="p">,</span> 203 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2024.emnlp-main.789}</span><span class="p">,</span> 204 <span class="na">pages</span> <span class="p">=</span> <span class="s">{14255--14271}</span> 205<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">COLM</a></abbr> </div> <div id="brassard2024acorn" class="col-sm-10"> <div class="title">ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation</div> <div class="author"> Ana Brassard, Benjamin Heinzerling, Keito Kudo, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>First Conference on Language Modeling</em> 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_colm_acorn.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Evaluating the quality of free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to evaluate how LLMs rate explanations. We observed that larger models outputted labels that maintained or increased the inter-annotator agreement, suggesting that they are within the expected variance between human raters. However, their correlation with majority-voted human ratings varied across different quality aspects, indicating that they are not a complete replacement. In turn, using LLMs as a supplement to a smaller group of human raters in some cases improved the correlation with the original majority labels. However, the effect was limited to cases where human raters were scarce, and an additional human rater had a more pronounced effect in all cases. Overall, we recommend against using LLMs as a complete replacement for human raters but encourage using them in configurations that end with targeted human involvement.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">brassard2024acorn</span><span class="p">,</span> 206 <span class="na">title</span> <span class="p">=</span> <span class="s">{{ACORN}
206: Aspect-wise Commonsense Reasoning Explanation Evaluation}</span><span class="p">,</span> 207 <span class="na">author</span> <span class="p">=</span> <span class="s">{Brassard, Ana and Heinzerling, Benjamin and Kudo, Keito and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 208 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{First Conference on Language Modeling}</span><span class="p">,</span> 209 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 210 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://openreview.net/forum?id=2oHnsM9M9D}</span> 211<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="kudo2024think" class="col-sm-10"> <div class="title">Think-to-Talk or Talk-to-Think? When LLMs Come Up with an Answer in Multi-Step Reasoning</div> <div class="author"> Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>arXiv</em> Dec 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_arxiv_think.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This study investigates the internal reasoning mechanism of language models during symbolic multi-step reasoning, motivated by the question of whether chain-of-thought (CoT) outputs are faithful to the modelâs internals. Specifically, we inspect when they internally determine their answers, particularly before or after CoT begins, to determine whether models follow a post-hoc "think-to-talk" mode or a step-by-step "talk-to-think" mode of explanation. Through causal probing experiments in controlled arithmetic reasoning tasks, we found systematic internal reasoning patterns across models; for example, simple subproblems are solved before CoT begins, and more complicated multi-hop calculations are performed during CoT.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">kudo2024think</span><span class="p">,</span> 212 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Kudo}, Keito and {Aoki}, Yoichi and {Kuribayashi}, Tatsuki and {Sone}, Shusaku and {Taniguchi}, Masaya and {Brassard}, Ana and {Sakaguchi}, Keisuke and {Inui}, Kentaro}</span><span class="p">,</span> 213 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Think-to-Talk or Talk-to-Think? When LLMs Come Up with an Answer in Multi-Step Reasoning}}</span><span class="p">,</span> 214 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 215 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 216 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span><span class="p">,</span> 217 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2412.01113}</span> 218<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">BEA</a></abbr> </div> <div id="mita2024tetra" class="col-sm-10"> <div class="title">Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond</div> <div class="author"> Masato Mita, <em>Keisuke Sakaguchi</em>, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, Kentaro Inui </div> <div class="periodical"> <em>Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)</em> Jun 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_bea_tetra.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/chemicaltree/tetra" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Natural language processing (NLP) technology has rapidly improved automated grammatical error correction (GEC) tasks, and the GEC community has begun to explore document-level revision. However, there are two major obstacles to going beyond automated \textitsentence-level GEC to NLP-based document-level revision support: (1) there are few public corpora with document-level revisions annotated by professional editors, and (2) it is infeasible to obtain all possible references and evaluate revision quality using such references because there are infinite revision possibilities. To address these challenges, this paper proposes a new document revision corpus, Text Revision of ACL papers (TETRA), in which multiple professional editors have revised academic papers sampled from the ACL anthology. This corpus enables us to focus on document-level and paragraph-level edits, such as edits related to coherence and consistency. Additionally, as a case study using the TETRA corpus, we investigate reference-less and interpretable methods for meta-evaluation to detect quality improvements according to document revisions. We show the uniqueness of TETRA compared with existing document revision corpora and demonstrate that a fine-tuned pre-trained language model can discriminate the quality of documents after revision even when the difference is subtle.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">mita2024tetra</span><span class="p">,</span> 219 <span class="na">title</span> <span class="p">=</span> <span class="s">{Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond}</span><span class="p">,</span> 220 <span class="na">author</span> <span class="p">=</span> <span class="s">{Mita, Masato and Sakaguchi, Keisuke and Hagiwara, Masato and Mizumoto, Tomoya and Suzuki, Jun and Inui, Kentaro}</span><span class="p">,</span> 221 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Ap
221plications (BEA 2024)}</span><span class="p">,</span> 222 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 223 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 224 <span class="na">address</span> <span class="p">=</span> <span class="s">{Mexico City, Mexico}</span><span class="p">,</span> 225 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 226 <span class="na">pages</span> <span class="p">=</span> <span class="s">{251--265}</span> 227<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">SigDial</a></abbr> </div> <div id="nozue2024multimodal" class="col-sm-10"> <div class="title">A Multimodal Dialogue System to Lead Consensus Building with Emotion-Displaying</div> <div class="author"> Shinnosuk Nozue, Yuto Nakano, Shoji Moriya, Tomoki Ariyama, Kazuma Kokuta, Suchun Xie, Kai Sato, Shusaku Sone, Ryohei Kamei, Reina Akama, Yuichiroh Matsubayashi, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue</em> Sep 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_sigdial_multimodal.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>The evolution of large language models has enabled fluent dialogue, increasing interest in the coexistence of humans and avatars. An essential aspect of achieving this coexistence involves developing sophisticated dialogue systems that can influence user behavior. In this background, we propose an effective multimodal dialogue system designed to promote consensus building with humans. Our system employs a slot-filling strategy to guide discussions and attempts to influence users with suggestions through emotional expression and intent conveyance via its avatar. These innovations have resulted in our system achieving the highest performance in a competition evaluating consensus building between humans and dialogue systems. We hope that our research will promote further discussion on the development of dialogue systems that enhance consensus building in human collaboration.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">nozue2024multimodal</span><span class="p">,</span> 228 <span class="na">title</span> <span class="p">=</span> <span class="s">{A Multimodal Dialogue System to Lead Consensus Building with Emotion-Displaying}</span><span class="p">,</span> 229 <span class="na">author</span> <span class="p">=</span> <span class="s">{Nozue, Shinnosuk and Nakano, Yuto and Moriya, Shoji and Ariyama, Tomoki and Kokuta, Kazuma and Xie, Suchun and Sato, Kai and Sone, Shusaku and Kamei, Ryohei and Ak
229ama, Reina and Matsubayashi, Yuichiroh and Sakaguchi, Keisuke}</span><span class="p">,</span> 230 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue}</span><span class="p">,</span> 231 <span class="na">month</span> <span class="p">=</span> <span class="nv">sep</span><span class="p">,</span> 232 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 233 <span class="na">address</span> <span class="p">=</span> <span class="s">{Kyoto, Japan}</span><span class="p">,</span> 234 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 235 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2024.sigdial-1.57}</span><span class="p">,</span> 236 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2024.sigdial-1.57}</span><span class="p">,</span> 237 <span class="na">pages</span> <span class="p">=</span> <span class="s">{669--673}</span> 238<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="kimura2024hijack" class="col-sm-10"> <div class="title">Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection</div> <div class="author"> Subaru Kimura, Ryota Tanaka, Shumpei Miyawaki, Jun Suzuki, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>arXiv</em> Aug 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_arxiv_hijack.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">kimura2024hijack</span><span class="p">,</span> 239 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Kimura}, Subaru and {Tanaka}, Ryota and {Miyawaki}, Shumpei and {Suzuki}, Jun and {Sakaguchi}, Keisuke}</span><span class="p">,</span> 240 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection}}</span><span class="p">,</span> 241 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 242 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 243 <span class="na">month</span> <span class="p">=</span> <span class="nv">aug</span><span class="p">,</span> 244 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2408.03554}</span> 245<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="llmjp2024" class="col-sm-10"> <div class="title">LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs</div> <div class="author"> LLM-jp, :, Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazu
245ki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, Hiroshi Kataoka, Satoru Katsumata, Daisuke Kawahara, Seiya Kawano, Atsushi Keyaki, Keisuke Kiryu, Hirokazu Kiyomaru, Takashi Kodama, Takahiro Kubo, Yohei Kuga, Ryoma Kumon, Shuhei Kurita, Sadao Kurohashi, Conglong Li, Taiki Maekawa, Hiroshi Matsuda, Yusuke Miyao, Kentaro Mizuki, Sakae Mizuki, Yugo Murawaki, Akim Mousterou, Ryo Nakamura, Taishi Nakamura, Kouta Nakayama, Tomoka Nakazato, Takuro Niitsuma, Jiro Nishitoba, Yusuke Oda, Hayato Ogawa, Takumi Okamoto, Naoaki Okazaki, Yohei Oseki, Shintaro Ozaki, Koki Ryu, Rafal Rzepka, <em>Keisuke Sakaguchi</em>, Shota Sasaki, Satoshi Sekine, Kohei Suda, Saku Sugawara, Issa Sugiura, Hiroaki Sugiyama, Hisami Suzuki, Jun Suzuki, Toyotaro Suzumura, Kensuke Tachibana, Yu Takagi, Kyosuke Takami, Koichi Takeda, Masashi Takeshita, Masahiro Tanaka, Kenjiro Taura, Arseny Tolmachev, Nobuhiro Ueda, Zhen Wan, Shuntaro Yada, Sakiko Yahata, Yuya Yamamoto, Yusuke Yamauchi, Hitomi Yanaka, Rio Yokota, Koichiro Yoshino </div> <div class="periodical"> <em>arXiv e-prints</em> Jul 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_arxiv_llmjp.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit this https URL.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">llmjp2024</span><span class="p">,</span> 246 <span class="na">author</span> <span class="p">=</span> <span class="s">{{LLM-jp} and {:} and {Aizawa}, Akiko and {Aramaki}, Eiji and {Chen}, Bowen and {Cheng}, Fei and {Deguchi}, Hiroyuki and {Enomoto}, Rintaro and {Fujii}, Kazuki and {Fukumoto}, Kensuke and {Fukushima}, Takuya and {Han}, Namgi and {Harada}, Yuto and {Hashimoto}, Chikara and {Hiraoka}, Tatsuya and {Hisada}, Shohei and {Hosokawa}, Sosuke and {Jie}, Lu and {Kamata}, Keisuke and {Kanazawa}, Teruhito and {Kanezashi}, Hiroki and {Kataoka}, Hiroshi and {Katsumata}, Satoru and {Kawahara}, Daisuke and {Kawano}, Seiya and {Keyaki}, Atsushi and {Kiryu}, Keisuke and {Kiyomaru}, Hirokazu and {Kodama}, Takashi and {Kubo}, Takahiro and {Kuga}, Yohei and {Kumon}, Ryoma and {Kurita}, Shuhei and {Kurohashi}, Sadao and {Li}, Conglong and {Maekawa}, Taiki and {Matsuda}, Hiroshi and {Miyao}, Yusuke and {Mizuki}, Kentaro and {Mizuki}, Sakae and {Murawaki}, Yugo and {Mousterou}, Akim and {Nakamura}, Ryo and {Nakamura}, Taishi and {Nakayama}, Kouta and {Nakazato}, Tomoka and {Niitsuma}, Takuro and {Nishitoba}, Jiro and {Oda}, Yusuke and {Ogawa}, Hayato and {Okamoto}, Takumi and {Okazaki}, Naoaki and {Oseki}, Yohei and {Ozaki}, Shintaro and {Ryu}, Koki and {Rzepka}, Rafal and {Sakaguchi}, Keisuke and {Sasaki}, Shota and {Sekine}, Satoshi and {Suda}, Kohei and {Sugawara}, Saku and {Sugiura}, Issa and {Sugiyama}, Hiroaki and {Suzuki}, Hisami and {Suzuki}, Jun and {Suzumura}, Toyotaro and {Tachibana}, Kensuke and {Takagi}, Yu and {Takami}, Kyosuke and {Takeda}, Koichi and {Takeshita}, Masashi and {Tanaka}, Masahiro and {Taura}, Kenjiro and {Tolmachev}, Arseny and {Ueda}, Nobuhiro and {Wan}, Zhen and {Yada}, Shuntaro and {Yahata}, Sakiko and {Yamamoto}, Yuya and {Yamauchi}, Yusuke and {Yanaka}, Hitomi and {Yokota}, Rio and {Yoshino}, Koichiro}</span><span class="p">,</span> 247 <span class="na">title</span> <span class="p">=</span> <span class="s">
247{{LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs}}</span><span class="p">,</span> 248 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv e-prints}</span><span class="p">,</span> 249 <span class="na">keywords</span> <span class="p">=</span> <span class="s">{Computer Science - Computation and Language, Computer Science - Artificial Intelligence}</span><span class="p">,</span> 250 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 251 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 252 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2407.03963}</span> 253<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="takahashi2024edit" class="col-sm-10"> <div class="title">The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models</div> <div class="author"> Ryosuke Takahashi, Go Kamoda, Benjamin Heinzerling, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>arXiv</em> Jun 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_arxiv_edit.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Language models (LMs) encode world knowledge in their internal parameters through training. However, LMs may learn personal and confidential information from the training data, leading to privacy concerns such as data leakage. Therefore, research on knowledge deletion from LMs is essential. This study focuses on the knowledge stored in LMs and analyzes the relationship between the side effects of knowledge deletion and the entities related to the knowledge. Our findings reveal that deleting knowledge related to popular entities can have catastrophic side effects. Furthermore, this research is the first to analyze knowledge deletion in models trained on synthetic knowledge graphs, indicating a new direction for controlled experiments.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">takahashi2024edit</span><span class="p">,</span> 254 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Takahashi}, Ryosuke and {Kamoda}, Go and {Heinzerling}, Benjamin and {Sakaguchi}, Keisuke and {Inui}, Kentaro}</span><span class="p">,</span> 255 <span class="na">title</span> <span class="p">=</span> <span class="s">{{The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models}}</span><span class="p">,</span> 256 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 257 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 258 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 259 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2406.06032}</span> 260<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">MORPHON</a></abbr> </div> <div id="matsuzaki2024junimorph" class="col-sm-10"> <div class="title">J-UniMorph: Japanese Morphological Annotation through the Universal Feature Schema</div> <div class="author"> Kosuke Matsuzaki, Masaya Taniguchi, Kentaro Inui, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology</em> Jun 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_sigmorphon_junimorph.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/cl-tohoku/J-UniMorph" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We introduce a Japanese Morphology dataset, J-UniMorph, developed based on the UniMorph feature schema. This dataset addresses the unique and rich verb forms characteristic of the languageâs agglutinative nature. J-UniMorph distinguishes itself from the existing Japanese subset of UniMorph, which is automatically extracted from Wiktionary. On average, the Wiktionary Edition features around 12 inflected forms for each word and is primarily dominated by denominal verbs (i.e., [noun] + suru (do-PRS)). Morphologically, this inflection pattern is same as the verb suru (do). In contrast, J-UniMorph explores a much broader and more frequently used range of verb forms, offering 118 inflected forms for each word on average. It includes honorifics, a range of politeness levels, and other linguistic nuances, emphasizing the distinctive characteristics of the Japanese language. This paper presents detailed statistics and characteristics of J-UniMorph, comparing it with the Wiktionary Edition. We will release J-UniMorph and its interactive visualizer publicly available, aiming to supp
260ort cross-linguistic research and various applications.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">matsuzaki2024junimorph</span><span class="p">,</span> 261 <span class="na">title</span> <span class="p">=</span> <span class="s">{{J}-{U}ni{M}orph: {J}apanese Morphological Annotation through the Universal Feature Schema}</span><span class="p">,</span> 262 <span class="na">author</span> <span class="p">=</span> <span class="s">{Matsuzaki, Kosuke and Taniguchi, Masaya and Inui, Kentaro and Sakaguchi, Keisuke}</span><span class="p">,</span> 263 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology}</span><span class="p">,</span> 264 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 265 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 266 <span class="na">address</span> <span class="p">=</span> <span class="s">{Mexico City, Mexico}</span><span class="p">,</span> 267 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 268 <span class="na">pages</span> <span class="p">=</span> <span class="s">{7--19}</span> 269<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">LREC-COLING</a></abbr> </div> <div id="kasai2024beam" class="col-sm-10"> <div class="title">A Call for Clarity in Beam Search: How It Works and When It Stops</div> <div class="author"> Jungo Kasai, <em>Keisuke Sakaguchi</em>, Ronan Le Bras, Dragomir Radev, Yejin Choi, Noah A. Smith </div> <div class="periodical"> <em>Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation</em> May 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_lrec_beam.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/jungokasai/beam_with_patience" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Text generation with beam search has proven successful in a wide range of applications. We point out that, though largely overlooked in the literature, the commonly-used implementation of beam decoding (e.g., Hugging Face Transformers and fairseq) uses a first come, first served heuristic: it keeps a set of already completed sequences over time steps and stops when the size of this set reaches the beam size. Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search. Empirical results demonstrate that adjusting this patience factor improves decoding performance of strong pretrained models on news text summarization and machine translation over diverse language pairs, with a negligible inference slowdown. Our approach only modifies one line of code and can be thu
269s readily incorporated in any implementation. Further, we find that different versions of beam decoding result in large performance differences in summarization, demonstrating the need for clarity in specifying the beam search implementation in research work. Our code will be available upon publication.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kasai2024beam</span><span class="p">,</span> 270 <span class="na">title</span> <span class="p">=</span> <span class="s">{A Call for Clarity in Beam Search: How It Works and When It Stops}</span><span class="p">,</span> 271 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kasai, Jungo and Sakaguchi, Keisuke and Bras, Ronan Le and Radev, Dragomir and Choi, Yejin and Smith, Noah A.}</span><span class="p">,</span> 272 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{{Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation}}</span><span class="p">,</span> 273 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 274 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span> 275<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ICLR</a></abbr> </div> <div id="brahman2024plasma" class="col-sm-10"> <div class="title">PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning</div> <div class="author"> Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona Jacqueline Arai, Soumya Sanyal, <em>Keisuke Sakaguchi</em>, Xiang Ren, Yejin Choi </div> <div class="periodical"> <em>The Twelfth International Conference on Learning Representations</em> May 2024 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2024_iclr_plasma.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Procedural planning, which entails decomposing a high-level goal into a sequence of temporally ordered steps, is an important yet intricate task for machines. It involves integrating common-sense knowledge to reason about complex and often contextualized situations, e.g. âscheduling a doctorâs appointment without a phoneâ. While current approaches show encouraging results using large language models (LLMs), they are hindered by drawbacks such as costly API calls and reproducibility issues. In this paper, we advocate planning using smaller language models. We present PlaSma, a novel two-pronged approach to endow small language models with procedural knowledge and (constrained) language planning capabilities. More concretely, we develop symbolic procedural knowledge distillation to enhance the commonsense knowledge in small language models and an inference-time algorithm to facilitate more structured and accurate reasoning. In addition, we introduce a new related task, Replanning, that requires a revision of a plan to cope with a constrained situation. In both the planning and replanning settings, we show that orders-of-magnitude smaller models (770M-11B parameters) can compete and often surpass their larger teacher modelsâ capabilities. Finally, we showcase successful application of PlaSma in an embodied environment, VirtualHome.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">brahman2024plasma</span><span class="p">,</span> 276 <span class="na">title</span> <span class="p">=</span> <span class="s">{PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning}</span><span class="p">,</span> 277 <span class="na">author</span> <span class="p">=</span> <span class="s">{Brahman, Faeze and Bhagavatula, Chandra and Pyatkin, Valentina and Hwang, Jena D. and Li, Xiang Lorraine and Arai, Hirona Jacqueline and Sanyal, Soumya and Sakaguchi, Keisuke and Ren, Xiang and Choi, Yejin}</span><span class="p">,</span> 278 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{The Twelfth International Conference on Learning Representations}</span><span class="p">,</span> 279 <span class="na">year</span> <span class="p">=</span> <span class="s">{2024}</span><span class="p">,</span> 280 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://github.com/allenai/PlaSma}</span><span class="p">,</span> 281 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span> 282<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2023</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NeurIPS</a></abbr> </div> <div id="kasai2023realtime" class="col-sm-10"> <div class="title">RealTime QA: Whatâs the Answer Right Now?</div> <div class="author"> Jungo Kasai, <em>Keisuke Sakaguchi</em>, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, Kentaro Inui </div> <div class="periodical"> <em>Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track</em> Dec 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_neurips_realtimeqa.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://realtimeqa.github.io/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We introduce RealTime QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). RealTime QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open domain QA datasets and pursues, instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this preliminary report presents real-time evaluation results over the past month. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that RealTime QA will spur progress in instantaneous applications of question answering and beyond.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kasai2023realtime</span><span class="p">,</span> 283 <span class="na">title</span> <span class="p">=</span> <span class="s">{RealTime {QA}: What's the Answer Right Now?}</span><span class="p">,</span> 284 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kasai, Jungo and Sakaguchi, Keisuke and Takahashi, Yoichi and Bras, Ronan Le and Asai, Akari and Yu, Xinyan Velocity and Radev, Dragomir and Smith, Noah A. and Choi, Yejin and Inui, Kentaro}</span><span class="p">,</span> 285 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track}</span><span class="p">,</span> 286 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 287 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://openreview.net/forum?id=HfKOIPCvsv}</span><span class="p">,</span> 288 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span> 289<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP Findings</a></abbr> </div> <div id="kamoda2023tta" class="col-sm-10"> <div class="title">Test-time Augmentation for Factual Probing</div> <div class="author"> Go Kamoda, Benjamin Heinzerling, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Findings of the Association for Computational Linguistics: EMNLP 2023</em> Dec 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_emnlp_tta.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/gokamoda/TTA4FactualProbing" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Factual probing is a method that uses prompts to test if a language model âknowsâ certain world knowledge facts. A problem in factual probing is that small changes to the prompt c
289an lead to large changes in model output. Previous work aimed to alleviate this problem by optimizing prompts via text mining or fine-tuning. However, such approaches are relation-specific and do not generalize to unseen relation types. Here, we propose to use test-time augmentation (TTA) as a relation-agnostic method for reducing sensitivity to prompt variations by automatically augmenting and ensembling prompts at test time. Experiments show improved model calibration, i.e., with TTA, model confidence better reflects prediction accuracy. Improvements in prediction accuracy are observed for some models, but for other models, TTA leads to degradation. Error analysis identifies the difficulty of producing high-quality prompt variations as the main challenge for TTA.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kamoda2023tta</span><span class="p">,</span> 290 <span class="na">title</span> <span class="p">=</span> <span class="s">{Test-time Augmentation for Factual Probing}</span><span class="p">,</span> 291 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kamoda, Go and Heinzerling, Benjamin and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 292 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Findings of the Association for Computational Linguistics: EMNLP 2023}</span><span class="p">,</span> 293 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span><span class="p">,</span> 294 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 295 <span class="na">address</span> <span class="p">=</span> <span class="s">{Singapore}</span><span class="p">,</span> 296 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 297 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2023.findings-emnlp.236}</span><span class="p">,</span> 298 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2023.findings-emnlp.236}</span><span class="p">,</span> 299 <span class="na">pages</span> <span class="p">=</span> <span class="s">{3650--3661}</span> 300<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="bhagavatula2023i2d2" class="col-sm-10"> <div class="title">I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation</div> <div class="author"> Chandra Bhagavatula, Jena D Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, <em>Keisuke Sakaguchi</em>, Swabha Swayamdipta, Peter West, Yejin Choi </div> <div class="periodical"> <em>Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em> Jul 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_acl_i2d2.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Commonsense capabilities of pre-trained language models dramatically improve with scale, leading many to believe that scale is the only winning recipe. But is it? Here, we investigate an alternative that a priori seems impossible: can smaller language models (e.g., GPT-2) win over models that are orders of magnitude larger and better (e.g., GPT-3), if powered with novel commonsense distillation algorithms?The key intellectual challenge is to design a learning algorithm that achieve a competitive level of commonsense acquisition, without relying on the benefits of scale. In particular, we study generative models of commonsense knowledge, focusing on the task of generating generics, statements of commonsense facts about everyday concepts, e.g., birds can fly.We introduce I2D2, a novel commonsense distillation framework that loosely follows the Symbolic Knowledge Distillation of West et al. but breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the modelâs own enhanced commonsense acquisition capabilities. Empirical results suggest that scale is not the only way, as novel algorithms can be a promising alternative. Moreover, our study leads to a new corpus of generics, Gen-A-tomic, that is the largest and highest quality available to date.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">bhagavatula2023i2d2</span><span class="p">,</span> 301 <span class="na">title</span> <span class="p">=</span> <span class="s">
301{I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation}</span><span class="p">,</span> 302 <span class="na">author</span> <span class="p">=</span> <span class="s">{Bhagavatula, Chandra and Hwang, Jena D and Downey, Doug and Bras, Ronan Le and Lu, Ximing and Sakaguchi, Keisuke and Swayamdipta, Swabha and West, Peter and Choi, Yejin}</span><span class="p">,</span> 303 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 304 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}</span><span class="p">,</span> 305 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 306 <span class="na">address</span> <span class="p">=</span> <span class="s">{Toronto, Canada}</span><span class="p">,</span> 307 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 308 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2023.acl-long.535}</span><span class="p">,</span> 309 <span class="na">pages</span> <span class="p">=</span> <span class="s">{9614--9630}</span> 310<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="behzad2023elqa" class="col-sm-10"> <div class="title">ELQA: A Corpus of Metalinguistic Questions and Answers about English</div> <div class="author"> Shabnam Behzad, <em>Keisuke Sakaguchi</em>, Nathan Schneider, Amir Zeldes </div> <div class="periodical"> <em>Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em> Jul 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_acl_elqa.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/shabnam-b/ELQA" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We present ELQA, a corpus of questions and answers in and about the English language. Collected from two online forums, the >70k questions (from English learners and others) cover wide-ranging topics including grammar, meaning, fluency, and etymology. The answers include descriptions of general properties of English vocabulary and grammar as well as explanations about specific (correct and incorrect) usage examples. Unlike most NLP datasets, this corpus is metalinguisticâit consists of language about language. As such, it can facilitate investigations of the metalinguistic capabilities of NLU models, as well as educational applications in the language learning domain. To study this, we define a free-form question answering task on our dataset and conduct evaluations on multiple LLMs (Large Language Models) to analyze their capacity to generate metalinguistic answers.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">behzad2023elqa</span><span class="p">,</span> 311 <span class="na">title</span> <span class="p">=</span> <span class="s">{ELQA: A Corpus of Metalinguistic Questions and Answers about English}</span><span class="p">,</span> 312 <span class="na">author</span> <span class="p">=</span> <span class="s">{Behzad, Shabnam and Sakaguchi, Keisuke and Schneider, Nathan and Zeldes, Amir}</span><span class="p">,</span> 313 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}}</span><span class="p">,</span> 314 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 315 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 316 <span class="na">address</span> <span class="p">=</span> <span class="s">{Toronto, Canada}</span><span class="p">,</span> 317 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 318 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2023.acl-long.113}</span><span class="p">,</span> 319 <span class="na">pages</span> <span class="p">=</span> <span class="s">{2031--2047}</span> 320<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="kasai2023med" class="col-sm-10"> <div class="title">Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations</div> <div class="author"> Jungo Kasai, Yuhei Kasai, <em>Keisuke Sakaguchi</em>, Yutaro Yamada, Dragomir Radev </div> <div class="periodical"> <em>arXiv</em> 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_arxiv_jmed.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all five years of the exams, highlighting LLMsâ potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">kasai2023med</span><span class="p">,</span> 321 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Kasai}, Jungo and {Kasai}, Yuhei and {Sakaguchi}, Keisuke and {Yamada}, Yutaro and {Radev}, Dragomir}</span><span class="p">,</span> 322 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations}}</span><span class="p">,</span> 323 <span class="na">
323journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 324 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 325 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2303.18027}</span> 326<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="coyne2023gptgec" class="col-sm-10"> <div class="title">An Analysis of GPT-3âs Performance in Grammatical Error Correction</div> <div class="author"> Steven Coyne, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>arXiv</em> 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_arxiv_gptgec.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of NaturalLanguage Processing tasks. However, there isa relative lack of detailed published analysisof their performance on the task of grammatical error correction (GEC). To address this,we perform experiments testing the capabilitiesof a GPT-3.5 model (text-davinci-003)and a GPT-4 model (gpt-4-0314) on major GEC benchmarks. We compare the performance of different prompts in both zero-shotand few-shot settings, analyzing intriguing orproblematic outputs encountered with differentprompt formats. We report the performance ofour best prompt on the BEA-2019 and JFLEGdatasets, finding that the GPT models can perform well in a sentence-level revision setting,with GPT-4 achieving a new high score on theJFLEG benchmark. Through human evaluation experiments, we compare the GPT modelsâcorrections to source, human reference, andbaseline GEC system sentences and observedifferences in editing strategies and how theyare scored by human raters.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">coyne2023gptgec</span><span class="p">,</span> 327 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Coyne}, Steven and {Sakaguchi}, Keisuke}</span><span class="p">,</span> 328 <span class="na">title</span> <span class="p">=</span> <span class="s">{{An Analysis of GPT-3's Performance in Grammatical Error Correction}}</span><span class="p">,</span> 329 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 330 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 331 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2303.14342}</span> 332<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="regan2023causalschema" class="col-sm-10"> <div class="title">Causal schema induction for knowledge discovery</div> <div class="author"> Michael Regan, Jena D. Hwang, <em>Keisuke Sakaguchi</em>, James Pustejovsky </div> <div class="periodical"> <em>arXiv</em> 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_arxiv_causalschema.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Making sense of familiar yet new situations typically involves making generalizations about causal schemas, stories that help humans reason about event sequences. Reasoning about events includes identifying cause and effect relations shared across event instances, a process we refer to as causal schema induction. Statistical schema induction systems may leverage structural knowledge encoded in discourse or the causal graphs associated with event meaning, however resources to study such causal structure are few in number and limited in size. In this work, we investigate how to apply schema induction models to the task of knowledge discovery for enhanced
332search of English-language news texts. To tackle the problem of data scarcity, we present Torquestra, a manually curated dataset of text-graph-schema units integrating temporal, event, and causal structures. We benchmark our dataset on three knowledge discovery tasks, building and evaluating models for each. Results show that systems that harness causal structure are effective at identifying texts sharing similar causal meaning components rather than relying on lexical cues alone. We make our dataset and models available for research purposes.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">regan2023causalschema</span><span class="p">,</span> 333 <span class="na">author</span> <span class="p">=</span> <span class="s">{{Regan}, Michael and {Hwang}, Jena D. and {Sakaguchi}, Keisuke and {Pustejovsky}, James}</span><span class="p">,</span> 334 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Causal schema induction for knowledge discovery}}</span><span class="p">,</span> 335 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 336 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 337 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/arXiv.2303.15381}</span> 338<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EACL</a></abbr> </div> <div id="Kudo2023eacl" class="col-sm-10"> <div class="title">Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?</div> <div class="author"> Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Proceedings of the 2023 Conference of the European Chapter of the Association for Computational Linguistics</em> May 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_eacl_skill_tree.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Compositionality is a pivotal property of symbolic reasoning. However, how well recent neural models capture compositionality remains underexplored in the symbolic reasoning tasks. This study empirically addresses this question by systematically examining recently published pre-trained seq2seq models with a carefully controlled dataset of multi-hop arithmetic symbolic reasoning. We introduce a skill tree on compositionality in arithmetic symbolic reasoning that defines the hierarchical levels of complexity along with three compositionality dimensions: systematicity, productivity, and substitutivity. Our experiments revealed that among the three types of composition, the models struggled most with systematicity, performing poorly even with relatively simple compositions. That difficulty was not resolved even after training the models with intermediate reasoning steps.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">Kudo2023eacl</span><span class="p">,</span> 339 <span class="na">title</span> <span class="p">=</span> <span class="s">{Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?}</span><span class="p">,</span> 340 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kudo, Keito and Aoki, Yoichi and Kuribayashi, Tatsuki and Brassard, Ana and Yoshikawa, Masashi and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 341 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 342 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2023 Conference of the {E}uropean Chapter of the Association for Computational Linguistics}</span><span class="p">,</span> 343 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span><span class="p">,</span> 344 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 345 <span class="na">address</span> <span class="p">=</span> <span class="s">{Dubrovnik, Croatia}</span><span class="p">,</span> 346 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1351--1362}</span><span class="p">,</span> 347 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2023.eacl-main.98}</span> 348<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EACL Findings</a></abbr> </div> <div id="Aoki2023eacl" class="col-sm-10"> <div class="title">Empirical Investigation of Neural Symbolic Reasoning Strategies</div> <div class="author"> Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, <em>Keisuke Sakaguchi</em>, Kentaro Inui </div> <div class="periodical"> <em>Findings of the Association for Computational Linguistics: EACL 2023 </em> May 2023 </div> <div class="award">
348 <img class="emoji" title=":trophy:" alt=":trophy:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f3c6.png" height="20" width="20"> <a href="https://aacl2022-srw.github.io/program" rel="external nofollow noopener" target="_blank">Best Paper Award at AACL-IJCNLP 2022 Student Research Workshop </a> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2023_eacl_strategy.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>Neural reasoning accuracy improves when generating intermediate reasoning steps. However, the source of this improvement is yet unclear.Here, we investigate and factorize the benefit of generating intermediate steps for symbolic reasoning.Specifically, we decompose the reasoning strategy w.r.t. step granularity and chaining strategy. With a purely symbolic numerical reasoning dataset (e.g., A=1, B=3, C=A+3, C?), we found that the choice of reasoning strategies significantly affects the performance, with the gap becoming even larger as the extrapolation length becomes longer.Surprisingly, we also found that certain configurations lead to nearly perfect performance, even in the case of length extrapolation.Our results indicate the importance of further exploring effective strategies for neural reasoning models.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">Aoki2023eacl</span><span class="p">,</span> 349 <span class="na">title</span> <span class="p">=</span> <span class="s">{Empirical Investigation of Neural Symbolic Reasoning Strategies}</span><span class="p">,</span> 350 <span class="na">author</span> <span class="p">=</span> <span class="s">{Aoki, Yoichi and Kudo, Keito and Kuribayashi, Tatsuki and Brassard, Ana and Yoshikawa, Masashi and Sakaguchi, Keisuke and Inui, Kentaro}</span><span class="p">,</span> 351 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 352 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Findings of the Association for Computational Linguistics: EACL 2023 }</span><span class="p">,</span> 353 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2023.findings-eacl.86}</span><span class="p">,</span> 354 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1154--1162}</span><span class="p">,</span> 355 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span><span class="p">,</span> 356 <span class="na">address</span> <span class="p">=</span> <span class="s">{Dubrovnik, Croatia}</span><span class="p">,</span> 357 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span> 358<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"><li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">Jxiv</arxiv> </div> <div id="choi_et_al_2023_j_bar_exam_en" class="col-sm-10"> <div class="title">Evaluating GPT in Japanese Bar Examination: Insights and Limitations</div> <div class="author"> Jungmin Choi, Jungo Kasai, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> Dec 2023 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="https://github.com/keisks/j_bar_exam" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Large-scale language models like ChatGPT have been reported to exceed the accuracy of human experts in a wide range of tasks. Recent research reports that ChatGPT passed the Japanese National Medical Examination, confirming its high performance in Japanese. We evaluated the accuracy of GPT-3, GPT-4, and ChatGPT in the Japanese Bar Examination (the multiple-choice format section), focusing on Constitutional Law,
358Civil Law, and Criminal Law over the past five years. The results revealed that the current correct answer rate for these exams is only 30-40% (compared to the average pass rate of 70%), which is significantly low. This study went beyond just the correct answer rate, dissecting the necessary reasoning and knowledge for the responses, and examining the performance of large-scale language models from each perspective. The findings show that 1) large-scale language models possess extensive knowledge of many statutes, 2) they have a high correct answer rate for questions that require understanding of legal theories but not specific knowledge of law, and 3) they have a low correct answer rate for questions requiring knowledge of case law. The primary reason for their lower performance compared to the American Bar Examination is thought to be a lack of knowledge in Japanese law, especially in case law.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@techreport</span><span class="p">{</span><span class="nl">choi_et_al_2023_j_bar_exam_en</span><span class="p">,</span> 359 <span class="na">author</span> <span class="p">=</span> <span class="s">{Choi, Jungmin and Kasai, Jungo and Sakaguchi, Keisuke}</span><span class="p">,</span> 360 <span class="na">title</span> <span class="p">=</span> <span class="s">{{Evaluating GPT in Japanese Bar Examination: Insights and Limitations}}</span><span class="p">,</span> 361 <span class="na">year</span> <span class="p">=</span> <span class="s">{2023}</span><span class="p">,</span> 362 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span><span class="p">,</span> 363 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Jxiv}</span> 364<span class="p">}</span></code></pre></figure> </div> </div> </div> </li></ol> <h6 class="year">2022</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP</a></abbr> </div> <div id="kasai2022twist" class="col-sm-10"> <div class="title">Twist Decoding: Diverse Generators Guide Each Other</div> <div class="author"> Jungo Kasai, <em>Keisuke Sakaguchi</em>, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, Noah A. Smith </div> <div class="periodical"> <em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em> Dec 2022 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2022_arxiv_twist.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/jungokasai/twist_decoding" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Many language generation models are now available for a wide range of generation tasks, including machine translation and summarization. Combining such diverse models may lead to further progress, but ensembling generation models is challenging during inference: conventional ensembling methods (e.g., shallow fusion) require that the models share vocabulary/tokenization schemes. We introduce Twist decoding, a simple and general text generation algorithm that benefits from diverse models at inference time. Our method does not assume the vocabulary, tokenization or even generation order is shared. Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available. Twist decoding also consistently outperforms the popular reranking heuristic where output candidates from one model are rescored by another. We hope that our work will encourage researchers and practitioners to examine generation models collectively, not just independently, and to seek out models with complementary strengths to the currently available models.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">kasai2022twist</span><span class="p">,</span> 365 <span class="na">title</span> <span class="p">=</span> <span class="s">{Twist Decoding: Diverse Generators Guide Each Other}</span><span class="p">,</span> 366 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kasai, Jungo and Sakaguchi, Keisuke and Bras, Ronan Le and Peng, Hao and Lu, Ximing and Radev, Dragomir and Choi, Yejin and
366Smith, Noah A.}</span><span class="p">,</span> 367 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)}</span><span class="p">,</span> 368 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span><span class="p">,</span> 369 <span class="na">year</span> <span class="p">=</span> <span class="s">{2022}</span><span class="p">,</span> 370 <span class="na">address</span> <span class="p">=</span> <span class="s">{Abu Dhabi, UAE}</span><span class="p">,</span> 371 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 372 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2022.emnlp-main.326}</span><span class="p">,</span> 373 <span class="na">pages</span> <span class="p">=</span> <span class="s">{4909--4923}</span> 374<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="jiang2022delphi" class="col-sm-10"> <div class="title">
374Can Machines Learn Morality? The Delphi Experiment</div> <div class="author"> Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, <em>Keisuke Sakaguchi</em>, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, Yejin Choi </div> <div class="periodical"> <em>arXiv</em> 2022 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2022_arxiv_delphi.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://delphi.allenai.org/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>As AI systems become increasingly powerful and pervasive, there are growing concerns about machinesâ morality or a lack thereof. Yet, teaching morality to machines is a formidable task, as morality remains among the most intensely debated questions in humanity, let alone for AI. Existing AI systems deployed to millions of users, however, are already making decisions loaded with moral implications, which poses a seemingly impossible challenge: teaching machines moral sense, while humanity continues to grapple with it. To explore this challenge, we introduce Delphi, an experimental framework based on deep neural networks trained directly to reason about descriptive ethical judgments, e.g., "helping a friend" is generally good, while "helping a friend spread fake news" is not. Empirical results shed novel insights on the promises and limits of machine ethics; Delphi demonstrates strong generalization capabilities in the face of novel ethical situations, while off-the-shelf neural network models exhibit markedly poor judgment including unjust biases, confirming the need for explicitly teaching machines moral sense. Yet, Delphi is not perfect, exhibiting susceptibility to pervasive biases and inconsistencies. Despite that, we demonstrate positive use cases of imperfect Delphi, including using it as a component model within other imperfect AI systems. Importantly, we interpret the operationalization of Delphi in light of prominent ethical theories, which leads us to important future research questions.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">jiang2022delphi</span><span class="p">,</span> 375 <span class="na">title</span> <span class="p">=</span> <span class="s">
375{Can Machines Learn Morality? The Delphi Experiment}</span><span class="p">,</span> 376 <span class="na">author</span> <span class="p">=</span> <span class="s">{Jiang, Liwei and Hwang, Jena D. and Bhagavatula, Chandra and Bras, Ronan Le and Liang, Jenny and Dodge, Jesse and Sakaguchi, Keisuke and Forbes, Maxwell and Borchardt, Jon and Gabriel, Saadia and Tsvetkov, Yulia and Etzioni, Oren and Sap, Maarten and Rini, Regina and Choi, Yejin}</span><span class="p">,</span> 377 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 378 <span class="na">year</span> <span class="p">=</span> <span class="s">{2022}</span><span class="p">,</span> 379 <span class="na">volume</span> <span class="p">=</span> <span class="s">{abs/2110.07574}</span><span class="p">,</span> 380 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.48550/ARXIV.2110.07574}</span> 381<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NAACL</a></abbr> </div> <div id="Kasai2022BidimensionalLG" class="col-sm-10"> <div class="title">Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand</div> <div class="author"> Jungo Kasai, <em>Keisuke Sakaguchi</em>, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R. Fabbri, Yejin Choi, Noah A. Smith </div> <div class="periodical"> <em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em> Jul 2022 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2022_naacl_billboards.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://nlp.cs.washington.edu/billboard/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Natural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to improve generation models tend to focus on simple n-gram overlap metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics should each more directly benefit and inform the other. We therefore propose a generalization of leaderboards, bidimensional leaderboards (BILLBOARDs), that simultaneously tracks progress in language generation tasks and metrics for their evaluation. Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a BILLBOARD accepts both generators and evaluation metrics as competing entries. A BILLBOARD automatically creates an ensemble metric that selects and linearly combines a few metrics based on a global analysis across generators. Further, metrics are ranked based on their correlations with human judgments. We release four BILLBOARDs for machine translation, summarization, and image captioning. We demonstrate that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation. Our mixed-effects model analysis shows that most automatic metrics, especially the reference-based ones, overrate machine over human generation, demonstrating the importance of updating metrics as generation models become stronger (and perhaps more similar to humans) in the future.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">Kasai2022BidimensionalLG</span><span class="p">,</span> 382 <span class="na">title</span> <span class="p">=</span> <span class="s">{Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand}</span><span class="p">,</span> 383 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kasai, Jungo and Sakaguchi, Keisuke and Bras, Ronan Le and Dunagan, Lavinia and Morrison, Jacob and Fabbri, Alexander R. and Choi, Yejin and
383Smith, Noah A.}</span><span class="p">,</span> 384 <span class="na">year</span> <span class="p">=</span> <span class="s">{2022}</span><span class="p">,</span> 385 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}</span><span class="p">,</span> 386 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 387 <span class="na">pages</span> <span class="p">=</span> <span class="s">{3540--3557}</span><span class="p">,</span> 388 <span class="na">address</span> <span class="p">=</span> <span class="s">{Seattle, United States}</span><span class="p">,</span> 389 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span> 390<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NAACL</a></abbr> </div> <div id="Kasai2022TransparentHE" class="col-sm-10"> <div class="title">Transparent Human Evaluation for Image Captioning</div> <div class="author"> Jungo Kasai, <em>Keisuke Sakaguchi</em>, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, Noah A. Smith </div> <div class="periodical"> <em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em> Jul 2022 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2022_naacl_thumb.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/jungokasai/THumB" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We establish THumB, a rubric-based human evaluation protocol for image captioning models. Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset. Each caption is evaluated along two main dimensions in a tradeoff (precision and recall) as well as other aspects that measure the text quality (fluency, conciseness, and inclusive language). Our evaluations demonstrate several critical problems of the current evaluation practice. Human-generated captions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite. Our rubric-based results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall. We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">Kasai2022TransparentHE</span><span class="p">,</span> 391 <span class="na">title</span> <span class="p">=</span> <span class="s">{Transparent Human Evaluation for Image Captioning}</span><span class="p">,</span> 392 <span class="na">author</span> <span class="p">=</span> <span class="s">{Kasai, Jungo and Sakaguchi, Keisuke and Dunagan, Lavinia and Morrison, Jacob and Bras, Ronan Le and Choi, Yejin and Smith, Noah A.}</span><span class="p">,</span> 393 <span class="na">year</span> <span class="p">=</span> <span class="s">{2022}</span><span class="p">,</span> 394 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}</span><span class="p">,</span> 395 <span class="na">pages</span> <span class="p">=</span> <span class="s">{3464--3478}</span><span class="p">,</span> 396 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 397 <span class="na">address</span> <span class="p">=</span> <span class="s">{Seattle, Washington}</span><span class="p">,</span> 398 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span> 399<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">IMLW@AAAI</a></abbr> </div> <div id="Tandon2021InterscriptAD" class="col-sm-10"> <div class="title">Interscript: A dataset for interactive learning of scripts through error feedback</div> <div class="author"> Niket Tandon, Aman Madaan, Peter Clark, <em>Keisuke Sakaguchi</em>, Yiming Yang </div> <div class="periodical"> <em>The AAAI-22 Workshop on Interactive Machine Learning</em> 2022 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2022_imlw_interscript.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/allenai/interscript" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>How can an end-user provide feedback if a deployed structured prediction model generates inconsistent output, ignoring the structural complexity of human language? This is an emerging topic with recent progress in synthetic or c
399onstrained settings, and the next big leap would require testing and tuning models in real-world settings. We present a new dataset, INTERSCRIPT, containing user feedback on a deployed model that generates complex everyday tasks. INTERSCRIPT contains 8,466 data pointsâ the input is a possibly erroneous script and a user feedback and the output is a modified script. We posit two use-cases of INTERSCRIPT that might significantly advance the state-of-the-art in interactive.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">Tandon2021InterscriptAD</span><span class="p">,</span> 400 <span class="na">title</span> <span class="p">=</span> <span class="s">{Interscript: A dataset for interactive learning of scripts through error feedback}</span><span class="p">,</span> 401 <span class="na">author</span> <span class="p">=</span> <span class="s">{Tandon, Niket and Madaan, Aman and Clark, Peter and Sakaguchi, Keisuke and Yang, Yiming}</span><span class="p">,</span> 402 <span class="na">year</span> <span class="p">=</span> <span class="s">{2022}</span><span class="p">,</span> 403 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{The AAAI-22 Workshop on Interactive Machine Learning}</span> 404<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2021</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="Madaan2021ImprovingNM" class="col-sm-10"> <div class="title">Improving Neural Model Performance through Natural Language Feedback on Their Explanations</div> <div class="author"> Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Yiming Yang, Peter Clark, <em>Keisuke Sakaguchi</em>, Eduard H. Hovy </div> <div class="periodical"> <em>arXiv</em> 2021 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2021_arxiv_mercurie.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>A class of explainable NLP models for reasoning tasks support their decisions by generating free-form or structured explanations, but what happens when these supporting structures contain errors? Our goal is to allow users to interactively correct explanation structures through natural language feedback. We introduce MERCURIEan interactive system that refines its explanations for a given reasoning task by getting human feedback in natural language. Our approach generates graphs that have 40% fewer inconsistencies as compared with the off-the-shelf system. Further, simply appending the corrected explanation structures to the output leads to a gain of 1.2 points on accuracy on defeasible reasoning across all three domains.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">Madaan2021ImprovingNM</span><span class="p">,</span> 405 <span class="na">title</span> <span class="p">=</span> <span class="s">{Improving Neural Model Performance through Natural Language Feedback on Their Explanations}</span><span class="p">,</span> 406 <span class="na">author</span> <span class="p">=</span> <span class="s">{Madaan, Aman and Tandon, Niket and Rajagopal, Dheeraj and Yang, Yiming and Clark, Peter and Sakaguchi, Keisuke and Hovy, Eduard H.}</span><span class="p">,</span> 407 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 408 <span class="na">year</span> <span class="p">=</span> <span class="s">{2021}</span><span class="p">,</span> 409 <span class="na">volume</span> <span class="p">=</span> <span class="s">{abs/2104.08765}</span> 410<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="Hagiwara2021GrammarTaggerAM" class="col-sm-10"> <div class="title">GrammarTagger: A Multilingual, Minimally-Supervised Grammar Profiler for Language Education</div> <div class="author"> Masato Hagiwara, Joshua Tanner, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>arXiv</em> 2021 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2021_arxiv_grammar_tagger.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/octanove/grammartagger" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We present GrammarTagger, an open-source grammar profiler which, given an input text, identifies grammatical features useful for language education. The model architecture enables it to learn from a small amount of texts annotated with spans and their labels, which 1) enables easier and more intuitive annotation, 2) supp
410orts overlapping spans, and 3) is less prone to error propagation, compared to complex hand-crafted rules defined on constituency/dependency parses. We show that we can bootstrap a grammar profiler model with F1 â 0.6 from only a couple hundred sentences both in English and Chinese, which can be further boosted via learning a multilingual model. With GrammarTagger, we also build Octanove Learn, a search engine of language learning materials indexed by their reading difficulty and grammatical features.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">Hagiwara2021GrammarTaggerAM</span><span class="p">,</span> 411 <span class="na">title</span> <span class="p">=</span> <span class="s">{GrammarTagger: A Multilingual, Minimally-Supervised Grammar Profiler for Language Education}</span><span class="p">,</span> 412 <span class="na">author</span> <span class="p">=</span> <span class="s">{Hagiwara, Masato and Tanner, Joshua and Sakaguchi, Keisuke}</span><span class="p">,</span> 413 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 414 <span class="na">year</span> <span class="p">=</span> <span class="s">{2021}</span><span class="p">,</span> 415 <span class="na">volume</span> <span class="p">=</span> <span class="s">{abs/2104.03190}</span> 416<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP Findings</a></abbr> </div> <div id="sakaguchi-etal-2021-proscript-partially" class="col-sm-10"> <div class="title">proScript: Partially Ordered Scripts Generation</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, Yejin Choi </div> <div class="periodical"> <em>Findings of the Association for Computational Linguistics: EMNLP 2021</em> Nov 2021 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2021_emnlp_proscript.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://keisuke-sakaguchi.github.io/proscript" class="btn btn-sm z-depth-0" role="button">Project</a> </div> <div class="abstract hidden"> <p>Scripts â prototypical event sequences describing everyday activities â have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information. However, to date they have proved hard to author or extract from text. In this work, we demonstrate for the first time that pre-trained neural language models can be finetuned to generate high-quality scripts, at varying levels of granularity, for a wide range of everyday scenarios (e.g., bake a cake). To do this, we collect a large (6.4k) crowdsourced partially ordered scripts (named proScript), that is substantially larger than prior datasets, and develop models that generate scripts by combining language generation and graph structure prediction. We define two complementary tasks: (i) edge prediction: given a scenario and unordered events, organize the events into a valid (possibly partial-order) script, and (ii) script generation: given only a scenario, generate events and organize them into a (possibly partial-order) script. Our experiments show that our models perform well (e.g., F1=75.7 on task (i)), illustrating a new approach to overcoming previous barriers to script collection. We also show that there is still significant room for improvement toward human level performance. Together, our tasks, dataset, and models offer a new research direction for learning script knowledge.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2021-proscript-partially</span><span class="p">,</span> 417 <span class="na">title</span> <span class="p">=</span> <span class="s">{pro{S}cript: Partially Ordered Scripts Generation}</span><span class="p">,</span> 418 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Bhagavatula, Chandra and Le Bras, Ronan and Tandon, Niket and Clark, Peter and Choi, Yejin}</span><span class="p">,</span> 419 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Findings of the Association for Computational Linguistics: EMNLP 2021}</span><span class="p">,</span> 420 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 421 <span class="na">year</span> <span class="p">=</span> <span class="s">{2021}</span><span class="p">,</span> 422 <span class="na">address</span> <span class="p">=</span> <span class="s">{Punta Cana, Dominican Republic}</span><span class="p">,</span> 423 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 424 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2021.findings-emnlp.184}</span><span class="p">,</span> 425 <span class="na">pages</span> <span class="p">=</span> <span class="s">{2138--2149}</span> 426<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">CACM</a></abbr> </div> <div id="10.1145/3474381" class="col-sm-10"> <div class="title">WinoGrande: An Adversarial Winograd Schema Challenge at Scale</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi </div> <div class="periodical"> <em>Commun. ACM</em> Aug 2021 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="https://cacm.acm.org/magazines/2021/9/255048-winogrande/fulltext" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">PDF</a> </div> <div class="abstract hidden"> <p>
426Commonsense reasoning remains a major challenge in AI, and yet, recent progresses on benchmarks may seem to suggest otherwise. In particular, the recent neural language models have reported above 90% accuracy on the Winograd Schema Challenge (WSC), a commonsense benchmark originally designed to be unsolvable for statistical models that rely simply on word associations. This raises an important questionâwhether these models have truly acquired robust commonsense capabilities or they rely on spurious biases in the dataset that lead to an overestimation of the true capabilities of machine commonsense.To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) large-scale crowdsourcing, followed by (2) systematic bias reduction using a novel AFLITE algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. Our experiments demonstrate that state-of-the-art models achieve considerably lower accuracy (59.4%-79.1%) on WINOGRANDE compared to humans (94%), confirming that the high performance on the original WSC was inflated by spurious biases in the dataset.Furthermore, we report new state-of-the-art results on five related benchmarks with emphasis on their dual implications. On the one hand, they demonstrate the effectiveness of WINOGRANDE when used as a resource for transfer learning. On the other hand, the high performance on all these benchmarks suggests the extent to which spurious biases are prevalent in all such datasets, which motivates further research on algorithmic bias reduction.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">10.1145/3474381</span><span class="p">,</span> 427 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin}</span><span class="p">,</span> 428 <span class="na">title</span> <span class="p">=</span> <span class="s">{WinoGrande: An Adversarial Winograd Schema Challenge at Scale}</span><span class="p">,</span> 429 <span class="na">year</span> <span class="p">=</span> <span class="s">{2021}</span><span class="p">,</span> 430 <span class="na">issue_date</span> <span class="p">=</span> <span class="s">{September 2021}</span><span class="p">,</span> 431 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computing Machinery}</span><span class="p">,</span> 432 <span class="na">address</span> <span class="p">=</span> <span class="s">{New York, NY, USA}</span><span class="p">,</span> 433 <span class="na">volume</span> <span class="p">=</span> <span class="s">{64}</span><span class="p">,</span> 434 <span class="na">number</span> <span class="p">=</span> <span class="s">{9}</span><span class="p">,</span> 435 <span class="na">issn</span> <span class="p">=</span> <span class="s">{0001-0782}</span><span class="p">,</span> 436 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://doi.org/10.1145/3474381}</span><span class="p">,</span> 437 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.1145/3474381}</span><span class="p">,</span> 438 <span class="na">journal</span> <span class="p">=</span> <span class="s">{Commun. ACM}</span><span class="p">,</span> 439 <span class="na">month</span> <span class="p">=</span> <span class="nv">aug</span><span class="p">,</span> 440 <span class="na">pages</span> <span class="p">=</span> <span class="s">{99--106}</span><span class="p">,</span> 441 <span class="na">numpages</span> <span class="p">=</span> <span class="s">{8}</span> 442<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">AAAI</a></abbr> </div> <div id="Hwang2021COMETATOMIC2O" class="col-sm-10"> <div class="title">COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs</div> <div class="author"> Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, <em>Keisuke Sakaguchi</em>, Antoine Bosselut, Yejin Choi </div> <div class="periodical"> <em>Proceedings of the AAAI Conference on Artificial Intelligence</em> May 2021 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2021_aaai_comet2020.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/allenai/comet-atomic-2020" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Recent years have brought about a renewed interest in commonsense representation and reasoning in the field of natural language understanding. The development of new commonsense knowledge graphs (CSKG) has been central to these advances as their diverse facts can be used and referenced by machine learning models for tackling new and challenging tasks. At the same time, there remain questions about the quality and coverage of these resources due to the massive scale required to comprehensively encompass general commonsense knowledge. In this work, we posit that manually constructed CSKGs will never achieve the coverage necessary to be applicable in all situations encountered by NLP agents. Therefore, we propose a new evaluation framework for testing the utility of KGs based on how effectively implicit knowledge representations can be learned from them. With this new goal, we propose ATOMIC 2020, a new CSKG of general-purpose commonsense knowledge containing knowledge that is not readily available in pretrained language models. We evaluate its properties in comparison with other leading CSKGs, performing the first large-scale pairwise study of commonsense knowledge resources. Next, we show that ATOMIC 2020 is better suited for training knowledge models that can generate accurate, representative knowledge for new, unseen entities and events. Finally, through human evaluation, we show that the few-shot performance of GPT-3 (175B parameters), while impressive, remains  12 absolute points lower than a BART-based knowledge model trained on ATOMIC 2020 despite using over 430x fewer parameters.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">Hwang2021COMETATOMIC2O</span><span class="p">,</span> 443 <span class="na">title</span> <span class="p">=</span> <span class="s">{COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs}</span><span class="p">,</span> 444 <span class="na">
444journal</span> <span class="p">=</span> <span class="s">{Proceedings of the AAAI Conference on Artificial Intelligence}</span><span class="p">,</span> 445 <span class="na">author</span> <span class="p">=</span> <span class="s">{Hwang, Jena D. and Bhagavatula, Chandra and Le Bras, Ronan and Da, Jeff and Sakaguchi, Keisuke and Bosselut, Antoine and Choi, Yejin}</span><span class="p">,</span> 446 <span class="na">volume</span> <span class="p">=</span> <span class="s">{35}</span><span class="p">,</span> 447 <span class="na">number</span> <span class="p">=</span> <span class="s">{7}</span><span class="p">,</span> 448 <span class="na">year</span> <span class="p">=</span> <span class="s">{2021}</span><span class="p">,</span> 449 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span><span class="p">,</span> 450 <span class="na">pages</span> <span class="p">=</span> <span class="s">{6384--6392}</span> 451<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2020</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP</a></abbr> </div> <div id="tandon-etal-2020-dataset" class="col-sm-10"> <div class="title">A Dataset for Tracking Entities in Open Domain Procedural Text</div> <div class="author"> Niket Tandon, <em>Keisuke Sakaguchi</em>, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, Eduard Hovy </div> <div class="periodical"> <em>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em> Nov 2020 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2020_emnlp_openpi.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://allenai.org/data/openpi" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We present the first dataset for tracking state changes in procedural text from arbitrary domains by using an unrestricted (open) vocabulary. For example, in a text describing fog removal using potatoes, a car window may transition between being foggy, sticky, opaque, and clear. Previous formulations of this task provide the text and entities involved, and ask how those entities change for just a small, pre-defined set of attributes (e.g., location), limiting their fidelity. Our solution is a new task formulation where given just a procedural text as input, the task is to generate a set of state change tuples (entity, attribute, before-state, after-state) for each step, where the entity, attribute, and state values must be predicted from an open vocabulary. Using crowdsourcing, we create OPENPI, a high-quality (91.5% coverage as judged by humans and completely vetted), and large-scale dataset comprising 29,928 state changes over 4,050 sentences from 810 procedural real-world paragraphs from WikiHow.com. A current state-of-the-art generation model on this task achieves 16.1% F1 based on BLEU metric, leaving enough room for novel model architectures.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">tandon-etal-2020-dataset</span><span class="p">,</span> 452 <span class="na">title</span> <span class="p">=</span> <span class="s">{A Dataset for Tracking Entities in Open Domain Procedural Text}</span><span class="p">,</span> 453 <span class="na">author</span> <span class="p">=</span> <span class="s">{Tandon, Niket and Sakaguchi, Keisuke and Dalvi, Bhavana and Rajagopal, Dheeraj and Clark, Peter and Guerquin, Michal and Richardson, Kyle and Hovy, Eduard}</span><span class="p">,</span> 454 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}</span><span class="p">,</span> 455 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 456 <span class="na">year</span> <span class="p">=</span> <span class="s">{2020}</span><span class="p">,</span> 457 <span class="na">address</span> <span class="p">=</span> <span class="s">{Online}</span><span class="p">,</span> 458 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 459 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2020.emnlp-main.520}</span><span class="p">,</span> 460 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2020.emnlp-main.520}</span><span class="p">,</span> 461 <span class="na">pages</span> <span class="p">=</span> <span class="s">{6408--6417}</span> 462<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="chen-etal-2020-uncertain" class="col-sm-10"> <div class="title">Uncertain Natural Language Inference</div> <div class="author"> Tongfei Chen, Zhengping Jiang, Adam Poliak, <em>Keisuke Sakaguchi</em>, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</em> Jul 2020 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2020_acl_unli.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://nlp.jhu.edu/unli/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments. We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically-labeled NLI data can be used in pre-training. Our best models correlate well with humans, demonstrating models are capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">chen-etal-2020-uncertain</span><span class="p">,</span> 463 <span class="na">title</span> <span class="p">=</span> <span class="s">{Uncertain Natural Language Inference}</span><span class="p">,</span> 464 <span class="na">author</span> <span class="p">=</span> <span class="s">{Chen, Tongfei and Jiang, Zhengping and Poliak, Adam and Sakaguchi, Keisuke and Van Durme, Benjamin}</span><span class="p">,</span> 465 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}</span><span class="p">,</span> 466 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 467 <span class="na">year</span> <span class="p">=</span> <span class="s">{2020}</span><span class="p">,</span> 468 <span class="na">address</span> <span class="p">=</span> <span class="s">{Online}</span><span class="p">,</span> 469 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 470 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2020.acl-main.774}</span><span class="p">,</span> 471 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/2020.acl-main.774}</span><span class="p">,</span> 472 <span class="na">pages</span> <span class="p">=</span> <span class="s">{8772--8779}</span> 473<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">LREC</a></abbr> </div> <div id="white-etal-2020-universal" class="col-sm-10"> <div class="title">The Universal Decompositional Semantics Dataset and Decomp Toolkit</div> <div class="author"> Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, Dee Ann Reisinger, Tim Vieira, <em>Keisuke Sakaguchi</em>, Sheng Zhang, Francis Ferraro, Rachel Rudinger, Kyle Rawlins, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the 12th Language Resources and Evaluation Conference</em> May 2020 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2020_lrec_decomp.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://decomp.io/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We present the Universal Decompositional Semantics (UDS) dataset (v1.0), which is bundled with the Decomp toolkit (v0.1). UDS1.0 unifies five high-quality, decompositional semantics-aligned annotation sets within a single semantic graph specificationâwith graph structures defined by the predicative patterns produced by the PredPatt tool and real-valued node and edge attributes constructed using sophisticated normalization procedures. The Decomp toolkit provides a suite of Python 3 tools for querying UDS graphs using SPARQL. Both UDS1.0 and Decomp0.1 are publicly available at http://decomp.io.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">
473white-etal-2020-universal</span><span class="p">,</span> 474 <span class="na">title</span> <span class="p">=</span> <span class="s">{The Universal Decompositional Semantics Dataset and Decomp Toolkit}</span><span class="p">,</span> 475 <span class="na">author</span> <span class="p">=</span> <span class="s">{White, Aaron Steven and Stengel-Eskin, Elias and Vashishtha, Siddharth and Govindarajan, Venkata Subrahmanyan and Reisinger, Dee Ann and Vieira, Tim and Sakaguchi, Keisuke and Zhang, Sheng and Ferraro, Francis and Rudinger, Rachel and Rawlins, Kyle and Van Durme, Benjamin}</span><span class="p">,</span> 476 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 12th Language Resources and Evaluation Conference}</span><span class="p">,</span> 477 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span><span class="p">,</span> 478 <span class="na">year</span> <span class="p">=</span> <span class="s">{2020}</span><span class="p">,</span> 479 <span class="na">address</span> <span class="p">=</span> <span class="s">{Marseille, France}</span><span class="p">,</span> 480 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{European Language Resources Association}</span><span class="p">,</span> 481 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/2020.lrec-1.699}</span><span class="p">,</span> 482 <span class="na">pages</span> <span class="p">=</span> <span class="s">{5698--5707}</span><span class="p">,</span> 483 <span class="na">language</span> <span class="p">=</span> <span class="s">{English}</span><span class="p">,</span> 484 <span class="na">isbn</span> <span class="p">=</span> <span class="s">{979-10-95546-34-4}</span> 485<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ICLR</a></abbr> </div> <div id="bhagavatula2020abductive" class="col-sm-10"> <div class="title">Abductive Commonsense Reasoning</div> <div class="author"> Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, <em>Keisuke Sakaguchi</em>, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, Yejin Choi </div> <div class="periodical"> <em>International Conference on Learning Representations</em> 2020 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2020_iclr_anli.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/allenai/abductive-commonsense-reasoning" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">bhagavatula2020abductive</span><span class="p">,</span> 486 <span class="na">title</span> <span class="p">=</span> <span class="s">{Abductive Commonsense Reasoning}</span><span class="p">,</span> 487 <span class="na">author</span> <span class="p">=</span> <span class="s">{Bhagavatula, Chandra and Bras, Ronan Le and Malaviya, Chaitanya and Sakaguchi, Keisuke and Holtzman, Ari and Rashkin, Hannah and Downey, Doug and Yih, Wen-tau and Choi, Yejin}</span><span class="p">,</span> 488 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{International Conference on Learning Representations}</span><span class="p">,</span> 489 <span class="na">year</span> <span class="p">=</span> <span class="s">{2020}</span><span class="p">,</span> 490 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://openreview.net/forum?id=Byg1v1HKDB}</span> 491<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">AAAI</a></abbr> </div> <div id="Sakaguchi-etal-2020-winogrande" class="col-sm-10"> <div class="title">WinoGrande: An Adversarial Winograd Schema Challenge at Scale</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi </div> <div class="periodical"> <em>Proceedings of the AAAI Conference on Artificial Intelligence</em> Apr 2020 </div> <div class="award">
491 <img class="emoji" title=":trophy:" alt=":trophy:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f3c6.png" height="20" width="20"> <a href="https://aaai.org/Awards/paper.php" rel="external nofollow noopener" target="_blank">Outstanding Paper Award (Single Best Paper) </a> </div> <div> <img class="emoji" title=":pencil:" alt=":pencil:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f4dd.png" height="20" width="20"> <a href="https://www.technologyreview.com/2020/01/31/304844/ai-common-sense-reads-human-language-ai2/" rel="external nofollow noopener" target="_blank">MIT Technology Review </a> </div> <div> <img class="emoji" title=":pencil:" alt=":pencil:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f4dd.png" height="20" width="20"> <a href="https://jack-clark.net/2019/07/29/import-ai-157-how-weather-can-break-self-driving-car-ai-modelling-traffic-via-deep-learning-and-satellites-and-chinese-scientists-make-a-smarter-smaller-yolov3/" rel="external nofollow noopener" target="_blank">Import AI </a> </div> <div> <img class="emoji" title=":pencil:" alt=":pencil:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f4dd.png" height="20" width="20"> <a href="https://www.forbes.com/sites/gilpress/2020/01/29/7-observations-about-ai-in-2019/?sh=e3a68c419063" rel="external nofollow noopener" target="_blank">Forbes </a> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2020_aaai_winogrande.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://winogrande.allenai.org/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4 â 79.1%, which are â¼15-35% (absolute) below human performance of 94.0%, depending on the amount of the training data allowed (2% â 100% respectively). Furthermore, we establish new state-of-the-art results on five related benchmarks â WSC (â 90.1%), DPR (â 93.1%), COPA(â 90.6%), KnowRef (â 85.6%), and Winogender (â 97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">Sakaguchi-etal-2020-winogrande</span><span class="p">,</span> 492 <span class="na">title</span> <span class="p">=</span> <span class="s">{WinoGrande: An Adversarial Winograd Schema Challenge at Scale}</span><span class="p">,</span> 493 <span class="na">volume</span> <span class="p">=</span> <span class="s">{34}</span><span class="p">,</span> 494 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://ojs.aaai.org/index.php/AAAI/article/view/6399}</span><span class="p">,</span> 495 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.1609/aaai.v34i05.6399}</span><span class="p">,</span> 496 <span class="na">number</span> <span class="p">=</span> <span class="s">{05}</span><span class="p">,</span> 497 <span class="na">
497journal</span> <span class="p">=</span> <span class="s">{Proceedings of the AAAI Conference on Artificial Intelligence}</span><span class="p">,</span> 498 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Le Bras, Ronan and Bhagavatula, Chandra and Choi, Yejin}</span><span class="p">,</span> 499 <span class="na">year</span> <span class="p">=</span> <span class="s">{2020}</span><span class="p">,</span> 500 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 501 <span class="na">pages</span> <span class="p">=</span> <span class="s">{8732--8740}</span> 502<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2019</h6> <ol class="bibliography" reversed="reversed"><li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP</a></abbr> </div> <div id="tandon-etal-2019-wiqa" class="col-sm-10"> <div class="title">WIQA: A dataset for âWhat if...â reasoning over procedural text</div> <div class="author"> Niket Tandon, Bhavana Dalvi, <em>Keisuke Sakaguchi</em>, Peter Clark, Antoine Bosselut </div> <div class="periodical"> <em>Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</em> Nov 2019 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2019_emnlp_wiqa.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://allenai.org/data/wiqa" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We introduce WIQA, the first large-scale dataset of âWhat if...â questions over procedural text. WIQA contains a collection of paragraphs, each annotated with multiple influence graphs describing how one change affects another, and a large (40k) collection of âWhat if...?â multiple-choice questions derived from these. For example, given a paragraph about beach erosion, would stormy weather hasten or decelerate erosion? WIQA contains three kinds of questions: perturbations to steps mentioned in the paragraph; external (out-of-paragraph) perturbations requiring commonsense knowledge; and irrelevant (no effect) perturbations. We find that state-of-the-art models achieve 73.8% accuracy, well below the human performance of 96.3%. We analyze the challenges, in particular tracking chains of influences, and present the dataset as an open challenge to the community.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">tandon-etal-2019-wiqa</span><span class="p">,</span> 503 <span class="na">title</span> <span class="p">=</span> <span class="s">{{WIQA}: A dataset for {``}What if...{''} reasoning over procedural text}</span><span class="p">,</span> 504 <span class="na">author</span> <span class="p">=</span> <span class="s">{Tandon, Niket and Dalvi, Bhavana and Sakaguchi, Keisuke and Clark, Peter and Bosselut, Antoine}</span><span class="p">,</span> 505 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}</span><span class="p">,</span> 506 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 507 <span class="na">year</span> <span class="p">=</span> <span class="s">{2019}</span><span class="p">,</span> 508 <span class="na">address</span> <span class="p">=</span> <span class="s">{Hong Kong, China}</span><span class="p">,</span> 509 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 510 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/D19-1629}</span><span class="p">,</span> 511 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/D19-1629}</span><span class="p">,</span> 512 <span class="na">pages</span> <span class="p">=</span> <span class="s">{6076--6085}</span> 513<span class="p">}</span></code></pre></figure> </div> </div> </div> </li></ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2018</h6> <ol class="bibliography" reversed="reversed"><li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="sakaguchi-van-durme-2018-efficient" class="col-sm-10"> <div class="title">Efficient Online Scalar Annotation with Bounded Support</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em> Jul 2018 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2018_acl_easl.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/decomp-sem/EASL" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="https://www.slideshare.net/keisks/acl18-sakaguchi" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-van-durme-2018-efficient</span><span class="p">,</span> 514 <span class="na">title</span> <span class="p">=</span> <span class="s">{Efficient Online Scalar Annotation with Bounded Support}</span><span class="p">,</span> 515 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Van Durme, Benjamin}</span><span class="p">,</span> 516 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}</span><span class="p">,</span> 517 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 518 <span class="na">year</span> <span class="p">=</span> <span class="s">{2018}</span><span class="p">,</span> 519 <span class="na">address</span> <span class="p">=</span> <span class="s">{Melbourne, Australia}</span><span class="p">,</span> 520 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 521 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/P18-1020}</span><span class="p">,</span> 522 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/P18-1020}</span><span class="p">,</span> 523 <span class="na">pages</span> <span class="p">=</span> <span class="s">{208--218}</span> 524<span class="p">}</span></code></pre></figure> </div> </div> </div> </li></ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2017</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">IJCNLP</a></abbr> </div> <div id="sakaguchi-etal-2017-grammatical" class="col-sm-10"> <div class="title">Grammatical Error Correction with Neural Reinforcement Learning</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Matt Post, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)</em> Nov 2017 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2017_ijcnlp_neural_rl.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/nematus/tree/nrl-gleu" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="https://www.slideshare.net/keisks/ijcnlp17-sakaguchi" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>We propose a neural encoder-decoder model with reinforcement learning (NRL) for grammatical error correction (GEC). Unlike conventional maximum likelihood estimation (MLE), the model directly optimizes towards an objective that considers a sentence-level, task-specific evaluation metric, avoiding the exposure bias issue in MLE. We demonstrate that NRL outperforms MLE both in human and automated evaluation metrics, achieving the state-of-the-art on a fluency-oriented GEC corpus.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2017-grammatical</span><span class="p">,</span> 525 <span class="na">title</span> <span class="p">=</span> <span class="s">{Grammatical Error Correction with Neural Reinforcement Learning}</span><span class="p">,</span> 526 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Post, Matt and Van Durme, Benjamin}</span><span class="p">,</span> 527 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)}</span><span class="p">,</span> 528 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 529 <span class="na">year</span> <span class="p">=</span> <span class="s">{2017}</span><span class="p">,</span> 530 <span class="na">address</span> <span class="p">=</span> <span class="s">{Taipei, Taiwan}</span><span class="p">,</span> 531 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Asian Federation of Natural Language Processing}</span><span class="p">,</span> 532 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/I17-2062}</span><span class="p">,</span> 533 <span class="na">pages</span> <span class="p">=</span> <span class="s">{366--372}</span> 534<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">BEA</a></abbr> </div> <div id="sakaguchi-etal-2017-gec" class="col-sm-10"> <div class="title">GEC into the future: Where are we going and how do we get there?</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Courtney Napoles, Joel Tetreault </div> <div class="periodical"> <em>Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Ap
534plications</em> Sep 2017 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2017_bea_gec_future.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>The field of grammatical error correction (GEC) has made tremendous bounds in the last ten years, but new questions and obstacles are revealing themselves. In this position paper, we discuss the issues that need to be addressed and provide recommendations for the field to continue to make progress, and propose a new shared task. We invite suggestions and critiques from the audience to make the new shared task a community-driven venture.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2017-gec</span><span class="p">,</span> 535 <span class="na">title</span> <span class="p">=</span> <span class="s">{{GEC} into the future: Where are we going and how do we get there?}</span><span class="p">,</span> 536 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Napoles, Courtney and Tetreault, Joel}</span><span class="p">,</span> 537 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 12th Workshop on Innovative Use of {NLP} for Building Educational Applications}</span><span class="p">,</span> 538 <span class="na">month</span> <span class="p">=</span> <span class="nv">sep</span><span class="p">,</span> 539 <span class="na">year</span> <span class="p">=</span> <span class="s">{2017}</span><span class="p">,</span> 540 <span class="na">address</span> <span class="p">=</span> <span class="s">{Copenhagen, Denmark}</span><span class="p">,</span> 541 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 542 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/W17-5019}</span><span class="p">,</span> 543 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/W17-5019}</span><span class="p">,</span> 544 <span class="na">pages</span> <span class="p">=</span> <span class="s">{180--187}</span> 545<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="sakaguchi-etal-2017-error" class="col-sm-10"> <div class="title">Error-repair Dependency Parsing for Ungrammatical Texts</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Matt Post, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</em> Jul 2017 </div> <div class="award"> <img class="emoji" title=":trophy:" alt=":trophy:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f3c6.png" height="20" width="20"> <a href="https://acl2017.wordpress.com/2017/08/03/outstanding-and-best-papers-and-the-decision-process/" rel="external nofollow noopener" target="_blank">Outstanding Paper Award (1.5% of the submissions) </a> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2017_acl_error_repair.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/error-repair-parsing" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="https://www.slideshare.net/keisks/201707-acl" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>
545We propose a new dependency parsing scheme which jointly parses a sentence and repairs grammatical errors by extending the non-directional transition-based formalism of Goldberg and Elhadad (2010) with three additional actions: SUBSTITUTE, DELETE, INSERT. Because these actions may cause an infinite loop in derivation, we also introduce simple constraints that ensure the parser termination. We evaluate our model with respect to dependency accuracy and grammaticality improvements for ungrammatical sentences, demonstrating the robustness and applicability of our scheme.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2017-error</span><span class="p">,</span> 546 <span class="na">title</span> <span class="p">=</span> <span class="s">{Error-repair Dependency Parsing for Ungrammatical Texts}</span><span class="p">,</span> 547 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Post, Matt and Van Durme, Benjamin}</span><span class="p">,</span> 548 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}</span><span class="p">,</span> 549 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 550 <span class="na">year</span> <span class="p">=</span> <span class="s">{2017}</span><span class="p">,</span> 551 <span class="na">address</span> <span class="p">=</span> <span class="s">{Vancouver, Canada}</span><span class="p">,</span> 552 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 553 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/P17-2030}</span><span class="p">,</span> 554 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/P17-2030}</span><span class="p">,</span> 555 <span class="na">pages</span> <span class="p">=</span> <span class="s">{189--195}</span> 556<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EACL</a></abbr> </div> <div id="napoles-etal-2017-jfleg" class="col-sm-10"> <div class="title">JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction</div> <div class="author"> Courtney Napoles, <em>Keisuke Sakaguchi</em>, Joel Tetreault </div> <div class="periodical"> <em>Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers</em> Apr 2017 </div> <div class="award"> </div> <div> <img class="emoji" title=":pencil:" alt=":pencil:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f4dd.png" height="20" width="20"> <a href="https://www.grammarly.com/blog/engineering/paving-the-way-for-human-level-sentence-corrections/" rel="external nofollow noopener" target="_blank">Grammarly blog </a> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2017_eacl_jfleg.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/jfleg" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We present a new parallel corpus, JHU FLuency-Extended GUG corpus (JFLEG) for developing and evaluating grammatical error correction (GEC). Unlike other corpora, it represents a broad range of language proficiency levels and uses holistic fluency edits to not only correct grammatical errors but also make the original text more native sounding. We describe the types of corrections made and benchmark four leading GEC systems on this corpus, identifying specific areas in which they do well and how they can improve. JFLEG fulfills the need for a new gold standard to properly assess the current state of GEC.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">napoles-etal-2017-jfleg</span><span class="p">,</span> 557 <span class="na">title</span> <span class="p">=</span> <span class="s">{{JFLEG}: A Fluency Corpus and Benchmark for Grammatical Error Correction}</span><span class="p">,</span> 558 <span class="na">author</span> <span class="p">=</span> <span class="s">{Napoles, Courtney and Sakaguchi, Keisuke and Tetreault, Joel}</span><span class="p">,</span> 559 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 15th Conference of the {E}uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers}</span><span class="p">,</span> 560 <span class="na">month</span> <span class="p">=</span> <span class="nv">apr</span><span class="p">,</span> 561 <span class="na">year</span> <span class="p">=</span> <span class="s">{2017}</span><span class="p">,</span> 562 <span class="na">address</span> <span class="p">=</span> <span class="s">{Valencia, Spain}</span><span class="p">,</span> 563 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 564 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/E17-2037}</span><span class="p">,</span> 565 <span class="na">pages</span> <span class="p">=</span> <span class="s">{229--234}</span> 566<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">AAAI</a></abbr> </div> <div id="10.5555/3298023.3298045" class="col-sm-10"> <div class="title">Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Kevin Duh, Matt Post, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence</em> 2017 </div> <div class="award"> </div> <div> <img class="emoji" title=":pencil:" alt=":pencil:" src="https://github.githubassets.com/images/icons/emoji/unicode/1f4dd.png" height="20" width="20"> <a href="https://x.com/tdietterich/status/827638126713544704" rel="external nofollow noopener" target="_blank">(The title contains intentional typos.) </a> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2017_aaai_robsut.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/robsut-wrod-reocginiton" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="/assets/pdf/2017_aaai_robsut_poster.pdf" class="btn btn-sm z-depth-0" role="button">Poster</a> </div> <div class="abstract hidden"> <p>Language processing mechanism by humans is generally more robust than computers. The Cmabrigde Uinervtisy (Cambridge University) effect from the psycholinguistics literature has demonstrated such a robust word processing mechanism, where jumbled words (e.g. Cmabrigde / Cambridge) are recognized
566with little cost. On the other hand, computational models for word recognition (e.g. spelling checkers) perform poorly on data with such noise.Inspired by the findings from the Cmabrigde Uinervtisy effect, we propose a word recognition model based on a semi-character level recurrent neural network (scRNN). In our experiments, we demonstrate that scRNN has significantly more robust performance in word spelling correction (i.e. word recognition) compared to existing spelling checkers and character-based convolutional neural network. Furthermore, we demonstrate that the model is cognitively plausible by replicating a psycholinguistics experiment about human reading difficulty using our model.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">10.5555/3298023.3298045</span><span class="p">,</span> 567 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Duh, Kevin and Post, Matt and Durme, Benjamin Van}</span><span class="p">,</span> 568 <span class="na">title</span> <span class="p">=</span> <span class="s">{Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network}</span><span class="p">,</span> 569 <span class="na">year</span> <span class="p">=</span> <span class="s">{2017}</span><span class="p">,</span> 570 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{AAAI Press}</span><span class="p">,</span> 571 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence}</span><span class="p">,</span> 572 <span class="na">pages</span> <span class="p">=</span> <span class="s">{3281--3287}</span><span class="p">,</span> 573 <span class="na">numpages</span> <span class="p">=</span> <span class="s">{7}</span><span class="p">,</span> 574 <span class="na">location</span> <span class="p">=</span> <span class="s">{San Francisco, California, USA}</span><span class="p">,</span> 575 <span class="na">series</span> <span class="p">=</span> <span class="s">{AAAI'17}</span> 576<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2016</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP</a></abbr> </div> <div id="white-etal-2016-universal" class="col-sm-10"> <div class="title">Universal Decompositional Semantics on Universal Dependencies</div> <div class="author"> Aaron Steven White, Drew Reisinger, <em>Keisuke Sakaguchi</em>, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing</em> Nov 2016 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2016_emnlp_decomp.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://decomp.io/" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>We present a framework for augmenting data sets from the Universal Dependencies project with Universal Decompositional Semantics. Where the Universal Dependencies project aims to provide a syntactic annotation standard that can be used consistently across many languages as well as a collection of corpora that use that standard, our extension has similar aims for semantic annotation. We describe results from annotating the English Universal Dependencies treebank, dealing with word senses, semantic roles, and event properties.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">
576white-etal-2016-universal</span><span class="p">,</span> 577 <span class="na">title</span> <span class="p">=</span> <span class="s">{Universal Decompositional Semantics on {U}niversal {D}ependencies}</span><span class="p">,</span> 578 <span class="na">author</span> <span class="p">=</span> <span class="s">{White, Aaron Steven and Reisinger, Drew and Sakaguchi, Keisuke and Vieira, Tim and Zhang, Sheng and Rudinger, Rachel and Rawlins, Kyle and Van Durme, Benjamin}</span><span class="p">,</span> 579 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}</span><span class="p">,</span> 580 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 581 <span class="na">year</span> <span class="p">=</span> <span class="s">{2016}</span><span class="p">,</span> 582 <span class="na">address</span> <span class="p">=</span> <span class="s">{Austin, Texas}</span><span class="p">,</span> 583 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 584 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/D16-1177}</span><span class="p">,</span> 585 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/D16-1177}</span><span class="p">,</span> 586 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1713--1723}</span> 587<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">EMNLP</a></abbr> </div> <div id="napoles-etal-2016-theres" class="col-sm-10"> <div class="title">Thereâs No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction</div> <div class="author"> Courtney Napoles, <em>Keisuke Sakaguchi</em>, Joel Tetreault </div> <div class="periodical"> <em>Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing</em> Nov 2016 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2016_emnlp_no_comparison.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/cnap/grammaticality-metrics" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metrics. By interpolating both methods, we achieve state-of-the-art correlation with human judgments. Finally, we show that GEC metrics are much more reliable when they are calculated at the sentence level instead of the corpus level. We have set up a CodaLab site for benchmarking GEC output using a common dataset and different evaluation metrics.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">napoles-etal-2016-theres</span><span class="p">,</span> 588 <span class="na">title</span> <span class="p">=</span> <span class="s">{There{'}s No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction}</span><span class="p">,</span> 589 <span class="na">author</span> <span class="p">=</span> <span class="s">{Napoles, Courtney and Sakaguchi, Keisuke and Tetreault, Joel}</span><span class="p">,</span> 590 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}</span><span class="p">,</span> 591 <span class="na">month</span> <span class="p">=</span> <span class="nv">nov</span><span class="p">,</span> 592 <span class="na">year</span> <span class="p">=</span> <span class="s">{2016}</span><span class="p">,</span> 593 <span class="na">address</span> <span class="p">=</span> <span class="s">{Austin, Texas}</span><span class="p">,</span> 594 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 595 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/D16-1228}</span><span class="p">,</span> 596 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/D16-1228}</span><span class="p">,</span> 597 <span class="na">pages</span> <span class="p">=</span> <span class="s">{2109--2115}</span> 598<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="nagata-sakaguchi-2016-phrase" class="col-sm-10"> <div class="title">Phrase Structure Annotation and Parsing for Learner English</div> <div class="author"> Ryo Nagata, <em>Keisuke Sakaguchi</em> </div> <div class="periodical"> <em>Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em> Aug 2016 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2016_acl_phrase_structure_annotation.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>There has been almost no work on phrase structure annotation and parsing specially designed for learner English despite the fact that they are useful for representing the structural characteristics of learner English. To address this problem, in this paper, we first propose a phrase structure annotation scheme for learner English and annotate two different learner corpora using it. Second, we show their usefulness, reporting on (a
598) inter-annotator agreement rate, (b) characteristic CFG rules in the corpora, and (c) parsing performance on them. In addition, we explore methods to improve phrase structure parsing for learner English (achieving an F -measure of 0.878). Finally, we release the full annotation guidelines, the annotated data, and the improved parser model for learner English to the public.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">nagata-sakaguchi-2016-phrase</span><span class="p">,</span> 599 <span class="na">title</span> <span class="p">=</span> <span class="s">{Phrase Structure Annotation and Parsing for Learner {E}nglish}</span><span class="p">,</span> 600 <span class="na">author</span> <span class="p">=</span> <span class="s">{Nagata, Ryo and Sakaguchi, Keisuke}</span><span class="p">,</span> 601 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}</span><span class="p">,</span> 602 <span class="na">month</span> <span class="p">=</span> <span class="nv">aug</span><span class="p">,</span> 603 <span class="na">year</span> <span class="p">=</span> <span class="s">{2016}</span><span class="p">,</span> 604 <span class="na">address</span> <span class="p">=</span> <span class="s">{Berlin, Germany}</span><span class="p">,</span> 605 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 606 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/P16-1173}</span><span class="p">,</span> 607 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.18653/v1/P16-1173}</span><span class="p">,</span> 608 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1837--1847}</span> 609<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">TACL</a></abbr> </div> <div id="sakaguchi-etal-2016-reassessing" class="col-sm-10"> <div class="title">Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Courtney Napoles, Matt Post, Joel Tetreault </div> <div class="periodical"> <em>Transactions of the Association for Computational Linguistics</em> 2016 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2016_tacl_reassessing.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/reassess-gec" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="https://www.slideshare.net/keisks/201608-tacl" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>The field of grammatical error correction (GEC) has grown substantially in recent years, with research directed at both evaluation metrics and improved system performance against those metrics. One unvisited assumption, however, is the reliance of GEC evaluation on error-coded corpora, which contain specific labeled corrections. We examine current practices and show that GECâs reliance on such corpora unnaturally constrains annotation and automatic evaluation, resulting in (a) sentences that do not sound acceptable to native speakers and (b) system rankings that do not correlate with human judgments. In light of this, we propose an alternate approach that jettisons costly error coding in favor of unannotated, whole-sentence rewrites. We compare the performance of existing metrics over different gold-standard annotations, and show that automatic evaluation with our new annotation scheme has very strong correlation with expert rankings (Ï = 0.82). As a result, we advocate for a fundamental and necessary shift in the goal of GEC, from correcting small, labeled error types, to producing text that has native fluency.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">sakaguchi-etal-2016-reassessing</span><span class="p">,</span> 610 <span class="na">title</span> <span class="p">=</span> <span class="s">{Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality}</span><span class="p">,</span> 611 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Napoles, Courtney and Post, Matt and Tetreault, Joel}</span><span class="p">,</span> 612 <span class="na">
612journal</span> <span class="p">=</span> <span class="s">{Transactions of the Association for Computational Linguistics}</span><span class="p">,</span> 613 <span class="na">volume</span> <span class="p">=</span> <span class="s">{4}</span><span class="p">,</span> 614 <span class="na">year</span> <span class="p">=</span> <span class="s">{2016}</span><span class="p">,</span> 615 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/Q16-1013}</span><span class="p">,</span> 616 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.1162/tacl_a_00091}</span><span class="p">,</span> 617 <span class="na">pages</span> <span class="p">=</span> <span class="s">{169--182}</span> 618<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">arXiv</arxiv> </div> <div id="Napoles2016GLEUWT" class="col-sm-10"> <div class="title">GLEU Without Tuning</div> <div class="author"> Courtney Napoles, <em>Keisuke Sakaguchi</em>, Matt Post, Joel R. Tetreault </div> <div class="periodical"> <em>arXiv</em> 2016 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2016_gleu_without_tuning.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>The GLEU metric was proposed for evaluating grammatical error corrections using n-gram overlap with a set of reference sentences, as opposed to precision/recall of specific annotated errors (Napoles et al., 2015). This paper describes improvements made to the GLEU metric that address problems that arise when using an increasing number of reference sets. Unlike the originally presented metric, the modified metric does not require tuning. We recommend that this version be used instead of the original version.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@article</span><span class="p">{</span><span class="nl">Napoles2016GLEUWT</span><span class="p">,</span> 619 <span class="na">title</span> <span class="p">=</span> <span class="s">{GLEU Without Tuning}</span><span class="p">,</span> 620 <span class="na">author</span> <span class="p">=</span> <span class="s">{Napoles, Courtney and Sakaguchi, Keisuke and Post, Matt and Tetreault, Joel R.}</span><span class="p">,</span> 621 <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span> 622 <span class="na">year</span> <span class="p">=</span> <span class="s">{2016}</span><span class="p">,</span> 623 <span class="na">volume</span> <span class="p">=</span> <span class="s">{abs/1605.02592}</span> 624<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2015</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="napoles-etal-2015-ground" class="col-sm-10"> <div class="title">Ground Truth for Grammatical Error Correction Metrics</div> <div class="author"> Courtney Napoles, <em>Keisuke Sakaguchi</em>, Matt Post, Joel Tetreault </div> <div class="periodical"> <em>Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)</em> Jul 2015 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2015_acl_groundtruth.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/cnap/gec-ranking" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> </div> <div class="abstract hidden"> <p>
624How do we know which grammatical error correction (GEC) system is best? A number of metrics have been proposed over the years, each motivated by weaknesses of previous metrics; however, the metrics themselves have not been compared to an empirical gold standard grounded in human judgments. We conducted the first human evaluation of GEC system outputs, and show that the rankings produced by metrics such as MaxMatch and I-measure do not correlate well with this ground truth. As a step towards better metrics, we also propose GLEU, a simple variant of BLEU, modified to account for both the source and the reference, and show that it hews much more closely to human judgments.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">napoles-etal-2015-ground</span><span class="p">,</span> 625 <span class="na">title</span> <span class="p">=</span> <span class="s">{Ground Truth for Grammatical Error Correction Metrics}</span><span class="p">,</span> 626 <span class="na">author</span> <span class="p">=</span> <span class="s">{Napoles, Courtney and Sakaguchi, Keisuke and Post, Matt and Tetreault, Joel}</span><span class="p">,</span> 627 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)}</span><span class="p">,</span> 628 <span class="na">month</span> <span class="p">=</span> <span class="nv">jul</span><span class="p">,</span> 629 <span class="na">year</span> <span class="p">=</span> <span class="s">{2015}</span><span class="p">,</span> 630 <span class="na">address</span> <span class="p">=</span> <span class="s">{Beijing, China}</span><span class="p">,</span> 631 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 632 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/P15-2097}</span><span class="p">,</span> 633 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.3115/v1/P15-2097}</span><span class="p">,</span> 634 <span class="na">pages</span> <span class="p">=</span> <span class="s">{588--593}</span> 635<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">NAACL</a></abbr> </div> <div id="sakaguchi-etal-2015-effective" class="col-sm-10"> <div class="title">Effective Feature Integration for Automated Short Answer Scoring</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Michael Heilman, Nitin Madnani </div> <div class="periodical"> <em>Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em> May 2015 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2015_naacl_stacking.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="http://www.slideshare.net/keisks/naacl15-sakaguchi" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>A major opportunity for NLP to have a realworld impact is in helping educators score student writing, particularly content-based writing (i.e., the task of automated short answer scoring). A major challenge in this enterprise is that scored responses to a particular question (i.e., labeled data) are valuable for modeling but limited in quantity. Additional information from the scoring guidelines for humans, such as exemplars for each score level and descriptions of key concepts, can also be used. Here, we explore methods for integrating scoring guidelines and labeled responses, and we find that stacked generalization (Wolpert, 1992) improves performance, especially for small training sets.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2015-effective</span><span class="p">,</span> 636 <span class="na">title</span> <span class="p">=</span> <span class="s">{Effective Feature Integration for Automated Short Answer Scoring}</span><span class="p">,</span> 637 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Heilman, Michael and Madnani, Nitin}</span><span class="p">,</span> 638 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 2015 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies}</span><span class="p">,</span> 639 <span class="na">month</span> <span class="p">=</span> <span class="nv">may</span><span class="p">,</span> 640 <span class="na">year</span> <span class="p">=</span> <span class="s">{2015}</span><span class="p">,</span> 641 <span class="na">address</span> <span class="p">=</span> <span class="s">{Denver, Colorado}</span><span class="p">,</span> 642 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 643 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/N15-1111}</span><span class="p">,</span> 644 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.3115/v1/N15-1111}</span><span class="p">,</span> 645 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1049--1054}</span> 646<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2014</h6> <ol class="bibliography" reversed="reversed"><li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">WMT</a></abbr> </div> <div id="sakaguchi-etal-2014-efficient" class="col-sm-10"> <div class="title">Efficient Elicitation of Annotations for Human Evaluation of Machine Translation</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Matt Post, Benjamin Van Durme </div> <div class="periodical"> <em>Proceedings of the Ninth Workshop on Statistical Machine Translation</em> Jun 2014 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2014_wmt_trueskill.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/wmt-trueskill" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="https://www.slideshare.net/keisks/wmt14sakaguchi" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>A main output of the annual Workshop on Statistical Machine Translation (WMT) is a ranking of the systems that participated in its shared translation tasks, produced by aggregating pairwise sentencelevel comparisons collected from human judges. Over the past few years, there have been a number of tweaks to the aggregation formula
646in attempts to address issues arising from the inherent ambiguity and subjectivity of the task, as well as weaknesses in the proposed models and the manner of model selection. We continue this line of work by adapting the TrueSkill TM algorithm â an online approach for modeling the relative skills of players in ongoing competitions, such as Microsoftâs Xbox Live â to the human evaluation of machine translation output. Our experimental results show that TrueSkill outperforms other recently proposed models on accuracy, and also can significantly reduce the number of pairwise annotations that need to be collected by sampling non-uniformly from the space of system competitions.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2014-efficient</span><span class="p">,</span> 647 <span class="na">title</span> <span class="p">=</span> <span class="s">{Efficient Elicitation of Annotations for Human Evaluation of Machine Translation}</span><span class="p">,</span> 648 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Post, Matt and Van Durme, Benjamin}</span><span class="p">,</span> 649 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the Ninth Workshop on Statistical Machine Translation}</span><span class="p">,</span> 650 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 651 <span class="na">year</span> <span class="p">=</span> <span class="s">{2014}</span><span class="p">,</span> 652 <span class="na">address</span> <span class="p">=</span> <span class="s">{Baltimore, Maryland, USA}</span><span class="p">,</span> 653 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 654 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/W14-3301}</span><span class="p">,</span> 655 <span class="na">doi</span> <span class="p">=</span> <span class="s">{10.3115/v1/W14-3301}</span><span class="p">,</span> 656 <span class="na">pages</span> <span class="p">=</span> <span class="s">{1--11}</span> 657<span class="p">}</span></code></pre></figure> </div> </div> </div> </li></ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2013</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">ACL</a></abbr> </div> <div id="sakaguchi-etal-2013-discriminative" class="col-sm-10"> <div class="title">Discriminative Approach to Fill-in-the-Blank Quiz Generation for Language Learners</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Yuki Arase, Mamoru Komachi </div> <div class="periodical"> <em>Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</em> Aug 2013 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2013_acl_discSimESL.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://github.com/keisks/disc-sim-esl" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Project</a> <a href="https://www.slideshare.net/keisks/acl2013-v2" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Poster</a> </div> <div class="abstract hidden"> <p>We propose discriminative methods to generate semantic distractors of fill-in-theblank quiz for language learners using a large-scale language learnersâ corpus. Unlike previous studies, the proposed methods aim at satisfying both reliability and validity of generated distractors; distractors should be exclusive against answers to avoid multiple answers in one quiz, and distractors should discriminate learnersâ proficiency. Detailed user evaluation with 3 native and 23 non-native speakers of English shows that our methods achieve better reliability and validity than previous methods.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2013-discriminative</span><span class="p">,</span> 658 <span class="na">title</span> <span class="p">=</span> <span class="s">{Discriminative Approach to Fill-in-the-Blank Quiz Generation for Language Learners}</span><span class="p">,</span> 659 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Arase, Yuki and Komachi, Mamoru}</span><span class="p">,</span> 660 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}</span><span class="p">,</span> 661 <span class="na">month</span> <span class="p">=</span> <span class="nv">aug</span><span class="p">,</span> 662 <span class="na">year</span> <span class="p">=</span> <span class="s">{2013}</span><span class="p">,</span> 663 <span class="na">address</span> <span class="p">=</span> <span class="s">{Sofia, Bulgaria}</span><span class="p">,</span> 664 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 665 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/P13-2043}</span><span class="p">,</span> 666 <span class="na">pages</span> <span class="p">=</span> <span class="s">{238--242}</span> 667<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">CoNLL</arxiv> </div> <div id="yoshimoto-etal-2013-naist" class="col-sm-10"> <div class="title">NAIST at 2013 CoNLL Grammatical Error Correction Shared Task</div> <div class="author"> Ippei Yoshimoto, Tomoya Kose, Kensuke Mitsuzawa, <em>Keisuke Sakaguchi</em>, Tomoya Mizumoto, Yuta Hayashibe, Mamoru Komachi, Yuji Matsumoto </div> <div class="periodical"> <em>Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task</em> Aug 2013 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2013_bea_naist_gec.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This paper describes the Nara Institute of Science and Technology (NAIST) error correction system in the CoNLL 2013 Shared Task. We constructed three systems: a system based on the Treelet Language Model for verb form and subjectverb agreement errors; a classifier trained on both learner and native corpora for noun number errors; a statistical machine translation (SMT)-based model for preposition and determiner errors. As for subject-verb agreement errors, we show that the Treelet Language Model-based approach can correct errors in which the target verb is distant from its subject. Our system ranked fourth on the official run.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">yoshimoto-etal-2013-naist</span><span class="p">,</span> 668 <span class="na">title</span> <span class="p">=</span> <span class="s">{{NAIST} at 2013 {C}o{NLL} Grammatical Error Correction Shared Task}</span><span class="p">,</span> 669 <span class="na">author</span> <span class="p">=</span> <span class="s">{Yoshimoto, Ippei and Kose, Tomoya and Mitsuzawa, Kensuke and Sakaguchi, Keisuke and Mizumoto, Tomoya and Hayashibe, Yuta and Komachi, Mamoru and Matsumoto, Yuji}</span><span class="p">,</span> 670 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task}</span><span class="p">,</span> 671 <span class="na">month</span> <span class="p">=</span> <span class="nv">aug</span><span class="p">,</span> 672 <span class="na">year</span> <span class="p">=</span> <span class="s">{2013}</span><span class="p">,</span> 673 <span class="na">address</span> <span class="p">=</span> <span class="s">{Sofia, Bulgaria}</span><span class="p">,</span> 674 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 675 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/W13-3604}</span><span class="p">,</span> 676 <span class="na">pages</span> <span class="p">=</span> <span class="s">{26--33}</span> 677<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">BEA</arxiv> </div> <div id="mizumoto-etal-2013-naist" class="col-sm-10"> <div class="title">NAIST at the NLI 2013 Shared Task</div> <div class="author"> Tomoya Mizumoto, Yuta Hayashibe, <em>Keisuke Sakaguchi</em>, Mamoru Komachi, Yuji Matsumoto </div> <div class="periodical"> <em>Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Ap
677plications</em> Jun 2013 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2013_bea_naist_nli.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This paper describes the Nara Institute of Science and Technology (NAIST) native language identification (NLI) system in the NLI 2013 Shared Task. We apply feature selection using a measure based on frequency for the closed track and try Capping and Sampling data methods for the open tracks. Our system ranked ninth in the closed track, third in open track 1 and fourth in open track 2.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">mizumoto-etal-2013-naist</span><span class="p">,</span> 678 <span class="na">title</span> <span class="p">=</span> <span class="s">{{NAIST} at the {NLI} 2013 Shared Task}</span><span class="p">,</span> 679 <span class="na">author</span> <span class="p">=</span> <span class="s">{Mizumoto, Tomoya and Hayashibe, Yuta and Sakaguchi, Keisuke and Komachi, Mamoru and Matsumoto, Yuji}</span><span class="p">,</span> 680 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the Eighth Workshop on Innovative Use of {NLP} for Building Educational Applications}</span><span class="p">,</span> 681 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 682 <span class="na">year</span> <span class="p">=</span> <span class="s">{2013}</span><span class="p">,</span> 683 <span class="na">address</span> <span class="p">=</span> <span class="s">{Atlanta, Georgia}</span><span class="p">,</span> 684 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 685 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/W13-1717}</span><span class="p">,</span> 686 <span class="na">pages</span> <span class="p">=</span> <span class="s">{134--139}</span> 687<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">MWE</a></abbr> </div> <div id="shigeto-etal-2013-construction" class="col-sm-10"> <div class="title">Construction of English MWE Dictionary and its Application to POS Tagging</div> <div class="author"> Yutaro Shigeto, Ai Azuma, Sorami Hisamoto, Shuhei Kondo, Tomoya Kose, <em>Keisuke Sakaguchi</em>, Akifumi Yoshimoto, Frances Yung, Yuji Matsumoto </div> <div class="periodical"> <em>Proceedings of the 9th Workshop on Multiword Expressions</em> Jun 2013 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2013_mwe.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> </div> <div class="abstract hidden"> <p>This paper reports our ongoing project for constructing an English multiword expression (MWE) dictionary and NLP tools based on the developed dictionary. We extracted functional MWEs from the English part of Wiktionary, annotated the Penn Treebank (PTB) with MWE information, and conducted POS tagging experiments. We report how the MWE annotation is done on PTB and the results of POS and MWE tagging experiments.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">shigeto-etal-2013-construction</span><span class="p">,</span> 688 <span class="na">title</span> <span class="p">=</span> <span class="s">{Construction of {E}nglish {MWE} Dictionary and its Application to {POS} Tagging}</span><span class="p">,</span> 689 <span class="na">author</span> <span class="p">=</span> <span class="s">{Shigeto, Yutaro and Azuma, Ai and Hisamoto, Sorami and Kondo, Shuhei and Kose, Tomoya and Sakaguchi, Keisuke and Yoshimoto, Akifumi and Yung, Frances and Matsumoto, Yuji}</span><span class="p">,</span> 690 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the 9th Workshop on Multiword Expressions}</span><span class="p">,</span> 691 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 692 <span class="na">year</span> <span class="p">=</span> <span class="s">{2013}</span><span class="p">,</span> 693 <span class="na">address</span> <span class="p">=</span> <span class="s">{Atlanta, Georgia, USA}</span><span class="p">,</span> 694 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 695 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/W13-1021}</span><span class="p">,</span> 696 <span class="na">pages</span> <span class="p">=</span> <span class="s">{139--144}</span> 697<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> <h6 class="year">2012</h6> <ol class="bibliography" reversed="reversed"> <li> <div class="row"> <div class="col-sm-2 abbr"> <abbr class="badge"><a href="">COLING</a></abbr> </div> <div id="sakaguchi-etal-2012-joint" class="col-sm-10"> <div class="title">Joint English Spelling Error Correction and POS Tagging for Language Learners Writing</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Tomoya Mizumoto, Mamoru Komachi, Yuji Matsumoto </div> <div class="periodical"> <em>Proceedings of COLING 2012</em> Dec 2012 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2012_coling_joint.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://www.slideshare.net/keisks/sakaguchi-coling12" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Slides</a> </div> <div class="abstract hidden"> <p>We propose an approach to correcting spelling errors and assigning part-of-speech (POS) tags simultaneously for sentences written by learners of English as a
697second language (ESL). In ESL writing, there are several types of errors such as preposition, determiner, verb, noun, and spelling errors. Spelling errors often interfere with POS tagging and syntactic parsing, which makes other error detection and correction tasks very difficult. In studies of grammatical error detection and correction in ESL writing, spelling correction has been regarded as a preprocessing step in a pipeline. However, several types of spelling errors in ESL are difficult to correct in the preprocessing, for example, homophones (e.g. *hear/here), confusion (*quiet/quite), split (*now a day/nowadays), merge (*swimingpool/swimming pool), inflection (*please/pleased) and derivation (*badly/bad), where the incorrect word is actually in the vocabulary and grammatical information is needed to disambiguate. In order to correct these spelling errors, and also typical typographical errors (*begginning/beginning), we propose a joint analysis of POS tagging and spelling error correction with a CRF (Conditional Random Field)-based model. We present an approach that achieves significantly better accuracies for both POS tagging and spelling correction, compared to existing approaches using either individual or pipeline analysis. We also show that the joint model can deal with novel types of misspelling in ESL writing.</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2012-joint</span><span class="p">,</span> 698 <span class="na">title</span> <span class="p">=</span> <span class="s">{Joint {E}nglish Spelling Error Correction and {POS} Tagging for Language Learners Writing}</span><span class="p">,</span> 699 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Mizumoto, Tomoya and Komachi, Mamoru and Matsumoto, Yuji}</span><span class="p">,</span> 700 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of {COLING} 2012}</span><span class="p">,</span> 701 <span class="na">month</span> <span class="p">=</span> <span class="nv">dec</span><span class="p">,</span> 702 <span class="na">year</span> <span class="p">=</span> <span class="s">{2012}</span><span class="p">,</span> 703 <span class="na">address</span> <span class="p">=</span> <span class="s">{Mumbai, India}</span><span class="p">,</span> 704 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{The COLING 2012 Organizing Committee}</span><span class="p">,</span> 705 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/C12-1144}</span><span class="p">,</span> 706 <span class="na">pages</span> <span class="p">=</span> <span class="s">{2357--2374}</span> 707<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> <li> <div class="row"> <div class="col-sm-2 abbr"> <arxiv class="badge">BEA</arxiv> </div> <div id="sakaguchi-etal-2012-naist" class="col-sm-10"> <div class="title">NAIST at the HOO 2012 Shared Task</div> <div class="author"> <em>Keisuke Sakaguchi</em>, Yuta Hayashibe, Shuhei Kondo, Lis Kanashiro, Tomoya Mizumoto, Mamoru Komachi, Yuji Matsumoto </div> <div class="periodical"> <em>Proceedings of the Seventh Workshop on Building Educational Applications Using NLP</em> Jun 2012 </div> <div class="award"> </div> <div> </div> <div> </div> <div> </div> <div class="links"> <a class="abstract btn btn-sm z-depth-0" role="button">Abs</a> <a class="bibtex btn btn-sm z-depth-0" role="button">Bib</a> <a href="/assets/pdf/2012_bea_naist_gec.pdf" class="btn btn-sm z-depth-0" role="button">PDF</a> <a href="https://www.slideshare.net/keisks/bea12" class="btn btn-sm z-depth-0" role="button" rel="external nofollow noopener" target="_blank">Poster</a> </div> <div class="abstract hidden"> <p>This paper describes the Nara Institute of Science and Technology (NAIST) error correction system in the Helping Our Own (HOO) 2012 Shared Task. Our system targets preposition and determiner errors with spelling correction as a pre-processing step. The result shows that spelling correction improves the Detection, Correction, and Recognition F-scores for preposition errors. With regard to preposition error correction, F-scores were not improved when using the training set with correction of all but preposition errors. As for determiner error correction, there was an improvement when the constituent parser was trained with a concatenation of treebank and modified treebank where all the articles appearing as the first word of an NP were removed. Our system ranked third in preposition and fourth in determiner error corrections.
707</p> </div> <div class="bibtex hidden"> <figure class="highlight"><pre><code class="language-bibtex" data-lang="bibtex"><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">sakaguchi-etal-2012-naist</span><span class="p">,</span> 708 <span class="na">title</span> <span class="p">=</span> <span class="s">{{NAIST} at the {HOO} 2012 Shared Task}</span><span class="p">,</span> 709 <span class="na">author</span> <span class="p">=</span> <span class="s">{Sakaguchi, Keisuke and Hayashibe, Yuta and Kondo, Shuhei and Kanashiro, Lis and Mizumoto, Tomoya and Komachi, Mamoru and Matsumoto, Yuji}</span><span class="p">,</span> 710 <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{Proceedings of the Seventh Workshop on Building Educational Applications Using {NLP}}</span><span class="p">,</span> 711 <span class="na">month</span> <span class="p">=</span> <span class="nv">jun</span><span class="p">,</span> 712 <span class="na">year</span> <span class="p">=</span> <span class="s">{2012}</span><span class="p">,</span> 713 <span class="na">address</span> <span class="p">=</span> <span class="s">{Montr{\'e}al, Canada}</span><span class="p">,</span> 714 <span class="na">publisher</span> <span class="p">=</span> <span class="s">{Association for Computational Linguistics}</span><span class="p">,</span> 715 <span class="na">url</span> <span class="p">=</span> <span class="s">{https://aclanthology.org/W12-2033}</span><span class="p">,</span> 716 <span class="na">pages</span> <span class="p">=</span> <span class="s">{281--288}</span> 717<span class="p">}</span></code></pre></figure> </div> </div> </div> </li> </ol> <ol class="bibliography" reversed="reversed"></ol> </div> </article> </div> </div> </body>
717<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js" integrity="sha512-bLT0Qm9VnAYZDflyKcBaQ2gg0hSYNQrJ8RilYldYQ1FxQYoCLtUjuuRuZo+fjqhx/qtq/1itJ0C2ejDxltZVFg==" crossorigin="anonymous"></script>
717
717<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/2.4.4/umd/popper.min.js" integrity="sha512-eUQ9hGdLjBjY3F41CScH3UX+4JDSI9zXeroz7hJ+RteoCaY+GP/LDoM8AO+Pt+DRFw3nXqsjh9Zsts8hnYv8/A==" crossorigin="anonymous"></script>
717
717<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js" integrity="sha512-M5KW3ztuIICmVIhjSqXe01oV2bpe248gOxqmlcYrEzAvws7Pw3z6BK0iGbrwvdrUQUhi3eXgtxp5I8PDo9YfjQ==" crossorigin="anonymous"></script>
717
717<script src="https://cdnjs.cloudflare.com/ajax/libs/mdbootstrap/4.19.1/js/mdb.min.js" integrity="sha512-Mug9KHKmroQFMLm93zGrjhibM2z2Obg9l6qFG2qKjXEXkMp/VDkI4uju9m4QKPjWSwQ6O2qzZEnJDEeCw0Blcw==" crossorigin="anonymous"></script>
717
717<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/medium-zoom.min.js" integrity="sha256-EdPgYcPk/IIrw7FYeuJQexva49pVRZNmt3LculEr7zM=" crossorigin="anonymous"></script>
717
717<script src="/assets/js/zoom.js"></script>
717
717<script src="/assets/js/common.js"></script>
717
717<script type="text/javascript"> 718 function play(audio_id){ 719 var audio = document.getElementById(audio_id); 720 audio.play(); 721 } 722</script>
722
722<script type="text/javascript"> 723 window.MathJax = { 724 tex: { 725 tags: 'ams' 726 } 727 }; 728</script>
728
728<script defer type="text/javascript" id="MathJax-script" src="https://cdn.jsdelivr.net/npm/[email protected]/es5/tex-mml-chtml.js"></script>
728 </html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.