1<!DOCTYPE html> 2<html> 3 4<head> 5 <meta charset="utf-8"> 6 <meta name="description" content="PrecisionHealthLLM"> 7 <meta name="keywords" content="LLM"> 8 <meta name="viewport" content="width=device-width, initial-scale=1"> 9 <meta property="og:title" content="PrecisionHealthLLM" /> 10 <meta property="og:type" content="website" /> 11 <meta property="og:url" content="https://PrecisionHealthLLM.github.io/" /> 12 <!-- <meta property="og:image" content="https://cdn-thumbnails.huggingface.co/social-thumbnails/Universal-NER.png" /> --> 13 <title>PrecisionHealthLLM</title> 14 15 <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"> 16 <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/bulma.min.css"> 17 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css"> 18 <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 19 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css"> 20 <link rel="stylesheet" href="./static/css/index.css"> 21 <link rel="icon" href="https://cdn-icons-png.flaticon.com/512/954/954591.png"> 22 <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"> 23 24 25
25<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
25 26
26<script defer src="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/js/all.min.js"></script>
26 27
27<script type="module" src="https://gradio.s3-us-west-2.amazonaws.com/3.27.0/gradio.js"></script>
27 28</head> 29 30<style> 31 .expandable-card .card-text-container { 32 max-height: 200px; 33 overflow-y: hidden; 34 position: relative; 35 } 36 37 .expandable-card.expanded .card-text-container { 38 max-height: none; 39 } 40 41 .expand-btn { 42 position: relative; 43 display: none; 44 background-color: rgba(255, 255, 255, 0.8); 45 /* margin-top: -20px; */ 46 /* justify-content: center; */ 47 color: #510c75; 48 border-color: transparent; 49 } 50 51 .expand-btn:hover { 52 background-color: rgba(200, 200, 200, 0.8); 53 text-decoration: none; 54 border-color: transparent; 55 color: #510c75; 56 } 57 58 .expand-btn:focus { 59 outline: none; 60 text-decoration: none; 61 } 62 63 .expandable-card:not(.expanded) .card-text-container:after { 64 content: ""; 65 position: absolute; 66 bottom: 0; 67 left: 0; 68 width: 100%; 69 height: 90px; 70 background: linear-gradient(rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 1)); 71 } 72 73 .expandable-card:not(.expanded) .expand-btn { 74 margin-top: -40px; 75 } 76 77 .card-body { 78 padding-bottom: 5px; 79 } 80 81 .vertical-flex-layout { 82 justify-content: center; 83 align-items: center; 84 height: 100%; 85 display: flex; 86 flex-direction: column; 87 gap: 5px; 88 } 89 90 .figure-img { 91 max-width: 100%; 92 height: auto; 93 } 94 95 .adjustable-font-size { 96 font-size: calc(0.5rem + 2vw); 97 } 98 99 .chat-history { 100 flex-grow: 1; 101 overflow-y: auto; 102 /* overflow-x: hidden; */ 103 padding: 5px; 104 border-bottom: 1px solid #ccc; 105 margin-bottom: 10px; 106 } 107 108 #gradio pre { 109 background-color: transparent; 110 } 111</style> 112 113<body> 114 115 116 <section class="hero"> 117 <div class="hero-body"> 118 <div class="container is-max-desktop"> 119 <div class="columns is-centered"> 120 <div class="column has-text-centered"> 121 <h1 class="title is-1 publication-title">Precision Health in the Age of LLMs</h1> 122 <h3 class="title is-3 publication-title">KDD 2023 Tutorial LS-21 | Thursday, August 10</h3> 123 <div class="is-size-5"> 124 <span class="author-block"> 125 <a href="https://aka.ms/hoifung" style="color:#008AD7;font-weight:normal;">Hoifung Poon</a>, 126 </span> 127 <span class="author-block"> 128 <a href="https://aka.ms/tristan" style="color:#008AD7;font-weight:normal;">Tristan Naumann</a>, 129 </span> 130 <span class="author-block"> 131 <a href="https://sheng-z.github.io/" style="color:#008AD7;font-weight:normal;">Sheng Zhang</a>, 132 </span> 133 <span class="author-block"> 134 <a href="https://javiergonzalezh.github.io/" style="color:#008AD7;font-weight:normal;">Javier González Hernández</a> 135 </span> 136 </div> 137 138 <br> 139 <div class="is-size-5 publication-authors"> 140 <!-- <span class="author-block"><b style="color:#f68946; font-weight:normal">▶ </b>University of Southern California</b></span> --> 141 <span class="author-block"><b style="color:#008AD7; font-weight:normal">  ▶ </b> Microsoft Research</span> 142 <!-- <span class="author-block">  <sup>*</sup>Equal Contribution</span> --> 143 </div> 144 145 146 <div class="column has-text-centered"> 147 <div class="publication-links"> 148 <!-- <span class="link-block"> 149 <a href="https://arxiv.org/abs/2308.03279" target="_blank" 150 class="external-link button is-normal is-rounded is-dark"> 151 <span class="icon"> 152 <i class="ai ai-arxiv"></i> 153 </span> 154 <span>arXiv</span> 155 </a> 156 </span> --> 157 <!-- <span class="link-block"> 158 <a href="https://github.com/universal-ner/universal-ner" target="_blank" 159 class="external-link button is-normal is-rounded is-dark"> 160 <span class="icon"> 161 <i class="fab fa-github"></i> 162 </span> 163 <span>Code</span> 164 </a> 165 </span> -->
166 <span class="link-block"> 167 <a href="https://raw.githubusercontent.com/PrecisionHealthLLM/PrecisionHealthLLM.github.io/main/PrecisionHealthLLM_KDD-tutorial.pdf" target="_blank" 168 class="external-link button is-normal is-rounded is-dark"> 169 <span class="icon"> 170 <i class="far fa-images"></i> 171 </span> 172 <span>Slides</span> 173 </a> 174 </span> 175 <span class="link-block"> 176 <a href="#" target="_blank" 177 class="external-link button is-normal is-rounded is-dark"> 178 <span class="icon"> 179 <i class="fas fa-database"></i> 180 </span> 181 <span>Resources</span> 182 </a> 183 </span> 184 <span class="link-block"> 185 <a href="https://dl.acm.org/doi/10.1145/3580305.3599568" target="_blank" 186 class="external-link button is-normal is-rounded is-dark"> 187 <span class="icon"> 188 <i class="fas fa-share-square"></i> 189 </span> 190 <span>DOI</span> 191 </a> 192 </span> 193 <!-- <span class="link-block"> 194 <a href="#" 195 class="external-link button is-normal is-rounded is-dark"> 196 <span class="icon"> 197 <i class="fab fa-youtube"></i> 198 </span> 199 <span>Video</span> 200 </a> 201 </span> --> 202 </div> 203 </div> 204 </div> 205 </div> 206 </div> 207 </div> 208 </section> 209 210 <section class="hero teaser"> 211 <div class="container is-max-desktop"> 212 <div class="hero-body"> 213 <h4 class="subtitle has-text-centered"> 214 Precision Health in the Age of Large Language Models (LLMs) was presented as tutorial LS-21 at <a href="https://kdd.org/kdd2023/">KDD 2023</a> on Thursday, August 10 10am-1pm. 215 We provide the materials presented as well as additional resources for those interested in this topic. 216 </h4> 217 </div> 218 </div> 219 </section> 220 221<section class="section" id="Abstract"> 222 <!-- Abstract. --> 223 <div class="columns is-centered has-text-centered"> 224 <div class="column is-six-fifths"> 225 <h2 class="title is-3">Abstract</h2> 226 </div> 227 </div> 228 <!-- </div> --> 229 <!--/ Abstract. --> 230<div class="container is-max-desktop"> 231 232 <div class="columns is-centered"> 233 <div class="column is-full-width"> 234 <div class="content has-text-justified"> 235 <p> 236 Medicine today is imprecise. Among the top 20 drugs in the U.S., 237 up to 80% of patients are non-responders. The goal of precision 238 health is to provide the right intervention for the right people at the 239 right time. The key to realize this dream is to develop a data-driven, 240 learning system that can instantly incorporate new health information to optimize care delivery and accelerate biomedical discovery. 241 In reality, however, the health ecosystem is mired in overwhelming 242 unstructured data and excruciating manual processing. For example, in cancer, standard of care often fails, and clinical trials are 243 the last hope. Yet less than 3% of patients can find a matching trial, 244 whereas 40% of trial failures simply stem from insufficient recruitment. Discovery is painfully slow as a new drug may take billions 245 of dollars and over a decade to develop. 246 </p> 247 <p> 248 In this tutorial, we will explore how large language models 249 (LLMs) can serve as a universal structuring tool to democratize 250 biomedical knowledge work and usher in an intelligence revolution in precision health. We first review background for precision 251 health and give a broad overview of the AI revolution that culminated in the development of large language models, highlighting 252 key technical innovations and prominent trends such as consolidation of AI methods across modalities. We then give an in-depth 253 review of biomedical LLMs and precision health applications, with 254 a particular focus on scaling real-world evidence generation and 255 drug discovery. To conclude, we discuss key technical challenges 256 (e.g., bias, hallucination, cost), societal ramifications (e.g., privacy, 257 regulation), as well as exciting research frontiers such as prompt 258 programming, knowledge distillation, multi-modal learning, causal 259 discovery. 260 </p> 261 262</section> 263 264 265 266 267<section class="section" id="Resources"> 268 <!-- Citation. --> 269 <div class="columns is-centered has-text-centered"> 270 <div class="column is-six-fifths"> 271 <h2 class="title is-3">Resources</h2> 272 </div> 273 </div> 274 <!-- </div> --> 275 <!--/ Results. --> 276<div class="container is-max-desktop"> 277 278 <div class="columns is-centered"> 279 <div class="column is-full-width"> 280 <div class="content has-text-justified"> 281 <p> 282 As a resource, we provide a non-exhaustive list of papers and other resources that we referred to during the tutorial. 283 We broadly categorize resources into three categories aligned with the structure of the tutorial: Precision Health, The Intelligence Revolution, LLMs for Precision Health, Application Challenges, and Research Frontiers. 284 </p> 285 286 <p><b>Precision Health</b></p> 287 <p> 288 <ul type="1"> 289 <li><a href="https://www.nature.com/articles/s41591-021-01614-0" target="_blank">AI in health and medicine</a></li> 290 <li><a href="https://www.nature.com/articles/s41586-023-05881-4" target="_blank">Foundation models for generalist medical artificial intelligence</a></li> 291 </ul> 292 </p> 293 294 <p><b>LLMs for Precision Health</b></p> 295 <p>GPT-4 in Medicine 296 <ul type="1"> 297 <li><a href="https://www.pearson.com/en-au/media/2yfj24is/9780138200046.pdf" target="_blank">The AI Revolution in Medicine</a></li> 298 <li><a href="https://arxiv.org/abs/2303.13375" target="_blank">Capabilities of GPT-4 on Medical Challenge Problems</a></li> 299 </ul> 300 </p> 301 <p> Biomedical LLMs 302 <ul type="1"> 303 <li>GPT-4: <a href="https://www.pearson.com/en-au/media/2yfj24is/9780138200046.pdf" target="_blank">The AI Revolution in Medicine</a></li> 304 <li>Med PaLM: <a href="https://arxiv.org/abs/2212.13138" target="_blank">Large Language Models Encode Clinical Knowledge</a></li> 305 <li>Med PaLM 2: <a href="https://arxiv.org/abs/2305.09617" target="_blank">Towards Expert-Level Medical Question Answering with Large Language Models</a></li> 306 <li>BioGPT: <a href="https://arxiv.org/abs/2210.10341" target="_blank">BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining</a></li> 307 <li>PubMedGPT / BioMedLM: <a href="https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b" target="_blank">Stanford CRFM Introduces PubMedGPT 2.7B</a></li> 308 <li>BioMedGPT: <a href="https://arxiv.org/pdf/2308.09442.pdf">BioMedGPT: Open Multimodal Generative Pre-trained Transformer for Biomedicine</a></li> 309 <li>BioMegatron: <a href="https://arxiv.org/abs/2010.06060" target="_blank">BioMegatron: Larger Biomedical Domain Language Model</a></li> 310 <li>GatorTronGPT: <a href="https://arxiv.org/abs/2305.13523" target="_blank">A Study of Generative Large Language Model for Medical Research and Healthcare</a></li> 311 <li>Galactica: <a href="https://arxiv.org/abs/2211.09085" target="_blank">Galactica: A Large Language Model for Science</a></li> 312 <li>PubMedBERT: <a href="https://arxiv.org/abs/2007.15779" target="_blank">Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing</a></li> 313 <li>ClinicalBERT: <a href="https://arxiv.org/abs/1904.03323" target="_blank">Publicly Available Clinical BERT Embeddings</a></li> 314 <li>BioBERT: <a href="https://arxiv.org/abs/1901.08746" target="_blank">BioBERT: a pre-trained biomedical language representation model for biomedical text mining</a></li> 315 <li>BioLinkBERT: <a href="https://arxiv.org/abs/2203.15827" target="_blank">LinkBERT: Pretraining Language Models with Document Links</a></li> 316 <li>SciBERT: <a href="https://arxiv.org/abs/1903.10676" target="_blank">SciBERT: A Pretrained Language Model for Scientific Text</a></li> 317 <li>DoT5: <a href="https://arxiv.org/abs/2303.13386" target="_blank">Compositional Zero-Shot Domain Transfer with Text-to-Text Models</a></li> 318 <li>SciFive: <a href="https://arxiv.org/abs/2106.03598" target="_blank">SciFive: a text-to-text transformer model for biomedical literature</a></li> 319 </ul> 320 </p> 321 <p>LLMs for Real-World Evidence 322 <ul type="1"> 323 <li><a href="https://www.medrxiv.org/content/10.1101/2023.01.30.23285067v1.full" target="_blank">The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model<
323/a></li> 324 <li><a href="https://www.nature.com/articles/s41523-023-00557-8" target="_blank">Large language model (ChatGPT) as a support tool for breast tumor board</a></li> 325 <li><a href="https://pubs.rsna.org/doi/full/10.1148/radiol.230725" target="_blank">Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study</a></li> 326 <li><a href="https://www.nytimes.com/2023/06/26/technology/ai-health-care-documentation.html?login=smartlock&auth=login-smartlock" target="_blank">A.I. May Someday Work Medical Miracles. For Now, It Helps Do Paperwork.</a></li> 327 </ul> 328 </p> 329 <p>LLMs for Drug Discovery 330 <ul type="1"> 331 <li><a href="https://www.nature.com/articles/s41587-023-01788-7" target="_blank">Drug discovery companies are customizing ChatGPT: hereâs how</a></li> 332 <li><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8604259/" target="_blank">AI-based language models powering drug discovery and development</a></li> 333 </ul> 334 </p> 335 336 <p><b>Application Challenges</b></p> 337 <p>Bias 338 <ul type="1"> 339 <li><a href="https://www.medrxiv.org/content/10.1101/2023.07.13.23292577v2.full" target="_blank">Coding Inequity: Assessing GPT-4âs Potential for Perpetuating Racial and Gender Biases in Healthcare</a></li> 340 <li><a href="https://www.statnews.com/2023/07/18/gpt4-health-disease-diagnosis-treatment-ai/" target="_blank">âDoctors need to get on top of thisâ: GPT-4 displays bias in medical tasks</a></li> 341 </ul> 342 </p> 343 <p>Hallucinations 344 <ul type="1"> 345 <li><a href="https://arxiv.org/abs/2202.03629" target="_blank">Survey of Hallucination in Natural Language Generation</a></li> 346 <li><a href="https://arxiv.org/abs/2305.13534" target="_blank">How Language Model Hallucinations Can Snowball</a></li> 347 <li><a href="https://arxiv.org/abs/2305.13669" target="_blank">Mitigating Language Model Hallucination with Interactive Question-Knowledge Alignment</a></li> 348 <li><a href="https://arxiv.org/abs/2305.10355" target="_blank">Evaluating Object Hallucination in Large Vision-Language Models</a></li> 349 </ul> 350 </p> 351 352 <p><b>Research Frontiers</b></p> 353 <p>Prompt Programming 354 <ul type="1"> 355 <li><a href="https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/" target="_blank">Prompt Engineering</a></li> 356 <li><a href="https://arxiv.org/abs/2005.14165" target="_blank">Language Models are Few-Shot Learners</a></li> 357 <li><a href="https://arxiv.org/abs/2201.11903" target="_blank">Chain-of-Thought Prompting Elicits Reasoning in Large Language Models</a></li> 358 <li><a href="https://arxiv.org/abs/2203.11171" target="_blank">Self-Consistency Improves Chain of Thought Reasoning in Language Models</a></li> 359 <li><a href="https://arxiv.org/abs/2210.03629" target="_blank">ReAct: Synergizing Reasoning and Acting in Language Models</a></li> 360 <li><a href="https://arxiv.org/abs/2303.17651" target="_blank">Self-Refine: Iterative Refinement with Self-Feedback</a></li> 361 <li><a href="https://arxiv.org/abs/2212.14024" target="_blank">Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP</a></li> 362 <li><a href="https://www.promptingguide.ai/papers" target="_blank">Prompt Engineering Guide Papers</a></li> 363 <li><a href="https://wenting-zhao.github.io/complex-reasoning-tutorial/" target="_blank">ACL 2023 Tutorial on Complex Reasoning in Natural Language</a></li> 364 <li><a href="https://arxiv.org/pdf/2102.07350.pdf" target="_blank">Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm</a></li> 365 <li><a href="https://arxiv.org/abs/2302.07842" target="_blank">Augmented Language Models: a Survey</a></li> 366 </ul> 367 </p> 368 <p>Retrieval-Augmented Generation (RAG) 369 <ul type="1"> 370 <li><a href="https://acl2023-retrieval-lm.github.io/" target="_blank">ACL 2023 Tutorial on Retrieval-based Language Models and Applications</a></li> 371 <li><a href="https://arxiv.org/abs/2302.07842" target="_blank">Augmented Language Models: a Survey</a></li> 372 <li><a href="https://www.nature.com/articles/s41586-023-05881-4" target="_blank">Foundation models for generalist medical artificial intelligence</a></li> 373 </ul> 374 </p> 375 <p>Knowledge Distillation 376 <ul type="1"> 377 <li><a href="https://arxiv.org/abs/2212.10560" target="_blank">Self-Instruct: Aligning Language Models with Self-Generated Instru
377ctions</a></li> 378 <li><a href="https://crfm.stanford.edu/2023/03/13/alpaca.html" target="_blank">Alpaca: A Strong, Replicable Instruction-Following Model</a></li> 379 <li><a href="https://lmsys.org/blog/2023-03-30-vicuna/" target="_blank">Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality</a></li> 380 <li><a href="https://arxiv.org/abs/2304.03277" target="_blank">Instruction Tuning with GPT-4</a></li> 381 <li><a href="https://arxiv.org/abs/2306.11644" target="_blank">Textbooks Are All You Need</a></li> 382 <li><a href="https://arxiv.org/abs/2306.08543" target="_blank">Knowledge Distillation of Large Language Models</a></li> 383 <li><a href="https://arxiv.org/abs/2305.15717" target="_blank">The False Promise of Imitating Proprietary LLMs</a></li> 384 <li><a href="https://arxiv.org/pdf/2302.02801.pdf" target="_blank">LAMPP: Language Models as Probabilistic Priors for Perception and Action</a></li> 385 <li><a href="https://arxiv.org/pdf/2305.02301.pdf" target="_blank">Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes</a></li> 386 </ul> 387 </p> 388 <p>Multi-modal learning 389 <ul type="1"> 390 <li><a href="https://arxiv.org/abs/2303.00915" target="_blank">Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing</a></li> 391 <li><a href="https://arxiv.org/abs/2306.00890" target="_blank">LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day</a></li> 392 <li><a href="https://www.nature.com/articles/s41467-023-36476-2" target="_blank">Multilingual translation for zero-shot biomedical classification using BioTranslator</a></li> 393 <li><a href="https://arxiv.org/abs/2305.10688" target="_blank">MolXPT: Wrapping Molecules with Text for Generative Pre-training</a></li> 394 <li><a href="https://www.nature.com/articles/s41591-022-01981-2" target="_blank">Multimodal biomedical AI</a></li> 395 <li><a href="https://www.nature.com/articles/s41586-023-05881-4" target="_blank">Foundation models for generalist medical artificial intelligence</a></li> 396 <li><a href="https://vlp-tutorial.github.io/" target="_blank">CVPR 2023 Tutorial on Recent Advances in Vision Foundation Models</a></li> 397 </ul> 398 </p> 399 <p>Causal Discovery 400 <ul type="1"> 401 <li><a href="https://arxiv.org/abs/2305.06850" target="_blank">A Causal Roadmap for Generating High-Quality Real-World Evidence</a></li> 402 <li><a href="https://arxiv.org/pdf/2305.00050.pdf" target="_blank">Causal Reasoning and Large Language Models: Opening a New Frontier for Causality</a></li> 403 <li><a href="https://arxiv.org/pdf/2306.05836.pdf" target="_blank">Can Large Language Models Infer Causation from Correlation?</a></li> 404 <li><a href="https://arxiv.org/pdf/2012.05453.pdf" target="_blank">Causal-BERT : Language models for causality detection between events expressed in text</a></li> 405 <li><a href="https://arxiv.org/pdf/2307.02390.pdf" target="_blank">Causal Discovery with Language Models as Imperfect Experts</a></li> 406 </ul> 407 </p> 408 409 410 <!-- <p> 411 Unlike the existing work that tunes the models to do diverse tasks, we present a general recipe of instruction tuning for a specific task, where the pretrained model is tuned for a broad application class such as open NER. 412 <ul type="1"> 413 <li><b>Conversation-style Instruction Tuning: </b> <span style="font-size: 95%;">We adopt a conversation-style tuning format, where the language model (LM) is presented with a passage as input. Then, for each entity type that appears in the output, we transform it into a natural language question. Subsequently, we tune the LM to generate a structured output in the form of a JSON list containing all entities of the query type in the passage. 414 We consider the <mark>reference entities</mark> (highlighted below) as gold tokens and apply a language modeling objective on these tokens.</span></li> 415 <div class="column is-six-fifths" style="display: flex; align-items: flex-start; justify-content: center;"> 416 <table class="GeneratedTable" style="width: 60%;"> 417 <thead> 418 <tr> 419 <th>Conversation-style Instruction Tuning Template</th> 420 </tr> 421 </thead> 422 <tbody> 423 <tr> 424 <td style="font-size: 90%;"> 425 A virtual assistant answers questions from a user based on the provided text.<br> 426 <b>User:</b> Text: <b><i>X</i></b><sub>passage</sub><br> 427 <b>Assistant:</b> I've read this text.<br> 428 <b>User:</b> What describes <b><i>t</i></b><sub>1</sub> in the text?<br> 429 <b>Assistant:</b> <mark><b><i>y</i></b><sub>1</sub></mark><br> 430 ...<br> 431 <b>User:</b> What describes <b><i>t</i></b><sub>T</sub> in the text?<br> 432 <b>Assistant:</b> <mark><b><i>y</i></b><sub>T</sub></mark><br> 433 </td> 434 </tr> 435 </tbody> 436 </table> 437 </div> 438 <li><b>Negative sampling:</b> <span style="font-size: 95%;">During tuning, we randomly sample negative entity types from the collection of all entity types that do not appear in the passage as queries and set the expected outputs as empty JSON lists. The sampling of negative entity types is done with a probability proportional to the frequency of entity types in the entire dataset. This approach greatly improves the instruction tuning results.</span></li> 439 <div class="column is-six-fifths" style="display: flex; align-items: flex-start; justify-content: center;"> 440 <table class="GeneratedTable" style="width: 90%;"> 441 <thead> 442 <tr> 443 <td>Negative Sampling Strategy</td> 444 <td>
444Movie</td> 445 <td>Restaurant</td> 446 <td>AI</td> 447 <td>Literature</td> 448 <td>Music</td> 449 <td>Politics</td> 450 <td>Science</td> 451 <td>Avg</td> 452 </tr> 453</thead> 454<tbody> 455 <tr> 456 <td>None</td> 457 <td>19.1</td> 458 <td>19.1</td> 459 <td>25.1</td> 460 <td>39.5</td> 461 <td>42.7</td> 462 <td>48.9</td> 463 <td>26.2</td> 464 <td>31.5</td> 465 </tr> 466 <tr> 467 <td>Uniform</td> 468 <td>42.5</td> 469 <td>29.0</td> 470 <td>42.5</td> 471 <td>53.3</td> 472 <td>57.4</td> 473 <td>56.8</td> 474 <td>52.6</td> 475 <td>47.7</td> 476 </tr> 477 <tr> 478 <td>Frequency</td> 479 <td>42.4</td> 480 <td>31.7</td> 481 <td>53.5</td> 482 <td>59.4</td> 483 <td>65.0</td> 484 <td>60.8</td> 485 <td>61.1</td> 486 <td>53.4</td> 487 </tr> 488 </tbody> 489</table> 490</div> 491 </ul> 492 Please check out "UniversalNER" model checkpoint on 493 <a href="https://huggingface.co/Universal-NER">[Models]</a>. 494 </p> --> 495 </div> 496 </div> 497 </div> 498 499 500</section> 501 502<section class="section" id="BibTeX"> 503 <div class="container is-max-desktop content"> 504 <h2 class="title">BibTeX</h2> 505 <pre><code> 506 @inproceedings{10.1145/3580305.3599568, 507 author = {Poon, Hoifung and Naumann, Tristan and Zhang, Sheng and Gonz\'{a}lez Hern\'{a}ndez, Javier}, 508 title = {Precision Health in the Age of Large Language Models}, 509 year = {2023}, 510 isbn = {9798400701030}, 511 publisher = {Association for Computing Machinery}, 512 address = {New York, NY, USA}, 513 url = {https://doi.org/10.1145/3580305.3599568}, 514 doi = {10.1145/3580305.3599568}, 515 booktitle = {Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, 516 pages = {5825â5826}, 517 numpages = {2}, 518 keywords = {artificial intelligence, large language model, precision health, machine learning}, 519 location = {Long Beach, CA, USA}, 520 series = {KDD '23} 521 } 522 </code></pre> 523 </div> 524</section> 525 526</body> 527 528</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.