PageSourceSearch

https://precisionhealthllm.github.io/

html precisionhealthllm.github.io collected 2026-10-03 09:47:00 UTC 28,219 bytes, 528 lines download raw bytes

1<!DOCTYPE html>
2<html>
3
4<head>
5  <meta charset="utf-8">
6  <meta name="description" content="PrecisionHealthLLM">
7  <meta name="keywords" content="LLM">
8  <meta name="viewport" content="width=device-width, initial-scale=1">
9  <meta property="og:title" content="PrecisionHealthLLM" />
10	<meta property="og:type" content="website" />
11	<meta property="og:url" content="https://PrecisionHealthLLM.github.io/" />
12	<!-- <meta property="og:image" content="https://cdn-thumbnails.huggingface.co/social-thumbnails/Universal-NER.png" /> -->
13  <title>PrecisionHealthLLM</title>
14
15  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro">
16  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/bulma.min.css">
17  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css">
18  <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
19  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css">
20  <link rel="stylesheet" href="./static/css/index.css">
21  <link rel="icon" href="https://cdn-icons-png.flaticon.com/512/954/954591.png">
22  <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
23
24
25  
25<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
25
26  
26<script defer src="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/js/all.min.js"></script>
26
27  
27<script type="module" src="https://gradio.s3-us-west-2.amazonaws.com/3.27.0/gradio.js"></script>
27
28</head>
29
30<style>
31  .expandable-card .card-text-container {
32    max-height: 200px;
33    overflow-y: hidden;
34    position: relative;
35  }
36
37  .expandable-card.expanded .card-text-container {
38    max-height: none;
39  }
40
41  .expand-btn {
42    position: relative;
43    display: none;
44    background-color: rgba(255, 255, 255, 0.8);
45    /* margin-top: -20px; */
46    /* justify-content: center; */
47    color: #510c75;
48    border-color: transparent;
49  }
50
51  .expand-btn:hover {
52    background-color: rgba(200, 200, 200, 0.8);
53    text-decoration: none;
54    border-color: transparent;
55    color: #510c75;
56  }
57
58  .expand-btn:focus {
59    outline: none;
60    text-decoration: none;
61  }
62
63  .expandable-card:not(.expanded) .card-text-container:after {
64    content: "";
65    position: absolute;
66    bottom: 0;
67    left: 0;
68    width: 100%;
69    height: 90px;
70    background: linear-gradient(rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 1));
71  }
72
73  .expandable-card:not(.expanded) .expand-btn {
74    margin-top: -40px;
75  }
76
77  .card-body {
78    padding-bottom: 5px;
79  }
80
81  .vertical-flex-layout {
82    justify-content: center;
83    align-items: center;
84    height: 100%;
85    display: flex;
86    flex-direction: column;
87    gap: 5px;
88  }
89
90  .figure-img {
91    max-width: 100%;
92    height: auto;
93  }
94
95  .adjustable-font-size {
96    font-size: calc(0.5rem + 2vw);
97  }
98
99  .chat-history {
100    flex-grow: 1;
101    overflow-y: auto;
102    /* overflow-x: hidden; */
103    padding: 5px;
104    border-bottom: 1px solid #ccc;
105    margin-bottom: 10px;
106  }
107
108  #gradio pre {
109    background-color: transparent;
110  }
111</style>
112
113<body>
114
115
116  <section class="hero">
117    <div class="hero-body">
118      <div class="container is-max-desktop">
119        <div class="columns is-centered">
120          <div class="column has-text-centered">
121            <h1 class="title is-1 publication-title">Precision Health in the Age of LLMs</h1>
122            <h3 class="title is-3 publication-title">KDD 2023 Tutorial LS-21 | Thursday, August 10</h3>
123            <div class="is-size-5">
124              <span class="author-block">
125                <a href="https://aka.ms/hoifung" style="color:#008AD7;font-weight:normal;">Hoifung Poon</a>,
126              </span>
127              <span class="author-block">
128                <a href="https://aka.ms/tristan" style="color:#008AD7;font-weight:normal;">Tristan Naumann</a>,
129              </span>
130              <span class="author-block">
131                <a href="https://sheng-z.github.io/" style="color:#008AD7;font-weight:normal;">Sheng Zhang</a>,
132              </span>
133              <span class="author-block">
134                <a href="https://javiergonzalezh.github.io/" style="color:#008AD7;font-weight:normal;">Javier González Hernández</a>
135              </span>
136            </div>
137
138            <br>
139            <div class="is-size-5 publication-authors">
140              <!-- <span class="author-block"><b style="color:#f68946; font-weight:normal">&#x25B6 </b>University of Southern California</b></span> -->
141              <span class="author-block"><b style="color:#008AD7; font-weight:normal">&nbsp&nbsp&#x25B6 </b> Microsoft Research</span>
142              <!-- <span class="author-block">&nbsp&nbsp<sup>*</sup>Equal Contribution</span> -->
143            </div>
144
145
146            <div class="column has-text-centered">
147              <div class="publication-links">
148                <!-- <span class="link-block">
149                  <a href="https://arxiv.org/abs/2308.03279" target="_blank"
150                    class="external-link button is-normal is-rounded is-dark">
151                    <span class="icon">
152                      <i class="ai ai-arxiv"></i>
153                    </span>
154                    <span>arXiv</span>
155                  </a>
156                </span> -->
157                <!-- <span class="link-block">
158                  <a href="https://github.com/universal-ner/universal-ner" target="_blank"
159                    class="external-link button is-normal is-rounded is-dark">
160                    <span class="icon">
161                      <i class="fab fa-github"></i>
162                    </span>
163                    <span>Code</span>
164                  </a>
165                </span> -->
166                <span class="link-block">
167                  <a href="https://raw.githubusercontent.com/PrecisionHealthLLM/PrecisionHealthLLM.github.io/main/PrecisionHealthLLM_KDD-tutorial.pdf" target="_blank"
168                    class="external-link button is-normal is-rounded is-dark">
169                    <span class="icon">
170                      <i class="far fa-images"></i>
171                    </span>
172                    <span>Slides</span>
173                  </a>
174                </span>
175                <span class="link-block">
176                  <a href="#" target="_blank"
177                    class="external-link button is-normal is-rounded is-dark">
178                    <span class="icon">
179                      <i class="fas fa-database"></i>
180                    </span>
181                    <span>Resources</span>
182                  </a>
183                </span>
184                <span class="link-block">
185                  <a href="https://dl.acm.org/doi/10.1145/3580305.3599568" target="_blank"
186                    class="external-link button is-normal is-rounded is-dark">
187                    <span class="icon">
188                      <i class="fas fa-share-square"></i>
189                    </span>
190                    <span>DOI</span>
191                  </a>
192                </span>
193                <!-- <span class="link-block">
194                <a href="#"
195                   class="external-link button is-normal is-rounded is-dark">
196                  <span class="icon">
197                    <i class="fab fa-youtube"></i>
198                  </span>
199                  <span>Video</span>
200                  </a>
201              </span> -->
202              </div>
203            </div>
204          </div>
205        </div>
206      </div>
207    </div>
208  </section>
209
210  <section class="hero teaser">
211    <div class="container is-max-desktop">
212      <div class="hero-body">
213        <h4 class="subtitle has-text-centered">
214            Precision Health in the Age of Large Language Models (LLMs) was presented as tutorial LS-21 at <a href="https://kdd.org/kdd2023/">KDD 2023</a> on Thursday, August 10 10am-1pm.
215            We provide the materials presented as well as additional resources for those interested in this topic.
216        </h4>
217      </div>
218    </div>
219  </section>
220
221<section class="section" id="Abstract">
222  <!-- Abstract. -->
223  <div class="columns is-centered has-text-centered">
224    <div class="column is-six-fifths">
225      <h2 class="title is-3">Abstract</h2>
226    </div>
227  </div>
228  <!-- </div> -->
229  <!--/ Abstract. -->    
230<div class="container is-max-desktop">
231
232  <div class="columns is-centered">
233    <div class="column is-full-width">
234      <div class="content has-text-justified">
235        <p>
236            Medicine today is imprecise. Among the top 20 drugs in the U.S.,
237            up to 80% of patients are non-responders. The goal of precision
238            health is to provide the right intervention for the right people at the
239            right time. The key to realize this dream is to develop a data-driven,
240            learning system that can instantly incorporate new health information to optimize care delivery and accelerate biomedical discovery.
241            In reality, however, the health ecosystem is mired in overwhelming
242            unstructured data and excruciating manual processing. For example, in cancer, standard of care often fails, and clinical trials are
243            the last hope. Yet less than 3% of patients can find a matching trial,
244            whereas 40% of trial failures simply stem from insufficient recruitment. Discovery is painfully slow as a new drug may take billions
245            of dollars and over a decade to develop.
246        </p>
247        <p>
248            In this tutorial, we will explore how large language models
249            (LLMs) can serve as a universal structuring tool to democratize
250            biomedical knowledge work and usher in an intelligence revolution in precision health. We first review background for precision
251            health and give a broad overview of the AI revolution that culminated in the development of large language models, highlighting
252            key technical innovations and prominent trends such as consolidation of AI methods across modalities. We then give an in-depth
253            review of biomedical LLMs and precision health applications, with
254            a particular focus on scaling real-world evidence generation and
255            drug discovery. To conclude, we discuss key technical challenges
256            (e.g., bias, hallucination, cost), societal ramifications (e.g., privacy,
257            regulation), as well as exciting research frontiers such as prompt
258            programming, knowledge distillation, multi-modal learning, causal
259            discovery.
260        </p>
261
262</section>
263 
264
265
266
267<section class="section" id="Resources">
268  <!-- Citation. -->
269  <div class="columns is-centered has-text-centered">
270    <div class="column is-six-fifths">
271      <h2 class="title is-3">Resources</h2>
272    </div>
273  </div>
274  <!-- </div> -->
275  <!--/ Results. -->    
276<div class="container is-max-desktop">
277
278  <div class="columns is-centered">
279    <div class="column is-full-width">
280      <div class="content has-text-justified"> 
281        <p>
282            As a resource, we provide a non-exhaustive list of papers and other resources that we referred to during the tutorial.
283            We broadly categorize resources into three categories aligned with the structure of the tutorial: Precision Health, The Intelligence Revolution, LLMs for Precision Health, Application Challenges, and Research Frontiers.
284        </p>
285
286        <p><b>Precision Health</b></p>
287        <p>
288            <ul type="1">
289                <li><a href="https://www.nature.com/articles/s41591-021-01614-0" target="_blank">AI in health and medicine</a></li>
290                <li><a href="https://www.nature.com/articles/s41586-023-05881-4" target="_blank">Foundation models for generalist medical artificial intelligence</a></li>
291            </ul>
292        </p>
293
294        <p><b>LLMs for Precision Health</b></p>
295        <p>GPT-4 in Medicine
296            <ul type="1">
297                <li><a href="https://www.pearson.com/en-au/media/2yfj24is/9780138200046.pdf" target="_blank">The AI Revolution in Medicine</a></li>
298                <li><a href="https://arxiv.org/abs/2303.13375" target="_blank">Capabilities of GPT-4 on Medical Challenge Problems</a></li>
299            </ul>
300        </p>
301        <p> Biomedical LLMs
302            <ul type="1">
303                <li>GPT-4: <a href="https://www.pearson.com/en-au/media/2yfj24is/9780138200046.pdf" target="_blank">The AI Revolution in Medicine</a></li>
304                <li>Med PaLM: <a href="https://arxiv.org/abs/2212.13138" target="_blank">Large Language Models Encode Clinical Knowledge</a></li>
305                <li>Med PaLM 2: <a href="https://arxiv.org/abs/2305.09617" target="_blank">Towards Expert-Level Medical Question Answering with Large Language Models</a></li>
306                <li>BioGPT: <a href="https://arxiv.org/abs/2210.10341" target="_blank">BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining</a></li>
307                <li>PubMedGPT / BioMedLM: <a href="https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b" target="_blank">Stanford CRFM Introduces PubMedGPT 2.7B</a></li>
308                <li>BioMedGPT: <a href="https://arxiv.org/pdf/2308.09442.pdf">BioMedGPT: Open Multimodal Generative Pre-trained Transformer for Biomedicine</a></li>
309                <li>BioMegatron: <a href="https://arxiv.org/abs/2010.06060" target="_blank">BioMegatron: Larger Biomedical Domain Language Model</a></li>
310                <li>GatorTronGPT: <a href="https://arxiv.org/abs/2305.13523" target="_blank">A Study of Generative Large Language Model for Medical Research and Healthcare</a></li>
311                <li>Galactica: <a href="https://arxiv.org/abs/2211.09085" target="_blank">Galactica: A Large Language Model for Science</a></li>
312                <li>PubMedBERT: <a href="https://arxiv.org/abs/2007.15779" target="_blank">Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing</a></li>
313                <li>ClinicalBERT: <a href="https://arxiv.org/abs/1904.03323" target="_blank">Publicly Available Clinical BERT Embeddings</a></li>
314                <li>BioBERT: <a href="https://arxiv.org/abs/1901.08746" target="_blank">BioBERT: a pre-trained biomedical language representation model for biomedical text mining</a></li>
315                <li>BioLinkBERT: <a href="https://arxiv.org/abs/2203.15827" target="_blank">LinkBERT: Pretraining Language Models with Document Links</a></li>
316                <li>SciBERT: <a href="https://arxiv.org/abs/1903.10676" target="_blank">SciBERT: A Pretrained Language Model for Scientific Text</a></li>
317                <li>DoT5: <a href="https://arxiv.org/abs/2303.13386" target="_blank">Compositional Zero-Shot Domain Transfer with Text-to-Text Models</a></li>
318                <li>SciFive: <a href="https://arxiv.org/abs/2106.03598" target="_blank">SciFive: a text-to-text transformer model for biomedical literature</a></li>
319            </ul>
320        </p>
321        <p>LLMs for Real-World Evidence
322            <ul type="1">
323                <li><a href="https://www.medrxiv.org/content/10.1101/2023.01.30.23285067v1.full" target="_blank">The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model<
323/a></li>
324                <li><a href="https://www.nature.com/articles/s41523-023-00557-8" target="_blank">Large language model (ChatGPT) as a support tool for breast tumor board</a></li>
325                <li><a href="https://pubs.rsna.org/doi/full/10.1148/radiol.230725" target="_blank">Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study</a></li>
326                <li><a href="https://www.nytimes.com/2023/06/26/technology/ai-health-care-documentation.html?login=smartlock&auth=login-smartlock" target="_blank">A.I. May Someday Work Medical Miracles. For Now, It Helps Do Paperwork.</a></li>
327            </ul>
328        </p>
329        <p>LLMs for Drug Discovery
330            <ul type="1">
331                <li><a href="https://www.nature.com/articles/s41587-023-01788-7" target="_blank">Drug discovery companies are customizing ChatGPT: here’s how</a></li>
332                <li><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8604259/" target="_blank">AI-based language models powering drug discovery and development</a></li>
333            </ul>
334        </p>
335
336        <p><b>Application Challenges</b></p>
337        <p>Bias
338            <ul type="1">
339                <li><a href="https://www.medrxiv.org/content/10.1101/2023.07.13.23292577v2.full" target="_blank">Coding Inequity: Assessing GPT-4’s Potential for Perpetuating Racial and Gender Biases in Healthcare</a></li>
340                <li><a href="https://www.statnews.com/2023/07/18/gpt4-health-disease-diagnosis-treatment-ai/" target="_blank">‘Doctors need to get on top of this’: GPT-4 displays bias in medical tasks</a></li>
341            </ul>
342        </p>
343        <p>Hallucinations
344            <ul type="1">
345                <li><a href="https://arxiv.org/abs/2202.03629" target="_blank">Survey of Hallucination in Natural Language Generation</a></li>
346                <li><a href="https://arxiv.org/abs/2305.13534" target="_blank">How Language Model Hallucinations Can Snowball</a></li>
347                <li><a href="https://arxiv.org/abs/2305.13669" target="_blank">Mitigating Language Model Hallucination with Interactive Question-Knowledge Alignment</a></li>
348                <li><a href="https://arxiv.org/abs/2305.10355" target="_blank">Evaluating Object Hallucination in Large Vision-Language Models</a></li>
349            </ul>
350        </p>
351
352        <p><b>Research Frontiers</b></p>
353        <p>Prompt Programming
354            <ul type="1">
355                <li><a href="https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/" target="_blank">Prompt Engineering</a></li>
356                <li><a href="https://arxiv.org/abs/2005.14165" target="_blank">Language Models are Few-Shot Learners</a></li>
357                <li><a href="https://arxiv.org/abs/2201.11903" target="_blank">Chain-of-Thought Prompting Elicits Reasoning in Large Language Models</a></li>
358                <li><a href="https://arxiv.org/abs/2203.11171" target="_blank">Self-Consistency Improves Chain of Thought Reasoning in Language Models</a></li>
359                <li><a href="https://arxiv.org/abs/2210.03629" target="_blank">ReAct: Synergizing Reasoning and Acting in Language Models</a></li>
360                <li><a href="https://arxiv.org/abs/2303.17651" target="_blank">Self-Refine: Iterative Refinement with Self-Feedback</a></li>
361                <li><a href="https://arxiv.org/abs/2212.14024" target="_blank">Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP</a></li>
362                <li><a href="https://www.promptingguide.ai/papers" target="_blank">Prompt Engineering Guide Papers</a></li>
363                <li><a href="https://wenting-zhao.github.io/complex-reasoning-tutorial/" target="_blank">ACL 2023 Tutorial on Complex Reasoning in Natural Language</a></li>
364                <li><a href="https://arxiv.org/pdf/2102.07350.pdf" target="_blank">Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm</a></li>
365                <li><a href="https://arxiv.org/abs/2302.07842" target="_blank">Augmented Language Models: a Survey</a></li>
366            </ul>
367        </p>
368        <p>Retrieval-Augmented Generation (RAG)
369            <ul type="1">
370                <li><a href="https://acl2023-retrieval-lm.github.io/" target="_blank">ACL 2023 Tutorial on Retrieval-based Language Models and Applications</a></li>
371                <li><a href="https://arxiv.org/abs/2302.07842" target="_blank">Augmented Language Models: a Survey</a></li>
372                <li><a href="https://www.nature.com/articles/s41586-023-05881-4" target="_blank">Foundation models for generalist medical artificial intelligence</a></li>
373            </ul>
374        </p>
375        <p>Knowledge Distillation
376            <ul type="1">
377                <li><a href="https://arxiv.org/abs/2212.10560" target="_blank">Self-Instruct: Aligning Language Models with Self-Generated Instru
377ctions</a></li>
378                <li><a href="https://crfm.stanford.edu/2023/03/13/alpaca.html" target="_blank">Alpaca: A Strong, Replicable Instruction-Following Model</a></li>
379                <li><a href="https://lmsys.org/blog/2023-03-30-vicuna/" target="_blank">Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality</a></li>
380                <li><a href="https://arxiv.org/abs/2304.03277" target="_blank">Instruction Tuning with GPT-4</a></li>
381                <li><a href="https://arxiv.org/abs/2306.11644" target="_blank">Textbooks Are All You Need</a></li>
382                <li><a href="https://arxiv.org/abs/2306.08543" target="_blank">Knowledge Distillation of Large Language Models</a></li>
383                <li><a href="https://arxiv.org/abs/2305.15717" target="_blank">The False Promise of Imitating Proprietary LLMs</a></li>
384                <li><a href="https://arxiv.org/pdf/2302.02801.pdf" target="_blank">LAMPP: Language Models as Probabilistic Priors for Perception and Action</a></li>
385                <li><a href="https://arxiv.org/pdf/2305.02301.pdf" target="_blank">Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes</a></li>
386            </ul>
387        </p>
388        <p>Multi-modal learning
389            <ul type="1">
390                <li><a href="https://arxiv.org/abs/2303.00915" target="_blank">Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing</a></li>
391                <li><a href="https://arxiv.org/abs/2306.00890" target="_blank">LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day</a></li>
392                <li><a href="https://www.nature.com/articles/s41467-023-36476-2" target="_blank">Multilingual translation for zero-shot biomedical classification using BioTranslator</a></li>
393                <li><a href="https://arxiv.org/abs/2305.10688" target="_blank">MolXPT: Wrapping Molecules with Text for Generative Pre-training</a></li>
394                <li><a href="https://www.nature.com/articles/s41591-022-01981-2" target="_blank">Multimodal biomedical AI</a></li>
395                <li><a href="https://www.nature.com/articles/s41586-023-05881-4" target="_blank">Foundation models for generalist medical artificial intelligence</a></li>
396                <li><a href="https://vlp-tutorial.github.io/" target="_blank">CVPR 2023 Tutorial on Recent Advances in Vision Foundation Models</a></li>
397            </ul>
398        </p>
399        <p>Causal Discovery
400            <ul type="1">
401                <li><a href="https://arxiv.org/abs/2305.06850" target="_blank">A Causal Roadmap for Generating High-Quality Real-World Evidence</a></li>
402                <li><a href="https://arxiv.org/pdf/2305.00050.pdf" target="_blank">Causal Reasoning and Large Language Models: Opening a New Frontier for Causality</a></li>
403                <li><a href="https://arxiv.org/pdf/2306.05836.pdf" target="_blank">Can Large Language Models Infer Causation from Correlation?</a></li>
404                <li><a href="https://arxiv.org/pdf/2012.05453.pdf" target="_blank">Causal-BERT : Language models for causality detection between events expressed in text</a></li>
405                <li><a href="https://arxiv.org/pdf/2307.02390.pdf" target="_blank">Causal Discovery with Language Models as Imperfect Experts</a></li>
406            </ul>
407        </p>
408
409
410        <!-- <p>
411          Unlike the existing work that tunes the models to do diverse tasks, we present a general recipe of instruction tuning for a specific task, where the pretrained model is tuned for a broad application class such as open NER.
412          <ul type="1">
413            <li><b>Conversation-style Instruction Tuning: </b> <span style="font-size: 95%;">We adopt a conversation-style tuning format, where the language model (LM) is presented with a passage as input. Then, for each entity type that appears in the output, we transform it into a natural language question. Subsequently, we tune the LM to generate a structured output in the form of a JSON list containing all entities of the query type in the passage.
414            We consider the <mark>reference entities</mark> (highlighted below) as gold tokens and apply a language modeling objective on these tokens.</span></li>
415 <div class="column is-six-fifths" style="display: flex; align-items: flex-start; justify-content: center;">  
416    <table class="GeneratedTable" style="width: 60%;">
417      <thead>
418        <tr>
419          <th>Conversation-style Instruction Tuning Template</th>
420        </tr>
421      </thead>
422      <tbody>
423        <tr>
424          <td style="font-size: 90%;">
425             A virtual assistant answers questions from a user based on the provided text.<br>
426    <b>User:</b> Text: <b><i>X</i></b><sub>passage</sub><br>
427    <b>Assistant:</b> I've read this text.<br>
428    <b>User:</b> What describes <b><i>t</i></b><sub>1</sub> in the text?<br>
429    <b>Assistant:</b> <mark><b><i>y</i></b><sub>1</sub></mark><br>
430    ...<br>
431    <b>User:</b> What describes <b><i>t</i></b><sub>T</sub> in the text?<br>
432    <b>Assistant:</b> <mark><b><i>y</i></b><sub>T</sub></mark><br>
433   </td>
434        </tr>
435      </tbody>
436    </table>
437    </div>
438            <li><b>Negative sampling:</b> <span style="font-size: 95%;">During tuning, we randomly sample negative entity types from the collection of all entity types that do not appear in the passage as queries and set the expected outputs as empty JSON lists. The sampling of negative entity types is done with a probability proportional to the frequency of entity types in the entire dataset. This approach greatly improves the instruction tuning results.</span></li>
439 <div class="column is-six-fifths" style="display: flex; align-items: flex-start; justify-content: center;">  
440    <table class="GeneratedTable" style="width: 90%;">
441      <thead>
442    <tr>
443        <td>Negative Sampling Strategy</td>
444        <td>
444Movie</td>
445        <td>Restaurant</td>
446        <td>AI</td>
447        <td>Literature</td>
448        <td>Music</td>
449        <td>Politics</td>
450        <td>Science</td>
451        <td>Avg</td>
452    </tr>
453</thead>
454<tbody>
455    <tr>
456        <td>None</td>
457        <td>19.1</td>
458        <td>19.1</td>
459        <td>25.1</td>
460        <td>39.5</td>
461        <td>42.7</td>
462        <td>48.9</td>
463        <td>26.2</td>
464        <td>31.5</td>
465    </tr>
466    <tr>
467        <td>Uniform</td>
468        <td>42.5</td>
469        <td>29.0</td>
470        <td>42.5</td>
471        <td>53.3</td>
472        <td>57.4</td>
473        <td>56.8</td>
474        <td>52.6</td>
475        <td>47.7</td>
476    </tr>
477    <tr>
478        <td>Frequency</td>
479        <td>42.4</td>
480        <td>31.7</td>
481        <td>53.5</td>
482        <td>59.4</td>
483        <td>65.0</td>
484        <td>60.8</td>
485        <td>61.1</td>
486        <td>53.4</td>
487    </tr>
488    </tbody>
489</table>
490</div>
491          </ul>  
492          Please check out "UniversalNER" model checkpoint on 
493          <a href="https://huggingface.co/Universal-NER">[Models]</a>.
494        </p> -->
495      </div>
496    </div>
497  </div>
498
499
500</section>
501
502<section class="section" id="BibTeX">
503    <div class="container is-max-desktop content">
504      <h2 class="title">BibTeX</h2>
505      <pre><code>
506        @inproceedings{10.1145/3580305.3599568,
507            author = {Poon, Hoifung and Naumann, Tristan and Zhang, Sheng and Gonz\'{a}lez Hern\'{a}ndez, Javier},
508            title = {Precision Health in the Age of Large Language Models},
509            year = {2023},
510            isbn = {9798400701030},
511            publisher = {Association for Computing Machinery},
512            address = {New York, NY, USA},
513            url = {https://doi.org/10.1145/3580305.3599568},
514            doi = {10.1145/3580305.3599568},
515            booktitle = {Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
516            pages = {5825–5826},
517            numpages = {2},
518            keywords = {artificial intelligence, large language model, precision health, machine learning},
519            location = {Long Beach, CA, USA},
520            series = {KDD '23}
521        }
522  </code></pre>
523    </div>
524</section>
525
526</body>
527
528</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.