PageSourceSearch

https://www.cloudraft.io/blog/llm-observability

html cloudraft.io collected 2026-10-04 00:24:59 UTC 203,503 bytes, 331 lines download raw bytes

1<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8" data-next-head=""/><meta name="viewport" content="width=device-width" data-next-head=""/><title data-next-head="">LLM Observability: Monitoring Large Language Models</title><meta name="robots" content="follow, index" data-next-head=""/><meta content="A comprehensive guide to implementing observability for Large Language Models in production environments, covering monitoring, tracing, security, and cost optimization strategies." name="description" data-next-head=""/><meta property="og:url" content="https://www.cloudraft.io/blog/llm-observability" data-next-head=""/><link rel="canonical" href="https://www.cloudraft.io/blog/llm-observability" data-next-head=""/><meta property="og:type" content="website" data-next-head=""/><meta property="og:site_name" content="CloudRaft" data-next-head=""/><meta property="og:description" content="A comprehensive guide to implementing observability for Large Language Models in production environments, covering monitoring, tracing, security, and cost optimization strategies." data-next-head=""/><meta property="og:title" content="LLM Observability: Monitoring Large Language Models" data-next-head=""/><meta property="og:image" content="https://res.cloudinary.com/dfee67kdq/image/upload/banner/llm-observability" data-next-head=""/><meta property="og:locale" content="en_US" data-next-head=""/><meta name="twitter:card" content="summary_large_image" data-next-head=""/><meta name="twitter:site" content="@cloudraftio" data-next-head=""/><meta name="twitter:title" content="LLM Observability: Monitoring Large Language Models" data-next-head=""/><meta name="twitter:description" content="A comprehensive guide to implementing observability for Large Language Models in production environments, covering monitoring, tracing, security, and cost optimization strategies." data-next-head=""/><meta name="twitter:image" content="https://res.cloudinary.com/dfee67kdq/image/upload/banner/llm-observability" data-next-head=""/><meta name="author" content="Rahul Agrawal" data-next-head=""/><meta name="article:published_time" property="article:published_time" content="2025-10-24T00:00:00.000Z" data-next-head=""/><meta name="published_date" property="article:published_time" content="2025-10-24T00:00:00.000Z" data-next-head=""/><meta name="published_time" property="article:published_time" content="2025-10-24T00:00:00.000Z" data-next-head=""/><link rel="apple-touch-icon" sizes="144x144" href="/favicon/apple-touch-icon.png" data-next-head=""/><link rel="icon" type="image/png" sizes="192x192" href="/favicon/android-icon-192x192.png" data-next-head=""/><link rel="icon" type="image/png" sizes="512x512" href="/favicon/android-icon-512x512.png" data-next-head=""/><link rel="icon" type="image/png" sizes="32x32" href="/favicon/favicon-32x32.png" data-next-head=""/><link rel="icon" type="image/png" sizes="16x16" href="/favicon/favicon-16x16.png" data-next-head=""/><link rel="icon" type="image/png" sizes="144x144" href="/favicon/favicon.ico" data-next-head=""/><link rel="manifest" href="/favicon/manifest.json" data-next-head=""/><meta name="msapplication-TileColor" content="#ffffff" data-next-head=""/><meta name="msapplication-TileImage" content="/favicon/ms-icon-144x144.png" data-next-head=""/><meta name="theme-color" content="#ffffff" data-next-head=""/>
1<script id="org-schema" type="application/ld+json" data-next-head="">{"@context":"https://schema.org","@type":"WebPage","copyrightYear":2026,"copyrightHolder":"CloudRaft","creator":"CloudRaft","description":"A comprehensive guide to implementing observability for Large Language Models in production environments, covering monitoring, tracing, security, and cost optimization strategies.","headline":"LLM Observability: Monitoring Large Language Models","inLanguage":"en-us","mainEntityOfPage":"https://www.cloudraft.io/blog/llm-observability","name":"LLM Observability: Monitoring Large Language Models","url":"https://www.cloudraft.io/blog/llm-observability","image":{"@type":"ImageObject","url":"https://res.cloudinary.com/dfee67kdq/image/upload/banner/llm-observability"}}</script>
1<script async="" src="https://platform.linkedin.com/in.js" data-next-head="" type="46210e8713f087f2d699c04e-text/javascript"></script>
1<link rel="preload" as="image" imageSrcSet="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fopen_source_logging_tools_vyqzdc.webp&amp;w=1200&amp;q=75 1x, /_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fopen_source_logging_tools_vyqzdc.webp&amp;w=3840&amp;q=75 2x" data-next-head=""/><link rel="preload" as="image" imageSrcSet="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2FBanner_dc8sql.webp&amp;w=1200&amp;q=75 1x, /_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2FBanner_dc8sql.webp&amp;w=3840&amp;q=75 2x" data-next-head=""/><link rel="preload" as="image" imageSrcSet="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fthanos-cloudraft_vvkxri.webp&amp;w=1200&amp;q=75 1x, /_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fthanos-cloudraft_vvkxri.webp&amp;w=3840&amp;q=75 2x" data-next-head=""/><link rel="preconnect" href="https://res.cloudinary.com" crossorigin="anonymous"/><link rel="preconnect" href="https://static.hsappstatic.net" crossorigin="anonymous"/><link rel="preconnect" href="https://www.clarity.ms" crossorigin="anonymous"/><link rel="preconnect" href="https://scripts.clarity.ms" crossorigin="anonymous"/><style type="text/css">@font-face {font-family:'Plus Jakarta Sans';font-style:normal;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/vietnamese/wght/normal.woff2);unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:normal;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/cyrillic-ext/wght/normal.woff2);unicode-range:U+0460-052F,U+1C80-1C8A,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:normal;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/latin/wght/normal.woff2);unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:normal;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/latin-ext/wght/normal.woff2);unicode-range:U+0100-02BA,U+02BD-02C5,U+02C7-02CC,U+02CE-02D7,U+02DD-02FF,U+0304,U+0308,U+0329,U+1D00-1DBF,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:italic;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/latin/wght/italic.woff2);unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:italic;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/cyrillic-ext/wght/italic.woff2);unicode-range:U+0460-052F,U+1C80-1C8A,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:italic;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/latin-ext/wght/italic.woff2);unicode-range:U+0100-02BA,U+02BD-02C5,U+02C7-02CC,U+02CE-02D7,U+02DD-02FF,U+0304,U+0308,U+0329,U+1D00-1DBF,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF;font-display:swap;}@font-face {font-family:'Plus Jakarta Sans';font-style:italic;font-weight:300 800;src:url(/cf-fonts/v/plus-jakarta-sans/5.2.8/vietnamese/wght/italic.woff2);unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB;font-display:swap;}</style>
1<script type="46210e8713f087f2d699c04e-text/javascript">
2                (function () {
3                  try {
4                    const theme = localStorage.getItem('theme') || 'light';
5                    if (theme === 'dark') {
6                      document.documentElement.classList.add('dark');
7                    } else {
8                      document.documentElement.classList.remove('dark');
9                    }
10                  } catch (_) {}
11                })();
12              </script>
12<link rel="preload" href="/_next/static/chunks/0y9o83s-jt7n2.css" as="style"/><link rel="stylesheet" href="/_next/static/chunks/0y9o83s-jt7n2.css" data-n-g=""/><link rel="preload" href="/_next/static/chunks/0mrre8cgb6rml.css" as="style"/><link rel="stylesheet" href="/_next/static/chunks/0mrre8cgb6rml.css" data-n-g=""/><link rel="preload" href="/_next/static/chunks/1476ugdwt5s2z.css" as="style"/><link rel="stylesheet" href="/_next/static/chunks/1476ugdwt5s2z.css" data-n-g=""/><link rel="preload" href="/_next/static/chunks/0enrfpp_y-kj0.css" as="style"/><link rel="stylesheet" href="/_next/static/chunks/0enrfpp_y-kj0.css" data-n-p=""/><noscript data-n-css=""></noscript>
12<script defer="" nomodule="" src="/_next/static/chunks/03~yq9q893hmn.js" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/09tqhgfc560qq.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0c~dn9.2bk429.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0pa2y6nbbr1au.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0ry7823wg2a.o.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0x-bmo8q_1d79.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0b.w8u5jwog3w.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/turbopack-0jcllicz_eqkf.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0~j70m_p4a8uv.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0-cv_rtkkvpri.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/03h2x2elikag4.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/109i226g~pba9.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0nrb34vf_me7c.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0osv5.sbga~fa.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0p.w4iipvo21g.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/07py9-6k~vly5.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0250uccnqkv.9.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0k5wswtrw7f.f.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0x3sou8b3txzs.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0jazr7g2mt4eh.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0a~~1tqa12b0~.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/0lp_7whj-dhrj.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/02jvz4ipsieeg.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/chunks/turbopack-03_bijqn710jo.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/IUuVij9Ol4yGFGm_99WoB/_buildManifest.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/IUuVij9Ol4yGFGm_99WoB/_ssgManifest.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<script src="/_next/static/IUuVij9Ol4yGFGm_99WoB/_clientMiddlewareManifest.js" defer="" type="46210e8713f087f2d699c04e-text/javascript"></script>
12<style id="__jsx-ff99253234eb226f">pre.jsx-ff99253234eb226f{padding-top:2.5rem;position:relative}</style>
12<script data-cfasync="false" nonce="5bffc8bc-16ba-4c45-8bd6-8ef0d72d2fa3">try{(function(w,d){!function(kP,kQ,kR,kS){if(kP.zaraz)console.error("zaraz is loaded twice");else{kP[kR]=kP[kR]||{};kP[kR].executed=[];kP.zaraz={deferred:[],listeners:[]};kP.zaraz._v="20";kP.zaraz._n="5bffc8bc-16ba-4c45-8bd6-8ef0d72d2fa3";kP.zaraz.q=[];kP.zaraz._f=function(kT){return async function(){var kU=Array.prototype.slice.call(arguments);kP.zaraz.q.push({m:kT,a:kU})}};for(const kV of["track","set","debug"])kP.zaraz[kV]=kP.zaraz._f(kV);kP.zaraz.init=()=>{var kW=kQ.getElementsByTagName(kS)[0],kX=kQ.createElement(kS),kY=kQ.getElementsByTagName("title")[0];kY&&(kP[kR].t=kQ.getElementsByTagName("title")[0].text);kP[kR].x=Math.random();kP[kR].w=kP.screen.width;kP[kR].h=kP.screen.height;kP[kR].j=kP.innerHeight;kP[kR].e=kP.innerWidth;kP[kR].l=kP.location.href;kP[kR].r=kQ.referrer;kP[kR].k=kP.screen.colorDepth;kP[kR].n=kQ.characterSet;kP[kR].o=(new Date).getTimezoneOffset();if(kP.dataLayer)for(const kZ of Object.entries(Object.entries(dataLayer).reduce((k$,la)=>({...k$[1],...la[1]}),{})))zaraz.set(kZ[0],kZ[1],{scope:"page"});kP[kR].q=[];for(;kP.zaraz.q.length;){const lb=kP.zaraz.q.shift();kP[kR].q.push(lb)}kX.defer=!0;for(const lc of[localStorage,sessionStorage])Object.keys(lc||{}).filter(le=>le.startsWith("_zaraz_")).forEach(ld=>{try{kP[kR]["z_"+ld.slice(7)]=JSON.parse(lc.getItem(ld))}catch{kP[kR]["z_"+ld.slice(7)]=lc.getItem(ld)}});kX.referrerPolicy="origin";kX.src="/cdn-cgi/zaraz/s.js?z="+btoa(encodeURIComponent(JSON.stringify(kP[kR])));kW.parentNode.insertBefore(kX,kW)};["complete","interactive"].includes(kQ.readyState)?zaraz.init():kP.addEventListener("DOMContentLoaded",zaraz.init)}}(w,d,"zarazData","script");window.zaraz._p=async kH=>new Promise(kI=>{if(kH){kH.e&&kH.e.forEach(kJ=>{try{const kK=d.querySelector("script[nonce]"),kL=kK?.nonce||kK?.getAttribute("nonce"),kM=d.createElement("script");kL&&(kM.nonce=kL);kM.innerHTML=kJ;kM.onload=()=>{d.head.removeChild(kM)};d.head.appendChild(kM)}catch(kN){console.error(`Error executing script: ${kJ}\n`,kN)}});Promise.allSettled((kH.f||[]).map(kO=>fetch(kO[0],kO[1])))}kI()});zaraz._p({"e":["(function(w,d){})(window,document)"]});})(window,document)}catch(e){throw fetch("/cdn-cgi/zaraz/t"),e;};</script>
12</head><body class="bg-canvas font-body text-ink antialiased transition-colors"><div id="__next">
12<script type="46210e8713f087f2d699c04e-text/javascript">((e,i,s,u,m,a,l,h)=>{let d=document.documentElement,w=["light","dark"];function p(n){(Array.isArray(e)?e:[e]).forEach(y=>{let k=y==="class",S=k&&a?m.map(f=>a[f]||f):m;k?(d.classList.remove(...S),d.classList.add(a&&a[n]?a[n]:n)):d.setAttribute(y,n)}),R(n)}function R(n){h&&w.includes(n)&&(d.style.colorScheme=n)}function c(){return window.matchMedia("(prefers-color-scheme: dark)").matches?"dark":"light"}if(u)p(u);else try{let n=localStorage.getItem(i)||s,y=l&&n==="system"?c():n;p(y)}catch(n){}})("class","theme","dark",null,["light","dark"],null,true,true)</script>
12<header class="dark sticky top-0 z-50 transition-shadow [color-scheme:dark]"><a href="#skip-nav" class="rounded-none p-2 transition font-medium text-dark dark:text-white bg-dark dark:bg-dark group dark:hover:text-primary-300 focus:outline-none focus:ring focus:ring-primary-300 absolute left-4 top-4 -translate-y-16 focus:translate-y-0"><span class="transition-colors bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">Skip to main content</span></a><div class="border-b border-line bg-canvas/90 text-ink backdrop-blur transition-colors"><nav class="bg-transparent"><div class="mx-auto  w-full px-4 sm:px-6 lg:px-8"><div class="relative flex h-16 items-center justify-between"><div class="flex items-center"><a class="text:text-primary-100 flex text-center text-3xl font-bold" href="/"><svg width="160" height="53" viewBox="0 0 473 53" fill="currentcolor" xmlns="http://www.w3.org/2000/svg"><path d="M39.624 4.912C35.664 2.464 30.84 1.024 25.8 1.024C11.4 1.024 0.24 11.896 0.24 26.8C0.24 42.784 12.048 52.72 25.8 52.72C30.768 52.72 36.024 51.064 39.696 48.832C41.208 48.04 41.568 46.168 40.416 44.584C39.48 43.432 37.968 43.216 36.672 43.936C33.432 45.952 29.76 47.032 25.8 47.032C14.928 47.032 6.432 39.472 6.432 26.8C6.432 14.992 14.496 6.712 25.8 6.712C29.832 6.712 33.504 7.936 36.672 9.808C38.04 10.528 39.48 10.456 40.344 9.232C41.496 7.72 41.208 5.848 39.624 4.912ZM87.3263 49.192C87.3263 47.536 86.0303 46.312 84.3743 46.312H62.5583V4.48C62.5583 2.824 61.1903 1.6 59.6063 1.6C57.8063 1.6 56.5823 2.824 56.5823 4.48V49.048C56.5823 50.704 57.8063 52 59.4623 52H84.3743C86.0303 52 87.3263 50.848 87.3263 49.192ZM147.175 26.8C147.175 11.968 137.023 0.879997 122.911 0.879997C108.727 0.879997 98.6466 11.968 98.6466 26.8C98.6466 41.632 108.727 52.72 122.911 52.72C137.023 52.72 147.175 41.632 147.175 26.8ZM140.983 26.8C140.983 38.464 133.495 46.96 122.911 46.96C112.255 46.96 104.839 38.536 104.839 26.8C104.839 15.064 112.255 6.64 122.911 6.64C133.495 6.64 140.983 15.064 140.983 26.8ZM199.279 1.528C197.551 1.528 196.543 2.824 196.543 4.48V33.064C196.543 40.984 190.207 46.744 182.863 46.744C175.663 46.744 169.543 40.984 169.543 33.064V4.48C169.543 2.824 168.175 1.528 166.591 1.528C164.791 1.528 163.567 2.824 163.567 4.48V33.064C163.567 44.152 171.847 52.504 182.863 52.504C193.807 52.504 202.087 44.152 202.087 33.064V4.48C202.087 2.824 200.935 1.528 199.279 1.528ZM224.404 1.6C222.748 1.6 221.524 2.824 221.524 4.48V49.048C221.524 50.704 222.748 52 224.404 52H242.404C256.588 52 263.644 40.696 263.644 26.8C263.644 13.264 256.588 1.6 242.404 1.6H224.404ZM241.684 7.144C252.772 7.144 257.452 16.36 257.452 26.8C257.452 37.6 252.772 46.384 241.684 46.384H227.5V7.144H241.684ZM283.356 52C285.012 52 286.02 50.704 286.02 49.048V31.552H301.212C305.388 31.552 309.708 33.352 309.708 39.472C309.708 47.608 312.156 50.2 314.46 51.496C315.612 52.216 317.7 52.216 318.636 50.848C319.572 49.552 319.068 47.824 317.7 47.032C316.044 46.096 315.684 44.224 315.468 38.896C315.324 33.856 312.66 30.328 308.988 28.6C313.668 26.512 317.124 21.616 317.124 15.856C317.124 7.576 310.788 1.6 302.796 1.6H282.924C281.268 1.6 280.044 2.824 280.044 4.48V49.048C280.044 50.704 281.556 52 283.356 52ZM302.436 7.144C307.332 7.144 310.932 10.96 310.932 16C310.932 22.048 306.972 26.152 302.796 26.368H286.02V7.144H302.436ZM373.988 49.336C373.988 48.976 373.916 48.616 373.772 48.184L355.196 3.256C354.764 2.104 353.684 1.384 352.46 1.384C351.236 1.312 350.156 1.888 349.652 3.256L331.004 48.616C330.86 48.976 330.788 49.336 330.86 49.696C330.86 51.064 331.724 52.144 333.524 52.144C334.604 52.144 335.612 51.424 336.044 50.272L340.868 38.536H363.692L368.516 50.2C369.02 51.424 370.028 52.144 371.18 52.144C372.62 52.144 373.988 51.064 373.988 49.336ZM343.172 32.92L352.172 10.96L361.316 32.92H343.172ZM391.668 52C393.324 52 394.62 50.704 394.62 49.048V29.032H413.628C415.284 29.032 416.58 27.736 416.58 26.296C416.58 24.64 415.284 23.416 413.628 23.416H394.62V7.144H416.868C418.524 7.144 419.82 5.92 419.82 4.408C419.82 2.752 418.524 1.6 416.868 1.6H391.596C389.94 1.6 388.716 2.824 388.716 4.48V49.048C388.716 50.704 389.868 52 391.668 52ZM452.946 52C454.746 52 456.042 50.704 456.042 49.048V7.072H469.722C471.378 7.072 472.674 6.064 472.674 4.336C472.674 2.68 471.378 1.6 469.722 
121.6H436.242C434.586 1.6 433.362 2.68 433.362 4.336C433.362 5.992 434.586 7.072 436.242 7.072H449.922V49.048C449.922 50.704 451.218 52 452.946 52Z" fill="rgb(var(--tw-clr-primary-300))" stroke="rgb(var(--tw-clr-primary-300))"></path></svg></a></div><div class="hidden flex-1 justify-center min-[900px]:flex"><ul class="flex items-center space-x-6 text-base text-ink"><li><button type="button" class="rounded-none py-2 transition-colors text-ink group hover:text-pine"><span class="px-2 transition-colors">What we do</span></button></li><li><a class="rounded-none py-2 transition-colors text-ink group hover:text-pine" href="/case-studies"><span class="px-2 transition-colors">Our Work</span></a></li><li><button type="button" class="rounded-none py-2 transition-colors text-ink group hover:text-pine"><span class="px-2 transition-colors">Insights</span></button></li><li><a class="rounded-none py-2 transition-colors text-ink group hover:text-pine" href="/careers"><span class="px-2 transition-colors">Careers</span></a></li><li><button type="button" class="rounded-none py-2 transition-colors text-ink group hover:text-pine"><span class="px-2 transition-colors">About</span></button></li></ul></div><div class="flex max-w-[360px] items-center gap-2"><div class="hidden items-center gap-2 min-[900px]:flex"><div class="flex-shrink-0"><a href="/contact-us"> <button class="text-md rounded-none border border-line-strong bg-transparent px-4 py-2 font-semibold text-ink shadow-sm transition-colors hover:border-pine hover:text-pine">Contact Us</button></a></div><button class="rounded-none p-2 focus:outline-none md:p-2.5 border border-line text-ink-2 text-lg md:text-xl" aria-hidden="true" tabindex="-1" aria-label="Toggle color theme"><svg stroke="currentColor" fill="none" stroke-width="2" viewBox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><circle cx="12" cy="12" r="5"></circle><line x1="12" y1="1" x2="12" y2="3"></line><line x1="12" y1="21" x2="12" y2="23"></line><line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line><line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line><line x1="1" y1="12" x2="3" y2="12"></line><line x1="21" y1="12" x2="23" y2="12"></line><line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line><line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line></svg></button></div><div class="flex items-center gap-2 min-[900px]:hidden"><button class="rounded-none p-2 focus:outline-none md:p-2.5 border border-line text-ink-2 text-lg md:text-xl p-2" aria-hidden="true" tabindex="-1" aria-label="Toggle color theme"><svg stroke="currentColor" fill="none" stroke-width="2" viewBox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><circle cx="12" cy="12" r="5"></circle><line x1="12" y1="1" x2="12" y2="3"></line><line x1="12" y1="21" x2="12" y2="23"></line><line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line><line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line><line x1="1" y1="12" x2="3" y2="12"></line><line x1="21" y1="12" x2="23" y2="12"></line><line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line><line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line></svg></button><button type="button" class="rounded-none border border-line bg-surface p-2 text-ink shadow-sm hover:border-pine hover:text-pine focus:outline-none" aria-controls="mobile-menu" aria-expanded="false"><span class="sr-only">Open main menu</span><svg class="block h-6 w-6" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor" aria-hidden="true"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16"></path></svg></button></div></div></div></div></nav></div></header><div class="fixed inset-0 flex items-center justify-center bg-white transition-opacity dark:bg-dark"></div><div id="skip-nav"><main><section class="layout mt-8"><div class="mx-auto max-w-6xl"><div class="dark:border-gray-600"><p class="text-md font-semibold uppercase tracking-widest dark:text-gray-300">blog</p><h1 class="py-8 font-bold leading-tight md:text-4xl lg:text-5xl">LLM Observability: Monitoring Large Language Models</h1><div class="flex flex-col justify-between text-sm text-gray-600 dark:text-gray-300"><div class="flex flex-wrap items-center gap-x-3 gap-y-1"><a class="text-md font-bold uppercase tracking-widest text-gray-600 hover:text-primary-500 dark:text-gray-300 dark:hover:text-primary-300" href="/profile/rahul-agrawal">Rahul Agrawal</a><span class="text-gray-400">•</span><time dateTime="2025-10-24T00:00:00.000Z" class="text-md font-bold uppercase tracking-widest text-gray-600 dark:text-gray-300">published<!-- --> <!-- -->October 24, 2025</time></div></div></div><hr class="my-8 h-2 w-full border-0 bg-gradient-to-r from-primary-300 via-primary-400 to-primary-500"/><div class="grid w-full grid-cols-1 gap-8 lg:grid-cols-[1fr_250px]"><article class="mdx prose w-full max-w-4xl dark:prose-invert md:prose-lg lg:prose-xl 2xl:prose-lg"><h2 id="introduction"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#introduction"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Introduction</h2>
13<p>Large Language Models (LLMs) have revolutionized cloud-native AI, powering applications from support bots to analytics engines. However, scaling LLMs in production introduces new monitoring and compliance complexities. Effe
13ctive observability bridges the gap between research and real-world reliability, ensuring models remain performant, cost-efficient, and secure in dynamic environments.</p>
14<p>The world of AI operations is rapidly evolving beyond traditional monitoring approaches. As organizations deploy LLMs at scale, they face unique challenges: unpredictable inference costs, model drift detection, security compliance, and the need for real-time performance insights. This comprehensive guide explores the essential observability strategies and tools needed to successfully monitor LLMs in production.</p>
15<h2 id="why-does-observability-matter-for-llms"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#why-does-observability-matter-for-llms"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Why Does Observability Matter for LLMs?</h2>
16<p>LLMs operate on massive datasets, require high-performance compute/storage, and serve unpredictable user loads. Traditional monitoring tools fall short—comprehensive observability is essential for:</p>
17<ul>
18<li><strong>Preventing unexpected downtime and performance bottlenecks</strong></li>
19<li><strong>Tracking model drift, accuracy, and prompt performance</strong></li>
20<li><strong>Enforcing security, privacy, and compliance for sensitive data</strong></li>
21<li><strong>Controlling costs and scaling efficiently</strong></li>
22</ul>
23<p>Unlike traditional applications, LLMs present unique observability challenges including token-based pricing models, variable inference times, and the need to monitor both technical metrics and model quality metrics.</p>
24<h2 id="key-observability-pillars"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#key-observability-pillars"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Key Observability Pillars</h2>
25<h3 id="metrics-collection--telemetry"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#metrics-collection--telemetry"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Metrics Collection &amp; Telemetry</h3>
26<p>Capture request latency, throughput, prompt complexity, GPU/memory utilization, token counts, and user feedback. Use Prometheus and OpenTelemetry for collection, with Grafana for dashboards.</p>
27<h3 id="distributed-tracing"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#distributed-tracing"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Distributed Tracing</h3>
28<p>LLMs typically run as microservices (often gRPC/REST APIs). Distributed traces pinpoint bottlenecks and enable root cause analysis. OpenTelemetry Auto Instrumentation streamlines tracing integration.</p>
29<h3 id="health-checks--canary-deployments"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#health-checks--canary-deployments"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Health Checks &amp; Canary Deployments</h3>
30<p>Use proactive, Kubernetes-native health checks (Canary Checker) to validate output quality for every new LLM build. Automate rollback and staged rollouts based on observability signals.</p>
31<h3 id="security--compliance-monitoring"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#security--compliance-monitoring"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Security &amp; Compliance Monitoring</h3>
32<p>LLM pipelines should support encryption, secure logging, and integrate policy-as-code tools (Kyverno). Runtime monitoring (with Tetragon, Cilium Hubble) addresses in-memory threats and zero trust.</p>
33<h3 id="usage-drift-and-cost-tracking"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#usage-drift-and-cost-tracking"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Usage, Drift, and Cost Tracking</h3>
34<p>Monitor resource/hardware usage and track model drift with vector databases and open-source logging tools (Loki, ELK). Implement usage-based billing for accurate cost attribution.</p>
35<h2 id="llm-observability-tools--platforms"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#llm-observability-tools--platforms"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>LLM Observability Tools &amp; Platforms</h2>
36<p>The ecosystem for LLM observability continues to grow, with several powerful commercial and open source solutions:</p>
37<table><thead><tr><th>
37Tool</th><th>Type</th><th>Key Features</th><th>Pricing/Freemium</th><th>Pros</th><th>Cons</th><th>Self-host Option</th></tr></thead><tbody><tr><td><strong>LangSmith</strong></td><td>Paid</td><td>LLM tracing, cost analytics, feedback, works natively with Langchain</td><td>Free tier up to 5,000 traces/month; paid SaaS tiers available; self-hosting only in enterprise</td><td>Robust integration with Langchain, manual/auto evals, SaaS simplicity</td><td>No open source backend, self-host for enterprise only, vendor lock-in risk</td><td>Limited (Enterprise)</td></tr><tr><td><strong>Lunary</strong></td><td>Free/Open Src</td><td>Model tracking, categorization (Radar), prompt analytics</td><td>Free up to 1,000 events/day; open source under Apache 2.0</td><td>Completely open source, can self-host for privacy, easy integration</td><td>Event limit on free cloud, limited advanced analytics compared to commercial</td><td>Yes</td></tr><tr><td><strong>Phoenix (Arize)</strong></td><td>Free/Open Src</td><td>Tracing, evaluation, hallucination detection</td><td>Free (ELv2 license), no full hosted SaaS; paid AX Pro starts at $50/m</td><td>Works out-of-box with LlamaIndex/LangChain/OpenAI, OTel compatible, built-in evals</td><td>Paid plan for hosted, may require infra management for self-host</td><td>Yes</td></tr><tr><td><strong>Langfuse</strong></td><td>Free/Open Src</td><td>Session tracking, tracing, evaluation, OpenTelemetry backend</td><td>Free self-host up to 50k events/mo; $59/m for 100k events (managed), $199/mo Pro</td><td>Most complete OSS feature set, SOC2 compliant, wide integrations</td><td>Hosted plans have data limits, advanced features priced</td><td>Yes</td></tr><tr><td><strong>Helicone</strong></td><td>Paid &amp; OSS</td><td>LLM monitoring, prompt management, caching, cost tracker</td><td>Free up to 10,000 requests; $20/m Pro, $200/m Team</td><td>Caching reduces API costs, SDK and proxy integration, security features</td><td>Limited requests in free; higher tiers unlock retention/features</td><td>Yes</td></tr><tr><td><strong>Grafana Cloud</strong></td><td>Paid/Open Src</td><td>Visualization, dashboards, multi-source metrics/logs/traces</td><td>Free up to 100GB data (3 active users); Pro $19/user/mo; Enterprise $8/user/mo</td><td>Flexible, massive plugin ecosystem, custom dashboards, active community</td><td>Usage tiers can get expensive, learning curve for advanced use</td><td>Yes</td></tr><tr><td><strong>Traceloop OpenLLMetry</strong></td><td>Free/Open Src</td><td>OTel style tracing, multi-tool compatibility</td><td>Free, open source (Apache 2.0), backend also free</td><td>Universal OTel-compatible, integrates with Langchain, LlamaIndex</td><td>Infra setup required, less advanced analytics</td><td>Yes</td></tr></tbody></table>
38<p>Recent surveys highlight these platforms&#x27; support for token counting, semantic traceability, drift detection, and GPU observation.</p>
39<h2 id="hands-on-demo-langfuse-in-action"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#hands-on-demo-langfuse-in-action"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Hands-On Demo: Langfuse in Action</h2>
40<p>To demonstrate LLM observability in practice, let&#x27;s walk through a complete setup using Langfuse—one of the most comprehensive open-source solutions. This demo showcases real-world tracing, session management, and analytics for LLM applications.</p>
41<h3 id="setting-up-langfuse-cloud"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#setting-up-langfuse-cloud"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>
41Setting Up Langfuse Cloud</h3>
42<p>Langfuse offers both self-hosted and cloud options. For this demo, we&#x27;ll use the cloud version for rapid setup:</p>
43<ol>
44<li><strong>Create Account</strong>: Visit <a target="_blank" rel="noopener noreferrer" href="https://cloud.langfuse.com" class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 cursor-newtab"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">cloud.langfuse.com</span></a> and sign up for a free account</li>
45<li><strong>Get API Keys</strong>: Navigate to Settings → API Keys and copy your Public Key and Secret Key</li>
46<li><strong>Configure Environment</strong>: Set up your environment variables:</li>
47</ol>
48<pre class="jsx-ff99253234eb226f language-bash"><code class="language-bash code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token assign-left variable">LANGFUSE_PUBLIC_KEY</span><span class="token operator">=</span>pk-lf-your-key-here
49</span><span class="code-line"><span class="token assign-left variable">LANGFUSE_SECRET_KEY</span><span class="token operator">=</span>sk-lf-your-key-here
50</span><span class="code-line"><span class="token assign-left variable">LANGFUSE_HOST</span><span class="token operator">=</span>https://cloud.langfuse.com
51</span><span class="code-line"><span class="token assign-left variable">OPENAI_API_KEY</span><span class="token operator">=</span>your-openai-api-key-here
52</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">bash</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
53<h3 id="demo-applications"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#demo-applications"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Demo Applications</h3>
54<p>We&#x27;ve created three comprehensive demo scenarios that showcase different aspects of LLM observability:</p>
55<h4 id="1-simple-chat-interface"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#1-simple-chat-interface"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>1. Simple Chat Interface</h4>
56<p>A basic conversational AI that demonstrates fundamental tracing concepts:</p>
57<pre class="jsx-ff99253234eb226f language-python"><code class="language-python code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token keyword">from</span> langfuse <span class="token keyword">import</span> Langfuse
58</span><span class="code-line"><span class="token keyword">import</span> openai
59</span><span class="code-line">
60</span><span class="code-line">langfuse <span class="token operator">=</span> Langfuse<span class="token punctuation">(</span>
61</span><span class="code-line">    public_key<span class="token operator">=</span><span class="token string">&quot;your-public-key&quot;</span><span class="token punctuation">,</span>
62</span><span class="code-line">    secret_key<span class="token operator">=</span><span class="token string">&quot;your-secret-key&quot;</span><span class="token punctuation">,</span>
63</span><span class="code-line">    host<span class="token operator">=</span><span class="token string">&quot;https://cloud.langfuse.com&quot;</span>
64</span><span class="code-line"><span class="token punctuation">)</span>
65</span><span class="code-line">
66</span><span class="code-line"><span class="token keyword">def</span> <span class="token function">chat_with_llm</span><span class="token punctuation">(</span>user_message<span class="token punctuation">:</span> <span class="token builtin">str</span><span class="token punctuation">,</span> model<span class="token punctuation">:</span> <span class="token builtin">str</span> <span class="token operator">=</span> <span class="token string">&quot;gpt-3.5-turbo&quot;</span><span class="token punctuation">)</span> <span class="token operator">-</span><span class="token operator">&gt;</span> <span class="token builtin">str</span><span class="token punctuation">:</span>
67</span><span class="code-line">    <span class="token comment"># Start a span for this chat completion</span>
68</span><span class="code-line">    span <span class="token operator">=</span> langfuse<span class="token punctuation">.</span>start_span<span class="token punctuation">(</span>name<span class="token operator">=</span><span class="token string">&quot;chat_completion&quot;</span><span class="token punctuation">,</span> <span class="token builtin">input</span><span class="token operator">=</span>user_message<span class="token punctuation">)</span>
69</span><span class="code-line">    <span class="token keyword">try</span><span class="token punctuation">:</span>
70</span><span class="code-line">
70        <span class="token comment"># Start a generation observation</span>
71</span><span class="code-line">        generation <span class="token operator">=</span> langfuse<span class="token punctuation">.</span>start_observation<span class="token punctuation">(</span>
72</span><span class="code-line">            name<span class="token operator">=</span><span class="token string">&quot;llm_call&quot;</span><span class="token punctuation">,</span>
73</span><span class="code-line">            model<span class="token operator">=</span>model<span class="token punctuation">,</span>
74</span><span class="code-line">            <span class="token builtin">input</span><span class="token operator">=</span>user_message<span class="token punctuation">,</span>
75</span><span class="code-line">            as_type<span class="token operator">=</span><span class="token string">&quot;generation&quot;</span>
76</span><span class="code-line">        <span class="token punctuation">)</span>
77</span><span class="code-line">
78</span><span class="code-line">        response <span class="token operator">=</span> openai<span class="token punctuation">.</span>chat<span class="token punctuation">.</span>completions<span class="token punctuation">.</span>create<span class="token punctuation">(</span>
79</span><span class="code-line">            model<span class="token operator">=</span>model<span class="token punctuation">,</span>
80</span><span class="code-line">            messages<span class="token operator">=</span><span class="token punctuation">[</span>
81</span><span class="code-line">                <span class="token punctuation">{</span><span class="token string">&quot;role&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;system&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;content&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;You are a helpful AI assistant.&quot;</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
82</span><span class="code-line">                <span class="token punctuation">{</span><span class="token string">&quot;role&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;user&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;content&quot;</span><span class="token punctuation">:</span> user_message<span class="token punctuation">}</span>
83</span><span class="code-line">            <span class="token punctuation">]</span><span class="token punctuation">,</span>
84</span><span class="code-line">            temperature<span class="token operator">=</span><span class="token number">0.7</span><span class="token punctuation">,</span>
85</span><span class="code-line">            max_tokens<span class="token operator">=</span><span class="token number">500</span>
86</span><span class="code-line">        <span class="token punctuation">)</span>
87</span><span class="code-line">
88</span><span class="code-line">        result <span class="token operator">=</span> response<span class="token punctuation">.</span>choices<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>message<span class="token punctuation">.</span>content
89</span><span class="code-line">        generation<span class="token punctuation">.</span>update<span class="token punctuation">(</span>output<span class="token operator">=</span>result<span class="token punctuation">)</span>
90</span><span class="code-line">        generation<span class="token punctuation">.</span>end<span class="token punctuation">(</span><span class="token punctuation">)</span>
91</span><span class="code-line">        span<span class="token punctuation">.</span>update<span class="token punctuation">(</span>output<span class="token operator">=</span>result<span class="token punctuation">)</span>
92</span><span class="code-line">        span<span class="token punctuation">.</span>end<span class="token punctuation">(</span><span class="token punctuation">)</span>
93</span><span class="code-line">
94</span><span class="code-line">
94        <span class="token keyword">return</span> result
95</span><span class="code-line">    <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e<span class="token punctuation">:</span>
96</span><span class="code-line">        error_msg <span class="token operator">=</span> <span class="token string-interpolation"><span class="token string">f&quot;Sorry, I encountered an error: </span><span class="token interpolation"><span class="token punctuation">{</span><span class="token builtin">str</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span><span class="token punctuation">}</span></span><span class="token string">&quot;</span></span>
97</span><span class="code-line">        generation<span class="token punctuation">.</span>update<span class="token punctuation">(</span>output<span class="token operator">=</span>error_msg<span class="token punctuation">,</span> level<span class="token operator">=</span><span class="token string">&quot;ERROR&quot;</span><span class="token punctuation">)</span>
98</span><span class="code-line">        generation<span class="token punctuation">.</span>end<span class="token punctuation">(</span><span class="token punctuation">)</span>
99</span><span class="code-line">        span<span class="token punctuation">.</span>update<span class="token punctuation">(</span>output<span class="token operator">=</span>error_msg<span class="token punctuation">,</span> level<span class="token operator">=</span><span class="token string">&quot;ERROR&quot;</span><span class="token punctuation">)</span>
100</span><span class="code-line">        span<span class="token punctuation">.</span>end<span class="token punctuation">(</span><span class="token punctuation">)</span>
101</span><span class="code-line">        <span class="token keyword">return</span> error_msg
102</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">python</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
103<h4 id="2-rag-retrieval-augmented-generation-pipeline"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#2-rag-retrieval-augmented-generation-pipeline"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>2. RAG (Retrieval Augmented Generation) Pipeline</h4>
104<p>A more complex workflow showing document retrieval, context assembly, and generation:</p>
105<pre class="jsx-ff99253234eb226f language-python"><code class="language-python code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token keyword">def</span> <span class="token function">rag_pipeline</span><span class="token punctuation">(</span>query<span class="token punctuation">:</span> <span class="token builtin">str</span><span class="token punctuation">)</span> <span class="token operator">-</span><span class="token operator">&gt;</span> Dict<span class="token punctuation">[</span><span class="token builtin">str</span><span class="token punctuation">,</span> <span class="token builtin">any</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
106</span><span class="code-line">    <span class="token comment"># Start main span for RAG pipeline</span>
107</span><span class="code-line">    trace <span class="token operator">=</span> langfuse<span class="token punctuation">.</span>start_span<span class="token punctuation">(</span>name<span class="token operator">=</span><span class="token string">&quot;rag_pipeline&quot;</span><span class="token punctuation">,</span> <span class="token builtin">input</span><span class="token operator">=</span>query<span class="token punctuation">)</span>
108</span><span class="code-line">
109</span><span class="code-line">    <span class="token keyword">try</span><span class="token punctuation">:</span>
110</span><span class="code-line">
110        <span class="token comment"># Step 1: Retrieve relevant documents</span>
111</span><span class="code-line">        documents <span class="token operator">=</span> retrieve_relevant_documents<span class="token punctuation">(</span>query<span class="token punctuation">,</span> trace<span class="token operator">=</span>trace<span class="token punctuation">)</span>
112</span><span class="code-line">
113</span><span class="code-line">        <span class="token comment"># Step 2: Assemble context</span>
114</span><span class="code-line">        context <span class="token operator">=</span> assemble_context<span class="token punctuation">(</span>documents<span class="token punctuation">,</span> query<span class="token punctuation">,</span> trace<span class="token operator">=</span>trace<span class="token punctuation">)</span>
115</span><span class="code-line">
116</span><span class="code-line">        <span class="token comment"># Step 3: Generate answer</span>
117</span><span class="code-line">        answer <span class="token operator">=</span> generate_answer<span class="token punctuation">(</span>context<span class="token punctuation">,</span> trace<span class="token operator">=</span>trace<span class="token punctuation">)</span>
118</span><span class="code-line">
119</span><span class="code-line">        result <span class="token operator">=</span> <span class="token punctuation">{</span>
120</span><span class="code-line">            <span class="token string">&quot;query&quot;</span><span class="token punctuation">:</span> query<span class="token punctuation">,</span>
121</span><span class="code-line">            <span class="token string">&quot;retrieved_documents&quot;</span><span class="token punctuation">:</span> documents<span class="token punctuation">,</span>
122</span><span class="code-line">            <span class="token string">&quot;context&quot;</span><span class="token punctuation">:</span> context<span class="token punctuation">,</span>
123</span><span class="code-line">            <span class="token string">&quot;answer&quot;</span><span class="token punctuation">:</span> answer
124</span><span class="code-line">        <span class="token punctuation">}</span>
125</span><span class="code-line">
126</span><span class="code-line">        trace<span class="token punctuation">.</span>update<span class="token punctuation">(</span>name<span class="token operator">=</span><span class="token string">&quot;rag_pipeline&quot;</span><span class="token punctuation">,</span> output<span class="token operator">=</span>result<span class="token punctuation">,</span> metadata<span class="token operator">=</span><span class="token punctuation">{</span><span class="token string">&quot;doc_count&quot;</span><span class="token punctuation">:</span> <span class="token builtin">len</span><span class="token punctuation">(</span>documents<span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
127</span><span class="code-line">        <span class="token keyword">return</span> result
128</span><span class="code-line">    <span class="token keyword">finally</span><span class="token punctuation">:</span>
129</span><span class="code-line">        trace<span class="token punctuation">.</span>end<span class="token punctuation">(</span><span class="token punctuation">)</span>
130</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">python</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
131<h4 id="3-multi-step-workflow"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#3-multi-step-workflow"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>3. Multi-Step Workflow</h4>
132<p>
132Demonstrates complex conversation chains and problem-solving workflows with nested spans and observations.</p>
133<h3 id="langfuse-dashboard-overview"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#langfuse-dashboard-overview"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Langfuse Dashboard Overview</h3>
134<p>Once you run the demo applications, the Langfuse dashboard provides comprehensive insights into your LLM operations:</p>
135<p><img src="https://res.cloudinary.com/dfee67kdq/image/upload/v1761306007/blogs/llm-observability/505232688-29705943-1671-4793-8a51-7ecfff4aeb82_e3hg4r.webp" alt="Langfuse Latency Dashboard"/>
136<em>Langfuse dashboard showing latency metrics and performance insights from our demo applications</em></p>
137<h3 id="trace-detail-view"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#trace-detail-view"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Trace Detail View</h3>
138<p>Individual traces reveal the complete request flow with nested spans, timing breakdown, and token usage:</p>
139<p><img src="https://res.cloudinary.com/dfee67kdq/image/upload/v1761306007/blogs/llm-observability/505232696-7be0c243-bf39-4576-a86f-6080adf03377_zwk0lx.webp" alt="Langfuse Trace Details"/>
140<em>Detailed trace view showing nested spans for RAG pipeline: document retrieval → context assembly → LLM generation</em></p>
141<h3 id="analytics-and-cost-tracking"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#analytics-and-cost-tracking"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Analytics and Cost Tracking</h3>
142<p>Built-in analytics track token usage, costs, and performance over time:</p>
143<p><img src="https://res.cloudinary.com/dfee67kdq/image/upload/v1761306007/blogs/llm-observability/505232672-808b17dc-3add-4be1-aee1-11747afde9ce_lqeszt.webp" alt="Langfuse Cost Dashboard"/>
144<em>Analytics dashboard displaying token usage, cost analysis, and performance metrics across different models</em></p>
145<h3 id="key-benefits-demonstrated"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#key-benefits-demonstrated"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Key Benefits Demonstrated</h3>
146<p>This hands-on demo showcases several critical LLM observability capabilities:</p>
147<ul>
148<li><strong>Distributed Tracing</strong>: Complete visibility into multi-step LLM workflows</li>
149<li><strong>Performance Monitoring</strong>: Real-time latency, throughput, and error tracking</li>
150<li><strong>Cost Management</strong>: Token usage and cost attribution across different models</li>
151<li><strong>Error Handling</strong>: Comprehensive error tracking and debugging information</li>
152</ul>
153<h3 id="running-the-demo"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#running-the-demo"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Running the Demo</h3>
154<p>To try this demo yourself:</p>
155<pre class="jsx-ff99253234eb226f language-bash"><code class="language-bash code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token comment"># Clone the demo repository</span>
156</span><span class="code-line"><span class="token function">git</span> clone https://github.com/cloudraftio/langfuse-demo.git
157</span><span class="code-line"><span class="token builtin class-name">cd</span> langfuse-demo
158</span><span class="code-line">
159</span><span class="code-line"><span class="token comment"># Install dependencies</span>
160</span><span class="code-line">pip <span class="token function">install</span> <span class="token parameter variable">-r</span> requirements.txt
161</span><span class="code-line">
162</span><span class="code-line"><span class="token comment"># Configure environment</span>
163</span><span class="code-line"><span class="token function">cp</span> env.example .env
164</span><span class="code-line"><span class="token comment"># Edit .env with your API keys</span>
165</span><span class="code-line">
166</span><span class="code-line"><span class="token comment"># Run all demos</span>
167</span><span class="code-line">python run_all_demos.py
168</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">bash</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
169<p>The demo generates realistic traces across different scenarios, providing a comprehensive view of LLM observability in action.</p>
170<h2 id="implementation-guide-llm-monitoring-on-kubernetes"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#implementation-guide-llm-monitoring-on-kubernetes"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Implementation Guide: LLM Monitoring on Kubernetes</h2>
171<p>Deploying and observing LLMs in Kubernetes requires integrating metrics collection, tracing, logging, alert
171ing, security, and visualization. Below is a detailed how-to guide with working code snippets and configurations:</p>
172<h3 id="1-exporting-llm-metrics-with-prometheus"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#1-exporting-llm-metrics-with-prometheus"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>1. Exporting LLM Metrics with Prometheus</h3>
173<p>Expose inference request counts and latency metrics from your LLM service. Here&#x27;s a minimal FastAPI example with Prometheus integration:</p>
174<pre class="jsx-ff99253234eb226f language-python"><code class="language-python code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token keyword">from</span> fastapi <span class="token keyword">import</span> FastAPI<span class="token punctuation">,</span> Request
175</span><span class="code-line"><span class="token keyword">from</span> prometheus_client <span class="token keyword">import</span> Counter<span class="token punctuation">,</span> Histogram<span class="token punctuation">,</span> make_asgi_app
176</span><span class="code-line"><span class="token keyword">import</span> time
177</span><span class="code-line">
178</span><span class="code-line">app <span class="token operator">=</span> FastAPI<span class="token punctuation">(</span><span class="token punctuation">)</span>
179</span><span class="code-line">
180</span><span class="code-line">REQUEST_COUNT <span class="token operator">=</span> Counter<span class="token punctuation">(</span><span class="token string">&quot;llm_requests&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;Number of LLM requests&quot;</span><span class="token punctuation">)</span>
181</span><span class="code-line">REQUEST_LATENCY <span class="token operator">=</span> Histogram<span class="token punctuation">(</span><span class="token string">&quot;llm_request_latency_seconds&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;Request latency in seconds&quot;</span><span class="token punctuation">)</span>
182</span><span class="code-line">
183</span><span class="code-line"><span class="token decorator annotation punctuation">@app<span class="token punctuation">.</span>post</span><span class="token punctuation">(</span><span class="token string">&quot;/generate&quot;</span><span class="token punctuation">)</span>
184</span><span class="code-line"><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">generate</span><span class="token punctuation">(</span>req<span class="token punctuation">:</span> Request<span class="token punctuation">)</span><span class="token punctuation">:</span>
185</span><span class="code-line">    start <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
186</span><span class="code-line">    data <span class="token operator">=</span> <span class="token keyword">await</span> req<span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span>
187</span><span class="code-line">    <span class="token comment"># Simulate call to LLM model</span>
188</span><span class="code-line">    response <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">&quot;output&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;Example LLM output&quot;</span><span class="token punctuation">}</span>
189</span><span class="code-line">    REQUEST_COUNT<span class="token punctuation">.</span>inc<span class="token punctuation">(</span><span class="token punctuation">)</span>
190</span><span class="code-line">    REQUEST_LATENCY<span class="token punctuation">.</span>observe<span class="token punctuation">(</span>time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">-</span> start<span class="token punctuation">)</span>
191</span><span class="code-line">    <span class="token keyword">return</span> response
192</span><span class="code-line">
193</span><span class="code-line"><span class="token comment"># Serve metrics at /metrics for Prometheus scraping</span>
194</span><span class="code-line">app<span class="token punctuation">.</span>
194mount<span class="token punctuation">(</span><span class="token string">&quot;/metrics&quot;</span><span class="token punctuation">,</span> make_asgi_app<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
195</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">python</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
196<p><strong>Key takeaways:</strong></p>
197<ul>
198<li>Metrics include request count and request latency</li>
199<li>Prometheus scrapes /metrics endpoint automatically</li>
200</ul>
201<h3 id="2-adding-distributed-tracing-with-opentelemetry"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#2-adding-distributed-tracing-with-opentelemetry"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>2. Adding Distributed Tracing with OpenTelemetry</h3>
202<p>Enable transparent tracing of requests through auto instrumentation:</p>
203<pre class="jsx-ff99253234eb226f language-python"><code class="language-python code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token keyword">from</span> fastapi <span class="token keyword">import</span> FastAPI
204</span><span class="code-line"><span class="token keyword">from</span> opentelemetry<span class="token punctuation">.</span>instrumentation<span class="token punctuation">.</span>fastapi <span class="token keyword">import</span> FastAPIInstrumentor
205</span><span class="code-line"><span class="token keyword">from</span> opentelemetry<span class="token punctuation">.</span>sdk<span class="token punctuation">.</span>resources <span class="token keyword">import</span> Resource
206</span><span class="code-line"><span class="token keyword">from</span> opentelemetry<span class="token punctuation">.</span>sdk<span class="token punctuation">.</span>trace <span class="token keyword">import</span> TracerProvider
207</span><span class="code-line"><span class="token keyword">from</span> opentelemetry<span class="token punctuation">.</span>exporter<span class="token punctuation">.</span>jaeger<span class="token punctuation">.</span>thrift <span class="token keyword">import</span> JaegerExporter
208</span><span class="code-line"><span class="token keyword">from</span> opentelemetry<span class="token punctuation">.</span>sdk<span class="token punctuation">.</span>trace<span class="token punctuation">.</span>export <span class="token keyword">import</span> BatchSpanProcessor
209</span><span class="code-line">
210</span><span class="code-line">app <span class="token operator">=</span> FastAPI<span class="token punctuation">(</span><span class="token punctuation">)</span>
211</span><span class="code-line">
212</span><span class="code-line"><span class="token comment"># Configure tracer provider</span>
213</span><span class="code-line">trace_provider <span class="token operator">=</span> TracerProvider<span class="token punctuation">(</span>resource<span class="token operator">=</span>Resource<span class="token punctuation">.</span>create<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">&quot;service.name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;llm-service&quot;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
214</span><span class="code-line">jaeger_exporter <span class="token operator">=</span> JaegerExporter<span class="token punctuation">(</span>
215</span><span class="code-line">    agent_host_name<span class="token operator">=</span><span class="token string">&quot;localhost&quot;</span><span class="token punctuation">,</span>
216</span><span class="code-line">    agent_port<span class="token operator">=</span><span class="token number">6831</span><span class="token punctuation">,</span>
217</span><span class="code-line"><span class="token punctuation">)</span>
218</span><span class="code-line">trace_provider<span class="token punctuation">.</span>
218add_span_processor<span class="token punctuation">(</span>BatchSpanProcessor<span class="token punctuation">(</span>jaeger_exporter<span class="token punctuation">)</span><span class="token punctuation">)</span>
219</span><span class="code-line">
220</span><span class="code-line"><span class="token comment"># Set tracer provider globally</span>
221</span><span class="code-line"><span class="token keyword">from</span> opentelemetry <span class="token keyword">import</span> trace
222</span><span class="code-line">trace<span class="token punctuation">.</span>set_tracer_provider<span class="token punctuation">(</span>trace_provider<span class="token punctuation">)</span>
223</span><span class="code-line">
224</span><span class="code-line"><span class="token comment"># Instrument FastAPI app</span>
225</span><span class="code-line">FastAPIInstrumentor<span class="token punctuation">.</span>instrument_app<span class="token punctuation">(</span>app<span class="token punctuation">)</span>
226</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">python</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
227<p><strong>Notes:</strong></p>
228<ul>
229<li>Sends traces to Jaeger (could be any other tracer backend)</li>
230<li>Captures detailed performance and call path info</li>
231</ul>
232<h3 id="3-defining-prometheus-alert-rules-for-latency"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#3-defining-prometheus-alert-rules-for-latency"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>3. Defining Prometheus Alert Rules for Latency</h3>
233<p>Alert on unusually high LLM response latency to proactively catch slowed inference:</p>
234<pre class="jsx-ff99253234eb226f language-yaml"><code class="language-yaml code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> monitoring.coreos.com/v1
235</span><span class="code-line"><span class="token key atrule">kind</span><span class="token punctuation">:</span> PrometheusRule
236</span><span class="code-line"><span class="token key atrule">metadata</span><span class="token punctuation">:</span>
237</span><span class="code-line">  <span class="token key atrule">name</span><span class="token punctuation">:</span> llm<span class="token punctuation">-</span>alerts
238</span><span class="code-line"><span class="token key atrule">spec</span><span class="token punctuation">:</span>
239</span><span class="code-line">  <span class="token key atrule">groups</span><span class="token punctuation">:</span>
240</span><span class="code-line">    <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> llm.rules
241</span><span class="code-line">      <span class="token key atrule">rules</span><span class="token punctuation">:</span>
242</span><span class="code-line">        <span class="token punctuation">-</span> <span class="token key atrule">alert</span><span class="token punctuation">:</span> HighLLMLatency
243</span><span class="code-line">          <span class="token key atrule">expr</span><span class="token punctuation">:</span> histogram_quantile(0.95<span class="token punctuation">,</span> sum(rate(llm_request_latency_seconds_bucket<span class="token punctuation">[</span>5m<span class="token punctuation">]</span>)) by (le)) <span class="token punctuation">&gt;</span> 2
244</span><span class="code-line">
244          <span class="token key atrule">for</span><span class="token punctuation">:</span> 5m
245</span><span class="code-line">          <span class="token key atrule">labels</span><span class="token punctuation">:</span>
246</span><span class="code-line">            <span class="token key atrule">severity</span><span class="token punctuation">:</span> warning
247</span><span class="code-line">          <span class="token key atrule">annotations</span><span class="token punctuation">:</span>
248</span><span class="code-line">            <span class="token key atrule">summary</span><span class="token punctuation">:</span> <span class="token string">&#x27;LLM inference latency at 95th percentile is greater than 2 seconds&#x27;</span>
249</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">yaml</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
250<h3 id="4-centralized-log-aggregation"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#4-centralized-log-aggregation"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>4. Centralized Log Aggregation</h3>
251<p>Use Fluentd or Promtail to ship container logs to Loki for easy search and parsing. Example Promtail config snippet:</p>
252<pre class="jsx-ff99253234eb226f language-yaml"><code class="language-yaml code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token key atrule">server</span><span class="token punctuation">:</span>
253</span><span class="code-line">  <span class="token key atrule">http_listen_port</span><span class="token punctuation">:</span> <span class="token number">9080</span>
254</span><span class="code-line"><span class="token key atrule">clients</span><span class="token punctuation">:</span>
255</span><span class="code-line">  <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//loki<span class="token punctuation">:</span>3100/loki/api/v1/push
256</span><span class="code-line"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
257</span><span class="code-line">  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> kubernetes<span class="token punctuation">-</span>pods
258</span><span class="code-line">    <span class="token key atrule">pipeline_stages</span><span class="token punctuation">:</span>
259</span><span class="code-line">      <span class="token punctuation">-</span> <span class="token key atrule">docker</span><span class="token punctuation">:</span> <span class="token punctuation">{</span><span class="token punctuation">}</span>
260</span><span class="code-line">    <span class="token key atrule">kubernetes_sd_configs</span><span class="token punctuation">:</span>
261</span><span class="code-line">      <span class="token punctuation">-</span> <span class="token key atrule">role</span><span class="token punctuation">:</span> pod
262</span><span class="code-line">    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
263</span><span class="code-line">      <span class="token punctuation">-</span> <span class="token key atrule">source_labels</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>__meta_kubernetes_pod_label_app<span class="token punctuation">]</span>
264</span><span class="code-line">
264        <span class="token key atrule">action</span><span class="token punctuation">:</span> keep
265</span><span class="code-line">        <span class="token key atrule">regex</span><span class="token punctuation">:</span> llm<span class="token punctuation">-</span>service
266</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">yaml</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
267<h3 id="5-kubernetes-native-health-checks-using-canary-checker"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#5-kubernetes-native-health-checks-using-canary-checker"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>5. Kubernetes Native Health Checks using Canary Checker</h3>
268<p>Install and configure Canary Checker to run quality assurance tests on model output before new versions go live:</p>
269<ul>
270<li>Write proactive test scripts for key prompt responses</li>
271<li>Define health check probes that measure model accuracy over test queries</li>
272<li>Automate canary deployments and rollbacks based on health status</li>
273</ul>
274<h3 id="6-security--compliance-integration"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#6-security--compliance-integration"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>6. Security &amp; Compliance Integration</h3>
275<p>Protect observability data and runtime environments with:</p>
276<ul>
277<li><strong>Kyverno</strong>: Policy enforcement for namespaces, secrets, and logs</li>
278<li><strong>Tetragon</strong>: eBPF runtime monitoring for suspicious system calls</li>
279<li><strong>Cilium Hubble</strong>: Network observability at packet and service granularity</li>
280</ul>
281<p>Example Kyverno policy to restrict access to metrics endpoint:</p>
282<pre class="jsx-ff99253234eb226f language-yaml"><code class="language-yaml code-highlight" data-code-type="code-block"><div class="overflow-x-auto"><span class="code-line"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> kyverno.io/v1
283</span><span class="code-line"><span class="token key atrule">kind</span><span class="token punctuation">:</span> ClusterPolicy
284</span><span class="code-line"><span class="token key atrule">metadata</span><span class="token punctuation">:</span>
285</span><span class="code-line">  <span class="token key atrule">name</span><span class="token punctuation">:</span> restrict<span class="token punctuation">-</span>metrics<span class="token punctuation">-</span>access
286</span><span class="code-line"><span class="token key atrule">spec</span><span class="token punctuation">:</span>
287</span><span class="code-line">  <span class="token key atrule">rules</span><span class="token punctuation">:</span>
288</span><span class="code-line">    <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> block<span class="token punctuation">-</span>public<span class="token punctuation">-</span>metrics
289</span><span class="code-line">      <span class="token key atrule">match</span><span class="token punctuation">:</span>
290</span><span class="code-line">
290        <span class="token key atrule">resources</span><span class="token punctuation">:</span>
291</span><span class="code-line">          <span class="token key atrule">kinds</span><span class="token punctuation">:</span>
292</span><span class="code-line">            <span class="token punctuation">-</span> Service
293</span><span class="code-line">          <span class="token key atrule">namespaces</span><span class="token punctuation">:</span>
294</span><span class="code-line">            <span class="token punctuation">-</span> default
295</span><span class="code-line">      <span class="token key atrule">validate</span><span class="token punctuation">:</span>
296</span><span class="code-line">        <span class="token key atrule">message</span><span class="token punctuation">:</span> <span class="token string">&#x27;Metrics service must not be publicly accessible.&#x27;</span>
297</span><span class="code-line">        <span class="token key atrule">pattern</span><span class="token punctuation">:</span>
298</span><span class="code-line">          <span class="token key atrule">spec</span><span class="token punctuation">:</span>
299</span><span class="code-line">            <span class="token key atrule">type</span><span class="token punctuation">:</span> ClusterIP
300</span></div><div class="absolute top-0 left-6 rounded-b-none border border-t-0 border-gray-600 px-3 py-1"><span class="select-none font-medium text-pine dark:text-primary-300">yaml</span></div><button class="absolute top-2 right-2 hidden rounded border border-gray-600 p-2 text-lg transition-colors hover:bg-gray-700 md:block"><svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 20 20" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z"></path><path d="M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z"></path></svg></button></code></pre>
301<h3 id="7-visualization-with-grafana"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#7-visualization-with-grafana"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>7. Visualization with Grafana</h3>
302<p>Connect Grafana to Prometheus, Loki, and Jaeger:</p>
303<ul>
304<li>Create dashboards to display request latency trends, error rates, and token usage per inference</li>
305<li>Use traced request flows to drill into problematic LLM interactions</li>
306<li>Set alerts in Grafana for SLA breaches</li>
307</ul>
308<h2 id="what-llm-observability-cant-do"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#what-llm-observability-cant-do"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>What LLM Observability Can&#x27;t Do</h2>
309<p>While powerful, LLM observability has limitations:</p>
310<ul>
311<li><strong>Model Quality Assessment</strong>: Observability tools can detect performance issues but cannot automatically assess the quality or accuracy of model outputs</li>
312<li><strong>Context-Aware Monitoring</strong>: Understanding the semantic meaning of prompts and responses requires specialized AI evaluation tools</li>
313<li><strong>Real-time Model Drift Detection</strong>: While tools can track metrics, detecting subtle model drift often requires domain expertise and manual analysis</li>
314<li><strong>Cross-Model Comparison</strong>: Comparing performance across different LLM providers or model versions requires custom analysis beyond standard observability tools</li>
315</ul>
316<p>In these cases, observability acts as a foundation, providing the data needed for deeper analysis and human expertise.</p>
317<div class="rounded-none bg-primary-300 px-4 py-4 shadow-md"><div class="mx-16 p-0 text-center"><p class="m-0 pt-2 text-3xl font-bold leading-tight text-gray-800">Need help with LLM observability?</p><p class="-mt-1 pt-0 text-center text-xl italic  text-gray-800">Our team specializes in implementing comprehensive observability solutions for AI workloads. Get expert guidance on monitoring your LLM deployments.</p><div class=" mx-auto flex items-center justify-center py-1"><a target="_blank" rel="noopener noreferrer" href="https://cal.link/anjul" aria-label="Meet the CEO — Book a meeting with Anjul" class="cursor-newtab"><div class="group flex items-center gap-3 rounded-none border border-line bg-surface px-3 py-2 shadow-sm transition-colors duration-150 hover:border-line-strong hover:bg-surface-alt cursor-pointer focus:outline-none focus-visible:ring-2 focus-visible:ring-pine" role="group" title="Meet the CEO"><div class="flex-shrink-0 overflow-hidden rounded-none border border-line shadow-sm" style="width:48px;height:48px;min-width:48px;min-height:48px;display:flex;align-items:center;justify-content:center" aria-hidden="false"><img src="https://res.cloudinary.com/dfee67kdq/image/upload/v1710944213/author/anjul_sahu_yxw8eh.webp" alt="Anjul Sahu — CEO" class="block h-full w-full object-cover" style="display:block" width="48" height="48"/></div><div class="min-w-0"><div class="truncate text-sm font-semibold leading-tight text-ink">Meet the CEO</div><div class="truncate text-xs text-ink-3">Book a meeting with Anjul</div></div></div></a></div></div></div>
318<h2 id="cost-and-limitations"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#cost-and-limitations"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Cost and Limitations</h2>
319<ul>
320<li><strong>Open Source Solutions</strong>: Free to use but require significant engineering effort for setup, maintenance, and customization</li>
321<li><strong>Commercial Platforms</strong>: Provide rapid deployment and advanced features but involve ongoing subscription costs</li>
322<li><strong>
322Infrastructure Overhead</strong>: Running observability tools in Kubernetes requires additional compute and storage resources</li>
323<li><strong>Data Retention</strong>: Long-term storage of observability data can become expensive, especially for high-volume LLM applications</li>
324<li><strong>Learning Curve</strong>: Effective use of observability tools requires understanding both the tools and LLM-specific monitoring requirements</li>
325</ul>
326<h2 id="conclusion"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0 hash-anchor" href="/blog/llm-observability#conclusion"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent"><span class="icon icon-link"></span></span></a>Conclusion</h2>
327<p>LLM observability is now a mission-critical capability for any team running generative AI in production—whether on open source frameworks or managed SaaS platforms. Free and open source solutions excel at privacy, flexibility, and customization, enabling technical teams to build tailored monitoring stacks and maintain control over their infrastructure. Paid commercial platforms, meanwhile, shine through rapid onboarding, advanced analytics, enterprise-grade security, managed scaling, and deep integrations with LLM agent ecosystems.</p>
328<p>The best choice depends on your organization&#x27;s scale, budget, compliance needs, and engineering bandwidth. For startups or research environments, open source often offers rapid innovation and complete data sovereignty. For enterprises or mission-critical deployments, commercial observability tools deliver rich feature sets, robust support, and compliance at scale.</p>
329<p>Ultimately, combining or layering both approaches—using open source for experimentation and commercial solutions for high-traffic production—can bring organizations the best of both worlds: agility, security, and operational excellence.</p>
330<p>If you are looking for a <a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0" href="/ai-agent-development"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">custom AI solution</span></a> to integrate in your cloud native stack, we can help you build a tailored solution that meets your specific needs. <a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0" href="/contact-us"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">Contact us today</span></a> to learn more.</p></article><aside class="py-4"><div class="sticky top-36"><div id="toc-container" class="max-h-[calc(100vh-200px)]  overflow-auto pb-4"><h3 class="sticky top-0 py-2 text-lg font-semibold text-gray-900  dark:text-gray-100">Table of Contents</h3><div class="mt-4 flex flex-col space-y-2 text-sm"><p class="text-xs text-gray-500 dark:text-gray-400">No headings found</p></div></div></div></aside></div><div class="mt-12 flex flex-row gap-x-4"><a target="_blank" rel="noopener noreferrer" href="https://twitter.com/intent/tweet?url=https%3A%2F%2Fwww.cloudraft.io%2Fblog%2Fllm-observability&amp;text=I%20just%20read%20an%20article%20about%20LLM%20Observability%3A%20Monitoring%20Large%20Language%20Models%20by%20%40cloudraftio.%0A%0A" class="inline-flex rounded px-2 py-2 font-bold border border-gray-300 shadow-sm dark:border-gray-600 focus:outline-none focus-visible:ring focus-visible:ring-primary-300 scale-100 hover:scale-[1.03] active:scale-[0.97] motion-safe:transform-gpu motion-reduce:hover:scale-100 motion-reduce:hover:brightness-90 transition duration-100 animate-shadow bg-white text-gray-800 disabled:bg-gray-200 dark:bg-dark dark:text-gray-100 dark:disabled:bg-gray-700 items-center gap-2 cursor-newtab"><svg stroke="currentColor" fill="currentColor" stroke-width="0" role="img" viewBox="0 0 24 24" class="text-[1.2em] text-sky-500" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M21.543 7.104c.015.211.015.423.015.636 0 6.507-4.954 14.01-14.01 14.01v-.003A13.94 13.94 0 0 1 0 19.539a9.88 9.88 0 0 0 7.287-2.041 4.93 4.93 0 0 1-4.6-3.42 4.916 4.916 0 0 0 2.223-.084A4.926 4.926 0 0 1 .96 9.167v-.062a4.887 4.887 0 0 0 2.235.616A4.928 4.928 0 0 1 1.67 3.148 13.98 13.98 0 0 0 11.82 8.292a4.929 4.929 0 0 1 8.39-4.49 9.868 9.868 
3300 0 0 3.128-1.196 4.941 4.941 0 0 1-2.165 2.724A9.828 9.828 0 0 0 24 4.555a10.019 10.019 0 0 1-2.457 2.549z"></path></svg>Tweet this article</a><a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.cloudraft.io%2Fblog%2Fllm-observability" class="inline-flex rounded px-2 py-2 font-bold border border-gray-300 shadow-sm dark:border-gray-600 focus:outline-none focus-visible:ring focus-visible:ring-primary-300 scale-100 hover:scale-[1.03] active:scale-[0.97] motion-safe:transform-gpu motion-reduce:hover:scale-100 motion-reduce:hover:brightness-90 transition duration-100 animate-shadow bg-white text-gray-800 disabled:bg-gray-200 dark:bg-dark dark:text-gray-100 dark:disabled:bg-gray-700 items-center gap-2 cursor-newtab"><svg stroke="currentColor" fill="currentColor" stroke-width="0" role="img" viewBox="0 0 24 24" class="text-[1.2em] text-sky-700" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M20.447 20.452h-3.554v-5.569c0-1.328-.027-3.037-1.852-3.037-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85 3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433c-1.144 0-2.063-.926-2.063-2.065 0-1.138.92-2.063 2.063-2.063 1.14 0 2.064.925 2.064 2.063 0 1.139-.925 2.065-2.064 2.065zm1.782 13.019H3.555V9h3.564v11.452zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0h.003z"></path></svg>Share on LinkedIn</a></div><figure class="mt-12"></figure><div class="mt-20"><h2><span class="transition-colors bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">Other posts that you might like</span></h2><ul class="mt-4 grid gap-4 sm:grid-cols-2 xl:grid-cols-3"><li class="flex flex-col overflow-hidden border border-gray-300 bg-white dark:border-gray-600 dark:bg-dark scale-100 hover:scale-[1.02] active:scale-[0.97] motion-safe:transform-gpu transition-all duration-200 hover:border-primary-300/40 hover:shadow-md hover:shadow-primary-300/5 motion-reduce:hover:scale-100 animate-shadow"><a class="flex flex-1 flex-col focus:outline-none focus-visible:ring focus-visible:ring-primary-300" href="/blog/top-open-source-logging-tools"><div class="relative overflow-hidden"><figure class="pointer-events-none overflow-hidden"><div style="position:relative;height:0;padding-top:40%;cursor:default" class="img-blur"><div class="absolute inset-0 flex items-center justify-center"><img alt="Top Open Source Logging Tools for Cloud Native Observability" title="Top Open Source Logging Tools for Cloud Native Observability" loading="eager" width="1200" height="480" decoding="async" data-nimg="1" style="color:transparent" srcSet="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fopen_source_logging_tools_vyqzdc.webp&amp;w=1200&amp;q=75 1x, /_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fopen_source_logging_tools_vyqzdc.webp&amp;w=3840&amp;q=75 2x" src="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fopen_source_logging_tools_vyqzdc.webp&amp;w=3840&amp;q=75"/></div></div><figcaption class="text-center"></figcaption></figure></div><div class="flex flex-1 flex-col p-4"><p class="text-xs font-medium text-gray-500 dark:text-gray-400">Jan 07, 2025</p><h3 class="mt-2 text-lg font-bold leading-snug text-gray-800 dark:text-gray-100 md:text-xl">Top Open Source Logging Tools for Cloud Native Observability</h3><p class="line-clamp-2 mt-2 flex-1 text-sm leading-relaxed text-gray-600 dark:text-gray-400">Explore the best open source logging tools tailored for cloud native observability, empowering developers and operators to monitor and debug applications effectively.</p><div class="mt-4 flex flex-wrap gap-1.5"><span class="inline-block bg-gray-100 px-2 py-0.5 text-[10px] font-medium uppercase tracking-wider text-gray-600 dark:bg-gray-800 dark:text-gray-400">Observability</span></div></div></a></li><li class="flex flex-col overflow-hidden border border-gray-300 bg-white dark:border-gray-600 dark:bg-dark scale-100 hover:scale-[1.02] active:scale-[0.97] motion-safe:transform-gpu transition-all duration-200 hover:border-primary-300/40 hover:shadow-md hover:shadow-
330primary-300/5 motion-reduce:hover:scale-100 animate-shadow"><a class="flex flex-1 flex-col focus:outline-none focus-visible:ring focus-visible:ring-primary-300" href="/blog/implement-compliance-first-observability-opentelemetry"><div class="relative overflow-hidden"><figure class="pointer-events-none overflow-hidden"><div style="position:relative;height:0;padding-top:40%;cursor:default" class="img-blur"><div class="absolute inset-0 flex items-center justify-center"><img alt="Implementing Compliance-first Observability with OpenTelemetry" title="Implementing Compliance-first Observability with OpenTelemetry" loading="eager" width="1200" height="480" decoding="async" data-nimg="1" style="color:transparent" srcSet="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2FBanner_dc8sql.webp&amp;w=1200&amp;q=75 1x, /_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2FBanner_dc8sql.webp&amp;w=3840&amp;q=75 2x" src="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2FBanner_dc8sql.webp&amp;w=3840&amp;q=75"/></div></div><figcaption class="text-center"></figcaption></figure></div><div class="flex flex-1 flex-col p-4"><p class="text-xs font-medium text-gray-500 dark:text-gray-400">Jun 17, 2025</p><h3 class="mt-2 text-lg font-bold leading-snug text-gray-800 dark:text-gray-100 md:text-xl">Implementing Compliance-first Observability with OpenTelemetry</h3><p class="line-clamp-2 mt-2 flex-1 text-sm leading-relaxed text-gray-600 dark:text-gray-400">Learn how to design observability pipelines that enforce data protection, support compliance regulations, and ensure secure telemetry using OpenTelemetry.</p><div class="mt-4 flex flex-wrap gap-1.5"><span class="inline-block bg-gray-100 px-2 py-0.5 text-[10px] font-medium uppercase tracking-wider text-gray-600 dark:bg-gray-800 dark:text-gray-400">Observability</span></div></div></a></li><li class="flex flex-col overflow-hidden border border-gray-300 bg-white dark:border-gray-600 dark:bg-dark scale-100 hover:scale-[1.02] active:scale-[0.97] motion-safe:transform-gpu transition-all duration-200 hover:border-primary-300/40 hover:shadow-md hover:shadow-primary-300/5 motion-reduce:hover:scale-100 animate-shadow hidden xl:block"><a class="flex flex-1 flex-col focus:outline-none focus-visible:ring focus-visible:ring-primary-300" href="/blog/scaling-prometheus-with-thanos"><div class="relative overflow-hidden"><figure class="pointer-events-none overflow-hidden"><div style="position:relative;height:0;padding-top:40%;cursor:default" class="img-blur"><div class="absolute inset-0 flex items-center justify-center"><img alt="Scaling Prometheus with Thanos" title="Scaling Prometheus with Thanos" loading="eager" width="1200" height="480" decoding="async" data-nimg="1" style="color:transparent" srcSet="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fthanos-cloudraft_vvkxri.webp&amp;w=1200&amp;q=75 1x, /_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fthanos-cloudraft_vvkxri.webp&amp;w=3840&amp;q=75 2x" src="/_next/image?url=https%3A%2F%2Fres.cloudinary.com%2Fdfee67kdq%2Fimage%2Fupload%2Fq_auto%2Cf_auto%2Fbanner%2Fthanos-cloudraft_vvkxri.webp&amp;w=3840&amp;q=75"/></div></div><figcaption class="text-center"></figcaption></figure></div><div class="flex flex-1 flex-col p-4"><p class="text-xs font-medium text-gray-500 dark:text-gray-400">Jul 22, 2024</p><h3 class="mt-2 text-lg font-bold leading-snug text-gray-800 dark:text-gray-100 md:text-xl">Scaling Prometheus with Thanos</h3><p class="line-clamp-2 mt-2 flex-1 text-sm leading-relaxed text-gray-600 dark:text-gray-400"> Learn how to scale Prometheus using Thanos for long-term storage, global view metrics, and improved performance in large-scale Kubernetes environments.</p><div class="mt-4 flex flex-wrap gap-1.5"><span class="inline-block bg-gray-100 px-2 py-0.5 text-[10px] font-medium uppercase tracking-wider text-gray-600 dark:bg-gray-800 dark:text-gray-400">Observability</span></div></div></a></li></ul></div><div class="rounded-none border-line border p-4 mt-12"><h3><span class="transition-colors bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">Enjoying this post?</span></h3><p class="mt-2 text-ink-2">Get our posts directly in your inbox.</p><form class="mt-2 flex items-end gap-2"><div class="flex-grow"><label class="text-sm font-medium" for="email">Subscribe to our newsletter</label><input name="email" class="w-full rounded-none bg-surface text-ink border border-line text-sm md:text-base placeholder:text-ink-3 focus:border-pine focus:outline-none focus:ring-0" type="email" placeholder="Email" required=""/></div><button type="submit" class="rounded-none border border-line-strong bg-surface px-4 py-2 font-bold text-ink focus:outline-none focus-visible:ring-2 focus-visible:ring-pine transition-colors duration-100 hover:border-pine disabled:border-line disabled:bg-surface-alt disabled:text-ink-3 disabled:transform-none disabled:cursor-not-allowed">Subscribe</button></form><p class="mt-3 text-sm text-ink-2"></p></div><div class="mt-8 flex flex-col items-start gap-4 md:flex-row-reverse md:justify-between"><a class="animated-underline custom-link inline-flex items-center font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 border-b border-dotted border-dark hover:border-black/0" href="/blog"><span class="bg-gradient-to-tr from-pine to-aqua bg-clip-text text-transparent">← Back to blog</span></a></div></div></section></main></div><footer class="dark mt-8 bg-dark pb-4 [color-scheme:dark]"><main class="layout flex flex-col items-center border-t border-line pt-8"><div class="container grid grid-cols-1 gap-8 md:grid-cols-2 lg:grid-cols-4"><div class="col-span-1 space-y-4 md:col-span-2 lg:col-span-1"><svg width="160" height="53" viewBox="0 0 473 53" fill="currentcolor" xmlns="http://www.w3.org/2000/svg"><path d="M39.624 4.912C35.664 2.464 30.84 1.024 25.8 1.024C11.4 1.024 0.24 11.896 0.24 26.8C0.24 42.784 12.048 52.72 25.8 52.72C30.768 52.72 36.024 51.064 39.696 48.832C41.208 48.04 41.568 46.168 40.416 44.584C39.48 43.432 37.968 43.216 36.672 43.936C33.432 45.952 29.76 47.032 25.8 47.032C14.928 47.032 6.432 39.472 6.432 26.8C6.432 14.992 14.496 6.712 25.8 6.712C29.832 6.712 33.504 7.936 36.672 9.808C38.04 10.528 39.48 10.456 40.344 9.232C41.496 7.72 41.208 5.848 39.624 4.912ZM87.3263 49.192C87.3263 47.536 86.0303 46.312 84.3743 46.312H62.5
330583V4.48C62.5583 2.824 61.1903 1.6 59.6063 1.6C57.8063 1.6 56.5823 2.824 56.5823 4.48V49.048C56.5823 50.704 57.8063 52 59.4623 52H84.3743C86.0303 52 87.3263 50.848 87.3263 49.192ZM147.175 26.8C147.175 11.968 137.023 0.879997 122.911 0.879997C108.727 0.879997 98.6466 11.968 98.6466 26.8C98.6466 41.632 108.727 52.72 122.911 52.72C137.023 52.72 147.175 41.632 147.175 26.8ZM140.983 26.8C140.983 38.464 133.495 46.96 122.911 46.96C112.255 46.96 104.839 38.536 104.839 26.8C104.839 15.064 112.255 6.64 122.911 6.64C133.495 6.64 140.983 15.064 140.983 26.8ZM199.279 1.528C197.551 1.528 196.543 2.824 196.543 4.48V33.064C196.543 40.984 190.207 46.744 182.863 46.744C175.663 46.744 169.543 40.984 169.543 33.064V4.48C169.543 2.824 168.175 1.528 166.591 1.528C164.791 1.528 163.567 2.824 163.567 4.48V33.064C163.567 44.152 171.847 52.504 182.863 52.504C193.807 52.504 202.087 44.152 202.087 33.064V4.48C202.087 2.824 200.935 1.528 199.279 1.528ZM224.404 1.6C222.748 1.6 221.524 2.824 221.524 4.48V49.048C221.524 50.704 222.748 52 224.404 52H242.404C256.588 52 263.644 40.696 263.644 26.8C263.644 13.264 256.588 1.6 242.404 1.6H224.404ZM241.684 7.144C252.772 7.144 257.452 16.36 257.452 26.8C257.452 37.6 252.772 46.384 241.684 46.384H227.5V7.144H241.684ZM283.356 52C285.012 52 286.02 50.704 286.02 49.048V31.552H301.212C305.388 31.552 309.708 33.352 309.708 39.472C309.708 47.608 312.156 50.2 314.46 51.496C315.612 52.216 317.7 52.216 318.636 50.848C319.572 49.552 319.068 47.824 317.7 47.032C316.044 46.096 315.684 44.224 315.468 38.896C315.324 33.856 312.66 30.328 308.988 28.6C313.668 26.512 317.124 21.616 317.124 15.856C317.124 7.576 310.788 1.6 302.796 1.6H282.924C281.268 1.6 280.044 2.824 280.044 4.48V49.048C280.044 50.704 281.556 52 283.356 52ZM302.436 7.144C307.332 7.144 310.932 10.96 310.932 16C310.932 22.048 306.972 26.152 302.796 26.368H286.02V7.144H302.436ZM373.988 49.336C373.988 48.976 373.916 48.616 373.772 48.184L355.196 3.256C354.764 2.104 353.684 1.384 352.46 1.384C351.236 1.312 350.156 1.888 349.652 3.256L331.004 48.616C330.86 48.976 330.788 49.336 330.86 49.696C330.86 51.064 331.724 52.144 333.524 52.144C334.604 52.144 335.612 51.424 336.044 50.272L340.868 38.536H363.692L368.516 50.2C369.02 51.424 370.028 52.144 371.18 52.144C372.62 52.144 373.988 51.064 373.988 49.336ZM343.172 32.92L352.172 10.96L361.316 32.92H343.172ZM391.668 52C393.324 52 394.62 50.704 394.62 49.048V29.032H413.628C415.284 29.032 416.58 27.736 416.58 26.296C416.58 24.64 415.284 23.416 413.628 23.416H394.62V7.144H416.868C418.524 7.144 419.82 5.92 419.82 4.408C419.82 2.752 418.524 1.6 416.868 1.6H391.596C389.94 1.6 388.716 2.824 388.716 4.48V49.048C388.716 50.704 389.868 52 391.668 52ZM452.946 52C454.746 52 456.042 50.704 456.042 49.048V7.072H469.722C471.378 7.072 472.674 6.064 472.674 4.336C472.674 2.68 471.378 1.6 469.722 1.6H436.242C434.586 1.6 433.362 2.68 433.362 4.336C433.362 5.992 434.586 7.072 436.242 7.072H449.922V49.048C449.922 50.704 451.218 52 452.946 52Z" fill="rgb(var(--tw-clr-primary-300))" stroke="rgb(var(--tw-clr-primary-300))"></path></svg><p class="text-sm leading-relaxed text-ink-2">Scale your business by leveraging AI and cloud-native technologies and modern platform engineering practices.</p><div class="flex flex-row justify-start pt-2"><div class="flex items-center gap-4"><div class="" style="display:inline"><button class="rounded-none align-middle focus:outline-none focus-visible:ring focus-visible:ring-primary-300" aria-label="Copy email address [email protected]"><svg stroke="currentColor" fill="none" stroke-width="2" viewBox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" class="h-6 w-6 text-ink-3 hover:text-pine" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M4 4h16c1.1 0 2 .9 2 2v12c0 1.1-.9 2-2 2H4c-1.1 0-2-.9-2-2V6c0-1.1.9-2 2-2z"></path><polyline points="22,6 12,13 2,6"></polyline></svg></button></div><div class="" style="display:inline"><a target="_blank" rel="noopener noreferrer" href="https://github.com/cloudraftio" aria-label="CloudRaft on Github" class="rounded-none focus:outline-none focus-visible:ring focus-visible:ring-primary-300 cursor-newtab"><svg stroke="currentColor" fill="currentColor" stroke-width="0" role="img" viewBox="0 0 24 24" class="h-5 w-5 text-ink-3 hover:text-pine" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12"></path></svg></a></div><div class="" style="display:inline"><a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/company/cloudraft" aria-label="CloudRaft on Linkedin" class="rounded-none focus:outline-none focus-visible:ring focus-visible:ring-primary-300 cursor-newtab"><svg stroke="currentColor" fill="currentColor" stroke-width="0" role="img" viewBox="0 0 24 24" class="h-5 w-5 text-ink-3 hover:text-pine" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M20.447 20.452h-3.554v-5.569c0-1.328-.027-3.037-1.852-3.037-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85 3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433c-1.144 0-2.063-.926-2.063-2.065 0-1.138.92-2.063 2.063-2.063 1.14 0 2.064.925 2.064 2.063 0 1.139-.925 2.065-2.064 2.065zm1.782 13.019H3.555V9h3.564v11.452zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0h.003z"></path></svg></a></div><div class="" style="display:inline"><a target="_blank" rel="noopener noreferrer" href="https://x.com/cloudraftio" aria-label="CloudRaft on Twitter" class="rounded-none focus:outline-none focus-visible:ring focus-visible:ring-primary-300 cursor-newtab"><svg stroke="currentColor" fill="currentColor" stroke-width="0" role="img" viewBox="0 0 24 24" class="h-5 w-5 text-ink-3 hover:text-pine" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M21.543 7.104c.015.211.015.423.015.636 0 6.507-4.954 14.01-14.01 14.01v-.003A13.94 13.94 0 0 1 0 19.539a9.88 9.88 0 0 0 7.287-2.041 4.93 4.93 0 0 1-4.6-3.42 4.916 4.916 0 0 0 2.223-.084A4.926 4.926 0 0 1 .96 9.167v-.062a4.887 4.887 0 0 0 2.235.616A4.928 4.928 0 0 1 1.67 3.148 13.98 13.98 0 0 0 11.82 8.292a4.929 4.929 0 0 1 8.39-4.49 9.868 9.868 
3300 0 0 3.128-1.196 4.941 4.941 0 0 1-2.165 2.724A9.828 9.828 0 0 0 24 4.555a10.019 10.019 0 0 1-2.457 2.549z"></path></svg></a></div><div class="" style="display:inline"><a target="_blank" rel="noopener noreferrer" href="https://www.cloudraft.io/rss.xml" aria-label="Subscribe to our blog RSS feed" class="rounded-none text-sm font-medium focus:outline-none focus-visible:ring focus-visible:ring-primary-300 dark:text-gray-200 cursor-newtab"><svg stroke="currentColor" fill="currentColor" stroke-width="0" role="img" viewBox="0 0 24 24" class="h-5 w-5 text-ink-3 hover:text-pine" aria-hidden="true" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><path d="M19.199 24C19.199 13.467 10.533 4.8 0 4.8V0c13.165 0 24 10.835 24 24h-4.801zM3.291 17.415c1.814 0 3.293 1.479 3.293 3.295 0 1.813-1.485 3.29-3.301 3.29C1.47 24 0 22.526 0 20.71s1.475-3.294 3.291-3.295zM15.909 24h-4.665c0-6.169-5.075-11.245-11.244-11.245V8.09c8.727 0 15.909 7.184 15.909 15.91z"></path></svg></a></div></div></div><div class="pt-4"><h2 class="mb-2 text-sm font-medium text-pine">Subscribe to the monthly Newsletter</h2><div class="rounded-none border-line"><p class="mt-2 text-ink-2"></p><form class="mt-2 flex items-end gap-2"><div class="flex-grow"><input name="email" class="w-full rounded-none bg-surface text-ink border border-line text-sm md:text-base placeholder:text-ink-3 focus:border-pine focus:outline-none focus:ring-0" type="email" placeholder="Email" required=""/></div><button type="submit" class="rounded-none border border-line-strong bg-surface px-4 py-2 font-bold text-ink focus:outline-none focus-visible:ring-2 focus-visible:ring-pine transition-colors duration-100 hover:border-pine disabled:border-line disabled:bg-surface-alt disabled:text-ink-3 disabled:transform-none disabled:cursor-not-allowed">Subscribe</button></form><p class="mt-3 text-sm text-ink-2"></p></div></div><div class="border-t border-line pt-6"><div class="grid grid-cols-1 gap-6 text-ink-2 md:grid-cols-2 lg:gap-12"><div class="text-left"><div class="mb-3 flex items-center"><span class="mr-2 text-lg">🇺🇸</span><h3 class="text-base font-semibold">United States</h3></div><div class="space-y-0.5 text-sm"><p class="font-medium">CloudRaft, Inc.</p><p>Lewes, Delaware 19958</p></div></div><div class="text-left"><div class="mb-3 flex items-center"><span class="mr-2 text-lg">🇮🇳</span><h3 class="text-base font-semibold">India</h3></div><div class="space-y-0.5 text-sm"><p class="font-medium">CloudRaft Technologies Pvt Ltd</p><p>Indore, MP 452018</p></div></div></div></div></div><div class="col-span-1 space-y-4"><h4 class="text-sm font-medium uppercase tracking-wider text-ink">Services</h4><div class="space-y-1"><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/observability-consulting">Observability Consulting</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/platform-engineering">Platform Engineering</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/kubernetes-consulting">Kubernetes Consulting</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/ai-cloud-consulting">AI Cloud Consulting</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/devops-consulting">DevOps Consulting</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/postgresql-consulting">PostgreSQL Consulting</a></div></div><div class="col-span-1 space-y-4"><h4 class="text-sm font-medium uppercase tracking-wider text-ink">Resources</h4><div class="space-y-1"><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/blog">Blog</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/case-studies">Case Studies</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/talks">Talks</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/observability-pricing-calculator">Pricing Calculator</a></div></div><div class="col-span-1 space-y-4"><h4 class="text-sm font-medium uppercase tracking-wider text-ink">Company</h4><div class="space-y-1"><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/company">About Us</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/contact-us">Contact</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/careers">Careers</a><a class="block py-1 text-sm text-ink-2 transition-colors hover:text-pine" href="/partners">Partners</a></div></div></div><div class="mt-8 flex flex-wrap items-center justify-center gap-x-6 gap-y-2 pb-4"><p class="text-sm text-ink-2">© CloudRaft <!-- -->2026</p><a class="text-sm text-ink-2 hover:text-pine" href="/privacy-terms">Privacy &amp; Terms</a><a class="text-sm text-ink-2 hover:text-pine" href="/sitemap.xml">Sitemap</a></div></main></footer></div>
330<script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"code":"var Component=(()=\u003e{var h=Object.create;var i=Object.defineProperty;var m=Object.getOwnPropertyDescriptor;var u=Object.getOwnPropertyNames;var N=Object.getPrototypeOf,k=Object.prototype.hasOwnProperty;var g=(a,e)=\u003e()=\u003e(e||a((e={exports:{}}).exports,e),e.exports),y=(a,e)=\u003e{for(var s in e)i(a,s,{get:e[s],enumerable:!0})},t=(a,e,s,l)=\u003e{if(e\u0026\u0026typeof e==\"object\"||typeof e==\"function\")for(let c of u(e))!k.call(a,c)\u0026\u0026c!==s\u0026\u0026i(a,c,{get:()=\u003ee[c],enumerable:!(l=m(e,c))||l.enumerable});return a};var f=(a,e,s)=\u003e(s=a!=null?h(N(a)):{},t(e||!a||!a.__esModule?i(s,\"default\",{value:a,enumerable:!0}):s,a)),b=a=\u003et(i({},\"__esModule\",{value:!0}),a);var o=g((M,r)=\u003e{r.exports=_jsx_runtime});var L={};y(L,{default:()=\u003ep,frontmatter:()=\u003ev});var n=f(o()),v={title:\"LLM Observability: Monitoring Large Language Models\",publishedAt:\"2025-10-24\",description:\"A comprehensive guide to implementing observability for Large Language Models in production environments, covering monitoring, tracing, security, and cost optimization strategies.\",englishOnly:\"true\",banner:\"llm-observability\",tags:\"Observability\",author:\"Rahul Agrawal\",keywords:\"LLM Observability, AI Monitoring, Large Language Models, Production AI, AI Observability, LLM Monitoring, AI Operations, Model Drift, AI Security, AI Cost Optimization, OpenTelemetry, Prometheus, Grafana\",img:\"https://res.cloudinary.com/dfee67kdq/image/upload/v1761306667/author/rahul_rnsjpe.webp\",featuredImage:\"background/abstract_3.avif\"};function d(a){let e={a:\"a\",code:\"code\",em:\"em\",h2:\"h2\",h3:\"h3\",h4:\"h4\",img:\"img\",li:\"li\",ol:\"ol\",p:\"p\",pre:\"pre\",span:\"span\",strong:\"strong\",table:\"table\",tbody:\"tbody\",td:\"td\",th:\"th\",thead:\"thead\",tr:\"tr\",ul:\"ul\",...a.components},{BlogCTA:s}=e;return s||w(\"BlogCTA\",!0),(0,n.jsxs)(n.Fragment,{children:[(0,n.jsxs)(e.h2,{id:\"introduction\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#introduction\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Introduction\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Large Language Models (LLMs) have revolutionized cloud-native AI, powering applications from support bots to analytics engines. However, scaling LLMs in production introduces new monitoring and compliance complexities. Effective observability bridges the gap between research and real-world reliability, ensuring models remain performant, cost-efficient, and secure in dynamic environments.\"}),`\n`,(0,n.jsx)(e.p,{children:\"The world of AI operations is rapidly evolving beyond traditional monitoring approaches. As organizations deploy LLMs at scale, they face unique challenges: unpredictable inference costs, model drift detection, security compliance, and the need for real-time performance insights. This comprehensive guide explores the essential observability strategies and tools needed to successfully monitor LLMs in production.\"}),`\n`,(0,n.jsxs)(e.h2,{id:\"why-does-observability-matter-for-llms\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#why-does-observability-matter-for-llms\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Why Does Observability Matter for LLMs?\"]}),`\n`,(0,n.jsx)(e.p,{children:\"LLMs operate on massive datasets, require high-performance compute/storage, and serve unpredictable user loads. Traditional monitoring tools fall short\\u2014comprehensive observability is essential for:\"}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsx)(e.li,{children:(0,n.jsx)(e.strong,{children:\"Preventing unexpected downtime and performance bottlenecks\"})}),`\n`,(0,n.jsx)(e.li,{children:(0,n.jsx)(e.strong,{children:\"Tracking model drift, accuracy, and prompt performance\"})}),`\n`,(0,n.jsx)(e.li,{children:(0,n.jsx)(e.strong,{children:\"Enforcing security, privacy, and compliance for sensitive data\"})}),`\n`,(0,n.jsx)(e.li,{children:(0,n.jsx)(e.strong,{children:\"Controlling costs and scaling efficiently\"})}),`\n`]}),`\n`,(0,n.jsx)(e.p,{children:\"Unlike traditional applications, LLMs present unique observability challenges including token-based pricing models, variable inference times, and the need to monitor both technical metrics and model quality metrics.\"}),`\n`,(0,n.jsxs)(e.h2,{id:\"key-observability-pillars\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#key-observability-pillars\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Key Observability Pillars\"]}),`\n`,(0,n.jsxs)(e.h3,{id:\"metrics-collection--telemetry\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#metrics-collection--telemetry\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Metrics Collection \u0026 Telemetry\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Capture request latency, throughput, prompt complexity, GPU/memory utilization, token counts, and user feedback. Use Prometheus and OpenTelemetry for collection, with Grafana for dashboards.\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"distributed-tracing\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#distributed-tracing\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Distributed Tracing\"]}),`\n`,(0,n.jsx)(e.p,{children:\"LLMs typically run as microservices (often gRPC/REST APIs). Distributed traces pinpoint bottlenecks and enable root cause analysis. OpenTelemetry Auto Instrumentation streamlines tracing integration.\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"health-checks--canary-deployments\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#health-checks--canary-deployments\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Health Checks \u0026 Canary Deployments\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Use proactive, Kubernetes-native health checks (Canary Checker) to validate output quality for every new LLM build. Automate rollback and staged rollouts based on observability signals.\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"security--compliance-monitoring\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#security--compliance-monitoring\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Security \u0026 Compliance Monitoring\"]}),`\n`,(0,n.jsx)(e.p,{children:\"LLM pipelines should support encryption, secure logging, and integrate policy-as-code tools (Kyverno). Runtime monitoring (with Tetragon, Cilium Hubble) addresses in-memory threats and zero trust.\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"usage-drift-and-cost-tracking\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#usage-drift-and-cost-tracking\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Usage, Drift, and Cost Tracking\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Monitor resource/hardware usage and track model drift with vector databases and open-source logging tools (Loki, ELK). Implement usage-based billing for accurate cost attribution.\"}),`\n`,(0,n.jsxs)(e.h2,{id:\"llm-observability-tools--platforms\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#llm-observability-tools--platforms\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"LLM Observability Tools \u0026 Platforms\"]}),`\n`,(0,n.jsx)(e.p,{children:\"The ecosystem for LLM observability continues to grow, with several powerful commercial and open source solutions:\"}),`\n`,(0,n.jsxs)(e.table,{children:[(0,n.jsx)(e.thead,{children:(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.th,{children:\"Tool\"}),(0,n.jsx)(e.th,{children:\"Type\"}),(0,n.jsx)(e.th,{children:\"Key Features\"}),(0,n.jsx)(e.th,{children:\"Pricing/Freemium\"}),(0,n.jsx)(e.th,{children:\"Pros\"}),(0,n.jsx)(e.th,{children:\"Cons\"}),(0,n.jsx)(e.th,{children:\"Self-host Option\"})]})}),(0,n.jsxs)(e.tbody,{children:[(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"LangSmith\"})}),(0,n.jsx)(e.td,{children:\"Paid\"}),(0,n.jsx)(e.td,{children:\"LLM tracing, cost analytics, feedback, works natively with Langchain\"}),(0,n.jsx)(e.td,{children:\"Free tier up to 5,000 traces/month; paid SaaS tiers available;
330 self-hosting only in enterprise\"}),(0,n.jsx)(e.td,{children:\"Robust integration with Langchain, manual/auto evals, SaaS simplicity\"}),(0,n.jsx)(e.td,{children:\"No open source backend, self-host for enterprise only, vendor lock-in risk\"}),(0,n.jsx)(e.td,{children:\"Limited (Enterprise)\"})]}),(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"Lunary\"})}),(0,n.jsx)(e.td,{children:\"Free/Open Src\"}),(0,n.jsx)(e.td,{children:\"Model tracking, categorization (Radar), prompt analytics\"}),(0,n.jsx)(e.td,{children:\"Free up to 1,000 events/day; open source under Apache 2.0\"}),(0,n.jsx)(e.td,{children:\"Completely open source, can self-host for privacy, easy integration\"}),(0,n.jsx)(e.td,{children:\"Event limit on free cloud, limited advanced analytics compared to commercial\"}),(0,n.jsx)(e.td,{children:\"Yes\"})]}),(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"Phoenix (Arize)\"})}),(0,n.jsx)(e.td,{children:\"Free/Open Src\"}),(0,n.jsx)(e.td,{children:\"Tracing, evaluation, hallucination detection\"}),(0,n.jsx)(e.td,{children:\"Free (ELv2 license), no full hosted SaaS; paid AX Pro starts at $50/m\"}),(0,n.jsx)(e.td,{children:\"Works out-of-box with LlamaIndex/LangChain/OpenAI, OTel compatible, built-in evals\"}),(0,n.jsx)(e.td,{children:\"Paid plan for hosted, may require infra management for self-host\"}),(0,n.jsx)(e.td,{children:\"Yes\"})]}),(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"Langfuse\"})}),(0,n.jsx)(e.td,{children:\"Free/Open Src\"}),(0,n.jsx)(e.td,{children:\"Session tracking, tracing, evaluation, OpenTelemetry backend\"}
330),(0,n.jsx)(e.td,{children:\"Free self-host up to 50k events/mo; $59/m for 100k events (managed), $199/mo Pro\"}),(0,n.jsx)(e.td,{children:\"Most complete OSS feature set, SOC2 compliant, wide integrations\"}),(0,n.jsx)(e.td,{children:\"Hosted plans have data limits, advanced features priced\"}),(0,n.jsx)(e.td,{children:\"Yes\"})]}),(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"Helicone\"})}),(0,n.jsx)(e.td,{children:\"Paid \u0026 OSS\"}),(0,n.jsx)(e.td,{children:\"LLM monitoring, prompt management, caching, cost tracker\"}),(0,n.jsx)(e.td,{children:\"Free up to 10,000 requests; $20/m Pro, $200/m Team\"}),(0,n.jsx)(e.td,{children:\"Caching reduces API costs, SDK and proxy integration, security features\"}),(0,n.jsx)(e.td,{children:\"Limited requests in free; higher tiers unlock retention/features\"}),(0,n.jsx)(e.td,{children:\"Yes\"})]}),(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"Grafana Cloud\"})}),(0,n.jsx)(e.td,{children:\"Paid/Open Src\"}),(0,n.jsx)(e.td,{children:\"Visualization, dashboards, multi-source metrics/logs/traces\"}),(0,n.jsx)(e.td,{children:\"Free up to 100GB data (3 active users); Pro $19/user/mo; Enterprise $8/user/mo\"}),(0,n.jsx)(e.td,{children:\"Flexible, massive plugin ecosystem, custom dashboards, active community\"}),(0,n.jsx)(e.td,{children:\"Usage tiers can get expensive, learning curve for advanced use\"}),(0,n.jsx)(e.td,{children:\"Yes\"})]}),(0,n.jsxs)(e.tr,{children:[(0,n.jsx)(e.td,{children:(0,n.jsx)(e.strong,{children:\"Traceloop OpenLLMetry\"})}),(0,n.jsx)(e.td,{children:\"Free/Open Src\"}),(0,n.jsx)(e.td,{children:\"OTel style tracing, multi-tool compatibility\"}),(0,n.jsx)(e.td,{children:\"Free, open source (Apache 2.0), backend also free\"}),(0,n.jsx)(e.td,{children:\"Universal OTel-compatible, integrates with Langchain, LlamaIndex\"}),(0,n.jsx)(e.td,{children:\"Infra setup required, less advanced analytics\"}),(0,n.jsx)(e.td,{children:\"Yes\"})]})]})]}),`\n`,(0,n.jsx)(e.p,{children:\"Recent surveys highlight these platforms' support for token counting, semantic traceability, drift detection, and GPU observation.\"}),`\n`,(0,n.jsxs)(e.h2,{id:\"hands-on-demo-langfuse-in-action\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#hands-on-demo-langfuse-in-action\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Hands-On Demo: Langfuse in Action\"]}),`\n`,(0,n.jsx)(e.p,{children:\"To demonstrate LLM observability in practice, let's walk through a complete setup using Langfuse\\u2014one of the most comprehensive open-source solutions. This demo showcases real-world tracing, session management, and analytics for LLM applications.\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"setting-up-langfuse-cloud\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#setting-up-langfuse-cloud\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Setting Up Langfuse Cloud\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Langfuse offers both self-hosted and cloud options. For this demo, we'll use the cloud version for rapid setup:\"}),`\n`,(0,n.jsxs)(e.ol,{children:[`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Create Account\"}),\": Visit \",(0,n.jsx)(e.a,{href:\"https://cloud.langfuse.com\",children:\"cloud.langfuse.com\"}),\" and sign up for a free account\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Get API Keys\"}),\": Navigate to Settings \\u2192 API Keys and copy your Public Key and Secret Key\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Configure Environment\"}),\": Set up your environment variables:\"]}),`\n`]}),`\n`,(0,n.jsx)(e.pre,{className:\"language-bash\",children:(0,n.jsxs)(e.code,{className:\"language-bash code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token assign-left variable\",children:\"LANGFUSE_PUBLIC_KEY\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),`pk-lf-your-key-here\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token assign-left variable\",children:\"LANGFUSE_SECRET_KEY\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),`sk-lf-your-key-here\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token assign-left variable\",children:\"LANGFUSE_HOST\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),`https://cloud.langfuse.com\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token assign-left variable\",children:\"OPENAI_API_KEY\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),`your-openai-api-key-here\n`]})]})}),`\n`,(0,n.jsxs)(e.h3,{id:\"demo-applications\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#demo-applications\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Demo Applications\"]}),`\n`,(0,n.jsx)(e.p,{children:\"We've created three comprehensive demo scenarios that showcase different aspects of LLM observability:\"}),`\n`,(0,n.jsxs)(e.h4,{id:\"1-simple-chat-interface\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#1-simple-chat-interface\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"1. Simple Chat Interface\"]}),`\n`,(0,n.jsx)(e.p,{children:\"A basic conversational AI that demonstrates fundamental tracing concepts:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-python\",children:(0,n.jsxs)(e.code,{className:\"language-python code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" langfuse \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` Langfuse\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` openai\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"langfuse \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" Langfuse\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    public_key\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"your-public-key\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    secret_key\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"your-secret-key\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    host\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"https://cloud.langfuse.com\"'}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"def\"}),\" \",(0,n.jsx)(e.span,{className:\"token function\",children:\"chat_with_llm\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"user_message\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"str\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" model\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"str\"}),\" \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"gpt-3.5-turbo\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),\" \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"-\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"\u003e\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"str\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Start a span for this chat completion\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    span \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" langfuse\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"start_span\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"name\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"chat_completion\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"input\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"user_message\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"try\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Start a generation observation\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        generation \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" langfuse\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"start_observation\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            name\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"llm_call\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            model\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"model\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"input\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"user_message\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            as_type\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"generation\"'}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        response \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" openai\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"chat\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"completions\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"create\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            model\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"model\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            messages\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"[\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"                \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"role\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"system\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"content\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"You are a helpful AI assistant.\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"                \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"role\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"user\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"content\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" user_message\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"]\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            temperature\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token number\",children:\"0.7\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            max_tokens\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token number\",children:\"500\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        result \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" response\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"choices\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"[\"}),(0,n.jsx)(e.span,{className:\"token number\",children:\"0\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"]\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"message\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),`content\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        generation\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"update\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"output\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"result\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        generation\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"end\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        span\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"update\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"output\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"result\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        span\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"end\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-lin
330e\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"return\"}),` result\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"except\"}),\" Exception \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"as\"}),\" e\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        error_msg \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" \",(0,n.jsxs)(e.span,{className:\"token string-interpolation\",children:[(0,n.jsx)(e.span,{className:\"token string\",children:'f\"Sorry, I encountered an error: '}),(0,n.jsxs)(e.span,{className:\"token interpolation\",children:[(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token builtin\",children:\"str\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"e\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"})]}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"'})]}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        generation\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"update\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"output\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"error_msg\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" level\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"ERROR\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        generation\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"end\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        span\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"update\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"output\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"error_msg\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" level\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"ERROR\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        span\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"end\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"return\"}),` error_msg\n`]})]})}),`\n`,(0,n.jsxs)(e.h4,{id:\"2-rag-retrieval-augmented-generation-pipeline\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#2-rag-retrieval-augmented-generation-pipeline\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"2. RAG (Retrieval Augmented Generation) Pipeline\"]}),`\n`,(0,n.jsx)(e.p,{children:\"A more complex workflow showing document retrieval, context assembly, and generation:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-python\",children:(0,n.jsxs)(e.code,{className:\"language-python code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"def\"}),\" \",(0,n.jsx)(e.span,{className:\"token function\",children:\"rag_pipeline\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"query\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"str\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),\" \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"-\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"\u003e\"}),\" Dict\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"[\"}),(0,n.jsx)(e.span,{className:\"token builtin\",children:\"str\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"any\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"]\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Start main span for RAG pipeline\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    trace \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" langfuse\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"start_span\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"name\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"rag_pipeline\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"input\"}),(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"query\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"try\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Step 1: Retrieve relevant documents\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        documents \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" retrieve_relevant_documents\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"query\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" trace\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Step 2: Assemble context\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        context \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" assemble_context\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"documents\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" query\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" trace\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Step 3: Generate answer\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        answer \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" generate_answer\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"context\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" trace\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        result \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"query\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" query\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"retrieved_documents\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" documents\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"context\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" context\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"answer\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` answer\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"update\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"name\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"rag_pipeline\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" output\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"result\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" metadata\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"doc_count\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token builtin\",children:\"len\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"documents\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"return\"}),` result\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"finally\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"end\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]})]})}),`\n`,(0,n.jsxs)(e.h4,{id:\"3-multi-step-workflow\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#3-multi-step-workflow\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"3. Multi-Step Workflow\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Demonstrates complex conversation chains and problem-solving workflows with 
330nested spans and observations.\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"langfuse-dashboard-overview\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#langfuse-dashboard-overview\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Langfuse Dashboard Overview\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Once you run the demo applications, the Langfuse dashboard provides comprehensive insights into your LLM operations:\"}),`\n`,(0,n.jsxs)(e.p,{children:[(0,n.jsx)(e.img,{src:\"https://res.cloudinary.com/dfee67kdq/image/upload/v1761306007/blogs/llm-observability/505232688-29705943-1671-4793-8a51-7ecfff4aeb82_e3hg4r.webp\",alt:\"Langfuse Latency Dashboard\"}),`\n`,(0,n.jsx)(e.em,{children:\"Langfuse dashboard showing latency metrics and performance insights from our demo applications\"})]}),`\n`,(0,n.jsxs)(e.h3,{id:\"trace-detail-view\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#trace-detail-view\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Trace Detail View\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Individual traces reveal the complete request flow with nested spans, timing breakdown, and token usage:\"}),`\n`,(0,n.jsxs)(e.p,{children:[(0,n.jsx)(e.img,{src:\"https://res.cloudinary.com/dfee67kdq/image/upload/v1761306007/blogs/llm-observability/505232696-7be0c243-bf39-4576-a86f-6080adf03377_zwk0lx.webp\",alt:\"Langfuse Trace Details\"}),`\n`,(0,n.jsx)(e.em,{children:\"Detailed trace view showing nested spans for RAG pipeline: document retrieval \\u2192 context assembly \\u2192 LLM generation\"})]}),`\n`,(0,n.jsxs)(e.h3,{id:\"analytics-and-cost-tracking\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#analytics-and-cost-tracking\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Analytics and Cost Tracking\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Built-in analytics track token usage, costs, and performance over time:\"}),`\n`,(0,n.jsxs)(e.p,{children:[(0,n.jsx)(e.img,{src:\"https://res.cloudinary.com/dfee67kdq/image/upload/v1761306007/blogs/llm-observability/505232672-808b17dc-3add-4be1-aee1-11747afde9ce_lqeszt.webp\",alt:\"Langfuse Cost Dashboard\"}),`\n`,(0,n.jsx)(e.em,{children:\"Analytics dashboard displaying token usage, cost analysis, and performance metrics across different models\"})]}),`\n`,(0,n.jsxs)(e.h3,{id:\"key-benefits-demonstrated\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#key-benefits-demonstrated\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Key Benefits Demonstrated\"]}),`\n`,(0,n.jsx)(e.p,{children:\"This hands-on demo showcases several critical LLM observability capabilities:\"}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Distributed Tracing\"}),\": Complete visibility into multi-step LLM workflows\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Performance Monitoring\"}),\": Real-time latency, throughput, and error tracking\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Cost Management\"}),\": Token usage and cost attribution across different models\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Error Handling\"}),\": Comprehensive error tracking and debugging information\"]}),`\n`]}),`\n`,(0,n.jsxs)(e.h3,{id:\"running-the-demo\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#running-the-demo\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Running the Demo\"]}),`\n`,(0,n.jsx)(e.p,{children:\"To try this demo yourself:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-bash\",children:(0,n.jsxs)(e.code,{className:\"language-bash code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Clone the demo repository\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token function\",children:\"git\"}),` clone https://github.com/cloudraftio/langfuse-demo.git\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token builtin class-name\",children:\"cd\"}),` langfuse-demo\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Install dependencies\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"pip \",(0,n.jsx)(e.span,{className:\"token function\",children:\"install\"}),\" \",(0,n.jsx)(e.span,{className:\"token parameter variable\",children:\"-r\"}),` requirements.txt\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Configure environment\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token function\",children:\"cp\"}),` env.example .env\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Edit .env with your API keys\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Run all demos\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`python run_all_demos.py\n`})]})}),`\n`,(0,n.jsx)(e.p,{children:\"The demo generates realistic traces across different scenarios, providing a comprehensive view of LLM observability in action.\"}),`\n`,(0,n.jsxs)(e.h2,{id:\"implementation-guide-llm-monitoring-on-kubernetes\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#implementation-guide-llm-monitoring-on-kubernetes\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Implementation Guide: LLM Monitoring on Kubernetes\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Deploying and observing LLMs in Kubernetes requires integrating metrics collection, tracing, logging, alert
330ing, security, and visualization. Below is a detailed how-to guide with working code snippets and configurations:\"}),`\n`,(0,n.jsxs)(e.h3,{id:\"1-exporting-llm-metrics-with-prometheus\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#1-exporting-llm-metrics-with-prometheus\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"1. Exporting LLM Metrics with Prometheus\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Expose inference request counts and latency metrics from your LLM service. Here's a minimal FastAPI example with Prometheus integration:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-python\",children:(0,n.jsxs)(e.code,{className:\"language-python code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" fastapi \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),\" FastAPI\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),` Request\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" prometheus_client \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),\" Counter\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" Histogram\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),` make_asgi_app\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` time\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"app \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" FastAPI\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"REQUEST_COUNT \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" Counter\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"llm_requests\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"Number of LLM requests\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"REQUEST_LATENCY \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" Histogram\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"llm_request_latency_seconds\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"Request latency in seconds\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsxs)(e.span,{className:\"token decorator annotation punctuation\",children:[\"@app\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"post\"]}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"/generate\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"async\"}),\" \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"def\"}),\" \",(0,n.jsx)(e.span,{className:\"token function\",children:\"generate\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"req\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" Request\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    start \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" time\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"time\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    data \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"await\"}),\" req\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"json\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Simulate call to LLM model\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    response \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"output\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"Example LLM output\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    REQUEST_COUNT\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"inc\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    REQUEST_LATENCY\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"observe\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"time\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"time\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),\" \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"-\"}),\" start\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"return\"}),` response\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Serve metrics at /metrics for Prometheus scraping\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"app\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"mount\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"/metrics\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" make_asgi_app\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]})]})}),`\n`,(0,n.jsx)(e.p,{children:(0,n.jsx)(e.strong,{children:\"Key takeaways:\"})}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsx)(e.li,{children:\"Metrics include request count and request latency\"}),`\n`,(0,n.jsx)(e.li,{children:\"Prometheus scrapes /metrics endpoint automatically\"}),`\n`]}),`\n`,(0,n.jsxs)(e.h3,{id:\"2-adding-distributed-tracing-with-opentelemetry\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#2-adding-distributed-tracing-with-opentelemetry\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"2. Adding Distributed Tracing with OpenTelemetry\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Enable transparent tracing of requests through auto instrumentation:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-python\",children:(0,n.jsxs)(e.code,{className:\"language-python code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" fastapi \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` FastAPI\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" opentelemetry\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"instrumentation\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"fastapi \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` FastAPIInstrumentor\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" opentelemetry\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"sdk\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"resources \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` Resource\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" opentelemetry\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"sdk\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"trace \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` TracerProvider\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" opentelemetry\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"exporter\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"jaeger\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"thrift \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` JaegerExporter\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" opentelemetry\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"sdk\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"export \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` BatchSpanProcessor\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"app \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" FastAPI\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Configure tracer provider\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"trace_provider \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" TracerProvider\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"resource\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\"Resource\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"create\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"service.name\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:'\"llm-service\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"jaeger_exporter \",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),\" JaegerExporter\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    agent_host_name\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}),(0,n.jsx)(e.span,{className:\"token string\",children:'\"localhost\"'}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    agent_port\",(0,n.jsx)(e.span,{className:\"token operator\",children:\"=\"}
330),(0,n.jsx)(e.span,{className:\"token number\",children:\"6831\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"trace_provider\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"add_span_processor\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"BatchSpanProcessor\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"jaeger_exporter\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Set tracer provider globally\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token keyword\",children:\"from\"}),\" opentelemetry \",(0,n.jsx)(e.span,{className:\"token keyword\",children:\"import\"}),` trace\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"trace\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"set_tracer_provider\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"trace_provider\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]}),(0,n.jsx)(e.span,{className:\"code-line\",children:`\n`}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token comment\",children:\"# Instrument FastAPI app\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"FastAPIInstrumentor\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\".\"}),\"instrument_app\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"(\"}),\"app\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\")\"}),`\n`]})]})}),`\n`,(0,n.jsx)(e.p,{children:(0,n.jsx)(e.strong,{children:\"Notes:\"})}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsx)(e.li,{children:\"Sends traces to Jaeger (could be any other tracer backend)\"}),`\n`,(0,n.jsx)(e.li,{children:\"Captures detailed performance and call path info\"}),`\n`]}),`\n`,(0,n.jsxs)(e.h3,{id:\"3-defining-prometheus-alert-rules-for-latency\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#3-defining-prometheus-alert-rules-for-latency\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"3. Defining Prometheus Alert Rules for Latency\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Alert on unusually high LLM response latency to proactively catch slowed inference:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-yaml\",children:(0,n.jsxs)(e.code,{className:\"language-yaml code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"apiVersion\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` monitoring.coreos.com/v1\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"kind\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` PrometheusRule\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"metadata\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"name\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" llm\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),`alerts\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"spec\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"groups\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"name\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` llm.rules\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"      \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"rules\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"alert\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` HighLLMLatency\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"expr\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" histogram_quantile(0.95\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\",\"}),\" sum(rate(llm_request_latency_seconds_bucket\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"[\"}),\"5m\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"]\"}),\")) by (le)) \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"\u003e\"}),` 2\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"for\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` 5m\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"labels\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"severity\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` warning\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"annotations\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"summary\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:\"'LLM inference latency at 95th percentile is greater than 2 seconds'\"}),`\n`]})]})}
330),`\n`,(0,n.jsxs)(e.h3,{id:\"4-centralized-log-aggregation\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#4-centralized-log-aggregation\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"4. Centralized Log Aggregation\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Use Fluentd or Promtail to ship container logs to Loki for easy search and parsing. Example Promtail config snippet:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-yaml\",children:(0,n.jsxs)(e.code,{className:\"language-yaml code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"server\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"http_listen_port\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token number\",children:\"9080\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"clients\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"url\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" http\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\"//loki\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`3100/loki/api/v1/push\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"scrape_configs\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"job_name\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" kubernetes\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),`pods\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"pipeline_stages\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"      \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"docker\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"{\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"}\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"kubernetes_sd_configs\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"      \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"role\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` pod\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"relabel_configs\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"      \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"source_labels\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"[\"}),\"__meta_kubernetes_pod_label_app\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"]\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"action\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` keep\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"regex\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" llm\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),`service\n`]})]})}),`\n`,(0,n.jsxs)(e.h3,{id:\"5-kubernetes-native-health-checks-using-canary-checker\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#5-kubernetes-native-health-checks-using-canary-checker\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"5. Kubernetes Native Health Checks using Canary Checker\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Install and configure Canary Checker to run quality assurance tests on model output before new versions go live:\"}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsx)(e.li,{children:\"Write proactive test scripts for key prompt responses\"}),`\n`,(0,n.jsx)(e.li,{children:\"Define health check probes that measure model accuracy over test queries\"}
330),`\n`,(0,n.jsx)(e.li,{children:\"Automate canary deployments and rollbacks based on health status\"}),`\n`]}),`\n`,(0,n.jsxs)(e.h3,{id:\"6-security--compliance-integration\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#6-security--compliance-integration\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"6. Security \u0026 Compliance Integration\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Protect observability data and runtime environments with:\"}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Kyverno\"}),\": Policy enforcement for namespaces, secrets, and logs\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Tetragon\"}),\": eBPF runtime monitoring for suspicious system calls\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Cilium Hubble\"}),\": Network observability at packet and service granularity\"]}),`\n`]}),`\n`,(0,n.jsx)(e.p,{children:\"Example Kyverno policy to restrict access to metrics endpoint:\"}),`\n`,(0,n.jsx)(e.pre,{className:\"language-yaml\",children:(0,n.jsxs)(e.code,{className:\"language-yaml code-highlight\",children:[(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"apiVersion\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` kyverno.io/v1\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"kind\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` ClusterPolicy\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"metadata\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"name\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" restrict\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\"metrics\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),`access\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"spec\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"  \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"rules\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"    \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\" \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"name\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" block\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),\"public\",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),`metrics\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"      \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"match\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"resources\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"kinds\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),` Service\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"namespaces\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token punctuation\",children:\"-\"}),` default\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"      \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"validate\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"message\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),\" \",(0,n.jsx)(e.span,{className:\"token string\",children:\"'Metrics service must not be publicly accessible.'\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"        \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"pattern\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"          \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"spec\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),`\n`]}),(0,n.jsxs)(e.span,{className:\"code-line\",children:[\"            \",(0,n.jsx)(e.span,{className:\"token key atrule\",children:\"type\"}),(0,n.jsx)(e.span,{className:\"token punctuation\",children:\":\"}),` ClusterIP\n`]})]})}),`\n`,(0,n.jsxs)(e.h3,{id:\"7-visualization-with-grafana\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#7-visualization-with-grafana\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"7. Visualization with Grafana\"]}),`\n`,(0,n.jsx)(e.p,{children:\"Connect Grafana to Prometheus, Loki, and Jaeger:\"}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsx)(e.li,{children:\"Create dashboards to display request latency trends, error rates, and token usage per inference\"}),`\n`,(0,n.jsx)(e.li,{children:\"Use traced request flows to drill into problematic LLM interactions\"}),`\n`,(0,n.jsx)(e.li,{children:\"Set alerts in Grafana for SLA breaches\"}),`\n`]}),`\n`,(0,n.jsxs)(e.h2,{id:\"what-llm-observability-cant-do\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#what-llm-observability-cant-do\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"What LLM Observability Can't Do\"]}),`\n`,(0,n.jsx)(e.p,{children:\"While powerful, LLM observability has limitations:\"}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Model Quality Assessment\"}),\": Observability tools can detect performance issues but cannot automatically assess the quality or accuracy of model outputs\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Context-Aware Monitoring\"}),\": Understanding the semantic meaning of prompts and responses requires specialized AI evaluation tools\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Real-time Model Drift Detection\"}),\": While tools can track metrics, detecting subtle model drift often requires domain expertise and manual analysis\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Cross-Model Comparison\"}),\": Comparing performance across different LLM providers or model versions requires custom analysis beyond standard observability tools\"]}),`\n`]}),`\n`,(0,n.jsx)(e.p,{children:\"In these cases, observability acts as a foundation, providing the data needed for deeper analysis and human expertise.\"}),`\n`,(0,n.jsx)(s,{title:\"Need help with LLM observability?\",description:\"Our team specializes in implementing comprehensive observability solutions for AI workloads. Get expert guidance on monitoring your LLM deployments.\",buttonText:\"Get Expert Help\"}),`\n`,(0,n.jsxs)(e.h2,{id:\"cost-and-limitations\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#cost-and-limitations\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Cost and Limitations\"]}),`\n`,(0,n.jsxs)(e.ul,{children:[`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Open Source Solutions\"}),\": Free to use but require significant engineering effort for setup, maintenance, and customization\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Commercial Platforms\"}),\": Provide rapid deployment and advanced features but involve ongoing subscription costs\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Infrastru
330cture Overhead\"}),\": Running observability tools in Kubernetes requires additional compute and storage resources\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Data Retention\"}),\": Long-term storage of observability data can become expensive, especially for high-volume LLM applications\"]}),`\n`,(0,n.jsxs)(e.li,{children:[(0,n.jsx)(e.strong,{children:\"Learning Curve\"}),\": Effective use of observability tools requires understanding both the tools and LLM-specific monitoring requirements\"]}),`\n`]}),`\n`,(0,n.jsxs)(e.h2,{id:\"conclusion\",children:[(0,n.jsx)(e.a,{className:\"hash-anchor\",href:\"#conclusion\",children:(0,n.jsx)(e.span,{className:\"icon icon-link\"})}),\"Conclusion\"]}),`\n`,(0,n.jsx)(e.p,{children:\"LLM observability is now a mission-critical capability for any team running generative AI in production\\u2014whether on open source frameworks or managed SaaS platforms. Free and open source solutions excel at privacy, flexibility, and customization, enabling technical teams to build tailored monitoring stacks and maintain control over their infrastructure. Paid commercial platforms, meanwhile, shine through rapid onboarding, advanced analytics, enterprise-grade security, managed scaling, and deep integrations with LLM agent ecosystems.\"}),`\n`,(0,n.jsx)(e.p,{children:\"The best choice depends on your organization's scale, budget, compliance needs, and engineering bandwidth. For startups or research environments, open source often offers rapid innovation and complete data sovereignty. For enterprises or mission-critical deployments, commercial observability tools deliver rich feature sets, robust support, and compliance at scale.\"}),`\n`,(0,n.jsx)(e.p,{children:\"Ultimately, combining or layering both approaches\\u2014using open source for experimentation and commercial solutions for high-traffic production\\u2014can bring organizations the best of both worlds: agility, security, and operational excellence.\"}),`\n`,(0,n.jsxs)(e.p,{children:[\"If you are looking for a \",(0,n.jsx)(e.a,{href:\"/ai-agent-development\",children:\"custom AI solution\"}),\" to integrate in your cloud native stack, we can help you build a tailored solution that meets your specific needs. \",(0,n.jsx)(e.a,{href:\"/contact-us\",children:\"Contact us today\"}),\" to learn more.\"]})]})}function p(a={}){let{wrapper:e}=a.components||{};return e?(0,n.jsx)(e,{...a,children:(0,n.jsx)(d,{...a})}):d(a)}function w(a,e){throw new Error(\"Expected \"+(e?\"component\":\"object\")+\" `\"+a+\"` to be defined: you likely forgot to import, pass, or provide it.\")}return b(L);})();\n;return Component;","frontmatter":{"wordCount":2315,"readingTime":{"text":"12 min read","minutes":11.57,"time":694200,"words":2314},"slug":"llm-observability","title":"LLM Observability: Monitoring Large Language Models","publishedAt":"2025-10-24","description":"A comprehensive guide to implementing observability for Large Language Models in production environments, covering monitoring, tracing, security, and cost optimization strategies.","englishOnly":"true","banner":"llm-observability","tags":"Observability","author":"Rahul Agrawal","keywords":"LLM Observability, AI Monitoring, Large Language Models, Production AI, AI Observability, LLM Monitoring, AI Operations, Model Drift, AI Security, AI Cost Optimization, OpenTelemetry, Prometheus, Grafana","img":"https://res.cloudinary.com/dfee67kdq/image/upload/v1761306667/author/rahul_rnsjpe.webp","featuredImage":"background/abstract_3.avif"},"recommendations":[{"title":"Top Open Source Logging Tools for Cloud Native Observability","publishedAt":"2025-01-07","description":"Explore the best open source logging tools tailored for cloud native observability, empowering developers and operators to monitor and debug applications effectively.","englishOnly":"true","banner":"open_source_logging_tools_vyqzdc.webp","tags":"Observability","author":"Anish Bista","keywords":"Cloud native logging tools, Open source observability tools, Kubernetes logging, Best logging tools for Kubernetes, Logging in cloud native environments, Open source monitoring and debugging, Kubernetes observability best practices, logging tool comparison, Centralized logging solutions, Open source logging for developers, loki, quickwit, victorialogs, signoz, opensearch, splunk alternatives, openobserve, siglens, parseable","img":"https://res.cloudinary.com/dfee67kdq/image/upload/v1753441734/author/image_11_klein7.webp","slug":"top-open-source-logging-tools","readingTime":{"text":"15 min read","minutes":14.275,"time":856500,"words":2855}},{"title":"Implementing Compliance-first Observability with OpenTelemetry","publishedAt":"2025-06-17","description":"Learn how to design observability pipelines that enforce data protection, support compliance regulations, and ensure secure telemetry using OpenTelemetry.","englishOnly":"true","banner":"Banner_dc8sql.webp","tags":"Observability","author":"Vibhuti Sharma","keywords":"OpenTelemetry, OpenTelemetry tutorial, OpenTelemetry integration, OpenTelemetry collector, OpenTelemetry tracing, OpenTelemetry metrics, OpenTelemetry logs, observability with OpenTelemetry, OpenTelemetry best practices, OpenTelemetry monitoring, compliance observability, observability and compliance, cloud compliance monitoring, compliance automation, regulatory compliance observability, security and compliance observability, compliance dashboards, observability tools, observability platform, observability for DevOps, cloud observability, end-to-e
330nd observability, distributed tracing, metrics and logging, observability compliance standards, GDPR observability, SOC2 observability, HIPAA observability, compliance monitoring tools, observability compliance framework","img":"https://res.cloudinary.com/dfee67kdq/image/upload/t_logo-200_200/v1745331196/author/vibhuti_sharma.webp","slug":"implement-compliance-first-observability-opentelemetry","readingTime":{"text":"15 min read","minutes":14.55,"time":873000,"words":2910}},{"title":"Scaling Prometheus with Thanos","publishedAt":"2024-07-22","description":" Learn how to scale Prometheus using Thanos for long-term storage, global view metrics, and improved performance in large-scale Kubernetes environments.","englishOnly":"true","banner":"thanos-cloudraft_vvkxri.webp","tags":"Observability","author":"Ritesh Sonawane","keywords":"Prometheus, Thanos, observability, monitoring at scale, Kubernetes monitoring, long-term metrics storage, scalable monitoring, Thanos architecture, global view metrics, cloud-native monitoring","img":"https://res.cloudinary.com/dfee67kdq/image/upload/v1714768168/author/ritesh_sonawane.webp","slug":"scaling-prometheus-with-thanos","readingTime":{"text":"10 min read","minutes":9.96,"time":597600,"words":1992}}],"authorsResolved":[{"slug":"rahul-agrawal","name":"Rahul Agrawal","photo":"","linkedin":"","x":"","bio":"SRE at CloudRaft focused on observability and cloud native infrastru
330cture."}]},"__N_SSG":true},"page":"/blog/[slug]","query":{"slug":"llm-observability"},"buildId":"IUuVij9Ol4yGFGm_99WoB","isFallback":false,"gsp":true,"scriptLoader":[]}</script>
vendor: 142 bytes, line 330
330<script src="/cdn-cgi/scripts/7d0fa10a/cloudflare-static/rocket-loader.min.js" data-cf-settings="46210e8713f087f2d699c04e-|49" defer></script>
330<script type="module" src="https://static.cloudflareinsights.com/beacon.min.js/v31edd6df95cf4e85bb4c19e7a9bdbcba1788362987495" integrity="sha512-iIg7k2xntmwu6/uSb5tpc/hySgZc4eoL31yB29W6tJFo2akwjPWcEqnCEdJvGexCL0KEQwVYv5BlowfhVz26hg==" data-cf-beacon='{"version":"2024.11.0","token":"2c726c1e07f5470693ceb6edef0cf5fb","r":1,"spa":2}' crossorigin="anonymous"></script>
330
331</body></html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.