PageSourceSearch

https://manumishra.com/blog/laya-on-lambda/

html manumishra.com collected 2026-10-03 21:14:27 UTC 41,374 bytes, 96 lines download raw bytes

1<!doctype html>
2<html lang="en" dir="ltr" class="blog-wrapper blog-post-page plugin-blog plugin-id-default" data-has-hydrated="false">
3<head>
4<meta charset="UTF-8">
5<meta name="generator" content="Docusaurus v3.8.0">
6<title data-rh="true">Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model | Manu Mishra</title><meta data-rh="true" name="viewport" content="width=device-width,initial-scale=1"><meta data-rh="true" property="og:locale" content="en"><meta data-rh="true" name="docusaurus_locale" content="en"><meta data-rh="true" name="docusaurus_tag" content="default"><meta data-rh="true" name="docsearch:language" content="en"><meta data-rh="true" name="docsearch:docusaurus_tag" content="default"><meta data-rh="true" name="author" content="Manu Mishra"><meta data-rh="true" name="msvalidate.01" content="8D78E08118D465445E0BA072DB19A547"><meta data-rh="true" property="og:url" content="https://manumishra.com"><meta data-rh="true" name="twitter:card" content="summary_large_image"><meta data-rh="true" name="twitter:title" content="Manu Mishra - Distinguished Solutions Architect, Author &amp; Researcher in AI &amp; Cloud"><meta data-rh="true" name="twitter:description" content="Discover cutting-edge insights, published research, and proven expertise in AI, machine learning, and cloud-native architectures from an industry-leading solutions architect, technical author, and researcher."><meta data-rh="true" property="og:title" content="Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model | Manu Mishra"><meta data-rh="true" name="description" content="Running the 421M-parameter Laya decision model on AWS Lambda with CPU-only PyTorch. What broke on ARM64, where the cold start really goes, warm latency, Banking77 accuracy, and how it compares with BitNet and EmbeddingGemma on Lambda."><meta data-rh="true" property="og:description" content="Running the 421M-parameter Laya decision model on AWS Lambda with CPU-only PyTorch. What broke on ARM64, where the cold start really goes, warm latency, Banking77 accuracy, and how it compares with BitNet and EmbeddingGemma on Lambda."><meta data-rh="true" name="keywords" content="laya model,aws lambda,serverless ai,decision model,pytorch on lambda,lambda container image,cold start,modernbert,banking77,cpu inference"><meta data-rh="true" property="og:image" content="https://manumishra.com/img/blog/laya-on-lambda.png"><meta data-rh="true" name="twitter:image" content="https://manumishra.com/img/blog/laya-on-lambda.png"><meta data-rh="true" property="og:type" content="article"><meta data-rh="true" property="article:published_time" content="2026-09-29T00:00:00.000Z"><meta data-rh="true" property="article:author" content="https://github.com/manu-mishra"><meta data-rh="true" property="article:tag" content="aws lambda,serverless,machine learning,pytorch,cold start,experiment"><link data-rh="true" rel="icon" href="/img/manu-mishra-fav-icon.png"><link data-rh="true" rel="canonical" href="https://manumishra.com/blog/laya-on-lambda"><link data-rh="true" rel="alternate" href="https://manumishra.com/blog/laya-on-lambda" hreflang="en"><link data-rh="true" rel="alternate" href="https://manumishra.com/blog/laya-on-lambda" hreflang="x-default">
6<script data-rh="true" type="application/ld+json">{"@context":"https://schema.org","@type":"BlogPosting","@id":"https://manumishra.com/blog/laya-on-lambda","mainEntityOfPage":"https://manumishra.com/blog/laya-on-lambda","url":"https://manumishra.com/blog/laya-on-lambda","headline":"Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model","name":"Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model","description":"Running the 421M-parameter Laya decision model on AWS Lambda with CPU-only PyTorch. What broke on ARM64, where the cold start really goes, warm latency, Banking77 accuracy, and how it compares with BitNet and EmbeddingGemma on Lambda.","datePublished":"2026-09-29T00:00:00.000Z","author":{"@type":"Person","name":"Manu Mishra","description":"Distinguished Solutions Architect, Author & Researcher in AI & Cloud","url":"https://github.com/manu-mishra","image":"/img/logo.png"},"image":{"@type":"ImageObject","@id":"https://manumishra.com/img/blog/laya-on-lambda.png","url":"https://manumishra.com/img/blog/laya-on-lambda.png","contentUrl":"https://manumishra.com/img/blog/laya-on-lambda.png","caption":"title image for the blog post: Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model"},"keywords":["laya model","aws lambda","serverless ai","decision model","pytorch on lambda","lambda container image","cold start","modernbert","banking77","cpu inference"],"isPartOf":{"@type":"Blog","@id":"https://manumishra.com/blog","name":"Blog"}}</script>
6<link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Manu Mishra RSS Feed">
7<link rel="alternate" type="application/atom+xml" href="/blog/atom.xml" title="Manu Mishra Atom Feed">
8<script src="/scripts/apply-theme.js"></script>
8<link rel="stylesheet" href="/assets/css/styles.e2a0b64d.css">
9<script src="/assets/js/runtime~main.774ca1e8.js" defer="defer"></script>
vendor: 1 bytes, line 9
9
10<script src="/assets/js/main.3aca5a99.js" defer="defer"></script>
10
11</head>
12<body class="navigation-with-keyboard">
13<svg xmlns="http://www.w3.org/2000/svg" style="display: none;"><defs>
14<symbol id="theme-svg-external-link" viewBox="0 0 24 24"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"/></symbol>
15</defs></svg>
16<script>!function(){var t=function(){try{return new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return window.localStorage.getItem("theme")}catch(t){}}();document.documentElement.setAttribute("data-theme",t||(window.matchMedia("(prefers-color-scheme: dark)").matches?"dark":"light")),document.documentElement.setAttribute("data-theme-choice",t||"system")}(),function(){try{const c=new URLSearchParams(window.location.search).entries();for(var[t,e]of c)if(t.startsWith("docusaurus-data-")){var a=t.replace("docusaurus-data-","data-");document.documentElement.setAttribute(a,e)}}catch(t){}}()</script>
16<div id="__docusaurus"><link rel="preload" as="image" href="/img/logo.png"><div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#__docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="theme-layout-navbar navbar navbar--fixed-top"><div class="navbar__inner"><div class="theme-layout-navbar-left navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo.png" alt="Manu Mishra Logo" class="themedComponent_mlkZ themedComponent--light_NVdE"><img src="/img/logo.png" alt="Manu Mishra Logo" class="themedComponent_mlkZ themedComponent--dark_xIcU"></div><b class="navbar__title text--truncate">Manu Mishra</b></a><a class="navbar__item navbar__link" href="/experience">Experience</a><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/about">About</a></div><div class="theme-layout-navbar-right navbar__items navbar__items--right"><a href="https://github.com/manu-mishra" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">GitHub<svg width="13.5" height="13.5" aria-hidden="true" class="iconExternalLink_nPIU"><use href="#theme-svg-external-link"></use></svg></a><a href="https://linkedin.com/in/manu-mishra" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">LinkedIn<svg width="13.5" height="13.5" aria-hidden="true" class="iconExternalLink_nPIU"><use href="#theme-svg-external-link"></use></svg></a><div class="toggle_vylO colorModeToggle_DEke"><button class="clean-btn toggleButton_gllP toggleButtonDisabled_aARS" type="button" disabled="" title="system mode" aria-label="Switch between dark and light mode (currently system mode)"><svg viewBox="0 0 24 24" width="24" height="24" aria-hidden="true" class="toggleIcon_g3eP lightToggleIcon_pyhR"><path fill="currentColor" d="M12,9c1.65,0,3,1.35,3,3s-1.35,3-3,3s-3-1.35-3-3S10.35,9,12,9 M12,7c-2.76,0-5,2.24-5,5s2.24,5,5,5s5-2.24,5-5 S14.76,7,12,7L12,7z M2,13l2,0c0.55,0,1-0.45,1-1s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S1.45,13,2,13z M20,13l2,0c0.55,0,1-0.45,1-1 s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S19.45,13,20,13z M11,2v2c0,0.55,0.45,1,1,1s1-0.45,1-1V2c0-0.55-0.45-1-1-1S11,1.45,11,2z M11,20v2c0,0.55,0.45,1,1,1s1-0.45,1-1v-2c0-0.55-0.45-1-1-1C11.45,19,11,19.45,11,20z M5.99,4.58c-0.39-0.39-1.03-0.39-1.41,0 c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0s0.39-1.03,0-1.41L5.99,4.58z M18.36,16.95 c-0.39-0.39-1.03-0.39-1.41,0c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0c0.39-0.39,0.39-1.03,0-1.41 L18.36,16.95z M19.42,5.99c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06c-0.39,0.39-0.39,1.03,0,1.41 s1.03,0.39,1.41,0L19.42,5.99z M7.05,18.36c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06 c-0.39,0.39-0.39,1.03,0,1.41s1.03,0.39,1.41,0L7.05,18.36z"></path></svg><svg viewBox="0 0 24 24" width="24" height="24" aria-hidden="true" class="toggleIcon_g3eP darkToggleIcon_wfgR"><path fill="currentColor" d="M9.37,5.51C9.19,6.15,9.1,6.82,9.1,7.5c0,4.08,3.32,7.4,7.4,7.4c0.68,0,1.35-0.09,1.99-0.27C17.45,17.19,14.93,19,12,19 c-3.86,0-7-3.14-7-7C5,9.07,6.81,6.55,9.37,5.51z M12,3c-4.97,0-9,4.03-9,9s4.03,9,9,9s9-4.03,9-9c0-0.46-0.04-0.92-0.1-1.36 c-0.98,1.37-2.58,2.26-4.4,2.26c-2.98,0-5.4-2.42-5.4-5.4c0-1.81,0.89-3.42,2.26-4.4C12.92,3.04,12.46,3,12,3L12,3z"></path></svg><svg viewBox="0 0 24 24" width="24" height="24" aria-hidden="true" class="toggleIcon_g3eP systemToggleIcon_QzmC"><path fill="currentColor" d="m12 21c4.971 0 9-4.029 9-9s-4.029-9-9-9-9 4.029-9 9 4.029 9 9 9zm4.95-13.95c1.313 1.313 2.05 3.093 2.05 4.95s-0.738 3.637-2.05 4.95c-1.313 1.313-3.093 2.05-4.95 2.05v-14c1.857 0 3.637 0.737 4.95 2.05z"></path></svg></button></div><div class="navbarSearchContainer_Bca1"></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="__docusaurus_skipToContent_fallback" class="theme-layout-main main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">All Blog Posts</div><div role="group"><h3 class="yearGroupHeading_rMGB">2026</h3><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a aria-current="page" class="sidebarItemLink_mo7H sidebarItemLinkActive_I1ZP" href="/blog/laya-on-lambda">Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/expectation-constructs-agentic-ai-primitive">Expectation Constructs: The Missing Primitive in Agentic AI</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/open-claw-fleet-virtual-organizations-ai-agents">Open-Claw-Fleet - Running Thousands of AI Agents on AWS</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/universal-image-mcp-multi-provider-image-generation">Universal Image MCP - One Server, Three AI Image Providers</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/s3-tables-streaming-analytics">Serverless Streaming Analytics with S3 Tables &amp; Firehose</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/recursive-language-models-strands-agentcore">Recursive Language Models on AWS with Strands Agents</a></li></ul></div><div role="group"><h3 class="yearGroupHeading_rMGB">2025</h3><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/aws-reinvent-2025-reinvented-powered-by-mcp">AWS Re:Invent 2025, Reinvented — Powered by MCP</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/embeddings-gemma-on-lambda">Google&#x27;s EmbeddingGemma on AWS Lambda - A Curiosity-Driven Experiment</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/deploy-microsoft-bitnet-llm-on-aws-lambda">Running 1.58-bit LLMs on AWS Lambda with BitNet</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/threat-modeling-autonomous-ai">Threat Modeling for Autonomous AI - What OWASP Wants You to Know</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/data-governance-playbook">Data Governance Playbook - From Chaos to Confidence</a></li></ul></div><div role="group"><h3 class="yearGroupHeading_rMGB">2024</h3><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/digital-standstill-theory-constraints">Digital Standstill &amp; Theory of Constraints</a></li></ul></div><div role="group"><h3 class="yearGroupHeading_rMGB">2023</h3><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/three-cs-of-coe">The Three &quot;C&quot;s of COE - From Center to Centering to Culture of Excellence</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/priming-business-flywheel-genai">Priming Business Flywheel with Gen-AI</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/unified-systems">Unified Systems - The Tech Trend You Never Knew You Needed</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/api-first-ai-era">Rethinking API-First - Unveiling Its True Power in the AI Era</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/llm-technical-debt">The Future with Large Language Models - A Technical Debt Worth Taking</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/software-craftsmanship">Software Craftsmanship - Objectify Tools, Not Debates</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/software-engineer-vs-developer">Software Engineer vs. Developer through the Lens of Socratic Questioning</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/digital-inertia">KTLO Can Lead to Digital Inertia and Hinder Digital Transformation</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/dont-keep-lights-on">Don&#x27;t Keep The Lights On</a></li></ul></div></nav></aside><main class="col col--7"><article class=""><header><h1 class="title_f1Hy">Laya on AWS Lambda - A Curiosity-Driven Experiment with a Decision Model</h1><div class="container_mt6G margin-vert--md"><time datetime="2026-09-29T00:00:00.000Z">September 29, 2026</time> · <!-- -->8 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--12 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/manu-mishra" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo authorImage_XqGP" src="/img/logo.png" alt="Manu Mishra"></a><div class="avatar__intro authorDetails_lV9A"><div class="avatar__name"><a href="https://github.com/manu-mishra" target="_blank" rel="noopener noreferrer"><span class="authorName_yefp">Manu Mishra</span></a></div><small class="authorTitle_nd0D" title="Distinguished Solutions Architect, Author &amp; Researcher in AI &amp; Cloud">Distinguished Solutions Architect, Author &amp; Researcher in AI &amp; Cloud</small><div class="authorSocials_rSDt"></div></div></div></div></div></header><div id="__blog-post-container" class="markdown"><p><img decoding="async" loading="lazy" alt="Laya on AWS Lambda" src="/assets/images/laya-on-lambda-6e6f774833a597c40bba5820126651a0.png" width="1594" height="1194" class="img_ev3q"></p>
17<p><em>Note: This is a curiosity-driven experiment, not a production recommendation. Cold starts are long, the ARM64 path doesn&#x27;t work, and confidence scores aren&#x27;t calibrated. For real workloads, look at <a href="https://aws.amazon.com/sagemaker/" target="_blank" rel="noopener noreferrer">Amazon SageMaker</a>, <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="noopener noreferrer">Amazon Bedrock</a>, or a warm, right-sized container service.</em></p>
18<h2 class="anchor anchorWithStickyNavbar_LWe7" id="1-the-idea">
181. The idea<a href="#1-the-idea" class="hash-link" aria-label="Direct link to 1. The idea" title="Direct link to 1. The idea">​</a></h2>
19<p>First I ran a <a href="/blog/deploy-microsoft-bitnet-llm-on-aws-lambda">1.58-bit LLM on Lambda</a>. Then <a href="/blog/embeddings-gemma-on-lambda">an embedding model</a>. Both worked, with the usual serverless caveats. That left one more kind of model I was curious about: one that doesn&#x27;t generate text or vectors, but makes a decision.</p>
20<p><a href="https://huggingface.co/convaiinnovations/laya" target="_blank" rel="noopener noreferrer">Laya</a> is exactly that. It&#x27;s an open-weight (Apache-2.0), 421M-parameter model that reads some text, answers a typed question about it, and returns a bounded answer in a single forward pass. Could that run on Lambda, CPU only, with no inference server? Only one way to find out.</p>
21<!-- -->
22<h2 class="anchor anchorWithStickyNavbar_LWe7" id="2-why-a-decision-model-is-interesting">2. Why a decision model is interesting<a href="#2-why-a-decision-model-is-interesting" class="hash-link" aria-label="Direct link to 2. Why a decision model is interesting" title="Direct link to 2. Why a decision model is interesting">​</a></h2>
23<p>Laya answers three kinds of questions:</p>
24<ul>
25<li><strong>choice</strong>: pick one of a set of options, such as which team should handle a support ticket</li>
26<li><strong>score</strong>: pick a level on an ordered scale, such as how urgent a request is</li>
27<li><strong>noul</strong>: give the probability that a yes/no statement is true</li>
28</ul>
29<p>That covers a lot of everyday plumbing: ticket routing, guardrails, triage, picking which downstream agent or model should handle a request. And the shape suits Lambda well. There&#x27;s no token-by-token loop and no GPU, just one forward pass per request.</p>
30<h2 class="anchor anchorWithStickyNavbar_LWe7" id="3-the-architecture">3. The architecture<a href="#3-the-architecture" class="hash-link" aria-label="Direct link to 3. The architecture" title="Direct link to 3. The architecture">​</a></h2>
31<p>The design stayed deliberately small. A Lambda container image holds Python 3.11, PyTorch (CPU), the Laya SDK, and the model checkpoint itself, pinned to one revision and verified by SHA-256 at build time. At runtime it&#x27;s fully offline: no Hugging Face downloads, no tokens.</p>
32<ul>
33<li><strong>Compute</strong>: Lambda, x86_64, 6 GB of memory (about 3.5 vCPUs), 300 s timeout</li>
34<li><strong>Image</strong>: about 1.04 GB compressed, 2.08 GB uncompressed, on the Lambda AL2023 base image, running as a non-root user with a read-only filesystem</li>
35<li><strong>Access</strong>: direct, IAM-authorized <code>aws lambda invoke</code> only, with no Function URL or API Gateway</li>
36<li><strong>Infrastructure</strong>: AWS CDK, a logs-only IAM role, and 7-day log retention</li>
37</ul>
38<p>The model loads once per execution environment and stays in memory for warm requests.</p>
39<h2 class="anchor anchorWithStickyNavbar_LWe7" id="4-arm64-first-and-it-didnt-work">4. ARM64 first, and it didn&#x27;t work<a href="#4-arm64-first-and-it-didnt-work" class="hash-link" aria-label="Direct link to 4. ARM64 first, and it didn&#x27;t work" title="Direct link to 4. ARM64 first, and it didn&#x27;t work">​</a></h2>
40<p>Like my earlier experiments, I started on ARM64 (Graviton). The image built fine and ran fine in a native ARM64 container on my laptop. On Lambda, every inference failed:</p>
41<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">Can&#x27;t open MIDR_EL1 sysfs entry</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Error in cpuinfo: failed to parse the list of possible processors in /sys/devices/system/cpu/possible</span><br></span></code></pre></div></div>
42<p>PyTorch&#x27;s CPU-detection library needs CPU-topology files that Lambda&#x27;s ARM64 sandbox hides. Forcing one thread didn&#x27;t help, and neither did skipping explicit thread settings. It&#x27;
42s a <a href="https://github.com/pytorch/cpuinfo/issues/143" target="_blank" rel="noopener noreferrer">known issue</a>. I switched to x86_64 and it worked on the first try.</p>
43<p>Lesson one: a container that works on ARM64 locally can still fail inside Lambda&#x27;s ARM64 sandbox. Test on Lambda early.</p>
44<h2 class="anchor anchorWithStickyNavbar_LWe7" id="5-it-worked-but-the-cold-start-hurt">5. It worked, but the cold start hurt<a href="#5-it-worked-but-the-cold-start-hurt" class="hash-link" aria-label="Direct link to 5. It worked, but the cold start hurt" title="Direct link to 5. It worked, but the cold start hurt">​</a></h2>
45<p>The first working deployment answered correctly, but its first request took about 94 seconds. Warm requests took about 0.37 seconds. A cold start that slow needed an explanation, so before blaming the model I went looking for problems in my own code. I found three:</p>
46<ul>
47<li><strong>A duplicated 205 MB layer.</strong> A <code>chmod</code> in the final Docker stage rewrote the whole Python runtime into a second image layer. Setting permissions in the builder stage instead cut the compressed image from 1.25 GB to 1.04 GB.</li>
48<li><strong>Hashing 843 MB on every cold start.</strong> The checkpoint was SHA-256-hashed every time the model loaded, even though the build had already verified it. It&#x27;
48s now opt-in.</li>
49<li><strong>A slow first inference.</strong> PyTorch does about 6 seconds of one-time setup on the first forward pass. A small warm-up call during initialization now absorbs that cost, so the first real user doesn&#x27;t.</li>
50</ul>
51<p>All three were worth fixing. None of them explained the cold start.</p>
52<h2 class="anchor anchorWithStickyNavbar_LWe7" id="6-the-real-culprit-a-freshly-deployed-image">6. The real culprit: a freshly deployed image<a href="#6-the-real-culprit-a-freshly-deployed-image" class="hash-link" aria-label="Direct link to 6. The real culprit: a freshly deployed image" title="Direct link to 6. The real culprit: a freshly deployed image">​</a></h2>
53<p>Adding per-stage timing to initialization made the picture clear:</p>
54<table><thead><tr><th>Stage</th><th style="text-align:right">First request after a new image deploy</th><th style="text-align:right">Later cold start</th></tr></thead><tbody><tr><td>Import torch</td><td style="text-align:right">3.3 s</td><td style="text-align:right">8.2 s</td></tr><tr><td><strong>Load model</strong></td><td style="text-align:right"><strong>91.3 s</strong></td><td style="text-align:right"><strong>15.6 s</strong></td></tr><tr><td>Warm-up inference</td><td style="text-align:right">5.0 s</td><td style="text-align:right">4.4 s</td></tr><tr><td><strong>Total cold request</strong></td><td style="text-align:right"><strong>100.2 s</strong></td><td style="text-align:right"><strong>28.9 s</strong></td></tr></tbody></table>
55<p>Lambda streams container image data from Amazon ECR on demand. Right after a new image is deployed nothing is cached yet, so the first execution environment pulls the 843 MB checkpoint while reading it. Later cold starts on the same image are much faster. The first request after a deploy actually hit the original 120-second timeout, which is why the timeout is now 300 seconds. (The 28.9 s figure comes from a single later cold start, so treat it as indicative, not a guarantee.)</p>
56<h2 class="anchor anchorWithStickyNavbar_LWe7" id="7-performance-results">7. Performance results<a href="#7-performance-results" class="hash-link" aria-label="Direct link to 7. Performance results" title="Direct link to 7. Performance results">​</a></h2>
57<p>With the model warm, 20 sequential requests measured:</p>
58<table><thead><tr><th style="text-align:right">p50</th><th style="text-align:right">p95</th><th style="text-align:right">Max</th><th style="text-align:right">Billed p95</th><th style="text-align:right">Peak memory</th></tr></thead><tbody><tr><td style="text-align:right">374 ms</td><td style="text-align:right">423 ms</td><td style="text-align:right">433 ms</td><td style="text-align:right">431 ms</td><td style="text-align:right">2,985 MB of 6,144 MB</td></tr></tbody></table>
59<p>Every benchmark call sent the same request:</p>
60<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">  </span><span class="token property">&quot;state&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token property">&quot;subject&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;Duplicate invoice&quot;</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> </span><span class="token property">&quot;body&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;We were charged twice. Please refund the duplicate.&quot;</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">  </span><span class="token property">&quot;questions&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">    </span><span class="token property">&quot;department&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">      </span><span class="token property">&quot;type&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;choice&quot;</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">      </span><span class="token property">&quot;instructions&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;Which team should handle this request?&quot;</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">      </span><span class="token property">&quot;criteria&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">        </span><span class="token property">&quot;billing&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;invoices, payments, and refunds&quot;</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">        </span><span class="token property">&quot;technical&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;bugs, outages, and errors&quot;</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">        </span><span class="token property">&quot;other&quot;</span><span class="token operator">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">&quot;everything else&quot;</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">      </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><br></span></code></pre></div></div>
61<p>Laya answered <code>billing</code> with 95.2% probability (technical 2.3%, other 2.5%).</p>
62<p>One example doesn&#x27;t tell you much about quality, so I also ran 100 queries from the public <a href="https://github.com/PolyAI-LDN/task-specific-datasets" target="_blank" rel="noopener noreferrer">Banking77</a> test set. Each query was scored against its true intent plus some distractor intents:</p>
63<table><thead><tr><th style="text-align:right">Candidate intents</th><th style="text-align:right">Accuracy</th></tr></thead><tbody><tr><td style="text-align:right">3</td><td style="text-align:right">93%</td></tr><tr><td style="text-align:right">10</td><td style="text-align:right">86%</td></tr><tr><td style="text-align:right">20</td><td style="text-align:right">70%</td></tr></tbody></table>
64<p>That&#x27;s zero-shot, with a closed list of candidates, not a trained 77-way classifier. Accuracy drops as the list of options grows, and the confidence scores aren&#x27;t calibrated.</p>
65<h2 class="anchor anchorWithStickyNavbar_LWe7" id="8-three-experiments-side-by-side">8. Three experiments, side by side<a href="#8-three-experiments-side-by-side" class="hash-l
65ink" aria-label="Direct link to 8. Three experiments, side by side" title="Direct link to 8. Three experiments, side by side">​</a></h2>
66<table><thead><tr><th></th><th>BitNet (generation)</th><th>EmbeddingGemma (embedding)</th><th>Laya (decision)</th></tr></thead><tbody><tr><td>Parameters</td><td>2B (1.58-bit)</td><td>300M</td><td>421M</td></tr><tr><td>Output</td><td>Generated text</td><td>768-dim vector</td><td>Choice, score, or probability</td></tr><tr><td>Lambda memory</td><td>2–10 GB tested</td><td>2 GB</td><td>6 GB</td></tr><tr><td>Warm model time</td><td>6–7 s for 10 tokens</td><td>0.12–0.33 s</td><td>~0.37 s</td></tr><tr><td>Cold start</td><td>~12 s</td><td>~12 s</td><td>~29 s; ~100 s right after a deploy</td></tr></tbody></table>
67<p>These numbers come from each project&#x27;s own measurements, taken with different methods and settings, so read this as a rough comparison.</p>
68<p>The pattern is still clear. Models that answer in one forward pass (embeddings and decisions) respond in well under a second. Token-by-token generation takes seconds even for a heavily quantized model. Laya&#x27;s cold start is the slowest of the three: PyTorch is a heavy runtime, and the checkpoint is larger. The earlier projects&#x27; 12-second cold starts may also have come from already-cached images.</p>
69<h2 class="anchor anchorWithStickyNavbar_LWe7" id="9-why-not-production">9. Why not production<a href="#9-why-not-production" class="hash-link" aria-label="Direct link to 9. Why not production" title="Direct link to 9. Why not production">​</a></h2>
70<p>It works, but I wouldn&#x27;t run anything serious this way:</p>
71<ul>
72<li><strong>Cold starts</strong>: about 30 seconds normally, about 100 seconds right after a deploy. SnapStart doesn&#x27;t support container images, so hiding them means provisioned concurrency or scheduled warm-up calls, and both cost money while idle.</li>
73<li><strong>ARM64 is off the table</strong> with this PyTorch build, which rules out Lambda&#x27;s usually cheaper architecture.</li>
74<li><strong>Quality needs work before real use</strong>: confidence isn&#x27;t calibrated, and accuracy on your own domain has to be measured, not assumed from Banking77.</li>
75<li><strong>Better options exist</strong> for sustained traffic: SageMaker endpoints, or a warm container on ECS or Fargate.</li>
76</ul>
77<h2 class="anchor anchorWithStickyNavbar_LWe7" id="10-wrapping-up">10. Wrapping up<a href="#10-wrapping-up" class="hash-link" aria-label="Direct link to 10. Wrapping up" title="Direct link to 10. Wrapping up">​</a></h2>
78<p>Running Laya on Lambda isn&#x27;t about beating dedicated inference infrastructure. The question was simply what happens when you put a small decision model on serverless compute. The answer: warm, it&#x27;s fast and cheap. Cold, you pay mostly for moving an 843 MB model into a fresh sandbox. And ARM64 can surprise you in ways a local container never will.</p>
79<p>The biggest lesson, again: measure before you optimize. I fixed three real issues in my own build, but caching dominated the cold start, and I only found that out by timing every stage.</p>
80<p>The complete implementation, including the benchmarks, the Banking77 evaluation, and an HTML report, is on <a href="https://github.com/manu-mishra/laya-on-lambda" target="_blank" rel="noopener noreferrer">GitHub</a>. Clone it, try it, break it.</p>
81<hr>
82<p><em>Built with <a href="https://kiro.dev" target="_blank" rel="noopener noreferrer">Kiro</a>, using AI-assisted development to go from idea to measured results quickly.</em></p>
83<h2 class="anchor anchorWithStickyNavbar_LWe7" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
84<ul>
85<li><a href="https://huggingface.co/convaiinnovations/laya" target="_blank" rel="noopener noreferrer">Laya model card</a></li>
86<li><a href="https://github.com/NandhaKishorM/laya" target="_blank" rel="noopener noreferrer">Laya source repository</a></li>
87<li><a href="https://github.com/manu-mishra/laya-on-lambda" target="_blank" rel="noopener noreferrer">Project repository</a></li>
88<li>
88<a href="https://github.com/pytorch/cpuinfo/issues/143" target="_blank" rel="noopener noreferrer">PyTorch cpuinfo issue on Lambda ARM64</a></li>
89<li><a href="https://github.com/PolyAI-LDN/task-specific-datasets" target="_blank" rel="noopener noreferrer">Banking77 dataset</a> (Casanueva et al., 2020, CC BY 4.0)</li>
90<li><a href="/blog/deploy-microsoft-bitnet-llm-on-aws-lambda">Previous post: BitNet on AWS Lambda</a></li>
91<li><a href="/blog/embeddings-gemma-on-lambda">Previous post: EmbeddingGemma on AWS Lambda</a></li>
92<li><a href="https://docs.aws.amazon.com/lambda/latest/dg/images-create.html" target="_blank" rel="noopener noreferrer">AWS Lambda container images</a></li>
93<li><a href="https://aws.amazon.com/sagemaker/" target="_blank" rel="noopener noreferrer">Amazon SageMaker</a></li>
94</ul></div><footer class="docusaurus-mt-lg"><div class="row margin-top--sm theme-blog-footer-edit-meta-row"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a rel="tag" class="tag_zVej tagRegular_sFm0" href="/blog/tags/aws-lambda">aws lambda</a></li><li class="tag_QGVx"><a rel="tag" class="tag_zVej tagRegular_sFm0" href="/blog/tags/serverless">serverless</a></li><li class="tag_QGVx"><a rel="tag" class="tag_zVej tagRegular_sFm0" href="/blog/tags/machine-learning">machine learning</a></li><li class="tag_QGVx"><a rel="tag" class="tag_zVej tagRegular_sFm0" href="/blog/tags/pytorch">pytorch</a></li><li class="tag_QGVx"><a rel="tag" class="tag_zVej tagRegular_sFm0" href="/blog/tags/cold-start">cold start</a></li><li class="tag_QGVx"><a rel="tag" class="tag_zVej tagRegular_sFm0" href="/blog/tags/experiment">experiment</a></li></ul></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Blog post page navigation"><a class="pagination-nav__link pagination-nav__link--next" href="/blog/expectation-constructs-agentic-ai-primitive"><div class="pagination-nav__sublabel">Older post</div><div class="pagination-nav__label">Expectation Constructs: The Missing Primitive in Agentic AI</div></a></nav></main><div class="col col--2"><div class="tableOfContents_bqdL thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-the-idea" class="table-of-contents__link toc-highlight">1. The idea</a></li><li><a href="#2-why-a-decision-model-is-interesting" class="table-of-contents__link toc-highlight">2. Why a decision model is interesting</a></li><li><a href="#3-the-architecture" class="table-of-contents__link toc-highlight">3. The architecture</a></li><li><a href="#4-arm64-first-and-it-didnt-work" class="table-of-contents__link toc-highlight">4. ARM64 first, and it didn&#39;t work</a></li><li><a href="#5-it-worked-but-the-cold-start-hurt" class="table-of-contents__link toc-highlight">5. It worked, but the cold start hurt</a></li><li><a href="#6-the-real-culprit-a-freshly-deployed-image" class="table-of-contents__link toc-highlight">6. The real culprit: a freshly deployed image</a></li><li><a href="#7-performance-results" class="table-of-contents__link toc-highlight">7. Performance results</a></li><li><a href="#8-three-experiments-side-by-side" class="table-of-contents__link toc-highlight">8. Three experiments, side by side</a></li><li><a href="#9-why-not-production" class="table-of-contents__link toc-highlight">9. Why not production</a></li><li><a href="#10-wrapping-up" class="table-of-contents__link toc-highlight">10. Wrapping up</a></li><li><a href="#references" class="table-of-contents__link toc-highlight">References</a></li></ul></div></div></div></div></div><footer class="theme-layout-footer footer footer--dark"><div class="container container-fluid"><div class="row footer__links"><div class="theme-layout-footer-column col footer__col"><div class="footer__title">Portfolio</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/experience">Experience</a></li><li class="footer__item"><a class="footer__link-item" href="/blog">Blog</a></li><li class="footer__item"><a class="footer__link-item" href="/about">About</a></li></ul></div>
94<div class="theme-layout-footer-column col footer__col"><div class="footer__title">Connect</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/manu-mishra" target="_blank" rel="noopener noreferrer" class="footer__link-item">GitHub<svg width="13.5" height="13.5" aria-hidden="true" class="iconExternalLink_nPIU"><use href="#theme-svg-external-link"></use></svg></a></li><li class="footer__item"><a href="https://linkedin.com/in/manu-mishra" target="_blank" rel="noopener noreferrer" class="footer__link-item">LinkedIn<svg width="13.5" height="13.5" aria-hidden="true" class="iconExternalLink_nPIU"><use href="#theme-svg-external-link"></use></svg></a></li><li class="footer__item"><a href="mailto:[email protected]" target="_blank" rel="noopener noreferrer" class="footer__link-item">Email<svg width="13.5" height="13.5" aria-hidden="true" class="iconExternalLink_nPIU"><use href="#theme-svg-external-link"></use></svg></a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">Copyright © 2026 Manu Mishra. Built with Docusaurus.</div></div></div></footer></div>
95</body>
96</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.