1<!DOCTYPE html> 2<html lang="en"><head> 3 <meta charset="utf-8"> 4 <meta http-equiv="X-UA-Compatible" content="IE=edge"> 5 <meta name="viewport" content="width=device-width, initial-scale=1"> 6
6<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
6<!-- Begin Jekyll SEO tag v2.8.0 --> 7<title>Uncertainty in Deep Learning, p.2 (Methods) | No-Range Space</title> 8<meta name="generator" content="Jekyll v3.10.0" /> 9<meta property="og:title" content="Uncertainty in Deep Learning, p.2 (Methods)" /> 10<meta name="author" content="Nikita Durasov" /> 11<meta property="og:locale" content="en_US" /> 12<meta name="description" content="In this post, we explores the main methods for estimating uncertainty in deep learning â from Bayesian inference and sampling-based ensembles to efficient deterministic and calibration techniques. We discuss how each approach captures different types of uncertainty, their computational trade-offs, and when to prefer one over another in practice." /> 13<meta property="og:description" content="In this post, we explores the main methods for estimating uncertainty in deep learning â from Bayesian inference and sampling-based ensembles to efficient deterministic and calibration techniques. We discuss how each approach captures different types of uncertainty, their computational trade-offs, and when to prefer one over another in practice." /> 14<link rel="canonical" href="https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html" /> 15<meta property="og:url" content="https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html" /> 16<meta property="og:site_name" content="No-Range Space" /> 17<meta property="og:type" content="article" /> 18<meta property="article:published_time" content="2026-08-13T00:00:00+00:00" /> 19<meta name="twitter:card" content="summary" /> 20<meta property="twitter:title" content="Uncertainty in Deep Learning, p.2 (Methods)" />
21<script type="application/ld+json"> 22{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Nikita Durasov"},"dateModified":"2026-08-13T00:00:00+00:00","datePublished":"2026-08-13T00:00:00+00:00","description":"In this post, we explores the main methods for estimating uncertainty in deep learning â from Bayesian inference and sampling-based ensembles to efficient deterministic and calibration techniques. We discuss how each approach captures different types of uncertainty, their computational trade-offs, and when to prefer one over another in practice.","headline":"Uncertainty in Deep Learning, p.2 (Methods)","mainEntityOfPage":{"@type":"WebPage","@id":"https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html"},"url":"https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html"}</script>
22 23<!-- End Jekyll SEO tag --> 24<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 25 rel="stylesheet"> 26 <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> 27 <link rel="stylesheet" href="/assets/css/style.css"> 28 <link rel="shortcut icon" type="image/png" href="/assets/images/logo.png"> 29 <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css" rel="stylesheet"> 30 <!--
30<script async src="https://busuanzi.icodeq.com/busuanzi.pure.mini.js"></script>
30 --> 31 <!--
31<script defer src="https://busuanzi.9420.ltd/js"></script>
31 --> 32 <!-- <meta name="referrer" content="no-referrer-when-downgrade"> --> 33 34 <meta property="og:image" content="/assets/images/logo.png" /><link type="application/atom+xml" rel="alternate" href="https://www.norange.ai/feed.xml" title="No-Range Space" />
vendor: 63 bytes, line 34
34<script async src="https://www.googletagmanager.com/gtag/js?id=
34UA-142853940-1
vendor: 12 bytes, line 34
34"></script>
35<script> 36 window['ga-disable-UA-142853940-1'] = window.doNotTrack === "1" || navigator.doNotTrack === "1" || navigator.doNotTrack === "yes" || navigator.msDoNotTrack === "1"; 37
vendor: 134 bytes, lines 37-41
37window.dataLayer = window.dataLayer || []; 38 function gtag(){dataLayer.push(arguments);} 39 gtag('js', new Date()); 40 41 gtag('config', '
41UA-142853940-1
vendor: 4 bytes, line 41
41');
42</script>
42 43<style> 44 .MathJax_Display, .MJXc-display, .MathJax_SVG_Display { 45 overflow-x: auto; 46 overflow-y: hidden; 47 } 48 </style> 49 <!-- 50<meta property="og:image" content="https://www.norange.ai/assets/logo.png" />--> 51</head> 52<body style="font-family: 'Lucida Console', Monaco, monospace"><header class="site-header"> 53 54 <div class="wrapper"><div class="row"> 55 <div class="col-6 col-sm-3"> 56 57 <a rel="author" href="/" class="blog-btn">Blog â</a> 58 59 <style> 60 .blog-btn { 61 display: inline-block; 62 margin: 2px 0; 63 line-height: 25px; /* space above & below the button */ 64 padding: 3px 10px; /* space inside the button around the text */ 65 font-family: 'Inter', sans-serif; 66 font-size: 18px; 67 font-weight: 500; 68 color: #fff; 69 text-decoration: none; 70 border-radius: 8px; 71 border: 1.5px solid transparent; 72 background: linear-gradient(#000, #000) padding-box, 73 linear-gradient(270deg, #096ce2, #5d41dc) border-box; 74 transition: all 0.4s ease; 75 box-shadow: 0 0 12px rgba(131, 56, 236, 0.6); 76 } 77 78 .blog-btn:hover { 79 background: linear-gradient(#000, #000) padding-box, 80 linear-gradient(90deg, #FFFFFF, #f285c1) border-box; 81 box-shadow: 0 0 18px #f285c1; 82 transform: translateY(-1px); 83 } 84 85 .blog-btn:hover::after { 86 position: relative; 87 left: 3px; 88 transition: left 0.0s ease; 89 } 90 91 </style> 92 93 </div> 94 95 <div class="col-6 col-sm-9"><nav class="site-nav" style="z-index: 1;"> 96 <input type="checkbox" id="nav-trigger" class="nav-trigger" /> 97 <label for="nav-trigger"> 98 <span class="menu-icon"> 99 <svg viewBox="0 0 18 15" width="18px" height="15px"> 100 <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/> 101 </svg> 102 </span> 103 </label> 104 105 <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div> 106 </nav></div> 107 </div> 108 109 <!-- <a class="site-title" rel="author" href="/">No-Range Space</a><nav class="site-nav" style="z-index: 1;"> 110 <input type="checkbox" id="nav-trigger" class="nav-trigger" /> 111 <label for="nav-trigger"> 112 <span class="menu-icon"> 113 <svg viewBox="0 0 18 15" width="18px" height="15px"> 114 <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/> 115 </svg> 116 </span> 117 </label> 118 119 <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div> 120 </nav>--> 121 122 </div> 123</header> 124<main class="page-content" aria-label="Content"> 125 <div class="wrapper"> 126 <div style="position: relative; z-index: 0;"> 127
127<script type="text/javascript" 128 src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_CHTML"></script>
vendor: 1 bytes, line 128
128
129<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
vendor: 1 bytes, line 129
129
130<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
130 131 132<nav class="post-toc" aria-label="Table of contents"> 133 <p class="post-toc-title">Contents</p> 134</nav> 135 136<article class="post h-entry" itemscope itemtype="http://schema.org/BlogPosting"> 137 138 <header class="post-header"> 139 <h1 class="post-title p-name" itemprop="name headline"><span class="post-title-gradient">Uncertainty in Deep Learning, p.2 (Methods)</span></h1> 140 <p class="post-meta" style="margin: 10px 0px"> 141 <span class="title-highlights" style="font-size:15px"><time class="dt-published" datetime="2026-08-13T00:00:00+00:00" itemprop="datePublished"> 142 Aug 13, 2026 143 </time>⢠144 <span itemprop="author" itemscope itemtype="http://schema.org/Person"> 145 <span class="p-author h-card" itemprop="name">Nikita Durasov</span></span><span style="margin-left: 8px;">â¢</span> <i class="fa-solid fa-eye"></i> 146 <span><span id="ga_site_pv">Loading...</span> views</span> 147 </span> 148 </p> 149 </header> 150 151 <div class="post-content e-content" itemprop="articleBody"> 152 \[\begin{array}{l|ccc|ccc} 153 & \text{Train. T.} & \text{Inf. T.} & \text{Mem.} & \text{Integr.} & \text{Regr.} & \text{Class.} \\ 154\hline 155\text{Deep Ensembles~[LPB17]} & N\times & N\times & N\times & \checkmark & \checkmark & \checkmark \\ 156\text{MC-Dropout~[GG16]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 157\text{Snapshot Ensembles~[HYL17]} & 1\times & N\times & N\times & \checkmark & \checkmark & \checkmark \\ 158\text{FGE~[GVR18]} & 1\times & N\times & N\times & \checkmark & \checkmark & \checkmark \\ 159\text{Bayes by Backprop~[BBC15]} & 1\times & N\times & 2\times & \checkmark & \checkmark & \checkmark \\ 160\text{BatchEnsemble~[WMW20]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 161\text{Masksembles~[DDG21]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 162\text{Packed-Ensembles~[LLF22]} & 1\times & 1\times & 1\times & \times & \checkmark & \checkmark \\ 163\text{MIMO~[HSG21]} & 1\times & 1\times & 1\times & \times & \checkmark & \checkmark \\ 164\text{Iterative Uncertainty~[DDF24]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 165\text{ABNN~[FFM24]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 166\text{Prediction Depth~[BSS21]} & 1\times & 1\times & 1\times & \checkmark & \checkmark & \checkmark \\ 167\text{Subnetwork Inference~[DXW21]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 168\text{BatchNorm Uncertainty~[ABT18]} & 1\times & N\times & 1\times & \checkmark & \checkmark & \checkmark \\ 169\hline 170\text{Single Model~[KG17]} & 1\times & 1\times & 1\times & \checkmark & \checkmark & \checkmark \\ 171\text{SNGP~[LDL20]} & 1\times & 1\times & 1\times & \checkmark & \times & \checkmark \\ 172\text{DUQ~[VA20]} & 1\times & 1\times & 1\times & \times & \times & \checkmark \\ 173\text{DDU~[MGE23]} & 1\times & 1\times & 1\times & \times & \times & \checkmark \\ 174\text{ZigZag~[DCF24]} & 1\times & 2\times & 1\times & \checkmark & \checkmark & \checkmark \\ 175\text{Experts Donât Cheat~[JGL24]} & 1\times & 2\times & 1\times & \checkmark & \times & \checkmark \\ 176\text{Evidential Learning~[SEK18]} & 1\times & 1\times & 1\times & \checkmark & \checkmark & \checkmark \\ 177\text{Prior Networks~[MMG18]} & 1\times & 1\times & 1\times & \times & \checkmark & \checkmark \\ 178\text{Posterior Networks~[CPK20]} & 1\times & 1\times & 1\times & \times & \checkmark & \checkmark \\ 179\text{Variance Propagation~[PCH19]} & 1\times & 1\times & 1\times & \times & \checkmark & \checkmark \\ 180\text{Orthonormal Certificates~[TG19]} & 1\times & 1\times & 1\times & \checkmark & \checkmark & \checkmark \\ 181\text{Conformal Predictions~[SW08]} & 1\times & 1\times & 1\times & \checkmark & \checkmark & \checkmark \\ 182\end{array}\] 183 184<blockquote> 185 <p><small><strong>Table 1:</strong> <strong>List of popular uncertainty methods for deep learning models.</strong> The leftmost column lists the uncertainty methods, categorized
185into sampling-based and sampling-free approaches. Each column represents different aspects of these methods:</small></p> 186</blockquote> 187 188<blockquote> 189 <p><small><strong>(1)</strong> Training time relative to training a single model (âTrain. T.â),<br /> 190<strong>(2)</strong> Inference time relative to a single model (âInf. T.â),<br /> 191<strong>(3)</strong> Memory requirements for the method (âMem.â),<br /> 192<strong>(4)</strong> A subjective metric indicating the ease of integrating the method into existing architectures and training pipelines (âIntegr.â),<br /> 193<strong>(5)</strong> Whether the approach supports regression tasks, and<br /> 194<strong>(6)</strong> Whether it works with classification tasks.</small></p> 195</blockquote> 196 197<blockquote> 198 <p><small>We round down the values for time and memory metrics unless they are <em>significantly</em> greater than one. For time and memory evaluations, we assume that models are run sequentially (not in parallel) and that the entire model resides in GPU memory throughout the process. The integration metric reflects the complexity of incorporating the method into existing training pipelines or model architectures. Examples of methods with difficult integration include those requiring <em>significant</em> changes to model architectures (e.g., replacing most layers), training pipelines, or additional data.</small></p> 199</blockquote> 200 201<h2 id="uncertainty-estimation-methods">Uncertainty Estimation Methods</h2> 202 203<p>Uncertainty estimation provides a structured way to measure how confident a deep learning model is in its predictions. These techniques play an essential role in handling noisy inputs, supporting decision-making in safety-critical settings, and identifying out-of-distribution (OOD) samples. Broadly, existing methods fall into sampling-based and sampling-free categories. Each class comes with its own advantages and limitations in terms of accuracy, computational overhead, and scalabilityâtopics discussed in the following sections.</p> 204 205<p>As discussed earlier, many uncertainty estimation techniques originate from probabilistic principles, yet most ultimately focus on producing a scalar quantity that reflects confidence in a prediction rather than modelling the full predictive distribution, which is often computationally prohibitive. In a simple 1D regression setting, for example, the model may output both a mean and an associated scale parameter such as a standard deviation [KG17]. Different approaches vary in how these quantities are obtained. The following sections outline how commonly used methods compute uncertainty and summarize them in a high-level comparison table (Table 1). Before introducing these methods, we first establish a more formal view of uncertainty from a Bayesian perspective.</p> 206 207<h3 id="bayesian-approach">Bayesian Approach</h3> 208 209<p>Bayesian methods offer a principled way to incorporate uncertainty into machine learning models by treating model parameters as random variables rather than fixed quantities [BBC15, GG16]. The central idea follows Bayesâ theorem, which connects the posterior distribution of the parameters \(\theta\) given observed data \(D = \{(x_i, y_i)\}_{i=1}^N\) to the prior distribution \(P(\theta)\) and the likelihood:</p> 210 211\[P(\theta \mid D) = \frac{P(\theta) P(D \mid \theta)}{P(D)} 212= \prod_{i=1}^N \frac{P(\theta) \, P(y_i \mid x_i, \theta)}{\int P(D \mid \theta) P(\theta) \, d\theta}\] 213 214<p>where \(P(\theta \mid D)\) is the posterior distribution, representing the updated belief about the parameters after observing the data, \(P(\theta)\) is the prior, capturing prior knowledge or assumptions about \(\theta\), \(P(D \mid \theta) = \prod_{i=1}^N P(y_i \mid x_i, \theta)\) (assuming the data points are independent and identically distributed) is the likelihood, describing how probable the observed data \(D\) is given the parameters, and \(P(D)\) is the evidence, a normalizing constant ensuring that the posterior is a valid probability distribution. From this formalization, many familiar loss functions and regularization techniques in machine learning can be derived by specifying assumptions about the likelihood \(P(y_i \mid x_i, \theta)\) and the prior \(P(\theta)\). These assumptions determine the modelâs behavior and the constraints imposed during training, providing a unified Bayesian perspective on various learning approaches.</p> 215 216<p>In the Bayesian approach, predictions for a new input \(x\) are made by marginalizing over the posterior distribution of the parameters. The predictive distribution is expressed as:</p> 217 218\[P(y \mid x, D) = \int
219\color{#2563eb}{ 220 \underbrace{P(y \mid x, \theta)}_{\text{Aleatoric}} \, 221} 222\color{#16a34a}{ 223 \underbrace{P(\theta \mid D)}_{\text{Epistemic}} 224} 225\, d\theta\qquad(1)\] 226 227<p>This equation highlights the two main sources of uncertainty in Bayesian modeling. The term \(P(y \mid x, \theta)\) represents aleatoric uncertainty, which arises from inherent noise or ambiguity in the data. It reflects the variability in predictions for a fixed model parameter \(\theta\) and is irreducible, even with additional data. In contrast, \(P(\theta \mid D)\) captures epistemic uncertainty, which stems from the modelâs lack of knowledge about the true parameter values due to limited or insufficient training data. This uncertainty can be reduced by gathering more diverse or representative data, leading to a more confident posterior.</p> 228 229<p>However, directly computing \(P(\theta \mid D)\) is intractable for complex models like deep neural networks due to the high-dimensional parameter space and the integral required to compute \(P(D)\). To address this, most existing uncertainty estimation methods for deep learning attempt to implicitly or explicitly approximate the Bayesian integral to make it computationally feasible. A classical strategy is to replace analytical computation with sampling. In this context, sampling from \(P(\theta \mid D)\) provides an estimate of the integral, and we refer to approaches based on this idea as sampling-based methods.</p> 230 231<h3 id="sampling-based-methods">Sampling-Based Methods</h3> 232 233<p>Sampling-based methods derive uncertainty estimates by generating multiple predictions from the model and analyzing their variability. These approaches are known for their flexibility and accuracy, particularly in capturing epistemic uncertainty [PCH19]. As mentioned earlier, for high-dimensional real-valued parameters \(\theta\) such as those in neural networks, the integral in Equation 1 becomes intractable and is typically approximated using Monte Carlo samples:</p> 234 235<p>\begin{equation} 236P(y \mid x, \mathcal{D}) \approx \frac{1}{N} \sum_{k=1}^{N} P(y \mid x, \theta^{(k)}); \quad \theta^{(k)} \sim p(\theta \mid \mathcal{D})\qquad(2) 237\label{eq:ensemble_approximation} 238\end{equation}</p> 239 240<p>based on \(N\) different sets of parameters \(\theta^{(k)}\). Different sampling-based methods differ in how they compute these parameters. Some approaches rely on explicit posterior approximations, such as Bayesian Neural Networks [BBC15], while others use implicit sampling techniques like MC-Dropout [GG16] or Deep Ensembles [LPB17]. Although sampling-based approaches are computationally expensive due to the need to train and perform inference with multiple models, they remain the gold standard for both accuracy and uncertainty quality [OHL+19], particularly in the case of Deep Ensembles. While many sampling-free methods discussed later aim to match the performance of sampling-based approaches while reducing computational overhead, they often fall short of achieving the same level of uncertainty quality.</p> 241 242<p><strong><u>Masksembles</u></strong>: Masksembles [DDG21] enhance MC-Dropout by carefully designing structured masks that selectively drop specific features. This approach increases diversity among predictions, resulting in improved uncertainty quantification.</p> 243 244<p><strong><u>BatchEnsembles</u></strong>: BatchEnsembles [WMW20] offer an efficient ensembling approach by representing multiple models within a single network using learnable rank-one perturbations of the weights. This design minimizes memory overhead while preserving diversity in predictions. By sharing parameters across models, BatchEnsembles strike a balance between scalability and uncertainty estimation quality.</p> 245 246<p><strong><u>Subnetwork Inference</u></strong>: Subnetwork Inference [DXW21] is a scalable Bayesian framework that performs posterior inference over a small subset of neural network weights, keeping the rest as point estimates. By performing inference on a selected subnetwork with expressive approximations like full-covariance Gaussians, it aims to balance computational efficiency with accurate uncertainty estimation.</p> 247 248<p><strong><u>Packed-Ensembles</u></strong>
248: Packed-Ensembles [LLF22] propose an efficient ensembling approach that uses grouped convolutions to create multiple independent subnetworks within a single shared architecture. The model is trained similarly to a standard single model but produces multiple predictions, mimicking ensembling while introducing only minor computational overhead.</p> 249 250<p><strong><u>MIMO</u></strong>: The Multiple Input Multiple Output framework (MIMO) [HSG21], along with similar approaches like MixMO [rame2021mixmo], enables efficient ensembling by training independent subnetworks within a single model. During training, \(N\) inputs are processed simultaneously, allowing the model to make \(N\) independent predictions in a single forward pass. At inference, the input is replicated \(N\) times to generate \(N\) predictions, which are ensembled to estimate uncertainty.</p> 251 252<p><strong><u>Iterative Uncertainty</u></strong>: The Iterative Uncertainty framework [DDF24] leverages the iterative refinement process of neural networks to estimate uncertainty without modifying the original architecture. By analyzing the variance in outputs across consecutive iterations, it uses the rate of convergence as a proxy for certainty, where faster convergence indicates higher confidence. A related concept, <strong>Prediction Depth</strong> [BSS21], measures the layer depth at which a modelâs prediction stabilizes, with harder examples requiring deeper layers for convergence.</p> 253 254<p><strong><u>ABNN</u></strong>: Adaptable Bayesian Neural Networks (ABNN) [FFM24], along with similar approaches like BatchNorm Uncertainty [ABT18], transform pre-trained DNNs into Bayesian Neural Networks by fine-tuning the modelâs normalization layers and introducing noise during inference to approximate sampling.</p> 255 256<p><strong><u>Bayes by Backprop</u></strong>: Bayes by Backprop [BBC15] is a variational inference method that approximates the posterior distribution of model parameters by minimizing a variational objective. Instead of learning point estimates, it learns a distribution over weights, typically using a Gaussian variational posterior.</p> 257 258<h3 id="sampling-free-methods">Sampling-Free Methods</h3> 259 260<p><strong><u>Sampling-free methods</u></strong> estimate uncertainty without generating multiple predictions, offering computational efficiency and scalability. As mentioned earlier, sampling-based approaches are often too costly for real-world applications like robotics and autonomous driving, necessitating faster alternatives. These methods achieve efficiency through architectural modifications, deterministic approximations, or specialized loss functions. Several popular approaches are discussed in the following subsections.</p> 261 262<p><strong><u>Distance-Aware Uncertainty</u></strong>: Recent approaches such as <strong>Deterministic Uncertainty Quantification</strong> (DUQ) [VA20] and <strong>SNGP</strong> [LDL20] focus on singleâforward-pass uncertainty estimation by introducing distance-aware output layers, such as radial basis functions (RBFs) in DUQ and Gaussian processes (GPs) in SNGP. DUQ employs a Jacobian penalty to encourage smoothness in the feature space, while SNGP uses spectral normalization to ensure sensitivity and stability in the latent space. Building on these ideas, <strong>Deep Deterministic Uncertainty</strong> (DDU) [MGE23] further extends the framework by directly modeling the feature density in the latent space.</p> 263 264<p><strong><u>Conformal Prediction</u></strong>: Conformal prediction [SW08] is a statistical framework that provides uncertainty estimates with formal coverage guarantees, ensuring that the true label lies within the predicted set with a specified confidence level. It operates by calibrating model outputs on a held-out dataset to produce prediction intervals or sets, making it agnostic to the underlying model architecture. This flexibility allows conformal prediction to be applied across a wide range of tasks, offering a principled and reliable approach to uncertainty quantification in real-world applications.</p> 265 266<p><strong><u>Second-Order Probabilistic Modeling</u></strong>: Second-order probabilistic modeling approachesâsuch as <strong>prior networks</strong> [MMG18], <strong>posterior networks</strong> [CPK20], and <strong>evidential learning</strong> [SEK18]âcapture uncertainty by modeling distributions over predictive probabilities. These methods leverage a factorization of the predictive distribution, as shown in Equation 3, to decompose uncertainty into aleatoric, distributional, and epistemic components.</p> 267 268\[p(y \mid x, \mathcal{D}) 269= \iint 270\underbrace{P(y \mid \pi)}_{\text{Aleatoric}} 271\, 272\underbrace{p(\pi \mid x, \theta)}_{\text{Distributional}} 273\, 274\underbrace{p(\theta \mid \mathcal{D})}_{\text{Epistemic}} 275\, d\pi \, d\theta 276\approx \\ 277\approx \int 278\underbrace{P(y \mid \pi)}_{\text{Aleatoric}} 279\, 280\underbrace{p(\pi \mid x, \hat{\theta})}_{p(\theta \mid \mathcal{D}) \approx \delta(\theta - \hat{\theta})} 281\, d\pi.\qquad(3)\] 282 283<p>Prior networks directly model the distributional uncertainty \(p(\pi \mid x, \hat{\theta})\) by parameterizing a Dirichlet distribution, enabling the network to represent confidence levels tied to predictive uncertainty. Posterior networks extend this by incorporating learned feature-level distributions, combining evidence from the data to produce refined predictions. Evidential learning follows a
283similar framework, interpreting input data as evidence to parameterize distributions over class probabilities, often using a Dirichlet or related family.</p> 284 285<p><strong><u>Additional Information Approaches</u></strong>: Recent methods leverage auxiliary information to probe model confidence, based on the idea that confident models remain unaffected by additional inputs, while uncertain ones show prediction shifts. ZigZag [DCF24] measures uncertainty by comparing predictions with and without auxiliary information (ground truth during training and model predictions during inference), revealing sensitivity to added data. Similarly, <strong>Experts Donât Cheat</strong> [JGL24] leverages paired responses, allowing models to âcheatâ by using one response to predict the other, thereby exposing uncertain predictions.</p> 286 287<h2 id="conclusion">Conclusion</h2> 288 289<h2 id="references">References</h2> 290 291<p><strong>[LPB17]</strong> B. Lakshminarayanan, A. Pritzel, and C. Blundell. âSimple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.â In Advances in Neural Information Processing Systems, 2017.</p> 292 293<p><strong>[GG16]</strong> Y. Gal and Z. Ghahramani. âDropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.â In International Conference on Machine Learning, 2016.</p> 294 295<p><strong>[HYL+17]</strong> G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. âSnapshot Ensembles: Train 1, Get M for Free.â In International Conference on Learning Representations, 2017.</p> 296 297<p><strong>[GVR+18]</strong> T. Garipov, P. Izmailov, D. Podoprikhin, T. Vetrov, and A. G. Wilson. âLoss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs.â In Advances in Neural Information Processing Systems, 2018.</p> 298 299<p><strong>[BBC15]</strong> C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. âWeight Uncertainty in Neural Networks.â In International Conference on Machine Learning, 2015.</p> 300 301<p><strong>[WMW+20]</strong> Y. Wen, Y. M. Wu, E. M. Moody, and S. L. Lee. âBatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning.â In International Conference on Learning Representations, 2020.</p> 302 303<p><strong>[DDG21]</strong> N. Durasov, D. Dvornik, and P. Fua. âMasksembles for Uncertainty Estimation.â In International Conference on Computer Vision, 2021.</p> 304 305<p><strong>[LLF22]</strong> C. Laurent, S. Luneau, and J. Forest. âPacked-Ensembles for Efficient Uncertainty Estimation.â In International Conference on Learning Representations, 2022.</p> 306 307<p><strong>[HSG+21]</strong> D. Havasi, R. Senanayake, J. Liu, G. Dillon, and A. Hartnett. âTraining Independent Subnetworks for Robust Prediction.â In International Conference on Learning Representations, 2021.</p> 308 309<p><strong>[DDF24]</strong> N. Durasov, D. Dvornik, and P. Fua. âIterative Uncertainty Estimation with Self-Consistency.â In Conference on Computer Vision and Pattern Recognition, 2024.</p> 310 311<p><strong>[FFM24]</strong> G. Franchi, S. Franchi, and A. Maki. âMake Uncertainty Great Again: Accounting for Approximation in Bayesian Neural Networks.â In International Conference on Learning Representations, 2024.</p> 312 313<p><strong>[BSS+21]</strong> A. Baldock, C. Shen, A. van den Hengel, and L. Liu. âDeep Regression Ensembles with Prediction Depth.â In Advances in Neural Information Processing Systems, 2021.</p> 314 315<p><strong>[DXW21]</strong> N. Daxberger, J. Kristiadi, S. Sun, A. Immer, R. Eschenhagen, A. Pooch, and P. Hennig. âBayesian Deep Learning via Subnetwork Inference.â In International Conference on Machine Learning, 2021.</p> 316 317<p><strong>[ABT18]</strong> A. Atanov, A. Ashukha, D. Vetrov, and M. Muhammad. âUncertainty Estimation via Batch Normalization in Deep Networks.â In International Conference on Learning Representations, 2018.</p> 318 319<p>Single-Model Predictive Uncertainty</p> 320 321<p><strong>[KG17]</strong> A. Kendall and Y. Gal. âWhat Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?â In Advances in Neural Information Processing Systems, 2017.</p> 322 323<p><strong>[LDL20]</strong> J. Liu, C. D. Tran, A. Tagasovska, D. J. Sutherland, and I. Davidson. âSimple and Efficient Gaussian Process Priors for Deep Models.â In Advances in Neural Information Processing Systems, 2020.</p> 324 325<p><strong>[VA20]</strong> L. van Amersfoort, M. Smith, M. C. C. Labouriau, N. Pitas, and Y. Gal. âUncertainty Estimation Using Deep Kernel Learning.â In International Conference on Machine Learning, 2020.</p> 326 327<p><strong>[MGE23]</strong> A. Mukhoti, C. Galesso, R. Guo, G. van Steenkiste, and J. M. Hernández-Lobato. âDeep Deterministic Uncertainty: A Simple Baseline.â In International Conference on Learning Representations, 2023.</p> 328 329<p><strong>[DCF24]</strong> N. Durasov, A. Charpentier, and P. Fua. âZigZag: Adversarial Inference for OOD Detection.â In International Conference on Learning Representations, 2024.</p> 330 331<p><strong>[JGL24]</strong> A. Johnson, A. Gouk, and T. Lotte. âExperts Donât Cheat: Reliable Deep Ensembles without Tricks.â In International Conference on Learning Representations, 2024.</p> 332 333<p><strong>[SEK18]</strong> Y. Sensoy, L. Kaplan, and M. Kandemir. âEvidential Deep Learning to Quantify Classification Uncertainty.â In Advances in Neural Information Processing Systems, 2018.</p> 334 335<p>[<strong>MMG18]</strong> A. Malinin and M. Gales. âPredictive Uncertainty Estimation via Prior Networks.â In Advances in Neural Information Processing Systems, 2018.</p> 336 337<p><strong>[CPK20]</strong> A. Charpentier, S. Pionnier, and M. K. Ghassemi. âPosterior Networks: Extracting Uncertainty from Deterministic Neural Networks.â In International Conference on Learning Representations, 2020.</p> 338 339<p><strong>[PCH19]</strong> B. Postels, C. Chitta, and L. Van Gool. âSampling-Free Epistemic Uncertainty Estimation Using Variance Propagation.â In International Conference on Computer Vision, 2019.</p> 340 341<p><strong>[TG19]</strong> A. Tagasovska and D. Gonzalez. âOrthonormal Certificates for Out-of-Distribution Detection.â In International Conference on Machine Learning, 2019.</p> 342 343<p><strong>[SW08]</strong> G. Shafer and V. Vovk. âA Tutorial on Conformal Prediction.â Journal of Machine Learning Research, 2008.</p> 344 345 </div><div id="disqus_thread"></div> 346
346<script> 347 var disqus_config = function () { 348 this.page.url = 'https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html'; 349 this.page.identifier = 'https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html'; 350 }; 351 352 (function() { 353 var d = document, s = d.createElement('script'); 354 355 s.src = 'https://nikitadurasov-github-io.disqus.com/embed.js'; 356 357 s.setAttribute('data-timestamp', +new Date()); 358 (d.head || d.body).appendChild(s); 359 })(); 360 </script>
360 361 <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript><a class="u-url" href="/uncertainty/2026/08/13/uncertainty-estimation-part-2.html" hidden></a> 362</article> 363
364<script> 365(function () { 366 var content = document.querySelector(".post-content"); 367 var nav = document.querySelector(".post-toc"); 368 if (!content) return; 369 370 var headings = Array.prototype.slice 371 .call(content.querySelectorAll("h2, h3, h4, h5, h6")) 372 .filter(function (heading) { 373 return heading.textContent.trim().length > 0; 374 }); 375 376 if (!headings.length) return; 377 378 function slugify(text) { 379 return text 380 .toLowerCase() 381 .replace(/[^\w\s-]/g, "") 382 .trim() 383 .replace(/\s+/g, "-"); 384 } 385 386 var usedIds = {}; 387 388 // Labels are captured before anchors are appended, so the "##" markers below 389 // never leak into the table of contents. 390 var sections = headings.map(function (heading) { 391 var label = heading.textContent.trim(); 392 393 if (!heading.id) { 394 var base = slugify(label) || "section"; 395 var id = base; 396 var suffix = 1; 397 while (document.getElementById(id) || usedIds[id]) { 398 id = base + "-" + suffix++; 399 } 400 heading.id = id; 401 } 402 usedIds[heading.id] = true; 403 404 return { 405 heading: heading, 406 level: parseInt(heading.tagName.charAt(1), 10), 407 label: label 408 }; 409 }); 410 411 function copyText(text) { 412 if (navigator.clipboard && navigator.clipboard.writeText) { 413 navigator.clipboard.writeText(text); 414 return; 415 } 416 var field = document.createElement("textarea"); 417 field.value = text; 418 field.setAttribute("readonly", ""); 419 field.style.position = "fixed"; 420 field.style.opacity = "0"; 421 document.body.appendChild(field); 422 field.select(); 423 try { 424 document.execCommand("copy"); 425 } catch (err) {} 426 document.body.removeChild(field); 427 } 428 429 var copiedTimer; 430 431 sections.forEach(function (section) { 432 var heading = section.heading; 433 434 var anchor = document.createElement("a"); 435 anchor.className = "heading-anchor"; 436 anchor.href = "#" + heading.id; 437 anchor.setAttribute("aria-label", "Copy link to this section"); 438 anchor.textContent = new Array(section.level + 1).join("+"); 439 440 heading.firstChild.nodeValue = " " + heading.firstChild.nodeValue; 441 heading.insertBefore(anchor, heading.firstChild); 442 heading.classList.add("is-linkable"); 443 444 heading.addEventListener("click", function (event) { 445 if (String(window.getSelection())) return; 446 event.preventDefault(); 447 448 copyText( 449 location.origin + location.pathname + "#" + heading.id 450 ); 451 history.replaceState(null, "", "#" + heading.id); 452 453 window.clearTimeout(copiedTimer); 454 Array.prototype.forEach.call( 455 content.querySelectorAll(".heading-anchor.is-copied"), 456 function (node) { 457 node.classList.remove("is-copied"); 458 } 459 ); 460 anchor.classList.add("is-copied"); 461 copiedTimer = window.setTimeout(function () { 462 anchor.classList.remove("is-copied"); 463 }, 1600); 464 }); 465 }); 466 467 if (!nav || sections.length < 3) return; 468 469 var root = document.createElement("ul"); 470 471 var stack = [{ level: 0, list: root }]; 472 var links = []; 473 474 sections.forEach(function (section) { 475 var heading = section.heading; 476 var level = section.level; 477 478 while (stack.length > 1 && level <= stack[stack.length - 1].level) { 479 stack.pop(); 480 } 481 482 var item = document.createElement("li"); 483 484 var link = document.createElement("a"); 485 link.href = "#" + heading.id; 486 link.textContent = section.label; 487 item.appendChild(link); 488 489 stack[stack.length - 1].list.appendChild(item); 490 491 var sublist = document.createElement("ul"); 492 item.appendChild(sublist); 493 stack.push({ level: level, list: sublist }); 494 495 links.push({ heading: heading, link: link, item: item }); 496 }); 497 498 Array.prototype.forEach.call(root.querySelectorAll("ul"), function (list) { 499 if (!list.children.length) list.parentNode.removeChild(list); 500 }); 501 502 nav.appendChild(root); 503 nav.classList.add("post-toc--ready"); 504 505 var active = null; 506 507 function keepVisible(link) { 508 var navBox = nav.getBoundingClientRect(); 509 var linkBox = link.getBoundingClientRect(); 510 if (linkBox.top < navBox.top) { 511 nav.scrollTop -= navBox.top - linkBox.top + 24; 512 } else if (linkBox.bottom > navBox.bottom) { 513 nav.scrollTop += linkBox.bottom - navBox.bottom + 24; 514 } 515 } 516 517 function highlight() { 518 var current = links[0]; 519 for (var i = 0; i < links.length; i++) { 520 if (links[i].heading.getBoundingClientRect().top > 120) break; 521 current = links[i]; 522 } 523 524 if (current === active) return; 525 active = current; 526 527 Array.prototype.forEach.call( 528 nav.querySelectorAll(".is-active, .is-parent"), 529 function (node) { 530 node.classList.remove("is-active", "is-parent"); 531 } 532 ); 533 534 current.link.classList.add("is-active"); 535 536 var parent = current.item.parentNode; 537 while (parent && parent !== root) { 538 if (parent.tagName === "LI") { 539 var parentLink = parent.querySelector("a"); 540 if (parentLink) parentLink.classList.add("is-parent"); 541 } 542 parent = parent.parentNode; 543 } 544 545 keepVisible(current.link); 546 } 547 548 var ticking = false;
549 function onScroll() { 550 if (ticking) return; 551 ticking = true; 552 window.requestAnimationFrame(function () { 553 ticking = false; 554 highlight(); 555 }); 556 } 557 558 window.addEventListener("scroll", onScroll, { passive: true }); 559 window.addEventListener("resize", onScroll); 560 highlight(); 561})(); 562</script>
562 563
564<script> 565fetch("/assets/views.json") 566 .then((res) => res.json()) 567 .then((data) => { 568 const path = window.location.pathname.replace(/\/$/, ""); 569 const basename = path.split("/").pop(); // ex: "uncertainty-estimation-intro.html", sums up all views ending with this basename 570 571 let total = 0; 572 for (const [k, v] of Object.entries(data)) { 573 if (typeof v === "number" && k.endsWith("/" + basename)) { 574 total += v; 575 } 576 } 577 578 document.getElementById("ga_site_pv").textContent = total.toLocaleString(); 579 }) 580 .catch((err) => { 581 console.error("Failed to load total view count", err); 582 document.getElementById("ga_site_pv").textContent = "N/A"; 583 }); 584 </script>
584 585 </div> 586 </div> 587 </main><footer class="site-footer h-card"> 588 <data class="u-url" href="/"></data> 589 590 <div class="wrapper"> 591 592 <div class="footer-col-wrapper"> 593 <div class="footer-col"> 594 <!-- <p class="feed-subscribe"> 595 <a href="/feed.xml"> 596 <svg class="svg-icon orange"> 597 <use xlink:href="/assets/minima-social-icons.svg#rss"></use> 598 </svg><span>Subscribe</span> 599 </a> 600 </p> --> 601 </div> 602 <div class="footer-col"> 603 <p style="opacity: 0.2">Thoughts on deep learning, tech, and whatever else seems interesting.</p> 604 </div> 605 </div> 606 607 <div class="social-links"><style> 608 .google-scholar-icon { 609 transform: rotate(180deg); 610 transform-origin: center; 611 display: inline-block; 612 } 613</style> 614 615<ul class="social-media-list"> 616 <!--<li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li>--><li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li><!----><li> 617 <a rel="me" href="https://scholar.google.com/citations?user=KMMvTfcAAAAJ%26hl" class="svg-a" title="Google Scholar Profile for KMMvTfcAAAAJ&hl"> 618 <svg class="svg-icon blue google-scholar-icon"> 619 <use xlink:href="/assets/minima-social-icons.svg#google-scholar"></use> 620 </svg> 621 </a> 622 </li></ul> 623 624 625</div> 626 627 </div> 628 629</footer> 630<!-- jQuery first, then Popper.js, then Bootstrap JS. --> 631
631<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
631 632
632<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
632 633
633<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
633 634 635 </body> 636 637</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.