PageSourceSearch

https://norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html

html norange.ai collected 2026-10-03 23:20:29 UTC 43,583 bytes, 637 lines download raw bytes

1<!DOCTYPE html>
2<html lang="en"><head>
3  <meta charset="utf-8">
4  <meta http-equiv="X-UA-Compatible" content="IE=edge">
5  <meta name="viewport" content="width=device-width, initial-scale=1">
6  
6<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
6<!-- Begin Jekyll SEO tag v2.8.0 -->
7<title>Uncertainty in Deep Learning, p.2 (Methods) | No-Range Space</title>
8<meta name="generator" content="Jekyll v3.10.0" />
9<meta property="og:title" content="Uncertainty in Deep Learning, p.2 (Methods)" />
10<meta name="author" content="Nikita Durasov" />
11<meta property="og:locale" content="en_US" />
12<meta name="description" content="In this post, we explores the main methods for estimating uncertainty in deep learning — from Bayesian inference and sampling-based ensembles to efficient deterministic and calibration techniques. We discuss how each approach captures different types of uncertainty, their computational trade-offs, and when to prefer one over another in practice." />
13<meta property="og:description" content="In this post, we explores the main methods for estimating uncertainty in deep learning — from Bayesian inference and sampling-based ensembles to efficient deterministic and calibration techniques. We discuss how each approach captures different types of uncertainty, their computational trade-offs, and when to prefer one over another in practice." />
14<link rel="canonical" href="https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html" />
15<meta property="og:url" content="https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html" />
16<meta property="og:site_name" content="No-Range Space" />
17<meta property="og:type" content="article" />
18<meta property="article:published_time" content="2026-08-13T00:00:00+00:00" />
19<meta name="twitter:card" content="summary" />
20<meta property="twitter:title" content="Uncertainty in Deep Learning, p.2 (Methods)" />
21<script type="application/ld+json">
22{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Nikita Durasov"},"dateModified":"2026-08-13T00:00:00+00:00","datePublished":"2026-08-13T00:00:00+00:00","description":"In this post, we explores the main methods for estimating uncertainty in deep learning — from Bayesian inference and sampling-based ensembles to efficient deterministic and calibration techniques. We discuss how each approach captures different types of uncertainty, their computational trade-offs, and when to prefer one over another in practice.","headline":"Uncertainty in Deep Learning, p.2 (Methods)","mainEntityOfPage":{"@type":"WebPage","@id":"https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html"},"url":"https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html"}</script>
22
23<!-- End Jekyll SEO tag -->
24<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
25  rel="stylesheet">
26  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
27  <link rel="stylesheet" href="/assets/css/style.css">
28  <link rel="shortcut icon" type="image/png" href="/assets/images/logo.png">
29  <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css" rel="stylesheet">
30  <!-- 
30<script async src="https://busuanzi.icodeq.com/busuanzi.pure.mini.js"></script>
30 -->
31  <!-- 
31<script defer src="https://busuanzi.9420.ltd/js"></script>
31 -->
32  <!-- <meta name="referrer" content="no-referrer-when-downgrade"> -->
33  
34  <meta property="og:image" content="/assets/images/logo.png" /><link type="application/atom+xml" rel="alternate" href="https://www.norange.ai/feed.xml" title="No-Range Space" />
vendor: 63 bytes, line 34
34<script async src="https://www.googletagmanager.com/gtag/js?id=
34UA-142853940-1
vendor: 12 bytes, line 34
34"></script>
35<script>
36  window['ga-disable-UA-142853940-1'] = window.doNotTrack === "1" || navigator.doNotTrack === "1" || navigator.doNotTrack === "yes" || navigator.msDoNotTrack === "1";
37  
vendor: 134 bytes, lines 37-41
37window.dataLayer = window.dataLayer || [];
38  function gtag(){dataLayer.push(arguments);}
39  gtag('js', new Date());
40
41  gtag('config', '
41UA-142853940-1
vendor: 4 bytes, line 41
41');
42</script>
42
43<style>
44      .MathJax_Display, .MJXc-display, .MathJax_SVG_Display {
45      overflow-x: auto;
46      overflow-y: hidden;
47  } 
48  </style>
49  <!--
50<meta property="og:image" content="https://www.norange.ai/assets/logo.png" />-->
51</head>
52<body style="font-family: 'Lucida Console', Monaco, monospace"><header class="site-header">
53
54  <div class="wrapper"><div class="row">
55      <div class="col-6 col-sm-3">
56
57        <a rel="author" href="/" class="blog-btn">Blog →</a>
58
59        <style> 
60          .blog-btn { 
61            display: inline-block; 
62            margin: 2px 0;   
63            line-height: 25px;     /* space above & below the button */
64            padding: 3px 10px;    /* space inside the button around the text */
65            font-family: 'Inter', sans-serif; 
66            font-size: 18px; 
67            font-weight: 500; 
68            color: #fff; 
69            text-decoration: none; 
70            border-radius: 8px; 
71            border: 1.5px solid transparent; 
72            background: linear-gradient(#000, #000) padding-box, 
73                        linear-gradient(270deg, #096ce2, #5d41dc) border-box; 
74            transition: all 0.4s ease; 
75            box-shadow: 0 0 12px rgba(131, 56, 236, 0.6); 
76          } 
77
78          .blog-btn:hover { 
79            background: linear-gradient(#000, #000) padding-box, 
80                        linear-gradient(90deg, #FFFFFF, #f285c1) border-box; 
81            box-shadow: 0 0 18px #f285c1; 
82            transform: translateY(-1px); 
83          } 
84
85          .blog-btn:hover::after { 
86            position: relative; 
87            left: 3px; 
88            transition: left 0.0s ease; 
89          }
90
91        </style> 
92
93      </div>
94
95      <div class="col-6 col-sm-9"><nav class="site-nav" style="z-index: 1;">
96            <input type="checkbox" id="nav-trigger" class="nav-trigger" />
97            <label for="nav-trigger">
98              <span class="menu-icon">
99                <svg viewBox="0 0 18 15" width="18px" height="15px">
100                  <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/>
101                </svg>
102              </span>
103            </label>
104
105            <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div>
106          </nav></div>
107    </div>
108
109    <!-- <a class="site-title" rel="author" href="/">No-Range Space</a><nav class="site-nav" style="z-index: 1;">
110        <input type="checkbox" id="nav-trigger" class="nav-trigger" />
111        <label for="nav-trigger">
112          <span class="menu-icon">
113            <svg viewBox="0 0 18 15" width="18px" height="15px">
114              <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/>
115            </svg>
116          </span>
117        </label>
118
119        <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div>
120      </nav>-->
121
122  </div>
123</header>
124<main class="page-content" aria-label="Content">
125      <div class="wrapper">
126        <div style="position: relative; z-index: 0;">
127          
127<script type="text/javascript"
128        src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_CHTML"></script>
vendor: 1 bytes, line 128
128
129<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
vendor: 1 bytes, line 129
129
130<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
130
131
132<nav class="post-toc" aria-label="Table of contents">
133  <p class="post-toc-title">Contents</p>
134</nav>
135
136<article class="post h-entry" itemscope itemtype="http://schema.org/BlogPosting">
137
138  <header class="post-header">
139    <h1 class="post-title p-name" itemprop="name headline"><span class="post-title-gradient">Uncertainty in Deep Learning, p.2 (Methods)</span></h1> 
140    <p class="post-meta" style="margin: 10px 0px">
141      <span class="title-highlights" style="font-size:15px"><time class="dt-published" datetime="2026-08-13T00:00:00+00:00" itemprop="datePublished">
142        Aug 13, 2026
143      </time>• 
144          <span itemprop="author" itemscope itemtype="http://schema.org/Person">
145            <span class="p-author h-card" itemprop="name">Nikita Durasov</span></span><span style="margin-left: 8px;">•</span> <i class="fa-solid fa-eye"></i>
146      <span><span id="ga_site_pv">Loading...</span> views</span>
147      </span>
148    </p>
149  </header>
150
151  <div class="post-content e-content" itemprop="articleBody">
152    \[\begin{array}{l|ccc|ccc}
153 &amp; \text{Train. T.} &amp; \text{Inf. T.} &amp; \text{Mem.} &amp; \text{Integr.} &amp; \text{Regr.} &amp; \text{Class.} \\
154\hline
155\text{Deep Ensembles~[LPB17]} &amp; N\times &amp; N\times &amp; N\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
156\text{MC-Dropout~[GG16]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
157\text{Snapshot Ensembles~[HYL17]} &amp; 1\times &amp; N\times &amp; N\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
158\text{FGE~[GVR18]} &amp; 1\times &amp; N\times &amp; N\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
159\text{Bayes by Backprop~[BBC15]} &amp; 1\times &amp; N\times &amp; 2\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
160\text{BatchEnsemble~[WMW20]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
161\text{Masksembles~[DDG21]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
162\text{Packed-Ensembles~[LLF22]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \checkmark &amp; \checkmark \\
163\text{MIMO~[HSG21]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \checkmark &amp; \checkmark \\
164\text{Iterative Uncertainty~[DDF24]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
165\text{ABNN~[FFM24]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
166\text{Prediction Depth~[BSS21]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
167\text{Subnetwork Inference~[DXW21]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
168\text{BatchNorm Uncertainty~[ABT18]} &amp; 1\times &amp; N\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
169\hline
170\text{Single Model~[KG17]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
171\text{SNGP~[LDL20]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \checkmark &amp; \times &amp; \checkmark \\
172\text{DUQ~[VA20]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \times &amp; \checkmark \\
173\text{DDU~[MGE23]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \times &amp; \checkmark \\
174\text{ZigZag~[DCF24]} &amp; 1\times &amp; 2\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
175\text{Experts Don’t Cheat~[JGL24]} &amp; 1\times &amp; 2\times &amp; 1\times &amp; \checkmark &amp; \times &amp; \checkmark \\
176\text{Evidential Learning~[SEK18]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
177\text{Prior Networks~[MMG18]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \checkmark &amp; \checkmark \\
178\text{Posterior Networks~[CPK20]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \checkmark &amp; \checkmark \\
179\text{Variance Propagation~[PCH19]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \times &amp; \checkmark &amp; \checkmark \\
180\text{Orthonormal Certificates~[TG19]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
181\text{Conformal Predictions~[SW08]} &amp; 1\times &amp; 1\times &amp; 1\times &amp; \checkmark &amp; \checkmark &amp; \checkmark \\
182\end{array}\]
183
184<blockquote>
185  <p><small><strong>Table 1:</strong> <strong>List of popular uncertainty methods for deep learning models.</strong>  The leftmost column lists the uncertainty methods, categorized 
185into sampling-based and sampling-free approaches. Each column represents different aspects of these methods:</small></p>
186</blockquote>
187
188<blockquote>
189  <p><small><strong>(1)</strong> Training time relative to training a single model (“Train. T.”),<br />
190<strong>(2)</strong> Inference time relative to a single model (“Inf. T.”),<br />
191<strong>(3)</strong> Memory requirements for the method (“Mem.”),<br />
192<strong>(4)</strong> A subjective metric indicating the ease of integrating the method into existing architectures and training pipelines (“Integr.”),<br />
193<strong>(5)</strong> Whether the approach supports regression tasks, and<br />
194<strong>(6)</strong> Whether it works with classification tasks.</small></p>
195</blockquote>
196
197<blockquote>
198  <p><small>We round down the values for time and memory metrics unless they are <em>significantly</em> greater than one. For time and memory evaluations, we assume that models are run sequentially (not in parallel) and that the entire model resides in GPU memory throughout the process. The integration metric reflects the complexity of incorporating the method into existing training pipelines or model architectures. Examples of methods with difficult integration include those requiring <em>significant</em> changes to model architectures (e.g., replacing most layers), training pipelines, or additional data.</small></p>
199</blockquote>
200
201<h2 id="uncertainty-estimation-methods">Uncertainty Estimation Methods</h2>
202
203<p>Uncertainty estimation provides a structured way to measure how confident a deep learning model is in its predictions. These techniques play an essential role in handling noisy inputs, supporting decision-making in safety-critical settings, and identifying out-of-distribution (OOD) samples. Broadly, existing methods fall into sampling-based and sampling-free categories. Each class comes with its own advantages and limitations in terms of accuracy, computational overhead, and scalability—topics discussed in the following sections.</p>
204
205<p>As discussed earlier, many uncertainty estimation techniques originate from probabilistic principles, yet most ultimately focus on producing a scalar quantity that reflects confidence in a prediction rather than modelling the full predictive distribution, which is often computationally prohibitive. In a simple 1D regression setting, for example, the model may output both a mean and an associated scale parameter such as a standard deviation [KG17]. Different approaches vary in how these quantities are obtained. The following sections outline how commonly used methods compute uncertainty and summarize them in a high-level comparison table (Table 1). Before introducing these methods, we first establish a more formal view of uncertainty from a Bayesian perspective.</p>
206
207<h3 id="bayesian-approach">Bayesian Approach</h3>
208
209<p>Bayesian methods offer a principled way to incorporate uncertainty into machine learning models by treating model parameters as random variables rather than fixed quantities [BBC15, GG16]. The central idea follows Bayes’ theorem, which connects the posterior distribution of the parameters \(\theta\) given observed data \(D = \{(x_i, y_i)\}_{i=1}^N\) to the prior distribution \(P(\theta)\) and the likelihood:</p>
210
211\[P(\theta \mid D) = \frac{P(\theta) P(D \mid \theta)}{P(D)} 
212= \prod_{i=1}^N \frac{P(\theta) \, P(y_i \mid x_i, \theta)}{\int P(D \mid \theta) P(\theta) \, d\theta}\]
213
214<p>where \(P(\theta \mid D)\) is the posterior distribution, representing the updated belief about the parameters after observing the data, \(P(\theta)\) is the prior, capturing prior knowledge or assumptions about \(\theta\), \(P(D \mid \theta) = \prod_{i=1}^N P(y_i \mid x_i, \theta)\) (assuming the data points are independent and identically distributed) is the likelihood, describing how probable the observed data \(D\) is given the parameters, and \(P(D)\) is the evidence, a normalizing constant ensuring that the posterior is a valid probability distribution. From this formalization, many familiar loss functions and regularization techniques in machine learning can be derived by specifying assumptions about the likelihood \(P(y_i \mid x_i, \theta)\) and the prior \(P(\theta)\). These assumptions determine the model’s behavior and the constraints imposed during training, providing a unified Bayesian perspective on various learning approaches.</p>
215
216<p>In the Bayesian approach, predictions for a new input \(x\) are made by marginalizing over the posterior distribution of the parameters. The predictive distribution is expressed as:</p>
217
218\[P(y \mid x, D) = \int 
219\color{#2563eb}{
220    \underbrace{P(y \mid x, \theta)}_{\text{Aleatoric}} \,
221}
222\color{#16a34a}{
223    \underbrace{P(\theta \mid D)}_{\text{Epistemic}} 
224}
225\, d\theta\qquad(1)\]
226
227<p>This equation highlights the two main sources of uncertainty in Bayesian modeling. The term \(P(y \mid x, \theta)\) represents aleatoric uncertainty, which arises from inherent noise or ambiguity in the data. It reflects the variability in predictions for a fixed model parameter \(\theta\) and is irreducible, even with additional data. In contrast, \(P(\theta \mid D)\) captures epistemic uncertainty, which stems from the model’s lack of knowledge about the true parameter values due to limited or insufficient training data. This uncertainty can be reduced by gathering more diverse or representative data, leading to a more confident posterior.</p>
228
229<p>However, directly computing \(P(\theta \mid D)\) is intractable for complex models like deep neural networks due to the high-dimensional parameter space and the integral required to compute \(P(D)\). To address this, most existing uncertainty estimation methods for deep learning attempt to implicitly or explicitly approximate the Bayesian integral to make it computationally feasible. A classical strategy is to replace analytical computation with sampling. In this context, sampling from \(P(\theta \mid D)\) provides an estimate of the integral, and we refer to approaches based on this idea as sampling-based methods.</p>
230
231<h3 id="sampling-based-methods">Sampling-Based Methods</h3>
232
233<p>Sampling-based methods derive uncertainty estimates by generating multiple predictions from the model and analyzing their variability. These approaches are known for their flexibility and accuracy, particularly in capturing epistemic uncertainty [PCH19]. As mentioned earlier, for high-dimensional real-valued parameters \(\theta\) such as those in neural networks, the integral in Equation 1 becomes intractable and is typically approximated using Monte Carlo samples:</p>
234
235<p>\begin{equation}
236P(y \mid x, \mathcal{D}) \approx \frac{1}{N} \sum_{k=1}^{N} P(y \mid x, \theta^{(k)}); \quad \theta^{(k)} \sim p(\theta \mid \mathcal{D})\qquad(2)
237\label{eq:ensemble_approximation}
238\end{equation}</p>
239
240<p>based on \(N\) different sets of parameters \(\theta^{(k)}\). Different sampling-based methods differ in how they compute these parameters. Some approaches rely on explicit posterior approximations, such as Bayesian Neural Networks [BBC15], while others use implicit sampling techniques like MC-Dropout [GG16] or Deep Ensembles [LPB17]. Although sampling-based approaches are computationally expensive due to the need to train and perform inference with multiple models, they remain the gold standard for both accuracy and uncertainty quality [OHL+19], particularly in the case of Deep Ensembles. While many sampling-free methods discussed later aim to match the performance of sampling-based approaches while reducing computational overhead, they often fall short of achieving the same level of uncertainty quality.</p>
241
242<p><strong><u>Masksembles</u></strong>: Masksembles [DDG21] enhance MC-Dropout by carefully designing structured masks that selectively drop specific features. This approach increases diversity among predictions, resulting in improved uncertainty quantification.</p>
243
244<p><strong><u>BatchEnsembles</u></strong>: BatchEnsembles [WMW20] offer an efficient ensembling approach by representing multiple models within a single network using learnable rank-one perturbations of the weights. This design minimizes memory overhead while preserving diversity in predictions. By sharing parameters across models, BatchEnsembles strike a balance between scalability and uncertainty estimation quality.</p>
245
246<p><strong><u>Subnetwork Inference</u></strong>: Subnetwork Inference [DXW21] is a scalable Bayesian framework that performs posterior inference over a small subset of neural network weights, keeping the rest as point estimates. By performing inference on a selected subnetwork with expressive approximations like full-covariance Gaussians, it aims to balance computational efficiency with accurate uncertainty estimation.</p>
247
248<p><strong><u>Packed-Ensembles</u></strong>
248: Packed-Ensembles [LLF22] propose an efficient ensembling approach that uses grouped convolutions to create multiple independent subnetworks within a single shared architecture. The model is trained similarly to a standard single model but produces multiple predictions, mimicking ensembling while introducing only minor computational overhead.</p>
249
250<p><strong><u>MIMO</u></strong>: The Multiple Input Multiple Output framework (MIMO) [HSG21], along with similar approaches like MixMO [rame2021mixmo], enables efficient ensembling by training independent subnetworks within a single model. During training, \(N\) inputs are processed simultaneously, allowing the model to make \(N\) independent predictions in a single forward pass. At inference, the input is replicated \(N\) times to generate \(N\) predictions, which are ensembled to estimate uncertainty.</p>
251
252<p><strong><u>Iterative Uncertainty</u></strong>: The Iterative Uncertainty framework [DDF24] leverages the iterative refinement process of neural networks to estimate uncertainty without modifying the original architecture. By analyzing the variance in outputs across consecutive iterations, it uses the rate of convergence as a proxy for certainty, where faster convergence indicates higher confidence. A related concept, <strong>Prediction Depth</strong> [BSS21], measures the layer depth at which a model’s prediction stabilizes, with harder examples requiring deeper layers for convergence.</p>
253
254<p><strong><u>ABNN</u></strong>: Adaptable Bayesian Neural Networks (ABNN) [FFM24], along with similar approaches like BatchNorm Uncertainty [ABT18], transform pre-trained DNNs into Bayesian Neural Networks by fine-tuning the model’s normalization layers and introducing noise during inference to approximate sampling.</p>
255
256<p><strong><u>Bayes by Backprop</u></strong>: Bayes by Backprop [BBC15] is a variational inference method that approximates the posterior distribution of model parameters by minimizing a variational objective. Instead of learning point estimates, it learns a distribution over weights, typically using a Gaussian variational posterior.</p>
257
258<h3 id="sampling-free-methods">Sampling-Free Methods</h3>
259
260<p><strong><u>Sampling-free methods</u></strong> estimate uncertainty without generating multiple predictions, offering computational efficiency and scalability. As mentioned earlier, sampling-based approaches are often too costly for real-world applications like robotics and autonomous driving, necessitating faster alternatives. These methods achieve efficiency through architectural modifications, deterministic approximations, or specialized loss functions. Several popular approaches are discussed in the following subsections.</p>
261
262<p><strong><u>Distance-Aware Uncertainty</u></strong>: Recent approaches such as <strong>Deterministic Uncertainty Quantification</strong> (DUQ) [VA20] and <strong>SNGP</strong> [LDL20] focus on single–forward-pass uncertainty estimation by introducing distance-aware output layers, such as radial basis functions (RBFs) in DUQ and Gaussian processes (GPs) in SNGP. DUQ employs a Jacobian penalty to encourage smoothness in the feature space, while SNGP uses spectral normalization to ensure sensitivity and stability in the latent space. Building on these ideas, <strong>Deep Deterministic Uncertainty</strong> (DDU) [MGE23] further extends the framework by directly modeling the feature density in the latent space.</p>
263
264<p><strong><u>Conformal Prediction</u></strong>: Conformal prediction [SW08] is a statistical framework that provides uncertainty estimates with formal coverage guarantees, ensuring that the true label lies within the predicted set with a specified confidence level. It operates by calibrating model outputs on a held-out dataset to produce prediction intervals or sets, making it agnostic to the underlying model architecture. This flexibility allows conformal prediction to be applied across a wide range of tasks, offering a principled and reliable approach to uncertainty quantification in real-world applications.</p>
265
266<p><strong><u>Second-Order Probabilistic Modeling</u></strong>: Second-order probabilistic modeling approaches—such as <strong>prior networks</strong> [MMG18], <strong>posterior networks</strong> [CPK20], and <strong>evidential learning</strong> [SEK18]—capture uncertainty by modeling distributions over predictive probabilities. These methods leverage a factorization of the predictive distribution, as shown in Equation 3, to decompose uncertainty into aleatoric, distributional, and epistemic components.</p>
267
268\[p(y \mid x, \mathcal{D})
269= \iint 
270\underbrace{P(y \mid \pi)}_{\text{Aleatoric}}
271\,
272\underbrace{p(\pi \mid x, \theta)}_{\text{Distributional}}
273\,
274\underbrace{p(\theta \mid \mathcal{D})}_{\text{Epistemic}}
275\, d\pi \, d\theta
276\approx \\
277\approx \int 
278\underbrace{P(y \mid \pi)}_{\text{Aleatoric}}
279\,
280\underbrace{p(\pi \mid x, \hat{\theta})}_{p(\theta \mid \mathcal{D}) \approx \delta(\theta - \hat{\theta})}
281\, d\pi.\qquad(3)\]
282
283<p>Prior networks directly model the distributional uncertainty \(p(\pi \mid x, \hat{\theta})\) by parameterizing a Dirichlet distribution, enabling the network to represent confidence levels tied to predictive uncertainty. Posterior networks extend this by incorporating learned feature-level distributions, combining evidence from the data to produce refined predictions. Evidential learning follows a 
283similar framework, interpreting input data as evidence to parameterize distributions over class probabilities, often using a Dirichlet or related family.</p>
284
285<p><strong><u>Additional Information Approaches</u></strong>: Recent methods leverage auxiliary information to probe model confidence, based on the idea that confident models remain unaffected by additional inputs, while uncertain ones show prediction shifts. ZigZag [DCF24] measures uncertainty by comparing predictions with and without auxiliary information (ground truth during training and model predictions during inference), revealing sensitivity to added data. Similarly, <strong>Experts Don’t Cheat</strong> [JGL24] leverages paired responses, allowing models to “cheat” by using one response to predict the other, thereby exposing uncertain predictions.</p>
286
287<h2 id="conclusion">Conclusion</h2>
288
289<h2 id="references">References</h2>
290
291<p><strong>[LPB17]</strong> B. Lakshminarayanan, A. Pritzel, and C. Blundell. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” In Advances in Neural Information Processing Systems, 2017.</p>
292
293<p><strong>[GG16]</strong> Y. Gal and Z. Ghahramani. “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.” In International Conference on Machine Learning, 2016.</p>
294
295<p><strong>[HYL+17]</strong> G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. “Snapshot Ensembles: Train 1, Get M for Free.” In International Conference on Learning Representations, 2017.</p>
296
297<p><strong>[GVR+18]</strong> T. Garipov, P. Izmailov, D. Podoprikhin, T. Vetrov, and A. G. Wilson. “Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs.” In Advances in Neural Information Processing Systems, 2018.</p>
298
299<p><strong>[BBC15]</strong> C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. “Weight Uncertainty in Neural Networks.” In International Conference on Machine Learning, 2015.</p>
300
301<p><strong>[WMW+20]</strong> Y. Wen, Y. M. Wu, E. M. Moody, and S. L. Lee. “BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning.” In International Conference on Learning Representations, 2020.</p>
302
303<p><strong>[DDG21]</strong> N. Durasov, D. Dvornik, and P. Fua. “Masksembles for Uncertainty Estimation.” In International Conference on Computer Vision, 2021.</p>
304
305<p><strong>[LLF22]</strong> C. Laurent, S. Luneau, and J. Forest. “Packed-Ensembles for Efficient Uncertainty Estimation.” In International Conference on Learning Representations, 2022.</p>
306
307<p><strong>[HSG+21]</strong> D. Havasi, R. Senanayake, J. Liu, G. Dillon, and A. Hartnett. “Training Independent Subnetworks for Robust Prediction.” In International Conference on Learning Representations, 2021.</p>
308
309<p><strong>[DDF24]</strong> N. Durasov, D. Dvornik, and P. Fua. “Iterative Uncertainty Estimation with Self-Consistency.” In Conference on Computer Vision and Pattern Recognition, 2024.</p>
310
311<p><strong>[FFM24]</strong> G. Franchi, S. Franchi, and A. Maki. “Make Uncertainty Great Again: Accounting for Approximation in Bayesian Neural Networks.” In International Conference on Learning Representations, 2024.</p>
312
313<p><strong>[BSS+21]</strong> A. Baldock, C. Shen, A. van den Hengel, and L. Liu. “Deep Regression Ensembles with Prediction Depth.” In Advances in Neural Information Processing Systems, 2021.</p>
314
315<p><strong>[DXW21]</strong> N. Daxberger, J. Kristiadi, S. Sun, A. Immer, R. Eschenhagen, A. Pooch, and P. Hennig. “Bayesian Deep Learning via Subnetwork Inference.” In International Conference on Machine Learning, 2021.</p>
316
317<p><strong>[ABT18]</strong> A. Atanov, A. Ashukha, D. Vetrov, and M. Muhammad. “Uncertainty Estimation via Batch Normalization in Deep Networks.” In International Conference on Learning Representations, 2018.</p>
318
319<p>Single-Model Predictive Uncertainty</p>
320
321<p><strong>[KG17]</strong> A. Kendall and Y. Gal. “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?” In Advances in Neural Information Processing Systems, 2017.</p>
322
323<p><strong>[LDL20]</strong> J. Liu, C. D. Tran, A. Tagasovska, D. J. Sutherland, and I. Davidson. “Simple and Efficient Gaussian Process Priors for Deep Models.” In Advances in Neural Information Processing Systems, 2020.</p>
324
325<p><strong>[VA20]</strong> L. van Amersfoort, M. Smith, M. C. C. Labouriau, N. Pitas, and Y. Gal. “Uncertainty Estimation Using Deep Kernel Learning.” In International Conference on Machine Learning, 2020.</p>
326
327<p><strong>[MGE23]</strong> A. Mukhoti, C. Galesso, R. Guo, G. van Steenkiste, and J. M. Hernández-Lobato. “Deep Deterministic Uncertainty: A Simple Baseline.” In International Conference on Learning Representations, 2023.</p>
328
329<p><strong>[DCF24]</strong> N. Durasov, A. Charpentier, and P. Fua. “ZigZag: Adversarial Inference for OOD Detection.” In International Conference on Learning Representations, 2024.</p>
330
331<p><strong>[JGL24]</strong> A. Johnson, A. Gouk, and T. Lotte. “Experts Don’t Cheat: Reliable Deep Ensembles without Tricks.” In International Conference on Learning Representations, 2024.</p>
332
333<p><strong>[SEK18]</strong> Y. Sensoy, L. Kaplan, and M. Kandemir. “Evidential Deep Learning to Quantify Classification Uncertainty.” In Advances in Neural Information Processing Systems, 2018.</p>
334
335<p>[<strong>MMG18]</strong> A. Malinin and M. Gales. “Predictive Uncertainty Estimation via Prior Networks.” In Advances in Neural Information Processing Systems, 2018.</p>
336
337<p><strong>[CPK20]</strong> A. Charpentier, S. Pionnier, and M. K. Ghassemi. “Posterior Networks: Extracting Uncertainty from Deterministic Neural Networks.” In International Conference on Learning Representations, 2020.</p>
338
339<p><strong>[PCH19]</strong> B. Postels, C. Chitta, and L. Van Gool. “Sampling-Free Epistemic Uncertainty Estimation Using Variance Propagation.” In International Conference on Computer Vision, 2019.</p>
340
341<p><strong>[TG19]</strong> A. Tagasovska and D. Gonzalez. “Orthonormal Certificates for Out-of-Distribution Detection.” In International Conference on Machine Learning, 2019.</p>
342
343<p><strong>[SW08]</strong> G. Shafer and V. Vovk. “A Tutorial on Conformal Prediction.” Journal of Machine Learning Research, 2008.</p>
344
345  </div><div id="disqus_thread"></div>
346  
346<script>
347    var disqus_config = function () {
348      this.page.url = 'https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html';
349      this.page.identifier = 'https://www.norange.ai/uncertainty/2026/08/13/uncertainty-estimation-part-2.html';
350    };
351
352    (function() {
353      var d = document, s = d.createElement('script');
354
355      s.src = 'https://nikitadurasov-github-io.disqus.com/embed.js';
356
357      s.setAttribute('data-timestamp', +new Date());
358      (d.head || d.body).appendChild(s);
359    })();
360  </script>
360
361  <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript><a class="u-url" href="/uncertainty/2026/08/13/uncertainty-estimation-part-2.html" hidden></a>
362</article>
363
364<script>
365(function () {
366  var content = document.querySelector(".post-content");
367  var nav = document.querySelector(".post-toc");
368  if (!content) return;
369
370  var headings = Array.prototype.slice
371    .call(content.querySelectorAll("h2, h3, h4, h5, h6"))
372    .filter(function (heading) {
373      return heading.textContent.trim().length > 0;
374    });
375
376  if (!headings.length) return;
377
378  function slugify(text) {
379    return text
380      .toLowerCase()
381      .replace(/[^\w\s-]/g, "")
382      .trim()
383      .replace(/\s+/g, "-");
384  }
385
386  var usedIds = {};
387
388  // Labels are captured before anchors are appended, so the "##" markers below
389  // never leak into the table of contents.
390  var sections = headings.map(function (heading) {
391    var label = heading.textContent.trim();
392
393    if (!heading.id) {
394      var base = slugify(label) || "section";
395      var id = base;
396      var suffix = 1;
397      while (document.getElementById(id) || usedIds[id]) {
398        id = base + "-" + suffix++;
399      }
400      heading.id = id;
401    }
402    usedIds[heading.id] = true;
403
404    return {
405      heading: heading,
406      level: parseInt(heading.tagName.charAt(1), 10),
407      label: label
408    };
409  });
410
411  function copyText(text) {
412    if (navigator.clipboard && navigator.clipboard.writeText) {
413      navigator.clipboard.writeText(text);
414      return;
415    }
416    var field = document.createElement("textarea");
417    field.value = text;
418    field.setAttribute("readonly", "");
419    field.style.position = "fixed";
420    field.style.opacity = "0";
421    document.body.appendChild(field);
422    field.select();
423    try {
424      document.execCommand("copy");
425    } catch (err) {}
426    document.body.removeChild(field);
427  }
428
429  var copiedTimer;
430
431  sections.forEach(function (section) {
432    var heading = section.heading;
433
434    var anchor = document.createElement("a");
435    anchor.className = "heading-anchor";
436    anchor.href = "#" + heading.id;
437    anchor.setAttribute("aria-label", "Copy link to this section");
438    anchor.textContent = new Array(section.level + 1).join("+");
439
440    heading.firstChild.nodeValue = " " + heading.firstChild.nodeValue;
441    heading.insertBefore(anchor, heading.firstChild);
442    heading.classList.add("is-linkable");
443
444    heading.addEventListener("click", function (event) {
445      if (String(window.getSelection())) return;
446      event.preventDefault();
447
448      copyText(
449        location.origin + location.pathname + "#" + heading.id
450      );
451      history.replaceState(null, "", "#" + heading.id);
452
453      window.clearTimeout(copiedTimer);
454      Array.prototype.forEach.call(
455        content.querySelectorAll(".heading-anchor.is-copied"),
456        function (node) {
457          node.classList.remove("is-copied");
458        }
459      );
460      anchor.classList.add("is-copied");
461      copiedTimer = window.setTimeout(function () {
462        anchor.classList.remove("is-copied");
463      }, 1600);
464    });
465  });
466
467  if (!nav || sections.length < 3) return;
468
469  var root = document.createElement("ul");
470
471  var stack = [{ level: 0, list: root }];
472  var links = [];
473
474  sections.forEach(function (section) {
475    var heading = section.heading;
476    var level = section.level;
477
478    while (stack.length > 1 && level <= stack[stack.length - 1].level) {
479      stack.pop();
480    }
481
482    var item = document.createElement("li");
483
484    var link = document.createElement("a");
485    link.href = "#" + heading.id;
486    link.textContent = section.label;
487    item.appendChild(link);
488
489    stack[stack.length - 1].list.appendChild(item);
490
491    var sublist = document.createElement("ul");
492    item.appendChild(sublist);
493    stack.push({ level: level, list: sublist });
494
495    links.push({ heading: heading, link: link, item: item });
496  });
497
498  Array.prototype.forEach.call(root.querySelectorAll("ul"), function (list) {
499    if (!list.children.length) list.parentNode.removeChild(list);
500  });
501
502  nav.appendChild(root);
503  nav.classList.add("post-toc--ready");
504
505  var active = null;
506
507  function keepVisible(link) {
508    var navBox = nav.getBoundingClientRect();
509    var linkBox = link.getBoundingClientRect();
510    if (linkBox.top < navBox.top) {
511      nav.scrollTop -= navBox.top - linkBox.top + 24;
512    } else if (linkBox.bottom > navBox.bottom) {
513      nav.scrollTop += linkBox.bottom - navBox.bottom + 24;
514    }
515  }
516
517  function highlight() {
518    var current = links[0];
519    for (var i = 0; i < links.length; i++) {
520      if (links[i].heading.getBoundingClientRect().top > 120) break;
521      current = links[i];
522    }
523
524    if (current === active) return;
525    active = current;
526
527    Array.prototype.forEach.call(
528      nav.querySelectorAll(".is-active, .is-parent"),
529      function (node) {
530        node.classList.remove("is-active", "is-parent");
531      }
532    );
533
534    current.link.classList.add("is-active");
535
536    var parent = current.item.parentNode;
537    while (parent && parent !== root) {
538      if (parent.tagName === "LI") {
539        var parentLink = parent.querySelector("a");
540        if (parentLink) parentLink.classList.add("is-parent");
541      }
542      parent = parent.parentNode;
543    }
544
545    keepVisible(current.link);
546  }
547
548  var ticking = false;
549  function onScroll() {
550    if (ticking) return;
551    ticking = true;
552    window.requestAnimationFrame(function () {
553      ticking = false;
554      highlight();
555    });
556  }
557
558  window.addEventListener("scroll", onScroll, { passive: true });
559  window.addEventListener("resize", onScroll);
560  highlight();
561})();
562</script>
562
563
564<script>
565fetch("/assets/views.json")
566  .then((res) => res.json())
567  .then((data) => {
568    const path = window.location.pathname.replace(/\/$/, "");
569    const basename = path.split("/").pop(); // ex: "uncertainty-estimation-intro.html", sums up all views ending with this basename
570
571    let total = 0;
572    for (const [k, v] of Object.entries(data)) {
573      if (typeof v === "number" && k.endsWith("/" + basename)) {
574        total += v;
575      }
576    }
577
578  document.getElementById("ga_site_pv").textContent = total.toLocaleString();
579    })
580    .catch((err) => {
581      console.error("Failed to load total view count", err);
582      document.getElementById("ga_site_pv").textContent = "N/A";
583    });
584  </script>
584
585        </div>
586      </div>
587    </main><footer class="site-footer h-card">
588  <data class="u-url" href="/"></data>
589
590  <div class="wrapper">
591
592    <div class="footer-col-wrapper">
593      <div class="footer-col">
594        <!-- <p class="feed-subscribe">
595          <a href="/feed.xml">
596            <svg class="svg-icon orange">
597              <use xlink:href="/assets/minima-social-icons.svg#rss"></use>
598            </svg><span>Subscribe</span>
599          </a>
600        </p> -->
601      </div>
602      <div class="footer-col">
603        <p style="opacity: 0.2">Thoughts on deep learning, tech, and whatever else seems interesting.</p>
604      </div>
605    </div>
606
607    <div class="social-links"><style>
608  .google-scholar-icon {
609    transform: rotate(180deg);
610    transform-origin: center;
611    display: inline-block;
612  }
613</style>
614
615<ul class="social-media-list">
616  <!--<li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li>--><li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li><!----><li>
617    <a rel="me" href="https://scholar.google.com/citations?user=KMMvTfcAAAAJ%26hl" class="svg-a" title="Google Scholar Profile for KMMvTfcAAAAJ&amp;hl">
618      <svg class="svg-icon blue google-scholar-icon">
619        <use xlink:href="/assets/minima-social-icons.svg#google-scholar"></use>
620      </svg>
621    </a>
622  </li></ul>
623
624
625</div>
626
627  </div>
628
629</footer>
630<!-- jQuery first, then Popper.js, then Bootstrap JS. -->
631    
631<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
631
632    
632<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
632
633    
633<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
633
634
635  </body>
636
637</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.