1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <meta name="description" 6 content="Repurposing Marigold for Zero-Shot Metric Depth 7Estimation via Defocus Blur Cues"> 8 <meta name="keywords" content="metric depth, defocus, marigold, blur"> 9 <meta name="viewport" content="width=device-width, initial-scale=1"> 10 <title>Repurposing Marigold for Zero-Shot Metric Depth 11Estimation via Defocus Blur Cues</title> 12 13 <!-- Global site tag (gtag.js) - Google Analytics --> 14 <!--
vendor: 64 bytes, line 14
14 <script async src="https://www.googletagmanager.com/gtag/js?id=
14G-PYVRSFMDRL
vendor: 12 bytes, line 14
14"></script>
14--> 15
15<script> 16
vendor: 155 bytes, lines 16-24
16window.dataLayer = window.dataLayer || []; 17 18 function gtag() { 19 dataLayer.push(arguments); 20 } 21 22 gtag('js', new Date()); 23 24 gtag('config', '
24G-PYVRSFMDRL
vendor: 6 bytes, lines 24-25
24'); 25
25</script>
25 26 27 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 28 rel="stylesheet"> 29 30 <link rel="stylesheet" href="./static/css/bulma.min.css"> 31 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 32 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 33 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 34 <link rel="stylesheet" 35 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 36 <link rel="stylesheet" href="./static/css/index.css"> 37 <link rel="icon" href="./static/images/favicon.png"> 38 39
39<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
39 40
40<script defer src="./static/js/fontawesome.all.min.js"></script>
40 41
41<script src="./static/js/bulma-carousel.min.js"></script>
41 42
42<script src="./static/js/bulma-slider.min.js"></script>
42 43
43<script src="./static/js/index.js"></script>
43 44</head> 45<body> 46 47 48<section class="hero"> 49 <div class="hero-body"> 50 <div class="container is-max-desktop"> 51 <div class="columns is-centered"> 52 <div class="column has-text-centered"> 53 <h1 class="title is-1 publication-title">Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues</h1> 54 <div class="is-size-5 publication-authors"> 55 <span class="author-block"> 56 <a href="https://chinmay0301.github.io/">Chinmay Talegaonkar</a><sup></sup>,</span> 57 <span class="author-block"> 58 <a href="https://gsnikhil.github.io/">Nikhil Gandudi Suresh</a><sup></sup>,</span> 59 <!-- TODO: Add author links --> 60 <span class="author-block"> 61 <a href="">Zachary Novack</a><sup></sup>, 62 </span> 63 <span class="author-block"> 64 <a href="">Yash Belhe</a><sup></sup>, 65 </span> 66 <span class="author-block"> 67 <a href="">Priyanka Nagasamudra</a><sup></sup>, 68 </span> 69 <span class="author-block"> 70 <a href="https://www.nickantipa.com/">Nicholas Antipa</a><sup></sup> 71 </span> 72 </div> 73 74 <div class="is-size-5 publication-authors"> 75 <span class="author-block"><sup></sup>University of California, San Diego</span> 76 </div> 77 <div class="is-size-5 publication-authors"> 78 <span class="author-block" style="color: red;">NeurIPS 2025 (Spotlight)</span> 79 </div> 80 <!-- TODO: Fix all these links --> 81 <div class="column has-text-centered"> 82 <div class="publication-links"> 83 <!-- PDF Link. --> 84 <span class="link-block"> 85 <a href="https://openreview.net/pdf?id=imO353Gyrl" 86 class="external-link button is-normal is-rounded is-dark"> 87 <span class="icon"> 88 <i class="fas fa-file-pdf"></i> 89 </span> 90 <span>Paper</span> 91 </a> 92 </span> 93 <span class="link-block"> 94 <a href="https://arxiv.org/pdf/2505.17358" 95 class="external-link button is-normal is-rounded is-dark"> 96 <span class="icon"> 97 <i class="ai ai-arxiv"></i> 98 </span> 99 <span>arXiv</span> 100 </a> 101 </span> 102 <!-- Video Link. 103 <span class="link-block"> 104 <a href="" 105 class="external-link button is-normal is-rounded is-dark"> 106 <span class="icon"> 107 <i class="fab fa-youtube"></i> 108 </span> 109 <span>Video</span> 110 </a> 111 </span> --> 112 <!-- Code Link. --> 113 <span class="link-block"> 114 <a href="https://github.com/chinmay0301ucsd/DiffusionCam" 115 class="external-link button is-normal is-rounded is-dark"> 116 <span class="icon"> 117 <i class="fab fa-github"></i> 118 </span> 119 <span>Code</span> 120 </a> 121 </span> 122 <!-- Dataset Link. --> 123 <span class="link-block"> 124 <a href="https://drive.google.com/drive/folders/1EKlT9LLfHVa8dpB2oEFkOUCXVrEuYA5P?usp=sharing" 125 class="external-link button is-normal is-rounded is-dark"> 126 <span class="icon"> 127 <i class="far fa-images"></i> 128 </span> 129 <span>Data</span> 130 </a> 131 </div> 132 133 </div> 134 </div> 135 </div> 136 </div> 137 </div> 138</section> 139 140<section class="section"> 141 <div class="container is-max-desktop"> 142 <!-- Abstract. --> 143 <div class="columns is-centered has-text-centered"> 144 <div class="column is-four-fifths"> 145 <h2 class="title is-3">Abstract</h2> 146 <div class="content has-text-justified"> 147 <p> 148 Recent monocular metric depth estimation (MMDE) methods have made notable progress towards zero-shot generalization. However, they still exhibit a significant performance drop on out-of-distribution datasets. We address this limitation by injecting defocus blur cues at inference time into Marigold, a pre-trained diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-free manner. 149 </p> 150 <p> 151 To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, we then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blur image formation model. We compare our method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements. 152 </p> 153 </div> 154 </div> 155 </div> 156 <!--/ Abstract. --> 157 </div> 158</section> 159 160<section class="hero teaser"> 161 <div class="container is-max-desktop"> 162 <div class="hero-body"> 163 <!-- <video id="teaser" autoplay muted loop playsinline height="100%"> 164 <source src="./static/images/depth_result_finalv2.jpg" 165 type="video/mp4"> 166 </video> --> 167 168 169 <img src="./static/images/synthetic_figure_v1.jpg" alt="Hero Image"> 170 <h2 class="subtitle has-text-centered">
171 <span class="dnerf">Correcting Texture-Depth Coupling: Given an all-in-focus (a) and defocused (b) image of textured, fronto-parallel planes with constant ground truth depth, our method accurately recovers depth maps (c) with low RMSE. Unlike competing approaches, our approach resolves texture-induced errors in the Marigold initialization and recovers both the correct relative depth and metric scale. </span> 172 </h2> 173 174 </div> 175 </div> 176</section> 177 178 179 180 181 182 183<section class="section"> 184 <div class="container is-max-desktop"> 185 <!-- Paper at a Glance. --> 186 <div class="columns is-centered has-text-centered"> 187 <div class="column"> 188 <h2 class="title is-3">Paper at a Glance</h2> 189 <div class="content has-text-justified"> 190 <img src="./static/images/pipeline_figure_v2.jpg" alt="Pipeline Figure"> 191 <p> 192 We capture two images of a scene from the same viewpoint using a camera focused at a fixed distance: a sharp, all-in-focus image (using a high F-stop number) and a defocused image (using a lower F-stop number to introduce blur). Given the sharp image and an initial learnable noise vector representing depth, the Marigold framework estimates a relative depth map. Importantly, Marigold itself is a fixed, training-free frameworkâits architecture and operations do not change during optimization. 193 </p> 194 <p> 195 The estimated relative depth is then converted to metric depth via an affine transformation with learnable parameters. Using the sharp image, the estimated metric depth, and known camera parameters, we synthesize a defocused image using a differentiable forward model of defocus blur. 196 </p> 197 <p> 198 Training is guided by minimizing the L2 loss between the synthesized defocused image and the actual captured defocused image. This loss is backpropagated to update the learnable noise vector and affine transformation parameters, enabling the recovery of scene depth without training the Marigold model itself. 199 </p> 200 </div> 201 </div> 202 </div> 203 <!--/ Paper at a Glance. --> 204 </div> 205</section> 206 207<section class="section"> 208 <div class="container is-max-desktop"> 209 <!-- Results --> 210 <div class="columns is-centered has-text-centered"> 211 <div class="column"> 212 <h2 class="title is-3">Results</h2> 213 <div class="content has-text-justified"> 214 <img src="./static/images/depth_result_finalv2.jpg" alt="Hero Image"> 215 <p class="subtitle has-text-centered"> 216 <span class="dnerf">Our method consistently estimates accurate metric depth across all the scenes </span> 217 </p> 218 <!-- Table --> 219 <table> 220 <thead> 221 <tr> 222 <th>Method</th> 223 <th>RMSE â</th> 224 <th>REL â</th> 225 <th>log10 â</th> 226 <th>δâ â</th> 227 <th>δâ â</th> 228 <th>δâ â</th> 229 <th>CD â</th> 230 <th>FA â</th> 231 </tr> 232 </thead> 233 <tbody> 234 <tr> 235 <td>MLPro</td> 236 <td>0.468</td> 237 <td>0.246</td> 238 <td>0.105</td> 239 <td>0.597</td> 240 <td>0.821</td> 241 <td>0.990</td> 242 <td>0.205</td> 243 <td>0.696</td> 244 </tr> 245 <tr> 246 <td>UniDepth</td> 247 <td>0.574</td> 248 <td>0.358</td> 249 <td>0.152</td> 250 <td>0.263</td> 251 <td>0.757</td> 252 <td>0.902</td> 253 <td>0.260</td> 254 <td>0.612</td> 255 </tr> 256 <tr> 257 <td>Metric3D</td> 258 <td>0.349</td> 259 <td>0.195</td> 260 <td>0.087</td> 261 <td>0.611</td> 262 <td>0.958</td> 263 <td>0.983</td> 264 <td>0.135</td> 265 <td>0.814</td> 266 </tr> 267 <tr> 268 <td>Ours - Gaussian</td> 269 <td>0.528</td> 270 <td>0.279</td> 271 <td>0.142</td> 272 <td>0.422</td> 273 <td>0.695</td> 274 <td>0.928</td> 275 <td>0.241</td> 276 <td>0.652</td> 277 </tr> 278 <tr> 279 <td class="bold">Ours - Disc</td> 280 <td class="bold">0.273</td> 281 <td class="bold">0.125</td> 282 <td class="bold">0.052</td> 283 <td class="bold">0.879</td> 284 <td class="bold">0.975</td> 285 <td class="bold">0.991</td> 286 <td class="bold">0.103</td> 287 <td class="bold">0.870</td> 288 </tr> 289 </tbody> 290 <caption><strong>Table 1:</strong> Our method with the Disc PSF outperforms all MMDE baselines averaged over all scenes in our dataset. The disc PSF, being more consistent with the real camera PSF, also outperforms the Gaussian PSF.</caption> 291 </table> 292 <!--/ Table --> 293 </div> 294 </div> 295 </div> 296 <!--/ Results --> 297 </div> 298</section> 299 300<section class="section"> 301 <div class="container is-max-desktop"> 302 <!-- Results --> 303 <div class="columns is-centered has-text-centered"> 304 <div class="column"> 305 <h2 class="title is-3">Dataset</h2> 306 <div class="content has-text-justified"> 307 <p> 308 The dataset comprises seven diverse real-world indoor scenes captured at multiple defocus blur levels. RGB images were acquired using a Canon EOS 5D Mark II with a 21MP sensor (5616 à 3744 resolution, 6.41â¯Î¼m pixel pitch) and an RGGB Bayer pattern. A 50mm Canon lens was used, with F-stop settings ranging from f/1.4 to f/22, and the focus distance fixed at 80â¯cm. 309 310 Ground truth depth was obtained using an Intel RealSense D435 stereo depth camera, which offers less than 2% depth error at 2 meters. To reduce noise such as flying pixels, each depth map is averaged over 60 frames. The dataset can be found at this <a href="https://drive.google.com/drive/folders/1EKlT9LLfHVa8dpB2oEFkOUCXVrEuYA5P?usp=sharing">link</a>. Please see the Readme in the dataset as well. 311 </p> 312 313 314 </div> 315 </div> 316 </div> 317 <!--/ Results --> 318 </div> 319</section> 320 321 322 323<!-- TODO: Check if the below details are correct --> 324<section class="section" id="BibTeX">
325 <div class="container is-max-desktop content"> 326 <h2 class="title">BibTeX</h2> 327 <pre><code>@misc{talegaonkar2025repurposingmarigoldzeroshotmetric, 328 title={Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues}, 329 author={Chinmay Talegaonkar and Nikhil Gandudi Suresh and Zachary Novack and Yash Belhe and Priyanka Nagasamudra and Nicholas Antipa}, 330 year={2025}, 331 eprint={2505.17358}, 332 archivePrefix={arXiv}, 333 primaryClass={cs.CV}, 334 url={https://arxiv.org/abs/2505.17358}, 335 336}</code></pre> 337 </div> 338</section> 339 340 341<footer class="footer"> 342 <div class="container"> 343 <div class="content has-text-centered"> 344 <!-- TODO: fix paper link --> 345 <a class="icon-link" 346 href=""> 347 <i class="fas fa-file-pdf"></i> 348 </a> 349 <!-- TODO: add code link --> 350 <a class="icon-link" href="" class="external-link" disabled> 351 <i class="fab fa-github"></i> 352 </a> 353 </div> 354 <div class="columns is-centered"> 355 <div class="column is-8"> 356 <div class="content"> 357 <p> 358 This website is licensed under a <a rel="license" 359 href="http://creativecommons.org/licenses/by-sa/4.0/">Creative 360 Commons Attribution-ShareAlike 4.0 International License</a>. 361 </p> 362 <p> 363 Template borrowed from <a 364 href="https://github.com/nerfies/nerfies.github.io">nerfies</a> website. 365 </p> 366 </div> 367 </div> 368 </div> 369 </div> 370</footer> 371 372</body> 373</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.