1<html> 2<head> 3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4<title>LTM-NeRF</title> 5<link href="./js/style.css" rel="stylesheet">
6<script type="text/javascript" src="./js/jquery.mlens-1.0.min.js"></script>
vendor: 1 bytes, line 6
6
7<script type="text/javascript" src="./js/jquery.js"></script>
7 8 9 10<style> 11 p.serif{ 12 font-family:"Times New Roman", Times, serif; 13 } 14 p.sansserif{ 15 font-family: Arial, Helvetica, sans-serif; 16 } 17 .text-center { 18 text-align: center; 19} 20</style> 21 22</head> 23 24<body> 25<div class="content"> 26 <h1><strong>LTM-NeRF: Embedding 3D Local Tone Mapping in HDR Neural Radiance Field</strong></h1> 27 <p id="authors" class="serif"> 28 <p id="authors"><a href="https://xhuangcv.github.io/">Xin Huang<sup>1</sup></a> <a href="https://qzhang-cv.github.io/">Qi Zhang<sup>2</sup></a> <a href="https://scholar.google.com/citations?user=PhkrqioAAAAJ&hl=en">Ying Feng<sup>2</sup></a> </p> 29 <p id="authors"><a href="http://users.cecs.anu.edu.au/~hongdong/">Hongdong Li<sup>3</sup></a> <a href="https://teacher.nwpu.edu.cn/qwang.html ">Qing Wang<sup>1</sup></a> <br> 30 <br> 31 <span style="font-size: 20px"><sup>1</sup>Northwestern Polytechnical University 32 </span> 33 <span style="font-size: 20px"> 34 <sup>2</sup>Tencent AI Lab 35 </span> 36 <span style="font-size: 20px"> 37 <sup>3</sup>The Australian National University 38 </span> 39 <br> 40 <br> 41 <span style="font-size: 20px">IEEE T-PAMI 2024</span> 42 </p> 43
44 <div class="row"> 45 <div class="col-full"> 46 <img class="summary-img" src="figs/teaser.png" style="width:100%;"> 47 </div> 48 </div> 49 50 <p style="font-size: 1.2em" class="serif"> 51 Our LTM-NeRF, which incorporates the Camera Response Function (CRF) module and the Neural Exposure Field, 52 collaborates seamlessly with NeRF. Using only (a) LDR views with different exposure settings as the supervision, 53 LTM-NeRF can reconstruct an HDR neural radiance field for HDR view rendering. 54 Furthermore, LTM-NeRF also supports directly producing (b) locally tone-mapped views, 55 or (c) the LDR views (globally tone-mapped using CRF) under a variety of exposure settings.</p> 56 57</div> 58 59<div class="content"> 60 <p style="text-align:center; font-size: 2em;" class="serif">Abstract</p> 61 <p> 62 Recent advances in Neural Radiance Fields (NeRF) have provided a new geometric primitive for novel view synthesis. High Dynamic Range NeRF (HDR NeRF) can render novel views with a higher dynamic range. However, effectively displaying the scene contents of HDR NeRF on diverse devices with limited dynamic range poses a significant challenge. To address this, we present LTM-NeRF, a method designed to recover HDR NeRF and support 3D local tone mapping. LTM-NeRF allows for the synthesis of HDR views, tone-mapped views, and LDR views under different exposure settings, using only the multi-view multi-exposure LDR inputs for supervision. Specifically, we propose a differentiable Camera Response Function (CRF) module for HDR NeRF reconstruction, globally mapping the scene's HDR radiance to LDR pixels. Moreover, we introduce a Neural Exposure Field (NeEF) to represent the spatially varying exposure time of an HDR NeRF to achieve 3D local tone mapping, for compatibility with various displays. Comprehensive experiments demonstrate that our method can not only synthesize HDR views and exposure-varying LDR views accurately but also render locally tone-mapped views naturally. 63 </p> 64</div> 65 66 67<div class="content"> 68 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Methodology</p> 69 <p style="font-size: 1.2em" class="serif"> 70 Overview of LTM-NeRF. The optimization of LTM-NeRF is composed of two stages. 71 In stage one, an HDR NeRF is recovered by integrating a NeRF-based framework with the CRF Network. 72 The first stage is optimized under the supervision of multi-view multi-exposure images. 73 After the optimization, the learned HDR NeRF and CRF are frozen. 74 In stage two, a Neural Exposure Field (NeEF) is introduced to represent the exposure time of the HDR NeRF and achieve 3D tone mapping. 75 To optimize NeEF, we conduct pseudo-ground truth as the supervision. 76 After the whole training, the system can render locally tone-mapped views, HDR views, and LDR views with different exposures.
77 </p> 78 <img class="summary-img" src="figs/pipeline.png" style="width:100%;"> <br> 79 80 81 82 83</div> 84 85<!-- results --> 86<div class="content"> 87 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Results -- LDR Views </p> 88 <p style="font-size: 1.2em" class="serif"> 89 After reconstructing the HDR NeRF from multi-exposure multi-view images, LTM-NeRF can efficiently render controllable LDR views with varying exposures due to the incorporating of the Camera Response Function during the rendering process.<br> 90 </p> 91 92 <div class="row"> 93 <div class="col-full"> 94 <video width="100%" loop controls> 95 <source src="results/main_results/ldr_rendering.mp4" type="video/mp4"> 96 </video> 97 </div> 98 </div> 99 100 101</div> 102 103<div class="content"> 104 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Results -- HDR Views </p> 105 <p style="font-size: 1.2em" class="serif"> 106 LTM-NeRF can render HDR novel views from the HDR NeRF. The HDR views are tone mapped with a 2D Tone Mapping Operation (TMO) for display. In comparison to LDR views, the HDR views contain the scene content of both over-exposure and under-exposure areas. <br> 107 </p> 108
109 <div class="row"> 110 <div class="col-full"> 111 <video width="100%" loop controls> 112 <source src="results/main_results/hdr_rendering.mp4" type="video/mp4"> 113 </video> 114 </div> 115 </div> 116 117 </div> 118 119<div class="content"> 120 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Results -- Locally Tone-mapped Views</p> 121 <p style="font-size: 1.2em" class="serif"> 122 LTM-NeRF can also directly render locally tone-mapped views. Thanks to the introduction of the neural exposure field, 123 LTM-NeRF preserves the consistency of rendered views. The Exposure Maps display the learned exposure time of each 3D point within the 124 HDR radiance field. The brighter the map, the longer the exposure time. The darker regions correspond with a longer exposure time while the brighter regions correspond with a shorter exposure time to achieve an appropriate contrast.<br> 125 </p> 126 127 <div class="row"> 128 <div class="col-full"> 129 <video width="100%" loop controls> 130 <source src="results/main_results/tm_video.mp4" type="video/mp4"> 131 </video> 132 </div> 133 </div> 134 135 </div> 136 137 138<!-- comparisons --> 139<div class="content"> 140 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">
140Comparisons for LDR/HDR View Synthesis</p> 141 <p style="font-size: 1.2em" class="serif"> 142 LTM-NeRF outperforms both NeRF and NeRF-W on novel LDR Views rendering. When compared with NeRF-GT (the upper bound of our method) 143 for novel LDR and HDR view rendering, our method achieves similar performance. 144 </p> 145 <div class="row"> 146 <div class="col-full"> 147 <video width="100%" loop controls> 148 <source src="results/comparisons/ldr_hdr_comparison.mp4" type="video/mp4"> 149 </video> 150 </div> 151 </div> 152</div> 153 154 155<div class="content"> 156 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Comparisons for Locally Tone-mapped View Synthesis</p> 157 <p style="font-size: 1.2em" class="serif"> 158 Compared with post-processing methods that render HDR video from HDR NeRF and then tone map it with 2D image and video methods, LTM-NeRF produces tone-mapped views with more vivid color and improved contrast. Most importantly, our 3D tone mapping method inherently preserves the view consistency. 159 </p> 160 <div class="row"> 161 <div class="col-full"> 162 <video width="100%" loop controls> 163 <source src="results/comparisons/tm_comparison.mp4" type="video/mp4"> 164 </video> 165 </div> 166 </div> 167 168 <p style="font-size: 1.2em" class="serif"> 169 To enhance the view consistency, a potential strategy is to train a NeRF from the images tone-mapped with 2D methods and then render views. 170 However, this strategy also exhibits undesirable luminance changes. The flicker artifacts among the training views are reconstructed because they are interpreted as the view-dependent color by NeRF, though the small flickers are smoothed due to the high-frequency information being difficult for NeRF to overfit. 171 <div class="row"> 172 <div class="col-full"> 173 <video width="100%" loop controls> 174 <source src="results/comparisons/tm_comparison_smooth.mp4" type="video/mp4"> 175 </video> 176 </div> 177 </div> 178 179</div> 180 181 182<div class="content"> 183 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Positions to Exposure Time V.S. Radiance to Exposure Time</p> 184 <p style="font-size: 1.2em" class="serif"> 185 We train NeEF by taking 3D positions as inputs. Another implementation is to learn NeEF from radiance. 186 However, when using radiance as input, high-frequency radiance may result in noisy exposure time and tone-mapped views. 187 Additionally, view-dependent effects and floating artifacts in the radiance may introduce artifacts in NeEF. 188 </p> 189 <div class="row"> 190 <div class="col-full"> 191 <video width="100%" loop controls> 192 <source src="results/comparisons/radiance2time.mp4" type="video/mp4"> 193 </video> 194 </div> 195 </div> 196</div> 197 198<div class="content"> 199 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Comparisons with pseudo-GTs</p> 200 <p style="font-size: 1.2em" class="serif"> 201 We compare our rendered tone-mapped view with the pseudo-ground truths generated by the exposure fusion methods. 202 The pseudo-ground truths appear visible seams and view inconsistency. 203 LTM-NeRF effectively smooths the inconsistency and produces better results without obvious seams and a more natural color. 204 </p> 205 <div class="row"> 206 <div class="col-full"> 207 <video width="100%" loop controls> 208 <source src="results/comparisons/exposure_fusion.mp4" type="video/mp4"> 209 </video> 210 </div> 211 </div> 212 213</div> 214 215<div class="content"> 216 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Comparisons with RawNeRF for Tone-mapped View Synthesis</p> 217 <p style="font-size: 1.2em" class="serif"> 218 RawNeRF is trained from multi-view HDR images without noise, since it is unable to take multi-exposure multi-view LDR images as input. 219 The rendered HDR views are tone-mapped using RawNeRF's tone mapping operation that employs a single gamma curve for the entire view. 220 As shown in the video, RawNeRF's global tone mapping results in a loss of local contrast, though it preserves view consistency. 221 </p> 222 <div class="row"> 223 <div class="col-full"> 224 <video width="100%" loop controls> 225 <source src="results/comparisons/nerfinthedark.mp4" type="video/mp4"> 226 </video> 227 </div> 228 </div> 229 230</div> 231 232 <!-- 3D representation --> 233<div class="content"> 234 <p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">Different 3D Representations</p> 235 <p style="font-size: 1.2em" class="serif"> 236 LTM-NeRF is also compatible with other 3D scene representations, such as Instant-NGP. 237 We showcase the results using Instant-NGP to represent both the radiance and exposure fields.
238 The exposure maps, rendered using the model integrated with Instant-NGP, exhibit superior quality, 239 attributable to the accurate geometry learned by Instant-NGP.<br> 240 </p> 241 242 <div class="row"> 243 <div class="col-gallery"> 244 <video width="100%" loop autoplay muted> 245 <source src="results/main_results/nerf_tm.mp4" type="video/mp4"> 246 </video> 247 <p class="text-center">Tone-mapped Views (NeRF)</p> 248 </div> 249 <div class="col-gallery"> 250 <video width="100%" loop autoplay muted> 251 <source src="results/main_results/nerf_exp.mp4" type="video/mp4"> 252 </video> 253 <p class="text-center">Exposure Maps (NeRF)</p> 254 </div> 255 <div class="col-gallery"> 256 <video width="100%" loop autoplay muted> 257 <source src="results/main_results/ngp_tm.mp4" type="video/mp4"> 258 </video> 259 <p class="text-center" style="font-size: 0.95em">Tone-mapped Views (Instant-NGP)</p> 260 </div> 261 <div class="col-gallery"> 262 <video width="100%" loop autoplay muted> 263 <source src="results/main_results/ngp_exp.mp4" type="video/mp4"> 264 </video> 265 <p class="text-center">Exposure Maps (Instant-NGP)</p> 266 </div> 267 </div> 268 269 270</div> 271 272 273 274</body> 275
276<script> 277var videos = document.getElementsByClassName("clickplay"); 278for (var i = 0; i < videos.length; i++) { 279 videos[i].addEventListener("click", function() { 280 this.play(); 281 }); 282 videos[i].addEventListener("ended", function() { 283 this.pause(); 284 this.currentTime = 0; 285 }); 286} 287 288document.querySelectorAll('.info-container').forEach(function(container) { 289 container.addEventListener('mouseover', function() { 290 var infoText = container.querySelector('.info-text'); 291 infoText.style.display = 'block'; 292 }); 293 294 container.addEventListener('mouseout', function() { 295 var infoText = container.querySelector('.info-text'); 296 infoText.style.display = 'none'; 297 }); 298}); 299</script>
299 300 301</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.